arvados.git
7 years ago9767: Update 'Run a pipeline' button in a project page to include workflows as well. 9767-workflows-in-chooser
radhika [Thu, 25 Aug 2016 14:59:41 +0000 (10:59 -0400)]
9767: Update 'Run a pipeline' button in a project page to include workflows as well.

7 years ago9767: update test to match preview pane update
radhika [Wed, 24 Aug 2016 18:39:48 +0000 (14:39 -0400)]
9767: update test to match preview pane update

7 years ago9767: set name of the container_request based on the workflow name.
radhika [Wed, 24 Aug 2016 18:31:55 +0000 (14:31 -0400)]
9767: set name of the container_request based on the workflow name.

7 years ago9767: add safe_yaml gem to workbench and update load_config
radhika [Wed, 24 Aug 2016 18:19:03 +0000 (14:19 -0400)]
9767: add safe_yaml gem to workbench and update load_config

7 years agoMerge branch 'master' into 9767-workflows-in-chooser
radhika [Wed, 24 Aug 2016 18:13:07 +0000 (14:13 -0400)]
Merge branch 'master' into 9767-workflows-in-chooser

7 years ago9767: json parse not needed after yaml load
radhika [Wed, 24 Aug 2016 17:47:46 +0000 (13:47 -0400)]
9767: json parse not needed after yaml load

7 years ago9767: need to parse the json string of the workflow yaml
radhika [Wed, 24 Aug 2016 11:23:44 +0000 (07:23 -0400)]
9767: need to parse the json string of the workflow yaml

7 years ago9767: use a workflow with inputs in test
radhika [Tue, 23 Aug 2016 23:18:01 +0000 (19:18 -0400)]
9767: use a workflow with inputs in test

7 years agoMerge branch '9804-drop-activity-pane'
Lucas Di Pentima [Tue, 23 Aug 2016 22:41:05 +0000 (19:41 -0300)]
Merge branch '9804-drop-activity-pane'

9804: Deleted activity pane from Workbench's collection view, a couple of queries that became unused on collection controller and updated the related tests. Closes #9804.

7 years ago9767: corrected workflow.json mount
radhika [Tue, 23 Aug 2016 21:41:30 +0000 (17:41 -0400)]
9767: corrected workflow.json mount

7 years ago9804: Tweaked the collection#show's pane widths. Fixed some integration_performance...
Lucas Di Pentima [Tue, 23 Aug 2016 20:36:34 +0000 (17:36 -0300)]
9804: Tweaked the collection#show's pane widths. Fixed some integration_performance tests that were failing after the activity pane removal and removed an unused query at collection controller.

7 years ago9767: add test to run a workflow from dashboard
radhika [Tue, 23 Aug 2016 20:12:34 +0000 (16:12 -0400)]
9767: add test to run a workflow from dashboard

7 years agoMerge branch '9706-package-systemd-files'
Tom Clegg [Tue, 23 Aug 2016 17:29:04 +0000 (13:29 -0400)]
Merge branch '9706-package-systemd-files'

refs #9706
refs #9745

7 years ago9706: Use ${} interpolation syntax consistently.
Tom Clegg [Mon, 22 Aug 2016 19:18:30 +0000 (15:18 -0400)]
9706: Use ${} interpolation syntax consistently.

7 years ago9706: 9745: Fix config struct to match example.
Tom Clegg [Tue, 16 Aug 2016 01:35:35 +0000 (21:35 -0400)]
9706: 9745: Fix config struct to match example.

7 years ago9706: Notify systemd when crunch-dispatch-slurm is ready.
Tom Clegg [Mon, 15 Aug 2016 20:58:02 +0000 (16:58 -0400)]
9706: Notify systemd when crunch-dispatch-slurm is ready.

7 years ago9706: Add systemd unit file for crunch-dispatch-slurm.
Tom Clegg [Mon, 15 Aug 2016 20:10:58 +0000 (16:10 -0400)]
9706: Add systemd unit file for crunch-dispatch-slurm.

7 years ago9706: Stop systemd services when their packages are uninstalled.
Tom Clegg [Mon, 15 Aug 2016 20:10:29 +0000 (16:10 -0400)]
9706: Stop systemd services when their packages are uninstalled.

7 years ago9706: Go service postinst improvements.
Brett Smith [Wed, 10 Aug 2016 17:22:02 +0000 (13:22 -0400)]
9706: Go service postinst improvements.

* Use /bin/sh, per Debian policy.
* Prefer basename for readability.
* Check for systemd init with [ -e /run/systemd/system ].
* Make configuration changes based on checking the unit file state.

7 years ago9706: Add postinst script to Go packages that have systemd unit files, so (on Debian...
Tom Clegg [Thu, 4 Aug 2016 22:43:17 +0000 (18:43 -0400)]
9706: Add postinst script to Go packages that have systemd unit files, so (on Debian) the service starts up without manual intervention.

7 years ago9706: If "foo.service" exists in the source dir for a Go program, package it so it...
Tom Clegg [Wed, 3 Aug 2016 21:06:55 +0000 (17:06 -0400)]
9706: If "foo.service" exists in the source dir for a Go program, package it so it installs to /lib/systemd/system/.

7 years agoMerge branch '9827-no-docker-caching' closes #9827
Peter Amstutz [Tue, 23 Aug 2016 13:27:28 +0000 (09:27 -0400)]
Merge branch '9827-no-docker-caching' closes #9827

7 years agoMerge branch '9773-faster-find-or-create'
Tom Clegg [Mon, 22 Aug 2016 20:39:05 +0000 (16:39 -0400)]
Merge branch '9773-faster-find-or-create'

refs #9773

7 years ago9773: If previous jobs disagree about outputs, but a new job is
Tom Clegg [Mon, 22 Aug 2016 20:25:14 +0000 (16:25 -0400)]
9773: If previous jobs disagree about outputs, but a new job is
already queued/running, reuse the new job (as we did before 9773).

7 years ago9773: Exclude script_parameters_digest from searchable columns -- it is not even...
Tom Clegg [Sat, 20 Aug 2016 22:20:29 +0000 (18:20 -0400)]
9773: Exclude script_parameters_digest from searchable columns -- it is not even visible to clients.

7 years ago9773: Use script_parameters_digest in reusable job query.
Tom Clegg [Fri, 19 Aug 2016 21:03:24 +0000 (17:03 -0400)]
9773: Use script_parameters_digest in reusable job query.

7 years ago9773: Add script_parameters_digest column to support faster job reuse.
Tom Clegg [Fri, 19 Aug 2016 21:02:03 +0000 (17:02 -0400)]
9773: Add script_parameters_digest column to support faster job reuse.

7 years ago9773: Fix up find-or-create-job code.
Tom Clegg [Fri, 19 Aug 2016 19:00:09 +0000 (15:00 -0400)]
9773: Fix up find-or-create-job code.

* Reduce pyramid of conditions

* Ask the database to filter script_parameters, instead of doing that in Ruby

* Ditto job state

* Check "output is readable?" only once, not once per candidate job

* If two matching jobs have different outputs, avoid reusing either of them,
  even if one of the outputs is not readable by current_user.

7 years agoMerge branch '9818-logstash'
Tom Clegg [Mon, 22 Aug 2016 17:06:58 +0000 (13:06 -0400)]
Merge branch '9818-logstash'

refs #9818
refs #9709

7 years ago9818: Log params when small, first 1K of json-encoded params when big.
Tom Clegg [Thu, 18 Aug 2016 20:47:19 +0000 (16:47 -0400)]
9818: Log params when small, first 1K of json-encoded params when big.

7 years ago9818: Use lograge/logstash for Workbench request logs.
Tom Clegg [Thu, 18 Aug 2016 19:18:45 +0000 (15:18 -0400)]
9818: Use lograge/logstash for Workbench request logs.

7 years ago9818: Use lograge to log API requests in logstash format.
Tom Clegg [Thu, 18 Aug 2016 19:12:03 +0000 (15:12 -0400)]
9818: Use lograge to log API requests in logstash format.

Example log line:

{"method":"GET","path":"/arvados/v1/keep_services/accessible","format":"html","controller":"arvados/v1/keep_services","action":"accessible","status":200,"duration":11.07,"view":0.36,"db":2.63,"@timestamp":"2016-08-18T19:04:17Z","@version":"1","message":"[200] GET /arvados/v1/keep_services/accessible (arvados/v1/keep_services#accessible)"}

7 years ago9804: Removed the "Activity" pane from Collection View @ Workbench. Also removed...
Lucas Di Pentima [Mon, 22 Aug 2016 14:05:00 +0000 (11:05 -0300)]
9804: Removed the "Activity" pane from Collection View @ Workbench. Also removed a related query on the controller's show method, and a test.

7 years agoMerge branch '9826-config-symbols'
Tom Clegg [Mon, 22 Aug 2016 13:42:44 +0000 (09:42 -0400)]
Merge branch '9826-config-symbols'

closes #9826

7 years ago9826: Deserialize symbols when loading server config YAML.
Tom Clegg [Mon, 22 Aug 2016 13:13:15 +0000 (09:13 -0400)]
9826: Deserialize symbols when loading server config YAML.

7 years agoAdd --compute-checksum to sdk/cwl/test_with_arvbox.sh refs #9552
Peter Amstutz [Mon, 22 Aug 2016 02:53:32 +0000 (22:53 -0400)]
Add --compute-checksum to sdk/cwl/test_with_arvbox.sh refs #9552

7 years ago9827: Use "arvbox rebuild" to ensure up to date arvados/arvbox-demo image.
Peter Amstutz [Mon, 22 Aug 2016 02:13:55 +0000 (22:13 -0400)]
9827: Use "arvbox rebuild" to ensure up to date arvados/arvbox-demo image.

7 years ago9767: create a pipeline instance or container request depending on the template selected.
radhika [Fri, 19 Aug 2016 21:06:41 +0000 (17:06 -0400)]
9767: create a pipeline instance or container request depending on the template selected.

7 years ago9767: include workflows in the template chooser.
radhika [Fri, 19 Aug 2016 17:54:19 +0000 (13:54 -0400)]
9767: include workflows in the template chooser.

7 years agoMerge branch '9820-cwl-poll-jobs' closes #9820
Peter Amstutz [Fri, 19 Aug 2016 15:13:51 +0000 (11:13 -0400)]
Merge branch '9820-cwl-poll-jobs' closes #9820

7 years ago9820: Directly poll job or container records that we are interested in. Benefit:...
Peter Amstutz [Fri, 19 Aug 2016 14:50:25 +0000 (10:50 -0400)]
9820: Directly poll job or container records that we are interested in. Benefit: puts less load on database than log table polling, and doesn't miss events.

7 years agoMerge branch '9684-workflows'
Tom Clegg [Fri, 19 Aug 2016 00:27:37 +0000 (20:27 -0400)]
Merge branch '9684-workflows'

refs #9684

7 years ago9684: Style nitpicks.
Tom Clegg [Fri, 19 Aug 2016 00:12:25 +0000 (20:12 -0400)]
9684: Style nitpicks.

7 years ago9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"
radhika [Thu, 18 Aug 2016 18:22:59 +0000 (14:22 -0400)]
9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"

7 years agoMerge branch '9333-api-server-invalid-sql'
Lucas Di Pentima [Thu, 18 Aug 2016 17:56:43 +0000 (14:56 -0300)]
Merge branch '9333-api-server-invalid-sql'

9333: Fixes to solve workbench integration tests failures. Refs #9333

7 years agoMerge branch 'master' into 9333-api-server-invalid-sql
Lucas Di Pentima [Thu, 18 Aug 2016 17:34:38 +0000 (14:34 -0300)]
Merge branch 'master' into 9333-api-server-invalid-sql

7 years ago9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"
radhika [Thu, 18 Aug 2016 16:47:50 +0000 (12:47 -0400)]
9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"

7 years ago9333: Adding collection model specific sttributes to be ignored on the validation...
Lucas Di Pentima [Thu, 18 Aug 2016 16:06:01 +0000 (13:06 -0300)]
9333: Adding collection model specific sttributes to be ignored on the validation mechanism to fix workbench integration test failures

7 years ago9684: Add safe_yaml initializer
radhika [Thu, 18 Aug 2016 13:43:23 +0000 (09:43 -0400)]
9684: Add safe_yaml initializer

7 years agocloses #9684
radhika [Thu, 18 Aug 2016 11:51:21 +0000 (07:51 -0400)]
closes #9684
Merge branch '9684-workflows'

7 years agoMerge branch '9333-api-server-invalid-sql'
Lucas Di Pentima [Wed, 17 Aug 2016 16:44:41 +0000 (13:44 -0300)]
Merge branch '9333-api-server-invalid-sql'

9333: Selected attributes list validation on the API Server, avoiding invalid SQL queries to the database. Closes #9333

7 years ago9333: Removed superfluous quote marks on exception message
Lucas Di Pentima [Wed, 17 Aug 2016 16:39:24 +0000 (13:39 -0300)]
9333: Removed superfluous quote marks on exception message

7 years ago9333: Treating the "attribute is not a string" error case the same as the other inval...
Lucas Di Pentima [Tue, 16 Aug 2016 21:22:04 +0000 (18:22 -0300)]
9333: Treating the "attribute is not a string" error case the same as the other invalid attributes. Moved the checks to ArvadosModel. Updated tests.

7 years ago9684: workflow yaml error logging
radhika [Tue, 16 Aug 2016 21:09:33 +0000 (17:09 -0400)]
9684: workflow yaml error logging

7 years ago9678: use safe_yaml and other updates.
radhika [Mon, 15 Aug 2016 22:19:00 +0000 (18:19 -0400)]
9678: use safe_yaml and other updates.

7 years agocloses #9709
radhika [Mon, 15 Aug 2016 14:26:44 +0000 (10:26 -0400)]
closes #9709
Merge branch '9709-dont-log-manifest-texts'

7 years ago9333: Attribute validation on "select" to avoid invalid SQL statements
Lucas Di Pentima [Fri, 12 Aug 2016 00:38:46 +0000 (21:38 -0300)]
9333: Attribute validation on "select" to avoid invalid SQL statements

7 years agoFix failing tests. refs #9776
Peter Amstutz [Thu, 11 Aug 2016 22:18:13 +0000 (18:18 -0400)]
Fix failing tests. refs #9776

7 years agoFix schema salad package build version. no issue #
Peter Amstutz [Thu, 11 Aug 2016 21:50:49 +0000 (17:50 -0400)]
Fix schema salad package build version.  no issue #

7 years agoMerge branch '9776-cwl-HOME-TMPDIR' closes #9776
Peter Amstutz [Thu, 11 Aug 2016 21:02:15 +0000 (17:02 -0400)]
Merge branch '9776-cwl-HOME-TMPDIR' closes #9776

7 years ago9776: Ensure that $HOME and $TMPDIR are set correctly.
Peter Amstutz [Thu, 11 Aug 2016 20:55:10 +0000 (16:55 -0400)]
9776: Ensure that $HOME and $TMPDIR are set correctly.

7 years agoMerge branch '9615-cwl-writable-unsupported' refs #9615
Peter Amstutz [Thu, 11 Aug 2016 20:33:06 +0000 (16:33 -0400)]
Merge branch '9615-cwl-writable-unsupported' refs #9615

7 years ago9615: Add check_writable to check for "writable" field and raise UnsupportedRequirement.
Peter Amstutz [Thu, 4 Aug 2016 15:36:56 +0000 (11:36 -0400)]
9615: Add check_writable to check for "writable" field and raise UnsupportedRequirement.

7 years agoMerge branch '9751-cwl-keep-dir' closes #9751
Peter Amstutz [Thu, 11 Aug 2016 20:30:34 +0000 (16:30 -0400)]
Merge branch '9751-cwl-keep-dir' closes #9751

7 years ago9552: import compute_checksums from cwltool.
Peter Amstutz [Thu, 11 Aug 2016 18:54:36 +0000 (14:54 -0400)]
9552: import compute_checksums from cwltool.

7 years ago9751: Fix handling local Directories
Peter Amstutz [Thu, 11 Aug 2016 19:35:15 +0000 (15:35 -0400)]
9751: Fix handling local Directories

7 years ago9751: Bump cwltool dependency.
Peter Amstutz [Thu, 11 Aug 2016 18:46:24 +0000 (14:46 -0400)]
9751: Bump cwltool dependency.

7 years ago9751: Fix unit tests. Rename dir6.cwl to keep-dir-test-input.cwl
Peter Amstutz [Thu, 11 Aug 2016 18:19:37 +0000 (14:19 -0400)]
9751: Fix unit tests.  Rename dir6.cwl to keep-dir-test-input.cwl

7 years ago9751: Override realpath in CollectionFsAccess
Peter Amstutz [Thu, 11 Aug 2016 17:01:41 +0000 (13:01 -0400)]
9751: Override realpath in CollectionFsAccess

7 years ago9751: Fix direct to keep directory references.
Peter Amstutz [Wed, 10 Aug 2016 17:45:56 +0000 (13:45 -0400)]
9751: Fix direct to keep directory references.
Add arvados-specific integration test for arvados-cwl-runner.

7 years ago9751: Provide make_fs_access=CollectionFsAccess and bump cwltool pin.
Peter Amstutz [Wed, 10 Aug 2016 15:51:44 +0000 (11:51 -0400)]
9751: Provide make_fs_access=CollectionFsAccess and bump cwltool pin.

7 years agocloses #9678
radhika [Thu, 11 Aug 2016 18:13:36 +0000 (14:13 -0400)]
closes #9678
Merge branch '9678-container-request-log-tab'

7 years ago9709: do not include manifest_text in collection logs.
radhika [Thu, 11 Aug 2016 18:10:00 +0000 (14:10 -0400)]
9709: do not include manifest_text in collection logs.

7 years agocloses #9658
radhika [Thu, 11 Aug 2016 13:50:36 +0000 (09:50 -0400)]
closes #9658
Merge branch '9658-cwl-runner-container'

7 years ago9678: enhance test to verify that the combined logs are displayed.
radhika [Wed, 10 Aug 2016 18:41:29 +0000 (14:41 -0400)]
9678: enhance test to verify that the combined logs are displayed.

7 years ago9678: enhance container log tab display to consider the "combined" object logs.
radhika [Wed, 10 Aug 2016 18:05:02 +0000 (14:05 -0400)]
9678: enhance container log tab display to consider the "combined" object logs.

7 years ago9658: when arvados-cwl-runner is invoked with --api containers, display the container...
radhika [Wed, 10 Aug 2016 16:29:31 +0000 (12:29 -0400)]
9658: when arvados-cwl-runner is invoked with --api containers, display the container_request uuid to console.

7 years agocloses #9745
radhika [Wed, 10 Aug 2016 15:58:22 +0000 (11:58 -0400)]
closes #9745
Merge branch '9745-cds-api-config'

7 years ago9684: add workflow resource to api server
radhika [Tue, 9 Aug 2016 19:51:50 +0000 (15:51 -0400)]
9684: add workflow resource to api server

7 years ago9745: Update example config.
Tom Clegg [Fri, 5 Aug 2016 15:45:55 +0000 (11:45 -0400)]
9745: Update example config.

7 years ago9745: Load API host and token from crunch-dispatch-slurm config file.
Tom Clegg [Fri, 5 Aug 2016 15:44:29 +0000 (11:44 -0400)]
9745: Load API host and token from crunch-dispatch-slurm config file.

7 years ago9706: Add example config file.
Tom Clegg [Fri, 5 Aug 2016 14:54:37 +0000 (10:54 -0400)]
9706: Add example config file.

No issue #

7 years agogofmt.
Tom Clegg [Thu, 4 Aug 2016 22:16:24 +0000 (18:16 -0400)]
gofmt.

No issue #

7 years agoFix race in crunchstat.
Tom Clegg [Thu, 4 Aug 2016 21:45:10 +0000 (17:45 -0400)]
Fix race in crunchstat.

To fulfill the "nothing more will be logged after Stop() returns"
guarantee, Stop() needs to wait for run() to finish.

Should fix https://ci.curoverse.com/job/run-tests-remainder/38/consoleText

No issue #

7 years agoSimplify write flusher using a time.Ticker.
Tom Clegg [Thu, 4 Aug 2016 21:44:34 +0000 (17:44 -0400)]
Simplify write flusher using a time.Ticker.

No issue #

7 years agoFix potential race in ThrottledLogger flusher.
Tom Clegg [Thu, 4 Aug 2016 20:53:32 +0000 (16:53 -0400)]
Fix potential race in ThrottledLogger flusher.

No issue #

7 years agoMerge branch '9552-compute-checksum-final-output' refs #9552
Peter Amstutz [Thu, 4 Aug 2016 16:01:37 +0000 (12:01 -0400)]
Merge branch '9552-compute-checksum-final-output' refs #9552

7 years agoMerge branch '9406-cgroup-parent'
Tom Clegg [Thu, 4 Aug 2016 15:31:44 +0000 (11:31 -0400)]
Merge branch '9406-cgroup-parent'

closes #9406

7 years ago9406: Explain why there are two separate CgroupParent fields.
Tom Clegg [Thu, 4 Aug 2016 15:31:34 +0000 (11:31 -0400)]
9406: Explain why there are two separate CgroupParent fields.

7 years ago9552: If compute_checksum is true, check if checksum needs to be computed on final...
Peter Amstutz [Thu, 4 Aug 2016 14:23:12 +0000 (10:23 -0400)]
9552: If compute_checksum is true, check if checksum needs to be computed on final output, and add it if missing.

7 years agoMerge branch '9595-mount-json'
Tom Clegg [Thu, 4 Aug 2016 14:10:12 +0000 (10:10 -0400)]
Merge branch '9595-mount-json'

closes #9595

7 years ago9595: Fix hardcoded temp dir in test. Add checks for CleanupDirs().
Tom Clegg [Thu, 4 Aug 2016 06:26:54 +0000 (02:26 -0400)]
9595: Fix hardcoded temp dir in test. Add checks for CleanupDirs().

7 years ago9595: Support "json" mount type.
Tom Clegg [Thu, 4 Aug 2016 05:48:33 +0000 (01:48 -0400)]
9595: Support "json" mount type.

7 years agoMerge branch '9677-update-install-docs'
Ward Vandewege [Thu, 4 Aug 2016 00:48:32 +0000 (20:48 -0400)]
Merge branch '9677-update-install-docs'

closes #9677

7 years agoDocument how to download and tag the arvados/jobs Docker image.
Ward Vandewege [Wed, 3 Aug 2016 19:54:34 +0000 (15:54 -0400)]
Document how to download and tag the arvados/jobs Docker image.

refs #9677

7 years agoMerge branch '9674-cwl-initialworkdir' closes #9674
Peter Amstutz [Wed, 3 Aug 2016 20:47:05 +0000 (16:47 -0400)]
Merge branch '9674-cwl-initialworkdir' closes #9674

7 years agoMerge branch '9617-reject-container-reqs-without-vcpus'
Lucas Di Pentima [Tue, 2 Aug 2016 23:29:04 +0000 (20:29 -0300)]
Merge branch '9617-reject-container-reqs-without-vcpus'

9617, 9618: Added runtime constraints to fixtures.

refs #9617, #9618

7 years ago9617, 9618: Added runtime constraints to container requests fixtures. Wrote a new...
Lucas Di Pentima [Tue, 2 Aug 2016 20:27:14 +0000 (17:27 -0300)]
9617, 9618: Added runtime constraints to container requests fixtures. Wrote a new test to load a CR from a fixture and update some attribute to test for the new validations

7 years agoMerge branch '9617-reject-container-reqs-without-vcpus'
Lucas Di Pentima [Tue, 2 Aug 2016 15:28:18 +0000 (12:28 -0300)]
Merge branch '9617-reject-container-reqs-without-vcpus'

9617, 9618: Require positive integers on both 'vcpus' and 'ram' runtime constraints when a container request is committed. Closes #9617, #9618

7 years ago9617: Corrected variable & test naming
Lucas Di Pentima [Tue, 2 Aug 2016 15:26:03 +0000 (12:26 -0300)]
9617: Corrected variable & test naming