arvados.git
8 years agoMerge branch '9826-config-symbols'
Tom Clegg [Mon, 22 Aug 2016 13:42:44 +0000 (09:42 -0400)]
Merge branch '9826-config-symbols'

closes #9826

8 years ago9826: Deserialize symbols when loading server config YAML.
Tom Clegg [Mon, 22 Aug 2016 13:13:15 +0000 (09:13 -0400)]
9826: Deserialize symbols when loading server config YAML.

8 years agoAdd --compute-checksum to sdk/cwl/test_with_arvbox.sh refs #9552
Peter Amstutz [Mon, 22 Aug 2016 02:53:32 +0000 (22:53 -0400)]
Add --compute-checksum to sdk/cwl/test_with_arvbox.sh refs #9552

8 years agoMerge branch '9820-cwl-poll-jobs' closes #9820
Peter Amstutz [Fri, 19 Aug 2016 15:13:51 +0000 (11:13 -0400)]
Merge branch '9820-cwl-poll-jobs' closes #9820

8 years ago9820: Directly poll job or container records that we are interested in. Benefit:...
Peter Amstutz [Fri, 19 Aug 2016 14:50:25 +0000 (10:50 -0400)]
9820: Directly poll job or container records that we are interested in. Benefit: puts less load on database than log table polling, and doesn't miss events.

8 years agoMerge branch '9684-workflows'
Tom Clegg [Fri, 19 Aug 2016 00:27:37 +0000 (20:27 -0400)]
Merge branch '9684-workflows'

refs #9684

8 years ago9684: Style nitpicks.
Tom Clegg [Fri, 19 Aug 2016 00:12:25 +0000 (20:12 -0400)]
9684: Style nitpicks.

8 years ago9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"
radhika [Thu, 18 Aug 2016 18:22:59 +0000 (14:22 -0400)]
9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"

8 years agoMerge branch '9333-api-server-invalid-sql'
Lucas Di Pentima [Thu, 18 Aug 2016 17:56:43 +0000 (14:56 -0300)]
Merge branch '9333-api-server-invalid-sql'

9333: Fixes to solve workbench integration tests failures. Refs #9333

8 years agoMerge branch 'master' into 9333-api-server-invalid-sql
Lucas Di Pentima [Thu, 18 Aug 2016 17:34:38 +0000 (14:34 -0300)]
Merge branch 'master' into 9333-api-server-invalid-sql

8 years ago9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"
radhika [Thu, 18 Aug 2016 16:47:50 +0000 (12:47 -0400)]
9684: update arvados_model -> recursive_stringify to convert ":foo" to "foo"

8 years ago9333: Adding collection model specific sttributes to be ignored on the validation...
Lucas Di Pentima [Thu, 18 Aug 2016 16:06:01 +0000 (13:06 -0300)]
9333: Adding collection model specific sttributes to be ignored on the validation mechanism to fix workbench integration test failures

8 years ago9684: Add safe_yaml initializer
radhika [Thu, 18 Aug 2016 13:43:23 +0000 (09:43 -0400)]
9684: Add safe_yaml initializer

8 years agocloses #9684
radhika [Thu, 18 Aug 2016 11:51:21 +0000 (07:51 -0400)]
closes #9684
Merge branch '9684-workflows'

8 years agoMerge branch '9333-api-server-invalid-sql'
Lucas Di Pentima [Wed, 17 Aug 2016 16:44:41 +0000 (13:44 -0300)]
Merge branch '9333-api-server-invalid-sql'

9333: Selected attributes list validation on the API Server, avoiding invalid SQL queries to the database. Closes #9333

8 years ago9333: Removed superfluous quote marks on exception message
Lucas Di Pentima [Wed, 17 Aug 2016 16:39:24 +0000 (13:39 -0300)]
9333: Removed superfluous quote marks on exception message

8 years ago9333: Treating the "attribute is not a string" error case the same as the other inval...
Lucas Di Pentima [Tue, 16 Aug 2016 21:22:04 +0000 (18:22 -0300)]
9333: Treating the "attribute is not a string" error case the same as the other invalid attributes. Moved the checks to ArvadosModel. Updated tests.

8 years ago9684: workflow yaml error logging
radhika [Tue, 16 Aug 2016 21:09:33 +0000 (17:09 -0400)]
9684: workflow yaml error logging

8 years ago9678: use safe_yaml and other updates.
radhika [Mon, 15 Aug 2016 22:19:00 +0000 (18:19 -0400)]
9678: use safe_yaml and other updates.

8 years agocloses #9709
radhika [Mon, 15 Aug 2016 14:26:44 +0000 (10:26 -0400)]
closes #9709
Merge branch '9709-dont-log-manifest-texts'

8 years ago9333: Attribute validation on "select" to avoid invalid SQL statements
Lucas Di Pentima [Fri, 12 Aug 2016 00:38:46 +0000 (21:38 -0300)]
9333: Attribute validation on "select" to avoid invalid SQL statements

8 years agoFix failing tests. refs #9776
Peter Amstutz [Thu, 11 Aug 2016 22:18:13 +0000 (18:18 -0400)]
Fix failing tests. refs #9776

8 years agoFix schema salad package build version. no issue #
Peter Amstutz [Thu, 11 Aug 2016 21:50:49 +0000 (17:50 -0400)]
Fix schema salad package build version.  no issue #

8 years agoMerge branch '9776-cwl-HOME-TMPDIR' closes #9776
Peter Amstutz [Thu, 11 Aug 2016 21:02:15 +0000 (17:02 -0400)]
Merge branch '9776-cwl-HOME-TMPDIR' closes #9776

8 years ago9776: Ensure that $HOME and $TMPDIR are set correctly.
Peter Amstutz [Thu, 11 Aug 2016 20:55:10 +0000 (16:55 -0400)]
9776: Ensure that $HOME and $TMPDIR are set correctly.

8 years agoMerge branch '9615-cwl-writable-unsupported' refs #9615
Peter Amstutz [Thu, 11 Aug 2016 20:33:06 +0000 (16:33 -0400)]
Merge branch '9615-cwl-writable-unsupported' refs #9615

8 years ago9615: Add check_writable to check for "writable" field and raise UnsupportedRequirement.
Peter Amstutz [Thu, 4 Aug 2016 15:36:56 +0000 (11:36 -0400)]
9615: Add check_writable to check for "writable" field and raise UnsupportedRequirement.

8 years agoMerge branch '9751-cwl-keep-dir' closes #9751
Peter Amstutz [Thu, 11 Aug 2016 20:30:34 +0000 (16:30 -0400)]
Merge branch '9751-cwl-keep-dir' closes #9751

8 years ago9552: import compute_checksums from cwltool.
Peter Amstutz [Thu, 11 Aug 2016 18:54:36 +0000 (14:54 -0400)]
9552: import compute_checksums from cwltool.

8 years ago9751: Fix handling local Directories
Peter Amstutz [Thu, 11 Aug 2016 19:35:15 +0000 (15:35 -0400)]
9751: Fix handling local Directories

8 years ago9751: Bump cwltool dependency.
Peter Amstutz [Thu, 11 Aug 2016 18:46:24 +0000 (14:46 -0400)]
9751: Bump cwltool dependency.

8 years ago9751: Fix unit tests. Rename dir6.cwl to keep-dir-test-input.cwl
Peter Amstutz [Thu, 11 Aug 2016 18:19:37 +0000 (14:19 -0400)]
9751: Fix unit tests.  Rename dir6.cwl to keep-dir-test-input.cwl

8 years ago9751: Override realpath in CollectionFsAccess
Peter Amstutz [Thu, 11 Aug 2016 17:01:41 +0000 (13:01 -0400)]
9751: Override realpath in CollectionFsAccess

8 years ago9751: Fix direct to keep directory references.
Peter Amstutz [Wed, 10 Aug 2016 17:45:56 +0000 (13:45 -0400)]
9751: Fix direct to keep directory references.
Add arvados-specific integration test for arvados-cwl-runner.

8 years ago9751: Provide make_fs_access=CollectionFsAccess and bump cwltool pin.
Peter Amstutz [Wed, 10 Aug 2016 15:51:44 +0000 (11:51 -0400)]
9751: Provide make_fs_access=CollectionFsAccess and bump cwltool pin.

8 years agocloses #9678
radhika [Thu, 11 Aug 2016 18:13:36 +0000 (14:13 -0400)]
closes #9678
Merge branch '9678-container-request-log-tab'

8 years ago9709: do not include manifest_text in collection logs.
radhika [Thu, 11 Aug 2016 18:10:00 +0000 (14:10 -0400)]
9709: do not include manifest_text in collection logs.

8 years agocloses #9658
radhika [Thu, 11 Aug 2016 13:50:36 +0000 (09:50 -0400)]
closes #9658
Merge branch '9658-cwl-runner-container'

8 years ago9678: enhance test to verify that the combined logs are displayed.
radhika [Wed, 10 Aug 2016 18:41:29 +0000 (14:41 -0400)]
9678: enhance test to verify that the combined logs are displayed.

8 years ago9678: enhance container log tab display to consider the "combined" object logs.
radhika [Wed, 10 Aug 2016 18:05:02 +0000 (14:05 -0400)]
9678: enhance container log tab display to consider the "combined" object logs.

8 years ago9658: when arvados-cwl-runner is invoked with --api containers, display the container...
radhika [Wed, 10 Aug 2016 16:29:31 +0000 (12:29 -0400)]
9658: when arvados-cwl-runner is invoked with --api containers, display the container_request uuid to console.

8 years agocloses #9745
radhika [Wed, 10 Aug 2016 15:58:22 +0000 (11:58 -0400)]
closes #9745
Merge branch '9745-cds-api-config'

8 years ago9684: add workflow resource to api server
radhika [Tue, 9 Aug 2016 19:51:50 +0000 (15:51 -0400)]
9684: add workflow resource to api server

8 years ago9745: Update example config.
Tom Clegg [Fri, 5 Aug 2016 15:45:55 +0000 (11:45 -0400)]
9745: Update example config.

8 years ago9745: Load API host and token from crunch-dispatch-slurm config file.
Tom Clegg [Fri, 5 Aug 2016 15:44:29 +0000 (11:44 -0400)]
9745: Load API host and token from crunch-dispatch-slurm config file.

8 years ago9706: Add example config file.
Tom Clegg [Fri, 5 Aug 2016 14:54:37 +0000 (10:54 -0400)]
9706: Add example config file.

No issue #

8 years agogofmt.
Tom Clegg [Thu, 4 Aug 2016 22:16:24 +0000 (18:16 -0400)]
gofmt.

No issue #

8 years agoFix race in crunchstat.
Tom Clegg [Thu, 4 Aug 2016 21:45:10 +0000 (17:45 -0400)]
Fix race in crunchstat.

To fulfill the "nothing more will be logged after Stop() returns"
guarantee, Stop() needs to wait for run() to finish.

Should fix https://ci.curoverse.com/job/run-tests-remainder/38/consoleText

No issue #

8 years agoSimplify write flusher using a time.Ticker.
Tom Clegg [Thu, 4 Aug 2016 21:44:34 +0000 (17:44 -0400)]
Simplify write flusher using a time.Ticker.

No issue #

8 years agoFix potential race in ThrottledLogger flusher.
Tom Clegg [Thu, 4 Aug 2016 20:53:32 +0000 (16:53 -0400)]
Fix potential race in ThrottledLogger flusher.

No issue #

8 years agoMerge branch '9552-compute-checksum-final-output' refs #9552
Peter Amstutz [Thu, 4 Aug 2016 16:01:37 +0000 (12:01 -0400)]
Merge branch '9552-compute-checksum-final-output' refs #9552

8 years agoMerge branch '9406-cgroup-parent'
Tom Clegg [Thu, 4 Aug 2016 15:31:44 +0000 (11:31 -0400)]
Merge branch '9406-cgroup-parent'

closes #9406

8 years ago9406: Explain why there are two separate CgroupParent fields.
Tom Clegg [Thu, 4 Aug 2016 15:31:34 +0000 (11:31 -0400)]
9406: Explain why there are two separate CgroupParent fields.

8 years ago9552: If compute_checksum is true, check if checksum needs to be computed on final...
Peter Amstutz [Thu, 4 Aug 2016 14:23:12 +0000 (10:23 -0400)]
9552: If compute_checksum is true, check if checksum needs to be computed on final output, and add it if missing.

8 years agoMerge branch '9595-mount-json'
Tom Clegg [Thu, 4 Aug 2016 14:10:12 +0000 (10:10 -0400)]
Merge branch '9595-mount-json'

closes #9595

8 years ago9595: Fix hardcoded temp dir in test. Add checks for CleanupDirs().
Tom Clegg [Thu, 4 Aug 2016 06:26:54 +0000 (02:26 -0400)]
9595: Fix hardcoded temp dir in test. Add checks for CleanupDirs().

8 years ago9595: Support "json" mount type.
Tom Clegg [Thu, 4 Aug 2016 05:48:33 +0000 (01:48 -0400)]
9595: Support "json" mount type.

8 years agoMerge branch '9677-update-install-docs'
Ward Vandewege [Thu, 4 Aug 2016 00:48:32 +0000 (20:48 -0400)]
Merge branch '9677-update-install-docs'

closes #9677

8 years agoDocument how to download and tag the arvados/jobs Docker image.
Ward Vandewege [Wed, 3 Aug 2016 19:54:34 +0000 (15:54 -0400)]
Document how to download and tag the arvados/jobs Docker image.

refs #9677

8 years agoMerge branch '9674-cwl-initialworkdir' closes #9674
Peter Amstutz [Wed, 3 Aug 2016 20:47:05 +0000 (16:47 -0400)]
Merge branch '9674-cwl-initialworkdir' closes #9674

8 years agoMerge branch '9617-reject-container-reqs-without-vcpus'
Lucas Di Pentima [Tue, 2 Aug 2016 23:29:04 +0000 (20:29 -0300)]
Merge branch '9617-reject-container-reqs-without-vcpus'

9617, 9618: Added runtime constraints to fixtures.

refs #9617, #9618

8 years ago9617, 9618: Added runtime constraints to container requests fixtures. Wrote a new...
Lucas Di Pentima [Tue, 2 Aug 2016 20:27:14 +0000 (17:27 -0300)]
9617, 9618: Added runtime constraints to container requests fixtures. Wrote a new test to load a CR from a fixture and update some attribute to test for the new validations

8 years agoMerge branch '9617-reject-container-reqs-without-vcpus'
Lucas Di Pentima [Tue, 2 Aug 2016 15:28:18 +0000 (12:28 -0300)]
Merge branch '9617-reject-container-reqs-without-vcpus'

9617, 9618: Require positive integers on both 'vcpus' and 'ram' runtime constraints when a container request is committed. Closes #9617, #9618

8 years ago9617: Corrected variable & test naming
Lucas Di Pentima [Tue, 2 Aug 2016 15:26:03 +0000 (12:26 -0300)]
9617: Corrected variable & test naming

8 years ago9674: InitialWorkDirPathMapper bug fix to set correct source path for files in
Peter Amstutz [Tue, 2 Aug 2016 15:17:18 +0000 (11:17 -0400)]
9674: InitialWorkDirPathMapper bug fix to set correct source path for files in
virtual working directory.

8 years ago9617: Reorganized code reusing the invalid_constraint list on both create & update...
Lucas Di Pentima [Tue, 2 Aug 2016 14:50:05 +0000 (11:50 -0300)]
9617: Reorganized code reusing the invalid_constraint list on both create & update tests. Separated the successful test case to be on its own.

8 years ago9406: Add -cgroup-parent-subsystem argument.
Tom Clegg [Tue, 2 Aug 2016 13:42:07 +0000 (09:42 -0400)]
9406: Add -cgroup-parent-subsystem argument.

This accommodates setups where the scheduling system (e.g., SLURM)
uses cgroups to impose resource limits on crunch-run itself, and those
limits are meant to include resources used by the container itself.

Normally, when docker creates a container, it creates new cgroups
inside the "docker" cgroup, which circumvents any resource limits set
on crunch-run's own cgroup.

If a subsystem is specified (e.g., -cgroup-parent-subsystem=memory),
crunch-run will look in /proc/self/cgroup to find the current cgroup
for the given subsystem, and instruct docker to use that cgroup as the
parent of the new container's cgroup instead.

8 years ago9617: Added one more invalid constraint case to the tests, also check for container_u...
Lucas Di Pentima [Tue, 2 Aug 2016 13:28:57 +0000 (10:28 -0300)]
9617: Added one more invalid constraint case to the tests, also check for container_uuid on successful test

8 years ago9406: Support passing additional arguments from crunch-dispatch-slurm to crunch-run.
Tom Clegg [Tue, 2 Aug 2016 01:42:37 +0000 (21:42 -0400)]
9406: Support passing additional arguments from crunch-dispatch-slurm to crunch-run.

8 years ago9617: Corrected style following suggestions and also compacted the code to avoid...
Lucas Di Pentima [Mon, 1 Aug 2016 21:30:16 +0000 (18:30 -0300)]
9617: Corrected style following suggestions and also compacted the code to avoid repetition.

8 years ago9617: Added validations on updates & tests on sucessful cases.
Lucas Di Pentima [Mon, 1 Aug 2016 19:44:16 +0000 (16:44 -0300)]
9617: Added validations on updates & tests on sucessful cases.

8 years ago9617, 9618: Added validations and wrote the corresponding test. Also, corrected some...
Lucas Di Pentima [Fri, 29 Jul 2016 21:07:15 +0000 (18:07 -0300)]
9617, 9618: Added validations and wrote the corresponding test. Also, corrected some other tests to conform this new requirement.

8 years agorefs #9552
radhika [Thu, 28 Jul 2016 18:27:27 +0000 (14:27 -0400)]
refs #9552
Merge branch '9552-compute-checksum-flag'

8 years ago9552: add compute-checksum flag to arvados-cwl-runner
radhika [Thu, 28 Jul 2016 17:36:26 +0000 (13:36 -0400)]
9552: add compute-checksum flag to arvados-cwl-runner

8 years agoWe have a new build job at ci.curoverse.com. Update the build status
Ward Vandewege [Thu, 28 Jul 2016 00:23:06 +0000 (20:23 -0400)]
We have a new build job at ci.curoverse.com. Update the build status
icon in README.md accordingly.

No issue #

8 years agocloses #9581
radhika [Wed, 27 Jul 2016 21:15:00 +0000 (17:15 -0400)]
closes #9581
Merge branch '9581-slurm-dispatcher-config'

8 years ago9581: add json config file handling to slurm dispatcher.
radhika [Tue, 26 Jul 2016 16:28:27 +0000 (12:28 -0400)]
9581: add json config file handling to slurm dispatcher.

8 years agoMerge branch '8555-s3-trash'
Tom Clegg [Wed, 27 Jul 2016 19:22:26 +0000 (15:22 -0400)]
Merge branch '8555-s3-trash'

closes #8555

8 years ago8555: Fail Go tests if not gofmted.
Tom Clegg [Wed, 27 Jul 2016 15:29:17 +0000 (11:29 -0400)]
8555: Fail Go tests if not gofmted.

8 years ago8555: golint
Tom Clegg [Tue, 26 Jul 2016 20:29:57 +0000 (16:29 -0400)]
8555: golint

8 years ago8555: Fix up comments.
Tom Clegg [Tue, 26 Jul 2016 20:26:15 +0000 (16:26 -0400)]
8555: Fix up comments.

8 years ago8555: gofmt
Tom Clegg [Tue, 26 Jul 2016 20:20:30 +0000 (16:20 -0400)]
8555: gofmt

8 years ago8555: Test Get() after successful Untrash. Test Put+Mtime in all scenarios.
Tom Clegg [Tue, 26 Jul 2016 16:25:35 +0000 (12:25 -0400)]
8555: Test Get() after successful Untrash. Test Put+Mtime in all scenarios.

8 years ago8555: Untrash to repair inconsistent state (trash/X without recentX).
Tom Clegg [Tue, 26 Jul 2016 15:07:46 +0000 (11:07 -0400)]
8555: Untrash to repair inconsistent state (trash/X without recentX).

8 years ago8555: Log statistics in EmptyTrash.
Tom Clegg [Tue, 26 Jul 2016 13:44:56 +0000 (09:44 -0400)]
8555: Log statistics in EmptyTrash.

8 years ago8555: Improve variable names and comments.
Tom Clegg [Tue, 26 Jul 2016 13:38:07 +0000 (09:38 -0400)]
8555: Improve variable names and comments.

8 years ago8555: Reformat test cases.
Tom Clegg [Tue, 26 Jul 2016 13:19:30 +0000 (09:19 -0400)]
8555: Reformat test cases.

8 years ago8555: De-obfuscate test case.
Tom Clegg [Fri, 22 Jul 2016 18:04:34 +0000 (14:04 -0400)]
8555: De-obfuscate test case.

8 years ago8555: Fix EmptyTrash so it can clean up old races.
Tom Clegg [Thu, 21 Jul 2016 15:44:17 +0000 (11:44 -0400)]
8555: Fix EmptyTrash so it can clean up old races.

8 years ago8555: golint
Tom Clegg [Wed, 20 Jul 2016 15:56:59 +0000 (11:56 -0400)]
8555: golint

8 years ago8555: gofmt
Tom Clegg [Wed, 20 Jul 2016 15:42:14 +0000 (11:42 -0400)]
8555: gofmt

8 years ago8555: Test various backend states. Update recent/X timestamp during Untrash.
Tom Clegg [Wed, 20 Jul 2016 15:40:41 +0000 (11:40 -0400)]
8555: Test various backend states. Update recent/X timestamp during Untrash.

8 years ago8555: Move checkRaceWindow out to a func, tweak comments
Tom Clegg [Mon, 18 Jul 2016 19:47:23 +0000 (15:47 -0400)]
8555: Move checkRaceWindow out to a func, tweak comments

8 years ago8555: Add log messages for time-parsing errors.
Tom Clegg [Thu, 14 Jul 2016 20:08:16 +0000 (16:08 -0400)]
8555: Add log messages for time-parsing errors.

8 years ago8555: Implement trash with race-recovery for S3 volumes.
Tom Clegg [Thu, 14 Jul 2016 19:53:20 +0000 (15:53 -0400)]
8555: Implement trash with race-recovery for S3 volumes.

See https://dev.arvados.org/projects/arvados/wiki/S3_bucket_volume_implementation

8 years agocloses #8568
radhika [Wed, 27 Jul 2016 17:26:39 +0000 (13:26 -0400)]
closes #8568
Merge branch '8568-docker-version-support'

8 years agoMerge branch 'master' into 8568-docker-version-support
radhika [Wed, 27 Jul 2016 17:26:06 +0000 (13:26 -0400)]
Merge branch 'master' into 8568-docker-version-support

8 years agocloses #9552
radhika [Wed, 27 Jul 2016 16:13:15 +0000 (12:13 -0400)]
closes #9552
Merge branch '9552-compute-checksum-flag'

8 years agoMerge branch 'master' into 9552-compute-checksum-flag
radhika [Wed, 27 Jul 2016 16:07:35 +0000 (12:07 -0400)]
Merge branch 'master' into 9552-compute-checksum-flag

Conflicts:
sdk/cwl/setup.py

8 years ago9659: Bump CWL SDK's versioned dependency on PySDK.
Brett Smith [Tue, 26 Jul 2016 20:54:39 +0000 (16:54 -0400)]
9659: Bump CWL SDK's versioned dependency on PySDK.

The CWL SDK depends on the change made to arv-run in
27816b602e9da83a2565e6fe8f87f250555b1ba5.  Update the version
dependency in setup.py to reflect this.  Refs #9570, #9659.