arvados.git
8 years ago8828: Move logic for checking $(which crunchrunner) into script that runs before...
Peter Amstutz [Thu, 31 Mar 2016 19:58:47 +0000 (15:58 -0400)]
8828: Move logic for checking $(which crunchrunner) into script that runs before invoking Docker on the compute node.

8 years agoBuild newer cwltool version.
Ward Vandewege [Thu, 31 Mar 2016 17:49:54 +0000 (13:49 -0400)]
Build newer cwltool version.

No issue #

8 years agoMerge branch '8840-lock-job-record' closes #8840
Peter Amstutz [Thu, 31 Mar 2016 15:54:40 +0000 (11:54 -0400)]
Merge branch '8840-lock-job-record' closes #8840

8 years agoMerge branch '8654-arv-jobs-cwl-runner' closes #8654
Peter Amstutz [Thu, 31 Mar 2016 15:35:49 +0000 (11:35 -0400)]
Merge branch '8654-arv-jobs-cwl-runner' closes #8654

8 years ago8654: Update test because input cwl files changed.
Peter Amstutz [Thu, 31 Mar 2016 15:30:31 +0000 (11:30 -0400)]
8654: Update test because input cwl files changed.

8 years ago8654: Fix versionstring(). Improve help text / comments / style tweaks.
Peter Amstutz [Thu, 31 Mar 2016 15:00:15 +0000 (11:00 -0400)]
8654: Fix versionstring().  Improve help text / comments / style tweaks.

8 years agoMerge branch 'master' into 8654-arv-jobs-cwl-runner
Peter Amstutz [Thu, 31 Mar 2016 14:32:09 +0000 (10:32 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner

8 years ago8654: Rename tests/inp/ to test/input/
Peter Amstutz [Thu, 31 Mar 2016 14:31:59 +0000 (10:31 -0400)]
8654: Rename tests/inp/ to test/input/

8 years ago8840: Use 'with_lock' instead of 'transaction' in Job.lock method.
Peter Amstutz [Wed, 30 Mar 2016 19:00:10 +0000 (15:00 -0400)]
8840: Use 'with_lock' instead of 'transaction' in Job.lock method.

8 years ago8654: Add missing test_submit
Peter Amstutz [Wed, 30 Mar 2016 18:58:30 +0000 (14:58 -0400)]
8654: Add missing test_submit

8 years ago8654: Add comments
Peter Amstutz [Wed, 30 Mar 2016 18:45:12 +0000 (14:45 -0400)]
8654: Add comments

8 years ago8654: Print uuid of uploaded docker image on stderr instead of stdout.
Peter Amstutz [Wed, 30 Mar 2016 18:00:24 +0000 (14:00 -0400)]
8654: Print uuid of uploaded docker image on stderr instead of stdout.

8 years agocloses #8703
radhika [Wed, 30 Mar 2016 13:47:29 +0000 (09:47 -0400)]
closes #8703
Merge branch '8703-job-components'

8 years agoMerge branch 'master' into 8703-job-components
radhika [Wed, 30 Mar 2016 13:47:08 +0000 (09:47 -0400)]
Merge branch 'master' into 8703-job-components

8 years ago8654: Update test_with_arvbox.sh
Peter Amstutz [Wed, 30 Mar 2016 13:33:49 +0000 (09:33 -0400)]
8654: Update test_with_arvbox.sh

8 years agocrunchrunner crunch script selects between $JOB_PARAMETER_CRUNCHRUNNER
Peter Amstutz [Wed, 30 Mar 2016 01:11:06 +0000 (21:11 -0400)]
crunchrunner crunch script selects between $JOB_PARAMETER_CRUNCHRUNNER
or /usr/local/bin/crunchrunner, refs #8827

8 years ago8654: Pin pyasn1_modules to version that is compatible with pyasn1==0.1.7.
Peter Amstutz [Tue, 29 Mar 2016 20:28:27 +0000 (16:28 -0400)]
8654: Pin pyasn1_modules to version that is compatible with pyasn1==0.1.7.

8 years ago8654: Fix version string produced by arvados-cwl-runner.
Peter Amstutz [Tue, 29 Mar 2016 20:28:19 +0000 (16:28 -0400)]
8654: Fix version string produced by arvados-cwl-runner.

8 years agoFix sdk/cwl test refs #8815
Peter Amstutz [Tue, 29 Mar 2016 19:34:48 +0000 (15:34 -0400)]
Fix sdk/cwl test refs #8815

8 years agoMerge branch '8815-crunchrunner-everywhere' closes #8815
Peter Amstutz [Tue, 29 Mar 2016 18:16:58 +0000 (14:16 -0400)]
Merge branch '8815-crunchrunner-everywhere' closes #8815

8 years agoMerge branch '8815-crunchrunner-everywhere' into 8654-arv-jobs-cwl-runner
Peter Amstutz [Tue, 29 Mar 2016 17:34:40 +0000 (13:34 -0400)]
Merge branch '8815-crunchrunner-everywhere' into 8654-arv-jobs-cwl-runner

Conflicts:
sdk/cwl/arvados_cwl/__init__.py

8 years ago8654: Make --submit --wait the default mode.
Peter Amstutz [Tue, 29 Mar 2016 17:30:17 +0000 (13:30 -0400)]
8654: Make --submit --wait the default mode.

8 years ago8703: better organized tests
radhika [Tue, 29 Mar 2016 17:10:49 +0000 (13:10 -0400)]
8703: better organized tests

8 years ago8815: Fix syntax errors.
Peter Amstutz [Tue, 29 Mar 2016 17:06:22 +0000 (13:06 -0400)]
8815: Fix syntax errors.

8 years ago8815: Now expect /usr/local/bin/crunchrunner. Bind mount host certificates to
Peter Amstutz [Tue, 29 Mar 2016 16:18:15 +0000 (12:18 -0400)]
8815: Now expect /usr/local/bin/crunchrunner.  Bind mount host certificates to
/etc/arvados/ca-certificates.crt

8 years agoMerge branch 'master' into 8703-job-components
radhika [Tue, 29 Mar 2016 15:54:01 +0000 (11:54 -0400)]
Merge branch 'master' into 8703-job-components

8 years agoBuild a package for crunchrunner.
Ward Vandewege [Tue, 29 Mar 2016 15:24:59 +0000 (11:24 -0400)]
Build a package for crunchrunner.

refs #8815

8 years ago8815: Rely on system-provided crunchrunner. Also use arvados/jobs by default if...
Peter Amstutz [Tue, 29 Mar 2016 13:38:16 +0000 (09:38 -0400)]
8815: Rely on system-provided crunchrunner.  Also use arvados/jobs by default if no docker provided.

8 years ago8815: Crunch-job bind mounts crunchrunner binary and certificates from host.
Peter Amstutz [Tue, 29 Mar 2016 13:23:51 +0000 (09:23 -0400)]
8815: Crunch-job bind mounts crunchrunner binary and certificates from host.
Updated arvbox to compile and install crunchrunner.

8 years ago8654: Bump cwltool version dependency and print cwl version string in cwl-runner...
Peter Amstutz [Mon, 28 Mar 2016 14:06:32 +0000 (10:06 -0400)]
8654: Bump cwltool version dependency and print cwl version string in cwl-runner crunch script.

8 years agoMerge branch 'master' into 8654-arv-jobs-cwl-runner
Peter Amstutz [Mon, 28 Mar 2016 13:37:37 +0000 (09:37 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner

Conflicts:
docker/jobs/Dockerfile

8 years agoMerge branch '8800-queue-query'
Brett Smith [Sun, 27 Mar 2016 20:47:07 +0000 (16:47 -0400)]
Merge branch '8800-queue-query'

Closes #8800, #8809.

8 years ago8800: Document the new queue_position implementation.
Brett Smith [Sun, 27 Mar 2016 20:43:55 +0000 (16:43 -0400)]
8800: Document the new queue_position implementation.

For the benefit of future readers.

8 years agoFix package build and test of the arvados-cwl-runner package for
Ward Vandewege [Sun, 27 Mar 2016 02:19:09 +0000 (22:19 -0400)]
Fix package build and test of the arvados-cwl-runner package for
ubuntu1204 and centos6.

refs #8671

8 years ago8800: Drop queue_position support.
Tom Clegg [Fri, 25 Mar 2016 19:59:34 +0000 (15:59 -0400)]
8800: Drop queue_position support.

8 years agoBump up the iteration for the python-arvados-cwl-runner package so that
Ward Vandewege [Sat, 26 Mar 2016 01:50:28 +0000 (21:50 -0400)]
Bump up the iteration for the python-arvados-cwl-runner package so that
it gets rebuilt.

refs #8671

8 years agoFix a few more dependencies for the python-arvados-cwl-runner package.
Ward Vandewege [Sat, 26 Mar 2016 01:36:19 +0000 (21:36 -0400)]
Fix a few more dependencies for the python-arvados-cwl-runner package.

refs #8671

8 years ago8654: Passes 100% CWL conformance tests using running cwl-runner in a crunch job!
Peter Amstutz [Fri, 25 Mar 2016 20:35:10 +0000 (16:35 -0400)]
8654: Passes 100% CWL conformance tests using running cwl-runner in a crunch job!

8 years ago8654: --version reports versions for arvados-cwl-runner, arvados-python-client,
Peter Amstutz [Fri, 25 Mar 2016 17:54:29 +0000 (13:54 -0400)]
8654: --version reports versions for arvados-cwl-runner, arvados-python-client,
and cwltool.

8 years ago8556: Update method signature for current SDK version.
Tom Clegg [Thu, 24 Mar 2016 19:38:07 +0000 (15:38 -0400)]
8556: Update method signature for current SDK version.

refs #8556

8 years ago8703: add "components" hash to job.
radhika [Thu, 24 Mar 2016 18:05:25 +0000 (14:05 -0400)]
8703: add "components" hash to job.

8 years agoMerge branch '8767-items-available'
Tom Clegg [Thu, 24 Mar 2016 17:58:21 +0000 (13:58 -0400)]
Merge branch '8767-items-available'

refs #8767

8 years ago8767: Make offset work properly in ApiClientAuthorizationsController#index.
Tom Clegg [Tue, 22 Mar 2016 20:31:56 +0000 (16:31 -0400)]
8767: Make offset work properly in ApiClientAuthorizationsController#index.

Before this, #index was ignoring the "offset" request param and was
not providing an "items_available" attribute in the response.  This
made Workbench's "get all pages" routine an infinite loop.

8 years agoDon't retry slurm subprocess calls that raise OSError, no issue #
Peter Amstutz [Wed, 23 Mar 2016 20:32:15 +0000 (16:32 -0400)]
Don't retry slurm subprocess calls that raise OSError, no issue #

8 years agoFix merge conflict. refs #8766
Peter Amstutz [Wed, 23 Mar 2016 19:42:21 +0000 (15:42 -0400)]
Fix merge conflict.  refs #8766

8 years ago8654: Add value for quiet to args.
Peter Amstutz [Wed, 23 Mar 2016 18:56:48 +0000 (14:56 -0400)]
8654: Add value for quiet to args.

8 years agoMerge branch '8766-cwl-collection-project'
Peter Amstutz [Wed, 23 Mar 2016 18:00:32 +0000 (14:00 -0400)]
Merge branch '8766-cwl-collection-project'

8 years agoMerge branch 'master' into 8766-cwl-collection-project
Peter Amstutz [Wed, 23 Mar 2016 18:00:21 +0000 (14:00 -0400)]
Merge branch 'master' into 8766-cwl-collection-project

Conflicts:
sdk/cwl/tests/test_job.py

8 years agoBuild script fix: make sure that run-build-docker-jobs-image.sh always
Ward Vandewege [Wed, 23 Mar 2016 15:15:17 +0000 (11:15 -0400)]
Build script fix: make sure that run-build-docker-jobs-image.sh always
rebuilds the jobs image when the git commit passed in via --tags
changes.

No issue #

8 years agoMerge branch '8766-cwl-collection-project' into 8654-arv-jobs-cwl-runner
Peter Amstutz [Wed, 23 Mar 2016 14:52:38 +0000 (10:52 -0400)]
Merge branch '8766-cwl-collection-project' into 8654-arv-jobs-cwl-runner

Conflicts:
sdk/cwl/arvados_cwl/__init__.py

8 years ago8766: Refactor logic to copy output collection and add tests.
Peter Amstutz [Wed, 23 Mar 2016 14:41:30 +0000 (10:41 -0400)]
8766: Refactor logic to copy output collection and add tests.

8 years agoFix sdk/cwl tests after commit 9df4d7da24ef8be639af1ab806cf833ee544fb45
Ward Vandewege [Wed, 23 Mar 2016 00:58:16 +0000 (20:58 -0400)]
Fix sdk/cwl tests after commit 9df4d7da24ef8be639af1ab806cf833ee544fb45

refs #8426

8 years ago8766: Port logic from arv-run-pipeline-instance for copy collection to parent
Peter Amstutz [Tue, 22 Mar 2016 20:41:19 +0000 (16:41 -0400)]
8766: Port logic from arv-run-pipeline-instance for copy collection to parent
project.

8 years ago8654: Bump cwltool version
Peter Amstutz [Tue, 22 Mar 2016 20:23:48 +0000 (16:23 -0400)]
8654: Bump cwltool version

8 years ago8654: Use iterator instead of readlines() when going through job log file.
Peter Amstutz [Tue, 22 Mar 2016 20:22:53 +0000 (16:22 -0400)]
8654: Use iterator instead of readlines() when going through job log file.

8 years agoClean up build/run-build-docker-jobs-image.sh a bit.
Ward Vandewege [Tue, 22 Mar 2016 19:55:21 +0000 (15:55 -0400)]
Clean up build/run-build-docker-jobs-image.sh a bit.

No issue #

8 years agoMerge branch 'master' into 8654-arv-jobs-cwl-runner
Peter Amstutz [Tue, 22 Mar 2016 16:24:02 +0000 (12:24 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner

8 years agoMerge branch '8747-fix-crash-on-update'
Tom Clegg [Tue, 22 Mar 2016 16:09:27 +0000 (12:09 -0400)]
Merge branch '8747-fix-crash-on-update'

refs #8747

8 years ago8345: Do not crash when update() called without an obj.
Tom Clegg [Thu, 10 Mar 2016 20:25:41 +0000 (15:25 -0500)]
8345: Do not crash when update() called without an obj.

8 years agoCrunchrunner fix connect os.Stderr to cmd.Stderr & update crunchrunner
Peter Amstutz [Tue, 22 Mar 2016 14:04:36 +0000 (10:04 -0400)]
Crunchrunner fix connect os.Stderr to cmd.Stderr & update crunchrunner
collection ref in arvados-cwl-runner refs #8426

8 years ago8654: Remove print statements.
Peter Amstutz [Tue, 22 Mar 2016 03:02:19 +0000 (23:02 -0400)]
8654: Remove print statements.

8 years ago8654: Handle error on capturing runner output
Peter Amstutz [Tue, 22 Mar 2016 03:01:38 +0000 (23:01 -0400)]
8654: Handle error on capturing runner output

8 years ago8654: Support --quiet
Peter Amstutz [Tue, 22 Mar 2016 02:01:35 +0000 (22:01 -0400)]
8654: Support --quiet

8 years ago8654: Support waiting for submitted runner job to complete
Peter Amstutz [Tue, 22 Mar 2016 01:43:25 +0000 (21:43 -0400)]
8654: Support waiting for submitted runner job to complete

8 years ago8654: Upload Docker images
Peter Amstutz [Tue, 22 Mar 2016 00:37:39 +0000 (20:37 -0400)]
8654: Upload Docker images

8 years ago8654: Bump arvados-cwl-runner version dependency on arvados python client.
Peter Amstutz [Tue, 22 Mar 2016 00:19:55 +0000 (20:19 -0400)]
8654: Bump arvados-cwl-runner version dependency on arvados python client.

8 years ago8654: When uploading, reuse existing collections with same owner/name/PDH
Peter Amstutz [Tue, 22 Mar 2016 00:16:10 +0000 (20:16 -0400)]
8654: When uploading, reuse existing collections with same owner/name/PDH

8 years ago8654: Bump arvados-cwl-runner version dependency on arvados python client.
Peter Amstutz [Mon, 21 Mar 2016 21:09:49 +0000 (17:09 -0400)]
8654: Bump arvados-cwl-runner version dependency on arvados python client.

8 years ago8654: Job has uuid and state fields
Peter Amstutz [Mon, 21 Mar 2016 21:05:08 +0000 (17:05 -0400)]
8654: Job has uuid and state fields

8 years ago8654: Add args.debug = True to get backtraces on errors
Peter Amstutz [Mon, 21 Mar 2016 21:02:51 +0000 (17:02 -0400)]
8654: Add args.debug = True to get backtraces on errors

8 years ago8654: Set args.submit = False
Peter Amstutz [Mon, 21 Mar 2016 21:01:15 +0000 (17:01 -0400)]
8654: Set args.submit = False

8 years ago8654: Add runner job to pipeline instance.
Peter Amstutz [Mon, 21 Mar 2016 20:55:12 +0000 (16:55 -0400)]
8654: Add runner job to pipeline instance.

8 years ago8654: Update cwl:tool entry
Peter Amstutz [Mon, 21 Mar 2016 20:46:26 +0000 (16:46 -0400)]
8654: Update cwl:tool entry

8 years ago8654: Fix keeppath and add ensure_unique_name to upload.
Peter Amstutz [Mon, 21 Mar 2016 20:44:31 +0000 (16:44 -0400)]
8654: Fix keeppath and add ensure_unique_name to upload.

8 years ago8654: Scanning for and uploading dependencies for submit works. Add test for
Peter Amstutz [Mon, 21 Mar 2016 20:24:06 +0000 (16:24 -0400)]
8654: Scanning for and uploading dependencies for submit works.  Add test for
--submit.  Fix cwl-runner to handle CWL File objects in script_parameters.

8 years agoMerge branch 'master' into 8654-arv-jobs-cwl-runner
Peter Amstutz [Mon, 21 Mar 2016 16:37:38 +0000 (12:37 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner

8 years agoMore small build script improvements.
Ward Vandewege [Sun, 20 Mar 2016 18:39:52 +0000 (14:39 -0400)]
More small build script improvements.

No issue #

8 years agoBuild script improvements:
Ward Vandewege [Sun, 20 Mar 2016 17:16:18 +0000 (13:16 -0400)]
Build script improvements:

* move some more functions into run-library.sh
* add a wrapper script for our single target package build/test/upload step

No issue #

8 years agoEnable BaseComputeNodeDriver.list_nodes to pass through keyword arguments.
Peter Amstutz [Sat, 19 Mar 2016 20:00:11 +0000 (16:00 -0400)]
Enable BaseComputeNodeDriver.list_nodes to pass through keyword arguments.
closes #8761

8 years agoMerge branch '8002-no-pyvcf-wip'
Brett Smith [Fri, 18 Mar 2016 18:50:59 +0000 (14:50 -0400)]
Merge branch '8002-no-pyvcf-wip'

Closes #8002, #8759.

8 years ago8002: Stop backporting PyVCF.
Brett Smith [Fri, 18 Mar 2016 18:37:13 +0000 (14:37 -0400)]
8002: Stop backporting PyVCF.

Arvados proper doesn't use it.  We only backported it as a convenience
previously.

8 years agoMerge branch '7852-readline-cache-fix-wip'
Brett Smith [Fri, 18 Mar 2016 15:31:00 +0000 (11:31 -0400)]
Merge branch '7852-readline-cache-fix-wip'

Closes #7852, #8755.

8 years ago7852: Bugfix _filepos tracking in PySDK readline method.
Brett Smith [Thu, 17 Mar 2016 22:03:21 +0000 (18:03 -0400)]
7852: Bugfix _filepos tracking in PySDK readline method.

Make sure self._filepos is always in the right position to return
uncached data before another read method could be called, to avoid
inconsistent results when interleaving those calls with readline
calls.

8 years ago7852: Added one more test "test_readline_then_readall" which does a readline followed...
radhika [Wed, 3 Feb 2016 17:19:54 +0000 (12:19 -0500)]
7852: Added one more test "test_readline_then_readall" which does a readline followed by a readall.
This test also fails exactly like the test "test_readline_then_readlines".

8 years ago7852: add a failing test that does: readline and then readlines. The expectation...
radhika [Mon, 25 Jan 2016 21:37:24 +0000 (16:37 -0500)]
7852: add a failing test that does: readline and then readlines. The expectation would be the second call,
readlines, would fetch all the remaining lines after the first line. However, the readlines call is only
fetching the data after the "first block" which was read during the first readline call. It appears that
self._readline_cache and self._filepos are playing a role here, which are set during the first readline.

8 years agoFix log scanning for "crunchrunner: $(task.outdir)" performed by arvados cwl
Peter Amstutz [Fri, 18 Mar 2016 14:21:55 +0000 (10:21 -0400)]
Fix log scanning for "crunchrunner: $(task.outdir)" performed by arvados cwl
runner.  no issue #

8 years agoMake the build of the schema_salad packages more quiet.
Ward Vandewege [Fri, 18 Mar 2016 02:19:24 +0000 (22:19 -0400)]
Make the build of the schema_salad packages more quiet.

refs #8671

8 years agogofmt. refs #8754
Tom Clegg [Thu, 17 Mar 2016 22:15:10 +0000 (18:15 -0400)]
gofmt. refs #8754

8 years ago8654: Resource upload WIP
Peter Amstutz [Thu, 17 Mar 2016 21:22:30 +0000 (17:22 -0400)]
8654: Resource upload WIP

8 years agoImprove exception reporting for arvados-cwl-runner, no issue #
Peter Amstutz [Thu, 17 Mar 2016 20:34:48 +0000 (16:34 -0400)]
Improve exception reporting for arvados-cwl-runner, no issue #

8 years agoBuild dependencies for arvados-cwl-runner, automatically.
Ward Vandewege [Thu, 17 Mar 2016 19:45:54 +0000 (15:45 -0400)]
Build dependencies for arvados-cwl-runner, automatically.

closes #8671

8 years ago8752: Skip unreliable Python SDK Keep test.
Brett Smith [Thu, 17 Mar 2016 18:42:00 +0000 (14:42 -0400)]
8752: Skip unreliable Python SDK Keep test.

Refs #8752.  See there for more discussion about the circumstances of
failure.

8 years agocloses #8561
radhika [Thu, 17 Mar 2016 15:30:01 +0000 (11:30 -0400)]
closes #8561
Merge branch '8561-node-pairing'

8 years agoMerge branch 'master' into 8561-node-pairing
radhika [Thu, 17 Mar 2016 15:29:44 +0000 (11:29 -0400)]
Merge branch 'master' into 8561-node-pairing

8 years ago8561: delete redundant "maxDiff = None"
radhika [Thu, 17 Mar 2016 15:28:23 +0000 (11:28 -0400)]
8561: delete redundant "maxDiff = None"

8 years agoMerge branch '8561-node-pairing' of git.curoverse.com:arvados into 8561-node-pairing
radhika [Thu, 17 Mar 2016 15:17:12 +0000 (11:17 -0400)]
Merge branch '8561-node-pairing' of git.curoverse.com:arvados into 8561-node-pairing

8 years ago8561: Set ec2_instance_id and cloud_factory().node_id to mock cloud/arvados id
Peter Amstutz [Thu, 17 Mar 2016 14:42:02 +0000 (10:42 -0400)]
8561: Set ec2_instance_id and cloud_factory().node_id to mock cloud/arvados id
instead of constant '2'.

8 years agoNode manager bugfix: late subscribers should get proxy() not _later (which is a tell_...
Peter Amstutz [Thu, 17 Mar 2016 13:59:10 +0000 (09:59 -0400)]
Node manager bugfix: late subscribers should get proxy() not _later (which is a tell_proxy())
fixes #8691 note-4

8 years agoMerge branch '8709-docker-cleaner-volumes-wip'
Brett Smith [Thu, 17 Mar 2016 13:57:26 +0000 (09:57 -0400)]
Merge branch '8709-docker-cleaner-volumes-wip'

Closes #8709, #8711.

8 years ago8709: Docker cleaner cleans container volumes.
Brett Smith [Tue, 15 Mar 2016 21:59:16 +0000 (17:59 -0400)]
8709: Docker cleaner cleans container volumes.

This prevents a compute node's Docker partition from filling with
scratch space volumes.