Peter Amstutz [Mon, 28 Mar 2016 13:37:37 +0000 (09:37 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner
Conflicts:
docker/jobs/Dockerfile
Brett Smith [Sun, 27 Mar 2016 20:47:07 +0000 (16:47 -0400)]
Merge branch '8800-queue-query'
Closes #8800, #8809.
Brett Smith [Sun, 27 Mar 2016 20:43:55 +0000 (16:43 -0400)]
8800: Document the new queue_position implementation.
For the benefit of future readers.
Ward Vandewege [Sun, 27 Mar 2016 02:19:09 +0000 (22:19 -0400)]
Fix package build and test of the arvados-cwl-runner package for
ubuntu1204 and centos6.
refs #8671
Tom Clegg [Fri, 25 Mar 2016 19:59:34 +0000 (15:59 -0400)]
8800: Drop queue_position support.
Ward Vandewege [Sat, 26 Mar 2016 01:50:28 +0000 (21:50 -0400)]
Bump up the iteration for the python-arvados-cwl-runner package so that
it gets rebuilt.
refs #8671
Ward Vandewege [Sat, 26 Mar 2016 01:36:19 +0000 (21:36 -0400)]
Fix a few more dependencies for the python-arvados-cwl-runner package.
refs #8671
Peter Amstutz [Fri, 25 Mar 2016 20:35:10 +0000 (16:35 -0400)]
8654: Passes 100% CWL conformance tests using running cwl-runner in a crunch job!
Peter Amstutz [Fri, 25 Mar 2016 17:54:29 +0000 (13:54 -0400)]
8654: --version reports versions for arvados-cwl-runner, arvados-python-client,
and cwltool.
Tom Clegg [Thu, 24 Mar 2016 19:38:07 +0000 (15:38 -0400)]
8556: Update method signature for current SDK version.
refs #8556
Tom Clegg [Thu, 24 Mar 2016 17:58:21 +0000 (13:58 -0400)]
Merge branch '8767-items-available'
refs #8767
Tom Clegg [Tue, 22 Mar 2016 20:31:56 +0000 (16:31 -0400)]
8767: Make offset work properly in ApiClientAuthorizationsController#index.
Before this, #index was ignoring the "offset" request param and was
not providing an "items_available" attribute in the response. This
made Workbench's "get all pages" routine an infinite loop.
Peter Amstutz [Wed, 23 Mar 2016 20:32:15 +0000 (16:32 -0400)]
Don't retry slurm subprocess calls that raise OSError, no issue #
Peter Amstutz [Wed, 23 Mar 2016 19:42:21 +0000 (15:42 -0400)]
Fix merge conflict. refs #8766
Peter Amstutz [Wed, 23 Mar 2016 18:56:48 +0000 (14:56 -0400)]
8654: Add value for quiet to args.
Peter Amstutz [Wed, 23 Mar 2016 18:00:32 +0000 (14:00 -0400)]
Merge branch '8766-cwl-collection-project'
Peter Amstutz [Wed, 23 Mar 2016 18:00:21 +0000 (14:00 -0400)]
Merge branch 'master' into 8766-cwl-collection-project
Conflicts:
sdk/cwl/tests/test_job.py
Ward Vandewege [Wed, 23 Mar 2016 15:15:17 +0000 (11:15 -0400)]
Build script fix: make sure that run-build-docker-jobs-image.sh always
rebuilds the jobs image when the git commit passed in via --tags
changes.
No issue #
Peter Amstutz [Wed, 23 Mar 2016 14:52:38 +0000 (10:52 -0400)]
Merge branch '8766-cwl-collection-project' into 8654-arv-jobs-cwl-runner
Conflicts:
sdk/cwl/arvados_cwl/__init__.py
Peter Amstutz [Wed, 23 Mar 2016 14:41:30 +0000 (10:41 -0400)]
8766: Refactor logic to copy output collection and add tests.
Ward Vandewege [Wed, 23 Mar 2016 00:58:16 +0000 (20:58 -0400)]
Fix sdk/cwl tests after commit
9df4d7da24ef8be639af1ab806cf833ee544fb45
refs #8426
Peter Amstutz [Tue, 22 Mar 2016 20:41:19 +0000 (16:41 -0400)]
8766: Port logic from arv-run-pipeline-instance for copy collection to parent
project.
Peter Amstutz [Tue, 22 Mar 2016 20:23:48 +0000 (16:23 -0400)]
8654: Bump cwltool version
Peter Amstutz [Tue, 22 Mar 2016 20:22:53 +0000 (16:22 -0400)]
8654: Use iterator instead of readlines() when going through job log file.
Ward Vandewege [Tue, 22 Mar 2016 19:55:21 +0000 (15:55 -0400)]
Clean up build/run-build-docker-jobs-image.sh a bit.
No issue #
Peter Amstutz [Tue, 22 Mar 2016 16:24:02 +0000 (12:24 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner
Tom Clegg [Tue, 22 Mar 2016 16:09:27 +0000 (12:09 -0400)]
Merge branch '8747-fix-crash-on-update'
refs #8747
Tom Clegg [Thu, 10 Mar 2016 20:25:41 +0000 (15:25 -0500)]
8345: Do not crash when update() called without an obj.
Peter Amstutz [Tue, 22 Mar 2016 14:04:36 +0000 (10:04 -0400)]
Crunchrunner fix connect os.Stderr to cmd.Stderr & update crunchrunner
collection ref in arvados-cwl-runner refs #8426
Peter Amstutz [Tue, 22 Mar 2016 03:02:19 +0000 (23:02 -0400)]
8654: Remove print statements.
Peter Amstutz [Tue, 22 Mar 2016 03:01:38 +0000 (23:01 -0400)]
8654: Handle error on capturing runner output
Peter Amstutz [Tue, 22 Mar 2016 02:01:35 +0000 (22:01 -0400)]
8654: Support --quiet
Peter Amstutz [Tue, 22 Mar 2016 01:43:25 +0000 (21:43 -0400)]
8654: Support waiting for submitted runner job to complete
Peter Amstutz [Tue, 22 Mar 2016 00:37:39 +0000 (20:37 -0400)]
8654: Upload Docker images
Peter Amstutz [Tue, 22 Mar 2016 00:19:55 +0000 (20:19 -0400)]
8654: Bump arvados-cwl-runner version dependency on arvados python client.
Peter Amstutz [Tue, 22 Mar 2016 00:16:10 +0000 (20:16 -0400)]
8654: When uploading, reuse existing collections with same owner/name/PDH
Peter Amstutz [Mon, 21 Mar 2016 21:09:49 +0000 (17:09 -0400)]
8654: Bump arvados-cwl-runner version dependency on arvados python client.
Peter Amstutz [Mon, 21 Mar 2016 21:05:08 +0000 (17:05 -0400)]
8654: Job has uuid and state fields
Peter Amstutz [Mon, 21 Mar 2016 21:02:51 +0000 (17:02 -0400)]
8654: Add args.debug = True to get backtraces on errors
Peter Amstutz [Mon, 21 Mar 2016 21:01:15 +0000 (17:01 -0400)]
8654: Set args.submit = False
Peter Amstutz [Mon, 21 Mar 2016 20:55:12 +0000 (16:55 -0400)]
8654: Add runner job to pipeline instance.
Peter Amstutz [Mon, 21 Mar 2016 20:46:26 +0000 (16:46 -0400)]
8654: Update cwl:tool entry
Peter Amstutz [Mon, 21 Mar 2016 20:44:31 +0000 (16:44 -0400)]
8654: Fix keeppath and add ensure_unique_name to upload.
Peter Amstutz [Mon, 21 Mar 2016 20:24:06 +0000 (16:24 -0400)]
8654: Scanning for and uploading dependencies for submit works. Add test for
--submit. Fix cwl-runner to handle CWL File objects in script_parameters.
Peter Amstutz [Mon, 21 Mar 2016 16:37:38 +0000 (12:37 -0400)]
Merge branch 'master' into 8654-arv-jobs-cwl-runner
Ward Vandewege [Sun, 20 Mar 2016 18:39:52 +0000 (14:39 -0400)]
More small build script improvements.
No issue #
Ward Vandewege [Sun, 20 Mar 2016 17:16:18 +0000 (13:16 -0400)]
Build script improvements:
* move some more functions into run-library.sh
* add a wrapper script for our single target package build/test/upload step
No issue #
Peter Amstutz [Sat, 19 Mar 2016 20:00:11 +0000 (16:00 -0400)]
Enable BaseComputeNodeDriver.list_nodes to pass through keyword arguments.
closes #8761
Brett Smith [Fri, 18 Mar 2016 18:50:59 +0000 (14:50 -0400)]
Merge branch '8002-no-pyvcf-wip'
Closes #8002, #8759.
Brett Smith [Fri, 18 Mar 2016 18:37:13 +0000 (14:37 -0400)]
8002: Stop backporting PyVCF.
Arvados proper doesn't use it. We only backported it as a convenience
previously.
Brett Smith [Fri, 18 Mar 2016 15:31:00 +0000 (11:31 -0400)]
Merge branch '7852-readline-cache-fix-wip'
Closes #7852, #8755.
Brett Smith [Thu, 17 Mar 2016 22:03:21 +0000 (18:03 -0400)]
7852: Bugfix _filepos tracking in PySDK readline method.
Make sure self._filepos is always in the right position to return
uncached data before another read method could be called, to avoid
inconsistent results when interleaving those calls with readline
calls.
radhika [Wed, 3 Feb 2016 17:19:54 +0000 (12:19 -0500)]
7852: Added one more test "test_readline_then_readall" which does a readline followed by a readall.
This test also fails exactly like the test "test_readline_then_readlines".
radhika [Mon, 25 Jan 2016 21:37:24 +0000 (16:37 -0500)]
7852: add a failing test that does: readline and then readlines. The expectation would be the second call,
readlines, would fetch all the remaining lines after the first line. However, the readlines call is only
fetching the data after the "first block" which was read during the first readline call. It appears that
self._readline_cache and self._filepos are playing a role here, which are set during the first readline.
Peter Amstutz [Fri, 18 Mar 2016 14:21:55 +0000 (10:21 -0400)]
Fix log scanning for "crunchrunner: $(task.outdir)" performed by arvados cwl
runner. no issue #
Ward Vandewege [Fri, 18 Mar 2016 02:19:24 +0000 (22:19 -0400)]
Make the build of the schema_salad packages more quiet.
refs #8671
Tom Clegg [Thu, 17 Mar 2016 22:15:10 +0000 (18:15 -0400)]
gofmt. refs #8754
Peter Amstutz [Thu, 17 Mar 2016 21:22:30 +0000 (17:22 -0400)]
8654: Resource upload WIP
Peter Amstutz [Thu, 17 Mar 2016 20:34:48 +0000 (16:34 -0400)]
Improve exception reporting for arvados-cwl-runner, no issue #
Ward Vandewege [Thu, 17 Mar 2016 19:45:54 +0000 (15:45 -0400)]
Build dependencies for arvados-cwl-runner, automatically.
closes #8671
Brett Smith [Thu, 17 Mar 2016 18:42:00 +0000 (14:42 -0400)]
8752: Skip unreliable Python SDK Keep test.
Refs #8752. See there for more discussion about the circumstances of
failure.
radhika [Thu, 17 Mar 2016 15:30:01 +0000 (11:30 -0400)]
closes #8561
Merge branch '8561-node-pairing'
radhika [Thu, 17 Mar 2016 15:29:44 +0000 (11:29 -0400)]
Merge branch 'master' into 8561-node-pairing
radhika [Thu, 17 Mar 2016 15:28:23 +0000 (11:28 -0400)]
8561: delete redundant "maxDiff = None"
radhika [Thu, 17 Mar 2016 15:17:12 +0000 (11:17 -0400)]
Merge branch '8561-node-pairing' of git.curoverse.com:arvados into 8561-node-pairing
Peter Amstutz [Thu, 17 Mar 2016 14:42:02 +0000 (10:42 -0400)]
8561: Set ec2_instance_id and cloud_factory().node_id to mock cloud/arvados id
instead of constant '2'.
Peter Amstutz [Thu, 17 Mar 2016 13:59:10 +0000 (09:59 -0400)]
Node manager bugfix: late subscribers should get proxy() not _later (which is a tell_proxy())
fixes #8691 note-4
Brett Smith [Thu, 17 Mar 2016 13:57:26 +0000 (09:57 -0400)]
Merge branch '8709-docker-cleaner-volumes-wip'
Closes #8709, #8711.
Brett Smith [Tue, 15 Mar 2016 21:59:16 +0000 (17:59 -0400)]
8709: Docker cleaner cleans container volumes.
This prevents a compute node's Docker partition from filling with
scratch space volumes.
radhika [Wed, 16 Mar 2016 19:46:38 +0000 (15:46 -0400)]
Merge branch 'master' into 8561-node-pairing
radhika [Wed, 16 Mar 2016 19:22:14 +0000 (15:22 -0400)]
closes #8554
Merge branch '8554-trash-untrash-unix-volume'
radhika [Wed, 16 Mar 2016 19:21:10 +0000 (15:21 -0400)]
Merge branch 'master' into 8554-trash-untrash-unix-volume
Tom Clegg [Wed, 16 Mar 2016 16:54:55 +0000 (12:54 -0400)]
8554: Report Walk errors and continue, instead of stopping silently.
Tom Clegg [Wed, 16 Mar 2016 16:41:56 +0000 (12:41 -0400)]
8554: Un-pyramid code
Tom Clegg [Wed, 16 Mar 2016 16:16:15 +0000 (12:16 -0400)]
8554: Dedup Get() checks, add comments, fix up regexp.
Peter Amstutz [Wed, 16 Mar 2016 16:11:34 +0000 (12:11 -0400)]
8561: Fix test_arvados_node_match
Tom Clegg [Wed, 16 Mar 2016 07:43:35 +0000 (03:43 -0400)]
Fix backports dir search for PACKAGE="llfuse==0.41.1".
Also, drop "-v 0.41.1" args from fpm-info.sh: cannot specify version
both here and in run-build-packages.sh.
refs #8345
Peter Amstutz [Wed, 16 Mar 2016 01:48:32 +0000 (21:48 -0400)]
Merge branch '6518-crunch2-dispatch-slurm' closes #6518
Peter Amstutz [Wed, 16 Mar 2016 01:43:52 +0000 (21:43 -0400)]
6518: Use ARVADOS_API_HOST when checking strigger command line in test.
Peter Amstutz [Wed, 16 Mar 2016 01:29:28 +0000 (21:29 -0400)]
6518: Add check for sbatch/strigger command line in test.
Peter Amstutz [Wed, 16 Mar 2016 00:03:40 +0000 (20:03 -0400)]
6518: Add crunch-dispatch-slurm to tests.
Peter Amstutz [Wed, 16 Mar 2016 00:02:30 +0000 (20:02 -0400)]
Merge branch 'master' into 6518-crunch2-dispatch-slurm
Peter Amstutz [Wed, 16 Mar 2016 00:01:19 +0000 (20:01 -0400)]
6518: Merge in tests. Code cleanup around variable naming and comments.
radhika [Tue, 15 Mar 2016 20:23:58 +0000 (16:23 -0400)]
8561: test tweaks
Tom Clegg [Tue, 15 Mar 2016 20:09:20 +0000 (16:09 -0400)]
Merge branch '8491-import-build-tools' refs #8491
Tom Clegg [Tue, 15 Mar 2016 20:09:03 +0000 (16:09 -0400)]
8491: Add default target "make help"
radhika [Tue, 15 Mar 2016 15:27:08 +0000 (11:27 -0400)]
8554: removed testTrashUntrashWithEmptyTrashGoroutine
Peter Amstutz [Tue, 15 Mar 2016 13:34:36 +0000 (09:34 -0400)]
Proxy objects held in node manager dict of booting nodes should be regular proxy(), not
tell_proxy(). fixes #8691
Brett Smith [Tue, 15 Mar 2016 01:05:17 +0000 (21:05 -0400)]
8203/8690: crunch-job uses `srun --nodes=1`.
Using `srun --nodelist=[one node]` is not sufficient to ensure that
the command runs on a single node; only that it will at least be
allocated the requested node. This is documented in the srun man
page, and we've observed additional node allocations in these calls.
When srun allocates additional nodes to the work, this can break later
parsing code in crunch-job.
Prefer --nodes, which sets a hard maximum on the number of nodes used.
We're already using this to run individual tasks (-N1), so it's a
known strategy. Tested live before pushing. Refs #8203, #8690.
radhika [Tue, 15 Mar 2016 00:19:28 +0000 (20:19 -0400)]
8561: rather than getting node's ipadd, use it's ec2_instance_id
Tom Clegg [Mon, 14 Mar 2016 19:54:32 +0000 (15:54 -0400)]
Fix ineffective test case
No issue #
Tom Clegg [Mon, 14 Mar 2016 19:06:02 +0000 (15:06 -0400)]
Merge branch '8079-lookup-token-uuid' closes #8079
Tom Clegg [Mon, 14 Mar 2016 19:04:12 +0000 (15:04 -0400)]
8079: Allow where(api_token: foo) and disallow where(api_token: ["contains", "f"])
Tom Clegg [Sat, 12 Mar 2016 22:32:17 +0000 (17:32 -0500)]
8079: Tidy up and document current_api_client_is_trusted.
Tom Clegg [Sun, 6 Mar 2016 19:50:22 +0000 (14:50 -0500)]
8079: Prevent users from changing their own token UUIDs.
Tom Clegg [Fri, 4 Mar 2016 22:30:53 +0000 (17:30 -0500)]
8079: Prevent users from looking up other users' tokens by UUID.
Previous code was allowing any user logging in through a "trusted
client" (typically Workbench) to retrieve the secret token for any
ApiClientAuthorization whose UUID is known. This won't be acceptable
when Container records start including those UUIDs.
Also added permission for any user to update (e.g., change expiration)
and delete their current token, even if the token wasn't assigned
through a "trusted client".
Peter Amstutz [Mon, 14 Mar 2016 17:19:56 +0000 (13:19 -0400)]
Really fix cwl-runner tests this time. refs #8673
Peter Amstutz [Mon, 14 Mar 2016 16:21:31 +0000 (12:21 -0400)]
Fix cwl-runner tests. refs #8673
Peter Amstutz [Mon, 14 Mar 2016 14:20:11 +0000 (10:20 -0400)]
Merge branch '8673-cwl-runner-project-uuid' closes #8673
radhika [Mon, 14 Mar 2016 14:02:50 +0000 (10:02 -0400)]
8554: test improvements
projects / arvados.git / log