Tom Clegg [Thu, 26 Jan 2017 19:37:54 +0000 (14:37 -0500)]
10704: Rate-limit startup attempts per container.
Tom Clegg [Wed, 25 Jan 2017 21:37:26 +0000 (16:37 -0500)]
10703: Un-pyramid run().
Tom Clegg [Wed, 25 Jan 2017 21:04:30 +0000 (16:04 -0500)]
10701: Remove unneeded complexity in squeue invocation.
Tom Clegg [Wed, 25 Jan 2017 15:59:48 +0000 (10:59 -0500)]
10703: Do not catch signals in crunch-dispatch-slurm. Simplify "stop dispatcher loop" API.
Tom Clegg [Tue, 24 Jan 2017 20:28:37 +0000 (15:28 -0500)]
Merge branch '9998-no-count-items-available'
closes #9998
Tom Clegg [Tue, 24 Jan 2017 20:28:04 +0000 (15:28 -0500)]
9998: Update CLI usage.
Tom Clegg [Tue, 24 Jan 2017 20:18:52 +0000 (15:18 -0500)]
9998: Remove superfluous word.
Tom Clegg [Tue, 24 Jan 2017 20:17:00 +0000 (15:17 -0500)]
Merge branch '10868-job-log-node-info'
refs #10868
Tom Clegg [Tue, 24 Jan 2017 18:51:40 +0000 (13:51 -0500)]
9998: Use links instead of specimens to test count param.
Tom Clegg [Tue, 24 Jan 2017 18:42:30 +0000 (13:42 -0500)]
9998: Update API docs.
Tom Clegg [Tue, 24 Jan 2017 03:47:57 +0000 (22:47 -0500)]
10868: Include node UUIDs and properties in job logs.
Tom Clegg [Mon, 23 Jan 2017 20:31:23 +0000 (15:31 -0500)]
9998: Handle timestamp collisions in collection index.
Tom Clegg [Mon, 23 Jan 2017 15:41:24 +0000 (10:41 -0500)]
Merge branch 'master' into 9998-no-count-items-available
Peter Amstutz [Mon, 23 Jan 2017 15:32:05 +0000 (10:32 -0500)]
Merge branch '10812-cwl-separate-dependencies' refs #10812
Peter Amstutz [Mon, 23 Jan 2017 15:20:46 +0000 (10:20 -0500)]
10812: Handle expected NotFoundError error when checking if file exists in keep.
Peter Amstutz [Mon, 23 Jan 2017 14:52:34 +0000 (09:52 -0500)]
10812: Add num_retries to CollectionFetcher and collectionResolver.
Peter Amstutz [Fri, 20 Jan 2017 16:36:05 +0000 (11:36 -0500)]
10812: Fix indentation typo.
Peter Amstutz [Thu, 19 Jan 2017 18:29:11 +0000 (13:29 -0500)]
10812: Update cwltool and schema-salad dependencies.
Peter Amstutz [Wed, 18 Jan 2017 22:46:54 +0000 (17:46 -0500)]
10812: Handle $schema references.
Peter Amstutz [Wed, 18 Jan 2017 16:47:07 +0000 (11:47 -0500)]
10812: Don't use cStringIO which doesn't like unicode strings.
Peter Amstutz [Mon, 16 Jan 2017 15:51:10 +0000 (10:51 -0500)]
10812: Check for dockerOutputDirectory and raise UnsupportedRequirement up front.
Peter Amstutz [Mon, 16 Jan 2017 15:22:30 +0000 (10:22 -0500)]
10812: Fix imports, reference #main in cwl:tool
Peter Amstutz [Wed, 11 Jan 2017 21:33:11 +0000 (16:33 -0500)]
10812: Improve check that already packed workflow collection exists with same
PDH and similar name. Don't crash when registering pipeline template that has
array inputs.
Peter Amstutz [Wed, 11 Jan 2017 20:51:54 +0000 (15:51 -0500)]
10812: Don't try to upload embedded tools separately unless they have an
identifier.
Peter Amstutz [Wed, 11 Jan 2017 13:41:08 +0000 (08:41 -0500)]
10812: Handle workflow keep references.
Peter Amstutz [Mon, 23 Jan 2017 15:31:08 +0000 (10:31 -0500)]
10812: Use packed workflows for all run modes.
Conflicts:
sdk/cwl/arvados_cwl/__init__.py
sdk/cwl/arvados_cwl/runner.py
Peter Amstutz [Mon, 23 Jan 2017 15:29:01 +0000 (10:29 -0500)]
Merge branch '10812-log-merge' refs #10812
Peter Amstutz [Mon, 23 Jan 2017 15:28:39 +0000 (10:28 -0500)]
Merge branch '10812-submit-runner-image' refs #10812
Peter Amstutz [Mon, 23 Jan 2017 15:28:06 +0000 (10:28 -0500)]
Merge branch '10935-noop-munge' closes #10935
Peter Amstutz [Fri, 20 Jan 2017 17:25:51 +0000 (12:25 -0500)]
10812: Add tests
Peter Amstutz [Thu, 12 Jan 2017 22:05:23 +0000 (17:05 -0500)]
10812: Perform merge of logs based on timestamp instead of sorting log lines.
Peter Amstutz [Fri, 20 Jan 2017 16:21:50 +0000 (11:21 -0500)]
10935: Add test that round tripping works after disabling deep_munge.
Peter Amstutz [Fri, 20 Jan 2017 14:46:50 +0000 (09:46 -0500)]
10812: Add --submit-runner-image to override default Docker image when submitting workflow runner.
Conflicts:
sdk/cwl/arvados_cwl/__init__.py
sdk/cwl/arvados_cwl/runner.py
Tom Clegg [Thu, 19 Jan 2017 20:51:18 +0000 (15:51 -0500)]
Merge branch 'master' into 9998-no-count-items-available
Tom Clegg [Thu, 19 Jan 2017 20:50:55 +0000 (15:50 -0500)]
9998: Change count=true/false to exact/none. Add tests.
Peter Amstutz [Thu, 19 Jan 2017 19:45:39 +0000 (14:45 -0500)]
10935: Make Rails method deep_munge a noop, so that empty arrays can round trip
through json instead of being turned into 'nil'.
Tom Clegg [Mon, 16 Jan 2017 16:13:32 +0000 (11:13 -0500)]
Merge branch 'master' into 9998-no-count-items-available
Conflicts:
sdk/go/arvados/resource_list.go
services/keep-balance/collection.go
radhika [Fri, 13 Jan 2017 03:46:50 +0000 (22:46 -0500)]
closes #5666
Merge branch '5666-wu-cancel-button-style'
Tom Clegg [Wed, 18 Jan 2017 22:29:13 +0000 (17:29 -0500)]
10883: scancel slurm job when erroring out early.
refs #10883
Lucas Di Pentima [Wed, 18 Jan 2017 21:32:30 +0000 (18:32 -0300)]
Merge branch '10813-arv-put-six-threads'
Refs #10813
Lucas Di Pentima [Wed, 18 Jan 2017 20:14:01 +0000 (17:14 -0300)]
10813: Merge branch 'master' into 10813-arv-put-six-threads
Lucas Di Pentima [Wed, 18 Jan 2017 16:29:46 +0000 (13:29 -0300)]
10813: Fixed typo on argument documentation.
Added only_committed parameter to Collection.manifest_text() so it passes along to _get_manifest_text().
Corrected put_queue's size back to 2 on BlockManager.
Peter Amstutz [Wed, 18 Jan 2017 15:04:33 +0000 (10:04 -0500)]
Merge branch '10292-cwl-deadlock' refs #10292
Peter Amstutz [Wed, 18 Jan 2017 14:57:25 +0000 (09:57 -0500)]
10292: Update cwltool for bugfix: intermittant workflow deadlock on subworkflow completion.
Lucas Di Pentima [Wed, 18 Jan 2017 14:14:59 +0000 (11:14 -0300)]
10813: Bug fixed on BlockManager's _get_manifest_text() when asked for only include committed blocks on collections with subcollections inside it.
Also, added --threads parameter to arv-put command so the default number of upload threads (2) can be overridden.
Changed the update thread to execute once every minute, because it blocks the local collection for a few seconds (depending on the manifest size)
Also, cleaned up some locking uses on the update thread to minimize the time with the locking in place.
Ward Vandewege [Wed, 18 Jan 2017 03:26:04 +0000 (22:26 -0500)]
Remove references to debian7 and centos6. Also remove some hardcoded
--iteration flags for fpm, now that #6885 is done.
refs #10785
Ward Vandewege [Wed, 18 Jan 2017 02:54:58 +0000 (21:54 -0500)]
Merge branch '10858-build-packages-improvements'
closes #10858
Ward Vandewege [Wed, 18 Jan 2017 02:35:54 +0000 (21:35 -0500)]
10858: Equivalent commit for 10881: Add python-futures as explicit
cwltest dependency and bump iteration.
refs #10858
Ward Vandewege [Wed, 18 Jan 2017 02:34:34 +0000 (21:34 -0500)]
Fix rpm architecture in the test_package_presence function. Expand on
comment.
refs #10858
Ward Vandewege [Wed, 18 Jan 2017 01:47:56 +0000 (20:47 -0500)]
Merge branch '10800-update-doc'
closes #10800
Ward Vandewege [Wed, 18 Jan 2017 01:46:44 +0000 (20:46 -0500)]
Merge branch 'master' into 10800-update-doc
refs #10800
Ward Vandewege [Wed, 18 Jan 2017 01:43:07 +0000 (20:43 -0500)]
More documentation updates to reflect the end of support for Debian
Wheezy and CentOS 6. Also remove obsolete arv-docker installation page,
which was no longer linked from anywhere.
refs #10800
Ward Vandewege [Tue, 17 Jan 2017 18:01:00 +0000 (13:01 -0500)]
Documentation updates to reflect the end of support for Debian Wheezy and CentOS 6.
refs #10800
Tom Clegg [Tue, 17 Jan 2017 16:01:42 +0000 (11:01 -0500)]
Merge branch '10868-log-dispatch-decisions'
refs #10868
Tom Clegg [Mon, 16 Jan 2017 21:53:16 +0000 (16:53 -0500)]
10868: Add fake uuid to test case for more realistic log messages.
Tom Clegg [Mon, 16 Jan 2017 21:38:29 +0000 (16:38 -0500)]
10868: Log job runtime constraints and node properties when making a dispatch decision.
Peter Amstutz [Mon, 16 Jan 2017 18:11:19 +0000 (13:11 -0500)]
Bump libcloud pin to 0.20.2.dev3 for Azure China support. refs #10833
Ward Vandewege [Sat, 14 Jan 2017 14:57:35 +0000 (09:57 -0500)]
If the fpm run fails, always print out the failed command so that we
don't have to run the build scripts in debug mode all the time.
refs #10858
Ward Vandewege [Sat, 14 Jan 2017 14:56:53 +0000 (09:56 -0500)]
Fix the shortcut that tests the need of rebuilding the arvados-src
package.
refs #10858
Ward Vandewege [Sat, 14 Jan 2017 03:54:34 +0000 (22:54 -0500)]
Handle rpm-based backports correctly.
refs #10858
Ward Vandewege [Sat, 14 Jan 2017 03:38:38 +0000 (22:38 -0500)]
Merge branch 'master' into 10858-build-packages-improvements
refs #10858
Ward Vandewege [Sat, 14 Jan 2017 03:36:12 +0000 (22:36 -0500)]
Refactor the build-packages scripts a bit to make them much more
efficient when there is nothing to do.
refs #10858
Lucas Di Pentima [Fri, 13 Jan 2017 18:40:00 +0000 (15:40 -0300)]
10813: arv-put with 6 upload threads, to be tested on production.
Peter Amstutz [Fri, 13 Jan 2017 15:44:45 +0000 (10:44 -0500)]
Merge branch '10881-build-python-futures-package' refs #10881
Peter Amstutz [Fri, 13 Jan 2017 15:43:48 +0000 (10:43 -0500)]
10881: Add python-futures as explicit cwltest dependency and bump iteration.
radhika [Fri, 13 Jan 2017 03:45:56 +0000 (22:45 -0500)]
5666: Use "warning" bootstrap style for Cancel button display.
Ward Vandewege [Fri, 13 Jan 2017 02:53:24 +0000 (21:53 -0500)]
Build the python-futures package, version 3.0.5, for our supported
distributions.
refs #10881
Tom Clegg [Fri, 13 Jan 2017 02:52:32 +0000 (21:52 -0500)]
Merge branch '10880-poll-test-races'
closes #10880
Tom Clegg [Fri, 13 Jan 2017 02:46:38 +0000 (21:46 -0500)]
10880: Fix races in event-polling tests.
The polling tests were broken because the API stub was returning its
fake events during the "find recent log ID" probe, so they were being
thrown away by PollClient and never sent to the test case's callback
function.
Most of the tests kept passing anyway, though, because subscribe()
itself sends a synthetic {'status':200} event, and this was being
misconstrued as confirmation that an event had been retrieved from the
API stub.
The false positive was not reliable for test_callback(), though,
because it cleared callback_called and tested another event without
calling subscribe() again. Sometimes, because the first "wait for
event" wasn't really waiting, the test would feed the second event to
the stub before PollClient had a chance to do its ID probe, so both
events would be dropped and the second "wait for event" would fail.
With the "wait for event" code fixed, it is no longer necessary to
close the poll client before checking which filters were used to
retrieve events.
radhika [Fri, 13 Jan 2017 01:59:34 +0000 (20:59 -0500)]
closes #10110
Merge branch '10110-arv-copy-workflow'
radhika [Fri, 13 Jan 2017 01:33:06 +0000 (20:33 -0500)]
10110: add ruamel.yaml to setup.py
radhika [Wed, 4 Jan 2017 12:50:20 +0000 (07:50 -0500)]
10110: also copy any docker images during arv-copy of a workflow and update documentation.
radhika [Wed, 4 Jan 2017 09:20:01 +0000 (04:20 -0500)]
10110: arv-copy workflows
radhika [Fri, 13 Jan 2017 01:42:25 +0000 (20:42 -0500)]
closes #10516
Merge branch '10516-workbench-stats-logic'
Tom Clegg [Fri, 13 Jan 2017 00:08:37 +0000 (19:08 -0500)]
Merge branch '10877-azure-base-url'
closes #10877
Tom Clegg [Thu, 12 Jan 2017 23:40:59 +0000 (18:40 -0500)]
10877: Add example using core.chinacloudapi.cn.
radhika [Thu, 12 Jan 2017 23:17:41 +0000 (18:17 -0500)]
10516: update work_unit test to compare cputime and walltime.
radhika [Fri, 13 Jan 2017 00:28:51 +0000 (19:28 -0500)]
10516: update test to expect 0 cputime, instead of nil.
radhika [Thu, 12 Jan 2017 23:44:18 +0000 (18:44 -0500)]
10516: compute cputime recursively using cputime of any children.
Tom Clegg [Thu, 12 Jan 2017 23:03:30 +0000 (18:03 -0500)]
10877: Configurable base URL for Azure storage, e.g., StorageBaseURL: core.usgovcloudapi.net
Ward Vandewege [Wed, 11 Jan 2017 00:36:08 +0000 (19:36 -0500)]
run-build-packages-one-target.sh should respect the --debug flag.
refs #10858
Ward Vandewege [Mon, 9 Jan 2017 21:52:18 +0000 (16:52 -0500)]
Pin the python-pathlib2 dependency to 2.1.0, because 2.2 introduces more
dependencies and we've already got enough of those.
No issue #
Peter Amstutz [Mon, 9 Jan 2017 15:13:11 +0000 (10:13 -0500)]
Merge branch 'arvbox-tmp-cache' no issue #
Tom Clegg [Mon, 9 Jan 2017 15:12:08 +0000 (10:12 -0500)]
Fix run-tests.sh --only-install option.
No issue #
Peter Amstutz [Mon, 9 Jan 2017 15:06:04 +0000 (10:06 -0500)]
Create tmp/cache dir after cleaning old one.
Ward Vandewege [Fri, 6 Jan 2017 22:31:50 +0000 (17:31 -0500)]
Bump up rails package iteration.
refs #10838
Ward Vandewege [Fri, 6 Jan 2017 19:17:20 +0000 (14:17 -0500)]
Fix run-tests.sh --only option for the workbench suite.
refs #8345
Peter Amstutz [Fri, 6 Jan 2017 18:41:11 +0000 (13:41 -0500)]
Merge branch '10793-cwl-continue-on-error' closes #10793
Peter Amstutz [Fri, 6 Jan 2017 18:40:32 +0000 (13:40 -0500)]
10793: Ensure on_error is set if not present in job_order_object.
Ward Vandewege [Fri, 6 Jan 2017 18:39:55 +0000 (13:39 -0500)]
Merge branch '10838-fix-rails-package-postinst'
closes #10838
Ward Vandewege [Fri, 6 Jan 2017 18:29:00 +0000 (13:29 -0500)]
In our rails postinstall package script, preserve newlines when looking
for the '^database:' line, so that we match every line of the rake
db:migrate:status output, not just the first one.
refs #10838
Lucas Di Pentima [Thu, 5 Jan 2017 22:05:36 +0000 (19:05 -0300)]
Merge branch '10223-cr-set-output-name'
Closes #10223
Lucas Di Pentima [Thu, 5 Jan 2017 21:23:03 +0000 (18:23 -0300)]
10223: Added CR's output_name to FTS index
Lucas Di Pentima [Thu, 5 Jan 2017 20:59:28 +0000 (17:59 -0300)]
10223: Added test to check for missing full text search indexes
Peter Amstutz [Thu, 5 Jan 2017 20:29:57 +0000 (15:29 -0500)]
10793: Fix to return tuple for alternate exit points --create/update-workflow and --no-wait
Tom Clegg [Thu, 5 Jan 2017 17:42:22 +0000 (12:42 -0500)]
Merge branch '10808-admin-cancel-job'
refs #10808
Tom Clegg [Thu, 5 Jan 2017 17:42:04 +0000 (12:42 -0500)]
10808: Move comment.
Peter Amstutz [Thu, 5 Jan 2017 16:17:57 +0000 (11:17 -0500)]
10793: Move output_callback to finally block to ensure it is always called.
Tom Clegg [Thu, 5 Jan 2017 15:55:53 +0000 (10:55 -0500)]
10808: Exempt "change state to Cancelled" from "locked by uuid" protection.
Lucas Di Pentima [Thu, 5 Jan 2017 15:54:30 +0000 (12:54 -0300)]
10223: Added relevant explanation on the rollback command