arvados.git
7 years ago11220: Fix deprecated assertEquals -> assertEqual.
Tom Clegg [Fri, 20 Oct 2017 02:30:40 +0000 (22:30 -0400)]
11220: Fix deprecated assertEquals -> assertEqual.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago11220: Don't bother trying to read manifests from Keep.
Tom Clegg [Fri, 20 Oct 2017 02:30:35 +0000 (22:30 -0400)]
11220: Don't bother trying to read manifests from Keep.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12461-cache-race'
Tom Clegg [Thu, 19 Oct 2017 18:15:47 +0000 (14:15 -0400)]
Merge branch '12461-cache-race'

refs #12461

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoFix wrong HTTP status code for early "impossible path" check.
Tom Clegg [Thu, 19 Oct 2017 18:12:07 +0000 (14:12 -0400)]
Fix wrong HTTP status code for early "impossible path" check.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12467-read-imgload-response' closes #12467
Peter Amstutz [Thu, 19 Oct 2017 18:09:34 +0000 (14:09 -0400)]
Merge branch '12467-read-imgload-response' closes #12467

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12467: Fix log messages. Defer response.Body.Close().
Peter Amstutz [Thu, 19 Oct 2017 17:59:24 +0000 (13:59 -0400)]
12467: Fix log messages.  Defer response.Body.Close().

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12467: Wait for response that image loading is complete
Peter Amstutz [Thu, 19 Oct 2017 17:35:23 +0000 (13:35 -0400)]
12467: Wait for response that image loading is complete

Need to do this before trying to start container.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12446-dispatcher-query' closes #12446
Peter Amstutz [Thu, 19 Oct 2017 15:41:40 +0000 (11:41 -0400)]
Merge branch '12446-dispatcher-query' closes #12446

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12446: Avoid listing every container uuid in status query.
Peter Amstutz [Tue, 17 Oct 2017 14:30:42 +0000 (10:30 -0400)]
12446: Avoid listing every container uuid in status query.

7 years ago12461: Avoid race bug in Rails file cache.
Tom Clegg [Wed, 18 Oct 2017 19:37:54 +0000 (15:37 -0400)]
12461: Avoid race bug in Rails file cache.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago11260: Fix tests.
Tom Clegg [Tue, 17 Oct 2017 19:49:14 +0000 (15:49 -0400)]
11260: Fix tests.

refs #11260

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoComment out sdk/go/health from run-tests.sh until it is fixed.
Peter Amstutz [Wed, 18 Oct 2017 13:38:56 +0000 (09:38 -0400)]
Comment out sdk/go/health from run-tests.sh until it is fixed.

refs #12260

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12452-nm-unpaired-down' refs #12452
Peter Amstutz [Tue, 17 Oct 2017 19:23:25 +0000 (15:23 -0400)]
Merge branch '12452-nm-unpaired-down' refs #12452

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12452: Fix test
Peter Amstutz [Tue, 17 Oct 2017 14:50:23 +0000 (10:50 -0400)]
12452: Fix test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12260-system-health'
Tom Clegg [Tue, 17 Oct 2017 14:24:19 +0000 (10:24 -0400)]
Merge branch '12260-system-health'

refs #12260

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Build arvados-health package.
Tom Clegg [Tue, 17 Oct 2017 14:23:37 +0000 (10:23 -0400)]
12260: Build arvados-health package.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Merge branch 'master' into 12260-system-health
Tom Clegg [Tue, 17 Oct 2017 13:06:50 +0000 (09:06 -0400)]
12260: Merge branch 'master' into 12260-system-health

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12452: Consider unpaired node older than boot grace period 'down'.
Peter Amstutz [Mon, 16 Oct 2017 19:50:36 +0000 (15:50 -0400)]
12452: Consider unpaired node older than boot grace period 'down'.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12447-crunch-run-leak' closes #12447
Peter Amstutz [Mon, 16 Oct 2017 18:01:50 +0000 (14:01 -0400)]
Merge branch '12447-crunch-run-leak' closes #12447

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12447: Simplify BlockCache locking.
Peter Amstutz [Mon, 16 Oct 2017 17:13:44 +0000 (13:13 -0400)]
12447: Simplify BlockCache locking.

Close response body when content-length is missing.

Tweak messages related to crunch-run -memprofile.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12447: Require Content-Length in KeepClient.Get
Peter Amstutz [Mon, 16 Oct 2017 17:00:55 +0000 (13:00 -0400)]
12447: Require Content-Length in KeepClient.Get

Fix test stub to report Content-Length.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12260: Update comments to match real func/type names.
Tom Clegg [Mon, 16 Oct 2017 14:44:54 +0000 (10:44 -0400)]
12260: Update comments to match real func/type names.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12447: Fix typo shadowing 'data' in BlockCache.Get
Peter Amstutz [Mon, 16 Oct 2017 14:41:18 +0000 (10:41 -0400)]
12447: Fix typo shadowing 'data' in BlockCache.Get

Move locking to BlockCache.setup()

Tweak tests to bind explicitly to loopback address.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12447: Remove .Bytes()
Peter Amstutz [Sat, 14 Oct 2017 16:47:42 +0000 (12:47 -0400)]
12447: Remove .Bytes()

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12447: Use ReadFull(). Use goimport for import formatting.
Peter Amstutz [Sat, 14 Oct 2017 15:58:14 +0000 (11:58 -0400)]
12447: Use ReadFull().  Use goimport for import formatting.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12447: Add ClearBlockCache method to KeepClient
Peter Amstutz [Fri, 13 Oct 2017 13:50:30 +0000 (09:50 -0400)]
12447: Add ClearBlockCache method to KeepClient

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12447: Reduce memory footprint of crunch-run.
Peter Amstutz [Fri, 13 Oct 2017 01:13:17 +0000 (21:13 -0400)]
12447: Reduce memory footprint of crunch-run.

* BlockCache reads into a fixed-capacity buffer, to avoid reallocations (this
  cuts resident memory usage by 1/2 and makes memory footprint extremely
  stable)
* Limit the crunch-run BlockCache to 2 blocks instead of 4 (cuts footprint by
  1/2 again)
* Clear the crunch-run BlockCache after loading the Docker image (probably cuts
  post-image-load footprint by 1/2 again).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12431-nm-delete-nic' refs #12431
Peter Amstutz [Thu, 12 Oct 2017 16:59:18 +0000 (12:59 -0400)]
Merge branch '12431-nm-delete-nic' refs #12431

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12431: Fix test
Peter Amstutz [Thu, 12 Oct 2017 16:58:35 +0000 (12:58 -0400)]
12431: Fix test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12416-s3-empty-object'
Tom Clegg [Thu, 12 Oct 2017 13:23:42 +0000 (09:23 -0400)]
Merge branch '12416-s3-empty-object'

refs #12416

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12416: Merge Put() into PutReader().
Tom Clegg [Wed, 11 Oct 2017 20:28:43 +0000 (16:28 -0400)]
12416: Merge Put() into PutReader().

Put() was only used for empty blocks, and passing a nil reader to
PutReader works just as well.

This ensures the "put empty block" fix applies to all writes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12416: Merge branch 'fix/keepstore-s3-radosgw-empty-object' of https://github.com...
Tom Clegg [Thu, 12 Oct 2017 13:14:26 +0000 (09:14 -0400)]
12416: Merge branch 'fix/keepstore-s3-radosgw-empty-object' of https://github.com/wtsi-hgi/arvados into 12416-s3-empty-object

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agofix s3_volume so that it can create empty recent/* objects on Ceph RadosGW
Joshua C. Randall [Sun, 8 Oct 2017 12:10:33 +0000 (12:10 +0000)]
fix s3_volume so that it can create empty recent/* objects on Ceph RadosGW

Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <jcrandall@alum.mit.edu>

7 years ago12431: Use libcloud fork 2.2.2.dev1
Peter Amstutz [Wed, 11 Oct 2017 17:03:20 +0000 (13:03 -0400)]
12431: Use libcloud fork 2.2.2.dev1

Should address issue deleting NICs.  Also add ex_fetch_power_state=False to
reduce Azure API calls.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12418-glob-empty-collection' closes #12418
Peter Amstutz [Tue, 10 Oct 2017 18:04:33 +0000 (14:04 -0400)]
Merge branch '12418-glob-empty-collection' closes #12418

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12418: Add regression test for glob output on empty collection.
Peter Amstutz [Tue, 10 Oct 2017 18:03:55 +0000 (14:03 -0400)]
12418: Add regression test for glob output on empty collection.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12418: Explicitly check if collection is not None
Peter Amstutz [Tue, 10 Oct 2017 15:19:47 +0000 (11:19 -0400)]
12418: Explicitly check if collection is not None

The empty collection evaluates to False, so it is necessary to explicitly check
for "not None".

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agopin ciso8601 >=1.0.0, <=1.0.4 closes #12422
Peter Amstutz [Tue, 10 Oct 2017 17:09:38 +0000 (13:09 -0400)]
pin ciso8601 >=1.0.0, <=1.0.4 closes #12422

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoUpdate BytesAllocated -> BytesAllocatedCumulative in status report.
Tom Clegg [Fri, 6 Oct 2017 14:15:39 +0000 (10:15 -0400)]
Update BytesAllocated -> BytesAllocatedCumulative in status report.

The number reported is the total of all buffer allocations since
startup. It doesn't take into account how many of those buffers have
since been freed.

refs #12402

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Add more services. Use existing package names.
Tom Clegg [Fri, 6 Oct 2017 03:22:09 +0000 (23:22 -0400)]
12260: Add more services. Use existing package names.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12355-make-cwl-conformance-tests-faster'
Ward Vandewege [Thu, 5 Oct 2017 15:17:35 +0000 (11:17 -0400)]
Merge branch '12355-make-cwl-conformance-tests-faster'

closes #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoMerge branch 'master' into 12355-make-cwl-conformance-tests-faster
Ward Vandewege [Thu, 5 Oct 2017 15:17:08 +0000 (11:17 -0400)]
Merge branch 'master' into 12355-make-cwl-conformance-tests-faster

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoarvbox: use overlay2 inside arvbox.
Ward Vandewege [Wed, 4 Oct 2017 18:49:26 +0000 (14:49 -0400)]
arvbox: use overlay2 inside arvbox.

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoarvbox: run the rails code in production mode when running in localdemo or
Ward Vandewege [Wed, 4 Oct 2017 15:45:09 +0000 (11:45 -0400)]
arvbox: run the rails code in production mode when running in localdemo or
publicdemo.

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoAdd missing test files refs #12213
Peter Amstutz [Wed, 4 Oct 2017 01:06:58 +0000 (21:06 -0400)]
Add missing test files refs #12213

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoarvbox: containers api: make crunch-dispatch-local poll more agressively
Ward Vandewege [Wed, 4 Oct 2017 00:44:43 +0000 (20:44 -0400)]
arvbox: containers api: make crunch-dispatch-local poll more agressively

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoarvbox: jobs api: crunch-dispatch: stop excessive stat of /etc/localtime
Ward Vandewege [Wed, 4 Oct 2017 00:43:13 +0000 (20:43 -0400)]
arvbox: jobs api: crunch-dispatch: stop excessive stat of /etc/localtime

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoarvbox: bump ruby version up to 2.3.5
Ward Vandewege [Wed, 4 Oct 2017 00:42:00 +0000 (20:42 -0400)]
arvbox: bump ruby version up to 2.3.5

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoRun cwl conformance tests with the containers api.
Ward Vandewege [Wed, 4 Oct 2017 00:39:07 +0000 (20:39 -0400)]
Run cwl conformance tests with the containers api.

refs #12355

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years ago12260: Add -config arg. Drop non-resolvable hostname support.
Tom Clegg [Tue, 3 Oct 2017 19:15:20 +0000 (15:15 -0400)]
12260: Add -config arg. Drop non-resolvable hostname support.

Change check key to svctype+http://host:port/path.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '8333-docker-repo-tag'
Tom Clegg [Tue, 3 Oct 2017 14:56:39 +0000 (10:56 -0400)]
Merge branch '8333-docker-repo-tag'

closes #8333

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Fix remote ping auth. Make server work.
Tom Clegg [Tue, 3 Oct 2017 14:55:42 +0000 (10:55 -0400)]
12260: Fix remote ping auth. Make server work.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12213-expression-keepref' closes #12213
Peter Amstutz [Tue, 3 Oct 2017 13:25:10 +0000 (09:25 -0400)]
Merge branch '12213-expression-keepref' closes #12213

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago8333: Support "myrepo.example:8888/repo/image"
Tom Clegg [Tue, 3 Oct 2017 13:15:24 +0000 (09:15 -0400)]
8333: Support "myrepo.example:8888/repo/image"

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago8333: Support "myrepo.example:8888/repo/image:tag"
Tom Clegg [Tue, 3 Oct 2017 13:03:33 +0000 (09:03 -0400)]
8333: Support "myrepo.example:8888/repo/image:tag"

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Improve data structures.
Tom Clegg [Tue, 3 Oct 2017 02:36:35 +0000 (22:36 -0400)]
12260: Improve data structures.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Test case for one unhealthy service.
Tom Clegg [Mon, 2 Oct 2017 21:17:29 +0000 (17:17 -0400)]
12260: Test case for one unhealthy service.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12260: Health check aggregator.
Tom Clegg [Fri, 29 Sep 2017 03:28:13 +0000 (23:28 -0400)]
12260: Health check aggregator.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12213: Check for correct test output.
Peter Amstutz [Mon, 2 Oct 2017 18:25:42 +0000 (14:25 -0400)]
12213: Check for correct test output.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12073-nodemanager-stale-nodes-recs-bis'
Lucas Di Pentima [Mon, 2 Oct 2017 14:24:59 +0000 (11:24 -0300)]
Merge branch '12073-nodemanager-stale-nodes-recs-bis'
Closes #12073

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years ago12073: Prioritize stale node records that have a slot_number when
Lucas Di Pentima [Sat, 30 Sep 2017 03:02:19 +0000 (00:02 -0300)]
12073: Prioritize stale node records that have a slot_number when
trying to assign one to a cloud node.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years ago12213: Remapping from source to destination collection needs to apply to files as...
Peter Amstutz [Thu, 28 Sep 2017 19:15:10 +0000 (15:15 -0400)]
12213: Remapping from source to destination collection needs to apply to files as well.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12213: Add test case.
Peter Amstutz [Thu, 28 Sep 2017 18:44:48 +0000 (14:44 -0400)]
12213: Add test case.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12278-cwl-debug-flag' closes #12278
Peter Amstutz [Fri, 29 Sep 2017 20:33:12 +0000 (16:33 -0400)]
Merge branch '12278-cwl-debug-flag' closes #12278

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12273-skip-special-files'
Tom Clegg [Fri, 29 Sep 2017 18:24:15 +0000 (14:24 -0400)]
Merge branch '12273-skip-special-files'

closes #12273

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago8333: Support "arv keep docker repo:tag" syntax.
Tom Clegg [Fri, 29 Sep 2017 18:20:54 +0000 (14:20 -0400)]
8333: Support "arv keep docker repo:tag" syntax.

"arv keep docker repo tag" still works, too.

"arv keep docker repo:tag tag" is an error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12341-slow-node-destroy'
Lucas Di Pentima [Fri, 29 Sep 2017 18:16:08 +0000 (15:16 -0300)]
Merge branch '12341-slow-node-destroy'
Closes #12341

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years ago12341: Ensure shutdown_actor.stop() is called on node_finished_shutdown.
Lucas Di Pentima [Fri, 29 Sep 2017 14:37:17 +0000 (11:37 -0300)]
12341: Ensure shutdown_actor.stop() is called on node_finished_shutdown.
Updated test related to shutdown race condition.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years agoMerge branch '12347-obey-wf-reuse-hint'
Tom Clegg [Fri, 29 Sep 2017 12:56:35 +0000 (08:56 -0400)]
Merge branch '12347-obey-wf-reuse-hint'

closes #12347

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12347: Don't reuse runner job/container if arv:enableReuse is false.
Tom Clegg [Thu, 28 Sep 2017 03:09:16 +0000 (23:09 -0400)]
12347: Don't reuse runner job/container if arv:enableReuse is false.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12341: If a node_destroy call succesfully destroys the VM but takes too long
Lucas Di Pentima [Thu, 28 Sep 2017 21:09:09 +0000 (18:09 -0300)]
12341: If a node_destroy call succesfully destroys the VM but takes too long
to finish with additional tasks, the cloud_node_update may be called
in the meantime, removing the node from the cloud_node list. If this happens,
node_finished_shutdown() won't be able to retrieve the node, and shouldn't
raise an exception.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years agoMerge branch '12347-disable-reuse'
Tom Clegg [Thu, 28 Sep 2017 20:54:28 +0000 (16:54 -0400)]
Merge branch '12347-disable-reuse'

refs #12347

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago11583: Fix hung goroutines when stubbed container never runs.
Tom Clegg [Thu, 28 Sep 2017 18:52:01 +0000 (14:52 -0400)]
11583: Fix hung goroutines when stubbed container never runs.

Test suite memory usage goes down from >8GB to 1.5GB.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago11583: Stop signal handlers on shutdown.
Tom Clegg [Thu, 28 Sep 2017 18:19:55 +0000 (14:19 -0400)]
11583: Stop signal handlers on shutdown.

Reduces noise in goroutine dumps during test suite.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12273: Skip non-regular files when uploading outputs to Keep.
Tom Clegg [Thu, 28 Sep 2017 17:57:21 +0000 (13:57 -0400)]
12273: Skip non-regular files when uploading outputs to Keep.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12278: Need to set debug logging in crunch_script.py
Peter Amstutz [Thu, 28 Sep 2017 16:52:44 +0000 (12:52 -0400)]
12278: Need to set debug logging in crunch_script.py

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12278: Bump schema-salad and cwltool for SourceLine fix.
Peter Amstutz [Thu, 28 Sep 2017 16:12:33 +0000 (12:12 -0400)]
12278: Bump schema-salad and cwltool for SourceLine fix.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12278: Filter nameext and nameroot from final cwl.output.json
Peter Amstutz [Tue, 26 Sep 2017 19:49:43 +0000 (15:49 -0400)]
12278: Filter nameext and nameroot from final cwl.output.json

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12278: SourceLine yields stack trace if debugging enabled
Peter Amstutz [Tue, 26 Sep 2017 13:44:57 +0000 (09:44 -0400)]
12278: SourceLine yields stack trace if debugging enabled

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12278: Can provide custom schema salad to dev jobs build.
Peter Amstutz [Mon, 25 Sep 2017 20:10:39 +0000 (16:10 -0400)]
12278: Can provide custom schema salad to dev jobs build.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12278: Propagate --debug flag to jobs.
Peter Amstutz [Mon, 25 Sep 2017 15:32:20 +0000 (11:32 -0400)]
12278: Propagate --debug flag to jobs.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12347: Fix --disable-reuse for jobs API.
Tom Clegg [Thu, 28 Sep 2017 02:24:06 +0000 (22:24 -0400)]
12347: Fix --disable-reuse for jobs API.

The --disable-reuse flag was creating a pipeline instance containing a
job that ran cwl with reuse disabled, but it didn't set
properties.run_options.enable_job_reuse on the pipeline instance
itself. If you had run before with --disable-reuse, that previous run
would be eligible for reuse.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12347: Fix formatting.
Tom Clegg [Thu, 28 Sep 2017 02:16:32 +0000 (22:16 -0400)]
12347: Fix formatting.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12347: Fix --disable-reuse for containers API.
Tom Clegg [Thu, 28 Sep 2017 02:11:11 +0000 (22:11 -0400)]
12347: Fix --disable-reuse for containers API.

The --disable-reuse flag was requesting a container to run
"arvados-cwl-runner --disable-reuse", but it didn't set
use_existing=False on the container request itself. If you had run
before with --disable-reuse, that previous run would be eligible for
reuse.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoUpdate documentation to reflect that we are packaging Debian 9
Ward Vandewege [Wed, 27 Sep 2017 21:57:16 +0000 (17:57 -0400)]
Update documentation to reflect that we are packaging Debian 9
('stretch').

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years ago12246: Make "possible causes" message easier to find and read. refs #12246
Peter Amstutz [Wed, 27 Sep 2017 18:21:32 +0000 (14:21 -0400)]
12246: Make "possible causes" message easier to find and read. refs #12246

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years agoMerge branch '12246-command-not-found'
Tom Clegg [Wed, 27 Sep 2017 17:01:36 +0000 (13:01 -0400)]
Merge branch '12246-command-not-found'

closes #12246

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12246: Add debugging clues when command is not found in container.
Tom Clegg [Wed, 27 Sep 2017 15:26:22 +0000 (11:26 -0400)]
12246: Add debugging clues when command is not found in container.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12195-nodemanager-quota-error'
Lucas Di Pentima [Wed, 27 Sep 2017 14:05:14 +0000 (11:05 -0300)]
Merge branch '12195-nodemanager-quota-error'
Closes #12195

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years ago12195: Fixed warning message when the Retry-After header value cannot
Lucas Di Pentima [Wed, 27 Sep 2017 13:39:36 +0000 (10:39 -0300)]
12195: Fixed warning message when the Retry-After header value cannot
be converted to integer.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

7 years agoSuppress superfluous error message.
Tom Clegg [Tue, 26 Sep 2017 20:29:42 +0000 (16:29 -0400)]
Suppress superfluous error message.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12032: Fix compatibility with PostgreSQL 9.4.
Tom Clegg [Tue, 26 Sep 2017 19:39:00 +0000 (15:39 -0400)]
12032: Fix compatibility with PostgreSQL 9.4.

9.4 doesn't have idle_in_transaction_session_timeout or row_security
configs, so couldn't execute structure.sql.

9.4 doesn't write "Dumped from/by version X", so 9.4 and 9.6 dev boxes
want an edit war.

refs #12032

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12287-preserve-json-numbers' closes #12287
Peter Amstutz [Tue, 26 Sep 2017 19:28:32 +0000 (15:28 -0400)]
Merge branch '12287-preserve-json-numbers' closes #12287

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12287: Add number round-trip test.
Peter Amstutz [Tue, 26 Sep 2017 17:56:04 +0000 (13:56 -0400)]
12287: Add number round-trip test.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12287: Preserve JSON number types/formats in log file.
Tom Clegg [Tue, 26 Sep 2017 02:43:10 +0000 (22:43 -0400)]
12287: Preserve JSON number types/formats in log file.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years agoMerge branch '12316-fix-provenance-graph'
Ward Vandewege [Tue, 26 Sep 2017 02:12:55 +0000 (22:12 -0400)]
Merge branch '12316-fix-provenance-graph'

closes #12316

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years agoDo not blow up the provenance graph if a PDH used in the workflow no
Ward Vandewege [Mon, 25 Sep 2017 20:53:21 +0000 (16:53 -0400)]
Do not blow up the provenance graph if a PDH used in the workflow no
longer exists.

refs #12316

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

7 years ago12276: Ensure dirent cache is disabled.
Tom Clegg [Fri, 22 Sep 2017 21:00:07 +0000 (17:00 -0400)]
12276: Ensure dirent cache is disabled.

Setting this to zero fixed flakiness in #7751, so we might still rely
on it being zero for reliability.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

7 years ago12276: Remove debug logging message.
Peter Amstutz [Fri, 22 Sep 2017 18:43:26 +0000 (14:43 -0400)]
12276: Remove debug logging message.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

7 years ago12276: Reduce number of spurious invalidations sent to kernel.
Peter Amstutz [Fri, 22 Sep 2017 15:21:07 +0000 (11:21 -0400)]
12276: Reduce number of spurious invalidations sent to kernel.

* New policy to only send invalidations on objects that have a nonzero
  kernel reference count.
* When clearing the contents of a CollectionDirectory, only invalidate the
  topmost directory entry, this should take care of invalidating all paths
  beneath it as well.
* Don't send invalidate_inode() when deleting an inode (by definition, the
  inode is unreferenced by the kernel).
* Remove allow_dirent_cache for now (this only relates to caching negative
  lookups, which we don't support).
* Set attribute attr_timeout based on the polling period of the underlying
  object.
* FuncToJSONFile sets allow_attr_cache = False instead of allow_dirent_cache =
  False (which is useless in this case).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>