arvados.git
6 years agoMerge branch 'master' of git.curoverse.com:arvados into 13429-cwl-runner-storage...
Fuad Muhic [Mon, 18 Jun 2018 09:03:33 +0000 (11:03 +0200)]
Merge branch 'master' of git.curoverse.com:arvados into 13429-cwl-runner-storage-classes-support

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years agoAdd storage_classes tests for arvados_cwl_runner
Fuad Muhic [Mon, 18 Jun 2018 08:58:33 +0000 (10:58 +0200)]
Add storage_classes tests for arvados_cwl_runner

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years agoMerge branch '13581-cr-not-satisfiable-report'
Lucas Di Pentima [Fri, 15 Jun 2018 19:58:29 +0000 (16:58 -0300)]
Merge branch '13581-cr-not-satisfiable-report'
Closes #13581

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13407-volume-replication'
Tom Clegg [Thu, 14 Jun 2018 17:16:37 +0000 (13:16 -0400)]
Merge branch '13407-volume-replication'

refs #13407

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13617-anm-instance-type-logging'
Lucas Di Pentima [Thu, 14 Jun 2018 17:09:34 +0000 (14:09 -0300)]
Merge branch '13617-anm-instance-type-logging'
Closes #13617

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13617: Replaces all logging about size names with size ids.
Lucas Di Pentima [Thu, 14 Jun 2018 15:39:08 +0000 (12:39 -0300)]
13617: Replaces all logging about size names with size ids.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13581: Adds log message when no instance types are configured.
Lucas Di Pentima [Thu, 14 Jun 2018 15:04:22 +0000 (12:04 -0300)]
13581: Adds log message when no instance types are configured.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '7478-anm-libcloud-deps-fix'
Lucas Di Pentima [Wed, 13 Jun 2018 20:04:15 +0000 (17:04 -0300)]
Merge branch '7478-anm-libcloud-deps-fix'
Refs #7478

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Fix nodemanager's libcloud install dependency.
Lucas Di Pentima [Wed, 13 Jun 2018 19:29:17 +0000 (16:29 -0300)]
7478: Fix nodemanager's libcloud install dependency.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '7478-invalid-size-not-defined'
Lucas Di Pentima [Wed, 13 Jun 2018 13:04:07 +0000 (10:04 -0300)]
Merge branch '7478-invalid-size-not-defined'
Refs #7478

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Fixes InvalidCloudSize creation. Adds wishlist related node info to logs.
Lucas Di Pentima [Wed, 13 Jun 2018 01:35:21 +0000 (22:35 -0300)]
7478: Fixes InvalidCloudSize creation. Adds wishlist related node info to logs.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '7478-anm-spot-instances'
Lucas Di Pentima [Tue, 12 Jun 2018 18:24:41 +0000 (15:24 -0300)]
Merge branch '7478-anm-spot-instances'
Closes #7478

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Moves invalid cloud size node's shutdown decision to proper method.
Lucas Di Pentima [Tue, 12 Jun 2018 17:44:24 +0000 (14:44 -0300)]
7478: Moves invalid cloud size node's shutdown decision to proper method.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13581: Available types reported to the user when CR is not satisfiable.
Lucas Di Pentima [Thu, 7 Jun 2018 17:11:00 +0000 (14:11 -0300)]
13581: Available types reported to the user when CR is not satisfiable.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13407: Test stats reporting for volumes with replication>1.
Tom Clegg [Tue, 12 Jun 2018 14:46:01 +0000 (10:46 -0400)]
13407: Test stats reporting for volumes with replication>1.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoUpdate storage_classes support for arvados_cwl_runner to work correctly
Fuad Muhic [Tue, 12 Jun 2018 13:10:23 +0000 (15:10 +0200)]
Update storage_classes support for arvados_cwl_runner to work correctly
when arvados_cwl_runner is run in submit mode.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago7478: Fixes GCE driver's arvados_node_size tag handling.
Lucas Di Pentima [Mon, 11 Jun 2018 22:49:21 +0000 (19:49 -0300)]
7478: Fixes GCE driver's arvados_node_size tag handling.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13407: Fix stats reporting for mounts with replication>1.
Tom Clegg [Mon, 11 Jun 2018 21:01:38 +0000 (17:01 -0400)]
13407: Fix stats reporting for mounts with replication>1.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago7478: Adds test to check that state is 'down' with 'invalid' size.
Lucas Di Pentima [Mon, 11 Jun 2018 20:51:38 +0000 (17:51 -0300)]
7478: Adds test to check that state is 'down' with 'invalid' size.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Adds tests on driver's create_node calls.
Lucas Di Pentima [Mon, 11 Jun 2018 19:47:50 +0000 (16:47 -0300)]
7478: Adds tests on driver's create_node calls.

* Check that arvados_node_size tag is passed
* On EC2 driver, check that ex_spot_market=True is used when requesting
spot instances.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13407: Take keepstore-reported volume replication into account.
Tom Clegg [Mon, 11 Jun 2018 19:36:21 +0000 (15:36 -0400)]
13407: Take keepstore-reported volume replication into account.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13562-group-properties' closes #13562
Peter Amstutz [Mon, 11 Jun 2018 19:32:22 +0000 (15:32 -0400)]
Merge branch '13562-group-properties' closes #13562

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch 'master' into 13562-group-properties
Peter Amstutz [Mon, 11 Jun 2018 19:31:56 +0000 (15:31 -0400)]
Merge branch 'master' into 13562-group-properties

6 years agoMerge branch '12960-migrate-properties' closes #12960
Peter Amstutz [Mon, 11 Jun 2018 19:30:05 +0000 (15:30 -0400)]
Merge branch '12960-migrate-properties' closes #12960

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago7478: Config load fix & test updates about instance type decoupling
Lucas Di Pentima [Mon, 11 Jun 2018 16:59:31 +0000 (13:59 -0300)]
7478: Config load fix & test updates about instance type decoupling

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch 'master' into 7478-anm-spot-instances
Lucas Di Pentima [Mon, 11 Jun 2018 15:34:22 +0000 (12:34 -0300)]
Merge branch 'master' into 7478-anm-spot-instances

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Assign invalid cloud sizes to nodes with stale arvados_node_size tags
Lucas Di Pentima [Mon, 11 Jun 2018 15:10:44 +0000 (12:10 -0300)]
7478: Assign invalid cloud sizes to nodes with stale arvados_node_size tags

When retrieving the node's list from the cloud provider, if the tag
refers to a non existant arvados cloud size, assign an invalid size
and return 'down' state when asked so that it can be properly shut down.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch 'master' of git.curoverse.com:arvados into 13429-cwl-runner-storage...
Fuad Muhic [Mon, 11 Jun 2018 13:50:27 +0000 (15:50 +0200)]
Merge branch 'master' of git.curoverse.com:arvados into 13429-cwl-runner-storage-classes-support

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago13594: Remove table lock.
Tom Clegg [Fri, 8 Jun 2018 19:05:12 +0000 (15:05 -0400)]
13594: Remove table lock.

refs #13594

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12960: Fix test due to slightly different serialization from jsonb
Peter Amstutz [Fri, 8 Jun 2018 18:59:41 +0000 (14:59 -0400)]
12960: Fix test due to slightly different serialization from jsonb

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13562: Migration needs to update full text search index on groups
Peter Amstutz [Fri, 8 Jun 2018 18:44:26 +0000 (14:44 -0400)]
13562: Migration needs to update full text search index on groups

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12960: Update documentation.
Peter Amstutz [Fri, 8 Jun 2018 14:13:03 +0000 (10:13 -0400)]
12960: Update documentation.

Also allow "properties" to be updated on committed container
requests (bugfix implementation to match documented behavior).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13562: Update documentation
Peter Amstutz [Fri, 8 Jun 2018 13:15:34 +0000 (09:15 -0400)]
13562: Update documentation

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13562: Add "properties" field to groups, supports subproperty query.
Peter Amstutz [Fri, 8 Jun 2018 12:59:56 +0000 (08:59 -0400)]
13562: Add "properties" field to groups, supports subproperty query.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoChange multiple storage classes error handling in cwl_runner
Fuad Muhic [Fri, 8 Jun 2018 09:05:22 +0000 (11:05 +0200)]
Change multiple storage classes error handling in cwl_runner

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years agoMerge branch '11907-acr-stable-pdh' refs #11907
Peter Amstutz [Thu, 7 Jun 2018 20:58:57 +0000 (16:58 -0400)]
Merge branch '11907-acr-stable-pdh' refs #11907

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13480-anm-tagless-azure-node'
Lucas Di Pentima [Thu, 7 Jun 2018 19:52:42 +0000 (16:52 -0300)]
Merge branch '13480-anm-tagless-azure-node'
Closes #13480

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago11907: Fix timezone handling for HTTP timestamps.
Peter Amstutz [Thu, 7 Jun 2018 19:26:36 +0000 (15:26 -0400)]
11907: Fix timezone handling for HTTP timestamps.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13427-multiple-mounts'
Tom Clegg [Thu, 7 Jun 2018 19:24:04 +0000 (15:24 -0400)]
Merge branch '13427-multiple-mounts'

refs #13427

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12960: Migrate properties fields to jsonb to support subproperty filters.
Peter Amstutz [Thu, 7 Jun 2018 18:33:03 +0000 (14:33 -0400)]
12960: Migrate properties fields to jsonb to support subproperty filters.

Specifically: links.properties and container_requests.properties,
nodes.properties, and nodes.info.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago11907: Fix tests
Peter Amstutz [Thu, 7 Jun 2018 17:13:53 +0000 (13:13 -0400)]
11907: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13330-collection-save'
Fuad Muhic [Thu, 7 Jun 2018 15:20:43 +0000 (17:20 +0200)]
Merge branch '13330-collection-save'

refs #13330

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago11907: Make collections containing dependencies have predictable PDHs
Peter Amstutz [Thu, 7 Jun 2018 13:54:47 +0000 (09:54 -0400)]
11907: Make collections containing dependencies have predictable PDHs

Removes optimization that tries to avoid redundant file
uploads (decided complexity tradeoff isn't worth it.)  Collections
created from local file inputs longer use block packing.

Likely to invalidate job reuse for jobs submitted by past versions of
a-c-r, but will have more stable job reuse going forward.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '11907-uploadfile-no-pack' refs #11907
Peter Amstutz [Thu, 7 Jun 2018 14:38:41 +0000 (10:38 -0400)]
Merge branch '11907-uploadfile-no-pack' refs #11907

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago11907: Add option to uploadfiles to disable packing.
Peter Amstutz [Thu, 7 Jun 2018 13:44:16 +0000 (09:44 -0400)]
11907: Add option to uploadfiles to disable packing.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFix ciso8601 issue in collections tests.
Fuad Muhic [Thu, 7 Jun 2018 09:39:46 +0000 (11:39 +0200)]
Fix ciso8601 issue in collections tests.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago13427: Use same index result N times if a device is mounted N times.
Tom Clegg [Wed, 6 Jun 2018 21:01:53 +0000 (17:01 -0400)]
13427: Use same index result N times if a device is mounted N times.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13427: More symlink hack
Tom Clegg [Wed, 6 Jun 2018 19:25:45 +0000 (15:25 -0400)]
13427: More symlink hack

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13427: Fix replication stats reporting for multiple-mounted devices.
Tom Clegg [Wed, 6 Jun 2018 19:19:35 +0000 (15:19 -0400)]
13427: Fix replication stats reporting for multiple-mounted devices.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13480: Ignore nodes with no tags instead of crashing.
Lucas Di Pentima [Wed, 6 Jun 2018 18:37:04 +0000 (15:37 -0300)]
13480: Ignore nodes with no tags instead of crashing.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13480: Adds test making the bug show itself.
Lucas Di Pentima [Wed, 6 Jun 2018 18:36:07 +0000 (15:36 -0300)]
13480: Adds test making the bug show itself.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13427: Handle same backend device mounted RW in multiple places.
Tom Clegg [Wed, 6 Jun 2018 17:14:26 +0000 (13:14 -0400)]
13427: Handle same backend device mounted RW in multiple places.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13427: Ignore readonly devices mounted read-write elsewhere.
Tom Clegg [Tue, 5 Jun 2018 02:10:37 +0000 (22:10 -0400)]
13427: Ignore readonly devices mounted read-write elsewhere.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13456-phantomjs-died'
Tom Clegg [Tue, 5 Jun 2018 19:56:57 +0000 (15:56 -0400)]
Merge branch '13456-phantomjs-died'

refs #13456

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMore gopath symlink hacks.
Tom Clegg [Tue, 5 Jun 2018 19:11:38 +0000 (15:11 -0400)]
More gopath symlink hacks.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13431-keepproxy-storage-classes-headers'
Lucas Di Pentima [Tue, 5 Jun 2018 19:10:11 +0000 (16:10 -0300)]
Merge branch '13431-keepproxy-storage-classes-headers'
Closes #13431

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13456: Restart driver after each test.
Tom Clegg [Tue, 5 Jun 2018 19:08:42 +0000 (15:08 -0400)]
13456: Restart driver after each test.

Shot in the dark from https://github.com/teampoltergeist/poltergeist/issues/232

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13431: Removes commented code
Lucas Di Pentima [Tue, 5 Jun 2018 18:43:40 +0000 (15:43 -0300)]
13431: Removes commented code

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13431: Fixes unrelated test
Lucas Di Pentima [Tue, 5 Jun 2018 18:37:05 +0000 (15:37 -0300)]
13431: Fixes unrelated test

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13431: Adds test confirming that keepproxy propagates X-Keep-Storage-Classes
Lucas Di Pentima [Tue, 5 Jun 2018 18:22:18 +0000 (15:22 -0300)]
13431: Adds test confirming that keepproxy propagates X-Keep-Storage-Classes

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13164-cr-locking'
Tom Clegg [Tue, 5 Jun 2018 16:57:23 +0000 (12:57 -0400)]
Merge branch '13164-cr-locking'

refs #13164

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoAdd check for multiple storage classes in arvados-cwl-runner
Fuad Muhic [Tue, 5 Jun 2018 15:30:05 +0000 (17:30 +0200)]
Add check for multiple storage classes in arvados-cwl-runner

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago13164: Remove locking. Clean up after races in background instead.
Tom Clegg [Tue, 5 Jun 2018 14:56:22 +0000 (10:56 -0400)]
13164: Remove locking. Clean up after races in background instead.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago7478: Fixes child CR default preemptable setting logic.
Lucas Di Pentima [Tue, 5 Jun 2018 14:38:12 +0000 (11:38 -0300)]
7478: Fixes child CR default preemptable setting logic.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoAdd getters for properties and trash_at attributes and small bugfix for
Fuad Muhic [Tue, 5 Jun 2018 12:03:38 +0000 (14:03 +0200)]
Add getters for properties and trash_at attributes and small bugfix for
save and save_new methods in Collection class.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago13431: Make keepproxy propagate X-Keep-Storage-Classes headers
Lucas Di Pentima [Tue, 5 Jun 2018 00:39:18 +0000 (21:39 -0300)]
13431: Make keepproxy propagate X-Keep-Storage-Classes headers

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13431: Adds storage class support to Keepclient.
Lucas Di Pentima [Mon, 4 Jun 2018 20:24:57 +0000 (17:24 -0300)]
13431: Adds storage class support to Keepclient.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '12061-nm-integration-tests'
Tom Clegg [Mon, 4 Jun 2018 13:22:13 +0000 (09:22 -0400)]
Merge branch '12061-nm-integration-tests'

refs #12061

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch 'master' into 7478-anm-spot-instances
Lucas Di Pentima [Mon, 4 Jun 2018 13:21:26 +0000 (10:21 -0300)]
Merge branch 'master' into 7478-anm-spot-instances

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13559-ciso8601-pin'
Lucas Di Pentima [Mon, 4 Jun 2018 13:20:29 +0000 (10:20 -0300)]
Merge branch '13559-ciso8601-pin'
Closes #13559

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13559: Pin ciso8601 to version <2.0.0 due to API changes.
Lucas Di Pentima [Fri, 1 Jun 2018 21:12:43 +0000 (18:12 -0300)]
13559: Pin ciso8601 to version <2.0.0 due to API changes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12061: Disable nodemanager integration tests.
Tom Clegg [Fri, 1 Jun 2018 17:35:10 +0000 (13:35 -0400)]
12061: Disable nodemanager integration tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Remove unreliable check.
Tom Clegg [Fri, 1 Jun 2018 17:36:43 +0000 (13:36 -0400)]
12061: Remove unreliable check.

"Shutdown because state=down" is possible during a race, when
nodemanager logs "Shutdown success" and polls sinfo again before the
test suite has finished updating the sinfo stub.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Reduce per-test timeout.
Tom Clegg [Fri, 1 Jun 2018 16:50:02 +0000 (12:50 -0400)]
12061: Reduce per-test timeout.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Fix nodes marked busy while expecting shutdown decision.
Tom Clegg [Fri, 1 Jun 2018 16:49:42 +0000 (12:49 -0400)]
12061: Fix nodes marked busy while expecting shutdown decision.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoAdd storage classes supper to arvados cwl runner
Fuad Muhic [Fri, 1 Jun 2018 16:06:49 +0000 (18:06 +0200)]
Add storage classes supper to arvados cwl runner

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years ago7478: Fixes integration tests.
Lucas Di Pentima [Fri, 1 Jun 2018 15:41:44 +0000 (12:41 -0300)]
7478: Fixes integration tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12061: Use shorter retry-after sleeps in integration test.
Tom Clegg [Thu, 31 May 2018 21:17:33 +0000 (17:17 -0400)]
12061: Use shorter retry-after sleeps in integration test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Don't wait around for test daemon to shutdown gracefully.
Tom Clegg [Thu, 31 May 2018 21:07:13 +0000 (17:07 -0400)]
12061: Don't wait around for test daemon to shutdown gracefully.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Poll aggressively in integration tests.
Tom Clegg [Thu, 31 May 2018 21:06:32 +0000 (17:06 -0400)]
12061: Poll aggressively in integration tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Allow sub-second poll times and sub-minute shutdown windows.
Tom Clegg [Thu, 31 May 2018 21:01:15 +0000 (17:01 -0400)]
12061: Allow sub-second poll times and sub-minute shutdown windows.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch 'master' into 7478-anm-spot-instances
Lucas Di Pentima [Thu, 31 May 2018 19:24:07 +0000 (16:24 -0300)]
Merge branch 'master' into 7478-anm-spot-instances

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13517-buffer-leak'
Tom Clegg [Thu, 31 May 2018 19:05:47 +0000 (15:05 -0400)]
Merge branch '13517-buffer-leak'

refs #13517

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12061-flaky-tests'
Tom Clegg [Thu, 31 May 2018 18:39:41 +0000 (14:39 -0400)]
Merge branch '12061-flaky-tests'

refs #12061

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Explicitly join actor thread after stopping.
Tom Clegg [Thu, 31 May 2018 18:26:59 +0000 (14:26 -0400)]
12061: Explicitly join actor thread after stopping.

Fixes the "nonfatal" test case the same way the "fatal" test case was
fixed in 6706c6d46c6cfb080371b147b60fcaa191bd55a6.

Since the nonfatal case just tests that kill_mock did *not* get
called, the race condition used to make the test ineffectual
occasionally, instead of failing occasionally.

The race condition became visible only after the status reporting test
was added.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13517: Fix leaked goroutine and buffer on failed S3 write.
Tom Clegg [Thu, 31 May 2018 17:44:18 +0000 (13:44 -0400)]
13517: Fix leaked goroutine and buffer on failed S3 write.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago7478: Unify arvados node size access across different drivers
Lucas Di Pentima [Thu, 31 May 2018 16:42:45 +0000 (13:42 -0300)]
7478: Unify arvados node size access across different drivers

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Test fixes.
Lucas Di Pentima [Wed, 30 May 2018 21:08:34 +0000 (18:08 -0300)]
7478: Test fixes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Set arvados-node-size-id tag on node creation.
Lucas Di Pentima [Wed, 30 May 2018 21:02:30 +0000 (18:02 -0300)]
7478: Set arvados-node-size-id tag on node creation.

This tag has the Arvados assigned node size id, not the cloud size.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Override CloudSizeWrapper id with config Size name.
Lucas Di Pentima [Wed, 30 May 2018 21:00:29 +0000 (18:00 -0300)]
7478: Override CloudSizeWrapper id with config Size name.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago7478: Ignore redundant instance_type field on CloudSizeWrapper
Lucas Di Pentima [Wed, 30 May 2018 20:55:59 +0000 (17:55 -0300)]
7478: Ignore redundant instance_type field on CloudSizeWrapper

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12061: Don't round down to 1s in timestamp parser.
Tom Clegg [Wed, 30 May 2018 20:32:20 +0000 (16:32 -0400)]
12061: Don't round down to 1s in timestamp parser.

ComputeNodeMonitorActor.offer_arvados_pair() pairs nodes only if

  first_ping_time >= cloud_node_start_time

However, first_ping_time is passed through arvados_timestamp() before
this comparison, which was truncating the subsecond part -- so the
comparison was effectively

  floor(first_ping_time) >= cloud_node_start_time

When FPT and CNST differed only in the subsecond part, this comparison
failed, and the nodes could never be paired. This caused sporadic
failures in tests, where the two values are often separated by less
than a second.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12061: Change busywait approach to preserve assertion messages.
Tom Clegg [Wed, 30 May 2018 20:30:45 +0000 (16:30 -0400)]
12061: Change busywait approach to preserve assertion messages.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '11162-wes-support' refs #11162
Peter Amstutz [Wed, 30 May 2018 11:39:30 +0000 (07:39 -0400)]
Merge branch '11162-wes-support' refs #11162

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13513-balance-deadlock'
Tom Clegg [Tue, 29 May 2018 20:05:27 +0000 (16:05 -0400)]
Merge branch '13513-balance-deadlock'

fixes #13513

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago11162: Remove debugging check
Peter Amstutz [Tue, 29 May 2018 17:16:55 +0000 (13:16 -0400)]
11162: Remove debugging check

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13513: Fix deadlocked workers.
Tom Clegg [Tue, 29 May 2018 13:46:04 +0000 (09:46 -0400)]
13513: Fix deadlocked workers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoAdded properties and trash_at parameters to save and save_new methods
Fuad Muhic [Tue, 29 May 2018 10:25:28 +0000 (12:25 +0200)]
Added properties and trash_at parameters to save and save_new methods

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

6 years agoFix crunch_script for jobs api refs #13516
Peter Amstutz [Sat, 26 May 2018 02:04:21 +0000 (22:04 -0400)]
Fix crunch_script for jobs api refs #13516

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoReplace unicode copyright character with ASCII in README.Rmd
Peter Amstutz [Sat, 26 May 2018 01:49:14 +0000 (21:49 -0400)]
Replace unicode copyright character with ASCII in README.Rmd

Fix documentation generation

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>