arvados.git
5 years ago14291: Adjust ExtraScratch (int) to AttachScratch (bool)
Peter Amstutz [Thu, 28 Feb 2019 18:05:57 +0000 (13:05 -0500)]
14291: Adjust ExtraScratch (int) to AttachScratch (bool)

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: minimal stub so that tests don't panic when run without live config
Peter Amstutz [Thu, 28 Feb 2019 15:35:00 +0000 (10:35 -0500)]
14291: minimal stub so that tests don't panic when run without live config

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Fix tag prefix typo
Peter Amstutz [Thu, 28 Feb 2019 15:18:26 +0000 (10:18 -0500)]
14291: Fix tag prefix typo

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Change instance type to a supported one, add SetTags test
Peter Amstutz [Thu, 28 Feb 2019 15:16:22 +0000 (10:16 -0500)]
14291: Change instance type to a supported one, add SetTags test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Support EBS attached storage and preemptible instances
Peter Amstutz [Wed, 27 Feb 2019 21:38:33 +0000 (16:38 -0500)]
14291: Support EBS attached storage and preemptible instances

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Vendoring
Peter Amstutz [Wed, 27 Feb 2019 19:30:16 +0000 (14:30 -0500)]
14291: Vendoring

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Still trying to nail down aws go sdk vendor
Peter Amstutz [Wed, 27 Feb 2019 19:27:20 +0000 (14:27 -0500)]
14291: Still trying to nail down aws go sdk vendor

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Add individual packages in github.com/aws/aws-sdk-go to vendor
Peter Amstutz [Wed, 27 Feb 2019 19:23:42 +0000 (14:23 -0500)]
14291: Add individual packages in github.com/aws/aws-sdk-go to vendor

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Add github.com/aws/aws-sdk-go to vendor
Peter Amstutz [Wed, 27 Feb 2019 19:17:41 +0000 (14:17 -0500)]
14291: Add github.com/aws/aws-sdk-go to vendor

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Working on instance creation
Peter Amstutz [Wed, 27 Feb 2019 18:57:18 +0000 (13:57 -0500)]
14291: Working on instance creation

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: EC2 driver WIP
Peter Amstutz [Tue, 26 Feb 2019 22:54:08 +0000 (17:54 -0500)]
14291: EC2 driver WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years agoMerge branch '14804-keepstore-transient-backend-errors'
Lucas Di Pentima [Mon, 25 Feb 2019 14:49:12 +0000 (11:49 -0300)]
Merge branch '14804-keepstore-transient-backend-errors'
Closes #14804

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14804: Adds test scenario comment
Lucas Di Pentima [Mon, 25 Feb 2019 14:47:48 +0000 (11:47 -0300)]
14804: Adds test scenario comment

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14804: Returns 503 status (instead of 404) on GET when a volume does the same.
Lucas Di Pentima [Fri, 22 Feb 2019 18:59:32 +0000 (15:59 -0300)]
14804: Returns 503 status (instead of 404) on GET when a volume does the same.

Also, translates Azure's "...StatusCode=503..." error to VolumeBusyError.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14804: Exposes the bug with a test.
Lucas Di Pentima [Fri, 22 Feb 2019 18:06:30 +0000 (15:06 -0300)]
14804: Exposes the bug with a test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoMerge branch '14807-dispatch-cloud-fixes'
Tom Clegg [Thu, 21 Feb 2019 19:41:56 +0000 (14:41 -0500)]
Merge branch '14807-dispatch-cloud-fixes'

refs #14807

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Force test SSH server to listen on IPv4.
Tom Clegg [Thu, 21 Feb 2019 18:49:09 +0000 (13:49 -0500)]
14807: Force test SSH server to listen on IPv4.

It seems net.Listen defaults to listening on IPv6 even on systems
where that doesn't work.

https://dev.arvados.org/issues/14807#note-19

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14853-chapmanb-subprocess-merge'
Eric Biagiotti [Thu, 21 Feb 2019 15:04:17 +0000 (10:04 -0500)]
Merge branch '14853-chapmanb-subprocess-merge'

refs #14853

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years ago14807: Avoid doing concurrent update requests per container.
Tom Clegg [Thu, 21 Feb 2019 06:00:00 +0000 (01:00 -0500)]
14807: Avoid doing concurrent update requests per container.

The previous code avoided concurrent Lock requests, but would
sometimes generate lots of overlapping Cancel requests instead of
waiting for the first one to finish.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoSome build script housekeeping: make the CentOS builds less noisy, and remove
Ward Vandewege [Wed, 20 Feb 2019 22:45:04 +0000 (17:45 -0500)]
Some build script housekeeping: make the CentOS builds less noisy, and remove
unused variables from the run-build-packages-python-and-ruby.sh script.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years ago14853: we need a newer version of setuptools to build on centos7, debian8 and ubuntu1...
Ward Vandewege [Wed, 20 Feb 2019 22:04:42 +0000 (17:04 -0500)]
14853: we need a newer version of setuptools to build on centos7, debian8 and ubuntu1404 (part 2).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoRemove a few files that were checked in erroneously, update the .gitignore
Ward Vandewege [Fri, 1 Feb 2019 14:46:24 +0000 (09:46 -0500)]
Remove a few files that were checked in erroneously, update the .gitignore
files for workbench and api to avoid this in the future.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years ago14807: Merge branch 'master'
Tom Clegg [Wed, 20 Feb 2019 16:49:18 +0000 (11:49 -0500)]
14807: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix missing import.
Tom Clegg [Wed, 20 Feb 2019 16:48:25 +0000 (11:48 -0500)]
14807: Fix missing import.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix logger type mismatch.
Tom Clegg [Wed, 20 Feb 2019 13:55:53 +0000 (08:55 -0500)]
14807: Fix logger type mismatch.

Broke in 91b39ff3fbba69e54a330a7949c9ed4bbb3117ff.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix admin permissions for containers.
Tom Clegg [Tue, 19 Feb 2019 22:28:26 +0000 (17:28 -0500)]
14807: Fix admin permissions for containers.

Admins are allowed to see all containers, even ones that aren't
associated with any container request.

Otherwise, when a container request retries, even the dispatcher can't
see the previously assigned container, and therefore never learns that
the container is supposed to be cancelled.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14860-fix-collection-versioning'
Lucas Di Pentima [Tue, 19 Feb 2019 20:21:32 +0000 (17:21 -0300)]
Merge branch '14860-fix-collection-versioning'
Closes #14860

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14860: Fixes bug by assigning the signed manifest text to the snapshot being saved.
Lucas Di Pentima [Tue, 19 Feb 2019 19:50:36 +0000 (16:50 -0300)]
14860: Fixes bug by assigning the signed manifest text to the snapshot being saved.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14860: Exposes the bug with a new test.
Lucas Di Pentima [Tue, 19 Feb 2019 18:24:46 +0000 (15:24 -0300)]
14860: Exposes the bug with a new test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoMerge branch '11561-limit-container-locks' refs #11561
Peter Amstutz [Tue, 19 Feb 2019 16:40:12 +0000 (11:40 -0500)]
Merge branch '11561-limit-container-locks' refs #11561

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14807: Send detached crunch-run logs to journal via systemd-cat.
Tom Clegg [Tue, 19 Feb 2019 16:35:22 +0000 (11:35 -0500)]
14807: Send detached crunch-run logs to journal via systemd-cat.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14853: we need a newer version of setuptools to build on centos7, debian8 and ubuntu1404.
Ward Vandewege [Tue, 19 Feb 2019 15:47:24 +0000 (10:47 -0500)]
14853: we need a newer version of setuptools to build on centos7, debian8 and ubuntu1404.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoMerge branch '14845-fix-flaky-test'
Tom Clegg [Mon, 18 Feb 2019 23:08:12 +0000 (18:08 -0500)]
Merge branch '14845-fix-flaky-test'

fixes #14845

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Remove errant rm.
Tom Clegg [Mon, 18 Feb 2019 19:48:24 +0000 (14:48 -0500)]
14807: Remove errant rm.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Update API endpoints: instance_id is always a query param.
Tom Clegg [Mon, 18 Feb 2019 19:32:31 +0000 (14:32 -0500)]
14807: Update API endpoints: instance_id is always a query param.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Log idle time in seconds instead of nanoseconds.
Tom Clegg [Mon, 18 Feb 2019 18:50:25 +0000 (13:50 -0500)]
14807: Log idle time in seconds instead of nanoseconds.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Cancel or requeue container when priority drops to zero.
Tom Clegg [Mon, 18 Feb 2019 18:17:08 +0000 (13:17 -0500)]
14807: Cancel or requeue container when priority drops to zero.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Use context to pass a suitable logger to all service commands.
Tom Clegg [Mon, 18 Feb 2019 17:33:21 +0000 (12:33 -0500)]
14807: Use context to pass a suitable logger to all service commands.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Log when a container is added/removed from the queue.
Tom Clegg [Sun, 17 Feb 2019 03:25:44 +0000 (22:25 -0500)]
14807: Log when a container is added/removed from the queue.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Split instance count/size/cost metrics by idle/hold status.
Tom Clegg [Sun, 17 Feb 2019 03:17:26 +0000 (22:17 -0500)]
14807: Split instance count/size/cost metrics by idle/hold status.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix up azure log message.
Tom Clegg [Fri, 15 Feb 2019 21:31:09 +0000 (16:31 -0500)]
14807: Fix up azure log message.

arvados.Duration's string representation includes units, so the log
message was "...older than 20s s".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Move secret-tag host key verify mechanism out of Azure driver.
Tom Clegg [Fri, 15 Feb 2019 21:15:27 +0000 (16:15 -0500)]
14807: Move secret-tag host key verify mechanism out of Azure driver.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Don't delete existing tags when updating.
Tom Clegg [Fri, 15 Feb 2019 20:28:34 +0000 (15:28 -0500)]
14807: Don't delete existing tags when updating.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Load API host/token directly from stdin without shell hack.
Tom Clegg [Fri, 15 Feb 2019 20:26:10 +0000 (15:26 -0500)]
14807: Load API host/token directly from stdin without shell hack.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Wait at least 1 second between retries on initial queue poll.
Tom Clegg [Thu, 14 Feb 2019 21:13:23 +0000 (16:13 -0500)]
14807: Wait at least 1 second between retries on initial queue poll.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Include more detail in errors.
Tom Clegg [Wed, 13 Feb 2019 19:38:07 +0000 (14:38 -0500)]
14807: Include more detail in errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix crunch-run --list output when /var/lock is a symlink.
Tom Clegg [Wed, 13 Feb 2019 16:11:03 +0000 (11:11 -0500)]
14807: Fix crunch-run --list output when /var/lock is a symlink.

filepath.Walk(/var/lock) does not return entries inside /var/lock if
/var/lock is a symlink, as it is on debian:9.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Always set node-token tag.
Tom Clegg [Wed, 13 Feb 2019 06:59:42 +0000 (01:59 -0500)]
14807: Always set node-token tag.

Azure driver expects it to be set to an unpredictable string.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Log full instance ID.
Tom Clegg [Wed, 13 Feb 2019 06:57:46 +0000 (01:57 -0500)]
14807: Log full instance ID.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Expose instance IP addresses in logs and management API.
Tom Clegg [Wed, 13 Feb 2019 06:56:49 +0000 (01:56 -0500)]
14807: Expose instance IP addresses in logs and management API.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix SSH target address.
Tom Clegg [Wed, 13 Feb 2019 06:55:32 +0000 (01:55 -0500)]
14807: Fix SSH target address.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Accept .../instances/_/drain?instance_id=X.
Tom Clegg [Wed, 13 Feb 2019 06:52:21 +0000 (01:52 -0500)]
14807: Accept .../instances/_/drain?instance_id=X.

Azure Instance IDs contain slashes, so they don't work well as path
parameters.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Pass SSH public key to driver.
Tom Clegg [Wed, 13 Feb 2019 06:49:59 +0000 (01:49 -0500)]
14807: Pass SSH public key to driver.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Allow driver to specify SSH username.
Tom Clegg [Wed, 13 Feb 2019 06:42:40 +0000 (01:42 -0500)]
14807: Allow driver to specify SSH username.

Use a non-root account with passwordless sudo on a provider (Azure)
that can easily set that up, but can't easily set up direct root
login.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14853: Adds os.name check for subprocess conditional import
Eric Biagiotti [Fri, 15 Feb 2019 16:28:09 +0000 (11:28 -0500)]
14853: Adds os.name check for subprocess conditional import

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoPython 3: only import subprocess32 on py27
chapmanb [Thu, 14 Feb 2019 16:23:39 +0000 (11:23 -0500)]
Python 3: only import subprocess32 on py27

Avoids the need to install and import subprocess32 when working
with any recent Python 3.x version. This helps with bioconda
install since subprocess32 is not built for py3.

Arvados-DCO-1.1-Signed-off-by: Brad Chapman <chapmanb@fastmail.com>

5 years ago14807: When ProviderType is unspecified, default to Arvados type.
Tom Clegg [Wed, 13 Feb 2019 06:34:43 +0000 (01:34 -0500)]
14807: When ProviderType is unspecified, default to Arvados type.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Pass env vars on stdin instead of using SSH feature.
Tom Clegg [Wed, 13 Feb 2019 06:32:34 +0000 (01:32 -0500)]
14807: Pass env vars on stdin instead of using SSH feature.

Arbitrary environment variables are typically not accepted by SSH
server configs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Match systemd description to component name.
Tom Clegg [Wed, 13 Feb 2019 03:00:21 +0000 (22:00 -0500)]
14807: Match systemd description to component name.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14325: Start up immediately if there are no stale locks.
Tom Clegg [Tue, 12 Feb 2019 19:17:10 +0000 (14:17 -0500)]
14325: Start up immediately if there are no stale locks.

...instead of waiting for the pool to send a notification to trigger
the first loop iteration.

refs #14325

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago11561: Add lock_count to containers, cancel container on unlock exceeded
Peter Amstutz [Thu, 14 Feb 2019 23:28:00 +0000 (18:28 -0500)]
11561: Add lock_count to containers, cancel container on unlock exceeded

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years agoMerge branch '14745-azure-cloud-driver-fixups'
Eric Biagiotti [Thu, 14 Feb 2019 19:56:47 +0000 (14:56 -0500)]
Merge branch '14745-azure-cloud-driver-fixups'

refs #14745

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Removes unnecessary test parameter
Eric Biagiotti [Thu, 14 Feb 2019 19:11:34 +0000 (14:11 -0500)]
14745: Removes unnecessary test parameter

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Adds support for passing nil params to StubDriver InstanceSet
Eric Biagiotti [Thu, 14 Feb 2019 17:31:53 +0000 (12:31 -0500)]
14745: Adds support for passing nil params to StubDriver InstanceSet

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoMerge branch '14828-fix-flaky-test'
Tom Clegg [Thu, 14 Feb 2019 16:31:32 +0000 (11:31 -0500)]
Merge branch '14828-fix-flaky-test'

fixes #14828

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14845: Fix inconsistent timeout error message.
Tom Clegg [Thu, 14 Feb 2019 07:07:21 +0000 (02:07 -0500)]
14845: Fix inconsistent timeout error message.

http.Client's Timeout feature is not needed because every outgoing
request runs in the context of an incoming request, which already has
a deadline. It only makes the "timed out" error response inconsistent.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14745: Improves azure driver exported var comment
Eric Biagiotti [Wed, 13 Feb 2019 22:38:54 +0000 (17:38 -0500)]
14745: Improves azure driver exported var comment

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Removes unneeded file and debug code and improves Duration comments
Eric Biagiotti [Wed, 13 Feb 2019 22:18:57 +0000 (17:18 -0500)]
14745: Removes unneeded file and debug code and improves Duration comments

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Adds back param loading to StubDriver InstanceSet
Eric Biagiotti [Wed, 13 Feb 2019 21:50:03 +0000 (16:50 -0500)]
14745: Adds back param loading to StubDriver InstanceSet

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14828: Fix double-cancel of unsatisfiable container.
Tom Clegg [Wed, 13 Feb 2019 21:47:36 +0000 (16:47 -0500)]
14828: Fix double-cancel of unsatisfiable container.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14745: Replaces debug println with proper return and doc clean up
Eric Biagiotti [Wed, 13 Feb 2019 20:10:48 +0000 (15:10 -0500)]
14745: Replaces debug println with proper return and doc clean up

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Updates StubDriver to match driver interface and updates vendor.json
Eric Biagiotti [Wed, 13 Feb 2019 18:51:18 +0000 (13:51 -0500)]
14745: Updates StubDriver to match driver interface and updates vendor.json

StubDriver InstanceSet was previously using mapstructure to decode DriverParameters into a StubInstanceSet, which doesn't make sense, but mapstructure would just return nil instead of an error.

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Updates drivers to use json.RawMessage instead of map[string]interface
Eric Biagiotti [Tue, 12 Feb 2019 22:09:47 +0000 (17:09 -0500)]
14745: Updates drivers to use json.RawMessage instead of map[string]interface

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com

5 years ago14325: Start up immediately if there are no stale locks.
Tom Clegg [Tue, 12 Feb 2019 19:17:10 +0000 (14:17 -0500)]
14325: Start up immediately if there are no stale locks.

...instead of waiting for the pool to send a notification to trigger
the first loop iteration.

refs #14325

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14325: Fix PrivateKey config type.
Tom Clegg [Tue, 12 Feb 2019 16:25:56 +0000 (11:25 -0500)]
14325: Fix PrivateKey config type.

JSON decoder expects []byte fields to be base64-encoded, which we
don't want here.

refs #14325

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14325-dispatch-cloud'
Tom Clegg [Tue, 12 Feb 2019 06:10:55 +0000 (01:10 -0500)]
Merge branch '14325-dispatch-cloud'

refs #14325

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14325: Rename crunch-dispatch-cloud → arvados-dispatch-cloud.
Tom Clegg [Tue, 12 Feb 2019 06:07:09 +0000 (01:07 -0500)]
14325: Rename crunch-dispatch-cloud → arvados-dispatch-cloud.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14745: Un-exports identifiers
Eric Biagiotti [Mon, 11 Feb 2019 21:46:29 +0000 (16:46 -0500)]
14745: Un-exports identifiers

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoMake sure to bump the dependency on the arvados gems to the 1.3 series.
Ward Vandewege [Mon, 11 Feb 2019 21:10:47 +0000 (16:10 -0500)]
Make sure to bump the dependency on the arvados gems to the 1.3 series.

refs #14559

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years ago14745: Makes DeleteDanglingResourcesAfter an arvados.Duration
Eric Biagiotti [Mon, 11 Feb 2019 19:59:15 +0000 (14:59 -0500)]
14745: Makes DeleteDanglingResourcesAfter an arvados.Duration

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoMerge branch '14826-cert-path' refs #14826
Peter Amstutz [Mon, 11 Feb 2019 19:54:53 +0000 (14:54 -0500)]
Merge branch '14826-cert-path' refs #14826

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14826: Fix copy and paste error
Peter Amstutz [Mon, 11 Feb 2019 18:05:34 +0000 (13:05 -0500)]
14826: Fix copy and paste error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years agoMerge branch '14725-wes' refs #14725
Peter Amstutz [Mon, 11 Feb 2019 16:32:01 +0000 (11:32 -0500)]
Merge branch '14725-wes'  refs #14725

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14725: Fix tests.
Peter Amstutz [Mon, 11 Feb 2019 16:23:48 +0000 (11:23 -0500)]
14725: Fix tests.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14745: Updates config keys, documentation, and addresses golint suggestions
Eric Biagiotti [Mon, 11 Feb 2019 16:07:28 +0000 (11:07 -0500)]
14745: Updates config keys, documentation, and addresses golint suggestions

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14826: API server looks in CA cert path that works on CentOS
Peter Amstutz [Mon, 11 Feb 2019 15:57:45 +0000 (10:57 -0500)]
14826: API server looks in CA cert path that works on CentOS

API server makes callbacks to remote clusters to validate remote
users, ensure it has the right system CA cert path to validate TLS
certs of remote clusters.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14725: Don't give a cluster_id when updating container request record
Peter Amstutz [Mon, 11 Feb 2019 15:09:24 +0000 (10:09 -0500)]
14725: Don't give a cluster_id when updating container request record

* When transferring files from http to keep, explicitly open in binary
  mode

* Bump arvados-python-client dependency for python3 support

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14745: Adds gocheck hook to go test for azure dirver
Eric Biagiotti [Mon, 11 Feb 2019 14:58:11 +0000 (09:58 -0500)]
14745: Adds gocheck hook to go test for azure dirver

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago14745: Makes the azure driver its own package
Eric Biagiotti [Mon, 11 Feb 2019 14:13:11 +0000 (09:13 -0500)]
14745: Makes the azure driver its own package

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoIn the arvados/jobs image, use the Python executable from the
Ward Vandewege [Thu, 7 Feb 2019 16:52:04 +0000 (11:52 -0500)]
In the arvados/jobs image, use the Python executable from the
python-arvados-python-client package by default.

refs #9945

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoMerge branch '13306-arvados-cwl-runner-py3-support'
Eric Biagiotti [Wed, 6 Feb 2019 22:38:17 +0000 (17:38 -0500)]
Merge branch '13306-arvados-cwl-runner-py3-support'

refs #13306
3

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoMerge branch '14325-dispatch-cloud'
Tom Clegg [Wed, 6 Feb 2019 21:29:38 +0000 (16:29 -0500)]
Merge branch '14325-dispatch-cloud'

closes #14325

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14325: Merge branch 'master'
Tom Clegg [Wed, 6 Feb 2019 21:27:57 +0000 (16:27 -0500)]
14325: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch 'master' into 13306-arvados-cwl-runner-py3-support
Eric Biagiotti [Wed, 6 Feb 2019 16:40:17 +0000 (11:40 -0500)]
Merge branch 'master' into 13306-arvados-cwl-runner-py3-support

refs #13306
3

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago13306: Removing unneeded import statements
Eric Biagiotti [Wed, 6 Feb 2019 16:40:04 +0000 (11:40 -0500)]
13306: Removing unneeded import statements

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoMake the python-arvados-cwl-runner package expose the cwltool executable, make
Ward Vandewege [Tue, 5 Feb 2019 22:14:05 +0000 (17:14 -0500)]
Make the python-arvados-cwl-runner package expose the cwltool executable, make
it conflict with the python-cwltool (and cwltool) packages, and add a note to
the upgrade documentation to that effect. Also document the Centos7 specific
upgrade from python33 to rh-python35 as part of story 9945.

refs #9945

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoMerge branch '13306-arvados-cwl-runner-py3-support'
Eric Biagiotti [Tue, 5 Feb 2019 18:25:14 +0000 (13:25 -0500)]
Merge branch '13306-arvados-cwl-runner-py3-support'

refs #13306

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago13306: Adds declaration of py2/3 support to arvados cwl and python sdks
Eric Biagiotti [Tue, 5 Feb 2019 17:55:54 +0000 (12:55 -0500)]
13306: Adds declaration of py2/3 support to arvados cwl and python sdks

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years ago13306: fixes test_with_arvados.sh pip executable with python2
Eric Biagiotti [Tue, 5 Feb 2019 16:53:07 +0000 (11:53 -0500)]
13306: fixes test_with_arvados.sh pip executable with python2

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <eric.biagiotti@gmail.com>

5 years agoMerge branch '14806-crunch1-unescape-manifest'
Tom Clegg [Tue, 5 Feb 2019 15:41:51 +0000 (10:41 -0500)]
Merge branch '14806-crunch1-unescape-manifest'

fixes #14806

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>