arvados.git
6 years agoFinish fixing setting project_uuid from runtimeContext, fix tests.
Peter Amstutz [Wed, 21 Nov 2018 21:39:59 +0000 (16:39 -0500)]
Finish fixing setting project_uuid from runtimeContext, fix tests.

refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFix ClusterTarget.project_uuid to correctly set the project uuid
Peter Amstutz [Wed, 21 Nov 2018 19:46:38 +0000 (14:46 -0500)]
Fix ClusterTarget.project_uuid to correctly set the project uuid

refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14345-nginx-proxy-request-buffering'
Tom Clegg [Wed, 21 Nov 2018 19:44:23 +0000 (14:44 -0500)]
Merge branch '14345-nginx-proxy-request-buffering'

refs #14345

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Disable proxy request buffering in Nginx keepproxy example.
Tom Clegg [Wed, 21 Nov 2018 19:34:17 +0000 (14:34 -0500)]
14345: Disable proxy request buffering in Nginx keepproxy example.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Disable Nginx proxy request buffering for keep-web requests.
Tom Clegg [Wed, 21 Nov 2018 19:14:04 +0000 (14:14 -0500)]
14345: Disable Nginx proxy request buffering for keep-web requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix --submit-runner-cluster tests refs #14198
Peter Amstutz [Wed, 21 Nov 2018 19:00:04 +0000 (14:00 -0500)]
Fix --submit-runner-cluster tests refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14345-proppatch'
Tom Clegg [Wed, 21 Nov 2018 18:28:00 +0000 (13:28 -0500)]
Merge branch '14345-proppatch'

refs #14345

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14198-fed-testing' refs #14198
Peter Amstutz [Wed, 21 Nov 2018 18:06:00 +0000 (13:06 -0500)]
Merge branch '14198-fed-testing' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Call validate_cluster_target on --submit-runner-cluster
Peter Amstutz [Wed, 21 Nov 2018 17:41:11 +0000 (12:41 -0500)]
14198: Call validate_cluster_target on --submit-runner-cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add check for uuidPrefix if local cluster isn't in the remoteHosts.
Peter Amstutz [Tue, 20 Nov 2018 22:26:17 +0000 (17:26 -0500)]
14198: Add check for uuidPrefix if local cluster isn't in the remoteHosts.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Update build.list for latest cwltool/schema-salad
Peter Amstutz [Tue, 20 Nov 2018 18:30:40 +0000 (13:30 -0500)]
14198: Update build.list for latest cwltool/schema-salad

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix secondaryFile tests
Peter Amstutz [Mon, 19 Nov 2018 20:05:01 +0000 (15:05 -0500)]
14198: Fix secondaryFile tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Tests for hints in different places
Peter Amstutz [Sat, 17 Nov 2018 03:01:45 +0000 (22:01 -0500)]
14198: Tests for hints in different places

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Tests prefer setting ClusterTarget hint on the workflow step.
Peter Amstutz [Fri, 16 Nov 2018 21:39:09 +0000 (16:39 -0500)]
14198: Tests prefer setting ClusterTarget hint on the workflow step.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Test cases for places that ClusterTarget can appear
Peter Amstutz [Fri, 16 Nov 2018 21:18:19 +0000 (16:18 -0500)]
14198: Test cases for places that ClusterTarget can appear

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set ClusterTarget hint on steps instead of tools
Peter Amstutz [Fri, 16 Nov 2018 15:59:52 +0000 (10:59 -0500)]
14198: Set ClusterTarget hint on steps instead of tools

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Rewrite README
Peter Amstutz [Wed, 14 Nov 2018 19:44:19 +0000 (14:44 -0500)]
14198: Rewrite README

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Connect test cases to outputs
Peter Amstutz [Wed, 14 Nov 2018 04:10:07 +0000 (23:10 -0500)]
14198: Connect test cases to outputs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add threestep-remote
Peter Amstutz [Wed, 14 Nov 2018 03:56:56 +0000 (22:56 -0500)]
14198: Add threestep-remote

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add scatter-gather
Peter Amstutz [Wed, 14 Nov 2018 03:06:36 +0000 (22:06 -0500)]
14198: Add scatter-gather

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add test twostep-remote-copy-to-home
Peter Amstutz [Tue, 13 Nov 2018 22:28:59 +0000 (17:28 -0500)]
14198: Add test twostep-remote-copy-to-home

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix collection PDHs for remote-case
Peter Amstutz [Tue, 13 Nov 2018 21:37:34 +0000 (16:37 -0500)]
14198: Fix collection PDHs for remote-case

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add some basic doc comments.
Peter Amstutz [Fri, 9 Nov 2018 20:48:14 +0000 (15:48 -0500)]
14198: Add some basic doc comments.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add remote-case and twostep-both-remote
Peter Amstutz [Fri, 9 Nov 2018 20:33:05 +0000 (15:33 -0500)]
14198: Add remote-case and twostep-both-remote

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add more twostep tests
Peter Amstutz [Fri, 9 Nov 2018 20:02:50 +0000 (15:02 -0500)]
14198: Add more twostep tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Working on two step test cases
Peter Amstutz [Fri, 9 Nov 2018 15:47:23 +0000 (10:47 -0500)]
14198: Working on two step test cases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Refactor arvbox federation setup
Peter Amstutz [Fri, 9 Nov 2018 03:19:15 +0000 (22:19 -0500)]
14198: Refactor arvbox federation setup

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Federation feature and testing
Peter Amstutz [Fri, 16 Nov 2018 21:15:11 +0000 (16:15 -0500)]
14198: Federation feature and testing

* Evaluate ClusterTarget in the job step that declares it

* Don't use get_requirement to inherit ClusterTarget.

* Workflow to start and configure a federation of N arvboxes, initialize test user

* Test cases.

* Prepare/scrub before running test

* Use --always-submit-runner

* Tests use isolated inputs.  Return success boolean

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set arvrunner before calling super constructor
Peter Amstutz [Fri, 16 Nov 2018 19:21:07 +0000 (14:21 -0500)]
14198: Set arvrunner before calling super constructor

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set arvrunner on ArvadosWorkflowStep
Peter Amstutz [Fri, 16 Nov 2018 19:12:51 +0000 (14:12 -0500)]
14198: Set arvrunner on ArvadosWorkflowStep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set fake 'class' so _init_job is happy
Peter Amstutz [Fri, 16 Nov 2018 18:55:49 +0000 (13:55 -0500)]
14198: Set fake 'class' so _init_job is happy

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix jobstep builder init
Peter Amstutz [Fri, 16 Nov 2018 16:44:55 +0000 (11:44 -0500)]
14198: Fix jobstep builder init

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix ruamel.yaml versioning
Peter Amstutz [Fri, 16 Nov 2018 16:28:27 +0000 (11:28 -0500)]
14198: Fix ruamel.yaml versioning

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Construct ArvadosWorkflowStep
Peter Amstutz [Fri, 16 Nov 2018 16:26:51 +0000 (11:26 -0500)]
14198: Construct ArvadosWorkflowStep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Update version deps for schema-salad, cwltool, ruamel.yaml, requests
Peter Amstutz [Fri, 16 Nov 2018 15:28:24 +0000 (10:28 -0500)]
14198: Update version deps for schema-salad, cwltool, ruamel.yaml, requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Update cwltool and schema-salad
Peter Amstutz [Wed, 14 Nov 2018 18:22:16 +0000 (13:22 -0500)]
14198: Update cwltool and schema-salad

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add --always-submit-runner
Peter Amstutz [Wed, 7 Nov 2018 21:24:02 +0000 (16:24 -0500)]
14198: Add --always-submit-runner

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add --submit-runner-cluster test
Peter Amstutz [Mon, 5 Nov 2018 22:44:23 +0000 (17:44 -0500)]
14198: Add --submit-runner-cluster test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Unit tests pass again
Peter Amstutz [Mon, 5 Nov 2018 15:16:48 +0000 (10:16 -0500)]
14198: Unit tests pass again

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: test_submit passes
Peter Amstutz [Fri, 2 Nov 2018 21:05:20 +0000 (17:05 -0400)]
14198: test_submit passes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: More test updates
Peter Amstutz [Fri, 2 Nov 2018 20:41:01 +0000 (16:41 -0400)]
14198: More test updates

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Bringing unit tests up to date WIP
Peter Amstutz [Fri, 2 Nov 2018 20:05:29 +0000 (16:05 -0400)]
14198: Bringing unit tests up to date WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add paging on container list, check for valid cluster id
Peter Amstutz [Thu, 1 Nov 2018 18:42:12 +0000 (14:42 -0400)]
14198: Add paging on container list, check for valid cluster id

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Use ArvRuntimeContext.submit_runner_cluster for target cluster
Peter Amstutz [Wed, 31 Oct 2018 17:25:24 +0000 (13:25 -0400)]
14198: Use ArvRuntimeContext.submit_runner_cluster for target cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix typo current -> current_container, add copyright header
Peter Amstutz [Thu, 25 Oct 2018 21:21:05 +0000 (17:21 -0400)]
14198: Fix typo current -> current_container, add copyright header

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Log current container
Peter Amstutz [Thu, 25 Oct 2018 21:11:59 +0000 (17:11 -0400)]
14198: Log current container

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Support expressions in TargetCluster[clusterID, ownerUUID]
Peter Amstutz [Thu, 25 Oct 2018 20:14:50 +0000 (16:14 -0400)]
14198: Support expressions in TargetCluster[clusterID, ownerUUID]

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Resolve to Docker images to PDH and set "http://arvados.org/cwl#dockerCollecti...
Peter Amstutz [Thu, 25 Oct 2018 19:19:05 +0000 (15:19 -0400)]
14198: Resolve to Docker images to PDH and set "arvados.org/cwl#dockerCollectionPDH"

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Refactor and add support for --submit-runner-cluster
Peter Amstutz [Thu, 25 Oct 2018 19:02:23 +0000 (15:02 -0400)]
14198: Refactor and add support for --submit-runner-cluster

Rename ArvCwlRunner to ArvCwlExecutor and move into its own file.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Use PDH for container_image instead of docker repo+tag
Peter Amstutz [Thu, 25 Oct 2018 14:36:47 +0000 (10:36 -0400)]
14198: Use PDH for container_image instead of docker repo+tag

Needed to support federated container requests.

This reverts 6ea807b2caf6c934f170b2e4d89c23c4a08ca69c

Based on the commit comment, that change was made to accomodate the
Docker v1 to v2 image format migration (to enable the API server to
select the image with the correct format).  However, the API server
subsequently gained the ability to detect if it needed to substitute a
PDH with a migrated image PDH in commit
a72205728f94f5261b657766e01f5767dc15d4b5 so now we want restore the
original behavior of locally resolving the image PDH and using that in
the container request.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Initial support ClusterTarget hint
Peter Amstutz [Wed, 24 Oct 2018 18:45:21 +0000 (14:45 -0400)]
14198: Initial support ClusterTarget hint

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14345: Support PROPPATCH method.
Tom Clegg [Wed, 21 Nov 2018 16:46:34 +0000 (11:46 -0500)]
14345: Support PROPPATCH method.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14516: when calculating the next available username, make sure to take merged
Ward Vandewege [Tue, 20 Nov 2018 18:45:14 +0000 (13:45 -0500)]
14516: when calculating the next available username, make sure to take merged
accounts into consideration - they have usernames too, and we have a unique
index on username in the users table.

refs #14516

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoCWL arvados-tests.sh uses --api=containers no issue #
Peter Amstutz [Tue, 20 Nov 2018 14:23:25 +0000 (09:23 -0500)]
CWL arvados-tests.sh uses --api=containers no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14497-rack-upgrade'
Lucas Di Pentima [Fri, 16 Nov 2018 16:44:42 +0000 (13:44 -0300)]
Merge branch '14497-rack-upgrade'
Closes #14497

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14497: Upgrades rack to 1.6.11 on api & workbench.
Lucas Di Pentima [Fri, 16 Nov 2018 15:07:32 +0000 (12:07 -0300)]
14497: Upgrades rack to 1.6.11 on api & workbench.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoAdd --no-tty to apt-key in arvbox dockerfile. no issue #
Peter Amstutz [Wed, 14 Nov 2018 21:35:30 +0000 (16:35 -0500)]
Add --no-tty to apt-key in arvbox dockerfile.  no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14419-concurrent-map-write'
Tom Clegg [Wed, 14 Nov 2018 21:05:34 +0000 (16:05 -0500)]
Merge branch '14419-concurrent-map-write'

fixes #14419

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoThe Dockerfiles for package building and testing on debian 9 seem to want
Ward Vandewege [Wed, 14 Nov 2018 20:57:32 +0000 (15:57 -0500)]
The Dockerfiles for package building and testing on debian 9 seem to want
--no-tty as an argument to gpg calls now.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoMerge branch '14198-request-token' refs #14198
Peter Amstutz [Wed, 14 Nov 2018 19:07:18 +0000 (14:07 -0500)]
Merge branch '14198-request-token' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add test that runtime token is set from auth for remote user
Peter Amstutz [Wed, 14 Nov 2018 18:53:47 +0000 (13:53 -0500)]
14198: Add test that runtime token is set from auth for remote user

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14419: Fix concurrent map write.
Tom Clegg [Wed, 14 Nov 2018 18:49:39 +0000 (13:49 -0500)]
14419: Fix concurrent map write.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoUpdate tests to match new template.
Tom Clegg [Tue, 13 Nov 2018 21:56:27 +0000 (16:56 -0500)]
Update tests to match new template.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix git command.
Tom Clegg [Tue, 13 Nov 2018 21:19:40 +0000 (16:19 -0500)]
Fix git command.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: Strip 4th position when setting runtime_token from provided auth
Peter Amstutz [Tue, 13 Nov 2018 21:18:09 +0000 (16:18 -0500)]
14198: Strip 4th position when setting runtime_token from provided auth

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14475-update-collection-error'
Tom Clegg [Tue, 13 Nov 2018 21:18:59 +0000 (16:18 -0500)]
Merge branch '14475-update-collection-error'

fixes #14475

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14475: Fix Content-Type header parsing.
Tom Clegg [Tue, 13 Nov 2018 20:49:31 +0000 (15:49 -0500)]
14475: Fix Content-Type header parsing.

"Content-Type: application/x-www-form-urlencoded; charset=UTF-8", for
example, means we should parse the request body as a form.

See https://github.com/golang/go/blob/master/src/net/http/request.go
-> parsePostForm().

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14397-keepstore-v4-signature'
Tom Clegg [Tue, 13 Nov 2018 19:51:08 +0000 (14:51 -0500)]
Merge branch '14397-keepstore-v4-signature'

fixes #14397

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Add comment about memory implications of providing SHA256.
Tom Clegg [Tue, 13 Nov 2018 16:43:28 +0000 (11:43 -0500)]
14397: Add comment about memory implications of providing SHA256.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: Remote user uses current token in runtime_token
Peter Amstutz [Tue, 13 Nov 2018 16:42:19 +0000 (11:42 -0500)]
14198: Remote user uses current token in runtime_token

Can't create a new token for a remote user, so use the token that was
supplied.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoIn the notification e-mail to the user that a shell account has been created,
Ward Vandewege [Wed, 7 Nov 2018 15:58:36 +0000 (10:58 -0500)]
In the notification e-mail to the user that a shell account has been created,
change the title of the e-mail to mention Arvados instead of Curoverse.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago14397: Pass SHA256 to s3 library.
Tom Clegg [Mon, 12 Nov 2018 18:12:28 +0000 (13:12 -0500)]
14397: Pass SHA256 to s3 library.

Otherwise, the s3 library copies the data into another new buffer,
doubling the memory needed for a write operation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Use {sha1}-dev as version in dev builds of Go programs.
Tom Clegg [Mon, 12 Nov 2018 15:40:27 +0000 (10:40 -0500)]
14397: Use {sha1}-dev as version in dev builds of Go programs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Fix unchecked errors in S3 block listing.
Tom Clegg [Mon, 12 Nov 2018 14:27:12 +0000 (09:27 -0500)]
14397: Fix unchecked errors in S3 block listing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Fix S3 ListOps stats.
Tom Clegg [Mon, 12 Nov 2018 14:24:20 +0000 (09:24 -0500)]
14397: Fix S3 ListOps stats.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Improve error reporting.
Tom Clegg [Fri, 9 Nov 2018 20:02:54 +0000 (15:02 -0500)]
14397: Improve error reporting.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Update goamz to fix v4 signature.
Tom Clegg [Fri, 9 Nov 2018 19:53:08 +0000 (14:53 -0500)]
14397: Update goamz to fix v4 signature.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: WIP handling various cases of remote users submitting container requests
Peter Amstutz [Fri, 9 Nov 2018 21:40:27 +0000 (16:40 -0500)]
14198: WIP handling various cases of remote users submitting container requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14198-fed-collection-list' refs #14198
Peter Amstutz [Fri, 9 Nov 2018 20:50:53 +0000 (15:50 -0500)]
Merge branch '14198-fed-collection-list' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Expand federation support for collections and links
Peter Amstutz [Fri, 9 Nov 2018 19:24:00 +0000 (14:24 -0500)]
14198: Expand federation support for collections and links

Add collections and links to the list of endpoints where requests
bearing cluster_id are routed through controller to remote clusters.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14328-containerd'
Tom Clegg [Fri, 9 Nov 2018 19:00:54 +0000 (14:00 -0500)]
Merge branch '14328-containerd'

closes #14328

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14328: Merge branch 'master'
Tom Clegg [Fri, 9 Nov 2018 18:59:29 +0000 (13:59 -0500)]
14328: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14328: Fix flaky test.
Tom Clegg [Fri, 9 Nov 2018 18:55:00 +0000 (13:55 -0500)]
14328: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14323-container-collection-mounts'
Lucas Di Pentima [Fri, 9 Nov 2018 16:44:55 +0000 (13:44 -0300)]
Merge branch '14323-container-collection-mounts'
Closes #14323

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '14198-crunch-run-tokens' refs #14198
Peter Amstutz [Fri, 9 Nov 2018 16:37:59 +0000 (11:37 -0500)]
Merge branch '14198-crunch-run-tokens' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add comment about different client objects used by runner
Peter Amstutz [Fri, 9 Nov 2018 16:36:01 +0000 (11:36 -0500)]
14198: Add comment about different client objects used by runner

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14259-pysdk-remote-block-copy'
Lucas Di Pentima [Fri, 9 Nov 2018 16:26:13 +0000 (13:26 -0300)]
Merge branch '14259-pysdk-remote-block-copy'
Closes #14259

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14323: Adds test for the empty collection mount resolution case.
Lucas Di Pentima [Fri, 9 Nov 2018 16:10:11 +0000 (13:10 -0300)]
14323: Adds test for the empty collection mount resolution case.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14198: Use container token for reading input/creating output
Peter Amstutz [Fri, 9 Nov 2018 02:32:17 +0000 (21:32 -0500)]
14198: Use container token for reading input/creating output

Refactor a bit to make it obvious which token is being used when
communicating with the API server.  Operations that could involve
federated collections (reading the Docker image, reading inputs,
copying remote blocks to local) must use the container token.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14328: Remove containerd watchdog.
Tom Clegg [Thu, 8 Nov 2018 22:05:22 +0000 (17:05 -0500)]
14328: Remove containerd watchdog.

No longer needed, now that we periodically ask docker-inspect to check
on the specific container we're running.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14345-webdav-lock-and-empty-dir'
Tom Clegg [Thu, 8 Nov 2018 21:57:19 +0000 (16:57 -0500)]
Merge branch '14345-webdav-lock-and-empty-dir'

refs #14345

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14323: Only get collection mount by UUID when PDH is missing and UUID provided.
Lucas Di Pentima [Thu, 8 Nov 2018 20:53:09 +0000 (17:53 -0300)]
14323: Only get collection mount by UUID when PDH is missing and UUID provided.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14259: Improvements on remote blocks copying logic.
Lucas Di Pentima [Thu, 8 Nov 2018 19:57:13 +0000 (16:57 -0300)]
14259: Improvements on remote blocks copying logic.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '14458-controller-panic' refs #14458
Peter Amstutz [Thu, 8 Nov 2018 18:25:05 +0000 (13:25 -0500)]
Merge branch '14458-controller-panic' refs #14458

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14458: Fix syntax error
Peter Amstutz [Thu, 8 Nov 2018 18:21:53 +0000 (13:21 -0500)]
14458: Fix syntax error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14458: Drain errorChan without waiting for close
Peter Amstutz [Thu, 8 Nov 2018 18:18:59 +0000 (13:18 -0500)]
14458: Drain errorChan without waiting for close

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14345: Handle "MOVE foo/ bar/" requests.
Tom Clegg [Tue, 6 Nov 2018 18:57:14 +0000 (13:57 -0500)]
14345: Handle "MOVE foo/ bar/" requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Accept "." placeholder: create parent dir, but no fake file.
Tom Clegg [Tue, 6 Nov 2018 14:54:10 +0000 (09:54 -0500)]
14345: Accept "." placeholder: create parent dir, but no fake file.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Accept lock/unlock requests as no-ops.
Tom Clegg [Thu, 18 Oct 2018 13:41:37 +0000 (09:41 -0400)]
14345: Accept lock/unlock requests as no-ops.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Unescape all \ooo in names in manifests, not just \040.
Tom Clegg [Mon, 5 Nov 2018 21:46:50 +0000 (16:46 -0500)]
14345: Unescape all \ooo in names in manifests, not just \040.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>