arvados.git
6 years ago14198: Resolve to Docker images to PDH and set "http://arvados.org/cwl#dockerCollecti...
Peter Amstutz [Thu, 25 Oct 2018 19:19:05 +0000 (15:19 -0400)]
14198: Resolve to Docker images to PDH and set "arvados.org/cwl#dockerCollectionPDH"

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Refactor and add support for --submit-runner-cluster
Peter Amstutz [Thu, 25 Oct 2018 19:02:23 +0000 (15:02 -0400)]
14198: Refactor and add support for --submit-runner-cluster

Rename ArvCwlRunner to ArvCwlExecutor and move into its own file.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Use PDH for container_image instead of docker repo+tag
Peter Amstutz [Thu, 25 Oct 2018 14:36:47 +0000 (10:36 -0400)]
14198: Use PDH for container_image instead of docker repo+tag

Needed to support federated container requests.

This reverts 6ea807b2caf6c934f170b2e4d89c23c4a08ca69c

Based on the commit comment, that change was made to accomodate the
Docker v1 to v2 image format migration (to enable the API server to
select the image with the correct format).  However, the API server
subsequently gained the ability to detect if it needed to substitute a
PDH with a migrated image PDH in commit
a72205728f94f5261b657766e01f5767dc15d4b5 so now we want restore the
original behavior of locally resolving the image PDH and using that in
the container request.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Initial support ClusterTarget hint
Peter Amstutz [Wed, 24 Oct 2018 18:45:21 +0000 (14:45 -0400)]
14198: Initial support ClusterTarget hint

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14516: when calculating the next available username, make sure to take merged
Ward Vandewege [Tue, 20 Nov 2018 18:45:14 +0000 (13:45 -0500)]
14516: when calculating the next available username, make sure to take merged
accounts into consideration - they have usernames too, and we have a unique
index on username in the users table.

refs #14516

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoCWL arvados-tests.sh uses --api=containers no issue #
Peter Amstutz [Tue, 20 Nov 2018 14:23:25 +0000 (09:23 -0500)]
CWL arvados-tests.sh uses --api=containers no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14497-rack-upgrade'
Lucas Di Pentima [Fri, 16 Nov 2018 16:44:42 +0000 (13:44 -0300)]
Merge branch '14497-rack-upgrade'
Closes #14497

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14497: Upgrades rack to 1.6.11 on api & workbench.
Lucas Di Pentima [Fri, 16 Nov 2018 15:07:32 +0000 (12:07 -0300)]
14497: Upgrades rack to 1.6.11 on api & workbench.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoAdd --no-tty to apt-key in arvbox dockerfile. no issue #
Peter Amstutz [Wed, 14 Nov 2018 21:35:30 +0000 (16:35 -0500)]
Add --no-tty to apt-key in arvbox dockerfile.  no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14419-concurrent-map-write'
Tom Clegg [Wed, 14 Nov 2018 21:05:34 +0000 (16:05 -0500)]
Merge branch '14419-concurrent-map-write'

fixes #14419

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoThe Dockerfiles for package building and testing on debian 9 seem to want
Ward Vandewege [Wed, 14 Nov 2018 20:57:32 +0000 (15:57 -0500)]
The Dockerfiles for package building and testing on debian 9 seem to want
--no-tty as an argument to gpg calls now.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoMerge branch '14198-request-token' refs #14198
Peter Amstutz [Wed, 14 Nov 2018 19:07:18 +0000 (14:07 -0500)]
Merge branch '14198-request-token' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add test that runtime token is set from auth for remote user
Peter Amstutz [Wed, 14 Nov 2018 18:53:47 +0000 (13:53 -0500)]
14198: Add test that runtime token is set from auth for remote user

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14419: Fix concurrent map write.
Tom Clegg [Wed, 14 Nov 2018 18:49:39 +0000 (13:49 -0500)]
14419: Fix concurrent map write.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoUpdate tests to match new template.
Tom Clegg [Tue, 13 Nov 2018 21:56:27 +0000 (16:56 -0500)]
Update tests to match new template.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix git command.
Tom Clegg [Tue, 13 Nov 2018 21:19:40 +0000 (16:19 -0500)]
Fix git command.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: Strip 4th position when setting runtime_token from provided auth
Peter Amstutz [Tue, 13 Nov 2018 21:18:09 +0000 (16:18 -0500)]
14198: Strip 4th position when setting runtime_token from provided auth

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14475-update-collection-error'
Tom Clegg [Tue, 13 Nov 2018 21:18:59 +0000 (16:18 -0500)]
Merge branch '14475-update-collection-error'

fixes #14475

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14475: Fix Content-Type header parsing.
Tom Clegg [Tue, 13 Nov 2018 20:49:31 +0000 (15:49 -0500)]
14475: Fix Content-Type header parsing.

"Content-Type: application/x-www-form-urlencoded; charset=UTF-8", for
example, means we should parse the request body as a form.

See https://github.com/golang/go/blob/master/src/net/http/request.go
-> parsePostForm().

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14397-keepstore-v4-signature'
Tom Clegg [Tue, 13 Nov 2018 19:51:08 +0000 (14:51 -0500)]
Merge branch '14397-keepstore-v4-signature'

fixes #14397

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Add comment about memory implications of providing SHA256.
Tom Clegg [Tue, 13 Nov 2018 16:43:28 +0000 (11:43 -0500)]
14397: Add comment about memory implications of providing SHA256.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: Remote user uses current token in runtime_token
Peter Amstutz [Tue, 13 Nov 2018 16:42:19 +0000 (11:42 -0500)]
14198: Remote user uses current token in runtime_token

Can't create a new token for a remote user, so use the token that was
supplied.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoIn the notification e-mail to the user that a shell account has been created,
Ward Vandewege [Wed, 7 Nov 2018 15:58:36 +0000 (10:58 -0500)]
In the notification e-mail to the user that a shell account has been created,
change the title of the e-mail to mention Arvados instead of Curoverse.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago14397: Pass SHA256 to s3 library.
Tom Clegg [Mon, 12 Nov 2018 18:12:28 +0000 (13:12 -0500)]
14397: Pass SHA256 to s3 library.

Otherwise, the s3 library copies the data into another new buffer,
doubling the memory needed for a write operation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Use {sha1}-dev as version in dev builds of Go programs.
Tom Clegg [Mon, 12 Nov 2018 15:40:27 +0000 (10:40 -0500)]
14397: Use {sha1}-dev as version in dev builds of Go programs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Fix unchecked errors in S3 block listing.
Tom Clegg [Mon, 12 Nov 2018 14:27:12 +0000 (09:27 -0500)]
14397: Fix unchecked errors in S3 block listing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Fix S3 ListOps stats.
Tom Clegg [Mon, 12 Nov 2018 14:24:20 +0000 (09:24 -0500)]
14397: Fix S3 ListOps stats.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Improve error reporting.
Tom Clegg [Fri, 9 Nov 2018 20:02:54 +0000 (15:02 -0500)]
14397: Improve error reporting.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Update goamz to fix v4 signature.
Tom Clegg [Fri, 9 Nov 2018 19:53:08 +0000 (14:53 -0500)]
14397: Update goamz to fix v4 signature.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: WIP handling various cases of remote users submitting container requests
Peter Amstutz [Fri, 9 Nov 2018 21:40:27 +0000 (16:40 -0500)]
14198: WIP handling various cases of remote users submitting container requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14198-fed-collection-list' refs #14198
Peter Amstutz [Fri, 9 Nov 2018 20:50:53 +0000 (15:50 -0500)]
Merge branch '14198-fed-collection-list' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Expand federation support for collections and links
Peter Amstutz [Fri, 9 Nov 2018 19:24:00 +0000 (14:24 -0500)]
14198: Expand federation support for collections and links

Add collections and links to the list of endpoints where requests
bearing cluster_id are routed through controller to remote clusters.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14328-containerd'
Tom Clegg [Fri, 9 Nov 2018 19:00:54 +0000 (14:00 -0500)]
Merge branch '14328-containerd'

closes #14328

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14328: Merge branch 'master'
Tom Clegg [Fri, 9 Nov 2018 18:59:29 +0000 (13:59 -0500)]
14328: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14328: Fix flaky test.
Tom Clegg [Fri, 9 Nov 2018 18:55:00 +0000 (13:55 -0500)]
14328: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14323-container-collection-mounts'
Lucas Di Pentima [Fri, 9 Nov 2018 16:44:55 +0000 (13:44 -0300)]
Merge branch '14323-container-collection-mounts'
Closes #14323

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '14198-crunch-run-tokens' refs #14198
Peter Amstutz [Fri, 9 Nov 2018 16:37:59 +0000 (11:37 -0500)]
Merge branch '14198-crunch-run-tokens' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add comment about different client objects used by runner
Peter Amstutz [Fri, 9 Nov 2018 16:36:01 +0000 (11:36 -0500)]
14198: Add comment about different client objects used by runner

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14259-pysdk-remote-block-copy'
Lucas Di Pentima [Fri, 9 Nov 2018 16:26:13 +0000 (13:26 -0300)]
Merge branch '14259-pysdk-remote-block-copy'
Closes #14259

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14323: Adds test for the empty collection mount resolution case.
Lucas Di Pentima [Fri, 9 Nov 2018 16:10:11 +0000 (13:10 -0300)]
14323: Adds test for the empty collection mount resolution case.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14198: Use container token for reading input/creating output
Peter Amstutz [Fri, 9 Nov 2018 02:32:17 +0000 (21:32 -0500)]
14198: Use container token for reading input/creating output

Refactor a bit to make it obvious which token is being used when
communicating with the API server.  Operations that could involve
federated collections (reading the Docker image, reading inputs,
copying remote blocks to local) must use the container token.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14328: Remove containerd watchdog.
Tom Clegg [Thu, 8 Nov 2018 22:05:22 +0000 (17:05 -0500)]
14328: Remove containerd watchdog.

No longer needed, now that we periodically ask docker-inspect to check
on the specific container we're running.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14345-webdav-lock-and-empty-dir'
Tom Clegg [Thu, 8 Nov 2018 21:57:19 +0000 (16:57 -0500)]
Merge branch '14345-webdav-lock-and-empty-dir'

refs #14345

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14323: Only get collection mount by UUID when PDH is missing and UUID provided.
Lucas Di Pentima [Thu, 8 Nov 2018 20:53:09 +0000 (17:53 -0300)]
14323: Only get collection mount by UUID when PDH is missing and UUID provided.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14259: Improvements on remote blocks copying logic.
Lucas Di Pentima [Thu, 8 Nov 2018 19:57:13 +0000 (16:57 -0300)]
14259: Improvements on remote blocks copying logic.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '14458-controller-panic' refs #14458
Peter Amstutz [Thu, 8 Nov 2018 18:25:05 +0000 (13:25 -0500)]
Merge branch '14458-controller-panic' refs #14458

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14458: Fix syntax error
Peter Amstutz [Thu, 8 Nov 2018 18:21:53 +0000 (13:21 -0500)]
14458: Fix syntax error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14458: Drain errorChan without waiting for close
Peter Amstutz [Thu, 8 Nov 2018 18:18:59 +0000 (13:18 -0500)]
14458: Drain errorChan without waiting for close

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14345: Handle "MOVE foo/ bar/" requests.
Tom Clegg [Tue, 6 Nov 2018 18:57:14 +0000 (13:57 -0500)]
14345: Handle "MOVE foo/ bar/" requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Accept "." placeholder: create parent dir, but no fake file.
Tom Clegg [Tue, 6 Nov 2018 14:54:10 +0000 (09:54 -0500)]
14345: Accept "." placeholder: create parent dir, but no fake file.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Accept lock/unlock requests as no-ops.
Tom Clegg [Thu, 18 Oct 2018 13:41:37 +0000 (09:41 -0400)]
14345: Accept lock/unlock requests as no-ops.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Unescape all \ooo in names in manifests, not just \040.
Tom Clegg [Mon, 5 Nov 2018 21:46:50 +0000 (16:46 -0500)]
14345: Unescape all \ooo in names in manifests, not just \040.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago"arvbox status" includes cluster id refs #14198
Peter Amstutz [Thu, 8 Nov 2018 16:16:53 +0000 (11:16 -0500)]
"arvbox status" includes cluster id refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14458: Drain errors channel on failure case instead of accumulating separately
Peter Amstutz [Thu, 8 Nov 2018 15:55:47 +0000 (10:55 -0500)]
14458: Drain errors channel on failure case instead of accumulating separately

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14458: Avoid panic by removing defer close()
Peter Amstutz [Wed, 7 Nov 2018 20:53:08 +0000 (15:53 -0500)]
14458: Avoid panic by removing defer close()

Channels will be garbage collected when they go unreferenced.

Make the errors channel buffered to avoid lingering goroutines trying
to send on an unbuffered error channel with no reciever.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14198-index-accepts-cluster-id' refs #14198
Peter Amstutz [Thu, 8 Nov 2018 16:03:55 +0000 (11:03 -0500)]
Merge branch '14198-index-accepts-cluster-id' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14439-federation-admin-docs'
Lucas Di Pentima [Wed, 7 Nov 2018 21:11:32 +0000 (18:11 -0300)]
Merge branch '14439-federation-admin-docs'
Refs #14439

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14323: Fixes indentation.
Lucas Di Pentima [Wed, 7 Nov 2018 21:10:19 +0000 (18:10 -0300)]
14323: Fixes indentation.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14439: Adds federation configuration page to the admin docs.
Lucas Di Pentima [Wed, 7 Nov 2018 20:43:20 +0000 (17:43 -0300)]
14439: Adds federation configuration page to the admin docs.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14198: Add cluster_id to index parameters
Peter Amstutz [Wed, 7 Nov 2018 19:38:45 +0000 (14:38 -0500)]
14198: Add cluster_id to index parameters

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14345: Use "." placeholder to persist empty directories.
Tom Clegg [Mon, 5 Nov 2018 21:15:15 +0000 (16:15 -0500)]
14345: Use "." placeholder to persist empty directories.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14323: Lookup collection mount by uuid when pdh is not given. Update tests.
Lucas Di Pentima [Wed, 7 Nov 2018 17:48:31 +0000 (14:48 -0300)]
14323: Lookup collection mount by uuid when pdh is not given. Update tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14323: Prioritize PDH over UUID for mount resolving when both provided.
Lucas Di Pentima [Wed, 7 Nov 2018 14:05:03 +0000 (11:05 -0300)]
14323: Prioritize PDH over UUID for mount resolving when both provided.

* Don't delete uuid after mount resolving.
* Update tests

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '14455-loofah-upgrade'
Lucas Di Pentima [Tue, 6 Nov 2018 19:29:48 +0000 (16:29 -0300)]
Merge branch '14455-loofah-upgrade'
Closes #14455

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '14406-remote-data-in-output'
Tom Clegg [Tue, 6 Nov 2018 19:26:56 +0000 (14:26 -0500)]
Merge branch '14406-remote-data-in-output'

refs #14406

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14406: Merge branch 'master'
Tom Clegg [Tue, 6 Nov 2018 18:58:53 +0000 (13:58 -0500)]
14406: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14455: Update loofah & nokogiri on workbench & api for security reasons.
Lucas Di Pentima [Tue, 6 Nov 2018 18:41:07 +0000 (15:41 -0300)]
14455: Update loofah & nokogiri on workbench & api for security reasons.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14406: Only call LocalLocator() once per block per stream.
Tom Clegg [Tue, 6 Nov 2018 16:13:05 +0000 (11:13 -0500)]
14406: Only call LocalLocator() once per block per stream.

Avoid excess calls when many files are packed in a single block.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch 'master' into 14259-pysdk-remote-block-copy
Lucas Di Pentima [Tue, 6 Nov 2018 14:41:06 +0000 (11:41 -0300)]
Merge branch 'master' into 14259-pysdk-remote-block-copy

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14259: Don't scan the entire collection for remote blocks when not needed.
Lucas Di Pentima [Tue, 6 Nov 2018 14:39:14 +0000 (11:39 -0300)]
14259: Don't scan the entire collection for remote blocks when not needed.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14406: Copy remote blocks to local storage during MarshalManifest.
Tom Clegg [Mon, 5 Nov 2018 14:58:40 +0000 (09:58 -0500)]
14406: Copy remote blocks to local storage during MarshalManifest.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14262-remote-container' refs #14262
Peter Amstutz [Thu, 1 Nov 2018 19:57:09 +0000 (15:57 -0400)]
Merge branch '14262-remote-container' refs #14262

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14406: Tone down slow stress test.
Tom Clegg [Thu, 1 Nov 2018 19:22:36 +0000 (15:22 -0400)]
14406: Tone down slow stress test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14262: Avoid out-of-bounds panics checking cluster prefixes
Peter Amstutz [Thu, 1 Nov 2018 19:14:14 +0000 (15:14 -0400)]
14262: Avoid out-of-bounds panics checking cluster prefixes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14406: Remove debug printfs.
Tom Clegg [Thu, 1 Nov 2018 18:54:13 +0000 (14:54 -0400)]
14406: Remove debug printfs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14262: Fix remote_user_test.rb so it doesn't mess up remote_hosts
Peter Amstutz [Thu, 1 Nov 2018 18:53:37 +0000 (14:53 -0400)]
14262: Fix remote_user_test.rb so it doesn't mess up remote_hosts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14406: Fix safety warning.
Tom Clegg [Thu, 1 Nov 2018 17:56:59 +0000 (13:56 -0400)]
14406: Fix safety warning.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14406: Pass output through collectionFS if it has remote block refs.
Tom Clegg [Thu, 1 Nov 2018 17:56:13 +0000 (13:56 -0400)]
14406: Pass output through collectionFS if it has remote block refs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14262: Revert changes to client.go
Peter Amstutz [Thu, 1 Nov 2018 15:04:01 +0000 (11:04 -0400)]
14262: Revert changes to client.go

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Move the context deadline to the top of the handler stack
Peter Amstutz [Thu, 1 Nov 2018 14:19:18 +0000 (10:19 -0400)]
14262: Move the context deadline to the top of the handler stack

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Fix permissions so runtime_token can set container progress/output
Peter Amstutz [Wed, 31 Oct 2018 20:40:19 +0000 (16:40 -0400)]
14262: Fix permissions so runtime_token can set container progress/output

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Only create runtime_token on home cluster for the authorization
Peter Amstutz [Wed, 31 Oct 2018 17:45:58 +0000 (13:45 -0400)]
14262: Only create runtime_token on home cluster for the authorization

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14259: Iterate just once when scanning for remote blocks.
Lucas Di Pentima [Wed, 31 Oct 2018 18:43:46 +0000 (15:43 -0300)]
14259: Iterate just once when scanning for remote blocks.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14259: Don't enforce bandwidth timeouts on keep client HEAD requests.
Lucas Di Pentima [Wed, 31 Oct 2018 18:16:22 +0000 (15:16 -0300)]
14259: Don't enforce bandwidth timeouts on keep client HEAD requests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14259: Test updates & additions related to keep client HEAD requests
Lucas Di Pentima [Wed, 31 Oct 2018 18:12:14 +0000 (15:12 -0300)]
14259: Test updates & additions related to keep client HEAD requests

* Test updates to confirm that when doing HEAD requests, no bandwidth timeout
are enforced.
* Test additions to prove that HEAD request responses don't get cached.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14262: Fix spillover of tests changing remote_hosts
Peter Amstutz [Wed, 31 Oct 2018 16:53:25 +0000 (12:53 -0400)]
14262: Fix spillover of tests changing remote_hosts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Rewrite collectionFederatedRequestHandler PDH search to use channels
Peter Amstutz [Mon, 29 Oct 2018 21:09:08 +0000 (17:09 -0400)]
14262: Rewrite collectionFederatedRequestHandler PDH search to use channels

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Make sure cancel() from proxy.Do() gets called
Peter Amstutz [Mon, 29 Oct 2018 19:36:45 +0000 (15:36 -0400)]
14262: Make sure cancel() from proxy.Do() gets called

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Only allow unknown PDH for images when there are remote_hosts
Peter Amstutz [Mon, 29 Oct 2018 18:56:26 +0000 (14:56 -0400)]
14262: Only allow unknown PDH for images when there are remote_hosts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Fix logic for when to allow unknown PDH for containers
Peter Amstutz [Wed, 24 Oct 2018 19:30:34 +0000 (15:30 -0400)]
14262: Fix logic for when to allow unknown PDH for containers

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix error responses in container POST
Peter Amstutz [Wed, 24 Oct 2018 18:44:25 +0000 (14:44 -0400)]
14198: Fix error responses in container POST

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Handle container_request posted as a string parameter
Peter Amstutz [Tue, 23 Oct 2018 18:30:39 +0000 (14:30 -0400)]
14262: Handle container_request posted as a string parameter

Needs to be parsed as json second time (this is how the Ruby 'arv'
client submits it, unfortunately.)

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Fix crunch-run tests
Peter Amstutz [Tue, 23 Oct 2018 17:29:53 +0000 (13:29 -0400)]
14262: Fix crunch-run tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Missing file
Peter Amstutz [Tue, 23 Oct 2018 13:26:48 +0000 (09:26 -0400)]
14262: Missing file

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Fix tests
Peter Amstutz [Mon, 22 Oct 2018 21:01:52 +0000 (17:01 -0400)]
14262: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Tests for setting and checking container tokens.
Peter Amstutz [Mon, 22 Oct 2018 16:13:38 +0000 (12:13 -0400)]
14262: Tests for setting and checking container tokens.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Add createAPIToken, with test
Peter Amstutz [Mon, 22 Oct 2018 15:02:02 +0000 (11:02 -0400)]
14262: Add createAPIToken, with test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Refactoring, split up federation code into smaller files
Peter Amstutz [Thu, 18 Oct 2018 21:17:33 +0000 (17:17 -0400)]
14262: Refactoring, split up federation code into smaller files

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Refactoring proxy
Peter Amstutz [Thu, 18 Oct 2018 20:08:28 +0000 (16:08 -0400)]
14262: Refactoring proxy

Split proxy.Do() into ForwardRequest() and ForwardResponse().

Inversion of control eliminates need for "filter" callback, since the
caller can now modify the response in between the calls to
ForwardRequest() and ForwardResponse().

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14262: Fix bug moving api_token to header
Peter Amstutz [Thu, 18 Oct 2018 18:34:49 +0000 (14:34 -0400)]
14262: Fix bug moving api_token to header

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>