arvados.git
3 years agoMerge branch 'master' into 17014-controller-container-requests 17014-controller-container-requests
Nico Cesar [Mon, 14 Dec 2020 18:34:23 +0000 (13:34 -0500)]
Merge branch 'master' into 17014-controller-container-requests

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agoImproved integration_test.go
Nico Cesar [Mon, 14 Dec 2020 18:29:42 +0000 (13:29 -0500)]
Improved integration_test.go

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 13 Dec 2020 19:26:57 +0000 (14:26 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sat, 12 Dec 2020 20:09:48 +0000 (15:09 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agocr.Container should be a string not a *string
Nico Cesar [Fri, 11 Dec 2020 21:55:49 +0000 (16:55 -0500)]
cr.Container should be a string not a *string

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agofirst commit with all test passing
Nico Cesar [Fri, 11 Dec 2020 20:46:40 +0000 (15:46 -0500)]
first commit with all test passing

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agoFailing tests for Ward to look at
Nico Cesar [Fri, 11 Dec 2020 18:26:21 +0000 (13:26 -0500)]
Failing tests for Ward to look at

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agoleftovers from a bad rebase
Nico Cesar [Thu, 10 Dec 2020 20:52:13 +0000 (15:52 -0500)]
leftovers from a bad rebase

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agocontainer requests in the new codepath for controller
Nico Cesar [Tue, 10 Nov 2020 14:00:52 +0000 (09:00 -0500)]
container requests in the new codepath for controller

Controller's "new" codepath is a step towards sunsetting railsapi. This
commit is the initial work to get Container Requests in that path

The reason for deleting some tests is intentional. This is part of the migration
from
   apps/workbench/test/controllers/container_requests_controller_test.rb
to
   lib/controller/integration_test.go

This single-commit is the result of collaboration with Lucas Di Pentima
and Tom Clegg in the branch 17014-controller-cr

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>

3 years agoCostanalyzer bugfix: when a collection uuid is supplied, write the csv
Ward Vandewege [Thu, 10 Dec 2020 15:33:57 +0000 (10:33 -0500)]
Costanalyzer bugfix: when a collection uuid is supplied, write the csv
files with the name of the associated container request uuid, not the
collection uuid. Also change tests to use a proper temporary directory
for the outputs, and do not reuse that directory between tests (which
hid the bug this commit fixed).

refs #17187

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17202: Test avoiding redirect for cross-origin inline images.
Tom Clegg [Wed, 9 Dec 2020 20:44:57 +0000 (15:44 -0500)]
17202: Test avoiding redirect for cross-origin inline images.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17202: Use explicit SameSite=Lax for 303-with-cookie.
Tom Clegg [Wed, 9 Dec 2020 14:34:14 +0000 (09:34 -0500)]
17202: Use explicit SameSite=Lax for 303-with-cookie.

This improves XSS protection on some browsers, including Safari and
Firefox for Android.

On most browsers, Lax is already the default.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17202: Bypass 303-with-token on cross-origin requests.
Tom Clegg [Wed, 9 Dec 2020 00:52:36 +0000 (19:52 -0500)]
17202: Bypass 303-with-token on cross-origin requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix ineffassign warning.
Ward Vandewege [Wed, 9 Dec 2020 22:06:06 +0000 (17:06 -0500)]
Fix ineffassign warning.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17152: Adds migration to fix collection versions' modified_at timestamps.
Lucas Di Pentima [Thu, 3 Dec 2020 20:42:22 +0000 (17:42 -0300)]
17152: Adds migration to fix collection versions' modified_at timestamps.

Also, with test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17152: Publishes the arvbox WebDAV download port.
Lucas Di Pentima [Thu, 3 Dec 2020 18:18:19 +0000 (15:18 -0300)]
17152: Publishes the arvbox WebDAV download port.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17152: Fixes old collection versions' modified_at handling and test.
Lucas Di Pentima [Tue, 1 Dec 2020 22:17:01 +0000 (19:17 -0300)]
17152: Fixes old collection versions' modified_at handling and test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago17199: Avoid returning same port twice from find_available_port().
Tom Clegg [Mon, 7 Dec 2020 18:43:23 +0000 (13:43 -0500)]
17199: Avoid returning same port twice from find_available_port().

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17161: Improve SystemRootToken docs.
Tom Clegg [Wed, 2 Dec 2020 22:14:37 +0000 (17:14 -0500)]
17161: Improve SystemRootToken docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17009: Fix s3 ListObjects endpoint with vhost-style requests.
Tom Clegg [Wed, 2 Dec 2020 22:36:35 +0000 (17:36 -0500)]
17009: Fix s3 ListObjects endpoint with vhost-style requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17009: Test virtual host-style S3 requests.
Tom Clegg [Thu, 3 Dec 2020 21:24:37 +0000 (16:24 -0500)]
17009: Test virtual host-style S3 requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix tests.
Tom Clegg [Mon, 7 Dec 2020 19:20:27 +0000 (14:20 -0500)]
Fix tests.

refs #17022

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agofix(provision): update arvados-formula's version
Javier Bértoli [Mon, 7 Dec 2020 15:46:06 +0000 (12:46 -0300)]
fix(provision): update arvados-formula's version

refs #17177

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): pin arvados-formula
Javier Bértoli [Mon, 7 Dec 2020 14:22:08 +0000 (11:22 -0300)]
fix(provision): pin arvados-formula

refs #17177

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): add port to workbench2 nginx's stanza
Javier Bértoli [Mon, 7 Dec 2020 12:20:29 +0000 (09:20 -0300)]
fix(provision): add port to workbench2 nginx's stanza

refs #17177

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): pin formulas' versions
Javier Bértoli [Fri, 4 Dec 2020 16:22:38 +0000 (13:22 -0300)]
fix(provision): pin formulas' versions

* Pin formulas versions, to prevent changes upstream breaking the installer
* Remove verbose/debug flags
* Remove unused entries in the config pillars

refs #17177

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): Document CA certificate purpose and installation
Javier Bértoli [Fri, 4 Dec 2020 12:17:34 +0000 (09:17 -0300)]
fix(provision): Document CA certificate purpose and installation

refs #17177

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(provision): Add a CA and sign certificates with it
Javier Bértoli [Thu, 3 Dec 2020 23:00:46 +0000 (20:00 -0300)]
fix(provision): Add a CA and sign certificates with it

refs #17177

As discussed [here](https://forum.arvados.org/t/debugging-arvados-deployed-with-salt/58/8)
and [here](https://gitter.im/arvados/community?at=5fc65683496ca3372e3474a3), Arvados needs
certs signed by a known CA to work correctly.

This PR adds a CA and leaves a copy of the certificate in the installer directory.

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17187: implement review feedback:
Ward Vandewege [Tue, 8 Dec 2020 00:36:54 +0000 (19:36 -0500)]
17187: implement review feedback:

* uuids are now specified at the end of the option list
* invalid uuids will cause the command to error out
* output directory is now optional

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17187: costanalyzer: print the total from the aggregate cost accounting
Ward Vandewege [Sun, 6 Dec 2020 15:54:11 +0000 (10:54 -0500)]
17187: costanalyzer: print the total from the aggregate cost accounting
       across all uuids to stdout.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17187: costanalyzer: allow specifying multiple comma-separated uuids via
Ward Vandewege [Sun, 6 Dec 2020 15:26:39 +0000 (10:26 -0500)]
17187: costanalyzer: allow specifying multiple comma-separated uuids via
       one -uuid argument

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17187: costanalyzer: add support for collection uuids.
Ward Vandewege [Sat, 5 Dec 2020 21:37:00 +0000 (16:37 -0500)]
17187: costanalyzer: add support for collection uuids.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUpgrade script/create_superuser_token.rb in the api server codebase to
Ward Vandewege [Thu, 3 Dec 2020 14:13:36 +0000 (09:13 -0500)]
Upgrade script/create_superuser_token.rb in the api server codebase to
generate v2 tokens.

refs #17022

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoClarify token details for the arv-user-activity tool.
Ward Vandewege [Thu, 3 Dec 2020 01:13:01 +0000 (20:13 -0500)]
Clarify token details for the arv-user-activity tool.

refs #17022

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUpdate error regexp in test case.
Tom Clegg [Wed, 2 Dec 2020 14:30:55 +0000 (09:30 -0500)]
Update error regexp in test case.

refs #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agofix(docs): broken link
Javier Bértoli [Wed, 2 Dec 2020 13:44:27 +0000 (10:44 -0300)]
fix(docs): broken link

reported on [github](https://github.com/arvados/arvados-formula/issues/3#issuecomment-733019750)

No issue #

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17072: Fix for added args on cwltool.docker.get_image
Peter Amstutz [Wed, 25 Nov 2020 21:45:53 +0000 (16:45 -0500)]
17072: Fix for added args on cwltool.docker.get_image

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17072: Increase default submit thread concurrency to 4
Peter Amstutz [Wed, 25 Nov 2020 21:13:58 +0000 (16:13 -0500)]
17072: Increase default submit thread concurrency to 4

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17072: Fix imports. Use task_queue from cwltool.
Peter Amstutz [Wed, 25 Nov 2020 21:10:57 +0000 (16:10 -0500)]
17072: Fix imports.  Use task_queue from cwltool.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17072: Bump cwltool version for unbound 'result' fix
Peter Amstutz [Wed, 25 Nov 2020 20:42:08 +0000 (15:42 -0500)]
17072: Bump cwltool version for unbound 'result' fix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Update install docs & fix Marc's email.
Peter Amstutz [Mon, 30 Nov 2020 19:16:56 +0000 (14:16 -0500)]
17022: Update install docs & fix Marc's email.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Add example usage of keyset_list_all to code cookbook.
Peter Amstutz [Tue, 24 Nov 2020 23:00:12 +0000 (18:00 -0500)]
17022: Add example usage of keyset_list_all to code cookbook.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Add test cases for keyset_list_all
Peter Amstutz [Tue, 24 Nov 2020 22:51:21 +0000 (17:51 -0500)]
17022: Add test cases for keyset_list_all

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Improve output. Added documentation page.
Peter Amstutz [Tue, 24 Nov 2020 22:03:32 +0000 (17:03 -0500)]
17022: Improve output.  Added documentation page.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Fix packaging
Peter Amstutz [Wed, 18 Nov 2020 20:32:21 +0000 (15:32 -0500)]
17022: Fix packaging

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Make this a fully packaged tool.
Peter Amstutz [Wed, 18 Nov 2020 20:23:36 +0000 (15:23 -0500)]
17022: Make this a fully packaged tool.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Python packaging for arv-user-activity
Peter Amstutz [Wed, 18 Nov 2020 20:02:55 +0000 (15:02 -0500)]
17022: Python packaging for arv-user-activity

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Specify time period in days on command line.
Peter Amstutz [Tue, 17 Nov 2020 22:46:42 +0000 (17:46 -0500)]
17022: Specify time period in days on command line.

Also print timestamps in local time zone.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Produce user activity report from the audit logs
Peter Amstutz [Fri, 13 Nov 2020 23:17:54 +0000 (18:17 -0500)]
17022: Produce user activity report from the audit logs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17022: Start work on user activity reporting script
Peter Amstutz [Mon, 9 Nov 2020 18:42:12 +0000 (13:42 -0500)]
17022: Start work on user activity reporting script

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoNow that our k8s helm charts retain state, update the documentation
Ward Vandewege [Mon, 30 Nov 2020 16:55:41 +0000 (11:55 -0500)]
Now that our k8s helm charts retain state, update the documentation
accordingly.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16774: text/plain response uses crlf. Tests check error codes.
Peter Amstutz [Tue, 24 Nov 2020 18:40:21 +0000 (13:40 -0500)]
16774: text/plain response uses crlf.  Tests check error codes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16774: Fix tests. Use encoder for xml error response.
Peter Amstutz [Mon, 23 Nov 2020 20:04:48 +0000 (15:04 -0500)]
16774: Fix tests.  Use encoder for xml error response.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16774: Keep-web errors include messages
Peter Amstutz [Fri, 20 Nov 2020 23:03:30 +0000 (18:03 -0500)]
16774: Keep-web errors include messages

Errors on the regular keep-web side return plain text responses.

Errors on the S3 side return XML error responses that S3 clients
expect.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix more golint warnings - update crunchrun tests.
Ward Vandewege [Mon, 30 Nov 2020 01:51:27 +0000 (20:51 -0500)]
Fix more golint warnings - update crunchrun tests.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 29 Nov 2020 02:26:28 +0000 (21:26 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Fri, 27 Nov 2020 13:28:41 +0000 (08:28 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Thu, 26 Nov 2020 22:17:10 +0000 (17:17 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix hardcoded initial user name and email
Peter van Heusden [Thu, 26 Nov 2020 18:35:16 +0000 (20:35 +0200)]
Fix hardcoded initial user name and email

Arvados-DCO-1.1-Signed-off-by: Peter Van Heusden <vh@sanbi.ac.za>

3 years agoFix more golint warnings.
Ward Vandewege [Wed, 25 Nov 2020 22:24:35 +0000 (17:24 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agofix(test): wrong variable comparison
Javier Bértoli [Wed, 25 Nov 2020 02:03:37 +0000 (23:03 -0300)]
fix(test): wrong variable comparison

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(docker): formula upgraded
Javier Bértoli [Wed, 25 Nov 2020 02:02:31 +0000 (23:02 -0300)]
fix(docker): formula upgraded

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoFix salt-install and test scripts
Javier Bértoli [Tue, 24 Nov 2020 20:17:25 +0000 (17:17 -0300)]
Fix salt-install and test scripts

* add checks to find existing resources
* run some shellcheck

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoFix salt-install's crunch-dispatch-local config and tests
Peter van Heusden [Wed, 18 Nov 2020 14:01:32 +0000 (16:01 +0200)]
Fix salt-install's crunch-dispatch-local config and tests

* added [@pvanheus fix]:https://github.com/arvados/arvados/pull/140
* added [a debug parameter to the provision script](https://gitter.im/arvados/community?at=5fb54b4bd37a1a13d6b46a05)
* added a test script to verify the cluster is able to run a CWL workflow
* document salt-install's workflow test

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17009: Fix bucket-level ops using virtual host-style requests.
Tom Clegg [Wed, 25 Nov 2020 17:14:34 +0000 (12:14 -0500)]
17009: Fix bucket-level ops using virtual host-style requests.

refs #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoDocumentation cleanups.
Ward Vandewege [Tue, 24 Nov 2020 18:17:44 +0000 (13:17 -0500)]
Documentation cleanups.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMake run-build-packages-python-and-ruby.sh say so when it decides to
Ward Vandewege [Tue, 24 Nov 2020 18:17:06 +0000 (13:17 -0500)]
Make run-build-packages-python-and-ruby.sh say so when it decides to
skip building Python packages.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17106: Add examples to S3 auth instructions.
Tom Clegg [Tue, 24 Nov 2020 15:26:36 +0000 (10:26 -0500)]
17106: Add examples to S3 auth instructions.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Fix key unescape: don't convert + to space.
Tom Clegg [Tue, 24 Nov 2020 04:45:47 +0000 (23:45 -0500)]
17106: Fix key unescape: don't convert + to space.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Allow use of URL-encoded token as S3 access/secret key.
Tom Clegg [Mon, 23 Nov 2020 19:52:57 +0000 (14:52 -0500)]
17106: Allow use of URL-encoded token as S3 access/secret key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Recommend using full tokens for S3 access.
Tom Clegg [Fri, 20 Nov 2020 15:40:30 +0000 (10:40 -0500)]
17106: Recommend using full tokens for S3 access.

Accept munged ("/" => "_") tokens in S3 requests with V2 signatures.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Update docs.
Tom Clegg [Wed, 18 Nov 2020 15:10:02 +0000 (10:10 -0500)]
17106: Update docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Comment on stored_secret behavior.
Tom Clegg [Wed, 18 Nov 2020 15:02:40 +0000 (10:02 -0500)]
17106: Comment on stored_secret behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Improve handling of bare tokens issued by remote clusters.
Tom Clegg [Tue, 17 Nov 2020 20:42:28 +0000 (15:42 -0500)]
17106: Improve handling of bare tokens issued by remote clusters.

When caching, use the remote cluster's original token UUID.

When returning the current api_client_authorization record, include
the secret supplied by the caller, even when only storing the HMAC in
the database.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Clean up test.
Tom Clegg [Tue, 17 Nov 2020 20:42:11 +0000 (15:42 -0500)]
17106: Clean up test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Skip s3cmd test if s3cmd not installed.
Tom Clegg [Tue, 17 Nov 2020 14:39:50 +0000 (09:39 -0500)]
17106: Skip s3cmd test if s3cmd not installed.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Test S3 with modified v2 token issued by LoginCluster.
Tom Clegg [Tue, 17 Nov 2020 02:25:12 +0000 (21:25 -0500)]
17106: Test S3 with modified v2 token issued by LoginCluster.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Accept v2 token with / replaced by _ as s3 access/secret key.
Tom Clegg [Tue, 17 Nov 2020 01:37:05 +0000 (20:37 -0500)]
17106: Accept v2 token with / replaced by _ as s3 access/secret key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix OS package iteration number in run-build-docker-jobs-image.sh
Ward Vandewege [Mon, 23 Nov 2020 19:48:35 +0000 (14:48 -0500)]
Fix OS package iteration number in run-build-docker-jobs-image.sh

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUpdate error checking in run-build-docker-jobs-image.sh
Ward Vandewege [Mon, 23 Nov 2020 16:51:21 +0000 (11:51 -0500)]
Update error checking in run-build-docker-jobs-image.sh

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoRemove now-unneeded code from sdk/cwl/test_with_arvbox.sh, add comment
Ward Vandewege [Mon, 23 Nov 2020 16:47:53 +0000 (11:47 -0500)]
Remove now-unneeded code from sdk/cwl/test_with_arvbox.sh, add comment
to clarify that calculate_python_sdk_cwl_package_versions now outputs
python-style package suffixes.

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17154: Add comment. Use strings.HasPrefix
Peter Amstutz [Mon, 23 Nov 2020 15:59:06 +0000 (10:59 -0500)]
17154: Add comment.  Use strings.HasPrefix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17154: Copy updates on federated users to local cluster
Peter Amstutz [Fri, 20 Nov 2020 21:01:25 +0000 (16:01 -0500)]
17154: Copy updates on federated users to local cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17154: Make sure most current user record is loaded.
Peter Amstutz [Fri, 20 Nov 2020 20:25:27 +0000 (15:25 -0500)]
17154: Make sure most current user record is loaded.

Convert 'option' items (which are symbols) to strings for value comparison

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoNow that calculate_python_sdk_cwl_package_versions returns python-style
Ward Vandewege [Mon, 23 Nov 2020 14:20:56 +0000 (09:20 -0500)]
Now that calculate_python_sdk_cwl_package_versions returns python-style
version string suffixes (.dev/rc), make sure the 2 scripts that use that
function are adapted accordingly.

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMove the handle_python_package() function from run-library.sh to
Ward Vandewege [Sun, 22 Nov 2020 13:35:06 +0000 (08:35 -0500)]
Move the handle_python_package() function from run-library.sh to
run-build-packages-python-and-ruby.sh, it is only used in the latter.
Update it to call python3 instead of just 'python'.

refs #15888

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sat, 21 Nov 2020 20:53:01 +0000 (15:53 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoRemove one more python2 remnant. This is a fix for the
Ward Vandewege [Fri, 20 Nov 2020 22:31:50 +0000 (17:31 -0500)]
Remove one more python2 remnant. This is a fix for the
python3-arvados-cwl-runner package, it now installs /usr/bin/cwltool
again.

refs #15888

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17015: Ensure that containers belong to this cluster
Peter Amstutz [Thu, 19 Nov 2020 20:36:29 +0000 (15:36 -0500)]
17015: Ensure that containers belong to this cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17009: Mention S3 considerations in keep-web install doc.
Tom Clegg [Thu, 19 Nov 2020 21:48:15 +0000 (16:48 -0500)]
17009: Mention S3 considerations in keep-web install doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17009: Support accessing S3 with virtual hosted-style URLs.
Tom Clegg [Wed, 18 Nov 2020 22:35:29 +0000 (17:35 -0500)]
17009: Support accessing S3 with virtual hosted-style URLs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix more golint warnings.
Ward Vandewege [Tue, 17 Nov 2020 01:40:47 +0000 (20:40 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: switch calculate_python_sdk_cwl_package_versions() to use the
Ward Vandewege [Tue, 17 Nov 2020 22:35:31 +0000 (17:35 -0500)]
17012: switch calculate_python_sdk_cwl_package_versions() to use the
       python version calculation code. This effectively is bash calling
       python calling bash, which I don't like, but the code is simpler.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: update calculation of python version number in run-library.sh
Ward Vandewege [Tue, 17 Nov 2020 15:26:58 +0000 (10:26 -0500)]
17012: update calculation of python version number in run-library.sh
       (used by a few docker image scripts).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: remove old gittaggers.py files.
Ward Vandewege [Tue, 17 Nov 2020 14:52:53 +0000 (09:52 -0500)]
17012: remove old gittaggers.py files.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: when calculating the version of our Python packages, take the
Ward Vandewege [Mon, 16 Nov 2020 20:08:26 +0000 (15:08 -0500)]
17012: when calculating the version of our Python packages, take the
       build directory into account.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: switch to os.UserHomeDir(), and send all output to stderr.
Ward Vandewege [Wed, 18 Nov 2020 21:20:40 +0000 (16:20 -0500)]
16950: switch to os.UserHomeDir(), and send all output to stderr.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: make specifying of the output directory mandatory (remove
Ward Vandewege [Wed, 18 Nov 2020 15:09:51 +0000 (10:09 -0500)]
16950: make specifying of the output directory mandatory (remove
       default).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: refactor caching; use ~/.cache/arvados/costanalyzer for that
Ward Vandewege [Sat, 14 Nov 2020 18:22:50 +0000 (13:22 -0500)]
16950: refactor caching; use ~/.cache/arvados/costanalyzer for that
       purpose.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: more changes after review.
Ward Vandewege [Fri, 13 Nov 2020 21:22:57 +0000 (16:22 -0500)]
16950: more changes after review.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>