arvados.git
3 years agoMerge branch '16774-keep-web-errors' refs #16774
Peter Amstutz [Mon, 30 Nov 2020 16:03:53 +0000 (11:03 -0500)]
Merge branch '16774-keep-web-errors' refs #16774

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix more golint warnings - update crunchrun tests.
Ward Vandewege [Mon, 30 Nov 2020 01:51:27 +0000 (20:51 -0500)]
Fix more golint warnings - update crunchrun tests.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 29 Nov 2020 02:26:28 +0000 (21:26 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Fri, 27 Nov 2020 13:28:41 +0000 (08:28 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Thu, 26 Nov 2020 22:17:10 +0000 (17:17 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch 'pvanheus-fix_cluster_test' from github.com/pvanheus:fix_cluster_test
Javier Bértoli [Thu, 26 Nov 2020 18:49:50 +0000 (15:49 -0300)]
Merge branch 'pvanheus-fix_cluster_test' from github.com/pvanheus:fix_cluster_test

No issue #

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoFix hardcoded initial user name and email
Peter van Heusden [Thu, 26 Nov 2020 18:35:16 +0000 (20:35 +0200)]
Fix hardcoded initial user name and email

Arvados-DCO-1.1-Signed-off-by: Peter Van Heusden <vh@sanbi.ac.za>

3 years agoFix more golint warnings.
Ward Vandewege [Wed, 25 Nov 2020 22:24:35 +0000 (17:24 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17150-system-root-token'
Javier Bértoli [Wed, 25 Nov 2020 18:06:50 +0000 (15:06 -0300)]
Merge branch '17150-system-root-token'

closes #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17009: Fix bucket-level ops using virtual host-style requests.
Tom Clegg [Wed, 25 Nov 2020 17:14:34 +0000 (12:14 -0500)]
17009: Fix bucket-level ops using virtual host-style requests.

refs #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agofix(test): wrong variable comparison
Javier Bértoli [Wed, 25 Nov 2020 02:03:37 +0000 (23:03 -0300)]
fix(test): wrong variable comparison

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(docker): formula upgraded
Javier Bértoli [Wed, 25 Nov 2020 02:02:31 +0000 (23:02 -0300)]
fix(docker): formula upgraded

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoFix salt-install and test scripts
Javier Bértoli [Tue, 24 Nov 2020 20:17:25 +0000 (17:17 -0300)]
Fix salt-install and test scripts

* add checks to find existing resources
* run some shellcheck

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago16774: text/plain response uses crlf. Tests check error codes.
Peter Amstutz [Tue, 24 Nov 2020 18:40:21 +0000 (13:40 -0500)]
16774: text/plain response uses crlf.  Tests check error codes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoDocumentation cleanups.
Ward Vandewege [Tue, 24 Nov 2020 18:17:44 +0000 (13:17 -0500)]
Documentation cleanups.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMake run-build-packages-python-and-ruby.sh say so when it decides to
Ward Vandewege [Tue, 24 Nov 2020 18:17:06 +0000 (13:17 -0500)]
Make run-build-packages-python-and-ruby.sh say so when it decides to
skip building Python packages.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17106-s3-fed-token'
Tom Clegg [Tue, 24 Nov 2020 16:55:48 +0000 (11:55 -0500)]
Merge branch '17106-s3-fed-token'

fixes #17106

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Add examples to S3 auth instructions.
Tom Clegg [Tue, 24 Nov 2020 15:26:36 +0000 (10:26 -0500)]
17106: Add examples to S3 auth instructions.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix salt-install's crunch-dispatch-local config and tests
Peter van Heusden [Wed, 18 Nov 2020 14:01:32 +0000 (16:01 +0200)]
Fix salt-install's crunch-dispatch-local config and tests

* added [@pvanheus fix]:https://github.com/arvados/arvados/pull/140
* added [a debug parameter to the provision script](https://gitter.im/arvados/community?at=5fb54b4bd37a1a13d6b46a05)
* added a test script to verify the cluster is able to run a CWL workflow
* document salt-install's workflow test

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17106: Fix key unescape: don't convert + to space.
Tom Clegg [Tue, 24 Nov 2020 04:45:47 +0000 (23:45 -0500)]
17106: Fix key unescape: don't convert + to space.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16774: Fix tests. Use encoder for xml error response.
Peter Amstutz [Mon, 23 Nov 2020 20:04:48 +0000 (15:04 -0500)]
16774: Fix tests.  Use encoder for xml error response.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17106: Allow use of URL-encoded token as S3 access/secret key.
Tom Clegg [Mon, 23 Nov 2020 19:52:57 +0000 (14:52 -0500)]
17106: Allow use of URL-encoded token as S3 access/secret key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix OS package iteration number in run-build-docker-jobs-image.sh
Ward Vandewege [Mon, 23 Nov 2020 19:48:35 +0000 (14:48 -0500)]
Fix OS package iteration number in run-build-docker-jobs-image.sh

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUpdate error checking in run-build-docker-jobs-image.sh
Ward Vandewege [Mon, 23 Nov 2020 16:51:21 +0000 (11:51 -0500)]
Update error checking in run-build-docker-jobs-image.sh

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoRemove now-unneeded code from sdk/cwl/test_with_arvbox.sh, add comment
Ward Vandewege [Mon, 23 Nov 2020 16:47:53 +0000 (11:47 -0500)]
Remove now-unneeded code from sdk/cwl/test_with_arvbox.sh, add comment
to clarify that calculate_python_sdk_cwl_package_versions now outputs
python-style package suffixes.

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17154-wb-profile' refs #17154
Peter Amstutz [Mon, 23 Nov 2020 16:23:51 +0000 (11:23 -0500)]
Merge branch '17154-wb-profile' refs #17154

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17154: Add comment. Use strings.HasPrefix
Peter Amstutz [Mon, 23 Nov 2020 15:59:06 +0000 (10:59 -0500)]
17154: Add comment.  Use strings.HasPrefix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoNow that calculate_python_sdk_cwl_package_versions returns python-style
Ward Vandewege [Mon, 23 Nov 2020 14:20:56 +0000 (09:20 -0500)]
Now that calculate_python_sdk_cwl_package_versions returns python-style
version string suffixes (.dev/rc), make sure the 2 scripts that use that
function are adapted accordingly.

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMove the handle_python_package() function from run-library.sh to
Ward Vandewege [Sun, 22 Nov 2020 13:35:06 +0000 (08:35 -0500)]
Move the handle_python_package() function from run-library.sh to
run-build-packages-python-and-ruby.sh, it is only used in the latter.
Update it to call python3 instead of just 'python'.

refs #15888

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sat, 21 Nov 2020 20:53:01 +0000 (15:53 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16774: Keep-web errors include messages
Peter Amstutz [Fri, 20 Nov 2020 23:03:30 +0000 (18:03 -0500)]
16774: Keep-web errors include messages

Errors on the regular keep-web side return plain text responses.

Errors on the S3 side return XML error responses that S3 clients
expect.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoRemove one more python2 remnant. This is a fix for the
Ward Vandewege [Fri, 20 Nov 2020 22:31:50 +0000 (17:31 -0500)]
Remove one more python2 remnant. This is a fix for the
python3-arvados-cwl-runner package, it now installs /usr/bin/cwltool
again.

refs #15888

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17154: Copy updates on federated users to local cluster
Peter Amstutz [Fri, 20 Nov 2020 21:01:25 +0000 (16:01 -0500)]
17154: Copy updates on federated users to local cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17154: Make sure most current user record is loaded.
Peter Amstutz [Fri, 20 Nov 2020 20:25:27 +0000 (15:25 -0500)]
17154: Make sure most current user record is loaded.

Convert 'option' items (which are symbols) to strings for value comparison

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '17015-cds-correct-containers' refs #17015
Peter Amstutz [Fri, 20 Nov 2020 17:59:12 +0000 (12:59 -0500)]
Merge branch '17015-cds-correct-containers' refs #17015

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17106: Recommend using full tokens for S3 access.
Tom Clegg [Fri, 20 Nov 2020 15:40:30 +0000 (10:40 -0500)]
17106: Recommend using full tokens for S3 access.

Accept munged ("/" => "_") tokens in S3 requests with V2 signatures.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '17009-s3-bucket-vhost'
Tom Clegg [Thu, 19 Nov 2020 21:50:26 +0000 (16:50 -0500)]
Merge branch '17009-s3-bucket-vhost'

closes #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17009: Mention S3 considerations in keep-web install doc.
Tom Clegg [Thu, 19 Nov 2020 21:48:15 +0000 (16:48 -0500)]
17009: Mention S3 considerations in keep-web install doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17015: Ensure that containers belong to this cluster
Peter Amstutz [Thu, 19 Nov 2020 20:36:29 +0000 (15:36 -0500)]
17015: Ensure that containers belong to this cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17009: Support accessing S3 with virtual hosted-style URLs.
Tom Clegg [Wed, 18 Nov 2020 22:35:29 +0000 (17:35 -0500)]
17009: Support accessing S3 with virtual hosted-style URLs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16950-add-costanalyzer' into master
Ward Vandewege [Wed, 18 Nov 2020 21:24:19 +0000 (16:24 -0500)]
Merge branch '16950-add-costanalyzer' into master

closes #16950

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch 'master' into 16950-add-costanalyzer
Ward Vandewege [Wed, 18 Nov 2020 21:23:51 +0000 (16:23 -0500)]
Merge branch 'master' into 16950-add-costanalyzer

refs #16950

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: switch to os.UserHomeDir(), and send all output to stderr.
Ward Vandewege [Wed, 18 Nov 2020 21:20:40 +0000 (16:20 -0500)]
16950: switch to os.UserHomeDir(), and send all output to stderr.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: make specifying of the output directory mandatory (remove
Ward Vandewege [Wed, 18 Nov 2020 15:09:51 +0000 (10:09 -0500)]
16950: make specifying of the output directory mandatory (remove
       default).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17106: Update docs.
Tom Clegg [Wed, 18 Nov 2020 15:10:02 +0000 (10:10 -0500)]
17106: Update docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Comment on stored_secret behavior.
Tom Clegg [Wed, 18 Nov 2020 15:02:40 +0000 (10:02 -0500)]
17106: Comment on stored_secret behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Improve handling of bare tokens issued by remote clusters.
Tom Clegg [Tue, 17 Nov 2020 20:42:28 +0000 (15:42 -0500)]
17106: Improve handling of bare tokens issued by remote clusters.

When caching, use the remote cluster's original token UUID.

When returning the current api_client_authorization record, include
the secret supplied by the caller, even when only storing the HMAC in
the database.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Clean up test.
Tom Clegg [Tue, 17 Nov 2020 20:42:11 +0000 (15:42 -0500)]
17106: Clean up test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Skip s3cmd test if s3cmd not installed.
Tom Clegg [Tue, 17 Nov 2020 14:39:50 +0000 (09:39 -0500)]
17106: Skip s3cmd test if s3cmd not installed.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Test S3 with modified v2 token issued by LoginCluster.
Tom Clegg [Tue, 17 Nov 2020 02:25:12 +0000 (21:25 -0500)]
17106: Test S3 with modified v2 token issued by LoginCluster.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17106: Accept v2 token with / replaced by _ as s3 access/secret key.
Tom Clegg [Tue, 17 Nov 2020 01:37:05 +0000 (20:37 -0500)]
17106: Accept v2 token with / replaced by _ as s3 access/secret key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '17012-fix-python-version-calculation' into master
Ward Vandewege [Tue, 17 Nov 2020 22:44:21 +0000 (17:44 -0500)]
Merge branch '17012-fix-python-version-calculation' into master

closes #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch 'master' into 17012-fix-python-version-calculation
Ward Vandewege [Tue, 17 Nov 2020 22:43:46 +0000 (17:43 -0500)]
Merge branch 'master' into 17012-fix-python-version-calculation

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: switch calculate_python_sdk_cwl_package_versions() to use the
Ward Vandewege [Tue, 17 Nov 2020 22:35:31 +0000 (17:35 -0500)]
17012: switch calculate_python_sdk_cwl_package_versions() to use the
       python version calculation code. This effectively is bash calling
       python calling bash, which I don't like, but the code is simpler.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: update calculation of python version number in run-library.sh
Ward Vandewege [Tue, 17 Nov 2020 15:26:58 +0000 (10:26 -0500)]
17012: update calculation of python version number in run-library.sh
       (used by a few docker image scripts).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: remove old gittaggers.py files.
Ward Vandewege [Tue, 17 Nov 2020 14:52:53 +0000 (09:52 -0500)]
17012: remove old gittaggers.py files.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17012: when calculating the version of our Python packages, take the
Ward Vandewege [Mon, 16 Nov 2020 20:08:26 +0000 (15:08 -0500)]
17012: when calculating the version of our Python packages, take the
       build directory into account.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Tue, 17 Nov 2020 01:40:47 +0000 (20:40 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 15 Nov 2020 19:18:40 +0000 (14:18 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: refactor caching; use ~/.cache/arvados/costanalyzer for that
Ward Vandewege [Sat, 14 Nov 2020 18:22:50 +0000 (13:22 -0500)]
16950: refactor caching; use ~/.cache/arvados/costanalyzer for that
       purpose.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: more changes after review.
Ward Vandewege [Fri, 13 Nov 2020 21:22:57 +0000 (16:22 -0500)]
16950: more changes after review.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17111-deselect-unsigned-manifest'
Tom Clegg [Fri, 13 Nov 2020 21:57:25 +0000 (16:57 -0500)]
Merge branch '17111-deselect-unsigned-manifest'

fixes #17111

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago17111: Deselect unsigned_manifest_text by default in groups#contents.
Tom Clegg [Fri, 13 Nov 2020 20:44:43 +0000 (15:44 -0500)]
17111: Deselect unsigned_manifest_text by default in groups#contents.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix more golint warnings.
Ward Vandewege [Fri, 13 Nov 2020 01:02:56 +0000 (20:02 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: changes after review.
Ward Vandewege [Thu, 12 Nov 2020 03:07:48 +0000 (22:07 -0500)]
16950: changes after review.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 8 Nov 2020 21:41:21 +0000 (16:41 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17086-add-focal-fossa' into master
Ward Vandewege [Sat, 7 Nov 2020 13:45:42 +0000 (08:45 -0500)]
Merge branch '17086-add-focal-fossa' into master

closes #17086

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17086: Merge branch 'master' into 17086-add-focal-fossa
Ward Vandewege [Sat, 7 Nov 2020 13:45:09 +0000 (08:45 -0500)]
17086: Merge branch 'master' into 17086-add-focal-fossa

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix require path in the 'script/rails' script for api server.
Ward Vandewege [Fri, 6 Nov 2020 13:34:38 +0000 (08:34 -0500)]
Fix require path in the 'script/rails' script for api server.

closes #17094

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17086: build packages for Ubuntu 20.04, Focal Fossa.
Ward Vandewege [Thu, 5 Nov 2020 19:43:37 +0000 (14:43 -0500)]
17086: build packages for Ubuntu 20.04, Focal Fossa.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17090-exclusive-lock' refs #17090
Peter Amstutz [Thu, 5 Nov 2020 20:15:13 +0000 (15:15 -0500)]
Merge branch '17090-exclusive-lock' refs #17090

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17090: Migration to refresh_permissions
Peter Amstutz [Thu, 5 Nov 2020 19:55:35 +0000 (14:55 -0500)]
17090: Migration to refresh_permissions

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17090: Use EXCLUSIVE lock for permission updates
Peter Amstutz [Thu, 5 Nov 2020 16:09:30 +0000 (11:09 -0500)]
17090: Use EXCLUSIVE lock for permission updates

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Thu, 5 Nov 2020 00:52:42 +0000 (19:52 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: add tests for the costanalyzer.
Ward Vandewege [Tue, 3 Nov 2020 21:34:16 +0000 (16:34 -0500)]
16950: add tests for the costanalyzer.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17040-slow-query' refs #17040
Peter Amstutz [Tue, 3 Nov 2020 18:29:21 +0000 (13:29 -0500)]
Merge branch '17040-slow-query' refs #17040

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17040: Update structure.sql
Peter Amstutz [Tue, 3 Nov 2020 18:01:57 +0000 (13:01 -0500)]
17040: Update structure.sql

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17040: RefreshTrashedGroups migration, clear_permissions calls forget_cached_group_perms
Peter Amstutz [Tue, 3 Nov 2020 17:12:59 +0000 (12:12 -0500)]
17040: RefreshTrashedGroups migration, clear_permissions calls forget_cached_group_perms

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17040: Cache results of User.group_permissions
Peter Amstutz [Mon, 2 Nov 2020 21:20:02 +0000 (16:20 -0500)]
17040: Cache results of User.group_permissions

When requesting a list of groups (either directly, or with the
"shared" endpoint) it calls writable_by for each group.  This
indirectly calls User.group_permissions, which makes a database query.
When it does this for every group (with a database query each time),
which gets very expensive.

To address this, this commit caches the result of group_permissions on
the user object.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17040: Get user_uuids and embed them as a constant in the main query
Peter Amstutz [Mon, 2 Nov 2020 19:25:28 +0000 (14:25 -0500)]
17040: Get user_uuids and embed them as a constant in the main query

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 1 Nov 2020 22:27:04 +0000 (17:27 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: add the costanalyzer to arvados-client.
Ward Vandewege [Sat, 3 Oct 2020 17:58:50 +0000 (13:58 -0400)]
16950: add the costanalyzer to arvados-client.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16950: a few cleanups for our container request fixtures: attach a real
Ward Vandewege [Fri, 30 Oct 2020 20:29:40 +0000 (16:29 -0400)]
16950: a few cleanups for our container request fixtures: attach a real
       log collection, and define CompletedContainerRequestUUID in
       arvadostest (go).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16750: Avoids using params on requests to make it compatible with federation.
Lucas Di Pentima [Thu, 1 Oct 2020 13:52:23 +0000 (10:52 -0300)]
16750: Avoids using params on requests to make it compatible with federation.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoMerge branch '17047-centos-remove-rh-python-dependencies' into master
Javier Bértoli [Wed, 28 Oct 2020 19:56:29 +0000 (16:56 -0300)]
Merge branch '17047-centos-remove-rh-python-dependencies' into master

closes #17047
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17047: bugfixes, and add an entry to the upgrade notes about the
Ward Vandewege [Wed, 28 Oct 2020 19:29:58 +0000 (15:29 -0400)]
17047: bugfixes, and add an entry to the upgrade notes about the
       packaging change.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agofeat(centos): use centos-provided python3
Javier Bértoli [Tue, 27 Oct 2020 13:13:53 +0000 (10:13 -0300)]
feat(centos): use centos-provided python3

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agobuild: Remove remaining references to the SSO server packages.
Ward Vandewege [Tue, 27 Oct 2020 15:24:23 +0000 (11:24 -0400)]
build: Remove remaining references to the SSO server packages.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agodocumentation: Mention distribution packages on the "Install the Slurm
Ward Vandewege [Tue, 27 Oct 2020 15:23:05 +0000 (11:23 -0400)]
documentation: Mention distribution packages on the "Install the Slurm
dispatcher" page.

closes #16883

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17044-remove-debian-stretch' into master
Ward Vandewege [Tue, 27 Oct 2020 15:14:15 +0000 (11:14 -0400)]
Merge branch '17044-remove-debian-stretch' into master

closes #17044

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17040: Refactor trash check
Peter Amstutz [Tue, 27 Oct 2020 02:55:46 +0000 (22:55 -0400)]
17040: Refactor trash check

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17044: implement review comments (part 2): update cwl tests to use
Ward Vandewege [Mon, 26 Oct 2020 23:35:12 +0000 (19:35 -0400)]
17044: implement review comments (part 2): update cwl tests to use
       Debian buster.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17044: implement review comments (part 1).
Ward Vandewege [Mon, 26 Oct 2020 23:33:10 +0000 (19:33 -0400)]
17044: implement review comments (part 1).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17044: remove Debian "stretch" from the build infrastructure. Update
Ward Vandewege [Mon, 26 Oct 2020 19:14:37 +0000 (15:14 -0400)]
17044: remove Debian "stretch" from the build infrastructure. Update
       documentation to mark stretch as EOL post the last release in the
       2.1.X series.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '16996-add-dispatch-local-service-file' into master
Javier Bértoli [Mon, 26 Oct 2020 18:32:23 +0000 (15:32 -0300)]
Merge branch '16996-add-dispatch-local-service-file' into master

closes #16996
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofeat(crunch-dispatch-local): add crunch-run dependency
Javier Bértoli [Mon, 26 Oct 2020 15:28:29 +0000 (12:28 -0300)]
feat(crunch-dispatch-local): add crunch-run dependency

refs #16995
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agofix(crunch-dispatch-local): add missing service file
Javier Bértoli [Mon, 19 Oct 2020 13:06:18 +0000 (10:06 -0300)]
fix(crunch-dispatch-local): add missing service file

refs #16996
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 25 Oct 2020 15:42:59 +0000 (11:42 -0400)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix more golint warnings.
Ward Vandewege [Sun, 25 Oct 2020 00:28:04 +0000 (20:28 -0400)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '16930-tagging' into master
Ward Vandewege [Fri, 23 Oct 2020 20:40:35 +0000 (16:40 -0400)]
Merge branch '16930-tagging' into master

closes #16930

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>