arvados.git
4 years ago17161: Improve SystemRootToken docs.
Tom Clegg [Wed, 2 Dec 2020 22:14:37 +0000 (17:14 -0500)]
17161: Improve SystemRootToken docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUpdate error regexp in test case.
Tom Clegg [Wed, 2 Dec 2020 14:30:55 +0000 (09:30 -0500)]
Update error regexp in test case.

refs #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agofix(docs): broken link
Javier Bértoli [Wed, 2 Dec 2020 13:44:27 +0000 (10:44 -0300)]
fix(docs): broken link

reported on [github](https://github.com/arvados/arvados-formula/issues/3#issuecomment-733019750)

No issue #

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years ago17152: Fixes old collection versions' modified_at handling and test.
Lucas Di Pentima [Tue, 1 Dec 2020 22:17:01 +0000 (19:17 -0300)]
17152: Fixes old collection versions' modified_at handling and test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

4 years agoMerge branch '17072-update-cwltool' refs #17072
Peter Amstutz [Tue, 1 Dec 2020 21:16:05 +0000 (16:16 -0500)]
Merge branch '17072-update-cwltool' refs #17072

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoMerge branch '17022-user-activity-report' refs #17022
Peter Amstutz [Mon, 30 Nov 2020 22:03:33 +0000 (17:03 -0500)]
Merge branch '17022-user-activity-report' refs #17022

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Update install docs & fix Marc's email.
Peter Amstutz [Mon, 30 Nov 2020 19:16:56 +0000 (14:16 -0500)]
17022: Update install docs & fix Marc's email.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoNow that our k8s helm charts retain state, update the documentation
Ward Vandewege [Mon, 30 Nov 2020 16:55:41 +0000 (11:55 -0500)]
Now that our k8s helm charts retain state, update the documentation
accordingly.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '16774-keep-web-errors' refs #16774
Peter Amstutz [Mon, 30 Nov 2020 16:03:53 +0000 (11:03 -0500)]
Merge branch '16774-keep-web-errors' refs #16774

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoFix more golint warnings - update crunchrun tests.
Ward Vandewege [Mon, 30 Nov 2020 01:51:27 +0000 (20:51 -0500)]
Fix more golint warnings - update crunchrun tests.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Sun, 29 Nov 2020 02:26:28 +0000 (21:26 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Fri, 27 Nov 2020 13:28:41 +0000 (08:28 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Thu, 26 Nov 2020 22:17:10 +0000 (17:17 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch 'pvanheus-fix_cluster_test' from github.com/pvanheus:fix_cluster_test
Javier Bértoli [Thu, 26 Nov 2020 18:49:50 +0000 (15:49 -0300)]
Merge branch 'pvanheus-fix_cluster_test' from github.com/pvanheus:fix_cluster_test

No issue #

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years agoFix hardcoded initial user name and email
Peter van Heusden [Thu, 26 Nov 2020 18:35:16 +0000 (20:35 +0200)]
Fix hardcoded initial user name and email

Arvados-DCO-1.1-Signed-off-by: Peter Van Heusden <vh@sanbi.ac.za>

4 years agoFix more golint warnings.
Ward Vandewege [Wed, 25 Nov 2020 22:24:35 +0000 (17:24 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17072: Fix for added args on cwltool.docker.get_image
Peter Amstutz [Wed, 25 Nov 2020 21:45:53 +0000 (16:45 -0500)]
17072: Fix for added args on cwltool.docker.get_image

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17072: Increase default submit thread concurrency to 4
Peter Amstutz [Wed, 25 Nov 2020 21:13:58 +0000 (16:13 -0500)]
17072: Increase default submit thread concurrency to 4

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17072: Fix imports. Use task_queue from cwltool.
Peter Amstutz [Wed, 25 Nov 2020 21:10:57 +0000 (16:10 -0500)]
17072: Fix imports.  Use task_queue from cwltool.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17072: Bump cwltool version for unbound 'result' fix
Peter Amstutz [Wed, 25 Nov 2020 20:42:08 +0000 (15:42 -0500)]
17072: Bump cwltool version for unbound 'result' fix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoMerge branch '17150-system-root-token'
Javier Bértoli [Wed, 25 Nov 2020 18:06:50 +0000 (15:06 -0300)]
Merge branch '17150-system-root-token'

closes #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years ago17009: Fix bucket-level ops using virtual host-style requests.
Tom Clegg [Wed, 25 Nov 2020 17:14:34 +0000 (12:14 -0500)]
17009: Fix bucket-level ops using virtual host-style requests.

refs #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agofix(test): wrong variable comparison
Javier Bértoli [Wed, 25 Nov 2020 02:03:37 +0000 (23:03 -0300)]
fix(test): wrong variable comparison

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years agofix(docker): formula upgraded
Javier Bértoli [Wed, 25 Nov 2020 02:02:31 +0000 (23:02 -0300)]
fix(docker): formula upgraded

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years ago17022: Add example usage of keyset_list_all to code cookbook.
Peter Amstutz [Tue, 24 Nov 2020 23:00:12 +0000 (18:00 -0500)]
17022: Add example usage of keyset_list_all to code cookbook.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Add test cases for keyset_list_all
Peter Amstutz [Tue, 24 Nov 2020 22:51:21 +0000 (17:51 -0500)]
17022: Add test cases for keyset_list_all

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Improve output. Added documentation page.
Peter Amstutz [Tue, 24 Nov 2020 22:03:32 +0000 (17:03 -0500)]
17022: Improve output.  Added documentation page.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoFix salt-install and test scripts
Javier Bértoli [Tue, 24 Nov 2020 20:17:25 +0000 (17:17 -0300)]
Fix salt-install and test scripts

* add checks to find existing resources
* run some shellcheck

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years ago16774: text/plain response uses crlf. Tests check error codes.
Peter Amstutz [Tue, 24 Nov 2020 18:40:21 +0000 (13:40 -0500)]
16774: text/plain response uses crlf.  Tests check error codes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoDocumentation cleanups.
Ward Vandewege [Tue, 24 Nov 2020 18:17:44 +0000 (13:17 -0500)]
Documentation cleanups.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMake run-build-packages-python-and-ruby.sh say so when it decides to
Ward Vandewege [Tue, 24 Nov 2020 18:17:06 +0000 (13:17 -0500)]
Make run-build-packages-python-and-ruby.sh say so when it decides to
skip building Python packages.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '17106-s3-fed-token'
Tom Clegg [Tue, 24 Nov 2020 16:55:48 +0000 (11:55 -0500)]
Merge branch '17106-s3-fed-token'

fixes #17106

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Add examples to S3 auth instructions.
Tom Clegg [Tue, 24 Nov 2020 15:26:36 +0000 (10:26 -0500)]
17106: Add examples to S3 auth instructions.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix salt-install's crunch-dispatch-local config and tests
Peter van Heusden [Wed, 18 Nov 2020 14:01:32 +0000 (16:01 +0200)]
Fix salt-install's crunch-dispatch-local config and tests

* added [@pvanheus fix]:https://github.com/arvados/arvados/pull/140
* added [a debug parameter to the provision script](https://gitter.im/arvados/community?at=5fb54b4bd37a1a13d6b46a05)
* added a test script to verify the cluster is able to run a CWL workflow
* document salt-install's workflow test

refs #17146, #17147, #17150

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

4 years ago17106: Fix key unescape: don't convert + to space.
Tom Clegg [Tue, 24 Nov 2020 04:45:47 +0000 (23:45 -0500)]
17106: Fix key unescape: don't convert + to space.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago16774: Fix tests. Use encoder for xml error response.
Peter Amstutz [Mon, 23 Nov 2020 20:04:48 +0000 (15:04 -0500)]
16774: Fix tests.  Use encoder for xml error response.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17106: Allow use of URL-encoded token as S3 access/secret key.
Tom Clegg [Mon, 23 Nov 2020 19:52:57 +0000 (14:52 -0500)]
17106: Allow use of URL-encoded token as S3 access/secret key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix OS package iteration number in run-build-docker-jobs-image.sh
Ward Vandewege [Mon, 23 Nov 2020 19:48:35 +0000 (14:48 -0500)]
Fix OS package iteration number in run-build-docker-jobs-image.sh

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoUpdate error checking in run-build-docker-jobs-image.sh
Ward Vandewege [Mon, 23 Nov 2020 16:51:21 +0000 (11:51 -0500)]
Update error checking in run-build-docker-jobs-image.sh

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoRemove now-unneeded code from sdk/cwl/test_with_arvbox.sh, add comment
Ward Vandewege [Mon, 23 Nov 2020 16:47:53 +0000 (11:47 -0500)]
Remove now-unneeded code from sdk/cwl/test_with_arvbox.sh, add comment
to clarify that calculate_python_sdk_cwl_package_versions now outputs
python-style package suffixes.

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '17154-wb-profile' refs #17154
Peter Amstutz [Mon, 23 Nov 2020 16:23:51 +0000 (11:23 -0500)]
Merge branch '17154-wb-profile' refs #17154

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17154: Add comment. Use strings.HasPrefix
Peter Amstutz [Mon, 23 Nov 2020 15:59:06 +0000 (10:59 -0500)]
17154: Add comment.  Use strings.HasPrefix

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoNow that calculate_python_sdk_cwl_package_versions returns python-style
Ward Vandewege [Mon, 23 Nov 2020 14:20:56 +0000 (09:20 -0500)]
Now that calculate_python_sdk_cwl_package_versions returns python-style
version string suffixes (.dev/rc), make sure the 2 scripts that use that
function are adapted accordingly.

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMove the handle_python_package() function from run-library.sh to
Ward Vandewege [Sun, 22 Nov 2020 13:35:06 +0000 (08:35 -0500)]
Move the handle_python_package() function from run-library.sh to
run-build-packages-python-and-ruby.sh, it is only used in the latter.
Update it to call python3 instead of just 'python'.

refs #15888

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Sat, 21 Nov 2020 20:53:01 +0000 (15:53 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16774: Keep-web errors include messages
Peter Amstutz [Fri, 20 Nov 2020 23:03:30 +0000 (18:03 -0500)]
16774: Keep-web errors include messages

Errors on the regular keep-web side return plain text responses.

Errors on the S3 side return XML error responses that S3 clients
expect.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoRemove one more python2 remnant. This is a fix for the
Ward Vandewege [Fri, 20 Nov 2020 22:31:50 +0000 (17:31 -0500)]
Remove one more python2 remnant. This is a fix for the
python3-arvados-cwl-runner package, it now installs /usr/bin/cwltool
again.

refs #15888

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17154: Copy updates on federated users to local cluster
Peter Amstutz [Fri, 20 Nov 2020 21:01:25 +0000 (16:01 -0500)]
17154: Copy updates on federated users to local cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17154: Make sure most current user record is loaded.
Peter Amstutz [Fri, 20 Nov 2020 20:25:27 +0000 (15:25 -0500)]
17154: Make sure most current user record is loaded.

Convert 'option' items (which are symbols) to strings for value comparison

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoMerge branch '17015-cds-correct-containers' refs #17015
Peter Amstutz [Fri, 20 Nov 2020 17:59:12 +0000 (12:59 -0500)]
Merge branch '17015-cds-correct-containers' refs #17015

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17106: Recommend using full tokens for S3 access.
Tom Clegg [Fri, 20 Nov 2020 15:40:30 +0000 (10:40 -0500)]
17106: Recommend using full tokens for S3 access.

Accept munged ("/" => "_") tokens in S3 requests with V2 signatures.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoMerge branch '17009-s3-bucket-vhost'
Tom Clegg [Thu, 19 Nov 2020 21:50:26 +0000 (16:50 -0500)]
Merge branch '17009-s3-bucket-vhost'

closes #17009

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17009: Mention S3 considerations in keep-web install doc.
Tom Clegg [Thu, 19 Nov 2020 21:48:15 +0000 (16:48 -0500)]
17009: Mention S3 considerations in keep-web install doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17015: Ensure that containers belong to this cluster
Peter Amstutz [Thu, 19 Nov 2020 20:36:29 +0000 (15:36 -0500)]
17015: Ensure that containers belong to this cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17009: Support accessing S3 with virtual hosted-style URLs.
Tom Clegg [Wed, 18 Nov 2020 22:35:29 +0000 (17:35 -0500)]
17009: Support accessing S3 with virtual hosted-style URLs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoMerge branch '16950-add-costanalyzer' into master
Ward Vandewege [Wed, 18 Nov 2020 21:24:19 +0000 (16:24 -0500)]
Merge branch '16950-add-costanalyzer' into master

closes #16950

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch 'master' into 16950-add-costanalyzer
Ward Vandewege [Wed, 18 Nov 2020 21:23:51 +0000 (16:23 -0500)]
Merge branch 'master' into 16950-add-costanalyzer

refs #16950

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16950: switch to os.UserHomeDir(), and send all output to stderr.
Ward Vandewege [Wed, 18 Nov 2020 21:20:40 +0000 (16:20 -0500)]
16950: switch to os.UserHomeDir(), and send all output to stderr.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16950: make specifying of the output directory mandatory (remove
Ward Vandewege [Wed, 18 Nov 2020 15:09:51 +0000 (10:09 -0500)]
16950: make specifying of the output directory mandatory (remove
       default).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17022: Fix packaging
Peter Amstutz [Wed, 18 Nov 2020 20:32:21 +0000 (15:32 -0500)]
17022: Fix packaging

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Make this a fully packaged tool.
Peter Amstutz [Wed, 18 Nov 2020 20:23:36 +0000 (15:23 -0500)]
17022: Make this a fully packaged tool.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Python packaging for arv-user-activity
Peter Amstutz [Wed, 18 Nov 2020 20:02:55 +0000 (15:02 -0500)]
17022: Python packaging for arv-user-activity

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Specify time period in days on command line.
Peter Amstutz [Tue, 17 Nov 2020 22:46:42 +0000 (17:46 -0500)]
17022: Specify time period in days on command line.

Also print timestamps in local time zone.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Produce user activity report from the audit logs
Peter Amstutz [Fri, 13 Nov 2020 23:17:54 +0000 (18:17 -0500)]
17022: Produce user activity report from the audit logs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17022: Start work on user activity reporting script
Peter Amstutz [Mon, 9 Nov 2020 18:42:12 +0000 (13:42 -0500)]
17022: Start work on user activity reporting script

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17106: Update docs.
Tom Clegg [Wed, 18 Nov 2020 15:10:02 +0000 (10:10 -0500)]
17106: Update docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Comment on stored_secret behavior.
Tom Clegg [Wed, 18 Nov 2020 15:02:40 +0000 (10:02 -0500)]
17106: Comment on stored_secret behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Improve handling of bare tokens issued by remote clusters.
Tom Clegg [Tue, 17 Nov 2020 20:42:28 +0000 (15:42 -0500)]
17106: Improve handling of bare tokens issued by remote clusters.

When caching, use the remote cluster's original token UUID.

When returning the current api_client_authorization record, include
the secret supplied by the caller, even when only storing the HMAC in
the database.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Clean up test.
Tom Clegg [Tue, 17 Nov 2020 20:42:11 +0000 (15:42 -0500)]
17106: Clean up test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Skip s3cmd test if s3cmd not installed.
Tom Clegg [Tue, 17 Nov 2020 14:39:50 +0000 (09:39 -0500)]
17106: Skip s3cmd test if s3cmd not installed.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Test S3 with modified v2 token issued by LoginCluster.
Tom Clegg [Tue, 17 Nov 2020 02:25:12 +0000 (21:25 -0500)]
17106: Test S3 with modified v2 token issued by LoginCluster.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17106: Accept v2 token with / replaced by _ as s3 access/secret key.
Tom Clegg [Tue, 17 Nov 2020 01:37:05 +0000 (20:37 -0500)]
17106: Accept v2 token with / replaced by _ as s3 access/secret key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoMerge branch '17012-fix-python-version-calculation' into master
Ward Vandewege [Tue, 17 Nov 2020 22:44:21 +0000 (17:44 -0500)]
Merge branch '17012-fix-python-version-calculation' into master

closes #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch 'master' into 17012-fix-python-version-calculation
Ward Vandewege [Tue, 17 Nov 2020 22:43:46 +0000 (17:43 -0500)]
Merge branch 'master' into 17012-fix-python-version-calculation

refs #17012

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17012: switch calculate_python_sdk_cwl_package_versions() to use the
Ward Vandewege [Tue, 17 Nov 2020 22:35:31 +0000 (17:35 -0500)]
17012: switch calculate_python_sdk_cwl_package_versions() to use the
       python version calculation code. This effectively is bash calling
       python calling bash, which I don't like, but the code is simpler.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17012: update calculation of python version number in run-library.sh
Ward Vandewege [Tue, 17 Nov 2020 15:26:58 +0000 (10:26 -0500)]
17012: update calculation of python version number in run-library.sh
       (used by a few docker image scripts).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17012: remove old gittaggers.py files.
Ward Vandewege [Tue, 17 Nov 2020 14:52:53 +0000 (09:52 -0500)]
17012: remove old gittaggers.py files.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17012: when calculating the version of our Python packages, take the
Ward Vandewege [Mon, 16 Nov 2020 20:08:26 +0000 (15:08 -0500)]
17012: when calculating the version of our Python packages, take the
       build directory into account.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Tue, 17 Nov 2020 01:40:47 +0000 (20:40 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Sun, 15 Nov 2020 19:18:40 +0000 (14:18 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16950: refactor caching; use ~/.cache/arvados/costanalyzer for that
Ward Vandewege [Sat, 14 Nov 2020 18:22:50 +0000 (13:22 -0500)]
16950: refactor caching; use ~/.cache/arvados/costanalyzer for that
       purpose.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16950: more changes after review.
Ward Vandewege [Fri, 13 Nov 2020 21:22:57 +0000 (16:22 -0500)]
16950: more changes after review.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '17111-deselect-unsigned-manifest'
Tom Clegg [Fri, 13 Nov 2020 21:57:25 +0000 (16:57 -0500)]
Merge branch '17111-deselect-unsigned-manifest'

fixes #17111

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years ago17111: Deselect unsigned_manifest_text by default in groups#contents.
Tom Clegg [Fri, 13 Nov 2020 20:44:43 +0000 (15:44 -0500)]
17111: Deselect unsigned_manifest_text by default in groups#contents.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix more golint warnings.
Ward Vandewege [Fri, 13 Nov 2020 01:02:56 +0000 (20:02 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16950: changes after review.
Ward Vandewege [Thu, 12 Nov 2020 03:07:48 +0000 (22:07 -0500)]
16950: changes after review.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Sun, 8 Nov 2020 21:41:21 +0000 (16:41 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '17086-add-focal-fossa' into master
Ward Vandewege [Sat, 7 Nov 2020 13:45:42 +0000 (08:45 -0500)]
Merge branch '17086-add-focal-fossa' into master

closes #17086

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17086: Merge branch 'master' into 17086-add-focal-fossa
Ward Vandewege [Sat, 7 Nov 2020 13:45:09 +0000 (08:45 -0500)]
17086: Merge branch 'master' into 17086-add-focal-fossa

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoFix require path in the 'script/rails' script for api server.
Ward Vandewege [Fri, 6 Nov 2020 13:34:38 +0000 (08:34 -0500)]
Fix require path in the 'script/rails' script for api server.

closes #17094

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago17086: build packages for Ubuntu 20.04, Focal Fossa.
Ward Vandewege [Thu, 5 Nov 2020 19:43:37 +0000 (14:43 -0500)]
17086: build packages for Ubuntu 20.04, Focal Fossa.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '17090-exclusive-lock' refs #17090
Peter Amstutz [Thu, 5 Nov 2020 20:15:13 +0000 (15:15 -0500)]
Merge branch '17090-exclusive-lock' refs #17090

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17090: Migration to refresh_permissions
Peter Amstutz [Thu, 5 Nov 2020 19:55:35 +0000 (14:55 -0500)]
17090: Migration to refresh_permissions

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17090: Use EXCLUSIVE lock for permission updates
Peter Amstutz [Thu, 5 Nov 2020 16:09:30 +0000 (11:09 -0500)]
17090: Use EXCLUSIVE lock for permission updates

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years agoFix more golint warnings.
Ward Vandewege [Thu, 5 Nov 2020 00:52:42 +0000 (19:52 -0500)]
Fix more golint warnings.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years ago16950: add tests for the costanalyzer.
Ward Vandewege [Tue, 3 Nov 2020 21:34:16 +0000 (16:34 -0500)]
16950: add tests for the costanalyzer.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

4 years agoMerge branch '17040-slow-query' refs #17040
Peter Amstutz [Tue, 3 Nov 2020 18:29:21 +0000 (13:29 -0500)]
Merge branch '17040-slow-query' refs #17040

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17040: Update structure.sql
Peter Amstutz [Tue, 3 Nov 2020 18:01:57 +0000 (13:01 -0500)]
17040: Update structure.sql

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17040: RefreshTrashedGroups migration, clear_permissions calls forget_cached_group_perms
Peter Amstutz [Tue, 3 Nov 2020 17:12:59 +0000 (12:12 -0500)]
17040: RefreshTrashedGroups migration, clear_permissions calls forget_cached_group_perms

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

4 years ago17040: Cache results of User.group_permissions
Peter Amstutz [Mon, 2 Nov 2020 21:20:02 +0000 (16:20 -0500)]
17040: Cache results of User.group_permissions

When requesting a list of groups (either directly, or with the
"shared" endpoint) it calls writable_by for each group.  This
indirectly calls User.group_permissions, which makes a database query.
When it does this for every group (with a database query each time),
which gets very expensive.

To address this, this commit caches the result of group_permissions on
the user object.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>