arvados.git
3 years agoMerge branch '16513-keep-exercise-improvements'
Ward Vandewege [Fri, 19 Jun 2020 15:37:42 +0000 (11:37 -0400)]
Merge branch '16513-keep-exercise-improvements'

refs #16513

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16513: Merge branch 'master' into 16513-keep-exercise-improvements
Ward Vandewege [Fri, 19 Jun 2020 15:36:55 +0000 (11:36 -0400)]
16513: Merge branch 'master' into 16513-keep-exercise-improvements

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '16526-ruby-and-python-build-script-updates'
Ward Vandewege [Fri, 19 Jun 2020 15:33:20 +0000 (11:33 -0400)]
Merge branch '16526-ruby-and-python-build-script-updates'

closes #16526

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16526: Merge branch 'master' into 16526-ruby-and-python-build-script-updates
Ward Vandewege [Fri, 19 Jun 2020 15:31:54 +0000 (11:31 -0400)]
16526: Merge branch 'master' into 16526-ruby-and-python-build-script-updates

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '16482-bump-cwltool-version'
Ward Vandewege [Thu, 18 Jun 2020 15:53:41 +0000 (11:53 -0400)]
Merge branch '16482-bump-cwltool-version'

closes #16482

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16482: Merge branch 'master' into 16482-bump-cwltool-version
Ward Vandewege [Thu, 18 Jun 2020 15:52:23 +0000 (11:52 -0400)]
16482: Merge branch 'master' into 16482-bump-cwltool-version

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16513: switch to CSV output only, send CSV to stdout and logs to stderr,
Ward Vandewege [Wed, 17 Jun 2020 16:31:39 +0000 (12:31 -0400)]
16513: switch to CSV output only, send CSV to stdout and logs to stderr,
       abort on ctrl-c, print final CSV line when ending.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '16171-oidc-config'
Tom Clegg [Wed, 17 Jun 2020 16:35:32 +0000 (12:35 -0400)]
Merge branch '16171-oidc-config'

closes #16171

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Link install docs to additional OpenIDConnect configs.
Tom Clegg [Wed, 17 Jun 2020 16:34:41 +0000 (12:34 -0400)]
16171: Link install docs to additional OpenIDConnect configs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Fix typo.
Tom Clegg [Wed, 17 Jun 2020 14:24:38 +0000 (10:24 -0400)]
16171: Fix typo.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16007-permission-table-rb' refs #16007
Peter Amstutz [Tue, 16 Jun 2020 21:25:38 +0000 (17:25 -0400)]
Merge branch '16007-permission-table-rb' refs #16007

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Update comments to discuss edge_id
Peter Amstutz [Tue, 16 Jun 2020 19:52:43 +0000 (15:52 -0400)]
16007: Update comments to discuss edge_id

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Handle overlapping permissions correctly
Peter Amstutz [Mon, 15 Jun 2020 22:17:09 +0000 (18:17 -0400)]
16007: Handle overlapping permissions correctly

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Add tests for overlapping permission links
Peter Amstutz [Mon, 15 Jun 2020 18:25:30 +0000 (14:25 -0400)]
16007: Add tests for overlapping permission links

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '16425-skip-unused-dirs'
Tom Clegg [Mon, 15 Jun 2020 15:17:48 +0000 (11:17 -0400)]
Merge branch '16425-skip-unused-dirs'

fixes #16425

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16427-undelete'
Tom Clegg [Fri, 12 Jun 2020 20:25:47 +0000 (16:25 -0400)]
Merge branch '16427-undelete'

closes #16427

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16425: Don't scan dirs that we don't write in.
Tom Clegg [Fri, 12 Jun 2020 20:14:20 +0000 (16:14 -0400)]
16425: Don't scan dirs that we don't write in.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16526: update the run-build-packages-python-and-ruby.sh script so that
Ward Vandewege [Fri, 12 Jun 2020 15:26:24 +0000 (11:26 -0400)]
16526: update the run-build-packages-python-and-ruby.sh script so that
it can be asked to do ruby gems or python packages alone. The default
behavior remains unchanged: both ruby gems and python packages will be
built.Some bash comparison cleanups.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16171: Merge branch 'master'
Tom Clegg [Fri, 12 Jun 2020 15:18:45 +0000 (11:18 -0400)]
16171: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Merge branch 'master'
Tom Clegg [Fri, 12 Jun 2020 15:17:41 +0000 (11:17 -0400)]
16427: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch 'master' into 16007-permission-table-rb
Peter Amstutz [Fri, 12 Jun 2020 14:43:53 +0000 (10:43 -0400)]
Merge branch 'master' into 16007-permission-table-rb

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16513: add an option to keep-exercise for a timed run, with a CSV output
Ward Vandewege [Fri, 12 Jun 2020 14:47:26 +0000 (10:47 -0400)]
16513: add an option to keep-exercise for a timed run, with a CSV output
       printed at the end.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16524: Pin rsa package to < 4.1
Peter Amstutz [Fri, 12 Jun 2020 14:13:24 +0000 (10:13 -0400)]
16524: Pin rsa package to < 4.1

Because >= 4.1 does not support python 2 any more.  refs #16524

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16427: Option to recover from given collection's last log entry.
Tom Clegg [Thu, 11 Jun 2020 20:29:33 +0000 (16:29 -0400)]
16427: Option to recover from given collection's last log entry.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoFix double slash in URL.
Tom Clegg [Thu, 11 Jun 2020 19:26:06 +0000 (15:26 -0400)]
Fix double slash in URL.

refs #16523

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16007: Add REVOKE_PERM and CAN_MANAGE_PERM constants
Peter Amstutz [Thu, 11 Jun 2020 17:12:04 +0000 (13:12 -0400)]
16007: Add REVOKE_PERM and CAN_MANAGE_PERM constants

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16427: Rename undelete -> recover-collection.
Tom Clegg [Wed, 10 Jun 2020 14:26:18 +0000 (10:26 -0400)]
16427: Rename undelete -> recover-collection.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Add new config keys to export list.
Tom Clegg [Tue, 9 Jun 2020 17:47:45 +0000 (13:47 -0400)]
16171: Add new config keys to export list.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Configurable "username" OIDC claim key.
Tom Clegg [Tue, 9 Jun 2020 14:55:08 +0000 (10:55 -0400)]
16171: Configurable "username" OIDC claim key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Configurable "email" and "email_verified" OIDC claim keys.
Tom Clegg [Tue, 9 Jun 2020 14:22:09 +0000 (10:22 -0400)]
16171: Configurable "email" and "email_verified" OIDC claim keys.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoadded type ContainerRequestList
Nico Cesar [Mon, 8 Jun 2020 20:35:22 +0000 (16:35 -0400)]
added type ContainerRequestList

refs #16462

Arvados-DCO-1.1-Signed-off-by: <nico@nicocesar.com>

3 years agoMerge branch '16171-oidc'
Tom Clegg [Mon, 8 Jun 2020 14:17:55 +0000 (10:17 -0400)]
Merge branch '16171-oidc'

refs #16171

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16007: Fix typo & use query parameters
Peter Amstutz [Fri, 5 Jun 2020 21:02:28 +0000 (17:02 -0400)]
16007: Fix typo & use query parameters

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16427: Add "undeleting collections" doc page in admin section.
Tom Clegg [Fri, 5 Jun 2020 20:23:01 +0000 (16:23 -0400)]
16427: Add "undeleting collections" doc page in admin section.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16007: Refactoring and update comments.
Peter Amstutz [Fri, 5 Jun 2020 20:20:24 +0000 (16:20 -0400)]
16007: Refactoring and update comments.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '16492-websocket-extensions-gem-upgrade'
Lucas Di Pentima [Fri, 5 Jun 2020 19:37:22 +0000 (16:37 -0300)]
Merge branch '16492-websocket-extensions-gem-upgrade'
Closes #16492

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16492: Upgrades websocket-extensions gem to fix a security alert.
Lucas Di Pentima [Fri, 5 Jun 2020 18:13:44 +0000 (15:13 -0300)]
16492: Upgrades websocket-extensions gem to fix a security alert.

We really don't use this gem as it's requested by actioncable from Rails 5
that we don't even load at runtime. But, as we cannot remove actioncable
from our Gemfile.lock files because of some other gems depending on 'rails',
we upgrade it to make GitHub happy.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16427: Support looking up old manifest for given log entry UUID.
Tom Clegg [Fri, 5 Jun 2020 17:10:22 +0000 (13:10 -0400)]
16427: Support looking up old manifest for given log entry UUID.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16427-undelete'
Tom Clegg [Fri, 5 Jun 2020 15:16:17 +0000 (11:16 -0400)]
Merge branch '16427-undelete'

refs #16427

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Update func comments.
Tom Clegg [Fri, 5 Jun 2020 15:15:44 +0000 (11:15 -0400)]
16427: Update func comments.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16007: Special handing for users with permissions on other users
Peter Amstutz [Thu, 4 Jun 2020 20:58:18 +0000 (16:58 -0400)]
16007: Special handing for users with permissions on other users

Revise & simplify permission traversal.  Don't traverse users except
when starting from the user (origin_uuid = starting_uuid).

This avoids disasterous queries where we re-traverse other users "just
in case" and end up recomputing the whole database.  As a tradeoff,
our epic readable_by query gets a touch more epic, as it now has to go
to the permissions table to check if there are other user permissions
the current user also is allowed to use.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16427: Merge branch 'master'
Tom Clegg [Fri, 5 Jun 2020 14:51:17 +0000 (10:51 -0400)]
16427: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Don't print legacy config path flags if they won't be used.
Tom Clegg [Thu, 4 Jun 2020 20:35:05 +0000 (16:35 -0400)]
16427: Don't print legacy config path flags if they won't be used.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Improve -help / usage message.
Tom Clegg [Thu, 4 Jun 2020 20:34:44 +0000 (16:34 -0400)]
16427: Improve -help / usage message.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Make test logging more obvious.
Tom Clegg [Thu, 4 Jun 2020 19:45:44 +0000 (15:45 -0400)]
16427: Make test logging more obvious.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Explain workerThreads choice.
Tom Clegg [Thu, 4 Jun 2020 19:44:13 +0000 (15:44 -0400)]
16427: Explain workerThreads choice.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Explain choice of blob ttl.
Tom Clegg [Thu, 4 Jun 2020 19:44:11 +0000 (15:44 -0400)]
16427: Explain choice of blob ttl.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Return error instead of ok bool from util funcs.
Tom Clegg [Thu, 4 Jun 2020 18:49:32 +0000 (14:49 -0400)]
16427: Return error instead of ok bool from util funcs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Change issuer config to string to avoid trailing-slash pain.
Tom Clegg [Thu, 4 Jun 2020 18:17:28 +0000 (14:17 -0400)]
16171: Change issuer config to string to avoid trailing-slash pain.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Warn about OIDC issuer URL spelling sensitivity.
Tom Clegg [Thu, 4 Jun 2020 15:44:29 +0000 (11:44 -0400)]
16171: Warn about OIDC issuer URL spelling sensitivity.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Test non-Google OIDC login with fake issuer.
Tom Clegg [Thu, 4 Jun 2020 15:21:43 +0000 (11:21 -0400)]
16171: Test non-Google OIDC login with fake issuer.

Ensures the proper credentials are used.

Exposes go-oidc's sensitivity to different spellings of equivalent
issuer URLs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Move more code to one-time setup func.
Tom Clegg [Thu, 4 Jun 2020 13:54:35 +0000 (09:54 -0400)]
16171: Move more code to one-time setup func.

Fix Google credentials used for non-Google OIDC provider.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16482: bump a-c-r's cwltool dependency to version 3.0.20200530110633 so
Ward Vandewege [Wed, 3 Jun 2020 19:42:03 +0000 (15:42 -0400)]
16482: bump a-c-r's cwltool dependency to version 3.0.20200530110633 so
that CWL v1.2.0-dev3 tests are supported. This also requires an update
to the schema-salad dependency, and some adjustment to account for code
that moved around in cwltool. Also, a fix to make sure the output from
resource_stream gets turned into a string, not a binary object.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agoBugfix: test_with_arvbox.sh in non-build mode now works again and uses
Ward Vandewege [Tue, 2 Jun 2020 21:43:40 +0000 (17:43 -0400)]
Bugfix: test_with_arvbox.sh in non-build mode now works again and uses
the logic from run-library.sh to determine the version of arvados/jobs
to pull. Also includes some extra error checking and a small comparison
operator bugfix.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agoFixes arvbox config, adding Login.SSO.Enable: true. No issue #
Lucas Di Pentima [Tue, 2 Jun 2020 18:00:40 +0000 (15:00 -0300)]
Fixes arvbox config, adding Login.SSO.Enable: true. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoMerge branch '16482-bump-cwltool-version'
Ward Vandewege [Wed, 3 Jun 2020 13:40:31 +0000 (09:40 -0400)]
Merge branch '16482-bump-cwltool-version'

closes #16482

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years ago16482: Merge branch 'master' into 16482-bump-cwltool-version
Ward Vandewege [Wed, 3 Jun 2020 13:39:41 +0000 (09:39 -0400)]
16482: Merge branch 'master' into 16482-bump-cwltool-version

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agoBugfix: test_with_arvbox.sh in non-build mode now works again and uses
Ward Vandewege [Tue, 2 Jun 2020 21:43:40 +0000 (17:43 -0400)]
Bugfix: test_with_arvbox.sh in non-build mode now works again and uses
the logic from run-library.sh to determine the version of arvados/jobs
to pull. Also includes some extra error checking and a small comparison
operator bugfix.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years ago16482: bump a-c-r's cwltool dependency to version 3.0.20200324120055 so
Ward Vandewege [Tue, 2 Jun 2020 21:30:56 +0000 (17:30 -0400)]
16482: bump a-c-r's cwltool dependency to version 3.0.20200324120055 so
       that CWL v1.2.0-dev3 tests are supported.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agoFixes arvbox config, adding Login.SSO.Enable: true. No issue #
Lucas Di Pentima [Tue, 2 Jun 2020 18:00:40 +0000 (15:00 -0300)]
Fixes arvbox config, adding Login.SSO.Enable: true. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoSome refactoring in the build scripts: move duplicated code to calculate
Ward Vandewege [Tue, 2 Jun 2020 14:05:44 +0000 (10:05 -0400)]
Some refactoring in the build scripts: move duplicated code to calculate
the cwl runner package version to a function. Remove unused argument to
the `version_from_git` function.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agoFix whitespace in tools/arvbox/lib/arvbox/docker/common.sh
Ward Vandewege [Mon, 1 Jun 2020 19:56:29 +0000 (15:56 -0400)]
Fix whitespace in tools/arvbox/lib/arvbox/docker/common.sh

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years ago16171: Add OIDC config keys to export whitelist.
Tom Clegg [Mon, 1 Jun 2020 15:08:06 +0000 (11:08 -0400)]
16171: Add OIDC config keys to export whitelist.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Tidy up config test.
Tom Clegg [Mon, 1 Jun 2020 14:28:15 +0000 (10:28 -0400)]
16171: Tidy up config test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Don't use Google as example of non-Google OIDC issuer.
Tom Clegg [Mon, 1 Jun 2020 14:17:21 +0000 (10:17 -0400)]
16171: Don't use Google as example of non-Google OIDC issuer.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Fix test order dependency.
Tom Clegg [Mon, 1 Jun 2020 13:42:22 +0000 (09:42 -0400)]
16427: Fix test order dependency.

Test was incorrectly assuming keep2, keep3 data dirs existed.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16171: Support non-Google OpenID Connect auth provider.
Tom Clegg [Sun, 31 May 2020 23:57:11 +0000 (19:57 -0400)]
16171: Support non-Google OpenID Connect auth provider.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Move blob signing test cases to arvados pkg.
Tom Clegg [Fri, 29 May 2020 20:38:24 +0000 (16:38 -0400)]
16427: Move blob signing test cases to arvados pkg.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Add test cases.
Tom Clegg [Fri, 29 May 2020 14:41:37 +0000 (10:41 -0400)]
16427: Add test cases.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Touch block timestamps to avoid garbage collection race.
Tom Clegg [Fri, 29 May 2020 14:36:13 +0000 (10:36 -0400)]
16427: Touch block timestamps to avoid garbage collection race.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16007: Add comment about override_edge_* parameters
Peter Amstutz [Fri, 29 May 2020 02:22:57 +0000 (22:22 -0400)]
16007: Add comment about override_edge_* parameters

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Ensure that updated permission edge overrides edges view
Peter Amstutz [Fri, 29 May 2020 01:41:46 +0000 (21:41 -0400)]
16007: Ensure that updated permission edge overrides edges view

An edge originating from a user can be traversed more than once, if
that edge is the same as the one being updated, ensure that it uses
the updated permission level and not the permission from the edges
view.  Necessary when revoking permissions.

Also moved comments into the body of the postgres functions to bring
them closer to the code, this also has the convenient effect of having
the comments appear in structure.sql function definitions so as to be
easier for future developers to find.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Enable permission correctness checking (only for tests)
Peter Amstutz [Thu, 28 May 2020 19:35:29 +0000 (15:35 -0400)]
16007: Enable permission correctness checking (only for tests)

* Explicitly set up a transaction in update_permissions
* Rename refresh_permission_view.rb -> update_permissions.rb
* Add skip_check_permissions_against_full_refresh

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: refresh_trashed uses a transaction
Peter Amstutz [Tue, 26 May 2020 19:12:56 +0000 (15:12 -0400)]
16007: refresh_trashed uses a transaction

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: More code comment detail about compute_permission_subgraph query
Peter Amstutz [Tue, 26 May 2020 18:11:24 +0000 (14:11 -0400)]
16007: More code comment detail about compute_permission_subgraph query

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Lots and lots lots of method documentation via code comments.
Peter Amstutz [Fri, 22 May 2020 19:00:22 +0000 (15:00 -0400)]
16007: Lots and lots lots of method documentation via code comments.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16007: Use incremental updates instead of materialized view for permissions
Peter Amstutz [Fri, 1 May 2020 18:26:35 +0000 (14:26 -0400)]
16007: Use incremental updates instead of materialized view for permissions

Separate 'trashed' from 'permissions' and remove 'trashed' from
permission computation.  Add postgres functions for computing trash
and update trashed_groups incrementally.  Make sure trash table gets
refreshed on database reset.  readable_by() now checks trash_at timestamp.

Drop materialized view and replace with a table that is updated
incrementally.  Add postgres functions for computing permissions.
Initialize materialized_permissions from search_permission_graph.
Call refresh_permissions in database_seeds.  Add index on
materialized_permissions.target_uuid.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16427: Add "touch" API to keepstore.
Tom Clegg [Thu, 28 May 2020 13:40:44 +0000 (09:40 -0400)]
16427: Add "touch" API to keepstore.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '15881-ldap'
Tom Clegg [Wed, 27 May 2020 20:10:06 +0000 (16:10 -0400)]
Merge branch '15881-ldap'

refs #15881

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16435-sync-groups-perm-levels'
Lucas Di Pentima [Wed, 27 May 2020 19:04:47 +0000 (16:04 -0300)]
Merge branch '16435-sync-groups-perm-levels'
Closes #16435

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16171: Rename googleLoginController to oidcLoginController.
Tom Clegg [Wed, 27 May 2020 13:44:36 +0000 (09:44 -0400)]
16171: Rename googleLoginController to oidcLoginController.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoThe arvados kubernetes repository has been renamed to arvados-k8s.
Ward Vandewege [Tue, 26 May 2020 13:40:48 +0000 (09:40 -0400)]
The arvados kubernetes repository has been renamed to arvados-k8s.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agodoc: fix typo on admin/federation page.
Ward Vandewege [Fri, 22 May 2020 12:18:06 +0000 (08:18 -0400)]
doc: fix typo on admin/federation page.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years ago16435: Merge branch 'master' into 16435-sync-groups-perm-levels
Lucas Di Pentima [Thu, 21 May 2020 23:09:38 +0000 (20:09 -0300)]
16435: Merge branch 'master' into 16435-sync-groups-perm-levels

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16435: Updates the documentation.
Lucas Di Pentima [Thu, 21 May 2020 23:08:40 +0000 (20:08 -0300)]
16435: Updates the documentation.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16435: Adds & updates tests.
Lucas Di Pentima [Thu, 21 May 2020 22:00:37 +0000 (19:00 -0300)]
16435: Adds & updates tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago16435: Avoids creating duplicated group->user links.
Lucas Di Pentima [Thu, 21 May 2020 21:11:10 +0000 (18:11 -0300)]
16435: Avoids creating duplicated group->user links.

When a user needs a permission change, the g->u link already exists.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years ago15881: Elaborate on SearchFilters behavior/uses.
Tom Clegg [Thu, 21 May 2020 20:10:12 +0000 (16:10 -0400)]
15881: Elaborate on SearchFilters behavior/uses.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16427: Add arvados-server undelete command.
Tom Clegg [Thu, 21 May 2020 19:48:19 +0000 (15:48 -0400)]
16427: Add arvados-server undelete command.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16435: Adds support for different permission levels: read, write & manage.
Lucas Di Pentima [Thu, 21 May 2020 19:21:12 +0000 (16:21 -0300)]
16435: Adds support for different permission levels: read, write & manage.

If the 3rd field isn't present on any record, it will fallback to 'can_write'
to maintain backwards compatibility.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agoWhen building/testing debian packages, run-build-packages-one-target.sh
Ward Vandewege [Thu, 21 May 2020 15:22:08 +0000 (11:22 -0400)]
When building/testing debian packages, run-build-packages-one-target.sh
now tests for the presence of dpkg-scanpackages and apt-ftparchive and
aborts if they are not present.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>

3 years agoMerge branch '16419-ssl-cert-file' refs #16419
Peter Amstutz [Thu, 21 May 2020 14:22:35 +0000 (10:22 -0400)]
Merge branch '16419-ssl-cert-file' refs #16419

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16419: Use CAINFO instead of CAPATH
Peter Amstutz [Thu, 21 May 2020 13:51:03 +0000 (09:51 -0400)]
16419: Use CAINFO instead of CAPATH

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago16427: Move signing code to sdk/go/arvados, add SignManifest.
Tom Clegg [Thu, 21 May 2020 13:48:24 +0000 (09:48 -0400)]
16427: Move signing code to sdk/go/arvados, add SignManifest.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years ago16435: Allows 2 or 3 fields per record on the CSV file.
Lucas Di Pentima [Wed, 20 May 2020 21:23:08 +0000 (18:23 -0300)]
16435: Allows 2 or 3 fields per record on the CSV file.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@di-pentima.com.ar>

3 years agokeep.py: python-api https certificate align pycurl with httplib2 certificate finder
Pjotr Prins [Wed, 20 May 2020 19:35:54 +0000 (14:35 -0500)]
keep.py: python-api https certificate align pycurl with httplib2 certificate finder

Arvados-DCO-1.1-Signed-off-by: Pjotr Prins <pjotr.public01@thebird.nl>

3 years agoutil.py: python-api https certificate openssl override as is used in GNU Guix
Pjotr Prins [Wed, 20 May 2020 16:12:20 +0000 (11:12 -0500)]
util.py: python-api https certificate openssl override as is used in GNU Guix

Arvados-DCO-1.1-Signed-off-by: Pjotr Prins <pjotr.public01@thebird.nl>

3 years agoMerge branch '16312-s3-signature-v4'
Tom Clegg [Wed, 20 May 2020 14:14:41 +0000 (10:14 -0400)]
Merge branch '16312-s3-signature-v4'

closes #16312

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16319-timestamp-precision'
Tom Clegg [Wed, 20 May 2020 14:10:58 +0000 (10:10 -0400)]
Merge branch '16319-timestamp-precision'

refs #16319

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

3 years agoMerge branch '16451-run-tests-interactive-run-test-many-times'
Ward Vandewege [Wed, 20 May 2020 14:00:47 +0000 (10:00 -0400)]
Merge branch '16451-run-tests-interactive-run-test-many-times'

closes #16451

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@jhvc.com>