arvados.git
2 years agoUpdate package versions for arvbox & provision.sh 2.3.2
Peter Amstutz [Thu, 9 Dec 2021 16:27:08 +0000 (11:27 -0500)]
Update package versions for arvbox & provision.sh

refs #18518

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoAdd (empty) upgrading section for 2.3.2
Peter Amstutz [Wed, 8 Dec 2021 21:22:12 +0000 (16:22 -0500)]
Add (empty) upgrading section for 2.3.2

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago17667: a round of edits after review feedback.
Ward Vandewege [Tue, 7 Dec 2021 00:46:01 +0000 (19:46 -0500)]
17667: a round of edits after review feedback.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years ago17667: add some missing information on the "Install prerequisites" page.
Ward Vandewege [Mon, 6 Dec 2021 17:48:57 +0000 (12:48 -0500)]
17667: add some missing information on the "Install prerequisites" page.
       Update the "Install PostgreSQL" page with a reference to Aurora
       RDS. Add a new page that details InternalURLs/ExternalURL.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years ago18547: Error out if two volumes return the same non-empty DeviceID.
Tom Clegg [Tue, 7 Dec 2021 00:32:34 +0000 (19:32 -0500)]
18547: Error out if two volumes return the same non-empty DeviceID.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago18547: Use volume UUID instead of DeviceID to deduplicate mounts.
Tom Clegg [Fri, 3 Dec 2021 15:54:59 +0000 (10:54 -0500)]
18547: Use volume UUID instead of DeviceID to deduplicate mounts.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago18376: Retry up to 4 times on EBADCOOKIE from readdir.
Tom Clegg [Thu, 18 Nov 2021 19:31:01 +0000 (14:31 -0500)]
18376: Retry up to 4 times on EBADCOOKIE from readdir.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago18491: Removes an unnecessary check that made the test fail on module upgrade.
Lucas Di Pentima [Tue, 30 Nov 2021 15:37:00 +0000 (12:37 -0300)]
18491: Removes an unnecessary check that made the test fail on module upgrade.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agogithub.com/Azure/go-autorest/autorest/azure/auth go-jwt security fix
Peter Amstutz [Mon, 6 Dec 2021 18:45:32 +0000 (13:45 -0500)]
github.com/Azure/go-autorest/autorest/azure/auth go-jwt security fix

refs #18491

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18511-java-sdk-groups-links' refs #18511
Peter Amstutz [Thu, 2 Dec 2021 15:31:27 +0000 (10:31 -0500)]
Merge branch '18511-java-sdk-groups-links' refs #18511

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18490-redundant-updates' refs #18490
Peter Amstutz [Wed, 1 Dec 2021 22:27:15 +0000 (17:27 -0500)]
Merge branch '18490-redundant-updates' refs #18490

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18480-arvput-special-files-handling' into main. Closes #18480
Lucas Di Pentima [Fri, 26 Nov 2021 16:21:34 +0000 (13:21 -0300)]
Merge branch '18480-arvput-special-files-handling' into main. Closes #18480

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago18488: Close pg connection when waiting for lock.
Tom Clegg [Tue, 30 Nov 2021 15:57:08 +0000 (10:57 -0500)]
18488: Close pg connection when waiting for lock.

In practice, pg_advisory_lock() calls pile up and consume database
connection slots, even when the corresponding processes have exited.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoDocumentation: fix AWS region name in the configuration reference and on
Ward Vandewege [Wed, 1 Dec 2021 20:48:54 +0000 (15:48 -0500)]
Documentation: fix AWS region name in the configuration reference and on
the S3 configuration page.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoUpdate provision.sh VERSION=2.3.1-1 refs #18361
Peter Amstutz [Wed, 24 Nov 2021 14:36:19 +0000 (09:36 -0500)]
Update provision.sh VERSION=2.3.1-1 refs #18361

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoUpdate arvbox DEFAULT_TAG to 2.3.1 refs #18361
Peter Amstutz [Wed, 24 Nov 2021 14:28:56 +0000 (09:28 -0500)]
Update arvbox DEFAULT_TAG to 2.3.1 refs #18361

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18313-arvbox-bootstrap-go-version' 2.3.1
Ward Vandewege [Tue, 2 Nov 2021 16:53:31 +0000 (12:53 -0400)]
Merge branch '18313-arvbox-bootstrap-go-version'

closes #18313

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoWe now rely on Go 1.16. Unbreak the arvbox build toolchain by switching
Ward Vandewege [Fri, 29 Oct 2021 13:29:29 +0000 (09:29 -0400)]
We now rely on Go 1.16. Unbreak the arvbox build toolchain by switching
to upstream Golang, the buster-backports version is too old.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoMerge branch '17742-provide-custom-certs'
Javier Bértoli [Thu, 21 Oct 2021 19:00:13 +0000 (16:00 -0300)]
Merge branch '17742-provide-custom-certs'

closes #17742
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

2 years agoMerge branch '18298-lsf-no-suitable-hosts'
Tom Clegg [Mon, 22 Nov 2021 18:45:21 +0000 (13:45 -0500)]
Merge branch '18298-lsf-no-suitable-hosts'

refs #18298

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '18298-lsf-no-suitable-hosts'
Tom Clegg [Fri, 19 Nov 2021 22:29:50 +0000 (17:29 -0500)]
Merge branch '18298-lsf-no-suitable-hosts'

refs #18298

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '18339-sweep-trash-lock'
Tom Clegg [Thu, 18 Nov 2021 20:01:06 +0000 (15:01 -0500)]
Merge branch '18339-sweep-trash-lock'

fixes #18339

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoUsing curl against https requires the ca-certificates package, update
Ward Vandewege [Fri, 19 Nov 2021 01:15:07 +0000 (20:15 -0500)]
Using curl against https requires the ca-certificates package, update
the docs accordingly. Also remove the 'silent' flag from curl so that
errors are shown.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoUpgrade notes are for 2.3.1 refs #18361
Peter Amstutz [Thu, 18 Nov 2021 19:38:46 +0000 (14:38 -0500)]
Upgrade notes are for 2.3.1 refs #18361

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoFix up "upgrading to {earlier version}" links.
Tom Clegg [Thu, 28 Oct 2021 14:28:44 +0000 (10:28 -0400)]
Fix up "upgrading to {earlier version}" links.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '17962-check-wf-definition' refs #17962
Peter Amstutz [Wed, 17 Nov 2021 22:03:57 +0000 (17:03 -0500)]
Merge branch '17962-check-wf-definition' refs #17962

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18285-cwl-hint-warning' refs #18285
Peter Amstutz [Wed, 17 Nov 2021 17:55:16 +0000 (12:55 -0500)]
Merge branch '18285-cwl-hint-warning' refs #18285

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18363-managed-properties-doc-improvement' into main.
Lucas Di Pentima [Wed, 17 Nov 2021 18:28:37 +0000 (15:28 -0300)]
Merge branch '18363-managed-properties-doc-improvement' into main.
Closes #18363

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '18340-delete-role-filter-groups' into main. Closes #18340.
Lucas Di Pentima [Wed, 17 Nov 2021 15:03:42 +0000 (12:03 -0300)]
Merge branch '18340-delete-role-filter-groups' into main. Closes #18340.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '18336-httplib2-pysdk-issues' into main. Closes #18336
Lucas Di Pentima [Wed, 17 Nov 2021 15:16:54 +0000 (12:16 -0300)]
Merge branch '18336-httplib2-pysdk-issues' into main. Closes #18336

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '18346-crunchrun-no-events' refs #18346
Peter Amstutz [Fri, 12 Nov 2021 14:38:05 +0000 (09:38 -0500)]
Merge branch '18346-crunchrun-no-events' refs #18346

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '18376-nfs-readdirent'
Tom Clegg [Tue, 16 Nov 2021 21:34:45 +0000 (16:34 -0500)]
Merge branch '18376-nfs-readdirent'

fixes #18376

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '17635-pysdk-collection-preserve-version' into main. Closes #17635
Lucas Di Pentima [Tue, 16 Nov 2021 18:48:47 +0000 (15:48 -0300)]
Merge branch '17635-pysdk-collection-preserve-version' into main. Closes #17635

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '18215-select-param-update-create' into main. Refs #18215
Lucas Di Pentima [Mon, 15 Nov 2021 18:12:19 +0000 (15:12 -0300)]
Merge branch '18215-select-param-update-create' into main. Refs #18215

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '18316-fuse-read-only' refs #18316
Peter Amstutz [Mon, 15 Nov 2021 17:50:54 +0000 (12:50 -0500)]
Merge branch '18316-fuse-read-only' refs #18316

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '16817-users-visible-upon-activation'
Tom Clegg [Thu, 11 Nov 2021 20:47:05 +0000 (15:47 -0500)]
Merge branch '16817-users-visible-upon-activation'

closes #16817

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago12859: Fix unclosed file descriptors in local filesystem driver.
Tom Clegg [Mon, 1 Nov 2021 18:49:29 +0000 (14:49 -0400)]
12859: Fix unclosed file descriptors in local filesystem driver.

Temporary file was not being closed/removed in the case where client
disconnection is detected while waiting for the volume-level serialize
lock.

Also, GetDeviceID was leaking one file descriptor per volume at
startup time.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago18318: Updates the nokogiri dependency on API & WB1.
Lucas Di Pentima [Tue, 2 Nov 2021 13:04:48 +0000 (10:04 -0300)]
18318: Updates the nokogiri dependency on API & WB1.

Addresses https://nvd.nist.gov/vuln/detail/CVE-2021-41098

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago18309: remove faraday dependency in the arvados-login-sync gem, instead
Ward Vandewege [Fri, 29 Oct 2021 15:29:34 +0000 (11:29 -0400)]
18309: remove faraday dependency in the arvados-login-sync gem, instead
       depend on the correct version of the arvados-google-api-client
       gem.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoController test fix.
Ward Vandewege [Thu, 28 Oct 2021 23:47:05 +0000 (19:47 -0400)]
Controller test fix.

refs #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoAnother rails test fix.
Ward Vandewege [Thu, 28 Oct 2021 18:10:40 +0000 (14:10 -0400)]
Another rails test fix.

refs #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoUpdate the test db structure.
Ward Vandewege [Thu, 28 Oct 2021 16:48:02 +0000 (12:48 -0400)]
Update the test db structure.

refs #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years ago18183: add a database migration that deletes tokens and ssh keys that
Ward Vandewege [Wed, 27 Oct 2021 19:48:54 +0000 (15:48 -0400)]
18183: add a database migration that deletes tokens and ssh keys that
       belong to inactive users.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years ago18183: When the user unsetup api endpoint is hit, any tokens owned by
Ward Vandewege [Wed, 27 Oct 2021 19:05:00 +0000 (15:05 -0400)]
18183: When the user unsetup api endpoint is hit, any tokens owned by
       the user should be deleted.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoMerge branch '17944-backend-vocabulary-validation-rebased' into main.
Lucas Di Pentima [Thu, 11 Nov 2021 17:51:31 +0000 (14:51 -0300)]
Merge branch '17944-backend-vocabulary-validation-rebased' into main.

Refs #17944

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago18346: Do not forward locally-issued token to own login cluster.
Tom Clegg [Wed, 10 Nov 2021 16:13:09 +0000 (11:13 -0500)]
18346: Do not forward locally-issued token to own login cluster.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoRemve last reference to terraform for now.
Ward Vandewege [Thu, 4 Nov 2021 20:06:03 +0000 (16:06 -0400)]
Remve last reference to terraform for now.

refs #17450

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoRemove reference to terraform code that was never checked in.
Ward Vandewege [Thu, 4 Nov 2021 19:59:26 +0000 (15:59 -0400)]
Remove reference to terraform code that was never checked in.

refs #17450

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

2 years agoMake the linkchecker more strict so it objects to relative links within
Ward Vandewege [Thu, 4 Nov 2021 19:55:17 +0000 (15:55 -0400)]
Make the linkchecker more strict so it objects to relative links within
our doc tree that do not have the {{ site.baseurl }} prefix.

Fix all the links that were incorrect that way.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoSet arvbox DEFAULT_TAG=2.3.0
Peter Amstutz [Fri, 29 Oct 2021 21:01:19 +0000 (17:01 -0400)]
Set arvbox DEFAULT_TAG=2.3.0

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix arvbox versioning. no issue #
Peter Amstutz [Fri, 29 Oct 2021 20:59:55 +0000 (16:59 -0400)]
Fix arvbox versioning.  no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoTweak docker image tagging
Peter Amstutz [Thu, 9 Sep 2021 14:16:49 +0000 (10:16 -0400)]
Tweak docker image tagging

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix top of upgrade notes to 2.3.0
Peter Amstutz [Wed, 27 Oct 2021 18:37:47 +0000 (14:37 -0400)]
Fix top of upgrade notes to 2.3.0

refs #18130

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoSince CWL defaults to 256M of ram, make that the suggested amount for
Ward Vandewege [Tue, 26 Oct 2021 18:28:23 +0000 (14:28 -0400)]
Since CWL defaults to 256M of ram, make that the suggested amount for
the `mksquashfs mem` setting in singularity.conf.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18290: address review comments. 2.3.0
Ward Vandewege [Mon, 25 Oct 2021 14:15:21 +0000 (10:15 -0400)]
18290: address review comments.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18290: LSF: make the bsub arguments completely configurable.
Ward Vandewege [Sat, 23 Oct 2021 15:42:14 +0000 (11:42 -0400)]
18290: LSF: make the bsub arguments completely configurable.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18281: Rephrase the help text for --force-rotate option.
Ward Vandewege [Thu, 21 Oct 2021 18:19:42 +0000 (14:19 -0400)]
18281: Rephrase the help text for --force-rotate option.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18281: make arvados-login-sync smart enough to replace expired tokens.
Ward Vandewege [Wed, 20 Oct 2021 20:28:30 +0000 (16:28 -0400)]
18281: make arvados-login-sync smart enough to replace expired tokens.
       Also add a --debug parameter and some debug output.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18288: when storing the anonymous user token, make sure to clear the
Ward Vandewege [Thu, 21 Oct 2021 20:27:59 +0000 (16:27 -0400)]
18288: when storing the anonymous user token, make sure to clear the
       expires_at field.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18288: when running the db:check_long_lived_tokens and
Ward Vandewege [Thu, 21 Oct 2021 20:16:04 +0000 (16:16 -0400)]
18288: when running the db:check_long_lived_tokens and
       db:fix_long_lived_tokens rake tasks, do not expire the
       anonymouspublic token.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: add support to set `mksquashfs mem` for Singularity in our
Ward Vandewege [Fri, 22 Oct 2021 16:47:17 +0000 (12:47 -0400)]
18289: add support to set `mksquashfs mem` for Singularity in our
       compute node image builder for cloud setups.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: Address review comments (documentation).
Ward Vandewege [Fri, 22 Oct 2021 16:46:45 +0000 (12:46 -0400)]
18289: Address review comments (documentation).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: Documentation changes to reflect we support both Slurm and LSF.
Ward Vandewege [Fri, 22 Oct 2021 14:29:16 +0000 (10:29 -0400)]
18289: Documentation changes to reflect we support both Slurm and LSF.
       Refactor the compute node installation instructions. Fix a few
       linkchecker warnings.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: documentation: singularity refinements.
Ward Vandewege [Thu, 21 Oct 2021 23:33:48 +0000 (19:33 -0400)]
18289: documentation: singularity refinements.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: only pass the --allow-other argument to arv-mount when the
Ward Vandewege [Thu, 21 Oct 2021 20:53:03 +0000 (16:53 -0400)]
18289: only pass the --allow-other argument to arv-mount when the
       runtime is Docker.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18286: Document requirement to add Services.DispatchLSF section.
Tom Clegg [Thu, 21 Oct 2021 18:38:48 +0000 (14:38 -0400)]
18286: Document requirement to add Services.DispatchLSF section.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoPin arvados 2.3.0 version in provision.sh. Refs #18130
Lucas Di Pentima [Mon, 20 Sep 2021 18:26:12 +0000 (15:26 -0300)]
Pin arvados 2.3.0 version in provision.sh. Refs #18130

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

3 years agoMerge branch '18260-update-cwltool' refs #18260
Peter Amstutz [Wed, 20 Oct 2021 18:28:23 +0000 (14:28 -0400)]
Merge branch '18260-update-cwltool' refs #18260

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18260: Update when we're going to actually run the workflow
Peter Amstutz [Wed, 20 Oct 2021 01:11:05 +0000 (21:11 -0400)]
18260: Update when we're going to actually run the workflow

Update cwltool version for bugfixes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoDocumentation: when starting the Arvados k8s helm chart on GKE, do not
Ward Vandewege [Tue, 19 Oct 2021 20:03:53 +0000 (16:03 -0400)]
Documentation: when starting the Arvados k8s helm chart on GKE, do not
specify the k8s version (use the default).

refs #18278

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18260: Update cwltool for fix for integration test 35
Peter Amstutz [Tue, 19 Oct 2021 20:04:29 +0000 (16:04 -0400)]
18260: Update cwltool for fix for integration test 35

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18259-doc-architecture-singularity'
Ward Vandewege [Tue, 19 Oct 2021 20:02:14 +0000 (16:02 -0400)]
Merge branch '18259-doc-architecture-singularity'

closes #18259

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18259: documentation: add more detail about Singularity.
Ward Vandewege [Fri, 15 Oct 2021 19:36:56 +0000 (15:36 -0400)]
18259: documentation: add more detail about Singularity.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUse docker inspect instead of iterating over images list
Peter Amstutz [Wed, 13 Oct 2021 19:34:29 +0000 (15:34 -0400)]
Use docker inspect instead of iterating over images list

refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoDocumentation: silence the curl command to get our Arvados package signing key.
Ward Vandewege [Sun, 10 Oct 2021 15:30:32 +0000 (11:30 -0400)]
Documentation: silence the curl command to get our Arvados package signing key.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoRemove long dead `run` and `pipeline` subcommands from the `arv`
Ward Vandewege [Sun, 10 Oct 2021 00:29:24 +0000 (20:29 -0400)]
Remove long dead `run` and `pipeline` subcommands from the `arv`
executable.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18259-singularity-doc'
Ward Vandewege [Fri, 8 Oct 2021 21:36:09 +0000 (17:36 -0400)]
Merge branch '18259-singularity-doc'

refs #18259

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18259: mention `mksquashfs mem` config setting in singularity.conf
Ward Vandewege [Fri, 8 Oct 2021 16:49:20 +0000 (12:49 -0400)]
18259: mention `mksquashfs mem` config setting in singularity.conf

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoThe minimum supported version of Python is 3.6
Ward Vandewege [Fri, 8 Oct 2021 21:20:43 +0000 (17:20 -0400)]
The minimum supported version of Python is 3.6

refs #18261

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix CWL tests mixed_version_v10_wf and command_input_file_expression
Peter Amstutz [Fri, 8 Oct 2021 20:06:58 +0000 (16:06 -0400)]
Fix CWL tests mixed_version_v10_wf and command_input_file_expression

refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18258-set-singularity-cachedir'
Ward Vandewege [Fri, 8 Oct 2021 16:50:08 +0000 (12:50 -0400)]
Merge branch '18258-set-singularity-cachedir'

closes #18258

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18258: also handle SINGULARITY_TMPDIR
Ward Vandewege [Fri, 8 Oct 2021 14:43:52 +0000 (10:43 -0400)]
18258: also handle SINGULARITY_TMPDIR

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18258: add missing error check on os.Mkdir
Ward Vandewege [Fri, 8 Oct 2021 14:38:17 +0000 (10:38 -0400)]
18258: add missing error check on os.Mkdir

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18258: set the SINGULARITY_CACHEDIR to a path underneath our temporary
Ward Vandewege [Fri, 8 Oct 2021 14:24:28 +0000 (10:24 -0400)]
18258: set the SINGULARITY_CACHEDIR to a path underneath our temporary
       space, and clean up after we're done building.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoarvbox fix, keepstore is part of arvados-server now
Peter Amstutz [Thu, 7 Oct 2021 18:01:15 +0000 (14:01 -0400)]
arvbox fix, keepstore is part of arvados-server now

refs #18216

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18113-change-cloudops-defaults'
Ward Vandewege [Thu, 7 Oct 2021 17:26:29 +0000 (13:26 -0400)]
Merge branch '18113-change-cloudops-defaults'

closes #18113

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18113: Merge branch 'main' into 18113-change-cloudops-defaults
Ward Vandewege [Thu, 7 Oct 2021 17:26:12 +0000 (13:26 -0400)]
18113: Merge branch 'main' into 18113-change-cloudops-defaults

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18113: change MaxCloudOpsPerSecond and MaxConcurrentInstanceCreateOps
Ward Vandewege [Wed, 29 Sep 2021 16:08:17 +0000 (12:08 -0400)]
18113: change MaxCloudOpsPerSecond and MaxConcurrentInstanceCreateOps
       defaults.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoRemove the failing pre-build (--only-deps) step for crunch
Peter Amstutz [Thu, 7 Oct 2021 14:52:30 +0000 (10:52 -0400)]
Remove the failing pre-build (--only-deps) step for crunch

It will just do what it needs to when the container starts.

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoRemove the 'experimental' label in our config reference from things that
Ward Vandewege [Thu, 7 Oct 2021 14:14:23 +0000 (10:14 -0400)]
Remove the 'experimental' label in our config reference from things that
are not (PAM login support, a-d-c, Singularity support). Update the docs
for our LSF and Singularity support along the same lines, and update
formatting a bit to make the current limitations of the implementation
clear.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoWe support Singularity 3.7.4.
Ward Vandewege [Thu, 7 Oct 2021 13:57:02 +0000 (09:57 -0400)]
We support Singularity 3.7.4.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18238-singularity-home' refs #18238
Peter Amstutz [Thu, 7 Oct 2021 14:08:43 +0000 (10:08 -0400)]
Merge branch '18238-singularity-home' refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18238: Another try at Singularity $HOME fix
Peter Amstutz [Thu, 7 Oct 2021 14:04:48 +0000 (10:04 -0400)]
18238: Another try at Singularity $HOME fix

Use just the --home parameter, which appears to have option of
behaving as a special case of --bind, and apparently (according to
cwltool experience) works across more Singularity versions.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18216-arvados-server-keepstore'
Tom Clegg [Wed, 6 Oct 2021 20:41:17 +0000 (16:41 -0400)]
Merge branch '18216-arvados-server-keepstore'

closes #18216

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '18248-fix-ubuntu-18.04-testing'
Javier Bértoli [Wed, 6 Oct 2021 20:08:01 +0000 (17:08 -0300)]
Merge branch '18248-fix-ubuntu-18.04-testing'

closes #18248
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago18248: update comment addressing review suggestions
Javier Bértoli [Wed, 6 Oct 2021 20:01:16 +0000 (17:01 -0300)]
18248: update comment addressing review suggestions

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoMerge branch '18238-containers-in-arvbox' refs #18238
Peter Amstutz [Wed, 6 Oct 2021 19:58:02 +0000 (15:58 -0400)]
Merge branch '18238-containers-in-arvbox' refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix setting HOME on singularity. Fix a-c-r secondaryFiles.
Peter Amstutz [Wed, 6 Oct 2021 19:48:19 +0000 (15:48 -0400)]
Fix setting HOME on singularity.  Fix a-c-r secondaryFiles.

To fix CWL conformance tests.

refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '17779-request-id-in-exception'
Tom Clegg [Wed, 6 Oct 2021 18:51:41 +0000 (14:51 -0400)]
Merge branch '17779-request-id-in-exception'

closes #17779

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '18247-faraday-version'
Ward Vandewege [Wed, 6 Oct 2021 13:49:50 +0000 (09:49 -0400)]
Merge branch '18247-faraday-version'

refs #18247

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>