arvados.git
3 years ago18318: Updates the nokogiri dependency on API & WB1.
Lucas Di Pentima [Tue, 2 Nov 2021 13:04:48 +0000 (10:04 -0300)]
18318: Updates the nokogiri dependency on API & WB1.

Addresses https://nvd.nist.gov/vuln/detail/CVE-2021-41098

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

3 years agoTweak docker image tagging
Peter Amstutz [Thu, 9 Sep 2021 14:16:49 +0000 (10:16 -0400)]
Tweak docker image tagging

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18309-faraday-dependency'
Ward Vandewege [Fri, 29 Oct 2021 17:22:43 +0000 (13:22 -0400)]
Merge branch '18309-faraday-dependency'

closes #18309

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18309: remove faraday dependency in the arvados-login-sync gem, instead
Ward Vandewege [Fri, 29 Oct 2021 15:29:34 +0000 (11:29 -0400)]
18309: remove faraday dependency in the arvados-login-sync gem, instead
       depend on the correct version of the arvados-google-api-client
       gem.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18273-java-timeout' refs #18273
Peter Amstutz [Fri, 29 Oct 2021 15:17:36 +0000 (11:17 -0400)]
Merge branch '18273-java-timeout' refs #18273

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18273: Tweak java sdk example
Peter Amstutz [Fri, 29 Oct 2021 14:54:19 +0000 (10:54 -0400)]
18273: Tweak java sdk example

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoWe now rely on Go 1.16. Unbreak the arvbox build toolchain by switching
Ward Vandewege [Fri, 29 Oct 2021 13:29:29 +0000 (09:29 -0400)]
We now rely on Go 1.16. Unbreak the arvbox build toolchain by switching
to upstream Golang, the buster-backports version is too old.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoController test fix.
Ward Vandewege [Thu, 28 Oct 2021 23:47:05 +0000 (19:47 -0400)]
Controller test fix.

refs #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18273: simplify and use version-at-commit.sh
Peter Amstutz [Thu, 28 Oct 2021 18:57:04 +0000 (14:57 -0400)]
18273: simplify and use version-at-commit.sh

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18273: Set docker gradle version
Peter Amstutz [Thu, 28 Oct 2021 18:37:28 +0000 (14:37 -0400)]
18273: Set docker gradle version

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoAnother rails test fix.
Ward Vandewege [Thu, 28 Oct 2021 18:10:40 +0000 (14:10 -0400)]
Another rails test fix.

refs #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18264-cwl-test-running-improvements'
Ward Vandewege [Thu, 28 Oct 2021 17:40:31 +0000 (13:40 -0400)]
Merge branch '18264-cwl-test-running-improvements'

refs #18264

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18264: Merge branch 'main' into 18264-cwl-test-running-improvements
Ward Vandewege [Thu, 28 Oct 2021 17:40:12 +0000 (13:40 -0400)]
18264: Merge branch 'main' into 18264-cwl-test-running-improvements

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUpdate the test db structure.
Ward Vandewege [Thu, 28 Oct 2021 16:48:02 +0000 (12:48 -0400)]
Update the test db structure.

refs #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18273: Increase default java sdk timeouts & make them configuable
Peter Amstutz [Thu, 14 Oct 2021 19:09:00 +0000 (15:09 -0400)]
18273: Increase default java sdk timeouts & make them configuable

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoFix up "upgrading to {earlier version}" links.
Tom Clegg [Thu, 28 Oct 2021 14:28:44 +0000 (10:28 -0400)]
Fix up "upgrading to {earlier version}" links.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '16347-container-local-keepstore'
Tom Clegg [Thu, 28 Oct 2021 14:14:09 +0000 (10:14 -0400)]
Merge branch '16347-container-local-keepstore'

closes #16347

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Clarify upgrade note.
Tom Clegg [Thu, 28 Oct 2021 13:58:36 +0000 (09:58 -0400)]
16347: Clarify upgrade note.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '18183-revoke-tokens-when-user-is-disabled'
Ward Vandewege [Wed, 27 Oct 2021 21:33:37 +0000 (17:33 -0400)]
Merge branch '18183-revoke-tokens-when-user-is-disabled'

closes #18183

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18183: Merge branch 'main' into 18183-revoke-tokens-when-user-is-disabled
Ward Vandewege [Wed, 27 Oct 2021 21:32:57 +0000 (17:32 -0400)]
18183: Merge branch 'main' into 18183-revoke-tokens-when-user-is-disabled

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix top of upgrade notes to 2.3.0
Peter Amstutz [Wed, 27 Oct 2021 18:37:47 +0000 (14:37 -0400)]
Fix top of upgrade notes to 2.3.0

refs #18130

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18183: add a database migration that deletes tokens and ssh keys that
Ward Vandewege [Wed, 27 Oct 2021 19:48:54 +0000 (15:48 -0400)]
18183: add a database migration that deletes tokens and ssh keys that
       belong to inactive users.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18183: When the user unsetup api endpoint is hit, any tokens owned by
Ward Vandewege [Wed, 27 Oct 2021 19:05:00 +0000 (15:05 -0400)]
18183: When the user unsetup api endpoint is hit, any tokens owned by
       the user should be deleted.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16347: Fix race in test helper func.
Tom Clegg [Wed, 27 Oct 2021 19:33:09 +0000 (15:33 -0400)]
16347: Fix race in test helper func.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Add upgrade note.
Tom Clegg [Wed, 27 Oct 2021 18:34:47 +0000 (14:34 -0400)]
16347: Add upgrade note.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Merge branch 'main'
Tom Clegg [Wed, 27 Oct 2021 18:12:56 +0000 (14:12 -0400)]
16347: Merge branch 'main'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago18264: add a script that uses the existing version logic in the build
Ward Vandewege [Tue, 26 Oct 2021 18:52:32 +0000 (14:52 -0400)]
18264: add a script that uses the existing version logic in the build
       scripts to generate a package version string from a local Arvados
       git checkout.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoSince CWL defaults to 256M of ram, make that the suggested amount for
Ward Vandewege [Tue, 26 Oct 2021 18:28:23 +0000 (14:28 -0400)]
Since CWL defaults to 256M of ram, make that the suggested amount for
the `mksquashfs mem` setting in singularity.conf.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18290-lsf-bsub-args-template'
Ward Vandewege [Mon, 25 Oct 2021 15:28:09 +0000 (11:28 -0400)]
Merge branch '18290-lsf-bsub-args-template'

closes #18290

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18290: Merge branch 'main' into 18290-lsf-bsub-args-template
Ward Vandewege [Mon, 25 Oct 2021 15:27:37 +0000 (11:27 -0400)]
18290: Merge branch 'main' into 18290-lsf-bsub-args-template

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18290: address review comments.
Ward Vandewege [Mon, 25 Oct 2021 14:15:21 +0000 (10:15 -0400)]
18290: address review comments.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18290: LSF: make the bsub arguments completely configurable.
Ward Vandewege [Sat, 23 Oct 2021 15:42:14 +0000 (11:42 -0400)]
18290: LSF: make the bsub arguments completely configurable.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18287-external-url-defaults'
Tom Clegg [Fri, 22 Oct 2021 19:29:04 +0000 (15:29 -0400)]
Merge branch '18287-external-url-defaults'

fixes #18287

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years agoMerge branch '18289-only-pass-allow-other-when-running-docker'
Ward Vandewege [Fri, 22 Oct 2021 18:19:19 +0000 (14:19 -0400)]
Merge branch '18289-only-pass-allow-other-when-running-docker'

closes #18289

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: add support to set `mksquashfs mem` for Singularity in our
Ward Vandewege [Fri, 22 Oct 2021 16:47:17 +0000 (12:47 -0400)]
18289: add support to set `mksquashfs mem` for Singularity in our
       compute node image builder for cloud setups.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: Address review comments (documentation).
Ward Vandewege [Fri, 22 Oct 2021 16:46:45 +0000 (12:46 -0400)]
18289: Address review comments (documentation).

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: Documentation changes to reflect we support both Slurm and LSF.
Ward Vandewege [Fri, 22 Oct 2021 14:29:16 +0000 (10:29 -0400)]
18289: Documentation changes to reflect we support both Slurm and LSF.
       Refactor the compute node installation instructions. Fix a few
       linkchecker warnings.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: documentation: singularity refinements.
Ward Vandewege [Thu, 21 Oct 2021 23:33:48 +0000 (19:33 -0400)]
18289: documentation: singularity refinements.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18289: only pass the --allow-other argument to arv-mount when the
Ward Vandewege [Thu, 21 Oct 2021 20:53:03 +0000 (16:53 -0400)]
18289: only pass the --allow-other argument to arv-mount when the
       runtime is Docker.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18288-fix-anonymouspublic-token-expiration'
Ward Vandewege [Thu, 21 Oct 2021 20:42:37 +0000 (16:42 -0400)]
Merge branch '18288-fix-anonymouspublic-token-expiration'

closes #18288

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18288: when storing the anonymous user token, make sure to clear the
Ward Vandewege [Thu, 21 Oct 2021 20:27:59 +0000 (16:27 -0400)]
18288: when storing the anonymous user token, make sure to clear the
       expires_at field.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18288: when running the db:check_long_lived_tokens and
Ward Vandewege [Thu, 21 Oct 2021 20:16:04 +0000 (16:16 -0400)]
18288: when running the db:check_long_lived_tokens and
       db:fix_long_lived_tokens rake tasks, do not expire the
       anonymouspublic token.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '17742-provide-custom-certs'
Javier Bértoli [Thu, 21 Oct 2021 19:00:13 +0000 (16:00 -0300)]
Merge branch '17742-provide-custom-certs'

closes #17742
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17742: update documentation addressing review suggestions
Javier Bértoli [Thu, 21 Oct 2021 18:59:04 +0000 (15:59 -0300)]
17742: update documentation addressing review suggestions

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoMerge branch '18281-login-sync-expired-tokens'
Ward Vandewege [Thu, 21 Oct 2021 18:47:04 +0000 (14:47 -0400)]
Merge branch '18281-login-sync-expired-tokens'

closes #18281

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch 'main' into 18281-login-sync-expired-tokens
Ward Vandewege [Thu, 21 Oct 2021 18:46:26 +0000 (14:46 -0400)]
Merge branch 'main' into 18281-login-sync-expired-tokens

closes #18281

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18286-lsf-doc'
Tom Clegg [Thu, 21 Oct 2021 18:37:23 +0000 (14:37 -0400)]
Merge branch '18286-lsf-doc'

refs #18286

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago18287: Remove superfluous {ExternalURL: "-"} values.
Tom Clegg [Thu, 21 Oct 2021 18:35:43 +0000 (14:35 -0400)]
18287: Remove superfluous {ExternalURL: "-"} values.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago18281: Rephrase the help text for --force-rotate option.
Ward Vandewege [Thu, 21 Oct 2021 18:19:42 +0000 (14:19 -0400)]
18281: Rephrase the help text for --force-rotate option.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18286: Document requirement to add Services.DispatchLSF section.
Tom Clegg [Thu, 21 Oct 2021 18:00:45 +0000 (14:00 -0400)]
18286: Document requirement to add Services.DispatchLSF section.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago18281: make arvados-login-sync smart enough to replace expired tokens.
Ward Vandewege [Wed, 20 Oct 2021 20:28:30 +0000 (16:28 -0400)]
18281: make arvados-login-sync smart enough to replace expired tokens.
       Also add a --debug parameter and some debug output.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18260-update-cwltool' refs #18260
Peter Amstutz [Wed, 20 Oct 2021 18:28:23 +0000 (14:28 -0400)]
Merge branch '18260-update-cwltool' refs #18260

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18260: Update when we're going to actually run the workflow
Peter Amstutz [Wed, 20 Oct 2021 01:11:05 +0000 (21:11 -0400)]
18260: Update when we're going to actually run the workflow

Update cwltool version for bugfixes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoDocumentation: when starting the Arvados k8s helm chart on GKE, do not
Ward Vandewege [Tue, 19 Oct 2021 20:03:53 +0000 (16:03 -0400)]
Documentation: when starting the Arvados k8s helm chart on GKE, do not
specify the k8s version (use the default).

refs #18278

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18260: Update cwltool for fix for integration test 35
Peter Amstutz [Tue, 19 Oct 2021 20:04:29 +0000 (16:04 -0400)]
18260: Update cwltool for fix for integration test 35

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18259-doc-architecture-singularity'
Ward Vandewege [Tue, 19 Oct 2021 20:02:14 +0000 (16:02 -0400)]
Merge branch '18259-doc-architecture-singularity'

closes #18259

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16347: Reserve memory for keepstore buffers when choosing node size.
Tom Clegg [Tue, 19 Oct 2021 15:57:19 +0000 (11:57 -0400)]
16347: Reserve memory for keepstore buffers when choosing node size.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Fix leaky state in test case.
Tom Clegg [Tue, 19 Oct 2021 14:08:40 +0000 (10:08 -0400)]
16347: Fix leaky state in test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Merge branch 'main'
Tom Clegg [Tue, 19 Oct 2021 13:21:44 +0000 (09:21 -0400)]
16347: Merge branch 'main'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17742: update script addressing review suggestions
Javier Bértoli [Mon, 18 Oct 2021 22:47:27 +0000 (19:47 -0300)]
17742: update script addressing review suggestions

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17742: update docs addressing review suggestions
Javier Bértoli [Mon, 18 Oct 2021 22:42:43 +0000 (19:42 -0300)]
17742: update docs addressing review suggestions

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago16347: Change default to enable local keepstore when possible.
Tom Clegg [Mon, 18 Oct 2021 19:30:29 +0000 (15:30 -0400)]
16347: Change default to enable local keepstore when possible.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Remove config comment made obsolete by logging option.
Tom Clegg [Mon, 18 Oct 2021 13:25:04 +0000 (09:25 -0400)]
16347: Remove config comment made obsolete by logging option.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago18259: documentation: add more detail about Singularity.
Ward Vandewege [Fri, 15 Oct 2021 19:36:56 +0000 (15:36 -0400)]
18259: documentation: add more detail about Singularity.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoUse docker inspect instead of iterating over images list
Peter Amstutz [Wed, 13 Oct 2021 19:34:29 +0000 (15:34 -0400)]
Use docker inspect instead of iterating over images list

refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago17742: add documentation about custom certs usage
Javier Bértoli [Tue, 12 Oct 2021 00:51:27 +0000 (21:51 -0300)]
17742: add documentation about custom certs usage

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17742: add custom certs to the local.params files
Javier Bértoli [Tue, 12 Oct 2021 00:49:43 +0000 (21:49 -0300)]
17742: add custom certs to the local.params files

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17742: modify multi_host/aws example's nginx
Javier Bértoli [Mon, 11 Oct 2021 22:03:26 +0000 (19:03 -0300)]
17742: modify multi_host/aws example's nginx

to use individual certificates and allow custom certs
Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoDocumentation: silence the curl command to get our Arvados package signing key.
Ward Vandewege [Sun, 10 Oct 2021 15:30:32 +0000 (11:30 -0400)]
Documentation: silence the curl command to get our Arvados package signing key.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoRemove long dead `run` and `pipeline` subcommands from the `arv`
Ward Vandewege [Sun, 10 Oct 2021 00:29:24 +0000 (20:29 -0400)]
Remove long dead `run` and `pipeline` subcommands from the `arv`
executable.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18259-singularity-doc'
Ward Vandewege [Fri, 8 Oct 2021 21:36:09 +0000 (17:36 -0400)]
Merge branch '18259-singularity-doc'

refs #18259

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18259: mention `mksquashfs mem` config setting in singularity.conf
Ward Vandewege [Fri, 8 Oct 2021 16:49:20 +0000 (12:49 -0400)]
18259: mention `mksquashfs mem` config setting in singularity.conf

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoThe minimum supported version of Python is 3.6
Ward Vandewege [Fri, 8 Oct 2021 21:20:43 +0000 (17:20 -0400)]
The minimum supported version of Python is 3.6

refs #18261

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoFix CWL tests mixed_version_v10_wf and command_input_file_expression
Peter Amstutz [Fri, 8 Oct 2021 20:06:58 +0000 (16:06 -0400)]
Fix CWL tests mixed_version_v10_wf and command_input_file_expression

refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18258-set-singularity-cachedir'
Ward Vandewege [Fri, 8 Oct 2021 16:50:08 +0000 (12:50 -0400)]
Merge branch '18258-set-singularity-cachedir'

closes #18258

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago16347: Add LocalKeepLogsToContainerLog config (none/errors/all).
Tom Clegg [Fri, 8 Oct 2021 15:04:53 +0000 (11:04 -0400)]
16347: Add LocalKeepLogsToContainerLog config (none/errors/all).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Add test case. Flush keepstore logs in CommitLogs.
Tom Clegg [Thu, 7 Oct 2021 20:22:54 +0000 (16:22 -0400)]
16347: Add test case. Flush keepstore logs in CommitLogs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Use health check endpoint to check keepstore readiness.
Tom Clegg [Thu, 7 Oct 2021 20:21:48 +0000 (16:21 -0400)]
16347: Use health check endpoint to check keepstore readiness.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Update test case to use context instead of CloseNotifier.
Tom Clegg [Wed, 6 Oct 2021 19:33:50 +0000 (15:33 -0400)]
16347: Update test case to use context instead of CloseNotifier.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Add new config key to whitelist map.
Tom Clegg [Wed, 6 Oct 2021 18:47:19 +0000 (14:47 -0400)]
16347: Add new config key to whitelist map.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Skip journald logging if systemd-cat is not installed.
Tom Clegg [Wed, 6 Oct 2021 18:44:19 +0000 (14:44 -0400)]
16347: Skip journald logging if systemd-cat is not installed.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Remove closenotifier shim, just use request context.
Tom Clegg [Tue, 5 Oct 2021 03:25:13 +0000 (23:25 -0400)]
16347: Remove closenotifier shim, just use request context.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Fix Azure storage driver log spam.
Tom Clegg [Tue, 5 Oct 2021 03:30:23 +0000 (23:30 -0400)]
16347: Fix Azure storage driver log spam.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago16347: Run a dedicated keepstore process for each container.
Tom Clegg [Mon, 4 Oct 2021 14:55:49 +0000 (10:55 -0400)]
16347: Run a dedicated keepstore process for each container.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 years ago17742: modify single_host/multiple_hostnames example's nginx
Javier Bértoli [Fri, 8 Oct 2021 14:57:50 +0000 (11:57 -0300)]
17742: modify single_host/multiple_hostnames example's nginx

to use individual certificates

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago18258: also handle SINGULARITY_TMPDIR
Ward Vandewege [Fri, 8 Oct 2021 14:43:52 +0000 (10:43 -0400)]
18258: also handle SINGULARITY_TMPDIR

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18258: add missing error check on os.Mkdir
Ward Vandewege [Fri, 8 Oct 2021 14:38:17 +0000 (10:38 -0400)]
18258: add missing error check on os.Mkdir

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18258: set the SINGULARITY_CACHEDIR to a path underneath our temporary
Ward Vandewege [Fri, 8 Oct 2021 14:24:28 +0000 (10:24 -0400)]
18258: set the SINGULARITY_CACHEDIR to a path underneath our temporary
       space, and clean up after we're done building.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoarvbox fix, keepstore is part of arvados-server now
Peter Amstutz [Thu, 7 Oct 2021 18:01:15 +0000 (14:01 -0400)]
arvbox fix, keepstore is part of arvados-server now

refs #18216

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18113-change-cloudops-defaults'
Ward Vandewege [Thu, 7 Oct 2021 17:26:29 +0000 (13:26 -0400)]
Merge branch '18113-change-cloudops-defaults'

closes #18113

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18113: Merge branch 'main' into 18113-change-cloudops-defaults
Ward Vandewege [Thu, 7 Oct 2021 17:26:12 +0000 (13:26 -0400)]
18113: Merge branch 'main' into 18113-change-cloudops-defaults

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago18113: change MaxCloudOpsPerSecond and MaxConcurrentInstanceCreateOps
Ward Vandewege [Wed, 29 Sep 2021 16:08:17 +0000 (12:08 -0400)]
18113: change MaxCloudOpsPerSecond and MaxConcurrentInstanceCreateOps
       defaults.

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years ago17742: add state to install custom certificates
Javier Bértoli [Wed, 29 Sep 2021 15:28:08 +0000 (12:28 -0300)]
17742: add state to install custom certificates

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years ago17742: modify snakeoil_certs to create individual certificates
Javier Bértoli [Wed, 29 Sep 2021 15:21:26 +0000 (12:21 -0300)]
17742: modify snakeoil_certs to create individual certificates

Arvados-DCO-1.1-Signed-off-by: Javier Bértoli <jbertoli@curii.com>

3 years agoRemove the failing pre-build (--only-deps) step for crunch
Peter Amstutz [Thu, 7 Oct 2021 14:52:30 +0000 (10:52 -0400)]
Remove the failing pre-build (--only-deps) step for crunch

It will just do what it needs to when the container starts.

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoRemove the 'experimental' label in our config reference from things that
Ward Vandewege [Thu, 7 Oct 2021 14:14:23 +0000 (10:14 -0400)]
Remove the 'experimental' label in our config reference from things that
are not (PAM login support, a-d-c, Singularity support). Update the docs
for our LSF and Singularity support along the same lines, and update
formatting a bit to make the current limitations of the implementation
clear.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoWe support Singularity 3.7.4.
Ward Vandewege [Thu, 7 Oct 2021 13:57:02 +0000 (09:57 -0400)]
We support Singularity 3.7.4.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <ward@curii.com>

3 years agoMerge branch '18238-singularity-home' refs #18238
Peter Amstutz [Thu, 7 Oct 2021 14:08:43 +0000 (10:08 -0400)]
Merge branch '18238-singularity-home' refs #18238

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years ago18238: Another try at Singularity $HOME fix
Peter Amstutz [Thu, 7 Oct 2021 14:04:48 +0000 (10:04 -0400)]
18238: Another try at Singularity $HOME fix

Use just the --home parameter, which appears to have option of
behaving as a special case of --bind, and apparently (according to
cwltool experience) works across more Singularity versions.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 years agoMerge branch '18216-arvados-server-keepstore'
Tom Clegg [Wed, 6 Oct 2021 20:41:17 +0000 (16:41 -0400)]
Merge branch '18216-arvados-server-keepstore'

closes #18216

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>