arvados.git
17 months ago20862: Merge branch 'main' into 20862-google-api-client
Tom Clegg [Tue, 29 Aug 2023 17:32:16 +0000 (13:32 -0400)]
20862: Merge branch 'main' into 20862-google-api-client

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20839-pysdk-deprecations'
Brett Smith [Tue, 29 Aug 2023 15:41:10 +0000 (11:41 -0400)]
Merge branch '20839-pysdk-deprecations'

Closes #20839.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20862: Re-run bundle after deleting bogus locally cached gem files.
Tom Clegg [Tue, 29 Aug 2023 15:25:57 +0000 (11:25 -0400)]
20862: Re-run bundle after deleting bogus locally cached gem files.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20757-ctr-shell-settings'
Tom Clegg [Tue, 29 Aug 2023 13:16:00 +0000 (09:16 -0400)]
Merge branch '20757-ctr-shell-settings'

fixes #20757

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20836: Update test_select for Py<3.8 compatibility 20839-pysdk-deprecations
Brett Smith [Mon, 28 Aug 2023 20:26:41 +0000 (16:26 -0400)]
20836: Update test_select for Py<3.8 compatibility

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20839: Mark deprecated PySDK functions as such
Brett Smith [Sun, 27 Aug 2023 02:13:39 +0000 (22:13 -0400)]
20839: Mark deprecated PySDK functions as such

This is in preparation to remove these from the SDK in Arvados 3.0.

These changes exacerbated the circular import between the top-level
arvados and util, which had knock-off effects on other import
statements, causing them to see the world differently (like whether
`arvados.api` is a module or function). Most of the import-related
shuffling is meant to mitigate all that.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20839: Add order_key to select in keyset_list_all
Brett Smith [Sun, 27 Aug 2023 03:59:21 +0000 (23:59 -0400)]
20839: Add order_key to select in keyset_list_all

Like uuid, the main loop requires this field.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20839: Replace arvados.util.mkdir_dash_p calls throughout
Brett Smith [Sun, 27 Aug 2023 02:52:13 +0000 (22:52 -0400)]
20839: Replace arvados.util.mkdir_dash_p calls throughout

In preparation to deprecate the function.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20839: Replace arvados.util.list_all calls throughout
Brett Smith [Sun, 27 Aug 2023 02:01:32 +0000 (22:01 -0400)]
20839: Replace arvados.util.list_all calls throughout

In preparation to deprecate this function. This does leave a couple
calls in arv-docker-migrate19, but that command will also be deprecated
and removed at least as soon as the function.

The changes to the the sdk/cwl/tests/test_submit.py mocks are necessary
because of mechanical differences between list_all and keyset_list_all.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

test fixup

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20757: Split tests into separate funcs. 20757-ctr-shell-settings
Tom Clegg [Mon, 28 Aug 2023 18:40:05 +0000 (14:40 -0400)]
20757: Split tests into separate funcs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20839: Optimize list sort
Brett Smith [Sun, 27 Aug 2023 01:55:39 +0000 (21:55 -0400)]
20839: Optimize list sort

Trivial optimization I noticed while working around here.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20839: Replace arvados.util.run_command calls in arv-copy
Brett Smith [Sun, 27 Aug 2023 01:42:20 +0000 (21:42 -0400)]
20839: Replace arvados.util.run_command calls in arv-copy

In preparation for deprecating this function.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20839: Remove old Python 2 TextIOWrapper shim
Brett Smith [Sat, 26 Aug 2023 23:32:53 +0000 (19:32 -0400)]
20839: Remove old Python 2 TextIOWrapper shim

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20757: `arvados-client shell` uses settings.conf if present.
Tom Clegg [Fri, 25 Aug 2023 16:28:56 +0000 (12:28 -0400)]
20757: `arvados-client shell` uses settings.conf if present.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20877-trashed-priority' refs #20877
Peter Amstutz [Fri, 25 Aug 2023 14:49:33 +0000 (10:49 -0400)]
Merge branch '20877-trashed-priority' refs #20877

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20894-instances-default' refs #20894
Peter Amstutz [Fri, 25 Aug 2023 14:04:25 +0000 (10:04 -0400)]
Merge branch '20894-instances-default' refs #20894

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20862: Update arvados gem version.
Tom Clegg [Thu, 24 Aug 2023 16:22:09 +0000 (12:22 -0400)]
20862: Update arvados gem version.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Update mocha gem.
Tom Clegg [Thu, 24 Aug 2023 14:25:49 +0000 (10:25 -0400)]
20862: Update mocha gem.

Previous versions are not compatible with new minitest.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoRevert "20894: Change default retention on prometheus to 90 days" 20894-instances-default
Peter Amstutz [Thu, 24 Aug 2023 14:10:35 +0000 (10:10 -0400)]
Revert "20894: Change default retention on prometheus to 90 days"

This reverts commit 15e554b5da9d4da9447e547abd536c53be069a21.

According to Lucas, this doesn't actually work.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20894: Change default retention on prometheus to 90 days
Peter Amstutz [Thu, 24 Aug 2023 14:06:41 +0000 (10:06 -0400)]
20894: Change default retention on prometheus to 90 days

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20894: Set LimitLogBytesPerJob to 0
Peter Amstutz [Wed, 23 Aug 2023 18:58:43 +0000 (14:58 -0400)]
20894: Set LimitLogBytesPerJob to 0

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20862: Add test stub func for sdk/ruby-google-api-client.
Tom Clegg [Wed, 23 Aug 2023 18:34:52 +0000 (14:34 -0400)]
20862: Add test stub func for sdk/ruby-google-api-client.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20894: Wordsmithing
Peter Amstutz [Wed, 23 Aug 2023 17:24:10 +0000 (13:24 -0400)]
20894: Wordsmithing

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20894: Change InitialQuotaEstimate and SupervisorFraction defaults.
Peter Amstutz [Wed, 23 Aug 2023 17:08:45 +0000 (13:08 -0400)]
20894: Change InitialQuotaEstimate and SupervisorFraction defaults.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20862: Improve exception messages.
Tom Clegg [Wed, 23 Aug 2023 14:41:58 +0000 (10:41 -0400)]
20862: Improve exception messages.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Update arvados gem version.
Tom Clegg [Wed, 23 Aug 2023 14:15:02 +0000 (10:15 -0400)]
20862: Update arvados gem version.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Update components to latest arvados, google-api-client gems.
Tom Clegg [Tue, 22 Aug 2023 15:26:22 +0000 (11:26 -0400)]
20862: Update components to latest arvados, google-api-client gems.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Fix ruby sdk deps.
Tom Clegg [Tue, 22 Aug 2023 15:16:29 +0000 (11:16 -0400)]
20862: Fix ruby sdk deps.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Fix charset middleware.
Tom Clegg [Tue, 22 Aug 2023 15:14:24 +0000 (11:14 -0400)]
20862: Fix charset middleware.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Fix faraday/gzip middleware.
Tom Clegg [Tue, 22 Aug 2023 15:03:13 +0000 (11:03 -0400)]
20862: Fix faraday/gzip middleware.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Fix ruby sdk dependencies.
Tom Clegg [Tue, 22 Aug 2023 14:55:20 +0000 (10:55 -0400)]
20862: Fix ruby sdk dependencies.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Fix arvados-google-api-client dependencies.
Tom Clegg [Tue, 22 Aug 2023 14:51:48 +0000 (10:51 -0400)]
20862: Fix arvados-google-api-client dependencies.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Build arvados-google-api-client gem in run-tests.sh.
Tom Clegg [Fri, 18 Aug 2023 15:24:13 +0000 (11:24 -0400)]
20862: Build arvados-google-api-client gem in run-tests.sh.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Disable license header check for ruby-google-api-client.
Tom Clegg [Tue, 22 Aug 2023 14:05:53 +0000 (10:05 -0400)]
20862: Disable license header check for ruby-google-api-client.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20877: Fix missing 'and'
Peter Amstutz [Wed, 23 Aug 2023 13:31:47 +0000 (09:31 -0400)]
20877: Fix missing 'and'

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20862: Add 'sdk/ruby-google-api-client/' from commit '2f4be67955e48bb65d008ecd9ff6da9...
Tom Clegg [Wed, 23 Aug 2023 13:31:48 +0000 (09:31 -0400)]
20862: Add 'sdk/ruby-google-api-client/' from commit '2f4be67955e48bb65d008ecd9ff6da9d8fe77028'

git-subtree-dir: sdk/ruby-google-api-client
git-subtree-mainline: 6e91ab41ffa1509dd1d4e2b8ecb559dd429c6ea4
git-subtree-split: 2f4be67955e48bb65d008ecd9ff6da9d8fe77028

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20877: Don't retry trashed container requests
Peter Amstutz [Mon, 21 Aug 2023 20:56:22 +0000 (16:56 -0400)]
20877: Don't retry trashed container requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20877: When trashing a project, update priorities on container requests
Peter Amstutz [Mon, 21 Aug 2023 20:42:39 +0000 (16:42 -0400)]
20877: When trashing a project, update priorities on container requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20877: Add test for trashing project
Peter Amstutz [Mon, 21 Aug 2023 18:02:23 +0000 (14:02 -0400)]
20877: Add test for trashing project

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20853-pdoc'
Brett Smith [Mon, 21 Aug 2023 16:15:50 +0000 (12:15 -0400)]
Merge branch '20853-pdoc'

Closes #20853.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoAdd pdoc to arvbox, refs #20853
Peter Amstutz [Fri, 18 Aug 2023 20:41:19 +0000 (16:41 -0400)]
Add pdoc to arvbox, refs #20853

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20862: Update apps to latest ruby SDK.
Tom Clegg [Fri, 18 Aug 2023 20:09:07 +0000 (16:09 -0400)]
20862: Update apps to latest ruby SDK.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Update ruby and cli gem dependencies.
Tom Clegg [Fri, 18 Aug 2023 17:21:27 +0000 (13:21 -0400)]
20862: Update ruby and cli gem dependencies.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Fix use of deprecated stdlib.
Tom Clegg [Fri, 18 Aug 2023 17:16:06 +0000 (13:16 -0400)]
20862: Fix use of deprecated stdlib.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Update dependencies, disable some Faraday usage.
Tom Clegg [Fri, 18 Aug 2023 19:50:36 +0000 (15:50 -0400)]
20862: Update dependencies, disable some Faraday usage.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20862: Update gemspec to reflect forked gem name.
Tom Clegg [Fri, 18 Aug 2023 19:36:14 +0000 (15:36 -0400)]
20862: Update gemspec to reflect forked gem name.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20878-rh-package-download'
Brett Smith [Fri, 18 Aug 2023 13:28:43 +0000 (09:28 -0400)]
Merge branch '20878-rh-package-download'

Closes #20878.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20878: Document package download change to add a target distribution 20878-rh-package-download
Brett Smith [Thu, 17 Aug 2023 23:53:33 +0000 (19:53 -0400)]
20878: Document package download change to add a target distribution

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20878: Improve processed RPM check style
Brett Smith [Thu, 17 Aug 2023 22:28:17 +0000 (18:28 -0400)]
20878: Improve processed RPM check style

This is nicer bash style that makes it more consistent with the
preceding download code.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20878: Improve RPM download logic
Brett Smith [Thu, 17 Aug 2023 22:27:12 +0000 (18:27 -0400)]
20878: Improve RPM download logic

Instead of downloading and parsing a package file listing, just try to
download the package directly, and check whether that succeeds.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20878: Parameterize URL for RPM package builds
Brett Smith [Thu, 17 Aug 2023 22:20:37 +0000 (18:20 -0400)]
20878: Parameterize URL for RPM package builds

* Add support for rocky8.
* Prevent future errors by refusing to attempt a download for unknown
  targets.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20858-jsonb-exists' refs #20858
Peter Amstutz [Wed, 16 Aug 2023 18:54:58 +0000 (14:54 -0400)]
Merge branch '20858-jsonb-exists' refs #20858

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20735-install-test'
Tom Clegg [Wed, 16 Aug 2023 13:32:32 +0000 (09:32 -0400)]
Merge branch '20735-install-test'

refs #20735

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Workaround nginx/glibc/docker issue.
Tom Clegg [Wed, 16 Aug 2023 13:04:18 +0000 (09:04 -0400)]
20735: Workaround nginx/glibc/docker issue.

Docker writes both "::1 localhost" and "127.0.0.1 localhost" in
/etc/hosts even when not enabling IPv6 [1], causing glibc to resolve
"localhost" to two identical IP addresses 127.0.0.1 and 127.0.0.1 [2],
causing nginx to error out on "listen localhost:45123;" [3]:

[emerg] a duplicate listen 127.0.0.1:45123 in /tmp/nginx.conf:29

[1] https://github.com/moby/moby/issues/35954#issuecomment-498406618
("you should be testing with IPv6 enabled")

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969 (open since
2012-12-17)

[2] https://trac.nginx.org/nginx/ticket/2400 (workaround added
2022-11-23)

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20680-cores-concurrency' refs #20680
Peter Amstutz [Tue, 15 Aug 2023 20:52:41 +0000 (16:52 -0400)]
Merge branch '20680-cores-concurrency' refs #20680

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20858: Enable 'exists' filter to use index 20858-jsonb-exists
Peter Amstutz [Tue, 15 Aug 2023 20:41:01 +0000 (16:41 -0400)]
20858: Enable 'exists' filter to use index

See comment in services/api/db/migrate/20230815160000_jsonb_exists_functions.rb

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20735: Fix default workbench2-source path.
Tom Clegg [Fri, 11 Aug 2023 13:47:10 +0000 (09:47 -0400)]
20735: Fix default workbench2-source path.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Override git directory ownership check.
Tom Clegg [Thu, 10 Aug 2023 21:18:27 +0000 (17:18 -0400)]
20735: Override git directory ownership check.

Without this, `git diff --shortstat` returns an error "warning: Not a
git repository."

The real problem, as reported by `git status`, is "fatal: detected
dubious ownership in repository at '/path'".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Fix auto version generator.
Tom Clegg [Thu, 10 Aug 2023 20:41:49 +0000 (16:41 -0400)]
20735: Fix auto version generator.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Revive lib/install dependency test.
Tom Clegg [Thu, 10 Aug 2023 19:22:55 +0000 (15:22 -0400)]
20735: Revive lib/install dependency test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Fix firefox package name on debian 11.
Tom Clegg [Thu, 10 Aug 2023 19:21:52 +0000 (15:21 -0400)]
20735: Fix firefox package name on debian 11.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Fixup singularity dependencies (test/dev-only).
Tom Clegg [Thu, 10 Aug 2023 19:20:15 +0000 (15:20 -0400)]
20735: Fixup singularity dependencies (test/dev-only).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '17244-cgroup2-cpu-max'
Tom Clegg [Tue, 15 Aug 2023 17:41:58 +0000 (13:41 -0400)]
Merge branch '17244-cgroup2-cpu-max'

fixes #17244

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Add upgrade note re number formatting change.
Tom Clegg [Tue, 15 Aug 2023 17:40:18 +0000 (13:40 -0400)]
17244: Add upgrade note re number formatting change.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoFurther fixes on single host installer deployment. No issue #
Lucas Di Pentima [Tue, 15 Aug 2023 17:39:44 +0000 (14:39 -0300)]
Further fixes on single host installer deployment. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months agoFixes deployment when using envvar BRANCH=main. No issue #
Lucas Di Pentima [Tue, 15 Aug 2023 13:53:05 +0000 (10:53 -0300)]
Fixes deployment when using envvar BRANCH=main. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20853: Modernize pdoc installation process 20853-pdoc
Brett Smith [Mon, 14 Aug 2023 21:04:10 +0000 (17:04 -0400)]
20853: Modernize pdoc installation process

Install to a virtualenv to accommodate PEP 668.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20853: Switch from pdoc3 to pdoc
Brett Smith [Mon, 14 Aug 2023 21:00:57 +0000 (17:00 -0400)]
20853: Switch from pdoc3 to pdoc

pdoc is better maintained, both in terms of release cadence and in terms
of community code of conduct.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago17244: New debian10 testdata, add cpu.max and cpuinfo for others. 17244-cgroup2-cpu-max
Tom Clegg [Mon, 14 Aug 2023 19:40:36 +0000 (15:40 -0400)]
17244: New debian10 testdata, add cpu.max and cpuinfo for others.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Use cpu.max to get available fraction of CPUs.
Tom Clegg [Mon, 14 Aug 2023 19:09:27 +0000 (15:09 -0400)]
17244: Use cpu.max to get available fraction of CPUs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20649-ssh-help'
Tom Clegg [Mon, 14 Aug 2023 20:29:48 +0000 (16:29 -0400)]
Merge branch '20649-ssh-help'

fixes #20649

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20457-careful-near-quota'
Tom Clegg [Mon, 14 Aug 2023 20:29:09 +0000 (16:29 -0400)]
Merge branch '20457-careful-near-quota'

fixes #20457

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20680: Set concurrency to num_cpus * 2 with a minimum of 8 20680-cores-concurrency
Peter Amstutz [Mon, 14 Aug 2023 18:41:39 +0000 (14:41 -0400)]
20680: Set concurrency to num_cpus * 2 with a minimum of 8

In practice it makes sense to have two workers per core because
workers usually spend at least half their time waiting on the
database.

Setting a minimum concurrency of 8 avoids a deadlock situation when
controller calls keep-web, which calls back to controller.

This doesn't completely eliminate the problem, but it at least avoids
having deadlocked requests with a single user in the default
configuration (!).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20649: Mention `arvados-server cloudtest` in boot timeout log msg. 20649-ssh-help
Tom Clegg [Mon, 14 Aug 2023 15:14:24 +0000 (11:14 -0400)]
20649: Mention `arvados-server cloudtest` in boot timeout log msg.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20649: Fix panic on race, worker shutdown vs. container startup.
Tom Clegg [Mon, 14 Aug 2023 15:11:51 +0000 (11:11 -0400)]
20649: Fix panic on race, worker shutdown vs. container startup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20649: Don't wait timeoutSignal before sending first TERM signal.
Tom Clegg [Mon, 14 Aug 2023 15:10:44 +0000 (11:10 -0400)]
20649: Don't wait timeoutSignal before sending first TERM signal.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoFixes comment formatting. No issue #
Lucas Di Pentima [Mon, 14 Aug 2023 14:07:19 +0000 (11:07 -0300)]
Fixes comment formatting. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20457: Fix flaky test.
Tom Clegg [Wed, 9 Aug 2023 21:07:35 +0000 (17:07 -0400)]
20457: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20649: Log actual error from last failed boot probe.
Tom Clegg [Fri, 11 Aug 2023 21:47:32 +0000 (17:47 -0400)]
20649: Log actual error from last failed boot probe.

Not just the stderr content, which is useless if the error was
something like "connection failed".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20649: Log boot probe errors other than usual wait-for-boot errors.
Tom Clegg [Fri, 11 Aug 2023 21:47:06 +0000 (17:47 -0400)]
20649: Log boot probe errors other than usual wait-for-boot errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20457: Update test to match new unlock-at-quota behavior. 20457-careful-near-quota
Tom Clegg [Fri, 11 Aug 2023 17:31:29 +0000 (13:31 -0400)]
20457: Update test to match new unlock-at-quota behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20457: Don't lock-cycle next-in-line ctr while waiting for quota.
Tom Clegg [Fri, 11 Aug 2023 14:17:30 +0000 (10:17 -0400)]
20457: Don't lock-cycle next-in-line ctr while waiting for quota.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20457: Fix total instances count.
Tom Clegg [Fri, 11 Aug 2023 14:12:25 +0000 (10:12 -0400)]
20457: Fix total instances count.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoAdd packages required to build newer singularity, refs #20735
Peter Amstutz [Thu, 10 Aug 2023 18:24:53 +0000 (14:24 -0400)]
Add packages required to build newer singularity, refs #20735

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20688-wb1-to-wb2-redirects' refs #20688
Peter Amstutz [Thu, 10 Aug 2023 17:11:30 +0000 (13:11 -0400)]
Merge branch '20688-wb1-to-wb2-redirects' refs #20688

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20649: cloudtest obeys DeployPublicKey config, like a-d-c.
Tom Clegg [Thu, 10 Aug 2023 14:45:46 +0000 (10:45 -0400)]
20649: cloudtest obeys DeployPublicKey config, like a-d-c.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20755-ec2-multiple-subnets'
Tom Clegg [Thu, 10 Aug 2023 14:34:11 +0000 (10:34 -0400)]
Merge branch '20755-ec2-multiple-subnets'

closes #20755

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20457: Fix flaky test.
Tom Clegg [Wed, 9 Aug 2023 21:07:35 +0000 (17:07 -0400)]
20457: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20457: Exercise quota handling in dispatcher chaos test.
Tom Clegg [Wed, 9 Aug 2023 21:06:40 +0000 (17:06 -0400)]
20457: Exercise quota handling in dispatcher chaos test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20457: When passing prior quota level, raise maxConcurrency slowly.
Tom Clegg [Mon, 7 Aug 2023 19:26:20 +0000 (15:26 -0400)]
20457: When passing prior quota level, raise maxConcurrency slowly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20680-rolling-deploy' refs #20680
Peter Amstutz [Wed, 9 Aug 2023 13:53:48 +0000 (09:53 -0400)]
Merge branch '20680-rolling-deploy' refs #20680

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20680: Increase MaxConcurrentRequests to 8 20680-rolling-deploy
Peter Amstutz [Wed, 9 Aug 2023 13:48:51 +0000 (09:48 -0400)]
20680: Increase MaxConcurrentRequests to 8

Necessary to avoid a API -> keep-web -> API loop causing a deadlock.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20755: Merge branch 'main' into 20755-ec2-multiple-subnets 20755-ec2-multiple-subnets
Tom Clegg [Wed, 9 Aug 2023 13:33:34 +0000 (09:33 -0400)]
20755: Merge branch 'main' into 20755-ec2-multiple-subnets

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20755: Fix test case by accepting empty DriverParameters.
Tom Clegg [Wed, 9 Aug 2023 13:32:37 +0000 (09:32 -0400)]
20755: Fix test case by accepting empty DriverParameters.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20755: Run cloudtest suite once per configured subnet, not just 1st.
Tom Clegg [Tue, 8 Aug 2023 21:31:24 +0000 (17:31 -0400)]
20755: Run cloudtest suite once per configured subnet, not just 1st.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20735-update-deps'
Tom Clegg [Tue, 8 Aug 2023 18:52:42 +0000 (14:52 -0400)]
Merge branch '20735-update-deps'

refs #20735

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20735: Clarify singularity version/compatibility expectations.
Tom Clegg [Tue, 8 Aug 2023 18:51:21 +0000 (14:51 -0400)]
20735: Clarify singularity version/compatibility expectations.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20680: Only do a rolling update when there is multiple controllers
Peter Amstutz [Tue, 8 Aug 2023 17:42:25 +0000 (13:42 -0400)]
20680: Only do a rolling update when there is multiple controllers

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoFix error check.
Tom Clegg [Tue, 8 Aug 2023 14:18:29 +0000 (10:18 -0400)]
Fix error check.

refs #17244

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20680: installer does rolling update of controller
Peter Amstutz [Tue, 8 Aug 2023 13:36:15 +0000 (09:36 -0400)]
20680: installer does rolling update of controller

When there is a load balancer, disable each controller node in turn
before updating it.

Also, don't reduce controller_nr when a controller is down, having
worker_connections go up and down during deploy seems like in could
cause problems.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>