arvados.git
10 months ago20839: Replace arvados.util.run_command calls in arv-copy
Brett Smith [Sun, 27 Aug 2023 01:42:20 +0000 (21:42 -0400)]
20839: Replace arvados.util.run_command calls in arv-copy

In preparation for deprecating this function.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20839: Remove old Python 2 TextIOWrapper shim
Brett Smith [Sat, 26 Aug 2023 23:32:53 +0000 (19:32 -0400)]
20839: Remove old Python 2 TextIOWrapper shim

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20889: Fixes Keep S3 bucket instructions. 20889-installer-fixes
Lucas Di Pentima [Fri, 25 Aug 2023 19:20:08 +0000 (16:20 -0300)]
20889: Fixes Keep S3 bucket instructions.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Adds documentation explaining the use of TLS cert keys encryption.
Lucas Di Pentima [Fri, 25 Aug 2023 19:13:22 +0000 (16:13 -0300)]
20889: Adds documentation explaining the use of TLS cert keys encryption.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Fixes installer.sh indentation to match the how provision.sh is written.
Lucas Di Pentima [Fri, 25 Aug 2023 18:30:04 +0000 (15:30 -0300)]
20889: Fixes installer.sh indentation to match the how provision.sh is written.

It had a mix of space-based and tab-based indentation that made code a bit
difficult to read.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Checks that cert files are present before trying to use them.
Lucas Di Pentima [Fri, 25 Aug 2023 18:22:56 +0000 (15:22 -0300)]
20889: Checks that cert files are present before trying to use them.

Also, some documentation fixes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20757: `arvados-client shell` uses settings.conf if present.
Tom Clegg [Fri, 25 Aug 2023 16:28:56 +0000 (12:28 -0400)]
20757: `arvados-client shell` uses settings.conf if present.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20877-trashed-priority' refs #20877
Peter Amstutz [Fri, 25 Aug 2023 14:49:33 +0000 (10:49 -0400)]
Merge branch '20877-trashed-priority' refs #20877

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months agoMerge branch '20894-instances-default' refs #20894
Peter Amstutz [Fri, 25 Aug 2023 14:04:25 +0000 (10:04 -0400)]
Merge branch '20894-instances-default' refs #20894

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20889: Fixes a bug when database & controller roles are on a single node.
Lucas Di Pentima [Fri, 25 Aug 2023 13:30:11 +0000 (10:30 -0300)]
20889: Fixes a bug when database & controller roles are on a single node.

The database role gets handled first, so the controller node gets removed
from the list and then when checking if multiple nodes were available, the
code path for a single controller didn't checked if the node was already
removed from the node list.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Allows customization on database name and user.
Lucas Di Pentima [Thu, 24 Aug 2023 21:47:26 +0000 (18:47 -0300)]
20889: Allows customization on database name and user.

Also, allows specifying external PG service for the single host cases.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Allows configurable S3 bucket name and IAM profile for Keep backend.
Lucas Di Pentima [Thu, 24 Aug 2023 21:05:10 +0000 (18:05 -0300)]
20889: Allows configurable S3 bucket name and IAM profile for Keep backend.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Adds config handling of external database service.
Lucas Di Pentima [Thu, 24 Aug 2023 20:56:05 +0000 (17:56 -0300)]
20889: Adds config handling of external database service.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Removes unnecessary documentation about rolling upgrades.
Lucas Di Pentima [Thu, 24 Aug 2023 20:35:39 +0000 (17:35 -0300)]
20889: Removes unnecessary documentation about rolling upgrades.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20889: Removes DISABLED_CONTROLLER from the user-editable local.params file.
Lucas Di Pentima [Thu, 24 Aug 2023 19:31:45 +0000 (16:31 -0300)]
20889: Removes DISABLED_CONTROLLER from the user-editable local.params file.

As the rolling upgrade procedure is done automatically, this is not needed
and could cause confusion by being on local.params

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20750: Fix documentation link typo 20750-sharing-token-scopes
Brett Smith [Thu, 24 Aug 2023 19:11:32 +0000 (15:11 -0400)]
20750: Fix documentation link typo

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20889: Sets default values for environment variables.
Lucas Di Pentima [Thu, 24 Aug 2023 19:07:41 +0000 (16:07 -0300)]
20889: Sets default values for environment variables.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20750: Reorder remote token validation strategy
Brett Smith [Wed, 23 Aug 2023 01:47:33 +0000 (21:47 -0400)]
20750: Reorder remote token validation strategy

See updated comments for an explanation of why we do queries in the
order that we do. Briefly, now that getting the current API client
authorization is no longer subject to token scopes, that query has the
best chance of telling us immediately whether or not the token is
valid. If that query succeeds, but the current user query fails, we
might still be able to load a user record from the database from the
token's owner_uuid, and use that.

The immediate application for this change is sharing tokens with limited
scopes. When the user creates such a token, we expect they're doing so
using a token with unlimited scopes, and their user record will have
been loaded into the database at that time. Then visitors using the
sharing token will be able to authorize from that database record,
without having permission to query the full user record (and see the
creator's name and email address) directly.

This may also provide better behavior in cases of network hiccups or
other remote server turbulence. Now only one query *needs* to succeed,
the current token API call.

With these changes, it seemed appropriate to propagate any remote error
status codes back to the client, so they can also distinguish permanent
authorization failures from temporary ones. To do that, we attach an
`http_status` singleton method to the original
HTTPClient::BadResponseError. ApplicationController#render_error already
uses that method to determine the status code to return to the client,
and this commit extends CurrentApiToken with similar behavior. Before
this commit, there were definitely other cases that could raise
unhandled exceptions; I assume they were previously all returning
generic error codes, and probably untested. I think this is improved
behavior that makes sense with the rest of the change.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20889: Adds configurable data retention parameter for Prometheus.
Lucas Di Pentima [Thu, 24 Aug 2023 18:36:27 +0000 (15:36 -0300)]
20889: Adds configurable data retention parameter for Prometheus.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20862: Update arvados gem version.
Tom Clegg [Thu, 24 Aug 2023 16:22:09 +0000 (12:22 -0400)]
20862: Update arvados gem version.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Update mocha gem.
Tom Clegg [Thu, 24 Aug 2023 14:25:49 +0000 (10:25 -0400)]
20862: Update mocha gem.

Previous versions are not compatible with new minitest.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoRevert "20894: Change default retention on prometheus to 90 days" 20894-instances-default
Peter Amstutz [Thu, 24 Aug 2023 14:10:35 +0000 (10:10 -0400)]
Revert "20894: Change default retention on prometheus to 90 days"

This reverts commit 15e554b5da9d4da9447e547abd536c53be069a21.

According to Lucas, this doesn't actually work.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20894: Change default retention on prometheus to 90 days
Peter Amstutz [Thu, 24 Aug 2023 14:06:41 +0000 (10:06 -0400)]
20894: Change default retention on prometheus to 90 days

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20889: Install Prometheus server from archives instead of package repo.
Lucas Di Pentima [Wed, 23 Aug 2023 20:27:21 +0000 (17:27 -0300)]
20889: Install Prometheus server from archives instead of package repo.

This avoids some incompatibilities (at least on Ubuntu 20.04) with grafana.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20894: Set LimitLogBytesPerJob to 0
Peter Amstutz [Wed, 23 Aug 2023 18:58:43 +0000 (14:58 -0400)]
20894: Set LimitLogBytesPerJob to 0

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20862: Add test stub func for sdk/ruby-google-api-client.
Tom Clegg [Wed, 23 Aug 2023 18:34:52 +0000 (14:34 -0400)]
20862: Add test stub func for sdk/ruby-google-api-client.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20894: Wordsmithing
Peter Amstutz [Wed, 23 Aug 2023 17:24:10 +0000 (13:24 -0400)]
20894: Wordsmithing

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20894: Change InitialQuotaEstimate and SupervisorFraction defaults.
Peter Amstutz [Wed, 23 Aug 2023 17:08:45 +0000 (13:08 -0400)]
20894: Change InitialQuotaEstimate and SupervisorFraction defaults.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20862: Improve exception messages.
Tom Clegg [Wed, 23 Aug 2023 14:41:58 +0000 (10:41 -0400)]
20862: Improve exception messages.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Update arvados gem version.
Tom Clegg [Wed, 23 Aug 2023 14:15:02 +0000 (10:15 -0400)]
20862: Update arvados gem version.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Update components to latest arvados, google-api-client gems.
Tom Clegg [Tue, 22 Aug 2023 15:26:22 +0000 (11:26 -0400)]
20862: Update components to latest arvados, google-api-client gems.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Fix ruby sdk deps.
Tom Clegg [Tue, 22 Aug 2023 15:16:29 +0000 (11:16 -0400)]
20862: Fix ruby sdk deps.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Fix charset middleware.
Tom Clegg [Tue, 22 Aug 2023 15:14:24 +0000 (11:14 -0400)]
20862: Fix charset middleware.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Fix faraday/gzip middleware.
Tom Clegg [Tue, 22 Aug 2023 15:03:13 +0000 (11:03 -0400)]
20862: Fix faraday/gzip middleware.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Fix ruby sdk dependencies.
Tom Clegg [Tue, 22 Aug 2023 14:55:20 +0000 (10:55 -0400)]
20862: Fix ruby sdk dependencies.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Fix arvados-google-api-client dependencies.
Tom Clegg [Tue, 22 Aug 2023 14:51:48 +0000 (10:51 -0400)]
20862: Fix arvados-google-api-client dependencies.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Build arvados-google-api-client gem in run-tests.sh.
Tom Clegg [Fri, 18 Aug 2023 15:24:13 +0000 (11:24 -0400)]
20862: Build arvados-google-api-client gem in run-tests.sh.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Disable license header check for ruby-google-api-client.
Tom Clegg [Tue, 22 Aug 2023 14:05:53 +0000 (10:05 -0400)]
20862: Disable license header check for ruby-google-api-client.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20877: Fix missing 'and'
Peter Amstutz [Wed, 23 Aug 2023 13:31:47 +0000 (09:31 -0400)]
20877: Fix missing 'and'

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20862: Add 'sdk/ruby-google-api-client/' from commit '2f4be67955e48bb65d008ecd9ff6da9...
Tom Clegg [Wed, 23 Aug 2023 13:31:48 +0000 (09:31 -0400)]
20862: Add 'sdk/ruby-google-api-client/' from commit '2f4be67955e48bb65d008ecd9ff6da9d8fe77028'

git-subtree-dir: sdk/ruby-google-api-client
git-subtree-mainline: 6e91ab41ffa1509dd1d4e2b8ecb559dd429c6ea4
git-subtree-split: 2f4be67955e48bb65d008ecd9ff6da9d8fe77028

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20750: DRY uncache token test code
Brett Smith [Tue, 22 Aug 2023 15:37:45 +0000 (11:37 -0400)]
20750: DRY uncache token test code

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20877: Don't retry trashed container requests
Peter Amstutz [Mon, 21 Aug 2023 20:56:22 +0000 (16:56 -0400)]
20877: Don't retry trashed container requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20877: When trashing a project, update priorities on container requests
Peter Amstutz [Mon, 21 Aug 2023 20:42:39 +0000 (16:42 -0400)]
20877: When trashing a project, update priorities on container requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20750: Valid tokens can always GET the current token
Brett Smith [Fri, 11 Aug 2023 20:07:10 +0000 (16:07 -0400)]
20750: Valid tokens can always GET the current token

This is required for clusters in a federation to be able to validate
tokens issued by other clusters. Rather than requiring all users to
specify this manually (and fail cryptically when they don't), simply
grant the permission to all tokens.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20750: Document the api_client_authorizations current API endpoint
Brett Smith [Fri, 11 Aug 2023 20:16:49 +0000 (16:16 -0400)]
20750: Document the api_client_authorizations current API endpoint

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20877: Add test for trashing project
Peter Amstutz [Mon, 21 Aug 2023 18:02:23 +0000 (14:02 -0400)]
20877: Add test for trashing project

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months agoMerge branch '20853-pdoc'
Brett Smith [Mon, 21 Aug 2023 16:15:50 +0000 (12:15 -0400)]
Merge branch '20853-pdoc'

Closes #20853.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months agoAdd pdoc to arvbox, refs #20853
Peter Amstutz [Fri, 18 Aug 2023 20:41:19 +0000 (16:41 -0400)]
Add pdoc to arvbox, refs #20853

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20862: Update apps to latest ruby SDK.
Tom Clegg [Fri, 18 Aug 2023 20:09:07 +0000 (16:09 -0400)]
20862: Update apps to latest ruby SDK.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Update ruby and cli gem dependencies.
Tom Clegg [Fri, 18 Aug 2023 17:21:27 +0000 (13:21 -0400)]
20862: Update ruby and cli gem dependencies.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Fix use of deprecated stdlib.
Tom Clegg [Fri, 18 Aug 2023 17:16:06 +0000 (13:16 -0400)]
20862: Fix use of deprecated stdlib.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Update dependencies, disable some Faraday usage.
Tom Clegg [Fri, 18 Aug 2023 19:50:36 +0000 (15:50 -0400)]
20862: Update dependencies, disable some Faraday usage.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20862: Update gemspec to reflect forked gem name.
Tom Clegg [Fri, 18 Aug 2023 19:36:14 +0000 (15:36 -0400)]
20862: Update gemspec to reflect forked gem name.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20878-rh-package-download'
Brett Smith [Fri, 18 Aug 2023 13:28:43 +0000 (09:28 -0400)]
Merge branch '20878-rh-package-download'

Closes #20878.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20878: Document package download change to add a target distribution 20878-rh-package-download
Brett Smith [Thu, 17 Aug 2023 23:53:33 +0000 (19:53 -0400)]
20878: Document package download change to add a target distribution

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20878: Improve processed RPM check style
Brett Smith [Thu, 17 Aug 2023 22:28:17 +0000 (18:28 -0400)]
20878: Improve processed RPM check style

This is nicer bash style that makes it more consistent with the
preceding download code.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20878: Improve RPM download logic
Brett Smith [Thu, 17 Aug 2023 22:27:12 +0000 (18:27 -0400)]
20878: Improve RPM download logic

Instead of downloading and parsing a package file listing, just try to
download the package directly, and check whether that succeeds.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20878: Parameterize URL for RPM package builds
Brett Smith [Thu, 17 Aug 2023 22:20:37 +0000 (18:20 -0400)]
20878: Parameterize URL for RPM package builds

* Add support for rocky8.
* Prevent future errors by refusing to attempt a download for unknown
  targets.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months agoMerge branch '20858-jsonb-exists' refs #20858
Peter Amstutz [Wed, 16 Aug 2023 18:54:58 +0000 (14:54 -0400)]
Merge branch '20858-jsonb-exists' refs #20858

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20497 adding editable to description
Alex Coleman [Wed, 16 Aug 2023 14:50:00 +0000 (08:50 -0600)]
20497 adding editable to description

Arvados-DCO-1.1-Signed-off-by: Alex Coleman <alex.coleman@curii.com>

10 months agoMerge branch '20735-install-test'
Tom Clegg [Wed, 16 Aug 2023 13:32:32 +0000 (09:32 -0400)]
Merge branch '20735-install-test'

refs #20735

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20735: Workaround nginx/glibc/docker issue.
Tom Clegg [Wed, 16 Aug 2023 13:04:18 +0000 (09:04 -0400)]
20735: Workaround nginx/glibc/docker issue.

Docker writes both "::1 localhost" and "127.0.0.1 localhost" in
/etc/hosts even when not enabling IPv6 [1], causing glibc to resolve
"localhost" to two identical IP addresses 127.0.0.1 and 127.0.0.1 [2],
causing nginx to error out on "listen localhost:45123;" [3]:

[emerg] a duplicate listen 127.0.0.1:45123 in /tmp/nginx.conf:29

[1] https://github.com/moby/moby/issues/35954#issuecomment-498406618
("you should be testing with IPv6 enabled")

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=14969 (open since
2012-12-17)

[2] https://trac.nginx.org/nginx/ticket/2400 (workaround added
2022-11-23)

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20680-cores-concurrency' refs #20680
Peter Amstutz [Tue, 15 Aug 2023 20:52:41 +0000 (16:52 -0400)]
Merge branch '20680-cores-concurrency' refs #20680

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20858: Enable 'exists' filter to use index 20858-jsonb-exists
Peter Amstutz [Tue, 15 Aug 2023 20:41:01 +0000 (16:41 -0400)]
20858: Enable 'exists' filter to use index

See comment in services/api/db/migrate/20230815160000_jsonb_exists_functions.rb

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20735: Fix default workbench2-source path.
Tom Clegg [Fri, 11 Aug 2023 13:47:10 +0000 (09:47 -0400)]
20735: Fix default workbench2-source path.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20735: Override git directory ownership check.
Tom Clegg [Thu, 10 Aug 2023 21:18:27 +0000 (17:18 -0400)]
20735: Override git directory ownership check.

Without this, `git diff --shortstat` returns an error "warning: Not a
git repository."

The real problem, as reported by `git status`, is "fatal: detected
dubious ownership in repository at '/path'".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20735: Fix auto version generator.
Tom Clegg [Thu, 10 Aug 2023 20:41:49 +0000 (16:41 -0400)]
20735: Fix auto version generator.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20735: Revive lib/install dependency test.
Tom Clegg [Thu, 10 Aug 2023 19:22:55 +0000 (15:22 -0400)]
20735: Revive lib/install dependency test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20735: Fix firefox package name on debian 11.
Tom Clegg [Thu, 10 Aug 2023 19:21:52 +0000 (15:21 -0400)]
20735: Fix firefox package name on debian 11.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20735: Fixup singularity dependencies (test/dev-only).
Tom Clegg [Thu, 10 Aug 2023 19:20:15 +0000 (15:20 -0400)]
20735: Fixup singularity dependencies (test/dev-only).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '17244-cgroup2-cpu-max'
Tom Clegg [Tue, 15 Aug 2023 17:41:58 +0000 (13:41 -0400)]
Merge branch '17244-cgroup2-cpu-max'

fixes #17244

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago17244: Add upgrade note re number formatting change.
Tom Clegg [Tue, 15 Aug 2023 17:40:18 +0000 (13:40 -0400)]
17244: Add upgrade note re number formatting change.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoFurther fixes on single host installer deployment. No issue #
Lucas Di Pentima [Tue, 15 Aug 2023 17:39:44 +0000 (14:39 -0300)]
Further fixes on single host installer deployment. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months agoFixes deployment when using envvar BRANCH=main. No issue #
Lucas Di Pentima [Tue, 15 Aug 2023 13:53:05 +0000 (10:53 -0300)]
Fixes deployment when using envvar BRANCH=main. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20853: Modernize pdoc installation process 20853-pdoc
Brett Smith [Mon, 14 Aug 2023 21:04:10 +0000 (17:04 -0400)]
20853: Modernize pdoc installation process

Install to a virtualenv to accommodate PEP 668.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago20853: Switch from pdoc3 to pdoc
Brett Smith [Mon, 14 Aug 2023 21:00:57 +0000 (17:00 -0400)]
20853: Switch from pdoc3 to pdoc

pdoc is better maintained, both in terms of release cadence and in terms
of community code of conduct.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

10 months ago17244: New debian10 testdata, add cpu.max and cpuinfo for others. 17244-cgroup2-cpu-max
Tom Clegg [Mon, 14 Aug 2023 19:40:36 +0000 (15:40 -0400)]
17244: New debian10 testdata, add cpu.max and cpuinfo for others.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago17244: Use cpu.max to get available fraction of CPUs.
Tom Clegg [Mon, 14 Aug 2023 19:09:27 +0000 (15:09 -0400)]
17244: Use cpu.max to get available fraction of CPUs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20649-ssh-help'
Tom Clegg [Mon, 14 Aug 2023 20:29:48 +0000 (16:29 -0400)]
Merge branch '20649-ssh-help'

fixes #20649

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20457-careful-near-quota'
Tom Clegg [Mon, 14 Aug 2023 20:29:09 +0000 (16:29 -0400)]
Merge branch '20457-careful-near-quota'

fixes #20457

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20680: Set concurrency to num_cpus * 2 with a minimum of 8 20680-cores-concurrency
Peter Amstutz [Mon, 14 Aug 2023 18:41:39 +0000 (14:41 -0400)]
20680: Set concurrency to num_cpus * 2 with a minimum of 8

In practice it makes sense to have two workers per core because
workers usually spend at least half their time waiting on the
database.

Setting a minimum concurrency of 8 avoids a deadlock situation when
controller calls keep-web, which calls back to controller.

This doesn't completely eliminate the problem, but it at least avoids
having deadlocked requests with a single user in the default
configuration (!).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20649: Mention `arvados-server cloudtest` in boot timeout log msg. 20649-ssh-help
Tom Clegg [Mon, 14 Aug 2023 15:14:24 +0000 (11:14 -0400)]
20649: Mention `arvados-server cloudtest` in boot timeout log msg.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20649: Fix panic on race, worker shutdown vs. container startup.
Tom Clegg [Mon, 14 Aug 2023 15:11:51 +0000 (11:11 -0400)]
20649: Fix panic on race, worker shutdown vs. container startup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20649: Don't wait timeoutSignal before sending first TERM signal.
Tom Clegg [Mon, 14 Aug 2023 15:10:44 +0000 (11:10 -0400)]
20649: Don't wait timeoutSignal before sending first TERM signal.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoFixes comment formatting. No issue #
Lucas Di Pentima [Mon, 14 Aug 2023 14:07:19 +0000 (11:07 -0300)]
Fixes comment formatting. No issue #

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

10 months ago20457: Fix flaky test.
Tom Clegg [Wed, 9 Aug 2023 21:07:35 +0000 (17:07 -0400)]
20457: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20649: Log actual error from last failed boot probe.
Tom Clegg [Fri, 11 Aug 2023 21:47:32 +0000 (17:47 -0400)]
20649: Log actual error from last failed boot probe.

Not just the stderr content, which is useless if the error was
something like "connection failed".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20649: Log boot probe errors other than usual wait-for-boot errors.
Tom Clegg [Fri, 11 Aug 2023 21:47:06 +0000 (17:47 -0400)]
20649: Log boot probe errors other than usual wait-for-boot errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20457: Update test to match new unlock-at-quota behavior. 20457-careful-near-quota
Tom Clegg [Fri, 11 Aug 2023 17:31:29 +0000 (13:31 -0400)]
20457: Update test to match new unlock-at-quota behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20457: Don't lock-cycle next-in-line ctr while waiting for quota.
Tom Clegg [Fri, 11 Aug 2023 14:17:30 +0000 (10:17 -0400)]
20457: Don't lock-cycle next-in-line ctr while waiting for quota.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20457: Fix total instances count.
Tom Clegg [Fri, 11 Aug 2023 14:12:25 +0000 (10:12 -0400)]
20457: Fix total instances count.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoAdd packages required to build newer singularity, refs #20735
Peter Amstutz [Thu, 10 Aug 2023 18:24:53 +0000 (14:24 -0400)]
Add packages required to build newer singularity, refs #20735

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months agoMerge branch '20688-wb1-to-wb2-redirects' refs #20688
Peter Amstutz [Thu, 10 Aug 2023 17:11:30 +0000 (13:11 -0400)]
Merge branch '20688-wb1-to-wb2-redirects' refs #20688

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20649: cloudtest obeys DeployPublicKey config, like a-d-c.
Tom Clegg [Thu, 10 Aug 2023 14:45:46 +0000 (10:45 -0400)]
20649: cloudtest obeys DeployPublicKey config, like a-d-c.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20755-ec2-multiple-subnets'
Tom Clegg [Thu, 10 Aug 2023 14:34:11 +0000 (10:34 -0400)]
Merge branch '20755-ec2-multiple-subnets'

closes #20755

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20457: Fix flaky test.
Tom Clegg [Wed, 9 Aug 2023 21:07:35 +0000 (17:07 -0400)]
20457: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20457: Exercise quota handling in dispatcher chaos test.
Tom Clegg [Wed, 9 Aug 2023 21:06:40 +0000 (17:06 -0400)]
20457: Exercise quota handling in dispatcher chaos test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months ago20457: When passing prior quota level, raise maxConcurrency slowly.
Tom Clegg [Mon, 7 Aug 2023 19:26:20 +0000 (15:26 -0400)]
20457: When passing prior quota level, raise maxConcurrency slowly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

10 months agoMerge branch '20680-rolling-deploy' refs #20680
Peter Amstutz [Wed, 9 Aug 2023 13:53:48 +0000 (09:53 -0400)]
Merge branch '20680-rolling-deploy' refs #20680

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

10 months ago20680: Increase MaxConcurrentRequests to 8 20680-rolling-deploy
Peter Amstutz [Wed, 9 Aug 2023 13:48:51 +0000 (09:48 -0400)]
20680: Increase MaxConcurrentRequests to 8

Necessary to avoid a API -> keep-web -> API loop causing a deadlock.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>