]> git.arvados.org - arvados.git/log
arvados.git
2 years ago20319: Move /containers/*/log to /container_requests/*/log.
Tom Clegg [Thu, 20 Apr 2023 13:32:30 +0000 (09:32 -0400)]
20319: Move /containers/*/log to /container_requests/*/log.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoUpdate R package version to 2.6.0
Peter Amstutz [Thu, 6 Apr 2023 18:46:00 +0000 (14:46 -0400)]
Update R package version to 2.6.0

Fortunately the actual release build script replaces the version
number automatically so this is just bringing it in alignment with the
released version.

refs #20145

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoAdd 2.6.0 release to upgrade notes 2.6.0
Peter Amstutz [Thu, 6 Apr 2023 16:08:37 +0000 (12:08 -0400)]
Add 2.6.0 release to upgrade notes

refs #20145

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19889: Add an id to the container log method documentation
Brett Smith [Wed, 5 Apr 2023 16:19:23 +0000 (12:19 -0400)]
19889: Add an id to the container log method documentation

To support the release notes.
Refs #19889

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years agoMerge branch '19889-live-log-webdav'
Tom Clegg [Tue, 4 Apr 2023 18:29:13 +0000 (14:29 -0400)]
Merge branch '19889-live-log-webdav'

closes #19889

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20240-container-update-deadlock'
Tom Clegg [Tue, 4 Apr 2023 18:27:22 +0000 (14:27 -0400)]
Merge branch '20240-container-update-deadlock'

fixes #20240

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20240: Change locking mode to EXCLUSIVE. 20240-container-update-deadlock
Tom Clegg [Tue, 4 Apr 2023 15:18:08 +0000 (11:18 -0400)]
20240: Change locking mode to EXCLUSIVE.

...in order to conflict with SELECT FOR UPDATE.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20240: Don't accept/retry after deadlock in test case.
Tom Clegg [Fri, 31 Mar 2023 15:16:10 +0000 (11:16 -0400)]
20240: Don't accept/retry after deadlock in test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20240: Lock containers table before cascading priority updates.
Tom Clegg [Fri, 31 Mar 2023 15:14:19 +0000 (11:14 -0400)]
20240: Lock containers table before cascading priority updates.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20240: Expose deadlock in multi-level workflow test case.
Tom Clegg [Thu, 30 Mar 2023 21:19:13 +0000 (17:19 -0400)]
20240: Expose deadlock in multi-level workflow test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20240: Fix test.
Tom Clegg [Thu, 30 Mar 2023 18:11:32 +0000 (14:11 -0400)]
20240: Fix test.

Automatic cancellation of child containers doesn't work as expected if
the parent container is in Queued state, which never happens in real
life, but did happen in the previous version of this test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20240: Update test to reveal bug.
Tom Clegg [Thu, 23 Mar 2023 21:40:30 +0000 (17:40 -0400)]
20240: Update test to reveal bug.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Fix crash on gateway connect error. 19889-live-log-webdav
Tom Clegg [Tue, 4 Apr 2023 15:00:22 +0000 (11:00 -0400)]
19889: Fix crash on gateway connect error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '16379-installer-prometheus-grafana'. Closes #16379
Lucas Di Pentima [Tue, 4 Apr 2023 13:38:50 +0000 (10:38 -0300)]
Merge branch '16379-installer-prometheus-grafana'. Closes #16379

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '20295-fix-collection-tree-caching-bug' refs #20295
Peter Amstutz [Tue, 4 Apr 2023 13:26:09 +0000 (09:26 -0400)]
Merge branch '20295-fix-collection-tree-caching-bug' refs #20295

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18790: Fix logging noise from ENOENT.
Tom Clegg [Mon, 3 Apr 2023 15:23:32 +0000 (11:23 -0400)]
18790: Fix logging noise from ENOENT.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago18790: Fix .../containers/.../log routing.
Tom Clegg [Mon, 3 Apr 2023 15:23:03 +0000 (11:23 -0400)]
18790: Fix .../containers/.../log routing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20264-trusted-clients-portnumber'
Tom Clegg [Fri, 31 Mar 2023 14:12:02 +0000 (10:12 -0400)]
Merge branch '20264-trusted-clients-portnumber'

fixes #20264
fixes #20140

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20235-probe-after-upgrade'
Tom Clegg [Fri, 31 Mar 2023 14:10:26 +0000 (10:10 -0400)]
Merge branch '20235-probe-after-upgrade'

fixes #20235

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoAdd unit test 20295-fix-collection-tree-caching-bug
Konrad Rudolph [Thu, 30 Mar 2023 17:25:25 +0000 (19:25 +0200)]
Add unit test

Arvados-DCO-1.1-Signed-off-by: Konrad Rudolph <konrad.rudolph@gmail.com>

2 years agoFix collection tree caching bug
Konrad Rudolph [Thu, 30 Mar 2023 16:33:15 +0000 (18:33 +0200)]
Fix collection tree caching bug

Before this fix, accessing two files inside the same collection in
succession via `collection$get` will fail on the second file because the
private tree inside the `collection` is only populated with the relative
path of the first file.

Arvados-DCO-1.1-Signed-off-by: Konrad Rudolph <konrad.rudolph@gmail.com>

2 years ago16379: Tweak doc on starring the dashboards on grafana 16379-installer-prometheus-grafana
Peter Amstutz [Thu, 30 Mar 2023 17:18:38 +0000 (13:18 -0400)]
16379: Tweak doc on starring the dashboards on grafana

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago16379: Reorganize local.params examples
Peter Amstutz [Thu, 30 Mar 2023 14:56:44 +0000 (10:56 -0400)]
16379: Reorganize local.params examples

Make sure the things that always need to be changed are at the top.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago16379: Small doc updates about prometheus and grafana
Peter Amstutz [Thu, 30 Mar 2023 14:30:41 +0000 (10:30 -0400)]
16379: Small doc updates about prometheus and grafana

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago16379: Updates for grafana install
Peter Amstutz [Thu, 30 Mar 2023 13:27:35 +0000 (09:27 -0400)]
16379: Updates for grafana install

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago16379: Add prometheus and grafana to workbench aliases
Peter Amstutz [Tue, 28 Mar 2023 22:07:43 +0000 (18:07 -0400)]
16379: Add prometheus and grafana to workbench aliases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20264: Reject redirect target with userinfo. 20264-trusted-clients-portnumber
Tom Clegg [Tue, 28 Mar 2023 21:05:26 +0000 (17:05 -0400)]
20264: Reject redirect target with userinfo.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago16379: doc updates
Peter Amstutz [Tue, 28 Mar 2023 20:38:25 +0000 (16:38 -0400)]
16379: doc updates

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19889: Preserve WebDAV path when proxying to keep-web.
Tom Clegg [Tue, 28 Mar 2023 19:51:59 +0000 (15:51 -0400)]
19889: Preserve WebDAV path when proxying to keep-web.

WebDAV clients expect the path in the server response to match the
request.

Previously when proxying to keep-web we were rewriting the request
from /arvados/v1/containers/{uuid}/log/stderr.txt to
/by_id/{pdh}/stderr.txt, so the response referred to
/by_id/{pdh}/stderr.txt.

With this change, we leave the request path alone and use a new
X-Webdav-Prefix request header (/arvados/v1/containers/{uuid}/log in
this case) to tell keep-web to strip that part when accessing the
virtual filesystem.

New test uses cadaver, which fails on the previous version with

Could not access /arvados/v1/containers/zzzzz-dz642-queuedcontainer/log/ (not WebDAV-enabled?):
Did not find a collection resource.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20270-salt-installer-less-instances'. Closes #20270
Lucas Di Pentima [Tue, 28 Mar 2023 19:40:01 +0000 (16:40 -0300)]
Merge branch '20270-salt-installer-less-instances'. Closes #20270

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Fixes missing 'monitoring' role on the workbench node.
Lucas Di Pentima [Tue, 28 Mar 2023 18:22:16 +0000 (15:22 -0300)]
16379: Fixes missing 'monitoring' role on the workbench node.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Removes prometheus config for keep1 & keepproxy nodes.
Lucas Di Pentima [Tue, 28 Mar 2023 18:08:15 +0000 (15:08 -0300)]
16379: Removes prometheus config for keep1 & keepproxy nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Sets grafana's admin user password through the CLI.
Lucas Di Pentima [Tue, 21 Mar 2023 22:07:18 +0000 (19:07 -0300)]
16379: Sets grafana's admin user password through the CLI.

Although it's being correctly set on the config file, the admin credentials
are not bein honored at first server start, and so the default password
is set in the database.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Makes sure the nginx formula is included in the keepweb's top file.
Lucas Di Pentima [Tue, 21 Mar 2023 20:30:58 +0000 (17:30 -0300)]
16379: Makes sure the nginx formula is included in the keepweb's top file.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Adds nginx dependency to the htpasswd setup.
Lucas Di Pentima [Fri, 17 Mar 2023 20:39:11 +0000 (17:39 -0300)]
16379: Adds nginx dependency to the htpasswd setup.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Adds grafana with default dashboards, moves prometheus hostname.
Lucas Di Pentima [Thu, 9 Mar 2023 20:46:53 +0000 (17:46 -0300)]
16379: Adds grafana with default dashboards, moves prometheus hostname.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Adds basic auth to the prometheus UI.
Lucas Di Pentima [Fri, 3 Mar 2023 19:24:02 +0000 (16:24 -0300)]
16379: Adds basic auth to the prometheus UI.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Moves prometheus to its own subdomain instead of 'mon.prefix.domain.tld'
Lucas Di Pentima [Tue, 28 Feb 2023 18:54:55 +0000 (15:54 -0300)]
16379: Moves prometheus to its own subdomain instead of 'mon.prefix.domain.tld'

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Adds prometheus' node_exporter to all nodes.
Lucas Di Pentima [Mon, 27 Feb 2023 16:29:06 +0000 (13:29 -0300)]
16379: Adds prometheus' node_exporter to all nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago16379: Adds prometheus service for monitoring arvados & postgresql.
Lucas Di Pentima [Tue, 21 Feb 2023 21:21:58 +0000 (18:21 -0300)]
16379: Adds prometheus service for monitoring arvados & postgresql.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago20270: Removes keep1 & keepproxy nodes. Uses SSH jumphost to deploy nodes. 20270-salt-installer-less-instances
Lucas Di Pentima [Tue, 28 Mar 2023 17:19:57 +0000 (14:19 -0300)]
20270: Removes keep1 & keepproxy nodes. Uses SSH jumphost to deploy nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago20270: Refactors the VPC code so that private nodes can access the Internet.
Lucas Di Pentima [Tue, 28 Mar 2023 17:17:30 +0000 (14:17 -0300)]
20270: Refactors the VPC code so that private nodes can access the Internet.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19889: Add www-authenticate header with 401 Unauthorized response.
Tom Clegg [Mon, 27 Mar 2023 23:24:02 +0000 (19:24 -0400)]
19889: Add www-authenticate header with 401 Unauthorized response.

Test webdav with cadaver.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Remove unused code.
Tom Clegg [Mon, 27 Mar 2023 23:23:42 +0000 (19:23 -0400)]
19889: Remove unused code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Fix comment.
Tom Clegg [Mon, 27 Mar 2023 23:23:32 +0000 (19:23 -0400)]
19889: Fix comment.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20242-dup-trash-lists'
Tom Clegg [Mon, 27 Mar 2023 18:31:40 +0000 (14:31 -0400)]
Merge branch '20242-dup-trash-lists'

fixes #20242

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20140: Accept wildcards in TrustedClients.
Tom Clegg [Mon, 27 Mar 2023 18:29:38 +0000 (14:29 -0400)]
20140: Accept wildcards in TrustedClients.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20270: Copies .gitignore file to avoid adding binaries to the repository.
Lucas Di Pentima [Mon, 27 Mar 2023 17:43:31 +0000 (14:43 -0300)]
20270: Copies .gitignore file to avoid adding binaries to the repository.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago20270: Splits hosts into public & private, saving on public IPs requirements.
Lucas Di Pentima [Mon, 27 Mar 2023 17:15:02 +0000 (14:15 -0300)]
20270: Splits hosts into public & private, saving on public IPs requirements.

Also, asks for less number of instances: there's no need to multiple keepstore
nodes, and keep-web can be run on the same node as workbench.
This makes the basic default cluster to go from 6 to 4 nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago20264: Ignore superfluous :443 and :80 in trusted origin check.
Tom Clegg [Mon, 27 Mar 2023 15:38:11 +0000 (11:38 -0400)]
20264: Ignore superfluous :443 and :80 in trusted origin check.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20259-banner-tooltip-doc' refs #20259
Peter Amstutz [Fri, 24 Mar 2023 19:46:57 +0000 (15:46 -0400)]
Merge branch '20259-banner-tooltip-doc' refs #20259

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20235: Install crunch-run before run-probe even if boot-probe fails. 20235-probe-after-upgrade
Tom Clegg [Fri, 24 Mar 2023 15:18:03 +0000 (11:18 -0400)]
20235: Install crunch-run before run-probe even if boot-probe fails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20187-cache-discovery-doc'
Tom Clegg [Fri, 24 Mar 2023 05:29:34 +0000 (01:29 -0400)]
Merge branch '20187-cache-discovery-doc'

closes #20187

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20242: Trash only one when identical replicas are eligible to trash. 20242-dup-trash-lists
Tom Clegg [Thu, 23 Mar 2023 18:36:55 +0000 (14:36 -0400)]
20242: Trash only one when identical replicas are eligible to trash.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Remove references to discovery doc cache. 20187-cache-discovery-doc
Tom Clegg [Thu, 23 Mar 2023 15:23:16 +0000 (11:23 -0400)]
20187: Remove references to discovery doc cache.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Fix webdav methods condition.
Tom Clegg [Thu, 23 Mar 2023 15:14:49 +0000 (11:14 -0400)]
19889: Fix webdav methods condition.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Add routing test.
Tom Clegg [Thu, 23 Mar 2023 14:15:46 +0000 (10:15 -0400)]
19889: Add routing test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Add comments.
Tom Clegg [Wed, 22 Mar 2023 21:37:47 +0000 (17:37 -0400)]
19889: Add comments.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Update docs.
Tom Clegg [Wed, 22 Mar 2023 21:15:27 +0000 (17:15 -0400)]
19889: Update docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20259: Add documentation for banner and tooltip features 20259-banner-tooltip-doc
Peter Amstutz [Wed, 22 Mar 2023 20:57:49 +0000 (16:57 -0400)]
20259: Add documentation for banner and tooltip features

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19889: Fix container update error when CR is already final somehow.
Tom Clegg [Wed, 22 Mar 2023 17:06:15 +0000 (13:06 -0400)]
19889: Fix container update error when CR is already final somehow.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Remove unneeded workaround for not saving empty dirs.
Tom Clegg [Wed, 22 Mar 2023 13:43:21 +0000 (09:43 -0400)]
19889: Remove unneeded workaround for not saving empty dirs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Give sensitive test a prand sequence that passes.
Tom Clegg [Wed, 22 Mar 2023 06:41:55 +0000 (02:41 -0400)]
19889: Give sensitive test a prand sequence that passes.

The previous version of the test relied on test input data,
superficially random but generated the same way each run using the
default-seeded global prand generator, that happens to avoid a bug
that would fail the test.

This change doesn't address the bug, it only fixes the problem of the
test starting to fail when something changes the pseudorandom input,
e.g., another unrelated module uses the math/rand package.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Stop using old x/net/context library.
Tom Clegg [Tue, 21 Mar 2023 21:17:00 +0000 (17:17 -0400)]
19889: Stop using old x/net/context library.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19889: Serve live logs via webdav.
Tom Clegg [Tue, 21 Mar 2023 19:04:19 +0000 (15:04 -0400)]
19889: Serve live logs via webdav.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '19967-crunch-run-cost-updates'
Brett Smith [Tue, 21 Mar 2023 15:14:27 +0000 (11:14 -0400)]
Merge branch '19967-crunch-run-cost-updates'

Closes #19967.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19967: Update container cost when crunch-run receives SIGUSR2 19967-crunch-run-cost-updates
Brett Smith [Mon, 20 Mar 2023 19:52:56 +0000 (15:52 -0400)]
19967: Update container cost when crunch-run receives SIGUSR2

The Crunch dispatcher sends this signal periodically, so this provides
regular cost updates for running containers.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago20187: Don't use rails cache for discovery doc.
Tom Clegg [Thu, 16 Mar 2023 16:25:08 +0000 (12:25 -0400)]
20187: Don't use rails cache for discovery doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Expire cache and start returning errors after 24h.
Tom Clegg [Thu, 16 Mar 2023 16:23:11 +0000 (12:23 -0400)]
20187: Expire cache and start returning errors after 24h.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Use validate func instead of making cache discoverydoc-aware.
Tom Clegg [Thu, 16 Mar 2023 15:05:16 +0000 (11:05 -0400)]
20187: Use validate func instead of making cache discoverydoc-aware.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Tidy up test case.
Tom Clegg [Thu, 16 Mar 2023 15:05:04 +0000 (11:05 -0400)]
20187: Tidy up test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '19937-build-packages-fix'. Closes #19937
Lucas Di Pentima [Thu, 16 Mar 2023 14:54:58 +0000 (11:54 -0300)]
Merge branch '19937-build-packages-fix'. Closes #19937

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago20187: Fix test race.
Tom Clegg [Wed, 15 Mar 2023 18:38:53 +0000 (14:38 -0400)]
20187: Fix test race.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Check that the discovery doc is really a discovery doc.
Tom Clegg [Wed, 15 Mar 2023 18:28:16 +0000 (14:28 -0400)]
20187: Check that the discovery doc is really a discovery doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20183-update-priority-thread'
Tom Clegg [Wed, 15 Mar 2023 14:08:08 +0000 (10:08 -0400)]
Merge branch '20183-update-priority-thread'

closes #20183

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Preserve CORS and other misc headers.
Tom Clegg [Wed, 15 Mar 2023 03:35:10 +0000 (23:35 -0400)]
20187: Preserve CORS and other misc headers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Test discovery doc cache.
Tom Clegg [Thu, 9 Mar 2023 20:39:00 +0000 (15:39 -0500)]
20187: Test discovery doc cache.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: 1 minute timeout for getting discovery doc from rails.
Tom Clegg [Thu, 9 Mar 2023 14:27:30 +0000 (09:27 -0500)]
20187: 1 minute timeout for getting discovery doc from rails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Update test.
Tom Clegg [Wed, 8 Mar 2023 18:35:35 +0000 (13:35 -0500)]
20187: Update test.

Discovery doc can no longer be counted on to proxy through to rails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20187: Validate and cache discovery doc in controller.
Tom Clegg [Wed, 8 Mar 2023 17:51:00 +0000 (12:51 -0500)]
20187: Validate and cache discovery doc in controller.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19937: Only output development versions. 19937-build-packages-fix
Lucas Di Pentima [Tue, 14 Mar 2023 17:45:02 +0000 (14:45 -0300)]
19937: Only output development versions.

Also, fixed a previously existing issue where a release branch commit with
its nearest tag being older than the merge base commit was getting versioned
as the most newest development version of any given time.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19937: Avoids processing empty input (e.g: when statically linked).
Lucas Di Pentima [Fri, 27 Jan 2023 20:22:01 +0000 (17:22 -0300)]
19937: Avoids processing empty input (e.g: when statically linked).

This prevents red herring error messages like:

...
== Packages dependencies for ./usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/schema_salad/ref_resolver.cpython-37m-x86_64-linux-gnu.so ==
dpkg-query: error: --search needs at least one file name pattern argument

Use --help for help about querying packages.
...

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '19972-go-client-retry'
Tom Clegg [Mon, 13 Mar 2023 14:57:40 +0000 (10:57 -0400)]
Merge branch '19972-go-client-retry'

closes #19972

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19972: Fix racy test case.
Tom Clegg [Mon, 13 Mar 2023 14:56:39 +0000 (10:56 -0400)]
19972: Fix racy test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20223-trash-sweep' refs #20223
Peter Amstutz [Fri, 10 Mar 2023 19:45:23 +0000 (14:45 -0500)]
Merge branch '20223-trash-sweep' refs #20223

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20227: Change default BalacePeriod to 6 hours.
Peter Amstutz [Fri, 10 Mar 2023 15:52:47 +0000 (10:52 -0500)]
20227: Change default BalacePeriod to 6 hours.

This matches BalanceTimeout.

The previous time of 10 minutes was much to short, since sweeps are
not incremental, a full sweep on a moderately sized cluster could
easily take 8 minutes out of the 10 minute interval meaning
keep-balance would be running almost constantly.

Add upgrading note about keep-balance.

refs #20227

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20223: Even more efficient to use pluck instead of batches 20223-trash-sweep
Peter Amstutz [Fri, 10 Mar 2023 18:52:51 +0000 (13:52 -0500)]
20223: Even more efficient to use pluck instead of batches

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19972: Lint.
Tom Clegg [Fri, 10 Mar 2023 18:37:51 +0000 (13:37 -0500)]
19972: Lint.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19972: Remove unused code.
Tom Clegg [Fri, 10 Mar 2023 18:35:09 +0000 (13:35 -0500)]
19972: Remove unused code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20223: Need to use each_record instead of each
Peter Amstutz [Fri, 10 Mar 2023 18:24:25 +0000 (13:24 -0500)]
20223: Need to use each_record instead of each

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20223: Use in_batches in used_by to tamp down memory usage
Peter Amstutz [Fri, 10 Mar 2023 18:19:09 +0000 (13:19 -0500)]
20223: Use in_batches in used_by to tamp down memory usage

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20223: Make trash sweep use in_batches
Peter Amstutz [Fri, 10 Mar 2023 18:13:48 +0000 (13:13 -0500)]
20223: Make trash sweep use in_batches

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '20188-insufficient-subnet'
Tom Clegg [Fri, 10 Mar 2023 16:07:17 +0000 (11:07 -0500)]
Merge branch '20188-insufficient-subnet'

fixes #20188

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20188: Handle InsufficientVolumeCapacity as quota error.
Tom Clegg [Fri, 10 Mar 2023 16:06:25 +0000 (11:06 -0500)]
20188: Handle InsufficientVolumeCapacity as quota error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '20223-container-bloat' refs #20223
Peter Amstutz [Fri, 10 Mar 2023 14:31:57 +0000 (09:31 -0500)]
Merge branch '20223-container-bloat' refs #20223

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20223: Use pluck to avoid loading full record just to get log 20223-container-bloat
Peter Amstutz [Fri, 10 Mar 2023 14:29:34 +0000 (09:29 -0500)]
20223: Use pluck to avoid loading full record just to get log

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago20183: Update container priority in creation order. 20183-update-priority-thread
Tom Clegg [Thu, 9 Mar 2023 21:50:03 +0000 (16:50 -0500)]
20183: Update container priority in creation order.

When multiple containers in a tree need updates, this is effectively
top-down order, which is more efficient. The topmost update will
update the whole tree anyway, so updating subtrees first would be a
waste.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20183: Clean up sql queries.
Tom Clegg [Thu, 9 Mar 2023 21:38:59 +0000 (16:38 -0500)]
20183: Clean up sql queries.

Remove "container_requests.uuid IS NOT NULL" (this is a tautology in
an inner join)

Use same "state=Committed" condition in Rails and controller
query. This is already implied by the "containers.state in (Queued,
Locked, Running)" condition since only a Committed request should
reference such a container. However, if the distinction ever comes
up (race?  manual database update?) we would want to skip the
non-Committed requests, so this serves as cheap insurance.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20183: Explain priority update strategy in comments.
Tom Clegg [Thu, 9 Mar 2023 21:38:24 +0000 (16:38 -0500)]
20183: Explain priority update strategy in comments.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago20223: Try to make priority update much lighter weight
Peter Amstutz [Thu, 9 Mar 2023 21:36:09 +0000 (16:36 -0500)]
20223: Try to make priority update much lighter weight

Uses select and pluck in several places to avoid loading full records.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>