arvados.git
20 months agoMerge branch '20235-probe-after-upgrade'
Tom Clegg [Fri, 31 Mar 2023 14:10:26 +0000 (10:10 -0400)]
Merge branch '20235-probe-after-upgrade'

fixes #20235

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months agoMerge branch '20270-salt-installer-less-instances'. Closes #20270
Lucas Di Pentima [Tue, 28 Mar 2023 19:40:01 +0000 (16:40 -0300)]
Merge branch '20270-salt-installer-less-instances'. Closes #20270

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

20 months ago20270: Removes keep1 & keepproxy nodes. Uses SSH jumphost to deploy nodes. 20270-salt-installer-less-instances
Lucas Di Pentima [Tue, 28 Mar 2023 17:19:57 +0000 (14:19 -0300)]
20270: Removes keep1 & keepproxy nodes. Uses SSH jumphost to deploy nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

20 months ago20270: Refactors the VPC code so that private nodes can access the Internet.
Lucas Di Pentima [Tue, 28 Mar 2023 17:17:30 +0000 (14:17 -0300)]
20270: Refactors the VPC code so that private nodes can access the Internet.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

20 months agoMerge branch '20242-dup-trash-lists'
Tom Clegg [Mon, 27 Mar 2023 18:31:40 +0000 (14:31 -0400)]
Merge branch '20242-dup-trash-lists'

fixes #20242

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20270: Copies .gitignore file to avoid adding binaries to the repository.
Lucas Di Pentima [Mon, 27 Mar 2023 17:43:31 +0000 (14:43 -0300)]
20270: Copies .gitignore file to avoid adding binaries to the repository.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

20 months ago20270: Splits hosts into public & private, saving on public IPs requirements.
Lucas Di Pentima [Mon, 27 Mar 2023 17:15:02 +0000 (14:15 -0300)]
20270: Splits hosts into public & private, saving on public IPs requirements.

Also, asks for less number of instances: there's no need to multiple keepstore
nodes, and keep-web can be run on the same node as workbench.
This makes the basic default cluster to go from 6 to 4 nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

20 months agoMerge branch '20259-banner-tooltip-doc' refs #20259
Peter Amstutz [Fri, 24 Mar 2023 19:46:57 +0000 (15:46 -0400)]
Merge branch '20259-banner-tooltip-doc' refs #20259

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20 months ago20235: Install crunch-run before run-probe even if boot-probe fails. 20235-probe-after-upgrade
Tom Clegg [Fri, 24 Mar 2023 15:18:03 +0000 (11:18 -0400)]
20235: Install crunch-run before run-probe even if boot-probe fails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months agoMerge branch '20187-cache-discovery-doc'
Tom Clegg [Fri, 24 Mar 2023 05:29:34 +0000 (01:29 -0400)]
Merge branch '20187-cache-discovery-doc'

closes #20187

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20242: Trash only one when identical replicas are eligible to trash. 20242-dup-trash-lists
Tom Clegg [Thu, 23 Mar 2023 18:36:55 +0000 (14:36 -0400)]
20242: Trash only one when identical replicas are eligible to trash.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Remove references to discovery doc cache. 20187-cache-discovery-doc
Tom Clegg [Thu, 23 Mar 2023 15:23:16 +0000 (11:23 -0400)]
20187: Remove references to discovery doc cache.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20259: Add documentation for banner and tooltip features 20259-banner-tooltip-doc
Peter Amstutz [Wed, 22 Mar 2023 20:57:49 +0000 (16:57 -0400)]
20259: Add documentation for banner and tooltip features

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20 months agoMerge branch '19967-crunch-run-cost-updates'
Brett Smith [Tue, 21 Mar 2023 15:14:27 +0000 (11:14 -0400)]
Merge branch '19967-crunch-run-cost-updates'

Closes #19967.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

20 months ago19967: Update container cost when crunch-run receives SIGUSR2 19967-crunch-run-cost-updates
Brett Smith [Mon, 20 Mar 2023 19:52:56 +0000 (15:52 -0400)]
19967: Update container cost when crunch-run receives SIGUSR2

The Crunch dispatcher sends this signal periodically, so this provides
regular cost updates for running containers.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

20 months ago20187: Don't use rails cache for discovery doc.
Tom Clegg [Thu, 16 Mar 2023 16:25:08 +0000 (12:25 -0400)]
20187: Don't use rails cache for discovery doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Expire cache and start returning errors after 24h.
Tom Clegg [Thu, 16 Mar 2023 16:23:11 +0000 (12:23 -0400)]
20187: Expire cache and start returning errors after 24h.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Use validate func instead of making cache discoverydoc-aware.
Tom Clegg [Thu, 16 Mar 2023 15:05:16 +0000 (11:05 -0400)]
20187: Use validate func instead of making cache discoverydoc-aware.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Tidy up test case.
Tom Clegg [Thu, 16 Mar 2023 15:05:04 +0000 (11:05 -0400)]
20187: Tidy up test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months agoMerge branch '19937-build-packages-fix'. Closes #19937
Lucas Di Pentima [Thu, 16 Mar 2023 14:54:58 +0000 (11:54 -0300)]
Merge branch '19937-build-packages-fix'. Closes #19937

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

20 months ago20187: Fix test race.
Tom Clegg [Wed, 15 Mar 2023 18:38:53 +0000 (14:38 -0400)]
20187: Fix test race.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Check that the discovery doc is really a discovery doc.
Tom Clegg [Wed, 15 Mar 2023 18:28:16 +0000 (14:28 -0400)]
20187: Check that the discovery doc is really a discovery doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months agoMerge branch '20183-update-priority-thread'
Tom Clegg [Wed, 15 Mar 2023 14:08:08 +0000 (10:08 -0400)]
Merge branch '20183-update-priority-thread'

closes #20183

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Preserve CORS and other misc headers.
Tom Clegg [Wed, 15 Mar 2023 03:35:10 +0000 (23:35 -0400)]
20187: Preserve CORS and other misc headers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Test discovery doc cache.
Tom Clegg [Thu, 9 Mar 2023 20:39:00 +0000 (15:39 -0500)]
20187: Test discovery doc cache.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: 1 minute timeout for getting discovery doc from rails.
Tom Clegg [Thu, 9 Mar 2023 14:27:30 +0000 (09:27 -0500)]
20187: 1 minute timeout for getting discovery doc from rails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Update test.
Tom Clegg [Wed, 8 Mar 2023 18:35:35 +0000 (13:35 -0500)]
20187: Update test.

Discovery doc can no longer be counted on to proxy through to rails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

20 months ago20187: Validate and cache discovery doc in controller.
Tom Clegg [Wed, 8 Mar 2023 17:51:00 +0000 (12:51 -0500)]
20187: Validate and cache discovery doc in controller.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19937: Only output development versions. 19937-build-packages-fix
Lucas Di Pentima [Tue, 14 Mar 2023 17:45:02 +0000 (14:45 -0300)]
19937: Only output development versions.

Also, fixed a previously existing issue where a release branch commit with
its nearest tag being older than the merge base commit was getting versioned
as the most newest development version of any given time.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

21 months ago19937: Avoids processing empty input (e.g: when statically linked).
Lucas Di Pentima [Fri, 27 Jan 2023 20:22:01 +0000 (17:22 -0300)]
19937: Avoids processing empty input (e.g: when statically linked).

This prevents red herring error messages like:

...
== Packages dependencies for ./usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/schema_salad/ref_resolver.cpython-37m-x86_64-linux-gnu.so ==
dpkg-query: error: --search needs at least one file name pattern argument

Use --help for help about querying packages.
...

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

21 months agoMerge branch '19972-go-client-retry'
Tom Clegg [Mon, 13 Mar 2023 14:57:40 +0000 (10:57 -0400)]
Merge branch '19972-go-client-retry'

closes #19972

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19972: Fix racy test case.
Tom Clegg [Mon, 13 Mar 2023 14:56:39 +0000 (10:56 -0400)]
19972: Fix racy test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '20223-trash-sweep' refs #20223
Peter Amstutz [Fri, 10 Mar 2023 19:45:23 +0000 (14:45 -0500)]
Merge branch '20223-trash-sweep' refs #20223

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20227: Change default BalacePeriod to 6 hours.
Peter Amstutz [Fri, 10 Mar 2023 15:52:47 +0000 (10:52 -0500)]
20227: Change default BalacePeriod to 6 hours.

This matches BalanceTimeout.

The previous time of 10 minutes was much to short, since sweeps are
not incremental, a full sweep on a moderately sized cluster could
easily take 8 minutes out of the 10 minute interval meaning
keep-balance would be running almost constantly.

Add upgrading note about keep-balance.

refs #20227

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20223: Even more efficient to use pluck instead of batches 20223-trash-sweep
Peter Amstutz [Fri, 10 Mar 2023 18:52:51 +0000 (13:52 -0500)]
20223: Even more efficient to use pluck instead of batches

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19972: Lint.
Tom Clegg [Fri, 10 Mar 2023 18:37:51 +0000 (13:37 -0500)]
19972: Lint.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19972: Remove unused code.
Tom Clegg [Fri, 10 Mar 2023 18:35:09 +0000 (13:35 -0500)]
19972: Remove unused code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20223: Need to use each_record instead of each
Peter Amstutz [Fri, 10 Mar 2023 18:24:25 +0000 (13:24 -0500)]
20223: Need to use each_record instead of each

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20223: Use in_batches in used_by to tamp down memory usage
Peter Amstutz [Fri, 10 Mar 2023 18:19:09 +0000 (13:19 -0500)]
20223: Use in_batches in used_by to tamp down memory usage

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20223: Make trash sweep use in_batches
Peter Amstutz [Fri, 10 Mar 2023 18:13:48 +0000 (13:13 -0500)]
20223: Make trash sweep use in_batches

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20188-insufficient-subnet'
Tom Clegg [Fri, 10 Mar 2023 16:07:17 +0000 (11:07 -0500)]
Merge branch '20188-insufficient-subnet'

fixes #20188

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20188: Handle InsufficientVolumeCapacity as quota error.
Tom Clegg [Fri, 10 Mar 2023 16:06:25 +0000 (11:06 -0500)]
20188: Handle InsufficientVolumeCapacity as quota error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '20223-container-bloat' refs #20223
Peter Amstutz [Fri, 10 Mar 2023 14:31:57 +0000 (09:31 -0500)]
Merge branch '20223-container-bloat' refs #20223

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20223: Use pluck to avoid loading full record just to get log 20223-container-bloat
Peter Amstutz [Fri, 10 Mar 2023 14:29:34 +0000 (09:29 -0500)]
20223: Use pluck to avoid loading full record just to get log

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20183: Update container priority in creation order. 20183-update-priority-thread
Tom Clegg [Thu, 9 Mar 2023 21:50:03 +0000 (16:50 -0500)]
20183: Update container priority in creation order.

When multiple containers in a tree need updates, this is effectively
top-down order, which is more efficient. The topmost update will
update the whole tree anyway, so updating subtrees first would be a
waste.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Clean up sql queries.
Tom Clegg [Thu, 9 Mar 2023 21:38:59 +0000 (16:38 -0500)]
20183: Clean up sql queries.

Remove "container_requests.uuid IS NOT NULL" (this is a tautology in
an inner join)

Use same "state=Committed" condition in Rails and controller
query. This is already implied by the "containers.state in (Queued,
Locked, Running)" condition since only a Committed request should
reference such a container. However, if the distinction ever comes
up (race?  manual database update?) we would want to skip the
non-Committed requests, so this serves as cheap insurance.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Explain priority update strategy in comments.
Tom Clegg [Thu, 9 Mar 2023 21:38:24 +0000 (16:38 -0500)]
20183: Explain priority update strategy in comments.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20223: Try to make priority update much lighter weight
Peter Amstutz [Thu, 9 Mar 2023 21:36:09 +0000 (16:36 -0500)]
20223: Try to make priority update much lighter weight

Uses select and pluck in several places to avoid loading full records.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20214-r-sdk' refs #20214
Peter Amstutz [Wed, 8 Mar 2023 18:28:44 +0000 (13:28 -0500)]
Merge branch '20214-r-sdk' refs #20214

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agospeed up write function 20214-r-sdk
AnetaSta22 [Thu, 23 Feb 2023 13:19:02 +0000 (14:19 +0100)]
speed up write function

Contributed by Aneta Stanczyk <aneta.stanczyk@contractors.roche.com>

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19972: Disable auto-retry in diagnostics and rpc.
Tom Clegg [Wed, 1 Mar 2023 20:16:39 +0000 (15:16 -0500)]
19972: Disable auto-retry in diagnostics and rpc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19972: Disable auto-retry in ws and dispatcher.
Tom Clegg [Tue, 28 Feb 2023 21:04:10 +0000 (16:04 -0500)]
19972: Disable auto-retry in ws and dispatcher.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19972: Auto retry with exponential backoff.
Tom Clegg [Tue, 28 Feb 2023 16:30:26 +0000 (11:30 -0500)]
19972: Auto retry with exponential backoff.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20188: Recognize InsufficientFreeAddressesInSubnet as quota error. 20188-insufficient-subnet
Tom Clegg [Thu, 2 Mar 2023 19:48:49 +0000 (14:48 -0500)]
20188: Recognize InsufficientFreeAddressesInSubnet as quota error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19975-oom-resubmit' refs #19975
Peter Amstutz [Tue, 7 Mar 2023 18:42:58 +0000 (13:42 -0500)]
Merge branch '19975-oom-resubmit' refs #19975

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: s/request/specify
Peter Amstutz [Tue, 7 Mar 2023 18:42:05 +0000 (13:42 -0500)]
19975: s/request/specify

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Add note about runtime.ram
Peter Amstutz [Tue, 7 Mar 2023 18:40:48 +0000 (13:40 -0500)]
19975: Add note about runtime.ram

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Typo fix.
Peter Amstutz [Tue, 7 Mar 2023 18:36:00 +0000 (13:36 -0500)]
19975: Typo fix.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Add OutOfMemoryRetry to CWL extensions page. 19975-oom-resubmit
Peter Amstutz [Tue, 7 Mar 2023 18:21:03 +0000 (13:21 -0500)]
19975: Add OutOfMemoryRetry to CWL extensions page.

Out of memory retry is now a warning for better visibility.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoFix keep-web, refs #19899
Peter Amstutz [Tue, 7 Mar 2023 15:38:07 +0000 (10:38 -0500)]
Fix keep-web, refs #19899

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20183-update-priority-thread' refs #20183
Peter Amstutz [Tue, 7 Mar 2023 14:54:38 +0000 (09:54 -0500)]
Merge branch '20183-update-priority-thread' refs #20183

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20200-log-limit' refs #20200
Peter Amstutz [Tue, 7 Mar 2023 14:46:43 +0000 (09:46 -0500)]
Merge branch '20200-log-limit' refs #20200

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '19899-webdav-cache-control' into main. Refs #19899
Stephen Smith [Mon, 6 Mar 2023 22:32:22 +0000 (17:32 -0500)]
Merge branch '19899-webdav-cache-control' into main. Refs #19899

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

21 months agoMerge branch '19981-reuse-flex-keep-cache'
Brett Smith [Mon, 6 Mar 2023 21:26:59 +0000 (16:26 -0500)]
Merge branch '19981-reuse-flex-keep-cache'

Refs #19981.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20200: Set MaxConcurrentRequests for test 20200-log-limit
Peter Amstutz [Mon, 6 Mar 2023 20:10:18 +0000 (15:10 -0500)]
20200: Set MaxConcurrentRequests for test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Add integration test for out-of-memory resubmit
Peter Amstutz [Mon, 6 Mar 2023 17:42:15 +0000 (12:42 -0500)]
19975: Add integration test for out-of-memory resubmit

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20200: Fix test typo
Peter Amstutz [Mon, 6 Mar 2023 18:18:55 +0000 (13:18 -0500)]
20200: Fix test typo

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20200: Fix defer mistake
Peter Amstutz [Mon, 6 Mar 2023 17:47:59 +0000 (12:47 -0500)]
20200: Fix defer mistake

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19899: Add Cache-Control header to keep-web CORS response 19899-webdav-cache-control
Stephen Smith [Mon, 6 Mar 2023 15:24:03 +0000 (10:24 -0500)]
19899: Add Cache-Control header to keep-web CORS response

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

21 months ago20200: Cleanups, mention changes in upgrade notes
Peter Amstutz [Mon, 6 Mar 2023 14:45:50 +0000 (09:45 -0500)]
20200: Cleanups, mention changes in upgrade notes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20183: Don't do a periodic priority-update check at startup.
Tom Clegg [Mon, 6 Mar 2023 14:31:28 +0000 (09:31 -0500)]
20183: Don't do a periodic priority-update check at startup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Fix test fixture.
Tom Clegg [Mon, 6 Mar 2023 04:18:02 +0000 (23:18 -0500)]
20183: Fix test fixture.

lsf test was relying on Rails to automatically reset "lockedcontainer"
fixture priority to 0.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Move update_priority tests. Fix updater starvation.
Tom Clegg [Mon, 6 Mar 2023 00:27:22 +0000 (19:27 -0500)]
20183: Move update_priority tests. Fix updater starvation.

Previously (in the Rails implementation) the "find containers that
have priority=0 but need priority>0" query was returning all
containers with active (committed, priority>0) requests. However, it
is possible for all such requests to have parent
containers (requesting_container_uuid) to have priority=0, in which
case Container.update_priority!() leaves the container priority at 0.

With the controller implementation, this was manifesting as lack of
progress, and warnings in logs.

This commit fixes it by not including such containers as needing
priority>0.

With the previous Rails setup, this could cause infinite recursion:
the updater thread called update_priority!(), which left
priority=0 but still triggered an after_commit hook, which
(if the updater thread had already been running for >5s) started a new
updater thread, which called update_priority!() on the same container,
etc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19981: Explain how keep_cache_* runtime constraints affect reuse 19981-reuse-flex-keep-cache
Brett Smith [Sun, 5 Mar 2023 22:06:41 +0000 (17:06 -0500)]
19981: Explain how keep_cache_* runtime constraints affect reuse

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19981: Fix verb tense typo
Brett Smith [Sun, 5 Mar 2023 21:57:11 +0000 (16:57 -0500)]
19981: Fix verb tense typo

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19981: Document the keep_cache_disk runtime constraint
Brett Smith [Sun, 5 Mar 2023 21:55:42 +0000 (16:55 -0500)]
19981: Document the keep_cache_disk runtime constraint

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19981: Add config note about impact of changing DefaultKeepCacheRAM
Brett Smith [Sun, 5 Mar 2023 21:44:40 +0000 (16:44 -0500)]
19981: Add config note about impact of changing DefaultKeepCacheRAM

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20183: Fixup auth contexts in tests.
Tom Clegg [Sun, 5 Mar 2023 04:39:17 +0000 (23:39 -0500)]
20183: Fixup auth contexts in tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Deduplicate test suite setup.
Tom Clegg [Sat, 4 Mar 2023 09:15:00 +0000 (04:15 -0500)]
20183: Deduplicate test suite setup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Move priority update thread from rails to controller.
Tom Clegg [Sat, 4 Mar 2023 08:49:55 +0000 (03:49 -0500)]
20183: Move priority update thread from rails to controller.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19975: Add OutOfMemoryRetry extension
Peter Amstutz [Sun, 5 Mar 2023 04:03:19 +0000 (23:03 -0500)]
19975: Add OutOfMemoryRetry extension

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19981: Reuse containers with various keep_cache constraints
Brett Smith [Sun, 5 Mar 2023 02:31:06 +0000 (21:31 -0500)]
19981: Reuse containers with various keep_cache constraints

See the comments for rationale and discussion.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19975: Retrying RAM wip
Peter Amstutz [Sat, 4 Mar 2023 22:46:37 +0000 (17:46 -0500)]
19975: Retrying RAM wip

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Initial work to auto-retry on OOM
Peter Amstutz [Sat, 4 Mar 2023 22:25:17 +0000 (17:25 -0500)]
19975: Initial work to auto-retry on OOM

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19981: Generalize vary_parameters test method
Brett Smith [Sat, 4 Mar 2023 15:54:21 +0000 (10:54 -0500)]
19981: Generalize vary_parameters test method

This is a pure reorganization commit intended to make this method
usable for tests that vary runtime constraints as well.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago16325: Document the purposes of the different log collections
Brett Smith [Sat, 4 Mar 2023 14:50:36 +0000 (09:50 -0500)]
16325: Document the purposes of the different log collections

From discussion in refs #16325.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months agoMerge branch '16325-test-container-collection-properties'
Brett Smith [Sat, 4 Mar 2023 14:24:59 +0000 (09:24 -0500)]
Merge branch '16325-test-container-collection-properties'

Closes #16325.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20200: Add test for limiting log create requests
Peter Amstutz [Fri, 3 Mar 2023 21:29:01 +0000 (16:29 -0500)]
20200: Add test for limiting log create requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago16325: Expand tests for container request collection finalization
Brett Smith [Fri, 3 Mar 2023 19:53:34 +0000 (14:53 -0500)]
16325: Expand tests for container request collection finalization

* Test both intermediate and final output containers.
* Test that collections are finalized after container success,
  failure, and cancellation.
* Test properties on both the output and log collections.

I removed the priority assertions from this test because they don't seem
to be the subject of this test, they seem to just be copy-pasted from
the test above where priority is the subject.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20200: Add limiter for log create requests
Peter Amstutz [Fri, 3 Mar 2023 19:46:48 +0000 (14:46 -0500)]
20200: Add limiter for log create requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20182-supervisor-limit' refs #20182
Peter Amstutz [Fri, 3 Mar 2023 17:03:30 +0000 (12:03 -0500)]
Merge branch '20182-supervisor-limit' refs #20182

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20182: API server sets "supervisor" flag now 20182-supervisor-limit
Peter Amstutz [Fri, 3 Mar 2023 15:44:30 +0000 (10:44 -0500)]
20182: API server sets "supervisor" flag now

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20198-passenger-restart'
Tom Clegg [Fri, 3 Mar 2023 14:20:58 +0000 (09:20 -0500)]
Merge branch '20198-passenger-restart'

fixes #20198

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20198: Run config-watcher thread in a worker process, not preloader. 20198-passenger-restart
Tom Clegg [Thu, 2 Mar 2023 23:41:21 +0000 (18:41 -0500)]
20198: Run config-watcher thread in a worker process, not preloader.

See https://www.phusionpassenger.com/library/indepth/ruby/spawn_methods/

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20198: Fix unclosed response body.
Tom Clegg [Thu, 2 Mar 2023 22:57:32 +0000 (17:57 -0500)]
20198: Fix unclosed response body.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19986-crunchstat-max-reporting'
Brett Smith [Thu, 2 Mar 2023 19:32:20 +0000 (14:32 -0500)]
Merge branch '19986-crunchstat-max-reporting'

Closes #19986.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19986: Prevent division by zero when reporting max disk usage
Brett Smith [Thu, 2 Mar 2023 19:31:50 +0000 (14:31 -0500)]
19986: Prevent division by zero when reporting max disk usage

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20182: Add 'supervisor' to default scheduling params
Peter Amstutz [Thu, 2 Mar 2023 05:16:19 +0000 (00:16 -0500)]
20182: Add 'supervisor' to default scheduling params

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoAdd --disable-binary to rvm install for debian 11 for upstream bug
Peter Amstutz [Thu, 2 Mar 2023 05:11:33 +0000 (00:11 -0500)]
Add --disable-binary to rvm install for debian 11 for upstream bug

refs #20185

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoAdd workaround for 'ffi' build in debian11 package container
Peter Amstutz [Thu, 2 Mar 2023 02:51:07 +0000 (21:51 -0500)]
Add workaround for 'ffi' build in debian11 package container

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>