arvados.git
21 months agoMerge branch '19975-oom-resubmit' refs #19975
Peter Amstutz [Tue, 7 Mar 2023 18:42:58 +0000 (13:42 -0500)]
Merge branch '19975-oom-resubmit' refs #19975

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: s/request/specify
Peter Amstutz [Tue, 7 Mar 2023 18:42:05 +0000 (13:42 -0500)]
19975: s/request/specify

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Add note about runtime.ram
Peter Amstutz [Tue, 7 Mar 2023 18:40:48 +0000 (13:40 -0500)]
19975: Add note about runtime.ram

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Typo fix.
Peter Amstutz [Tue, 7 Mar 2023 18:36:00 +0000 (13:36 -0500)]
19975: Typo fix.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Add OutOfMemoryRetry to CWL extensions page. 19975-oom-resubmit
Peter Amstutz [Tue, 7 Mar 2023 18:21:03 +0000 (13:21 -0500)]
19975: Add OutOfMemoryRetry to CWL extensions page.

Out of memory retry is now a warning for better visibility.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoFix keep-web, refs #19899
Peter Amstutz [Tue, 7 Mar 2023 15:38:07 +0000 (10:38 -0500)]
Fix keep-web, refs #19899

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20183-update-priority-thread' refs #20183
Peter Amstutz [Tue, 7 Mar 2023 14:54:38 +0000 (09:54 -0500)]
Merge branch '20183-update-priority-thread' refs #20183

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20200-log-limit' refs #20200
Peter Amstutz [Tue, 7 Mar 2023 14:46:43 +0000 (09:46 -0500)]
Merge branch '20200-log-limit' refs #20200

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '19899-webdav-cache-control' into main. Refs #19899
Stephen Smith [Mon, 6 Mar 2023 22:32:22 +0000 (17:32 -0500)]
Merge branch '19899-webdav-cache-control' into main. Refs #19899

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

21 months agoMerge branch '19981-reuse-flex-keep-cache'
Brett Smith [Mon, 6 Mar 2023 21:26:59 +0000 (16:26 -0500)]
Merge branch '19981-reuse-flex-keep-cache'

Refs #19981.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20200: Set MaxConcurrentRequests for test 20200-log-limit
Peter Amstutz [Mon, 6 Mar 2023 20:10:18 +0000 (15:10 -0500)]
20200: Set MaxConcurrentRequests for test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Add integration test for out-of-memory resubmit
Peter Amstutz [Mon, 6 Mar 2023 17:42:15 +0000 (12:42 -0500)]
19975: Add integration test for out-of-memory resubmit

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20200: Fix test typo
Peter Amstutz [Mon, 6 Mar 2023 18:18:55 +0000 (13:18 -0500)]
20200: Fix test typo

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20200: Fix defer mistake
Peter Amstutz [Mon, 6 Mar 2023 17:47:59 +0000 (12:47 -0500)]
20200: Fix defer mistake

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19899: Add Cache-Control header to keep-web CORS response 19899-webdav-cache-control
Stephen Smith [Mon, 6 Mar 2023 15:24:03 +0000 (10:24 -0500)]
19899: Add Cache-Control header to keep-web CORS response

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

21 months ago20200: Cleanups, mention changes in upgrade notes
Peter Amstutz [Mon, 6 Mar 2023 14:45:50 +0000 (09:45 -0500)]
20200: Cleanups, mention changes in upgrade notes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20183: Don't do a periodic priority-update check at startup.
Tom Clegg [Mon, 6 Mar 2023 14:31:28 +0000 (09:31 -0500)]
20183: Don't do a periodic priority-update check at startup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Fix test fixture.
Tom Clegg [Mon, 6 Mar 2023 04:18:02 +0000 (23:18 -0500)]
20183: Fix test fixture.

lsf test was relying on Rails to automatically reset "lockedcontainer"
fixture priority to 0.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Move update_priority tests. Fix updater starvation.
Tom Clegg [Mon, 6 Mar 2023 00:27:22 +0000 (19:27 -0500)]
20183: Move update_priority tests. Fix updater starvation.

Previously (in the Rails implementation) the "find containers that
have priority=0 but need priority>0" query was returning all
containers with active (committed, priority>0) requests. However, it
is possible for all such requests to have parent
containers (requesting_container_uuid) to have priority=0, in which
case Container.update_priority!() leaves the container priority at 0.

With the controller implementation, this was manifesting as lack of
progress, and warnings in logs.

This commit fixes it by not including such containers as needing
priority>0.

With the previous Rails setup, this could cause infinite recursion:
the updater thread called update_priority!(), which left
priority=0 but still triggered an after_commit hook, which
(if the updater thread had already been running for >5s) started a new
updater thread, which called update_priority!() on the same container,
etc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19981: Explain how keep_cache_* runtime constraints affect reuse 19981-reuse-flex-keep-cache
Brett Smith [Sun, 5 Mar 2023 22:06:41 +0000 (17:06 -0500)]
19981: Explain how keep_cache_* runtime constraints affect reuse

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19981: Fix verb tense typo
Brett Smith [Sun, 5 Mar 2023 21:57:11 +0000 (16:57 -0500)]
19981: Fix verb tense typo

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19981: Document the keep_cache_disk runtime constraint
Brett Smith [Sun, 5 Mar 2023 21:55:42 +0000 (16:55 -0500)]
19981: Document the keep_cache_disk runtime constraint

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19981: Add config note about impact of changing DefaultKeepCacheRAM
Brett Smith [Sun, 5 Mar 2023 21:44:40 +0000 (16:44 -0500)]
19981: Add config note about impact of changing DefaultKeepCacheRAM

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20183: Fixup auth contexts in tests.
Tom Clegg [Sun, 5 Mar 2023 04:39:17 +0000 (23:39 -0500)]
20183: Fixup auth contexts in tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Deduplicate test suite setup.
Tom Clegg [Sat, 4 Mar 2023 09:15:00 +0000 (04:15 -0500)]
20183: Deduplicate test suite setup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20183: Move priority update thread from rails to controller.
Tom Clegg [Sat, 4 Mar 2023 08:49:55 +0000 (03:49 -0500)]
20183: Move priority update thread from rails to controller.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19975: Add OutOfMemoryRetry extension
Peter Amstutz [Sun, 5 Mar 2023 04:03:19 +0000 (23:03 -0500)]
19975: Add OutOfMemoryRetry extension

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19981: Reuse containers with various keep_cache constraints
Brett Smith [Sun, 5 Mar 2023 02:31:06 +0000 (21:31 -0500)]
19981: Reuse containers with various keep_cache constraints

See the comments for rationale and discussion.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19975: Retrying RAM wip
Peter Amstutz [Sat, 4 Mar 2023 22:46:37 +0000 (17:46 -0500)]
19975: Retrying RAM wip

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19975: Initial work to auto-retry on OOM
Peter Amstutz [Sat, 4 Mar 2023 22:25:17 +0000 (17:25 -0500)]
19975: Initial work to auto-retry on OOM

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19981: Generalize vary_parameters test method
Brett Smith [Sat, 4 Mar 2023 15:54:21 +0000 (10:54 -0500)]
19981: Generalize vary_parameters test method

This is a pure reorganization commit intended to make this method
usable for tests that vary runtime constraints as well.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago16325: Document the purposes of the different log collections
Brett Smith [Sat, 4 Mar 2023 14:50:36 +0000 (09:50 -0500)]
16325: Document the purposes of the different log collections

From discussion in refs #16325.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months agoMerge branch '16325-test-container-collection-properties'
Brett Smith [Sat, 4 Mar 2023 14:24:59 +0000 (09:24 -0500)]
Merge branch '16325-test-container-collection-properties'

Closes #16325.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20200: Add test for limiting log create requests
Peter Amstutz [Fri, 3 Mar 2023 21:29:01 +0000 (16:29 -0500)]
20200: Add test for limiting log create requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago16325: Expand tests for container request collection finalization
Brett Smith [Fri, 3 Mar 2023 19:53:34 +0000 (14:53 -0500)]
16325: Expand tests for container request collection finalization

* Test both intermediate and final output containers.
* Test that collections are finalized after container success,
  failure, and cancellation.
* Test properties on both the output and log collections.

I removed the priority assertions from this test because they don't seem
to be the subject of this test, they seem to just be copy-pasted from
the test above where priority is the subject.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20200: Add limiter for log create requests
Peter Amstutz [Fri, 3 Mar 2023 19:46:48 +0000 (14:46 -0500)]
20200: Add limiter for log create requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20182-supervisor-limit' refs #20182
Peter Amstutz [Fri, 3 Mar 2023 17:03:30 +0000 (12:03 -0500)]
Merge branch '20182-supervisor-limit' refs #20182

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20182: API server sets "supervisor" flag now 20182-supervisor-limit
Peter Amstutz [Fri, 3 Mar 2023 15:44:30 +0000 (10:44 -0500)]
20182: API server sets "supervisor" flag now

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '20198-passenger-restart'
Tom Clegg [Fri, 3 Mar 2023 14:20:58 +0000 (09:20 -0500)]
Merge branch '20198-passenger-restart'

fixes #20198

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20198: Run config-watcher thread in a worker process, not preloader. 20198-passenger-restart
Tom Clegg [Thu, 2 Mar 2023 23:41:21 +0000 (18:41 -0500)]
20198: Run config-watcher thread in a worker process, not preloader.

See https://www.phusionpassenger.com/library/indepth/ruby/spawn_methods/

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago20198: Fix unclosed response body.
Tom Clegg [Thu, 2 Mar 2023 22:57:32 +0000 (17:57 -0500)]
20198: Fix unclosed response body.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19986-crunchstat-max-reporting'
Brett Smith [Thu, 2 Mar 2023 19:32:20 +0000 (14:32 -0500)]
Merge branch '19986-crunchstat-max-reporting'

Closes #19986.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19986: Prevent division by zero when reporting max disk usage
Brett Smith [Thu, 2 Mar 2023 19:31:50 +0000 (14:31 -0500)]
19986: Prevent division by zero when reporting max disk usage

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago20182: Add 'supervisor' to default scheduling params
Peter Amstutz [Thu, 2 Mar 2023 05:16:19 +0000 (00:16 -0500)]
20182: Add 'supervisor' to default scheduling params

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoAdd --disable-binary to rvm install for debian 11 for upstream bug
Peter Amstutz [Thu, 2 Mar 2023 05:11:33 +0000 (00:11 -0500)]
Add --disable-binary to rvm install for debian 11 for upstream bug

refs #20185

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoAdd workaround for 'ffi' build in debian11 package container
Peter Amstutz [Thu, 2 Mar 2023 02:51:07 +0000 (21:51 -0500)]
Add workaround for 'ffi' build in debian11 package container

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago20182: Add option to limit the number of supervisor containers
Peter Amstutz [Wed, 1 Mar 2023 22:01:55 +0000 (17:01 -0500)]
20182: Add option to limit the number of supervisor containers

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19986: Log max resource usage after a container finishes
Brett Smith [Wed, 1 Mar 2023 20:24:10 +0000 (15:24 -0500)]
19986: Log max resource usage after a container finishes

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19986: Log when a container uses nearly max RAM
Brett Smith [Wed, 1 Mar 2023 16:35:31 +0000 (11:35 -0500)]
19986: Log when a container uses nearly max RAM

This is meant to help users diagnose when their container likely failed
of OOM.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19986: DRY up logger setup in crunchstat tests
Brett Smith [Fri, 10 Feb 2023 19:47:19 +0000 (14:47 -0500)]
19986: DRY up logger setup in crunchstat tests

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19986: Separate collection of cgroup memory stats
Brett Smith [Fri, 10 Feb 2023 15:28:07 +0000 (10:28 -0500)]
19986: Separate collection of cgroup memory stats

This is scaffolding to help us report promptly when a container is
approaching OOM. This commit does not change any public interface or
reporting.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months agoMerge branch '19961-spot-interruption'
Tom Clegg [Tue, 28 Feb 2023 19:19:57 +0000 (14:19 -0500)]
Merge branch '19961-spot-interruption'

closes #19961

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19961: Update preemptionNotice text. 19961-spot-interruption
Tom Clegg [Mon, 27 Feb 2023 19:14:16 +0000 (14:14 -0500)]
19961: Update preemptionNotice text.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19961: Clarify preemptionNotice significance.
Tom Clegg [Mon, 27 Feb 2023 19:10:56 +0000 (14:10 -0500)]
19961: Clarify preemptionNotice significance.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19963-keep-web-auth-flow' into main
Tom Clegg [Thu, 23 Feb 2023 22:03:06 +0000 (17:03 -0500)]
Merge branch '19963-keep-web-auth-flow' into main

fixes #19963

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19961: Save separate preemptionNotice key in runtime_status.
Tom Clegg [Thu, 23 Feb 2023 21:59:03 +0000 (16:59 -0500)]
19961: Save separate preemptionNotice key in runtime_status.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19963: Test workbench2 login flow starting with no token. 19963-keep-web-auth-flow
Tom Clegg [Thu, 23 Feb 2023 16:26:36 +0000 (11:26 -0500)]
19963: Test workbench2 login flow starting with no token.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19963: Enable login flow without anonymous token configured.
Tom Clegg [Thu, 23 Feb 2023 15:45:14 +0000 (10:45 -0500)]
19963: Enable login flow without anonymous token configured.

Response is now 401 (was 404) when the requested collection is not
even loaded because the token is either invalid or passed in an
unsupported way (e.g., in an Authorization header with a
public-data-only URL path).

This revealed some testing issues, e.g., Test404 was attempting to
check that authenticated requests for nonexistent
files/dirs/collections returned 404, but it was in fact exercising a
code path that returned 404 because the token was not accepted.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19963: Dedup interface type.
Tom Clegg [Wed, 22 Feb 2023 15:56:52 +0000 (10:56 -0500)]
19963: Dedup interface type.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19963: Fix crash on no-cache + invalid token.
Tom Clegg [Wed, 22 Feb 2023 15:52:13 +0000 (10:52 -0500)]
19963: Fix crash on no-cache + invalid token.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '19973-dispatch-throttle' into main
Tom Clegg [Tue, 21 Feb 2023 22:04:30 +0000 (17:04 -0500)]
Merge branch '19973-dispatch-throttle' into main

fixes #19973
fixes #19984

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19961: Mention interrupt handling on admin doc page.
Tom Clegg [Tue, 21 Feb 2023 21:58:02 +0000 (16:58 -0500)]
19961: Mention interrupt handling on admin doc page.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '19980-arv-prefix-properties'
Brett Smith [Tue, 21 Feb 2023 21:27:28 +0000 (16:27 -0500)]
Merge branch '19980-arv-prefix-properties'

Closes #19980.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19980: System properties only require arv: prefix
Brett Smith [Tue, 21 Feb 2023 21:24:17 +0000 (16:24 -0500)]
19980: System properties only require arv: prefix

The decision to require a following alphabetic character was
well-intentioned but misplaced: it doesn't stop these properties from
existing, just changes the layer they would be defined at.

It might make sense to add some name rules to properties, but that would
be better done across the board as a separate story, not as a specific
corner case of system properties.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19961: Reuse metadata token within expected TTL, adjust retry logic.
Tom Clegg [Tue, 21 Feb 2023 21:14:04 +0000 (16:14 -0500)]
19961: Reuse metadata token within expected TTL, adjust retry logic.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19961: Merge branch 'main'
Tom Clegg [Tue, 21 Feb 2023 20:45:12 +0000 (15:45 -0500)]
19961: Merge branch 'main'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19973: Split unit test cases into multiple funcs. 19973-dispatch-throttle
Tom Clegg [Tue, 21 Feb 2023 16:57:27 +0000 (11:57 -0500)]
19973: Split unit test cases into multiple funcs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19973: Merge branch 'main'
Tom Clegg [Tue, 21 Feb 2023 15:24:52 +0000 (10:24 -0500)]
19973: Merge branch 'main'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19973: un-embed requestLimiter.
Tom Clegg [Tue, 21 Feb 2023 15:22:24 +0000 (10:22 -0500)]
19973: un-embed requestLimiter.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '20138-fix-sync-test-race'
Brett Smith [Tue, 21 Feb 2023 14:40:33 +0000 (09:40 -0500)]
Merge branch '20138-fix-sync-test-race'

Closes #20138.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago20138: Avoid timing failures in TestSyncNonCanonicalManifest 20138-fix-sync-test-race
Brett Smith [Mon, 20 Feb 2023 21:13:03 +0000 (16:13 -0500)]
20138: Avoid timing failures in TestSyncNonCanonicalManifest

See comments for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago20137: Fix incorrect keyword in Workbench1 config reload loop
Brett Smith [Mon, 20 Feb 2023 20:30:57 +0000 (15:30 -0500)]
20137: Fix incorrect keyword in Workbench1 config reload loop

This has the same rationale as e03ebe77c. I don't know if this code
is being exercised anywhere, but I don't want to find out the hard way
that the answer is yes.
Refs #20137.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19980: Expand documentation of reserved properties
Brett Smith [Mon, 20 Feb 2023 16:55:32 +0000 (11:55 -0500)]
19980: Expand documentation of reserved properties

* Note they will be prefixed with `arv:` from now on.
* Explain why and how these are set.
* Explain these can be set even with a strict vocabulary.
* Add the arvados-cwl-runner Git properties.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19980: Recognize the arv: prefix for all system properties
Brett Smith [Mon, 20 Feb 2023 15:53:06 +0000 (10:53 -0500)]
19980: Recognize the arv: prefix for all system properties

arvados-cwl-runner currently uses this prefix to record Git information,
and we want to use it for all future system properties.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago20137: Fix incorrect keyword in RailsAPI config reload loop
Brett Smith [Mon, 20 Feb 2023 19:46:41 +0000 (14:46 -0500)]
20137: Fix incorrect keyword in RailsAPI config reload loop

The reason rails_restart_test.go has been racey is because the code
under test is actually buggy. If the configuration reload loop ever
encounters a condition where it wants to restart (probably
`hash == hash_lastload`), it tries to call a nonexistent method
`continue`, the thread crashes, the configuration never gets reloaded,
and the test fails.

Use the intended Ruby keyword `next` instead.
Closes #20137.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19961: Fix races in tests.
Tom Clegg [Thu, 16 Feb 2023 22:26:14 +0000 (17:26 -0500)]
19961: Fix races in tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19961: Fix tests sabotaging other tests.
Tom Clegg [Thu, 16 Feb 2023 21:56:14 +0000 (16:56 -0500)]
19961: Fix tests sabotaging other tests.

Goroutines were calling "s.executor.exit <- exitCode" after their own
test was finished, at which point s.executor belonged to a subsequent
test, which would then exit too early or with the wrong exit code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '20083-sync-readonly'
Tom Clegg [Thu, 16 Feb 2023 20:56:06 +0000 (15:56 -0500)]
Merge branch '20083-sync-readonly'

fixes #20083

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '20121-update-prom-client'
Tom Clegg [Thu, 16 Feb 2023 16:37:39 +0000 (11:37 -0500)]
Merge branch '20121-update-prom-client'

fixes #20121

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago20121: Update prometheus client.
Tom Clegg [Thu, 16 Feb 2023 16:37:17 +0000 (11:37 -0500)]
20121: Update prometheus client.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19973: Fix tests.
Tom Clegg [Thu, 16 Feb 2023 05:24:46 +0000 (00:24 -0500)]
19973: Fix tests.

Stub pool AtQuota() was miscounting instances.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoAdds 'awscli' package to the salt installer state for encrypted ssl privkeys.
Lucas Di Pentima [Wed, 15 Feb 2023 20:58:24 +0000 (17:58 -0300)]
Adds 'awscli' package to the salt installer state for encrypted ssl privkeys.

refs #20035

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago19973: Test throttling after receiving 503 error.
Tom Clegg [Wed, 15 Feb 2023 17:09:42 +0000 (12:09 -0500)]
19973: Test throttling after receiving 503 error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoFixes statement_timeout unit handling on int->bigint migration. Refs #20074
Lucas Di Pentima [Wed, 15 Feb 2023 14:42:23 +0000 (11:42 -0300)]
Fixes statement_timeout unit handling on int->bigint migration. Refs #20074

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago19961: Detect and log EC2 spot interruption notices.
Tom Clegg [Tue, 14 Feb 2023 20:53:54 +0000 (15:53 -0500)]
19961: Detect and log EC2 spot interruption notices.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '20074-bigint-migration-fix'. Refs #20074
Lucas Di Pentima [Tue, 14 Feb 2023 18:57:47 +0000 (15:57 -0300)]
Merge branch '20074-bigint-migration-fix'. Refs #20074

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20074: Disables statement timeout on rails migration. 20074-bigint-migration-fix
Lucas Di Pentima [Tue, 14 Feb 2023 18:34:16 +0000 (15:34 -0300)]
20074: Disables statement timeout on rails migration.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months agoMerge branch '20035-salt-installer-privkey'. Closes #20035
Lucas Di Pentima [Tue, 14 Feb 2023 15:43:52 +0000 (12:43 -0300)]
Merge branch '20035-salt-installer-privkey'. Closes #20035

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20035: Fixes systemd unit Exec* value quoting. 20035-salt-installer-privkey
Lucas Di Pentima [Tue, 14 Feb 2023 15:37:03 +0000 (12:37 -0300)]
20035: Fixes systemd unit Exec* value quoting.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20035: Adds a warning on systemd's unit file re: runtime dirs.
Lucas Di Pentima [Tue, 14 Feb 2023 15:22:22 +0000 (12:22 -0300)]
20035: Adds a warning on systemd's unit file re: runtime dirs.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20035: Add missing jinja vars quoting on systemd's unit file template.
Lucas Di Pentima [Tue, 14 Feb 2023 15:19:42 +0000 (12:19 -0300)]
20035: Add missing jinja vars quoting on systemd's unit file template.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months agoMerge branch '20096-cwltest' refs #20096
Peter Amstutz [Mon, 13 Feb 2023 21:02:43 +0000 (16:02 -0500)]
Merge branch '20096-cwltest' refs #20096

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago20096: Report cwl runner version too
Peter Amstutz [Mon, 13 Feb 2023 20:53:40 +0000 (15:53 -0500)]
20096: Report cwl runner version too

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago20035: Manages named pipe file from systemd instead of salt.
Lucas Di Pentima [Mon, 13 Feb 2023 20:37:05 +0000 (17:37 -0300)]
20035: Manages named pipe file from systemd instead of salt.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20096: Tweak test_with_arvbox.sh 20096-cwltest
Peter Amstutz [Mon, 13 Feb 2023 19:38:38 +0000 (14:38 -0500)]
20096: Tweak test_with_arvbox.sh

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago20035: Enhances code redability, quotes jinja vars as yaml values.
Lucas Di Pentima [Fri, 10 Feb 2023 19:23:13 +0000 (16:23 -0300)]
20035: Enhances code redability, quotes jinja vars as yaml values.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months agoMerge branch '19385-cwl-fast-pack' refs #19385
Peter Amstutz [Fri, 10 Feb 2023 19:26:27 +0000 (14:26 -0500)]
Merge branch '19385-cwl-fast-pack' refs #19385

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago20035: Improves file/dir permission setting.
Lucas Di Pentima [Fri, 10 Feb 2023 19:10:27 +0000 (16:10 -0300)]
20035: Improves file/dir permission setting.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20035: Improves the connector script in various ways.
Lucas Di Pentima [Fri, 10 Feb 2023 18:16:31 +0000 (15:16 -0300)]
20035: Improves the connector script in various ways.

* Passes /dev/null to AWS CLI as credential file instead of an non-existent
  path.
* Single-quotes every Jinja variable to avoid side shell related effects.
* Passes AWS region name as a pillar.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

22 months ago20035: Updates nginx pillars to read the certificate privkey password.
Lucas Di Pentima [Tue, 7 Feb 2023 19:15:40 +0000 (16:15 -0300)]
20035: Updates nginx pillars to read the certificate privkey password.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>