arvados.git
16 months ago20343: Define Python regexps with raw strings 20343-regexp-prefixes
Brett Smith [Wed, 5 Jul 2023 20:58:07 +0000 (16:58 -0400)]
20343: Define Python regexps with raw strings

Prevent the DeprecationWarning added to Python 3.6 for unrecognized
string escape sequences. Found candidates by running:

    git grep --line-number -E '\bre\.\w+\([^r]*\\'

Most of the changes just add the raw string prefix `r` to these regexps,
but I did make some other readability improvements while I was at it,
including switching from regexps to plain string test methods where
appropriate.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months agoMerge branch '20710-shlex-quoting'
Brett Smith [Wed, 5 Jul 2023 19:46:21 +0000 (15:46 -0400)]
Merge branch '20710-shlex-quoting'

Closes #20710.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20710: Remove old backwards compatibility import fallback
Brett Smith [Wed, 5 Jul 2023 19:45:25 +0000 (15:45 -0400)]
20710: Remove old backwards compatibility import fallback

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20710: Switch from pipes.quote to shlex.quote 20710-shlex-quoting
Brett Smith [Wed, 5 Jul 2023 16:20:57 +0000 (12:20 -0400)]
20710: Switch from pipes.quote to shlex.quote

pipes is deprecated in Python 3.11+ and slated to be removed in Python
3.13. Fortunately we only use pipes.quote, which is literally the same
as shlex.quote:

    >>> import pipes
    <stdin>:1: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
    >>> import shlex
    >>> pipes.quote is shlex.quote
    True

Switch to that instead for continued support.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months agoMerge branch '20606-reuse-preemptible'
Tom Clegg [Thu, 29 Jun 2023 20:47:57 +0000 (16:47 -0400)]
Merge branch '20606-reuse-preemptible'

fixes #20606

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20606: Update retry tests. 20606-reuse-preemptible
Tom Clegg [Tue, 27 Jun 2023 14:36:08 +0000 (10:36 -0400)]
20606: Update retry tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20665-installer-secrets-handling'. Closes #20665
Lucas Di Pentima [Tue, 27 Jun 2023 13:31:38 +0000 (10:31 -0300)]
Merge branch '20665-installer-secrets-handling'. Closes #20665

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20665: Fixes logic script error & minor docs improvements. 20665-installer-secrets-handling
Lucas Di Pentima [Tue, 27 Jun 2023 13:29:27 +0000 (10:29 -0300)]
20665: Fixes logic script error & minor docs improvements.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20606: Don't reuse unstarted preemptible ctr for non-preemptible cr.
Tom Clegg [Mon, 26 Jun 2023 23:00:40 +0000 (19:00 -0400)]
20606: Don't reuse unstarted preemptible ctr for non-preemptible cr.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20619-arv-client-libfuse' closes #20619
Peter Amstutz [Mon, 26 Jun 2023 17:24:39 +0000 (13:24 -0400)]
Merge branch '20619-arv-client-libfuse'  closes #20619

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

16 months ago20665: Provides examples on how to securely handle the secrets file.
Lucas Di Pentima [Mon, 26 Jun 2023 15:52:56 +0000 (12:52 -0300)]
20665: Provides examples on how to securely handle the secrets file.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20665: Code style improvements.
Lucas Di Pentima [Mon, 26 Jun 2023 15:47:54 +0000 (12:47 -0300)]
20665: Code style improvements.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20619: Add test arvados-client executable starts 20619-arv-client-libfuse
Peter Amstutz [Fri, 23 Jun 2023 19:46:31 +0000 (15:46 -0400)]
20619: Add test arvados-client executable starts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20663-login-sync-config'
Brett Smith [Fri, 23 Jun 2023 18:44:17 +0000 (14:44 -0400)]
Merge branch '20663-login-sync-config'

Closes #20663.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20619: Add fpm-info for arvados-client to depend on fuse
Peter Amstutz [Fri, 23 Jun 2023 18:42:04 +0000 (14:42 -0400)]
20619: Add fpm-info for arvados-client to depend on fuse

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20663: Expand the default SyncIgnoredGroups 20663-login-sync-config
Brett Smith [Fri, 23 Jun 2023 18:14:38 +0000 (14:14 -0400)]
20663: Expand the default SyncIgnoredGroups

We discussed at standup that we prioritize the security of the default
configuration over backwards compatibility. This new default does that.

The list of groups is primarily informed by:
<https://wiki.debian.org/SystemGroups#Other_System_Groups>
<https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/#_list_of_statically_allocated_uidgid_and_corresponding_package>

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Merge 2.6.3 upgrading notes
Brett Smith [Fri, 23 Jun 2023 18:01:21 +0000 (14:01 -0400)]
20663: Merge 2.6.3 upgrading notes

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Add IgnoredGroups configuration to arvados-login-sync
Brett Smith [Fri, 23 Jun 2023 15:24:22 +0000 (11:24 -0400)]
20663: Add IgnoredGroups configuration to arvados-login-sync

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Bugfix configuration lookup
Brett Smith [Fri, 23 Jun 2023 13:25:24 +0000 (09:25 -0400)]
20663: Bugfix configuration lookup

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20425-fed-pdh-retry'
Tom Clegg [Thu, 22 Jun 2023 17:27:24 +0000 (13:27 -0400)]
Merge branch '20425-fed-pdh-retry'

fixes #20425

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20425: Merge branch 'main' into 20425-fed-pdh-retry 20425-fed-pdh-retry
Tom Clegg [Thu, 22 Jun 2023 16:55:05 +0000 (12:55 -0400)]
20425: Merge branch 'main' into 20425-fed-pdh-retry

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20663: Add configuration for arvados-login-sync
Brett Smith [Thu, 22 Jun 2023 15:10:29 +0000 (11:10 -0400)]
20663: Add configuration for arvados-login-sync

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Clarify comment about what "safe" config means
Brett Smith [Thu, 22 Jun 2023 14:52:05 +0000 (10:52 -0400)]
20663: Clarify comment about what "safe" config means

I might've liked to reveal configuration to authenticated clients, but
not unauthenticated ones. Unfortunately the code doesn't currently
support that.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Make arvados-login-sync actions configurable
Brett Smith [Thu, 22 Jun 2023 14:43:36 +0000 (10:43 -0400)]
20663: Make arvados-login-sync actions configurable

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Use set operations for group management
Brett Smith [Thu, 22 Jun 2023 14:18:02 +0000 (10:18 -0400)]
20663: Use set operations for group management

Make code more consistent for easier readability.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Make minimum groups configurable
Brett Smith [Thu, 22 Jun 2023 14:16:40 +0000 (10:16 -0400)]
20663: Make minimum groups configurable

Note this no longer adds the user's own group to the membership
list. The administrator can control that by adjusting USERGROUPS_ENAB
in /etc/login.defs.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20663: Improve permissions and ownership handling
Brett Smith [Thu, 22 Jun 2023 13:48:44 +0000 (09:48 -0400)]
20663: Improve permissions and ownership handling

* Set permissions of everything at creation time.
* Only change ownership for things we touch.
* Manage group ownership as well as user
  (having things owned by user:root is weird).
* Modernize style.

This is preparation for allowing administrators to configure what
resources arvados-login-sync manages.

Note this means arvados-login-sync no longer changes permissions for a
user's home directory. The administrator can do that by setting UMASK
in /etc/login.defs.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20374-workflow-main-prop' refs #20374
Peter Amstutz [Thu, 22 Jun 2023 14:43:12 +0000 (10:43 -0400)]
Merge branch '20374-workflow-main-prop' refs #20374

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20660-r-sdk-pr-204' refs #20660
Peter Amstutz [Thu, 22 Jun 2023 14:41:10 +0000 (10:41 -0400)]
Merge branch '20660-r-sdk-pr-204'  refs #20660

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20374: add arv:workflowMain to metadata properties doc. 20374-workflow-main-prop
Peter Amstutz [Thu, 22 Jun 2023 14:29:53 +0000 (10:29 -0400)]
20374: add arv:workflowMain to metadata properties doc.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20660: Tweaks to eliminate check warnings 20660-r-sdk-pr-204
Peter Amstutz [Thu, 22 Jun 2023 13:54:01 +0000 (09:54 -0400)]
20660: Tweaks to eliminate check warnings

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20665: Updates single-host docs to reflect local.params.secrets addition.
Lucas Di Pentima [Wed, 21 Jun 2023 20:51:32 +0000 (17:51 -0300)]
20665: Updates single-host docs to reflect local.params.secrets addition.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20665: Updates documentation reflecting the new local.params.secrets file.
Lucas Di Pentima [Wed, 21 Jun 2023 20:27:41 +0000 (17:27 -0300)]
20665: Updates documentation reflecting the new local.params.secrets file.

Also, fixes some old issues related to the use of ${CLUSTER} and ${DOMAIN}.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20665: Handles secrets on a separate file.
Lucas Di Pentima [Wed, 21 Jun 2023 19:11:11 +0000 (16:11 -0300)]
20665: Handles secrets on a separate file.

This allows the admin to give special treatment to it, to comply with security
policies that the local organization may have.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months agoMerge branch '20664-java-sdk-pr-205' refs #20664
Peter Amstutz [Wed, 21 Jun 2023 20:42:51 +0000 (16:42 -0400)]
Merge branch '20664-java-sdk-pr-205' refs #20664

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20660: Add devtools::check to run_test.R
Peter Amstutz [Wed, 21 Jun 2023 20:34:26 +0000 (16:34 -0400)]
20660: Add devtools::check to run_test.R

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoUpdate README.md
AnetaSta22 [Thu, 8 Jun 2023 12:20:35 +0000 (14:20 +0200)]
Update README.md

correct order

Add \dontrun{} in documentation in Arvados.R

Add \dontrun{} in documentation in ArvadosFile.R

Add \dontrun{} in documentation in Collection.R

Arvados-DCO-1.1-Signed-off-by: Aneta Stanczyk <aneta.stanczyk@contractors.roche.com>

17 months agoMerge branch '20643-login-sync-homedir-check'. Closes #20643
Lucas Di Pentima [Tue, 20 Jun 2023 14:45:40 +0000 (11:45 -0300)]
Merge branch '20643-login-sync-homedir-check'. Closes #20643

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20643: Skips user account processing earlier to avoid any partial configs. 20643-login-sync-homedir-check
Lucas Di Pentima [Tue, 20 Jun 2023 14:44:08 +0000 (11:44 -0300)]
20643: Skips user account processing earlier to avoid any partial configs.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months agoAdd new list method parameters: include_old_versions, include_trash 20664-java-sdk-pr-205
Krzysztof Majewski [Tue, 20 Jun 2023 14:07:47 +0000 (16:07 +0200)]
Add new list method parameters: include_old_versions, include_trash

Arvados-DCO-1.1-Signed-off-by: Krzysztof Majewski majewski.kielce@gmail.com

17 months agoMerge branch '20602-queue-metrics'
Tom Clegg [Tue, 20 Jun 2023 14:03:20 +0000 (10:03 -0400)]
Merge branch '20602-queue-metrics'

refs #20602

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20647-cr-logs-preflight'
Tom Clegg [Tue, 20 Jun 2023 13:43:10 +0000 (09:43 -0400)]
Merge branch '20647-cr-logs-preflight'

fixes #20647

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Allow lock requests to queue up to 2s (configurable). 20602-queue-metrics
Tom Clegg [Mon, 19 Jun 2023 20:41:18 +0000 (16:41 -0400)]
20602: Allow lock requests to queue up to 2s (configurable).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20652-cwl-poll-select' refs #20652
Peter Amstutz [Mon, 19 Jun 2023 20:07:56 +0000 (16:07 -0400)]
Merge branch '20652-cwl-poll-select' refs #20652

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20647: Improve comment on header deduplication. 20647-cr-logs-preflight
Tom Clegg [Mon, 19 Jun 2023 19:29:58 +0000 (15:29 -0400)]
20647: Improve comment on header deduplication.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20637-prefetch-threads' refs #20637
Peter Amstutz [Mon, 19 Jun 2023 18:56:48 +0000 (14:56 -0400)]
Merge branch '20637-prefetch-threads' refs #20637

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20602: Report queue usage bucketed by priority.
Tom Clegg [Mon, 19 Jun 2023 18:37:50 +0000 (14:37 -0400)]
20602: Report queue usage bucketed by priority.

Also report queue time stats for requests that are abandoned before
getting a processing slot.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Report queue time for pos/neg/zero-priority reqs.
Tom Clegg [Mon, 19 Jun 2023 15:53:38 +0000 (11:53 -0400)]
20602: Report queue time for pos/neg/zero-priority reqs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20647: Fix duplicate response headers via reverse proxy.
Tom Clegg [Sat, 17 Jun 2023 02:19:41 +0000 (22:19 -0400)]
20647: Fix duplicate response headers via reverse proxy.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20637: Fix tests 20637-prefetch-threads
Peter Amstutz [Fri, 16 Jun 2023 19:46:48 +0000 (15:46 -0400)]
20637: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20637: Fix arv-get
Peter Amstutz [Fri, 16 Jun 2023 18:53:49 +0000 (14:53 -0400)]
20637: Fix arv-get

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20637: Move the prefetch thread pool to KeepClient
Peter Amstutz [Fri, 16 Jun 2023 18:35:20 +0000 (14:35 -0400)]
20637: Move the prefetch thread pool to KeepClient

This avoids the problem of every Collection with its own BlockManager
creating its own prefetch thread pool, which becomes a resource leak
when reading files from 1000s of separate Collection objects.

The 'put' thread pool remains with the BlockManager but it now stops
the put threads on 'BlockManager.commit_all'.  This is because this
method always flushes pending blocks anyway, and is called before the
collection record is written to the API server -- so we can assume
we've just finished a batch of writes to that collection, and might
not need the put thread pool any more, and if we do, we can just make
a new one.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20602: Add queue metrics.
Tom Clegg [Fri, 16 Jun 2023 17:58:16 +0000 (13:58 -0400)]
20602: Add queue metrics.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20652: When polling the API server, limit fields returned 20652-cwl-poll-select
Peter Amstutz [Fri, 16 Jun 2023 15:48:24 +0000 (11:48 -0400)]
20652: When polling the API server, limit fields returned

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20647-cr-logs-preflight'
Tom Clegg [Fri, 16 Jun 2023 15:10:06 +0000 (11:10 -0400)]
Merge branch '20647-cr-logs-preflight'

fixes #20647

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20601-supervisor-fraction' refs #20601
Peter Amstutz [Fri, 16 Jun 2023 14:26:12 +0000 (10:26 -0400)]
Merge branch '20601-supervisor-fraction' refs #20601

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20601: Rename Scheduler.maxSupervisors -> Scheduler.supervisorFraction 20601-supervisor-fraction
Peter Amstutz [Fri, 16 Jun 2023 14:00:25 +0000 (10:00 -0400)]
20601: Rename Scheduler.maxSupervisors -> Scheduler.supervisorFraction

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20647: Test CORS preflight request routing.
Tom Clegg [Fri, 16 Jun 2023 06:06:35 +0000 (02:06 -0400)]
20647: Test CORS preflight request routing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20647: Fix CORS preflight request handling.
Tom Clegg [Fri, 16 Jun 2023 06:05:59 +0000 (02:05 -0400)]
20647: Fix CORS preflight request handling.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20601: MaxSupervisorFraction is relative to maxConcurrency.
Tom Clegg [Thu, 15 Jun 2023 19:53:34 +0000 (15:53 -0400)]
20601: MaxSupervisorFraction is relative to maxConcurrency.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20601-big-ctr-queue'
Tom Clegg [Thu, 15 Jun 2023 18:59:49 +0000 (14:59 -0400)]
Merge branch '20601-big-ctr-queue'

refs #20601

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20643: Improves error message.
Lucas Di Pentima [Thu, 15 Jun 2023 13:05:01 +0000 (10:05 -0300)]
20643: Improves error message.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20643: Skip (with a log error) users that don't have their HOME dirs created.
Lucas Di Pentima [Wed, 14 Jun 2023 15:07:44 +0000 (12:07 -0300)]
20643: Skip (with a log error) users that don't have their HOME dirs created.

Some systems may be able to resolve users from centralized services, but
need additional steps to allow them to log into the shell node. We then
expect that allowed users should already have their HOME directories created
before doing any syncing.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months agoMerge branch '20602-controller-qos'
Tom Clegg [Tue, 13 Jun 2023 16:13:25 +0000 (12:13 -0400)]
Merge branch '20602-controller-qos'

refs #20602

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20601: Add comment re fetch-by-uuid loop. 20601-big-ctr-queue
Tom Clegg [Tue, 13 Jun 2023 13:58:51 +0000 (09:58 -0400)]
20601: Add comment re fetch-by-uuid loop.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20601: Slow down container queue polling when busy/slow.
Tom Clegg [Mon, 12 Jun 2023 19:47:57 +0000 (15:47 -0400)]
20601: Slow down container queue polling when busy/slow.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Add IneligibleForQueuePriority const. 20602-controller-qos
Tom Clegg [Mon, 12 Jun 2023 18:47:30 +0000 (14:47 -0400)]
20602: Add IneligibleForQueuePriority const.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Use stdlib heap implementation.
Tom Clegg [Mon, 12 Jun 2023 18:43:39 +0000 (14:43 -0400)]
20602: Use stdlib heap implementation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20601: Fetch mounts field only once per container.
Tom Clegg [Mon, 12 Jun 2023 18:18:13 +0000 (14:18 -0400)]
20601: Fetch mounts field only once per container.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20601: Fetch only high-priority containers when queue is long.
Tom Clegg [Mon, 12 Jun 2023 15:29:33 +0000 (11:29 -0400)]
20601: Fetch only high-priority containers when queue is long.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20425: Reduce timeout for keep-web->controller calls from 5m to 1m.
Tom Clegg [Fri, 9 Jun 2023 14:07:23 +0000 (10:07 -0400)]
20425: Reduce timeout for keep-web->controller calls from 5m to 1m.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20425: Fix response status for mixed-status backend errors.
Tom Clegg [Fri, 9 Jun 2023 13:31:37 +0000 (09:31 -0400)]
20425: Fix response status for mixed-status backend errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20425: Test response status for mixed-status backend errors.
Tom Clegg [Mon, 5 Jun 2023 19:15:17 +0000 (15:15 -0400)]
20425: Test response status for mixed-status backend errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20425: Revert "12684/20425: Skip TestContainerInputOnDifferentCluster"
Tom Clegg [Mon, 5 Jun 2023 14:50:07 +0000 (10:50 -0400)]
20425: Revert "12684/20425: Skip TestContainerInputOnDifferentCluster"

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20614-no-retry-cancel' refs #20614
Peter Amstutz [Thu, 8 Jun 2023 20:12:22 +0000 (16:12 -0400)]
Merge branch '20614-no-retry-cancel'  refs #20614

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20602: Disable queue when testing DumpRequests.
Tom Clegg [Thu, 8 Jun 2023 18:30:23 +0000 (14:30 -0400)]
20602: Disable queue when testing DumpRequests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Prioritize backend traffic from keep-web as interactive.
Tom Clegg [Thu, 8 Jun 2023 14:31:11 +0000 (10:31 -0400)]
20602: Prioritize backend traffic from keep-web as interactive.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Attach assigned priority to response log entry.
Tom Clegg [Thu, 8 Jun 2023 14:30:35 +0000 (10:30 -0400)]
20602: Attach assigned priority to response log entry.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Prioritize interactive (browser) requests.
Tom Clegg [Wed, 7 Jun 2023 21:13:30 +0000 (17:13 -0400)]
20602: Prioritize interactive (browser) requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Priority MinInt64 means never wait in queue.
Tom Clegg [Wed, 7 Jun 2023 21:12:17 +0000 (17:12 -0400)]
20602: Priority MinInt64 means never wait in queue.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Add MaxQueuedRequests config.
Tom Clegg [Wed, 7 Jun 2023 16:37:40 +0000 (12:37 -0400)]
20602: Add MaxQueuedRequests config.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20602: Add priority queue to RequestLimiter.
Tom Clegg [Wed, 7 Jun 2023 16:37:25 +0000 (12:37 -0400)]
20602: Add priority queue to RequestLimiter.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20614: Better query for live container requests when deciding to retry 20614-no-retry-cancel
Peter Amstutz [Thu, 8 Jun 2023 16:31:48 +0000 (12:31 -0400)]
20614: Better query for live container requests when deciding to retry

Now takes into account the state and priority of the requesting
container, if any.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20613-googleapiclient-init-logs'
Brett Smith [Wed, 7 Jun 2023 20:28:53 +0000 (16:28 -0400)]
Merge branch '20613-googleapiclient-init-logs'

Closes #20613.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Update unconfigured logger check to accommodate NullHandler 20613-googleapiclient-init-logs
Brett Smith [Wed, 7 Jun 2023 19:00:43 +0000 (15:00 -0400)]
20613: Update unconfigured logger check to accommodate NullHandler

googleapiclient sets up its own NullHandler, as recommended by the
logging documentation. That makes the hasHandlers() test not work as
intended, so instead we traverse the hierarchy ourselves looking for a
real handler.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20548-installer-deploy' refs #20548
Peter Amstutz [Wed, 7 Jun 2023 17:51:21 +0000 (13:51 -0400)]
Merge branch '20548-installer-deploy' refs #20548

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoInitialize rcode variable, refs #20611
Peter Amstutz [Mon, 5 Jun 2023 22:10:07 +0000 (18:10 -0400)]
Initialize rcode variable, refs #20611

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20613: Add an upgrade note about new PySDK logging
Brett Smith [Wed, 7 Jun 2023 16:39:44 +0000 (12:39 -0400)]
20613: Add an upgrade note about new PySDK logging

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Fix test syntax for older Pythons
Brett Smith [Wed, 7 Jun 2023 14:50:58 +0000 (10:50 -0400)]
20613: Fix test syntax for older Pythons

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Deconfigure logger after an exception
Brett Smith [Wed, 7 Jun 2023 14:24:04 +0000 (10:24 -0400)]
20613: Deconfigure logger after an exception

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Bugfix boolean tucnaround
Brett Smith [Wed, 7 Jun 2023 12:04:13 +0000 (08:04 -0400)]
20613: Bugfix boolean tucnaround

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Use GoogleHTTPClientFilter in a-c-r
Brett Smith [Tue, 6 Jun 2023 19:35:01 +0000 (15:35 -0400)]
20613: Use GoogleHTTPClientFilter in a-c-r

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Log early discovery document retries
Brett Smith [Tue, 6 Jun 2023 19:33:03 +0000 (15:33 -0400)]
20613: Log early discovery document retries

See comments for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Introduce arvados.logging.GoogleHTTPClientFilter
Brett Smith [Tue, 6 Jun 2023 18:55:21 +0000 (14:55 -0400)]
20613: Introduce arvados.logging.GoogleHTTPClientFilter

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20613: Start PySDK logging module with existing constants
Brett Smith [Tue, 6 Jun 2023 18:54:22 +0000 (14:54 -0400)]
20613: Start PySDK logging module with existing constants

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20522-load-dispatch-priv-key'
Tom Clegg [Mon, 5 Jun 2023 19:16:11 +0000 (15:16 -0400)]
Merge branch '20522-load-dispatch-priv-key'

closes #20522

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20594-scaling-nginx-settings'. Closes #20594
Lucas Di Pentima [Mon, 5 Jun 2023 18:47:32 +0000 (15:47 -0300)]
Merge branch '20594-scaling-nginx-settings'. Closes #20594

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20548: Emit the git commit in the deploy log. 20548-installer-deploy
Peter Amstutz [Fri, 2 Jun 2023 20:37:53 +0000 (16:37 -0400)]
20548: Emit the git commit in the deploy log.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20548: Rework git deployment a bit
Peter Amstutz [Fri, 2 Jun 2023 20:18:07 +0000 (16:18 -0400)]
20548: Rework git deployment a bit

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20531-cwl-log-tail' refs #20531
Peter Amstutz [Fri, 2 Jun 2023 17:59:52 +0000 (13:59 -0400)]
Merge branch '20531-cwl-log-tail' refs #20531

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>