Peter Amstutz [Thu, 3 Aug 2023 14:31:05 +0000 (10:31 -0400)]
20688: Update single host/multiple hostname template
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 3 Aug 2023 13:44:41 +0000 (09:44 -0400)]
20688: "workbench" host is now workbench2
"workbench2" host redirects to workbench
Right now, both workbench and workbench2 roles still need to be
assigned, because the 'workbench' role is now configuring nginx but
the 'workbench2' state is the one that actually installs the package.
Will need to make some upstream changes to the formula.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 2 Aug 2023 21:18:32 +0000 (17:18 -0400)]
20688: Reorder redirects by priority
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 2 Aug 2023 18:39:20 +0000 (14:39 -0400)]
20688: check for ssh ahead of time, sync only before deploy
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 21 Jul 2023 17:44:36 +0000 (13:44 -0400)]
20688: Add nginx redirects from wb1 paths to wb2 paths to salt
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 21 Jul 2023 17:15:11 +0000 (13:15 -0400)]
20688: Fix quoting issue
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 18 Jul 2023 18:57:39 +0000 (14:57 -0400)]
20688: Add wb1 to wb2 redirects to arvbox
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 3 Aug 2023 18:26:51 +0000 (14:26 -0400)]
20680: Rework worker settings for new controller behavior
We now have separate values for concurrent requests and queued
requests. Ensure the arvados, passenger and nginx configurations align
with the correct values.
Renamed CONTROLLER_NGINX_WORKERS to CONTROLLER_MAX_WORKERS
Renamed CONTROLLER_MAX_CONCURRENT_REQUESTS to
CONTROLLER_MAX_QUEUED_REQUESTS
Adjusted config.yml defaults to reflect that MaxConcurrentRequests
means something different.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Lucas Di Pentima [Thu, 3 Aug 2023 20:53:26 +0000 (17:53 -0300)]
20610: Documentation fixes & additions.
Also, removed the 'api' role from the multi host local.params example.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 3 Aug 2023 19:58:18 +0000 (16:58 -0300)]
20610: Fixes balancer nginx's request queue config depending on # of backends
Also, fixes templating issue on arvados' config.yml file related to this
setting.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 3 Aug 2023 18:51:13 +0000 (15:51 -0300)]
20610: Unifies the 'api' role with the 'controller' role.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 3 Aug 2023 15:06:51 +0000 (12:06 -0300)]
20610: Improves provision.sh code readability.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 3 Aug 2023 14:05:24 +0000 (11:05 -0300)]
20610: Deduplicates variable substitution code.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 2 Aug 2023 21:21:22 +0000 (18:21 -0300)]
20610: Renames the ROLES map variable because the name was being used already.
Also, adds "set -eu" to provision.sh and fixes related unbound var issues.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 2 Aug 2023 13:38:07 +0000 (10:38 -0300)]
20610: Moves code from local.params to its own common.sh file.
provision & installer scripts now loads this new file that in turn loads
the params files.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Fri, 28 Jul 2023 14:43:39 +0000 (11:43 -0300)]
20610: Updates docs to reflect changes on arvados.sls configuration.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Fri, 28 Jul 2023 14:32:07 +0000 (11:32 -0300)]
20610: Adds documentation for load-balancing & rolling upgrades.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Fri, 28 Jul 2023 13:48:40 +0000 (10:48 -0300)]
20610: Removes the need to manually set up the ENABLE_BALANCER variable.
Having the role->nodes map allows us to simplify manual configuration.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 27 Jul 2023 21:17:09 +0000 (18:17 -0300)]
20610: Fixes deployment order to avoid failures.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 27 Jul 2023 19:32:36 +0000 (16:32 -0300)]
20610: Allows disabling backend controllers for rolling updates.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 27 Jul 2023 18:39:56 +0000 (15:39 -0300)]
20610: Restricts backends' HTTP access by nginx rules.
I think this is better than implementing those controls through security
groups via Terraform, because the node's role information is already available
on the salt code.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 26 Jul 2023 22:43:57 +0000 (19:43 -0300)]
20610: Fixes prometheus configuration to track the controller nodes.
Also, fixes a-d-c config, as it was also assumed to be on the controller node.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 26 Jul 2023 21:06:41 +0000 (18:06 -0300)]
20610: Removes duplication on balancer & backend declarations.
Instead of having to explicitly declare those, just use the ROLES map
to get the balancer and controller nodes.
Also, explicitly gives PG access to Keep-Balance instead of assuming it's
installed on the same node as the controller.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 26 Jul 2023 19:57:09 +0000 (16:57 -0300)]
20610: Adds node list & role->nodes map variables.
Uses the node list variable to configure prometheus node exporter from the
configuration in local.params.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 26 Jul 2023 13:53:31 +0000 (10:53 -0300)]
20610: Fixes a template variable usage.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 26 Jul 2023 13:51:25 +0000 (10:51 -0300)]
20610: Removes the need to directly edit the arvados.sls pillar.
All params are now in local.params.* files.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 29 Jun 2023 19:38:04 +0000 (16:38 -0300)]
20610: Tweaks postgresql access control to allow multiple controller hosts.
Also, explicitly adds the dispatcher's IP to the ACL, in case is located
on a different node.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 29 Jun 2023 19:33:28 +0000 (16:33 -0300)]
20610: Tweaks controller's nginx pillar to support "balanced mode".
When balancer is enabled:
* SSL nginx is not needed
* Each controller should answer to its own hostname instead of the cluster's
domain.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 29 Jun 2023 19:13:06 +0000 (16:13 -0300)]
20610: Updates arvados' pillar to handle the dispatcher's IP separately.
Also, re-arranges the _INT_IP envvars to be more explicit about IP sharing.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 29 Jun 2023 19:05:31 +0000 (16:05 -0300)]
20610: Adds the new pillars for the balancer role.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 29 Jun 2023 18:59:53 +0000 (15:59 -0300)]
20610: Adds 'balancer' role to the installer scripts.
Also, fixes a salt bootstrap issue dur to the fact that as of today, the 3004
version is no longer considered 'stable'.
Also, changes the way we use SSH so that no envvars are forwarded to remote
hosts, to avoid having 'setlocale' warnings mixed with salt cmd.run
outputs when using Debian distros to run the installer. This avoids issues
on the first run, before salt configures the remote node's locale.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Thu, 29 Jun 2023 18:57:40 +0000 (15:57 -0300)]
20610: Removes unused keepstore reference from Terraform.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Tom Clegg [Thu, 29 Jun 2023 20:47:57 +0000 (16:47 -0400)]
Merge branch '20606-reuse-preemptible'
fixes #20606
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 27 Jun 2023 14:36:08 +0000 (10:36 -0400)]
20606: Update retry tests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Lucas Di Pentima [Tue, 27 Jun 2023 13:31:38 +0000 (10:31 -0300)]
Merge branch '20665-installer-secrets-handling'. Closes #20665
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Tue, 27 Jun 2023 13:29:27 +0000 (10:29 -0300)]
20665: Fixes logic script error & minor docs improvements.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Tom Clegg [Mon, 26 Jun 2023 23:00:40 +0000 (19:00 -0400)]
20606: Don't reuse unstarted preemptible ctr for non-preemptible cr.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Mon, 26 Jun 2023 17:24:39 +0000 (13:24 -0400)]
Merge branch '20619-arv-client-libfuse' closes #20619
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Lucas Di Pentima [Mon, 26 Jun 2023 15:52:56 +0000 (12:52 -0300)]
20665: Provides examples on how to securely handle the secrets file.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Mon, 26 Jun 2023 15:47:54 +0000 (12:47 -0300)]
20665: Code style improvements.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Peter Amstutz [Fri, 23 Jun 2023 19:46:31 +0000 (15:46 -0400)]
20619: Add test arvados-client executable starts
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Fri, 23 Jun 2023 18:44:17 +0000 (14:44 -0400)]
Merge branch '20663-login-sync-config'
Closes #20663.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Fri, 23 Jun 2023 18:42:04 +0000 (14:42 -0400)]
20619: Add fpm-info for arvados-client to depend on fuse
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Fri, 23 Jun 2023 18:14:38 +0000 (14:14 -0400)]
20663: Expand the default SyncIgnoredGroups
We discussed at standup that we prioritize the security of the default
configuration over backwards compatibility. This new default does that.
The list of groups is primarily informed by:
<https://wiki.debian.org/SystemGroups#Other_System_Groups>
<https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/#_list_of_statically_allocated_uidgid_and_corresponding_package>
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 23 Jun 2023 18:01:21 +0000 (14:01 -0400)]
20663: Merge 2.6.3 upgrading notes
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 23 Jun 2023 15:24:22 +0000 (11:24 -0400)]
20663: Add IgnoredGroups configuration to arvados-login-sync
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 23 Jun 2023 13:25:24 +0000 (09:25 -0400)]
20663: Bugfix configuration lookup
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Thu, 22 Jun 2023 17:27:24 +0000 (13:27 -0400)]
Merge branch '20425-fed-pdh-retry'
fixes #20425
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 22 Jun 2023 16:55:05 +0000 (12:55 -0400)]
20425: Merge branch 'main' into 20425-fed-pdh-retry
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Thu, 22 Jun 2023 15:10:29 +0000 (11:10 -0400)]
20663: Add configuration for arvados-login-sync
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 22 Jun 2023 14:52:05 +0000 (10:52 -0400)]
20663: Clarify comment about what "safe" config means
I might've liked to reveal configuration to authenticated clients, but
not unauthenticated ones. Unfortunately the code doesn't currently
support that.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 22 Jun 2023 14:43:36 +0000 (10:43 -0400)]
20663: Make arvados-login-sync actions configurable
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 22 Jun 2023 14:18:02 +0000 (10:18 -0400)]
20663: Use set operations for group management
Make code more consistent for easier readability.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 22 Jun 2023 14:16:40 +0000 (10:16 -0400)]
20663: Make minimum groups configurable
Note this no longer adds the user's own group to the membership
list. The administrator can control that by adjusting USERGROUPS_ENAB
in /etc/login.defs.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 22 Jun 2023 13:48:44 +0000 (09:48 -0400)]
20663: Improve permissions and ownership handling
* Set permissions of everything at creation time.
* Only change ownership for things we touch.
* Manage group ownership as well as user
(having things owned by user:root is weird).
* Modernize style.
This is preparation for allowing administrators to configure what
resources arvados-login-sync manages.
Note this means arvados-login-sync no longer changes permissions for a
user's home directory. The administrator can do that by setting UMASK
in /etc/login.defs.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Thu, 22 Jun 2023 14:43:12 +0000 (10:43 -0400)]
Merge branch '20374-workflow-main-prop' refs #20374
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 22 Jun 2023 14:41:10 +0000 (10:41 -0400)]
Merge branch '20660-r-sdk-pr-204' refs #20660
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 22 Jun 2023 14:29:53 +0000 (10:29 -0400)]
20374: add arv:workflowMain to metadata properties doc.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 22 Jun 2023 13:54:01 +0000 (09:54 -0400)]
20660: Tweaks to eliminate check warnings
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Lucas Di Pentima [Wed, 21 Jun 2023 20:51:32 +0000 (17:51 -0300)]
20665: Updates single-host docs to reflect local.params.secrets addition.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 21 Jun 2023 20:27:41 +0000 (17:27 -0300)]
20665: Updates documentation reflecting the new local.params.secrets file.
Also, fixes some old issues related to the use of ${CLUSTER} and ${DOMAIN}.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 21 Jun 2023 19:11:11 +0000 (16:11 -0300)]
20665: Handles secrets on a separate file.
This allows the admin to give special treatment to it, to comply with security
policies that the local organization may have.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Peter Amstutz [Wed, 21 Jun 2023 20:42:51 +0000 (16:42 -0400)]
Merge branch '20664-java-sdk-pr-205' refs #20664
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 21 Jun 2023 20:34:26 +0000 (16:34 -0400)]
20660: Add devtools::check to run_test.R
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
AnetaSta22 [Thu, 8 Jun 2023 12:20:35 +0000 (14:20 +0200)]
Update README.md
correct order
Add \dontrun{} in documentation in Arvados.R
Add \dontrun{} in documentation in ArvadosFile.R
Add \dontrun{} in documentation in Collection.R
Arvados-DCO-1.1-Signed-off-by: Aneta Stanczyk <aneta.stanczyk@contractors.roche.com>
Lucas Di Pentima [Tue, 20 Jun 2023 14:45:40 +0000 (11:45 -0300)]
Merge branch '20643-login-sync-homedir-check'. Closes #20643
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Tue, 20 Jun 2023 14:44:08 +0000 (11:44 -0300)]
20643: Skips user account processing earlier to avoid any partial configs.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Krzysztof Majewski [Tue, 20 Jun 2023 14:07:47 +0000 (16:07 +0200)]
Add new list method parameters: include_old_versions, include_trash
Arvados-DCO-1.1-Signed-off-by: Krzysztof Majewski majewski.kielce@gmail.com
Tom Clegg [Tue, 20 Jun 2023 14:03:20 +0000 (10:03 -0400)]
Merge branch '20602-queue-metrics'
refs #20602
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 20 Jun 2023 13:43:10 +0000 (09:43 -0400)]
Merge branch '20647-cr-logs-preflight'
fixes #20647
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 19 Jun 2023 20:41:18 +0000 (16:41 -0400)]
20602: Allow lock requests to queue up to 2s (configurable).
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Mon, 19 Jun 2023 20:07:56 +0000 (16:07 -0400)]
Merge branch '20652-cwl-poll-select' refs #20652
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Mon, 19 Jun 2023 19:29:58 +0000 (15:29 -0400)]
20647: Improve comment on header deduplication.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Mon, 19 Jun 2023 18:56:48 +0000 (14:56 -0400)]
Merge branch '20637-prefetch-threads' refs #20637
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Mon, 19 Jun 2023 18:37:50 +0000 (14:37 -0400)]
20602: Report queue usage bucketed by priority.
Also report queue time stats for requests that are abandoned before
getting a processing slot.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 19 Jun 2023 15:53:38 +0000 (11:53 -0400)]
20602: Report queue time for pos/neg/zero-priority reqs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Sat, 17 Jun 2023 02:19:41 +0000 (22:19 -0400)]
20647: Fix duplicate response headers via reverse proxy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 16 Jun 2023 19:46:48 +0000 (15:46 -0400)]
20637: Fix tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Jun 2023 18:53:49 +0000 (14:53 -0400)]
20637: Fix arv-get
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Jun 2023 18:35:20 +0000 (14:35 -0400)]
20637: Move the prefetch thread pool to KeepClient
This avoids the problem of every Collection with its own BlockManager
creating its own prefetch thread pool, which becomes a resource leak
when reading files from 1000s of separate Collection objects.
The 'put' thread pool remains with the BlockManager but it now stops
the put threads on 'BlockManager.commit_all'. This is because this
method always flushes pending blocks anyway, and is called before the
collection record is written to the API server -- so we can assume
we've just finished a batch of writes to that collection, and might
not need the put thread pool any more, and if we do, we can just make
a new one.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Fri, 16 Jun 2023 17:58:16 +0000 (13:58 -0400)]
20602: Add queue metrics.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 16 Jun 2023 15:48:24 +0000 (11:48 -0400)]
20652: When polling the API server, limit fields returned
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Fri, 16 Jun 2023 15:10:06 +0000 (11:10 -0400)]
Merge branch '20647-cr-logs-preflight'
fixes #20647
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 16 Jun 2023 14:26:12 +0000 (10:26 -0400)]
Merge branch '20601-supervisor-fraction' refs #20601
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Jun 2023 14:00:25 +0000 (10:00 -0400)]
20601: Rename Scheduler.maxSupervisors -> Scheduler.supervisorFraction
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Fri, 16 Jun 2023 06:06:35 +0000 (02:06 -0400)]
20647: Test CORS preflight request routing.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 16 Jun 2023 06:05:59 +0000 (02:05 -0400)]
20647: Fix CORS preflight request handling.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 15 Jun 2023 19:53:34 +0000 (15:53 -0400)]
20601: MaxSupervisorFraction is relative to maxConcurrency.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 15 Jun 2023 18:59:49 +0000 (14:59 -0400)]
Merge branch '20601-big-ctr-queue'
refs #20601
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Lucas Di Pentima [Thu, 15 Jun 2023 13:05:01 +0000 (10:05 -0300)]
20643: Improves error message.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Wed, 14 Jun 2023 15:07:44 +0000 (12:07 -0300)]
20643: Skip (with a log error) users that don't have their HOME dirs created.
Some systems may be able to resolve users from centralized services, but
need additional steps to allow them to log into the shell node. We then
expect that allowed users should already have their HOME directories created
before doing any syncing.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Tom Clegg [Tue, 13 Jun 2023 16:13:25 +0000 (12:13 -0400)]
Merge branch '20602-controller-qos'
refs #20602
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Jun 2023 13:58:51 +0000 (09:58 -0400)]
20601: Add comment re fetch-by-uuid loop.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jun 2023 19:47:57 +0000 (15:47 -0400)]
20601: Slow down container queue polling when busy/slow.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jun 2023 18:47:30 +0000 (14:47 -0400)]
20602: Add IneligibleForQueuePriority const.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jun 2023 18:43:39 +0000 (14:43 -0400)]
20602: Use stdlib heap implementation.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jun 2023 18:18:13 +0000 (14:18 -0400)]
20601: Fetch mounts field only once per container.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jun 2023 15:29:33 +0000 (11:29 -0400)]
20601: Fetch only high-priority containers when queue is long.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jun 2023 14:07:23 +0000 (10:07 -0400)]
20425: Reduce timeout for keep-web->controller calls from 5m to 1m.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jun 2023 13:31:37 +0000 (09:31 -0400)]
20425: Fix response status for mixed-status backend errors.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
projects / arvados.git / log