arvados.git
16 months ago20457: Update test to match new unlock-at-quota behavior. 20457-careful-near-quota
Tom Clegg [Fri, 11 Aug 2023 17:31:29 +0000 (13:31 -0400)]
20457: Update test to match new unlock-at-quota behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20457: Don't lock-cycle next-in-line ctr while waiting for quota.
Tom Clegg [Fri, 11 Aug 2023 14:17:30 +0000 (10:17 -0400)]
20457: Don't lock-cycle next-in-line ctr while waiting for quota.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20457: Fix total instances count.
Tom Clegg [Fri, 11 Aug 2023 14:12:25 +0000 (10:12 -0400)]
20457: Fix total instances count.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20457: Fix flaky test.
Tom Clegg [Wed, 9 Aug 2023 21:07:35 +0000 (17:07 -0400)]
20457: Fix flaky test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20457: Exercise quota handling in dispatcher chaos test.
Tom Clegg [Wed, 9 Aug 2023 21:06:40 +0000 (17:06 -0400)]
20457: Exercise quota handling in dispatcher chaos test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20457: When passing prior quota level, raise maxConcurrency slowly.
Tom Clegg [Mon, 7 Aug 2023 19:26:20 +0000 (15:26 -0400)]
20457: When passing prior quota level, raise maxConcurrency slowly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20735-update-deps'
Tom Clegg [Tue, 8 Aug 2023 18:52:42 +0000 (14:52 -0400)]
Merge branch '20735-update-deps'

refs #20735

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20735: Clarify singularity version/compatibility expectations.
Tom Clegg [Tue, 8 Aug 2023 18:51:21 +0000 (14:51 -0400)]
20735: Clarify singularity version/compatibility expectations.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoFix error check.
Tom Clegg [Tue, 8 Aug 2023 14:18:29 +0000 (10:18 -0400)]
Fix error check.

refs #17244

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20754-docker-py-upgrade'
Brett Smith [Mon, 7 Aug 2023 21:18:56 +0000 (17:18 -0400)]
Merge branch '20754-docker-py-upgrade'

Closes #20754.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20735: Merge branch 'main' into 20735-update-deps 20735-update-deps
Tom Clegg [Mon, 7 Aug 2023 20:58:45 +0000 (16:58 -0400)]
20735: Merge branch 'main' into 20735-update-deps

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20735: Update singularity version in docs and build script.
Tom Clegg [Mon, 7 Aug 2023 20:57:23 +0000 (16:57 -0400)]
20735: Update singularity version in docs and build script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20705-crunchstat-warn-missing-data'
Tom Clegg [Mon, 7 Aug 2023 20:49:17 +0000 (16:49 -0400)]
Merge branch '20705-crunchstat-warn-missing-data'

fixes #20705

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20680-default-config-updates' refs #20680
Peter Amstutz [Mon, 7 Aug 2023 20:29:15 +0000 (16:29 -0400)]
Merge branch '20680-default-config-updates' refs #20680

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

16 months ago20680: Fix nginx_balancer_configuration & add note about defaults 20680-default-config-updates
Peter Amstutz [Mon, 7 Aug 2023 20:28:13 +0000 (16:28 -0400)]
20680: Fix nginx_balancer_configuration & add note about defaults

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

16 months ago20705: 10% failure on stub Create to ensure running error path. 20705-crunchstat-warn-missing-data
Tom Clegg [Mon, 7 Aug 2023 14:31:56 +0000 (10:31 -0400)]
20705: 10% failure on stub Create to ensure running error path.

Fixes occasional test failures due to absence of rate-limiting errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20735: Sync bundler version.
Tom Clegg [Mon, 7 Aug 2023 14:04:38 +0000 (10:04 -0400)]
20735: Sync bundler version.

See https://dev.arvados.org/issues/20735#note-6

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoFixes instance profile assignment. Refs #20610
Lucas Di Pentima [Fri, 4 Aug 2023 20:55:13 +0000 (17:55 -0300)]
Fixes instance profile assignment. Refs #20610

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months agoMerge branch '20610-installer-load-balancer'. Refs #20610
Lucas Di Pentima [Fri, 4 Aug 2023 19:08:32 +0000 (16:08 -0300)]
Merge branch '20610-installer-load-balancer'. Refs #20610

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20754: Upgrade docker module for docker-cleaner 20754-docker-py-upgrade
Brett Smith [Fri, 4 Aug 2023 18:28:18 +0000 (14:28 -0400)]
20754: Upgrade docker module for docker-cleaner

The immediate motivation is dealing with the bug discussed in the
comments. Upgrading generally seems like a good idea since we've been
pinned at a specific version for so long.

Changing the Docker API version is required by the library: 1.21 is the
oldest version currently supported. Our Go stack is standardized on
1.35, so use that here too.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20610: Documentation changes, including simplification & fixes. 20610-installer-load-balancer
Lucas Di Pentima [Fri, 4 Aug 2023 18:08:04 +0000 (15:08 -0300)]
20610: Documentation changes, including simplification & fixes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20754: Remove test requirements from docker-cleaner
Brett Smith [Fri, 4 Aug 2023 18:05:26 +0000 (14:05 -0400)]
20754: Remove test requirements from docker-cleaner

Neither are required with our current Python stack.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20610: Changes the default installer config in order to simplify documentation.
Lucas Di Pentima [Fri, 4 Aug 2023 17:48:29 +0000 (14:48 -0300)]
20610: Changes the default installer config in order to simplify documentation.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20680: Rework worker settings for new controller behavior
Peter Amstutz [Thu, 3 Aug 2023 18:26:51 +0000 (14:26 -0400)]
20680: Rework worker settings for new controller behavior

We now have separate values for concurrent requests and queued
requests.  Ensure the arvados, passenger and nginx configurations align
with the correct values.

Renamed CONTROLLER_NGINX_WORKERS to CONTROLLER_MAX_WORKERS

Renamed CONTROLLER_MAX_CONCURRENT_REQUESTS to
CONTROLLER_MAX_QUEUED_REQUESTS

Adjusted config.yml defaults to reflect that MaxConcurrentRequests
means something different.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

16 months ago20610: Documentation fixes & additions.
Lucas Di Pentima [Thu, 3 Aug 2023 20:53:26 +0000 (17:53 -0300)]
20610: Documentation fixes & additions.

Also, removed the 'api' role from the multi host local.params example.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Fixes balancer nginx's request queue config depending on # of backends
Lucas Di Pentima [Thu, 3 Aug 2023 19:58:18 +0000 (16:58 -0300)]
20610: Fixes balancer nginx's request queue config depending on # of backends

Also, fixes templating issue on arvados' config.yml file related to this
setting.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Unifies the 'api' role with the 'controller' role.
Lucas Di Pentima [Thu, 3 Aug 2023 18:51:13 +0000 (15:51 -0300)]
20610: Unifies the 'api' role with the 'controller' role.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Improves provision.sh code readability.
Lucas Di Pentima [Thu, 3 Aug 2023 15:06:51 +0000 (12:06 -0300)]
20610: Improves provision.sh code readability.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Deduplicates variable substitution code.
Lucas Di Pentima [Thu, 3 Aug 2023 14:05:24 +0000 (11:05 -0300)]
20610: Deduplicates variable substitution code.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Renames the ROLES map variable because the name was being used already.
Lucas Di Pentima [Wed, 2 Aug 2023 21:21:22 +0000 (18:21 -0300)]
20610: Renames the ROLES map variable because the name was being used already.

Also, adds "set -eu" to provision.sh and fixes related unbound var issues.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20735: Update Go to 1.20.6.
Tom Clegg [Wed, 2 Aug 2023 20:11:11 +0000 (16:11 -0400)]
20735: Update Go to 1.20.6.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20735: Update ruby and singularity.
Tom Clegg [Wed, 12 Jul 2023 15:35:48 +0000 (11:35 -0400)]
20735: Update ruby and singularity.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20735: Update to go 1.20.
Tom Clegg [Wed, 12 Jul 2023 15:34:45 +0000 (11:34 -0400)]
20735: Update to go 1.20.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20610: Moves code from local.params to its own common.sh file.
Lucas Di Pentima [Wed, 2 Aug 2023 13:38:07 +0000 (10:38 -0300)]
20610: Moves code from local.params to its own common.sh file.

provision & installer scripts now loads this new file that in turn loads
the params files.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months agoMerge branch '17244-cgroup2'
Tom Clegg [Wed, 2 Aug 2023 13:24:32 +0000 (09:24 -0400)]
Merge branch '17244-cgroup2'

refs #17244

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago17244: Fix doc link.
Tom Clegg [Wed, 2 Aug 2023 13:23:40 +0000 (09:23 -0400)]
17244: Fix doc link.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months agoMerge branch '20797-rhel8-packages'
Brett Smith [Wed, 2 Aug 2023 01:08:36 +0000 (21:08 -0400)]
Merge branch '20797-rhel8-packages'

Closes #20797.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Merge Red Hat package install instructions 20797-rhel8-packages
Brett Smith [Tue, 1 Aug 2023 18:07:32 +0000 (14:07 -0400)]
20797: Merge Red Hat package install instructions

For now, since the instructions are *exactly* the same across distros
and versions, this makes sense as a streamlining measure. If we add more
variation to the process in the future, it probably makes sense to split
these out again.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Run microdnf with --assumeyes
Brett Smith [Tue, 1 Aug 2023 16:06:59 +0000 (12:06 -0400)]
20797: Run microdnf with --assumeyes

`docker run --rm -ti rockylinux:8-minimal microdnf --help` says,

> The "--assumeyes" option is turned on by default.

However, that behavior apparently changes in a future version:
<https://github.com/rpm-software-management/microdnf/pull/97>,
microdnf commit ca778a0b34c5e15b5b0de1e0af0f8892280fc0c2.

Passing `--assumeyes` provides more consistency and will help keep the
code working in later releases.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Prefer dnf in PostgreSQL install instructions
Brett Smith [Tue, 1 Aug 2023 16:02:30 +0000 (12:02 -0400)]
20797: Prefer dnf in PostgreSQL install instructions

I believe this is the current preferred tool, and provides consistency
with other RH8 documentation.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago17244: Update docs for -cgroup-parent-subsystem flag.
Tom Clegg [Mon, 31 Jul 2023 19:52:50 +0000 (15:52 -0400)]
17244: Update docs for -cgroup-parent-subsystem flag.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago17244: Clarify log message.
Tom Clegg [Mon, 31 Jul 2023 19:33:18 +0000 (15:33 -0400)]
17244: Clarify log message.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20705: Warn if some stats are missing.
Tom Clegg [Mon, 31 Jul 2023 17:54:24 +0000 (13:54 -0400)]
20705: Warn if some stats are missing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

16 months ago20797: Update cross compilation detection logic
Brett Smith [Mon, 31 Jul 2023 15:37:38 +0000 (11:37 -0400)]
20797: Update cross compilation detection logic

The immediate bug that needs to be fixed is that we currently don't
support cross compilation on rocky8.

DRY up the code to try to make it easier to follow what's going on
here.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Note Alma/CentOS/Red Hat/Rocky 8 support in install docs
Brett Smith [Fri, 28 Jul 2023 19:53:45 +0000 (15:53 -0400)]
20797: Note Alma/CentOS/Red Hat/Rocky 8 support in install docs

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Copyedit "CentOS" for consistent capitalization
Brett Smith [Fri, 28 Jul 2023 19:12:44 +0000 (15:12 -0400)]
20797: Copyedit "CentOS" for consistent capitalization

I left the upgrade notes alone since they're more anchored in time, but
this should get the rest of the documentation.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Add basic documentation for how to add a new target distro
Brett Smith [Fri, 28 Jul 2023 18:51:18 +0000 (14:51 -0400)]
20797: Add basic documentation for how to add a new target distro

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Add basic intro documentation to build README
Brett Smith [Fri, 28 Jul 2023 18:41:15 +0000 (14:41 -0400)]
20797: Add basic intro documentation to build README

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Update bundler version in rocky8 Docker images
Brett Smith [Fri, 28 Jul 2023 19:32:32 +0000 (15:32 -0400)]
20797: Update bundler version in rocky8 Docker images

This updates the version to match our `Gemfile.lock` files and avoids a
warning.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Update for new createrepo package+executable name
Brett Smith [Fri, 28 Jul 2023 19:01:17 +0000 (15:01 -0400)]
20797: Update for new createrepo package+executable name

In Debian 12 the package is named createrepo-c and it installs
/usr/bin/createrepo_c. With this change, the build script prefers that
name, but still looks for and uses `createrepo`.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Streamline setup of Python build variables
Brett Smith [Fri, 28 Jul 2023 18:09:02 +0000 (14:09 -0400)]
20797: Streamline setup of Python build variables

Instead of setting Debian defaults and overriding them,

1. set global defaults
2. set target-specific variables
3. set deb/rpm-specific defaults as needed

I hope this makes clearer what's actually a target-specific variable
vs. a regular difference between Debian and Red Hat, and so makes it
easier to add targets in the future.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20797: Add initial support for rocky8 package builds
Brett Smith [Fri, 28 Jul 2023 17:48:08 +0000 (13:48 -0400)]
20797: Add initial support for rocky8 package builds

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20610: Updates docs to reflect changes on arvados.sls configuration.
Lucas Di Pentima [Fri, 28 Jul 2023 14:43:39 +0000 (11:43 -0300)]
20610: Updates docs to reflect changes on arvados.sls configuration.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Adds documentation for load-balancing & rolling upgrades.
Lucas Di Pentima [Fri, 28 Jul 2023 14:32:07 +0000 (11:32 -0300)]
20610: Adds documentation for load-balancing & rolling upgrades.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Removes the need to manually set up the ENABLE_BALANCER variable.
Lucas Di Pentima [Fri, 28 Jul 2023 13:48:40 +0000 (10:48 -0300)]
20610: Removes the need to manually set up the ENABLE_BALANCER variable.

Having the role->nodes map allows us to simplify manual configuration.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Fixes deployment order to avoid failures.
Lucas Di Pentima [Thu, 27 Jul 2023 21:17:09 +0000 (18:17 -0300)]
20610: Fixes deployment order to avoid failures.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Allows disabling backend controllers for rolling updates.
Lucas Di Pentima [Thu, 27 Jul 2023 19:32:36 +0000 (16:32 -0300)]
20610: Allows disabling backend controllers for rolling updates.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Restricts backends' HTTP access by nginx rules.
Lucas Di Pentima [Thu, 27 Jul 2023 18:39:56 +0000 (15:39 -0300)]
20610: Restricts backends' HTTP access by nginx rules.

I think this is better than implementing those controls through security
groups via Terraform, because the node's role information is already available
on the salt code.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Fixes prometheus configuration to track the controller nodes.
Lucas Di Pentima [Wed, 26 Jul 2023 22:43:57 +0000 (19:43 -0300)]
20610: Fixes prometheus configuration to track the controller nodes.

Also, fixes a-d-c config, as it was also assumed to be on the controller node.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Removes duplication on balancer & backend declarations.
Lucas Di Pentima [Wed, 26 Jul 2023 21:06:41 +0000 (18:06 -0300)]
20610: Removes duplication on balancer & backend declarations.

Instead of having to explicitly declare those, just use the ROLES map
to get the balancer and controller nodes.
Also, explicitly gives PG access to Keep-Balance instead of assuming it's
installed on the same node as the controller.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Adds node list & role->nodes map variables.
Lucas Di Pentima [Wed, 26 Jul 2023 19:57:09 +0000 (16:57 -0300)]
20610: Adds node list & role->nodes map variables.

Uses the node list variable to configure prometheus node exporter from the
configuration in local.params.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Fixes a template variable usage.
Lucas Di Pentima [Wed, 26 Jul 2023 13:53:31 +0000 (10:53 -0300)]
20610: Fixes a template variable usage.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months ago20610: Removes the need to directly edit the arvados.sls pillar.
Lucas Di Pentima [Wed, 26 Jul 2023 13:51:25 +0000 (10:51 -0300)]
20610: Removes the need to directly edit the arvados.sls pillar.

All params are now in local.params.* files.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

16 months agoMerge branch '20543-pip-install-virtualenv'
Brett Smith [Wed, 26 Jul 2023 13:40:39 +0000 (09:40 -0400)]
Merge branch '20543-pip-install-virtualenv'

Closes #20543.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

16 months ago20543: Recommend users install Python packages in a virtualenv
Brett Smith [Fri, 7 Jul 2023 19:44:03 +0000 (15:44 -0400)]
20543: Recommend users install Python packages in a virtualenv

`pip install --user` doesn't work on distributions that have adopted
PEP 668, including Debian 12+ and Ubuntu 23.04+.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20343-regexp-prefixes'
Brett Smith [Mon, 24 Jul 2023 14:53:13 +0000 (10:53 -0400)]
Merge branch '20343-regexp-prefixes'

Closes #20343.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago17244: Use crunch-run's cgroup, not init's, for "host" stats.
Tom Clegg [Fri, 21 Jul 2023 23:05:33 +0000 (19:05 -0400)]
17244: Use crunch-run's cgroup, not init's, for "host" stats.

This gets much closer to "everything on the host" with minimal
effort.

Tracking the furthest ancestor of crunch-run's cgroup might be even
better, but will require additional support in crunchstat.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Test findCgroup on cgroup files in crunchstat testdata.
Tom Clegg [Fri, 21 Jul 2023 13:53:34 +0000 (09:53 -0400)]
17244: Test findCgroup on cgroup files in crunchstat testdata.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20726-s3-session-release'
Tom Clegg [Fri, 21 Jul 2023 13:21:41 +0000 (09:21 -0400)]
Merge branch '20726-s3-session-release'

refs #20726

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20726: Fix unreleased session in s3 handler. 20726-s3-session-release
Tom Clegg [Thu, 20 Jul 2023 15:26:51 +0000 (11:26 -0400)]
20726: Fix unreleased session in s3 handler.

Caused subsequent requests with the same token to hang after reaching
expiry time.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Fix singularity reporting host stats as container stats.
Tom Clegg [Wed, 19 Jul 2023 14:40:07 +0000 (10:40 -0400)]
17244: Fix singularity reporting host stats as container stats.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Make -cgroup-parent-subsystem=X work in cgroups v2.
Tom Clegg [Tue, 18 Jul 2023 20:28:30 +0000 (16:28 -0400)]
17244: Make -cgroup-parent-subsystem=X work in cgroups v2.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Refactor crunchstat to use cgroup unified/hybrid modes.
Tom Clegg [Tue, 11 Jul 2023 14:03:58 +0000 (10:03 -0400)]
17244: Refactor crunchstat to use cgroup unified/hybrid modes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago17244: Remove crunchstat debian package.
Tom Clegg [Wed, 19 Jul 2023 04:54:42 +0000 (00:54 -0400)]
17244: Remove crunchstat debian package.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20731-user-guide-wb2' refs #20731
Peter Amstutz [Tue, 18 Jul 2023 14:20:50 +0000 (10:20 -0400)]
Merge branch '20731-user-guide-wb2'  refs #20731

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months agoMerge branch '20726-s3list-pages'
Tom Clegg [Thu, 13 Jul 2023 20:48:04 +0000 (16:48 -0400)]
Merge branch '20726-s3list-pages'

refs #20726

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20726: Fix traversing projects/collections that precede page marker. 20726-s3list-pages
Tom Clegg [Wed, 12 Jul 2023 04:19:08 +0000 (00:19 -0400)]
20726: Fix traversing projects/collections that precede page marker.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20731: Make some text changes & reorganization based on workbench 2 updates 20731-user-guide-wb2
Peter Amstutz [Tue, 11 Jul 2023 21:06:47 +0000 (17:06 -0400)]
20731: Make some text changes & reorganization based on workbench 2 updates

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

17 months ago20726: Fix ListObjects[V2] pages duplicating last item on next page.
Tom Clegg [Tue, 11 Jul 2023 19:25:22 +0000 (15:25 -0400)]
20726: Fix ListObjects[V2] pages duplicating last item on next page.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20726: Update s3ListObjects paging test.
Tom Clegg [Tue, 11 Jul 2023 17:38:38 +0000 (13:38 -0400)]
20726: Update s3ListObjects paging test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20667-maxsuper-atquota'
Tom Clegg [Tue, 11 Jul 2023 17:39:20 +0000 (13:39 -0400)]
Merge branch '20667-maxsuper-atquota'

fixes #20667

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20684-user-install-doc-fixes'
Brett Smith [Mon, 10 Jul 2023 15:56:21 +0000 (11:56 -0400)]
Merge branch '20684-user-install-doc-fixes'

Closes #20684.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Modernize pycurl dependency discussion
Brett Smith [Mon, 10 Jul 2023 15:50:56 +0000 (11:50 -0400)]
20684: Modernize pycurl dependency discussion

These instructions should be good for Debian 10+ and I'm assuming
corresponding Ubuntu too.

* Update libssl package name
* Clarify the command is good for multiple distros
* Prefer apt over apt-get

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Simplify curl dependency background
Brett Smith [Mon, 10 Jul 2023 15:48:21 +0000 (11:48 -0400)]
20684: Simplify curl dependency background

Specifying that the "Python SDK" depends on curl is needless detail
here.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Note tutorial expectations for all Crunch CLI tools 20684-user-install-doc-fixes
Brett Smith [Fri, 7 Jul 2023 21:19:13 +0000 (17:19 -0400)]
20684: Note tutorial expectations for all Crunch CLI tools

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Note crunchstat-summary can be installed in a virtualenv
Brett Smith [Fri, 7 Jul 2023 21:18:15 +0000 (17:18 -0400)]
20684: Note crunchstat-summary can be installed in a virtualenv

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Copyedit tutorial expectations include
Brett Smith [Fri, 7 Jul 2023 21:03:46 +0000 (17:03 -0400)]
20684: Copyedit tutorial expectations include

* Fix run-on sentence
* Clean up punctuation spacing

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Expand "Getting started at the command line" documentation
Brett Smith [Fri, 7 Jul 2023 20:57:38 +0000 (16:57 -0400)]
20684: Expand "Getting started at the command line" documentation

* Add more structure to the page
* Be explicit that we usually recommend using a VM
* Highlight that using our package repository is the simplest way to
  install tools
* List all the tools you can install, with more details

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months ago20684: Clean up FUSE references in a-c-r install instructions
Brett Smith [Fri, 7 Jul 2023 20:27:19 +0000 (16:27 -0400)]
20684: Clean up FUSE references in a-c-r install instructions

It looks like this page started as a copy of the corresponding FUSE
page, and we didn't update all the references.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17 months agoMerge branch '20559-dav-concurrent-writes'
Tom Clegg [Fri, 7 Jul 2023 13:49:24 +0000 (09:49 -0400)]
Merge branch '20559-dav-concurrent-writes'

refs #20559

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoFixes salt bootstrapping on nodes by the installer. Refs #20665
Lucas Di Pentima [Thu, 6 Jul 2023 19:34:43 +0000 (16:34 -0300)]
Fixes salt bootstrapping on nodes by the installer. Refs #20665

Salt v3004 is no more regarded as 'stable', so the boostrapping script started
to fail. Asking for 'old-stable' fixes this, at least for some time.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months agoMerge branch '20722-docker-upgrade'
Tom Clegg [Thu, 6 Jul 2023 18:01:12 +0000 (14:01 -0400)]
Merge branch '20722-docker-upgrade'

fixes #20722

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months ago20722: Update ContainerCreate usage for new docker client library. 20722-docker-upgrade
Tom Clegg [Thu, 6 Jul 2023 15:54:40 +0000 (11:54 -0400)]
20722: Update ContainerCreate usage for new docker client library.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17 months agoMerge branch '20707-aws-compute-ami-cgroupsv1'. Closes #20707
Lucas Di Pentima [Thu, 6 Jul 2023 15:15:01 +0000 (12:15 -0300)]
Merge branch '20707-aws-compute-ami-cgroupsv1'. Closes #20707

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20722: Cleans up dependencies with 'go mod tidy'
Lucas Di Pentima [Thu, 6 Jul 2023 14:52:53 +0000 (11:52 -0300)]
20722: Cleans up dependencies with 'go mod tidy'

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20722: Upgrades github.com/docker/docker to latest version.
Lucas Di Pentima [Thu, 6 Jul 2023 14:51:40 +0000 (11:51 -0300)]
20722: Upgrades github.com/docker/docker to latest version.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months agoMerge branch '20718-grpc-upgrade'. Closes #20718
Lucas Di Pentima [Thu, 6 Jul 2023 14:26:05 +0000 (11:26 -0300)]
Merge branch '20718-grpc-upgrade'. Closes #20718

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20718: Cleans up by running "go mod tidy -compat=1.17" 20718-grpc-upgrade
Lucas Di Pentima [Thu, 6 Jul 2023 13:41:50 +0000 (10:41 -0300)]
20718: Cleans up by running "go mod tidy -compat=1.17"

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20718: Upgrades google.golang.org/grpc to 1.56.1
Lucas Di Pentima [Thu, 6 Jul 2023 13:36:31 +0000 (10:36 -0300)]
20718: Upgrades google.golang.org/grpc to 1.56.1

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

17 months ago20343: Define Python regexps with raw strings 20343-regexp-prefixes
Brett Smith [Wed, 5 Jul 2023 20:58:07 +0000 (16:58 -0400)]
20343: Define Python regexps with raw strings

Prevent the DeprecationWarning added to Python 3.6 for unrecognized
string escape sequences. Found candidates by running:

    git grep --line-number -E '\bre\.\w+\([^r]*\\'

Most of the changes just add the raw string prefix `r` to these regexps,
but I did make some other readability improvements while I was at it,
including switching from regexps to plain string test methods where
appropriate.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>