arvados.git
18 months agoMerge branch '20548-installer-deploy' refs #20548
Peter Amstutz [Wed, 7 Jun 2023 17:51:21 +0000 (13:51 -0400)]
Merge branch '20548-installer-deploy' refs #20548

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoInitialize rcode variable, refs #20611
Peter Amstutz [Mon, 5 Jun 2023 22:10:07 +0000 (18:10 -0400)]
Initialize rcode variable, refs #20611

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoMerge branch '20522-load-dispatch-priv-key'
Tom Clegg [Mon, 5 Jun 2023 19:16:11 +0000 (15:16 -0400)]
Merge branch '20522-load-dispatch-priv-key'

closes #20522

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoMerge branch '20594-scaling-nginx-settings'. Closes #20594
Lucas Di Pentima [Mon, 5 Jun 2023 18:47:32 +0000 (15:47 -0300)]
Merge branch '20594-scaling-nginx-settings'. Closes #20594

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

18 months ago20548: Emit the git commit in the deploy log. 20548-installer-deploy
Peter Amstutz [Fri, 2 Jun 2023 20:37:53 +0000 (16:37 -0400)]
20548: Emit the git commit in the deploy log.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20548: Rework git deployment a bit
Peter Amstutz [Fri, 2 Jun 2023 20:18:07 +0000 (16:18 -0400)]
20548: Rework git deployment a bit

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoMerge branch '20531-cwl-log-tail' refs #20531
Peter Amstutz [Fri, 2 Jun 2023 17:59:52 +0000 (13:59 -0400)]
Merge branch '20531-cwl-log-tail' refs #20531

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20531: Add --reinstall option to test_with_arvbox.sh 20531-cwl-log-tail
Peter Amstutz [Fri, 2 Jun 2023 17:39:04 +0000 (13:39 -0400)]
20531: Add --reinstall option to test_with_arvbox.sh

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoMerge branch '20541-less-fetching-mounts'
Tom Clegg [Fri, 2 Jun 2023 16:11:02 +0000 (12:11 -0400)]
Merge branch '20541-less-fetching-mounts'

fixes #20541

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20531: Bump cwltool and schema-salad versions
Peter Amstutz [Fri, 2 Jun 2023 15:21:35 +0000 (11:21 -0400)]
20531: Bump cwltool and schema-salad versions

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20531: Suppress googleapiclient retry warning, better download error
Peter Amstutz [Fri, 2 Jun 2023 15:19:25 +0000 (11:19 -0400)]
20531: Suppress googleapiclient retry warning, better download error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoIncrease WatchdogInterval and WatchdogThreshold
Peter Amstutz [Fri, 26 May 2023 19:26:12 +0000 (15:26 -0400)]
Increase WatchdogInterval and WatchdogThreshold

Quick workaround for customer, we should add these to the config file.

refs #20595

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoMerge branch '20540-crunch-run-retry'
Tom Clegg [Thu, 1 Jun 2023 19:13:57 +0000 (15:13 -0400)]
Merge branch '20540-crunch-run-retry'

fixes #20540

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20531: Remove urllib3 pin, add cwltest pin.
Peter Amstutz [Thu, 1 Jun 2023 18:32:47 +0000 (14:32 -0400)]
20531: Remove urllib3 pin, add cwltest pin.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20540: Increase crunch-run API call timeouts to 10m. 20540-crunch-run-retry
Tom Clegg [Thu, 1 Jun 2023 17:54:41 +0000 (13:54 -0400)]
20540: Increase crunch-run API call timeouts to 10m.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoMerge branch '20545-drop-if-none-match'
Brett Smith [Thu, 1 Jun 2023 15:53:27 +0000 (11:53 -0400)]
Merge branch '20545-drop-if-none-match'

Closes #20545.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

18 months agoMerge branch '20433-crunch-log-zero-bytes-job'
Brett Smith [Thu, 1 Jun 2023 15:53:23 +0000 (11:53 -0400)]
Merge branch '20433-crunch-log-zero-bytes-job'

Closes #20433.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

18 months agoMerge branch '20561-file-copy-logging' refs #20561
Peter Amstutz [Thu, 1 Jun 2023 14:23:41 +0000 (10:23 -0400)]
Merge branch '20561-file-copy-logging' refs #20561

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20545: Drop If-None-Match from requests proxied by controller 20545-drop-if-none-match
Brett Smith [Thu, 1 Jun 2023 13:51:29 +0000 (09:51 -0400)]
20545: Drop If-None-Match from requests proxied by controller

See comments for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

18 months agoMerge branch '20485-optional-deploy-ssh-key'
Tom Clegg [Thu, 1 Jun 2023 13:33:48 +0000 (09:33 -0400)]
Merge branch '20485-optional-deploy-ssh-key'

closes #20485

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20433: Document that LimitLogBytesPerJob=0 disables live logging 20433-crunch-log-zero-bytes-job
Brett Smith [Wed, 31 May 2023 20:46:36 +0000 (16:46 -0400)]
20433: Document that LimitLogBytesPerJob=0 disables live logging

Except for the throttle message, which is a nice user affordance.

Add a test for this specific case.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

18 months ago20485: Test handling of nil publickey argument. 20485-optional-deploy-ssh-key
Tom Clegg [Wed, 31 May 2023 19:55:33 +0000 (15:55 -0400)]
20485: Test handling of nil publickey argument.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20540: Ensure delay>0 at attempt>0, even if requested min=0.
Tom Clegg [Wed, 31 May 2023 19:24:30 +0000 (15:24 -0400)]
20540: Ensure delay>0 at attempt>0, even if requested min=0.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20541: Auto unselect fields if response body will not be read. 20541-less-fetching-mounts
Tom Clegg [Wed, 31 May 2023 13:59:30 +0000 (09:59 -0400)]
20541: Auto unselect fields if response body will not be read.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20540: Implement nearly-full jitter for exponential backoff.
Tom Clegg [Tue, 30 May 2023 20:54:03 +0000 (16:54 -0400)]
20540: Implement nearly-full jitter for exponential backoff.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20561: Don't fail when inspecting a file outside the output directory 20561-file-copy-logging
Peter Amstutz [Tue, 30 May 2023 18:58:13 +0000 (14:58 -0400)]
20561: Don't fail when inspecting a file outside the output directory

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20541: Unselect unneeded fields in API calls from crunch-run.
Tom Clegg [Tue, 30 May 2023 14:33:56 +0000 (10:33 -0400)]
20541: Unselect unneeded fields in API calls from crunch-run.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoMerge branch '20229-doc-inspect-requests'
Tom Clegg [Mon, 29 May 2023 19:43:17 +0000 (15:43 -0400)]
Merge branch '20229-doc-inspect-requests'

closes #20229

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20229: Improve formatting.
Tom Clegg [Mon, 29 May 2023 19:42:51 +0000 (15:42 -0400)]
20229: Improve formatting.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20485: Option to skip automatic SSH key deployment on cloud VMs.
Tom Clegg [Mon, 29 May 2023 19:24:52 +0000 (15:24 -0400)]
20485: Option to skip automatic SSH key deployment on cloud VMs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20540: Use arvados.Client for arvadosclient.ArvadosClient reqs.
Tom Clegg [Mon, 29 May 2023 19:11:27 +0000 (15:11 -0400)]
20540: Use arvados.Client for arvadosclient.ArvadosClient reqs.

Caller-specified Retries is used as a timeout in minutes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoMerge branch '20520-instance-init-command'
Tom Clegg [Fri, 26 May 2023 19:30:25 +0000 (15:30 -0400)]
Merge branch '20520-instance-init-command'

refs #20520

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20540: Remove arvadosclient usage in copier and git_tree.
Tom Clegg [Fri, 26 May 2023 19:22:01 +0000 (15:22 -0400)]
20540: Remove arvadosclient usage in copier and git_tree.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20595: Scales nginx settings depending on max concurrent requests config. 20594-scaling-nginx-settings
Lucas Di Pentima [Fri, 26 May 2023 18:45:31 +0000 (15:45 -0300)]
20595: Scales nginx settings depending on max concurrent requests config.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

18 months ago20520: Fix broken link. 20520-instance-init-command
Tom Clegg [Fri, 26 May 2023 14:25:36 +0000 (10:25 -0400)]
20520: Fix broken link.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20520: Fix unreleased mutex on error importing SSH key.
Tom Clegg [Fri, 26 May 2023 14:10:09 +0000 (10:10 -0400)]
20520: Fix unreleased mutex on error importing SSH key.

Any error listing or importing keys (which, luckily, only happens the
first time a arvados-dispatch-cloud process creates a new instance)
would cause Create() call to fail, and cause all subsequent Create()
calls to hang forever until the service is restarted.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoMerge branch '20511-aborted-boot'
Tom Clegg [Fri, 26 May 2023 14:29:16 +0000 (10:29 -0400)]
Merge branch '20511-aborted-boot'

refs #20511

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoMerge branch '20474-installer-api-reqs-limit'. Closes #20474
Lucas Di Pentima [Fri, 26 May 2023 13:42:20 +0000 (10:42 -0300)]
Merge branch '20474-installer-api-reqs-limit'. Closes #20474

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

18 months ago20511: Fix allowing too many supervisor processes. 20511-aborted-boot
Tom Clegg [Thu, 25 May 2023 21:45:50 +0000 (17:45 -0400)]
20511: Fix allowing too many supervisor processes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20511: Don't shutdown excess instances just because MaxSupervisors.
Tom Clegg [Thu, 25 May 2023 21:10:48 +0000 (17:10 -0400)]
20511: Don't shutdown excess instances just because MaxSupervisors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20531: Add toplevel dependency to solve install error
Peter Amstutz [Thu, 25 May 2023 20:12:39 +0000 (16:12 -0400)]
20531: Add toplevel dependency to solve install error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20531: Add an extra newline to make the error section easier to see
Peter Amstutz [Thu, 25 May 2023 19:57:56 +0000 (15:57 -0400)]
20531: Add an extra newline to make the error section easier to see

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20474: Updates documentation stating that RailsAPI supports /metrics endpoint. 20474-installer-api-reqs-limit
Lucas Di Pentima [Thu, 25 May 2023 18:58:53 +0000 (15:58 -0300)]
20474: Updates documentation stating that RailsAPI supports /metrics endpoint.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

18 months ago20474: Changes RailsAPI queue size to be 10% more than controller's.
Lucas Di Pentima [Thu, 25 May 2023 18:52:32 +0000 (15:52 -0300)]
20474: Changes RailsAPI queue size to be 10% more than controller's.

Also, adds comment on config file clarifying why we're setting up this
increased value.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

18 months ago20520: Add test for custom init command.
Tom Clegg [Thu, 25 May 2023 18:42:15 +0000 (14:42 -0400)]
20520: Add test for custom init command.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20511: Fix unclosed response body in test.
Tom Clegg [Thu, 25 May 2023 17:57:58 +0000 (13:57 -0400)]
20511: Fix unclosed response body in test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20511: Don't shutdown/unlock based on dynamic maxConcurrency.
Tom Clegg [Thu, 25 May 2023 17:43:34 +0000 (13:43 -0400)]
20511: Don't shutdown/unlock based on dynamic maxConcurrency.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20531: don't print crunch-run log when the error is with the tool
Peter Amstutz [Thu, 25 May 2023 17:34:13 +0000 (13:34 -0400)]
20531: don't print crunch-run log when the error is with the tool

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20561: crunch-run logs files/directories propagated from keep
Peter Amstutz [Thu, 25 May 2023 17:30:36 +0000 (13:30 -0400)]
20561: crunch-run logs files/directories propagated from keep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20511: Limit server-to-server client to 1/4 of API req capacity.
Tom Clegg [Thu, 25 May 2023 15:25:02 +0000 (11:25 -0400)]
20511: Limit server-to-server client to 1/4 of API req capacity.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20511: Fix slow-expansion logic.
Tom Clegg [Thu, 25 May 2023 14:33:53 +0000 (10:33 -0400)]
20511: Fix slow-expansion logic.

Limit was always being raised to 2x known-working, instead of
min(+10%, 2x known-working) as intended.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20511: Start with 8 concurrent outgoing API calls, not unlimited.
Tom Clegg [Thu, 25 May 2023 14:12:40 +0000 (10:12 -0400)]
20511: Start with 8 concurrent outgoing API calls, not unlimited.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20520: Add Containers.CloudVMs.InstanceInitCommand config.
Tom Clegg [Tue, 23 May 2023 21:40:06 +0000 (17:40 -0400)]
20520: Add Containers.CloudVMs.InstanceInitCommand config.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months ago20522: Load dispatch key from file if configured as file:///... 20522-load-dispatch-priv-key
Tom Clegg [Tue, 23 May 2023 17:56:06 +0000 (13:56 -0400)]
20522: Load dispatch key from file if configured as file:///...

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

18 months agoAdd 2.6.2 to the upgrading notes refs #20393
Peter Amstutz [Mon, 22 May 2023 21:16:12 +0000 (17:16 -0400)]
Add 2.6.2 to the upgrading notes refs #20393

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months ago20474: Adds 5 to controller's request queue size for RailsAPI.
Lucas Di Pentima [Mon, 22 May 2023 21:13:53 +0000 (18:13 -0300)]
20474: Adds 5 to controller's request queue size for RailsAPI.

This size difference would allow extra requests (like metrics) to happen
even on heavily loaded clusters.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

18 months agoMerge branch '20529-container-deadlocks' refs #20529
Peter Amstutz [Mon, 22 May 2023 15:33:46 +0000 (11:33 -0400)]
Merge branch '20529-container-deadlocks' refs #20529

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

18 months agoMerge branch '20527-group-contents-select-doc'
Brett Smith [Mon, 22 May 2023 14:47:29 +0000 (10:47 -0400)]
Merge branch '20527-group-contents-select-doc'

Closes #20527.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

19 months ago20529: Lock direct parent containers on updates 20529-container-deadlocks
Peter Amstutz [Fri, 19 May 2023 19:01:34 +0000 (15:01 -0400)]
20529: Lock direct parent containers on updates

This is because cost is propagated up to the parent container on
container completion.

Handle case of resource attrs at top level when deciding whether to
lock or not (resource attrs appear as strings, not symbols)

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

19 months agoMerge branch '20482-installer-improvements'. Closes #20482
Lucas Di Pentima [Fri, 19 May 2023 18:11:57 +0000 (15:11 -0300)]
Merge branch '20482-installer-improvements'. Closes #20482

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: For some reason, symlinks don't seem to be ignored for copyright checks. 20482-installer-improvements
Lucas Di Pentima [Fri, 19 May 2023 16:16:56 +0000 (13:16 -0300)]
20482: For some reason, symlinks don't seem to be ignored for copyright checks.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Updates the upgrade notice to include the DOMAIN envvar at local.params.
Lucas Di Pentima [Fri, 19 May 2023 15:06:33 +0000 (12:06 -0300)]
20482: Updates the upgrade notice to include the DOMAIN envvar at local.params.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Updates installer's documentation to reflect latest changes.
Lucas Di Pentima [Thu, 18 May 2023 20:36:10 +0000 (17:36 -0300)]
20482: Updates installer's documentation to reflect latest changes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Code cleanup for readability.
Lucas Di Pentima [Thu, 18 May 2023 20:35:17 +0000 (17:35 -0300)]
20482: Code cleanup for readability.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Re-exports VPC's CIDR.
Lucas Di Pentima [Thu, 18 May 2023 20:31:37 +0000 (17:31 -0300)]
20482: Re-exports VPC's CIDR.

Previously exported as 'vpc_cidr' and removed when preexisting vpc usage
was added. This config data is used on local.params and was mentioned on the
documentation page.
Now, it's exported as 'cluster_int_cidr' and its value is requested from AWS
so that we get the correct one whether the vpc was just created or a
previously existing one is being in use.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Overrides shellinabox config templates to fix the domain name usage.
Lucas Di Pentima [Fri, 19 May 2023 14:57:03 +0000 (11:57 -0300)]
20482: Overrides shellinabox config templates to fix the domain name usage.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows the cluster operator to use an arbitrary domain.
Lucas Di Pentima [Thu, 18 May 2023 14:22:10 +0000 (11:22 -0300)]
20482: Allows the cluster operator to use an arbitrary domain.

Instead of making domains like cluster_prefix.domain mandatory, let the site
admin to select whichever domain they need for the deployment.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20527: Document the select argument of the groups contents API method 20527-group-contents-select-doc
Brett Smith [Fri, 19 May 2023 14:39:03 +0000 (10:39 -0400)]
20527: Document the select argument of the groups contents API method

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

19 months agoMerge branch '12684-pysdk-auto-retry'
Brett Smith [Thu, 18 May 2023 15:40:55 +0000 (11:40 -0400)]
Merge branch '12684-pysdk-auto-retry'

Closes #12684.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

19 months ago12684: Test that plain 403 responses are not retried 12684-pysdk-auto-retry
Brett Smith [Thu, 18 May 2023 12:48:13 +0000 (08:48 -0400)]
12684: Test that plain 403 responses are not retried

Requested in review.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

19 months ago12684: Stop retrying 422 responses in PySDK
Brett Smith [Thu, 18 May 2023 12:36:56 +0000 (08:36 -0400)]
12684: Stop retrying 422 responses in PySDK

The original motivation for this was to retry when the API server was
having database connectivity problems. The feeling eight years later is
that things have changed enough that, on balance, this isn't worth
retrying anymore.

I don't think this will have any real impact on current Arvados
software. In the main branch as I write this,
`check_http_response_status` only gets called in five places. Three of
those are in the main `arvados` module for job and task utilities, which
presumably nobody is using anymore. The other two talk to Keep, which
only returns 422 for hash mismatches, where a retry will definitely
never succeed.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

19 months agoMerge branch '20457-queue-churn'
Tom Clegg [Wed, 17 May 2023 15:19:35 +0000 (11:19 -0400)]
Merge branch '20457-queue-churn'

refs #20457
refs #20511

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

19 months agoMerge branch '20482-further-terraform-improvements'. Refs #20482
Lucas Di Pentima [Wed, 17 May 2023 13:28:16 +0000 (10:28 -0300)]
Merge branch '20482-further-terraform-improvements'. Refs #20482

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months agoMerge branch '20325-go-docker-distribution-upgrade2'. Refs #20325
Lucas Di Pentima [Wed, 17 May 2023 13:20:54 +0000 (10:20 -0300)]
Merge branch '20325-go-docker-distribution-upgrade2'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20325: Upgrades github.com/docker/distribution module. 20325-go-docker-distribution-upgrade2
Lucas Di Pentima [Tue, 16 May 2023 18:43:48 +0000 (15:43 -0300)]
20325: Upgrades github.com/docker/distribution module.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20457: Don't displace locked containers with queued containers. 20457-queue-churn
Tom Clegg [Tue, 16 May 2023 13:45:44 +0000 (09:45 -0400)]
20457: Don't displace locked containers with queued containers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

19 months ago20229: Add "inspecting active requests" page. 20229-doc-inspect-requests
Tom Clegg [Mon, 15 May 2023 20:39:16 +0000 (16:39 -0400)]
20229: Add "inspecting active requests" page.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

19 months ago20482: Makes installer.sh compatible with older (Ubuntu 18.04) git versions. 20482-further-terraform-improvements
Lucas Di Pentima [Mon, 15 May 2023 13:24:00 +0000 (10:24 -0300)]
20482: Makes installer.sh compatible with older (Ubuntu 18.04) git versions.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Don't create S3 endpoint if using a preexisting VPC.
Lucas Di Pentima [Mon, 15 May 2023 13:21:29 +0000 (10:21 -0300)]
20482: Don't create S3 endpoint if using a preexisting VPC.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Fixes formatting.
Lucas Di Pentima [Mon, 15 May 2023 13:20:52 +0000 (10:20 -0300)]
20482: Fixes formatting.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months agoRefine PySDK collection walk recipe
Brett Smith [Thu, 11 May 2023 15:53:41 +0000 (11:53 -0400)]
Refine PySDK collection walk recipe

Use PurePosixPath to clarify that we're strictly doing path manipulation.
(It will also behave better on Windows, although I'm not sure if the SDK
itself is Windows-ready yet.)

Keep Path objects in the queue to reduce local state.

No issue #

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

19 months agoMerge branch '20482-terraform-custom-configs'. Closes #20482
Lucas Di Pentima [Thu, 11 May 2023 15:48:33 +0000 (12:48 -0300)]
Merge branch '20482-terraform-custom-configs'. Closes #20482

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Improves upgrade note's title. 20482-terraform-custom-configs
Lucas Di Pentima [Thu, 11 May 2023 15:47:28 +0000 (12:47 -0300)]
20482: Improves upgrade note's title.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Improves file/dir creation on user_data script.
Lucas Di Pentima [Thu, 11 May 2023 14:52:36 +0000 (11:52 -0300)]
20482: Improves file/dir creation on user_data script.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Adds upgrade notes on domain_name variable changes.
Lucas Di Pentima [Thu, 11 May 2023 14:49:46 +0000 (11:49 -0300)]
20482: Adds upgrade notes on domain_name variable changes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Pins terraform and AWS provider versions.
Lucas Di Pentima [Thu, 11 May 2023 14:15:11 +0000 (11:15 -0300)]
20482: Pins terraform and AWS provider versions.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows the site admin to specify instance volume sizes per node.
Lucas Di Pentima [Thu, 11 May 2023 13:03:20 +0000 (10:03 -0300)]
20482: Allows the site admin to specify instance volume sizes per node.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Improves readability of instance profile assignment code.
Lucas Di Pentima [Thu, 11 May 2023 12:48:07 +0000 (09:48 -0300)]
20482: Improves readability of instance profile assignment code.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows setting instance types per service node.
Lucas Di Pentima [Thu, 11 May 2023 12:40:05 +0000 (09:40 -0300)]
20482: Allows setting instance types per service node.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Fixes PassRole's target to point to the newly created compute node role.
Lucas Di Pentima [Wed, 10 May 2023 22:09:39 +0000 (19:09 -0300)]
20482: Fixes PassRole's target to point to the newly created compute node role.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Sets output as sensitive to make Terraform happy.
Lucas Di Pentima [Wed, 10 May 2023 21:42:08 +0000 (18:42 -0300)]
20482: Sets output as sensitive to make Terraform happy.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Adds proper compute node instance profile instead of using keepstore's.
Lucas Di Pentima [Wed, 10 May 2023 20:38:48 +0000 (17:38 -0300)]
20482: Adds proper compute node instance profile instead of using keepstore's.

We first used keepstore's instance profile because compute nodes run a local
keepstore now.
We also need to give compute nodes permission to change resources related to
the EBS Autoscaler.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Extracts DNS aliases map as configurable variables. 20482-terraform-private-only-infra
Lucas Di Pentima [Wed, 10 May 2023 20:05:55 +0000 (17:05 -0300)]
20482: Extracts DNS aliases map as configurable variables.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Extracts the private IP addr map as configurable variables.
Lucas Di Pentima [Wed, 10 May 2023 19:51:55 +0000 (16:51 -0300)]
20482: Extracts the private IP addr map as configurable variables.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows the site admin to customize tags applied to every resource.
Lucas Di Pentima [Wed, 10 May 2023 19:29:10 +0000 (16:29 -0300)]
20482: Allows the site admin to customize tags applied to every resource.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows the site admin to specify a custom AMI for the nodes.
Lucas Di Pentima [Wed, 10 May 2023 18:25:47 +0000 (15:25 -0300)]
20482: Allows the site admin to specify a custom AMI for the nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows the admin to specify the user for deployment.
Lucas Di Pentima [Wed, 10 May 2023 17:45:40 +0000 (14:45 -0300)]
20482: Allows the admin to specify the user for deployment.

Also, removes the need to use AWS key pairs, by directly storing the SSH
pubkey in the user's ~/.ssh/ directory via the user-data script.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months agoMerge branch '20325-jquery-rails-upgrade'. Refs #20325
Lucas Di Pentima [Tue, 9 May 2023 19:49:09 +0000 (16:49 -0300)]
Merge branch '20325-jquery-rails-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20325: Upgrades jquery-rails and dependencies on RailsAPI & Workbench1. 20325-jquery-rails-upgrade
Lucas Di Pentima [Tue, 9 May 2023 18:47:23 +0000 (15:47 -0300)]
20325: Upgrades jquery-rails and dependencies on RailsAPI & Workbench1.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

19 months ago20482: Allows deploying on known VPC & subnets.
Lucas Di Pentima [Tue, 9 May 2023 00:10:09 +0000 (21:10 -0300)]
20482: Allows deploying on known VPC & subnets.

Instead of creating everything new, the admin now has the option to deploy
the resources on preexisting networks.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>