arvados.git
11 months ago12684: Test that plain 403 responses are not retried 12684-pysdk-auto-retry
Brett Smith [Thu, 18 May 2023 12:48:13 +0000 (08:48 -0400)]
12684: Test that plain 403 responses are not retried

Requested in review.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684: Stop retrying 422 responses in PySDK
Brett Smith [Thu, 18 May 2023 12:36:56 +0000 (08:36 -0400)]
12684: Stop retrying 422 responses in PySDK

The original motivation for this was to retry when the API server was
having database connectivity problems. The feeling eight years later is
that things have changed enough that, on balance, this isn't worth
retrying anymore.

I don't think this will have any real impact on current Arvados
software. In the main branch as I write this,
`check_http_response_status` only gets called in five places. Three of
those are in the main `arvados` module for job and task utilities, which
presumably nobody is using anymore. The other two talk to Keep, which
only returns 422 for hash mismatches, where a retry will definitely
never succeed.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684: Use mock services in arvfile sparse write tests
Brett Smith [Tue, 9 May 2023 15:16:22 +0000 (11:16 -0400)]
12684: Use mock services in arvfile sparse write tests

Without these mocks, Jenkins seems to spend a lot of time retrying
requests—although weirdly, I don't see that in my own development
environment.

I believe the mocks were always intended to be used, since they're
instantiated and already used in other sparse write tests. To me this
looks like an oversight when the previous tests were adapted to write
new collections.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684/20425: Skip TestContainerInputOnDifferentCluster
Brett Smith [Tue, 9 May 2023 14:04:41 +0000 (10:04 -0400)]
12684/20425: Skip TestContainerInputOnDifferentCluster

See comments for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684: Check for no log case in controller integration tests
Brett Smith [Fri, 5 May 2023 13:49:04 +0000 (09:49 -0400)]
12684: Check for no log case in controller integration tests

Without this guard, tests fail with a message "API endpoint not found,"
which sounds scary and makes you think you broke all of Arvados until
you see the test code is just looking up a collection with an empty
UUID.

And by "you," I mean me.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684: PySDK client retries specific 4xx errors
Brett Smith [Thu, 4 May 2023 21:41:23 +0000 (17:41 -0400)]
12684: PySDK client retries specific 4xx errors

The rationale for retrying these codes is the same as for retrying them
in the retry module.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684: Support num_retries in PySDK client constructors
Brett Smith [Thu, 4 May 2023 20:21:08 +0000 (16:21 -0400)]
12684: Support num_retries in PySDK client constructors

This lets users set their preferred retry strategy once, rather than in
every call to execute(), which is error-prone. The default num_retries
is 10 because we expect most users to care more about eventual success
than responsiveness. See the added release notes for further discussion
and rationale.

Changes to the rest of the code are mostly about supporting this
consistently. Tests that relied on the old no-default-num_retries
behavior now specify that explicitly.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months ago12684: Remove custom retry logic from PySDK
Brett Smith [Wed, 3 May 2023 18:19:06 +0000 (14:19 -0400)]
12684: Remove custom retry logic from PySDK

This logic traces its roots back to
`5722c604c6f5dc1553674d179ec016ec12e2b090`. The goal of that commit was to
work around a bug in httplib, which we no longer use as a client
library. `31eb1bdc31e1d030844a6fdc7f4ba4286ec79d4f` made an analogous
change for httplib2.

`8a0eb69984a93852ec888cd3e02b778b0be758ed` made three major changes:

1. Proactively close sockets if they seem likely to be stale
2. Wrap the retry logic in a loop
3. Generalize catching `httplib.BadStatusLine` to `httplib.HTTPException`
   (which covers all kinds of malformed HTTP responses)

However, #1 functionally obsoletes the exception handlers added in the
earlier commits. Preemptively closing the sockets prevents httplib/2
from trying to reuse stale ones. So these exception handlers, along with
their retry loops, no longer serve their original purpose.

Remove this logic in favor of using the retry logic built into
googleapiclient. That logic is easier to configure and more refined.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

11 months agoMerge branch '20475-dump-busy-queue'
Tom Clegg [Thu, 4 May 2023 20:17:08 +0000 (16:17 -0400)]
Merge branch '20475-dump-busy-queue'

closes #20475

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20475: Option to dump active requests when queue is >=90% full. 20475-dump-busy-queue
Tom Clegg [Wed, 3 May 2023 20:40:59 +0000 (16:40 -0400)]
20475: Option to dump active requests when queue is >=90% full.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '18790-log-client'
Tom Clegg [Wed, 3 May 2023 13:41:58 +0000 (09:41 -0400)]
Merge branch '18790-log-client'

closes #18790

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20457-logs-and-mem-usage'
Tom Clegg [Mon, 1 May 2023 20:11:07 +0000 (16:11 -0400)]
Merge branch '20457-logs-and-mem-usage'

refs #20457

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20457: Don't keep non-"tmp" mounts in memory at all. 20457-logs-and-mem-usage
Tom Clegg [Mon, 1 May 2023 19:40:12 +0000 (15:40 -0400)]
20457: Don't keep non-"tmp" mounts in memory at all.

Only "tmp" mounts are relevant for dispatch.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20432-getting-containers' refs #20432
Peter Amstutz [Mon, 1 May 2023 19:17:37 +0000 (15:17 -0400)]
Merge branch '20432-getting-containers' refs #20432

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20432: Tweak "error checking states on API server" message
Peter Amstutz [Mon, 1 May 2023 19:17:05 +0000 (15:17 -0400)]
20432: Tweak "error checking states on API server" message

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months agoMerge branch '20447-less-table-locking'
Tom Clegg [Mon, 1 May 2023 16:07:45 +0000 (12:07 -0400)]
Merge branch '20447-less-table-locking'

fixes #20447

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20457: Add logging in test case wrt 503 responses.
Tom Clegg [Mon, 1 May 2023 15:54:59 +0000 (11:54 -0400)]
20457: Add logging in test case wrt 503 responses.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20457: Don't keep ctr mounts in memory after computing node size.
Tom Clegg [Mon, 1 May 2023 15:54:23 +0000 (11:54 -0400)]
20457: Don't keep ctr mounts in memory after computing node size.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20457: Implement MaxDispatchAttempts in test stub.
Tom Clegg [Mon, 1 May 2023 15:53:34 +0000 (11:53 -0400)]
20457: Implement MaxDispatchAttempts in test stub.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20462-workflow-prefix' refs #20462
Peter Amstutz [Mon, 1 May 2023 15:44:16 +0000 (11:44 -0400)]
Merge branch '20462-workflow-prefix' refs #20462

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20462: Move to util & use Brett's implementation
Peter Amstutz [Mon, 1 May 2023 15:42:29 +0000 (11:42 -0400)]
20462: Move to util & use Brett's implementation

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20462: Refactor "common prefix" function & fix fencepost errors 20462-workflow-prefix
Peter Amstutz [Mon, 1 May 2023 14:58:25 +0000 (10:58 -0400)]
20462: Refactor "common prefix" function & fix fencepost errors

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20457: Log at info level when flapping lock at concurrency limit.
Tom Clegg [Mon, 1 May 2023 13:36:01 +0000 (09:36 -0400)]
20457: Log at info level when flapping lock at concurrency limit.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20447: Don't lock containers table for CR name/description updates. 20447-less-table-locking
Tom Clegg [Sat, 29 Apr 2023 04:23:36 +0000 (00:23 -0400)]
20447: Don't lock containers table for CR name/description updates.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20447: Don't lock containers table for cost updates.
Tom Clegg [Fri, 28 Apr 2023 17:53:57 +0000 (13:53 -0400)]
20447: Don't lock containers table for cost updates.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20422-cache-slot' refs #20422
Peter Amstutz [Fri, 28 Apr 2023 15:31:49 +0000 (11:31 -0400)]
Merge branch '20422-cache-slot' refs #20422

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago18790: Enable follow mode with -f flag. 18790-log-client
Tom Clegg [Fri, 28 Apr 2023 15:29:01 +0000 (11:29 -0400)]
18790: Enable follow mode with -f flag.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18790: Fix usage messages.
Tom Clegg [Fri, 28 Apr 2023 15:12:15 +0000 (11:12 -0400)]
18790: Fix usage messages.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18799: Version PySDK's typing_extensions dependency
Brett Smith [Wed, 26 Apr 2023 18:29:16 +0000 (14:29 -0400)]
18799: Version PySDK's typing_extensions dependency

3.7.4 added TypedDict, which is exactly what we use.
Refs #18799, #20446.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago18790: Move to container_requests/*/log endpoint.
Tom Clegg [Thu, 20 Apr 2023 13:26:52 +0000 (09:26 -0400)]
18790: Move to container_requests/*/log endpoint.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18790: Merge branch 'main' into 18790-log-client
Tom Clegg [Tue, 25 Apr 2023 14:47:06 +0000 (10:47 -0400)]
18790: Merge branch 'main' into 18790-log-client

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20319-container-request-logs'
Tom Clegg [Tue, 25 Apr 2023 14:33:29 +0000 (10:33 -0400)]
Merge branch '20319-container-request-logs'

closes #20319

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoImport dataclasses for py 3.6, refs #20257
Peter Amstutz [Fri, 21 Apr 2023 17:44:53 +0000 (13:44 -0400)]
Import dataclasses for py 3.6, refs #20257

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20422: Do GET when cache slot is empty instead of raising an error 20422-cache-slot
Peter Amstutz [Mon, 24 Apr 2023 20:59:27 +0000 (16:59 -0400)]
20422: Do GET when cache slot is empty instead of raising an error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months agoMerge branch '14070-name-index'
Tom Clegg [Mon, 24 Apr 2023 15:24:16 +0000 (11:24 -0400)]
Merge branch '14070-name-index'

fixes #14070

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoAdjust default value of BalanceCollectionBuffers down to 4
Peter Amstutz [Fri, 21 Apr 2023 19:19:29 +0000 (15:19 -0400)]
Adjust default value of BalanceCollectionBuffers down to 4

Buffering a large number of collections in RAM is much less helpful
since keep-balance started accessing the database directly instead of
using the API.

refs #20421

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months agoAdjust trash_at of log collection to now+2 weeks refs #20378
Peter Amstutz [Fri, 21 Apr 2023 19:06:24 +0000 (15:06 -0400)]
Adjust trash_at of log collection to now+2 weeks refs #20378

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months agoMerge branch '20388-installer-prom-targets'. Closes #20388
Lucas Di Pentima [Fri, 21 Apr 2023 18:08:36 +0000 (15:08 -0300)]
Merge branch '20388-installer-prom-targets'. Closes #20388

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months agoMerge branch '20257-http-import' refs #20257
Peter Amstutz [Fri, 21 Apr 2023 16:23:27 +0000 (12:23 -0400)]
Merge branch '20257-http-import' refs #20257

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Fix use of dataclass 20257-http-import
Peter Amstutz [Fri, 21 Apr 2023 16:02:04 +0000 (12:02 -0400)]
20257: Fix use of dataclass

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Mark functions and classes internal not intended for use
Peter Amstutz [Fri, 21 Apr 2023 14:57:30 +0000 (10:57 -0400)]
20257: Mark functions and classes internal not intended for use

Adjust return value of http_to_keep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago14070: Add trigram indexes on collections.name and projects.name. 14070-name-index
Tom Clegg [Fri, 21 Apr 2023 14:38:19 +0000 (10:38 -0400)]
14070: Add trigram indexes on collections.name and projects.name.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20388: Adds missing arvados services as prometheus targets. 20388-installer-prom-targets
Lucas Di Pentima [Fri, 21 Apr 2023 14:19:26 +0000 (11:19 -0300)]
20388: Adds missing arvados services as prometheus targets.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago20319: Update documentation. 20319-container-request-logs
Tom Clegg [Thu, 20 Apr 2023 23:38:04 +0000 (19:38 -0400)]
20319: Update documentation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20241-validate-ssh-keys'
Tom Clegg [Thu, 20 Apr 2023 14:49:16 +0000 (10:49 -0400)]
Merge branch '20241-validate-ssh-keys'

fixes #20241

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20319: Merge branch 'main' into 20319-container-request-logs
Tom Clegg [Thu, 20 Apr 2023 13:33:33 +0000 (09:33 -0400)]
20319: Merge branch 'main' into 20319-container-request-logs

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20319: Move /containers/*/log to /container_requests/*/log.
Tom Clegg [Thu, 20 Apr 2023 13:32:30 +0000 (09:32 -0400)]
20319: Move /containers/*/log to /container_requests/*/log.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20257: Fix test
Peter Amstutz [Wed, 19 Apr 2023 21:59:18 +0000 (17:59 -0400)]
20257: Fix test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Don't log arvados.keep at INFO
Peter Amstutz [Wed, 19 Apr 2023 20:59:16 +0000 (16:59 -0400)]
20257: Don't log arvados.keep at INFO

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Set INFO log level explicitly
Peter Amstutz [Wed, 19 Apr 2023 20:43:17 +0000 (16:43 -0400)]
20257: Set INFO log level explicitly

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Add message before doing any HTTP activity
Peter Amstutz [Wed, 19 Apr 2023 20:08:39 +0000 (16:08 -0400)]
20257: Add message before doing any HTTP activity

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Tweak runtime status for 'activity'
Peter Amstutz [Wed, 19 Apr 2023 19:44:11 +0000 (15:44 -0400)]
20257: Tweak runtime status for 'activity'

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20241: Add ed25519-sk test key 20241-validate-ssh-keys
Brett Smith [Wed, 19 Apr 2023 19:26:35 +0000 (15:26 -0400)]
20241: Add ed25519-sk test key

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago20257: Record basic workflow status in "activity"
Peter Amstutz [Wed, 19 Apr 2023 19:18:44 +0000 (15:18 -0400)]
20257: Record basic workflow status in "activity"

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Need to parse status code out of the header, because pyCurl
Peter Amstutz [Wed, 19 Apr 2023 19:02:55 +0000 (15:02 -0400)]
20257: Need to parse status code out of the header, because pyCurl

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20257: Refactor http-to-keep downloader
Peter Amstutz [Wed, 19 Apr 2023 14:07:05 +0000 (10:07 -0400)]
20257: Refactor http-to-keep downloader

* Moved into the SDK, so that other programs besides cwl-runner can use
it
* Migrated to pyCurl
* Updated tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago18799: Force UTF-8 encoding for discovery2pydoc output
Brett Smith [Wed, 19 Apr 2023 13:58:49 +0000 (09:58 -0400)]
18799: Force UTF-8 encoding for discovery2pydoc output

Refs #18799.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months agoMerge branch '18799-api-pydoc'
Brett Smith [Wed, 19 Apr 2023 12:55:56 +0000 (08:55 -0400)]
Merge branch '18799-api-pydoc'

Closes #18799.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago20241: Move authorized_keys to new code path, validate public key.
Tom Clegg [Tue, 18 Apr 2023 20:34:47 +0000 (16:34 -0400)]
20241: Move authorized_keys to new code path, validate public key.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18799: Make schema docstring more specific about how to select object keys 18799-api-pydoc
Brett Smith [Tue, 18 Apr 2023 15:13:08 +0000 (11:13 -0400)]
18799: Make schema docstring more specific about how to select object keys

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago18799: Link back to resource class from schema docstring
Brett Smith [Tue, 18 Apr 2023 15:07:54 +0000 (11:07 -0400)]
18799: Link back to resource class from schema docstring

This makes the connection bidirectional, which helps both understanding
and navigation.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months agoMerge branch '19620-remove-old-s3-driver'
Tom Clegg [Tue, 18 Apr 2023 15:13:41 +0000 (11:13 -0400)]
Merge branch '19620-remove-old-s3-driver'

closes #19620

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoAdd 2.6.1 to upgrade notes. refs #20345
Peter Amstutz [Mon, 17 Apr 2023 19:30:54 +0000 (15:30 -0400)]
Add 2.6.1 to upgrade notes. refs #20345

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months agoMerge branch '20123-access-token-doc' refs #20123
Peter Amstutz [Mon, 17 Apr 2023 18:43:01 +0000 (14:43 -0400)]
Merge branch '20123-access-token-doc' refs #20123

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20123: Fix tabs vs spaces 20123-access-token-doc
Peter Amstutz [Mon, 17 Apr 2023 18:41:57 +0000 (14:41 -0400)]
20123: Fix tabs vs spaces

refs #20123

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20123: Add note about scopes
Peter Amstutz [Mon, 17 Apr 2023 18:37:10 +0000 (14:37 -0400)]
20123: Add note about scopes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months ago20123: Make OpenID connect token a proper subsection
Peter Amstutz [Mon, 17 Apr 2023 18:25:05 +0000 (14:25 -0400)]
20123: Make OpenID connect token a proper subsection

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

12 months agoMerge branch '20203-rails-cache'
Tom Clegg [Mon, 17 Apr 2023 15:06:54 +0000 (11:06 -0400)]
Merge branch '20203-rails-cache'

fixes #20203

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago19620: Remove goamz dependency in non-testing code. 19620-remove-old-s3-driver
Tom Clegg [Mon, 17 Apr 2023 14:50:05 +0000 (10:50 -0400)]
19620: Remove goamz dependency in non-testing code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18799: Make the PySDK's build subcommand extend build_py
Brett Smith [Fri, 14 Apr 2023 18:03:45 +0000 (14:03 -0400)]
18799: Make the PySDK's build subcommand extend build_py

See the added comments for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months agoMerge branch '20325-go-docker-distribution-upgrade'. Refs #20325
Lucas Di Pentima [Fri, 14 Apr 2023 18:00:56 +0000 (15:00 -0300)]
Merge branch '20325-go-docker-distribution-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago18799: Add Python 3.7 compatibility for TypedDict
Brett Smith [Fri, 14 Apr 2023 17:55:54 +0000 (13:55 -0400)]
18799: Add Python 3.7 compatibility for TypedDict

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago18799: Clean up redundant setuptools requirement listing
Brett Smith [Fri, 14 Apr 2023 17:54:52 +0000 (13:54 -0400)]
18799: Clean up redundant setuptools requirement listing

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago18799: Add missing f-string prefix
Brett Smith [Fri, 14 Apr 2023 17:53:32 +0000 (13:53 -0400)]
18799: Add missing f-string prefix

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago20325: Upgrades github.com/docker/distribution module dependency. 20325-go-docker-distribution-upgrade
Lucas Di Pentima [Fri, 14 Apr 2023 17:25:07 +0000 (14:25 -0300)]
20325: Upgrades github.com/docker/distribution module dependency.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months agoMerge branch '20325-globalid-gem-upgrade'. Refs #20325
Lucas Di Pentima [Fri, 14 Apr 2023 16:15:17 +0000 (13:15 -0300)]
Merge branch '20325-globalid-gem-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months agoMerge branch '20334-slow-dedup-migration'
Tom Clegg [Fri, 14 Apr 2023 16:02:25 +0000 (12:02 -0400)]
Merge branch '20334-slow-dedup-migration'

fixes #20334

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20336-filter-bigint'
Tom Clegg [Fri, 14 Apr 2023 16:00:58 +0000 (12:00 -0400)]
Merge branch '20336-filter-bigint'

fixes #20336

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20203: Fix global var reset on db reset; skip in non-test mode. 20203-rails-cache
Tom Clegg [Fri, 14 Apr 2023 15:59:13 +0000 (11:59 -0400)]
20203: Fix global var reset on db reset; skip in non-test mode.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20325: Upgrades the globalid gem on both RailsAPI & WB1. 20325-globalid-gem-upgrade
Lucas Di Pentima [Fri, 14 Apr 2023 15:46:45 +0000 (12:46 -0300)]
20325: Upgrades the globalid gem on both RailsAPI & WB1.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months agoMerge branch '20325-go-containerd-upgrade'. Refs #20325
Lucas Di Pentima [Fri, 14 Apr 2023 15:18:07 +0000 (12:18 -0300)]
Merge branch '20325-go-containerd-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago20325: Upgrades github.com/containerd/containerd dependency. 20325-go-containerd-upgrade
Lucas Di Pentima [Fri, 14 Apr 2023 14:28:47 +0000 (11:28 -0300)]
20325: Upgrades github.com/containerd/containerd dependency.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago18799: Mark self parameters as POSITIONAL_OR_KEYWORD
Brett Smith [Fri, 14 Apr 2023 14:06:23 +0000 (10:06 -0400)]
18799: Mark self parameters as POSITIONAL_OR_KEYWORD

On Python 3.7, if you mark a parameter POSITIONAL_ONLY, it will generate
a signature that uses the positional-only `/` syntax, *even though*
Python 3.7 cannot parse that syntax (it was added in Python 3.8, PEP 570).

POSITIONAL_OR_KEYWORD avoids that problem, and is probably more accurate
anyway.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago19620: Remove old "v1" S3 keepstore driver.
Tom Clegg [Thu, 13 Apr 2023 21:23:33 +0000 (17:23 -0400)]
19620: Remove old "v1" S3 keepstore driver.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18799: Move canonical discovery document to sdk/python
Brett Smith [Thu, 13 Apr 2023 20:36:10 +0000 (16:36 -0400)]
18799: Move canonical discovery document to sdk/python

The Python SDK build is currently the only code that needs it, and
moving the file here simplifies the build implementation. We can
reevaluate where it lives if another need arises.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago20336: Allow filtering on set of integers. 20336-filter-bigint
Tom Clegg [Thu, 13 Apr 2023 19:42:22 +0000 (15:42 -0400)]
20336: Allow filtering on set of integers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20336: Reject >64 bit operands when filtering on integer columns.
Tom Clegg [Thu, 13 Apr 2023 17:27:34 +0000 (13:27 -0400)]
20336: Reject >64 bit operands when filtering on integer columns.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months agoMerge branch '20325-nokogiri-upgrade'. Refs #20325
Lucas Di Pentima [Thu, 13 Apr 2023 17:54:26 +0000 (14:54 -0300)]
Merge branch '20325-nokogiri-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago23025: Upgrades nokogiri gem on RailsAPI & Workbench1. 20325-nokogiri-upgrade
Lucas Di Pentima [Thu, 13 Apr 2023 17:32:01 +0000 (14:32 -0300)]
23025: Upgrades nokogiri gem on RailsAPI & Workbench1.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months agoMerge branch '20325-go-x-net-upgrade'. Refs #20325
Lucas Di Pentima [Thu, 13 Apr 2023 17:25:27 +0000 (14:25 -0300)]
Merge branch '20325-go-x-net-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago20325: Upgrades golang.org/x/net. 20325-go-x-net-upgrade
Lucas Di Pentima [Thu, 13 Apr 2023 16:54:56 +0000 (13:54 -0300)]
20325: Upgrades golang.org/x/net.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months agoMerge branch '20325-rack-upgrade'. Refs #20325
Lucas Di Pentima [Thu, 13 Apr 2023 16:41:33 +0000 (13:41 -0300)]
Merge branch '20325-rack-upgrade'. Refs #20325

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago20325: Upgrades rack gem on RailsAPI & Workbench1 20325-rack-upgrade
Lucas Di Pentima [Thu, 13 Apr 2023 16:16:22 +0000 (13:16 -0300)]
20325: Upgrades rack gem on RailsAPI & Workbench1

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

12 months ago20334: Batch permission updates during link dedup migration. 20334-slow-dedup-migration
Tom Clegg [Thu, 13 Apr 2023 15:47:46 +0000 (11:47 -0400)]
20334: Batch permission updates during link dedup migration.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18799: Make discovery2pydoc status check Py3.7-compatible
Brett Smith [Thu, 13 Apr 2023 15:42:09 +0000 (11:42 -0400)]
18799: Make discovery2pydoc status check Py3.7-compatible

The `status` attribute was added in Python 3.9.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago18799: Add discovery2pydoc to Python SDK manifest
Brett Smith [Thu, 13 Apr 2023 15:24:45 +0000 (11:24 -0400)]
18799: Add discovery2pydoc to Python SDK manifest

Same rationale as arvados_version: setup.py pulls it in.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago18799: Add background to API docstrings
Brett Smith [Thu, 13 Apr 2023 14:51:46 +0000 (10:51 -0400)]
18799: Add background to API docstrings

These explanations are intended to help orient users reading this
documentation module (whether via Pydoc or the web) to understand how it
relates to the API client object and the dictionaries it returns.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

12 months ago20123: Add detail about OIDC token validation/cache implementation.
Tom Clegg [Thu, 13 Apr 2023 14:50:13 +0000 (10:50 -0400)]
20123: Add detail about OIDC token validation/cache implementation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago20123: Add hints about OpenID Connect configuration.
Tom Clegg [Wed, 5 Apr 2023 13:50:36 +0000 (09:50 -0400)]
20123: Add hints about OpenID Connect configuration.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

12 months ago18799: Build Python SDK before generating web documentation
Brett Smith [Thu, 13 Apr 2023 14:07:05 +0000 (10:07 -0400)]
18799: Build Python SDK before generating web documentation

This gives preprocessing steps like discovery2pydoc a chance to run.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>