Brett Smith [Mon, 2 Jan 2023 20:27:07 +0000 (15:27 -0500)]
19792: Revamp the Python SDK cookbook
* Organize the recipes into related sections based on what they work
on.
* Add a preface to each recipe to explain when you might use it, give
API background, etc.
* Write the recipes in a style that emphasizes readability with long
variable names, lots of line breaks, and comments highlighting
key mechanics and variations.
* Modernize SDK usage in the recipes.
* Add links to supporting documentation where available.
As much as practical, I have maintained existing anchor names, even when
I updated header names for more consistent style.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 16:21:28 +0000 (11:21 -0500)]
19886: Describe what component updates container logs more generically
We want to make clear that an Arvados system component does this,
without being overly committal about which component it is. "Dispatcher"
might be taken to mean the Crunch dispatcher specifically. Right now
it's crunch-run, but we don't need to commit to that level of
detail. Settle on "Crunch system" per discussion with Tom in chat.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 14:11:59 +0000 (09:11 -0500)]
19886: Slim down API documentation about container logs
While the API server will let you set this field to a UUID, that
behavior is already poorly supported, and on reflection we're not sure
we want to support it. Document this the way crunch-run uses it and
avoid presenting too much detail for most readers.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 19 Jan 2023 20:39:29 +0000 (15:39 -0500)]
19917: Synthesize scheduling parameters when retrying a container
When we retry a cancelled container, there may be any number of
container requests that want it, each with their own scheduling
parameters. Create a new set of scheduling parameters for the new
container where each parameter has the most lax setting from all the
outstanding container requests.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:51:05 +0000 (14:51 -0500)]
19886: Update API documentation notes about container request logs
The existing note was already out-of-date: crunch-run would record logs
about 30 minutes after starting the container. With recent changes,
crunch-run now records logs shortly after starting the container. Update
the note accordingly.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:44:55 +0000 (14:44 -0500)]
19886: Correct API documentation notes about container logs
The documented limitation does not exist. In fact, there's a limitation
in the API server that if you want a container's logs propagated to its
associated container request(s), you *must* pass in a portable data
hash. crunch-run consistently updates container records with a portable
data hash in the log field for this reason. Update the note to reflect
this.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:35:58 +0000 (14:35 -0500)]
19886: crunch-run records initial log with PDH
The API server will only propagate a container's log collection to
container requests when it is specified with a portable data hash. See
the top of ContainerRequest#update_collections.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 13 Jan 2023 14:33:18 +0000 (09:33 -0500)]
19886: Refine crunch-run output collection tests
These tests are currently not working as designed because the API call
they're looking for is in a different API client. They happen to be
passing because they never assert they actually found the collection
they're looking for. Restructure the search loop to fix both those
problems.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 9 Jan 2023 14:14:22 +0000 (09:14 -0500)]
19895: Upgrade google-api-python-client
Version 2.0 tries to load all discovery documents from a local cache. It
added a new argument to discovery.build(static_discovery=False) to
retrieve the discovery document from the network.
Version 2.1 improves backwards compatibility by having static_discovery
default to False when you pass an explicit discoveryServiceUrl to
discovery.build(). We always do this (see
arvados.api.normalize_api_kwargs), so we can upgrade to version 2.1
without making any changes to our own SDK.
Doing so is just good hygiene, and makes life easier for users who want
to build projects that use both Google services and Arvados.
Version 2.1 was released on 2021-03-31, so this new version requirement
should be easy enough for most users to satisfy.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Tue, 13 Dec 2022 22:35:44 +0000 (22:35 +0000)]
19872: Set max_slots lower because mmap uses another FD
- Reduce max slots to 3/8 max fds instead of 1/2 because mmap() uses a
second file descriptor, and we keep the original file descriptor open
for flock()
- Rework how cache slots are allocated to try evicting things _before_
allocating a new cache slot, so the cache should be somewhat better
behaved about staying within its configured limits.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Wed, 7 Dec 2022 20:21:43 +0000 (15:21 -0500)]
19686: Fix test scaffolding after API changes
Without clearing arv-put's API client this way, multiple calls to main()
reuse the same, pre-patch Keep object from ThreadSafeApiCache, so calls
don't get recorded as intended.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 7 Dec 2022 04:16:28 +0000 (23:16 -0500)]
19686: Add tests for new Arvados client API
This commit ensures all public-facing functions have basic tests for
good inputs, as well as known-bad argument combinations (if any). This
includes testing functions like api_from_config that were previously
tested implicitly by other tests, but did not have their own dedicated
tests.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 29 Nov 2022 20:01:43 +0000 (15:01 -0500)]
19686: api constructor returns ThreadSafeApiCache
This is an API-compatible wrapper object that provides thread
safety. Returning this from api() helps keep users out of
trouble.
The changes to ThreadSafeApiCache are required to keep it API-compatible
with the original and keep tests passing. Assignments to the request_id
attribute need to be used for all future requests. It's a little unclear
if this is an intended API or just test scaffolding, but it's not too
difficult to keep working so I just did that.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 28 Nov 2022 21:30:28 +0000 (16:30 -0500)]
19686: Introduce low-level api_client constructor
When api() returns a ThreadSafeApiCache, we still want to provide a
mechanism to get a plain Resource object. The api_client() function is
that mechanism. It *just* builds the Resource object as api() did
before.
normalize_api_kwargs() takes the arguments passed to api() and turns
them into keyword arguments for api_client(). api_config_to_kwargs()
takes a configuration mapping nand turns them into keyword arguments for
api_client(). Both of these are small APIs, just returning one
dictionary from another.
With this reorganization, api(), api_from_config(), and
ThreadSafeApiClient() can all have simpler implementations.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 25 Nov 2022 16:56:44 +0000 (11:56 -0500)]
19791: Expand Python SDK examples into API client overview
This change takes the skeleton of the existing examples page and expands
it into a fuller overview of the API client provided by the Python
SDK. It provides more explanation of how the client works; how it
corresponds to the Arvados API; more concrete examples of real tasks you
might actually want to do with the client; and a few different examples
to demonstrate how you might use different client code patterns for
different situations.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
projects / arvados.git / log