Tom Clegg [Fri, 27 Jan 2023 18:30:09 +0000 (13:30 -0500)]
19923: Report heap and stack stats from Go runtime.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 27 Jan 2023 16:01:15 +0000 (11:01 -0500)]
19923: Sanity check provided chunk prefix.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 27 Jan 2023 15:50:29 +0000 (10:50 -0500)]
19923: Remove debug sleep.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 26 Jan 2023 22:47:17 +0000 (17:47 -0500)]
19923: Log keep-balance memory usage.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 26 Jan 2023 16:14:28 +0000 (11:14 -0500)]
19923: Log progress while reading keepstore indexes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 24 Jan 2023 16:29:46 +0000 (11:29 -0500)]
19923: keep-balance option to process a subset of blocks.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Fri, 20 Jan 2023 16:23:59 +0000 (11:23 -0500)]
Merge branch '19886-crunch-run-early-log-commit'
Closes #19886.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 16:21:28 +0000 (11:21 -0500)]
19886: Describe what component updates container logs more generically
We want to make clear that an Arvados system component does this,
without being overly committal about which component it is. "Dispatcher"
might be taken to mean the Crunch dispatcher specifically. Right now
it's crunch-run, but we don't need to commit to that level of
detail. Settle on "Crunch system" per discussion with Tom in chat.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 14:11:59 +0000 (09:11 -0500)]
19886: Slim down API documentation about container logs
While the API server will let you set this field to a UUID, that
behavior is already poorly supported, and on reflection we're not sure
we want to support it. Document this the way crunch-run uses it and
avoid presenting too much detail for most readers.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 14:07:28 +0000 (09:07 -0500)]
Merge branch '19917-retry-scheduling-parameters'
Closes #19917.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 19 Jan 2023 20:39:29 +0000 (15:39 -0500)]
19917: Synthesize scheduling parameters when retrying a container
When we retry a cancelled container, there may be any number of
container requests that want it, each with their own scheduling
parameters. Create a new set of scheduling parameters for the new
container where each parameter has the most lax setting from all the
outstanding container requests.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:51:05 +0000 (14:51 -0500)]
19886: Update API documentation notes about container request logs
The existing note was already out-of-date: crunch-run would record logs
about 30 minutes after starting the container. With recent changes,
crunch-run now records logs shortly after starting the container. Update
the note accordingly.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:44:55 +0000 (14:44 -0500)]
19886: Correct API documentation notes about container logs
The documented limitation does not exist. In fact, there's a limitation
in the API server that if you want a container's logs propagated to its
associated container request(s), you *must* pass in a portable data
hash. crunch-run consistently updates container records with a portable
data hash in the log field for this reason. Update the note to reflect
this.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:35:58 +0000 (14:35 -0500)]
19886: crunch-run records initial log with PDH
The API server will only propagate a container's log collection to
container requests when it is specified with a portable data hash. See
the top of ContainerRequest#update_collections.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:34:00 +0000 (14:34 -0500)]
19886: Tighten test regexps for better validation
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:33:41 +0000 (14:33 -0500)]
19886: DRY up test code for better readability
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Wed, 18 Jan 2023 15:11:06 +0000 (10:11 -0500)]
Merge branch '18693-dedup-permissions'
fixes #18693
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 18 Jan 2023 14:26:55 +0000 (09:26 -0500)]
Merge branch '19896-ldap-tls-downgrade'
fixes #19896
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 21:58:33 +0000 (16:58 -0500)]
18693: Use row locking only if enabled in config.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:35:29 +0000 (11:35 -0500)]
19896: Mention TLS version error message in config doc.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:27:37 +0000 (11:27 -0500)]
19896: Use StartTLS + MinTLSVersion regardless of Insecure flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:20:51 +0000 (11:20 -0500)]
Merge branch '19890-id-bigint'
fixes #19890
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 16 Jan 2023 22:35:41 +0000 (17:35 -0500)]
19890: Note slow upgrade due to schema migration.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 16 Jan 2023 22:26:57 +0000 (17:26 -0500)]
Merge branch '19597-multipart-request'
fixes #19597
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 16 Jan 2023 22:26:41 +0000 (17:26 -0500)]
19597: Comment explicit ParseForm usage.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Fri, 13 Jan 2023 20:10:56 +0000 (15:10 -0500)]
Merge branch '19895-google-api-python-client-2'
Closes #19895.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 12 Jan 2023 21:51:12 +0000 (16:51 -0500)]
19886: Save log collection when crunch-run starts a container
This allows users to see information about the node where the container
started, including through Workbench 2.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 13 Jan 2023 14:33:18 +0000 (09:33 -0500)]
19886: Refine crunch-run output collection tests
These tests are currently not working as designed because the API call
they're looking for is in a different API client. They happen to be
passing because they never assert they actually found the collection
they're looking for. Restructure the search loop to fix both those
problems.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 12 Jan 2023 21:48:12 +0000 (16:48 -0500)]
19886: Bugfix UUID generation in crunch-run tests
The previous code was generating UUIDs with a 30-character last part.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Fri, 13 Jan 2023 05:27:14 +0000 (00:27 -0500)]
Merge branch '19894-pg-access'
fixes #19894
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Tue, 10 Jan 2023 16:53:56 +0000 (11:53 -0500)]
19894: Copyedit the 2.5.0 PostgreSQL upgrade note
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Mon, 9 Jan 2023 16:43:31 +0000 (11:43 -0500)]
19894: Update documentation re: dispatcher connecting to postgresql.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Mon, 9 Jan 2023 14:14:22 +0000 (09:14 -0500)]
19895: Upgrade google-api-python-client
Version 2.0 tries to load all discovery documents from a local cache. It
added a new argument to discovery.build(static_discovery=False) to
retrieve the discovery document from the network.
Version 2.1 improves backwards compatibility by having static_discovery
default to False when you pass an explicit discoveryServiceUrl to
discovery.build(). We always do this (see
arvados.api.normalize_api_kwargs), so we can upgrade to version 2.1
without making any changes to our own SDK.
Doing so is just good hygiene, and makes life easier for users who want
to build projects that use both Google services and Arvados.
Version 2.1 was released on 2021-03-31, so this new version requirement
should be easy enough for most users to satisfy.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Wed, 4 Jan 2023 23:35:03 +0000 (18:35 -0500)]
19896: Configurable minimum TLS version for LDAP connection.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 2 Jan 2023 15:11:55 +0000 (10:11 -0500)]
19890: Use 64 bit ints for id columns.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Wed, 28 Dec 2022 20:15:03 +0000 (15:15 -0500)]
Update container state table headers
"Sgnificance" was a typo, and the immediate problem I wanted to
fix. "Description" is what we usually use and works fine here, so use
that.
I expanded "State" to "State value" to try to help clarify this is the
actual value in the data.
No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Tue, 20 Dec 2022 15:03:52 +0000 (10:03 -0500)]
Update version numbers
refs #19647
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Wed, 21 Dec 2022 19:31:26 +0000 (14:31 -0500)]
19597: Parse multipart/form-data request body.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 20 Dec 2022 16:39:01 +0000 (11:39 -0500)]
18693: Add row locking.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Tue, 20 Dec 2022 15:13:18 +0000 (10:13 -0500)]
Merge branch '19844-paging' refs #19844
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Tue, 20 Dec 2022 01:23:21 +0000 (20:23 -0500)]
18693: Remove existing redundant permission links from db.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 19 Dec 2022 16:31:00 +0000 (11:31 -0500)]
18693: Delete redundant login permission links on update/delete.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 19 Dec 2022 01:19:47 +0000 (20:19 -0500)]
18693: Return existing instead of creating redundant can_login link.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Sun, 18 Dec 2022 13:55:03 +0000 (08:55 -0500)]
18693: De-duplicate permission links on create/update/delete.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 22:18:33 +0000 (17:18 -0500)]
Merge branch '19880-wb1-pw-login-fix' refs #19880
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 21:53:02 +0000 (16:53 -0500)]
19880: Fix tests by restoring test hook
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 21:08:26 +0000 (16:08 -0500)]
19880: Fix tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 19:36:50 +0000 (14:36 -0500)]
19880: Password based login fix
Now sends user to wb1 front page instead of controller login endpoint
that returns an error.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 15:46:19 +0000 (10:46 -0500)]
Add missing test file refs #19688
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 15:25:20 +0000 (10:25 -0500)]
Test needs to ignore varying "acrContainerImage" field
refs #19688
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 22:09:12 +0000 (17:09 -0500)]
Merge branch '19704-aneta-r-sdk' refs #19704
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 19:05:14 +0000 (14:05 -0500)]
Merge branch '19872-mnt-cache-limits' refs #19872
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:55:48 +0000 (13:55 -0500)]
19704: Fix R SDK documentation build.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:21:41 +0000 (13:21 -0500)]
19872: Fix call to setrlimit
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:18:12 +0000 (13:18 -0500)]
19872: Fix typo
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:14:02 +0000 (13:14 -0500)]
19872: Use fewer file handles by default, but increase the limit in arv-mount
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 17:57:31 +0000 (12:57 -0500)]
Don't record varying git data for copy_deps test
refs #19688
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Thu, 15 Dec 2022 15:56:03 +0000 (10:56 -0500)]
19704: Update call signature of test fake.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Wed, 14 Dec 2022 20:48:53 +0000 (15:48 -0500)]
Merge branch '19847-cwl-disk-cache-size' refs #19847
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 14 Dec 2022 20:33:58 +0000 (15:33 -0500)]
Update copy_deps test, refs #19688
It now creates an additional collection containing the workflow text,
the test needed to be updated to reflect that.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Wed, 14 Dec 2022 15:09:52 +0000 (10:09 -0500)]
19847: Update test for new API default disk cache behavior.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Wed, 14 Dec 2022 14:36:54 +0000 (14:36 +0000)]
19872: Add a timeout to wait()
Add a timeout when waiting for a cache slot to avoid the possibility
of waiting forever on wait().
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 13 Dec 2022 22:35:44 +0000 (22:35 +0000)]
19872: Set max_slots lower because mmap uses another FD
- Reduce max slots to 3/8 max fds instead of 1/2 because mmap() uses a
second file descriptor, and we keep the original file descriptor open
for flock()
- Rework how cache slots are allocated to try evicting things _before_
allocating a new cache slot, so the cache should be somewhat better
behaved about staying within its configured limits.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Tue, 13 Dec 2022 21:54:11 +0000 (16:54 -0500)]
Merge branch '19709-boot-db-migrate'
closes #19709
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Dec 2022 21:38:54 +0000 (16:38 -0500)]
19847: Let server apply default disk cache size.
Use disk cache when DefaultKeepCacheRAM==0 (instead of
DefaultKeepCacheDisk>0).
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Dec 2022 21:31:31 +0000 (16:31 -0500)]
19847: Default to disk cache size = RAM size for all containers.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Lucas Di Pentima [Tue, 13 Dec 2022 20:34:20 +0000 (21:34 +0100)]
Merge branch '19878-rails-html-sanitizer-upgrade'. Closes #19878
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Tue, 13 Dec 2022 20:12:59 +0000 (21:12 +0100)]
19878: Upgrades rails-html-sanitizer on RailsAPI & Workbench.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Tue, 13 Dec 2022 20:06:59 +0000 (21:06 +0100)]
Merge branch '19877-loofah-upgrade'. Closes #19877
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Lucas Di Pentima [Tue, 13 Dec 2022 19:45:52 +0000 (20:45 +0100)]
19877: Upgrades loofah gem on RailsAPI & Woekbench
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Tom Clegg [Tue, 13 Dec 2022 19:24:33 +0000 (14:24 -0500)]
19709: Warn, not eror, on unexpected files in db/migrate/.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Dec 2022 21:46:03 +0000 (16:46 -0500)]
19709: Test schema_migrations vs. db/migrate/*.rb.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Dec 2022 21:05:48 +0000 (16:05 -0500)]
Merge branch '19840-docker-port-number'
fixes #19840
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Dec 2022 20:57:46 +0000 (15:57 -0500)]
Update resource requirements in test cases.
New disk cache defaults to 8 GiB, which does not necessarily fit in
/tmp on a dev/test machine (mine has a 4 GiB tmpfs).
refs #19847
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Dec 2022 20:01:17 +0000 (15:01 -0500)]
19709: Explain design in comments. Improve migration filename check.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Dec 2022 19:41:50 +0000 (14:41 -0500)]
19844: Get multiple pages of child CRs if necessary.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Dec 2022 19:06:15 +0000 (14:06 -0500)]
19844: Use 127.0.0.1 instead of localhost for test servers.
Otherwise, depending on host config, nginx futilely tries ::1 first.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Dec 2022 07:09:06 +0000 (02:09 -0500)]
19840: Cleanup link added during test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Dec 2022 22:23:53 +0000 (17:23 -0500)]
19840: Test listing saved images with names containing port numbers.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Dec 2022 21:52:52 +0000 (16:52 -0500)]
19840: Fix image name confusion when specified with explicit :443.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Nov 2022 20:31:42 +0000 (15:31 -0500)]
19709: Apply pending rails migrations at service start/restart.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Lucas Di Pentima [Thu, 8 Dec 2022 17:29:05 +0000 (18:29 +0100)]
Merge branch '19862-nokogiri-upgrade'. Closes #19862
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Tom Clegg [Thu, 8 Dec 2022 17:06:55 +0000 (12:06 -0500)]
19862: Fix RandomUUID length.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Lucas Di Pentima [Thu, 8 Dec 2022 15:15:42 +0000 (16:15 +0100)]
19862: Upgrades nokogiri and related gem on RailsAPI and Workbench.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Brett Smith [Thu, 8 Dec 2022 16:10:33 +0000 (11:10 -0500)]
19791: Fix broken link to old PySDK examples page
No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 8 Dec 2022 16:05:57 +0000 (11:05 -0500)]
Merge branch '19686-threadsafe-api-default'
Closes #19686.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 8 Dec 2022 15:51:58 +0000 (10:51 -0500)]
19686: Specify ThreadSafeApiCache API version throughout
This avoids logging a warning now that ThreadSafeApiCache accepts a
version argument just like other API client constructors.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 7 Dec 2022 20:21:43 +0000 (15:21 -0500)]
19686: Fix test scaffolding after API changes
Without clearing arv-put's API client this way, multiple calls to main()
reuse the same, pre-patch Keep object from ThreadSafeApiCache, so calls
don't get recorded as intended.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 7 Dec 2022 04:16:28 +0000 (23:16 -0500)]
19686: Add tests for new Arvados client API
This commit ensures all public-facing functions have basic tests for
good inputs, as well as known-bad argument combinations (if any). This
includes testing functions like api_from_config that were previously
tested implicitly by other tests, but did not have their own dedicated
tests.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 29 Nov 2022 20:01:43 +0000 (15:01 -0500)]
19686: api constructor returns ThreadSafeApiCache
This is an API-compatible wrapper object that provides thread
safety. Returning this from api() helps keep users out of
trouble.
The changes to ThreadSafeApiCache are required to keep it API-compatible
with the original and keep tests passing. Assignments to the request_id
attribute need to be used for all future requests. It's a little unclear
if this is an intended API or just test scaffolding, but it's not too
difficult to keep working so I just did that.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 28 Nov 2022 21:30:28 +0000 (16:30 -0500)]
19686: Introduce low-level api_client constructor
When api() returns a ThreadSafeApiCache, we still want to provide a
mechanism to get a plain Resource object. The api_client() function is
that mechanism. It *just* builds the Resource object as api() did
before.
normalize_api_kwargs() takes the arguments passed to api() and turns
them into keyword arguments for api_client(). api_config_to_kwargs()
takes a configuration mapping nand turns them into keyword arguments for
api_client(). Both of these are small APIs, just returning one
dictionary from another.
With this reorganization, api(), api_from_config(), and
ThreadSafeApiClient() can all have simpler implementations.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 7 Dec 2022 15:57:41 +0000 (10:57 -0500)]
19791: Remove duplicate word
No issue #
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Tue, 6 Dec 2022 22:13:19 +0000 (17:13 -0500)]
19847: Fix tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 6 Dec 2022 21:41:15 +0000 (16:41 -0500)]
19847: Change disk cache size heuristic
Now sets disk cache to (2 GB <= the size of the RAM request <= 32 GiB)
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 6 Dec 2022 20:26:47 +0000 (15:26 -0500)]
19847: crunch-run should provide --ram-cache explicitly
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Tue, 6 Dec 2022 20:18:11 +0000 (15:18 -0500)]
Merge branch '19791-python-api-overview'
Closes #19791.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Tue, 6 Dec 2022 18:41:42 +0000 (13:41 -0500)]
19847: Fix KeepCacheTypeRequirement
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Tue, 6 Dec 2022 18:23:07 +0000 (13:23 -0500)]
19791: Clarify the background discussion for using keyset_list_all
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 6 Dec 2022 18:16:39 +0000 (13:16 -0500)]
19791: Add a note explaining the page's intended scope
I wrote a note to this effect during the branch review, and I hope
putting it here can help guide future authors (possibly including future
me).
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 25 Nov 2022 16:56:44 +0000 (11:56 -0500)]
19791: Expand Python SDK examples into API client overview
This change takes the skeleton of the existing examples page and expands
it into a fuller overview of the API client provided by the Python
SDK. It provides more explanation of how the client works; how it
corresponds to the Arvados API; more concrete examples of real tasks you
might actually want to do with the client; and a few different examples
to demonstrate how you might use different client code patterns for
different situations.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
projects / arvados.git / log