Brett Smith [Wed, 25 Jan 2023 15:05:32 +0000 (10:05 -0500)]
19792: Reorganize cookbook vocabulary section into properties section
Presenting this as a section about properties gives it more room to grow
and gives more context to the vocabulary discussion.
It's okay to drop the working-with-properties anchor because it was
never merged with main.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 25 Jan 2023 14:20:13 +0000 (09:20 -0500)]
19792: Update sharing link recipe to handle v2 API tokens
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 2 Jan 2023 20:27:07 +0000 (15:27 -0500)]
19792: Revamp the Python SDK cookbook
* Organize the recipes into related sections based on what they work
on.
* Add a preface to each recipe to explain when you might use it, give
API background, etc.
* Write the recipes in a style that emphasizes readability with long
variable names, lots of line breaks, and comments highlighting
key mechanics and variations.
* Modernize SDK usage in the recipes.
* Add links to supporting documentation where available.
As much as practical, I have maintained existing anchor names, even when
I updated header names for more consistent style.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 2 Jan 2023 20:24:32 +0000 (15:24 -0500)]
19792: Add anchors to some dedicated API methods
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 28 Dec 2022 21:10:18 +0000 (16:10 -0500)]
19791: Remove Python SDK "Notes" section
This is now covered more helpfully, and in more detail, in the API
client overview.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Wed, 25 Jan 2023 02:05:12 +0000 (21:05 -0500)]
19385: Passing the v1.2.1 conformance tests!
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 25 Jan 2023 00:40:25 +0000 (19:40 -0500)]
19385: Down to just a few failing conformance tests, I hope
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Tue, 24 Jan 2023 23:04:07 +0000 (18:04 -0500)]
19320: Deduplicate instance types in spot price request.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 24 Jan 2023 22:44:27 +0000 (17:44 -0500)]
19320: Comment re future use of spot attr in priceKey.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 24 Jan 2023 22:40:34 +0000 (17:40 -0500)]
19320: Disable spot price checks if configured update interval <= 0.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 24 Jan 2023 22:31:33 +0000 (17:31 -0500)]
19320: Account for AddedScratch in spot instance cost estimates.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Tue, 24 Jan 2023 22:27:30 +0000 (17:27 -0500)]
19385: Finally passing schemadef test.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 24 Jan 2023 22:08:58 +0000 (17:08 -0500)]
19385: more WIP
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 24 Jan 2023 21:03:56 +0000 (16:03 -0500)]
19385: WIP for complex schemadef + $import case
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Tue, 24 Jan 2023 16:29:46 +0000 (11:29 -0500)]
19923: keep-balance option to process a subset of blocks.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Tue, 24 Jan 2023 14:06:44 +0000 (09:06 -0500)]
19960: Fix container_uuid property name for intermediate collections
container_uuid is the name documented in our API reference, and now
expected by vocabulary.go's system tags.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 24 Jan 2023 13:59:05 +0000 (08:59 -0500)]
19960: Add container_uuid to systemTagKeys
This is set by arvados-cwl-runner, and already documented in our API
reference.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 24 Jan 2023 13:52:52 +0000 (08:52 -0500)]
19960: Organize systemTagKeys
Note what properties go with what resource types, and why.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Mon, 23 Jan 2023 15:31:41 +0000 (10:31 -0500)]
19320: Log instance price changes in crunch-run.txt.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 20 Jan 2023 19:20:14 +0000 (14:20 -0500)]
19385: Reference rewriting fixups
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Fri, 20 Jan 2023 16:23:59 +0000 (11:23 -0500)]
Merge branch '19886-crunch-run-early-log-commit'
Closes #19886.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 16:21:28 +0000 (11:21 -0500)]
19886: Describe what component updates container logs more generically
We want to make clear that an Arvados system component does this,
without being overly committal about which component it is. "Dispatcher"
might be taken to mean the Crunch dispatcher specifically. Right now
it's crunch-run, but we don't need to commit to that level of
detail. Settle on "Crunch system" per discussion with Tom in chat.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 14:11:59 +0000 (09:11 -0500)]
19886: Slim down API documentation about container logs
While the API server will let you set this field to a UUID, that
behavior is already poorly supported, and on reflection we're not sure
we want to support it. Document this the way crunch-run uses it and
avoid presenting too much detail for most readers.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 20 Jan 2023 14:07:28 +0000 (09:07 -0500)]
Merge branch '19917-retry-scheduling-parameters'
Closes #19917.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 19 Jan 2023 20:39:29 +0000 (15:39 -0500)]
19917: Synthesize scheduling parameters when retrying a container
When we retry a cancelled container, there may be any number of
container requests that want it, each with their own scheduling
parameters. Create a new set of scheduling parameters for the new
container where each parameter has the most lax setting from all the
outstanding container requests.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Thu, 19 Jan 2023 17:49:02 +0000 (12:49 -0500)]
19385: Unit tests pass
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 19 Jan 2023 16:22:32 +0000 (11:22 -0500)]
19385: Fixing tests WIP
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 18 Jan 2023 22:14:47 +0000 (17:14 -0500)]
19385: Fix submit_secret
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 18 Jan 2023 21:55:57 +0000 (16:55 -0500)]
19385: update_refs fixes prefixes
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 18 Jan 2023 21:50:06 +0000 (16:50 -0500)]
19385: Fix submit secrets test
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:51:05 +0000 (14:51 -0500)]
19886: Update API documentation notes about container request logs
The existing note was already out-of-date: crunch-run would record logs
about 30 minutes after starting the container. With recent changes,
crunch-run now records logs shortly after starting the container. Update
the note accordingly.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:44:55 +0000 (14:44 -0500)]
19886: Correct API documentation notes about container logs
The documented limitation does not exist. In fact, there's a limitation
in the API server that if you want a container's logs propagated to its
associated container request(s), you *must* pass in a portable data
hash. crunch-run consistently updates container records with a portable
data hash in the log field for this reason. Update the note to reflect
this.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:35:58 +0000 (14:35 -0500)]
19886: crunch-run records initial log with PDH
The API server will only propagate a container's log collection to
container requests when it is specified with a portable data hash. See
the top of ContainerRequest#update_collections.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:34:00 +0000 (14:34 -0500)]
19886: Tighten test regexps for better validation
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 18 Jan 2023 19:33:41 +0000 (14:33 -0500)]
19886: DRY up test code for better readability
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Wed, 18 Jan 2023 15:11:06 +0000 (10:11 -0500)]
Merge branch '18693-dedup-permissions'
fixes #18693
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 18 Jan 2023 14:58:49 +0000 (09:58 -0500)]
19320: Fix AWS InstanceStatus and SpotPrice API usage.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 18 Jan 2023 14:26:55 +0000 (09:26 -0500)]
Merge branch '19896-ldap-tls-downgrade'
fixes #19896
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 22:51:23 +0000 (17:51 -0500)]
19320: Fix env var leak in test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 21:58:33 +0000 (16:58 -0500)]
18693: Use row locking only if enabled in config.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:35:29 +0000 (11:35 -0500)]
19896: Mention TLS version error message in config doc.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:27:37 +0000 (11:27 -0500)]
19896: Use StartTLS + MinTLSVersion regardless of Insecure flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:20:51 +0000 (11:20 -0500)]
Merge branch '19890-id-bigint'
fixes #19890
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 17 Jan 2023 16:16:56 +0000 (11:16 -0500)]
19320: Use AWS spot price data to calculate container cost.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 16 Jan 2023 22:35:41 +0000 (17:35 -0500)]
19890: Note slow upgrade due to schema migration.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 16 Jan 2023 22:26:57 +0000 (17:26 -0500)]
Merge branch '19597-multipart-request'
fixes #19597
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 16 Jan 2023 22:26:41 +0000 (17:26 -0500)]
19597: Comment explicit ParseForm usage.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 13 Jan 2023 20:14:56 +0000 (15:14 -0500)]
19385: Continuing to fix tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Fri, 13 Jan 2023 20:10:56 +0000 (15:10 -0500)]
Merge branch '19895-google-api-python-client-2'
Closes #19895.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Fri, 13 Jan 2023 17:44:08 +0000 (12:44 -0500)]
19385: More test fixing
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Thu, 12 Jan 2023 21:51:12 +0000 (16:51 -0500)]
19886: Save log collection when crunch-run starts a container
This allows users to see information about the node where the container
started, including through Workbench 2.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 13 Jan 2023 14:33:18 +0000 (09:33 -0500)]
19886: Refine crunch-run output collection tests
These tests are currently not working as designed because the API call
they're looking for is in a different API client. They happen to be
passing because they never assert they actually found the collection
they're looking for. Restructure the search loop to fix both those
problems.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 12 Jan 2023 21:48:12 +0000 (16:48 -0500)]
19886: Bugfix UUID generation in crunch-run tests
The previous code was generating UUIDs with a 30-character last part.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Fri, 13 Jan 2023 05:27:14 +0000 (00:27 -0500)]
Merge branch '19894-pg-access'
fixes #19894
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Thu, 12 Jan 2023 22:09:54 +0000 (17:09 -0500)]
19385: Fixing tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 12 Jan 2023 20:53:20 +0000 (15:53 -0500)]
19385: Work in progress checkpoint, submitting uses wrappers
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tested-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Tue, 10 Jan 2023 16:53:56 +0000 (11:53 -0500)]
19894: Copyedit the 2.5.0 PostgreSQL upgrade note
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Mon, 9 Jan 2023 20:39:54 +0000 (15:39 -0500)]
19385: new_upload_workflow work in progress
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Mon, 9 Jan 2023 16:43:31 +0000 (11:43 -0500)]
19894: Update documentation re: dispatcher connecting to postgresql.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Mon, 9 Jan 2023 14:14:22 +0000 (09:14 -0500)]
19895: Upgrade google-api-python-client
Version 2.0 tries to load all discovery documents from a local cache. It
added a new argument to discovery.build(static_discovery=False) to
retrieve the discovery document from the network.
Version 2.1 improves backwards compatibility by having static_discovery
default to False when you pass an explicit discoveryServiceUrl to
discovery.build(). We always do this (see
arvados.api.normalize_api_kwargs), so we can upgrade to version 2.1
without making any changes to our own SDK.
Doing so is just good hygiene, and makes life easier for users who want
to build projects that use both Google services and Arvados.
Version 2.1 was released on 2021-03-31, so this new version requirement
should be easy enough for most users to satisfy.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Fri, 6 Jan 2023 22:21:07 +0000 (17:21 -0500)]
19385: Messy work in progress for uploading workflows to collections
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Wed, 4 Jan 2023 23:35:03 +0000 (18:35 -0500)]
19896: Configurable minimum TLS version for LDAP connection.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Wed, 4 Jan 2023 15:31:04 +0000 (10:31 -0500)]
19425: Fix rebase error
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 13 Sep 2022 20:50:09 +0000 (16:50 -0400)]
19425: Bump to released cwltool & schema-salad
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Mon, 12 Sep 2022 03:05:37 +0000 (23:05 -0400)]
19425: Set loadingContext.skip_resolve_all
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Mon, 12 Sep 2022 02:07:50 +0000 (22:07 -0400)]
19425: Clear codegen_idx before reloading
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Sat, 10 Sep 2022 16:29:51 +0000 (12:29 -0400)]
19425: Fix passing --fast-parser
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 9 Sep 2022 21:09:55 +0000 (17:09 -0400)]
19425: cwltool dev
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 9 Sep 2022 19:26:43 +0000 (15:26 -0400)]
19425: cwltool dev version bump
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 9 Sep 2022 18:16:44 +0000 (14:16 -0400)]
19425: Add experimental --fast-parser
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Mon, 2 Jan 2023 15:11:55 +0000 (10:11 -0500)]
19890: Use 64 bit ints for id columns.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Wed, 28 Dec 2022 20:15:03 +0000 (15:15 -0500)]
Update container state table headers
"Sgnificance" was a typo, and the immediate problem I wanted to
fix. "Description" is what we usually use and works fine here, so use
that.
I expanded "State" to "State value" to try to help clarify this is the
actual value in the data.
No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Tue, 20 Dec 2022 15:03:52 +0000 (10:03 -0500)]
Update version numbers
refs #19647
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Wed, 21 Dec 2022 19:31:26 +0000 (14:31 -0500)]
19597: Parse multipart/form-data request body.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 20 Dec 2022 16:39:01 +0000 (11:39 -0500)]
18693: Add row locking.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Tue, 20 Dec 2022 15:13:18 +0000 (10:13 -0500)]
Merge branch '19844-paging' refs #19844
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Tue, 20 Dec 2022 01:23:21 +0000 (20:23 -0500)]
18693: Remove existing redundant permission links from db.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 19 Dec 2022 16:31:00 +0000 (11:31 -0500)]
18693: Delete redundant login permission links on update/delete.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 19 Dec 2022 01:19:47 +0000 (20:19 -0500)]
18693: Return existing instead of creating redundant can_login link.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Sun, 18 Dec 2022 13:55:03 +0000 (08:55 -0500)]
18693: De-duplicate permission links on create/update/delete.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 22:18:33 +0000 (17:18 -0500)]
Merge branch '19880-wb1-pw-login-fix' refs #19880
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 21:53:02 +0000 (16:53 -0500)]
19880: Fix tests by restoring test hook
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 21:08:26 +0000 (16:08 -0500)]
19880: Fix tests
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 19:36:50 +0000 (14:36 -0500)]
19880: Password based login fix
Now sends user to wb1 front page instead of controller login endpoint
that returns an error.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 15:46:19 +0000 (10:46 -0500)]
Add missing test file refs #19688
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Fri, 16 Dec 2022 15:25:20 +0000 (10:25 -0500)]
Test needs to ignore varying "acrContainerImage" field
refs #19688
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 22:09:12 +0000 (17:09 -0500)]
Merge branch '19704-aneta-r-sdk' refs #19704
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 19:05:14 +0000 (14:05 -0500)]
Merge branch '19872-mnt-cache-limits' refs #19872
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:55:48 +0000 (13:55 -0500)]
19704: Fix R SDK documentation build.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:21:41 +0000 (13:21 -0500)]
19872: Fix call to setrlimit
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:18:12 +0000 (13:18 -0500)]
19872: Fix typo
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 18:14:02 +0000 (13:14 -0500)]
19872: Use fewer file handles by default, but increase the limit in arv-mount
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Thu, 15 Dec 2022 17:57:31 +0000 (12:57 -0500)]
Don't record varying git data for copy_deps test
refs #19688
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Thu, 15 Dec 2022 15:56:03 +0000 (10:56 -0500)]
19704: Update call signature of test fake.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Wed, 14 Dec 2022 20:48:53 +0000 (15:48 -0500)]
Merge branch '19847-cwl-disk-cache-size' refs #19847
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Wed, 14 Dec 2022 20:33:58 +0000 (15:33 -0500)]
Update copy_deps test, refs #19688
It now creates an additional collection containing the workflow text,
the test needed to be updated to reflect that.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Wed, 14 Dec 2022 15:09:52 +0000 (10:09 -0500)]
19847: Update test for new API default disk cache behavior.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Peter Amstutz [Wed, 14 Dec 2022 14:36:54 +0000 (14:36 +0000)]
19872: Add a timeout to wait()
Add a timeout when waiting for a cache slot to avoid the possibility
of waiting forever on wait().
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Peter Amstutz [Tue, 13 Dec 2022 22:35:44 +0000 (22:35 +0000)]
19872: Set max_slots lower because mmap uses another FD
- Reduce max slots to 3/8 max fds instead of 1/2 because mmap() uses a
second file descriptor, and we keep the original file descriptor open
for flock()
- Rework how cache slots are allocated to try evicting things _before_
allocating a new cache slot, so the cache should be somewhat better
behaved about staying within its configured limits.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Tom Clegg [Tue, 13 Dec 2022 21:54:11 +0000 (16:54 -0500)]
Merge branch '19709-boot-db-migrate'
closes #19709
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>