arvados.git
21 months ago19385: Clean up debug prints
Peter Amstutz [Mon, 6 Feb 2023 19:50:11 +0000 (14:50 -0500)]
19385: Clean up debug prints

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19385: Fix tests because wrapper collection now includes original files
Peter Amstutz [Fri, 3 Feb 2023 21:38:50 +0000 (16:38 -0500)]
19385: Fix tests because wrapper collection now includes original files

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch 'main' into 19385-cwl-fast-pack
Peter Amstutz [Fri, 3 Feb 2023 19:11:09 +0000 (14:11 -0500)]
Merge branch 'main' into 19385-cwl-fast-pack

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '19954-permission-dedup-doc'
Tom Clegg [Fri, 3 Feb 2023 14:50:41 +0000 (09:50 -0500)]
Merge branch '19954-permission-dedup-doc'

closes #19954

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19954: Update wording to admit multiple overlapping permission links. 19954-permission-dedup-doc
Tom Clegg [Thu, 2 Feb 2023 21:58:10 +0000 (16:58 -0500)]
19954: Update wording to admit multiple overlapping permission links.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19624-priority-doc'
Tom Clegg [Thu, 2 Feb 2023 21:53:40 +0000 (16:53 -0500)]
Merge branch '19624-priority-doc'

fixes #19624

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19926-pysdk-install-docs-fix'
Brett Smith [Thu, 2 Feb 2023 21:02:25 +0000 (16:02 -0500)]
Merge branch '19926-pysdk-install-docs-fix'

Closes #19926.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19926: Remove "option 1/2" remnants from the Python SDK install docs
Brett Smith [Thu, 2 Feb 2023 21:01:31 +0000 (16:01 -0500)]
19926: Remove "option 1/2" remnants from the Python SDK install docs

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months agoMerge branch '18075-max-compute-vms'
Tom Clegg [Thu, 2 Feb 2023 20:20:12 +0000 (15:20 -0500)]
Merge branch '18075-max-compute-vms'

fixes #18075

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19954: Move special behavior comments into the method sections.
Tom Clegg [Thu, 2 Feb 2023 20:17:40 +0000 (15:17 -0500)]
19954: Move special behavior comments into the method sections.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19624: Layout container request lifecycle in two columns 19624-priority-doc
Brett Smith [Thu, 2 Feb 2023 16:31:36 +0000 (11:31 -0500)]
19624: Layout container request lifecycle in two columns

Provide a more readable layout. Normal full lifecycle on the left, early
end states on the right.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19926: Modernize pip install instructions
Brett Smith [Thu, 2 Feb 2023 15:39:37 +0000 (10:39 -0500)]
19926: Modernize pip install instructions

Running `python3 -m pip` instead of just plain `pip` helps ensure that
you install packages for the version of Python that you intend.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19926: Use noteboxes consistently in PySDK install instructions
Brett Smith [Thu, 2 Feb 2023 15:38:20 +0000 (10:38 -0500)]
19926: Use noteboxes consistently in PySDK install instructions

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19926: Flatten structure of PySDK install instructions
Brett Smith [Thu, 2 Feb 2023 15:33:32 +0000 (10:33 -0500)]
19926: Flatten structure of PySDK install instructions

The way there were "two tracks" of install and test instructions on the
page was confusing to readers. (Especially since we did not number those
options consistently.) Incorporate the test instructions into the
install instructions, so the reader only has to follow and care about
one specific method, and then they're done.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago18075: Note obsolete/deprecated config. 18075-max-compute-vms
Tom Clegg [Wed, 1 Feb 2023 18:51:35 +0000 (13:51 -0500)]
18075: Note obsolete/deprecated config.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago18075: Update AtQuota comment, check quota in Create, log at limit.
Tom Clegg [Wed, 1 Feb 2023 18:46:40 +0000 (13:46 -0500)]
18075: Update AtQuota comment, check quota in Create, log at limit.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19624: Layout graph L->R, say "update" instead of "assign".
Tom Clegg [Wed, 1 Feb 2023 19:14:14 +0000 (14:14 -0500)]
19624: Layout graph L->R, say "update" instead of "assign".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19954: Update API docs to reflect permission link deduplication.
Tom Clegg [Wed, 1 Feb 2023 18:34:51 +0000 (13:34 -0500)]
19954: Update API docs to reflect permission link deduplication.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19897-fuse-enotsup'
Brett Smith [Wed, 1 Feb 2023 15:49:41 +0000 (10:49 -0500)]
Merge branch '19897-fuse-enotsup'

Closes #19897.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Use binary mode in cookbook download+upload examples
Brett Smith [Wed, 1 Feb 2023 14:10:33 +0000 (09:10 -0500)]
19792: Use binary mode in cookbook download+upload examples

This incorporates a suggestion from review to use a binary mode in a way
that makes the recipes more robust. It's very likely that readers will
want to work with binary files at least as much as text files, and these
recipes still work for text files too for verbatim copies.

Refs #19792.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Fix typos throughout
Brett Smith [Wed, 1 Feb 2023 14:07:08 +0000 (09:07 -0500)]
19792: Fix typos throughout

Refs #19792.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19624: Redo diagram to show CR and container states.
Tom Clegg [Tue, 31 Jan 2023 21:26:06 +0000 (16:26 -0500)]
19624: Redo diagram to show CR and container states.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19624: Redo diagram to show container states.
Tom Clegg [Tue, 31 Jan 2023 15:57:07 +0000 (10:57 -0500)]
19624: Redo diagram to show container states.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19792-pysdk-cookbook'
Brett Smith [Mon, 30 Jan 2023 23:40:43 +0000 (18:40 -0500)]
Merge branch '19792-pysdk-cookbook'

Closes #19792.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Add 'wb' as an example Collection.open mode argument
Brett Smith [Mon, 30 Jan 2023 23:38:02 +0000 (18:38 -0500)]
19792: Add 'wb' as an example Collection.open mode argument

Give readers a hint that we fully support the mode argument, not just
the basic r/w/a options. Ideally they can look up the reference for full
details.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19897: Add tests for writing immutable mount objects
Brett Smith [Mon, 30 Jan 2023 23:29:47 +0000 (18:29 -0500)]
19897: Add tests for writing immutable mount objects

I originally wrote these expecting to get them to return ENOTSUP as
well. However, that doesn't happen now because of how the Operations
class calls Directory.writable() and turns False into EPERM. I'm
committing the tests since they're useful as-is, with a note that I'd
like to convert them to ENOTSUP in the future.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19897: Return ENOTSUP when the user tries to make a file in a project
Brett Smith [Mon, 30 Jan 2023 23:24:12 +0000 (18:24 -0500)]
19897: Return ENOTSUP when the user tries to make a file in a project

Before this commit, the NotImplementedError is propagating up to
catch_exceptions, which turns it into EIO. "I/O error" sounds like a
problem with the underlying systems and stresses out users. "Operation
not supported" should better communicate to the user that they're doing
something unexpected and should double-check their work.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19624: Container request state transition / lifecycle diagram.
Tom Clegg [Mon, 30 Jan 2023 20:43:08 +0000 (15:43 -0500)]
19624: Container request state transition / lifecycle diagram.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago18075: Remove tests for crunch1 Node.
Tom Clegg [Mon, 30 Jan 2023 15:23:21 +0000 (10:23 -0500)]
18075: Remove tests for crunch1 Node.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19792: Improve wording of keyset_list_all first argument comment
Brett Smith [Sun, 29 Jan 2023 18:23:32 +0000 (13:23 -0500)]
19792: Improve wording of keyset_list_all first argument comment

Explain the purpose of the argument in addition to giving the reader
guardrails to help them use it correctly.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Split container request queue recipes into separate sections
Brett Smith [Sun, 29 Jan 2023 18:22:52 +0000 (13:22 -0500)]
19792: Split container request queue recipes into separate sections

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Add PySDK cookbook recipe to walk a collection
Brett Smith [Sun, 29 Jan 2023 18:28:51 +0000 (13:28 -0500)]
19792: Add PySDK cookbook recipe to walk a collection

After working on the container request recipes, I realized this was a
gap in the cookbook: how do you work on a collection if you don't know
its structure? This starts to document that.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Split container request recipes into separate cookbook sections
Brett Smith [Sun, 29 Jan 2023 18:03:48 +0000 (13:03 -0500)]
19792: Split container request recipes into separate cookbook sections

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Split collection recipes into separate cookbook sections
Brett Smith [Sun, 29 Jan 2023 18:01:53 +0000 (13:01 -0500)]
19792: Split collection recipes into separate cookbook sections

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Expand explanation of vocabulary in PySDK cookbook
Brett Smith [Sun, 29 Jan 2023 18:00:28 +0000 (13:00 -0500)]
19792: Expand explanation of vocabulary in PySDK cookbook

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Remove "none" from list of permission names
Brett Smith [Sun, 29 Jan 2023 17:59:20 +0000 (12:59 -0500)]
19792: Remove "none" from list of permission names

I misunderstood the API documentation. When it described "none" as a
possible permission level, I took that to mean it was a possible
permission name.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Add overview sentence to PySDK cookbook introduction
Brett Smith [Sun, 29 Jan 2023 17:59:00 +0000 (12:59 -0500)]
19792: Add overview sentence to PySDK cookbook introduction

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago18075: Kill off MaxComputeVMs properly.
Tom Clegg [Fri, 27 Jan 2023 21:33:30 +0000 (16:33 -0500)]
18075: Kill off MaxComputeVMs properly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19993-groupby-forupdate'
Tom Clegg [Fri, 27 Jan 2023 19:56:36 +0000 (14:56 -0500)]
Merge branch '19993-groupby-forupdate'

fixes #19993

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19923-keep-balance-chunk'
Tom Clegg [Fri, 27 Jan 2023 19:56:01 +0000 (14:56 -0500)]
Merge branch '19923-keep-balance-chunk'

closes #19923

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago18075: Add CloudVMs.MaxInstances config, retire MaxComputeVMs.
Tom Clegg [Fri, 27 Jan 2023 19:54:47 +0000 (14:54 -0500)]
18075: Add CloudVMs.MaxInstances config, retire MaxComputeVMs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19923: Report heap and stack stats from Go runtime. 19923-keep-balance-chunk
Tom Clegg [Fri, 27 Jan 2023 18:30:09 +0000 (13:30 -0500)]
19923: Report heap and stack stats from Go runtime.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19993: Fix postgresql<12 incompatibility. 19993-groupby-forupdate
Tom Clegg [Fri, 27 Jan 2023 16:26:59 +0000 (11:26 -0500)]
19993: Fix postgresql<12 incompatibility.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19923: Sanity check provided chunk prefix.
Tom Clegg [Fri, 27 Jan 2023 16:01:15 +0000 (11:01 -0500)]
19923: Sanity check provided chunk prefix.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19923: Remove debug sleep.
Tom Clegg [Fri, 27 Jan 2023 15:50:29 +0000 (10:50 -0500)]
19923: Remove debug sleep.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19385: Update cwl dependencies to stable releases
Peter Amstutz [Fri, 27 Jan 2023 15:02:09 +0000 (10:02 -0500)]
19385: Update cwl dependencies to stable releases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months agoMerge branch '19320-spot-pricing'
Tom Clegg [Fri, 27 Jan 2023 14:51:58 +0000 (09:51 -0500)]
Merge branch '19320-spot-pricing'

refs #19320

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19923: Log keep-balance memory usage.
Tom Clegg [Thu, 26 Jan 2023 22:47:17 +0000 (17:47 -0500)]
19923: Log keep-balance memory usage.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months agoMerge branch '19960-vocabulary-system-keys'
Brett Smith [Thu, 26 Jan 2023 19:23:36 +0000 (14:23 -0500)]
Merge branch '19960-vocabulary-system-keys'

Refs #19960.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19385: Fix typo, passing with --fast-parser now
Peter Amstutz [Thu, 26 Jan 2023 17:56:12 +0000 (12:56 -0500)]
19385: Fix typo, passing with --fast-parser now

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19923: Log progress while reading keepstore indexes.
Tom Clegg [Thu, 26 Jan 2023 16:14:28 +0000 (11:14 -0500)]
19923: Log progress while reading keepstore indexes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19385: fast-parser fixups
Peter Amstutz [Thu, 26 Jan 2023 16:06:23 +0000 (11:06 -0500)]
19385: fast-parser fixups

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19792: Correct filter for revoking permissions example
Brett Smith [Thu, 26 Jan 2023 14:17:19 +0000 (09:17 -0500)]
19792: Correct filter for revoking permissions example

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Simplify example for modifying permissions
Brett Smith [Thu, 26 Jan 2023 14:16:53 +0000 (09:16 -0500)]
19792: Simplify example for modifying permissions

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Add missing words
Brett Smith [Thu, 26 Jan 2023 14:16:40 +0000 (09:16 -0500)]
19792: Add missing words

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19320: Fix live test. 19320-spot-pricing
Tom Clegg [Wed, 25 Jan 2023 22:03:37 +0000 (17:03 -0500)]
19320: Fix live test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19792: Add cookbook recipes for managing permissions
Brett Smith [Wed, 25 Jan 2023 21:54:50 +0000 (16:54 -0500)]
19792: Add cookbook recipes for managing permissions

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Add cookbook recipe to update properties generally
Brett Smith [Wed, 25 Jan 2023 18:37:10 +0000 (13:37 -0500)]
19792: Add cookbook recipe to update properties generally

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Reorganize cookbook vocabulary section into properties section
Brett Smith [Wed, 25 Jan 2023 15:05:32 +0000 (10:05 -0500)]
19792: Reorganize cookbook vocabulary section into properties section

Presenting this as a section about properties gives it more room to grow
and gives more context to the vocabulary discussion.

It's okay to drop the working-with-properties anchor because it was
never merged with main.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Update sharing link recipe to handle v2 API tokens
Brett Smith [Wed, 25 Jan 2023 14:20:13 +0000 (09:20 -0500)]
19792: Update sharing link recipe to handle v2 API tokens

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Revamp the Python SDK cookbook
Brett Smith [Mon, 2 Jan 2023 20:27:07 +0000 (15:27 -0500)]
19792: Revamp the Python SDK cookbook

* Organize the recipes into related sections based on what they work
  on.

* Add a preface to each recipe to explain when you might use it, give
  API background, etc.

* Write the recipes in a style that emphasizes readability with long
  variable names, lots of line breaks, and comments highlighting
  key mechanics and variations.

* Modernize SDK usage in the recipes.

* Add links to supporting documentation where available.

As much as practical, I have maintained existing anchor names, even when
I updated header names for more consistent style.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19792: Add anchors to some dedicated API methods
Brett Smith [Mon, 2 Jan 2023 20:24:32 +0000 (15:24 -0500)]
19792: Add anchors to some dedicated API methods

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19791: Remove Python SDK "Notes" section
Brett Smith [Wed, 28 Dec 2022 21:10:18 +0000 (16:10 -0500)]
19791: Remove Python SDK "Notes" section

This is now covered more helpfully, and in more detail, in the API
client overview.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19385: Passing the v1.2.1 conformance tests!
Peter Amstutz [Wed, 25 Jan 2023 02:05:12 +0000 (21:05 -0500)]
19385: Passing the v1.2.1 conformance tests!

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19385: Down to just a few failing conformance tests, I hope
Peter Amstutz [Wed, 25 Jan 2023 00:40:25 +0000 (19:40 -0500)]
19385: Down to just a few failing conformance tests, I hope

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19320: Deduplicate instance types in spot price request.
Tom Clegg [Tue, 24 Jan 2023 23:04:07 +0000 (18:04 -0500)]
19320: Deduplicate instance types in spot price request.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19320: Comment re future use of spot attr in priceKey.
Tom Clegg [Tue, 24 Jan 2023 22:44:27 +0000 (17:44 -0500)]
19320: Comment re future use of spot attr in priceKey.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19320: Disable spot price checks if configured update interval <= 0.
Tom Clegg [Tue, 24 Jan 2023 22:40:34 +0000 (17:40 -0500)]
19320: Disable spot price checks if configured update interval <= 0.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19320: Account for AddedScratch in spot instance cost estimates.
Tom Clegg [Tue, 24 Jan 2023 22:31:33 +0000 (17:31 -0500)]
19320: Account for AddedScratch in spot instance cost estimates.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19385: Finally passing schemadef test.
Peter Amstutz [Tue, 24 Jan 2023 22:27:30 +0000 (17:27 -0500)]
19385: Finally passing schemadef test.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19385: more WIP
Peter Amstutz [Tue, 24 Jan 2023 22:08:58 +0000 (17:08 -0500)]
19385: more WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19385: WIP for complex schemadef + $import case
Peter Amstutz [Tue, 24 Jan 2023 21:03:56 +0000 (16:03 -0500)]
19385: WIP for complex schemadef + $import case

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

21 months ago19923: keep-balance option to process a subset of blocks.
Tom Clegg [Tue, 24 Jan 2023 16:29:46 +0000 (11:29 -0500)]
19923: keep-balance option to process a subset of blocks.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

21 months ago19960: Fix container_uuid property name for intermediate collections
Brett Smith [Tue, 24 Jan 2023 14:06:44 +0000 (09:06 -0500)]
19960: Fix container_uuid property name for intermediate collections

container_uuid is the name documented in our API reference, and now
expected by vocabulary.go's system tags.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19960: Add container_uuid to systemTagKeys
Brett Smith [Tue, 24 Jan 2023 13:59:05 +0000 (08:59 -0500)]
19960: Add container_uuid to systemTagKeys

This is set by arvados-cwl-runner, and already documented in our API
reference.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

21 months ago19960: Organize systemTagKeys
Brett Smith [Tue, 24 Jan 2023 13:52:52 +0000 (08:52 -0500)]
19960: Organize systemTagKeys

Note what properties go with what resource types, and why.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19320: Log instance price changes in crunch-run.txt.
Tom Clegg [Mon, 23 Jan 2023 15:31:41 +0000 (10:31 -0500)]
19320: Log instance price changes in crunch-run.txt.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19385: Reference rewriting fixups
Peter Amstutz [Fri, 20 Jan 2023 19:20:14 +0000 (14:20 -0500)]
19385: Reference rewriting fixups

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months agoMerge branch '19886-crunch-run-early-log-commit'
Brett Smith [Fri, 20 Jan 2023 16:23:59 +0000 (11:23 -0500)]
Merge branch '19886-crunch-run-early-log-commit'

Closes #19886.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19886: Describe what component updates container logs more generically
Brett Smith [Fri, 20 Jan 2023 16:21:28 +0000 (11:21 -0500)]
19886: Describe what component updates container logs more generically

We want to make clear that an Arvados system component does this,
without being overly committal about which component it is. "Dispatcher"
might be taken to mean the Crunch dispatcher specifically. Right now
it's crunch-run, but we don't need to commit to that level of
detail. Settle on "Crunch system" per discussion with Tom in chat.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19886: Slim down API documentation about container logs
Brett Smith [Fri, 20 Jan 2023 14:11:59 +0000 (09:11 -0500)]
19886: Slim down API documentation about container logs

While the API server will let you set this field to a UUID, that
behavior is already poorly supported, and on reflection we're not sure
we want to support it. Document this the way crunch-run uses it and
avoid presenting too much detail for most readers.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months agoMerge branch '19917-retry-scheduling-parameters'
Brett Smith [Fri, 20 Jan 2023 14:07:28 +0000 (09:07 -0500)]
Merge branch '19917-retry-scheduling-parameters'

Closes #19917.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19917: Synthesize scheduling parameters when retrying a container
Brett Smith [Thu, 19 Jan 2023 20:39:29 +0000 (15:39 -0500)]
19917: Synthesize scheduling parameters when retrying a container

When we retry a cancelled container, there may be any number of
container requests that want it, each with their own scheduling
parameters. Create a new set of scheduling parameters for the new
container where each parameter has the most lax setting from all the
outstanding container requests.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19385: Unit tests pass
Peter Amstutz [Thu, 19 Jan 2023 17:49:02 +0000 (12:49 -0500)]
19385: Unit tests pass

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago19385: Fixing tests WIP
Peter Amstutz [Thu, 19 Jan 2023 16:22:32 +0000 (11:22 -0500)]
19385: Fixing tests WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago19385: Fix submit_secret
Peter Amstutz [Wed, 18 Jan 2023 22:14:47 +0000 (17:14 -0500)]
19385: Fix submit_secret

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago19385: update_refs fixes prefixes
Peter Amstutz [Wed, 18 Jan 2023 21:55:57 +0000 (16:55 -0500)]
19385: update_refs fixes prefixes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago19385: Fix submit secrets test
Peter Amstutz [Wed, 18 Jan 2023 21:50:06 +0000 (16:50 -0500)]
19385: Fix submit secrets test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22 months ago19886: Update API documentation notes about container request logs
Brett Smith [Wed, 18 Jan 2023 19:51:05 +0000 (14:51 -0500)]
19886: Update API documentation notes about container request logs

The existing note was already out-of-date: crunch-run would record logs
about 30 minutes after starting the container. With recent changes,
crunch-run now records logs shortly after starting the container. Update
the note accordingly.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19886: Correct API documentation notes about container logs
Brett Smith [Wed, 18 Jan 2023 19:44:55 +0000 (14:44 -0500)]
19886: Correct API documentation notes about container logs

The documented limitation does not exist. In fact, there's a limitation
in the API server that if you want a container's logs propagated to its
associated container request(s), you *must* pass in a portable data
hash. crunch-run consistently updates container records with a portable
data hash in the log field for this reason. Update the note to reflect
this.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19886: crunch-run records initial log with PDH
Brett Smith [Wed, 18 Jan 2023 19:35:58 +0000 (14:35 -0500)]
19886: crunch-run records initial log with PDH

The API server will only propagate a container's log collection to
container requests when it is specified with a portable data hash. See
the top of ContainerRequest#update_collections.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19886: Tighten test regexps for better validation
Brett Smith [Wed, 18 Jan 2023 19:34:00 +0000 (14:34 -0500)]
19886: Tighten test regexps for better validation

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months ago19886: DRY up test code for better readability
Brett Smith [Wed, 18 Jan 2023 19:33:41 +0000 (14:33 -0500)]
19886: DRY up test code for better readability

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22 months agoMerge branch '18693-dedup-permissions'
Tom Clegg [Wed, 18 Jan 2023 15:11:06 +0000 (10:11 -0500)]
Merge branch '18693-dedup-permissions'

fixes #18693

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19320: Fix AWS InstanceStatus and SpotPrice API usage.
Tom Clegg [Wed, 18 Jan 2023 14:58:49 +0000 (09:58 -0500)]
19320: Fix AWS InstanceStatus and SpotPrice API usage.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months agoMerge branch '19896-ldap-tls-downgrade'
Tom Clegg [Wed, 18 Jan 2023 14:26:55 +0000 (09:26 -0500)]
Merge branch '19896-ldap-tls-downgrade'

fixes #19896

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19320: Fix env var leak in test.
Tom Clegg [Tue, 17 Jan 2023 22:51:23 +0000 (17:51 -0500)]
19320: Fix env var leak in test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago18693: Use row locking only if enabled in config. 18693-dedup-permissions
Tom Clegg [Tue, 17 Jan 2023 21:58:33 +0000 (16:58 -0500)]
18693: Use row locking only if enabled in config.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19896: Mention TLS version error message in config doc. 19896-ldap-tls-downgrade
Tom Clegg [Tue, 17 Jan 2023 16:35:29 +0000 (11:35 -0500)]
19896: Mention TLS version error message in config doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22 months ago19896: Use StartTLS + MinTLSVersion regardless of Insecure flag.
Tom Clegg [Tue, 17 Jan 2023 16:27:37 +0000 (11:27 -0500)]
19896: Use StartTLS + MinTLSVersion regardless of Insecure flag.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>