arvados.git
23 months ago19886: crunch-run records initial log with PDH
Brett Smith [Wed, 18 Jan 2023 19:35:58 +0000 (14:35 -0500)]
19886: crunch-run records initial log with PDH

The API server will only propagate a container's log collection to
container requests when it is specified with a portable data hash. See
the top of ContainerRequest#update_collections.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months ago19886: Tighten test regexps for better validation
Brett Smith [Wed, 18 Jan 2023 19:34:00 +0000 (14:34 -0500)]
19886: Tighten test regexps for better validation

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months ago19886: DRY up test code for better readability
Brett Smith [Wed, 18 Jan 2023 19:33:41 +0000 (14:33 -0500)]
19886: DRY up test code for better readability

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months ago19886: Save log collection when crunch-run starts a container
Brett Smith [Thu, 12 Jan 2023 21:51:12 +0000 (16:51 -0500)]
19886: Save log collection when crunch-run starts a container

This allows users to see information about the node where the container
started, including through Workbench 2.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months ago19886: Refine crunch-run output collection tests
Brett Smith [Fri, 13 Jan 2023 14:33:18 +0000 (09:33 -0500)]
19886: Refine crunch-run output collection tests

These tests are currently not working as designed because the API call
they're looking for is in a different API client. They happen to be
passing because they never assert they actually found the collection
they're looking for. Restructure the search loop to fix both those
problems.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months ago19886: Bugfix UUID generation in crunch-run tests
Brett Smith [Thu, 12 Jan 2023 21:48:12 +0000 (16:48 -0500)]
19886: Bugfix UUID generation in crunch-run tests

The previous code was generating UUIDs with a 30-character last part.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months agoMerge branch '19894-pg-access'
Tom Clegg [Fri, 13 Jan 2023 05:27:14 +0000 (00:27 -0500)]
Merge branch '19894-pg-access'

fixes #19894

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

23 months ago19894: Copyedit the 2.5.0 PostgreSQL upgrade note 19894-pg-access
Brett Smith [Tue, 10 Jan 2023 16:53:56 +0000 (11:53 -0500)]
19894: Copyedit the 2.5.0 PostgreSQL upgrade note

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months ago19894: Update documentation re: dispatcher connecting to postgresql.
Tom Clegg [Mon, 9 Jan 2023 16:43:31 +0000 (11:43 -0500)]
19894: Update documentation re: dispatcher connecting to postgresql.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

23 months agoUpdate container state table headers
Brett Smith [Wed, 28 Dec 2022 20:15:03 +0000 (15:15 -0500)]
Update container state table headers

"Sgnificance" was a typo, and the immediate problem I wanted to
fix. "Description" is what we usually use and works fine here, so use
that.

I expanded "State" to "State value" to try to help clarify this is the
actual value in the data.

No issue #.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

23 months agoUpdate version numbers
Peter Amstutz [Tue, 20 Dec 2022 15:03:52 +0000 (10:03 -0500)]
Update version numbers

refs #19647

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19844-paging' refs #19844 2.5.0
Peter Amstutz [Tue, 20 Dec 2022 15:13:18 +0000 (10:13 -0500)]
Merge branch '19844-paging' refs #19844

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19880-wb1-pw-login-fix' refs #19880
Peter Amstutz [Fri, 16 Dec 2022 22:18:33 +0000 (17:18 -0500)]
Merge branch '19880-wb1-pw-login-fix' refs #19880

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19880: Fix tests by restoring test hook 19880-wb1-pw-login-fix
Peter Amstutz [Fri, 16 Dec 2022 21:53:02 +0000 (16:53 -0500)]
19880: Fix tests by restoring test hook

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19880: Fix tests
Peter Amstutz [Fri, 16 Dec 2022 21:08:26 +0000 (16:08 -0500)]
19880: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19880: Password based login fix
Peter Amstutz [Fri, 16 Dec 2022 19:36:50 +0000 (14:36 -0500)]
19880: Password based login fix

Now sends user to wb1 front page instead of controller login endpoint
that returns an error.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoAdd missing test file refs #19688
Peter Amstutz [Fri, 16 Dec 2022 15:46:19 +0000 (10:46 -0500)]
Add missing test file refs #19688

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoTest needs to ignore varying "acrContainerImage" field
Peter Amstutz [Fri, 16 Dec 2022 15:25:20 +0000 (10:25 -0500)]
Test needs to ignore varying "acrContainerImage" field

refs #19688

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19704-aneta-r-sdk' refs #19704
Peter Amstutz [Thu, 15 Dec 2022 22:09:12 +0000 (17:09 -0500)]
Merge branch '19704-aneta-r-sdk' refs #19704

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19872-mnt-cache-limits' refs #19872
Peter Amstutz [Thu, 15 Dec 2022 19:05:14 +0000 (14:05 -0500)]
Merge branch '19872-mnt-cache-limits' refs #19872

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19704: Fix R SDK documentation build. 19704-aneta-r-sdk
Peter Amstutz [Thu, 15 Dec 2022 18:55:48 +0000 (13:55 -0500)]
19704: Fix R SDK documentation build.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19872: Fix call to setrlimit 19872-mnt-cache-limits
Peter Amstutz [Thu, 15 Dec 2022 18:21:41 +0000 (13:21 -0500)]
19872: Fix call to setrlimit

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19872: Fix typo
Peter Amstutz [Thu, 15 Dec 2022 18:18:12 +0000 (13:18 -0500)]
19872: Fix typo

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19872: Use fewer file handles by default, but increase the limit in arv-mount
Peter Amstutz [Thu, 15 Dec 2022 18:14:02 +0000 (13:14 -0500)]
19872: Use fewer file handles by default, but increase the limit in arv-mount

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoDon't record varying git data for copy_deps test
Peter Amstutz [Thu, 15 Dec 2022 17:57:31 +0000 (12:57 -0500)]
Don't record varying git data for copy_deps test

refs #19688

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19704: Update call signature of test fake.
Tom Clegg [Thu, 15 Dec 2022 15:56:03 +0000 (10:56 -0500)]
19704: Update call signature of test fake.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '19847-cwl-disk-cache-size' refs #19847
Peter Amstutz [Wed, 14 Dec 2022 20:48:53 +0000 (15:48 -0500)]
Merge branch '19847-cwl-disk-cache-size' refs #19847

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoUpdate copy_deps test, refs #19688
Peter Amstutz [Wed, 14 Dec 2022 20:33:58 +0000 (15:33 -0500)]
Update copy_deps test, refs #19688

It now creates an additional collection containing the workflow text,
the test needed to be updated to reflect that.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19847: Update test for new API default disk cache behavior. 19847-cwl-disk-cache-size
Tom Clegg [Wed, 14 Dec 2022 15:09:52 +0000 (10:09 -0500)]
19847: Update test for new API default disk cache behavior.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19872: Add a timeout to wait()
Peter Amstutz [Wed, 14 Dec 2022 14:36:54 +0000 (14:36 +0000)]
19872: Add a timeout to wait()

Add a timeout when waiting for a cache slot to avoid the possibility
of waiting forever on wait().

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19872: Set max_slots lower because mmap uses another FD
Peter Amstutz [Tue, 13 Dec 2022 22:35:44 +0000 (22:35 +0000)]
19872: Set max_slots lower because mmap uses another FD

- Reduce max slots to 3/8 max fds instead of 1/2 because mmap() uses a
second file descriptor, and we keep the original file descriptor open
for flock()

- Rework how cache slots are allocated to try evicting things _before_
allocating a new cache slot, so the cache should be somewhat better
behaved about staying within its configured limits.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19709-boot-db-migrate'
Tom Clegg [Tue, 13 Dec 2022 21:54:11 +0000 (16:54 -0500)]
Merge branch '19709-boot-db-migrate'

closes #19709

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19847: Let server apply default disk cache size.
Tom Clegg [Tue, 13 Dec 2022 21:38:54 +0000 (16:38 -0500)]
19847: Let server apply default disk cache size.

Use disk cache when DefaultKeepCacheRAM==0 (instead of
DefaultKeepCacheDisk>0).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19847: Default to disk cache size = RAM size for all containers.
Tom Clegg [Tue, 13 Dec 2022 21:31:31 +0000 (16:31 -0500)]
19847: Default to disk cache size = RAM size for all containers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '19878-rails-html-sanitizer-upgrade'. Closes #19878
Lucas Di Pentima [Tue, 13 Dec 2022 20:34:20 +0000 (21:34 +0100)]
Merge branch '19878-rails-html-sanitizer-upgrade'. Closes #19878

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19878: Upgrades rails-html-sanitizer on RailsAPI & Workbench. 19878-rails-html-sanitizer-upgrade
Lucas Di Pentima [Tue, 13 Dec 2022 20:12:59 +0000 (21:12 +0100)]
19878: Upgrades rails-html-sanitizer on RailsAPI & Workbench.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoMerge branch '19877-loofah-upgrade'. Closes #19877
Lucas Di Pentima [Tue, 13 Dec 2022 20:06:59 +0000 (21:06 +0100)]
Merge branch '19877-loofah-upgrade'. Closes #19877

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19877: Upgrades loofah gem on RailsAPI & Woekbench 19877-loofah-upgrade
Lucas Di Pentima [Tue, 13 Dec 2022 19:45:52 +0000 (20:45 +0100)]
19877: Upgrades loofah gem on RailsAPI & Woekbench

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19709: Warn, not eror, on unexpected files in db/migrate/. 19709-boot-db-migrate
Tom Clegg [Tue, 13 Dec 2022 19:24:33 +0000 (14:24 -0500)]
19709: Warn, not eror, on unexpected files in db/migrate/.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19709: Test schema_migrations vs. db/migrate/*.rb.
Tom Clegg [Mon, 12 Dec 2022 21:46:03 +0000 (16:46 -0500)]
19709: Test schema_migrations vs. db/migrate/*.rb.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '19840-docker-port-number'
Tom Clegg [Fri, 9 Dec 2022 21:05:48 +0000 (16:05 -0500)]
Merge branch '19840-docker-port-number'

fixes #19840

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoUpdate resource requirements in test cases.
Tom Clegg [Fri, 9 Dec 2022 20:57:46 +0000 (15:57 -0500)]
Update resource requirements in test cases.

New disk cache defaults to 8 GiB, which does not necessarily fit in
/tmp on a dev/test machine (mine has a 4 GiB tmpfs).

refs #19847

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19709: Explain design in comments. Improve migration filename check.
Tom Clegg [Fri, 9 Dec 2022 20:01:17 +0000 (15:01 -0500)]
19709: Explain design in comments. Improve migration filename check.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19844: Get multiple pages of child CRs if necessary. 19844-paging
Tom Clegg [Thu, 8 Dec 2022 19:41:50 +0000 (14:41 -0500)]
19844: Get multiple pages of child CRs if necessary.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19844: Use 127.0.0.1 instead of localhost for test servers.
Tom Clegg [Thu, 8 Dec 2022 19:06:15 +0000 (14:06 -0500)]
19844: Use 127.0.0.1 instead of localhost for test servers.

Otherwise, depending on host config, nginx futilely tries ::1 first.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19840: Cleanup link added during test. 19840-docker-port-number
Tom Clegg [Fri, 9 Dec 2022 07:09:06 +0000 (02:09 -0500)]
19840: Cleanup link added during test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19840: Test listing saved images with names containing port numbers.
Tom Clegg [Thu, 8 Dec 2022 22:23:53 +0000 (17:23 -0500)]
19840: Test listing saved images with names containing port numbers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19840: Fix image name confusion when specified with explicit :443.
Tom Clegg [Thu, 8 Dec 2022 21:52:52 +0000 (16:52 -0500)]
19840: Fix image name confusion when specified with explicit :443.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19709: Apply pending rails migrations at service start/restart.
Tom Clegg [Mon, 14 Nov 2022 20:31:42 +0000 (15:31 -0500)]
19709: Apply pending rails migrations at service start/restart.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '19862-nokogiri-upgrade'. Closes #19862
Lucas Di Pentima [Thu, 8 Dec 2022 17:29:05 +0000 (18:29 +0100)]
Merge branch '19862-nokogiri-upgrade'. Closes #19862

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19862: Fix RandomUUID length. 19862-nokogiri-upgrade
Tom Clegg [Thu, 8 Dec 2022 17:06:55 +0000 (12:06 -0500)]
19862: Fix RandomUUID length.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years ago19862: Upgrades nokogiri and related gem on RailsAPI and Workbench.
Lucas Di Pentima [Thu, 8 Dec 2022 15:15:42 +0000 (16:15 +0100)]
19862: Upgrades nokogiri and related gem on RailsAPI and Workbench.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19791: Fix broken link to old PySDK examples page
Brett Smith [Thu, 8 Dec 2022 16:10:33 +0000 (11:10 -0500)]
19791: Fix broken link to old PySDK examples page

No issue #.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years agoMerge branch '19686-threadsafe-api-default'
Brett Smith [Thu, 8 Dec 2022 16:05:57 +0000 (11:05 -0500)]
Merge branch '19686-threadsafe-api-default'

Closes #19686.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19686: Specify ThreadSafeApiCache API version throughout
Brett Smith [Thu, 8 Dec 2022 15:51:58 +0000 (10:51 -0500)]
19686: Specify ThreadSafeApiCache API version throughout

This avoids logging a warning now that ThreadSafeApiCache accepts a
version argument just like other API client constructors.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19686: Fix test scaffolding after API changes
Brett Smith [Wed, 7 Dec 2022 20:21:43 +0000 (15:21 -0500)]
19686: Fix test scaffolding after API changes

Without clearing arv-put's API client this way, multiple calls to main()
reuse the same, pre-patch Keep object from ThreadSafeApiCache, so calls
don't get recorded as intended.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19686: Add tests for new Arvados client API
Brett Smith [Wed, 7 Dec 2022 04:16:28 +0000 (23:16 -0500)]
19686: Add tests for new Arvados client API

This commit ensures all public-facing functions have basic tests for
good inputs, as well as known-bad argument combinations (if any). This
includes testing functions like api_from_config that were previously
tested implicitly by other tests, but did not have their own dedicated
tests.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19686: api constructor returns ThreadSafeApiCache
Brett Smith [Tue, 29 Nov 2022 20:01:43 +0000 (15:01 -0500)]
19686: api constructor returns ThreadSafeApiCache

This is an API-compatible wrapper object that provides thread
safety. Returning this from api() helps keep users out of
trouble.

The changes to ThreadSafeApiCache are required to keep it API-compatible
with the original and keep tests passing. Assignments to the request_id
attribute need to be used for all future requests. It's a little unclear
if this is an intended API or just test scaffolding, but it's not too
difficult to keep working so I just did that.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19686: Introduce low-level api_client constructor
Brett Smith [Mon, 28 Nov 2022 21:30:28 +0000 (16:30 -0500)]
19686: Introduce low-level api_client constructor

When api() returns a ThreadSafeApiCache, we still want to provide a
mechanism to get a plain Resource object. The api_client() function is
that mechanism. It *just* builds the Resource object as api() did
before.

normalize_api_kwargs() takes the arguments passed to api() and turns
them into keyword arguments for api_client(). api_config_to_kwargs()
takes a configuration mapping nand turns them into keyword arguments for
api_client(). Both of these are small APIs, just returning one
dictionary from another.

With this reorganization, api(), api_from_config(), and
ThreadSafeApiClient() can all have simpler implementations.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19791: Remove duplicate word
Brett Smith [Wed, 7 Dec 2022 15:57:41 +0000 (10:57 -0500)]
19791: Remove duplicate word

No issue #

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19847: Fix tests
Peter Amstutz [Tue, 6 Dec 2022 22:13:19 +0000 (17:13 -0500)]
19847: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19847: Change disk cache size heuristic
Peter Amstutz [Tue, 6 Dec 2022 21:41:15 +0000 (16:41 -0500)]
19847: Change disk cache size heuristic

Now sets disk cache to (2 GB <= the size of the RAM request <= 32 GiB)

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19847: crunch-run should provide --ram-cache explicitly
Peter Amstutz [Tue, 6 Dec 2022 20:26:47 +0000 (15:26 -0500)]
19847: crunch-run should provide --ram-cache explicitly

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19791-python-api-overview'
Brett Smith [Tue, 6 Dec 2022 20:18:11 +0000 (15:18 -0500)]
Merge branch '19791-python-api-overview'

Closes #19791.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19847: Fix KeepCacheTypeRequirement
Peter Amstutz [Tue, 6 Dec 2022 18:41:42 +0000 (13:41 -0500)]
19847: Fix KeepCacheTypeRequirement

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19791: Clarify the background discussion for using keyset_list_all
Brett Smith [Tue, 6 Dec 2022 18:23:07 +0000 (13:23 -0500)]
19791: Clarify the background discussion for using keyset_list_all

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19791: Add a note explaining the page's intended scope
Brett Smith [Tue, 6 Dec 2022 18:16:39 +0000 (13:16 -0500)]
19791: Add a note explaining the page's intended scope

I wrote a note to this effect during the branch review, and I hope
putting it here can help guide future authors (possibly including future
me).

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19791: Expand Python SDK examples into API client overview
Brett Smith [Fri, 25 Nov 2022 16:56:44 +0000 (11:56 -0500)]
19791: Expand Python SDK examples into API client overview

This change takes the skeleton of the existing examples page and expands
it into a fuller overview of the API client provided by the Python
SDK. It provides more explanation of how the client works; how it
corresponds to the Arvados API; more concrete examples of real tasks you
might actually want to do with the client; and a few different examples
to demonstrate how you might use different client code patterns for
different situations.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

2 years ago19847: Add calculation for choosing keep disk cache size.
Peter Amstutz [Tue, 6 Dec 2022 17:15:11 +0000 (12:15 -0500)]
19847: Add calculation for choosing keep disk cache size.

Add KeepCacheTypeRequirement to control which cache to use, to
facilitate performance comparison.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19513-create-role-admin-only'
Tom Clegg [Tue, 6 Dec 2022 16:22:46 +0000 (11:22 -0500)]
Merge branch '19513-create-role-admin-only'

closes #19513

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoFix broken documentation link refs #19215
Peter Amstutz [Mon, 5 Dec 2022 22:02:59 +0000 (17:02 -0500)]
Fix broken documentation link refs #19215

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19215-installer' refs #19215
Peter Amstutz [Mon, 5 Dec 2022 18:43:01 +0000 (13:43 -0500)]
Merge branch '19215-installer' refs #19215

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Adjusts the documentation to reflect recent changes. 19215-installer
Lucas Di Pentima [Mon, 5 Dec 2022 18:24:32 +0000 (19:24 +0100)]
19215: Adjusts the documentation to reflect recent changes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19215: Properly outputs LE's secret access key.
Lucas Di Pentima [Mon, 5 Dec 2022 18:05:09 +0000 (19:05 +0100)]
19215: Properly outputs LE's secret access key.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19215: Fixes one remaining IP related variable.
Lucas Di Pentima [Fri, 2 Dec 2022 22:58:07 +0000 (19:58 -0300)]
19215: Fixes one remaining IP related variable.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19215: Adds private subnet & NAT gateway for compute nodes.
Lucas Di Pentima [Fri, 2 Dec 2022 22:54:22 +0000 (19:54 -0300)]
19215: Adds private subnet & NAT gateway for compute nodes.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years ago19513: Rename flag to CanCreateRoleGroups, add test case. 19513-create-role-admin-only
Tom Clegg [Fri, 2 Dec 2022 20:09:55 +0000 (15:09 -0500)]
19513: Rename flag to CanCreateRoleGroups, add test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 years agoMerge branch '18842-arv-mount-disk-config' refs #18842
Peter Amstutz [Fri, 2 Dec 2022 18:28:28 +0000 (13:28 -0500)]
Merge branch '18842-arv-mount-disk-config' refs #18842

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Add -o pipefail to installer.sh
Peter Amstutz [Fri, 2 Dec 2022 17:11:09 +0000 (12:11 -0500)]
19215: Add -o pipefail to installer.sh

This avoids having "| tee" suppress errors that should have caused the
script to stop.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18842: One more refactor 18842-arv-mount-disk-config
Peter Amstutz [Fri, 2 Dec 2022 16:30:43 +0000 (11:30 -0500)]
18842: One more refactor

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18842: Need to call ready.set() on base cases
Peter Amstutz [Fri, 2 Dec 2022 16:24:46 +0000 (11:24 -0500)]
18842: Need to call ready.set() on base cases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18842: Clean up keep cache set logic a little more
Peter Amstutz [Fri, 2 Dec 2022 16:14:51 +0000 (11:14 -0500)]
18842: Clean up keep cache set logic a little more

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18842: Clean up keep cache set() a little bit
Peter Amstutz [Fri, 2 Dec 2022 16:04:27 +0000 (11:04 -0500)]
18842: Clean up keep cache set() a little bit

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Fix *_INT_IP defaults and Terraform CLI install text.
Peter Amstutz [Thu, 1 Dec 2022 22:37:28 +0000 (17:37 -0500)]
19215: Fix *_INT_IP defaults and Terraform CLI install text.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19704: Add readArvFile to README
Peter Amstutz [Thu, 1 Dec 2022 21:23:15 +0000 (16:23 -0500)]
19704: Add readArvFile to README

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoGet the README to render better
Peter Amstutz [Thu, 1 Dec 2022 19:58:11 +0000 (14:58 -0500)]
Get the README to render better

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoarvbox changes: yank Composer. Add link to documentation.
Peter Amstutz [Thu, 1 Dec 2022 18:41:38 +0000 (13:41 -0500)]
arvbox changes: yank Composer.  Add link to documentation.

Also set locale.

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoSet up ruby sdk, no issue #
Peter Amstutz [Thu, 3 Nov 2022 14:48:29 +0000 (10:48 -0400)]
Set up ruby sdk, no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19704: Merge contributed updates to R SDK
AnetaSta22 [Tue, 16 Aug 2022 10:34:07 +0000 (12:34 +0200)]
19704: Merge contributed updates to R SDK

* Add roxygen documentation
* Add readArvFile() function
* Add files via upload
* New README
* Add parameter relativePath in getCollectionContent()

Contributed by Aneta Stanczyk <aneta.stanczyk@contractors.roche.com>

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Move the note about installing Terraform CLI
Peter Amstutz [Thu, 1 Dec 2022 16:40:30 +0000 (11:40 -0500)]
19215: Move the note about installing Terraform CLI

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: A few tweaks about setting up compute image.
Peter Amstutz [Thu, 1 Dec 2022 16:15:52 +0000 (11:15 -0500)]
19215: A few tweaks about setting up compute image.

Make packer script create a log file.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: More documentation details
Peter Amstutz [Wed, 30 Nov 2022 22:13:04 +0000 (17:13 -0500)]
19215: More documentation details

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Polishing install process
Peter Amstutz [Wed, 30 Nov 2022 21:25:00 +0000 (16:25 -0500)]
19215: Polishing install process

* Coordinate terraform using installer.sh
* Doc updates (WIP)

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Adds region_name as output.
Lucas Di Pentima [Wed, 30 Nov 2022 15:00:18 +0000 (12:00 -0300)]
19215: Adds region_name as output.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

2 years agoWhen comparing manifest_text, strip signatures
Peter Amstutz [Tue, 29 Nov 2022 15:47:57 +0000 (10:47 -0500)]
When comparing manifest_text, strip signatures

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years agoMerge branch '19699-cwl-dl-docs' refs #19699
Peter Amstutz [Tue, 29 Nov 2022 15:26:36 +0000 (10:26 -0500)]
Merge branch '19699-cwl-dl-docs' refs #19699

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19699: Update from review comments 19699-cwl-dl-docs
Peter Amstutz [Tue, 29 Nov 2022 14:58:50 +0000 (09:58 -0500)]
19699: Update from review comments

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18842: Add a couple of error handling tests
Peter Amstutz [Mon, 28 Nov 2022 22:29:34 +0000 (17:29 -0500)]
18842: Add a couple of error handling tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago18842: Turn disk write errors into KeepCacheError
Peter Amstutz [Mon, 28 Nov 2022 21:39:03 +0000 (16:39 -0500)]
18842: Turn disk write errors into KeepCacheError

* DiskCacheSlot properly keeps track of the file handle now
* Failure to write disk block to cache will attempt to shrink cache
and try again, if it still fails, turns into KeepCacheError
* init_cache writes a test block to check for working cache

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 years ago19215: Adds hardcoded deploy_user output set to "admin".
Lucas Di Pentima [Mon, 28 Nov 2022 21:33:21 +0000 (18:33 -0300)]
19215: Adds hardcoded deploy_user output set to "admin".

Debian AMIs have their default non-root user set to "admin", so admins will
need to set up the local.params DEPLOY_USER var accordingly.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>