20650: Improve test and move boto3 dep to python sdk

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Test WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Add option to control AWS credential capture behavior

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Log the access key id used

This is the non-secret portion of the credentials so it is safe to
include in the log.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Pass AWS credentials if needed

If there are s3:// URLs, and --defer-download is used, then pass the
environment's AWS credentials as a secret mount.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Need to recognize s3: URLs in a few places

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Add boto3 dependency

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Initial try saving downloading from s3 in a-c-r

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: boto session is passed in to s3_to_keep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Add a Downloader abstract base instead of duck typing

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Tinkering with AWS credentials

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Implemented s3_to_keep

Added support importing from s3 to arv-copy

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

20650: Initial refactor spitting up http_to_keep to share code

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

Build Java SDK with jdk11, refs #22084

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

Merge branch '22689-bump-babel-dependencies' into main. Closes #22689

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22689: Bump @babel/runtime-corejs2 to address CVE-2025-27789

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22689: Bump @babel/helpers to address CVE-2025-27789

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22689: Bump @babel/runtime to address CVE-2025-27789

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

21926: Add upgrade note about cuda→gpu API changes

Refs #21926.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

Merge branch '22621-upload-error-message' into main. Closes #22621

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22621: Change collection file upload error to be generic

Upload errors do not generate any reason in the exceptions so we are unable to
display specific error reasons.

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

Merge branch '22320-cached-only'

Refs #22320. Closes #22344.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

Merge branch '22228-separate-expire-and-refresh'

fixes #22228

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '22655-keepclient-test'

closes #22655

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Add tests for CachedOnly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Add CheckCacheOnly option to BlockReadOptions.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '22320-repack-small-blocks'

refs #22320

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22655: Fix Python 3 usage -- raise exceptions, not strings.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22655: Fix func signature to force kwargs, remove unused arg.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '22602-arv-copy' refs #22602

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22320: Check final blocksInManifest.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Run slow tests and write chart data only if env var is set.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22602: DRY search() to use search_paths() and add test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22320: Fix comment typo.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22228: Update comments.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22228: Test that remote expires_at is preserved in local cache.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '22660-acr-cred-leak' refs #22660

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22228: Update tests to use refreshes_at to induce cache refresh.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22228: Add refreshes_at field, preserve upstream expires_at.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22660: Simpler sanitize implementation from brett

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22602: Further improve error reporting for getting cluster creds

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22602: Fix test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22602: Improve error handling when instance info is wrong

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22660: Sanitize the git remote URL

Remove the username/password section of the netloc part of the URL, if
present.  This was reported by a user that it was leaking github
tokens, as one way of providing a github token is write it as

https://x-access-token:blahblahblah@github.com/foo/bar.git

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

22655: Propagate errors running run_test_servers.py from Go suites.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22655: Don't keep waiting for a subprocess to listen after it exits.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22655: In test environment, retry in-use addr for 5 minutes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Bump rack from 3.1.10 to 3.1.12 in /services/api

refs #22657

Bumps [rack](https://github.com/rack/rack) from 3.1.10 to 3.1.12.
- [Release notes](https://github.com/rack/rack/releases)
- [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rack/rack/compare/v3.1.10...v3.1.12)

---
updated-dependencies:
- dependency-name: rack
  dependency-type: indirect
...

Arvados-DCO-1.1-Signed-off-by: dependabot[bot] <support@github.com>

22655: Try a bit harder to detect port conflicts.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22655: Avoid needlessly changing global state in test suites.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '17209-http-forward'

closes #17209

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17209: Remove unused commented-out code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '22644-arvados-server-install-singularity'

Refs #22644.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22602: arv-copy checks config file first then falls back on default creds

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

Merge branch '22647-postinst-ignores-gem-install-errors'

Closes #22647, #22648.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22647: Wordsmith comment and status message

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22320: Export Repack() method.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Remove unimplemented Underutilized option.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Add test cases for 32MiB and 32MiB-1 sized blocks.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Fix maxBlockSize restored to wrong value after tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22647: Ignore `gem install` errors in RailsAPI postinst

See comments for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22320: Fix test case cheating by repeating content.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Implement full-repack, like old tests expected.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Parameterize RepackCost test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Improve stats output.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Fix mkdir in test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Fix test panic.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Write more stats for charting.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Refine repacking algorithm to handle a range of block sizes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Fix Repack bug causing segment replacements to be skipped.

repackTree must call replaceSegments with the newly computed
replacements -- not the merged set of new replacements plus
previously-applied replacements.  If it passes the merged set to
replaceSegments, some of the new replacements will be skipped.

For example, if a previous repack applied A->B, and the current repack
applies B->D and C->D, then the merged set includes A->D, and A is no
longer referenced, so replaceSegments will skip any->D.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Refactor test case.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Add repack-after-write test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Track repack results.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22320: Naive Repack implementation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

22644: Install Singularity from archive instead of Git

Ansible is already doing this, and it looks like something has maybe
happened with one of the submodules so it no longer gets cloned
correctly. Using the source archive is less error-prone.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Add PowerTools setup to the API install docs

Mirroring the upgrade note. Refs #22608.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

Merge branch '22608-enable-powertools'

Refs #22608.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Add upgrade note about libyaml/PowerTools dependency

This is a consequence of the upgrade to Rails 7.1.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Enable powertools repo in rocky8 test image

This is necessary to get libyaml-devel to install arvados-api-server.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

17209: Allow admin access to all container ports.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17209: Merge branch 'main' into 17209-http-forward

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

17209: Restrict HTTP proxy access to the CR submitter.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

Merge branch '22608-upgrade-rails'

Refs #22608, #22629.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

Merge branch '22624-left-panel-uncollapse-bug' into main. Closes #22624

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22608: Upgrade Passenger to 6.0.26

To address CVE-2025-26803.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Guard against no uuid param in find_object_by_uuid

See comments for rationale. The old code causes bad query construction
and crashes in Rails 7.2 because of
  <https://github.com/rails/rails/issues/52643>

While we can't upgrade to Rails 7.2 right now, the code is safe for
older versions. Since I spent the time to track this down, we might as
well make the change now and make a future upgrade easier.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Upgrade Rails to 7.1

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Unroll container requests token test loop

Make the separate test cases individual tests for nicer diagnostics.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22608: Codify Ruby 2.7 requirements in RailsAPI Gemfile

We've had a few incidents of these gems getting upgraded and then
needing to be reverted, e.g.:

* f2d7a04756206c6812f71225a6d782c23db1c7d9
* 4a42e4fd1f7cc2cd5cc1938d4bdc39e3a4800306
* 2f4fb1522c89c29a94854bf9f26fb6d13959f2d4

Write these in the Gemfile to prevent future mistakes like these.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

22624: Switch to using max-width constraint when left panel collapsed

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

Merge branch '22566-left-sidebar-scroll-bug' into main. Closes #22566

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22566: Replace custom class with disabled property

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22566: Fix horizontal scrolling on collapsed sidepanel

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22566: Refactor to combine get/set panel sizes, save localstorage keys and
panel settings in a type safe const

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22566: Lower min left panel size slightly, trigger details size recalculate
on left panel collapse

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

22566: Remove unused imports

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

Merge branch '22614-railsapi-service-packaging'

Refs #22614, #22349.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

Merge branch '22359-toolbar-overflow'

closes #22359

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

Merge branch '22408-wrong-project-toolbar'

closes #22408

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

22408: changed currentPathIsTrash calculation

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>