Tom Clegg [Mon, 4 Aug 2025 17:59:57 +0000 (13:59 -0400)]
23025: Fix double redirect.
Previous code did not use the query token to look up the container, so
the first request would fail the container lookup and send a 303
moving the token to a cookie, and the resulting second request would
succeed at the container lookup only to find the container UUID does
not match the client's cookie, and send another 303 to clear site
data.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 31 Jul 2025 14:09:48 +0000 (10:09 -0400)]
23044: Pass ARVADOS_TEST_PRIVESC through in test cluster.
My intent was to enable the use of sudo in integration tests with
{RuntimeEngine: singularity}. This still doesn't work with
non-passwordless sudo, but at least it tries to use sudo when so
configured, and logs something in stderr.txt if it fails.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 24 Jul 2025 17:51:51 +0000 (13:51 -0400)]
23044: Enable use of crunch-run gateway with loopback driver.
Normally a gateway address of 127.0.0.1:* means crunch-run couldn't
figure out its external IP address, but in the case of the loopback
driver, it's correct.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Thu, 24 Jul 2025 00:01:17 +0000 (20:01 -0400)]
Run `bundle install` for multi_json
Should've been included in the previous commit. I thought this would be
a noop because we already had an indirect dependency on
`multi_json (1.15.0)`, but I guess (to its credit) bundler does check
everything. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 19:58:28 +0000 (15:58 -0400)]
Only set Docker data-root for cloud compute nodes
b6d8d5665243578986cefc5c3fec850d6cf25b05 could cause single-node
installs to configure Docker with a data root at /tmp/docker-data, which
is probably not what people want. Instead set this in
build-compute-image.yml, and introduce a system to extend the Docker
daemon.json selectively. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 19:14:17 +0000 (15:14 -0400)]
Do not chown/chmod /etc/arvados
On dedicated compute nodes this is unlikely to exist and no big deal,
but on single-node installs the admin may choose to change the
ownership (especially group) of this directory. Remove those specifiers
to avoid interfering with that. Ansible's defaults should be fine for
a newly-created directory. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 19:12:35 +0000 (15:12 -0400)]
Configure Docker when installing it with crunch-dispatch-local
In order for crunch-dispatch-local to work, the local node needs to be
configured like any other compute node. Most of this is already done via
the dependency on arvados_compute, but that does not configure Docker
services, that's the job of compute_docker. Pull that in when necessary.
No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 13:32:13 +0000 (09:32 -0400)]
Merge Python .gitignore files into top level
At least in my development environment, the symlinks generally don't
work and confuse tools. Most of the temporary-type files are already
covered by patterns in .gitignore, so the easiest fix from here is just
to ignore the full build and dist paths. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 01:56:09 +0000 (21:56 -0400)]
23064: Start contrib directory
The goal of this directory is to help users better understand what
components are "core" Arvados vs. additional tools and to give us a
place to start building more of the latter.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 22 Jul 2025 15:16:41 +0000 (11:16 -0400)]
22817: Remove Java SDK Docker test scripts
These were used by previous Jenkins configuration. Our current nodes are
sure to have Gradle, so we have eliminated the need for this indirection.
Closes #22817.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Sun, 20 Jul 2025 17:47:24 +0000 (13:47 -0400)]
23006: Hardcode runner image inside arvbox
It is already a known limitation that arvbox can't build its own
arvados/jobs image because it runs Singularity. This pins it to a
known-good version while we work on transitioning away from arvbox.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 21 Jul 2025 20:37:38 +0000 (16:37 -0400)]
21389: Unify Python Docker images
Fundamentally these images all build a Docker image with a Python
virtualenv. There are minor details in how they go about it, but none of
them have any real consequence for the final built image. We can use the
same code to build all these images and just parametrize it in the build
context.
* Port build-dev-docker-jobs-image.sh to Python (build_docker_image.py).
This makes it easier to share code between them; provides better
argument parsing; gives us a better base to build from; and makes
sense for build tooling that works so much with Python.
* Update the Dockerfile to use a two-stage build. It still builds a
virtualenv with wheels provided by the build script, but a two-stage
build keeps image sizes in line with what we had with the package-based
approach, which is appropriate for an image that gets distributed as
widely as arvados/jobs.
* Synchronize everything on bookworm-slim.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 21 Jul 2025 20:34:18 +0000 (16:34 -0400)]
21389: Remove Docker image build from test_with_arvbox
This code has been noop for a while: arvbox doesn't include Docker, it
uses Singularity, to avoid the usual Docker-in-Docker problems. Since
there's no way it can work, we should stop maintaining it.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 17 Jul 2025 18:29:05 +0000 (14:29 -0400)]
21389: Add requirements.txt files to build
This makes it easier for different build tools to install all their
dependencies and keep them all consistent.
With this change, I have chosen to flatten the distinction between "env"
and "deps" in run-tests.sh. It's already a little shaky, and it didn't
seem worth splitting out two files to keep it going.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 15 Jul 2025 18:40:48 +0000 (14:40 -0400)]
23044: Add external service container port routing to handler
This is an initial pass to address the bug in refs #23044.
This might not be the right solution, but it seems to at least get
things functioning, and it's self-contained so it's easy to replace
later as desired.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 15 Jul 2025 13:18:25 +0000 (09:18 -0400)]
23041: Refine Ansible's strategy for installing FUSE
`/etc/fuse.conf` could be provided by either `fuse` or `fuse3`. If it
already exists, don't install a package, to avoid messing with the
administrator's selections. If it doesn't, install `fuse3` to get the
most modern version. All of our supported distributions have this except
maybe Ubuntu 20.04, where we're not going to run the Ansible installer
anyway.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 14 Jul 2025 16:03:04 +0000 (12:03 -0400)]
23033: Remove fuse dependency from arvados-client
The arv-mount dependency notes it is there because arv-mount can
sometimes call FUSE command line tools. As best I can tell, this is not
true for arvados-client. Declaring the library dependency is more
accurate. Refs #23033.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 9 Jul 2025 18:33:23 +0000 (14:33 -0400)]
22866: Document single-node install with Ansible
This is tested and we expect this will be easier for most people to
deploy.
Part of what's changing here is the documentation page does not have a
bunch of forks for different options. Instead it prioritizes getting the
simplest possible install working with minimum fuss. I think this is
what most people installing with this option want. The provided
configuration templates give pointers for ways you might change the
configuration in more advanced ways.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 8 Jul 2025 18:23:01 +0000 (14:23 -0400)]
22866: Touch up example cluster config
* Use `xurid` ("your id") instead of the test cluster id `zzzzz`.
* Clarify that the cluster identifier must be lowercase.
* Correct the documentation for nginx internal networks.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
projects / arvados.git / log