Lucas Di Pentima [Tue, 5 Aug 2025 21:50:45 +0000 (18:50 -0300)]
23010: Uses new pg-formula supporting Ubuntu 24.04
Also, avoid installing postgresql-contrib as it depends on the currently
supported PG version on each distro, eg: install postgresql-16 on Ubuntu
24.04 so both v15 and v16 get installed (because v15 is the one
requested by the installer) and that creates issues with service ports.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
Brett Smith [Tue, 5 Aug 2025 20:59:10 +0000 (16:59 -0400)]
22436: Remove arvados-package
Because Arvados is deployed with third-party software like nginx, a
configuration management tool like Salt or Ansible is better suited to
orchestrate the install compared to a distribution package. We are
focusing future development on the Ansible installer.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 4 Aug 2025 19:33:32 +0000 (15:33 -0400)]
Restart Docker after updating daemon.json
This was not in the original version of the code because it was written
to build cloud compute images where we just needed to write the right
configuration. Now that it supports single-node installs and maybe more
in the future, we need to make sure services get new state. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Mon, 4 Aug 2025 17:59:57 +0000 (13:59 -0400)]
23025: Fix double redirect.
Previous code did not use the query token to look up the container, so
the first request would fail the container lookup and send a 303
moving the token to a cookie, and the resulting second request would
succeed at the container lookup only to find the container UUID does
not match the client's cookie, and send another 303 to clear site
data.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Mon, 4 Aug 2025 01:47:10 +0000 (21:47 -0400)]
20311: Remove deprecated setup.py invocations
This is possible now that we define pyproject.toml for all our Python
modules. I'm not bothering with `arvados-server install` and `arvbox`
since they're both slated for removal in Arvados 3.2.0.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 4 Aug 2025 00:48:17 +0000 (20:48 -0400)]
20311: Remove unused get-package-version script
In addition to not having references in the repository, it was never
updated to remove the `development-` prefix following b81baa970ca14b1ace4b9f0dc2ea0eae8330e788.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 1 Aug 2025 21:24:14 +0000 (17:24 -0400)]
20311: Generate _version.py from a setuptools subcommand
This restructures arvados_version.py so that metadata generation happens
in a dedicated class (useful for other build tooling like
build_docker_image.py) and then it provides a setuptools subcommand to
write _version.py where needed. This provides cleaner separation of
concerns and gives us a better base to build more tooling from.
As part of this change, we take care to no longer install tests as part
of the wheel we build. We shouldn't have been doing this anyway, and now
it's required for the tests to run as expected.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 1 Aug 2025 21:21:32 +0000 (17:21 -0400)]
20311: Remove sys.path manipulation from run_test_server.py
This has been obsolete since run-tests.sh started installing the PySDK
to its virtualenv as part of `install env`, and that's a better approach
since it also covers dependencies, etc.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 31 Jul 2025 14:22:41 +0000 (10:22 -0400)]
20311: Make discovery pydoc build a subcommand
The removed comment is no longer a concern now that we consistently use
`pip install`, and this makes it easier to further extend the build
process for PEP 517.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Thu, 31 Jul 2025 14:09:48 +0000 (10:09 -0400)]
23044: Pass ARVADOS_TEST_PRIVESC through in test cluster.
My intent was to enable the use of sudo in integration tests with
{RuntimeEngine: singularity}. This still doesn't work with
non-passwordless sudo, but at least it tries to use sudo when so
configured, and logs something in stderr.txt if it fails.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 24 Jul 2025 17:51:51 +0000 (13:51 -0400)]
23044: Enable use of crunch-run gateway with loopback driver.
Normally a gateway address of 127.0.0.1:* means crunch-run couldn't
figure out its external IP address, but in the case of the loopback
driver, it's correct.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Thu, 24 Jul 2025 00:01:17 +0000 (20:01 -0400)]
Run `bundle install` for multi_json
Should've been included in the previous commit. I thought this would be
a noop because we already had an indirect dependency on
`multi_json (1.15.0)`, but I guess (to its credit) bundler does check
everything. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 19:58:28 +0000 (15:58 -0400)]
Only set Docker data-root for cloud compute nodes
b6d8d5665243578986cefc5c3fec850d6cf25b05 could cause single-node
installs to configure Docker with a data root at /tmp/docker-data, which
is probably not what people want. Instead set this in
build-compute-image.yml, and introduce a system to extend the Docker
daemon.json selectively. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 19:14:17 +0000 (15:14 -0400)]
Do not chown/chmod /etc/arvados
On dedicated compute nodes this is unlikely to exist and no big deal,
but on single-node installs the admin may choose to change the
ownership (especially group) of this directory. Remove those specifiers
to avoid interfering with that. Ansible's defaults should be fine for
a newly-created directory. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 19:12:35 +0000 (15:12 -0400)]
Configure Docker when installing it with crunch-dispatch-local
In order for crunch-dispatch-local to work, the local node needs to be
configured like any other compute node. Most of this is already done via
the dependency on arvados_compute, but that does not configure Docker
services, that's the job of compute_docker. Pull that in when necessary.
No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 13:32:13 +0000 (09:32 -0400)]
Merge Python .gitignore files into top level
At least in my development environment, the symlinks generally don't
work and confuse tools. Most of the temporary-type files are already
covered by patterns in .gitignore, so the easiest fix from here is just
to ignore the full build and dist paths. No issue #.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 23 Jul 2025 01:56:09 +0000 (21:56 -0400)]
23064: Start contrib directory
The goal of this directory is to help users better understand what
components are "core" Arvados vs. additional tools and to give us a
place to start building more of the latter.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 22 Jul 2025 15:16:41 +0000 (11:16 -0400)]
22817: Remove Java SDK Docker test scripts
These were used by previous Jenkins configuration. Our current nodes are
sure to have Gradle, so we have eliminated the need for this indirection.
Closes #22817.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Sun, 20 Jul 2025 17:47:24 +0000 (13:47 -0400)]
23006: Hardcode runner image inside arvbox
It is already a known limitation that arvbox can't build its own
arvados/jobs image because it runs Singularity. This pins it to a
known-good version while we work on transitioning away from arvbox.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 21 Jul 2025 20:37:38 +0000 (16:37 -0400)]
21389: Unify Python Docker images
Fundamentally these images all build a Docker image with a Python
virtualenv. There are minor details in how they go about it, but none of
them have any real consequence for the final built image. We can use the
same code to build all these images and just parametrize it in the build
context.
* Port build-dev-docker-jobs-image.sh to Python (build_docker_image.py).
This makes it easier to share code between them; provides better
argument parsing; gives us a better base to build from; and makes
sense for build tooling that works so much with Python.
* Update the Dockerfile to use a two-stage build. It still builds a
virtualenv with wheels provided by the build script, but a two-stage
build keeps image sizes in line with what we had with the package-based
approach, which is appropriate for an image that gets distributed as
widely as arvados/jobs.
* Synchronize everything on bookworm-slim.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 21 Jul 2025 20:34:18 +0000 (16:34 -0400)]
21389: Remove Docker image build from test_with_arvbox
This code has been noop for a while: arvbox doesn't include Docker, it
uses Singularity, to avoid the usual Docker-in-Docker problems. Since
there's no way it can work, we should stop maintaining it.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Thu, 17 Jul 2025 18:29:05 +0000 (14:29 -0400)]
21389: Add requirements.txt files to build
This makes it easier for different build tools to install all their
dependencies and keep them all consistent.
With this change, I have chosen to flatten the distinction between "env"
and "deps" in run-tests.sh. It's already a little shaky, and it didn't
seem worth splitting out two files to keep it going.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 15 Jul 2025 18:40:48 +0000 (14:40 -0400)]
23044: Add external service container port routing to handler
This is an initial pass to address the bug in refs #23044.
This might not be the right solution, but it seems to at least get
things functioning, and it's self-contained so it's easy to replace
later as desired.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 15 Jul 2025 13:18:25 +0000 (09:18 -0400)]
23041: Refine Ansible's strategy for installing FUSE
`/etc/fuse.conf` could be provided by either `fuse` or `fuse3`. If it
already exists, don't install a package, to avoid messing with the
administrator's selections. If it doesn't, install `fuse3` to get the
most modern version. All of our supported distributions have this except
maybe Ubuntu 20.04, where we're not going to run the Ansible installer
anyway.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 14 Jul 2025 16:03:04 +0000 (12:03 -0400)]
23033: Remove fuse dependency from arvados-client
The arv-mount dependency notes it is there because arv-mount can
sometimes call FUSE command line tools. As best I can tell, this is not
true for arvados-client. Declaring the library dependency is more
accurate. Refs #23033.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
projects / arvados.git / log