Stephen Smith [Tue, 23 Jan 2024 20:00:51 +0000 (15:00 -0500)]
21336: Add type checking to output data reducer instead of accepting any, fix
misalignments in payload type. Rename some properties for clarity to align with
passing in uuid in addition to output details to reducer.
Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>
Tom Clegg [Wed, 17 Jan 2024 21:41:24 +0000 (16:41 -0500)]
20318: Fix arvados-login-sync and arvados-cli dependencies.
Previous requirement '~> 2.7.0.rc1' excluded the current prerelease
version '2.8.0.dev20231214204948' that the test suite deliberately
installs ahead of time, so bundler overrode that and selected the last
released published version 2.7.1 from rubygems, which is not
compatible with Ruby 3.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Tue, 16 Jan 2024 16:14:02 +0000 (11:14 -0500)]
21230: Update virtualenv in fed-migrate test
I don't believe this is strictly necessary, but `python3 -m venv` is
more likely to work than `virtualenv` and better matches all our other
code. Everything else is just style updates.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Stephen Smith [Mon, 15 Jan 2024 21:44:12 +0000 (16:44 -0500)]
21353: Move url location check into waitUntil
It seems possible for window.location to be defined, then later undefined.
Moving the url location check should solve the problem by waiting for redirects
to be done.
Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>
Brett Smith [Mon, 15 Jan 2024 14:52:29 +0000 (09:52 -0500)]
21230: Activate virtualenv in arvbox doc service
Generating the PySDK docs needs to actually import pdoc, not just run
it. Given this, linking pdoc in /usr/local/bin/ is probably more
confusing than helpful, so don't.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 12 Jan 2024 19:30:45 +0000 (14:30 -0500)]
21230: Fix binstubs generation in arvbox
At some point, bundle started installing binstubs in `bin/` by
default. Various pieces of arvbox assume they go in `binstubs/` and
break when they don't. Explicitly pass options to get the old behavior
where needed.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 12 Jan 2024 15:52:00 +0000 (10:52 -0500)]
21230: Use virtualenv in arvbox
This is preparation to let us build arvbox on Debian 12, which has
adopted PEP 668 and doesn't permit installing packages outside a
virtualenv anymore.
This does move pdoc installation out of the doc service and into the
Dockerfile. The doc code currently doesn't have any way to version this
dependency, and it rarely changes in development, so this seems fine as
a code simplification.
All the code that tries to "pip install from cache only, then use the
network" is gone because as best I can tell it's already non-functional:
the pip cache isn't structured in the way that it assumes
anymore. Generally pip looks at what's installed in the virtualenv and
only installs what it *must* to satisfy dependencies, so this seems
fine.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Fri, 12 Jan 2024 16:06:19 +0000 (11:06 -0500)]
20318: Avoid error when cache file is deleted while being written.
The previous code had one goroutine writing the cache file, and
another goroutine(s) waiting for it to finish, then opening the file
for reading. If the cache file was deleted before being opened for
reading, ReadAt would return an error.
The new code shares the writer's filehandle with all of the goroutines
that are waiting for it to finish writing. This ensures ReadAt can
always read from the newly written file even if it is deleted
mid-write.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Fri, 12 Jan 2024 14:39:41 +0000 (09:39 -0500)]
21230: Remove pip install from get-package-version
Reasons to remove this: it can't work on Debian 12+, and the code has
obviously gotten stale with its original rationale ('setuptools<45' is
definitely not "latest setuptools").
Nobody can tell me what context this runs in. I'm going to hope that it
runs in some context where setuptools is already available. If it turns
out I'm wrong, then hopefully that will break the build in a way that
lets me assess our options.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 12 Jan 2024 14:31:27 +0000 (09:31 -0500)]
21230: Clean up unnecessary Python packages from package build Dockerfiles
* python3-pip: `ensurepip` is standard as of Python 3.4+.
* python3-setuptools: We want to install this in our build
virtualenv anyway.
* python3-virtualenv: python3-venv is sufficient.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Thu, 11 Jan 2024 18:10:47 +0000 (13:10 -0500)]
20318: Fix settled condition.
Intention was to stop iterating when *all* of the keepstore servers
need 0 pulls, but the code as written stops on the N>1st iteration
when *any* of the keepstore servers need 0 pulls. This caused a false
negative when it took more than one iteration for keepstore to
complete the requested pulls.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 11 Jan 2024 17:01:11 +0000 (12:01 -0500)]
20318: Route (*KeepClient)Get() through disk cache layer.
The updated implementation maintains the existing calling signature
but has some semantic differences:
The block size returned by Get is now -1 if the block size is not
indicated by the supplied locator. All locators generated by Arvados
include sizes, and Get() doesn't work on bare hashes because they have
no permission signature, so in practice this only comes up for test
cases and keepstore's pull worker.
The url returned by Get is now "" because
* in general it is not necessarily available in principle (data often
comes from the cache instead of a backend server)
* in the cases where it is available in principle, it would be a lot
of trouble to propagate it
* the only known reason to reveal the url is to provide detail about
an error, in which case the error message should already include the
relevant url.
In some cases where the upstream server responds 200 but an error is
detected early in the response content, Get() now returns the error
itself, where the old Get() implementation would have returned a
reader whose Read method returns an error.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 11 Jan 2024 16:22:27 +0000 (11:22 -0500)]
20318: Fix BadChecksum error not propagated through cache layer.
Previously, ReadAt would not return an error as long as enough bytes
were read, because the error might mean the read was truncated after
the point needed for the current read. This had the unintended
consequence that even a BadChecksum error returned on the final read
would not be propagated.
The updated version propagates errors even if the requested bytes were
retrieved, so the only case where a bad checksum error can be
undetected is when the read does not reach the end of the block *and*
the block has not been fully retrieved yet.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Wed, 10 Jan 2024 18:43:20 +0000 (13:43 -0500)]
21367/21230: Set up dev-jobs with virtualenv
The functional goal here is to do all pip installs inside the Dockerfile
inside a virtualenv to avoid the global pip install issues.
Changes that fall out of that:
* Installing python3-venv is all we need to go back to the -slim Debian
image.
* Stop installing other Python packages we'll just install inside the
virtualenv anyway.
* Rather than having separate arguments and code blocks for every
package we might want to install, just set up a dedicated build
context for the Docker image with all the packages we want to install,
and install them all unconditionally during the image build. This is
much less code and hopefully easier to follow.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 10 Jan 2024 18:33:36 +0000 (13:33 -0500)]
21367: Remove environment check
You only need for this for `arv-keepdocker` at the end of the process,
which can read from settings.conf just fine too. Even if it fails, the
developer will have the command handy to fix their environment and try
that again easily.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Lucas Di Pentima [Tue, 9 Jan 2024 22:03:05 +0000 (19:03 -0300)]
21285: Updates overview dashboard to include concurrent reqs queue names.
At the moment we have 2 queues: "api" & "tunnel", I think this change will
allow the dashboard to adapt to potential new queues (instead of hardcoding
queue names in the query)
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>
projects / arvados.git / log