Tom Clegg [Mon, 27 May 2024 14:44:54 +0000 (10:44 -0400)]
22567: Use --mount instead of -v/--volume for docker bind mounts.
If the source path does not exist, -v/--volume assumes the source path
is the name of a volume and creates an empty one. We would rather get
a "source path does not exist" error.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Brett Smith [Mon, 10 Feb 2025 16:33:12 +0000 (11:33 -0500)]
22489/22556: Do not reinstall dependencies for login-sync
This is a weird exception to our usual rule that the `install` recipes
in run-tests.sh assume you've already installed the dependencies. Even
`install_sdk/cli`, which has the exact same gem dependencies, does not
do this. So do not do it here either, for consistency and less work.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Mon, 10 Feb 2025 16:30:44 +0000 (11:30 -0500)]
22489/22556: Do not uninstall gems in run-tests.sh
Just overwrite them instead. I have manually confirmed that, at least as
of Ruby 2.7.4/Gems 3.2.5, running `gem install PATH.gem` when the gem is
already installed will reinstall it, incorporating any changes. This
makes less work and avoids situations where we have broken gems because
we uninstalled dependencies out from underneath them.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Sun, 9 Feb 2025 15:05:58 +0000 (10:05 -0500)]
22489: Conditionally set fs.inotify.max_user_watches in install-test-env
This matches the current behavior of `arvados-server install` to avoid
setting sysctls in unprivileged containers. I looked at using Ansible's
built-in virtualization facts to see if that could let us know whether
we could/should set these, but it's not as simple as "is this
virtualized" since that covers everything from KVM/Xen to tightly
restricted cgroups.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Tue, 4 Feb 2025 20:56:24 +0000 (15:56 -0500)]
22008: Improve csv_to_list API
Instead of having a single hardcoded function in util, have two
functions parse_seq and uniq in _internal. Add a UniqueSplit class to
commands._util for use as an argparse argument type to improve error
reporting.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Zoë Ma [Mon, 1 Jul 2024 07:34:33 +0000 (15:34 +0800)]
CWL SDK: Do not hard-code default storage class in command-line tool
For arvados-cwl-runner, the options --storage-classes and
--intermediate-storage-classes used the hard-coded string `"default"`
internally as the stand-in for the storage class when none was given
when the tool was invoked.
This works when the storage class named "default" is the only default
storage class configured (i.e. with `Default: True` property). However,
this could be confusing, because, for example, there can be more than
one default storage classes in addition to the one _named_ "default".
Furthermore, the it's possible that the class named "default" may not
have the `Default: True` property (although the latter is certainly not
recommended).
By using the empty string, instead of the hard-coded "default", to
indicate that "no valid storage class is explicitly specified", testing
for the condition "we should use default storage classes" is actually
simplified, because the empty string is not ambiguous.
With this change, the decision about default intermediate storage
classes is made when container is requested. This can be beneficial, for
example, when the stages are run on different clusters with different
storage class configurations.
In addition, like in 8cb5ef17 (for Python SDK), the input option value
(comma-separated list of storage classes) is subjected to more robust
validation, to filter out any duplicate or empty values.
Arvados-DCO-1.1-Signed-off-by: Zoë Ma <zoe.ma@curii.com>
Zoë Ma [Mon, 1 Jul 2024 07:17:49 +0000 (15:17 +0800)]
Python SDK: more validation for options taking comma-separated values
For command-line tools such as arv-put and arv-copy, options like
--storage-classes and --varying-url-params take a string of
comma-separated values. The current implementation may provide
insufficient validation for such input when, for example,
- input contains duplicates values
- input is empty or does not contain any valid value after removing
leading and trailing space characters, e.g. --storage-class=", ,,,"
This is fixed by filtering the input string through a clean-up and
deduplication function. A test is added for the arv-put command to
illustrate the updated behavior.
For convenience, the filter function is added as
arvados.util.csv_to_list().
The minor changes to arvados/keep.py and arvados/http_to_keep.py are
for reusing the same code in certain internal string processing.
Arvados-DCO-1.1-Signed-off-by: Zoë Ma <zoe.ma@curii.com>
Zoë Ma [Wed, 19 Jun 2024 03:01:11 +0000 (11:01 +0800)]
Support setting replication level at destination in arv-copy options
arv-copy effectively used a hard-coded replication level 2 for the
copied collections at the destination, bypassing the default replication
level set in the destination cluster's configuration file. There was no
command-line option to override this behavior.
A new command-line option, --replication, is added to the arv-copy
command, following the arv-put command's semantics. If left unspecified,
the destination's default replication setting is used. If that setting
cannot be found, use the fallback value of 2.
Arvados-DCO-1.1-Signed-off-by: Zoë Ma <zoe.ma@curii.com>
Brett Smith [Tue, 4 Feb 2025 19:19:55 +0000 (14:19 -0500)]
22406: Remove hardcoding of arvados_test database name
Users should be allowed to use a different database if they
want. Testing that this actually works is outside the scope of this
branch, but we would like to remove obstacles to it happening, rather
than adding more.
zzzzz is assumed to always be a cluster running automated tests. If you
want a development cluster in your config.yml using a development
database, you need to give it a different cluster ID.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Fri, 31 Jan 2025 21:45:44 +0000 (16:45 -0500)]
22406: Reorganize localdb login tests
* Rename login_ldap_docker_test→login_docker_test, following the suite
rename for the same reason (it tests both LDAP and PAM).
* Remove individual test suites for these login methods. They're not
testing anything the Docker suite can't test better thanks to
controlling more of the authentication stack.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Peter Amstutz [Wed, 29 Jan 2025 20:48:28 +0000 (15:48 -0500)]
22466: Reorganize block processing patterns to use exceptions
Now uses exceptions to indicate any condition where it cannot
determine a correct value for output_glob, which should make the logic
a bit easier to follow.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Wed, 29 Jan 2025 16:13:33 +0000 (11:13 -0500)]
22433: Update package build to Node 14.21.3
This matches lib/install/deps.go and tools/ansible/install-test-env.yml.
I don't want to parse it out of deps.go a la goversion because we're
looking to get rid of that soon-ish.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
It seems like retrying a deadlock is counter productive -- it ties up
a request slot and holds a transaction open. I believe returning a
500 error and having the client retry will be more robust.
This also adjusts the locking query to select and order by the primary
key (id). This is speculative and based on a stack overflow comment
suggesting that the doing lock ordering on a field other than the
primary key causes rows to be first acquired with share locks (out of
order) and then subsequently converted to exclusive locks (in order),
and this can be avoided by using the primary key.
Since I haven't been able to determine any other reason the ordered
locking, which is specifically implemented to prevent deadlocks but
deadlocks anyway, it's worth a shot.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
In circumstances where we can safely identify the primary and
secondary file (the filenames are sufficiently well characterized in
advance) then we can use output_glob. If not, capture everything, and
make sure we don't throw any spurious errors.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
projects / arvados.git / log