Brett Smith [Tue, 4 Feb 2025 20:56:24 +0000 (15:56 -0500)]
22008: Improve csv_to_list API
Instead of having a single hardcoded function in util, have two
functions parse_seq and uniq in _internal. Add a UniqueSplit class to
commands._util for use as an argparse argument type to improve error
reporting.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Zoë Ma [Mon, 1 Jul 2024 07:34:33 +0000 (15:34 +0800)]
CWL SDK: Do not hard-code default storage class in command-line tool
For arvados-cwl-runner, the options --storage-classes and
--intermediate-storage-classes used the hard-coded string `"default"`
internally as the stand-in for the storage class when none was given
when the tool was invoked.
This works when the storage class named "default" is the only default
storage class configured (i.e. with `Default: True` property). However,
this could be confusing, because, for example, there can be more than
one default storage classes in addition to the one _named_ "default".
Furthermore, the it's possible that the class named "default" may not
have the `Default: True` property (although the latter is certainly not
recommended).
By using the empty string, instead of the hard-coded "default", to
indicate that "no valid storage class is explicitly specified", testing
for the condition "we should use default storage classes" is actually
simplified, because the empty string is not ambiguous.
With this change, the decision about default intermediate storage
classes is made when container is requested. This can be beneficial, for
example, when the stages are run on different clusters with different
storage class configurations.
In addition, like in 8cb5ef17 (for Python SDK), the input option value
(comma-separated list of storage classes) is subjected to more robust
validation, to filter out any duplicate or empty values.
Arvados-DCO-1.1-Signed-off-by: Zoë Ma <zoe.ma@curii.com>
Zoë Ma [Mon, 1 Jul 2024 07:17:49 +0000 (15:17 +0800)]
Python SDK: more validation for options taking comma-separated values
For command-line tools such as arv-put and arv-copy, options like
--storage-classes and --varying-url-params take a string of
comma-separated values. The current implementation may provide
insufficient validation for such input when, for example,
- input contains duplicates values
- input is empty or does not contain any valid value after removing
leading and trailing space characters, e.g. --storage-class=", ,,,"
This is fixed by filtering the input string through a clean-up and
deduplication function. A test is added for the arv-put command to
illustrate the updated behavior.
For convenience, the filter function is added as
arvados.util.csv_to_list().
The minor changes to arvados/keep.py and arvados/http_to_keep.py are
for reusing the same code in certain internal string processing.
Arvados-DCO-1.1-Signed-off-by: Zoë Ma <zoe.ma@curii.com>
Zoë Ma [Wed, 19 Jun 2024 03:01:11 +0000 (11:01 +0800)]
Support setting replication level at destination in arv-copy options
arv-copy effectively used a hard-coded replication level 2 for the
copied collections at the destination, bypassing the default replication
level set in the destination cluster's configuration file. There was no
command-line option to override this behavior.
A new command-line option, --replication, is added to the arv-copy
command, following the arv-put command's semantics. If left unspecified,
the destination's default replication setting is used. If that setting
cannot be found, use the fallback value of 2.
Arvados-DCO-1.1-Signed-off-by: Zoë Ma <zoe.ma@curii.com>
Peter Amstutz [Wed, 29 Jan 2025 20:48:28 +0000 (15:48 -0500)]
22466: Reorganize block processing patterns to use exceptions
Now uses exceptions to indicate any condition where it cannot
determine a correct value for output_glob, which should make the logic
a bit easier to follow.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Wed, 29 Jan 2025 16:13:33 +0000 (11:13 -0500)]
22433: Update package build to Node 14.21.3
This matches lib/install/deps.go and tools/ansible/install-test-env.yml.
I don't want to parse it out of deps.go a la goversion because we're
looking to get rid of that soon-ish.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
It seems like retrying a deadlock is counter productive -- it ties up
a request slot and holds a transaction open. I believe returning a
500 error and having the client retry will be more robust.
This also adjusts the locking query to select and order by the primary
key (id). This is speculative and based on a stack overflow comment
suggesting that the doing lock ordering on a field other than the
primary key causes rows to be first acquired with share locks (out of
order) and then subsequently converted to exclusive locks (in order),
and this can be avoided by using the primary key.
Since I haven't been able to determine any other reason the ordered
locking, which is specifically implemented to prevent deadlocks but
deadlocks anyway, it's worth a shot.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
In circumstances where we can safely identify the primary and
secondary file (the filenames are sufficiently well characterized in
advance) then we can use output_glob. If not, capture everything, and
make sure we don't throw any spurious errors.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
Brett Smith [Wed, 22 Jan 2025 18:00:34 +0000 (13:00 -0500)]
22433: Report a failure status if RailsAPI postinst does not complete
In the past, we specifically wanted to return exit code 0 if RailsAPI
couldn't be installed because of missing configuration or supporting
database. This meant the message we reported about it was the last
thing users saw, and they could get to work fixing it.
However, we are really pushing to get people to use higher-level
installers like Salt and Ansible. In this scenario, the exit code 0 is a
problem: it means the installer will assume RailsAPI is good to go even
if it isn't.
Now we prioritize this case over the by-hand install and exit with an
error code if postinst cannot complete.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Brett Smith [Wed, 22 Jan 2025 17:51:47 +0000 (12:51 -0500)]
22433: Flatten RailsAPI postinst script
The previous version had most of the work in a giant `configure_version`
function and then just did a little status reporting at the end. Reverse
that relationship: have the main functionality be the top level of the
script, and move incomplete install reporting to a dedicated function.
Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>
Tom Clegg [Mon, 6 Jan 2025 16:08:55 +0000 (11:08 -0500)]
22411: Do not apply RequestTimeout to keepstore index handler.
This updates our code to account for the ResponseController pattern
(see https://tip.golang.org/doc/go1.20#http_responsecontroller):
Update prometheus client library to a newer version whose middleware
supports Unwrap.
Use http.NewResponseController(w).Hijack() instead of probing for the
Hijack() interface.
Use a explicit function call to disable the request timeout when
needed, instead of magically detecting whether the connection has been
hijacked.
(Our old way of detecting hijacking doesn't work if it's done via
ResponseController. Since we're adding an explicit "disable timeout"
mechanism anyway, it's much simpler to just write "hijack and cancel
timeout" when needed, rather than implement a new way to detect
hijacking.)
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
projects / arvados.git / log