arvados.git
2 months ago21639: Improve critical path of read() from cache
Peter Amstutz [Mon, 1 Apr 2024 19:58:06 +0000 (15:58 -0400)]
21639: Improve critical path of read() from cache

* Don't use tobytes(), it makes a copy, and it should be be zero-copy.
* Prefetching adds a lot of overhead.  Don't do it.
* Don't use a list comprehension to calculate cache size

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 months ago21508: Split mounts and param table styles
Stephen Smith [Sun, 7 Apr 2024 19:48:00 +0000 (15:48 -0400)]
21508: Split mounts and param table styles

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

2 months ago21508: Remove image preview from io panel
Stephen Smith [Mon, 1 Apr 2024 14:41:35 +0000 (10:41 -0400)]
21508: Remove image preview from io panel

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

2 months ago21508: Expand secondary param values into separate parameters
Stephen Smith [Fri, 29 Mar 2024 23:34:05 +0000 (19:34 -0400)]
21508: Expand secondary param values into separate parameters

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

2 months ago21613: Fix handling of expired token re-validating with new UUID. 21613-fed-token-exp
Tom Clegg [Mon, 1 Apr 2024 21:51:43 +0000 (17:51 -0400)]
21613: Fix handling of expired token re-validating with new UUID.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

2 months ago21448: fixed user-profile test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox...
Lisa Knox [Mon, 1 Apr 2024 17:36:16 +0000 (13:36 -0400)]
21448: fixed user-profile test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

2 months ago21448: removed move to from processes Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa...
Lisa Knox [Mon, 1 Apr 2024 16:59:47 +0000 (12:59 -0400)]
21448: removed move to from processes Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

2 months ago21448: fixed user profile test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox...
Lisa Knox [Mon, 1 Apr 2024 15:50:54 +0000 (11:50 -0400)]
21448: fixed user profile test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

2 months ago21639: Don't return broken cache slots
Peter Amstutz [Mon, 1 Apr 2024 15:10:52 +0000 (11:10 -0400)]
21639: Don't return broken cache slots

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 months ago21448: changed int tests to new menu orders Arvados-DCO-1.1-Signed-off-by: Lisa Knox...
Lisa Knox [Mon, 1 Apr 2024 15:07:01 +0000 (11:07 -0400)]
21448: changed int tests to new menu orders Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

2 months ago21448: fixed toolbar divider so it renders consistently Arvados-DCO-1.1-Signed-off...
Lisa Knox [Mon, 1 Apr 2024 13:55:50 +0000 (09:55 -0400)]
21448: fixed toolbar divider so it renders consistently Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

2 months ago21639: Use a more efficient data structure for the keep block cache
Peter Amstutz [Mon, 1 Apr 2024 13:52:53 +0000 (09:52 -0400)]
21639: Use a more efficient data structure for the keep block cache

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

2 months ago21636: Test http status returned for unknown backend errors.
Tom Clegg [Mon, 1 Apr 2024 13:20:02 +0000 (09:20 -0400)]
21636: Test http status returned for unknown backend errors.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21636: Set ExpiryWindow to avoid credential expiry races.
Tom Clegg [Fri, 29 Mar 2024 20:58:23 +0000 (16:58 -0400)]
21636: Set ExpiryWindow to avoid credential expiry races.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoMerge branch '21617-fed-content'
Tom Clegg [Fri, 29 Mar 2024 20:39:09 +0000 (16:39 -0400)]
Merge branch '21617-fed-content'

fixes #21617
fixes #21533

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21617: Split test cases.
Tom Clegg [Fri, 29 Mar 2024 20:37:43 +0000 (16:37 -0400)]
21617: Split test cases.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21448: divider in place for toolbar menu Arvados-DCO-1.1-Signed-off-by: Lisa Knox...
Lisa Knox [Fri, 29 Mar 2024 18:44:24 +0000 (14:44 -0400)]
21448: divider in place for toolbar menu Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: divider in place for context menu Arvados-DCO-1.1-Signed-off-by: Lisa Knox...
Lisa Knox [Fri, 29 Mar 2024 17:54:48 +0000 (13:54 -0400)]
21448: divider in place for context menu Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: expanded copy to oclipboard change Arvados-DCO-1.1-Signed-off-by: Lisa Knox...
Lisa Knox [Fri, 29 Mar 2024 15:20:41 +0000 (11:20 -0400)]
21448: expanded copy to oclipboard change Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: channged wording Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Fri, 29 Mar 2024 15:15:03 +0000 (11:15 -0400)]
21448: channged wording Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: ms toolbar sorted Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Fri, 29 Mar 2024 15:08:11 +0000 (11:08 -0400)]
21448: ms toolbar sorted Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: removed share from process menu Arvados-DCO-1.1-Signed-off-by: Lisa Knox ...
Lisa Knox [Fri, 29 Mar 2024 14:26:14 +0000 (10:26 -0400)]
21448: removed share from process menu Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: running process also sorted Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa...
Lisa Knox [Fri, 29 Mar 2024 14:07:02 +0000 (10:07 -0400)]
21448: running process also sorted Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: applied sort to all context menus in ticket Arvados-DCO-1.1-Signed-off-by...
Lisa Knox [Fri, 29 Mar 2024 13:48:10 +0000 (09:48 -0400)]
21448: applied sort to all context menus in ticket Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21583: Override base64 dependency so it doesn't appear in Gemfile.lock
Brett Smith [Thu, 14 Mar 2024 15:17:24 +0000 (11:17 -0400)]
21583: Override base64 dependency so it doesn't appear in Gemfile.lock

See comments and linked bug for rationale.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21448: sorter sorts Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>
Lisa Knox [Thu, 28 Mar 2024 18:51:28 +0000 (14:51 -0400)]
21448: sorter sorts Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: initial sorting algo in place Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa...
Lisa Knox [Thu, 28 Mar 2024 17:37:54 +0000 (13:37 -0400)]
21448: initial sorting algo in place Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: adjusted specs to fit capitalization standardization Arvados-DCO-1.1-Signed...
Lisa Knox [Thu, 28 Mar 2024 15:01:26 +0000 (11:01 -0400)]
21448: adjusted specs to fit capitalization standardization Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21598: Don't run trash workers in container-local keepstore. 21598-local-keepstore-emptytrash
Tom Clegg [Thu, 28 Mar 2024 14:57:44 +0000 (10:57 -0400)]
21598: Don't run trash workers in container-local keepstore.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21448: combined contextmenunames and mstoolbaractionnames since they were mostly...
Lisa Knox [Thu, 28 Mar 2024 14:36:21 +0000 (10:36 -0400)]
21448: combined contextmenunames and mstoolbaractionnames since they were mostly duplicated and serve the same purpose Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: replaced all context menu names with string constants Arvados-DCO-1.1-Signed...
Lisa Knox [Thu, 28 Mar 2024 13:21:21 +0000 (09:21 -0400)]
21448: replaced all context menu names with string constants Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21541: Fix typo
Peter Amstutz [Thu, 28 Mar 2024 12:17:59 +0000 (08:17 -0400)]
21541: Fix typo

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months ago21541: Revert change to safeapi.py
Peter Amstutz [Wed, 27 Mar 2024 21:14:28 +0000 (17:14 -0400)]
21541: Revert change to safeapi.py

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months ago21541: Code cleanup and additional memory usage improvements
Peter Amstutz [Mon, 25 Mar 2024 18:19:32 +0000 (14:19 -0400)]
21541: Code cleanup and additional memory usage improvements

* Add slots to major Directory classes

* Disconnect FuseArvadosFile from ArvadosFile to reduce cyclic
references.

* Clean up _remove_inode loop and use dataclasses for the inode
operations.

* Now calls del_entry on collection_record_file and project_object_file.

It looks like collection_record_file was holding a reference to the
Collection object (and was remaining in the inodes table) even when
CollectionDirectory was cleared.  I believe this is the memory leak I
have been looking for.

* Remove the "dead" flag and set parent_inode to None instead.  This
clarifies the behavior that directory entries keep their (numeric)
inodes until they are detached from the directory which may have
contributed to infrequent "file not found" errors.

* Adjust cache behavior to only hold objects that are cache-eligible
and have non-zero cache_size.  This avoids filling the cache with
entries that are just going to be skipped over.

Overall: Memory usage is mostly stable but does tend to creep up over
time.  My best guess is that this is forced because we need to keep
inodes in RAM as long as the kernel maintains a reference to them, so
with multiple processes accessing different filesystem locations, this
is simply RAM required for the working set.

I'm also cautiously optimistic that issues I observed with performance
slowing down with long-lived processes are improved (e.g. fixing
memory leaks means no more unbounded growth of cache_entries, which
means no more time wasted iterating over huge lists).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months ago21448: expanded string constants Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox...
Lisa Knox [Wed, 27 Mar 2024 15:47:59 +0000 (11:47 -0400)]
21448: expanded string constants Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21448: created string constants for menu items Arvados-DCO-1.1-Signed-off-by: Lisa...
Lisa Knox [Wed, 27 Mar 2024 14:01:51 +0000 (10:01 -0400)]
21448: created string constants for menu items Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21617: Test unauthenticated endpoint + LoginCluster + bad token. 21617-fed-content
Tom Clegg [Tue, 26 Mar 2024 14:01:28 +0000 (10:01 -0400)]
21617: Test unauthenticated endpoint + LoginCluster + bad token.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21617: Interpret any 4xx as token rejection, not an error.
Tom Clegg [Tue, 26 Mar 2024 02:30:10 +0000 (22:30 -0400)]
21617: Interpret any 4xx as token rejection, not an error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21617: Accept bogus tokens at endpoints that do not require auth.
Tom Clegg [Mon, 25 Mar 2024 21:25:10 +0000 (17:25 -0400)]
21617: Accept bogus tokens at endpoints that do not require auth.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21508: Fix cross origin io parameter image previews
Stephen Smith [Mon, 25 Mar 2024 15:47:19 +0000 (11:47 -0400)]
21508: Fix cross origin io parameter image previews

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

3 months ago21508: Normalize io panel row height
Stephen Smith [Mon, 25 Mar 2024 15:46:52 +0000 (11:46 -0400)]
21508: Normalize io panel row height

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

3 months ago21601: Ensure pip is new enough to support package building
Brett Smith [Sun, 24 Mar 2024 03:26:58 +0000 (23:26 -0400)]
21601: Ensure pip is new enough to support package building

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21601: Build Python distro packages from wheels
Brett Smith [Sat, 23 Mar 2024 18:16:35 +0000 (14:16 -0400)]
21601: Build Python distro packages from wheels

The immediate problem this solves is that, by building and installing
from a repository of wheels, Python packages can find their
interdependencies without any special logic in the build process.

Other benefits:

* Eliminates some redundant work. We don't have to build the Python SDK
  from source multiple times. We can use the published cwltest wheel
  instead of building our own.

* Prepares the code for PEP 517 compliance. We only invoke setup.py to
  build packages that have not been updated yet. We introspect packages
  from their wheels, so we no longer have to introspect the source to
  build distro packages.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21601: Remove PySDK from `install deps`
Brett Smith [Sat, 23 Mar 2024 18:17:47 +0000 (14:17 -0400)]
21601: Remove PySDK from `install deps`

It's already installed by `install env`. ðŸ¤·

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21601: Install wheel in run-tests.sh VENV3DIR
Brett Smith [Tue, 19 Mar 2024 03:04:59 +0000 (23:04 -0400)]
21601: Install wheel in run-tests.sh VENV3DIR

We do this basically everywhere else in our build infrastructure, so
doing it here helps avoid discrepancies between those environments.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21601: Specify Python interdependencies with ~=
Brett Smith [Sat, 16 Mar 2024 23:23:44 +0000 (19:23 -0400)]
21601: Specify Python interdependencies with ~=

This has the same rationale as using <= before, but it's stricter. It
should prevent pip from using release versions to satisfy development
dependencies in the future, and help root out bugs in our build
processes.

DRY up this logic in arvados_version.py.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21601: Move --short-tests-only check to arvados_version
Brett Smith [Sat, 16 Mar 2024 23:10:15 +0000 (19:10 -0400)]
21601: Move --short-tests-only check to arvados_version

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months ago21601: Make arvados_version.py more declarative
Brett Smith [Sat, 16 Mar 2024 23:02:59 +0000 (19:02 -0400)]
21601: Make arvados_version.py more declarative

The main goal of this change is to introduce the metadata maps near the
top of the file, which we will use to build additional
functionality. The rest of the changes are just modernization or
clean-up based on that.

Arvados-DCO-1.1-Signed-off-by: Brett Smith <brett.smith@curii.com>

3 months agoFix open-ended bundler version in ruby install recipe.
Tom Clegg [Thu, 21 Mar 2024 19:11:41 +0000 (15:11 -0400)]
Fix open-ended bundler version in ruby install recipe.

refs #21463

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoMerge branch '21600-banner-tests'
Lisa Knox [Thu, 21 Mar 2024 17:17:45 +0000 (13:17 -0400)]
Merge branch '21600-banner-tests'

closes #21600

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch '21578-mount-debug'
Tom Clegg [Thu, 21 Mar 2024 15:39:15 +0000 (11:39 -0400)]
Merge branch '21578-mount-debug'

closes #21578

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoMerge branch '21603-ec2-subnet-error'
Tom Clegg [Thu, 21 Mar 2024 15:37:16 +0000 (11:37 -0400)]
Merge branch '21603-ec2-subnet-error'

fixes #21603

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoUpdate test failing on debian 12 if s3cmd is installed.
Tom Clegg [Thu, 21 Mar 2024 15:21:22 +0000 (11:21 -0400)]
Update test failing on debian 12 if s3cmd is installed.

Also, update `arvados-server install -type test` to install s3cmd if
available.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoMerge branch '21357-favorites-names'
Lisa Knox [Thu, 21 Mar 2024 13:10:23 +0000 (09:10 -0400)]
Merge branch '21357-favorites-names'

closes #21357

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21357: changed dispatch in loadFavoritesTRee to a string instead of an object Arvados... 21357-favorites-names
Lisa Knox [Wed, 20 Mar 2024 13:53:28 +0000 (09:53 -0400)]
21357: changed dispatch in loadFavoritesTRee to a string instead of an object Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch 'main' into 21357-favorites-names
Lisa Knox [Tue, 19 Mar 2024 18:41:12 +0000 (14:41 -0400)]
Merge branch 'main' into 21357-favorites-names

refs #21357

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21357: added loadsidepaneltree to updatecollection Arvados-DCO-1.1-Signed-off-by...
Lisa Knox [Tue, 19 Mar 2024 17:57:04 +0000 (13:57 -0400)]
21357: added loadsidepaneltree to updatecollection Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21357: moved resource fetches into a single await Arvados-DCO-1.1-Signed-off-by:...
Lisa Knox [Tue, 19 Mar 2024 17:26:20 +0000 (13:26 -0400)]
21357: moved resource fetches into a single await Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: combined changes by Stephen and Lucas Arvados-DCO-1.1-Signed-off-by: Lisa... 21600-banner-tests
Lisa Knox [Tue, 19 Mar 2024 14:52:22 +0000 (10:52 -0400)]
21600: combined changes by Stephen and Lucas Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch '21412-user-profile-bugs'
Lisa Knox [Tue, 19 Mar 2024 14:16:09 +0000 (10:16 -0400)]
Merge branch '21412-user-profile-bugs'

closes #21412

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: incorporated Stephens changes Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa...
Lisa Knox [Tue, 19 Mar 2024 13:21:01 +0000 (09:21 -0400)]
21600: incorporated Stephens changes Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: added waits everywhere it could possibly matter Arvados-DCO-1.1-Signed-off...
Lisa Knox [Mon, 18 Mar 2024 19:59:49 +0000 (15:59 -0400)]
21600: added waits everywhere it could possibly matter Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21603: Recognize subnet error despite generic error code. 21603-ec2-subnet-error
Tom Clegg [Mon, 18 Mar 2024 19:01:16 +0000 (15:01 -0400)]
21603: Recognize subnet error despite generic error code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21600: tweaked test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>
Lisa Knox [Mon, 18 Mar 2024 18:42:54 +0000 (14:42 -0400)]
21600: tweaked test  Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch 'main' into 21541-arv-mount-keyerror-rebase refs #21541
Peter Amstutz [Mon, 18 Mar 2024 18:35:21 +0000 (14:35 -0400)]
Merge branch 'main' into 21541-arv-mount-keyerror-rebase refs #21541

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months agoMerge branch '20455-noopener' refs #20455
Peter Amstutz [Mon, 18 Mar 2024 18:22:05 +0000 (14:22 -0400)]
Merge branch '20455-noopener' refs #20455

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months ago21600: removed tests to test spec itself Arvados-DCO-1.1-Signed-off-by: Lisa Knox...
Lisa Knox [Mon, 18 Mar 2024 18:13:45 +0000 (14:13 -0400)]
21600: removed tests to test spec itself Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch '21597-cypress-reply-fix' into main. Closes #21597
Stephen Smith [Mon, 18 Mar 2024 17:39:51 +0000 (13:39 -0400)]
Merge branch '21597-cypress-reply-fix' into main. Closes #21597

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

3 months ago2960: Remove unused code.
Tom Clegg [Mon, 18 Mar 2024 15:39:11 +0000 (11:39 -0400)]
2960: Remove unused code.

refs #2960

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21600: changed tooltip expectation Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa...
Lisa Knox [Mon, 18 Mar 2024 15:02:11 +0000 (11:02 -0400)]
21600: changed tooltip expectation Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch '21449-lsf-maxruntime'
Tom Clegg [Mon, 18 Mar 2024 14:48:05 +0000 (10:48 -0400)]
Merge branch '21449-lsf-maxruntime'

closes #21449

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21578: Add -debug alias for -log-level=debug. 21578-mount-debug
Tom Clegg [Mon, 18 Mar 2024 14:44:58 +0000 (10:44 -0400)]
21578: Add -debug alias for -log-level=debug.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21578: Merge branch 'main' into 21578-mount-debug
Tom Clegg [Mon, 18 Mar 2024 14:31:16 +0000 (10:31 -0400)]
21578: Merge branch 'main' into 21578-mount-debug

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21600: used the force Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>
Lisa Knox [Mon, 18 Mar 2024 14:20:55 +0000 (10:20 -0400)]
21600: used the force Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: set cypress data tag Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Mon, 18 Mar 2024 13:55:44 +0000 (09:55 -0400)]
21600: set cypress data tag Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: testing wait removal Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Mon, 18 Mar 2024 13:16:18 +0000 (09:16 -0400)]
21600: testing wait removal Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: removed only and changed wait to 1 second Arvados-DCO-1.1-Signed-off-by: Lisa...
Lisa Knox [Fri, 15 Mar 2024 20:33:39 +0000 (16:33 -0400)]
21600: removed only and changed wait to 1 second Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: added 3 second wait Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Fri, 15 Mar 2024 20:06:45 +0000 (16:06 -0400)]
21600: added 3 second wait Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: added only for testing Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox...
Lisa Knox [Fri, 15 Mar 2024 19:37:54 +0000 (15:37 -0400)]
21600: added only for testing Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: tweaked banner test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Fri, 15 Mar 2024 19:27:54 +0000 (15:27 -0400)]
21600: tweaked banner test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21600: tweaked banner test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii...
Lisa Knox [Fri, 15 Mar 2024 19:07:35 +0000 (15:07 -0400)]
21600: tweaked banner test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch '21165-installer-uninstalls-wb1'. Closes #21165
Lucas Di Pentima [Fri, 15 Mar 2024 18:13:11 +0000 (15:13 -0300)]
Merge branch '21165-installer-uninstalls-wb1'. Closes #21165

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

3 months ago21597: Use req.on('reply' instead of req.reply to workaround cypress bug 21597-cypress-reply-fix
Stephen Smith [Fri, 15 Mar 2024 17:15:08 +0000 (13:15 -0400)]
21597: Use req.on('reply' instead of req.reply to workaround cypress bug

https://github.com/cypress-io/cypress/issues/26248#issuecomment-1623005400

Arvados-DCO-1.1-Signed-off-by: Stephen Smith <stephen@curii.com>

3 months ago21541: Adjust dependency to arvados-llfuse fork
Peter Amstutz [Fri, 15 Mar 2024 13:56:09 +0000 (09:56 -0400)]
21541: Adjust dependency to arvados-llfuse fork

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months ago21541: Fix KeyError, segfaults, and memory use issues
Peter Amstutz [Wed, 6 Mar 2024 20:03:38 +0000 (15:03 -0500)]
21541: Fix KeyError, segfaults, and memory use issues

* Fixes a segfault on startup due to multiple threads fetching the
cluster config using the same http2 object, which is not threadafe.
Now fetches the relevant configuration
item once (ForwardSlashNameSubstitution), and stores it where all the
threads can access it.  (bug #21568)

* Fixes KeyError thrown where a parent inode is removed from the
inodes table before its children.

* In the process of testing, re-discovered a bug where, if the llfuse
_notify_queue fills up, the entire mount process deadlocks.

The previous fix worked by monkey-patching llfuse to replace a
limited-length queue with an unlimited length queue, however changes
in subsequent llfuse versions caused that variable to be hidden from
Python (so the monkey patch didn't fail but it no longer had any
effect either).  The solution is to introduce an additional
unlimited-size queue in between the operation handlers and the
limited-size kernel notification queue.

* Because cache management and inode cleanup interact with kernel
notifications (which were moved into a separate thread), I decided
they should also be asynchronous from the operation handlers, so they
are now part of the same thread that processes kernel notifications.

* Attempting to remove an inode that is in use will now at minimum
send a kernel invalidation, which will sometimes nudge the kernel to
forget the inode, enabling us to remove it.

* Filter groups now check if the filter group contains itself so it
doesn't create an infinite path loop that breaks filesystem traversal
tools.

* In the process of testing, found that llfuse didn't wait for the
_notify_queue to drain before closing the FUSE channel, resulting in a
segfault if the _notify_loop thread tried to process any events after
shutdown started.  This bug cannot be worked around on the Arvados
side, so I have prepared an arvados-llfuse fork with a bug fix.

* Testing with arv-mount-stress-test (which creates 8 subprocesses that all
traverse the filesystem at the same time) now passes with no
filesystem errors, no deadlocks, no segfaults, and modest memory
usage.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months agosdk services installs crunchstat-summary refs #19744
Peter Amstutz [Fri, 15 Mar 2024 14:45:03 +0000 (10:45 -0400)]
sdk services installs crunchstat-summary refs #19744

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>

3 months agoReorder python install steps to address dependency issue.
Tom Clegg [Fri, 15 Mar 2024 16:01:22 +0000 (12:01 -0400)]
Reorder python install steps to address dependency issue.

refs #19744

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago19744: Reorder python install steps to address dependency issue. 21449-lsf-maxruntime
Tom Clegg [Fri, 15 Mar 2024 13:56:03 +0000 (09:56 -0400)]
19744: Reorder python install steps to address dependency issue.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoMerge branch '21507-long-project-title-wrap'
Lisa Knox [Fri, 15 Mar 2024 13:05:56 +0000 (09:05 -0400)]
Merge branch '21507-long-project-title-wrap'

closes #21507

Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21165: Uninstalls WB1 from single host installations too. 21165-installer-uninstalls-wb1
Lucas Di Pentima [Thu, 14 Mar 2024 20:54:02 +0000 (17:54 -0300)]
21165: Uninstalls WB1 from single host installations too.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

3 months ago21165: Adds extra state to uninstall wb1's package from workbench node.
Lucas Di Pentima [Thu, 14 Mar 2024 19:33:33 +0000 (16:33 -0300)]
21165: Adds extra state to uninstall wb1's package from workbench node.

I opted for specifically uninstalling the package instead of adding the
arvados.workbench.package.clean state, because that state also removes
packages that might be needed for other services.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>

3 months ago21449: Fix ordering in "install deps".
Tom Clegg [Thu, 14 Mar 2024 19:25:46 +0000 (15:25 -0400)]
21449: Fix ordering in "install deps".

"install sdk/cli" is meant to use the current version of the arvados
gem from this checkout, but that can only happen if "install sdk/ruby"
runs first.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21507: tweaked banner test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii... 21507-long-project-title-wrap
Lisa Knox [Thu, 14 Mar 2024 17:55:27 +0000 (13:55 -0400)]
21507: tweaked banner test Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21449: Propagate max_run_time to LSF scheduler.
Tom Clegg [Wed, 13 Mar 2024 21:22:39 +0000 (17:22 -0400)]
21449: Propagate max_run_time to LSF scheduler.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months agoMerge branch '21123-scheduling-status'
Tom Clegg [Thu, 14 Mar 2024 14:59:37 +0000 (10:59 -0400)]
Merge branch '21123-scheduling-status'

closes #21123

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21123: Add comment re arv-mount dependency.
Tom Clegg [Thu, 14 Mar 2024 14:58:47 +0000 (10:58 -0400)]
21123: Add comment re arv-mount dependency.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>

3 months ago21507: cleanup Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>
Lisa Knox [Thu, 14 Mar 2024 14:36:48 +0000 (10:36 -0400)]
21507: cleanup Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21507: expanded nowrap to all of the places it should apply Arvados-DCO-1.1-Signed...
Lisa Knox [Thu, 14 Mar 2024 14:35:30 +0000 (10:35 -0400)]
21507: expanded nowrap to all of the places it should apply Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months ago21507: removed word wrap on collection owner field Arvados-DCO-1.1-Signed-off-by...
Lisa Knox [Thu, 14 Mar 2024 14:22:22 +0000 (10:22 -0400)]
21507: removed word wrap on collection owner field Arvados-DCO-1.1-Signed-off-by: Lisa Knox <lisa.knox@curii.com>

3 months agoMerge branch '21585-installer-imdsv2'. Closes #21585
Lucas Di Pentima [Wed, 13 Mar 2024 20:13:42 +0000 (17:13 -0300)]
Merge branch '21585-installer-imdsv2'. Closes #21585

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas.dipentima@curii.com>