Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Added function to work with projects.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

12755: Stop building arvados-docker-cleaner for centos7.

(until we make the package installable again)

refs #12755

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Added feature to move File-like object inside collection.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

12377: Update workbench test breaking since 484246ac0.

refs #12377

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12715-slow-propfind'

fixes #12715

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12822-keepproxy-token-scope'

refs #12822

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12377-workbench-composer-link' refs #12377

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '12821-sbatch-script-race'

refs #12821

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Prepare for packaging.

refs #12837

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Made Subcollection and ArvadosFile more flexible.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

12821: Fix race in concurrent calls to submit().

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12721-build-test-version'

refs #12721

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12721: Add run-build-test-packages-one-target.sh --build-version arg

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12715: Rename flag !reading => alwaysReadEOF.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Add composer_url to arvbox workbench config refs #12377

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Added feature to delete file from a collection.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

12715: Fix slow PROPFIND by avoiding content probing.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12822: Allow reading from keepproxy with collection sharing tokens.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12377: Add link to composer on workbench

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '8311-mount-git'

closes #8311

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Fix gitUrl discovery doc for config git_repo_https_base: true.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Ensure git tree is readable even with umask 077.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Remove support for writable git_tree mount.

Feature is of dubious value, and expected behavior isn't clear --
e.g., should all files be world-writable?

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Ensure git tree tmpdir is readable by container.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Pass in container token to extractTree

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

8311: Ensure gitUrl is always a string in the discovery doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Fix API tests broken in 4343e1f.

* Restore arvados repo dropped from tarball.

* Fix git repo fixtures being removed in API test teardown.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Tidy up integration test services.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Update docs to describe current implementation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Reject invalid/unsupported git mounts. Default to path="/".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Added feature to create new file in a particular collection.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

8311: Try some git_tree mounts in TestSetupMounts.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Add git_tree mount type.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Add auth handler tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Access git repos by UUID.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Fix "read" or "write" mistakenly appended to response data.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

8311: Remove debug printf.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Fix Access-Control-Allow-Headers CORS header tests refs #12060

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Add Content-Type to Access-Control-Allow-Headers CORS headers refs #12060

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Refactored collection tree into separate file.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Merge branch '4019-query-properties' closes #4019

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

4019: More tests for error cases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Added ability to write to a collection file.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

4019: Improve error checking, add tests for some error cases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '12614-node-states' refs #12614

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Tag arvados/jobs image with $cwl_runner_version instead of git hash refs #12811

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12614: Can shutdown failed nodes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12614: Support daemon actor aware of 'fail' node state

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12614: Make node manager aware of "fail" node state.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Merge branch '12803-unescaped-colon-filename'

fixes #12803

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12765-workbench-404-trashed' refs #12765

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12803: Accept unescaped colons in filenames.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

4019: Add index for efficient jsonb query on collection properties

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

4019: Add URI quoting option for keys.  Update docs.  Fix tests.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Implemented feature to get size of ArvadosFile or Subcollection

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

4019: Update docs.  Tweak syntax of 'exists' and add alternate form.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '12550-crunch1-exit-race'

refs #12550

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12511-is-trashed-query' refs #12511

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

4019: Expand testing.  Make behavior consistent for != and "not in"

Rename "?" to "exists" and add "not exists".

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '12167-python-request-id'

refs #12167

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Allow --silent to be combined with -*-progress flags.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Test arv-get and arv-put X-Request-Id logging.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12511: readable_by filters on is_trashed directly

Fix performance problem in the negative join NOT EXISTS(...) used to filter out
trashed items for admin users.  The clause
(uuid = target_uuid OR owner_uuid = target_uuid) doesn't use the index
efficiently and results in a very expensive seq scan.

However, the "OR" in the negative clause is not necessary.  We only need
materialized_permission_view to know if "owner_uuid" is trashed (directly or
indirectly), not "uuid".  If the record type has an is_trashed flag (currently
only collections and groups) then it is more efficient to filter on that
directly.

This commit refactors the readable_by query to only check owner_uuid, and
additionally filter on "is_trashed" for collections and groups.

As an additional cleanup, because "readable_by" now explicitly knows to filter
on "is_trashed" this makes the default_scope in Collections redundant.  This
commit also removes default_scope & various uses of unscoped reduces complexity.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12550: Fix race: read buffered data from stderr after child exits.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Removed debug logs.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Moved all classes from RefClass OOP model to R6 and made improvements
to tree structure representing collection content.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

12721: Add missing _version.py, now part of source tree.

refs #12721

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

4019: Initial support for queries on jsonb subproperties.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '12765-permission-graph-trash' refs #12765

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12765: Add explicit flag whether "trashed" propagates across an edge

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12765: Add test case that "trashed" should not propagate over permission links.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch '11908-properties-column-json'

refs #11908

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Merge branch '12721-build-package-version'

refs #12721

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Test X-Request-Id request headers in Keep get/put/head reqs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Use one X-Request-Id per arv-put; display it if not --silent.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Use one X-Request-Id per arv-get process; display it if -v.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Send caller-provided or random X-Request-Id in Keep requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

12167: Send caller-provided or random X-Request-Id in API requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Write composer.yml instead of arvados-configuration.yml refs #12353

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12765: Fix untrash button on 404 page.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

12765: Add test for untrash on 404 page

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

Merge branch 'master' of git.curoverse.com:arvados into 11876-r-sdk

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Added ability to read file from a collection and made minor fixes to
collection content tree.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

Merge branch '11453-federated-tokens'

refs #11453

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

11453: Restore update of email/prefs/name on subsequent remote auth.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

11453: Test username is not updated once assigned.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

11453: Assign username when creating mirror of remote user.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

11453: "sso_insecure" config applies to remote Arvados auth, too.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Updated README and did some minor fixes to Arvados.R and Collection.R

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

11453: Merge branch 'master' into 11453-federated-tokens

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

Added ability to extract Subcollection from collection content tree.

Arvados-DCO-1.1-Signed-off-by: Fuad Muhic <fmuhic@capeannenterprises.com>

11908: Add comment about dropping/recreating FT index.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>