#
# 1. commit is directly tagged. print that.
#
-# 2. commit is on master or a development branch, the nearest tag is older
-# than commit where this branch joins master.
+# 2. commit is on main or a development branch, the nearest tag is older
+# than commit where this branch joins main.
# -> take greatest version tag in repo X.Y.Z and assign X.(Y+1).0
#
# 3. commit is on a release branch, the nearest tag is newer
-# than the commit where this branch joins master.
+# than the commit where this branch joins main.
# -> take nearest tag X.Y.Z and assign X.Y.(Z+1)
tagged=$(git tag --points-at "$commit")
echo $tagged
else
# 1. get the nearest tag with 'git describe'
- # 2. get the merge base between this commit and master
+ # 2. get the merge base between this commit and main
# 3. if the tag is an ancestor of the merge base,
# (tag is older than merge base) increment minor version
# else, tag is newer than merge base, so increment point version
nearest_tag=$(git describe --tags --abbrev=0 --match "$versionglob" "$commit")
- merge_base=$(git merge-base origin/master "$commit")
+ merge_base=$(git merge-base origin/main "$commit")
if git merge-base --is-ancestor "$nearest_tag" "$merge_base" ; then
# x.(y+1).0~devTIMESTAMP, where x.y.z is the newest version that does not contain $commit
* Delete the affected collections so that job reuse doesn't attempt to reuse them (it's likely that if one block is missing, they all are, so they're unlikely to contain any useful data)
* Resubmit any container requests for which you want the output collections regenerated
-The Arvados repository contains a tool that can be used to generate a report to help with this task at "arvados/tools/keep-xref/keep-xref.py":https://github.com/arvados/arvados/blob/master/tools/keep-xref/keep-xref.py
+The Arvados repository contains a tool that can be used to generate a report to help with this task at "arvados/tools/keep-xref/keep-xref.py":https://github.com/arvados/arvados/blob/main/tools/keep-xref/keep-xref.py
--- /dev/null
+---
+layout: default
+navsection: admin
+title: Restricting upload or download
+...
+
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+For some use cases, you may want to limit the ability of users to upload or download data from outside the cluster. (By "outside" we mean from networks other than the cluster's own private network). For example, this makes it possible to share restricted data sets with users so that they may run their own data analysis on the cluster, while preventing them from easily downloading the data set to their local workstation.
+
+This feature exists in addition to the existing Arvados permission system. Users can only download from collections they have @read@ access to, and can only upload to projects and collections they have @write@ access to.
+
+There are two services involved in accessing data from outside the cluster.
+
+h2. Keepproxy Permissions
+
+Permitting @keeproxy@ makes it possible to use @arv-put@ and @arv-get@, and upload from Workbench 1. It works in terms of individual 64 MiB keep blocks. It prints a log each time a user uploads or downloads an individual block.
+
+The default policy allows anyone to upload or download.
+
+<pre>
+ Collections:
+ KeepproxyPermission:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+</pre>
+
+If you create a sharing link as an admin user, and then give someone the token from the sharing link to download a file using @arv-get@, because the downloader is anonymous, the download permission will be restricted based on the "User" role and not the "Admin" role.
+
+h2. WebDAV and S3 API Permissions
+
+Permitting @WebDAV@ makes it possible to use WebDAV, S3 API, download from Workbench 1, and upload/download with Workbench 2. It works in terms of individual files. It prints a log each time a user uploads or downloads a file. When @WebDAVLogEvents@ (default true) is enabled, it also adds an entry into the API server @logs@ table.
+
+When a user attempts to upload or download from a service without permission, they will receive a @403 Forbidden@ response. This only applies to file content.
+
+Denying download permission does not deny access to access to XML file listings with PROPFIND, or auto-generated HTML documents containing file listings.
+
+Denying upload permission does not deny other operations that modify collections without directly accessing file content, such as MOVE and COPY.
+
+The default policy allows anyone to upload or download.
+
+<pre>
+ Collections:
+ WebDAVPermisison:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+ WebDAVLogEvents: true
+</pre>
+
+If you create a sharing link as an admin user, and then give someone the token from the sharing link to download a file over HTTP (WebDAV or S3 API), because the downloader is anonymous, the download permission will be restricted based on the "User" role and not the "Admin" role.
+
+h2. Shell node and container permissions
+
+Be aware that even when upload and download from outside the network is not allowed, a user who has access to a shell node or runs a container still has internal access to Keep. (This is necessary to be able to run workflows). From the shell node or container, a user could send data outside the network by some other method, although this requires more intent than accidentally clicking on a link and downloading a file. It is possible to set up a firewall to prevent shell and compute nodes from making connections to hosts outside the private network. Exactly how to configure firewalls is out of scope for this page, as it depends on the specific network infrastructure of your cluster.
+
+h2. Choosing a policy
+
+This distinction between WebDAV and Keepproxy is important for auditing. WebDAV records 'upload' and 'download' events on the API server that are included in the "User Activity Report":user-activity.html , whereas @keepproxy@ only logs upload and download of individual blocks, which require a reverse lookup to determine the collection(s) and file(s) a block is associated with.
+
+You set separate permissions for @WebDAV@ and @Keepproxy@, with separate policies for regular users and admin users.
+
+These policies apply to only access from outside the cluster, using Workbench or Arvados CLI tools.
+
+The @WebDAVLogEvents@ option should be enabled if you intend to the run the "User Activity Report":user-activity.html . If you don't need audits, or you are running a site that is mostly serving public data to anonymous downloaders, you can disable in to avoid the extra API server request.
+
+h3. Audited downloads
+
+For ease of access auditing, this policy prevents downloads using @arv-get@. Downloads through WebDAV and S3 API are permitted, but logged. Uploads are allowed.
+
+<pre>
+ Collections:
+ WebDAVPermisison:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+
+ KeepproxyPermission:
+ User:
+ Download: false
+ Upload: true
+ Admin:
+ Download: false
+ Upload: true
+ WebDAVLogEvents: true
+</pre>
+
+h3. Disallow downloads by regular users
+
+This policy prevents regular users (non-admin) from downloading data. Uploading is allowed. This supports the case where restricted data sets are shared with users so that they may run their own data analysis on the cluster, while preventing them from downloading the data set to their local workstation. Be aware that users won't be able to download the results of their analysis, either, requiring an admin in the loop or some other process to release results.
+
+<pre>
+ Collections:
+ WebDAVPermisison:
+ User:
+ Download: false
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+
+ KeepproxyPermission:
+ User:
+ Download: false
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+ WebDAVLogEvents: true
+</pre>
+
+h3. Disallow uploads by regular users
+
+This policy is suitable for an installation where data is being shared with a group of users who are allowed to download the data, but not permitted to store their own data on the cluster.
+
+<pre>
+ Collections:
+ WebDAVPermisison:
+ User:
+ Download: true
+ Upload: false
+ Admin:
+ Download: true
+ Upload: true
+
+ KeepproxyPermission:
+ User:
+ Download: true
+ Upload: false
+ Admin:
+ Download: true
+ Upload: true
+ WebDAVLogEvents: true
+</pre>
h3. Storage management notes
-The "keep-balance":{{site.baseurl}}/install/install-keep-balance.html service is responsible for deciding which blocks should be placed on which keepstore volumes. As part of the rebalancing behavior, it will determine where a block should go in order to satisfy the desired storage classes, and issue pull requests to copy the block from its original volume to the desired volume. The block will subsequently be moved to trash on the original volume.
+When uploading data, if a data block cannot be uploaded to all desired storage classes, it will result in a fatal error. Data blocks will not be uploaded to volumes that do not have the desired storage class.
-If a block appears in multiple collections with different storage classes, the block will be stored in separate volumes for each storage class, even if that results in overreplication, unless there is a volume which has all the desired storage classes.
+If you change the storage classes for a collection, the data is not moved immediately. The "keep-balance":{{site.baseurl}}/install/install-keep-balance.html service is responsible for deciding which blocks should be placed on which keepstore volumes. As part of the rebalancing behavior, it will determine where a block should go in order to satisfy the desired storage classes, and issue pull requests to copy the block from its original volume to the desired volume. The block will subsequently be moved to trash on the original volume.
-If a collection has a desired storage class which is not available in any keepstore volume, the collection's blocks will remain in place, and an error will appear in the @keep-balance@ logs.
+If a block is assigned to multiple storage classes, the block will be stored on @desired_replication@ number of volumes for storage class, even if that results in overreplication.
-This feature does not provide a hard guarantee on where data will be stored. Data may be written to default storage and moved to the desired storage class later. If controlling data locality is a hard requirement (such as legal restrictions on the location of data) we recommend setting up multiple Arvados clusters.
+If a collection has a desired storage class which is not available in any keepstore volume, the collection's blocks will remain in place, and an error will appear in the @keep-balance@ logs.
This release corrects a potential data loss issue, if you are running Arvados 1.3.0 or 1.3.1 we strongly recommended disabling @keep-balance@ until you can upgrade to 1.3.3 or 1.4.0. With keep-balance disabled, there is no chance of data loss.
-We've put together a "wiki page":https://dev.arvados.org/projects/arvados/wiki/Recovering_lost_data which outlines how to recover blocks which have been put in the trash, but not yet deleted, as well as how to identify any collections which have missing blocks so that they can be regenerated. The keep-balance component has been enhanced to provide a list of missing blocks and affected collections and we've provided a "utility script":https://github.com/arvados/arvados/blob/master/tools/keep-xref/keep-xref.py which can be used to identify the workflows that generated those collections and who ran those workflows, so that they can be rerun.
+We've put together a "wiki page":https://dev.arvados.org/projects/arvados/wiki/Recovering_lost_data which outlines how to recover blocks which have been put in the trash, but not yet deleted, as well as how to identify any collections which have missing blocks so that they can be regenerated. The keep-balance component has been enhanced to provide a list of missing blocks and affected collections and we've provided a "utility script":https://github.com/arvados/arvados/blob/main/tools/keep-xref/keep-xref.py which can be used to identify the workflows that generated those collections and who ran those workflows, so that they can be rerun.
h2(#v1_3_0). v1.3.0 (2018-12-05)
|runtime_token|string|A v2 token to be passed into the container itself, used to access Keep-backed mounts, etc. |Not returned in API responses. Reset to null when state is "Complete" or "Cancelled".|
|runtime_user_uuid|string|The user permission that will be granted to this container.||
|runtime_auth_scopes|array of string|The scopes associated with the auth token used to run this container.||
+|output_storage_classes|array of strings|The storage classes that will be used for the log and output collections of this container request|default is ["default"]|
h2(#priority). Priority
|runtime_token|string|A v2 token to be passed into the container itself, used to access Keep-backed mounts, etc.|Not returned in API responses. Reset to null when state is "Complete" or "Cancelled".|
|gateway_address|string|Address (host:port) of gateway server.|Internal use only.|
|interactive_session_started|boolean|Indicates whether @arvados-client shell@ has been used to run commands in the container, which may have altered the container's behavior and output.||
+|output_storage_classes|array of strings|The storage classes that will be used for the log and output collections of this container||
h2(#container_states). Container states
table(table table-bordered table-condensed).
|\3=. *Core*|
-|"Postgres database":install-postgresql.html |Stores data for the API server.|Required.|
+|"PostgreSQL database":install-postgresql.html |Stores data for the API server.|Required.|
|"API server":install-api-server.html |Core Arvados logic for managing users, groups, collections, containers, and enforcing permissions.|Required.|
|\3=. *Keep (storage)*|
|"Keepstore":install-keepstore.html |Stores content-addressed blocks in a variety of backends (local filesystem, cloud object storage).|Required.|
* LDAP login to authenticate users by username/password using the LDAP protocol, supported by many services such as OpenLDAP and Active Directory.
* PAM login to authenticate users by username/password according to the PAM configuration on the controller node.
+h2(#postgresql). PostgreSQL
+
+Arvados works well with a standalone PostgreSQL installation. When deploying on AWS, Aurora RDS also works but Aurora Serverless is not recommended.
+
h2(#storage). Storage backend
Choose which backend you will use for storing and retrieving content-addressed Keep blocks.
<div class="offset1">
table(table table-bordered table-condensed).
|_. Function|_. Number of nodes|_. Recommended specs|
-|Postgres database, Arvados API server, Arvados controller, Git, Websockets, Container dispatcher|1|16+ GiB RAM, 4+ cores, fast disk for database|
+|PostgreSQL database, Arvados API server, Arvados controller, Git, Websockets, Container dispatcher|1|16+ GiB RAM, 4+ cores, fast disk for database|
|Workbench, Keepproxy, Keep-web, Keep-balance|1|8 GiB RAM, 2+ cores|
|Keepstore servers ^1^|2+|4 GiB RAM|
|Compute worker nodes ^1^|0+ |Depends on workload; scaled dynamically in the cloud|
h3(#hosts_setup_using_terraform). Hosts setup using terraform (AWS, experimental)
-We added a few "terraform":https://terraform.io/ scripts (https://github.com/arvados/arvados/tree/master/tools/terraform) to let you create these instances easier in an AWS account. Check "the Arvados terraform documentation":/doc/install/terraform.html for more details.
+We added a few "terraform":https://terraform.io/ scripts (https://github.com/arvados/arvados/tree/main/tools/terraform) to let you create these instances easier in an AWS account. Check "the Arvados terraform documentation":/doc/install/terraform.html for more details.
{% if site.current_version %}
{% assign branchname = site.current_version | slice: 1, 5 | append: '-dev' %}
{% else %}
-{% assign branchname = 'master' %}
+{% assign branchname = 'main' %}
{% endif %}
This is a package-based installation method. Start with the @provision.sh@ script which is available by cloning the @{{ branchname }}@ branch from "https://git.arvados.org/arvados.git":https://git.arvados.org/arvados.git . The @provision.sh@ script and its supporting files can be found in the "arvados/tools/salt-install":https://git.arvados.org/arvados.git/tree/refs/heads/{{ branchname }}:/tools/salt-install directory in the Arvados git repository.
h3(#create_a_compute_image). Create a compute image
-In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/master/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image ID in the Arvados configuration in the next steps.
+In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/main/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image ID in the Arvados configuration in the next steps.
h2(#choose_configuration). Choose the desired configuration
{% if site.current_version %}
{% assign branchname = site.current_version | slice: 1, 5 | append: '-dev' %}
{% else %}
-{% assign branchname = 'master' %}
+{% assign branchname = 'main' %}
{% endif %}
This is a package-based installation method. Start with the @provision.sh@ script which is available by cloning the @{{ branchname }}@ branch from "https://git.arvados.org/arvados.git":https://git.arvados.org/arvados.git . The @provision.sh@ script and its supporting files can be found in the "arvados/tools/salt-install":https://git.arvados.org/arvados.git/tree/refs/heads/{{ branchname }}:/tools/salt-install directory in the Arvados git repository.
h2(#vagrant). Vagrant
-This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/main/tools/salt-install directory in the Arvados git repository.
A @Vagrantfile@ is provided to install Arvados in a virtual machine on your computer using "Vagrant":https://www.vagrantup.com/.
<notextile>{% code example_sdk_go as go %}</notextile>
-A few more usage examples can be found in the "services/keepproxy":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/services/keepproxy and "sdk/go/keepclient":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/go/keepclient directories in the arvados source tree.
+A few more usage examples can be found in the "services/keepproxy":https://dev.arvados.org/projects/arvados/repository/revisions/main/show/services/keepproxy and "sdk/go/keepclient":https://dev.arvados.org/projects/arvados/repository/revisions/main/show/sdk/go/keepclient directories in the arvados source tree.
h2(#get-files). Get the tutorial files
-The tutorial files are located in the documentation section of the Arvados source repository, which can be found on "git.arvados.org":https://git.arvados.org/arvados.git/tree/HEAD:/doc/user/cwl/bwa-mem or "github":https://github.com/arvados/arvados/tree/master/doc/user/cwl/bwa-mem
+The tutorial files are located in the documentation section of the Arvados source repository, which can be found on "git.arvados.org":https://git.arvados.org/arvados.git/tree/HEAD:/doc/user/cwl/bwa-mem or "github":https://github.com/arvados/arvados/tree/main/doc/user/cwl/bwa-mem
<notextile>
<pre><code>~$ <span class="userinput">git clone https://git.arvados.org/arvados.git</span>
h2. Get the example files
-The tutorial files are located in the "documentation section of the Arvados source repository:":https://github.com/arvados/arvados/tree/master/doc/user/cwl/federated or "see below":#fed-example
+The tutorial files are located in the "documentation section of the Arvados source repository:":https://github.com/arvados/arvados/tree/main/doc/user/cwl/federated or "see below":#fed-example
<notextile>
<pre><code>~$ <span class="userinput">git clone https://github.com/arvados/arvados</span>
# Persistent sessions.
MaxSessions: 100
+ # Selectively set permissions for regular users and admins to
+ # download or upload data files using the upload/download
+ # features for Workbench, WebDAV and S3 API support.
+ WebDAVPermission:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+
+ # Selectively set permissions for regular users and admins to be
+ # able to download or upload blocks using arv-put and
+ # arv-get from outside the cluster.
+ KeepproxyPermission:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+
+ # Post upload / download events to the API server logs table, so
+ # that they can be included in the arv-user-activity report.
+ # You can disable this if you find that it is creating excess
+ # load on the API server and you don't need it.
+ WebDAVLogEvents: true
+
Login:
# One of the following mechanisms (Google, PAM, LDAP, or
# LoginCluster) should be enabled; see
"Collections.TrashSweepInterval": false,
"Collections.TrustAllContent": false,
"Collections.WebDAVCache": false,
+ "Collections.KeepproxyPermission": false,
+ "Collections.WebDAVPermission": false,
+ "Collections.WebDAVLogEvents": false,
"Containers": true,
"Containers.CloudVMs": false,
"Containers.CrunchRunArgumentsList": false,
# Persistent sessions.
MaxSessions: 100
+ # Selectively set permissions for regular users and admins to
+ # download or upload data files using the upload/download
+ # features for Workbench, WebDAV and S3 API support.
+ WebDAVPermission:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+
+ # Selectively set permissions for regular users and admins to be
+ # able to download or upload blocks using arv-put and
+ # arv-get from outside the cluster.
+ KeepproxyPermission:
+ User:
+ Download: true
+ Upload: true
+ Admin:
+ Download: true
+ Upload: true
+
+ # Post upload / download events to the API server logs table, so
+ # that they can be included in the arv-user-activity report.
+ # You can disable this if you find that it is creating excess
+ # load on the API server and you don't need it.
+ WebDAVLogEvents: true
+
Login:
# One of the following mechanisms (Google, PAM, LDAP, or
# LoginCluster) should be enabled; see
ManifestFileReader(m manifest.Manifest, filename string) (arvados.File, error)
LocalLocator(locator string) (string, error)
ClearBlockCache()
+ SetStorageClasses(sc []string)
}
// NewLogWriter is a factory function to create a new log writer.
"--foreground",
"--allow-other",
"--read-write",
+ "--storage-classes", strings.Join(runner.Container.OutputStorageClasses, ","),
fmt.Sprintf("--crunchstat-interval=%v", runner.statInterval.Seconds())}
if runner.Container.RuntimeConstraints.KeepCacheRAM > 0 {
return fmt.Errorf("error creating container API client: %v", err)
}
+ runner.ContainerKeepClient.SetStorageClasses(runner.Container.OutputStorageClasses)
+ runner.DispatcherKeepClient.SetStorageClasses(runner.Container.OutputStorageClasses)
+
err = runner.ContainerArvClient.Call("GET", "containers", runner.Container.UUID, "secret_mounts", nil, &sm)
if err != nil {
if apierr, ok := err.(arvadosclient.APIServerError); !ok || apierr.HttpStatusCode != 404 {
var _ = Suite(&TestSuite{})
type TestSuite struct {
- client *arvados.Client
- api *ArvTestClient
- runner *ContainerRunner
- executor *stubExecutor
- keepmount string
+ client *arvados.Client
+ api *ArvTestClient
+ runner *ContainerRunner
+ executor *stubExecutor
+ keepmount string
+ testDispatcherKeepClient KeepTestClient
+ testContainerKeepClient KeepTestClient
}
func (s *TestSuite) SetUpTest(c *C) {
s.executor = &stubExecutor{}
var err error
s.api = &ArvTestClient{}
- s.runner, err = NewContainerRunner(s.client, s.api, &KeepTestClient{}, "zzzzz-zzzzz-zzzzzzzzzzzzzzz")
+ s.runner, err = NewContainerRunner(s.client, s.api, &s.testDispatcherKeepClient, "zzzzz-zzzzz-zzzzzzzzzzzzzzz")
c.Assert(err, IsNil)
s.runner.executor = s.executor
s.runner.MkArvClient = func(token string) (IArvadosClient, IKeepClient, *arvados.Client, error) {
- return s.api, &KeepTestClient{}, s.client, nil
+ return s.api, &s.testContainerKeepClient, s.client, nil
}
s.runner.RunArvMount = func(cmd []string, tok string) (*exec.Cmd, error) {
s.runner.ArvMountPoint = s.keepmount
}
type KeepTestClient struct {
- Called bool
- Content []byte
+ Called bool
+ Content []byte
+ StorageClasses []string
}
type stubExecutor struct {
client.Content = nil
}
+func (client *KeepTestClient) SetStorageClasses(sc []string) {
+ client.StorageClasses = sc
+}
+
type FileWrapper struct {
io.ReadCloser
len int64
s.runner.NewLogWriter = logs.NewTestLoggingWriter
s.runner.Container.ContainerImage = arvadostest.DockerImage112PDH
s.runner.Container.Command = []string{"./hw"}
+ s.runner.Container.OutputStorageClasses = []string{"default"}
imageID, err := s.runner.LoadImage()
c.Assert(err, IsNil)
return d, err
}
s.runner.MkArvClient = func(token string) (IArvadosClient, IKeepClient, *arvados.Client, error) {
- return &ArvTestClient{secretMounts: secretMounts}, &KeepTestClient{}, nil, nil
+ return &ArvTestClient{secretMounts: secretMounts}, &s.testContainerKeepClient, nil, nil
}
if extraMounts != nil && len(extraMounts) > 0 {
"output_path": "/tmp",
"priority": 1,
"runtime_constraints": {"vcpus":1,"ram":1000000},
- "state": "Locked"
+ "state": "Locked",
+ "output_storage_classes": ["default"]
}`, nil, 0, func() {
c.Check(s.executor.created.Command, DeepEquals, []string{"echo", "hello world"})
c.Check(s.executor.created.Image, Equals, "sha256:d8309758b8fe2c81034ffc8a10c36460b77db7bc5e7b448c4e5b684f9d95a678")
c.Check(s.api.CalledWith("container.exit_code", 0), NotNil)
c.Check(s.api.CalledWith("container.state", "Complete"), NotNil)
c.Check(s.api.Logs["stdout"].String(), Matches, ".*hello world\n")
-
+ c.Check(s.testDispatcherKeepClient.StorageClasses, DeepEquals, []string{"default"})
+ c.Check(s.testContainerKeepClient.StorageClasses, DeepEquals, []string{"default"})
}
func (s *TestSuite) TestRunAlreadyRunning(c *C) {
c.Check(s.api.Logs["stdout"].String(), Matches, ".*/bin\n")
}
+func (s *TestSuite) TestFullRunSetOutputStorageClasses(c *C) {
+ s.fullRunHelper(c, `{
+ "command": ["pwd"],
+ "container_image": "`+arvadostest.DockerImage112PDH+`",
+ "cwd": "/bin",
+ "environment": {},
+ "mounts": {"/tmp": {"kind": "tmp"} },
+ "output_path": "/tmp",
+ "priority": 1,
+ "runtime_constraints": {},
+ "state": "Locked",
+ "output_storage_classes": ["foo", "bar"]
+}`, nil, 0, func() {
+ fmt.Fprintln(s.executor.created.Stdout, s.executor.created.WorkingDir)
+ })
+
+ c.Check(s.api.CalledWith("container.exit_code", 0), NotNil)
+ c.Check(s.api.CalledWith("container.state", "Complete"), NotNil)
+ c.Check(s.api.Logs["stdout"].String(), Matches, ".*/bin\n")
+ c.Check(s.testDispatcherKeepClient.StorageClasses, DeepEquals, []string{"foo", "bar"})
+ c.Check(s.testContainerKeepClient.StorageClasses, DeepEquals, []string{"foo", "bar"})
+}
+
func (s *TestSuite) TestStopOnSignal(c *C) {
s.executor.runFunc = func() {
s.executor.created.Stdout.Write([]byte("foo\n"))
cr.RunArvMount = am.ArvMountTest
cr.ContainerArvClient = &ArvTestClient{}
cr.ContainerKeepClient = &KeepTestClient{}
+ cr.Container.OutputStorageClasses = []string{"default"}
realTemp := c.MkDir()
certTemp := c.MkDir()
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "default", "--crunchstat-interval=5",
"--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{"/tmp": {realTemp + "/tmp2", false}})
os.RemoveAll(cr.ArvMountPoint)
cr.Container.Mounts["/out"] = arvados.Mount{Kind: "tmp"}
cr.Container.Mounts["/tmp"] = arvados.Mount{Kind: "tmp"}
cr.Container.OutputPath = "/out"
+ cr.Container.OutputStorageClasses = []string{"foo", "bar"}
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "foo,bar", "--crunchstat-interval=5",
"--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{"/out": {realTemp + "/tmp2", false}, "/tmp": {realTemp + "/tmp3", false}})
os.RemoveAll(cr.ArvMountPoint)
cr.Container.Mounts["/tmp"] = arvados.Mount{Kind: "tmp"}
cr.Container.OutputPath = "/tmp"
cr.Container.RuntimeConstraints.API = true
+ cr.Container.OutputStorageClasses = []string{"default"}
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "default", "--crunchstat-interval=5",
"--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{"/tmp": {realTemp + "/tmp2", false}, "/etc/arvados/ca-certificates.crt": {stubCertPath, true}})
os.RemoveAll(cr.ArvMountPoint)
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "default", "--crunchstat-interval=5",
"--mount-tmp", "tmp0", "--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{"/keeptmp": {realTemp + "/keep1/tmp0", false}})
os.RemoveAll(cr.ArvMountPoint)
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "default", "--crunchstat-interval=5",
"--mount-tmp", "tmp0", "--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{
"/keepinp": {realTemp + "/keep1/by_id/59389a8f9ee9d399be35462a0f92541c+53", true},
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "default", "--crunchstat-interval=5",
"--file-cache", "512", "--mount-tmp", "tmp0", "--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{
"/keepinp": {realTemp + "/keep1/by_id/59389a8f9ee9d399be35462a0f92541c+53", true},
bindmounts, err := cr.SetupMounts()
c.Check(err, IsNil)
c.Check(am.Cmd, DeepEquals, []string{"--foreground", "--allow-other",
- "--read-write", "--crunchstat-interval=5",
+ "--read-write", "--storage-classes", "default", "--crunchstat-interval=5",
"--file-cache", "512", "--mount-tmp", "tmp0", "--mount-by-pdh", "by_id", realTemp + "/keep1"})
c.Check(bindmounts, DeepEquals, map[string]bindmount{
"/tmp": {realTemp + "/tmp2", false},
MaxSessions int
}
+type UploadDownloadPermission struct {
+ Upload bool
+ Download bool
+}
+
+type UploadDownloadRolePermissions struct {
+ User UploadDownloadPermission
+ Admin UploadDownloadPermission
+}
+
type Cluster struct {
ClusterID string `json:"-"`
ManagementToken string
BalanceTimeout Duration
WebDAVCache WebDAVCacheConfig
+
+ KeepproxyPermission UploadDownloadRolePermissions
+ WebDAVPermission UploadDownloadRolePermissions
+ WebDAVLogEvents bool
}
Git struct {
GitCommand string
FinishedAt *time.Time `json:"finished_at"` // nil if not yet finished
GatewayAddress string `json:"gateway_address"`
InteractiveSessionStarted bool `json:"interactive_session_started"`
+ OutputStorageClasses []string `json:"output_storage_classes"`
}
// ContainerRequest is an arvados#container_request resource.
ExpiresAt time.Time `json:"expires_at"`
Filters []Filter `json:"filters"`
ContainerCount int `json:"container_count"`
+ OutputStorageClasses []string `json:"output_storage_classes"`
}
// Mount is special behavior to attach to a filesystem path or device.
Sync() error
}
-func debugPanicIfNotLocked(l sync.Locker) {
+func debugPanicIfNotLocked(l sync.Locker, writing bool) {
if !DebugLocksPanicMode {
return
}
race := false
- go func() {
- l.Lock()
- race = true
- l.Unlock()
- }()
- time.Sleep(10)
+ if rl, ok := l.(interface {
+ RLock()
+ RUnlock()
+ }); ok && writing {
+ go func() {
+ // Fail if we can grab the read lock during an
+ // operation that purportedly has write lock.
+ rl.RLock()
+ race = true
+ rl.RUnlock()
+ }()
+ } else {
+ go func() {
+ l.Lock()
+ race = true
+ l.Unlock()
+ }()
+ }
+ time.Sleep(100)
if race {
panic("bug: caller-must-have-lock func called, but nobody has lock")
}
}
func (n *treenode) Child(name string, replace func(inode) (inode, error)) (child inode, err error) {
- debugPanicIfNotLocked(n)
+ debugPanicIfNotLocked(n, false)
child = n.inodes[name]
if name == "" || name == "." || name == ".." {
err = ErrInvalidArgument
return
}
if newchild == nil {
+ debugPanicIfNotLocked(n, true)
delete(n.inodes, name)
} else if newchild != child {
+ debugPanicIfNotLocked(n, true)
n.inodes[name] = newchild
n.fileinfo.modTime = time.Now()
child = newchild
func (n *treenode) MemorySize() (size int64) {
n.RLock()
defer n.RUnlock()
- debugPanicIfNotLocked(n)
+ debugPanicIfNotLocked(n, false)
for _, inode := range n.inodes {
size += inode.MemorySize()
}
}
}
createMode := flag&os.O_CREATE != 0
- if createMode {
- parent.Lock()
- defer parent.Unlock()
- } else {
- parent.RLock()
- defer parent.RUnlock()
- }
+ // We always need to take Lock() here, not just RLock(). Even
+ // if we know we won't be creating a file, parent might be a
+ // lookupnode, which sometimes populates its inodes map during
+ // a Child() call.
+ parent.Lock()
+ defer parent.Unlock()
n, err := parent.Child(name, nil)
if err != nil {
return nil, err
if err != nil {
return nil, err
}
+ coll.UUID = dn.fs.uuid
data, err := json.Marshal(&coll)
if err == nil {
data = append(data, '\n')
ActiveToken = "3kg6k6lzmp9kj5cpkcoxie963cmvjahbt2fod9zru30k1jqdmi"
ActiveTokenUUID = "zzzzz-gj3su-077z32aux8dg2s1"
ActiveTokenV2 = "v2/zzzzz-gj3su-077z32aux8dg2s1/3kg6k6lzmp9kj5cpkcoxie963cmvjahbt2fod9zru30k1jqdmi"
+ AdminUserUUID = "zzzzz-tpzed-d9tiejq69daie8f"
AdminToken = "4axaw8zxe0qm22wa6urpp5nskcne8z88cvbupv653y1njyi05h"
AdminTokenUUID = "zzzzz-gj3su-027z32aux8dg2s1"
AnonymousToken = "4kg6k6lzmp9kj4cpkcoxie964cmvjahbt4fod9zru44k4jqdmi"
UserAgreementPDH = "b519d9cb706a29fc7ea24dbea2f05851+93"
HelloWorldPdh = "55713e6a34081eb03609e7ad5fcad129+62"
+ MultilevelCollection1 = "zzzzz-4zz18-pyw8yp9g3pr7irn"
+
AProjectUUID = "zzzzz-j7d0g-v955i6s2oi1cbso"
ASubprojectUUID = "zzzzz-j7d0g-axqo7eu9pwvna1x"
kc.cache().Clear()
}
+func (kc *KeepClient) SetStorageClasses(sc []string) {
+ // make a copy so the caller can't mess with it.
+ kc.StorageClasses = append([]string{}, sc...)
+}
+
var (
// There are four global http.Client objects for the four
// possible permutations of TLS behavior (verify/skip-verify)
attribute :secret_mounts, :jsonbHash, default: {}
attribute :runtime_status, :jsonbHash, default: {}
attribute :runtime_auth_scopes, :jsonbHash, default: {}
+ attribute :output_storage_classes, :jsonbArray, default: ["default"]
serialize :environment, Hash
serialize :mounts, Hash
t.add :lock_count
t.add :gateway_address
t.add :interactive_session_started
+ t.add :output_storage_classes
end
# Supported states for a container
end
def self.full_text_searchable_columns
- super - ["secret_mounts", "secret_mounts_md5", "runtime_token", "gateway_address"]
+ super - ["secret_mounts", "secret_mounts_md5", "runtime_token", "gateway_address", "output_storage_classes"]
end
def self.searchable_columns *args
- super - ["secret_mounts_md5", "runtime_token", "gateway_address"]
+ super - ["secret_mounts_md5", "runtime_token", "gateway_address", "output_storage_classes"]
end
def logged_attributes
secret_mounts: req.secret_mounts,
runtime_token: req.runtime_token,
runtime_user_uuid: runtime_user.uuid,
- runtime_auth_scopes: runtime_auth_scopes
+ runtime_auth_scopes: runtime_auth_scopes,
+ output_storage_classes: req.output_storage_classes,
}
end
act_as_system_user do
:environment, :mounts, :output_path, :priority,
:runtime_constraints, :scheduling_parameters,
:secret_mounts, :runtime_token,
- :runtime_user_uuid, :runtime_auth_scopes)
+ :runtime_user_uuid, :runtime_auth_scopes,
+ :output_storage_classes)
end
case self.state
# already know how to properly treat them.
attribute :properties, :jsonbHash, default: {}
attribute :secret_mounts, :jsonbHash, default: {}
+ attribute :output_storage_classes, :jsonbArray, default: ["default"]
serialize :environment, Hash
serialize :mounts, Hash
t.add :scheduling_parameters
t.add :state
t.add :use_existing
+ t.add :output_storage_classes
end
# Supported states for a container request
:container_image, :cwd, :environment, :filters, :mounts,
:output_path, :priority, :runtime_token,
:runtime_constraints, :state, :container_uuid, :use_existing,
- :scheduling_parameters, :secret_mounts, :output_name, :output_ttl]
+ :scheduling_parameters, :secret_mounts, :output_name, :output_ttl,
+ :output_storage_classes]
def self.limit_index_columns_read
["mounts"]
'container_uuid' => container_uuid,
},
portable_data_hash: log_col.portable_data_hash,
- manifest_text: log_col.manifest_text)
+ manifest_text: log_col.manifest_text,
+ storage_classes_desired: self.output_storage_classes
+ )
completed_coll.save_with_unique_name!
end
end
owner_uuid: self.owner_uuid,
name: coll_name,
manifest_text: "",
+ storage_classes_desired: self.output_storage_classes,
properties: {
'type' => out_type,
'container_request' => uuid,
end
def self.full_text_searchable_columns
- super - ["mounts", "secret_mounts", "secret_mounts_md5", "runtime_token"]
+ super - ["mounts", "secret_mounts", "secret_mounts_md5", "runtime_token", "output_storage_classes"]
end
protected
log_coll = Collection.new(
owner_uuid: self.owner_uuid,
name: coll_name = "Container log for request #{uuid}",
- manifest_text: "")
+ manifest_text: "",
+ storage_classes_desired: self.output_storage_classes)
end
# copy logs from old container into CR's log collection
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+class AddContainerOutputStorageClass < ActiveRecord::Migration[5.2]
+ def change
+ add_column :container_requests, :output_storage_classes, :jsonb, :default => ["default"]
+ add_column :containers, :output_storage_classes, :jsonb, :default => ["default"]
+ end
+end
SET default_with_oids = false;
+--
+-- Name: groups; Type: TABLE; Schema: public; Owner: -
+--
+
+CREATE TABLE public.groups (
+ id integer NOT NULL,
+ uuid character varying(255),
+ owner_uuid character varying(255),
+ created_at timestamp without time zone NOT NULL,
+ modified_by_client_uuid character varying(255),
+ modified_by_user_uuid character varying(255),
+ modified_at timestamp without time zone,
+ name character varying(255) NOT NULL,
+ description character varying(524288),
+ updated_at timestamp without time zone NOT NULL,
+ group_class character varying(255),
+ trash_at timestamp without time zone,
+ is_trashed boolean DEFAULT false NOT NULL,
+ delete_at timestamp without time zone,
+ properties jsonb DEFAULT '{}'::jsonb
+);
+
+
--
-- Name: api_client_authorizations; Type: TABLE; Schema: public; Owner: -
--
output_name character varying(255) DEFAULT NULL::character varying,
output_ttl integer DEFAULT 0 NOT NULL,
secret_mounts jsonb DEFAULT '{}'::jsonb,
- runtime_token text
+ runtime_token text,
+ output_storage_classes jsonb DEFAULT '["default"]'::jsonb
);
runtime_token text,
lock_count integer DEFAULT 0 NOT NULL,
gateway_address character varying,
- interactive_session_started boolean DEFAULT false NOT NULL
+ interactive_session_started boolean DEFAULT false NOT NULL,
+ output_storage_classes jsonb DEFAULT '["default"]'::jsonb
);
ALTER SEQUENCE public.containers_id_seq OWNED BY public.containers.id;
---
--- Name: groups; Type: TABLE; Schema: public; Owner: -
---
-
-CREATE TABLE public.groups (
- id integer NOT NULL,
- uuid character varying(255),
- owner_uuid character varying(255),
- created_at timestamp without time zone NOT NULL,
- modified_by_client_uuid character varying(255),
- modified_by_user_uuid character varying(255),
- modified_at timestamp without time zone,
- name character varying(255) NOT NULL,
- description character varying(524288),
- updated_at timestamp without time zone NOT NULL,
- group_class character varying(255),
- trash_at timestamp without time zone,
- is_trashed boolean DEFAULT false NOT NULL,
- delete_at timestamp without time zone,
- properties jsonb DEFAULT '{}'::jsonb
-);
-
-
--
-- Name: groups_id_seq; Type: SEQUENCE; Schema: public; Owner: -
--
('20201105190435'),
('20201202174753'),
('20210108033940'),
-('20210126183521');
+('20210126183521'),
+('20210621204455');
script: hash
repository: active/foo
script_version: 7def43a4d3f20789dda4700f703b5514cc3ed250
- supplied_script_version: master
+ supplied_script_version: main
script_parameters:
input: 1f4b0bc7583c2a7f9102c395f4ffc5e3+45
created_at: <%= 3.minute.ago.to_s(:db) %>
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters: {}
has_component_with_empty_script_parameters:
components:
foo:
script: foo
- script_version: master
+ script_version: main
has_component_with_completed_jobs:
# Test that the job "started_at" and "finished_at" fields are parsed
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters: {}
job:
uuid: zzzzz-8i9sb-rft1xdewxkwgxnz
- script_version: master
+ script_version: main
created_at: <%= 10.minute.ago.to_s(:db) %>
started_at: <%= 10.minute.ago.to_s(:db) %>
finished_at: <%= 9.minute.ago.to_s(:db) %>
done: 1
bar:
script: bar
- script_version: master
+ script_version: main
script_parameters: {}
job:
uuid: zzzzz-8i9sb-r2dtbzr6bfread7
- script_version: master
+ script_version: main
created_at: <%= 9.minute.ago.to_s(:db) %>
started_at: <%= 9.minute.ago.to_s(:db) %>
state: Running
done: 3
baz:
script: baz
- script_version: master
+ script_version: main
script_parameters: {}
job:
uuid: zzzzz-8i9sb-c7408rni11o7r6s
- script_version: master
+ script_version: main
created_at: <%= 9.minute.ago.to_s(:db) %>
state: Queued
tasks_summary: {}
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters: {}
job: {
uuid: zzzzz-8i9sb-pshmckwoma9plh7,
- script_version: master
+ script_version: main
}
components_is_jobspec:
state: RunningOnServer
components:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
uuid: zzzzz-8i9sb-jyq01m7in1jlofj
repository: active/foo
script: foo
- script_version: master
+ script_version: main
script_parameters:
input: zzzzz-4zz18-4en62shvi99lxd4
log: zzzzz-4zz18-4en62shvi99lxd4
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
uuid: zzzzz-8i9sb-aceg2bnq7jt7kon
repository: active/foo
script: foo
- script_version: master
+ script_version: main
script_parameters:
input: zzzzz-4zz18-bv31uwvy3neko21
log: zzzzz-4zz18-bv31uwvy3neko21
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters: {}
job:
uuid: zzzzz-8i9sb-pshmckwoma9plh7
- script_version: master
+ script_version: main
running_pipeline_with_complete_job:
uuid: zzzzz-d1hrv-partdonepipelin
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters: {}
job:
uuid: zzzzz-8i9sb-job1atlevel3noc
- script_version: master
+ script_version: main
created_at: <%= 12.hour.ago.to_s(:db) %>
started_at: <%= 12.hour.ago.to_s(:db) %>
state: Running
done: 1
bar:
script: bar
- script_version: master
+ script_version: main
script_parameters: {}
job:
uuid: zzzzz-8i9sb-job2atlevel3noc
- script_version: master
+ script_version: main
created_at: <%= 12.hour.ago.to_s(:db) %>
started_at: <%= 12.hour.ago.to_s(:db) %>
state: Running
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
part-one:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
title: "Foo/bar pair"
part-two:
script: bar
- script_version: master
+ script_version: main
script_parameters:
input:
output_of: part-one
name: Pipeline Template with Jobspec Components
components:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
with-search:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
title: foo template input
bar:
script: bar
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo_component:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
part-one:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
description: "Provide an input file"
part-two:
script: bar
- script_version: master
+ script_version: main
script_parameters:
input:
output_of: part-one
components:
work:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
foo_component:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
name: Template to test owner uuid and name unique key violation upon removal
components:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
name: Template to test owner uuid and name unique key violation upon removal
components:
script: foo
- script_version: master
+ script_version: main
script_parameters:
input:
required: true
components:
part-one:
script: foo
- script_version: master
+ script_version: main
script_parameters:
ex_string:
required: true
ex_string_def:
required: true
dataclass: string
- default: hello-testing-123
\ No newline at end of file
+ default: hello-testing-123
BASE_FILTERS = {
'repository' => ['=', 'active/foo'],
'script' => ['=', 'hash'],
- 'script_version' => ['in git', 'master'],
+ 'script_version' => ['in git', 'main'],
'docker_image_locator' => ['=', nil],
'arvados_sdk_version' => ['=', nil],
}
require 'tmpdir'
# Commit log for "foo" repository in test.git.tar
-# master is the main branch
-# b1 is a branch off of master
+# main is the main branch
+# b1 is a branch off of main
# tag1 is a tag
#
# 1de84a8 * b1
-# 077ba2a * master
+# 077ba2a * main
# 4fe459a * tag1
# 31ce37f * foo
test 'find_commit_range does not bypass permissions' do
authorize_with :inactive
assert_raises ArgumentError do
- CommitsHelper::find_commit_range 'foo', nil, 'master', []
+ CommitsHelper::find_commit_range 'foo', nil, 'main', []
end
end
fake_gitdir = repositories(:foo).server_path
CommitsHelper::expects(:cache_dir_for).once.with(url).returns fake_gitdir
CommitsHelper::expects(:fetch_remote_repository).once.with(fake_gitdir, url).returns true
- c = CommitsHelper::find_commit_range url, nil, 'master', []
+ c = CommitsHelper::find_commit_range url, nil, 'main', []
refute_empty c
end
end
test "find_commit_range skips fetch_remote_repository for #{url}" do
CommitsHelper::expects(:fetch_remote_repository).never
assert_raises ArgumentError do
- CommitsHelper::find_commit_range url, nil, 'master', []
+ CommitsHelper::find_commit_range url, nil, 'main', []
end
end
end
test 'fetch_remote_repository does not leak commits across repositories' do
url = "http://localhost:1/fake/fake.git"
fetch_remote_from_local_repo url, :foo
- c = CommitsHelper::find_commit_range url, nil, 'master', []
+ c = CommitsHelper::find_commit_range url, nil, 'main', []
assert_equal ['077ba2ad3ea24a929091a9e6ce545c93199b8e57'], c
url = "http://localhost:2/fake/fake.git"
def with_foo_repository
Dir.chdir("#{Rails.configuration.Git.Repositories}/#{repositories(:foo).uuid}") do
- must_pipe("git checkout master 2>&1")
+ must_pipe("git checkout main 2>&1")
yield
end
end
assert_equal ['31ce37fe365b3dc204300a3e4c396ad333ed0556'], a
#test "test_branch1" do
- a = CommitsHelper::find_commit_range('active/foo', nil, 'master', nil)
+ a = CommitsHelper::find_commit_range('active/foo', nil, 'main', nil)
assert_includes(a, '077ba2ad3ea24a929091a9e6ce545c93199b8e57')
#test "test_branch2" do
#test "test_tag" do
# complains "fatal: ambiguous argument 'tag1': unknown revision or path
# not in the working tree."
- a = CommitsHelper::find_commit_range('active/foo', 'tag1', 'master', nil)
+ a = CommitsHelper::find_commit_range('active/foo', 'tag1', 'main', nil)
assert_equal ['077ba2ad3ea24a929091a9e6ce545c93199b8e57', '4fe459abe02d9b365932b8f5dc419439ab4e2577'], a
#test "test_multi_revision_exclude" do
cr.save!
end
end
+
+ test "default output_storage_classes" do
+ act_as_user users(:active) do
+ cr = create_minimal_req!(priority: 1,
+ state: ContainerRequest::Committed,
+ output_name: 'foo')
+ run_container(cr)
+ cr.reload
+ output = Collection.find_by_uuid(cr.output_uuid)
+ assert_equal ["default"], output.storage_classes_desired
+ end
+ end
+
+ test "setting output_storage_classes" do
+ act_as_user users(:active) do
+ cr = create_minimal_req!(priority: 1,
+ state: ContainerRequest::Committed,
+ output_name: 'foo',
+ output_storage_classes: ["foo_storage_class", "bar_storage_class"])
+ run_container(cr)
+ cr.reload
+ output = Collection.find_by_uuid(cr.output_uuid)
+ assert_equal ["foo_storage_class", "bar_storage_class"], output.storage_classes_desired
+ log = Collection.find_by_uuid(cr.log_uuid)
+ assert_equal ["foo_storage_class", "bar_storage_class"], log.storage_classes_desired
+ end
+ end
+
+ test "reusing container with different container_request.output_storage_classes" do
+ common_attrs = {cwd: "test",
+ priority: 1,
+ command: ["echo", "hello"],
+ output_path: "test",
+ runtime_constraints: {"vcpus" => 4,
+ "ram" => 12000000000},
+ mounts: {"test" => {"kind" => "json"}},
+ environment: {"var" => "value1"},
+ output_storage_classes: ["foo_storage_class"]}
+ set_user_from_auth :active
+ cr1 = create_minimal_req!(common_attrs.merge({state: ContainerRequest::Committed}))
+ cont1 = run_container(cr1)
+ cr1.reload
+
+ output1 = Collection.find_by_uuid(cr1.output_uuid)
+
+ # Testing with use_existing default value
+ cr2 = create_minimal_req!(common_attrs.merge({state: ContainerRequest::Uncommitted,
+ output_storage_classes: ["bar_storage_class"]}))
+
+ assert_not_nil cr1.container_uuid
+ assert_nil cr2.container_uuid
+
+ # Update cr2 to commited state, check for reuse, then run it
+ cr2.update_attributes!({state: ContainerRequest::Committed})
+ assert_equal cr1.container_uuid, cr2.container_uuid
+
+ cr2.reload
+ output2 = Collection.find_by_uuid(cr2.output_uuid)
+
+ # the original CR output has the original storage class,
+ # but the second CR output has the new storage class.
+ assert_equal ["foo_storage_class"], cont1.output_storage_classes
+ assert_equal ["foo_storage_class"], output1.storage_classes_desired
+ assert_equal ["bar_storage_class"], output2.storage_classes_desired
+ end
end
# Default (valid) set of attributes, with given overrides
{
script: "hash",
- script_version: "master",
+ script_version: "main",
repository: "active/foo",
}.merge(merge_me)
end
}
func (s *GitoliteSuite) TestFetch(c *check.C) {
- err := s.RunGit(c, activeToken, "fetch", "active/foo.git")
+ err := s.RunGit(c, activeToken, "fetch", "active/foo.git", "refs/heads/main")
c.Check(err, check.Equals, nil)
}
}
func (s *GitoliteSuite) TestPush(c *check.C) {
- err := s.RunGit(c, activeToken, "push", "active/foo.git", "master:gitolite-push")
+ err := s.RunGit(c, activeToken, "push", "active/foo.git", "main:gitolite-push")
c.Check(err, check.Equals, nil)
// Check that the commit hash appears in the gitolite log, as
}
func (s *GitoliteSuite) TestPushUnwritable(c *check.C) {
- err := s.RunGit(c, spectatorToken, "push", "active/foo.git", "master:gitolite-push-fail")
+ err := s.RunGit(c, spectatorToken, "push", "active/foo.git", "main:gitolite-push-fail")
c.Check(err, check.ErrorMatches, `.*HTTP (code = )?403.*`)
}
c.Assert(err, check.Equals, nil)
_, err = exec.Command("git", "init", "--bare", s.tmpRepoRoot+"/zzzzz-s0uqq-382brsig8rp3666.git").Output()
c.Assert(err, check.Equals, nil)
+ // we need git 2.28 to specify the initial branch with -b; Buster only has 2.20; so we do it in 2 steps
_, err = exec.Command("git", "init", s.tmpWorkdir).Output()
c.Assert(err, check.Equals, nil)
+ _, err = exec.Command("sh", "-c", "cd "+s.tmpWorkdir+" && git checkout -b main").Output()
+ c.Assert(err, check.Equals, nil)
_, err = exec.Command("sh", "-c", "cd "+s.tmpWorkdir+" && echo initial >initial && git add initial && git -c user.name=Initial -c user.email=Initial commit -am 'foo: initial commit'").CombinedOutput()
c.Assert(err, check.Equals, nil)
- _, err = exec.Command("sh", "-c", "cd "+s.tmpWorkdir+" && git push "+s.tmpRepoRoot+"/zzzzz-s0uqq-382brsig8rp3666.git master:master").CombinedOutput()
+ _, err = exec.Command("sh", "-c", "cd "+s.tmpWorkdir+" && git push "+s.tmpRepoRoot+"/zzzzz-s0uqq-382brsig8rp3666.git main:main").CombinedOutput()
c.Assert(err, check.Equals, nil)
_, err = exec.Command("sh", "-c", "cd "+s.tmpWorkdir+" && echo work >work && git add work && git -c user.name=Foo -c user.email=Foo commit -am 'workdir: test'").CombinedOutput()
c.Assert(err, check.Equals, nil)
func (s *GitSuite) TestPathVariants(c *check.C) {
s.makeArvadosRepo(c)
for _, repo := range []string{"active/foo.git", "active/foo/.git", "arvados.git", "arvados/.git"} {
- err := s.RunGit(c, spectatorToken, "fetch", repo)
+ err := s.RunGit(c, spectatorToken, "fetch", repo, "refs/heads/main")
c.Assert(err, check.Equals, nil)
}
}
func (s *GitSuite) TestReadonly(c *check.C) {
- err := s.RunGit(c, spectatorToken, "fetch", "active/foo.git")
+ err := s.RunGit(c, spectatorToken, "fetch", "active/foo.git", "refs/heads/main")
c.Assert(err, check.Equals, nil)
- err = s.RunGit(c, spectatorToken, "push", "active/foo.git", "master:newbranchfail")
+ err = s.RunGit(c, spectatorToken, "push", "active/foo.git", "main:newbranchfail")
c.Assert(err, check.ErrorMatches, `.*HTTP (code = )?403.*`)
_, err = os.Stat(s.tmpRepoRoot + "/zzzzz-s0uqq-382brsig8rp3666.git/refs/heads/newbranchfail")
c.Assert(err, check.FitsTypeOf, &os.PathError{})
}
func (s *GitSuite) TestReadwrite(c *check.C) {
- err := s.RunGit(c, activeToken, "fetch", "active/foo.git")
+ err := s.RunGit(c, activeToken, "fetch", "active/foo.git", "refs/heads/main")
c.Assert(err, check.Equals, nil)
- err = s.RunGit(c, activeToken, "push", "active/foo.git", "master:newbranch")
+ err = s.RunGit(c, activeToken, "push", "active/foo.git", "main:newbranch")
c.Assert(err, check.Equals, nil)
_, err = os.Stat(s.tmpRepoRoot + "/zzzzz-s0uqq-382brsig8rp3666.git/refs/heads/newbranch")
c.Assert(err, check.Equals, nil)
}
func (s *GitSuite) TestNonexistent(c *check.C) {
- err := s.RunGit(c, spectatorToken, "fetch", "thisrepodoesnotexist.git")
+ err := s.RunGit(c, spectatorToken, "fetch", "thisrepodoesnotexist.git", "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.* not found.*`)
}
func (s *GitSuite) TestMissingGitdirReadableRepository(c *check.C) {
- err := s.RunGit(c, activeToken, "fetch", "active/foo2.git")
+ err := s.RunGit(c, activeToken, "fetch", "active/foo2.git", "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.* not found.*`)
}
func (s *GitSuite) TestNoPermission(c *check.C) {
for _, repo := range []string{"active/foo.git", "active/foo/.git"} {
- err := s.RunGit(c, anonymousToken, "fetch", repo)
+ err := s.RunGit(c, anonymousToken, "fetch", repo, "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.* not found.*`)
}
}
func (s *GitSuite) TestExpiredToken(c *check.C) {
for _, repo := range []string{"active/foo.git", "active/foo/.git"} {
- err := s.RunGit(c, expiredToken, "fetch", repo)
+ err := s.RunGit(c, expiredToken, "fetch", repo, "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.* (500 while accessing|requested URL returned error: 500).*`)
}
}
func (s *GitSuite) TestInvalidToken(c *check.C) {
for _, repo := range []string{"active/foo.git", "active/foo/.git"} {
- err := s.RunGit(c, "s3cr3tp@ssw0rd", "fetch", repo)
+ err := s.RunGit(c, "s3cr3tp@ssw0rd", "fetch", repo, "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.* requested URL returned error.*`)
}
}
func (s *GitSuite) TestShortToken(c *check.C) {
for _, repo := range []string{"active/foo.git", "active/foo/.git"} {
- err := s.RunGit(c, "s3cr3t", "fetch", repo)
+ err := s.RunGit(c, "s3cr3t", "fetch", repo, "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.* (500 while accessing|requested URL returned error: 500).*`)
}
}
func (s *GitSuite) TestShortTokenBadReq(c *check.C) {
for _, repo := range []string{"bogus"} {
- err := s.RunGit(c, "s3cr3t", "fetch", repo)
+ err := s.RunGit(c, "s3cr3t", "fetch", repo, "refs/heads/main")
c.Assert(err, check.ErrorMatches, `.*not found.*`)
}
}
msg, err := exec.Command("git", "init", "--bare", s.tmpRepoRoot+"/zzzzz-s0uqq-arvadosrepo0123.git").CombinedOutput()
c.Log(string(msg))
c.Assert(err, check.Equals, nil)
- msg, err = exec.Command("git", "--git-dir", s.tmpRepoRoot+"/zzzzz-s0uqq-arvadosrepo0123.git", "fetch", "../../.git", "HEAD:master").CombinedOutput()
+ msg, err = exec.Command("git", "--git-dir", s.tmpRepoRoot+"/zzzzz-s0uqq-arvadosrepo0123.git", "fetch", "../../.git", "HEAD:main").CombinedOutput()
c.Log(string(msg))
c.Assert(err, check.Equals, nil)
}
}
type cachedSession struct {
- expire time.Time
- fs atomic.Value
+ expire time.Time
+ fs atomic.Value
+ client *arvados.Client
+ arvadosclient *arvadosclient.ArvadosClient
+ keepclient *keepclient.KeepClient
+ user atomic.Value
}
func (c *cache) setup() {
// Get a long-lived CustomFileSystem suitable for doing a read operation
// with the given token.
-func (c *cache) GetSession(token string) (arvados.CustomFileSystem, error) {
+func (c *cache) GetSession(token string) (arvados.CustomFileSystem, *cachedSession, error) {
c.setupOnce.Do(c.setup)
now := time.Now()
ent, _ := c.sessions.Get(token)
sess = &cachedSession{
expire: now.Add(c.config.TTL.Duration()),
}
+ var err error
+ sess.client, err = arvados.NewClientFromConfig(c.cluster)
+ if err != nil {
+ return nil, nil, err
+ }
+ sess.client.AuthToken = token
+ sess.arvadosclient, err = arvadosclient.New(sess.client)
+ if err != nil {
+ return nil, nil, err
+ }
+ sess.keepclient = keepclient.New(sess.arvadosclient)
c.sessions.Add(token, sess)
} else if sess.expire.Before(now) {
c.metrics.sessionMisses.Inc()
go c.pruneSessions()
fs, _ := sess.fs.Load().(arvados.CustomFileSystem)
if fs != nil && !expired {
- return fs, nil
+ return fs, sess, nil
}
- ac, err := arvados.NewClientFromConfig(c.cluster)
- if err != nil {
- return nil, err
- }
- ac.AuthToken = token
- arv, err := arvadosclient.New(ac)
- if err != nil {
- return nil, err
- }
- kc := keepclient.New(arv)
- fs = ac.SiteFileSystem(kc)
+ fs = sess.client.SiteFileSystem(sess.keepclient)
fs.ForwardSlashNameSubstitution(c.cluster.Collections.ForwardSlashNameSubstitution)
sess.fs.Store(fs)
- return fs, nil
+ return fs, sess, nil
}
// Remove all expired session cache entries, then remove more entries
c.metrics.collectionHits.Inc()
return ent.collection
}
+
+func (c *cache) GetTokenUser(token string) (*arvados.User, error) {
+ // Get and cache user record associated with this
+ // token. We need to know their UUID for logging, and
+ // whether they are an admin or not for certain
+ // permission checks.
+
+ // Get/create session entry
+ _, sess, err := c.GetSession(token)
+ if err != nil {
+ return nil, err
+ }
+
+ // See if the user is already set, and if so, return it
+ user, _ := sess.user.Load().(*arvados.User)
+ if user != nil {
+ return user, nil
+ }
+
+ // Fetch the user record
+ c.metrics.apiCalls.Inc()
+ var current arvados.User
+
+ err = sess.client.RequestAndDecode(¤t, "GET", "/arvados/v1/users/current", nil, nil)
+ if err != nil {
+ return nil, err
+ }
+
+ // Stash the user record for next time
+ sess.user.Store(¤t)
+ return ¤t, nil
+}
defer h.clientPool.Put(arv)
var collection *arvados.Collection
+ var tokenUser *arvados.User
tokenResult := make(map[string]int)
for _, arv.ApiToken = range tokens {
var err error
return
}
+ // Check configured permission
+ _, sess, err := h.Config.Cache.GetSession(arv.ApiToken)
+ tokenUser, err = h.Config.Cache.GetTokenUser(arv.ApiToken)
+
if webdavMethod[r.Method] {
+ if !h.userPermittedToUploadOrDownload(r.Method, tokenUser) {
+ http.Error(w, "Not permitted", http.StatusForbidden)
+ return
+ }
+ h.logUploadOrDownload(r, sess.arvadosclient, nil, strings.Join(targetPath, "/"), collection, tokenUser)
+
if writeMethod[r.Method] {
// Save the collection only if/when all
// webdav->filesystem operations succeed --
} else if stat.IsDir() {
h.serveDirectory(w, r, collection.Name, fs, openPath, true)
} else {
+ if !h.userPermittedToUploadOrDownload(r.Method, tokenUser) {
+ http.Error(w, "Not permitted", http.StatusForbidden)
+ return
+ }
+ h.logUploadOrDownload(r, sess.arvadosclient, nil, strings.Join(targetPath, "/"), collection, tokenUser)
+
http.ServeContent(w, r, basename, stat.ModTime(), f)
if wrote := int64(w.WroteBodyBytes()); wrote != stat.Size() && w.WroteStatus() == http.StatusOK {
// If we wrote fewer bytes than expected, it's
http.Error(w, errReadOnly.Error(), http.StatusMethodNotAllowed)
return
}
- fs, err := h.Config.Cache.GetSession(tokens[0])
+
+ fs, sess, err := h.Config.Cache.GetSession(tokens[0])
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
return
}
+
+ tokenUser, err := h.Config.Cache.GetTokenUser(tokens[0])
+ if !h.userPermittedToUploadOrDownload(r.Method, tokenUser) {
+ http.Error(w, "Not permitted", http.StatusForbidden)
+ return
+ }
+ h.logUploadOrDownload(r, sess.arvadosclient, fs, r.URL.Path, nil, tokenUser)
+
if r.Method == "GET" {
_, basename := filepath.Split(r.URL.Path)
applyContentDispositionHdr(w, r, basename, attachment)
io.WriteString(w, html.EscapeString(redir))
io.WriteString(w, `">Continue</A>`)
}
+
+func (h *handler) userPermittedToUploadOrDownload(method string, tokenUser *arvados.User) bool {
+ var permitDownload bool
+ var permitUpload bool
+ if tokenUser != nil && tokenUser.IsAdmin {
+ permitUpload = h.Config.cluster.Collections.WebDAVPermission.Admin.Upload
+ permitDownload = h.Config.cluster.Collections.WebDAVPermission.Admin.Download
+ } else {
+ permitUpload = h.Config.cluster.Collections.WebDAVPermission.User.Upload
+ permitDownload = h.Config.cluster.Collections.WebDAVPermission.User.Download
+ }
+ if (method == "PUT" || method == "POST") && !permitUpload {
+ // Disallow operations that upload new files.
+ // Permit webdav operations that move existing files around.
+ return false
+ } else if method == "GET" && !permitDownload {
+ // Disallow downloading file contents.
+ // Permit webdav operations like PROPFIND that retrieve metadata
+ // but not file contents.
+ return false
+ }
+ return true
+}
+
+func (h *handler) logUploadOrDownload(
+ r *http.Request,
+ client *arvadosclient.ArvadosClient,
+ fs arvados.CustomFileSystem,
+ filepath string,
+ collection *arvados.Collection,
+ user *arvados.User) {
+
+ log := ctxlog.FromContext(r.Context())
+ props := make(map[string]string)
+ props["reqPath"] = r.URL.Path
+ var useruuid string
+ if user != nil {
+ log = log.WithField("user_uuid", user.UUID).
+ WithField("user_full_name", user.FullName)
+ useruuid = user.UUID
+ } else {
+ useruuid = fmt.Sprintf("%s-tpzed-anonymouspublic", h.Config.cluster.ClusterID)
+ }
+ if collection == nil && fs != nil {
+ collection, filepath = h.determineCollection(fs, filepath)
+ }
+ if collection != nil {
+ log = log.WithField("collection_uuid", collection.UUID).
+ WithField("collection_file_path", filepath)
+ props["collection_uuid"] = collection.UUID
+ props["collection_file_path"] = filepath
+ }
+ if r.Method == "PUT" || r.Method == "POST" {
+ log.Info("File upload")
+ if h.Config.cluster.Collections.WebDAVLogEvents {
+ go func() {
+ lr := arvadosclient.Dict{"log": arvadosclient.Dict{
+ "object_uuid": useruuid,
+ "event_type": "file_upload",
+ "properties": props}}
+ err := client.Create("logs", lr, nil)
+ if err != nil {
+ log.WithError(err).Error("Failed to create upload log event on API server")
+ }
+ }()
+ }
+ } else if r.Method == "GET" {
+ if collection != nil && collection.PortableDataHash != "" {
+ log = log.WithField("portable_data_hash", collection.PortableDataHash)
+ props["portable_data_hash"] = collection.PortableDataHash
+ }
+ log.Info("File download")
+ if h.Config.cluster.Collections.WebDAVLogEvents {
+ go func() {
+ lr := arvadosclient.Dict{"log": arvadosclient.Dict{
+ "object_uuid": useruuid,
+ "event_type": "file_download",
+ "properties": props}}
+ err := client.Create("logs", lr, nil)
+ if err != nil {
+ log.WithError(err).Error("Failed to create download log event on API server")
+ }
+ }()
+ }
+ }
+}
+
+func (h *handler) determineCollection(fs arvados.CustomFileSystem, path string) (*arvados.Collection, string) {
+ segments := strings.Split(path, "/")
+ var i int
+ for i = 0; i < len(segments); i++ {
+ dir := append([]string{}, segments[0:i]...)
+ dir = append(dir, ".arvados#collection")
+ f, err := fs.OpenFile(strings.Join(dir, "/"), os.O_RDONLY, 0)
+ if f != nil {
+ defer f.Close()
+ }
+ if err != nil {
+ if !os.IsNotExist(err) {
+ return nil, ""
+ }
+ continue
+ }
+ // err is nil so we found it.
+ decoder := json.NewDecoder(f)
+ var collection arvados.Collection
+ err = decoder.Decode(&collection)
+ if err != nil {
+ return nil, ""
+ }
+ return &collection, strings.Join(segments[i:], "/")
+ }
+ return nil, ""
+}
"context"
"fmt"
"html"
+ "io"
"io/ioutil"
"net/http"
"net/http/httptest"
// If we return no content because the client sent an
// If-Modified-Since header, our response should be
- // 304, and we should not emit a log message.
- {true, true, http.StatusNotModified, ``},
+ // 304. We still expect a "File download" log since it
+ // counts as a file access for auditing.
+ {true, true, http.StatusNotModified, `(?ms).*msg="File download".*`},
} {
c.Logf("trial: %+v", trial)
arvadostest.StartKeep(2, true)
}
return hc
}
+
+func (s *IntegrationSuite) checkUploadDownloadRequest(c *check.C, h *handler, req *http.Request,
+ successCode int, direction string, perm bool, userUuid string, collectionUuid string, filepath string) {
+
+ client := s.testServer.Config.Client
+ client.AuthToken = arvadostest.AdminToken
+ var logentries arvados.LogList
+ limit1 := 1
+ err := client.RequestAndDecode(&logentries, "GET", "arvados/v1/logs", nil,
+ arvados.ResourceListParams{
+ Limit: &limit1,
+ Order: "created_at desc"})
+ c.Check(err, check.IsNil)
+ c.Check(logentries.Items, check.HasLen, 1)
+ lastLogId := logentries.Items[0].ID
+ nextLogId := lastLogId
+
+ var logbuf bytes.Buffer
+ logger := logrus.New()
+ logger.Out = &logbuf
+ resp := httptest.NewRecorder()
+ req = req.WithContext(ctxlog.Context(context.Background(), logger))
+ h.ServeHTTP(resp, req)
+
+ if perm {
+ c.Check(resp.Result().StatusCode, check.Equals, successCode)
+ c.Check(logbuf.String(), check.Matches, `(?ms).*msg="File `+direction+`".*`)
+ c.Check(logbuf.String(), check.Not(check.Matches), `(?ms).*level=error.*`)
+
+ count := 0
+ for ; nextLogId == lastLogId && count < 20; count++ {
+ time.Sleep(50 * time.Millisecond)
+ err = client.RequestAndDecode(&logentries, "GET", "arvados/v1/logs", nil,
+ arvados.ResourceListParams{
+ Filters: []arvados.Filter{arvados.Filter{Attr: "event_type", Operator: "=", Operand: "file_" + direction}},
+ Limit: &limit1,
+ Order: "created_at desc",
+ })
+ c.Check(err, check.IsNil)
+ if len(logentries.Items) > 0 {
+ nextLogId = logentries.Items[0].ID
+ }
+ }
+ c.Check(count, check.Not(check.Equals), 20)
+ c.Check(logentries.Items[0].ObjectUUID, check.Equals, userUuid)
+ c.Check(logentries.Items[0].Properties["collection_uuid"], check.Equals, collectionUuid)
+ c.Check(logentries.Items[0].Properties["collection_file_path"], check.Equals, filepath)
+ } else {
+ c.Check(resp.Result().StatusCode, check.Equals, http.StatusForbidden)
+ c.Check(logbuf.String(), check.Equals, "")
+ }
+}
+
+func (s *IntegrationSuite) TestDownloadLoggingPermission(c *check.C) {
+ config := newConfig(s.ArvConfig)
+ h := handler{Config: config}
+ u := mustParseURL("http://" + arvadostest.FooCollection + ".keep-web.example/foo")
+
+ config.cluster.Collections.TrustAllContent = true
+
+ for _, adminperm := range []bool{true, false} {
+ for _, userperm := range []bool{true, false} {
+ config.cluster.Collections.WebDAVPermission.Admin.Download = adminperm
+ config.cluster.Collections.WebDAVPermission.User.Download = userperm
+
+ // Test admin permission
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + arvadostest.AdminToken},
+ },
+ }
+ s.checkUploadDownloadRequest(c, &h, req, http.StatusOK, "download", adminperm,
+ arvadostest.AdminUserUUID, arvadostest.FooCollection, "foo")
+
+ // Test user permission
+ req = &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + arvadostest.ActiveToken},
+ },
+ }
+ s.checkUploadDownloadRequest(c, &h, req, http.StatusOK, "download", userperm,
+ arvadostest.ActiveUserUUID, arvadostest.FooCollection, "foo")
+ }
+ }
+
+ config.cluster.Collections.WebDAVPermission.User.Download = true
+
+ for _, tryurl := range []string{"http://" + arvadostest.MultilevelCollection1 + ".keep-web.example/dir1/subdir/file1",
+ "http://keep-web/users/active/multilevel_collection_1/dir1/subdir/file1"} {
+
+ u = mustParseURL(tryurl)
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + arvadostest.ActiveToken},
+ },
+ }
+ s.checkUploadDownloadRequest(c, &h, req, http.StatusOK, "download", true,
+ arvadostest.ActiveUserUUID, arvadostest.MultilevelCollection1, "dir1/subdir/file1")
+ }
+
+ u = mustParseURL("http://" + strings.Replace(arvadostest.FooCollectionPDH, "+", "-", 1) + ".keep-web.example/foo")
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + arvadostest.ActiveToken},
+ },
+ }
+ s.checkUploadDownloadRequest(c, &h, req, http.StatusOK, "download", true,
+ arvadostest.ActiveUserUUID, arvadostest.FooCollection, "foo")
+}
+
+func (s *IntegrationSuite) TestUploadLoggingPermission(c *check.C) {
+ config := newConfig(s.ArvConfig)
+ h := handler{Config: config}
+
+ for _, adminperm := range []bool{true, false} {
+ for _, userperm := range []bool{true, false} {
+
+ arv := s.testServer.Config.Client
+ arv.AuthToken = arvadostest.ActiveToken
+
+ var coll arvados.Collection
+ err := arv.RequestAndDecode(&coll,
+ "POST",
+ "/arvados/v1/collections",
+ nil,
+ map[string]interface{}{
+ "ensure_unique_name": true,
+ "collection": map[string]interface{}{
+ "name": "test collection",
+ },
+ })
+ c.Assert(err, check.Equals, nil)
+
+ u := mustParseURL("http://" + coll.UUID + ".keep-web.example/bar")
+
+ config.cluster.Collections.WebDAVPermission.Admin.Upload = adminperm
+ config.cluster.Collections.WebDAVPermission.User.Upload = userperm
+
+ // Test admin permission
+ req := &http.Request{
+ Method: "PUT",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + arvadostest.AdminToken},
+ },
+ Body: io.NopCloser(bytes.NewReader([]byte("bar"))),
+ }
+ s.checkUploadDownloadRequest(c, &h, req, http.StatusCreated, "upload", adminperm,
+ arvadostest.AdminUserUUID, coll.UUID, "bar")
+
+ // Test user permission
+ req = &http.Request{
+ Method: "PUT",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + arvadostest.ActiveToken},
+ },
+ Body: io.NopCloser(bytes.NewReader([]byte("bar"))),
+ }
+ s.checkUploadDownloadRequest(c, &h, req, http.StatusCreated, "upload", userperm,
+ arvadostest.ActiveUserUUID, coll.UUID, "bar")
+ }
+ }
+}
"time"
"git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/arvadosclient"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
+ "git.arvados.org/arvados.git/sdk/go/keepclient"
"github.com/AdRoll/goamz/s3"
)
var err error
var fs arvados.CustomFileSystem
+ var arvclient *arvadosclient.ArvadosClient
if r.Method == http.MethodGet || r.Method == http.MethodHead {
// Use a single session (cached FileSystem) across
// multiple read requests.
- fs, err = h.Config.Cache.GetSession(token)
+ var sess *cachedSession
+ fs, sess, err = h.Config.Cache.GetSession(token)
if err != nil {
s3ErrorResponse(w, InternalError, err.Error(), r.URL.Path, http.StatusInternalServerError)
return true
}
+ arvclient = sess.arvadosclient
} else {
// Create a FileSystem for this request, to avoid
// exposing incomplete write operations to concurrent
// requests.
- _, kc, client, release, err := h.getClients(r.Header.Get("X-Request-Id"), token)
+ var kc *keepclient.KeepClient
+ var release func()
+ var client *arvados.Client
+ arvclient, kc, client, release, err = h.getClients(r.Header.Get("X-Request-Id"), token)
if err != nil {
s3ErrorResponse(w, InternalError, err.Error(), r.URL.Path, http.StatusInternalServerError)
return true
s3ErrorResponse(w, NoSuchKey, "The specified key does not exist.", r.URL.Path, http.StatusNotFound)
return true
}
+
+ tokenUser, err := h.Config.Cache.GetTokenUser(token)
+ if !h.userPermittedToUploadOrDownload(r.Method, tokenUser) {
+ http.Error(w, "Not permitted", http.StatusForbidden)
+ return true
+ }
+ h.logUploadOrDownload(r, arvclient, fs, fspath, nil, tokenUser)
+
// shallow copy r, and change URL path
r := *r
r.URL.Path = fspath
return true
}
defer f.Close()
+
+ tokenUser, err := h.Config.Cache.GetTokenUser(token)
+ if !h.userPermittedToUploadOrDownload(r.Method, tokenUser) {
+ http.Error(w, "Not permitted", http.StatusForbidden)
+ return true
+ }
+ h.logUploadOrDownload(r, arvclient, fs, fspath, nil, tokenUser)
+
_, err = io.Copy(f, r.Body)
if err != nil {
err = fmt.Errorf("write to %q failed: %w", r.URL.Path, err)
// IntegrationSuite tests need an API server and a keep-web server
type IntegrationSuite struct {
testServer *server
+ ArvConfig *arvados.Config
}
func (s *IntegrationSuite) TestNoToken(c *check.C) {
c.Check(summaries["request_duration_seconds/get/404"].SampleCount, check.Equals, "1")
c.Check(summaries["time_to_status_seconds/get/404"].SampleCount, check.Equals, "1")
c.Check(counters["arvados_keepweb_collectioncache_requests//"].Value, check.Equals, int64(2))
- c.Check(counters["arvados_keepweb_collectioncache_api_calls//"].Value, check.Equals, int64(1))
+ c.Check(counters["arvados_keepweb_collectioncache_api_calls//"].Value, check.Equals, int64(2))
c.Check(counters["arvados_keepweb_collectioncache_hits//"].Value, check.Equals, int64(1))
c.Check(counters["arvados_keepweb_collectioncache_pdh_hits//"].Value, check.Equals, int64(1))
c.Check(counters["arvados_keepweb_collectioncache_permission_hits//"].Value, check.Equals, int64(1))
cfg.cluster.ManagementToken = arvadostest.ManagementToken
cfg.cluster.SystemRootToken = arvadostest.SystemRootToken
cfg.cluster.Users.AnonymousUserToken = arvadostest.AnonymousToken
+ s.ArvConfig = arvCfg
s.testServer = &server{Config: cfg}
err = s.testServer.Start(ctxlog.TestLogger(c))
c.Assert(err, check.Equals, nil)
"os/signal"
"regexp"
"strings"
- "sync"
"syscall"
"time"
"github.com/coreos/go-systemd/daemon"
"github.com/ghodss/yaml"
"github.com/gorilla/mux"
+ lru "github.com/hashicorp/golang-lru"
log "github.com/sirupsen/logrus"
)
signal.Notify(term, syscall.SIGINT)
// Start serving requests.
- router = MakeRESTRouter(kc, time.Duration(keepclient.DefaultProxyRequestTimeout), cluster.ManagementToken)
+ router, err = MakeRESTRouter(kc, time.Duration(keepclient.DefaultProxyRequestTimeout), cluster, logger)
+ if err != nil {
+ return err
+ }
return http.Serve(listener, httpserver.AddRequestIDs(httpserver.LogRequests(router)))
}
+type TokenCacheEntry struct {
+ expire int64
+ user *arvados.User
+}
+
type APITokenCache struct {
- tokens map[string]int64
- lock sync.Mutex
+ tokens *lru.TwoQueueCache
expireTime int64
}
-// RememberToken caches the token and set an expire time. If we already have
-// an expire time on the token, it is not updated.
-func (cache *APITokenCache) RememberToken(token string) {
- cache.lock.Lock()
- defer cache.lock.Unlock()
-
+// RememberToken caches the token and set an expire time. If the
+// token is already in the cache, it is not updated.
+func (cache *APITokenCache) RememberToken(token string, user *arvados.User) {
now := time.Now().Unix()
- if cache.tokens[token] == 0 {
- cache.tokens[token] = now + cache.expireTime
+ _, ok := cache.tokens.Get(token)
+ if !ok {
+ cache.tokens.Add(token, TokenCacheEntry{
+ expire: now + cache.expireTime,
+ user: user,
+ })
}
}
// RecallToken checks if the cached token is known and still believed to be
// valid.
-func (cache *APITokenCache) RecallToken(token string) bool {
- cache.lock.Lock()
- defer cache.lock.Unlock()
+func (cache *APITokenCache) RecallToken(token string) (bool, *arvados.User) {
+ val, ok := cache.tokens.Get(token)
+ if !ok {
+ return false, nil
+ }
+ cacheEntry := val.(TokenCacheEntry)
now := time.Now().Unix()
- if cache.tokens[token] == 0 {
- // Unknown token
- return false
- } else if now < cache.tokens[token] {
+ if now < cacheEntry.expire {
// Token is known and still valid
- return true
+ return true, cacheEntry.user
} else {
// Token is expired
- cache.tokens[token] = 0
- return false
+ cache.tokens.Remove(token)
+ return false, nil
}
}
return req.RemoteAddr
}
-func CheckAuthorizationHeader(kc *keepclient.KeepClient, cache *APITokenCache, req *http.Request) (pass bool, tok string) {
+func (h *proxyHandler) CheckAuthorizationHeader(req *http.Request) (pass bool, tok string, user *arvados.User) {
parts := strings.SplitN(req.Header.Get("Authorization"), " ", 2)
if len(parts) < 2 || !(parts[0] == "OAuth2" || parts[0] == "Bearer") || len(parts[1]) == 0 {
- return false, ""
+ return false, "", nil
}
tok = parts[1]
op = "write"
}
- if cache.RecallToken(op + ":" + tok) {
+ if ok, user := h.APITokenCache.RecallToken(op + ":" + tok); ok {
// Valid in the cache, short circuit
- return true, tok
+ return true, tok, user
}
var err error
- arv := *kc.Arvados
+ arv := *h.KeepClient.Arvados
arv.ApiToken = tok
arv.RequestID = req.Header.Get("X-Request-Id")
- if op == "read" {
- err = arv.Call("HEAD", "keep_services", "", "accessible", nil, nil)
- } else {
- err = arv.Call("HEAD", "users", "", "current", nil, nil)
+ user = &arvados.User{}
+ userCurrentError := arv.Call("GET", "users", "", "current", nil, user)
+ err = userCurrentError
+ if err != nil && op == "read" {
+ apiError, ok := err.(arvadosclient.APIServerError)
+ if ok && apiError.HttpStatusCode == http.StatusForbidden {
+ // If it was a scoped "sharing" token it will
+ // return 403 instead of 401 for the current
+ // user check. If it is a download operation
+ // and they have permission to read the
+ // keep_services table, we can allow it.
+ err = arv.Call("HEAD", "keep_services", "", "accessible", nil, nil)
+ }
}
if err != nil {
log.Printf("%s: CheckAuthorizationHeader error: %v", GetRemoteAddress(req), err)
- return false, ""
+ return false, "", nil
+ }
+
+ if userCurrentError == nil && user.IsAdmin {
+ // checking userCurrentError is probably redundant,
+ // IsAdmin would be false anyway. But can't hurt.
+ if op == "read" && !h.cluster.Collections.KeepproxyPermission.Admin.Download {
+ return false, "", nil
+ }
+ if op == "write" && !h.cluster.Collections.KeepproxyPermission.Admin.Upload {
+ return false, "", nil
+ }
+ } else {
+ if op == "read" && !h.cluster.Collections.KeepproxyPermission.User.Download {
+ return false, "", nil
+ }
+ if op == "write" && !h.cluster.Collections.KeepproxyPermission.User.Upload {
+ return false, "", nil
+ }
}
// Success! Update cache
- cache.RememberToken(op + ":" + tok)
+ h.APITokenCache.RememberToken(op+":"+tok, user)
- return true, tok
+ return true, tok, user
}
// We need to make a private copy of the default http transport early
*APITokenCache
timeout time.Duration
transport *http.Transport
+ logger log.FieldLogger
+ cluster *arvados.Cluster
}
// MakeRESTRouter returns an http.Handler that passes GET and PUT
// requests to the appropriate handlers.
-func MakeRESTRouter(kc *keepclient.KeepClient, timeout time.Duration, mgmtToken string) http.Handler {
+func MakeRESTRouter(kc *keepclient.KeepClient, timeout time.Duration, cluster *arvados.Cluster, logger log.FieldLogger) (http.Handler, error) {
rest := mux.NewRouter()
transport := defaultTransport
transport.TLSClientConfig = arvadosclient.MakeTLSConfig(kc.Arvados.ApiInsecure)
transport.TLSHandshakeTimeout = keepclient.DefaultTLSHandshakeTimeout
+ cacheQ, err := lru.New2Q(500)
+ if err != nil {
+ return nil, fmt.Errorf("Error from lru.New2Q: %v", err)
+ }
+
h := &proxyHandler{
Handler: rest,
KeepClient: kc,
timeout: timeout,
transport: &transport,
APITokenCache: &APITokenCache{
- tokens: make(map[string]int64),
+ tokens: cacheQ,
expireTime: 300,
},
+ logger: logger,
+ cluster: cluster,
}
rest.HandleFunc(`/{locator:[0-9a-f]{32}\+.*}`, h.Get).Methods("GET", "HEAD")
rest.HandleFunc(`/`, h.Options).Methods("OPTIONS")
rest.Handle("/_health/{check}", &health.Handler{
- Token: mgmtToken,
+ Token: cluster.ManagementToken,
Prefix: "/_health/",
}).Methods("GET")
rest.NotFoundHandler = InvalidPathHandler{}
- return h
+ return h, nil
}
var errLoopDetected = errors.New("loop detected")
-func (*proxyHandler) checkLoop(resp http.ResponseWriter, req *http.Request) error {
+func (h *proxyHandler) checkLoop(resp http.ResponseWriter, req *http.Request) error {
if via := req.Header.Get("Via"); strings.Index(via, " "+viaAlias) >= 0 {
- log.Printf("proxy loop detected (request has Via: %q): perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?", via)
+ h.logger.Printf("proxy loop detected (request has Via: %q): perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?", via)
http.Error(resp, errLoopDetected.Error(), http.StatusInternalServerError)
return errLoopDetected
}
SetCorsHeaders(resp)
}
-var errBadAuthorizationHeader = errors.New("Missing or invalid Authorization header")
+var errBadAuthorizationHeader = errors.New("Missing or invalid Authorization header, or method not allowed")
var errContentLengthMismatch = errors.New("Actual length != expected content length")
var errMethodNotSupported = errors.New("Method not supported")
var pass bool
var tok string
- if pass, tok = CheckAuthorizationHeader(kc, h.APITokenCache, req); !pass {
+ var user *arvados.User
+ if pass, tok, user = h.CheckAuthorizationHeader(req); !pass {
status, err = http.StatusForbidden, errBadAuthorizationHeader
return
}
locator = removeHint.ReplaceAllString(locator, "$1")
+ if locator != "" {
+ parts := strings.SplitN(locator, "+", 3)
+ if len(parts) >= 2 {
+ logger := h.logger
+ if user != nil {
+ logger = logger.WithField("user_uuid", user.UUID).
+ WithField("user_full_name", user.FullName)
+ }
+ logger.WithField("locator", fmt.Sprintf("%s+%s", parts[0], parts[1])).Infof("Block download")
+ }
+ }
+
switch req.Method {
case "HEAD":
expectLength, proxiedURI, err = kc.Ask(locator)
for _, sc := range strings.Split(req.Header.Get("X-Keep-Storage-Classes"), ",") {
scl = append(scl, strings.Trim(sc, " "))
}
- kc.StorageClasses = scl
+ kc.SetStorageClasses(scl)
}
_, err = fmt.Sscanf(req.Header.Get("Content-Length"), "%d", &expectLength)
var pass bool
var tok string
- if pass, tok = CheckAuthorizationHeader(kc, h.APITokenCache, req); !pass {
+ var user *arvados.User
+ if pass, tok, user = h.CheckAuthorizationHeader(req); !pass {
err = errBadAuthorizationHeader
status = http.StatusForbidden
return
locatorOut, wroteReplicas, err = kc.PutHR(locatorIn, req.Body, expectLength)
}
+ if locatorOut != "" {
+ parts := strings.SplitN(locatorOut, "+", 3)
+ if len(parts) >= 2 {
+ logger := h.logger
+ if user != nil {
+ logger = logger.WithField("user_uuid", user.UUID).
+ WithField("user_full_name", user.FullName)
+ }
+ logger.WithField("locator", fmt.Sprintf("%s+%s", parts[0], parts[1])).Infof("Block upload")
+ }
+ }
+
// Tell the client how many successful PUTs we accomplished
resp.Header().Set(keepclient.XKeepReplicasStored, fmt.Sprintf("%d", wroteReplicas))
}()
kc := h.makeKeepClient(req)
- ok, token := CheckAuthorizationHeader(kc, h.APITokenCache, req)
+ ok, token, _ := h.CheckAuthorizationHeader(req)
if !ok {
status, err = http.StatusForbidden, errBadAuthorizationHeader
return
"git.arvados.org/arvados.git/sdk/go/keepclient"
log "github.com/sirupsen/logrus"
+ "gopkg.in/check.v1"
. "gopkg.in/check.v1"
)
arvadostest.StopAPI()
}
-func runProxy(c *C, bogusClientToken bool, loadKeepstoresFromConfig bool) *keepclient.KeepClient {
+func runProxy(c *C, bogusClientToken bool, loadKeepstoresFromConfig bool, kp *arvados.UploadDownloadRolePermissions) (*keepclient.KeepClient, *bytes.Buffer) {
cfg, err := config.NewLoader(nil, ctxlog.TestLogger(c)).Load()
c.Assert(err, Equals, nil)
cluster, err := cfg.GetCluster("")
cluster.Services.Keepproxy.InternalURLs = map[arvados.URL]arvados.ServiceInstance{{Host: ":0"}: {}}
+ if kp != nil {
+ cluster.Collections.KeepproxyPermission = *kp
+ }
+
listener = nil
+ logbuf := &bytes.Buffer{}
+ logger := log.New()
+ logger.Out = logbuf
go func() {
- run(log.New(), cluster)
+ run(logger, cluster)
defer closeListener()
}()
waitForListener()
kc.SetServiceRoots(sr, sr, sr)
kc.Arvados.External = true
- return kc
+ return kc, logbuf
}
func (s *ServerRequiredSuite) TestResponseViaHeader(c *C) {
- runProxy(c, false, false)
+ runProxy(c, false, false, nil)
defer closeListener()
req, err := http.NewRequest("POST",
}
func (s *ServerRequiredSuite) TestLoopDetection(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
sr := map[string]string{
}
func (s *ServerRequiredSuite) TestStorageClassesHeader(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
// Set up fake keepstore to record request headers
}
func (s *ServerRequiredSuite) TestStorageClassesConfirmedHeader(c *C) {
- runProxy(c, false, false)
+ runProxy(c, false, false, nil)
defer closeListener()
content := []byte("foo")
}
func (s *ServerRequiredSuite) TestDesiredReplicas(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
content := []byte("TestDesiredReplicas")
}
func (s *ServerRequiredSuite) TestPutWrongContentLength(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
content := []byte("TestPutWrongContentLength")
// fixes the invalid Content-Length header. In order to test
// our server behavior, we have to call the handler directly
// using an httptest.ResponseRecorder.
- rtr := MakeRESTRouter(kc, 10*time.Second, "")
+ rtr, err := MakeRESTRouter(kc, 10*time.Second, &arvados.Cluster{}, log.New())
+ c.Assert(err, check.IsNil)
type testcase struct {
sendLength string
}
func (s *ServerRequiredSuite) TestManyFailedPuts(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
router.(*proxyHandler).timeout = time.Nanosecond
}
func (s *ServerRequiredSuite) TestPutAskGet(c *C) {
- kc := runProxy(c, false, false)
+ kc, logbuf := runProxy(c, false, false, nil)
defer closeListener()
hash := fmt.Sprintf("%x", md5.Sum([]byte("foo")))
c.Check(rep, Equals, 2)
c.Check(err, Equals, nil)
c.Log("Finished PutB (expected success)")
+
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block upload" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="TestCase Administrator" user_uuid=zzzzz-tpzed-d9tiejq69daie8f.*`)
+ logbuf.Reset()
}
{
c.Assert(err, Equals, nil)
c.Check(blocklen, Equals, int64(3))
c.Log("Finished Ask (expected success)")
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block download" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="TestCase Administrator" user_uuid=zzzzz-tpzed-d9tiejq69daie8f.*`)
+ logbuf.Reset()
}
{
c.Check(all, DeepEquals, []byte("foo"))
c.Check(blocklen, Equals, int64(3))
c.Log("Finished Get (expected success)")
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block download" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="TestCase Administrator" user_uuid=zzzzz-tpzed-d9tiejq69daie8f.*`)
+ logbuf.Reset()
}
{
}
func (s *ServerRequiredSuite) TestPutAskGetForbidden(c *C) {
- kc := runProxy(c, true, false)
+ kc, _ := runProxy(c, true, false, nil)
defer closeListener()
hash := fmt.Sprintf("%x+3", md5.Sum([]byte("bar")))
blocklen, _, err := kc.Ask(hash)
c.Check(err, FitsTypeOf, &keepclient.ErrNotFound{})
- c.Check(err, ErrorMatches, ".*not found.*")
+ c.Check(err, ErrorMatches, ".*HTTP 403.*")
c.Check(blocklen, Equals, int64(0))
_, blocklen, _, err = kc.Get(hash)
c.Check(err, FitsTypeOf, &keepclient.ErrNotFound{})
- c.Check(err, ErrorMatches, ".*not found.*")
+ c.Check(err, ErrorMatches, ".*HTTP 403.*")
c.Check(blocklen, Equals, int64(0))
+}
+
+func testPermission(c *C, admin bool, perm arvados.UploadDownloadPermission) {
+ kp := arvados.UploadDownloadRolePermissions{}
+ if admin {
+ kp.Admin = perm
+ kp.User = arvados.UploadDownloadPermission{Upload: true, Download: true}
+ } else {
+ kp.Admin = arvados.UploadDownloadPermission{Upload: true, Download: true}
+ kp.User = perm
+ }
+
+ kc, logbuf := runProxy(c, false, false, &kp)
+ defer closeListener()
+ if admin {
+ kc.Arvados.ApiToken = arvadostest.AdminToken
+ } else {
+ kc.Arvados.ApiToken = arvadostest.ActiveToken
+ }
+
+ hash := fmt.Sprintf("%x", md5.Sum([]byte("foo")))
+ var hash2 string
+
+ {
+ var rep int
+ var err error
+ hash2, rep, err = kc.PutB([]byte("foo"))
+ if perm.Upload {
+ c.Check(hash2, Matches, fmt.Sprintf(`^%s\+3(\+.+)?$`, hash))
+ c.Check(rep, Equals, 2)
+ c.Check(err, Equals, nil)
+ c.Log("Finished PutB (expected success)")
+ if admin {
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block upload" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="TestCase Administrator" user_uuid=zzzzz-tpzed-d9tiejq69daie8f.*`)
+ } else {
+
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block upload" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="Active User" user_uuid=zzzzz-tpzed-xurymjxw79nv3jz.*`)
+ }
+ } else {
+ c.Check(hash2, Equals, "")
+ c.Check(rep, Equals, 0)
+ c.Check(err, FitsTypeOf, keepclient.InsufficientReplicasError(errors.New("")))
+ }
+ logbuf.Reset()
+ }
+ if perm.Upload {
+ // can't test download without upload.
+
+ reader, blocklen, _, err := kc.Get(hash2)
+ if perm.Download {
+ c.Assert(err, Equals, nil)
+ all, err := ioutil.ReadAll(reader)
+ c.Check(err, IsNil)
+ c.Check(all, DeepEquals, []byte("foo"))
+ c.Check(blocklen, Equals, int64(3))
+ c.Log("Finished Get (expected success)")
+ if admin {
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block download" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="TestCase Administrator" user_uuid=zzzzz-tpzed-d9tiejq69daie8f.*`)
+ } else {
+ c.Check(logbuf.String(), Matches, `(?ms).*msg="Block download" locator=acbd18db4cc2f85cedef654fccc4a4d8\+3 user_full_name="Active User" user_uuid=zzzzz-tpzed-xurymjxw79nv3jz.*`)
+ }
+ } else {
+ c.Check(err, FitsTypeOf, &keepclient.ErrNotFound{})
+ c.Check(err, ErrorMatches, ".*Missing or invalid Authorization header, or method not allowed.*")
+ c.Check(blocklen, Equals, int64(0))
+ }
+ logbuf.Reset()
+ }
+
+}
+
+func (s *ServerRequiredSuite) TestPutGetPermission(c *C) {
+
+ for _, adminperm := range []bool{true, false} {
+ for _, userperm := range []bool{true, false} {
+
+ testPermission(c, true,
+ arvados.UploadDownloadPermission{
+ Upload: adminperm,
+ Download: true,
+ })
+ testPermission(c, true,
+ arvados.UploadDownloadPermission{
+ Upload: true,
+ Download: adminperm,
+ })
+ testPermission(c, false,
+ arvados.UploadDownloadPermission{
+ Upload: true,
+ Download: userperm,
+ })
+ testPermission(c, false,
+ arvados.UploadDownloadPermission{
+ Upload: true,
+ Download: userperm,
+ })
+ }
+ }
}
func (s *ServerRequiredSuite) TestCorsHeaders(c *C) {
- runProxy(c, false, false)
+ runProxy(c, false, false, nil)
defer closeListener()
{
}
func (s *ServerRequiredSuite) TestPostWithoutHash(c *C) {
- runProxy(c, false, false)
+ runProxy(c, false, false, nil)
defer closeListener()
{
}
func getIndexWorker(c *C, useConfig bool) {
- kc := runProxy(c, false, useConfig)
+ kc, _ := runProxy(c, false, useConfig, nil)
defer closeListener()
// Put "index-data" blocks
}
func (s *ServerRequiredSuite) TestCollectionSharingToken(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
hash, _, err := kc.PutB([]byte("shareddata"))
c.Check(err, IsNil)
}
func (s *ServerRequiredSuite) TestPutAskGetInvalidToken(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
// Put a test block
_, _, _, err = kc.Get(hash)
c.Assert(err, FitsTypeOf, &keepclient.ErrNotFound{})
c.Check(err.(*keepclient.ErrNotFound).Temporary(), Equals, false)
- c.Check(err, ErrorMatches, ".*HTTP 403 \"Missing or invalid Authorization header\".*")
+ c.Check(err, ErrorMatches, ".*HTTP 403 \"Missing or invalid Authorization header, or method not allowed\".*")
}
_, _, err = kc.PutB([]byte("foo"))
- c.Check(err, ErrorMatches, ".*403.*Missing or invalid Authorization header")
+ c.Check(err, ErrorMatches, ".*403.*Missing or invalid Authorization header, or method not allowed")
}
}
func (s *ServerRequiredSuite) TestAskGetKeepProxyConnectionError(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
// Point keepproxy at a non-existent keepstore
}
func (s *NoKeepServerSuite) TestAskGetNoKeepServerError(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
hash := fmt.Sprintf("%x", md5.Sum([]byte("foo")))
}
func (s *ServerRequiredSuite) TestPing(c *C) {
- kc := runProxy(c, false, false)
+ kc, _ := runProxy(c, false, false, nil)
defer closeListener()
- rtr := MakeRESTRouter(kc, 10*time.Second, arvadostest.ManagementToken)
+ rtr, err := MakeRESTRouter(kc, 10*time.Second, &arvados.Cluster{ManagementToken: arvadostest.ManagementToken}, log.New())
+ c.Assert(err, check.IsNil)
req, err := http.NewRequest("GET",
"http://"+listener.Addr().String()+"/_health/ping",
FROM arvados/arvbox-base
ARG arvados_version
ARG composer_version=arvados-fork
-ARG workbench2_version=master
+ARG workbench2_version=main
RUN cd /usr/src && \
git clone --no-checkout https://git.arvados.org/arvados.git && \
if ! test -d $ARVADOS_CONTAINER_PATH/git/repositories/$repo_uuid.git ; then
git clone --bare /usr/src/arvados $ARVADOS_CONTAINER_PATH/git/repositories/$repo_uuid.git
else
- git --git-dir=$ARVADOS_CONTAINER_PATH/git/repositories/$repo_uuid.git fetch -f /usr/src/arvados master:master
+ git --git-dir=$ARVADOS_CONTAINER_PATH/git/repositories/$repo_uuid.git fetch -f /usr/src/arvados main:main
fi
cd /usr/src/arvados/services/api
- ca-certificates
arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run:
- # Taken from https://github.com/arvados/arvados/blob/master/tools/arvbox/lib/arvbox/docker/service/certificate/run
+ # Taken from https://github.com/arvados/arvados/blob/main/tools/arvbox/lib/arvbox/docker/service/certificate/run
cmd.run:
- name: |
# These dirs are not to CentOS-ish, but this is a helper script
- ca-certificates
arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run:
- # Taken from https://github.com/arvados/arvados/blob/master/tools/arvbox/lib/arvbox/docker/service/certificate/run
+ # Taken from https://github.com/arvados/arvados/blob/main/tools/arvbox/lib/arvbox/docker/service/certificate/run
cmd.run:
- name: |
# These dirs are not to CentOS-ish, but this is a helper script
# Extra states to apply. If you use your own subdir, change this value accordingly
# EXTRA_STATES_DIR="${CONFIG_DIR}/states"
+# These are ARVADOS-related settings.
# Which release of Arvados repo you want to use
RELEASE="production"
# Which version of Arvados you want to install. Defaults to latest stable
# This is an arvados-formula setting.
# If branch is set, the script will switch to it before running salt
# Usually not needed, only used for testing
-# BRANCH="master"
+# BRANCH="main"
##########################################################
# Usually there's no need to modify things below this line
# Formulas versions
-# ARVADOS_TAG="v1.1.4"
+# ARVADOS_TAG="2.2.0"
# POSTGRES_TAG="v0.41.6"
# NGINX_TAG="temp-fix-missing-statements-in-pillar"
# DOCKER_TAG="v1.0.0"
# Extra states to apply. If you use your own subdir, change this value accordingly
# EXTRA_STATES_DIR="${CONFIG_DIR}/states"
+# These are ARVADOS-related settings.
# Which release of Arvados repo you want to use
RELEASE="production"
# Which version of Arvados you want to install. Defaults to latest stable
# This is an arvados-formula setting.
# If branch is set, the script will switch to it before running salt
# Usually not needed, only used for testing
-# BRANCH="master"
+# BRANCH="main"
##########################################################
# Usually there's no need to modify things below this line
# Formulas versions
-# ARVADOS_TAG="v1.1.4"
+# ARVADOS_TAG="2.2.0"
# POSTGRES_TAG="v0.41.6"
# NGINX_TAG="temp-fix-missing-statements-in-pillar"
# DOCKER_TAG="v1.0.0"
# Extra states to apply. If you use your own subdir, change this value accordingly
# EXTRA_STATES_DIR="${CONFIG_DIR}/states"
+# These are ARVADOS-related settings.
# Which release of Arvados repo you want to use
RELEASE="production"
# Which version of Arvados you want to install. Defaults to latest stable
# This is an arvados-formula setting.
# If branch is set, the script will switch to it before running salt
# Usually not needed, only used for testing
-# BRANCH="master"
+# BRANCH="main"
##########################################################
# Usually there's no need to modify things below this line
# Formulas versions
-# ARVADOS_TAG="v1.1.4"
+# ARVADOS_TAG="2.2.0"
# POSTGRES_TAG="v0.41.6"
# NGINX_TAG="temp-fix-missing-statements-in-pillar"
# DOCKER_TAG="v1.0.0"
-#!/bin/bash -x
+#!/usr/bin/env bash
# Copyright (C) The Arvados Authors. All rights reserved.
#
echo >&2
echo >&2 "${0} options:"
echo >&2 " -d, --debug Run salt installation in debug mode"
- echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
echo >&2 " -c <local.params>, --config <local.params> Path to the local.params config file"
echo >&2 " -t, --test Test installation running a CWL workflow"
echo >&2 " -r, --roles List of Arvados roles to apply to the host, comma separated"
echo >&2 " workbench2"
echo >&2 " Defaults to applying them all"
echo >&2 " -h, --help Display this help and exit"
+ echo >&2 " --dump-config <dest_dir> Dumps the pillars and states to a directory"
+ echo >&2 " This parameter does not perform any installation at all. It's"
+ echo >&2 " intended to give you a parsed sot of configuration files so"
+ echo >&2 " you can inspect them or use them in you Saltstack infrastructure."
+ echo >&2 " It"
+ echo >&2 " - parses the pillar and states templates,"
+ echo >&2 " - downloads the helper formulas with their desired versions,"
+ echo >&2 " - prepares the 'top.sls' files both for pillars and states"
+ echo >&2 " for the selected role/s"
+ echo >&2 " - writes the resulting files into <dest_dir>"
echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2
}
arguments() {
# NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
+ if ! which getopt > /dev/null; then
+ echo >&2 "GNU getopt is required to run this script. Please install it and re-reun it"
+ exit 1
+ fi
+
TEMP=$(getopt -o c:dhp:r:tv \
- --long config:,debug,help,ssl-port:,roles:,test,vagrant \
+ --long config:,debug,dump-config:,help,roles:,test,vagrant \
-n "${0}" -- "${@}")
- if [ ${?} != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
+ if [ ${?} != 0 ];
+ then echo "Please check the parameters you entered and re-run again"
+ exit 1
+ fi
# Note the quotes around `$TEMP': they are essential!
eval set -- "$TEMP"
-d | --debug)
LOG_LEVEL="debug"
shift
+ set -x
;;
- -p | --ssl-port)
- CONTROLLER_EXT_SSL_PORT=${2}
+ --dump-config)
+ if [[ ${2} = /* ]]; then
+ DUMP_SALT_CONFIG_DIR=${2}
+ else
+ DUMP_SALT_CONFIG_DIR=${PWD}/${2}
+ fi
+ ## states
+ S_DIR="${DUMP_SALT_CONFIG_DIR}/salt"
+ ## formulas
+ F_DIR="${DUMP_SALT_CONFIG_DIR}/formulas"
+ ## pillars
+ P_DIR="${DUMP_SALT_CONFIG_DIR}/pillars"
+ ## tests
+ T_DIR="${DUMP_SALT_CONFIG_DIR}/tests"
+ DUMP_CONFIG="yes"
shift 2
;;
-r | --roles)
CONFIG_FILE="${SCRIPT_DIR}/local.params"
CONFIG_DIR="local_config_dir"
+DUMP_CONFIG="no"
LOG_LEVEL="info"
CONTROLLER_EXT_SSL_PORT=443
TESTS_DIR="tests"
WORKBENCH1_EXT_SSL_PORT=443
WORKBENCH2_EXT_SSL_PORT=3001
+## These are ARVADOS-related parameters
# For a stable release, change RELEASE "production" and VERSION to the
# package version (including the iteration, e.g. X.Y.Z-1) of the
# release.
+# The "local.params.example.*" files already set "RELEASE=production"
+# to deploy production-ready packages
RELEASE="development"
VERSION="latest"
-# The arvados-formula version. For a stable release, this should be a
+# These are arvados-formula-related parameters
+# An arvados-formula tag. For a stable release, this should be a
# branch name (e.g. X.Y-dev) or tag for the release.
-ARVADOS_TAG="main"
+# ARVADOS_TAG="2.2.0"
+# BRANCH="main"
# Other formula versions we depend on
POSTGRES_TAG="v0.41.6"
-NGINX_TAG="v2.7.4"
+NGINX_TAG="temp-fix-missing-statements-in-pillar"
DOCKER_TAG="v1.0.0"
LOCALE_TAG="v0.3.4"
LETSENCRYPT_TAG="v2.1.0"
# Salt's dir
+DUMP_SALT_CONFIG_DIR=""
## states
S_DIR="/srv/salt"
## formulas
F_DIR="/srv/formulas"
-##pillars
+## pillars
P_DIR="/srv/pillars"
+## tests
+T_DIR="/tmp/cluster_tests"
arguments ${@}
if [ -s ${CONFIG_FILE} ]; then
source ${CONFIG_FILE}
else
- echo >&2 "Please create a '${CONFIG_FILE}' file with initial values, as described in"
+ echo >&2 "You don't seem to have a config file with initial values."
+ echo >&2 "Please create a '${CONFIG_FILE}' file as described in"
echo >&2 " * https://doc.arvados.org/install/salt-single-host.html#single_host, or"
echo >&2 " * https://doc.arvados.org/install/salt-multi-host.html#multi_host_multi_hostnames"
exit 1
fi
if [ ! -d ${CONFIG_DIR} ]; then
- echo >&2 "Please create a '${CONFIG_DIR}' with initial values, as described in"
+ echo >&2 "You don't seem to have a config directory with pillars and states."
+ echo >&2 "Please create a '${CONFIG_DIR}' directory (as configured in your '${CONFIG_FILE}'). Please see"
echo >&2 " * https://doc.arvados.org/install/salt-single-host.html#single_host, or"
echo >&2 " * https://doc.arvados.org/install/salt-multi-host.html#multi_host_multi_hostnames"
exit 1
exit 1
fi
-if ! grep -E '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
+if ! grep -qE '^[[:alnum:]]{5}$' <<<${CLUSTER} ; then
echo >&2 "ERROR: <CLUSTER> must be exactly 5 alphanumeric characters long"
echo >&2 "Fix the cluster name in the 'local.params' file and re-run the provision script"
exit 1
HOSTNAME_EXT="${CLUSTER}.${DOMAIN}"
fi
-apt-get update
-apt-get install -y curl git jq
-
-if which salt-call; then
- echo "Salt already installed"
+if [ "${DUMP_CONFIG}" = "yes" ]; then
+ echo "The provision installer will just dump a config under ${DUMP_SALT_CONFIG_DIR} and exit"
else
- curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
- sh /tmp/bootstrap_salt.sh -XdfP -x python3
- /bin/systemctl stop salt-minion.service
- /bin/systemctl disable salt-minion.service
-fi
+ apt-get update
+ apt-get install -y curl git jq
+
+ if which salt-call; then
+ echo "Salt already installed"
+ else
+ curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
+ sh /tmp/bootstrap_salt.sh -XdfP -x python3
+ /bin/systemctl stop salt-minion.service
+ /bin/systemctl disable salt-minion.service
+ fi
-# Set salt to masterless mode
-cat > /etc/salt/minion << EOFSM
+ # Set salt to masterless mode
+ cat > /etc/salt/minion << EOFSM
file_client: local
file_roots:
base:
base:
- ${P_DIR}
EOFSM
+fi
-mkdir -p ${S_DIR} ${F_DIR} ${P_DIR}
+mkdir -p ${S_DIR} ${F_DIR} ${P_DIR} ${T_DIR}
# Get the formula and dependencies
cd ${F_DIR} || exit 1
-git clone --branch "${ARVADOS_TAG}" https://git.arvados.org/arvados-formula.git
-git clone --branch "${DOCKER_TAG}" https://github.com/saltstack-formulas/docker-formula.git
-git clone --branch "${LOCALE_TAG}" https://github.com/saltstack-formulas/locale-formula.git
-git clone --branch "${NGINX_TAG}" https://github.com/saltstack-formulas/nginx-formula.git
-git clone --branch "${POSTGRES_TAG}" https://github.com/saltstack-formulas/postgres-formula.git
-git clone --branch "${LETSENCRYPT_TAG}" https://github.com/saltstack-formulas/letsencrypt-formula.git
+echo "Cloning formulas"
+rm -rf ${F_DIR}/* || exit 1
+git clone --quiet https://github.com/saltstack-formulas/docker-formula.git ${F_DIR}/docker
+( cd docker && git checkout --quiet tags/"${DOCKER_TAG}" -b "${DOCKER_TAG}" )
+
+git clone --quiet https://github.com/saltstack-formulas/locale-formula.git ${F_DIR}/locale
+( cd locale && git checkout --quiet tags/"${LOCALE_TAG}" -b "${LOCALE_TAG}" )
+
+git clone --quiet https://github.com/netmanagers/nginx-formula.git ${F_DIR}/nginx
+( cd nginx && git checkout --quiet tags/"${NGINX_TAG}" -b "${NGINX_TAG}" )
+
+git clone --quiet https://github.com/saltstack-formulas/postgres-formula.git ${F_DIR}/postgres
+( cd postgres && git checkout --quiet tags/"${POSTGRES_TAG}" -b "${POSTGRES_TAG}" )
+
+git clone --quiet https://github.com/saltstack-formulas/letsencrypt-formula.git ${F_DIR}/letsencrypt
+( cd letsencrypt && git checkout --quiet tags/"${LETSENCRYPT_TAG}" -b "${LETSENCRYPT_TAG}" )
+
+git clone --quiet https://git.arvados.org/arvados-formula.git ${F_DIR}/arvados
# If we want to try a specific branch of the formula
if [ "x${BRANCH}" != "x" ]; then
- cd ${F_DIR}/arvados-formula || exit 1
- git checkout -t origin/"${BRANCH}" -b "${BRANCH}"
- cd -
+ ( cd ${F_DIR}/arvados && git checkout --quiet -t origin/"${BRANCH}" -b "${BRANCH}" )
+elif [ "x${ARVADOS_TAG}" != "x" ]; then
+( cd ${F_DIR}/arvados && git checkout --quiet tags/"${ARVADOS_TAG}" -b "${ARVADOS_TAG}" )
fi
if [ "x${VAGRANT}" = "xyes" ]; then
SOURCE_STATES_DIR="${EXTRA_STATES_DIR}"
+echo "Writing pillars and states"
+
# Replace variables (cluster, domain, etc) in the pillars, states and tests
# to ease deployment for newcomers
if [ ! -d "${SOURCE_PILLARS_DIR}" ]; then
echo "You requested to run tests, but ${SOURCE_TESTS_DIR} does not exist or is not a directory. Exiting."
exit 1
fi
-mkdir -p /tmp/cluster_tests
+mkdir -p ${T_DIR}
# Replace cluster and domain name in the test files
for f in $(ls "${SOURCE_TESTS_DIR}"/*); do
sed "s#__CLUSTER__#${CLUSTER}#g;
s#__INITIAL_USER__#${INITIAL_USER}#g;
s#__DATABASE_PASSWORD__#${DATABASE_PASSWORD}#g;
s#__SYSTEM_ROOT_TOKEN__#${SYSTEM_ROOT_TOKEN}#g" \
- "${f}" > "/tmp/cluster_tests"/$(basename "${f}")
+ "${f}" > ${T_DIR}/$(basename "${f}")
done
-chmod 755 /tmp/cluster_tests/run-test.sh
+chmod 755 ${T_DIR}/run-test.sh
# Replace helper state files that differ from the formula's examples
if [ -d "${SOURCE_STATES_DIR}" ]; then
done
fi
+if [ "${DUMP_CONFIG}" = "yes" ]; then
+ # We won't run the rest of the script because we're just dumping the config
+ exit 0
+fi
+
# FIXME! #16992 Temporary fix for psql call in arvados-api-server
if [ -e /root/.psqlrc ]; then
if ! ( grep 'pset pager off' /root/.psqlrc ); then
# Test that the installation finished correctly
if [ "x${TEST}" = "xyes" ]; then
- cd /tmp/cluster_tests
+ cd ${T_DIR}
./run-test.sh
fi
arv.config()["Services"]["Workbench1"]["ExternalURL"],
uuid, prof)
+collectionNameCache = {}
+def getCollectionName(arv, uuid):
+ if uuid not in collectionNameCache:
+ u = arv.collections().get(uuid=uuid).execute()
+ collectionNameCache[uuid] = u["name"]
+ return collectionNameCache[uuid]
+
def getname(u):
return "\"%s\" (%s)" % (u["name"], u["uuid"])
else:
users[owner].append("%s Deleted collection %s %s" % (event_at, getname(e["properties"]["old_attributes"]), loguuid))
+ elif e["event_type"] == "file_download":
+ users[e["object_uuid"]].append("%s Downloaded file \"%s\" from \"%s\" (%s) (%s)" % (event_at,
+ e["properties"].get("collection_file_path") or e["properties"].get("reqPath"),
+ getCollectionName(arv, e["properties"].get("collection_uuid")),
+ e["properties"].get("collection_uuid"),
+ e["properties"].get("portable_data_hash")))
+
+ elif e["event_type"] == "file_upload":
+ users[e["object_uuid"]].append("%s Uploaded file \"%s\" to \"%s\" (%s)" % (event_at,
+ e["properties"].get("collection_file_path") or e["properties"].get("reqPath"),
+ getCollectionName(arv, e["properties"].get("collection_uuid")),
+ e["properties"].get("collection_uuid")))
+
else:
users[owner].append("%s %s %s %s" % (e["event_type"], e["object_kind"], e["object_uuid"], loguuid))
projects / arvados.git / commitdiff