mimemagic (0.3.8)
nokogiri (~> 1)
mini_mime (1.0.2)
- mini_portile2 (2.5.0)
+ mini_portile2 (2.5.1)
minitest (5.10.3)
mocha (1.8.0)
metaclass (~> 0.0.1)
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
nio4r (2.5.7)
- nokogiri (1.11.2)
+ nokogiri (1.11.5)
mini_portile2 (~> 2.5.0)
racc (~> 1.4)
npm-rails (0.2.1)
api_params[:filters] = @filters if @filters
api_params[:distinct] = @distinct if @distinct
api_params[:include_trash] = @include_trash if @include_trash
+ api_params[:cluster_id] = Rails.configuration.ClusterID
if @fetch_multiple_pages
# Default limit to (effectively) api server's MAX_LIMIT
api_params[:limit] = 2**(0.size*8 - 1) - 1
<%= raw(Rails.configuration.Workbench.WelcomePageHTML) %>
<% case %>
- <% when Rails.configuration.Login.Google.Enable %>
- <% when Rails.configuration.Login.OpenIDConnect.Enable %>
- <% when Rails.configuration.Login.SSO.Enable %>
- <div class="pull-right">
- <%= link_to arvados_api_client.arvados_login_url(return_to: request.url), class: "btn btn-primary" do %>
- Log in to <%= Rails.configuration.Workbench.SiteName %>
- <i class="fa fa-fw fa-arrow-circle-right"></i>
- <% end %>
- </div>
- <% when Rails.configuration.Login.PAM.Enable %>
- <% when Rails.configuration.Login.LDAP.Enable %>
- <% when Rails.configuration.Login.Test.Enable %>
+ <% when Rails.configuration.Login.PAM.Enable,
+ Rails.configuration.Login.LDAP.Enable,
+ Rails.configuration.Login.Test.Enable %>
<form id="login-form-tag" onsubmit="controller_password_authenticate(event)">
<p>username <input type="text" class="form-control" name="login-username"
value="" id="login-username" style="width: 50%"
<span style="color: red"><p id="login-authenticate-error"></p></span>
<button type="submit" class="btn btn-primary">Log in</button>
</form>
+ <% else %>
+ <div class="pull-right">
+ <%= link_to arvados_api_client.arvados_login_url(return_to: request.url), class: "btn btn-primary" do %>
+ Log in to <%= Rails.configuration.Workbench.SiteName %>
+ <i class="fa fa-fw fa-arrow-circle-right"></i>
+ <% end %>
+ </div>
<% end %>
</div>
- user/cwl/federated-workflows.html.textile.liquid
- user/cwl/cwl-versions.html.textile.liquid
- user/cwl/crunchstat-summary.html.textile.liquid
+ - user/cwl/costanalyzer.html.textile.liquid
+ - user/debugging/container-shell-access.html.textile.liquid
- Working with git repositories:
- user/tutorials/add-new-repository.html.textile.liquid
- user/tutorials/git-arvados-guide.html.textile.liquid
- install/crunch2-slurm/configure-slurm.html.textile.liquid
- install/crunch2-slurm/install-compute-node.html.textile.liquid
- install/crunch2-slurm/install-test.html.textile.liquid
+ - Additional configuration:
+ - install/container-shell-access.html.textile.liquid
- External dependencies:
- install/install-postgresql.html.textile.liquid
- install/ruby.html.textile.liquid
|old_user_uuid|uuid|The uuid of the "old" account|query||
|new_owner_uuid|uuid|The uuid of a project to which objects owned by the "old" user will be reassigned.|query||
|redirect_to_new_user|boolean|If true, also redirect login and reassign authorization credentials from "old" user to the "new" user|query||
+
+h3. authenticate
+
+Create a new API token based on username/password credentials. Returns an "API client authorization":api_client_authorizations.html object containing the API token, or an "error object.":../requests.html#errors
+
+Valid credentials are determined by the choice of "configured login backend.":{{site.baseurl}}/install/setup-login.html
+
+Note: this endpoint cannot be used with login backends that use web-based third party authentication, such as Google or OpenID Connect.
+
+Arguments:
+
+table(table table-bordered table-condensed).
+|_. Argument |_. Type |_. Description |_. Location |_. Example |
+{background:#ccffcc}.|username|string|The username.|body||
+{background:#ccffcc}.|password|string|The password.|body||
API requests must provide the API token using the @Authorization@ header in the following format:
<pre>
-$ curl -v -H "Authorization: OAuth2 xxxxapitokenxxxx" https://192.168.5.2:8000/arvados/v1/collections
+$ curl -v -H "Authorization: Bearer xxxxapitokenxxxx" https://192.168.5.2:8000/arvados/v1/collections
> GET /arvados/v1/collections HTTP/1.1
> ...
-> Authorization: OAuth2 xxxxapitokenxxxx
+> Authorization: Bearer xxxxapitokenxxxx
> ...
</pre>
+On a cluster configured to use an OpenID Connect provider (other than Google) as a login backend, Arvados can be configured to accept an OpenID Connect access token in place of an Arvados API token. OIDC access tokens are also accepted by a cluster that delegates login to another cluster (LoginCluster) which in turn has this feature configured. See @Login.OpenIDConnect.AcceptAccessTokenScope@ in the "default config.yml file":{{site.baseurl}}/admin/config.html for details.
+
+<pre>
+$ curl -v -H "Authorization: Bearer xxxx-openid-connect-access-token-xxxx" https://192.168.5.2:8000/arvados/v1/collections
+</pre>
+
h3. Parameters
Request parameters may be provided in one of two ways. They may be provided in the "query" section of request URI, or they may be provided in the body of the request with application/x-www-form-urlencoded encoding. If parameters are provided in both places, their values will be merged. Parameter names must be unique. If a parameter appears multiple times, the behavior is undefined.
Results are returned JSON-encoded in the response body.
-h3. Errors
+h3(#errors). Errors
If a request cannot be fulfilled, the API will return 4xx or 5xx HTTP status code. Be aware that the API server may return a 404 (Not Found) status for resources that exist but for which the client does not have read access. The API will also return an error record:
h3. Create a new record
<pre>
-$ curl -v -X POST --data-urlencode 'collection={"name":"empty collection"}' -H "Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections | jq .
+$ curl -v -X POST --data-urlencode 'collection={"name":"empty collection"}' -H "Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections | jq .
> POST /arvados/v1/collections HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 192.168.5.2:8000
> Accept: */*
-> Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
+> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
> Content-Length: 54
> Content-Type: application/x-www-form-urlencoded
>
h3. Delete a record
<pre>
-$ curl -X DELETE -v -H "Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-m1ma0mxxfg3mbcc | jq .
+$ curl -X DELETE -v -H "Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-m1ma0mxxfg3mbcc | jq .
> DELETE /arvados/v1/collections/962eh-4zz18-m1ma0mxxfg3mbcc HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 192.168.5.2:8000
> Accept: */*
-> Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
+> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
h3. Get a specific record
<pre>
-$ curl -v -H "Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km | jq .
+$ curl -v -H "Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km | jq .
> GET /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 192.168.5.2:8000
> Accept: */*
-> Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
+> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
(Note, return result is truncated).
<pre>
-$ curl -v -G --data-urlencode 'filters=[["created_at",">","2016-11-08T21:38:24.124834000Z"]]' -H "Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections | jq .
+$ curl -v -G --data-urlencode 'filters=[["created_at",">","2016-11-08T21:38:24.124834000Z"]]' -H "Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections | jq .
> GET /arvados/v1/collections?filters=%5B%5B%22uuid%22%2C%20%22%3D%22%2C%20%22962eh-4zz18-xi32mpz2621o8km%22%5D%5D HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 192.168.5.2:8000
> Accept: */*
-> Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
+> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
h3. Update a field
<pre>
-$ curl -v -X PUT --data-urlencode 'collection={"name":"rna.SRR948778.bam"}' -H "Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km | jq .
+$ curl -v -X PUT --data-urlencode 'collection={"name":"rna.SRR948778.bam"}' -H "Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km | jq .
> PUT /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 192.168.5.2:8000
> Accept: */*
-> Authorization: OAuth2 oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
+> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr
> Content-Length: 53
> Content-Type: application/x-www-form-urlencoded
>
All requests to the API server must have an API token. API tokens can be issued by going though the login flow, or created via the API. At this time, only browser based applications can perform login from email/password. Command line applications and services must use an API token provided via the @ARVADOS_API_TOKEN@ environment variable or configuration file.
-h2. Browser login
+h2. Login
-Browser based applications can perform log in via the following highlevel flow:
+Browser based applications can log in using one of the two possible flows:
-# The web application presents a "login" link to @/login@ on the API server with a @return_to@ parameter provided in the query portion of the URL. For example @https://{{ site.arvados_api_host }}/login?return_to=XXX@ , where @return_to=XXX@ is the URL of the login page for the web application.
-# The "login" link takes the browser to the login page (this may involve several redirects)
-# The user logs in. API server authenticates the user and issues a new API token.
-# The browser is redirected to the login page URL provided in @return_to=XXX@ with the addition of @?api_token=xxxxapitokenxxxx@.
-# The web application gets the login request with the included authorization token.
+h3. Authenticate via a third party
-!{{site.baseurl}}/images/Session_Establishment.svg!
+# The web application instructs the user to click on a link to the @/login@ endpoint on the API server. This link should include the @return_to@ parameter in the query portion of the URL. For example @https://{{ site.arvados_api_host }}/login?return_to=XXX@ , where @return_to=XXX@ is a page in the web application.
+# The @/login@ endpoint redirects the user to the configured third party authentication provider (e.g. Google or other OpenID Connect provider).
+# The user logs in to the third party provider, then they are redirected back to the API server.
+# The API server authenticates the user, issues a new API token, and redirects the browser to the URL provided in @return_to=XXX@ with the addition of @?api_token=xxxxapitokenxxxx@.
+# The web application gets the authorization token from the query and uses it to access the API server on the user's behalf.
+
+h3. Direct username/password authentication
+
+# The web application presents username and password fields.
+# When the submit button is pressed, using Javascript, the browser sends a POST request to @/arvados/v1/users/authenticate@
+** The request payload type is @application/javascript@
+** The request body is a JSON object with @username@ and @password@ fields.
+# The API server receives the username and password, authenticates them with the upstream provider (such as LDAP or PAM), and responds with the @api_client_authorization@ object for the new API token.
+# The web application receives the authorization token in the response and uses it to access the API server on the user's behalf.
+
+h3. Using an OpenID Connect access token
-The "browser authentication process is documented in detail on the Arvados wiki.":https://dev.arvados.org/projects/arvados/wiki/Workbench_authentication_process
+A cluster that uses OpenID Connect as a login provider can be configured to accept OIDC access tokens as well as Arvados API tokens (this is disabled by default; see @Login.OpenIDConnect.AcceptAccessToken@ in the "default config.yml file":{{site.baseurl}}/admin/config.html).
+# The client obtains an access token from the OpenID Connect provider via some method outside of Arvados.
+# The client presents the access token with an Arvados API request (e.g., request header @Authorization: Bearer xxxxaccesstokenxxxx@).
+# Depending on configuration, the API server decodes the access token (which must be a signed JWT) and confirms that it includes the required scope (see @Login.OpenIDConnect.AcceptAccessTokenScope@ in the "default config.yml file":{{site.baseurl}}/admin/config.html).
+# The API server uses the provider's UserInfo endpoint to validate the presented token.
+# If the token is valid, it is cached in the Arvados database and accepted in subsequent API calls for the next 10 minutes.
+
+h3. Diagram
+
+!{{site.baseurl}}/images/Session_Establishment.svg!
h2. User activation
--- /dev/null
+---
+layout: default
+navsection: installguide
+title: Configure container shell access
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Arvados can be configured to permit shell access to running containers. This can be handy for debugging, but it could affect reproducability of workflows. This feature can be enabled for admin users, or for all users. By default, it is entirely disabled.
+
+The relevant configuration section is
+
+<notextile>
+<pre><code> Containers:
+ ShellAccess:
+ # An admin user can use "arvados-client shell" to start an
+ # interactive shell (with any user ID) in any running
+ # container.
+ Admin: false
+
+ # Any user can use "arvados-client shell" to start an
+ # interactive shell (with any user ID) in any running
+ # container that they started, provided it isn't also
+ # associated with a different user's container request.
+ #
+ # Interactive sessions make it easy to alter the container's
+ # runtime environment in ways that aren't recorded or
+ # reproducible. Consider the implications for automatic
+ # container reuse before enabling and using this feature. In
+ # particular, note that starting an interactive session does
+ # not disqualify a container from being reused by a different
+ # user/workflow in the future.
+ User: false
+</code></pre>
+</notextile>
+
+To enable the feature a firewall change may also be required. This feature requires the opening of tcp connections from @arvados-controller@ to the range specified in the @net.ipv4.ip_local_port_range@ sysctl on compute nodes. If that range is unknown or hard to determine, it will be sufficient to allow tcp connections from @arvados-controller@ to port 1024-65535 on compute nodes, while allowing traffic that is part of existing tcp connections.
+
+After changing the configuration, @arvados-controller@ must be restarted for the change to take effect. When enabling, shell access will be enabled for any running containers. When disabling, access is removed immediately for any running containers, as well as any containers started subsequently. Restarting @arvados-controller@ will kill any active connections.
+
+Usage instructions for this feature are available in the "User guide":{{site.baseurl}}/user/debugging/container-shell-access.html.
|_. Distribution|_. State|_. Last supported version|
|CentOS 7|Supported|Latest|
|Debian 10 ("buster")|Supported|Latest|
+|Ubuntu 20.04 ("focal")|Supported|Latest|
|Ubuntu 18.04 ("bionic")|Supported|Latest|
-|Ubuntu 16.04 ("xenial")|Supported|Latest|
-|Debian 9 ("stretch")|EOL|Latest 2.1.X release|
+|Ubuntu 16.04 ("xenial")|EOL|2.1.2|
+|Debian 9 ("stretch")|EOL|2.1.2|
|Debian 8 ("jessie")|EOL|1.4.3|
|Ubuntu 14.04 ("trusty")|EOL|1.4.3|
|Ubuntu 12.04 ("precise")|EOL|8ed7b6dd5d4df93a3f37096afe6d6f81c2a7ef6e (2017-05-03)|
Check the LDAP section in the "default config file":{{site.baseurl}}/admin/config.html for more details and configuration options.
-h2(#pam). PAM (experimental)
+h2(#pam). PAM
With this configuration, authentication is done according to the Linux PAM ("Pluggable Authentication Modules") configuration on your controller host.
Check the "default config file":{{site.baseurl}}/admin/config.html for more PAM configuration options.
-The default PAM configuration on most Linux systems uses the local password database in @/etc/shadow@ for all logins. In this case, in order to log in to Arvados, users must have a UNIX account and password on the controller host itself. This can be convenient for a single-user or test cluster. User accounts can have @/dev/false@ as the shell in order to allow the user to log into Arvados but not log into a shell on the controller host.
+The default PAM configuration on most Linux systems uses the local user/password database in @/etc/passwd@ and @/etc/shadow@ for all logins. In this case, in order to log in to Arvados, users must have a UNIX account and password on the controller host itself. This can be convenient for a single-user or test cluster. Configuring a user account with a shell of @/bin/false@ will enable the user to log into Arvados but not log into shell login on the controller host.
-PAM can also be configured to use different backends like LDAP. In a production environment, PAM configuration should use the service name ("arvados" by default) to set a separate policy for Arvados logins: generally, Arvados users should not have shell accounts on the controller node.
+PAM can also be configured to use other authentication systems such such as NIS or Kerberos. In a production environment, PAM configuration should use the service name ("arvados" by default) and set a separate policy for Arvados login. In this case, Arvados users should not have shell accounts on the controller node.
For information about configuring PAM, refer to the "PAM System Administrator's Guide":http://www.linux-pam.org/Linux-PAM-html/Linux-PAM_SAG.html.
--- /dev/null
+---
+layout: default
+navsection: userguide
+title: Analyzing workflow cost (cloud only)
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+{% include 'notebox_begin' %}
+
+This is only applicable when Arvados runs in a cloud environment and @arvados-dispatch-cloud@ is used to dispatch @crunch@ jobs. The per node-hour price for each defined InstanceType most be supplied in "config.yml":{{site.baseurl}}/admin/config.html.
+
+{% include 'notebox_end' %}
+
+The @arvados-client@ program can be used to analyze the cost of a workflow. It can be installed from packages (@apt install arvados-client@ or @yum install arvados-client@). The @arvados-client costanalyzer@ command analyzes the cost accounting information associated with Arvados container requests.
+
+h2(#syntax). Syntax
+
+The @arvados-client costanalyzer@ tool has a number of command line arguments:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arvados-client costanalyzer -h</span>
+Usage:
+ arvados-client costanalyzer [options ...] uuid [uuid ...]
+
+ This program analyzes the cost of Arvados container requests. For each uuid
+ supplied, it creates a CSV report that lists all the containers used to
+ fulfill the container request, together with the machine type and cost of
+ each container. At least one uuid must be specified.
+
+ When supplied with the uuid of a container request, it will calculate the
+ cost of that container request and all its children.
+
+ When supplied with the uuid of a collection, it will see if there is a
+ container_request uuid in the properties of the collection, and if so, it
+ will calculate the cost of that container request and all its children.
+
+ When supplied with a project uuid or when supplied with multiple container
+ request or collection uuids, it will create a CSV report for each supplied
+ uuid, as well as a CSV file with aggregate cost accounting for all supplied
+ uuids. The aggregate cost report takes container reuse into account: if a
+ container was reused between several container requests, its cost will only
+ be counted once.
+
+ Caveats:
+
+ - This program uses the cost data from config.yml at the time of the
+ execution of the container, stored in the 'node.json' file in its log
+ collection. If the cost data was not correctly configured at the time the
+ container was executed, the output from this program will be incorrect.
+
+ - If a container was run on a preemptible ("spot") instance, the cost data
+ reported by this program may be wildly inaccurate, because it does not have
+ access to the spot pricing in effect for the node then the container ran. The
+ UUID report file that is generated when the '-output' option is specified has
+ a column that indicates the preemptible state of the instance that ran the
+ container.
+
+ - This program does not take into account overhead costs like the time spent
+ starting and stopping compute nodes that run containers, the cost of the
+ permanent cloud nodes that provide the Arvados services, the cost of data
+ stored in Arvados, etc.
+
+ - When provided with a project uuid, subprojects will not be considered.
+
+ In order to get the data for the uuids supplied, the ARVADOS_API_HOST and
+ ARVADOS_API_TOKEN environment variables must be set.
+
+ This program prints the total dollar amount from the aggregate cost
+ accounting across all provided uuids on stdout.
+
+ When the '-output' option is specified, a set of CSV files with cost details
+ will be written to the provided directory.
+
+Options:
+ -cache
+ create and use a local disk cache of Arvados objects (default true)
+ -log-level level
+ logging level (debug, info, ...) (default "info")
+ -output directory
+ output directory for the CSV reports
+</code></pre>
+</notextile>
--- /dev/null
+---
+layout: default
+navsection: userguide
+title: Debugging workflows - shell access
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+{% include 'notebox_begin' %}
+
+To use this feature, your Arvados installation must be configured to allow container shell access. See "the install guide":{{site.baseurl}}/install/container-shell-access.html for more information.
+
+{% include 'notebox_end' %}
+
+The @arvados-client@ program can be used to connect to a container in a running workflow. It can be installed from packages (@apt install arvados-client@ or @yum install arvados-client@). The @arvados-client shell@ command provides an ssh connection into a running container.
+
+h2(#syntax). Syntax
+
+The @arvados-client shell@ tool has the following syntax:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arvados-client shell -h</span>
+arvados-client shell: open an interactive shell on a running container.
+
+Usage: arvados-client shell [options] [username@]container-uuid [ssh-options] [remote-command [args...]]
+
+Options:
+ -detach-keys string
+ set detach key sequence, as in docker-attach(1) (default "ctrl-],ctrl-]")
+
+</code></pre>
+</notextile>
+
+The @arvados-client shell@ command calls the ssh binary on your system to make the connection. Everything after _[username@]container-uuid_ is passed through to your OpenSSH client.
+
+h2(#Examples). Examples
+
+Connect to a running container, using the container request UUID:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arvados-client shell ce8i5-xvhdp-e6wnujfslyyqn4b</span>
+root@0f13dcd755fa:~#
+</code></pre>
+</notextile>
+
+The container UUID also works:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arvados-client shell ce8i5-dz642-h1cl0sa62d4i430</span>
+root@0f13dcd755fa:~#
+</code></pre>
+</notextile>
+
+SSH port forwarding is supported:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arvados-client shell ce8i5-dz642-h1cl0sa62d4i430 -L8888:localhost:80</span>
+root@0f13dcd755fa:~# nc -l -p 80
+</code></pre>
+</notextile>
+
+And then, connecting to port 8888 locally:
+
+<notextile>
+<pre><code>~$ <span class="userinput">echo hello | nc localhost 8888</span>
+</code></pre>
+</notextile>
+
+Which appears on the other end:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arvados-client shell ce8i5-dz642-h1cl0sa62d4i430 -L8888:localhost:80</span>
+root@0f13dcd755fa:~# nc -l -p 80
+hello
+</code></pre>
+</notextile>
UUIDTTL: 5s
# Block cache entries. Each block consumes up to 64 MiB RAM.
- MaxBlockEntries: 4
+ MaxBlockEntries: 20
# Collection cache entries.
MaxCollectionEntries: 1000
AuthenticationRequestParameters:
SAMPLE: ""
+ # Accept an OIDC access token as an API token if the OIDC
+ # provider's UserInfo endpoint accepts it.
+ #
+ # AcceptAccessTokenScope should also be used when enabling
+ # this feature.
+ AcceptAccessToken: false
+
+ # Before accepting an OIDC access token as an API token, first
+ # check that it is a JWT whose "scope" value includes this
+ # value. Example: "https://zzzzz.example.com/" (your Arvados
+ # API endpoint).
+ #
+ # If this value is empty and AcceptAccessToken is true, all
+ # access tokens will be accepted regardless of scope,
+ # including non-JWT tokens. This is not recommended.
+ AcceptAccessTokenScope: ""
+
PAM:
# (Experimental) Use PAM to authenticate users.
Enable: false
"Login.LDAP.UsernameAttribute": false,
"Login.LoginCluster": true,
"Login.OpenIDConnect": true,
+ "Login.OpenIDConnect.AcceptAccessToken": false,
+ "Login.OpenIDConnect.AcceptAccessTokenScope": false,
"Login.OpenIDConnect.AuthenticationRequestParameters": false,
"Login.OpenIDConnect.ClientID": false,
"Login.OpenIDConnect.ClientSecret": false,
UUIDTTL: 5s
# Block cache entries. Each block consumes up to 64 MiB RAM.
- MaxBlockEntries: 4
+ MaxBlockEntries: 20
# Collection cache entries.
MaxCollectionEntries: 1000
AuthenticationRequestParameters:
SAMPLE: ""
+ # Accept an OIDC access token as an API token if the OIDC
+ # provider's UserInfo endpoint accepts it.
+ #
+ # AcceptAccessTokenScope should also be used when enabling
+ # this feature.
+ AcceptAccessToken: false
+
+ # Before accepting an OIDC access token as an API token, first
+ # check that it is a JWT whose "scope" value includes this
+ # value. Example: "https://zzzzz.example.com/" (your Arvados
+ # API endpoint).
+ #
+ # If this value is empty and AcceptAccessToken is true, all
+ # access tokens will be accepted regardless of scope,
+ # including non-JWT tokens. This is not recommended.
+ AcceptAccessTokenScope: ""
+
PAM:
# (Experimental) Use PAM to authenticate users.
Enable: false
cluster.Login.OpenIDConnect.ClientSecret = s.fakeProvider.ValidClientSecret
cluster.Login.OpenIDConnect.EmailClaim = "email"
cluster.Login.OpenIDConnect.EmailVerifiedClaim = "email_verified"
+ cluster.Login.OpenIDConnect.AcceptAccessToken = true
+ cluster.Login.OpenIDConnect.AcceptAccessTokenScope = ""
s.testHandler = &Handler{Cluster: cluster}
s.testServer = newServerFromIntegrationTestEnv(c)
_, err := fn(ctx, conn.cluster.ClusterID, conn.local, opts)
return err
}
+ if opts.ClusterID != "" {
+ // Client explicitly selected cluster
+ _, err := fn(ctx, conn.cluster.ClusterID, conn.chooseBackend(opts.ClusterID), opts)
+ return err
+ }
cannotSplit := false
var matchAllFilters map[string]bool
ClientSecret: ` + s.oidcprovider.ValidClientSecret + `
EmailClaim: email
EmailVerifiedClaim: email_verified
+ AcceptAccessToken: true
+ AcceptAccessTokenScope: ""
`
} else {
yaml += `
}
case wantOpenIDConnect:
return &oidcLoginController{
- Cluster: cluster,
- Parent: parent,
- Issuer: cluster.Login.OpenIDConnect.Issuer,
- ClientID: cluster.Login.OpenIDConnect.ClientID,
- ClientSecret: cluster.Login.OpenIDConnect.ClientSecret,
- AuthParams: cluster.Login.OpenIDConnect.AuthenticationRequestParameters,
- EmailClaim: cluster.Login.OpenIDConnect.EmailClaim,
- EmailVerifiedClaim: cluster.Login.OpenIDConnect.EmailVerifiedClaim,
- UsernameClaim: cluster.Login.OpenIDConnect.UsernameClaim,
+ Cluster: cluster,
+ Parent: parent,
+ Issuer: cluster.Login.OpenIDConnect.Issuer,
+ ClientID: cluster.Login.OpenIDConnect.ClientID,
+ ClientSecret: cluster.Login.OpenIDConnect.ClientSecret,
+ AuthParams: cluster.Login.OpenIDConnect.AuthenticationRequestParameters,
+ EmailClaim: cluster.Login.OpenIDConnect.EmailClaim,
+ EmailVerifiedClaim: cluster.Login.OpenIDConnect.EmailVerifiedClaim,
+ UsernameClaim: cluster.Login.OpenIDConnect.UsernameClaim,
+ AcceptAccessToken: cluster.Login.OpenIDConnect.AcceptAccessToken,
+ AcceptAccessTokenScope: cluster.Login.OpenIDConnect.AcceptAccessTokenScope,
}
case wantSSO:
return &ssoLoginController{Parent: parent}
"golang.org/x/oauth2"
"google.golang.org/api/option"
"google.golang.org/api/people/v1"
+ "gopkg.in/square/go-jose.v2/jwt"
)
var (
)
type oidcLoginController struct {
- Cluster *arvados.Cluster
- Parent *Conn
- Issuer string // OIDC issuer URL, e.g., "https://accounts.google.com"
- ClientID string
- ClientSecret string
- UseGooglePeopleAPI bool // Use Google People API to look up alternate email addresses
- EmailClaim string // OpenID claim to use as email address; typically "email"
- EmailVerifiedClaim string // If non-empty, ensure claim value is true before accepting EmailClaim; typically "email_verified"
- UsernameClaim string // If non-empty, use as preferred username
- AuthParams map[string]string // Additional parameters to pass with authentication request
+ Cluster *arvados.Cluster
+ Parent *Conn
+ Issuer string // OIDC issuer URL, e.g., "https://accounts.google.com"
+ ClientID string
+ ClientSecret string
+ UseGooglePeopleAPI bool // Use Google People API to look up alternate email addresses
+ EmailClaim string // OpenID claim to use as email address; typically "email"
+ EmailVerifiedClaim string // If non-empty, ensure claim value is true before accepting EmailClaim; typically "email_verified"
+ UsernameClaim string // If non-empty, use as preferred username
+ AcceptAccessToken bool // Accept access tokens as API tokens
+ AcceptAccessTokenScope string // If non-empty, don't accept access tokens as API tokens unless they contain this scope
+ AuthParams map[string]string // Additional parameters to pass with authentication request
// override Google People API base URL for testing purposes
// (normally empty, set by google pkg to
if !ok {
return loginError(errors.New("error in OAuth2 exchange: no ID token in OAuth2 token"))
}
+ ctxlog.FromContext(ctx).WithField("rawIDToken", rawIDToken).Debug("oauth2Token provided ID token")
idToken, err := ctrl.verifier.Verify(ctx, rawIDToken)
if err != nil {
return loginError(fmt.Errorf("error verifying ID token: %s", err))
if err != nil {
return fmt.Errorf("error setting up OpenID Connect provider: %s", err)
}
+ if ok, err := ta.checkAccessTokenScope(ctx, tok); err != nil || !ok {
+ ta.cache.Add(tok, time.Now().Add(tokenCacheNegativeTTL))
+ return err
+ }
oauth2Token := &oauth2.Token{
AccessToken: tok,
}
ta.cache.Add(tok, aca)
return nil
}
+
+// Check that the provided access token is a JWT with the required
+// scope. If it is a valid JWT but missing the required scope, we
+// return a 403 error, otherwise true (acceptable as an API token) or
+// false (pass through unmodified).
+//
+// Return false if configured not to accept access tokens at all.
+//
+// Note we don't check signature or expiry here. We are relying on the
+// caller to verify those separately (e.g., by calling the UserInfo
+// endpoint).
+func (ta *oidcTokenAuthorizer) checkAccessTokenScope(ctx context.Context, tok string) (bool, error) {
+ if !ta.ctrl.AcceptAccessToken {
+ return false, nil
+ } else if ta.ctrl.AcceptAccessTokenScope == "" {
+ return true, nil
+ }
+ var claims struct {
+ Scope string `json:"scope"`
+ }
+ if t, err := jwt.ParseSigned(tok); err != nil {
+ ctxlog.FromContext(ctx).WithError(err).Debug("error parsing jwt")
+ return false, nil
+ } else if err = t.UnsafeClaimsWithoutVerification(&claims); err != nil {
+ ctxlog.FromContext(ctx).WithError(err).Debug("error extracting jwt claims")
+ return false, nil
+ }
+ for _, s := range strings.Split(claims.Scope, " ") {
+ if s == ta.ctrl.AcceptAccessTokenScope {
+ return true, nil
+ }
+ }
+ ctxlog.FromContext(ctx).WithFields(logrus.Fields{"have": claims.Scope, "need": ta.ctrl.AcceptAccessTokenScope}).Infof("unacceptable access token scope")
+ return false, httpserver.ErrorWithStatus(errors.New("unacceptable access token scope"), http.StatusUnauthorized)
+}
json.Unmarshal([]byte(fmt.Sprintf("%q", s.fakeProvider.Issuer.URL)), &s.cluster.Login.OpenIDConnect.Issuer)
s.cluster.Login.OpenIDConnect.ClientID = "oidc#client#id"
s.cluster.Login.OpenIDConnect.ClientSecret = "oidc#client#secret"
+ s.cluster.Login.OpenIDConnect.AcceptAccessToken = true
+ s.cluster.Login.OpenIDConnect.AcceptAccessTokenScope = ""
s.fakeProvider.ValidClientID = "oidc#client#id"
s.fakeProvider.ValidClientSecret = "oidc#client#secret"
db := arvadostest.DB(c, s.cluster)
tokenCacheTTL = time.Millisecond
tokenCacheRaceWindow = time.Millisecond
+ tokenCacheNegativeTTL = time.Millisecond
oidcAuthorizer := OIDCAccessTokenAuthorizer(s.cluster, func(context.Context) (*sqlx.DB, error) { return db, nil })
accessToken := s.fakeProvider.ValidAccessToken()
mac := hmac.New(sha256.New, []byte(s.cluster.SystemRootToken))
io.WriteString(mac, accessToken)
- hmac := fmt.Sprintf("%x", mac.Sum(nil))
+ apiToken := fmt.Sprintf("%x", mac.Sum(nil))
cleanup := func() {
- _, err := db.Exec(`delete from api_client_authorizations where api_token=$1`, hmac)
+ _, err := db.Exec(`delete from api_client_authorizations where api_token=$1`, apiToken)
c.Check(err, check.IsNil)
}
cleanup()
c.Assert(creds.Tokens, check.HasLen, 1)
c.Check(creds.Tokens[0], check.Equals, accessToken)
- err := db.QueryRowContext(ctx, `select expires_at at time zone 'UTC' from api_client_authorizations where api_token=$1`, hmac).Scan(&exp1)
+ err := db.QueryRowContext(ctx, `select expires_at at time zone 'UTC' from api_client_authorizations where api_token=$1`, apiToken).Scan(&exp1)
c.Check(err, check.IsNil)
c.Check(exp1.Sub(time.Now()) > -time.Second, check.Equals, true)
c.Check(exp1.Sub(time.Now()) < time.Second, check.Equals, true)
})(ctx, nil)
// If the token is used again after the in-memory cache
- // expires, oidcAuthorizer must re-checks the token and update
+ // expires, oidcAuthorizer must re-check the token and update
// the expires_at value in the database.
time.Sleep(3 * time.Millisecond)
oidcAuthorizer.WrapCalls(func(ctx context.Context, opts interface{}) (interface{}, error) {
var exp time.Time
- err := db.QueryRowContext(ctx, `select expires_at at time zone 'UTC' from api_client_authorizations where api_token=$1`, hmac).Scan(&exp)
+ err := db.QueryRowContext(ctx, `select expires_at at time zone 'UTC' from api_client_authorizations where api_token=$1`, apiToken).Scan(&exp)
c.Check(err, check.IsNil)
c.Check(exp.Sub(exp1) > 0, check.Equals, true)
c.Check(exp.Sub(exp1) < time.Second, check.Equals, true)
return nil, nil
})(ctx, nil)
+
+ s.fakeProvider.AccessTokenPayload = map[string]interface{}{"scope": "openid profile foobar"}
+ accessToken = s.fakeProvider.ValidAccessToken()
+ ctx = auth.NewContext(context.Background(), &auth.Credentials{Tokens: []string{accessToken}})
+
+ mac = hmac.New(sha256.New, []byte(s.cluster.SystemRootToken))
+ io.WriteString(mac, accessToken)
+ apiToken = fmt.Sprintf("%x", mac.Sum(nil))
+
+ for _, trial := range []struct {
+ configEnable bool
+ configScope string
+ acceptable bool
+ shouldRun bool
+ }{
+ {true, "foobar", true, true},
+ {true, "foo", false, false},
+ {true, "", true, true},
+ {false, "", false, true},
+ {false, "foobar", false, true},
+ } {
+ c.Logf("trial = %+v", trial)
+ cleanup()
+ s.cluster.Login.OpenIDConnect.AcceptAccessToken = trial.configEnable
+ s.cluster.Login.OpenIDConnect.AcceptAccessTokenScope = trial.configScope
+ oidcAuthorizer = OIDCAccessTokenAuthorizer(s.cluster, func(context.Context) (*sqlx.DB, error) { return db, nil })
+ checked := false
+ oidcAuthorizer.WrapCalls(func(ctx context.Context, opts interface{}) (interface{}, error) {
+ var n int
+ err := db.QueryRowContext(ctx, `select count(*) from api_client_authorizations where api_token=$1`, apiToken).Scan(&n)
+ c.Check(err, check.IsNil)
+ if trial.acceptable {
+ c.Check(n, check.Equals, 1)
+ } else {
+ c.Check(n, check.Equals, 0)
+ }
+ checked = true
+ return nil, nil
+ })(ctx, nil)
+ c.Check(checked, check.Equals, trial.shouldRun)
+ }
}
func (s *OIDCLoginSuite) TestGenericOIDCLogin(c *check.C) {
flags.Usage = func() {
fmt.Fprintf(flags.Output(), `
Usage:
- %s [options ...] <uuid> ...
+ %s [options ...] uuid [uuid ...]
This program analyzes the cost of Arvados container requests. For each uuid
supplied, it creates a CSV report that lists all the containers used to
container was reused between several container requests, its cost will only
be counted once.
- To get the node costs, the progam queries the Arvados API for current cost
- data for each node type used. This means that the reported cost always
- reflects the cost data as currently defined in the Arvados API configuration
- file.
-
Caveats:
- - the Arvados API configuration cost data may be out of sync with the cloud
- provider.
- - when generating reports for older container requests, the cost data in the
- Arvados API configuration file may have changed since the container request
- was fulfilled. This program uses the cost data stored at the time of the
+
+ - This program uses the cost data from config.yml at the time of the
execution of the container, stored in the 'node.json' file in its log
- collection.
- - if a container was run on a preemptible ("spot") instance, the cost data
+ collection. If the cost data was not correctly configured at the time the
+ container was executed, the output from this program will be incorrect.
+
+ - If a container was run on a preemptible ("spot") instance, the cost data
reported by this program may be wildly inaccurate, because it does not have
access to the spot pricing in effect for the node then the container ran. The
UUID report file that is generated when the '-output' option is specified has
a column that indicates the preemptible state of the instance that ran the
container.
+ - This program does not take into account overhead costs like the time spent
+ starting and stopping compute nodes that run containers, the cost of the
+ permanent cloud nodes that provide the Arvados services, the cost of data
+ stored in Arvados, etc.
+
+ - When provided with a project uuid, subprojects will not be considered.
+
In order to get the data for the uuids supplied, the ARVADOS_API_HOST and
ARVADOS_API_TOKEN environment variables must be set.
})
return nil
}
-
- return fmt.Errorf("Unsupported file type (mode %o) in output dir: %q", fi.Mode(), src)
+ cp.logger.Printf("Skipping unsupported file type (mode %o) in output dir: %q", fi.Mode(), src)
+ return nil
}
// Return the host path that was mounted at the given path in the
package crunchrun
import (
+ "bytes"
"io"
"io/ioutil"
"os"
+ "syscall"
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/arvadosclient"
"git.arvados.org/arvados.git/sdk/go/arvadostest"
+ "github.com/sirupsen/logrus"
check "gopkg.in/check.v1"
)
var _ = check.Suite(&copierSuite{})
type copierSuite struct {
- cp copier
+ cp copier
+ log bytes.Buffer
}
func (s *copierSuite) SetUpTest(c *check.C) {
- tmpdir, err := ioutil.TempDir("", "crunch-run.test.")
- c.Assert(err, check.IsNil)
+ tmpdir := c.MkDir()
api, err := arvadosclient.MakeArvadosClient()
c.Assert(err, check.IsNil)
+ s.log = bytes.Buffer{}
s.cp = copier{
client: arvados.NewClientFromEnv(),
arvClient: api,
secretMounts: map[string]arvados.Mount{
"/secret_text": {Kind: "text", Content: "xyzzy"},
},
+ logger: &logrus.Logger{Out: &s.log, Formatter: &logrus.TextFormatter{}, Level: logrus.InfoLevel},
}
}
-func (s *copierSuite) TearDownTest(c *check.C) {
- os.RemoveAll(s.cp.hostOutputDir)
-}
-
func (s *copierSuite) TestEmptyOutput(c *check.C) {
err := s.cp.walkMount("", s.cp.ctrOutputDir, 10, true)
c.Check(err, check.IsNil)
_, err = io.WriteString(f, "foo")
c.Assert(err, check.IsNil)
c.Assert(f.Close(), check.IsNil)
+ err = syscall.Mkfifo(s.cp.hostOutputDir+"/dir1/fifo", 0644)
+ c.Assert(err, check.IsNil)
err = s.cp.walkMount("", s.cp.ctrOutputDir, 10, true)
c.Check(err, check.IsNil)
{src: os.DevNull, dst: "/dir1/dir2/dir3/.keep"},
{src: s.cp.hostOutputDir + "/dir1/foo", dst: "/dir1/foo", size: 3},
})
+ c.Check(s.log.String(), check.Matches, `.* msg="Skipping unsupported file type \(mode 200000644\) in output dir: \\"/ctr/outdir/dir1/fifo\\""\n`)
}
func (s *copierSuite) TestSymlinkCycle(c *check.C) {
EmailClaim string
EmailVerifiedClaim string
UsernameClaim string
+ AcceptAccessToken bool
+ AcceptAccessTokenScope string
AuthenticationRequestParameters map[string]string
}
PAM struct {
"gopkg.in/check.v1"
"gopkg.in/square/go-jose.v2"
+ "gopkg.in/square/go-jose.v2/jwt"
)
type OIDCProvider struct {
ValidClientID string
ValidClientSecret string
// desired response from token endpoint
- AuthEmail string
- AuthEmailVerified bool
- AuthName string
+ AuthEmail string
+ AuthEmailVerified bool
+ AuthName string
+ AccessTokenPayload map[string]interface{}
PeopleAPIResponse map[string]interface{}
c.Assert(err, check.IsNil)
p.Issuer = httptest.NewServer(http.HandlerFunc(p.serveOIDC))
p.PeopleAPI = httptest.NewServer(http.HandlerFunc(p.servePeopleAPI))
+ p.AccessTokenPayload = map[string]interface{}{"sub": "example"}
return p
}
func (p *OIDCProvider) ValidAccessToken() string {
- return p.fakeToken([]byte("fake access token"))
+ buf, _ := json.Marshal(p.AccessTokenPayload)
+ return p.fakeToken(buf)
}
func (p *OIDCProvider) serveOIDC(w http.ResponseWriter, req *http.Request) {
case "/auth":
w.WriteHeader(http.StatusInternalServerError)
case "/userinfo":
- if authhdr := req.Header.Get("Authorization"); strings.TrimPrefix(authhdr, "Bearer ") != p.ValidAccessToken() {
+ authhdr := req.Header.Get("Authorization")
+ if _, err := jwt.ParseSigned(strings.TrimPrefix(authhdr, "Bearer ")); err != nil {
p.c.Logf("OIDCProvider: bad auth %q", authhdr)
w.WriteHeader(http.StatusUnauthorized)
return
mimemagic (0.3.8)
nokogiri (~> 1)
mini_mime (1.0.2)
- mini_portile2 (2.5.0)
+ mini_portile2 (2.5.1)
minitest (5.10.3)
mocha (1.8.0)
metaclass (~> 0.0.1)
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
nio4r (2.5.7)
- nokogiri (1.11.2)
+ nokogiri (1.11.5)
mini_portile2 (~> 2.5.0)
racc (~> 1.4)
oauth2 (1.4.1)
},
})
if err == nil {
- c.collections.Add(client.AuthToken+"\000"+coll.PortableDataHash, &cachedCollection{
+ c.collections.Add(client.AuthToken+"\000"+updated.PortableDataHash, &cachedCollection{
expire: time.Now().Add(time.Duration(c.config.TTL)),
collection: &updated,
})
c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Equals, 42)
}
+// Writing to a collection shouldn't affect its entry in the
+// PDH-to-manifest cache.
+func (s *IntegrationSuite) TestCacheWriteCollectionSamePDH(c *check.C) {
+ arv, err := arvadosclient.MakeArvadosClient()
+ c.Assert(err, check.Equals, nil)
+ arv.ApiToken = arvadostest.ActiveToken
+
+ u := mustParseURL("http://x.example/testfile")
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{"Authorization": {"Bearer " + arv.ApiToken}},
+ }
+
+ checkWithID := func(id string, status int) {
+ req.URL.Host = strings.Replace(id, "+", "-", -1) + ".example"
+ req.Host = req.URL.Host
+ resp := httptest.NewRecorder()
+ s.testServer.Handler.ServeHTTP(resp, req)
+ c.Check(resp.Code, check.Equals, status)
+ }
+
+ var colls [2]arvados.Collection
+ for i := range colls {
+ err := arv.Create("collections",
+ map[string]interface{}{
+ "ensure_unique_name": true,
+ "collection": map[string]interface{}{
+ "name": "test collection",
+ },
+ }, &colls[i])
+ c.Assert(err, check.Equals, nil)
+ }
+
+ // Populate cache with empty collection
+ checkWithID(colls[0].PortableDataHash, http.StatusNotFound)
+
+ // write a file to colls[0]
+ reqPut := *req
+ reqPut.Method = "PUT"
+ reqPut.URL.Host = colls[0].UUID + ".example"
+ reqPut.Host = req.URL.Host
+ reqPut.Body = ioutil.NopCloser(bytes.NewBufferString("testdata"))
+ resp := httptest.NewRecorder()
+ s.testServer.Handler.ServeHTTP(resp, &reqPut)
+ c.Check(resp.Code, check.Equals, http.StatusCreated)
+
+ // new file should not appear in colls[1]
+ checkWithID(colls[1].PortableDataHash, http.StatusNotFound)
+ checkWithID(colls[1].UUID, http.StatusNotFound)
+
+ checkWithID(colls[0].UUID, http.StatusOK)
+}
+
func copyHeader(h http.Header) http.Header {
hc := http.Header{}
for k, v := range h {
* CLUSTER.DOMAIN
* collections.CLUSTER.DOMAIN
-* \*\-\-collections.CLUSTER.DOMAIN
+* \*.collections.CLUSTER.DOMAIN
* download.CLUSTER.DOMAIN
* keep.CLUSTER.DOMAIN
* workbench.CLUSTER.DOMAIN
InternalURLs:
'http://localhost:8004': {}
WebDAV:
- ExternalURL: 'https://*--collections.__CLUSTER__.__DOMAIN__:__KEEPWEB_EXT_SSL_PORT__/'
+ ExternalURL: 'https://*.collections.__CLUSTER__.__DOMAIN__:__KEEPWEB_EXT_SSL_PORT__/'
InternalURLs:
'http://localhost:9002': {}
WebDAVDownload:
use_package: true
pkgs:
- certbot: latest
- - python3-certbot-nginx
+ - python3-certbot-dns-route53
config:
server: https://acme-v02.api.letsencrypt.org/directory
email: __INITIAL_USER_EMAIL__
- authenticator: nginx
- webroot-path: /var/www
+ authenticator: dns-route53
agree-tos: true
keep-until-expiring: true
expand: true
max-log-backups: 0
deploy-hook: systemctl reload nginx
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- ### LETSENCRYPT DEFAULT PATH
- letsencrypt_well_known.conf:
- - location /.well-known:
- - root: /var/www
- download.__CLUSTER__.__DOMAIN__
collections.__CLUSTER__.__DOMAIN__:
- collections.__CLUSTER__.__DOMAIN__
+ - *.collections.__CLUSTER__.__DOMAIN__
### NGINX
nginx:
- server_name: __CLUSTER__.__DOMAIN__
- listen:
- 80 default
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
- server_name: keep.__CLUSTER__.__DOMAIN__
- listen:
- 80
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
overwrite: true
config:
- server:
- - server_name: '~^((.*--)?collections|download)\.__CLUSTER__\.__DOMAIN__'
+ - server_name: '~^((.*\.)?collections|download)\.__CLUSTER__\.__DOMAIN__'
- listen:
- 80
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
cmd: create-initial-cert-collections.__CLUSTER__.__DOMAIN__-collections.__CLUSTER__.__DOMAIN__
config:
- server:
- - server_name: '~^(.*--)?collections\.__CLUSTER__\.__DOMAIN__'
+ - server_name: '*.collections.__CLUSTER__.__DOMAIN__'
- listen:
- __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- server_name: webshell.__CLUSTER__.__DOMAIN__
- listen:
- 80
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
- server_name: ws.__CLUSTER__.__DOMAIN__
- listen:
- 80
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
- server_name: workbench2.__CLUSTER__.__DOMAIN__
- listen:
- 80
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
- server_name: workbench.__CLUSTER__.__DOMAIN__
- listen:
- 80
- - include: snippets/letsencrypt_well_known.conf
- location /:
- return: '301 https://$host$request_uri'
echo "${SOURCE_PILLARS_DIR} does not exist or is not a directory. Exiting."
exit 1
fi
-for f in "${SOURCE_PILLARS_DIR}"/*; do
+for f in $(ls "${SOURCE_PILLARS_DIR}"/*); do
sed "s#__ANONYMOUS_USER_TOKEN__#${ANONYMOUS_USER_TOKEN}#g;
s#__BLOB_SIGNING_KEY__#${BLOB_SIGNING_KEY}#g;
s#__CONTROLLER_EXT_SSL_PORT__#${CONTROLLER_EXT_SSL_PORT}#g;
fi
mkdir -p /tmp/cluster_tests
# Replace cluster and domain name in the test files
-for f in "${SOURCE_TESTS_DIR}"/*; do
+for f in $(ls "${SOURCE_TESTS_DIR}"/*); do
sed "s#__CLUSTER__#${CLUSTER}#g;
s#__CONTROLLER_EXT_SSL_PORT__#${CONTROLLER_EXT_SSL_PORT}#g;
s#__DOMAIN__#${DOMAIN}#g;
if [ -d "${SOURCE_STATES_DIR}" ]; then
mkdir -p "${F_DIR}"/extra/extra
- for f in "${SOURCE_STATES_DIR}"/*; do
+ for f in $(ls "${SOURCE_STATES_DIR}"/*); do
sed "s#__ANONYMOUS_USER_TOKEN__#${ANONYMOUS_USER_TOKEN}#g;
s#__CLUSTER__#${CLUSTER}#g;
s#__BLOB_SIGNING_KEY__#${BLOB_SIGNING_KEY}#g;
# States, extra states
if [ -d "${F_DIR}"/extra/extra ]; then
- for f in "${F_DIR}"/extra/extra/*.sls; do
+ for f in $(ls "${F_DIR}"/extra/extra/*.sls); do
echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
done
fi
grep -q "letsencrypt" ${S_DIR}/top.sls || echo " - letsencrypt" >> ${S_DIR}/top.sls
fi
echo " - postgres" >> ${S_DIR}/top.sls
- echo " - docker" >> ${S_DIR}/top.sls
+ echo " - docker.software" >> ${S_DIR}/top.sls
echo " - arvados" >> ${S_DIR}/top.sls
# Pillars
;;
"shell")
# States
- grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker.software" >> ${S_DIR}/top.sls
grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
# Pillars
grep -q "" ${P_DIR}/top.sls || echo " - docker" >> ${P_DIR}/top.sls
;;
"dispatcher")
# States
- grep -q "docker" ${S_DIR}/top.sls || echo " - docker" >> ${S_DIR}/top.sls
+ grep -q "docker" ${S_DIR}/top.sls || echo " - docker.software" >> ${S_DIR}/top.sls
grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
# Pillars
# ATM, no specific pillar needed
projects / arvados.git / commitdiff