---
layout: default
navsection: installguide
-title: Install the keep-web server
+title: Install Keep-web server
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
-The keep-web server provides read-only HTTP access to files stored in Keep. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, typically behind a web proxy that provides SSL support. See the "godoc page":http://godoc.org/github.com/curoverse/arvados/services/keep-web for more detail.
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
-By convention, we use the following hostname for the keep-web service:
+# "Introduction":#introduction
+# "Configure DNS":#introduction
+# "Configure anonymous user token.yml":#update-config
+# "Update nginx configuration":#update-nginx
+# "Install keep-web package":#install-packages
+# "Start the service":#start-service
+# "Restart the API server and controller":#restart-api
+# "Confirm working installation":#confirm-working
-<notextile>
-<pre><code>collections.<span class="userinput">uuid_prefix</span>.your.domain
-</code></pre>
-</notextile>
+h2(#introduction). Introduction
-This hostname should resolve from anywhere on the internet.
+The Keep-web server provides read/write HTTP (WebDAV) access to files stored in Keep. This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, typically behind a web proxy that provides TLS support. See the "godoc page":http://godoc.org/github.com/curoverse/arvados/services/keep-web for more detail.
-h2. Install keep-web
+h2(#dns). Configure DNS
-On Debian-based systems:
+It is important to properly configure the keep-web service to so it does not open up cross-site-scripting (XSS) attacks. A HTML file can be stored in collection. If an attacker causes a victim to visit that HTML file through Workbench, it will be rendered by the browser. If all collections are served at the same domain, the browser will consider collections as coming from the same origin and thus have access to the same browsing data (such as API token), enabling malicious Javascript in the HTML file to access Arvados as the victim.
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install keep-web</span>
-</code></pre>
-</notextile>
+There are two approaches to mitigate this.
-On Red Hat-based systems:
+# The service can tell the browser that all files should go to download instead of in-browser preview, except in situations where an attacker is unlikely to be able to gain access to anything they didn't already have access to.
+# Each each collection served by @keep-web@ is served on its own virtual host. This allows for file with executable content to be displayed in-browser securely. The virtual host embeds the collection uuid or portable data hash in the hostname. For example, a collection with uuid @xxxxx-4zz18-tci4vn4fa95w0zx@ could be served as @xxxxx-4zz18-tci4vn4fa95w0zx.collections.ClusterID.example.com@ . The portable data hash @dd755dbc8d49a67f4fe7dc843e4f10a6+54@ could be served at @dd755dbc8d49a67f4fe7dc843e4f10a6-54.collections.ClusterID.example.com@ . This requires "wildcard DNS record":https://en.wikipedia.org/wiki/Wildcard_DNS_record and "wildcard TLS certificate.":https://en.wikipedia.org/wiki/Wildcard_certificate
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install keep-web</span>
-</code></pre>
-</notextile>
+h3. Collections download URL
-Verify that @keep-web@ is functional:
+Downloads links will served from the the URL in @Services.WebDAVDownload.ExternalURL@ . The collection uuid or PDH is put in the URL path.
-<notextile>
-<pre><code>~$ <span class="userinput">keep-web -h</span>
-Usage of keep-web:
- -address string
- Address to listen on: "host:port", or ":port" to listen on all interfaces. (default ":80")
- -anonymous-token value
- API token to try when none of the tokens provided in an HTTP request succeed in reading the desired collection. If this flag is used more than once, each token will be attempted in turn until one works. (default [])
- -attachment-only-host string
- Accept credentials, and add "Content-Disposition: attachment" response headers, for requests at this hostname:port. Prohibiting inline display makes it possible to serve untrusted and non-public content from a single origin, i.e., without wildcard DNS or SSL.
- -trust-all-content
- Serve non-public content from a single origin. Dangerous: read docs before using!
-</code></pre>
-</notextile>
+If blank, serve links to WebDAV with @disposition=attachment@ query param. Unlike preview links, browsers do not render attachments, so there is no risk of XSS.
-If you intend to use Keep-web to serve public data to anonymous clients, configure it with an anonymous token. You can use the same one you used when you set up your Keepproxy server, or use the following command on the <strong>API server</strong> to create another:
+If @WebDAVDownload@ is blank, and @WebDAV@ has a single origin (not wildcard, see below), then Workbench will show an error page
-<notextile>
-<pre><code>/var/www/arvados-api/current/script$ <span class="userinput">RAILS_ENV=production bundle exec ./get_anonymous_user_token.rb</span>
-hoShoomoo2bai3Ju1xahg6aeng1siquuaZ1yae2gi2Uhaeng2r
-</code></pre></notextile>
+<pre>
+ Services:
+ WebDAVDownload:
+ ExternalURL: https://download.ClusterID.example.com
+</pre>
+
+h3. Collections preview URL
+
+Collections will be served using the URL pattern in @Services.WebDAV.ExternalURL@ . If blank, use @Services.WebDAVDownload.ExternalURL@ instead, and disable inline preview. If both are empty, downloading collections from workbench will be impossible.
+
+h4. In their own subdomain
+
+Collections can be served from their own subdomain:
+
+<pre>
+ Services:
+ WebDAV:
+ ExternalURL: https://*.collections.ClusterID.example.com
+</pre>
+
+h4. Under the main domain
-We recommend running @keep-web@ under "runit":https://packages.debian.org/search?keywords=runit or a similar supervisor. The basic command to start @keep-web@ is:
+Alternately, they can go under the main domain by including @--@:
+
+<pre>
+ Services:
+ WebDAV:
+ ExternalURL: https://*--collections.ClusterID.example.com
+</pre>
+
+h4. From a single domain
+
+Serve preview links from a single domain, setting uuid or pdh in the path (similar to downloads). This configuration only allows previews of public data or collection-sharing links, because these use the anonymous user token or the token is already embedded in the URL. Authenticated requests will always result in file downloads from @Services.WebDAVDownload.ExternalURL@.
+
+<pre>
+ Services:
+ WebDAV:
+ ExternalURL: https://collections.ClusterID.example.com
+</pre>
+
+h2(#update-config). Configure anonymous user token
+
+{% assign railscmd = "bundle exec ./script/get_anonymous_user_token.rb --get" %}
+{% assign railsout = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" %}
+If you intend to use Keep-web to serve public data to anonymous clients, configure it with an anonymous token. Use the following command on the <strong>API server</strong> to create an anonymous user token. {% include 'install_rails_command' %}
<notextile>
-<pre><code>export ARVADOS_API_HOST=<span class="userinput">uuid_prefix</span>.your.domain
-exec sudo -u nobody keep-web -address=<span class="userinput">:9002</span> -anonymous-token=<span class="userinput">hoShoomoo2bai3Ju1xahg6aeng1siquuaZ1yae2gi2Uhaeng2r</span> 2>&1
+<pre><code> Users:
+ AnonymousUserToken: "{{railsout}}"
</code></pre>
</notextile>
-Omit the @-anonymous-token@ arguments if you do not want to serve public data.
-
-Set @ARVADOS_API_HOST_INSECURE=1@ if your API server's SSL certificate is not signed by a recognized CA.
+Set @Users.AnonymousUserToken: ""@ (empty string) or leave it out if you do not want to serve public data.
-h3. Set up a reverse proxy with SSL support
+h2. Set InternalURL
-The keep-web service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption.
+<pre>
+ Services:
+ WebDAV:
+ InternalURL:
+ "http://collections.ClusterID.example.com:9002": {}
+</pre>
-This is best achieved by putting a reverse proxy with SSL support in front of keep-web, running on port 443 and passing requests to keep-web on port 9002 (or whatever port you chose in your run script).
+h3. Update nginx configuration
-Note: A wildcard SSL certificate is required in order to proxy keep-web effectively.
+Put a reverse proxy with SSL support in front of keep-web. Keep-web itself runs on the port 25107 (or whatever is specified in @Services.Keepproxy.InternalURL@) the reverse proxy runs on port 443 and forwards requests to Keepproxy.
-For example, using Nginx:
+Use a text editor to create a new file @/etc/nginx/conf.d/keep-web.conf@ with the following configuration. Options that need attention are marked with “TODO”.
<notextile><pre>
upstream keep-web {
}
server {
- listen <span class="userinput">[your public IP address]</span>:443 ssl;
- server_name collections.<span class="userinput">uuid_prefix</span>.your.domain *.collections.<span class="userinput">uuid_prefix</span>.your.domain ~.*--collections.<span class="userinput">uuid_prefix</span>.your.domain;
+ listen <span class="userinput">[TODO: your public IP address]</span>:443 ssl;
+ server_name download.<span class="userinput">ClusterID</span>.example.com
+ collections.<span class="userinput">ClusterID</span>.example.com
+ *.collections.<span class="userinput">ClusterID</span>.example.com
+ ~.*--collections.<span class="userinput">ClusterID</span>.example.com;
proxy_connect_timeout 90s;
proxy_read_timeout 300s;
ssl on;
- ssl_certificate <span class="userinput"/>YOUR/PATH/TO/cert.pem</span>;
- ssl_certificate_key <span class="userinput"/>YOUR/PATH/TO/cert.key</span>;
+ ssl_certificate <span class="userinput">/TODO/YOUR/PATH/TO/cert.pem</span>;
+ ssl_certificate_key <span class="userinput">/TODO/YOUR/PATH/TO/cert.key</span>;
location / {
proxy_pass http://keep-web;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+ client_max_body_size 0;
+ proxy_http_version 1.1;
+ proxy_request_buffering off;
}
}
</pre></notextile>
-h3. Configure DNS
+{% include 'notebox_begin' %}
+If you restrict access to your Arvados services based on network topology -- for example, your proxy server is not reachable from the public internet -- additional proxy configuration might be needed to thwart cross-site scripting attacks that would circumvent your restrictions.
-Configure your DNS servers so the following names resolve to your Nginx proxy's public IP address.
-* @*--collections.uuid_prefix.your.domain@, if your DNS server allows this without interfering with other DNS names; or
-* @*.collections.uuid_prefix.your.domain@, if you have a wildcard SSL certificate valid for these names; or
-* @collections.uuid_prefix.your.domain@, if neither of the above options is feasible. In this case, only unauthenticated requests will be served, i.e., public data and collection sharing links.
+Normally, Keep-web accepts requests for multiple collections using the same host name, provided the client's credentials are not being used. This provides insufficient XSS protection in an installation where the "anonymously accessible" data is not truly public, but merely protected by network topology.
-h3. Tell Workbench about the keep-web service
+In such cases -- for example, a site which is not reachable from the internet, where some data is world-readable from Arvados's perspective but is intended to be available only to users within the local network -- the downstream proxy should configured to return 401 for all paths beginning with "/c="
+{% include 'notebox_end' %}
-Add *one* of the following entries to your Workbench configuration file (@/etc/arvados/workbench/application.yml@), depending on your DNS setup:
+{% assign arvados_component = 'keep-web' %}
-<notextile>
-<pre><code>keep_web_url: https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://%{uuid_or_pdh}.collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://collections.<span class="userinput">uuid_prefix</span>.your.domain
-</code></pre>
-</notextile>
+{% include 'install_packages' %}
+
+{% include 'start_service' %}
+
+{% include 'restart_api' %}
+
+h2(#confirm-working). Confirm working installation
+
+Adjust for your configuration.
+
+<pre>
+$ curl -H "Authorization: Bearer $system_root_token" https://download.ClusterID.example.com/c=59389a8f9ee9d399be35462a0f92541c-53/_/hello.txt
+</pre>
+
+<pre>
+$ curl -H "Authorization: Bearer $system_root_token" https://collections.ClusterID.example.com/c=59389a8f9ee9d399be35462a0f92541c-53/_/hello.txt
+</pre>
+
+<pre>
+$ curl -H "Authorization: Bearer $system_root_token" https://59389a8f9ee9d399be35462a0f92541c-53.collections.ClusterID.example.com/hello.txt
+</pre>
projects / arvados.git / blobdiff