# "Introduction":#introduction
# "Configure DNS":#introduction
-# "Configure anonymous user token.yml":#update-config
+# "Configure anonymous user token":#update-config
# "Update nginx configuration":#update-nginx
# "Install keep-web package":#install-packages
# "Start the service":#start-service
h2(#introduction). Introduction
-The Keep-web server provides read/write HTTP (WebDAV) access to files stored in Keep. This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, typically behind a web proxy that provides TLS support. See the "godoc page":http://godoc.org/github.com/curoverse/arvados/services/keep-web for more detail.
+The Keep-web server provides read/write access to files stored in Keep using WebDAV and S3 protocols. This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, typically behind a web proxy that provides TLS support. See the "godoc page":https://pkg.go.dev/git.arvados.org/arvados.git/services/keep-web for more detail.
h2(#dns). Configure DNS
There are two approaches to mitigate this.
# The service can tell the browser that all files should go to download instead of in-browser preview, except in situations where an attacker is unlikely to be able to gain access to anything they didn't already have access to.
-# Each each collection served by @keep-web@ is served on its own virtual host. This allows for file with executable content to be displayed in-browser securely. The virtual host embeds the collection uuid or portable data hash in the hostname. For example, a collection with uuid @xxxxx-4zz18-tci4vn4fa95w0zx@ could be served as @xxxxx-4zz18-tci4vn4fa95w0zx.collections.ClusterID.example.com@ . The portable data hash @dd755dbc8d49a67f4fe7dc843e4f10a6+54@ could be served at @dd755dbc8d49a67f4fe7dc843e4f10a6-54.collections.ClusterID.example.com@ . This requires "wildcard DNS record":https://en.wikipedia.org/wiki/Wildcard_DNS_record and "wildcard TLS certificate.":https://en.wikipedia.org/wiki/Wildcard_certificate
+# Each collection served by @keep-web@ is served on its own virtual host. This allows for file with executable content to be displayed in-browser securely. The virtual host embeds the collection uuid or portable data hash in the hostname. For example, a collection with uuid @xxxxx-4zz18-tci4vn4fa95w0zx@ could be served as @xxxxx-4zz18-tci4vn4fa95w0zx.collections.ClusterID.example.com@ . The portable data hash @dd755dbc8d49a67f4fe7dc843e4f10a6+54@ could be served at @dd755dbc8d49a67f4fe7dc843e4f10a6-54.collections.ClusterID.example.com@ . This requires "wildcard DNS record":https://en.wikipedia.org/wiki/Wildcard_DNS_record and "wildcard TLS certificate.":https://en.wikipedia.org/wiki/Wildcard_certificate
h3. Collections download URL
-Downloads links will served from the the URL in @Services.WebDAVDownload.ExternalURL@ . The collection uuid or PDH is put in the URL path.
+Downloads links will served from the URL in @Services.WebDAVDownload.ExternalURL@ . The collection uuid or PDH is put in the URL path.
If blank, serve links to WebDAV with @disposition=attachment@ query param. Unlike preview links, browsers do not render attachments, so there is no risk of XSS.
<notextile>
<pre><code> Services:
WebDAVDownload:
- ExternalURL: <span class="userinput">https://download.ClusterID.example.com</span>
+ ExternalURL: https://<span class="userinput">download.ClusterID.example.com</span>
</code></pre>
</notextile>
h3. Collections preview URL
-Collections will be served using the URL pattern in @Services.WebDAV.ExternalURL@ . If blank, use @Services.WebDAVDownload.ExternalURL@ instead, and disable inline preview. If both are empty, downloading collections from workbench will be impossible.
+Collections will be served using the URL pattern in @Services.WebDAV.ExternalURL@ . If blank, use @Services.WebDAVDownload.ExternalURL@ instead, and disable inline preview. If both are empty, downloading collections from workbench will be impossible. When wildcard domains configured, credentials are still required to access non-public data.
h4. In their own subdomain
<notextile>
<pre><code> Services:
WebDAV:
- ExternalURL: <span class="userinput">https://*.collections.ClusterID.example.com</span>
+ ExternalURL: https://<span class="userinput">*.collections.ClusterID.example.com/</span>
</code></pre>
</notextile>
+This option is preferred if you plan to access Keep using third-party S3 client software, because it accommodates S3 virtual host-style requests and path-style requests without any special client configuration.
+
h4. Under the main domain
Alternately, they can go under the main domain by including @--@:
<notextile>
<pre><code> Services:
WebDAV:
- ExternalURL: <span class="userinput">https://*--collections.ClusterID.example.com</span>
+ ExternalURL: https://<span class="userinput">*--collections.ClusterID.example.com/</span>
</code></pre>
</notextile>
h4. From a single domain
-Serve preview links from a single domain, setting uuid or pdh in the path (similar to downloads). This configuration only allows previews of public data or collection-sharing links, because these use the anonymous user token or the token is already embedded in the URL. Authenticated requests will always result in file downloads from @Services.WebDAVDownload.ExternalURL@.
+Serve preview links from a single domain, setting uuid or pdh in the path (similar to downloads). This configuration only allows previews of public data (data accessible by the anonymous user) and collection-sharing links (where the token is already embedded in the URL); it will ignore authorization headers, so a request for non-public data may return "404 Not Found" even if normally valid credentials were provided.
<notextile>
<pre><code> Services:
WebDAV:
- ExternalURL: <span class="userinput">https://collections.ClusterID.example.com/</span>
+ ExternalURL: https://<span class="userinput">collections.ClusterID.example.com/</span>
</code></pre>
</notextile>
Note the trailing slash.
+{% include 'notebox_begin' %}
+Whether you choose to serve collections from their own subdomain or from a single domain, it's important to keep in mind that they should be served from me same _site_ as Workbench for the inline previews to work.
+
+Please check "keep-web's URL pattern guide":../api/keep-web-urls.html#same-site to learn more.
+{% include 'notebox_end' %}
+
h2. Set InternalURLs
<notextile>
<pre><code> Services:
WebDAV:
InternalURLs:
- <span class="userinput">"http://localhost:9002"</span>: {}
+ http://<span class="userinput">localhost:9002</span>: {}
</code></pre>
</notextile>
h2(#update-config). Configure anonymous user token
-{% assign railscmd = "bundle exec ./script/get_anonymous_user_token.rb --get" %}
-{% assign railsout = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" %}
-If you intend to use Keep-web to serve public data to anonymous clients, configure it with an anonymous token. Use the following command on the <strong>API server</strong> to create an anonymous user token. {% include 'install_rails_command' %}
+If you intend to use Keep-web to serve public data to anonymous clients, configure it with an anonymous token.
+
+Generate a random string (>= 32 characters long) and put it in the @config.yml@ file, in the @AnonymousUserToken@ field.
<notextile>
<pre><code> Users:
- AnonymousUserToken: <span class="userinput">"{{railsout}}"</span>
+ AnonymousUserToken: <span class="userinput">"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"</span>
</code></pre>
</notextile>
h3. Update nginx configuration
-Put a reverse proxy with SSL support in front of keep-web. Keep-web itself runs on the port 25107 (or whatever is specified in @Services.Keepproxy.InternalURL@) the reverse proxy runs on port 443 and forwards requests to Keepproxy.
+Put a reverse proxy with SSL support in front of keep-web. Keep-web itself runs on the port 9002 (or whatever is specified in @Services.WebDAV.InternalURL@) while the reverse proxy runs on port 443 and forwards requests to Keep-web.
-Use a text editor to create a new file @/etc/nginx/conf.d/keep-web.conf@ with the following configuration. Options that need attention are marked with “TODO”.
+Use a text editor to create a new file @/etc/nginx/conf.d/keep-web.conf@ with the following configuration. Options that need attention are marked in <span class="userinput">red</span>.
<notextile><pre>
upstream keep-web {
}
server {
- listen *:443 ssl;
- server_name download.<span class="userinput">ClusterID</span>.example.com
- collections.<span class="userinput">ClusterID</span>.example.com
- *.collections.<span class="userinput">ClusterID</span>.example.com
- ~.*--collections.<span class="userinput">ClusterID</span>.example.com;
+ listen 443 ssl;
+ server_name <span class="userinput">download.ClusterID.example.com</span>
+ <span class="userinput">collections.ClusterID.example.com</span>
+ <span class="userinput">*.collections.ClusterID.example.com</span>
+ <span class="userinput">~.*--collections.ClusterID.example.com</span>;
proxy_connect_timeout 90s;
proxy_read_timeout 300s;
ssl on;
- ssl_certificate <span class="userinput">/TODO/YOUR/PATH/TO/cert.pem</span>;
- ssl_certificate_key <span class="userinput">/TODO/YOUR/PATH/TO/cert.key</span>;
+ ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
+ ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
location / {
proxy_pass http://keep-web;
client_max_body_size 0;
proxy_http_version 1.1;
proxy_request_buffering off;
+ proxy_max_temp_file_size 0;
}
}
</pre></notextile>
h2(#confirm-working). Confirm working installation
-<pre>
-$ curl -H "Authorization: Bearer $system_root_token" https://download.ClusterID.example.com/c=59389a8f9ee9d399be35462a0f92541c-53/_/hello.txt
-</pre>
+<notextile><code><pre>
+$ curl -H "Authorization: Bearer $system_root_token" https://<span class="userinput">download.ClusterID.example.com</span>/c=59389a8f9ee9d399be35462a0f92541c-53/_/hello.txt
+</code></pre></notextile>
If wildcard collections domains are configured:
-<pre>
-$ curl -H "Authorization: Bearer $system_root_token" https://59389a8f9ee9d399be35462a0f92541c-53.collections.ClusterID.example.com/hello.txt
-</pre>
+<notextile><code><pre>
+$ curl -H "Authorization: Bearer $system_root_token" https://<span class="userinput">59389a8f9ee9d399be35462a0f92541c-53.collections.ClusterID.example.com</span>/hello.txt
+</code></pre></notextile>
If using a single collections preview domain:
-<pre>
-$ curl https://collections.ClusterID.example.com/c=59389a8f9ee9d399be35462a0f92541c-53/t=$system_root_token/_/hello.txt
-</pre>
+<notextile><code><pre>
+$ curl https://<span class="userinput">collections.ClusterID.example.com</span>/c=59389a8f9ee9d399be35462a0f92541c-53/t=$system_root_token/_/hello.txt
+</code></pre></notextile>
projects / arvados.git / blobdiff