1 // Keep-web provides read-only HTTP access to files stored in Keep. It
2 // serves public data to anonymous and unauthenticated clients, and
3 // serves private data to clients that supply Arvados API tokens. It
4 // can be installed anywhere with access to Keep services, typically
5 // behind a web proxy that supports TLS.
7 // See http://doc.arvados.org/install/install-keep-web.html.
11 // The default configuration file location is
12 // /etc/arvados/keep-web/config.json.
14 // Example configuration file
18 // "APIHost": "zzzzz.arvadosapi.com:443",
23 // "AnonymousTokens":["xxxxxxxxxxxxxxxxxxxx"],
24 // "AttachmentOnlyHost":"",
25 // "TrustAllContent":false
28 // Starting the server
30 // Start a server using the default config file
31 // /etc/arvados/keep-web/config.json:
35 // Start a server using the config file /path/to/config.json:
37 // keep-web -config /path/to/config.json
39 // Proxy configuration
41 // Keep-web does not support SSL natively. Typically, it is installed
42 // behind a proxy like nginx.
44 // Here is an example nginx configuration.
47 // upstream keep-web {
48 // server localhost:1234;
52 // server_name collections.example.com *.collections.example.com ~.*--collections.example.com;
53 // ssl_certificate /root/wildcard.example.com.crt;
54 // ssl_certificate_key /root/wildcard.example.com.key;
56 // proxy_pass http://keep-web;
57 // proxy_set_header Host $host;
58 // proxy_set_header X-Forwarded-For $remote_addr;
63 // It is not necessary to run keep-web on the same host as the nginx
64 // proxy. However, TLS is not used between nginx and keep-web, so
65 // intervening networks must be secured by other means.
67 // Anonymous downloads
69 // The "AnonymousTokens" configuration entry is an array of tokens to
70 // use when processing anonymous requests, i.e., whenever a web client
71 // does not supply its own Arvados API token via path, query string,
72 // cookie, or request header.
74 // "AnonymousTokens":["xxxxxxxxxxxxxxxxxxxxxxx"]
76 // See http://doc.arvados.org/install/install-keep-web.html for examples.
80 // The following "same origin" URL patterns are supported for public
81 // collections and collections shared anonymously via secret links
82 // (i.e., collections which can be served by keep-web without making
83 // use of any implicit credentials like cookies). See "Same-origin
86 // http://collections.example.com/c=uuid_or_pdh/path/file.txt
87 // http://collections.example.com/c=uuid_or_pdh/t=TOKEN/path/file.txt
89 // The following "multiple origin" URL patterns are supported for all
92 // http://uuid_or_pdh--collections.example.com/path/file.txt
93 // http://uuid_or_pdh--collections.example.com/t=TOKEN/path/file.txt
95 // In the "multiple origin" form, the string "--" can be replaced with
96 // "." with identical results (assuming the downstream proxy is
97 // configured accordingly). These two are equivalent:
99 // http://uuid_or_pdh--collections.example.com/path/file.txt
100 // http://uuid_or_pdh.collections.example.com/path/file.txt
102 // The first form (with "--" instead of ".") avoids the cost and
103 // effort of deploying a wildcard TLS certificate for
104 // *.collections.example.com at sites that already have a wildcard
105 // certificate for *.example.com. The second form is likely to be
106 // easier to configure, and more efficient to run, on a downstream
109 // In all of the above forms, the "collections.example.com" part can
110 // be anything at all: keep-web itself ignores everything after the
111 // first "." or "--". (Of course, in order for clients to connect at
112 // all, DNS and any relevant proxies must be configured accordingly.)
114 // In all of the above forms, the "uuid_or_pdh" part can be either a
115 // collection UUID or a portable data hash with the "+" character
116 // optionally replaced by "-". (When "uuid_or_pdh" appears in the
117 // domain name, replacing "+" with "-" is mandatory, because "+" is
118 // not a valid character in a domain name.)
120 // In all of the above forms, a top level directory called "_" is
121 // skipped. In cases where the "path/file.txt" part might start with
122 // "t=" or "c=" or "_/", links should be constructed with a leading
123 // "_/" to ensure the top level directory is not interpreted as a
124 // token or collection ID.
126 // Assuming there is a collection with UUID
127 // zzzzz-4zz18-znfnqtbbv4spc3w and portable data hash
128 // 1f4b0bc7583c2a7f9102c395f4ffc5e3+45, the following URLs are
131 // http://zzzzz-4zz18-znfnqtbbv4spc3w.collections.example.com/foo/bar.txt
132 // http://zzzzz-4zz18-znfnqtbbv4spc3w.collections.example.com/_/foo/bar.txt
133 // http://zzzzz-4zz18-znfnqtbbv4spc3w--collections.example.com/_/foo/bar.txt
134 // http://1f4b0bc7583c2a7f9102c395f4ffc5e3-45--foo.example.com/foo/bar.txt
135 // http://1f4b0bc7583c2a7f9102c395f4ffc5e3-45--.invalid/foo/bar.txt
137 // An additional form is supported specifically to make it more
138 // convenient to maintain support for existing Workbench download
141 // http://collections.example.com/collections/download/uuid_or_pdh/TOKEN/foo/bar.txt
143 // A regular Workbench "download" link is also accepted, but
144 // credentials passed via cookie, header, etc. are ignored. Only
145 // public data can be served this way:
147 // http://collections.example.com/collections/uuid_or_pdh/foo/bar.txt
149 // Authorization mechanisms
151 // A token can be provided in an Authorization header:
153 // Authorization: OAuth2 o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK
155 // A base64-encoded token can be provided in a cookie named "api_token":
157 // Cookie: api_token=bzA3ajRweDdSbEpLNEN1TVlwN0MwTERUNEN6UjFKMXFCRTVBdm83ZUNjVWpPVGlreEs=
159 // A token can be provided in an URL-encoded query string:
161 // GET /foo/bar.txt?api_token=o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK
163 // A suitably encoded token can be provided in a POST body if the
164 // request has a content type of application/x-www-form-urlencoded or
165 // multipart/form-data:
168 // Content-Type: application/x-www-form-urlencoded
170 // api_token=o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK
172 // If a token is provided in a query string or in a POST request, the
173 // response is an HTTP 303 redirect to an equivalent GET request, with
174 // the token stripped from the query string and added to a cookie
179 // Currently, keep-web does not generate HTML index listings, nor does
180 // it serve a default file like "index.html" when a directory is
181 // requested. These features are likely to be added in future
182 // versions. Until then, keep-web responds with 404 if a directory
183 // name (or any path ending with "/") is requested.
187 // Client-provided authorization tokens are ignored if the client does
188 // not provide a Host header.
190 // In order to use the query string or a POST form authorization
191 // mechanisms, the client must follow 303 redirects; the client must
192 // accept cookies with a 303 response and send those cookies when
193 // performing the redirect; and either the client or an intervening
194 // proxy must resolve a relative URL ("//host/path") if given in a
195 // response Location header.
199 // Normally, Keep-web accepts requests for multiple collections using
200 // the same host name, provided the client's credentials are not being
201 // used. This provides insufficient XSS protection in an installation
202 // where the "anonymously accessible" data is not truly public, but
203 // merely protected by network topology.
205 // In such cases -- for example, a site which is not reachable from
206 // the internet, where some data is world-readable from Arvados's
207 // perspective but is intended to be available only to users within
208 // the local network -- the downstream proxy should configured to
209 // return 401 for all paths beginning with "/c=".
213 // Without the same-origin protection outlined above, a web page
214 // stored in collection X could execute JavaScript code that uses the
215 // current viewer's credentials to download additional data from
216 // collection Y -- data which is accessible to the current viewer, but
217 // not to the author of collection X -- from the same origin
218 // (``https://collections.example.com/'') and upload it to some other
219 // site chosen by the author of collection X.
221 // Attachment-Only host
223 // It is possible to serve untrusted content and accept user
224 // credentials at the same origin as long as the content is only
225 // downloaded, never executed by browsers. A single origin (hostname
226 // and port) can be designated as an "attachment-only" origin: cookies
227 // will be accepted and all responses will have a
228 // "Content-Disposition: attachment" header. This behavior is invoked
229 // only when the designated origin matches exactly the Host header
230 // provided by the client or downstream proxy.
232 // "AttachmentOnlyHost":"domain.example:9999"
234 // Trust All Content mode
236 // In TrustAllContent mode, Keep-web will accept credentials (API
237 // tokens) and serve any collection X at
238 // "https://collections.example.com/c=X/path/file.ext". This is
239 // UNSAFE except in the special case where everyone who is able write
240 // ANY data to Keep, and every JavaScript and HTML file written to
241 // Keep, is also trusted to read ALL of the data in Keep.
243 // In such cases you can enable trust-all-content mode.
245 // "TrustAllContent":true
247 // When TrustAllContent is enabled, the only effect of the
248 // AttachmentOnlyHost flag is to add a "Content-Disposition:
249 // attachment" header.
251 // "AttachmentOnlyHost":"domain.example:9999",
252 // "TrustAllContent":true
254 // Depending on your site configuration, you might also want to enable
255 // the "trust all content" setting in Workbench. Normally, Workbench
256 // avoids redirecting requests to keep-web if they depend on
257 // TrustAllContent being enabled.