14716: Updates documentation, add test check.
authorLucas Di Pentima <ldipentima@veritasgenetics.com>
Mon, 5 Aug 2019 18:44:34 +0000 (15:44 -0300)
committerLucas Di Pentima <ldipentima@veritasgenetics.com>
Mon, 5 Aug 2019 18:44:34 +0000 (15:44 -0300)
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

doc/install/install-keep-web.html.textile.liquid
services/keep-web/doc.go
services/keep-web/server_test.go

index 2991d7b0dc2df95aef92aaf191856067d3362adf..08dc186e8da41ec89cc834dfbc3c4cc56262234b 100644 (file)
@@ -45,14 +45,22 @@ Verify that @Keep-web@ is functional:
 <notextile>
 <pre><code>~$ <span class="userinput">keep-web -h</span>
 Usage of keep-web:
-  -allow-anonymous
-        Serve public data to anonymous clients. Try the token supplied in the ARVADOS_API_TOKEN environment variable when none of the tokens provided in an HTTP request succeed in reading the desired collection. (default false)
-  -attachment-only-host string
-        Accept credentials, and add "Content-Disposition: attachment" response headers, for requests at this hostname:port. Prohibiting inline display makes it possible to serve untrusted and non-public content from a single origin, i.e., without wildcard DNS or TLS.
-  -listen string
-        Address to listen on: "host:port", or ":port" to listen on all interfaces. (default ":80")
-  -trust-all-content
-        Serve non-public content from a single origin. Dangerous: read docs before using!
+  -config file
+       Site configuration file (default may be overridden by setting an ARVADOS_CONFIG environment variable) (default "/etc/arvados/config.yml")
+  -dump-config
+       write current configuration to stdout and exit
+  -legacy-crunch-dispatch-slurm-config file
+       Legacy crunch-dispatch-slurm configuration file (default "/etc/arvados/crunch-dispatch-slurm/crunch-dispatch-slurm.yml")
+  -legacy-keepstore-config file
+       Legacy keepstore configuration file (default "/etc/arvados/keepstore/keepstore.yml")
+  -legacy-keepweb-config file
+       Legacy keep-web configuration file (default "/etc/arvados/keep-web/keep-web.yml")
+  -legacy-ws-config file
+       Legacy arvados-ws configuration file (default "/etc/arvados/ws/ws.yml")
+  -skip-legacy
+       Don't load legacy config files
+  -version
+       print version information and exit.
 </code></pre>
 </notextile>
 
@@ -62,22 +70,40 @@ If you intend to use Keep-web to serve public data to anonymous clients, configu
 
 Install runit to supervise the Keep-web daemon.  {% include 'install_runit' %}
 
+Set the cluster config file like the following:
+
+<notextile>
+<pre><code>Clusters:
+  <span class="userinput">uuid_prefix</span>:
+    SystemRootToken: "{{railsout}}"
+    Services:
+      Controller:
+        ExternalURL: "https://<span class="userinput">uuid_prefix</span>.your.domain"
+        Insecure: false
+      WebDAV:
+        InternalURLs:
+          "http://:9002/": {}
+      WebDAVDownload:
+        InternalURLs:
+          "http://:9002/": {}
+        ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/"
+    Users:
+      AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxx"
+    Collections:
+      TrustAllContent: false
+</code></pre>
+</notextile>
+
 The basic command to start Keep-web in the service run script is:
 
 <notextile>
-<pre><code>export ARVADOS_API_HOST=<span class="userinput">uuid_prefix</span>.your.domain
-export ARVADOS_API_TOKEN="<span class="userinput">{{railsout}}</span>"
-exec sudo -u nobody keep-web \
- -listen=<span class="userinput">:9002</span> \
- -attachment-only-host=<span class="userinput">download.uuid_prefix.your.domain</span> \
- -allow-anonymous \
- 2&gt;&amp;1
+<pre><code>exec sudo -u nobody keep-web -config=/path/to/arvados.yml
 </code></pre>
 </notextile>
 
-Omit the @-allow-anonymous@ argument if you do not want to serve public data.
+Set @Users.AnonymousUserToken: ""@ (empty string) if you do not want to serve public data.
 
-Set @ARVADOS_API_HOST_INSECURE=1@ if your API server's TLS certificate is not signed by a recognized CA.
+Set @Services.Controller.Insecure: true@ if your API server's TLS certificate is not signed by a recognized CA.
 
 h3. Set up a reverse proxy with TLS support
 
@@ -134,24 +160,32 @@ Configure your DNS servers so the following names resolve to your Nginx proxy's
 
 If neither of the above wildcard options is feasible, you have two choices:
 # Serve web content at @collections.uuid_prefix.your.domain@, but only for unauthenticated requests (public data and collection sharing links). Authenticated requests will always result in file downloads, using the @download@ name. For example, the Workbench "preview" button and the "view entire log file" link will invoke file downloads instead of displaying content in the browser window.
-# In the special case where you know you are immune to XSS exploits, you can enable the "trust all content" mode in Keep-web (with the @-trust-all-content@ command line flag) and Workbench (with the @trust_all_content@ item in @application.yml@). With both of these enabled, inline web content can be served from a single @collections@ host name; no wildcard DNS or certificate is needed. Do not do this without understanding the security implications described in the "Keep-web documentation":http://godoc.org/github.com/curoverse/arvados/services/keep-web.
+# In the special case where you know you are immune to XSS exploits, you can enable the "trust all content" mode in Keep-web and Workbench (setting @Collections.TrustAllContent: true@ on the config file). With this enabled, inline web content can be served from a single @collections@ host name; no wildcard DNS or certificate is needed. Do not do this without understanding the security implications described in the "Keep-web documentation":http://godoc.org/github.com/curoverse/arvados/services/keep-web.
 
 h3. Tell Workbench about the Keep-web service
 
 Workbench has features like "download file from collection" and "show image" which work better if the content is served by Keep-web rather than Workbench itself. We recommend using the two different hostnames ("download" and "collections" above) for file downloads and inline content respectively.
 
-Add the following entry to your Workbench configuration file (@/etc/arvados/workbench/application.yml@). This URL will be used for file downloads.
+Add the following entry to your Workbench cluster configuration file (@/etc/arvados/config.yml@). This URL will be used for file downloads.
 
 <notextile>
-<pre><code>keep_web_download_url: https://download.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}
+<pre><code>Clusters:
+  zzzzz:
+    Services:
+      WebDAVDownload:
+        ExternalURL: "https://download.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}"
 </code></pre>
 </notextile>
 
-Additionally, add *one* of the following entries to your Workbench configuration file, depending on your DNS setup. This URL will be used to serve user content that can be displayed in the browser, like image previews and static HTML pages.
+Additionally, add *one* of the following entries to your Workbench cluster configuration file, depending on your DNS setup. This URL will be used to serve user content that can be displayed in the browser, like image previews and static HTML pages.
 
 <notextile>
-<pre><code>keep_web_url: https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://%{uuid_or_pdh}.collections.<span class="userinput">uuid_prefix</span>.your.domain
-keep_web_url: https://collections.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}
+<pre><code>Clusters:
+  zzzzz:
+    Services:
+      WebDAV:
+        ExternalURL: "https://%{uuid_or_pdh}--collections.<span class="userinput">uuid_prefix</span>.your.domain"
+        ExternalURL: "https://%{uuid_or_pdh}.collections.<span class="userinput">uuid_prefix</span>.your.domain"
+        ExternalURL: "https://collections.<span class="userinput">uuid_prefix</span>.your.domain/c=%{uuid_or_pdh}"
 </code></pre>
 </notextile>
index d65156f98781f99cd3fbc4a20b2f0ba144ea8f97..8682eac2dd08b5aaa8f308330ca4a2eba06cf34e 100644 (file)
 //
 // Configuration
 //
-// The default configuration file location is
-// /etc/arvados/keep-web/keep-web.yml.
+// The default cluster configuration file location is
+// /etc/arvados/config.yml.
 //
 // Example configuration file
 //
-//     Client:
-//       APIHost: "zzzzz.arvadosapi.com:443"
-//       AuthToken: ""
-//       Insecure: false
-//     Listen: :1234
-//     AnonymousTokens:
-//       - xxxxxxxxxxxxxxxxxxxx
-//     AttachmentOnlyHost: ""
-//     TrustAllContent: false
+//   Clusters:
+//     zzzzz:
+//       SystemRootToken: ""
+//       Services:
+//         Controller:
+//           ExternalURL: "https://example.com"
+//           Insecure: false
+//         WebDAV:
+//           InternalURLs:
+//             "http://:1234/": {}
+//         WebDAVDownload:
+//           InternalURLs:
+//             "http://:1234/": {}
+//           ExternalURL: "https://download.example.com/"
+//       Users:
+//         AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxx"
+//       Collections:
+//         TrustAllContent: false
 //
 // Starting the server
 //
 // Start a server using the default config file
-// /etc/arvados/keep-web/keep-web.yml:
+// /etc/arvados/config.yml:
 //
 //   keep-web
 //
-// Start a server using the config file /path/to/keep-web.yml:
+// Start a server using the config file /path/to/config.yml:
 //
-//   keep-web -config /path/to/keep-web.yml
+//   keep-web -config /path/to/config.yml
 //
 // Proxy configuration
 //
 //
 // Anonymous downloads
 //
-// The "AnonymousTokens" configuration entry is an array of tokens to
-// use when processing anonymous requests, i.e., whenever a web client
+// The "Users.AnonymousUserToken" configuration entry used when
+// when processing anonymous requests, i.e., whenever a web client
 // does not supply its own Arvados API token via path, query string,
 // cookie, or request header.
 //
-//   "AnonymousTokens":["xxxxxxxxxxxxxxxxxxxxxxx"]
+//   Clusters:
+//     zzzzz:
+//       Users:
+//         AnonymousUserToken: "xxxxxxxxxxxxxxxxxxxxxxx"
 //
 // See http://doc.arvados.org/install/install-keep-web.html for examples.
 //
 // only when the designated origin matches exactly the Host header
 // provided by the client or downstream proxy.
 //
-//   "AttachmentOnlyHost":"domain.example:9999"
+//   Clusters:
+//     zzzzz:
+//       Services:
+//         WebDAVDownload:
+//           ExternalURL: "https://domain.example:9999"
 //
 // Trust All Content mode
 //
 //
 // In such cases you can enable trust-all-content mode.
 //
-//   "TrustAllContent":true
+//   Clusters:
+//     zzzzz:
+//       Collections:
+//         TrustAllContent: true
 //
 // When TrustAllContent is enabled, the only effect of the
-// AttachmentOnlyHost flag is to add a "Content-Disposition:
+// Attachment-Only host setting is to add a "Content-Disposition:
 // attachment" header.
 //
-//   "AttachmentOnlyHost":"domain.example:9999",
-//   "TrustAllContent":true
+//   Clusters:
+//     zzzzz:
+//       Services:
+//         WebDAVDownload:
+//           ExternalURL: "https://domain.example:9999"
+//       Collections:
+//         TrustAllContent: true
 //
 // Depending on your site configuration, you might also want to enable
 // the "trust all content" setting in Workbench. Normally, Workbench
index ab9ff60b503eb12a1f325175b6d124a055e18222..534ed9b3022d01e7d169fc5212ca1d973dae963f 100644 (file)
@@ -435,7 +435,7 @@ func (s *UnitSuite) TestLegacyConfig(c *check.C) {
                "anonusertoken"
        ],
        "AttachmentOnlyHost": "download.example.com",
-       "TrustAllContent": false,
+       "TrustAllContent": true,
        "Cache": {
                "TTL": "1m",
                "UUIDTTL": "1s",
@@ -477,6 +477,7 @@ func (s *UnitSuite) TestLegacyConfig(c *check.C) {
        c.Check(cfg.cluster.Services.WebDAVDownload.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
        c.Check(cfg.cluster.Services.WebDAV.InternalURLs[arvados.URL{Host: ":80"}], check.NotNil)
 
+       c.Check(cfg.cluster.Collections.TrustAllContent, check.Equals, true)
        c.Check(cfg.cluster.Users.AnonymousUserToken, check.Equals, "anonusertoken")
        c.Check(cfg.cluster.ManagementToken, check.Equals, "xyzzy")
 }