X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/6e5158294c7e19af9031fc43fefc21275e758911..c73f87c5fedba20b82a091152353f718d81ba6c3:/doc/install/install-keepproxy.html.textile.liquid diff --git a/doc/install/install-keepproxy.html.textile.liquid b/doc/install/install-keepproxy.html.textile.liquid index 69a298a7a7..d3a60ad0f3 100644 --- a/doc/install/install-keepproxy.html.textile.liquid +++ b/doc/install/install-keepproxy.html.textile.liquid @@ -3,10 +3,18 @@ layout: default navsection: installguide title: Install Keepproxy server ... +{% comment %} +Copyright (C) The Arvados Authors. All rights reserved. -The Keepproxy server is a gateway into your Keep storage. Unlike the Keepstore servers, which are only accessible on the local LAN, Keepproxy is designed to provide secure access into Keep from anywhere on the internet. +SPDX-License-Identifier: CC-BY-SA-3.0 +{% endcomment %} -By convention, we use the following hostname for the Keepproxy: +The Keepproxy server is a gateway into your Keep storage. Unlike the Keepstore servers, which are only accessible on the local LAN, Keepproxy is suitable for clients located elsewhere on the internet. Specifically, in contrast to Keepstore: +* A client writing through Keepproxy generates less network traffic: the client sends a single copy of a data block, and Keepproxy sends copies to the appropriate Keepstore servers. +* A client can write through Keepproxy without precomputing content hashes. Notably, the browser-based upload feature in Workbench requires Keepproxy. +* Keepproxy checks API token validity before processing requests. (Clients that can connect directly to Keepstore can use it as scratch space even without a valid API token.) + +By convention, we use the following hostname for the Keepproxy server:
table(table table-bordered table-condensed). @@ -36,33 +44,30 @@ Verify that Keepproxy is functional: 
~$ keepproxy -h
-Usage of default:
-  -default-replicas=2: Default number of replicas to write if not specified by the client.
-  -listen=":25107": Interface on which to listen for requests, in the format ipaddr:port. e.g. -listen=10.0.1.24:8000. Use -listen=:port to listen on all network interfaces.
-  -no-get=false: If set, disable GET operations
-  -no-put=false: If set, disable PUT operations
-  -pid="": Path to write pid file
+Usage of keepproxy:
+  -config file
+    	Site configuration file (default may be overridden by setting an ARVADOS_CONFIG environment variable) (default "/etc/arvados/config.yml")
+  -dump-config
+    	write current configuration to stdout and exit
+[...]
+  -version
+    	print version information and exit.
-h3. Create an API token for the Keepproxy server - -The Keepproxy server needs a token to talk to the API server. - -On the API server, use the following command to create the token: - - -
~/arvados/services/api/script$ RAILS_ENV=production bundle exec ./get_anonymous_user_token.rb
-hoShoomoo2bai3Ju1xahg6aeng1siquuaZ1yae2gi2Uhaeng2r
-
 - -h3. Set up the Keepproxy service +h3. Update the cluster config -We recommend you run Keepproxy under "runit":http://smarden.org/runit/ or a similar supervisor. Make sure the launcher sets the envirnoment variables @ARVADOS_API_TOKEN@ (with the token you just generated), @ARVADOS_API_HOST@, and, if needed, @ARVADOS_API_HOST_INSECURE@. The core keepproxy command to run is: +Edit the cluster config at @/etc/arvados/config.yml@ and set @Services.Keepproxy.ExternalURL@ and @Services.Keepproxy.InternalURLs@. Replace @uuid_prefix@ with your cluster id. -
ARVADOS_API_TOKEN=[generated token] ARVADOS_API_HOST=uuid_prefix.your.domain exec keepproxy
-
+
Clusters:
+  uuid_prefix:
+    Services:
+      Keepproxy:
+        ExternalURL: https://keep.uuid_prefix.your.domain
+        InternalURLs:
+	  "http://localhost:25107": {}
+
 h3. Set up a reverse proxy with SSL support @@ -73,32 +78,28 @@ This is best achieved by putting a reverse proxy with SSL support in front of Ke 
 upstream keepproxy {
-  server     127.0.0.1:25107  fail_timeout=10s;
+  server                127.0.0.1:25107;
 }
 
 server {
-  listen       [your public IP address]:443 ssl;
-  server_name keep.example.com
+  listen                  [your public IP address]:443 ssl;
+  server_name             keep.uuid_prefix.your.domain;
+
+  proxy_connect_timeout   90s;
+  proxy_read_timeout      300s;
+  proxy_set_header        X-Real-IP $remote_addr;
+  proxy_http_version      1.1;
+  proxy_request_buffering off;
 
-  ssl on;
-  ssl_certificate           /etc/nginx/keep.example.com-ssl.crt;
-  ssl_certificate_key       /etc/nginx/keep.example.com-ssl.key;
+  ssl                     on;
+  ssl_certificate         /etc/nginx/keep.uuid_prefix.your.domain-ssl.crt;
+  ssl_certificate_key     /etc/nginx/keep.uuid_prefix.your.domain-ssl.key;
 
   # Clients need to be able to upload blocks of data up to 64MiB in size.
-  client_max_body_size 64m;
-  client_body_buffer_size 64M;
+  client_max_body_size    64m;
 
   location / {
     proxy_pass            http://keepproxy;
-    proxy_redirect        off;
-    proxy_connect_timeout 90;
-    proxy_read_timeout    300;
-
-    proxy_set_header      X-Forwarded-Proto https;
-    proxy_set_header      Host $http_host;
-    proxy_set_header      X-External-Client $external_client;
-    proxy_set_header      X-Real-IP $remote_addr;
-    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
   }
 }
@@ -107,17 +108,64 @@ Note: if the Web uploader is failing to upload data and there are no logs from k h3. Tell the API server about the Keepproxy server -The API server needs to be informed about the presence of your Keepproxy server. Please execute the following commands on your shell server. +The API server needs to be informed about the presence of your Keepproxy server. + +First, if you don't already have an admin token, create a superuser token. + +{% include 'create_superuser_token' %} + +Configure your environment to run @arv@ using the output of create_superuser_token.rb: + +
+export ARVADOS_API_HOST=zzzzz.example.com
+export ARVADOS_API_TOKEN=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+
-
~$ prefix=`arv --format=uuid user current | cut -d- -f1`
-~$ echo "Site prefix is '$prefix'"
+
~$ uuid_prefix=`arv --format=uuid user current | cut -d- -f1`
+~$ echo "Site prefix is '$uuid_prefix'"
 ~$ read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice"
 {
- "service_host":"keep.$prefix.your.domain",
+ "service_host":"keep.$uuid_prefix.your.domain",
  "service_port":443,
  "service_ssl_flag":true,
  "service_type":"proxy"
 }
 EOF
 

+
+h2. Run Keepproxy
+
+h3. Start the service (option 1: systemd)
+
+If your system does not use systemd, skip this section and follow the "runit instructions":#runit instead.
+
+If your system uses systemd, the keepproxy service should already be set up. Start it and check its status:
+
+
+
~$ sudo systemctl restart keepproxy
+~$ sudo systemctl status keepproxy
+● keepproxy.service - Arvados Keep Proxy
+   Loaded: loaded (/lib/systemd/system/keepproxy.service; enabled)
+   Active: active (running) since Tue 2019-07-23 09:33:47 EDT; 3 weeks 1 days ago
+     Docs: https://doc.arvados.org/
+ Main PID: 1150 (Keepproxy)
+   CGroup: /system.slice/keepproxy.service
+           └─1150 /usr/bin/keepproxy
+[...]
+

+
+
+h3(#runit). Start the service (option 2: runit)
+
+Install runit to supervise the Keep-web daemon.  {% include 'install_runit' %}
+
+h3. Testing keepproxy
+
+Log into a host that is on an external network from your private Arvados network.  The host should be able to contact your keepproxy server (eg keep.$uuid_prefix.arvadosapi.com), but not your keepstore servers (eg keep[0-9].$uuid_prefix.arvadosapi.com).
+
+Install the "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html
+
+@ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ must be set in the environment.
+
+You should now be able to use @arv-put@ to upload collections and @arv-get@ to fetch collections, for an example see "Testing keep.":install-keepstore.html#testing on the keepstore install page.