3 navsection: installguide
4 title: InternalURLs and ExternalURL
8 Copyright (C) The Arvados Authors. All rights reserved.
10 SPDX-License-Identifier: CC-BY-SA-3.0
13 The Arvados configuration is stored at @/etc/arvados/config.yml@. See the "Configuration reference":config.html for more detail.
15 The @Services@ section lists a number of Arvados services, each with an @InternalURLs@ and/or @ExternalURL@ configuration key. This document explains the precise meaning of these configuration keys, and how they are used by the Arvados services.
17 Generally speaking, the keys under @InternalURLs@ are the endpoints where the service should be listening, and reachable from other hosts inside the Arvados cluster. The @ExternalURL@ value is the URL that the service advertises as its own URL. The @ExternalURL@ is the address where the service should be reachable from outside the Arvados cluster.
19 Because many of the Arvados services live behind a reverse proxy (e.g. Nginx as used in this guide), configuring the reverse proxy and the @InternalURLs@ and @ExternalURL@ values must be done in concert.
21 We'll walk through a number of examples to explain in more detail.
25 Consider this section for the @Keep-balance@ service:
27 {% codeblock as yaml %}
30 "http://ClusterID.example.com:9005/": {}
33 @Keep-balance@ has an API endpoint, but it is only used to expose "Prometheus":https://prometheus.io metrics.
35 There is no @ExternalURL@ key because @Keep-balance@ does not have an Arvados API, no Arvados services need to connect to @Keep-balance@.
37 The value for @InternalURLs@ tells the @Keep-balance@ service to start up and listen on port 9005, if it is started on a host where @ClusterID.example.com@ resolves to a local IP address. If @Keep-balance@ is started on a machine where the @ClusterID.example.com@ hostname does not resolve to a local IP address, it would refuse to start up, because it would not be able to find a local IP address to listen on.
39 It is also possible to use IP addresses in @InternalURLs@, for example:
41 {% codeblock as yaml %}
44 "http://127.0.0.1:9005/": {}
47 In this example, @Keep-balance@ would start up and listen on port 9005 at the @127.0.0.1@ IP address. Prometheus would only be able to access the @Keep-balance@ metrics if it could reach that IP and port, e.g. if it runs on the same machine.
49 Finally, it is also possible to listen on all interfaces, for example:
51 {% codeblock as yaml %}
54 "http://0.0.0.0:9005/": {}
57 In this case, @Keep-balance@ will listen on port 9005 on all IP addresses local to the machine.
61 Consider this section for the @Keepstore@ service:
63 {% codeblock as yaml %}
66 "http://keep0.ClusterID.example.com:25107": {}
67 "http://keep1.ClusterID.example.com:25107": {}
70 There is no @ExternalURL@ key because @Keepstore@ is only accessed from inside the Arvados cluster. For access from outside, all traffic goes via @Keepproxy@.
72 When @Keepstore@ is installed on the host where @keep0.ClusterID.example.com@ resolves to a local IP address, it will listen on port 25107 on that IP address. Likewise on the @keep1.ClusterID.example.com@ host. On all other systems, @Keepstore@ will refuse to start.
76 Consider this section for the @Keepproxy@ service:
78 {% codeblock as yaml %}
80 ExternalURL: https://keep.ClusterID.example.com
82 "http://localhost:25107": {}
85 The @ExternalURL@ advertised is @https://keep.ClusterID.example.com@. The @Keepproxy@ service will start up on @localhost@ port 25107, however. This is possible because we also configure Nginx to terminate SSL and sit in front of the @Keepproxy@ service:
87 <notextile><pre><code>upstream keepproxy {
88 server 127.0.0.1:<span class="userinput">25107</span>;
93 server_name <span class="userinput">keep.ClusterID.example.com</span>;
95 proxy_connect_timeout 90s;
96 proxy_read_timeout 300s;
97 proxy_set_header X-Real-IP $remote_addr;
98 proxy_http_version 1.1;
99 proxy_request_buffering off;
100 proxy_max_temp_file_size 0;
102 ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
103 ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
105 # Clients need to be able to upload blocks of data up to 64MiB in size.
106 client_max_body_size 64m;
109 proxy_pass http://keepproxy;
112 </code></pre></notextile>
114 If a client connects to the @Keepproxy@ service, it will talk to Nginx which will reverse proxy the traffic to the @Keepproxy@ service.
118 Consider this section for the @Workbench@ service:
120 {% codeblock as yaml %}
122 ExternalURL: "https://workbench.ClusterID.example.com"
125 The @ExternalURL@ advertised is @https://workbench.ClusterID.example.com@. There is no value for @InternalURLs@ because Workbench1 is a Rails application served by Passenger, and the listening host/post is configured in the Nginx configuration:
127 <notextile><pre><code>
130 server_name workbench.ClusterID.example.com;
132 ssl_certificate /YOUR/PATH/TO/cert.pem;
133 ssl_certificate_key /YOUR/PATH/TO/cert.key;
135 root /var/www/arvados-workbench/current/public;
138 passenger_enabled on;
139 # If you're using RVM, uncomment the line below.
140 #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
142 # `client_max_body_size` should match the corresponding setting in
143 # the API.MaxRequestSize and Controller's server's Nginx configuration.
144 client_max_body_size 128m;
146 </code></pre></notextile>
150 Consider this section for the @API server@ service:
152 {% codeblock as yaml %}
155 "http://localhost:8004": {}
158 There is no @ExternalURL@ defined because the @API server@ is not directly accessible and does not need to advertise a URL: all traffic to it flows via @Controller@, which is the only client that talks to it.
160 The @API server@ is also a Rails application, and its listening host/port is defined in the Nginx configuration:
162 <notextile><pre><code>
164 # This configures the Arvados API server. It is written using Ruby
165 # on Rails and uses the Passenger application server.
167 listen localhost:8004;
168 server_name localhost-api;
170 root /var/www/arvados-api/current/public;
171 index index.html index.htm index.php;
173 passenger_enabled on;
175 # If you are using RVM, uncomment the line below.
176 # If you're using system ruby, leave it commented out.
177 #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
179 # This value effectively limits the size of API objects users can
180 # create, especially collections. If you change this, you should
181 # also ensure the following settings match it:
182 # * `client_max_body_size` in the previous server section
183 # * `API.MaxRequestSize` in config.yml
184 client_max_body_size 128m;
186 </code></pre></notextile>
188 So then, why is there a need to specify @InternalURLs@ for the @RailsAPI@ service? It is there because this is how the @Controller@ service locates the @RailsAPI@ service it should talk to. Since this connection is internal to the Avados cluster, @Controller@ uses @InternalURLs@ to find the @RailsAPI@ endpoint.
192 Consider this section for the @Controller@ service:
194 {% codeblock as yaml %}
197 "http://localhost:8003": {}
198 ExternalURL: "https://ClusterID.example.com"
201 The @ExternalURL@ advertised is @https://ClusterID.example.com@. The @Controller@ service will start up on @localhost@ port 8003. Nginx is configured to terminate SSL and sit in front of the @Controller@ service:
203 <notextile><pre><code>
204 # This is the port where nginx expects to contact arvados-controller.
205 upstream controller {
206 server localhost:8003 fail_timeout=10s;
210 # This configures the public https port that clients will actually connect to,
211 # the request is reverse proxied to the upstream 'controller'
214 server_name ClusterID.example.com;
216 ssl_certificate /YOUR/PATH/TO/cert.pem;
217 ssl_certificate_key /YOUR/PATH/TO/cert.key;
219 # Refer to the comment about this setting in the passenger (arvados
220 # api server) section of your Nginx configuration.
221 client_max_body_size 128m;
224 proxy_pass http://controller;
226 proxy_connect_timeout 90s;
227 proxy_read_timeout 300s;
229 proxy_set_header Host $http_host;
230 proxy_set_header Upgrade $http_upgrade;
231 proxy_set_header Connection "upgrade";
232 proxy_set_header X-External-Client $external_client;
233 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
234 proxy_set_header X-Forwarded-Proto https;
235 proxy_set_header X-Real-IP $remote_addr;
238 </code></pre></notextile>
240 @Controller@ provides the main Arvados API endpoint. As such, it is used extensively by Arvados clients inside and outside of the cluster. It is recommended to use a split-horizon DNS setup where @ClusterID.example.com@ resolves to an internal IP address from inside the Arvados cluster, and a publicly routed external IP address when resolved from outside the cluster. This will simplify firewalling and traffic routing. In a cloud environment where traffic that flows via public IP addresses is charged, using split horizon DNS can also save a lot of money.
242 h2. Connection overview
244 When a client wants to talk to an Arvados cluster, it needs to look up the endpoint of the particular API it wants to connect to. If the client does not have access to the @config.yml@ file, it connects to @Controller@ and retrieves the value of @InternalURLs@ or @ExternalURL@ for the service it wants to talk to. Arvados clients typically get the URL of the @Controller@ from the @ARVADOS_API_HOST@ environment variable.
246 When an Arvados service with access to @config.yml@ needs to talk to another Arvados service, it looks up the value of @InternalURLs@ or @ExternalURL@ for the service it wants to talk to.
248 Below is a list of Arvados clients (or services acting as a client), the Arvados service they connect to, and the configuration value that they use to find the appropriate endpoint.
250 <div class="offset1">
251 table(table table-bordered table-condensed).
252 |_.Client |_.Destination service|_.Target URL from |
253 |arv |Controller |$ARVADOS_API_HOST |
254 |API client (e.g. SDK, arv-put, arv-get, etc) |Controller |$ARVADOS_API_HOST |
255 |arv-ws |Controller |$ARVADOS_API_HOST |
256 |arv-ws |Websocket |Websocket.ExternalURL |
257 |arv-mount |Controller |$ARVADOS_API_HOST |
258 |arv-mount |Keepstore |Keepstore.InternalURLs |
259 |arv-mount |Keepproxy |Keepproxy.ExternalURL |
260 |Controller |RailsAPI |RailsAPI.InternalURLs |
261 |Keep-balance|Keepstore |Keepstore.InternalURLs |
262 |Keep client|Keepstore |Keepstore.InternalURLs |
263 |Keep client|Keepproxy |Keepproxy.ExternalURL |
264 |Nginx |Controller |Controller.InternalURLs |
265 |Nginx |Keepproxy |Keepproxy.InternalURLs |
266 |Nginx |Keep-web |WebDAV.InternalURLs |
267 |Prometheus |Keep-balance |Keepbalance.InternalURLs|
268 |Workbench2 |Keep-web |WebDAV.ExternalURL |
269 |Workbench2 |Websocket |Websocket.ExternalURL |
270 |webdav client|Keep-web |WebDAV.ExternalURL |