3 navsection: installguide
4 title: Install the API server
7 h2. Install prerequisites
9 The Arvados package repository includes an API server package that can help automate much of the deployment.
11 h3(#install_ruby_and_bundler). Install Ruby and Bundler
13 {% include 'install_ruby_and_bundler' %}
15 h3(#install_postgres). Install PostgreSQL
17 {% include 'install_postgres' %}
19 h2(#install_apiserver). Install API server and dependencies
21 On a Debian-based system, install the following packages:
24 <pre><code>~$ <span class="userinput">sudo apt-get install bison build-essential libcurl4-openssl-dev git arvados-api-server</span>
28 On a Red Hat-based system, install the following packages:
31 <pre><code>~$ <span class="userinput">sudo yum install bison make automake gcc gcc-c++ libcurl-devel git arvados-api-server</span>
35 {% include 'install_git' %}
37 h2(#configure). Set up the database
39 Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one:
42 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**128).to_s(36)'</span>
43 6gqa1vu492idd7yca9tfandj3
44 </code></pre></notextile>
46 Create a new database user.
49 <pre><code>~$ <span class="userinput">sudo -u postgres createuser --encrypted -R -S --pwprompt arvados</span>
50 [sudo] password for <b>you</b>: <span class="userinput">yourpassword</span>
51 Enter password for new role: <span class="userinput">paste-password-you-generated</span>
52 Enter it again: <span class="userinput">paste-password-again</span>
53 </code></pre></notextile>
55 {% assign pg_hba_path = "/opt/rh/postgresql92/root/var/lib/pgsql/data/pg_hba.conf" %}
56 {% assign pg_service = "postgresql92-postgresql" %}
57 {% include 'install_redhat_postgres_auth' %}
62 <pre><code>~$ <span class="userinput">sudo -u postgres createdb arvados_production -T template0 -E UTF8 -O arvados</span>
66 h2. Configure the database connection
68 Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above.
70 h2(#configure_application). Configure the API server
72 Edit @/etc/arvados/api/application.yml@ to configure the settings described in the following sections. The API server reads both @application.yml@ and its own @config/application.default.yml@ file. The settings in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@. The @config/application.yml.example@ file is not read by the API server and is provided as a starting template only.
74 @config/application.default.yml@ documents additional configuration settings not listed here. You can "view the current source version":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml for reference.
76 Only put local configuration in @application.yml@. Do not edit @application.default.yml@.
78 h3(#uuid_prefix). uuid_prefix
80 Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment. This prefix is used for all database identifiers to identify the record as originating from this site. It must be exactly 5 lowercase ASCII letters and digits.
82 Example @application.yml@:
85 <pre><code> uuid_prefix: <span class="userinput">zzzzz</span></code></pre>
90 The @secret_token@ is used for for signing cookies. IMPORTANT: This is a site secret. It should be at least 50 characters. Generate a random value and set it in @application.yml@:
93 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
94 yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
95 </code></pre></notextile>
97 Example @application.yml@:
100 <pre><code> secret_token: <span class="userinput">yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy</span></code></pre>
103 h3(#blob_signing_key). blob_signing_key
105 The @blob_signing_key@ is used to enforce access control to Keep blocks. This same key must be provided to the Keepstore daemons when "installing Keepstore servers.":install-keepstore.html IMPORTANT: This is a site secret. It should be at least 50 characters. Generate a random value and set it in @application.yml@:
108 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
109 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
110 </code></pre></notextile>
112 Example @application.yml@:
115 <pre><code> blob_signing_key: <span class="userinput">xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</span></code></pre>
118 h3(#omniauth). sso_app_secret, sso_app_id, sso_provider_url
120 The following settings enable the API server to communicate with the "Single Sign On (SSO) server":install-sso.html to authenticate user log in.
122 Set @sso_provider_url@ to the base URL where your SSO server is installed. This should be a URL consisting of the scheme and host (and optionally, port), without a trailing slash.
124 Set @sso_app_secret@ and @sso_app_id@ to the corresponding values for @app_secret@ and @app_id@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step.
126 Example @application.yml@:
129 <pre><code> sso_app_id: <span class="userinput">arvados-server</span>
130 sso_app_secret: <span class="userinput">wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww</span>
131 sso_provider_url: <span class="userinput">https://sso.example.com</span>
135 h3. workbench_address
137 Set @workbench_address@ to the URL of your workbench application after following "Install Workbench.":install-workbench-app.html
139 Example @application.yml@:
142 <pre><code> workbench_address: <span class="userinput">https://workbench.zzzzz.example.com</span></code></pre>
145 h3. websocket_address
147 Set @websocket_address@ to the @wss://@ URL of the API server websocket endpoint after following "Set up Web servers":#set_up. The path of the default endpoint is @/websocket@.
149 Example @application.yml@:
152 <pre><code> websocket_address: <span class="userinput">wss://ws.zzzzz.example.com</span>/websocket</code></pre>
155 h3(#git_repositories_dir). git_repositories_dir
157 The @git_repositories_dir@ setting specifies the directory where user git repositories will be stored.
159 The git server setup process is covered on "its own page":install-arv-git-httpd.html. For now, create an empty directory in the default location:
162 <pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git/repositories</span>
163 </code></pre></notextile>
165 If you intend to store your git repositories in a different location, specify that location in @application.yml@.
167 Default setting in @application.default.yml@:
170 <pre><code> git_repositories_dir: <span class="userinput">/var/lib/arvados/git/repositories</span>
174 h3(#git_internal_dir). git_internal_dir
176 The @git_internal_dir@ setting specifies the location of Arvados' internal git repository. By default this is @/var/lib/arvados/internal.git@. This repository stores git commits that have been used to run Crunch jobs. It should _not_ be a subdirectory of @git_repositories_dir@.
178 Example @application.yml@:
181 <pre><code> git_internal_dir: <span class="userinput">/var/lib/arvados/internal.git</span>
185 h2(#set_up). Set up Web servers
187 For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets. To do that:
191 <li><a href="https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/install_passenger_main.html">Install Nginx and Phusion Passenger</a>.</li>
193 <li><p>Install runit to supervise the Puma daemon. {% include 'install_runit' %}<notextile></p></li>
195 <li><p>Install the script below as the run script for the Puma service, modifying it as directed by the comments.</p>
197 <pre><code>#!/bin/bash
202 # Uncomment the line below if you're using RVM.
203 #source /etc/profile.d/rvm.sh
207 echo ws-only > "$envdir/ARVADOS_WEBSOCKETS"
209 cd /var/www/arvados-api/current
210 echo "Starting puma in `pwd`"
212 # Change arguments below to match your deployment, "webserver-user" and
213 # "webserver-group" should be changed to the user and group of the web server
214 # process. This is typically "www-data:www-data" on Debian systems by default,
215 # other systems may use different defaults such the name of the web server
216 # software (for example, "nginx:nginx").
217 exec chpst -m 1073741824 -u webserver-user:webserver-group -e "$envdir" \
218 bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
222 <li><p>Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for both it and Puma. You might add a block like the following, adding SSL and logging parameters to taste:</p>
225 listen 127.0.0.1:8000;
226 server_name localhost-api;
228 root /var/www/arvados-api/current/public;
229 index index.html index.htm index.php;
231 passenger_enabled on;
232 # If you're using RVM, uncomment the line below.
233 #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
235 # This value effectively limits the size of API objects users can
236 # create, especially collections. If you change this, you should
237 # also ensure the following settings match it:
238 # * `client_max_body_size` in the server section below
239 # * `client_max_body_size` in the Workbench Nginx configuration (twice)
240 # * `max_request_size` in the API server's application.yml file
241 client_max_body_size 128m;
245 server 127.0.0.1:8000 fail_timeout=10s;
248 upstream websockets {
249 # The address below must match the one specified in puma's -b option.
250 server 127.0.0.1:8100 fail_timeout=10s;
253 proxy_http_version 1.1;
255 # When Keep clients request a list of Keep services from the API server, the
256 # server will automatically return the list of available proxies if
257 # the request headers include X-External-Client: 1. Following the example
258 # here, at the end of this section, add a line for each netmask that has
259 # direct access to Keep storage daemons to set this header value to 0.
260 geo $external_client {
262 <span class="userinput">10.20.30.0/24</span> 0;
266 listen <span class="userinput">[your public IP address]</span>:443 ssl;
267 server_name <span class="userinput">uuid_prefix.your.domain</span>;
270 ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
271 ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
273 index index.html index.htm index.php;
275 # Refer to the comment about this setting in the server section above.
276 client_max_body_size 128m;
279 proxy_pass http://api;
281 proxy_connect_timeout 90s;
282 proxy_read_timeout 300s;
284 proxy_set_header X-Forwarded-Proto https;
285 proxy_set_header Host $http_host;
286 proxy_set_header X-External-Client $external_client;
287 proxy_set_header X-Real-IP $remote_addr;
288 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
293 listen <span class="userinput">[your public IP address]</span>:443 ssl;
294 server_name ws.<span class="userinput">uuid_prefix.your.domain</span>;
297 ssl_certificate <span class="userinput">/YOUR/PATH/TO/cert.pem</span>;
298 ssl_certificate_key <span class="userinput">/YOUR/PATH/TO/cert.key</span>;
300 index index.html index.htm index.php;
303 proxy_pass http://websockets;
305 proxy_connect_timeout 90s;
306 proxy_read_timeout 300s;
308 proxy_set_header Upgrade $http_upgrade;
309 proxy_set_header Connection "upgrade";
310 proxy_set_header Host $host;
311 proxy_set_header X-Real-IP $remote_addr;
312 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
318 <li><p>Restart Nginx:</p>
320 <pre><code>~$ <span class="userinput">sudo nginx -s reload</span>
328 h2. Prepare the API server deployment
330 {% assign railspkg = "arvados-api-server" %}
331 {% include 'install_rails_reconfigure' %}
333 {% include 'notebox_begin' %}
334 You can safely ignore the following messages if they appear while this command runs:
335 <pre>Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will
336 break this application for all non-root users on this machine.</pre>
337 <pre>fatal: Not a git repository (or any of the parent directories): .git</pre>
338 {% include 'notebox_end' %}