3 navsection: installguide
4 title: Install the API server
7 h2. Install prerequisites
9 The Arvados package repository includes an API server package that can help automate much of the deployment.
11 h3(#install_ruby_and_bundler). Install Ruby and Bundler
13 {% include 'install_ruby_and_bundler' %}
15 h3(#install_postgres). Install PostgreSQL
17 {% include 'install_postgres' %}
19 h3(#build_tools_apiserver). Build tools
21 On older distributions, you may need to use a backports repository to satisfy these requirements. For example, on older Red Hat-based systems, consider using the "postgresql92":https://www.softwarecollections.org/en/scls/rhscl/postgresql92/ and "nginx16":https://www.softwarecollections.org/en/scls/rhscl/nginx16/ Software Collections.
23 On a Debian-based system, install the following packages:
26 <pre><code>~$ <span class="userinput">sudo apt-get install bison build-essential libcurl4-openssl-dev git nginx arvados-api-server</span>
30 On a Red Hat-based system, install the following packages:
33 <pre><code>~$ <span class="userinput">sudo yum install bison make automake gcc gcc-c++ libcurl-devel nginx git arvados-api-server</span>
37 h2. Set up the database
39 Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one:
42 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**128).to_s(36)'</span>
43 6gqa1vu492idd7yca9tfandj3
44 </code></pre></notextile>
46 Create a new database user.
49 <pre><code>~$ <span class="userinput">sudo -u postgres createuser --encrypted -R -S --pwprompt arvados</span>
50 [sudo] password for <b>you</b>: <span class="userinput">yourpassword</span>
51 Enter password for new role: <span class="userinput">paste-password-you-generated</span>
52 Enter it again: <span class="userinput">paste-password-again</span>
53 </code></pre></notextile>
55 {% include 'notebox_begin' %}
57 This user setup assumes that your PostgreSQL is configured to accept password authentication. Red Hat systems use ident-based authentication by default. You may need to either adapt the user creation, or reconfigure PostgreSQL (in @pg_hba.conf@) to accept password authentication.
59 {% include 'notebox_end' %}
64 <pre><code>~$ <span class="userinput">sudo -u postgres createdb arvados_production -T template0 -E UTF8 -O arvados</span>
68 h2. Set up configuration files
70 The API server package uses configuration files that you write to @/etc/arvados/api@ and ensures they're consistently deployed. Create this directory and copy the example configuration files to it:
73 <pre><code>~$ <span class="userinput">sudo mkdir -p /etc/arvados/api</span>
74 ~$ <span class="userinput">sudo chmod 700 /etc/arvados/api</span>
75 ~$ <span class="userinput">cd /var/www/arvados-api/current</span>
76 /var/www/arvados-api/current$ <span class="userinput">sudo cp config/database.yml.sample /etc/arvados/api/database.yml</span>
77 /var/www/arvados-api/current$ <span class="userinput">sudo cp config/application.yml.example /etc/arvados/api/application.yml</span>
81 h2. Configure the database connection
83 Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above.
85 h2(#configure_application). Configure the API server
87 Edit @/etc/arvados/api/application.yml@ and to configure the settings described in the following sections. The deployment script will consistently deploy this to the API server's configuration directory. The API server reads both @application.yml@ and its own @config/application.default.yml@ file. The settings in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@. The @config/application.yml.example@ file is not read by the API server and is provided as a starting template only.
89 Only put local configuration in @application.yml@, do not edit @application.default.yml@.
91 h3(#uuid_prefix). uuid_prefix
93 The @uuid_prefix@ is used for all database identifiers to identify the record as originating from this site. It is a string consisting of exactly 5 lowercase ASCII letters and digits. Generate a random value and set it in @application.yml@:
96 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**40).to_s(36)[0,5]'</span>
98 </code></pre></notextile>
100 Example @application.yml@:
103 <pre><code> uuid_prefix: <span class="userinput">zzzzz</span></code></pre>
108 The @secret_token@ is used for for signing cookies. IMPORTANT: This is a site secret. It should be at least 50 characters. Generate a random value and set it to @application.yml@:
111 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
112 yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
113 </code></pre></notextile>
115 Example @application.yml@:
118 <pre><code> secret_token: <span class="userinput">yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy</span></code></pre>
121 h3(#blob_signing_key). blob_signing_key
123 The @blob_signing_key@ is used to enforce access control to Keep blocks. This same key must be provided to the Keepstore daemons when "installing Keepstore servers.":install-keepstore.html IMPORTANT: This is a site secret. It should be at least 50 characters. Generate a random value and set it to @application.yml@:
126 <pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
127 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
128 </code></pre></notextile>
130 Example @application.yml@:
133 <pre><code> blob_signing_key: <span class="userinput">xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</span></code></pre>
136 h3(#omniauth). sso_app_secret, sso_app_id, sso_provider_url
138 The following settings enable the API server to communicate with the "Single Sign On (SSO) server":install-sso.html to authenticate user log in.
140 Set @sso_provider_url@ to the base URL where your SSO server is installed. This should be a URL consisting of the scheme and host (and optionally, port), without a trailing slash.
142 Set @sso_app_secret@ and @sso_app_id@ to the corresponding values for @app_secret@ and @app_id@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step.
144 Example @application.yml@:
147 <pre><code> sso_app_id: <span class="userinput">arvados-server</span>
148 sso_app_secret: <span class="userinput">wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww</span>
149 sso_provider_url: <span class="userinput">https://sso.example.com</span>
153 h3. workbench_address
155 Set @workbench_address@ to the URL of your workbench application after following "Install Workbench.":install-workbench-app.html
157 Example @application.yml@:
160 <pre><code> workbench_address: <span class="userinput">https://workbench.zzzzz.example.com</span></code></pre>
163 h3. websockets_address
165 Set @websockets_address@ to the @@wss://@ URL of the API server websocket endpoint after following "Set up Web servers.":#set_up
167 Example @application.yml@:
170 <pre><code> websockets_address: <span class="userinput">https://ws.zzzzz.example.com</span></code></pre>
173 h3(#git_repositories_dir). git_repositories_dir, git_internal_dir
175 The @git_repositories_dir@ setting specifies the directory is where user git repositories will be stored. By default this is @/var/lib/arvados/git@. You can change the location by defining it in @application.yml@.
177 The @git_internal_dir@ setting specifies the directory is where the system internal git repository will be stored. By default this is @/var/lib/arvados/internal.git@. This repository stores git branches that have been used to run crunch jobs.
179 Example @application.yml@:
182 <pre><code> git_repositories_dir: <span class="userinput">/var/lib/arvados/git</span>
183 git_internal_dir: <span class="userinput">/var/lib/arvados/internal.git</span>
187 h3. Additional settings
189 The file @config/application.default.yml@ (online at "https://arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml":https://arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml documents a number of additional configuration settings. Only put local configuration in @application.yml@, do not edit @application.default.yml@.
191 h2. Prepare the API server deployment
193 Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh@. This will install and check your configuration, install necessary gems, and run any necessary database setup.
195 {% include 'notebox_begin' %}
196 You can safely ignore the following error message you may see when loading the database structure:
198 <pre><code>ERROR: must be owner of extension plpgsql</code></pre></notextile>
199 {% include 'notebox_end' %}
201 This command aborts when it encounters an error. It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.
203 h2(#set_up). Set up Web servers
205 For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets. To do that:
209 <li>Install Nginx via your distribution or a backports repository.</li>
211 <li><a href="https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html">Install Phusion Passenger for Nginx</a>.</li>
213 <li><p>Puma is already included with the API server's gems. We recommend you use a tool like <a href="http://smarden.org/runit/">runit</a> or something similar. Here's a sample run script for that:</p>
215 <pre><code>#!/bin/bash
220 # Uncomment the line below if you're using RVM.
221 #source /etc/profile.d/rvm.sh
225 echo ws-only > "$envdir/ARVADOS_WEBSOCKETS"
227 cd /var/www/arvados-api/current
228 echo "Starting puma in `pwd`"
230 # You may need to change arguments below to match your deployment, especially -u.
231 exec chpst -m 1073741824 -u www-data:www-data -e "$envdir" \
232 bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
236 <li><p>Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for both it and Puma. You might add a block like the following, adding SSL and logging parameters to taste:</p>
239 listen 127.0.0.1:8000;
240 server_name localhost-api;
242 root /var/www/arvados-api/current/public;
243 index index.html index.htm index.php;
245 passenger_enabled on;
246 # If you're using RVM, uncomment the line below.
247 #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
251 server 127.0.0.1:8000 fail_timeout=10s;
254 upstream websockets {
255 # The address below must match the one specified in puma's -b option.
256 server 127.0.0.1:8100 fail_timeout=10s;
259 proxy_http_version 1.1;
262 listen <span class="userinput">[your public IP address]</span>:443 ssl;
263 server_name <span class="userinput">uuid_prefix.your.domain</span>;
267 index index.html index.htm index.php;
270 proxy_pass http://api;
273 proxy_set_header X-Forwarded-Proto https;
274 proxy_set_header Host $http_host;
275 proxy_set_header X-External-Client $external_client;
276 proxy_set_header X-Real-IP $remote_addr;
277 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
282 listen <span class="userinput">[your public IP address]</span>:443 ssl;
283 server_name ws.<span class="userinput">uuid_prefix.your.domain</span>;
287 index index.html index.htm index.php;
290 proxy_pass http://websockets;
293 proxy_set_header Upgrade $http_upgrade;
294 proxy_set_header Connection "upgrade";
295 proxy_set_header Host $host;
296 proxy_set_header X-Real-IP $remote_addr;
297 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
303 <li>Restart Nginx.</li>