Merge branch 'git-commit-debug' of https://github.com/wtsi-hgi/arvados

[arvados.git] / doc / install / install-api-server.html.textile.liquid

---
layout: default
navsection: installguide
title: Install the API server
...

h2. Install prerequisites

The Arvados package repository includes an API server package that can help automate much of the deployment.

h3(#install_ruby_and_bundler). Install Ruby and Bundler

{% include 'install_ruby_and_bundler' %}

h3(#install_postgres). Install PostgreSQL

{% include 'install_postgres' %}

h3(#build_tools_apiserver). Build tools

On older distributions, you may need to use a backports repository to satisfy these requirements.  For example, on older Red Hat-based systems, consider using the "postgresql92":https://www.softwarecollections.org/en/scls/rhscl/postgresql92/ and "nginx16":https://www.softwarecollections.org/en/scls/rhscl/nginx16/ Software Collections.

On a Debian-based system, install the following packages:

<notextile>
<pre><code>~$ <span class="userinput">sudo apt-get install bison build-essential libcurl4-openssl-dev git nginx arvados-api-server</span>
</code></pre>
</notextile>

On a Red Hat-based system, install the following packages:

<notextile>
<pre><code>~$ <span class="userinput">sudo yum install bison make automake gcc gcc-c++ libcurl-devel nginx git arvados-api-server</span>
</code></pre>
</notextile>

h2. Set up the database

Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one:

<notextile>
<pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**128).to_s(36)'</span>
6gqa1vu492idd7yca9tfandj3
</code></pre></notextile>

Create a new database user.

<notextile>
<pre><code>~$ <span class="userinput">sudo -u postgres createuser --encrypted -R -S --pwprompt arvados</span>
[sudo] password for <b>you</b>: <span class="userinput">yourpassword</span>
Enter password for new role: <span class="userinput">paste-password-you-generated</span>
Enter it again: <span class="userinput">paste-password-again</span>
</code></pre></notextile>

{% include 'notebox_begin' %}

This user setup assumes that your PostgreSQL is configured to accept password authentication.  Red Hat systems use ident-based authentication by default.  You may need to either adapt the user creation, or reconfigure PostgreSQL (in @pg_hba.conf@) to accept password authentication.

{% include 'notebox_end' %}

Create the database:

<notextile>
<pre><code>~$ <span class="userinput">sudo -u postgres createdb arvados_production -T template0 -E UTF8 -O arvados</span>
</code></pre>
</notextile>

h2. Set up configuration files

The API server package uses configuration files that you write to @/etc/arvados/api@ and ensures they're consistently deployed.  Create this directory and copy the example configuration files to it:

<notextile>
<pre><code>~$ <span class="userinput">sudo mkdir -p /etc/arvados/api</span>
~$ <span class="userinput">sudo chmod 700 /etc/arvados/api</span>
~$ <span class="userinput">cd /var/www/arvados-api/current</span>
/var/www/arvados-api/current$ <span class="userinput">sudo cp config/database.yml.sample /etc/arvados/api/database.yml</span>
/var/www/arvados-api/current$ <span class="userinput">sudo cp config/application.yml.example /etc/arvados/api/application.yml</span>
</code></pre>
</notextile>

h2. Configure the database connection

Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above.

h2. Configure the API server

Edit @/etc/arvados/api/application.yml@ following the instructions below.  The deployment script will consistently deploy this to the API server's configuration directory.  The API server reads both @application.yml@ and its own @config/application.default.yml@ file.  Values in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@.  The @config/application.yml.example@ file is not read by the API server and is provided for installation convenience only.

Always put your local configuration in @application.yml@ instead of editing @application.default.yml@.

h3(#uuid_prefix). uuid_prefix

Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment.  This prefix is used for all database identifiers to identify the record as originating from this site.  It must be exactly 5 alphanumeric characters (lowercase ASCII letters and digits).

h3(#git_repositories_dir). git_repositories_dir

This field defaults to @/var/lib/arvados/git@. You can override the value by defining it in @application.yml@.

Make sure a clone of the arvados repository exists in @git_repositories_dir@.

<notextile>
<pre><code>~$ <span class="userinput">sudo mkdir -p /var/lib/arvados/git</span>
~$ <span class="userinput">sudo git clone --bare git://git.curoverse.com/arvados.git /var/lib/arvados/git/arvados.git</span>
</code></pre></notextile>

h3. secret_token

Generate a new secret token for signing cookies:

<notextile>
<pre><code>~$ <span class="userinput">ruby -e 'puts rand(2**400).to_s(36)'</span>
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
</code></pre></notextile>

Then put that value in the @secret_token@ field.

h3. blob_signing_key

If you want access control on your "Keepstore":install-keepstore.html server(s), you should set @blob_signing_key@ to the same value as the permission key you provide to your Keepstore daemon(s).

h3. workbench_address

Fill in the url of your workbench application in @workbench_address@, for example

&nbsp;&nbsp;https://workbench.@uuid_prefix@.your.domain

h3(#omniauth). sso_app_id, sso_app_secret, sso_provider_url

For @sso_app_id@ and @sso_app_secret@, provide the same @app_id@ and @app_secret@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step.

For @sso_provider_url@, provide the base URL where your SSO server is installed: just the scheme and host, with no trailing slash.

<notextile>
<pre><code>  sso_app_id: arvados-server
  sso_app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  sso_provider_url: https://sso.example.com
</code></pre>
</notextile>

h3. Other options

Consult @/var/www/arvados-api/current/config/application.default.yml@ for a full list of configuration options. (But don't edit it. Edit @application.yml@ instead.)

h2. Prepare the API server deployment

Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh@.  This will install and check your configuration, install necessary gems, and run any necessary database setup.

{% include 'notebox_begin' %}
You can safely ignore the following error message you may see when loading the database structure:
<notextile>
<pre><code>ERROR:  must be owner of extension plpgsql</code></pre></notextile>
{% include 'notebox_end' %}

This command aborts when it encounters an error.  It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.

h2. Set up Web servers

For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets.  To do that:

<notextile>
<ol>
<li>Install Nginx via your distribution or a backports repository.</li>

<li><a href="https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html">Install Phusion Passenger for Nginx</a>.</li>

<li><p>Puma is already included with the API server's gems.  We recommend you use a tool like <a href="http://smarden.org/runit/">runit</a> or something similar.  Here's a sample run script for that:</p>

<pre><code>#!/bin/bash

set -e
exec 2>&1

# Uncomment the line below if you're using RVM.
#source /etc/profile.d/rvm.sh

envdir="`pwd`/env"
mkdir -p "$envdir"
echo ws-only > "$envdir/ARVADOS_WEBSOCKETS"

cd /var/www/arvados-api/current
echo "Starting puma in `pwd`"

# You may need to change arguments below to match your deployment, especially -u.
exec chpst -m 1073741824 -u www-data:www-data -e "$envdir" \
  bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
</code></pre>
</li>

<li><p>Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for both it and Puma.  You might add a block like the following, adding SSL and logging parameters to taste:</p>

<pre><code>server {
  listen 127.0.0.1:8000;
  server_name localhost-api;

  root /var/www/arvados-api/current/public;
  index  index.html index.htm index.php;

  passenger_enabled on;
  # If you're using RVM, uncomment the line below.
  #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
}

upstream api {
  server     127.0.0.1:8000  fail_timeout=10s;
}

upstream websockets {
  # The address below must match the one specified in puma's -b option.
  server     127.0.0.1:8100  fail_timeout=10s;
}

proxy_http_version 1.1;

server {
  listen       <span class="userinput">[your public IP address]</span>:443 ssl;
  server_name  <span class="userinput">uuid_prefix.your.domain</span>;

  ssl on;

  index  index.html index.htm index.php;

  location / {
    proxy_pass            http://api;
    proxy_redirect        off;

    proxy_set_header      X-Forwarded-Proto https;
    proxy_set_header      Host $http_host;
    proxy_set_header      X-External-Client $external_client;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

server {
  listen       <span class="userinput">[your public IP address]</span>:443 ssl;
  server_name  ws.<span class="userinput">uuid_prefix.your.domain</span>;

  ssl on;

  index  index.html index.htm index.php;

  location / {
    proxy_pass            http://websockets;
    proxy_redirect        off;

    proxy_set_header      Upgrade $http_upgrade;
    proxy_set_header      Connection "upgrade";
    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}
</code></pre>
</li>

<li>Restart Nginx.</li>

</ol>
</notextile>