X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/17f836f358b242798a34e34abb0ec4b12cdff1df..2b41829bf0a889558c320121710ef3fd2e90ef7e:/doc/install/install-api-server.html.textile.liquid diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid index 3b398356d3..77a90e0f52 100644 --- a/doc/install/install-api-server.html.textile.liquid +++ b/doc/install/install-api-server.html.textile.liquid @@ -4,163 +4,255 @@ navsection: installguide title: Install the API server ... -h2. Prerequisites: +h2. Install prerequisites -# A GNU/Linux (virtual) machine -# A domain name for your api server +The Arvados package repository includes an API server package that can help automate much of the deployment. It requires: -h2(#dependencies). Install dependencies +* PostgreSQL 9.0+ +* "Ruby 2.1 and bundler":install-manual-prerequisites-ruby.html +* Build tools and the curl and PostgreSQL development libraries, to build gem dependencies +* Nginx + +On older distributions, you may need to use a backports repository to satisfy these requirements. For example, on older Red Hat-based systems, consider using the "postgresql92":https://www.softwarecollections.org/en/scls/rhscl/postgresql92/ and "nginx16":https://www.softwarecollections.org/en/scls/rhscl/nginx16/ Software Collections. + +On a Debian-based system, install the following packages: + + +
~$ sudo apt-get install bison build-essential libpq-dev libcurl4-openssl-dev postgresql git nginx arvados-api-server
+
+
+ +On a Red Hat-based system, install the following packages: -
~$ sudo apt-get install \
-    bison build-essential gettext libcurl3 libcurl3-gnutls \
-    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
-    libsqlite3-dev libssl-dev libxslt1.1 postgresql sqlite3 sudo \
-    wget zlib1g-dev
-
+
~$ sudo yum install bison make automake gcc gcc-c++ libcurl-devel postgresql-server postgresql-devel nginx git arvados-api-server
+
+ + +{% include 'notebox_begin' %} -h2(#ruby). Install Ruby and bundler +If you intend to use specific versions of these packages from Software Collections, you may have to adapt some of the package names to match; e.g., @postgresql92-postgresql-server postgresql92-postgresql-devel nginx16@. -We recommend Ruby >= 2.1. +{% include 'notebox_end' %} + +h2. Set up the database + +Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one: -
mkdir -p ~/src
-cd ~/src
-wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz
-tar xzf ruby-2.1.2.tar.gz
-cd ruby-2.1.2
-./configure
-make
-sudo make install
-
-sudo gem install bundler
+
~$ ruby -e 'puts rand(2**128).to_s(36)'
+6gqa1vu492idd7yca9tfandj3
 
-h2. Download the source tree +Create a new database user. -
~$ cd $HOME # (or wherever you want to install)
-~$ git clone https://github.com/curoverse/arvados.git
+
~$ sudo -u postgres createuser --encrypted -R -S --pwprompt arvados
+[sudo] password for you: yourpassword
+Enter password for new role: paste-password-you-generated
+Enter it again: paste-password-again
 
-See also: "Downloading the source code":https://arvados.org/projects/arvados/wiki/Download on the Arvados wiki. +{% include 'notebox_begin' %} + +This user setup assumes that your PostgreSQL is configured to accept password authentication. Red Hat systems use ident-based authentication by default. You may need to either adapt the user creation, or reconfigure PostgreSQL (in @pg_hba.conf@) to accept password authentication. -h2. Install gem dependencies +{% include 'notebox_end' %} + +Create the database: -
~$ cd arvados/services/api
-~/arvados/services/api$ bundle install
-
+
~$ sudo -u postgres createdb arvados_production -T template0 -E UTF8 -O arvados
+
+ -h2. Configure the API server +h2. Set up configuration files -Edit the main configuration: +The API server package uses configuration files that you write to @/etc/arvados/api@ and ensures they're consistently deployed. Create this directory and copy the example configuration files to it: -
~/arvados/services/api$ cp -i config/application.yml.example config/application.yml
-
+
~$ sudo mkdir -p /etc/arvados/api
+~$ sudo chmod 700 /etc/arvados/api
+~$ cd /var/www/arvados-api/current
+/var/www/arvados-api/current$ sudo cp config/initializers/omniauth.rb.example /etc/arvados/api/omniauth.rb
+/var/www/arvados-api/current$ sudo cp config/database.yml.sample /etc/arvados/api/database.yml
+/var/www/arvados-api/current$ sudo cp config/application.yml.example /etc/arvados/api/application.yml
+
+ + +h2. Configure the database connection + +Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above. + +h2. Configure the API server -Choose a unique 5-character alphanumeric string to use as your @uuid_prefix@. An example is given that generates a 5-character string based on a hash of your hostname. The @uuid_prefix@ is a unique identifier for your API server. It also serves as the first part of the hostname for your API server. +Edit @/etc/arvados/api/application.yml@ following the instructions below. The deployment script will consistently deploy this to the API server's configuration directory. The API server reads both @application.yml@ and its own @config/application.default.yml@ file. Values in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@. The @config/application.yml.example@ file is not read by the API server and is provided for installation convenience only. -For a development site, use your own domain instead of arvadosapi.com. +Always put your local configuration in @application.yml@ instead of editing @application.default.yml@. -Make sure a clone of the arvados repository exists in @git_repositories_dir@: +h3(#uuid_prefix). uuid_prefix + +Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment. This prefix is used for all database identifiers to identify the record as originating from this site. It must be exactly 5 alphanumeric characters (lowercase ASCII letters and digits). + +h3(#git_repositories_dir). git_repositories_dir + +This field defaults to @/var/lib/arvados/git@. You can override the value by defining it in @application.yml@. + +Make sure a clone of the arvados repository exists in @git_repositories_dir@. -
~/arvados/services/api$ sudo mkdir -p /var/cache/git
-~/arvados/services/api$ sudo git clone --bare ../../.git /var/cache/git/arvados.git
+
~$ sudo mkdir -p /var/lib/arvados/git
+~$ sudo git clone --bare git://git.curoverse.com/arvados.git /var/lib/arvados/git/arvados.git
 
+h3. secret_token + Generate a new secret token for signing cookies: -
~/arvados/services/api$ rake secret
+
~$ ruby -e 'puts rand(2**400).to_s(36)'
 zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
 
-Put it in @config/application.yml@ in the production or common section: +Then put that value in the @secret_token@ field. + +h3. blob_signing_key + +If you want access control on your "Keepstore":install-keepstore.html server(s), you should set @blob_signing_key@ to the same value as the permission key you provide to your Keepstore daemon(s). + +h3. workbench_address + +Fill in the url of your workbench application in @workbench_address@, for example + +  https://workbench.@prefix_uuid@.your.domain + +h3(#omniauth). sso_app_id, sso_app_secret, sso_provider_url + +For @sso_app_id@ and @sso_app_secret@, provide the same @app_id@ and @app_secret@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step. + +For @sso_provider_url@, provide the base URL where your SSO server is installed: just the scheme and host, with no trailing slash. -
    secret_token: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+
  sso_app_id: arvados-server
+  sso_app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+  sso_provider_url: https://sso.example.com
 
-Consult @application.default.yml@ for a full list of configuration options. Always put your local configuration in @application.yml@ instead of editing @application.default.yml@. +h3. Other options -Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one: +Consult @/var/www/arvados-api/current/config/application.default.yml@ for a full list of configuration options. (But don't edit it. Edit @application.yml@ instead.) - -
~/arvados/services/api$ ruby -e 'puts rand(2**128).to_s(36)'
-6gqa1vu492idd7yca9tfandj3
-
+h2. Prepare the API server deployment -Create a new database user with permission to create its own databases. +Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh@. This will install and check your configuration, install necessary gems, and run any necessary database setup. +{% include 'notebox_begin' %} +You can safely ignore the following error message you may see when loading the database structure: -
~/arvados/services/api$ sudo -u postgres createuser --createdb --encrypted --pwprompt arvados
-[sudo] password for you: yourpassword
-Enter password for new role: paste-password-you-generated
-Enter it again: paste-password-again
-Shall the new role be a superuser? (y/n) n
-Shall the new role be allowed to create more new roles? (y/n) n
-
+
ERROR:  must be owner of extension plpgsql
+{% include 'notebox_end' %} -Configure API server to connect to your database by creating and updating @config/database.yml@. Replace the @xxxxxxxx@ database password placeholders with the new password you generated above. +This command aborts when it encounters an error. It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again. - -
~/arvados/services/api$ cp -i config/database.yml.sample config/database.yml
-~/arvados/services/api$ edit config/database.yml
-
+h2. Set up Web servers -Create and initialize the database. +For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets. To do that: -
~/arvados/services/api$ RAILS_ENV=development bundle exec rake db:setup
-
+
    +
  1. Install Nginx via your distribution or a backports repository.
  2. -Set up omniauth: +
  3. Install Phusion Passenger for Nginx.
  4. - -
    ~/arvados/services/api$ cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
    -
    +
  5. Puma is already included with the API server's gems. We recommend you use a tool like runit or something similar. Here's a sample run script for that:

    -Edit @config/initializers/omniauth.rb@. Set @APP_SECRET@ to the value of @app_secret@ from "installing the single sign on server":install-sso.html . +
    #!/bin/bash
     
    -You can now run the development server:
    +set -e
    +# Uncomment the line below if you're using RVM.
    +#source /etc/profile.d/rvm.sh
     
    -
    -
    ~/arvados/services/api$ bundle exec rails server --port=3030
    -
    +envdir="/etc/sv/puma/env" +root=/etc/sv/puma +echo "Starting puma from ${root}" +cd $root +mkdir -p "${envdir}" +exec 2>&1 +cd /var/www/arvados-api/current +# You may need to change arguments below to match your deployment, especially -u. +exec chpst -e "${envdir}" -m 1073741824 -u www-data:www-data bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100 +
    +
  6. -h3. Apache/Passenger (optional) +
  7. Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for both it and Puma. You might add a block like the following, adding SSL and logging parameters to taste:

    -You can use "Passenger":https://www.phusionpassenger.com/ for deployment. Point it to the services/api directory in the source tree. +
    server {
    +  listen 127.0.0.1:8000;
    +  server_name localhost-api;
     
    -To enable streaming so users can monitor crunch jobs in real time, add to your Passenger configuration in Apache:
    +  root /var/www/arvados-api/current/public;
    +  index  index.html index.htm index.php;
     
    -
    -
    PassengerBufferResponse off
    -
    -
    + passenger_enabled on; + # If you're using RVM, uncomment the line below. + #passenger_ruby /usr/local/rvm/wrappers/default/ruby; +} -h2(#admin-user). Add an admin user +upstream api { + server 127.0.0.1:8000 fail_timeout=10s; +} -Point your browser to the API server's login endpoint: +upstream websockets { + # The address below must match the one specified in puma's -b option. + server 127.0.0.1:8100 fail_timeout=10s; +} - -
    https://localhost:3030/login
    -
    -
    +proxy_http_version 1.1; -Log in with your google account. +server { + listen [your public IP address]:443 ssl; + server_name uuid-prefix.your.domain; -Use the rails console to give yourself admin privileges: + ssl on; - -
    ~/arvados/services/api$ bundle exec rails console
    -irb(main):001:0> Thread.current[:user] = User.all.select(&:identity_url).last
    -irb(main):002:0> Thread.current[:user].is_admin = true
    -irb(main):003:0> Thread.current[:user].update_attributes is_admin: true, is_active: true
    -irb(main):004:0> User.where(is_admin: true).collect &:email
    -=> ["root", "your_address@example.com"]
    -
    + index index.html index.htm index.php; + + location / { + proxy_pass http://api; + proxy_redirect off; + + proxy_set_header X-Forwarded-Proto https; + proxy_set_header Host $http_host; + proxy_set_header X-External-Client $external_client; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } +} + +server { + listen [your public IP address]:443 ssl; + server_name ws.uuid-prefix.your.domain; + + ssl on; + + index index.html index.htm index.php; + + location / { + proxy_pass http://websockets; + proxy_redirect off; + + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } +} +
    +
  8. + +
  9. Restart Nginx.
  10. + +
+