X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/cbfc8eea7f3fc96f478530c77441b7175a043a17..a9335a762f70e30affdb259e2ff487f27963f1c8:/doc/install/install-api-server.html.textile.liquid?ds=sidebyside diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid index ccfc58ecf4..91e2c69892 100644 --- a/doc/install/install-api-server.html.textile.liquid +++ b/doc/install/install-api-server.html.textile.liquid @@ -4,125 +4,335 @@ navsection: installguide title: Install the API server ... -h2. Prerequisites: +h2. Install prerequisites -# A GNU/Linux (virtual) machine -# A domain name for your api server -# Ruby >= 2.0.0 -# Bundler: @gem install bundler@ -# Curl libraries: @sudo apt-get install libcurl3 libcurl3-gnutls libcurl4-openssl-dev@ +The Arvados package repository includes an API server package that can help automate much of the deployment. -h2. Download the source tree +h3(#install_ruby_and_bundler). Install Ruby and Bundler - -
~$ git clone https://github.com/curoverse/arvados.git
-
+{% include 'install_ruby_and_bundler' %} -See also: "Downloading the source code:https://arvados.org/projects/arvados/wiki/Download on the Arvados wiki. +h3(#install_postgres). Install PostgreSQL -h2. Install gem dependencies +{% include 'install_postgres' %} - -
~$ cd arvados/services/api
-~/arvados/services/api$ bundle install
-
+h2(#install_apiserver). Install API server and dependencies + +On a Debian-based system, install the following packages: -h2. Configure the API server + +
~$ sudo apt-get install bison build-essential libcurl4-openssl-dev git arvados-api-server
+
+
-Edit the main configuration: +On a Red Hat-based system, install the following packages: -
~/arvados/services/api$ cp -i config/application.yml.example config/application.yml
-
+
~$ sudo yum install bison make automake gcc gcc-c++ libcurl-devel git arvados-api-server
+
+ -Choose a unique 5-character alphanumeric string to use as your @uuid_prefix@. An example is given that generates a 5-character string based on a hash of your hostname. The @uuid_prefix@ is a unique identifier for your API server. It also serves as the first part of the hostname for your API server. +{% include 'install_git' %} -For a development site, use your own domain instead of arvadosapi.com. +h2(#configure). Set up the database -Make sure a clone of the arvados repository exists in @git_repositories_dir@: +Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one: -
~/arvados/services/api$ sudo mkdir -p /var/cache/git
-~/arvados/services/api$ sudo git clone --bare ../../.git /var/cache/git/arvados.git
+
~$ ruby -e 'puts rand(2**128).to_s(36)'
+6gqa1vu492idd7yca9tfandj3
 
-Generate a new secret token for signing cookies: +Create a new database user. -
~/arvados/services/api$ rake secret
-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+
~$ sudo -u postgres createuser --encrypted -R -S --pwprompt arvados
+[sudo] password for you: yourpassword
+Enter password for new role: paste-password-you-generated
+Enter it again: paste-password-again
 
-Put it in @config/application.yml@ in the production or common section: +{% assign pg_hba_path = "/opt/rh/postgresql92/root/var/lib/pgsql/data/pg_hba.conf" %} +{% assign pg_service = "postgresql92-postgresql" %} +{% include 'install_redhat_postgres_auth' %} + +Create the database: -
    secret_token: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+
~$ sudo -u postgres createdb arvados_production -T template0 -E UTF8 -O arvados
 
-Consult @application.default.yml@ for a full list of configuration options. Always put your local configuration in @application.yml@ instead of editing @application.default.yml@. +h2. Configure the database connection + +Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above. + +h2(#configure_application). Configure the API server + +Edit @/etc/arvados/api/application.yml@ to configure the settings described in the following sections. The API server reads both @application.yml@ and its own @config/application.default.yml@ file. The settings in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@. The @config/application.yml.example@ file is not read by the API server and is provided as a starting template only. + +@config/application.default.yml@ documents additional configuration settings not listed here. You can "view the current source version":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml for reference. + +Only put local configuration in @application.yml@. Do not edit @application.default.yml@. + +h3(#uuid_prefix). uuid_prefix + +Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment. This prefix is used for all database identifiers to identify the record as originating from this site. It must be exactly 5 lowercase ASCII letters and digits. -Configure the database: +Example @application.yml@: -
~/arvados/services/api$ cp -i config/database.yml.sample config/database.yml
-
+
  uuid_prefix: zzzzz
+ + +h3. secret_token -By default, the development database will use the sqlite3 driver, so no configuration is necessary. If you wish to use mysql or postgres, edit @config/database.yml@ to your liking and make sure the database and db user exist. Then initialize the database: +The @secret_token@ is used for for signing cookies. IMPORTANT: This is a site secret. It should be at least 50 characters. Generate a random value and set it in @application.yml@: -
~/arvados/services/api$ RAILS_ENV=development bundle exec rake db:setup
+
~$ ruby -e 'puts rand(2**400).to_s(36)'
+yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
 
-Set up omniauth: +Example @application.yml@: -
~/arvados/services/api$ cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
-
+
  secret_token: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
+ -Edit @config/initializers/omniauth.rb@. Set @APP_SECRET@ to the value of @app_secret@ from "installing the single sign on server":install-sso.html . +h3(#blob_signing_key). blob_signing_key -You can now run the development server: +The @blob_signing_key@ is used to enforce access control to Keep blocks. This same key must be provided to the Keepstore daemons when "installing Keepstore servers.":install-keepstore.html IMPORTANT: This is a site secret. It should be at least 50 characters. Generate a random value and set it in @application.yml@: -
~/arvados/services/api$ rails server
+
~$ ruby -e 'puts rand(2**400).to_s(36)'
+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
-h3. Apache/Passenger (optional) +Example @application.yml@: + + +
  blob_signing_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+ +h3(#omniauth). sso_app_secret, sso_app_id, sso_provider_url + +The following settings enable the API server to communicate with the "Single Sign On (SSO) server":install-sso.html to authenticate user log in. -You can use "Passenger":https://www.phusionpassenger.com/ for deployment. Point it to the services/api directory in the source tree. +Set @sso_provider_url@ to the base URL where your SSO server is installed. This should be a URL consisting of the scheme and host (and optionally, port), without a trailing slash. -To enable streaming so users can monitor crunch jobs in real time, add to your Passenger configuration in Apache: +Set @sso_app_secret@ and @sso_app_id@ to the corresponding values for @app_secret@ and @app_id@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step. + +Example @application.yml@: -
PassengerBufferResponse off
+
  sso_app_id: arvados-server
+  sso_app_secret: wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
+  sso_provider_url: https://sso.example.com
 
-h2. Add an admin user +h3. workbench_address -Point browser to the API endpoint. Log in with a google account. +Set @workbench_address@ to the URL of your workbench application after following "Install Workbench.":install-workbench-app.html -In the rails console: +Example @application.yml@: -
~/arvados/services/api$ rails console
-irb(main):001:0> Thread.current[:user] = User.find(1)
-irb(main):002:0> Thread.current[:user].is_admin = true
-irb(main):003:0> User.find(1).update_attributes is_admin: true, is_active: true
-irb(main):004:0> User.find(1).is_admin
-=> true
-
+
  workbench_address: https://workbench.zzzzz.example.com
+ -h2. Create an API token +h3. websocket_address -In rails console: +Set @websocket_address@ to the @wss://@ URL of the API server websocket endpoint after following "Set up Web servers":#set_up. The path of the default endpoint is @/websocket@. + +Example @application.yml@: -
~/arvados/services/api$ rails console
-irb(main):001:0> a = ApiClient.new(owner_uuid:'0')
-irb(main):002:0> a.save!
-irb(main):003:0> x = ApiClientAuthorization.new(api_client_id:a.id, user_id:1)
-irb(main):004:0> x.save
-irb(main):005:0> x.api_token
+
  websocket_address: wss://ws.zzzzz.example.com/websocket
+ + +h3(#git_repositories_dir). git_repositories_dir + +The @git_repositories_dir@ setting specifies the directory where user git repositories will be stored. + +The git server setup process is covered on "its own page":install-arv-git-httpd.html. For now, create an empty directory in the default location: + + +
~$ sudo mkdir -p /var/lib/arvados/git/repositories
 
+ +If you intend to store your git repositories in a different location, specify that location in @application.yml@. + +Default setting in @application.default.yml@: + + +
  git_repositories_dir: /var/lib/arvados/git/repositories
+
+
+ +h3(#git_internal_dir). git_internal_dir + +The @git_internal_dir@ setting specifies the location of Arvados' internal git repository. By default this is @/var/lib/arvados/internal.git@. This repository stores git commits that have been used to run Crunch jobs. It should _not_ be a subdirectory of @git_repositories_dir@. + +Example @application.yml@: + + +
  git_internal_dir: /var/lib/arvados/internal.git
+
+
+ +h2(#set_up). Set up Web servers + +For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets. To do that: + + +
    +
  1. Install Nginx and Phusion Passenger.
  2. + +
  3. Install runit to supervise the Puma daemon. {% include 'install_runit' %}

  4. + +
  5. Install the script below as the run script for the Puma service, modifying it as directed by the comments.

    + +
    #!/bin/bash
    +
    +set -e
    +exec 2>&1
    +
    +# Uncomment the line below if you're using RVM.
    +#source /etc/profile.d/rvm.sh
    +
    +envdir="`pwd`/env"
    +mkdir -p "$envdir"
    +echo ws-only > "$envdir/ARVADOS_WEBSOCKETS"
    +
    +cd /var/www/arvados-api/current
    +echo "Starting puma in `pwd`"
    +
    +# Change arguments below to match your deployment, "webserver-user" and
    +# "webserver-group" should be changed to the user and group of the web server
    +# process.  This is typically "www-data:www-data" on Debian systems by default,
    +# other systems may use different defaults such the name of the web server
    +# software (for example, "nginx:nginx").
    +exec chpst -m 1073741824 -u webserver-user:webserver-group -e "$envdir" \
    +  bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
    +
    +
  6. + +
  7. Edit the http section of your Nginx configuration to run the Passenger server, and act as a front-end for both it and Puma. You might add a block like the following, adding SSL and logging parameters to taste:

    + +
    server {
    +  listen 127.0.0.1:8000;
    +  server_name localhost-api;
    +
    +  root /var/www/arvados-api/current/public;
    +  index  index.html index.htm index.php;
    +
    +  passenger_enabled on;
    +  # If you're using RVM, uncomment the line below.
    +  #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
    +
    +  # This value effectively limits the size of API objects users can
    +  # create, especially collections.  If you change this, you should
    +  # also ensure the following settings match it:
    +  # * `client_max_body_size` in the server section below
    +  # * `client_max_body_size` in the Workbench Nginx configuration (twice)
    +  # * `max_request_size` in the API server's application.yml file
    +  client_max_body_size 128m;
    +}
    +
    +upstream api {
    +  server     127.0.0.1:8000  fail_timeout=10s;
    +}
    +
    +upstream websockets {
    +  # The address below must match the one specified in puma's -b option.
    +  server     127.0.0.1:8100  fail_timeout=10s;
    +}
    +
    +proxy_http_version 1.1;
    +
    +# When Keep clients request a list of Keep services from the API server, the
    +# server will automatically return the list of available proxies if
    +# the request headers include X-External-Client: 1.  Following the example
    +# here, at the end of this section, add a line for each netmask that has
    +# direct access to Keep storage daemons to set this header value to 0.
    +geo $external_client {
    +  default        1;
    +  10.20.30.0/24  0;
    +}
    +
    +server {
    +  listen       [your public IP address]:443 ssl;
    +  server_name  uuid_prefix.your.domain;
    +
    +  ssl on;
    +  ssl_certificate     /YOUR/PATH/TO/cert.pem;
    +  ssl_certificate_key /YOUR/PATH/TO/cert.key;
    +
    +  index  index.html index.htm index.php;
    +
    +  # Refer to the comment about this setting in the server section above.
    +  client_max_body_size 128m;
    +
    +  location / {
    +    proxy_pass            http://api;
    +    proxy_redirect        off;
    +    proxy_connect_timeout 90s;
    +    proxy_read_timeout    300s;
    +
    +    proxy_set_header      X-Forwarded-Proto https;
    +    proxy_set_header      Host $http_host;
    +    proxy_set_header      X-External-Client $external_client;
    +    proxy_set_header      X-Real-IP $remote_addr;
    +    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    +  }
    +}
    +
    +server {
    +  listen       [your public IP address]:443 ssl;
    +  server_name  ws.uuid_prefix.your.domain;
    +
    +  ssl on;
    +  ssl_certificate     /YOUR/PATH/TO/cert.pem;
    +  ssl_certificate_key /YOUR/PATH/TO/cert.key;
    +
    +  index  index.html index.htm index.php;
    +
    +  location / {
    +    proxy_pass            http://websockets;
    +    proxy_redirect        off;
    +    proxy_connect_timeout 90s;
    +    proxy_read_timeout    300s;
    +
    +    proxy_set_header      Upgrade $http_upgrade;
    +    proxy_set_header      Connection "upgrade";
    +    proxy_set_header      Host $host;
    +    proxy_set_header      X-Real-IP $remote_addr;
    +    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    +  }
    +}
    +
    +
  8. + +
  9. Restart Nginx:

    + +
    ~$ sudo nginx -s reload
    +
    + +
  10. + +
+
+ +h2. Prepare the API server deployment + +{% assign railspkg = "arvados-api-server" %} +{% include 'install_rails_reconfigure' %} + +{% include 'notebox_begin' %} +You can safely ignore the following messages if they appear while this command runs: +
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will
+break this application for all non-root users on this machine.
+
fatal: Not a git repository (or any of the parent directories): .git
+{% include 'notebox_end' %}