X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/847f47a1a86d40dc6ae5d13f62039e55d1afa36d..9024a5b5eb247e3a552dbb92a61f7c916c0d4349:/doc/install/install-api-server.html.textile.liquid diff --git a/doc/install/install-api-server.html.textile.liquid b/doc/install/install-api-server.html.textile.liquid index bdd9379612..464559a74c 100644 --- a/doc/install/install-api-server.html.textile.liquid +++ b/doc/install/install-api-server.html.textile.liquid @@ -6,39 +6,35 @@ title: Install the API server h2. Install prerequisites -The Arvados package repository includes an API server package that can help automate much of the deployment. It requires: +The Arvados package repository includes an API server package that can help automate much of the deployment. -* PostgreSQL 9.0+ -* "Ruby 2.1 and bundler":install-manual-prerequisites-ruby.html -* Build tools and the curl and PostgreSQL development libraries, to build gem dependencies -* Nginx +h3(#install_ruby_and_bundler). Install Ruby and Bundler -On older distributions, you may need to use a backports repository to satisfy these requirements. For example, on older Red Hat-based systems, consider using the "postgresql92":https://www.softwarecollections.org/en/scls/rhscl/postgresql92/ and "nginx16":https://www.softwarecollections.org/en/scls/rhscl/nginx16/ Software Collections. +{% include 'install_ruby_and_bundler' %} + +h3(#install_postgres). Install PostgreSQL + +{% include 'install_postgres' %} + +h2(#install_apiserver). Install API server and dependencies On a Debian-based system, install the following packages: -
~$ sudo apt-get install bison build-essential libpq-dev libcurl4-openssl-dev postgresql git nginx arvados-api-server
+
~$ sudo apt-get install bison build-essential libcurl4-openssl-dev git arvados-api-server
 

 
 
 On a Red Hat-based system, install the following packages:
 
 
-
~$ sudo yum install bison make automake gcc gcc-c++ libcurl-devel postgresql-server postgresql-devel nginx git arvados-api-server
+
~$ sudo yum install bison make automake gcc gcc-c++ libcurl-devel git arvados-api-server
 

 
 
-{% include 'notebox_begin' %}
-
-If you intend to use specific versions of these packages from Software Collections, you may have to adapt some of the package names to match. For example:
+{% include 'install_git' %}
 
-
-
~$ sudo yum install postgresql92 postgresql92-postgresql-devel nginx16

-
-{% include 'notebox_end' %}
-
-h2. Set up the database
+h2(#configure). Set up the database
 
 Generate a new database password. Nobody ever needs to memorize it or type it, so we'll make a strong one:
 
@@ -56,11 +52,9 @@ Enter password for new role: paste-password-you-generate
 Enter it again: paste-password-again
 

 
-{% include 'notebox_begin' %}
-
-This user setup assumes that your PostgreSQL is configured to accept password authentication.  Red Hat systems use ident-based authentication by default.  You may need to either adapt the user creation, or reconfigure PostgreSQL (in @pg_hba.conf@) to accept password authentication.
-
-{% include 'notebox_end' %}
+{% assign pg_hba_path = "/opt/rh/postgresql92/root/var/lib/pgsql/data/pg_hba.conf" %}
+{% assign pg_service = "postgresql92-postgresql" %}
+{% include 'install_redhat_postgres_auth' %}
 
 Create the database:
 
@@ -77,8 +71,7 @@ The API server package uses configuration files that you write to @/etc/arvados/
 
~$ sudo mkdir -p /etc/arvados/api
 ~$ sudo chmod 700 /etc/arvados/api
 ~$ cd /var/www/arvados-api/current
-/var/www/arvados-api/current$ sudo cp config/initializers/omniauth.rb.example /etc/arvados/api/omniauth.rb
-/var/www/arvados-api/current$ sudo cp config/database.yml.sample /etc/arvados/api/database.yml
+/var/www/arvados-api/current$ sudo cp config/database.yml.example /etc/arvados/api/database.yml
 /var/www/arvados-api/current$ sudo cp config/application.yml.example /etc/arvados/api/application.yml
 

 
@@ -87,104 +80,155 @@ h2. Configure the database connection
 
 Edit @/etc/arvados/api/database.yml@ and replace the @xxxxxxxx@ database password placeholders with the PostgreSQL password you generated above.
 
-h2. Configure the API server
+h2(#configure_application). Configure the API server
 
-Edit @/etc/arvados/api/application.yml@ following the instructions below.  The deployment script will consistently deploy this to the API server's configuration directory.  The API server reads both @application.yml@ and its own @config/application.default.yml@ file.  Values in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@.  The @config/application.yml.example@ file is not read by the API server and is provided for installation convenience only.
+Edit @/etc/arvados/api/application.yml@ to configure the settings described in the following sections.  The deployment script will consistently deploy this to the API server's configuration directory.  The API server reads both @application.yml@ and its own @config/application.default.yml@ file.  The settings in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@.  The @config/application.yml.example@ file is not read by the API server and is provided as a starting template only.
 
-Always put your local configuration in @application.yml@ instead of editing @application.default.yml@.
+@config/application.default.yml@ documents additional configuration settings not listed here.  You can "view the current source version":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml for reference.
+
+Only put local configuration in @application.yml@.  Do not edit @application.default.yml@.
 
 h3(#uuid_prefix). uuid_prefix
 
-Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment.  This prefix is used for all database identifiers to identify the record as originating from this site.  It must be exactly 5 alphanumeric characters (lowercase ASCII letters and digits).
+Define your @uuid_prefix@ in @application.yml@ by setting the @uuid_prefix@ field in the section for your environment.  This prefix is used for all database identifiers to identify the record as originating from this site.  It must be exactly 5 lowercase ASCII letters and digits.
 
-h3(#git_repositories_dir). git_repositories_dir
+Example @application.yml@:
 
-This field defaults to @/var/lib/arvados/git@. You can override the value by defining it in @application.yml@.
+
+
  uuid_prefix: zzzzz

+
+
+h3. secret_token
 
-Make sure a clone of the arvados repository exists in @git_repositories_dir@.
+The @secret_token@ is used for for signing cookies.  IMPORTANT: This is a site secret. It should be at least 50 characters.  Generate a random value and set it in @application.yml@:
 
 
-
~$ sudo mkdir -p /var/lib/arvados/git
-~$ sudo git clone --bare git://git.curoverse.com/arvados.git /var/lib/arvados/git/arvados.git
+
~$ ruby -e 'puts rand(2**400).to_s(36)'
+yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
 

 
-h3. secret_token
+Example @application.yml@:
+
+
+
  secret_token: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

+
 
-Generate a new secret token for signing cookies:
+h3(#blob_signing_key). blob_signing_key
+
+The @blob_signing_key@ is used to enforce access control to Keep blocks.  This same key must be provided to the Keepstore daemons when "installing Keepstore servers.":install-keepstore.html  IMPORTANT: This is a site secret. It should be at least 50 characters.  Generate a random value and set it in @application.yml@:
 
 
 
~$ ruby -e 'puts rand(2**400).to_s(36)'
-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 

 
-Then put that value in the @secret_token@ field.
+Example @application.yml@:
+
+
+
  blob_signing_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

+
+
+h3(#omniauth). sso_app_secret, sso_app_id, sso_provider_url
+
+The following settings enable the API server to communicate with the "Single Sign On (SSO) server":install-sso.html to authenticate user log in.
+
+Set @sso_provider_url@ to the base URL where your SSO server is installed.  This should be a URL consisting of the scheme and host (and optionally, port), without a trailing slash.
 
-h3. blob_signing_key
+Set @sso_app_secret@ and @sso_app_id@ to the corresponding values for @app_secret@ and @app_id@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step.
 
-If you want access control on your "Keepstore":install-keepstore.html server(s), you should set @blob_signing_key@ to the same value as the permission key you provide to your Keepstore daemon(s).
+Example @application.yml@:
+
+
+
  sso_app_id: arvados-server
+  sso_app_secret: wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
+  sso_provider_url: https://sso.example.com
+

+
 
 h3. workbench_address
 
-Fill in the url of your workbench application in @workbench_address@, for example
+Set @workbench_address@ to the URL of your workbench application after following "Install Workbench.":install-workbench-app.html
 
-  https://workbench.@prefix_uuid@.your.domain
+Example @application.yml@:
 
-h3(#omniauth). sso_app_id, sso_app_secret, sso_provider_url
+
+
  workbench_address: https://workbench.zzzzz.example.com

+
 
-For @sso_app_id@ and @sso_app_secret@, provide the same @app_id@ and @app_secret@ used in the "Create arvados-server client for Single Sign On (SSO)":install-sso.html#client step.
+h3. websocket_address
 
-For @sso_provider_url@, provide the base URL where your SSO server is installed: just the scheme and host, with no trailing slash.
+Set @websocket_address@ to the @wss://@ URL of the API server websocket endpoint after following "Set up Web servers":#set_up.  The path of the default endpoint is @/websocket@.
+
+Example @application.yml@:
 
 
-
  sso_app_id: arvados-server
-  sso_app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-  sso_provider_url: https://sso.example.com
-

+
  websocket_address: wss://ws.zzzzz.example.com/websocket

 
 
-h3. Other options
+h3(#git_repositories_dir). git_repositories_dir
 
-Consult @/var/www/arvados-api/current/config/application.default.yml@ for a full list of configuration options. (But don't edit it. Edit @application.yml@ instead.)
+The @git_repositories_dir@ setting specifies the directory where user git repositories will be stored.
 
-h2. Prepare the API server deployment
+The git server setup process is covered on "its own page":install-arv-git-httpd.html. For now, create an empty directory in the default location:
 
-Now that all your configuration is in place, run @/usr/local/bin/arvados-api-server-upgrade.sh@.  This will install and check your configuration, install necessary gems, and run any necessary database setup.
+
+
~$ sudo mkdir -p /var/lib/arvados/git/repositories
+

+
+If you intend to store your git repositories in a different location, specify that location in @application.yml@.
+
+Default setting in @application.default.yml@:
 
-{% include 'notebox_begin' %}
-You can safely ignore the following error message you may see when loading the database structure:
 
-
ERROR:  must be owner of extension plpgsql

-{% include 'notebox_end' %}
+
  git_repositories_dir: /var/lib/arvados/git/repositories
+

+
+
+h3(#git_internal_dir). git_internal_dir
+
+The @git_internal_dir@ setting specifies the location of Arvados' internal git repository.  By default this is @/var/lib/arvados/internal.git@.  This repository stores git commits that have been used to run Crunch jobs.  It should _not_ be a subdirectory of @git_repositories_dir@.
 
-This command aborts when it encounters an error.  It's safe to rerun multiple times, so if there's a problem with your configuration, you can fix that and try again.
+Example @application.yml@:
+
+
+
  git_internal_dir: /var/lib/arvados/internal.git
+

+
 
-h2. Set up Web servers
+h2(#set_up). Set up Web servers
 
 For best performance, we recommend you use Nginx as your Web server front-end, with a Passenger backend for the main API server and a Puma backend for API server Websockets.  To do that:
 
 
 
    
-
  1. Install Nginx via your distribution or a backports repository.
    2. 
+
  2. Install Nginx and Phusion Passenger.
    3. 
 
-
  3. Install Phusion Passenger for Nginx.
    4. 
+
  4. Install runit to supervise the Puma daemon.  {% include 'install_runit' %}
    5. 
 
-
  5. Puma is already included with the API server's gems.  We recommend you use a tool like runit or something similar.  Here's a sample run script for that:
    
+
  6. Install the script below as the run script for the Puma service, modifying it as directed by the comments.
    
 
 
    #!/bin/bash
 
 set -e
+exec 2>&1
+
 # Uncomment the line below if you're using RVM.
 #source /etc/profile.d/rvm.sh
 
-envdir="/etc/sv/puma/env"
-root=/etc/sv/puma
-echo "Starting puma from ${root}"
-cd $root
-mkdir -p "${envdir}"
-exec 2>&1
+envdir="`pwd`/env"
+mkdir -p "$envdir"
+echo ws-only > "$envdir/ARVADOS_WEBSOCKETS"
+
 cd /var/www/arvados-api/current
-# You may need to change arguments below to match your deployment, especially -u.
-exec chpst -e "${envdir}" -m 1073741824 -u www-data:www-data bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
+echo "Starting puma in `pwd`"
+
+# Change arguments below to match your deployment, "webserver-user" and
+# "webserver-group" should be changed to the user and group of the web server
+# process.  This is typically "www-data:www-data" on Debian systems by default,
+# other systems may use different defaults such the name of the web server
+# software (for example, "nginx:nginx").
+exec chpst -m 1073741824 -u webserver-user:webserver-group -e "$envdir" \
+  bundle exec puma -t 0:512 -e production -b tcp://127.0.0.1:8100
 
    
 
    7. 
 
@@ -200,6 +244,14 @@ exec chpst -e "${envdir}" -m 1073741824 -u www-data:www-data bundle exec puma -t
   passenger_enabled on;
   # If you're using RVM, uncomment the line below.
   #passenger_ruby /usr/local/rvm/wrappers/default/ruby;
+
+  # This value effectively limits the size of API objects users can
+  # create, especially collections.  If you change this, you should
+  # also ensure the following settings match it:
+  # * `client_max_body_size` in the server section below
+  # * `client_max_body_size` in the Workbench Nginx configuration (twice)
+  # * `max_request_size` in the API server's application.yml file
+  client_max_body_size 128m;
 }
 
 upstream api {
@@ -213,17 +265,34 @@ upstream websockets {
 
 proxy_http_version 1.1;
 
+# When Keep clients request a list of Keep services from the API server, the
+# server will automatically return the list of available proxies if
+# the request headers include X-External-Client: 1.  Following the example
+# here, at the end of this section, add a line for each netmask that has
+# direct access to Keep storage daemons to set this header value to 0.
+geo $external_client {
+  default        1;
+  10.20.30.0/24  0;
+}
+
 server {
   listen       [your public IP address]:443 ssl;
-  server_name  uuid-prefix.your.domain;
+  server_name  uuid_prefix.your.domain;
 
   ssl on;
+  ssl_certificate     /YOUR/PATH/TO/cert.pem;
+  ssl_certificate_key /YOUR/PATH/TO/cert.key;
 
   index  index.html index.htm index.php;
 
+  # Refer to the comment about this setting in the server section above.
+  client_max_body_size 128m;
+
   location / {
     proxy_pass            http://api;
     proxy_redirect        off;
+    proxy_connect_timeout 90s;
+    proxy_read_timeout    300s;
 
     proxy_set_header      X-Forwarded-Proto https;
     proxy_set_header      Host $http_host;
@@ -235,15 +304,19 @@ server {
 
 server {
   listen       [your public IP address]:443 ssl;
-  server_name  ws.uuid-prefix.your.domain;
+  server_name  ws.uuid_prefix.your.domain;
 
   ssl on;
+  ssl_certificate     /YOUR/PATH/TO/cert.pem;
+  ssl_certificate_key /YOUR/PATH/TO/cert.key;
 
   index  index.html index.htm index.php;
 
   location / {
     proxy_pass            http://websockets;
     proxy_redirect        off;
+    proxy_connect_timeout 90s;
+    proxy_read_timeout    300s;
 
     proxy_set_header      Upgrade $http_upgrade;
     proxy_set_header      Connection "upgrade";
@@ -255,7 +328,24 @@ server {
 

 
 
-
  • Restart Nginx.
    • 
+
  • Restart Nginx:
    
+
+
    ~$ sudo nginx -s reload
+
    
+
+
    • 
 
 
     
+
+h2. Prepare the API server deployment
+
+{% assign railspkg = "arvados-api-server" %}
+{% include 'install_rails_reconfigure' %}
+
+{% include 'notebox_begin' %}
+You can safely ignore the following messages if they appear while this command runs:
+
    Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will
+break this application for all non-root users on this machine.
    
+
    fatal: Not a git repository (or any of the parent directories): .git
    
+{% include 'notebox_end' %}