19215: Incorporate many details into install doc

[arvados.git] / doc / install / salt-multi-host.html.textile.liquid

---
layout: default
navsection: installguide
title: Multi host Arvados
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.

SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}

# "Introduction":#introduction
# "Prerequisites and planning":#prerequisites
# "Download the installer":#download
# "Initialize the installer":#copy_config
# "Choose the SSL configuration":#certificates
## "Using a self-signed certificates":#self-signed
## "Using a Let's Encrypt certificates":#lets-encrypt
## "Bring your own certificates":#bring-your-own
# "Create a compute image":#create_a_compute_image
# "Further customization of the installation (modifying the salt pillars and states)":#further_customization
# "Begin installation":#installation
# "Install the CA root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
# "After the installation":#post_install

h2(#introduction). Introduction

This multi host installer is the recommendend way to set up a production Arvados cluster.  These instructions include speciic details for installing on Amazon Web Services (AWS), which are marked as "AWS specific".  However with additional customization the installer can be used as a template for deployment on other cloud provider or HPC systems.

h2(#prerequisites). Prerequisites and planning

h3. Cluster ID and base domain

Choose a 5-character cluster identifier that will represent the cluster.  Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id .  Only lowercase letters and digits 0-9 are allowed.  Examples will use @xarv1@ or ${CLUSTER}, you should substitute the cluster id you have selected.

Determine the base domain for the cluster.  This will be referred to as ${DOMAIN}

For example, if CLUSTER is "xarv1" and DOMAIN is "example.com", then "controller.${CLUSTER}.${DOMAIN}" means "controller.xargv1.example.com".

h3. Virtual Private Cloud (AWS specific)

We recommend setting Arvados up in a "Virtual Private Cloud (VPC)":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html

When you do so, you need to configure a couple of additional things:

# "Create a subnet for the compute nodes":https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html
# You should set up a "security group which allows SSH access (port 22)":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
# Make sure to add a "VPC S3 endpoint":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html

h3. S3 Bucket (AWS specific)

We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@

Then create an IAM role called @${CLUSTER}-keepstore-00-iam-role@ which has "permission to read and write the bucket":https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html

h3. Machines

You will need to allocate (virtual) machines for the fixed infrastructure of the Arvados cluster.  These machines should have at least 2 cores and 8 GiB of RAM, running a "supported Arvados distribution":{{site.baseurl}}/install/install-manual-prerequisites.html#supportedlinux

Allocate these as appropriate for your site.  On AWS you may choose to do it manually with the AWS console, or using a DevOps tool such as CloudFormation or Terraform.

The installer will set up the Arvados services on your machines.  Here is the default assignment of services to machines:

# API node
## postgresql server
## arvados api server
## arvados controller  (recommendend hostname @controller.${CLUSTER}.${DOMAIN}@)
## arvados websocket   (recommendend hostname @ws.${CLUSTER}.${DOMAIN}@)
## arvados cloud dispatcher
## arvados keepbalance
# KEEPSTORE nodes (at least 2)
## arvados keepstore   (recommendend hostnames @keep0.${CLUSTER}.${DOMAIN}@ and @keep1.${CLUSTER}.${DOMAIN}@)
# KEEPPROXY node
## arvados keepproxy   (recommendend hostname @keep.${CLUSTER}.${DOMAIN}@)
## arvados keepweb     (recommendend hostname @download.${CLUSTER}.${DOMAIN}@ and @*.collections.${CLUSTER}.${DOMAIN}@)
# WORKBENCH node
## arvados workbench   (recommendend hostname @workbench.${CLUSTER}.${DOMAIN}@)
## arvados workbench2  (recommendend hostname @workbench2.${CLUSTER}.${DOMAIN}@)
## arvados webshell    (recommendend hostname @webshell.${CLUSTER}.${DOMAIN}@)
# SHELL node  (optional)
## arvados shell       (recommended hostname @shell.${CLUSTER}.${DOMAIN}@)

Additional prerequisites when preparing machines to run the installer:

# root or passwordless sudo access
# from the account where you are performing the install, passwordless @ssh@ to each machine (meaning, the client's public key added to @~/.ssh/authorized_keys@ on each node)
# @git@ installed on each machine
# port 443 reachable by clients
# DNS hostnames for each service
## @controller.${CLUSTER}.${DOMAIN}@
## @ws.${CLUSTER}.${DOMAIN}@
## @keep0.${CLUSTER}.${DOMAIN}@
## @keep1.${CLUSTER}.${DOMAIN}@
## @keep.${CLUSTER}.${DOMAIN}@
## @download.${CLUSTER}.${DOMAIN}@
## @*.collections.${CLUSTER}.${DOMAIN}@  -- important note, this should be a wildcard DNS, going to the keepweb service
## @workbench.${CLUSTER}.${DOMAIN}@
## @workbench2.${CLUSTER}.${DOMAIN}@
## @webshell.${CLUSTER}.${DOMAIN}@
## @shell.${CLUSTER}.${DOMAIN}@

(AWS specific) The machine that runs the arvados cloud dispatcher will need an "IAM role that allows it to create EC2 instances, see here for details .":https://doc.arvados.org/v2.4/install/crunch2-cloud/install-dispatch-cloud.html

If your infrastructure differs from the setup proposed above (ie, different hostnames, or using AWS RDS or an existing DB server), you can still use the installer, but additional customization will be necessary.

h2(#download). Download the installer

{% assign local_params_src = 'multiple_hosts' %}
{% assign config_examples_src = 'multi_host/aws'%}
{% include 'download_installer' %}

h2. Edit @local.params@

This can be found wherever you choose to initialize the install files (@~/setup-arvados-xarv1@ in these examples).

# Set @CLUSTER@ to the 5-character cluster identifier (e.g "xarv1")
# Set @DOMAIN@ to the base DNS domain of the environment, e.g. "example.com"
# Edit Internal IP settings. Since services share hosts, some hosts are the same.
# Edit @CLUSTER_INT_CIDR@, this should be the CIDR of the private network that Arvados is running on, e.g. the VPC
        AWS Specific: Go to the AWS console and into the VPC service, there is a column in
        this table view of the VPCs that gives the CIDR for the VPC (IPv4 CIDR).
# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.
# Set each @KEY@ / @TOKEN@ to a random string
        Here's an easy way to create five random tokens:
<pre><code>for i in 1 2 3 4 5; do
  tr -dc A-Za-z0-9 </dev/urandom | head -c 32 ; echo ''
done
</code></pre>
# Set @DATABASE_PASSWORD@ to a random string
   Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.
   For example, if the password is `Cq&WU<A']p?j`
   With backslash quoting the special characters it should appear like this in local.params:
<pre><code>DATABASE_PASSWORD="Cq\&WU\<A\'\]p\?j"</code></pre>

{% include 'ssl_config_multi' %}

h2(#create_a_compute_image). Configure Keep on S3 (AWS specific)

Once you have that image created, Open @local_config_dir/pillars/arvados.sls@ and edit as follows:

1. In the @arvados.cluster.Volumes@ section, set @Region@ to the appropriate AWS region (e.g. 'us-east-1')

h2(#create_a_compute_image). Create a compute image

{% include 'branchname' %}

On cloud installations, containers are dispatched in Docker daemons running in the <i>compute instances</i>, which need some special setup.  Follow "the instructions build a cloud compute node image":https://doc.arvados.org/install/crunch2-cloud/install-compute-node.html using the "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images  .

Once you have that image created, Open @local_config_dir/pillars/arvados.sls@ and edit as follows (AWS specific settings described here, configuration for Azure is similar):

# In the @arvados.cluster.Containers.CloudVMs@ section:
## Set @ImageID@ to the AMI output from Packer
## Set @Region@ to the appropriate AWS region
## Set @AdminUsername@ to the admin user account on the image
## Set the @SecurityGroupIDs@ list to the VPC security group which you set up to allow SSH connections to these nodes
## Set @SubnetID@ to the value of SubnetId of your VPC
# Update @arvados.cluster.Containers.DispatchPrivateKey@ and paste the contents of the @~/.ssh/id_dispatcher@ file you generated in an earlier step.
# Update @arvados.cluster.InstanceTypes@ as necessary. If t3 and m5/c5 node types are not available, replace them with t2 and m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.

h2(#further_customization). Further customization of the installation

If you are installing on AWS and following the naming conventions recommend in this guide, then likely no further configuration is necessary and you can begin installation.

If your infrastructure differs from the setup proposed above (ie, using AWS RDS or an existing DB server), you can still use the installer, but additional customization will be necessary.

This is done by editing the Saltstack pillars and states files found in @local_config_dir@.  In particular, @local_config_dir/pillars/arvados.sls@ has the template used to produce the Arvados configuration file that is distributed to all the nodes.

Any extra salt <i>state</i> file you add under @local_config_dir/states@ will be added to the salt run and applied to the hosts.

h2(#installation). Begin installation

At this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.

Run this in ~/arvados-setup-xarv1:

<pre>
./installer.sh deploy
</pre>

This will deploy all the nodes.  It will take a while and produce a lot of logging.  If it runs into an error, it will stop.

When everything has finished, you can run the diagnostics.

Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.

You are probably an "internal client" if you are running the diagnostics from one of the Arvados machines inside the VPC.

You are an "external client" if you running the diagnostics from your workstation outside of the VPC.

<pre>
./installer.sh diagnostics (-internal-client|-external-client)
</pre>

h3. Diagnosing issues

Most service logs go to @/var/log/syslog@

The logs for Rails API server and for Workbench can be found in

@/var/www/arvados-api/current/log/production.log@
and
@/var/www/arvados-workbench/current/log/production.log@

on the appropriate instances.

Workbench2 is a client-side Javascript application, if it having trouble loading, check the browser's developer console.

h3(#iterating). Iterating on config changes

You can iterate on the config and maintain the cluster by making changes to @local.params@ and @local_config_dir@ and running @installer.sh deploy@ again.

If you are debugging a configuration issue on a specific node, you can speed up the cycle a bit by deploying just one node:

@installer.sh deploy keep0.xarv1.example.com@

However, once you have a final configuration, you should run a full deploy to ensure that the configuration has been synchronized on all the nodes.

h3. Common problems and solutions

* (AWS Specific) If the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start.  You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@.  The solution is to build a new AMI with --aws-ena-support true

* The arvados-api-server package sets up the database as a post-install script.  If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.

This will manifest as an error like this:

<pre>
#<ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation \"api_clients\" does not exist
</pre>

If this happens, you need to

1. correct the database information
2. run "installer.sh deploy xngs2.rdcloud.bms.com" to update the configuration on the API/controller node
3. On the API/controller server node, run this command to re-run the post-install script, which will set up the database:

<pre>
dpkg-reconfigure arvados-api-server
</pre>

4. Re-run 'installer.sh deploy' again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.

{% include 'install_ca_cert' %}

h2(#initial_user). Initial user and login

At this point you should be able to log into the Arvados cluster. The initial URL will be

* https://workbench.${CLUSTER}.${DOMAIN}

By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.

Assuming you didn't change these values in the @local.params@ file, the initial credentials are:

* User: 'admin'
* Password: 'password'
* Email: 'admin@${CLUSTER}.${DOMAIN}'

h2(#test_install). Test the installed cluster running a simple workflow

As part of the installation, the @provision.sh@ script saves a simple example test workflow in the @/tmp/cluster_tests@ directory in the @shell@ node. If you want to run it, just ssh to the node, then run:

<notextile>
<pre><code>cd /tmp/cluster_tests
sudo /run-test.sh
</code></pre>
</notextile>

It will create a test user (by default, the same one as the admin user), upload a small workflow and run it. If everything goes OK, the output should similar to this (some output was shortened for clarity):

<notextile>
<pre><code>Creating Arvados Standard Docker Images project
Arvados project uuid is 'arva2-j7d0g-0prd8cjlk6kfl7y'
{
 ...
 "uuid":"arva2-o0j2j-n4zu4cak5iifq2a",
 "owner_uuid":"arva2-tpzed-000000000000000",
 ...
}
Uploading arvados/jobs' docker image to the project
2.1.1: Pulling from arvados/jobs
8559a31e96f4: Pulling fs layer
...
Status: Downloaded newer image for arvados/jobs:2.1.1
docker.io/arvados/jobs:2.1.1
2020-11-23 21:43:39 arvados.arv_put[32678] INFO: Creating new cache file at /home/vagrant/.cache/arvados/arv-put/c59256eda1829281424c80f588c7cc4d
2020-11-23 21:43:46 arvados.arv_put[32678] INFO: Collection saved as 'Docker image arvados jobs:2.1.1 sha256:0dd50'
arva2-4zz18-1u5pvbld7cvxuy2
Creating initial user ('admin')
Setting up user ('admin')
{
 "items":[
  {
   ...
   "owner_uuid":"arva2-tpzed-000000000000000",
   ...
   "uuid":"arva2-o0j2j-1ownrdne0ok9iox"
  },
  {
   ...
   "owner_uuid":"arva2-tpzed-000000000000000",
   ...
   "uuid":"arva2-o0j2j-1zbeyhcwxc1tvb7"
  },
  {
   ...
   "email":"admin@arva2.arv.local",
   ...
   "owner_uuid":"arva2-tpzed-000000000000000",
   ...
   "username":"admin",
   "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
   ...
  }
 ],
 "kind":"arvados#HashList"
}
Activating user 'admin'
{
 ...
 "email":"admin@arva2.arv.local",
 ...
 "username":"admin",
 "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
 ...
}
Running test CWL workflow
INFO /usr/bin/cwl-runner 2.1.1, arvados-python-client 2.1.1, cwltool 3.0.20200807132242
INFO Resolved 'hasher-workflow.cwl' to 'file:///tmp/cluster_tests/hasher-workflow.cwl'
...
INFO Using cluster arva2 (https://arva2.arv.local:8443/)
INFO Upload local files: "test.txt"
INFO Uploaded to ea34d971b71d5536b4f6b7d6c69dc7f6+50 (arva2-4zz18-c8uvwqdry4r8jao)
INFO Using collection cache size 256 MiB
INFO [container hasher-workflow.cwl] submitted container_request arva2-xvhdp-v1bkywd58gyocwm
INFO [container hasher-workflow.cwl] arva2-xvhdp-v1bkywd58gyocwm is Final
INFO Overall process status is success
INFO Final output collection d6c69a88147dde9d52a418d50ef788df+123
{
    "hasher_out": {
        "basename": "hasher3.md5sum.txt",
        "class": "File",
        "location": "keep:d6c69a88147dde9d52a418d50ef788df+123/hasher3.md5sum.txt",
        "size": 95
    }
}
INFO Final process status is success
</code></pre>
</notextile>

h2(#post_install). After the installation

As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates.  You should retain this repository but be aware that it contains sensitive information (passwords and tokens used by the Arvados services).

As described in "Iterating on config changes":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades.  However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.

When deploying upgrades, consult the "Arvados upgrade notes":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@.

See "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.