---
layout: default
navsection: installguide
title: Multi host Arvados
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
# "Introduction":#introduction
# "Prerequisites and planning":#prerequisites
# "Required hosts":#hosts
# "Download the installer":#download
# "Initialize the installer":#copy_config
# "Edit local.params":#localparams
# "Configure Keep storage":#keep
# "Choose the SSL configuration":#certificates
## "Using a self-signed certificates":#self-signed
## "Using a Let's Encrypt certificates":#lets-encrypt
## "Bring your own certificates":#bring-your-own
# "Create a compute image":#create_a_compute_image
# "Further customization of the installation":#further_customization
# "Begin installation":#installation
# "Confirm the cluster is working":#test-install
## "Debugging issues":#debugging
## "Iterating on config changes":#iterating
## "Common problems and solutions":#common-problems
# "Install the CA root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "After the installation":#post_install
h2(#introduction). Introduction
This multi host installer is the recommendend way to set up a production Arvados cluster. These instructions include specific details for installing on Amazon Web Services (AWS), which are marked as "AWS specific". However with additional customization the installer can be used as a template for deployment on other cloud provider or HPC systems.
h2(#prerequisites). Prerequisites and planning
h3. Cluster ID and base domain
Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or ${CLUSTER}, you should substitute the cluster id you have selected.
Determine the base domain for the cluster. This will be referred to as ${DOMAIN}
For example, if CLUSTER is "xarv1" and DOMAIN is "example.com", then "controller.${CLUSTER}.${DOMAIN}" means "controller.xargv1.example.com".
h3. Virtual Private Cloud (AWS specific)
We recommend setting Arvados up in a "Virtual Private Cloud (VPC)":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html
When you do so, you need to configure a couple of additional things:
# "Create a subnet for the compute nodes":https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html
# You should set up a "security group which allows SSH access (port 22)":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
# Make sure to add a "VPC S3 endpoint":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
h3(#keep-bucket). S3 Bucket (AWS specific)
We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@
Then create an IAM role called @${CLUSTER}-keepstore-00-iam-role@ which has "permission to read and write the bucket":https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html . Here is an example policy:
{
"Id": "arvados-keepstore policy",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": "arn:aws:s3:::xarv1-nyw5e-000000000000000-volume"
}
]
}
for i in 1 2 3 4 5; do
tr -dc A-Za-z0-9
# Set @DATABASE_PASSWORD@ to a random string
Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.
For example, if the password is `Cq&WUDATABASE_PASSWORD="Cq\&WU\
h2(#keep). Configure Keep storage
The @multi_host/aws@ template uses S3 for storage. Arvados also supports "filesystem storage":configure-fs-storage.html and "Azure blob storage":configure-azure-blob-storage.html . Keep storage configuration can be found in in the section @arvados.cluster.Volumes@ of @local_config_dir/pillars/arvados.sls@.
h3. Object storage in S3 (AWS Specific)
Open @local_config_dir/pillars/arvados.sls@ and edit as follows:
# In the @arvados.cluster.Volumes@ section, set @Region@ to the appropriate AWS region (e.g. 'us-east-1')
# Set @Bucket@ to the value of "keepstore role you created earlier":#keep-bucket
# Set @IAMRole@ to "keepstore role you created earlier":#keep-bucket
{% include 'ssl_config_multi' %}
h2(#create_a_compute_image). Create a compute image
{% include 'branchname' %}
On cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup.
*Start by following "the instructions build a cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images* .
Once you have that image created, Open @local_config_dir/pillars/arvados.sls@ and edit as follows (AWS specific settings described here, configuration for Azure is similar):
# In the @arvados.cluster.Containers.CloudVMs@ section:
## Set @ImageID@ to the AMI produced by Packer
## Set @Region@ to the appropriate AWS region
## Set @AdminUsername@ to the admin user account on the image
## Set the @SecurityGroupIDs@ list to the VPC security group which you set up to allow SSH connections to these nodes
## Set @SubnetID@ to the value of SubnetId of your VPC
# Update @arvados.cluster.Containers.DispatchPrivateKey@ and paste the contents of the @~/.ssh/id_dispatcher@ file you generated in an earlier step.
# Update @arvados.cluster.InstanceTypes@ as necessary. If m5/c5 node types are not available, replace them with m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.
h2(#further_customization). Further customization of the installation (optional)
If you are installing on AWS and following the naming conventions recommend in this guide, then likely no further configuration is necessary and you can begin installation.
A couple of common customizations are described here. Other changes may require editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ has the template used to produce the Arvados configuration file that is distributed to all the nodes.
Any extra salt _state_ files you add under @local_config_dir/states@ will be added to the salt run and applied to the hosts.
h3(#authentication). Using a different authentication provider
By default, the installer will use the "Test" provider, which is a list of usernames and cleartext passwords stored in the Arvados config file. *This is low security configuration and you are strongly advised to configure one of the other "supported authentication methods":setup-login.html* .
h3(#ext-database). Using an external database (optional)
Arvados requires a database that is compatible with PostgreSQL 9.5 or later.
For example, Arvados is known to work with Amazon Aurora (note: even idle, Arvados constantly accesses the database, so we strongly advise using "provisioned" mode).
# In @local.params@, remove 'database' from the list of roles assigned to the controller node:
NODES=(
[controller.${CLUSTER}.${DOMAIN}]=api,controller,websocket,dispatcher,keepbalance
...
)
# In @local.params@, set @DATABASE_INT_IP@ to the database endpoint (can be a hostname, does not have to be an IP address).
DATABASE_INT_IP=...
# In @local.params@, set @DATABASE_PASSWORD@ to the correct value. "See the previous section describing correct quoting":#localparams
# In @local_config_dir/pillars/arvados.sls@ you may need to adjust the database name and user. This can be found in the section @arvados.cluster.database@.
h2(#installation). Begin installation
At this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.
Run this in @~/arvados-setup-xarv1@:
./installer.sh deploy
This will deploy all the nodes. It will take a while and produce a lot of logging. If it runs into an error, it will stop.
{% include 'install_ca_cert' %}
h2(#test-install). Confirm the cluster is working
When everything has finished, you can run the diagnostics.
Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.
If you are running the diagnostics from one of the Arvados machines inside the VPC, you want @-internal-client@ .
You are an "external client" if you running the diagnostics from your workstation outside of the VPC.
./installer.sh diagnostics (-internal-client|-external-client)
h3(#debugging). Debugging issues
Most service logs go to @/var/log/syslog@.
The logs for Rails API server and for Workbench can be found in
@/var/www/arvados-api/current/log/production.log@
and
@/var/www/arvados-workbench/current/log/production.log@
on the appropriate instances.
Workbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in "Tools → Developer Tools").
h3(#iterating). Iterating on config changes
You can iterate on the config and maintain the cluster by making changes to @local.params@ and @local_config_dir@ and running @installer.sh deploy@ again.
If you are debugging a configuration issue on a specific node, you can speed up the cycle a bit by deploying just one node:
./installer.sh deploy keep0.xarv1.example.com@
However, once you have a final configuration, you should run a full deploy to ensure that the configuration has been synchronized on all the nodes.
h3(#common-problems). Common problems and solutions
h4. PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
The arvados-api-server package sets up the database as a post-install script. If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.
This will manifest as an error like this:
#
If this happens, you need to
1. correct the database information
2. run @./installer.sh deploy xarv1.example.com@ to update the configuration on the API/controller node
3. On the API/controller server node, run this command to re-run the post-install script, which will set up the database:
dpkg-reconfigure arvados-api-server
4. Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.
h4. Missing ENA support (AWS Specific)
If the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start. You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@. The solution is to build a new AMI with --aws-ena-support true
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster. The initial URL will be
https://workbench.${CLUSTER}.${DOMAIN}
If you did not "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
If you did configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
h2(#post_install). After the installation
As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates. You should retain this repository but be aware that it contains sensitive information (passwords and tokens used by the Arvados services).
As described in "Iterating on config changes":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades. However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.
When deploying upgrades, consult the "Arvados upgrade notes":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@.
See also "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.