---
layout: default
navsection: installguide
-title: Multi host Arvados
+title: Multi-Host Arvados
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-# "Install Saltstack":#saltstack
-# "Install dependencies":#dependencies
-# "Install Arvados using Saltstack":#saltstack
-# "DNS configuration":#final_steps
+# "Introduction":#introduction
+# "Prerequisites and planning":#prerequisites
+# "Download the installer":#download
+# "Initialize the installer":#copy_config
+# "Set up your infrastructure":#setup-infra
+## "Create AWS infrastructure with Terraform":#terraform
+## "Create required infrastructure manually":#inframanual
+# "Edit local.params* files":#localparams
+# "Configure Keep storage":#keep
+# "Choose the SSL configuration":#certificates
+## "Using a Let's Encrypt certificates":#lets-encrypt
+## "Bring your own certificates":#bring-your-own
+### "Securing your TLS certificate keys":#secure-tls-keys
+# "Create a compute image":#create_a_compute_image
+# "Begin installation":#installation
+# "Further customization of the installation":#further_customization
+# "Confirm the cluster is working":#test-install
+## "Debugging issues":#debugging
+## "Iterating on config changes":#iterating
+## "Common problems and solutions":#common-problems
# "Initial user and login":#initial_user
+# "Monitoring and Metrics":#monitoring
+# "Load balancing controllers":#load_balancing
+# "After the installation":#post_install
-h2(#saltstack). Install Saltstack
+h2(#introduction). Introduction
-If you already have a Saltstack environment you can skip this section.
+This multi host installer is the recommendend way to set up a production Arvados cluster. These instructions include specific details for installing on Amazon Web Services (AWS), which are marked as "AWS specific". However with additional customization the installer can be used as a template for deployment on other cloud provider or HPC systems.
-The simplest way to get Salt up and running on a node is to use the bootstrap script they provide:
+h2(#prerequisites). Prerequisites and planning
-<notextile>
-<pre><code>curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
-sudo sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+h3. Cluster ID and base domain
+
+Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or @${CLUSTER}@, you should substitute the cluster id you have selected.
+
+Determine the base domain for the cluster. This will be referred to as @${DOMAIN}@.
+
+For example, if DOMAIN is @xarv1.example.com@, then @controller.${DOMAIN}@ means @controller.xarv1.example.com@.
+
+h3(#DNS). DNS hostnames for each service
+
+You will need a DNS entry for each service. When using the "Terraform script":#terraform to set up your infrastructure, these domains will be created automatically using AWS Route 53.
+
+In the default configuration these are:
+
+# @controller.${DOMAIN}@
+# @ws.${DOMAIN}@
+# @keep0.${DOMAIN}@
+# @keep1.${DOMAIN}@
+# @keep.${DOMAIN}@
+# @download.${DOMAIN}@
+# @*.collections.${DOMAIN}@ -- important note, this must be a wildcard DNS, resolving to the @keepweb@ service
+# @workbench.${DOMAIN}@
+# @workbench2.${DOMAIN}@
+# @webshell.${DOMAIN}@
+# @shell.${DOMAIN}@
+# @prometheus.${DOMAIN}@
+# @grafana.${DOMAIN}@
+
+For more information, see "DNS entries and TLS certificates":install-manual-prerequisites.html#dnstls.
+
+h2(#download). Download the installer
+
+{% assign local_params_src = 'multiple_hosts' %}
+{% assign config_examples_src = 'multi_host/aws' %}
+{% assign terraform_src = 'terraform/aws' %}
+{% include 'download_installer' %}
+
+h2(#setup-infra). Set up your infrastructure
+
+## "Create AWS infrastructure with Terraform":#terraform
+## "Create required infrastructure manually":#inframanual
+
+h3(#terraform). Create AWS infrastructure with Terraform (AWS specific)
+
+We provide a set of Terraform code files that you can run to create the necessary infrastructure on Amazon Web Services.
+
+These files are located in the @terraform@ installer directory and are divided in three sections:
+
+# The @terraform/vpc/@ subdirectory controls the network related infrastructure of your cluster, including firewall rules and split-horizon DNS resolution.
+# The @terraform/data-storage/@ subdirectory controls the stateful part of your cluster, currently only sets up the S3 bucket for holding the Keep blocks and in the future it'll also manage the database service.
+# The @terraform/services/@ subdirectory controls the hosts that will run the different services on your cluster, makes sure that they have the required software for the installer to do its job.
+
+h4. Software requirements & considerations
+
+{% include 'notebox_begin' %}
+The Terraform state files (that keep crucial infrastructure information from the cloud) will be saved inside each subdirectory, under the @terraform.tfstate@ name. These will be committed to the git repository used to coordinate deployment. It is very important to keep this git repository secure, only sysadmins that will be responsible for maintaining your Arvados cluster should have access to it.
+{% include 'notebox_end' %}
+
+h4. Terraform code configuration
+
+Each section described above contain a @terraform.tfvars@ file with some configuration values that you should set before applying each configuration. You should at least set the AWS region, cluster prefix and domain name in @terraform/vpc/terraform.tfvars@:
+
+<pre><code>{% include 'terraform_vpc_tfvars' %}</code></pre>
+
+If you don't set the main configuration variables at @vpc/terraform.tfvars@ file, you will be asked to re-enter these parameters every time you run Terraform.
+
+The @data-storage/terraform.tfvars@ and @services/terraform.tfvars@ let you configure additional details, including the SSH public key for deployment, instance & volume sizes, etc. All these configurations are provided with sensible defaults:
+
+<pre><code>{% include 'terraform_datastorage_tfvars' %}</code></pre>
+
+<pre><code>{% include 'terraform_services_tfvars' %}</code></pre>
+
+h4. Set credentials
+
+You will need an AWS access key and secret key to create the infrastructure.
+
+<pre><code class="userinput">export AWS_ACCESS_KEY_ID="anaccesskey"
+export AWS_SECRET_ACCESS_KEY="asecretkey"</code></pre>
+
+h4. Create the infrastructure
+
+Build the infrastructure by running @./installer.sh terraform@. The last stage will output the information needed to set up the cluster's domain and continue with the installer. for example:
+
+<pre><code class="userinput">./installer.sh terraform
+...
+Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
+
+Outputs:
+
+arvados_sg_id = "sg-02f999a99973999d7"
+arvados_subnet_id = "subnet-01234567abc"
+cluster_int_cidr = "10.1.0.0/16"
+cluster_name = "xarv1"
+compute_subnet_id = "subnet-abcdef12345"
+deploy_user = "admin"
+domain_name = "xarv1.example.com"
+letsencrypt_iam_access_key_id = "AKAA43MAAAWAKAADAASD"
+private_ip = {
+ "controller" = "10.1.1.1"
+ "keep0" = "10.1.1.3"
+ "keep1" = "10.1.1.4"
+ "keepproxy" = "10.1.1.2"
+ "shell" = "10.1.1.7"
+ "workbench" = "10.1.1.5"
+}
+public_ip = {
+ "controller" = "18.235.116.23"
+ "keep0" = "34.202.85.86"
+ "keep1" = "38.22.123.98"
+ "keepproxy" = "34.231.9.201"
+ "shell" = "44.208.155.240"
+ "workbench" = "52.204.134.136"
+}
+region_name = "us-east-1"
+route53_dns_ns = tolist([
+ "ns-1119.awsdns-11.org",
+ "ns-1812.awsdns-34.co.uk",
+ "ns-437.awsdns-54.com",
+ "ns-809.awsdns-37.net",
+])
+ssl_password_secret_name = "xarv1-arvados-ssl-privkey-password"
+vpc_id = "vpc-0999994998399923a"
+letsencrypt_iam_secret_access_key = "XXXXXSECRETACCESSKEYXXXX"
</code></pre>
-</notextile>
-For more information check "Saltstack's documentation":https://docs.saltstack.com/en/latest/topics/installation/index.html
-h2(#dependencies). Install dependencies
+h4. Additional DNS configuration
+
+Once Terraform has completed, the infrastructure for your Arvados cluster is up and running. One last piece of DNS configuration is required.
+
+The domain names for your cluster (e.g.: controller.xarv1.example.com) are managed via "Route 53":https://aws.amazon.com/route53/ and the TLS certificates will be issued using "Let's Encrypt":https://letsencrypt.org/ .
+
+You need to configure the parent domain to delegate to the newly created zone. For example, you need to configure "example.com" to delegate the subdomain "xarv1.example.com" to the nameservers for the Arvados hostname records created by Terraform. You do this by creating a @NS@ record on the parent domain that refers to the name servers listed in the Terraform output parameter @route53_dns_ns@.
+
+If your parent domain is also controlled by Route 53, the process will be like this:
+
+# Log in to the AWS Console and navigate to the service page for *Route 53*
+# Go to the list of *Hosted zones* and click on the zone for the parent domain
+# Click on *Create record*
+# For *Record name* put the cluster id
+# For *Record type* choose @NS - Name servers for a hosted zone@
+# For *Value* add the values from Terraform output parameter @route53_dns_ns@, one hostname per line, with punctuation (quotes and commas) removed.
+# Click *Create records*
+
+If the parent domain is controlled by some other service, follow the guide for the the appropriate service.
+
+h4. Other important output parameters
+
+The certificates will be requested from Let's Encrypt when you run the installer.
+
+* @cluster_int_cidr@ will be used to set @CLUSTER_INT_CIDR@
+
+* You'll also need @compute_subnet_id@ and @arvados_sg_id@ to set @COMPUTE_SUBNET@ and @COMPUTE_SG@ in @local.params@ and when you "create a compute image":#create_a_compute_image.
+
+You can now proceed to "edit local.params* files":#localparams.
+
+h3(#inframanual). Create required infrastructure manually
+
+If you will be setting up infrastructure without using the provided Terraform script, here are the recommendations you will need to consider.
+
+h4. Virtual Private Cloud (AWS specific)
+
+We recommend setting Arvados up in its own "Virtual Private Cloud (VPC)":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html
+
+When you do so, you need to configure a couple of additional things:
+
+# "Create a subnet for the compute nodes":https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html
+# You should set up a "security group which allows SSH access (port 22)":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
+# Make sure to add a "VPC S3 endpoint":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
+
+h4(#keep-bucket). S3 Bucket (AWS specific)
+
+We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@. We recommend creating an IAM role called @${CLUSTER}-keepstore-00-iam-role@ with a "policy that can read, write, list and delete objects in the bucket":configure-s3-object-storage.html#IAM . With the example cluster id @xarv1@ the bucket would be called @xarv1-nyw5e-000000000000000-volume@ and the role would be called @xarv1-keepstore-00-iam-role@.
+
+These names are recommended because they are default names used in the configuration template. If you use different names, you will need to edit the configuration template later.
+
+h4(#hosts). Required hosts
+
+You will need to allocate several hosts (physical or virtual machines) for the fixed infrastructure of the Arvados cluster. These machines should have at least 2 cores and 8 GiB of RAM, running a supported Linux distribution.
+
+{% include 'supportedlinux' %}
+
+Allocate the following hosts as appropriate for your site. On AWS you may choose to do it manually with the AWS console, or using a DevOps tool such as CloudFormation or Terraform. With the exception of "keep0" and "keep1", all of these hosts should have external (public) IP addresses if you intend for them to be accessible outside of the private network or VPC.
+
+The installer will set up the Arvados services on your machines. Here is the default assignment of services to machines:
+
+# API node
+## postgresql server
+## arvados api server
+## arvados controller (recommendend hostname @controller.${DOMAIN}@)
+# KEEPSTORE nodes (at least 1 if using S3 as a Keep backend, else 2)
+## arvados keepstore (recommendend hostnames @keep0.${DOMAIN}@ and @keep1.${DOMAIN}@)
+# WORKBENCH node
+## arvados legacy workbench URLs (recommendend hostname @workbench.${DOMAIN}@)
+## arvados workbench2 (recommendend hostname @workbench2.${DOMAIN}@)
+## arvados webshell (recommendend hostname @webshell.${DOMAIN}@)
+## arvados websocket (recommendend hostname @ws.${DOMAIN}@)
+## arvados cloud dispatcher
+## arvados keepbalance
+## arvados keepproxy (recommendend hostname @keep.${DOMAIN}@)
+## arvados keepweb (recommendend hostname @download.${DOMAIN}@ and @*.collections.${DOMAIN}@)
+# SHELL node (optional)
+## arvados shell (recommended hostname @shell.${DOMAIN}@)
+
+When using the database installed by Arvados (and not an "external database":#ext-database), the database is stored under @/var/lib/postgresql@. Arvados logs are also kept in @/var/log@ and @/var/www/arvados-api/shared/log@. Accordingly, you should ensure that the disk partition containing @/var@ has adequate storage for your planned usage. We suggest starting with 50GiB of free space on the database host.
+
+h4. Additional prerequisites when preparing machines to run the installer
+
+# From the account where you are performing the install, passwordless @ssh@ to each machine
+This means the client's public key should added to @~/.ssh/authorized_keys@ on each node.
+# Passwordless @sudo@ access on the account on each machine you will @ssh@ in to
+This usually means adding the account to the @sudo@ group and having a rule like this in @/etc/sudoers.d/arvados_passwordless@ that allows members of group @sudo@ to execute any command without entering a password.
+<pre>%sudo ALL=(ALL:ALL) NOPASSWD:ALL</pre>
+# @git@ installed on each machine
+# Port 443 reachable by clients
+
+(AWS specific) The machine that runs the arvados cloud dispatcher will need an "IAM role that allows it to manage EC2 instances.":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html#IAM
+
+If your infrastructure differs from the setup proposed above (ie, different hostnames), you can still use the installer, but "additional customization may be necessary":#further_customization .
+
+h2(#localparams). Edit @local.params*@ files
+
+The cluster configuration parameters are included in two files: @local.params@ and @local.params.secrets@. These files can be found wherever you choose to initialize the installation files (e.g., @~/setup-arvados-xarv1@ in these examples).
+
+The @local.params.secrets@ file is intended to store security-sensitive data such as passwords, private keys, tokens, etc. Depending on the security requirements of the cluster deployment, you may wish to store this file in a secrets store like AWS Secrets Manager or Jenkins credentials.
+
+h3. Parameters from @local.params@:
+
+# Set @CLUSTER@ to the 5-character cluster identifier. (e.g. "xarv1")
+# Set @DOMAIN@ to the base DNS domain of the environment. (e.g. "xarv1.example.com")
+# Set the @*_INT_IP@ variables with the internal (private) IP addresses of each host. Since services share hosts, some hosts are the same. See "note about /etc/hosts":#etchosts
+# Edit @CLUSTER_INT_CIDR@, this should be the CIDR of the private network that Arvados is running on, e.g. the VPC. If you used terraform, this is emitted as @cluster_int_cidr@.
+_CIDR stands for "Classless Inter-Domain Routing" and describes which portion of the IP address that refers to the network. For example 192.168.3.0/24 means that the first 24 bits are the network (192.168.3) and the last 8 bits are a specific host on that network._
+_AWS Specific: Go to the AWS console and into the VPC service, there is a column in this table view of the VPCs that gives the CIDR for the VPC (IPv4 CIDR)._
+# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.
-Arvados depends in a few applications and packages (postgresql, nginx+passenger, ruby) that can also be installed using their respective Saltstack formulas.
+h3. Parameters from @local.params.secrets@:
-The formulas we use are:
+# Set each @KEY@ / @TOKEN@ / @PASSWORD@ to a random string. You can use @installer.sh generate-tokens@
+<pre><code class="userinput">./installer.sh generate-tokens
+BLOB_SIGNING_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+MANAGEMENT_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+SYSTEM_ROOT_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+ANONYMOUS_USER_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+DATABASE_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+</code></pre>
+# Set @DATABASE_PASSWORD@ to a random string (unless you "already have a database":#ext-database then you should set it to that database's password)
+ Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.
+ For example, if the password is @Lq&MZ<V']d?j@
+ With backslash quoting the special characters it should appear like this in local.params:
+<pre><code>DATABASE_PASSWORD="Lq\&MZ\<V\'\]d\?j"</code></pre>
+# Set @DISPATCHER_SSH_PRIVKEY@ to a SSH private key that @arvados-dispatch-cloud@ will use to connect to the compute nodes:
+<pre><code>DISPATCHER_SSH_PRIVKEY="-----BEGIN OPENSSH PRIVATE KEY-----
+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
+...
+s4VY40kNxs6MsAAAAPbHVjYXNAaW5zdGFsbGVyAQIDBA==
+-----END OPENSSH PRIVATE KEY-----"
+</code></pre>You can create one by following the steps described on the "building a compute node documentation":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html#sshkeypair page.
+
+h3(#etchosts). Note on @/etc/hosts@
-* "postgres":https://github.com/saltstack-formulas/postgres-formula.git
-* "nginx":https://github.com/saltstack-formulas/nginx-formula.git
-* "docker":https://github.com/saltstack-formulas/docker-formula.git
-* "locale":https://github.com/saltstack-formulas/locale-formula.git
+Because Arvados services are typically accessed by external clients, they are likely to have both a public IP address and a internal IP address.
-There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/saltstack-formulas/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+On cloud providers such as AWS, sending internal traffic to a service's public IP address can incur egress costs and throttling. Thus it is very important for internal traffic to stay on the internal network. The installer implements this by updating @/etc/hosts@ on each node to associate each service's hostname with the internal IP address, so that when Arvados services communicate with one another, they always use the internal network address. This is NOT a substitute for DNS, you still need to set up DNS names for all of the services that have public IP addresses (it does, however, avoid a complex "split-horizon" DNS configuration).
-h2(#saltstack). Install Arvados using Saltstack
+It is important to be aware of this because if you mistype the IP address for any of the @*_INT_IP@ variables, hosts may unexpectedly fail to be able to communicate with one another. If this happens, check and edit as necessary the file @/etc/hosts@ on the host that is failing to make an outgoing connection.
-This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+h2(#keep). Configure Keep storage
-The Arvados formula we maintain is located in the Saltstack's community repository of formulas:
+The @multi_host/aws@ template uses S3 for storage. Arvados also supports "filesystem storage":configure-fs-storage.html and "Azure blob storage":configure-azure-blob-storage.html . Keep storage configuration can be found in in the @arvados.cluster.Volumes@ section of @local_config_dir/pillars/arvados.sls@.
-* "arvados-formula":https://github.com/saltstack-formulas/arvados-formula.git
+h3. Object storage in S3 (AWS Specific)
-The @development@ version lives in our own repository
+If you "followed the recommendend naming scheme":#keep-bucket for both the bucket and role (or used the provided Terraform script), you're done.
-* "arvados-formula development":https://github.com/arvados/arvados-formula.git
+If you did not follow the recommendend naming scheme for either the bucket or role, you'll need to update these parameters in @local.params@:
-This last one might break from time to time, as we try and add new features. Use with caution.
+# Set @KEEP_AWS_S3_BUCKET@ to the value of "keepstore bucket you created earlier":#keep-bucket
+# Set @KEEP_AWS_IAM_ROLE@ to "keepstore role you created earlier":#keep-bucket
-As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados.
+You can also configure a specific AWS Region for the S3 bucket by setting @KEEP_AWS_REGION@.
-For those familiar with Saltstack, the process to get it deployed is similar to any other formula:
+{% include 'ssl_config_multi' %}
-1. Fork/copy the formula to your Salt master host.
-2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
-3. Run a @state.apply@ to get it deployed.
+h2(#authentication). Configure your authentication provider (optional, recommended)
-h2(#final_steps). DNS configuration
+By default, the installer will use the "Test" provider, which is a list of usernames and cleartext passwords stored in the Arvados config file. *This is low security configuration and you are strongly advised to configure one of the other "supported authentication methods":setup-login.html* .
-After the setup is done, you need to set up your DNS to be able to access the cluster's nodes.
+h2(#ext-database). Using an external database (optional)
-The simplest way to do this is to add entries in the @/etc/hosts@ file of every host:
+The standard behavior of the installer is to install and configure PostgreSQL for use by Arvados. You can optionally configure it to use a separately managed database instead.
-<notextile>
-<pre><code>export CLUSTER="arva2"
-export DOMAIN="arv.local"
+Arvados requires a database that is compatible with PostgreSQL 9.5 or later. For example, Arvados is known to work with Amazon Aurora (note: even idle, Arvados services will periodically poll the database, so we strongly advise using "provisioned" mode).
-echo A.B.C.a api ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.b keep keep.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.c keep0 keep0.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.d collections collections.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.e download download.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.f ws ws.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.g workbench workbench.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.h workbench2 workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+# In @local.params@, remove 'database' from the list of roles assigned to the controller node:
+<pre><code>NODES=(
+ [controller.${DOMAIN}]=controller,websocket,dispatcher,keepbalance
+ ...
+)
+</code></pre>
+# In @local.params@, set @DATABASE_INT_IP@ to empty string and @DATABASE_EXTERNAL_SERVICE_HOST_OR_IP@ to the database endpoint (can be a hostname, does not have to be an IP address).
+<pre><code>DATABASE_INT_IP=""
+...
+DATABASE_EXTERNAL_SERVICE_HOST_OR_IP="arvados.xxxxxxx.eu-east-1.rds.amazonaws.com"
</code></pre>
-</notextile>
+# In @local.params.secrets@, set @DATABASE_PASSWORD@ to the correct value. "See the previous section describing correct quoting":#localparams
+# In @local.params@ you may need to adjust the database name and user.
+
+h2(#further_customization). Further customization of the installation (optional)
+
+If you are installing on AWS and have followed all of the naming conventions recommend in this guide, you probably don't need to do any further customization.
+
+If you are installing on a different cloud provider or on HPC, other changes may require editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ contains the template (in the @arvados.cluster@ section) used to produce the Arvados configuration file that is distributed to all the nodes. Consult the "Configuration reference":config.html for a comprehensive list of configuration keys.
+
+Any extra Salt "state" files you add under @local_config_dir/states@ will be added to the Salt run and applied to the hosts.
+
+h2(#create_a_compute_image). Configure compute nodes
+
+{% include 'branchname' %}
+
+If you will use fixed compute nodes with an HPC scheduler such as SLURM or LSF, you will need to "Set up your compute nodes with Docker":{{site.baseurl}}/install/crunch2/install-compute-node-docker.html or "Set up your compute nodes with Singularity":{{site.baseurl}}/install/crunch2/install-compute-node-singularity.html.
+
+On cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup.
+
+h3. Build the compute image
+
+Follow "the instructions to build a cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the compute image builder script found in @arvados/tools/compute-images@ in your Arvados clone from "step 3":#download.
+
+h3. Configure the compute image
+
+Once the image has been created, open @local.params@ and edit as follows (AWS specific settings described here, you will need to make custom changes for other cloud providers):
+
+# Set @COMPUTE_AMI@ to the AMI produced by Packer
+# Set @COMPUTE_AWS_REGION@ to the appropriate AWS region
+# Set @COMPUTE_USER@ to the admin user account on the image
+# Set the @COMPUTE_SG@ list to the VPC security group which you set up to allow SSH connections to these nodes
+# Set @COMPUTE_SUBNET@ to the value of SubnetId of your VPC
+# Update @arvados.cluster.InstanceTypes@ in @local_config_dir/pillars/arvados.sls@ as necessary. The example instance types are for AWS, other cloud providers will of course have different instance types with different names and specifications.
+(AWS specific) If m5/c5 node types are not available, replace them with m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.
+
+h2(#installation). Begin installation
+
+At this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.
+
+Run this in the @~/arvados-setup-xarv1@ directory:
+
+<pre><code class="userinput">./installer.sh deploy</code></pre>
+
+This will install and configure Arvados on all the nodes. It will take a while and produce a lot of logging. If it runs into an error, it will stop.
+
+h2(#test-install). Confirm the cluster is working
+
+When everything has finished, you can run the diagnostics. There's a couple ways of doing this listed below.
+
+h3. Running diagnostics from the same system as the installer
+
+The requirements to run diagnostics are having @arvados-client@ and @docker@ installed. If this is not possible you can run them on your Arvados shell node as explained in the next section.
+
+Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@. If you are running the installer from a host connected to the Arvados private network, use @-internal-client@. Otherwise, use @-external-client@.
+
+<pre><code class="userinput">./installer.sh diagnostics (-internal-client|-external-client)</code></pre>
+
+h3. Running diagnostics from a cluster node
+
+You can run the diagnostics from the cluster's shell node. This has the advantage that you don't need to manage any software on your local system, but might not be a possibility if your Arvados cluster doesn't include a shell node.
+
+<pre><code class="userinput">./installer.sh diagnostics-internal</code></pre>
+
+h3(#debugging). Debugging issues
+
+The installer records log files for each deployment.
-Replacing in each case de @A.B.C.x@ IP with the corresponding IP of the node.
+Most service logs go to @/var/log/syslog@.
-If your infrastructure uses another DNS service setup, add the corresponding entries accordingly.
+The logs for Rails API server can be found in @/var/www/arvados-api/current/log/production.log@ on the appropriate instance(s).
+
+Workbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in "Tools → Developer Tools").
+
+h3(#iterating). Iterating on config changes
+
+You can iterate on the config and maintain the cluster by making changes to @local.params@ and @local_config_dir@ and running @installer.sh deploy@ again.
+
+If you are debugging a configuration issue on a specific node, you can speed up the cycle a bit by deploying just one node:
+
+<pre><code class="userinput">./installer.sh deploy keep0.xarv1.example.com</code></pre>
+
+However, once you have a final configuration, you should run a full deploy to ensure that the configuration has been synchronized on all the nodes.
+
+h3(#common-problems). Common problems and solutions
+
+h4. PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
+
+The arvados-api-server package sets up the database as a post-install script. If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.
+
+This will manifest as an error like this:
+
+<pre>
+#<ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
+</pre>
+
+If this happens, you need to
+
+1. correct the database information
+2. run @./installer.sh deploy xarv1.example.com@ to update the configuration on the API/controller node
+3. Log in to the API/controller server node, then run this command to re-run the post-install script, which will set up the database:
+<pre><code class="userinput">dpkg-reconfigure arvados-api-server</code></pre>
+4. Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.
+
+h4. Missing ENA support (AWS Specific)
+
+If the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start. You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@. The solution is to build a new AMI with --aws-ena-support true
h2(#initial_user). Initial user and login
-At this point you should be able to log into the Arvados cluster.
+At this point you should be able to log into the Arvados cluster. The initial URL will be
+
+@https://workbench.${DOMAIN}@
+
+If you did *not* "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ from the @local.params*@ file.
+
+If you *did* configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
+
+h2(#monitoring). Monitoring and Metrics
+
+You can monitor the health and performance of the system using the admin dashboard:
+
+@https://grafana.${DOMAIN}@
+
+To log in, use username "admin" and @${INITIAL_USER_PASSWORD}@ from @local.params.secrets@.
+
+Once logged in, you will want to add the dashboards to the front page.
+
+# On the left icon bar, click on "Browse"
+# You should see a folder called "Arvados Cluster", click to open it
+## If you don't see anything, make sure the check box next to "Starred" is not selected
+# You should see three dashboards "Arvados cluster overview", "Node exporter" and "Postgres exporter"
+# Visit each dashboard, at the top of the page click on the star next to the title to "Mark as favorite"
+# They should now be linked on the front page.
+
+h2(#load_balancing). Load balancing controllers (optional)
+
+In order to handle high loads and perform rolling upgrades, the controller service can be scaled to a number of hosts and the installer make this implementation a fairly simple task.
+
+First, you should take care of the infrastructure deployment: if you use our Terraform code, you will need to set up the @terraform.tfvars@ in @terraform/vpc/@ so that in addition to the node named @controller@ (the load-balancer), a number of @controllerN@ nodes (backends) are defined as needed, and added to the @internal_service_hosts@ list.
+
+We suggest that the backend nodes just hold the controller service and nothing else, so they can be easily created or destroyed as needed without other service disruption.
+
+The following is an example @terraform/vpc/terraform.tfvars@ file that describes a cluster with a load-balancer, 2 backend nodes, a separate database node, a shell node, a keepstore node and a workbench node that will also hold other miscelaneous services:
+
+<pre><code>region_name = "us-east-1"
+cluster_name = "xarv1"
+domain_name = "xarv1.example.com"
+# Include controller nodes in this list so instances are assigned to the
+# private subnet. Only the balancer node should be connecting to them.
+internal_service_hosts = [ "keep0", "shell", "database", "controller1", "controller2" ]
+
+# Assign private IPs for the controller nodes. These will be used to create
+# internal DNS resolutions that will get used by the balancer and database nodes.
+private_ip = {
+ controller = "10.1.1.11"
+ workbench = "10.1.1.15"
+ database = "10.1.2.12"
+ controller1 = "10.1.2.21"
+ controller2 = "10.1.2.22"
+ shell = "10.1.2.17"
+ keep0 = "10.1.2.13"
+}</code></pre>
+
+Once the infrastructure is deployed, you'll then need to define which node will be using the @balancer@ role and which will be the @controller@ nodes in @local.params@, as it's being shown in this partial example:
+
+<pre><code>NODES=(
+ [controller.${DOMAIN}]=balancer
+ [controller1.${DOMAIN}]=controller
+ [controller2.${DOMAIN}]=controller
+ [database.${DOMAIN}]=database
+ ...
+)
+</code></pre>
-If you did not change the defaults, the initial URL will be:
+Note that we also set the @database@ role to its own node instead of just leaving it in a shared controller node.
-* https://workbench.arva2.arv.local
+Each time you run @installer.sh deploy@, the system will automatically do rolling upgrades. This means it will make changes to one controller node at a time, after removing it from the balancer so that there's no downtime.
-or, in general, the url format will be:
+h2(#post_install). After the installation
-* https://workbench.@<cluster>.<domain>@
+As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates. You should retain this repository but *be aware that it contains sensitive information* (passwords and tokens used by the Arvados services as well as cloud credentials if you used Terraform to create the infrastructure).
-By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+As described in "Iterating on config changes":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades. However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.
-Assuming you didn't change the defaults, the initial credentials are:
+When deploying upgrades, consult the "Arvados upgrade notes":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@. To specify the version to upgrade to, set the @VERSION@ parameter in @local.params@.
-* User: 'admin'
-* Password: 'password'
-* Email: 'admin@arva2.arv.local'
+See also "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
projects / arvados.git / blobdiff