---
layout: default
navsection: installguide
-title: Multi host Arvados
+title: Multi-Host Arvados
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
{% endcomment %}
# "Introduction":#introduction
-# "Hosts preparation":#hosts_preparation
-## "Create a compute image":#create_a_compute_image
-# "Multi host install using the provision.sh script":#multi_host
-# "Choose the desired configuration":#choose_configuration
-## "Multiple hosts / multiple hostnames":#multi_host_multi_hostnames
-## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
-# "Installation order":#installation_order
-# "Run the provision.sh script":#run_provision_script
+# "Prerequisites and planning":#prerequisites
+# "Required hosts":#hosts
+# "Download the installer":#download
+# "Initialize the installer":#copy_config
+# "Edit local.params":#localparams
+# "Configure Keep storage":#keep
+# "Choose the SSL configuration":#certificates
+## "Using a Let's Encrypt certificates":#lets-encrypt
+## "Bring your own certificates":#bring-your-own
+# "Create a compute image":#create_a_compute_image
+# "Begin installation":#installation
+# "Further customization of the installation":#further_customization
+# "Confirm the cluster is working":#test-install
+## "Debugging issues":#debugging
+## "Iterating on config changes":#iterating
+## "Common problems and solutions":#common-problems
# "Initial user and login":#initial_user
-# "Test the installed cluster running a simple workflow":#test_install
+# "After the installation":#post_install
+h2(#introduction). Introduction
+This multi host installer is the recommendend way to set up a production Arvados cluster. These instructions include specific details for installing on Amazon Web Services (AWS), which are marked as "AWS specific". However with additional customization the installer can be used as a template for deployment on other cloud provider or HPC systems.
-h2(#introduction). Introduction
+h2(#prerequisites). Prerequisites and planning
+
+h3. Cluster ID and base domain
+
+Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or ${CLUSTER}, you should substitute the cluster id you have selected.
+
+Determine the base domain for the cluster. This will be referred to as ${DOMAIN}
+
+For example, if CLUSTER is "xarv1" and DOMAIN is "example.com", then "controller.${CLUSTER}.${DOMAIN}" means "controller.xargv1.example.com".
+
+h3. Virtual Private Cloud (AWS specific)
-Arvados components can be installed in a distributed infrastructure, whether it is an "on-prem" with physical or virtual hosts, or a cloud environment.
+We recommend setting Arvados up in a "Virtual Private Cloud (VPC)":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html
-As infrastructures vary a great deal from site to site, these instructions should be considered more as 'guidelines' than fixed steps to follow.
+When you do so, you need to configure a couple of additional things:
-We provide an "installer script":salt.html that can help you deploy the different Arvados components. At the time of writing, the provided examples are suitable to install Arvados on AWS.
+# "Create a subnet for the compute nodes":https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html
+# You should set up a "security group which allows SSH access (port 22)":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
+# Make sure to add a "VPC S3 endpoint":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
+h3(#keep-bucket). S3 Bucket (AWS specific)
+We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@. We recommend creating an IAM role called @${CLUSTER}-keepstore-00-iam-role@ with a "policy that can read, write, list and delete objects in the bucket":configure-s3-object-storage.html#IAM . With the example cluster id @xargv1@ the bucket would be called @xarv1-nyw5e-000000000000000-volume@ and the role would be called @xarv1-keepstore-00-iam-role@.
-h2(#hosts_preparation). Hosts preparation
+These names are recommended because they are default names used in the configuration template. If you use different names, you will need to edit the configuration template later.
-In order to run Arvados on a multi-host installation, there are a few requirements that your infrastructure has to fulfill.
+h2(#hosts). Required hosts
-These instructions explain how to setup a multi-host environment that is suitable for production use of Arvados.
+You will need to allocate several hosts (physical or virtual machines) for the fixed infrastructure of the Arvados cluster. These machines should have at least 2 cores and 8 GiB of RAM, running a supported Linux distribution.
-We suggest distributing the Arvados components in the following way, creating at least 6 hosts:
+{% include 'supportedlinux' %}
-# Database server:
+Allocate the following hosts as appropriate for your site. On AWS you may choose to do it manually with the AWS console, or using a DevOps tool such as CloudFormation or Terraform. With the exception of "keep0" and "keep1", all of these hosts should have external (public) IP addresses if you intend for them to be accessible outside of the private network or VPC.
+
+The installer will set up the Arvados services on your machines. Here is the default assignment of services to machines:
+
+# API node
## postgresql server
-# API node:
## arvados api server
-## arvados controller
-## arvados websocket
+## arvados controller (recommendend hostname @controller.${CLUSTER}.${DOMAIN}@)
+## arvados websocket (recommendend hostname @ws.${CLUSTER}.${DOMAIN}@)
## arvados cloud dispatcher
-# WORKBENCH node:
-## arvados workbench
-## arvados workbench2
-## arvados webshell
-# KEEPPROXY node:
-## arvados keepproxy
-## arvados keepweb
-# KEEPSTOREs (at least 2)
-## arvados keepstore
-# SHELL node (optional):
-## arvados shell
+## arvados keepbalance
+# KEEPSTORE nodes (at least 2)
+## arvados keepstore (recommendend hostnames @keep0.${CLUSTER}.${DOMAIN}@ and @keep1.${CLUSTER}.${DOMAIN}@)
+# KEEPPROXY node
+## arvados keepproxy (recommendend hostname @keep.${CLUSTER}.${DOMAIN}@)
+## arvados keepweb (recommendend hostname @download.${CLUSTER}.${DOMAIN}@ and @*.collections.${CLUSTER}.${DOMAIN}@)
+# WORKBENCH node
+## arvados workbench (recommendend hostname @workbench.${CLUSTER}.${DOMAIN}@)
+## arvados workbench2 (recommendend hostname @workbench2.${CLUSTER}.${DOMAIN}@)
+## arvados webshell (recommendend hostname @webshell.${CLUSTER}.${DOMAIN}@)
+# SHELL node (optional)
+## arvados shell (recommended hostname @shell.${CLUSTER}.${DOMAIN}@)
+
+h3(#DNS). DNS hostnames for each service
+
+You will need a DNS entry for each service. In the default configuration these are:
+
+# @controller.${CLUSTER}.${DOMAIN}@
+# @ws.${CLUSTER}.${DOMAIN}@
+# @keep0.${CLUSTER}.${DOMAIN}@
+# @keep1.${CLUSTER}.${DOMAIN}@
+# @keep.${CLUSTER}.${DOMAIN}@
+# @download.${CLUSTER}.${DOMAIN}@
+# @*.collections.${CLUSTER}.${DOMAIN}@ -- important note, this must be a wildcard DNS, resolving to the @keepweb@ service
+# @workbench.${CLUSTER}.${DOMAIN}@
+# @workbench2.${CLUSTER}.${DOMAIN}@
+# @webshell.${CLUSTER}.${DOMAIN}@
+# @shell.${CLUSTER}.${DOMAIN}@
+
+h3. Additional prerequisites when preparing machines to run the installer
+
+# From the account where you are performing the install, passwordless @ssh@ to each machine
+This means the client's public key should added to @~/.ssh/authorized_keys@ on each node.
+# Passwordless @sudo@ access on the account on each machine you will @ssh@ in to
+This usually means adding the account to the @sudo@ group and having a rule like this in @/etc/sudoers.d/arvados_passwordless@ that allows members of group @sudo@ to execute any command without entering a password.
+<pre>%sudo ALL=(ALL:ALL) NOPASSWD:ALL</pre>
+# @git@ installed on each machine
+# Port 443 reachable by clients
+
+(AWS specific) The machine that runs the arvados cloud dispatcher will need an "IAM role that allows it to manage EC2 instances.":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html#IAM
+
+If your infrastructure differs from the setup proposed above (ie, different hostnames), you can still use the installer, but "additional customization may be necessary":#further_customization .
+
+h2(#download). Download the installer
+
+{% assign local_params_src = 'multiple_hosts' %}
+{% assign config_examples_src = 'multi_host/aws'%}
+{% include 'download_installer' %}
+
+h2(#localparams). Edit @local.params@
+
+This can be found wherever you choose to initialize the install files (@~/setup-arvados-xarv1@ in these examples).
+
+# Set @CLUSTER@ to the 5-character cluster identifier (e.g "xarv1")
+# Set @DOMAIN@ to the base DNS domain of the environment, e.g. "example.com"
+# Edit Internal IP settings. Since services share hosts, some hosts are the same. See "note about /etc/hosts":#etchosts
+# Edit @CLUSTER_INT_CIDR@, this should be the CIDR of the private network that Arvados is running on, e.g. the VPC.
+CIDR stands for "Classless Inter-Domain Routing" and describes which portion of the IP address that refers to the network. For example 192.168.3.0/24 means that the first 24 bits are the network (192.168.3) and the last 8 bits are a specific host on that network.
+_AWS Specific: Go to the AWS console and into the VPC service, there is a column in this table view of the VPCs that gives the CIDR for the VPC (IPv4 CIDR)._
+# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.
+# Set each @KEY@ / @TOKEN@ to a random string
+ Here's an easy way to create five random tokens:
+<pre><code>for i in 1 2 3 4 5; do
+ tr -dc A-Za-z0-9 </dev/urandom | head -c 32 ; echo ''
+done
+</code></pre>
+# Set @DATABASE_PASSWORD@ to a random string (unless you "already have a database":#ext-database then you should set it to that database's password)
+ Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.
+ For example, if the password is @Lq&MZ<V']d?j@
+ With backslash quoting the special characters it should appear like this in local.params:
+<pre><code>DATABASE_PASSWORD="Lq\&MZ\<V\'\]d\?j"</code></pre>
-Note that these hosts can be virtual machines in your infrastructure and they don't need to be physical machines.
+h3(#etchosts). Note on @/etc/hosts@
-Again, if your infrastructure differs from the setup proposed above (ie, using RDS or an existing DB server), remember that you will need to edit the configuration files for the scripts so they work with your infrastructure.
+Because Arvados services are typically accessed by external clients, they are likely to have both a public IP address and a internal IP address.
-h2(#multi_host). Multi host install using the provision.sh script
+On cloud providers such as AWS, sending internal traffic to a service's public IP address can incur egress costs and throttling. Thus it is very important for internal traffic to stay on the internal network. The installer implements this by updating @/etc/hosts@ on each node to associate each service's hostname with the internal IP address, so that when Arvados services communicate with one another, they always use the internal network address. This is NOT a substitute for DNS, you still need to set up DNS names for all of the services that have public IP addresses (it does, however, avoid a complex "split-horizon" DNS configuration).
-{% include 'branchname' %}
+It is important to be aware of this because if you mistype the IP address for any of the @*_INT_IP@ variables, hosts may unexpectedly fail to be able to communicate with one another. If this happens, check and edit as necessary the file @/etc/hosts@ on the host that is failing to make an outgoing connection.
+
+h2(#keep). Configure Keep storage
+
+The @multi_host/aws@ template uses S3 for storage. Arvados also supports "filesystem storage":configure-fs-storage.html and "Azure blob storage":configure-azure-blob-storage.html . Keep storage configuration can be found in in the @arvados.cluster.Volumes@ section of @local_config_dir/pillars/arvados.sls@.
-This is a package-based installation method. Start with the @provision.sh@ script which is available by cloning the @{{ branchname }}@ branch from "https://git.arvados.org/arvados.git":https://git.arvados.org/arvados.git . The @provision.sh@ script and its supporting files can be found in the "arvados/tools/salt-install":https://git.arvados.org/arvados.git/tree/refs/heads/{{ branchname }}:/tools/salt-install directory in the Arvados git repository.
+h3. Object storage in S3 (AWS Specific)
-This procedure will install all the main Arvados components to get you up and running in a multi-host environment.
+Open @local_config_dir/pillars/arvados.sls@ and edit as follows:
-The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located at "arvados-formula":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
+# In the @arvados.cluster.Volumes.DriverParameters@ section, set @Region@ to the appropriate AWS region (e.g. 'us-east-1')
-After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
+If you did not "follow the recommendend naming scheme":#keep-bucket for either the bucket or role, you'll need to update these parameters as well:
-h3(#create_a_compute_image). Create a compute image
+# Set @Bucket@ to the value of "keepstore bucket you created earlier":#keep-bucket
+# Set @IAMRole@ to "keepstore role you created earlier":#keep-bucket
-In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/main/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image ID in the Arvados configuration in the next steps.
+{% include 'ssl_config_multi' %}
-h2(#choose_configuration). Choose the desired configuration
+h2(#authentication). Configure your authentication provider (optional, recommended)
-For documentation's sake, we will use the cluster name <i>arva2</i> and the domain <i>arv.local</i>. If you don't change them as required in the next steps, installation won't proceed.
+By default, the installer will use the "Test" provider, which is a list of usernames and cleartext passwords stored in the Arvados config file. *This is low security configuration and you are strongly advised to configure one of the other "supported authentication methods":setup-login.html* .
-We will try to provide a few Arvados' multi host installation configurations examples for different infrastructure providers. Currently only AWS is available but they can be used with almost any provider with little changes.
+h2(#ext-database). Using an external database (optional)
-You need to copy one of the example configuration files and directory, and edit them to suit your needs.
+The standard behavior of the installer is to install and configure PostgreSQL for use by Arvados. You can optionally configure it to use a separately managed database instead.
-h3(#multi_host_multi_hostnames). Multiple hosts / multiple hostnames
-<notextile>
-<pre><code>cp local.params.example.multiple_hosts local.params
-cp -r config_examples/multi_host/aws local_config_dir
+Arvados requires a database that is compatible with PostgreSQL 9.5 or later. For example, Arvados is known to work with Amazon Aurora (note: even idle, Arvados services will periodically poll the database, so we strongly advise using "provisioned" mode).
+
+# In @local.params@, remove 'database' from the list of roles assigned to the controller node:
+<pre><code>NODES=(
+ [controller.${CLUSTER}.${DOMAIN}]=api,controller,websocket,dispatcher,keepbalance
+ ...
+)
+</code></pre>
+# In @local.params@, set @DATABASE_INT_IP@ to the database endpoint (can be a hostname, does not have to be an IP address).
+<pre><code>DATABASE_INT_IP=...
</code></pre>
-</notextile>
+# In @local.params@, set @DATABASE_PASSWORD@ to the correct value. "See the previous section describing correct quoting":#localparams
+# In @local_config_dir/pillars/arvados.sls@ you may need to adjust the database name and user. This can be found in the section @arvados.cluster.database@.
-Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_INT_IP, *_TOKEN</b> and <b>*KEY</b> variables. Those variables will be used to do a search and replace on the <i>pillars/*</i> in place of any matching __VARIABLE__.
+h2(#further_customization). Further customization of the installation (optional)
-The <i>multi_host</i> example includes Let's Encrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
+If you are installing on AWS and have followed all of the naming conventions recommend in this guide, you probably don't need to do any further customization.
-{% include 'install_custom_certificates' %}
+If you are installing on a different cloud provider or on HPC, other changes may require editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ contains the template (in the @arvados.cluster@ section) used to produce the Arvados configuration file that is distributed to all the nodes. Consult the "Configuration reference":config.html for a comprehensive list of configuration keys.
-h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
+Any extra Salt "state" files you add under @local_config_dir/states@ will be added to the Salt run and applied to the hosts.
-You will need further customization to suit your environment, which can be done editing the Saltstack pillars and states files. Pay particular attention to the <i>pillars/arvados.sls</i> file, where you will need to provide some information that describes your environment.
+h2(#create_a_compute_image). Create a compute image
-Any extra <i>state</i> file you add under <i>local_config_dir/states</i> will be added to the salt run and applied to the hosts.
+{% include 'branchname' %}
-h2(#installation_order). Installation order
+On cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup. If you will use a HPC scheduler such as SLURM you can skip this section.
-A few Arvados nodes need to be installed in certain order. The required order is
+*Start by following "the instructions to build a cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images* .
-* Database
-* API server
-* The other nodes can be installed in any order after the two above
+Once you have that image created, Open @local_config_dir/pillars/arvados.sls@ and edit as follows (AWS specific settings described here, other cloud providers will have similar settings in their respective configuration section):
-h2(#run_provision_script). Run the provision.sh script
+# In the @arvados.cluster.Containers.CloudVMs@ section:
+## Set @ImageID@ to the AMI produced by Packer
+## Set @DriverParameters.Region@ to the appropriate AWS region
+## Set @DriverParameters.AdminUsername@ to the admin user account on the image
+## Set the @DriverParameters.SecurityGroupIDs@ list to the VPC security group which you set up to allow SSH connections to these nodes
+## Set @DriverParameters.SubnetID@ to the value of SubnetId of your VPC
+# Update @arvados.cluster.Containers.DispatchPrivateKey@ and paste the contents of the @~/.ssh/id_dispatcher@ file you generated in an earlier step.
+# Update @arvados.cluster.InstanceTypes@ as necessary. The example instance types are for AWS, other cloud providers will of course have different instance types with different names and specifications.
+(AWS specific) If m5/c5 node types are not available, replace them with m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.
-When you finished customizing the configuration, you are ready to copy the files to the hosts and run the @provision.sh@ script. The script allows you to specify the <i>role/s</i> a node will have and it will install only the Arvados components required for such role. The general format of the command is:
+h2(#installation). Begin installation
-<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-# if you use custom certificates (not Let's Encrypt), make sure to copy those too:
-# scp -r certs user@host:
-ssh user@host sudo ./provision.sh --roles comma,separated,list,of,roles,to,apply
-</code></pre>
-</notextile>
+At this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.
-and wait for it to finish.
+Run this in the @~/arvados-setup-xarv1@ directory:
-If everything goes OK, you'll get some final lines stating something like:
+<pre>
+./installer.sh deploy
+</pre>
-<notextile>
-<pre><code>arvados: Succeeded: 109 (changed=9)
-arvados: Failed: 0
-</code></pre>
-</notextile>
+This will install and configure Arvados on all the nodes. It will take a while and produce a lot of logging. If it runs into an error, it will stop.
-The distribution of role as described above can be applied running these commands:
+h2(#test-install). Confirm the cluster is working
-h4. Database
-<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo ./provision.sh --config local.params --roles database
-</code></pre>
-</notextile>
+When everything has finished, you can run the diagnostics.
-h4. API
-<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo ./provision.sh --config local.params --roles api,controller,websocket,dispatcher
-</code></pre>
-</notextile>
+Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.
-h4. Keepstore(s)
-<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo ./provision.sh --config local.params --roles keepstore
-</code></pre>
-</notextile>
+If you are running the diagnostics from one of the Arvados machines inside the private network, you want @-internal-client@ .
-h4. Workbench
-<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo ./provision.sh --config local.params --roles workbench,workbench2,webshell
-</code></pre>
-</notextile>
+You are an "external client" if you running the diagnostics from your workstation outside of the private network.
-h4. Keepproxy / Keepweb
-<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo ./provision.sh --config local.params --roles keepproxy,keepweb
-</code></pre>
-</notextile>
+<pre>
+./installer.sh diagnostics (-internal-client|-external-client)
+</pre>
-h4. Shell (here we copy the CLI test workflow too)
-<notextile>
-<pre><code>scp -r provision.sh local* tests user@host:
-ssh user@host sudo ./provision.sh --config local.params --roles shell
-</code></pre>
-</notextile>
+h3(#debugging). Debugging issues
-h2(#initial_user). Initial user and login
+The installer records log files for each deployment.
-At this point you should be able to log into the Arvados cluster. The initial URL will be:
+Most service logs go to @/var/log/syslog@.
-* https://workbench.arva2.arv.local
+The logs for Rails API server and for Workbench can be found in
-or, in general, the url format will be:
+@/var/www/arvados-api/current/log/production.log@
+and
+@/var/www/arvados-workbench/current/log/production.log@
-* https://workbench.@<cluster>.<domain>@
+on the appropriate instances.
-By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+Workbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in "Tools → Developer Tools").
-Assuming you didn't change these values in the @local.params@ file, the initial credentials are:
+h3(#iterating). Iterating on config changes
-* User: 'admin'
-* Password: 'password'
-* Email: 'admin@arva2.arv.local'
+You can iterate on the config and maintain the cluster by making changes to @local.params@ and @local_config_dir@ and running @installer.sh deploy@ again.
-h2(#test_install). Test the installed cluster running a simple workflow
+If you are debugging a configuration issue on a specific node, you can speed up the cycle a bit by deploying just one node:
-If you followed the instructions above, the @provision.sh@ script saves a simple example test workflow in the @/tmp/cluster_tests@ directory in the @shell@ node. If you want to run it, just ssh to the node, change to that directory and run:
+<pre>
+./installer.sh deploy keep0.xarv1.example.com@
+</pre>
-<notextile>
-<pre><code>cd /tmp/cluster_tests
-sudo /run-test.sh
-</code></pre>
-</notextile>
-
-It will create a test user (by default, the same one as the admin user), upload a small workflow and run it. If everything goes OK, the output should similar to this (some output was shortened for clarity):
-
-<notextile>
-<pre><code>Creating Arvados Standard Docker Images project
-Arvados project uuid is 'arva2-j7d0g-0prd8cjlk6kfl7y'
-{
- ...
- "uuid":"arva2-o0j2j-n4zu4cak5iifq2a",
- "owner_uuid":"arva2-tpzed-000000000000000",
- ...
-}
-Uploading arvados/jobs' docker image to the project
-2.1.1: Pulling from arvados/jobs
-8559a31e96f4: Pulling fs layer
-...
-Status: Downloaded newer image for arvados/jobs:2.1.1
-docker.io/arvados/jobs:2.1.1
-2020-11-23 21:43:39 arvados.arv_put[32678] INFO: Creating new cache file at /home/vagrant/.cache/arvados/arv-put/c59256eda1829281424c80f588c7cc4d
-2020-11-23 21:43:46 arvados.arv_put[32678] INFO: Collection saved as 'Docker image arvados jobs:2.1.1 sha256:0dd50'
-arva2-4zz18-1u5pvbld7cvxuy2
-Creating initial user ('admin')
-Setting up user ('admin')
-{
- "items":[
- {
- ...
- "owner_uuid":"arva2-tpzed-000000000000000",
- ...
- "uuid":"arva2-o0j2j-1ownrdne0ok9iox"
- },
- {
- ...
- "owner_uuid":"arva2-tpzed-000000000000000",
- ...
- "uuid":"arva2-o0j2j-1zbeyhcwxc1tvb7"
- },
- {
- ...
- "email":"admin@arva2.arv.local",
- ...
- "owner_uuid":"arva2-tpzed-000000000000000",
- ...
- "username":"admin",
- "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
- ...
- }
- ],
- "kind":"arvados#HashList"
-}
-Activating user 'admin'
-{
- ...
- "email":"admin@arva2.arv.local",
- ...
- "username":"admin",
- "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
- ...
-}
-Running test CWL workflow
-INFO /usr/bin/cwl-runner 2.1.1, arvados-python-client 2.1.1, cwltool 3.0.20200807132242
-INFO Resolved 'hasher-workflow.cwl' to 'file:///tmp/cluster_tests/hasher-workflow.cwl'
-...
-INFO Using cluster arva2 (https://arva2.arv.local:8443/)
-INFO Upload local files: "test.txt"
-INFO Uploaded to ea34d971b71d5536b4f6b7d6c69dc7f6+50 (arva2-4zz18-c8uvwqdry4r8jao)
-INFO Using collection cache size 256 MiB
-INFO [container hasher-workflow.cwl] submitted container_request arva2-xvhdp-v1bkywd58gyocwm
-INFO [container hasher-workflow.cwl] arva2-xvhdp-v1bkywd58gyocwm is Final
-INFO Overall process status is success
-INFO Final output collection d6c69a88147dde9d52a418d50ef788df+123
-{
- "hasher_out": {
- "basename": "hasher3.md5sum.txt",
- "class": "File",
- "location": "keep:d6c69a88147dde9d52a418d50ef788df+123/hasher3.md5sum.txt",
- "size": 95
- }
-}
-INFO Final process status is success
-</code></pre>
-</notextile>
+However, once you have a final configuration, you should run a full deploy to ensure that the configuration has been synchronized on all the nodes.
+
+h3(#common-problems). Common problems and solutions
+
+h4. PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
+
+The arvados-api-server package sets up the database as a post-install script. If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.
+
+This will manifest as an error like this:
+
+<pre>
+#<ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
+</pre>
+
+If this happens, you need to
+
+1. correct the database information
+2. run @./installer.sh deploy xarv1.example.com@ to update the configuration on the API/controller node
+3. Log in to the API/controller server node, then run this command to re-run the post-install script, which will set up the database:
+<pre>dpkg-reconfigure arvados-api-server</pre>
+4. Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.
+
+h4. Missing ENA support (AWS Specific)
+
+If the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start. You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@. The solution is to build a new AMI with --aws-ena-support true
+
+h2(#initial_user). Initial user and login
+
+At this point you should be able to log into the Arvados cluster. The initial URL will be
+
+https://workbench.${CLUSTER}.${DOMAIN}
+
+If you did *not* "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
+
+If you *did* configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
+
+h2(#post_install). After the installation
+
+As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates. You should retain this repository but be aware that it contains sensitive information (passwords and tokens used by the Arvados services).
+
+As described in "Iterating on config changes":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades. However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.
+
+When deploying upgrades, consult the "Arvados upgrade notes":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@. To specify the version to upgrade to, set the @VERSION@ parameter in @local.params@.
+
+See also "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
projects / arvados.git / blobdiff