# "Introduction":#introduction
# "Prerequisites and planning":#prerequisites
-# "Required hosts":#hosts
# "Download the installer":#download
# "Initialize the installer":#copy_config
+# "Set up your infrastructure":#setup-infra
+## "Create AWS infrastructure with Terraform":#terraform
+## "Create required infrastructure manually":#inframanual
# "Edit local.params":#localparams
# "Configure Keep storage":#keep
# "Choose the SSL configuration":#certificates
h3. Cluster ID and base domain
-Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or ${CLUSTER}, you should substitute the cluster id you have selected.
+Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or @${CLUSTER}@, you should substitute the cluster id you have selected.
-Determine the base domain for the cluster. This will be referred to as ${DOMAIN}
+Determine the base domain for the cluster. This will be referred to as @${DOMAIN}@.
-For example, if CLUSTER is "xarv1" and DOMAIN is "example.com", then "controller.${CLUSTER}.${DOMAIN}" means "controller.xargv1.example.com".
+For example, if CLUSTER is @xarv1@ and DOMAIN is @example.com@, then @controller.${CLUSTER}.${DOMAIN}@ means @controller.xarv1.example.com@.
+
+h3(#DNS). DNS hostnames for each service
+
+You will need a DNS entry for each service. When using the "Terraform script":#terraform to set up your infrastructure, these domains will be created automatically using AWS Route 53.
+
+In the default configuration these are:
+
+# @controller.${CLUSTER}.${DOMAIN}@
+# @ws.${CLUSTER}.${DOMAIN}@
+# @keep0.${CLUSTER}.${DOMAIN}@
+# @keep1.${CLUSTER}.${DOMAIN}@
+# @keep.${CLUSTER}.${DOMAIN}@
+# @download.${CLUSTER}.${DOMAIN}@
+# @*.collections.${CLUSTER}.${DOMAIN}@ -- important note, this must be a wildcard DNS, resolving to the @keepweb@ service
+# @workbench.${CLUSTER}.${DOMAIN}@
+# @workbench2.${CLUSTER}.${DOMAIN}@
+# @webshell.${CLUSTER}.${DOMAIN}@
+# @shell.${CLUSTER}.${DOMAIN}@
+
+For more information, see "DNS entries and TLS certificates":install-manual-prerequisites.html#dnstls.
+
+h2(#download). Download the installer
+
+{% assign local_params_src = 'multiple_hosts' %}
+{% assign config_examples_src = 'multi_host/aws' %}
+{% assign terraform_src = 'terraform/aws' %}
+{% include 'download_installer' %}
+
+h2(#setup-infra). Set up your infrastructure
+
+## "Create AWS infrastructure with Terraform":#terraform
+## "Create required infrastructure manually":#inframanual
+
+h3(#terraform). Create AWS infrastructure with Terraform (AWS specific)
+
+We provide a set of Terraform code files that you can run to create the necessary infrastructure on Amazon Web Services.
+
+These files are located in the @terraform@ installer directory and are divided in three sections:
+
+# The @terraform/vpc/@ subdirectory controls the network related infrastructure of your cluster, including firewall rules and split-horizon DNS resolution.
+# The @terraform/data-storage/@ subdirectory controls the stateful part of your cluster, currently only sets up the S3 bucket for holding the Keep blocks and in the future it'll also manage the database service.
+# The @terraform/services/@ subdirectory controls the hosts that will run the different services on your cluster, makes sure that they have the required software for the installer to do its job.
+
+h4. Software requirements & considerations
+
+{% include 'notebox_begin' %}
+The Terraform state files (that keep crucial infrastructure information from the cloud) will be saved inside each subdirectory, under the @terraform.tfstate@ name. These will be committed to the git repository used to coordinate deployment. It is very important to keep this git repository secure, only sysadmins that will be responsible for maintaining your Arvados cluster should have access to it.
+{% include 'notebox_end' %}
+
+h4. Terraform code configuration
+
+Each section described above contain a @terraform.tfvars@ file with some configuration values that you should set before applying each configuration. You should set the cluster prefix and domain name in @vpc/terraform.tfvars@:
+
+<pre><code>region_name = "us-east-1"
+# cluster_name = "xarv1"
+# domain_name = "example.com"</code></pre>
+
+If you don't set the variables @vpc/terraform.tfvars@ file, you will be asked to re-enter these parameters every time you run Terraform.
+
+The @data-storage/terraform.tfvars@ and @services/terraform.tfvars@ let you configure the location of your ssh public key (default @~/.ssh/id_rsa.pub@) and the instance type to use (default @m5a.large@).
+
+h4. Create the infrastructure
+
+Build the infrastructure by running @./installer.sh terraform@. The last stage will output the information needed to set up the cluster's domain and continue with the installer. for example:
+
+<pre><code>$ ./installer.sh terraform
+...
+Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
+
+Outputs:
+
+arvados_sg_id = "sg-02f999a99973999d7"
+arvados_subnet_id = "subnet-01234567abc"
+cluster_name = "xarv1"
+compute_subnet_id = "subnet-abcdef12345"
+deploy_user = "admin"
+domain_name = "example.com"
+letsencrypt_iam_access_key_id = "AKAA43MAAAWAKAADAASD"
+private_ip = {
+ "controller" = "10.1.1.1"
+ "keep0" = "10.1.1.3"
+ "keep1" = "10.1.1.4"
+ "keepproxy" = "10.1.1.2"
+ "shell" = "10.1.1.7"
+ "workbench" = "10.1.1.5"
+}
+public_ip = {
+ "controller" = "18.235.116.23"
+ "keep0" = "34.202.85.86"
+ "keep1" = "38.22.123.98"
+ "keepproxy" = "34.231.9.201"
+ "shell" = "44.208.155.240"
+ "workbench" = "52.204.134.136"
+}
+region_name = "us-east-1"
+route53_dns_ns = tolist([
+ "ns-1119.awsdns-11.org",
+ "ns-1812.awsdns-34.co.uk",
+ "ns-437.awsdns-54.com",
+ "ns-809.awsdns-37.net",
+])
+vpc_cidr = "10.1.0.0/16"
+vpc_id = "vpc-0999994998399923a"
+letsencrypt_iam_secret_access_key = "XXXXXSECRETACCESSKEYXXXX"
+</code></pre>
-h3. Virtual Private Cloud (AWS specific)
-We recommend setting Arvados up in a "Virtual Private Cloud (VPC)":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html
+h4. Additional DNS configuration
+
+Once Terraform has completed, the infrastructure for your Arvados cluster is up and running. One last piece of DNS configuration is required.
+
+The domain names for your cluster (e.g.: controller.xarv1.example.com) are managed via "Route 53":https://aws.amazon.com/route53/ and the TLS certificates will be issued using "Let's Encrypt":https://letsencrypt.org/ .
+
+You need to configure the parent domain to delegate to the newly created zone. In other words, you need to configure @${DOMAIN}@ (e.g. "example.com") to delegate the subdomain @${CLUSTER}.${DOMAIN}@ (e.g. "xarv1.example.com") to the nameservers for the Arvados hostname records created by Terraform. You do this by creating a @NS@ record on the parent domain that refers to the name servers listed in the Terraform output parameter @route53_dns_ns@.
+
+If your parent domain is also controlled by Route 53, the process will be like this:
+
+# Log in to the AWS Console and navigate to the service page for *Route 53*
+# Go to the list of *Hosted zones* and click on the zone for the parent domain
+# Click on *Create record*
+# For *Record name* put the cluster id
+# For *Record type* choose @NS - Name servers for a hosted zone@
+# For *Value* add the values from Terraform output parameter @route53_dns_ns@, one hostname per line, with punctuation (quotes and commas) removed.
+# Click *Create records*
+
+If the parent domain is controlled by some other service, follow the guide for the the appropriate service.
+
+h4. Other important output parameters
+
+The certificates will be requested from Let's Encrypt when you run the installer.
+
+* @vpc_cidr@ will be used to set @CLUSTER_INT_CIDR@
+
+* You'll also need @compute_subnet_id@ and @arvados_sg_id@ to set @DriverParameters.SubnetID@ and @DriverParameters.SecurityGroupIDs@ in @local_config_dir/pillars/arvados.sls@ and when you "create a compute image":#create_a_compute_image.
+
+You can now proceed to "edit local.params":#localparams.
+
+h3(#inframanual). Create required infrastructure manually
+
+If you will be setting up infrastructure without using the provided Terraform script, here are the recommendations you will need to consider.
+
+h4. Virtual Private Cloud (AWS specific)
+
+We recommend setting Arvados up in its own "Virtual Private Cloud (VPC)":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html
When you do so, you need to configure a couple of additional things:
# You should set up a "security group which allows SSH access (port 22)":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
# Make sure to add a "VPC S3 endpoint":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
-h3(#keep-bucket). S3 Bucket (AWS specific)
+h4(#keep-bucket). S3 Bucket (AWS specific)
-We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@. We recommend creating an IAM role called @${CLUSTER}-keepstore-00-iam-role@ with a "policy that can read, write, list and delete objects in the bucket":configure-s3-object-storage.html#IAM . With the example cluster id @xargv1@ the bucket would be called @xarv1-nyw5e-000000000000000-volume@ and the role would be called @xarv1-keepstore-00-iam-role@.
+We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@. We recommend creating an IAM role called @${CLUSTER}-keepstore-00-iam-role@ with a "policy that can read, write, list and delete objects in the bucket":configure-s3-object-storage.html#IAM . With the example cluster id @xarv1@ the bucket would be called @xarv1-nyw5e-000000000000000-volume@ and the role would be called @xarv1-keepstore-00-iam-role@.
These names are recommended because they are default names used in the configuration template. If you use different names, you will need to edit the configuration template later.
-h2(#hosts). Required hosts
+h4(#hosts). Required hosts
You will need to allocate several hosts (physical or virtual machines) for the fixed infrastructure of the Arvados cluster. These machines should have at least 2 cores and 8 GiB of RAM, running a supported Linux distribution.
# SHELL node (optional)
## arvados shell (recommended hostname @shell.${CLUSTER}.${DOMAIN}@)
-h3(#DNS). DNS hostnames for each service
+When using the database installed by Arvados (and not an "external database":#ext-database), the database is stored under @/var/lib/postgresql@. Arvados logs are also kept in @/var/log@ and @/var/www/arvados-api/shared/log@. Accordingly, you should ensure that the disk partition containing @/var@ has adequate storage for your planned usage. We suggest starting with 50GiB of free space on the database host.
-You will need a DNS entry for each service. In the default configuration these are:
-
-# @controller.${CLUSTER}.${DOMAIN}@
-# @ws.${CLUSTER}.${DOMAIN}@
-# @keep0.${CLUSTER}.${DOMAIN}@
-# @keep1.${CLUSTER}.${DOMAIN}@
-# @keep.${CLUSTER}.${DOMAIN}@
-# @download.${CLUSTER}.${DOMAIN}@
-# @*.collections.${CLUSTER}.${DOMAIN}@ -- important note, this must be a wildcard DNS, resolving to the @keepweb@ service
-# @workbench.${CLUSTER}.${DOMAIN}@
-# @workbench2.${CLUSTER}.${DOMAIN}@
-# @webshell.${CLUSTER}.${DOMAIN}@
-# @shell.${CLUSTER}.${DOMAIN}@
-
-This is described in more detail in "DNS entries and TLS certificates":install-manual-prerequisites.html#dnstls.
-
-h3. Additional prerequisites when preparing machines to run the installer
+h4. Additional prerequisites when preparing machines to run the installer
# From the account where you are performing the install, passwordless @ssh@ to each machine
This means the client's public key should added to @~/.ssh/authorized_keys@ on each node.
If your infrastructure differs from the setup proposed above (ie, different hostnames), you can still use the installer, but "additional customization may be necessary":#further_customization .
-h2(#download). Download the installer
-
-{% assign local_params_src = 'multiple_hosts' %}
-{% assign config_examples_src = 'multi_host/aws'%}
-{% include 'download_installer' %}
-
h2(#localparams). Edit @local.params@
This can be found wherever you choose to initialize the install files (@~/setup-arvados-xarv1@ in these examples).
# Set @CLUSTER@ to the 5-character cluster identifier (e.g "xarv1")
# Set @DOMAIN@ to the base DNS domain of the environment, e.g. "example.com"
-# Edit Internal IP settings. Since services share hosts, some hosts are the same. See "note about /etc/hosts":#etchosts
+# Set the @*_INT_IP@ variables with the internal (private) IP addresses of each host. Since services share hosts, some hosts are the same. See "note about /etc/hosts":#etchosts
# Edit @CLUSTER_INT_CIDR@, this should be the CIDR of the private network that Arvados is running on, e.g. the VPC.
CIDR stands for "Classless Inter-Domain Routing" and describes which portion of the IP address that refers to the network. For example 192.168.3.0/24 means that the first 24 bits are the network (192.168.3) and the last 8 bits are a specific host on that network.
_AWS Specific: Go to the AWS console and into the VPC service, there is a column in this table view of the VPCs that gives the CIDR for the VPC (IPv4 CIDR)._
# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.
-# Set each @KEY@ / @TOKEN@ to a random string
- Here's an easy way to create five random tokens:
-<pre><code>for i in 1 2 3 4 5; do
- tr -dc A-Za-z0-9 </dev/urandom | head -c 32 ; echo ''
-done
+# Set each @KEY@ / @TOKEN@ / @PASSWORD@ to a random string. You can use @installer.sh generate-tokens@
+<pre><code>$ ./installer.sh generate-tokens
+BLOB_SIGNING_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+MANAGEMENT_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+SYSTEM_ROOT_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+ANONYMOUS_USER_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+WORKBENCH_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+DATABASE_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
</code></pre>
# Set @DATABASE_PASSWORD@ to a random string (unless you "already have a database":#ext-database then you should set it to that database's password)
Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.
# In the @arvados.cluster.Volumes.DriverParameters@ section, set @Region@ to the appropriate AWS region (e.g. 'us-east-1')
-If you did not "follow the recommendend naming scheme":#keep-bucket for either the bucket or role, you'll need to update these parameters as well:
+If "followed the recommendend naming scheme":#keep-bucket for both the bucket and role (or used the provided Terraform script), you're done.
+
+If you did not follow the recommendend naming scheme for either the bucket or role, you'll need to update these parameters as well:
# Set @Bucket@ to the value of "keepstore bucket you created earlier":#keep-bucket
# Set @IAMRole@ to "keepstore role you created earlier":#keep-bucket
Any extra Salt "state" files you add under @local_config_dir/states@ will be added to the Salt run and applied to the hosts.
-h2(#create_a_compute_image). Create a compute image
+h2(#create_a_compute_image). Configure compute nodes
{% include 'branchname' %}
-On cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup. If you will use a HPC scheduler such as SLURM you can skip this section.
+If you will use fixed compute nodes with an HPC scheduler such as SLURM or LSF, you will need to "Set up your compute nodes with Docker":{{site.baseurl}}/install/crunch2/install-compute-node-docker.html or "Set up your compute nodes with Singularity":{{site.baseurl}}/install/crunch2/install-compute-node-singularity.html.
+
+On cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup.
+
+h3. Build the compute image
+
+Follow "the instructions to build a cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the compute image builder script found in @arvados/tools/compute-images@ in your Arvados clone from "step 3":#download.
-*Start by following "the instructions to build a cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images* .
+h3. Configure the compute image
-Once you have that image created, Open @local_config_dir/pillars/arvados.sls@ and edit as follows (AWS specific settings described here, other cloud providers will have similar settings in their respective configuration section):
+Once the image has been created, open @local_config_dir/pillars/arvados.sls@ and edit as follows (AWS specific settings described here, other cloud providers will have similar settings in their respective configuration section):
# In the @arvados.cluster.Containers.CloudVMs@ section:
## Set @ImageID@ to the AMI produced by Packer
At this point you should be able to log into the Arvados cluster. The initial URL will be
-https://workbench.${CLUSTER}.${DOMAIN}
+https://workbench.@${CLUSTER}.${DOMAIN}@
If you did *not* "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
h2(#post_install). After the installation
-As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates. You should retain this repository but be aware that it contains sensitive information (passwords and tokens used by the Arvados services).
+As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates. You should retain this repository but *be aware that it contains sensitive information* (passwords and tokens used by the Arvados services as well as cloud credentials if you used Terraform to create the infrastructure).
As described in "Iterating on config changes":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades. However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.
projects / arvados.git / blobdiff