h3. Configure CloudVMs
-Add or update the following portions of your cluster configuration file, @config.yml@. Refer to "config.defaults.yml":{{site.baseurl}}/admin/config.html for information about additional configuration options. The @DispatchPrivateKey@ should be the *private* key generated in "the previous section":install-compute-node.html#sshkeypair.
+Add or update the following portions of your cluster configuration file, @config.yml@. Refer to "config.defaults.yml":{{site.baseurl}}/admin/config.html for information about additional configuration options. The @DispatchPrivateKey@ should be the *private* key generated in "Create a SSH keypair":install-compute-node.html#sshkeypair .
<notextile>
<pre><code> Services:
h3(#GPUsupport). NVIDIA GPU support
-To specify instance types with NVIDIA GPUs, you must include an additional @CUDA@ section:
+To specify instance types with NVIDIA GPUs, "the compute image must be built with CUDA support":install-compute-node.html#nvidia , and you must include an additional @CUDA@ section:
<notextile>
<pre><code> InstanceTypes:
The @DriverVersion@ is the version of the CUDA toolkit installed in your compute image (in X.Y format, do not include the patchlevel). The @HardwareCapability@ is the CUDA compute capability of the GPUs available for this instance type. The @DeviceCount@ is the number of GPU cores available for this instance type.
+h3(#aws-ebs-autoscaler). EBS Autoscale configuration
+
+See "Autoscaling compute node scratch space":install-compute-node.html#aws-ebs-autoscaler for details about compute image configuration.
+
+The @Containers.InstanceTypes@ list should be modified so that all @AddedScratch@ lines are removed, and the @IncludedScratch@ value should be set to 5 TB. This way, the scratch space requirements will be met by all the defined instance type. For example:
+
+<notextile><pre><code> InstanceTypes:
+ c5large:
+ ProviderType: c5.large
+ VCPUs: 2
+ RAM: 4GiB
+ IncludedScratch: 5TB
+ Price: 0.085
+ m5large:
+ ProviderType: m5.large
+ VCPUs: 2
+ RAM: 8GiB
+ IncludedScratch: 5TB
+ Price: 0.096
+...
+</code></pre></notextile>
+
+You will also need to create an IAM role in AWS with these permissions:
+
+<notextile><pre><code>{
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "ec2:AttachVolume",
+ "ec2:DescribeVolumeStatus",
+ "ec2:DescribeVolumes",
+ "ec2:DescribeTags",
+ "ec2:ModifyInstanceAttribute",
+ "ec2:DescribeVolumeAttribute",
+ "ec2:CreateVolume",
+ "ec2:DeleteVolume",
+ "ec2:CreateTags"
+ ],
+ "Resource": "*"
+ }
+ ]
+}
+</code></pre></notextile>
+
+Then set @Containers.CloudVMs.DriverParameters.IAMInstanceProfile@ to the name of the IAM role. This will make @arvados-dispatch-cloud@ pass an IAM instance profile to the compute nodes when they start up, giving them sufficient permissions to attach and grow EBS volumes.
+
h3. AWS Credentials for Local Keepstore on Compute node
-When @Containers/LocalKeepBlobBuffersPerVCPU@ is non-zero, the compute node will spin up a local Keepstore service for faster storage access. If Keep is backed by S3, the compute node will need to be able to access the S3 bucket.
+When @Containers.LocalKeepBlobBuffersPerVCPU@ is non-zero, the compute node will spin up a local Keepstore service for direct storage access. If Keep is backed by S3, the compute node will need to be able to access the S3 bucket.
+
+If the AWS credentials for S3 access are configured in @config.yml@ (i.e. @Volumes.DriverParameters.AccessKeyID@ and @Volumes.DriverParameters.SecretAccessKey@), these credentials will be made available to the local Keepstore on the compute node to access S3 directly and no further configuration is necessary.
-If the AWS credentials for S3 access are configured in @config.yml@ (i.e. @Volumes/DriverParameters/AccessKeyID@ and @Volumes/DriverParameters/SecretAccessKey@), these credentials will be made available to the local Keepstore on the compute node to access S3 directly and no further configuration is necessary.
+Alternatively, if an IAM role is configured in @config.yml@ (i.e. @Volumes.DriverParameters.IAMRole@), the name of an instance profile that corresponds to this role ("often identical to the name of the IAM role":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#ec2-instance-profile) must be configured in the @CloudVMs.DriverParameters.IAMInstanceProfile@ parameter.
-Alternatively, if an IAM role is configured in @config.yml@ (i.e. @Volumes/DriverParameters/IAMRole@), the name of an instance profile that corresponds to this role ("often identical to the name of the IAM role":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#ec2-instance-profile) must be configured in the @CloudVMs/DriverParameters/IAMInstanceProfile@ parameter.
+*If you are also using EBS Autoscale feature, the role in IAMInstanceProfile must have both ec2 and s3 permissions.*
-Finally, if @config.yml@ does not have @Volumes/DriverParameters/AccessKeyID@, @Volumes/DriverParameters/SecretAccessKey@ or @Volumes/DriverParameters/IAMRole@ defined, Keepstore uses the IAM role attached to the node, whatever it may be called. The @CloudVMs/DriverParameters/IAMInstanceProfile@ parameter must then still be configured with the name of a profile whose IAM role has permission to access the S3 bucket(s). That way, @arvados-dispatch-cloud@ can attach the IAM role to the compute node as it is created.
+Finally, if @config.yml@ does not have @Volumes.DriverParameters.AccessKeyID@, @Volumes.DriverParameters.SecretAccessKey@ or @Volumes.DriverParameters.IAMRole@ defined, Keepstore uses the IAM role attached to the node, whatever it may be called. The @CloudVMs.DriverParameters.IAMInstanceProfile@ parameter must then still be configured with the name of a profile whose IAM role has permission to access the S3 bucket(s). That way, @arvados-dispatch-cloud@ can attach the IAM role to the compute node as it is created.
h3. Minimal configuration example for Amazon EC2
</code></pre>
</notextile>
-h3(#IAM). Example IAM policy
+h3(#IAM). Example IAM policy for cloud dispatcher
Example policy for the IAM role used by the cloud dispatcher:
<notextile>
<pre>
{
- "Version": "2012-10-17",
"Id": "arvados-dispatch-cloud policy",
"Statement": [
{
# "Introduction":#introduction
# "Prerequisites and planning":#prerequisites
-# "Hosts":#hosts
+# "Required hosts":#hosts
# "Download the installer":#download
# "Initialize the installer":#copy_config
# "Edit local.params":#localparams
# "Create a compute image":#create_a_compute_image
# "Further customization of the installation":#further_customization
# "Begin installation":#installation
-## "Run diagnostics to confirming the cluster is working":#test-install
+# "Confirm the cluster is working":#test-install
## "Debugging issues":#debugging
## "Iterating on config changes":#iterating
## "Common problems and solutions":#common-problems
# You should set up a "security group which allows SSH access (port 22)":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html
# Make sure to add a "VPC S3 endpoint":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
-h3. S3 Bucket (AWS specific)
+h3(#keep-bucket). S3 Bucket (AWS specific)
We recommend "creating an S3 bucket":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@
-Then create an IAM role called @${CLUSTER}-keepstore-00-iam-role@ which has "permission to read and write the bucket":https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html
-
-h3. Other IAM Roles (AWS specific)
-
+Then create an IAM role called @${CLUSTER}-keepstore-00-iam-role@ which has "permission to read and write the bucket":https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html . Here is an example policy:
+<notextile>
+<pre>
+{
+ "Id": "arvados-keepstore policy",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:*"
+ ],
+ "Resource": "arn:aws:s3:::xarv1-nyw5e-000000000000000-volume"
+ }
+ ]
+}
+</pre>
+</notextile>
-h2(#hosts). Hosts
+h2(#hosts). Required hosts
You will need to allocate several hosts (physical or virtual machines) for the fixed infrastructure of the Arvados cluster. These machines should have at least 2 cores and 8 GiB of RAM, running a supported Linux distribution.
{% include 'supportedlinux' %}
-Allocate these as appropriate for your site. On AWS you may choose to do it manually with the AWS console, or using a DevOps tool such as CloudFormation or Terraform.
+Allocate the following hosts as appropriate for your site. On AWS you may choose to do it manually with the AWS console, or using a DevOps tool such as CloudFormation or Terraform.
The installer will set up the Arvados services on your machines. Here is the default assignment of services to machines:
## @webshell.${CLUSTER}.${DOMAIN}@
## @shell.${CLUSTER}.${DOMAIN}@
-(AWS specific) The machine that runs the arvados cloud dispatcher will need an "IAM role that allows it to create EC2 instances, see here for details .":https://doc.arvados.org/v2.4/install/crunch2-cloud/install-dispatch-cloud.html#IAM
+(AWS specific) The machine that runs the arvados cloud dispatcher will need an "IAM role that allows it to create EC2 instances, see here for details .":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html
-If your infrastructure differs from the setup proposed above (ie, different hostnames, or using AWS RDS or an existing DB server), you can still use the installer, but additional customization will be necessary.
+If your infrastructure differs from the setup proposed above (ie, different hostnames, or using an external DB server such as AWS RDS), you can still use the installer, but "additional customization may be necessary":#further_customization .
h2(#download). Download the installer
h2(#keep). Configure Keep storage
-The @multi_host/aws@ template uses S3 for storage. Arvados also supports "filesystem storage":configure-fs-storage.html and "Azure blob storage":configure-azure-blob-storage.html . Keep storage configuration can be found in @local_config_dir/pillars/arvados.sls@ in the section @arvados.cluster.Volumes@.
+The @multi_host/aws@ template uses S3 for storage. Arvados also supports "filesystem storage":configure-fs-storage.html and "Azure blob storage":configure-azure-blob-storage.html . Keep storage configuration can be found in in the section @arvados.cluster.Volumes@ of @local_config_dir/pillars/arvados.sls@.
h3. Object storage in S3 (AWS Specific)
Open @local_config_dir/pillars/arvados.sls@ and edit as follows:
# In the @arvados.cluster.Volumes@ section, set @Region@ to the appropriate AWS region (e.g. 'us-east-1')
-# Set @IAMRole@ to the name of the `KeepstoreRole` generated by CloudFormation. Just use the part after the '/' (not the arn:aws:iam.... stuff at the beginning).
-# Set @Bucket@ to the value of `KeepBucket1`
+# Set @Bucket@ to the value of "keepstore role you created earlier":#keep-bucket
+# Set @IAMRole@ to "keepstore role you created earlier":#keep-bucket
{% include 'ssl_config_multi' %}
{% include 'branchname' %}
-On cloud installations, containers are dispatched in Docker daemons running in the <i>compute instances</i>, which need some special setup. Follow "the instructions build a cloud compute node image":https://doc.arvados.org/install/crunch2-cloud/install-compute-node.html using the "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images .
+On cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup.
+
+*Start by following "the instructions build a cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images* .
Once you have that image created, Open @local_config_dir/pillars/arvados.sls@ and edit as follows (AWS specific settings described here, configuration for Azure is similar):
# In the @arvados.cluster.Containers.CloudVMs@ section:
-## Set @ImageID@ to the AMI output from Packer
+## Set @ImageID@ to the AMI produced by Packer
## Set @Region@ to the appropriate AWS region
## Set @AdminUsername@ to the admin user account on the image
## Set the @SecurityGroupIDs@ list to the VPC security group which you set up to allow SSH connections to these nodes
## Set @SubnetID@ to the value of SubnetId of your VPC
# Update @arvados.cluster.Containers.DispatchPrivateKey@ and paste the contents of the @~/.ssh/id_dispatcher@ file you generated in an earlier step.
-# Update @arvados.cluster.InstanceTypes@ as necessary. If t3 and m5/c5 node types are not available, replace them with t2 and m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.
+# Update @arvados.cluster.InstanceTypes@ as necessary. If m5/c5 node types are not available, replace them with m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.
-h2(#further_customization). Further customization of the installation
+h2(#further_customization). Further customization of the installation (optional)
If you are installing on AWS and following the naming conventions recommend in this guide, then likely no further configuration is necessary and you can begin installation.
-If your infrastructure differs from the setup proposed above (ie, using AWS RDS or an existing DB server), you can still use the installer, but additional customization will be necessary.
+A couple of common customizations are described here. Other changes may require editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ has the template used to produce the Arvados configuration file that is distributed to all the nodes.
+
+Any extra salt _state_ files you add under @local_config_dir/states@ will be added to the salt run and applied to the hosts.
-This is done by editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ has the template used to produce the Arvados configuration file that is distributed to all the nodes.
+h3(#authentication). Using a different authentication provider
-Any extra salt <i>state</i> file you add under @local_config_dir/states@ will be added to the salt run and applied to the hosts.
+By default, the installer will use the "Test" provider, which is a list of usernames and cleartext passwords stored in the Arvados config file. *This is low security configuration and you are strongly advised to configure one of the other "supported authentication methods":setup-login.html* .
+
+h3(#ext-database). Using an external database (optional)
+
+Arvados requires a database that is compatible with PostgreSQL 9.5 or later.
+
+For example, Arvados is known to work with Amazon Aurora (note: even idle, Arvados constantly accesses the database, so we strongly advise using "provisioned" mode).
+
+# In @local.params@, remove 'database' from the list of roles assigned to the controller node:
+<pre><code>NODES=(
+ [controller.${CLUSTER}.${DOMAIN}]=api,controller,websocket,dispatcher,keepbalance
+ ...
+)
+</code></pre>
+# In @local.params@, set @DATABASE_INT_IP@ to the database endpoint (can be a hostname, does not have to be an IP address).
+<pre><code>DATABASE_INT_IP=...
+</code></pre>
+# In @local.params@, set @DATABASE_PASSWORD@ to the correct value. "See the previous section describing correct quoting":#localparams
+# In @local_config_dir/pillars/arvados.sls@ you may need to adjust the database name and user. This can be found in the section @arvados.cluster.database@.
h2(#installation). Begin installation
At this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.
-Run this in ~/arvados-setup-xarv1:
+Run this in @~/arvados-setup-xarv1@:
<pre>
./installer.sh deploy
This will deploy all the nodes. It will take a while and produce a lot of logging. If it runs into an error, it will stop.
-h3(#test-install). Run diagnostics to confirming the cluster is working
+{% include 'install_ca_cert' %}
+
+h2(#test-install). Confirm the cluster is working
When everything has finished, you can run the diagnostics.
Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.
-You are probably an "internal client" if you are running the diagnostics from one of the Arvados machines inside the VPC.
+If you are running the diagnostics from one of the Arvados machines inside the VPC, you want @-internal-client@ .
You are an "external client" if you running the diagnostics from your workstation outside of the VPC.
h3(#debugging). Debugging issues
-Most service logs go to @/var/log/syslog@
+Most service logs go to @/var/log/syslog@.
The logs for Rails API server and for Workbench can be found in
on the appropriate instances.
-Workbench2 is a client-side Javascript application, if it having trouble loading, check the browser's developer console.
+Workbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in "Tools → Developer Tools").
h3(#iterating). Iterating on config changes
h3(#common-problems). Common problems and solutions
-h4. Missing ENA support (AWS Specific)
-
-If the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start. You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@. The solution is to build a new AMI with --aws-ena-support true
-
h4. PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
The arvados-api-server package sets up the database as a post-install script. If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.
4. Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.
-{% include 'install_ca_cert' %}
+h4. Missing ENA support (AWS Specific)
+
+If the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start. You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@. The solution is to build a new AMI with --aws-ena-support true
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster. The initial URL will be
-* https://workbench.${CLUSTER}.${DOMAIN}
-
-By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+https://workbench.${CLUSTER}.${DOMAIN}
-Assuming you didn't change these values in the @local.params@ file, the initial credentials are:
+If you did not "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
-* User: 'admin'
-* Password: 'password'
-* Email: 'admin@${CLUSTER}.${DOMAIN}'
+If you did configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
h2(#post_install). After the installation
When deploying upgrades, consult the "Arvados upgrade notes":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@.
-See "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
+See also "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
projects / arvados.git / commitdiff