3 navsection: installguide
4 title: Build a cloud compute node image
7 Copyright (C) The Arvados Authors. All rights reserved.
9 SPDX-License-Identifier: CC-BY-SA-3.0
12 {% include 'notebox_begin_warning' %}
13 @arvados-dispatch-cloud@ is only relevant for cloud installations. Skip this section if you are installing an on premises cluster that will spool jobs to Slurm or LSF.
14 {% include 'notebox_end' %}
16 # "Introduction":#introduction
17 # "Create an SSH keypair":#sshkeypair
18 # "Compute image requirements":#requirements
19 # "The build script":#building
20 # "DNS resolution":#dns-resolution
21 # "Singularity mksquashfs configuration":#singularity_mksquashfs_configuration
22 # "Build an AWS image":#aws
23 ## "Autoscaling compute node scratch space":#aws-ebs-autoscaler
24 # "Build an Azure image":#azure
26 h2(#introduction). Introduction
28 This page describes how to build a compute node image that can be used to run containers dispatched by Arvados in the cloud.
30 Packer templates for AWS and Azure are provided with Arvados. To use them, the following are needed:
32 * "Packer":https://www.packer.io/
33 * credentials for your cloud account
34 * configuration details for your cloud account
37 h2(#sshkeypair). Create a SSH keypair
39 @arvados-dispatch-cloud@ communicates with the compute nodes via SSH. To do this securely, a SSH keypair is needed.
41 Generate a SSH keypair with no passphrase. The private key needs to be stored in the cluster configuration file (see @Containers/DispatchPrivateKey@) for use by @arvados-dispatch-cloud@, as described in the "next section":install-dispatch-cloud.html#update-config. The public key will be baked into the compute node images, see the cloud-specific documentation below.
44 <pre><code>~$ <span class="userinput">ssh-keygen -N '' -f ~/.ssh/id_dispatcher</span>
45 Generating public/private rsa key pair.
46 Your identification has been saved in /home/user/.ssh/id_dispatcher.
47 Your public key has been saved in /home/user/.ssh/id_dispatcher.pub.
48 The key fingerprint is:
50 ~$ <span class="userinput">cat ~/.ssh/id_dispatcher</span>
51 -----BEGIN RSA PRIVATE KEY-----
52 MIIEpQIBAAKCAQEAqXoCzcOBkFQ7w4dvXf9B++1ctgZRqEbgRYL3SstuMV4oawks
53 ttUuxJycDdsPmeYcHsKo8vsEZpN6iYsX6ZZzhkO5nEayUTU8sBjmg1ZCTo4QqKXr
55 oFyAjVoexx0RBcH6BveTfQtJKbktP1qBO4mXo2dP0cacuZEtlAqW9Eb06Pvaw/D9
56 foktmqOY8MyctzFgXBpGTxPliGjqo8OkrOyQP2g+FL7v+Km31Xs61P8=
57 -----END RSA PRIVATE KEY-----
61 h2(#requirements). Compute image requirements
63 Arvados comes with a build script to automate the creation of a suitable compute node image (see "The build script":#building below). It is provided as a convenience. It is also possible to create a compute node image via other means. These are the requirements:
65 * for AWS: the SSH public key for @arvados-dispatch-cloud@ (the one that corresponds with @Containers.DispatchPrivateKey@ in the Arvados config file) needs to go into ~/.ssh/authorized_keys for the SSH user you want @arvados-dispatch-cloud@ to use (cf. @CloudVMs.DriverParameters.AdminUsername@ in the Arvados config file) and that user needs to be able to sudo without password prompt, unless you use `root` in which case sudo is not used.
66 * for Azure: @arvados-dispatch-cloud@ automatically extracts the SSH public key from the value of @Containers.DispatchPrivateKey@ and uses an API call to create the user specified in @CloudVMs.DriverParameters.AdminUsername@ with that SSH public key and password-less sudo enabled.
67 * SSH needs to be running and reachable by @arvados-dispatch-cloud@ on port 22 (or a custom port, see @CloudVMS.SSHPort@ to in the Arvados config file)
68 * the @python3-arvados-fuse@ package needs to be installed
69 * @Docker@ or @Singularity@ needs to be installed (cf. @Containers.RuntimeEngine@ in the Arvados config file).
70 * all available scratch space should be made available under `/tmp`.
72 h2(#building). The build script
74 The necessary files are located in the @arvados/tools/compute-images@ directory in the source tree. A build script is provided to generate the image. The @--help@ argument lists all available options:
76 <notextile><pre><code>~$ <span class="userinput">./build.sh --help</span>
77 build.sh: Build cloud images for arvados-dispatch-cloud
84 --json-file (required)
85 Path to the packer json file
86 --arvados-cluster-id (required)
87 The ID of the Arvados cluster, e.g. zzzzz
88 --aws-profile (default: false)
89 AWS profile to use (valid profile from ~/.aws/config
90 --aws-secrets-file (default: false, required if building for AWS)
91 AWS secrets file which will be sourced from this script
92 --aws-source-ami (default: false, required if building for AWS)
93 The AMI to use as base for building the images
94 --aws-region (default: us-east-1)
95 The AWS region to use for building the images
96 --aws-vpc-id (optional)
97 VPC id for AWS, otherwise packer will pick the default one
99 Subnet id for AWS otherwise packer will pick the default one for the VPC
100 --aws-ebs-autoscale (default: false)
101 Install the AWS EBS autoscaler daemon.
102 --gcp-project-id (default: false, required if building for GCP)
104 --gcp-account-file (default: false, required if building for GCP)
106 --gcp-zone (default: us-central1-f)
108 --azure-secrets-file (default: false, required if building for Azure)
109 Azure secrets file which will be sourced from this script
110 --azure-resource-group (default: false, required if building for Azure)
112 --azure-location (default: false, required if building for Azure)
113 Azure location, e.g. centralus, eastus, westeurope
114 --azure-sku (default: unset, required if building for Azure, e.g. 16.04-LTS)
115 Azure SKU image to use
116 --ssh_user (default: packer)
117 The user packer will use to log into the image
118 --resolver (default: host's network provided)
119 The dns resolver for the machine
120 --reposuffix (default: unset)
121 Set this to "-dev" to track the unstable/dev Arvados repositories
122 --public-key-file (required)
123 Path to the public key file that a-d-c will use to log into the compute node
124 --mksquashfs-mem (default: 256M)
125 Only relevant when using Singularity. This is the amount of memory mksquashfs is allowed to use.
127 Output debug information (default: false)
128 </code></pre></notextile>
130 h2(#dns-resolution). DNS resolution
132 Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should be passed as the value for the @--resolver@ argument to "the build script":#building.
134 Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
136 <notextile><pre><code>10.20.30.40 <span class="userinput">ClusterID.example.com</span>
137 10.20.30.41 <span class="userinput">keep1.ClusterID.example.com</span>
138 10.20.30.42 <span class="userinput">keep2.ClusterID.example.com</span>
139 </code></pre></notextile>
141 Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
143 {% assign show_docker_warning = true %}
145 {% include 'singularity_mksquashfs_configuration' %}
147 The desired amount of memory to make available for @mksquashfs@ can be configured in an argument to "the build script":#building. It defaults to @256M@.
149 h2(#aws). Build an AWS image
151 <notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-aws.json \
152 --arvados-cluster-id ClusterID \
153 --aws-profile AWSProfile \
154 --aws-source-ami AMI \
156 --aws-subnet-id Subnet \
158 --resolver ResolverIP \
159 --public-key-file ArvadosDispatchCloudPublicKeyPath
161 </code></pre></notextile>
163 For @ClusterID@, fill in your cluster ID. The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored. The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
165 @AWSProfile@ should be replaced with the name of an AWS profile with sufficient permissions to create the image.
167 @ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
169 h3(#aws-ebs-autoscaler). Autoscaling compute node scratch space
171 If you want to add the "AWS EBS autoscaler":https://github.com/awslabs/amazon-ebs-autoscale daemon in your images, add the @--aws-ebs-autoscale@ flag to the "the build script":#building. Doing so will make the compute image scratch space scale automatically as needed.
173 The AWS EBS autoscaler daemon will be installed with this configuration:
175 <notextile><pre><code>{
176 "mountpoint": "/tmp",
177 "filesystem": "lvm.ext4",
179 "volume_group": "autoscale_vg",
180 "logical_volume": "autoscale_lv"
187 "detection_interval": 2,
189 "max_ebs_volume_size": 1500,
190 "max_logical_volume_size": 8000,
191 "max_ebs_volume_count": 16
194 "log_file": "/var/log/ebs-autoscale.log",
198 </code></pre></notextile>
200 Changing the configuration is left as an exercise for the reader.
202 Using this feature also requires a few Arvados configuration changes in @config.yml@:
204 * The @Containers/InstanceTypes@ list should be modified so that all @AddedScratch@ lines are removed, and the @IncludedScratch@ value should be set to a (fictional) high number. This way, the scratch space requirements will be met by all the defined instance type. For example:
206 <notextile><pre><code> InstanceTypes:
208 ProviderType: c5.large
211 IncludedScratch: 16TB
214 ProviderType: m5.large
217 IncludedScratch: 16TB
220 </code></pre></notextile>
222 * You will also need to create an IAM role in AWS with these permissions:
224 <notextile><pre><code>{
225 "Version": "2012-10-17",
231 "ec2:DescribeVolumeStatus",
232 "ec2:DescribeVolumes",
234 "ec2:ModifyInstanceAttribute",
235 "ec2:DescribeVolumeAttribute",
244 </code></pre></notextile>
246 Then, in @config.yml@ set @Containers/CloudVMs/DriverParameters/IAMInstanceProfile@ to the name of the IAM role. This will make @arvados-dispatch-cloud@ pass an IAMInstanceProfile to the compute nodes as they start up, giving them sufficient permissions to attach and grow EBS volumes.
248 h2(#azure). Build an Azure image
250 <notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-azure.json \
251 --arvados-cluster-id ClusterID \
252 --azure-resource-group ResourceGroup \
253 --azure-location AzureRegion \
254 --azure-sku AzureSKU \
255 --azure-secrets-file AzureSecretsFilePath \
256 --resolver ResolverIP \
257 --public-key-file ArvadosDispatchCloudPublicKeyPath
259 </code></pre></notextile>
261 For @ClusterID@, fill in your cluster ID. The @ResourceGroup@ and @AzureRegion@ (e.g. 'eastus2') should be configured for where you want the compute image to be generated and stored. The @AzureSKU@ is the SKU of the base image to be used, e.g. '18.04-LTS' for Ubuntu 18.04.
263 @AzureSecretsFilePath@ should be replaced with the path to a shell script that loads the Azure secrets with sufficient permissions to create the image. The file would look like this:
265 <notextile><pre><code>export ARM_CLIENT_ID=...
266 export ARM_CLIENT_SECRET=...
267 export ARM_SUBSCRIPTION_ID=...
268 export ARM_TENANT_ID=...
269 </code></pre></notextile>
271 These secrets can be generated from the Azure portal, or with the cli using a command like this:
273 <notextile><pre><code>~$ <span class="userinput">az ad sp create-for-rbac --name Packer --password ...</span>
274 </code></pre></notextile>
276 @ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.