{% endcomment %}
{% include 'notebox_begin_warning' %}
-arvados-dispatch-cloud is only relevant for cloud installations. Skip this section if you are installing an on premises cluster that will spool jobs to Slurm.
+@arvados-dispatch-cloud@ is only relevant for cloud installations. Skip this section if you are installing an on premises cluster that will spool jobs to Slurm or LSF.
{% include 'notebox_end' %}
# "Introduction":#introduction
+# "Install Packer":#install-packer
# "Create an SSH keypair":#sshkeypair
+# "Compute image requirements":#requirements
# "The build script":#building
-# "Build an Azure image":#azure
+# "DNS resolution":#dns-resolution
+# "NVIDIA GPU support":#nvidia
+# "Singularity mksquashfs configuration":#singularity_mksquashfs_configuration
# "Build an AWS image":#aws
+## "Autoscaling compute node scratch space":#aws-ebs-autoscaler
+# "Build an Azure image":#azure
h2(#introduction). Introduction
* credentials for your cloud account
* configuration details for your cloud account
+h2(#install-packer). Install Packer
+
+"Download Packer here":https://developer.hashicorp.com/packer/downloads
h2(#sshkeypair). Create a SSH keypair
</code></pre>
</notextile>
+h2(#requirements). Compute image requirements
+
+Arvados comes with a build script to automate the creation of a suitable compute node image (see "The build script":#building below). It is provided as a convenience. It is also possible to create a compute node image via other means. These are the requirements:
+
+* for AWS: the SSH public key for @arvados-dispatch-cloud@ (the one that corresponds with @Containers.DispatchPrivateKey@ in the Arvados config file) needs to go into ~/.ssh/authorized_keys for the SSH user you want @arvados-dispatch-cloud@ to use (cf. @CloudVMs.DriverParameters.AdminUsername@ in the Arvados config file) and that user needs to be able to sudo without password prompt, unless you use `root` in which case sudo is not used.
+* for Azure: @arvados-dispatch-cloud@ automatically extracts the SSH public key from the value of @Containers.DispatchPrivateKey@ and uses an API call to create the user specified in @CloudVMs.DriverParameters.AdminUsername@ with that SSH public key and password-less sudo enabled.
+* SSH needs to be running and reachable by @arvados-dispatch-cloud@ on port 22 (or a custom port, see @CloudVMS.SSHPort@ to in the Arvados config file)
+* the @python3-arvados-fuse@ package needs to be installed
+* @Docker@ or @Singularity@ needs to be installed (cf. @Containers.RuntimeEngine@ in the Arvados config file).
+* all available scratch space should be made available under `/tmp`.
+
h2(#building). The build script
The necessary files are located in the @arvados/tools/compute-images@ directory in the source tree. A build script is provided to generate the image. The @--help@ argument lists all available options:
Options:
- --json-file (required)
- Path to the packer json file
- --arvados-cluster-id (required)
- The ID of the Arvados cluster, e.g. zzzzz
- --aws-profile (default: false)
- AWS profile to use (valid profile from ~/.aws/config
- --aws-secrets-file (default: false, required if building for AWS)
- AWS secrets file which will be sourced from this script
- --aws-source-ami (default: false, required if building for AWS)
- The AMI to use as base for building the images
- --aws-region (default: us-east-1)
+ --json-file <path>
+ Path to the packer json file (required)
+ --arvados-cluster-id <xxxxx>
+ The ID of the Arvados cluster, e.g. zzzzz(required)
+ --aws-profile <profile>
+ AWS profile to use (valid profile from ~/.aws/config (optional)
+ --aws-secrets-file <path>
+ AWS secrets file which will be sourced from this script (optional)
+ When building for AWS, either an AWS profile or an AWS secrets file
+ must be provided.
+ --aws-source-ami <ami-xxxxxxxxxxxxxxxxx>
+ The AMI to use as base for building the images (required if building for AWS)
+ --aws-region <region> (default: us-east-1)
The AWS region to use for building the images
- --aws-vpc-id (optional)
- VPC id for AWS, otherwise packer will pick the default one
- --aws-subnet-id
- Subnet id for AWS otherwise packer will pick the default one for the VPC
- --gcp-project-id (default: false, required if building for GCP)
- GCP project id
- --gcp-account-file (default: false, required if building for GCP)
- GCP account file
- --gcp-zone (default: us-central1-f)
+ --aws-vpc-id <vpc-id>
+ VPC id for AWS, if not specified packer will derive from the subnet id or pick the default one.
+ --aws-subnet-id <subnet-xxxxxxxxxxxxxxxxx>
+ Subnet id for AWS, if not specified packer will pick the default one for the VPC.
+ --aws-ebs-autoscale
+ Install the AWS EBS autoscaler daemon (default: do not install the AWS EBS autoscaler).
+ --aws-associate-public-ip <true|false>
+ Associate a public IP address with the node used for building the compute image.
+ Required when the machine running packer can not reach the node used for building
+ the compute image via its private IP. (default: true if building for AWS)
+ Note: if the subnet has "Auto-assign public IPv4 address" enabled, disabling this
+ flag will have no effect.
+ --aws-ena-support <true|false>
+ Enable enhanced networking (default: true if building for AWS)
+ --gcp-project-id <project-id>
+ GCP project id (required if building for GCP)
+ --gcp-account-file <path>
+ GCP account file (required if building for GCP)
+ --gcp-zone <zone> (default: us-central1-f)
GCP zone
- --azure-secrets-file (default: false, required if building for Azure)
- Azure secrets file which will be sourced from this script
- --azure-resource-group (default: false, required if building for Azure)
- Azure resource group
- --azure-location (default: false, required if building for Azure)
- Azure location, e.g. centralus, eastus, westeurope
- --azure-sku (default: unset, required if building for Azure, e.g. 16.04-LTS)
+ --azure-secrets-file <patch>
+ Azure secrets file which will be sourced from this script (required if building for Azure)
+ --azure-resource-group <resouce-group>
+ Azure resource group (required if building for Azure)
+ --azure-location <location>
+ Azure location, e.g. centralus, eastus, westeurope (required if building for Azure)
+ --azure-sku <sku> (required if building for Azure, e.g. 16.04-LTS)
Azure SKU image to use
- --ssh_user (default: packer)
- The user packer will use lo log into the image
- --domain (default: arvadosapi.com)
- The domain part of the FQDN for the cluster
- --resolver (default: 8.8.8.8)
- The dns resolver for the machine
- --reposuffix (default: unset)
+ --ssh_user <user> (default: packer)
+ The user packer will use to log into the image
+ --resolver <resolver_IP>
+ The dns resolver for the machine (default: host's network provided)
+ --reposuffix <suffix>
Set this to "-dev" to track the unstable/dev Arvados repositories
- --public-key-file (required)
- Path to the public key file that a-d-c will use to log into the compute node
+ --public-key-file <path>
+ Path to the public key file that a-d-c will use to log into the compute node (required)
+ --mksquashfs-mem (default: 256M)
+ Only relevant when using Singularity. This is the amount of memory mksquashfs is allowed to use.
+ --nvidia-gpu-support
+ Install all the necessary tooling for Nvidia GPU support (default: do not install Nvidia GPU support)
--debug
- Output debug information (default: false)
+ Output debug information (default: no debug output is printed)
</code></pre></notextile>
-h2(#azure). Build an Azure image
+h2(#dns-resolution). DNS resolution
-<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-azure.json \
- --arvados-cluster-id ClusterID \
- --azure-resource-group ResourceGroup \
- --azure-location AzureRegion \
- --azure-sku AzureSKU \
- --azure-secrets-file AzureSecretsFilePath \
- --resolver ResolverIP \
- --public-key-file ArvadosDispatchCloudPublicKeyPath
-</span>
-</code></pre></notextile>
+Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. If you are on AWS and using Route 53 for your DNS, the default resolver configuration can be used with no extra options.
-For @ClusterID@, fill in your cluster ID. The @ResourceGroup@ and @AzureRegion@ (e.g. 'eastus2') should be configured for where you want the compute image to be generated and stored. The @AzureSKU@ is the SKU of the base image to be used, e.g. '18.04-LTS' for Ubuntu 18.04.
+You can also run your own internal DNS resolver. In that case, the IP address of the resolver should be passed as the value for the @--resolver@ argument to "the build script":#building.
-@AzureSecretsFilePath@ should be replaced with the path to a shell script that loads the Azure secrets with sufficient permissions to create the image. The file would look like this:
+As a third option, the services could be hardcoded into an @/etc/hosts@ file. For example:
-<notextile><pre><code>export ARM_CLIENT_ID=...
-export ARM_CLIENT_SECRET=...
-export ARM_SUBSCRIPTION_ID=...
-export ARM_TENANT_ID=...
+<notextile><pre><code>10.20.30.40 <span class="userinput">ClusterID.example.com</span>
+10.20.30.41 <span class="userinput">keep1.ClusterID.example.com</span>
+10.20.30.42 <span class="userinput">keep2.ClusterID.example.com</span>
</code></pre></notextile>
-These secrets can be generated from the Azure portal, or with the cli using a command like this:
-
-<notextile><pre><code>~$ <span class="userinput">az ad sp create-for-rbac --name Packer --password ...</span>
-</code></pre></notextile>
+Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
-@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
+h2(#nvidia). NVIDIA GPU support
-Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should replace the string @ResolverIP@ in the command above.
+If you plan on using instance types with NVIDIA GPUs, add @--nvidia-gpu-support@ to the build command line. Arvados uses the same compute image for both GPU and non-GPU instance types. The GPU tooling is ignored when using the image with a non-GPU instance type.
-Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
+{% assign show_docker_warning = true %}
-<notextile><pre><code>10.20.30.40 <span class="userinput">ClusterID.example.com</span>
-10.20.30.41 <span class="userinput">keep1.ClusterID.example.com</span>
-10.20.30.42 <span class="userinput">keep2.ClusterID.example.com</span>
-</code></pre></notextile>
+{% include 'singularity_mksquashfs_configuration' %}
-Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
+The desired amount of memory to make available for @mksquashfs@ can be configured in an argument to "the build script":#building. It defaults to @256M@.
h2(#aws). Build an AWS image
+For @ClusterID@, fill in your cluster ID.
+
+@AWSProfile@ is the name of an AWS profile in your "credentials file":https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#shared-credentials-file (@~/.aws/credentials@) listing the @aws_access_key_id@ and @aws_secret_access_key@ to use.
+
+The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
+
+The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored.
+
+@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
+
<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-aws.json \
--arvados-cluster-id ClusterID \
--aws-profile AWSProfile \
--aws-vpc-id VPC \
--aws-subnet-id Subnet \
--ssh_user admin \
- --resolver ResolverIP \
--public-key-file ArvadosDispatchCloudPublicKeyPath
</span>
</code></pre></notextile>
-For @ClusterID@, fill in your cluster ID. The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored. The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
-@AWSProfile@ should be replaced with the name of an AWS profile with sufficient permissions to create the image.
+h3(#aws-ebs-autoscaler). Autoscaling compute node scratch space
+
+Arvados supports "AWS EBS autoscaler":https://github.com/awslabs/amazon-ebs-autoscale. This feature automatically expands the scratch space on the compute node on demand by 200 GB at a time, up to 5 TB.
+
+If you want to add the daemon in your images, add the @--aws-ebs-autoscale@ flag to the "the build script":#building.
+
+The AWS EBS autoscaler daemon will be installed with this configuration:
+
+<notextile><pre><code>{
+ "mountpoint": "/tmp",
+ "filesystem": "lvm.ext4",
+ "lvm": {
+ "volume_group": "autoscale_vg",
+ "logical_volume": "autoscale_lv"
+ },
+ "volume": {
+ "type": "gp3",
+ "iops": 3000,
+ "encrypted": 1
+ },
+ "detection_interval": 2,
+ "limits": {
+ "max_ebs_volume_size": 1500,
+ "max_logical_volume_size": 8000,
+ "max_ebs_volume_count": 16
+ },
+ "logging": {
+ "log_file": "/var/log/ebs-autoscale.log",
+ "log_interval": 300
+ }
+}
+</code></pre></notextile>
-@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
+Changing the ebs-autoscale configuration is left as an exercise for the reader.
-Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should replace the string @ResolverIP@ in the command above.
+This feature also requires a few Arvados configuration changes, described in "EBS Autoscale configuration":install-dispatch-cloud.html#aws-ebs-autoscaler.
-Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
+h2(#azure). Build an Azure image
-<notextile><pre><code>10.20.30.40 <span class="userinput">ClusterID.example.com</span>
-10.20.30.41 <span class="userinput">keep1.ClusterID.example.com</span>
-10.20.30.42 <span class="userinput">keep2.ClusterID.example.com</span>
+<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-azure.json \
+ --arvados-cluster-id ClusterID \
+ --azure-resource-group ResourceGroup \
+ --azure-location AzureRegion \
+ --azure-sku AzureSKU \
+ --azure-secrets-file AzureSecretsFilePath \
+ --resolver ResolverIP \
+ --public-key-file ArvadosDispatchCloudPublicKeyPath
+</span>
</code></pre></notextile>
-Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
+For @ClusterID@, fill in your cluster ID. The @ResourceGroup@ and @AzureRegion@ (e.g. 'eastus2') should be configured for where you want the compute image to be generated and stored. The @AzureSKU@ is the SKU of the base image to be used, e.g. '18.04-LTS' for Ubuntu 18.04.
+
+@AzureSecretsFilePath@ should be replaced with the path to a shell script that loads the Azure secrets with sufficient permissions to create the image. The file would look like this:
+
+<notextile><pre><code>export ARM_CLIENT_ID=...
+export ARM_CLIENT_SECRET=...
+export ARM_SUBSCRIPTION_ID=...
+export ARM_TENANT_ID=...
+</code></pre></notextile>
+
+These secrets can be generated from the Azure portal, or with the cli using a command like this:
+
+<notextile><pre><code>~$ <span class="userinput">az ad sp create-for-rbac --name Packer --password ...</span>
+</code></pre></notextile>
+
+@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
projects / arvados.git / blobdiff