export ARM_CLIENT_ID=...
-export ARM_CLIENT_SECRET=...
-export ARM_SUBSCRIPTION_ID=...
-export ARM_TENANT_ID=...
+10.20.30.40 ClusterID.example.com
+10.20.30.41 keep1.ClusterID.example.com
+10.20.30.42 keep2.ClusterID.example.com
-These secrets can be generated from the Azure portal, or with the cli using a command like this:
-
-~$ az ad sp create-for-rbac --name Packer --password ...
-
+Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
-@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
+h2(#nvidia). NVIDIA GPU support
-Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should replace the string @ResolverIP@ in the command above.
+If you plan on using instance types with NVIDIA GPUs, add @--nvidia-gpu-support@ to the build command line. Arvados uses the same compute image for both GPU and non-GPU instance types. The GPU tooling is ignored when using the image with a non-GPU instance type.
-Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
+{% assign show_docker_warning = true %}
-10.20.30.40 ClusterID.example.com
-10.20.30.41 keep1.ClusterID.example.com
-10.20.30.42 keep2.ClusterID.example.com
-
+{% include 'singularity_mksquashfs_configuration' %}
-Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
+The desired amount of memory to make available for @mksquashfs@ can be configured in an argument to "the build script":#building. It defaults to @256M@.
h2(#aws). Build an AWS image
+For @ClusterID@, fill in your cluster ID.
+
+@AWSProfile@ is the name of an AWS profile in your "credentials file":https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#shared-credentials-file (@~/.aws/credentials@) listing the @aws_access_key_id@ and @aws_secret_access_key@ to use.
+
+The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
+
+The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored.
+
+@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
+
~$ ./build.sh --json-file arvados-images-aws.json \
--arvados-cluster-id ClusterID \
--aws-profile AWSProfile \
@@ -160,24 +188,74 @@ h2(#aws). Build an AWS image
--aws-vpc-id VPC \
--aws-subnet-id Subnet \
--ssh_user admin \
- --resolver ResolverIP \
--public-key-file ArvadosDispatchCloudPublicKeyPath
-For @ClusterID@, fill in your cluster ID. The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored. The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
-@AWSProfile@ should be replaced with the name of an AWS profile with sufficient permissions to create the image.
+h3(#aws-ebs-autoscaler). Autoscaling compute node scratch space
+
+Arvados supports "AWS EBS autoscaler":https://github.com/awslabs/amazon-ebs-autoscale. This feature automatically expands the scratch space on the compute node on demand by 200 GB at a time, up to 5 TB.
+
+If you want to add the daemon in your images, add the @--aws-ebs-autoscale@ flag to the "the build script":#building.
+
+The AWS EBS autoscaler daemon will be installed with this configuration:
+
+{
+ "mountpoint": "/tmp",
+ "filesystem": "lvm.ext4",
+ "lvm": {
+ "volume_group": "autoscale_vg",
+ "logical_volume": "autoscale_lv"
+ },
+ "volume": {
+ "type": "gp3",
+ "iops": 3000,
+ "encrypted": 1
+ },
+ "detection_interval": 2,
+ "limits": {
+ "max_ebs_volume_size": 1500,
+ "max_logical_volume_size": 8000,
+ "max_ebs_volume_count": 16
+ },
+ "logging": {
+ "log_file": "/var/log/ebs-autoscale.log",
+ "log_interval": 300
+ }
+}
+
-@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
+Changing the ebs-autoscale configuration is left as an exercise for the reader.
-Compute nodes must be able to resolve the hostnames of the API server and any keepstore servers to your internal IP addresses. You can do this by running an internal DNS resolver. The IP address of the resolver should replace the string @ResolverIP@ in the command above.
+This feature also requires a few Arvados configuration changes, described in "EBS Autoscale configuration":install-dispatch-cloud.html#aws-ebs-autoscaler.
-Alternatively, the services could be hardcoded into an @/etc/hosts@ file. For example:
+h2(#azure). Build an Azure image
-10.20.30.40 ClusterID.example.com
-10.20.30.41 keep1.ClusterID.example.com
-10.20.30.42 keep2.ClusterID.example.com
+~$ ./build.sh --json-file arvados-images-azure.json \
+ --arvados-cluster-id ClusterID \
+ --azure-resource-group ResourceGroup \
+ --azure-location AzureRegion \
+ --azure-sku AzureSKU \
+ --azure-secrets-file AzureSecretsFilePath \
+ --resolver ResolverIP \
+ --public-key-file ArvadosDispatchCloudPublicKeyPath
+
-Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
+For @ClusterID@, fill in your cluster ID. The @ResourceGroup@ and @AzureRegion@ (e.g. 'eastus2') should be configured for where you want the compute image to be generated and stored. The @AzureSKU@ is the SKU of the base image to be used, e.g. '18.04-LTS' for Ubuntu 18.04.
+
+@AzureSecretsFilePath@ should be replaced with the path to a shell script that loads the Azure secrets with sufficient permissions to create the image. The file would look like this:
+
+export ARM_CLIENT_ID=...
+export ARM_CLIENT_SECRET=...
+export ARM_SUBSCRIPTION_ID=...
+export ARM_TENANT_ID=...
+
+
+These secrets can be generated from the Azure portal, or with the cli using a command like this:
+
+~$ az ad sp create-for-rbac --name Packer --password ...
+
+
+@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.