From: Peter Amstutz <peter.amstutz@curii.com>
Date: Tue, 26 Apr 2022 13:28:43 +0000 (-0400)
Subject: Merge branch '18835-uploadfiles-test' refs #18835
X-Git-Tag: 2.5.0~193
X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/95252e3481369d48d26c71dabc4475d12a805b3e?hp=1dd56e127c9460713f10996a5ff8d4ddf6244b29

Merge branch '18835-uploadfiles-test' refs #18835

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
---

diff --git a/doc/install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid b/doc/install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid
index 0ed7a599fc..2a7e105905 100644
--- a/doc/install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid
+++ b/doc/install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid
@@ -74,7 +74,7 @@ Add or update the following portions of your cluster configuration file, @config
 </code></pre>
 </notextile>
 
-h4(#GPUsupport). NVIDIA GPU support
+h3(#GPUsupport). NVIDIA GPU support
 
 To specify instance types with NVIDIA GPUs, you must include an additional @CUDA@ section:
 
@@ -95,7 +95,17 @@ To specify instance types with NVIDIA GPUs, you must include an additional @CUDA
 
 The @DriverVersion@ is the version of the CUDA toolkit installed in your compute image (in X.Y format, do not include the patchlevel).  The @HardwareCapability@ is the CUDA compute capability of the GPUs available for this instance type.  The @DeviceCount@ is the number of GPU cores available for this instance type.
 
-h4. Minimal configuration example for Amazon EC2
+h3. AWS Credentials for Local Keepstore on Compute node
+
+When @Containers/LocalKeepBlobBuffersPerVCPU@ is non-zero, the compute node will spin up a local Keepstore service for faster storage access. If Keep is backed by S3, the compute node will need to be able to access the S3 bucket.
+
+If the AWS credentials for S3 access are configured in @config.yml@ (i.e. @Volumes/DriverParameters/AccessKeyID@ and @Volumes/DriverParameters/SecretAccessKey@), these credentials will be made available to the local Keepstore on the compute node to access S3 directly and no further configuration is necessary.
+
+Alternatively, if an IAM role is configured in @config.yml@ (i.e. @Volumes/DriverParameters/IAMRole@), the name of an instance profile that corresponds to this role ("often identical to the name of the IAM role":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#ec2-instance-profile) must be configured in the @CloudVMs/DriverParameters/IAMInstanceProfile@ parameter.
+
+Finally, if @config.yml@ does not have @Volumes/DriverParameters/AccessKeyID@, @Volumes/DriverParameters/SecretAccessKey@ or @Volumes/DriverParameters/IAMRole@ defined, Keepstore uses the IAM role attached to the node, whatever it may be called. The @CloudVMs/DriverParameters/IAMInstanceProfile@ parameter must then still be configured with the name of a profile whose IAM role has permission to access the S3 bucket(s). That way, @arvados-dispatch-cloud@ can attach the IAM role to the compute node as it is created.
+
+h3. Minimal configuration example for Amazon EC2
 
 The <span class="userinput">ImageID</span> value is the compute node image that was built in "the previous section":install-compute-node.html#aws.
 
@@ -146,7 +156,7 @@ Example policy for the IAM role used by the cloud dispatcher:
 </pre>
 </notextile>
 
-h4. Minimal configuration example for Azure
+h3. Minimal configuration example for Azure
 
 Using managed disks:
 
diff --git a/tools/salt-install/config_examples/multi_host/aws/pillars/arvados.sls b/tools/salt-install/config_examples/multi_host/aws/pillars/arvados.sls
index 2e85be7905..f41b6ac5b3 100644
--- a/tools/salt-install/config_examples/multi_host/aws/pillars/arvados.sls
+++ b/tools/salt-install/config_examples/multi_host/aws/pillars/arvados.sls
@@ -127,11 +127,12 @@ arvados:
         Driver: ec2
         DriverParameters:
           Region: FIXME
-          EBSVolumeType: gp2
+          EBSVolumeType: gp3
           AdminUsername: FIXME
           ### This SG should allow SSH from the dispatcher to the compute nodes
           SecurityGroupIDs: ['sg-FIXMEFIXMEFIXMEFI']
           SubnetID: subnet-FIXMEFIXMEFIXMEFI
+          IAMInstanceProfile: __CLUSTER__-keepstore-00-iam-role
       DispatchPrivateKey: |
         -----BEGIN OPENSSH PRIVATE KEY-----
         Read https://doc.arvados.org/install/crunch2-cloud/install-compute-node.html#sshkeypair
@@ -148,16 +149,10 @@ arvados:
         Replication: 2
         Driver: S3
         DriverParameters:
+          UseAWSS3v2Driver: true
           Bucket: __CLUSTER__-nyw5e-000000000000000-volume
           IAMRole: __CLUSTER__-keepstore-00-iam-role
           Region: FIXME
-      __CLUSTER__-nyw5e-0000000000000001:
-        Replication: 2
-        Driver: S3
-        DriverParameters:
-          Bucket: __CLUSTER__-nyw5e-000000000000001-volume
-          IAMRole: __CLUSTER__-keepstore-01-iam-role
-          Region: FIXME
 
     Users:
       NewUsersAreActive: true