h2. Configuration
-To use preemptible instances, set @UsePreemptibleInstances: true@ and add entries to @InstanceTypes@ with @Preemptible: true@ to @config.yml@. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and will likely be lower. For example:
+First, configure some @InstanceTypes@ that have @Preemptible: true@. For a preemptible instance, @Price@ determines the maximum bid price; the actual price paid is dynamic and will likely be lower.
+
+Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. To do this automatically, use @PreemptiblePriceFactor@ to enable a preemptible version of each listed type, using the given factor to set the maximum bid price relative to the non-preemptible price. Alternatively, you can configure preemptible instance types explicitly. For example, the following two configurations are equivalent:
<pre>
Clusters:
- ClusterID:
+ ClusterID:
Containers:
- UsePreemptibleInstances: true
+ PreemptiblePriceFactor: 0.8
InstanceTypes:
m4.large:
- Preemptible: false
ProviderType: m4.large
VCPUs: 2
RAM: 8GiB
AddedScratch: 32GB
Price: 0.1
- m4.large.spot:
- Preemptible: true
+</pre>
+
+<pre>
+Clusters:
+ ClusterID:
+ InstanceTypes:
+ m4.large:
ProviderType: m4.large
VCPUs: 2
RAM: 8GiB
AddedScratch: 32GB
Price: 0.1
+ m4.large.preemptible:
+ Preemptible: true
+ ProviderType: m4.large
+ VCPUs: 2
+ RAM: 8GiB
+ AddedScratch: 32GB
+ Price: 0.08
</pre>
-When @UsePreemptibleInstances@ is enabled, child containers (workflow steps) will automatically be made preemptible. Note that because preempting the workflow runner would cancel the entire workflow, the workflow runner runs in a reserved (non-preemptible) instance.
+Next, you can choose to enable automatic use of preemptible instances:
+
+<pre>
+Clusters:
+ ClusterID:
+ Containers:
+ AlwaysUsePreemptibleInstances: true
+</pre>
+
+If @AlwaysUsePreemptibleInstances@ is "true", child containers (workflow steps) will always select preemptible instances, regardless of user option.
+
+If @AlwaysUsePreemptibleInstances@ is "false" (the default) or unspecified, preemptible instance are "used when requested by the user.":{{site.baseurl}}/user/cwl/cwl-run-options.html#preemptible
+
+Note that regardless of the value of @AlwaysUsePreemptibleInstances@, the top level workflow runner container always runs in a reserved (non-preemptible) instance, to avoid situations where the workflow runner is killed requiring the entire to be restarted.
No additional configuration is required, "arvados-dispatch-cloud":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html will now start preemptible instances where appropriate.
The account needs to have a service linked role created. This can be done by logging into the AWS account, go to _IAM Management_ → _Roles_ and create the @AWSServiceRoleForEC2Spot@ role by clicking on the @Create@ button, selecting @EC2@ service and @EC2 - Spot Instances@ use case.
+h3. Interruption notices
+
+When running a container on a spot instance, Arvados monitors the EC2 metadata endpoint for interruption notices. When an interruption notice is received, it is reported in a log entry in the @crunch-run.txt@ file as well as @warning@ and @preemptionNotice@ keys in the @runtime_status@ field of the affected container.
+
+Example excerpt from @crunch-run.txt@:
+
+<pre>
+2023-02-21T21:12:42.350719824Z Cloud provider scheduled instance stop at 2023-02-21T21:14:42Z
+</pre>
+
+Example @runtime_status@:
+
+<pre>
+{
+ "warning": "preemption notice",
+ "warningDetail": "Cloud provider scheduled instance stop at 2023-02-21T21:14:42Z",
+ "preemptionNotice": "Cloud provider scheduled instance stop at 2023-02-21T21:14:42Z"
+}
+</pre>
+
h2. Preemptible instances on Azure
For general information, see "Use Spot VMs in Azure":https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms.
projects / arvados.git / blobdiff