From: Peter Amstutz Date: Fri, 13 Aug 2021 18:11:33 +0000 (-0400) Subject: 17390: Document arv:OutputStorageClass extension X-Git-Tag: 2.3.0~108^2 X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/cc77a0c01b38353d09d4f09a84434a60f1de1c45 17390: Document arv:OutputStorageClass extension Arvados-DCO-1.1-Signed-off-by: Peter Amstutz --- diff --git a/doc/user/cwl/cwl-extensions.html.textile.liquid b/doc/user/cwl/cwl-extensions.html.textile.liquid index 0987218e3c..206b75d58f 100644 --- a/doc/user/cwl/cwl-extensions.html.textile.liquid +++ b/doc/user/cwl/cwl-extensions.html.textile.liquid @@ -24,29 +24,34 @@ For portability, most Arvados extensions should go into the @hints@ section of y {% codeblock as yaml %} hints: arv:RunInSingleContainer: {} + arv:RuntimeConstraints: keep_cache: 123456 outputDirType: keep_output_dir + arv:PartitionRequirement: partition: dev_partition + arv:APIRequirement: {} - cwltool:LoadListingRequirement: - loadListing: shallow_listing + arv:IntermediateOutput: outputTTL: 3600 - arv:ReuseRequirement: - enableReuse: false + cwltool:Secrets: secrets: [input1, input2] - cwltool:TimeLimit: - timelimit: 14400 + arv:WorkflowRunnerResources: ramMin: 2048 coresMin: 2 keep_cache: 512 + arv:ClusterTarget: cluster_id: clsr1 project_uuid: clsr1-j7d0g-qxc4jcji7n4lafx + + arv:OutputStorageClass: + intermediateStorageClass: fast_storage + finalStorageClass: robust_storage {% endcodeblock %} h2(#RunInSingleContainer). arv:RunInSingleContainer @@ -120,6 +125,15 @@ table(table table-bordered table-condensed). |cluster_id|string|The five-character alphanumeric cluster id (uuid prefix) where a container or subworkflow will execute. May be an expression.| |project_uuid|string|The uuid of the project which will own container request and output of the container. May be an expression.| +h2(#OutputStorageClass). arv:OutputStorageClass + +Specify the "storage class":{{site.baseurl}}/user/topics/storage-classes.html to use for intermediate and final outputs. + +table(table table-bordered table-condensed). +|_. Field |_. Type |_. Description | +|intermediateStorageClass|string or array of strings|The storage class for output of intermediate steps. For example, faster "hot" storage.| +|finalStorageClass_uuid|string or array of strings|The storage class for the final output. | + h2. arv:dockerCollectionPDH This is an optional extension field appearing on the standard @DockerRequirement@. It specifies the portable data hash of the Arvados collection containing the Docker image. If present, it takes precedence over @dockerPull@ or @dockerImageId@. @@ -135,6 +149,16 @@ h1. Deprecated extensions The following extensions are deprecated because equivalent features are part of the CWL v1.1 standard. +{% codeblock as yaml %} +hints: + cwltool:LoadListingRequirement: + loadListing: shallow_listing + arv:ReuseRequirement: + enableReuse: false + cwltool:TimeLimit: + timelimit: 14400 +{% endcodeblock %} + h2. cwltool:LoadListingRequirement For CWL v1.1 scripts, this is deprecated in favor of "loadListing":https://www.commonwl.org/v1.1/CommandLineTool.html#CommandInputParameter or "LoadListingRequirement":https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingRequirement diff --git a/doc/user/topics/storage-classes.html.textile.liquid b/doc/user/topics/storage-classes.html.textile.liquid index 650c370955..06fd4d811c 100644 --- a/doc/user/topics/storage-classes.html.textile.liquid +++ b/doc/user/topics/storage-classes.html.textile.liquid @@ -10,9 +10,11 @@ Copyright (C) The Arvados Authors. All rights reserved. SPDX-License-Identifier: CC-BY-SA-3.0 {% endcomment %} -Storage classes (alternately known as "storage tiers") allow you to control which volumes should be used to store particular collection data blocks. This can be used to implement data storage policies such as moving data to archival storage. +Storage classes (sometimes called as "storage tiers") allow you to control which back-end storage volumes should be used to store the data blocks of a particular collection. This can be used to implement data storage policies such as assigning data collections to "fast", "robust" or "archival" storage. -Names of storage classes are internal to the cluster and decided by the administrator. Aside from "default", Arvados currently does not define any standard storage class names. +Names of storage classes are internal to the cluster and decided by the administrator. Aside from "default", Arvados currently does not define any standard storage class names. Consult your cluster administrator for guidance on what storage classes are available to use on your specific Arvados instance. + +Note that when changing the storage class of an existing collection, it does not take effect immediately, the blocks are asynchronously copied to the new storage class and removed from the old one. The collection field "storage_classes_confirmed" is updated to reflect when data blocks have been successfully copied. h3. arv-put @@ -32,14 +34,12 @@ $ arv-mount --storage-classes=transient --mount-tmp=scratch keep h3. arvados-cwl-runner -You may also specify the desired storage class for the final output collection produced by @arvados-cwl-runner@: +You may specify the desired storage class for the intermediate and final output collections produced by @arvados-cwl-runner@ on the command line or using the "arv:OutputStorageClass hint":{{site.baseurl}}/user/cwl/cwl-extensions.html#OutputStorageClass . 
-$ arvados-cwl-runner --storage-classes=hot myworkflow.cwl myinput.yml
+$ arvados-cwl-runner --intermediate-storage-classes=hot_storage --storage-classes=robust_storage myworkflow.cwl myinput.yml
-(Note: intermediate collections produced by a workflow run will use the cluster's default storage class(es).) - h3. arv command line You may set the storage class on an existing collection by setting the "storage_classes_desired" field of a Collection. For example, at the command line: