X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/e36e81150649a6457c9cbf0101130cfdb776336f..a24e4872dc015304b76c33b1246e4a3fc5163711:/doc/user/cwl/cwl-run-options.html.textile.liquid diff --git a/doc/user/cwl/cwl-run-options.html.textile.liquid b/doc/user/cwl/cwl-run-options.html.textile.liquid index 27970f440a..94e46ae1bc 100644 --- a/doc/user/cwl/cwl-run-options.html.textile.liquid +++ b/doc/user/cwl/cwl-run-options.html.textile.liquid @@ -1,7 +1,7 @@ --- layout: default navsection: userguide -title: "Using arvados-cwl-runner" +title: "arvados-cwl-runner options" ... {% comment %} Copyright (C) The Arvados Authors. All rights reserved. @@ -32,36 +32,40 @@ table(table table-bordered table-condensed). |==--debug==| Print even more logging| |==--metrics==| Print timing metrics| |==--tool-help==| Print command line help for tool| -|==--enable-reuse==| Enable job or container reuse (default)| -|==--disable-reuse==| Disable job or container reuse| -|==--project-uuid UUID==| Project that will own the workflow jobs, if not provided, will go to home project.| +|==--enable-reuse==| Enable container reuse (default)| +|==--disable-reuse==| Disable container reuse| +|==--project-uuid UUID==| Project that will own the workflow containers, if not provided, will go to home project.| |==--output-name OUTPUT_NAME==|Name to use for collection that stores the final output.| |==--output-tags OUTPUT_TAGS==|Tags for the final output collection separated by commas, e.g., =='--output-tags tag0,tag1,tag2'==.| -|==--ignore-docker-for-reuse==|Ignore Docker image version when deciding whether to reuse past jobs.| +|==--ignore-docker-for-reuse==|Ignore Docker image version when deciding whether to reuse past containers.| |==--submit==| Submit workflow runner to Arvados to manage the workflow (default).| -|==--local==| Run workflow on local host (still submits jobs to Arvados).| +|==--local==| Run workflow on local host (still submits containers to Arvados).| |==--create-template==| (Deprecated) synonym for --create-workflow.| -|==--create-workflow==| Create an Arvados workflow (if using the 'containers' API) or pipeline template (if using the 'jobs' API). See --api.| +|==--create-workflow==| Register an Arvados workflow that can be run from Workbench| |==--update-workflow== UUID|Update an existing Arvados workflow or pipeline template with the given UUID.| -|==--wait==| After submitting workflow runner job, wait for completion.| -|==--no-wait==| Submit workflow runner job and exit.| +|==--wait==| After submitting workflow runner, wait for completion.| +|==--no-wait==| Submit workflow runner and exit.| |==--log-timestamps==| Prefix logging lines with timestamp| |==--no-log-timestamps==| No timestamp on logging lines| -|==--api== {jobs,containers}|Select work submission API. Default is 'jobs' if that API is available, otherwise 'containers'.| |==--compute-checksum==| Compute checksum of contents while collecting outputs| -|==--submit-runner-ram== SUBMIT_RUNNER_RAM|RAM (in MiB) required for the workflow runner job (default 1024)| -|==--submit-runner-image== SUBMIT_RUNNER_IMAGE|Docker image for workflow runner job| +|==--submit-runner-ram== SUBMIT_RUNNER_RAM|RAM (in MiB) required for the workflow runner (default 1024)| +|==--submit-runner-image== SUBMIT_RUNNER_IMAGE|Docker image for workflow runner| |==--always-submit-runner==|When invoked with --submit --wait, always submit a runner to manage the workflow, even when only running a single CommandLineTool| -|==--submit-request-uuid== UUID|Update and commit to supplied container request instead of creating a new one (containers API only).| -|==--submit-runner-cluster== CLUSTER_ID|Submit workflow runner to a remote cluster (containers API only)| +|==--match-submitter-images==|Where Arvados has more than one Docker image of the same name, use image from the Docker instance on the submitting node.| +|==--submit-request-uuid== UUID|Update and commit to supplied container request instead of creating a new one.| +|==--submit-runner-cluster== CLUSTER_ID|Submit workflow runner to a remote cluster| |==--name NAME==|Name to use for workflow execution instance.| |==--on-error== {stop,continue}|Desired workflow behavior when a step fails. One of 'stop' (do not submit any more steps) or 'continue' (may submit other steps that are not downstream from the error). Default is 'continue'.| |==--enable-dev==|Enable loading and running development versions of CWL spec.| -|==--storage-classes== STORAGE_CLASSES|Specify comma separated list of storage classes to be used when saving workflow output to Keep.| +|==--storage-classes== STORAGE_CLASSES|Specify comma separated list of storage classes to be used when saving the final workflow output to Keep.| +|==--intermediate-storage-classes== STORAGE_CLASSES|Specify comma separated list of storage classes to be used when intermediate workflow output to Keep.| |==--intermediate-output-ttl== N|If N > 0, intermediate output collections will be trashed N seconds after creation. Default is 0 (don't trash).| -|==--priority== PRIORITY|Workflow priority (range 1..1000, higher has precedence over lower, containers api only)| -|==--thread-count== THREAD_COUNT|Number of threads to use for job submit and output collection.| +|==--priority== PRIORITY|Workflow priority (range 1..1000, higher has precedence over lower)| +|==--thread-count== THREAD_COUNT|Number of threads to use for container submit and output collection.| |==--http-timeout== HTTP_TIMEOUT|API request timeout in seconds. Default is 300 seconds (5 minutes).| +|==--enable-preemptible==|Use preemptible instances. Control individual steps with "arv:UsePreemptible":cwl-extensions.html#UsePreemptible hint.| +|==--disable-preemptible==|Don't use preemptible instances.| +|==--skip-schemas==|Skip loading of extension schemas (the $schemas section).| |==--trash-intermediate==|Immediately trash intermediate outputs on workflow success.| |==--no-trash-intermediate==|Do not trash intermediate outputs (default).| @@ -74,10 +78,10 @@ Use the @--name@ and @--output-name@ options to specify the name of the workflow
~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --name "Example bwa run" --output-name "Example bwa output" bwa-mem.cwl bwa-mem-input.yml
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
@@ -98,23 +102,23 @@ To submit a workflow and exit immediately, use the @--no-wait@ option. This wil
~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --no-wait bwa-mem.cwl bwa-mem-input.yml
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to qr1hi-4zz18-eqnfwrow8aysa9q
-2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-qr1hi-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to zzzzz-4zz18-eqnfwrow8aysa9q
+2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+zzzzz-8i9sb-fm2n3b1w0l6bskg
h3(#local). Control a workflow locally
-To run a workflow with local control, use @--local@. This means that the host where you run @arvados-cwl-runner@ will be responsible for submitting jobs, however, the jobs themselves will still run on the Arvados cluster. With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow will be terminated.
+To run a workflow with local control, use @--local@. This means that the host where you run @arvados-cwl-runner@ will be responsible for submitting containers, however, the containers themselves will still run on the Arvados cluster. With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow will be terminated.
~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance qr1hi-d1hrv-92wcu6ldtio74r4
-2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Queued
-2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Running
-2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Complete
+2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance zzzzz-d1hrv-92wcu6ldtio74r4
+2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Queued
+2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Running
+2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Complete
2016-07-01 10:05:46 arvados.cwl-runner[16290] INFO: Overall process status is success
{
"aligned_sam": {
@@ -142,3 +146,19 @@ Using @--intermediate-output-ttl@ without @--trash-intermediate@ means that inte
h3(#federation). Run workflow on a remote federated cluster
By default, the workflow runner will run on the local (home) cluster. Using @--submit-runner-cluster@ you can specify that the runner should be submitted to a remote federated cluster. When doing this, @--project-uuid@ should specify a project on that cluster. Steps making up the workflow will be submitted to the remote federated cluster by default, but the behavior of @arv:ClusterTarget@ is unchanged. Note: when using this option, any resources that need to be uploaded in order to run the workflow (such as files or Docker images) will be uploaded to the local (home) cluster, and streamed to the federated cluster on demand.
+
+h3(#preemptible). Using preemptible (spot) instances
+
+Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance.
+
+If the sitewide configuration @Containers.AlwaysUsePreemptibleInstances@ is true, workflow steps will always select preemptible instances, regardless of user option.
+
+If @Containers.AlwaysUsePreemptibleInstances@ is false, you can request preemptible instances for a specific run with the @arvados-cwl-runner --enable-preemptible@ option.
+
+Within the workflow, you can control whether individual steps should be preemptible with the "arv:UsePreemptible":cwl-extensions.html#UsePreemptible hint.
+
+If a workflow requests preemptible instances with "arv:UsePreemptible":cwl-extensions.html#UsePreemptible , but you _do not_ want to use preemptible instances, you can override it for a specific run with the @arvados-cwl-runner --disable-preemptible@ option.
+
+h3(#gpu). Use CUDA GPU instances
+
+See "cwltool:CUDARequirement":cwl-extensions.html#CUDARequirement .