X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/bee4fa82c15b2188c2105fff3b52c305b38f04a6..4325310d63e8f674ba9ed30fda65f61c4ca1cacf:/doc/user/cwl/cwl-run-options.html.textile.liquid diff --git a/doc/user/cwl/cwl-run-options.html.textile.liquid b/doc/user/cwl/cwl-run-options.html.textile.liquid index 8d1ec19988..94e46ae1bc 100644 --- a/doc/user/cwl/cwl-run-options.html.textile.liquid +++ b/doc/user/cwl/cwl-run-options.html.textile.liquid @@ -1,46 +1,76 @@ --- layout: default navsection: userguide -title: "Using arvados-cwl-runner" +title: "arvados-cwl-runner options" ... +{% comment %} +Copyright (C) The Arvados Authors. All rights reserved. + +SPDX-License-Identifier: CC-BY-SA-3.0 +{% endcomment %} + +# "*Command line options*":#options +# "*Specify workflow and output names*":#names +# "*Submit a workflow without waiting for the result*":#nowait +# "*Control a workflow locally*":#local +# "*Automatically delete intermediate outputs*":#delete +# "*Run workflow on a remote federated cluster*":#federation + +h3(#options). Command line options The following command line options are available for @arvados-cwl-runner@: table(table table-bordered table-condensed). |_. Option |_. Description | |==--basedir== BASEDIR| Base directory used to resolve relative references in the input, default to directory of input object file or current directory (if inputs piped/provided on command line).| +|==--eval-timeout EVAL_TIMEOUT==|Time to wait for a Javascript expression to evaluate before giving an error, default 20s.| +|==--print-dot==| Print workflow visualization in graphviz format and exit| |==--version==| Print version and exit| |==--validate==| Validate CWL document only.| |==--verbose==| Default logging| |==--quiet==| Only print warnings and errors.| |==--debug==| Print even more logging| +|==--metrics==| Print timing metrics| |==--tool-help==| Print command line help for tool| -|==--enable-reuse==|Enable job reuse (default)| -|==--disable-reuse==|Disable job reuse (always run new jobs).| -|==--project-uuid UUID==| Project that will own the workflow jobs, if not provided, will go to home project.| +|==--enable-reuse==| Enable container reuse (default)| +|==--disable-reuse==| Disable container reuse| +|==--project-uuid UUID==| Project that will own the workflow containers, if not provided, will go to home project.| |==--output-name OUTPUT_NAME==|Name to use for collection that stores the final output.| |==--output-tags OUTPUT_TAGS==|Tags for the final output collection separated by commas, e.g., =='--output-tags tag0,tag1,tag2'==.| -|==--ignore-docker-for-reuse==|Ignore Docker image version when deciding whether to reuse past jobs.| -|==--submit==| Submit workflow to run on Arvados.| -|==--local==| Control workflow from local host (submits jobs to Arvados).| -|==--create-template==| (Deprecated) synonym for ==--create-workflow.==| -|==--create-workflow==| Create an Arvados workflow (if using the 'containers' API) or pipeline template (if using the 'jobs' API). See ==--api==.| +|==--ignore-docker-for-reuse==|Ignore Docker image version when deciding whether to reuse past containers.| +|==--submit==| Submit workflow runner to Arvados to manage the workflow (default).| +|==--local==| Run workflow on local host (still submits containers to Arvados).| +|==--create-template==| (Deprecated) synonym for --create-workflow.| +|==--create-workflow==| Register an Arvados workflow that can be run from Workbench| |==--update-workflow== UUID|Update an existing Arvados workflow or pipeline template with the given UUID.| -|==--wait==| After submitting workflow runner job, wait for completion.| -|==--no-wait==| Submit workflow runner job and exit.| -|==--api== WORK_API| Select work submission API, one of 'jobs' or 'containers'. Default is 'jobs' if that API is available, otherwise 'containers'.| +|==--wait==| After submitting workflow runner, wait for completion.| +|==--no-wait==| Submit workflow runner and exit.| +|==--log-timestamps==| Prefix logging lines with timestamp| +|==--no-log-timestamps==| No timestamp on logging lines| |==--compute-checksum==| Compute checksum of contents while collecting outputs| -|==--submit-runner-ram== SUBMIT_RUNNER_RAM|RAM (in MiB) required for the workflow runner job (default 1024)| -|==--submit-runner-image== SUBMIT_RUNNER_IMAGE|Docker image for workflow runner job, default arvados/jobs| -|==--name== NAME| Name to use for workflow execution instance.| -|==--on-error {stop,continue}|Desired workflow behavior when a step fails. One of 'stop' or 'continue'. Default is 'continue'.| -|==--enable-dev==| Enable loading and running development versions of CWL spec.| +|==--submit-runner-ram== SUBMIT_RUNNER_RAM|RAM (in MiB) required for the workflow runner (default 1024)| +|==--submit-runner-image== SUBMIT_RUNNER_IMAGE|Docker image for workflow runner| +|==--always-submit-runner==|When invoked with --submit --wait, always submit a runner to manage the workflow, even when only running a single CommandLineTool| +|==--match-submitter-images==|Where Arvados has more than one Docker image of the same name, use image from the Docker instance on the submitting node.| +|==--submit-request-uuid== UUID|Update and commit to supplied container request instead of creating a new one.| +|==--submit-runner-cluster== CLUSTER_ID|Submit workflow runner to a remote cluster| +|==--name NAME==|Name to use for workflow execution instance.| +|==--on-error== {stop,continue}|Desired workflow behavior when a step fails. One of 'stop' (do not submit any more steps) or 'continue' (may submit other steps that are not downstream from the error). Default is 'continue'.| +|==--enable-dev==|Enable loading and running development versions of CWL spec.| +|==--storage-classes== STORAGE_CLASSES|Specify comma separated list of storage classes to be used when saving the final workflow output to Keep.| +|==--intermediate-storage-classes== STORAGE_CLASSES|Specify comma separated list of storage classes to be used when intermediate workflow output to Keep.| |==--intermediate-output-ttl== N|If N > 0, intermediate output collections will be trashed N seconds after creation. Default is 0 (don't trash).| -|==--trash-intermediate==| Immediately trash intermediate outputs on workflow success.| +|==--priority== PRIORITY|Workflow priority (range 1..1000, higher has precedence over lower)| +|==--thread-count== THREAD_COUNT|Number of threads to use for container submit and output collection.| +|==--http-timeout== HTTP_TIMEOUT|API request timeout in seconds. Default is 300 seconds (5 minutes).| +|==--enable-preemptible==|Use preemptible instances. Control individual steps with "arv:UsePreemptible":cwl-extensions.html#UsePreemptible hint.| +|==--disable-preemptible==|Don't use preemptible instances.| +|==--skip-schemas==|Skip loading of extension schemas (the $schemas section).| +|==--trash-intermediate==|Immediately trash intermediate outputs on workflow success.| |==--no-trash-intermediate==|Do not trash intermediate outputs (default).| -h3. Specify workflow and output names +h3(#names). Specify workflow and output names Use the @--name@ and @--output-name@ options to specify the name of the workflow and name of the output collection. @@ -48,10 +78,10 @@ Use the @--name@ and @--output-name@ options to specify the name of the workflow
~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --name "Example bwa run" --output-name "Example bwa output" bwa-mem.cwl bwa-mem-input.yml
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to qr1hi-4zz18-h7ljh5u76760ww2
-2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Running
-2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-fm2n3b1w0l6bskg) is Complete
+2016-06-30 14:56:36 arvados.arv-run[27002] INFO: Uploaded to zzzzz-4zz18-h7ljh5u76760ww2
+2016-06-30 14:56:40 arvados.cwl-runner[27002] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 14:56:41 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Running
+2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-fm2n3b1w0l6bskg) is Complete
2016-06-30 14:57:12 arvados.cwl-runner[27002] INFO: Overall process status is success
{
"aligned_sam": {
@@ -64,7 +94,7 @@ arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107,
-h3. Submit a workflow with no waiting
+h3(#nowait). Submit a workflow without waiting for the result
To submit a workflow and exit immediately, use the @--no-wait@ option. This will submit the workflow to Arvados, print out the UUID of the job that was submitted to standard output, and exit.
@@ -72,23 +102,23 @@ To submit a workflow and exit immediately, use the @--no-wait@ option. This wil
~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --no-wait bwa-mem.cwl bwa-mem-input.yml
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Upload local files: "bwa-mem.cwl"
-2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to qr1hi-4zz18-eqnfwrow8aysa9q
-2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job qr1hi-8i9sb-fm2n3b1w0l6bskg
-qr1hi-8i9sb-fm2n3b1w0l6bskg
+2016-06-30 15:07:52 arvados.arv-run[12480] INFO: Uploaded to zzzzz-4zz18-eqnfwrow8aysa9q
+2016-06-30 15:07:52 arvados.cwl-runner[12480] INFO: Submitted job zzzzz-8i9sb-fm2n3b1w0l6bskg
+zzzzz-8i9sb-fm2n3b1w0l6bskg
-h3. Control a workflow locally
+h3(#local). Control a workflow locally
-To run a workflow with local control, use @--local@. This means that the host where you run @arvados-cwl-runner@ will be responsible for submitting jobs, however, the jobs themselves will still run on the Arvados cluster. With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow will be terminated.
+To run a workflow with local control, use @--local@. This means that the host where you run @arvados-cwl-runner@ will be responsible for submitting containers, however, the containers themselves will still run on the Arvados cluster. With @--local@, if you interrupt @arvados-cwl-runner@ or log out, the workflow will be terminated.
~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml
arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107, cwltool 1.0.20160629140624
-2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance qr1hi-d1hrv-92wcu6ldtio74r4
-2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Queued
-2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Running
-2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (qr1hi-8i9sb-2nzzfbuf9zjrj4g) is Complete
+2016-07-01 10:05:19 arvados.cwl-runner[16290] INFO: Pipeline instance zzzzz-d1hrv-92wcu6ldtio74r4
+2016-07-01 10:05:28 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Queued
+2016-07-01 10:05:29 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Running
+2016-07-01 10:05:45 arvados.cwl-runner[16290] INFO: Job bwa-mem.cwl (zzzzz-8i9sb-2nzzfbuf9zjrj4g) is Complete
2016-07-01 10:05:46 arvados.cwl-runner[16290] INFO: Overall process status is success
{
"aligned_sam": {
@@ -101,7 +131,7 @@ arvados-cwl-runner 1.0.20160628195002, arvados-python-client 0.1.20160616015107,