---
-title: "Running and debugging a workflow"
-teaching: 0
-exercises: 0
+title: "Running and Debugging a Workflow"
+teaching: 15
+exercises: 20
questions:
-- "Key question (FIXME)"
+- "How do I provide input to run a workflow?"
+- "What should I do if the workflow fails?"
objectives:
-- "First learning objective. (FIXME)"
+- "Write an input parameter file."
+- "Execute the workflow."
+- "Diagnose workflow errors."
keypoints:
-- "First key point. Brief Answer to questions. (FIXME)"
+- "The input parameter file is a YAML file with values for each input parameter."
+- "A common reason for a workflow step fails is insufficient RAM."
+- "Use ResourceRequirement to set the amount of RAM to be allocated to the job."
+- "Output parameter values are printed as JSON to standard output at the end of the run."
---
-# Running and debugging a workflow
-
-### 1. The input parameter file
+# The input parameter file
CWL input values are provided in the form of a YAML or JSON file.
-Create one by right clicking on the explorer, select "New File" and
-create a called file "main-input.yaml".
+create a called file
This file gives the values for parameters declared in the `inputs`
section of our workflow. Our workflow takes `fq`, `genome` and `gtf`
`class: File` or `class: Directory`. This distinguishes them from
plain strings that may or may not be file paths.
-Note: if you don't have example sequence data or the STAR index files, see the Appendix below.
+Note: if you don't have example sequence data or the STAR index files, see [setup](/setup.html).
+{% capture generic_input_tab_content %}
+main-input.yaml
```
fq:
class: File
class: File
location: rnaseq/reference_data/chr1-hg19_genes.gtf
```
-
-On Arvados, do this:
-
+{: .language-yaml }
+
+> ## Running the workflow
+>
+> Type this into the terminal:
+>
+> ```
+> cwl-runner main.cwl main-input.yaml
+> ```
+> {: .language-bash }
+>
+> This may take a few minutes to run, and will print some amount of
+> logging. The logging you see, how access other logs, and how to
+> track workflow progress will depend on your CWL runner platform.
+{: .challenge }
+{% endcapture %}
+
+{% capture arvados_input_tab_content %}
+main-input.yaml
```
fq:
class: File
class: File
location: keep:9178fe1b80a08a422dbe02adfd439764+925/reference_data/chr1-hg19_genes.gtf
```
+{: .language-yaml }
-### 2. Running the workflow
+> ## Running the workflow
+>
+> If you are using VSCode with Arvados tasks, select `main.cwl` and
+> then use the `Run CWL Workflow on Arvados` task.
+>
+{: .challenge }
+{% endcapture %}
-Type this into the terminal:
+<div class="tabbed">
+ <ul class="tab">
+ <li><a href="#section-arvados-input">arvados</a></li>
+ <li><a href="#section-generic-input">generic</a></li>
+ </ul>
-```
-cwl-runner main.cwl main-input.yaml
-```
+ <section id="section-arvados-input">{{ arvados_input_tab_content | markdownify}}</section>
+ <section id="section-generic-input">{{ generic_input_tab_content | markdownify}}</section>
+</div>
+
+# Debugging the workflow
-### 3. Debugging the workflow
+Depending on whether and how your workflow platform enforces memory
+limits, your workflow may fail. Let's talk about what to do when a
+workflow fails.
A workflow can fail for many reasons: some possible reasons include
-bad input, bugs in the code, or running out memory. In this case, the
-STAR workflow might fail with an out of memory error.
+bad input, bugs in the code, or running out memory. In our example,
+the STAR workflow may fail with an out of memory error.
To help diagnose these errors, the workflow runner produces logs that
record what happened, either in the terminal or the web interface.
If this happens, you will need to request more RAM.
-### 4. Setting runtime RAM requirements
+# Setting runtime RAM requirements
By default, a step is allocated 256 MB of RAM. From the STAR error message:
> Check if you have enough RAM 5711762337 bytes
-We can see that STAR requires quite a bit more RAM than that. To
+We can see that STAR requires quite a bit more RAM than 256 MB. To
request more RAM, add a "requirements" section with
"ResourceRequirement" to the "STAR" step:
STAR:
requirements:
ResourceRequirement:
- ramMin: 8000
+ ramMin: 9000
run: bio-cwl-tools/STAR/STAR-Align.cwl
+ ...
```
+{: .language-yaml }
Resource requirements you can set include:
* tmpdirMin: temporary directory available space
* outdirMin: output directory available space
-After setting the RAM requirements, re-run the workflow.
+> ## Running the workflow
+>
+> Now that you've fixed the workflow, run it again.
+>
+{: .challenge }
-### 5. Workflow results
+> ## Episode solution
+> * <a href="../assets/answers/ep3/main.cwl">main.cwl</a>
+{: .solution}
+
+# Workflow results
The CWL runner will print a results JSON object to standard output. It will look something like this (it may include additional fields).
+{% capture generic_output_tab_content %}
```
{
}
}
```
+{: .language-yaml }
+{% endcapture %}
-This has the same structure as `main-input.yaml`. The each output
-parameter is listed, with the `location` field of each `File` object
-indicating where the output file can be found.
-
-# Appendix
-
-## Downloading sample and reference data
-
-Start from your rnaseq-cwl-exercises directory.
-
-```
-mkdir rnaseq
-cd rnaseq
-wget --mirror --no-parent --no-host --cut-dirs=1 https://download.pirca.arvadosapi.com/c=9178fe1b80a08a422dbe02adfd439764+925/
+{% capture arvados_output_tab_content %}
```
-
-## Downloading or generating STAR index
-
-Running STAR requires index files generated from the reference.
-
-This is a rather large download (4 GB). Depending on your bandwidth, it may be faster to generate it yourself.
-
-### Downloading
-
-```
-mkdir hg19-chr1-STAR-index
-cd hg19-chr1-STAR-index
-wget --mirror --no-parent --no-host --cut-dirs=1 https://download.pirca.arvadosapi.com/c=02a12ce9e2707610991bd29d38796b57+2912/
+{
+ "bam_sorted_indexed": {
+ "basename": "Aligned.sortedByCoord.out.bam",
+ "class": "File",
+ "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Aligned.sortedByCoord.out.bam",
+ "secondaryFiles": [
+ {
+ "basename": "Aligned.sortedByCoord.out.bam.bai",
+ "class": "File",
+ "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Aligned.sortedByCoord.out.bam.bai"
+ }
+ ],
+ "size": 25370695
+ },
+ "qc_html": {
+ "basename": "Mov10_oe_1.subset_fastqc.html",
+ "class": "File",
+ "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Mov10_oe_1.subset_fastqc.html",
+ "size": 383589
+ }
+}
```
+{: .language-yaml }
+{% endcapture %}
-### Generating
-
-Create `chr1-star-index.yaml`:
+<div class="tabbed">
+ <ul class="tab">
+ <li><a href="#section-arvados-output">arvados</a></li>
+ <li><a href="#section-generic-output">generic</a></li>
+ </ul>
-```
-InputFiles:
- - class: File
- location: rnaseq/reference_data/chr1.fa
- format: http://edamontology.org/format_1930
-IndexName: 'hg19-chr1-STAR-index'
-Gtf:
- class: File
- location: rnaseq/reference_data/chr1-hg19_genes.gtf
-Overhang: 99
-```
+ <section id="section-arvados-output">{{ arvados_output_tab_content | markdownify}}</section>
+ <section id="section-generic-output">{{ generic_output_tab_content | markdownify}}</section>
+</div>
-Generate the index with your local cwl-runner.
-
-```
-cwl-runner bio-cwl-tools/STAR/STAR-Index.cwl chr1-star-index.yaml
-```
+This has a similar structure as `main-input.yaml`. The each output
+parameter is listed, with the `location` field of each `File` object
+indicating where the output file can be found.
projects / rnaseq-cwl-training.git / blobdiff