2 title: "Running and Debugging a Workflow"
6 - "How do I provide input to run a workflow?"
7 - "What should I do if the workflow fails?"
9 - "Write an input parameter file."
10 - "Execute the workflow."
11 - "Diagnose workflow errors."
13 - "The input parameter file is a YAML file with values for each input parameter."
14 - "A common reason for a workflow step fails is insufficient RAM."
15 - "Use ResourceRequirement to set the amount of RAM to be allocated to the job."
16 - "Output parameter values are printed as JSON to standard output at the end of the run."
19 # The input parameter file
21 CWL input values are provided in the form of a YAML or JSON file.
24 This file gives the values for parameters declared in the `inputs`
25 section of our workflow. Our workflow takes `fq`, `genome` and `gtf`
28 When setting inputs, Files and Directories are given as an object with
29 `class: File` or `class: Directory`. This distinguishes them from
30 plain strings that may or may not be file paths.
32 Note: if you don't have example sequence data or the STAR index files, see [setup](/setup.html).
37 {% tab input generic %}
42 location: rnaseq/raw_fastq/Mov10_oe_1.subset.fq
43 format: http://edamontology.org/format_1930
46 location: hg19-chr1-STAR-index
49 location: rnaseq/reference_data/chr1-hg19_genes.gtf
53 > ## Running the workflow
55 > Type this into the terminal:
58 > cwl-runner main.cwl main-input.yaml
62 > This may take a few minutes to run, and will print some amount of
63 > logging. The logging you see, how access other logs, and how to
64 > track workflow progress will depend on your CWL runner platform.
69 {% tab input arvados %}
74 location: keep:9178fe1b80a08a422dbe02adfd439764+925/raw_fastq/Mov10_oe_1.subset.fq
75 format: http://edamontology.org/format_1930
78 location: keep:02a12ce9e2707610991bd29d38796b57+2912
81 location: 9178fe1b80a08a422dbe02adfd439764+925/reference_data/chr1-hg19_genes.gtf
85 > ## Running the workflow
87 > If you are using VSCode with Arvados tasks, select `main.cwl` and
88 > then use the `Run CWL Workflow on Arvados` task.
95 # Debugging the workflow
97 Depending on whether and how your workflow platform enforces memory
98 limits, your workflow may fail. Let's talk about what to do when a
101 A workflow can fail for many reasons: some possible reasons include
102 bad input, bugs in the code, or running out memory. In our example,
103 the STAR workflow may fail with an out of memory error.
105 To help diagnose these errors, the workflow runner produces logs that
106 record what happened, either in the terminal or the web interface.
108 Some errors you might see in the logs that would indicate an out of
112 EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc
113 Possible cause 1: not enough RAM. Check if you have enough RAM 5711762337 bytes
114 Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 5711762337
120 Container exited with code: 137
123 (Exit code 137 most commonly occurs when a container goes "out of memory" and is terminated by the operating system).
125 If this happens, you will need to request more RAM.
127 # Setting runtime RAM requirements
129 By default, a step is allocated 256 MB of RAM. From the STAR error message:
131 > Check if you have enough RAM 5711762337 bytes
133 We can see that STAR requires quite a bit more RAM than 256 MB. To
134 request more RAM, add a "requirements" section with
135 "ResourceRequirement" to the "STAR" step:
142 run: bio-cwl-tools/STAR/STAR-Align.cwl
147 Resource requirements you can set include:
149 * coresMin: CPU cores
150 * ramMin: RAM (in megabytes)
151 * tmpdirMin: temporary directory available space
152 * outdirMin: output directory available space
154 > ## Running the workflow
156 > Now that you've fixed the workflow, run it again.
160 > ## Episode solution
161 > * <a href="{% link assets/answers/ep3/main.cwl %}">main.cwl</a>
166 The CWL runner will print a results JSON object to standard output. It will look something like this (it may include additional fields).
171 {% tab output generic %}
174 "bam_sorted_indexed": {
175 "location": "file:///home/username/rnaseq-cwl-training-exercises/Aligned.sortedByCoord.out.bam",
176 "basename": "Aligned.sortedByCoord.out.bam",
181 "basename": "Aligned.sortedByCoord.out.bam.bai",
182 "location": "file:///home/username/rnaseq-cwl-training-exercises/Aligned.sortedByCoord.out.bam.bai",
189 "location": "file:///home/username/rnaseq-cwl-training-exercises/Mov10_oe_1.subset_fastqc.html",
190 "basename": "Mov10_oe_1.subset_fastqc.html",
199 {% tab output arvados %}
202 "bam_sorted_indexed": {
203 "basename": "Aligned.sortedByCoord.out.bam",
205 "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Aligned.sortedByCoord.out.bam",
208 "basename": "Aligned.sortedByCoord.out.bam.bai",
210 "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Aligned.sortedByCoord.out.bam.bai"
216 "basename": "Mov10_oe_1.subset_fastqc.html",
218 "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Mov10_oe_1.subset_fastqc.html",
228 This has a similar structure as `main-input.yaml`. The each output
229 parameter is listed, with the `location` field of each `File` object
230 indicating where the output file can be found.