2 title: "Running and Debugging a Workflow"
6 - "How do I provide input to run a workflow?"
7 - "What should I do if the workflow fails?"
9 - "Write an input parameter file."
10 - "Execute the workflow."
11 - "Diagnose workflow errors."
13 - "The input parameter file is a YAML file with values for each input parameter."
14 - "A common reason for a workflow step fails is insufficient RAM."
15 - "Use ResourceRequirement to set the amount of RAM to be allocated to the job."
16 - "Output parameter values are printed as JSON to standard output at the end of the run."
19 # The input parameter file
21 CWL input values are provided in the form of a YAML or JSON file.
24 This file gives the values for parameters declared in the `inputs`
25 section of our workflow. Our workflow takes `fq`, `genome` and `gtf`
28 When setting inputs, Files and Directories are given as an object with
29 `class: File` or `class: Directory`. This distinguishes them from
30 plain strings that may or may not be file paths.
32 Note: if you don't have example sequence data or the STAR index files, see [setup](/setup.html).
34 {% assign tabs = "generic, arvados" | split: ", " %}
35 {% capture generic_tab_content %}
40 location: rnaseq/raw_fastq/Mov10_oe_1.subset.fq
41 format: http://edamontology.org/format_1930
44 location: hg19-chr1-STAR-index
47 location: rnaseq/reference_data/chr1-hg19_genes.gtf
51 > ## Running the workflow
53 > Type this into the terminal:
56 > cwl-runner main.cwl main-input.yaml
60 > This may take a few minutes to run, and will print some amount of
61 > logging. The logging you see, how access other logs, and how to
62 > track workflow progress will depend on your CWL runner platform.
66 {% capture arvados_tab_content %}
71 location: keep:9178fe1b80a08a422dbe02adfd439764+925/raw_fastq/Mov10_oe_1.subset.fq
72 format: http://edamontology.org/format_1930
75 location: keep:02a12ce9e2707610991bd29d38796b57+2912
78 location: keep:9178fe1b80a08a422dbe02adfd439764+925/reference_data/chr1-hg19_genes.gtf
82 > ## Running the workflow
84 > If you are using VSCode with Arvados tasks, select `main.cwl` and
85 > then use the `Run CWL Workflow on Arvados` task.
93 <li><a href="#section-{{ tab }}">{{ tab }}</a></li>
97 <section id="section-generic">{{ generic_tab_content | markdownify}}</section>
98 <section id="section-arvados">{{ arvados_tab_content | markdownify}}</section>
101 # Debugging the workflow
103 Depending on whether and how your workflow platform enforces memory
104 limits, your workflow may fail. Let's talk about what to do when a
107 A workflow can fail for many reasons: some possible reasons include
108 bad input, bugs in the code, or running out memory. In our example,
109 the STAR workflow may fail with an out of memory error.
111 To help diagnose these errors, the workflow runner produces logs that
112 record what happened, either in the terminal or the web interface.
114 Some errors you might see in the logs that would indicate an out of
118 EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc
119 Possible cause 1: not enough RAM. Check if you have enough RAM 5711762337 bytes
120 Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 5711762337
126 Container exited with code: 137
129 (Exit code 137 most commonly occurs when a container goes "out of memory" and is terminated by the operating system).
131 If this happens, you will need to request more RAM.
133 # Setting runtime RAM requirements
135 By default, a step is allocated 256 MB of RAM. From the STAR error message:
137 > Check if you have enough RAM 5711762337 bytes
139 We can see that STAR requires quite a bit more RAM than 256 MB. To
140 request more RAM, add a "requirements" section with
141 "ResourceRequirement" to the "STAR" step:
148 run: bio-cwl-tools/STAR/STAR-Align.cwl
153 Resource requirements you can set include:
155 * coresMin: CPU cores
156 * ramMin: RAM (in megabytes)
157 * tmpdirMin: temporary directory available space
158 * outdirMin: output directory available space
160 > ## Running the workflow
162 > Now that you've fixed the workflow, run it again.
166 > ## Episode solution
167 > * <a href="../assets/answers/ep3/main.cwl">main.cwl</a>
172 The CWL runner will print a results JSON object to standard output. It will look something like this (it may include additional fields).
177 {% tab output generic %}
180 "bam_sorted_indexed": {
181 "location": "file:///home/username/rnaseq-cwl-training-exercises/Aligned.sortedByCoord.out.bam",
182 "basename": "Aligned.sortedByCoord.out.bam",
187 "basename": "Aligned.sortedByCoord.out.bam.bai",
188 "location": "file:///home/username/rnaseq-cwl-training-exercises/Aligned.sortedByCoord.out.bam.bai",
195 "location": "file:///home/username/rnaseq-cwl-training-exercises/Mov10_oe_1.subset_fastqc.html",
196 "basename": "Mov10_oe_1.subset_fastqc.html",
205 {% tab output arvados %}
208 "bam_sorted_indexed": {
209 "basename": "Aligned.sortedByCoord.out.bam",
211 "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Aligned.sortedByCoord.out.bam",
214 "basename": "Aligned.sortedByCoord.out.bam.bai",
216 "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Aligned.sortedByCoord.out.bam.bai"
222 "basename": "Mov10_oe_1.subset_fastqc.html",
224 "location": "keep:2dbaaef5aefd558e37f14280e47091a9+327/Mov10_oe_1.subset_fastqc.html",
234 This has a similar structure as `main-input.yaml`. The each output
235 parameter is listed, with the `location` field of each `File` object
236 indicating where the output file can be found.
240 // Get relevant elements and collections
241 const tabbed = document.querySelector('.tabbed');
242 const tablist = tabbed.querySelector('ul');
243 const tabs = tablist.querySelectorAll('a');
244 const panels = tabbed.querySelectorAll('[id^="section"]');
246 // The tab switching function
247 const switchTab = (oldTab, newTab) => {
249 // Make the active tab focusable by the user (Tab key)
250 newTab.removeAttribute('tabindex');
251 // Set the selected state
252 newTab.setAttribute('aria-selected', 'true');
253 newTab.classList.add('active');
254 oldTab.removeAttribute('aria-selected');
255 oldTab.setAttribute('tabindex', '-1');
256 oldTab.classList.remove('active');
257 // Get the indices of the new and old tabs to find the correct
258 // tab panels to show and hide
259 let index = Array.prototype.indexOf.call(tabs, newTab);
260 let oldIndex = Array.prototype.indexOf.call(tabs, oldTab);
261 panels[oldIndex].hidden = true;
262 panels[index].hidden = false;
265 // Add the tablist role to the first <ul> in the .tabbed container
266 tablist.setAttribute('role', 'tablist');
268 // Add semantics are remove user focusability for each tab
269 Array.prototype.forEach.call(tabs, (tab, i) => {
270 tab.setAttribute('role', 'tab');
271 tab.setAttribute('id', 'tab' + (i + 1));
272 tab.setAttribute('tabindex', '-1');
273 // tab.setAttribute('class', 'active');
274 tab.parentNode.setAttribute('role', 'presentation');
276 // Handle clicking of tabs for mouse users
277 tab.addEventListener('click', e => {
279 let currentTab = tablist.querySelector('[aria-selected]');
280 if (e.currentTarget !== currentTab) {
281 switchTab(currentTab, e.currentTarget);
285 // Handle keydown events for keyboard users
286 tab.addEventListener('keydown', e => {
287 // Get the index of the current tab in the tabs node list
288 let index = Array.prototype.indexOf.call(tabs, e.currentTarget);
289 // Work out which key the user is pressing and
290 // Calculate the new tab's index where appropriate
291 let dir = e.which === 37 ? index - 1 : e.which === 39 ? index + 1 : e.which === 40 ? 'down' : null;
294 // If the down key is pressed, move focus to the open panel,
295 // otherwise switch to the adjacent tab
296 dir === 'down' ? panels[i].focus() : tabs[dir] ? switchTab(e.currentTarget, tabs[dir]) : void 0;
301 // Add tab panel semantics and hide them all
302 Array.prototype.forEach.call(panels, (panel, i) => {
303 panel.setAttribute('role', 'tabpanel');
304 panel.setAttribute('tabindex', '-1');
305 let id = panel.getAttribute('id');
306 panel.setAttribute('aria-labelledby', tabs[i].id);
310 // Initially activate the first tab and reveal the first tab panel
311 tabs[0].removeAttribute('tabindex');
312 tabs[0].setAttribute('aria-selected', 'true');
313 tabs[0].setAttribute('class', 'active');
314 panels[0].hidden = false;