2 title: "Running and debugging a workflow"
6 - "Key question (FIXME)"
8 - "First learning objective. (FIXME)"
10 - "First key point. Brief Answer to questions. (FIXME)"
13 # Running and debugging a workflow
15 ### 1. The input parameter file
17 CWL input values are provided in the form of a YAML or JSON file.
18 Create one by right clicking on the explorer, select "New File" and
19 create a called file "main-input.yaml".
21 This file gives the values for parameters declared in the `inputs`
22 section of our workflow. Our workflow takes `fq`, `genome` and `gtf`
25 When setting inputs, Files and Directories are given as an object with
26 `class: File` or `class: Directory`. This distinguishes them from
27 plain strings that may or may not be file paths.
29 Note: if you don't have example sequence data or the STAR index files, see the Appendix below.
34 location: rnaseq/raw_fastq/Mov10_oe_1.subset.fq
35 format: http://edamontology.org/format_1930
38 location: hg19-chr1-STAR-index
41 location: rnaseq/reference_data/chr1-hg19_genes.gtf
49 location: keep:9178fe1b80a08a422dbe02adfd439764+925/raw_fastq/Mov10_oe_1.subset.fq
50 format: http://edamontology.org/format_1930
53 location: keep:02a12ce9e2707610991bd29d38796b57+2912
56 location: keep:9178fe1b80a08a422dbe02adfd439764+925/reference_data/chr1-hg19_genes.gtf
59 ### 2. Running the workflow
61 Type this into the terminal:
64 cwl-runner main.cwl main-input.yaml
67 ### 3. Debugging the workflow
69 A workflow can fail for many reasons: some possible reasons include
70 bad input, bugs in the code, or running out memory. In this case, the
71 STAR workflow might fail with an out of memory error.
73 To help diagnose these errors, the workflow runner produces logs that
74 record what happened, either in the terminal or the web interface.
76 Some errors you might see in the logs that would indicate an out of
80 EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc
81 Possible cause 1: not enough RAM. Check if you have enough RAM 5711762337 bytes
82 Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 5711762337
88 Container exited with code: 137
91 (Exit code 137 most commonly occurs when a container goes "out of memory" and is terminated by the operating system).
93 If this happens, you will need to request more RAM.
95 ### 4. Setting runtime RAM requirements
97 By default, a step is allocated 256 MB of RAM. From the STAR error message:
99 > Check if you have enough RAM 5711762337 bytes
101 We can see that STAR requires quite a bit more RAM than that. To
102 request more RAM, add a "requirements" section with
103 "ResourceRequirement" to the "STAR" step:
110 run: bio-cwl-tools/STAR/STAR-Align.cwl
113 Resource requirements you can set include:
115 * coresMin: CPU cores
116 * ramMin: RAM (in megabytes)
117 * tmpdirMin: temporary directory available space
118 * outdirMin: output directory available space
120 After setting the RAM requirements, re-run the workflow.
122 ### 5. Workflow results
124 The CWL runner will print a results JSON object to standard output. It will look something like this (it may include additional fields).
129 "bam_sorted_indexed": {
130 "location": "file:///home/username/rnaseq-cwl-training-exercises/Aligned.sortedByCoord.out.bam",
131 "basename": "Aligned.sortedByCoord.out.bam",
136 "basename": "Aligned.sortedByCoord.out.bam.bai",
137 "location": "file:///home/username/rnaseq-cwl-training-exercises/Aligned.sortedByCoord.out.bam.bai",
144 "location": "file:///home/username/rnaseq-cwl-training-exercises/Mov10_oe_1.subset_fastqc.html",
145 "basename": "Mov10_oe_1.subset_fastqc.html",
152 This has the same structure as `main-input.yaml`. The each output
153 parameter is listed, with the `location` field of each `File` object
154 indicating where the output file can be found.
158 ## Downloading sample and reference data
160 Start from your rnaseq-cwl-exercises directory.
165 wget --mirror --no-parent --no-host --cut-dirs=1 https://download.pirca.arvadosapi.com/c=9178fe1b80a08a422dbe02adfd439764+925/
168 ## Downloading or generating STAR index
170 Running STAR requires index files generated from the reference.
172 This is a rather large download (4 GB). Depending on your bandwidth, it may be faster to generate it yourself.
177 mkdir hg19-chr1-STAR-index
178 cd hg19-chr1-STAR-index
179 wget --mirror --no-parent --no-host --cut-dirs=1 https://download.pirca.arvadosapi.com/c=02a12ce9e2707610991bd29d38796b57+2912/
184 Create `chr1-star-index.yaml`:
189 location: rnaseq/reference_data/chr1.fa
190 format: http://edamontology.org/format_1930
191 IndexName: 'hg19-chr1-STAR-index'
194 location: rnaseq/reference_data/chr1-hg19_genes.gtf
198 Generate the index with your local cwl-runner.
201 cwl-runner bio-cwl-tools/STAR/STAR-Index.cwl chr1-star-index.yaml