2 title: "Dynamic Workflow behavior with expressions"
6 - "Key question (FIXME)"
8 - "First learning objective. (FIXME)"
10 - "First key point. Brief Answer to questions. (FIXME)"
13 ### 1. Expressions on step inputs
15 You might have noticed that the output bam files are all named
16 `Aligned.sortedByCoord.out.bam`. This happens because because when we
17 call STAR, it gives the output a default file name.
19 During workflow execution, this is usually not a problem. The
20 workflow runner is smart enough to know that these files are different
21 and keep them separate. This can even make development easier by not
22 having to worry about assigning unique file names to every file.
24 However, it is a problem for humans interpreting the output. We can
25 fix this by setting the parameter `OutFileNamePrefix` on STAR. We
26 want the output filename to be based on the input filename.
28 In `alignment.cwl`, we can use `valueFrom` on the `OutFileNamePrefix`
29 input parameter to construct the output prefix from the input
34 StepInputExpressionRequirement: {}
43 OutFileNamePrefix: {valueFrom: "$(inputs.ForwardReads.nameroot)."}
46 The code between `$(...)` is called an "expression". It is evaluated
47 when setting up the step to run, and the expression is replaced by the
48 result to get the parameter value.
50 An expression can refer to other inputs to the step that are either
51 directly connected to another value, or have a default value. Here,
52 we refer to the input parameter ForwardReads, which is our fastq input
55 ForwardReads is a File object, not a plain file name, so it has some
56 fields of its own. The file object has a number of fields that we can
57 use. These include `basename` (the name of the file, without a
58 directory), `size` (file size, in bytes), `nameext` (the last file
59 extension) and `nameroot` (the name with `nameext` removed). Using
61 Finally, our expression is embedded in a string, so after replacing
62 the expression with the value of `inputs.ForwardReads.nameroot`, it
63 adds the remainder of the string, which just is a dot `.`. This is to
64 separate the leading part of our filename from the "Aligned.bam"
65 extension that will be added by STAR.
67 ### 2. Organizing output files into Directories
69 You probably noticed that all the output files appear in the same
70 directory. You might prefer that each file appears in its own
71 directory. This will show you how to do that.
73 Unlike shell scripts, in CWL you cannot call `mkdir` and `mv` to
74 organize files into directories. This is because the output files, at
75 this point, do not actually exist together in one directory. They may
76 exist on different nodes, in different directories, or different cloud
77 buckets. In CWL, instead of moving files around directly, you tell
78 the runner you want your directory to look like, and it will create it
81 We can use an "expression" to create a `Directory` object describing
82 each of our directories. An expression is a piece of Javascript code
83 which is executed by the workflow runner to produce values that affect
84 the workflow. These can be a simple as substituting the value of an
85 input variable, or as complex as en entire function that generates new
88 Javscript code must be bracketed inside `$(...)` or `${...}`. The
89 difference comes down to syntax. The `$()` form is more compact but
90 can only include code that can appear on the right side of an
91 assignment (`=`), which cannot include control blocks like `if` or
92 `for`. The `${}` form is a Javascript function, which can include
93 control blocks, and must end in a `return` statement.
95 Dxpressions can both appear in `valueFrom` fields as well as some
96 other fields, or in an `ExpressionTool` which, like `Workflow` or
97 `CommandLineTool` has explicitly defined `inputs` and `outputs`
100 The approach here is to define an expression tool which takes a
102 The `Directory` object has two fields, `basename` and `listing`. The
103 `basename` is the name of the directory, and the `listing` is the
104 contents, which consists of other File and Directory objects.
106 Create `subdirs.cwl`:
110 class: ExpressionTool
112 InlineJavascriptRequirement: {}
122 for (var i = 0; i < inputs.bams.length; i++) {
124 "class": "Directory",
125 "basename": inputs.fq[i].nameroot,
126 "listing": [inputs.bams[i], inputs.qc[i]]
129 return {"dirs": dirs};
133 Then change `main.cwl`:
142 bams: alignment/bam_sorted_indexed
143 qc: alignment/qc_html
148 outputSource: output-subdirs/dirs
151 outputSource: featureCounts/featurecounts