From b76063e83a876049d13d1f97cb1f1d7b07e0ebb7 Mon Sep 17 00:00:00 2001 From: Peter Amstutz Date: Mon, 25 Jan 2021 15:55:58 -0500 Subject: [PATCH] More text for lesson 5. Working on formatting. Arvados-DCO-1.1-Signed-off-by: Peter Amstutz --- lesson1/lesson1.md | 31 +++++++++++++++---------------- lesson5/lesson5.md | 42 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 54 insertions(+), 19 deletions(-) diff --git a/lesson1/lesson1.md b/lesson1/lesson1.md index b989855..8163823 100644 --- a/lesson1/lesson1.md +++ b/lesson1/lesson1.md @@ -2,7 +2,7 @@ In this lesson we will turn `rnaseq_analysis_on_input_file.sh` into a workflow. -# Setting up +## Setting up We will create a new git repository and import a library of existing tool definitions that will help us build our workflow. @@ -11,19 +11,16 @@ tool definitions that will help us build our workflow. 2. Create a new git repository to hold our workflow with this command: -## Arvados - ``` -git clone https://github.com/arvados/arvados-vscode-cwl-template.git rnaseq-cwl-training-exercises +git init rnaseq-cwl-training-exercises ``` -## Generic +On Arvados use this: ``` -git init rnaseq-cwl-training-exercises +git clone https://github.com/arvados/arvados-vscode-cwl-template.git rnaseq-cwl-training-exercises ``` - 3. Go to File->Open Folder and select rnaseq-cwl-training-exercises 4. Go to the terminal window @@ -34,11 +31,13 @@ git init rnaseq-cwl-training-exercises git submodule add https://github.com/common-workflow-library/bio-cwl-tools.git ``` -# Writing the workflow +## Writing the workflow + +### 1. -1. Create a new file "main.cwl" +Create a new file "main.cwl" -2. Start with this header. +Start with this header. ``` @@ -47,7 +46,7 @@ class: Workflow label: RNAseq CWL practice workflow ``` -3. Workflow Inputs +### 2. Workflow Inputs The purpose of a workflow is to consume some input parameters, run a series of steps, and produce output values. @@ -80,7 +79,7 @@ inputs: gtf: File ``` -4. Workflow Steps +### 3. Workflow Steps A workflow consists of one or more steps. This is the `steps` section. @@ -115,7 +114,7 @@ steps: out: [html_file] ``` -5. Running alignment with STAR +### 4. Running alignment with STAR STAR has more parameters. Sometimes we want to provide input values to a step without making them as workflow-level inputs. We can do @@ -137,7 +136,7 @@ this with `{default: N}` out: [alignment] ``` -6. Running samtools +### 5. Running samtools The third step is to generate an index for the aligned BAM. @@ -156,7 +155,7 @@ step will not run until the `STAR` step has completed successfully. out: [bam_sorted_indexed] ``` -7. featureCounts +### 6. featureCounts As of this writing, the `subread` package that provides `featureCounts` is not available in bio-cwl-tools (and if it has been @@ -164,7 +163,7 @@ added since writing this, let's pretend that it isn't there.) We will dive into how to write a CWL wrapper for a command line tool in lesson 2. For now, we will leave off the final step. -8. Workflow Outputs +### 7. Workflow Outputs The last thing to do is declare the workflow outputs in the `outputs` section. diff --git a/lesson5/lesson5.md b/lesson5/lesson5.md index fe3442d..54df134 100644 --- a/lesson5/lesson5.md +++ b/lesson5/lesson5.md @@ -6,13 +6,14 @@ You might have noticed that the output bam files are all named `Aligned.sortedByCoord.out.bam`. This happens because because when we call STAR, it gives the output a default file name. -Now, during workflow execution, this is usually not a problem. The +During workflow execution, this is usually not a problem. The workflow runner is smart enough to know that these files are different and keep them separate. This can even make development easier by not having to worry about assigning unique file names to every file. However, it is a problem for humans interpreting the output. We can -fix this by setting the parameter `OutFileNamePrefix` on STAR. +fix this by setting the parameter `OutFileNamePrefix` on STAR. We +want the output filename to be based on the input filename. In `alignment.cwl`, we can use `valueFrom` on the `OutFileNamePrefix` input parameter to construct the output prefix from the input @@ -54,7 +55,42 @@ extension that will be added by STAR. 2. Organizing output files into Directories -This is a more advanced example. +You probably noticed that all the output files appear in the same +directory. You might prefer that each file appears in its own +directory. This will show you how to do that. + +Unlike shell scripts, in CWL you cannot call `mkdir` and `mv` to +organize files into directories. This is because the output files, at +this point, do not actually exist together in one directory. They may +exist on different nodes, in different directories, or different cloud +buckets. In CWL, instead of moving files around directly, you tell +the runner you want your directory to look like, and it will create it +for you. + +We can use an "expression" to create a `Directory` object describing +each of our directories. An expression is a piece of Javascript code +which is executed by the workflow runner to produce values that affect +the workflow. These can be a simple as substituting the value of an +input variable, or as complex as en entire function that generates new +objects. + +Javscript code must be bracketed inside `$(...)` or `${...}`. The +difference comes down to syntax. The `$()` form is more compact but +can only include code that can appear on the right side of an +assignment (`=`), which cannot include control blocks like `if` or +`for`. The `${}` form is a Javascript function, which can include +control blocks, and must end in a `return` statement. + +Dxpressions can both appear in `valueFrom` fields as well as some +other fields, or in an `ExpressionTool` which, like `Workflow` or +`CommandLineTool` has explicitly defined `inputs` and `outputs` +sections. + +The approach here is to define an expression tool which takes a + +The `Directory` object has two fields, `basename` and `listing`. The +`basename` is the name of the directory, and the `listing` is the +contents, which consists of other File and Directory objects. Create `subdirs.cwl`: -- 2.30.2