Below are diagrams of the main workflow which runs the processing across multiple sets of fastq and the main subworkflow (run multiple times in parallel by the main workflow) which processes a single set of FASTQs. This main subworkflow also calls other additional subworkflows including subworkflows that perform variant calling using GATK in parallel by regions and generate the ClinVar HTML variant report. These CWL diagrams (generated using "CWL viewer":https://view.commonwl.org) will give you a basic idea of the flow, input/outputs and workflow steps involved in the tutorial example. However, if you aren’t used to looking at CWL workflow diagrams and/or aren’t particularly interested in this level of detail, do not worry. You will not need to know these particulars to run the workflow.
-!{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image2.png!
+<figure> !{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image2.png!
+<figcaption> _*Figure 1*: Main CWL Workflow for WGS Processing Tutorial. This runs the same WGS subworkflow over multiple pairs FASTQs files._ </figcaption> </figure>
-_*Figure 1*: Main CWL Workflow for WGS Processing Tutorial. This runs the same WGS subworkflow over multiple pairs FASTQs files._
-
-!{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image3.png!
-
-_*Figure 2*: Main subworkflow for the WGS Processing Tutorial. This subworkflow does alignment, reduplication, variant calling and reporting._
+<figure> !{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image3.png!
+<figcaption> _*Figure 2*: Main subworkflow for the WGS Processing Tutorial. This subworkflow does alignment, deduplication, variant calling and reporting._ </figcaption> </figure>
_Ways to Learn More About CWL_
To create a new project, go to the Projects dropdown menu and select “Add a New Project”.
-!{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image4.png!
-
-_*Figure 3*: Screenshot of adding a new project using Arvados Workbench._
-
+<figure> !{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image4.png!
+<figcaption> _*Figure 3*: Adding a new project using Arvados Workbench._ </figcaption> </figure>
-Let’s name your project “WGS Processing Tutorial”. You can also add a description of your project using the Edit button. The universally unique identifier (UUID) of the project can be found in the URL.
+Let’s name your project “WGS Processing Tutorial”. You can also add a description of your project using the *Edit* button. The universally unique identifier (UUID) of the project can be found in the URL.
-!{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image6.png!
-
-_*Figure 4*: Screenshot of renaming new project using Arvados Workbench. The UUID of the project can be found in the URL and is highlighted in yellow in this image for emphasis._
+<figure> !{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image6.png!
+<figcaption> _*Figure 4*: Renaming new project using Arvados Workbench. The UUID of the project can be found in the URL and is highlighted in yellow in this image for emphasis._ </figcaption> </figure>
If you choose to use another name for your project, just keep in mind when the project name is referenced in the walkthrough later on.
Arvados uses a content-addressable filesystem (i.e. Keep) where the addresses of files are derived from their contents. A major benefit of this is that Arvados can then verify that when a dataset is retrieved it is the dataset you requested and can track the exact datasets that were used for each of our previous calculations. This is what allows you to be certain that we are always working with the data that you think you are using. You use the content address of a collection when you want to guarantee that you use the same version as input to your workflow.
-!{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image1.png!
-
-_*Figure 5*: Screenshot of a collection in Arvados as viewed via the Arvados Workbench. On the upper left you will find a panel that contains: the name of the collection (editable), a description of the collection (editable), the collection UUID and the content address and content size._
+<figure> !{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image1.png!
+<figcaption> _*Figure 5*: A collection in Arvados as viewed via the Arvados Workbench. On the upper left you will find a panel that contains: the name of the collection (editable), a description of the collection (editable), the collection UUID and the content address and content size._ </figcaption> </figure>
Let’s start working with collections by copying the existing collection that stores the FASTQ data being processed into our new “WGS Processing Tutorial” project.
In this case it is called “PGP UK FASTQs” and by searching for it in the “search this site” box. It will come up and you can navigate to it. You would do similarly if you would want to search by UUID or content address.
-Now that you have found the collection of FASTQs you want to copy to your project, you can simply use the Copy to project... button and select your new project to copy the collection there. You can rename your collection whatever you wish, or use the default name on copy and add whatever description you would like.
+Now that you have found the collection of FASTQs you want to copy to your project, you can simply use the <span class="btn btn-sm btn-primary" >Copy to project...</span> button and select your new project to copy the collection there. You can rename your collection whatever you wish, or use the default name on copy and add whatever description you would like.
-We want to do the same thing for the other inputs to our WGS workflow. Similar to the “PGP UK FASTQs” collection there is a collection of inputs entitled “WGS Tutorial References” and that collection can be copied over in a similar fashion.
+We want to do the same thing for the other inputs to our WGS workflow. Similar to the “PGP UK FASTQs” collection there is a collection of inputs entitled “WGS Processing reference data” and that collection can be copied over in a similar fashion.
Now that we are a bit more familiar with the Arvados Workbench, projects and collections. Let’s move onto running a workflow.
h3. 4a. Interactively Running a Workflow Using Workbench
-Workflows can be registered in Arvados. Registration allows you to share a workflow with other Arvados users, and let’s them run the workflow by clicking the Run a process… button on the Workbench Dashboard and on the command line by specifying the workflow UUID. Default values can be specified for workflow inputs.
+Workflows can be registered in Arvados. Registration allows you to share a workflow with other Arvados users, and let’s them run the workflow by clicking the <span class="btn btn-sm btn-primary" >Run a process…</span> button on the Workbench Dashboard and on the command line by specifying the workflow UUID. Default values can be specified for workflow inputs.
We have already previously registered the WGS workflow and set default input values for this set of the walkthrough.
Let’s find the the registered WGS Processing Workflow and run it interactively in our newly created project.
-* To find the registered workflow, you can search for it in the search box located in the top right corner of the Arvados Workbench by looking for the name “WGS Processing Workflow”.
-* Once you have found the registered workflow, you can run it your project by using the Run this workflow.. button and selecting your New Project you set up in Section 3a.
-* Default inputs to the registered workflow will automatically be filled in. These inputs will still work. You can verify this by checking the addresses of the collections you copied over to your New Project. However, if you wanted to change the inputs to the workflow, this is where you would do it.
-* Now, you can submit your workflow by hitting the Run button.
+# To find the registered workflow, you can search for it in the search box located in the top right corner of the Arvados Workbench by looking for the name “WGS Processing Workflow”.
+# Once you have found the registered workflow, you can run it your project by using the <span class="btn btn-sm btn-primary" >Run this workflow..</span> button and selecting your project ("WGS Processing Tutorial") that you set up in Section 3a.
+# Default inputs to the registered workflow will be automatically filled in. These inputs will still work. You can verify this by checking the addresses of the collections you copied over to your New Project.
+# The input *Directory of paired FASTQ files* will need to be set. Click on <span class="btn btn-sm btn-primary" >Choose</span> button, select "PGP UK FASTQs" in the *Choose a dataset* dialog and then click <span class="btn btn-sm btn-primary" >OK</span>.
+# Now, you can submit your workflow by scrolling to the bottom of the page and hitting the <span class="btn btn-sm btn-primary" >Run</span> button.
Congratulations! You have now submitted your workflow to run. You can move to Section 5 to learn how to check the state of your submitted workflow and Section 6 to learn how to examine the results of and logs from your workflow.
Arvados provides a virtual machine which has all the necessary client-side libraries installed to submit to your Arvados cluster using the command line. Webshell gives you access to an Arvados Virtual Machine (VM) from your browser with no additional setup. You can access webshell through the Arvados Workbench. It is the easiest way to try out submitting a workflow to Arvados via the command line.
-To get access to webshell on the Arvados Playground, you need to contact a Curii Arvados Playground Administrator to get access to an Arvados shell node by emailing "support@curii.com.":mailto:support@curii.com
+To get access to webshell on the Arvados Playground, you need to contact a Curii Arvados Playground Administrator to get access to an Arvados shell node by emailing "info@curii.com.":mailto:info@curii.com
Once you receive an email letting you know your access has been set up and you should be able to access the shell virtual machine. You can follow the instructions here to access the machine using the browser (also known as using webshell):
-* "{{ site.baseurl }}/user/getting_started/vm-login-with-webshell.html":{{ site.baseurl }}/user/getting_started/vm-login-with-webshell.html
+* "Accessing an Arvados VM with Webshell":{{ site.baseurl }}/user/getting_started/vm-login-with-webshell.html
Arvados also allows you to ssh into the shell machine and other hosted VMs instead of using the webshell capabilities. However this tutorial does not cover that option in-depth. If you like to explore it on your own, you can allow the instructions in the documentation here:
-* "{{ site.baseurl }}/user/getting_started/ssh-access-unix.html":{{ site.baseurl }}/user/getting_started/ssh-access-unix.html
-(Unix)
-* "{{ site.baseurl }}/user/getting_started/ssh-access-windows.html (Windows)":{{ site.baseurl }}/user/getting_started/ssh-access-windows.html (Windows)
+* "Accessing an Arvados VM with SSH - Unix Environments":{{ site.baseurl }}/user/getting_started/ssh-access-unix.html
+* "Accessing an Arvados VM with SSH - Windows Environments":{{ site.baseurl }}/user/getting_started/ssh-access-windows.html
Once you can use webshell, you can proceed to section *“4d. Running a Workflow Using the Command Line”* .
To be able to submit workflows to the Arvados cluster, you will need to install the Python SDK on your machine. Additional features can be made available by installing additional libraries, but this is the bare minimum you need to install to do this walkthrough tutorial. You can follow the instructions in the Arvados documentment to install the Python SDK and set the appropriate configurations to access the Arvados Playground.
-* Installing the Python SDK: "{{ site.baseurl }}/sdk/python/sdk-python.html":{{ site.baseurl }}/sdk/python/sdk-python.html
-** Note: You will only need to complete through the end of the Test section.
-* Setting Configurations to Access the Arvados Playground:
-"{{ site.baseurl }}/user/reference/api-tokens.html":{{ site.baseurl }}/user/reference/api-tokens.html
+* "Installing the Arvados CWL Runner":{{ site.baseurl }}/sdk/python/arvados-cwl-runner.html
+* "Setting Configurations to Access the Arvados Playground":{{ site.baseurl }}/user/reference/api-tokens.html
Once you have your machine set up to submit to the Arvados Playground Cluster, you can proceed to section *“4d. Running a Workflow Using the Command Line”* .
Common states you will see are as follows:
-* Queued - Workflow or step is waiting to run
-* Running or Active - Workflow is currently running
-* Complete - Workflow or step has successfully completed
-* Failed - Workflow or step has not successfully completed
-* Failing -- Workflow is running but has steps that have failed
-* Cancelled - Workflow or step has been either manually cancelled or has been canceled by Arvados due to an Issue
+* <span class="label label-default">Queued</span> - Workflow or step is waiting to run
+* <span class="label label-info">Running</span> or <span class="label label-info">Active</span> - Workflow is currently running
+* <span class="label label-success">Complete</span> - Workflow or step has successfully completed
+* <span class="label label-warning">Failing</span> - Workflow is running but has steps that have failed
+* <span class="label label-danger">Failed</span> - Workflow or step did not complete successfully
+* <span class="label label-danger">Cancelled</span> - Workflow or step was either manually cancelled or was canceled by Arvados due to a system error
Since Arvados Crunch reuses steps and workflows if possible, this workflow should run relatively quickly since this workflow has been run before and you have access to those previously run steps. You may notice an initial period where the top level job shows the option of canceling while the other steps are filled in with already finished steps.
Once your workflow has finished, you can see how long it took the workflow to run, see scaling information, and examine the logs and outputs. Outputs will be only available for steps that have been successfully completed. Outputs will be saved for every step in the workflow and be saved for the workflow itself. Outputs are saved in collections. You can access each collection by clicking on the link corresponding to the output.
-!{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image5.png!
-
-_*Figure 6*: Screenshot of a completed workflow process in Arvados as viewed via the Arvados Workbench. You can click on the outputs link (highlighted in yellow) to view the outputs. Outputs of a workflow are stored in a collection._
+<figure> !{width: 100%}{{ site.baseurl }}/images/wgs-tutorial/image5.png!
+<figcaption> _*Figure 6*: A completed workflow process in Arvados as viewed via the Arvados Workbench. You can click on the outputs link (highlighted in yellow) to view the outputs. Outputs of a workflow are stored in a collection._ </figcaption> </figure>
If we click on the outputs of the workflow, we will see the output collection.
** Contains information using Arvados Keep on the node executing the container
* @hoststat.txt@
** Contains about resource consumption (RAM, cpu, disk, network) on the node while it was running
-
This is different from the log crunchstat.txt because it includes resource consumption of Arvados components that run on the node outside the container such as crunch-run and other processes related to the Keep file system.
-For the highest level logs, the logs are tracking the container that ran the arvados-cwl-runner process which you can think of as the “mastermind” behind tracking which parts of the CWL workflow need to be run when, which have been run already, what order they need to be run, which can be run simultaneously, and so forth and then sending out the related container requests. Each step then has their own logs related to containers running a CWL step of the workflow including a log of standard error that contains the standard error of the code run in that CWL step. Those logs can be found by expanding the steps and clicking on the link to the log collection.
+For the highest level logs, the logs are tracking the container that ran the @arvados-cwl-runner@ process which you can think of as the “mastermind” behind tracking which parts of the CWL workflow need to be run when, which have been run already, what order they need to be run, which can be run simultaneously, and so forth and then sending out the related container requests. Each step then has their own logs related to containers running a CWL step of the workflow including a log of standard error that contains the standard error of the code run in that CWL step. Those logs can be found by expanding the steps and clicking on the link to the log collection.
-Let’s take a peek at a few of these logs to get you more familiar with them. First, we can look at the stderr.txt of the highest level process. Again recall this should be of the “mastermind” arvados-cwl-runner process. You can click on the log to download it to your local machine, and when you look at the contents - you should see something like the following...
+Let’s take a peek at a few of these logs to get you more familiar with them. First, we can look at the @stderr.txt@ of the highest level process. Again recall this should be of the “mastermind” @arvados-cwl-runner@ process. You can click on the log to download it to your local machine, and when you look at the contents - you should see something like the following...
<pre><code>2020-06-22T20:30:04.737703197Z INFO /usr/bin/arvados-cwl-runner 2.0.3, arvados-python-client 2.0.3, cwltool 1.0.20190831161204
2020-06-22T20:30:04.743250012Z INFO Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'
You can see the output of all the work that arvados-cwl-runner does by managing the execution of the CWL workflow and all the underlying steps and subworkflows.
-Now, let’s explore the logs for a step in the workflow. Remember that those logs can be found by expanding the steps and clicking on the link to the log collection. Let’s look at the log for the step that does the alignment. That step is named bwamem-samtools-view. We can see there are 10 of them because we are aligning 10 genomes. Let’s look at bwamem-samtools-view2.
+Now, let’s explore the logs for a step in the workflow. Remember that those logs can be found by expanding the steps and clicking on the link to the log collection. Let’s look at the log for the step that does the alignment. That step is named bwamem-samtools-view. We can see there are 10 of them because we are aligning 10 genomes. Let’s look at *bwamem-samtools-view2.*
We click the arrow to open up the step, and then can click on the log collection to access the logs. You may notice there are two sets of seemingly identical logs. One listed under a directory named for a container and one up in the main directory. This is done in case your step had to be automatically re-run due to any issues and gives the logs of each re-run. The logs in the main directory are the logs for the successful run. In most cases this does not happen, you will just see one directory and one those logs will match the logs in the main directory. Let’s open the logs labeled node-info.txt and stderr.txt.
Thank you for working through this walkthrough tutorial. Hopefully this tutorial has helped you get a feel for working with Arvados. This tutorial just covered the basic capabilities of Arvados. There are many more capabilities to explore. Please see the links featured at the end of Section 1 for ways to learn more about Arvados or get help while you are working with Arvados.
-If you would like help setting up your own production instance of Arvados, please contact us at "info@curii.com.":info@curii.com
+If you would like help setting up your own production instance of Arvados, please contact us at "info@curii.com.":mailto:info@curii.com
</div>
projects / arvados.git / commitdiff