---
layout: default
navsection: userguide
-title: "Using Crunch to run external programs"
+title: "Writing a pipeline template"
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
-This tutorial demonstrates how to use Crunch to run an external program by writting a wrapper using the Python SDK.
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+{% include 'pipeline_deprecation_notice' %}
+
+This tutorial demonstrates how to construct a two stage pipeline template that uses the "bwa mem":http://bio-bwa.sourceforge.net/ tool to produce a "Sequence Alignment/Map (SAM)":https://samtools.github.io/ file, then uses the "Picard SortSam tool":http://picard.sourceforge.net/command-line-overview.shtml#SortSam to produce a BAM (Binary Alignment/Map) file.
{% include 'tutorial_expectations' %}
-In this tutorial, you will use the external program @md5sum@ to compute hashes instead of the built-in Python library used in earlier tutorials.
+Use the following command to create an empty template using @arv create pipeline_template@:
+
+<notextile>
+<pre><code>~$ <span class="userinput">arv create pipeline_template</span></code></pre>
+</notextile>
-Start by entering the @crunch_scripts@ directory of your Git working tree:
+This will open the template record in an interactive text editor (as specified by $EDITOR or $VISUAL, otherwise defaults to @nano@). Now, update the contents of the editor with the following content:
+
+<notextile>{% code 'tutorial_bwa_sortsam_pipeline' as javascript %}</notextile>
+
+* @"name"@ is a human-readable name for the pipeline.
+* @"components"@ is a set of scripts or commands that make up the pipeline. Each component is given an identifier (@"bwa-mem"@ and @"SortSam"@) in this example).
+** Each entry in components @"components"@ is an Arvados job submission. For more information about individual jobs, see the "job resource reference.":{{site.baseurl}}/api/methods/jobs.html
+* @"repository"@, @"script_version"@, and @"script"@ indicate that we intend to use the external @"run-command"@ tool wrapper that is part of the Arvados. These parameters are described in more detail in "Writing a script":tutorial-firstscript.html.
+* @"runtime_constraints"@ describes runtime resource requirements for the component.
+** @"docker_image"@ specifies the "Docker":https://www.docker.com/ runtime environment in which to run the job. The Docker image @"bcosc/arv-base-java"@ supplied here has the Java runtime environment, bwa, and samtools installed.
+** @"arvados_sdk_version"@ specifies a version of the Arvados SDK to load alongside the job's script. The example uses 'master'. If you would like to use a specific version of the sdk, you can find it in the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
+* @"script_parameters"@ describes the component parameters.
+** @"command"@ is the actual command line to invoke the @bwa@ and then @SortSam@. The notation @$()@ denotes macro substitution commands evaluated by the run-command tool wrapper.
+** @"task.stdout"@ indicates that the output of this command should be captured to a file.
+** @$(node.cores)@ evaluates to the number of cores available on the compute node at time the command is run.
+** @$(tmpdir)@ evaluates to the local path for temporary directory the command should use for scratch data.
+** @$(reference_collection)@ evaluates to the script_parameter @"reference_collection"@
+** @$(dir $(...))@ constructs a local path to a directory representing the supplied Arvados collection.
+** @$(file $(...))@ constructs a local path to a given file within the supplied Arvados collection.
+** @$(glob $(...))@ searches the specified path based on a file glob pattern and evalutes to the first result.
+** @$(basename $(...))@ evaluates to the supplied path with leading path portion and trailing filename extensions stripped
+* @"output_of"@ indicates that the @output@ of the @bwa-mem@ component should be used as the @"input"@ script parameter of @SortSam@. Arvados uses these dependencies between components to automatically determine the correct order to run them.
+
+When using @run-command@, the tool should write its output to the current working directory. The output will be automatically uploaded to Keep when the job completes.
+
+See the "run-command reference":{{site.baseurl}}/user/topics/run-command.html for more information about using @run-command@.
+
+*Note:* When trying to get job reproducibility without re-computation, you need to set these parameters to their specific hashes. Using a version such as master in @"arvados_sdk_version"@ will grab the latest version hash, which will allow Arvados to re-compute your job if the sdk gets updated.
+* @"arvados_sdk_version"@ : The latest version can be found on the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
+* @"script_version"@ : The current version of your script in your git repository can be found by using the following command:
<notextile>
-<pre><code>~$ <span class="userinput">cd <b>you</b>/crunch_scripts</span>
-</code></pre>
+<pre><code>~$ <span class="userinput">git rev-parse HEAD</span></code></pre>
</notextile>
-Next, using @nano@ or your favorite Unix text editor, create a new file called @run-md5sum.py@ in the @crunch_scripts@ directory.
+* @"docker_image"@ : The docker image hash used is found on the "Collection page":https://playground.arvados.org/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address*.
-notextile. <pre>~/<b>you</b>/crunch_scripts$ <code class="userinput">nano run-md5sum.py</code></pre>
+h2. Running your pipeline
-Add the following code to use the @md5sum@ program to compute the hash of each file in a collection:
+Your new pipeline template should appear at the top of the Workbench "pipeline templates":{{site.arvados_workbench_host}}/pipeline_templates page. You can run your pipeline "using Workbench":tutorial-workflow-workbench.html or the "command line.":{{site.baseurl}}/user/topics/running-pipeline-command-line.html
-<notextile> {% code 'run_md5sum_py' as python %} </notextile>
+Test data is available in the "Arvados Tutorial":{{site.arvados_workbench_host}}/projects/qr1hi-j7d0g-u7zg1qdaowykd8d project:
-Make the file executable:
+* Choose <i class="fa fa-fw fa-archive"></i> "Tutorial chromosome 19 reference (2463fa9efeb75e099685528b3b9071e0+438)":{{site.arvados_workbench_host}}/collections/2463fa9efeb75e099685528b3b9071e0+438 for the "reference_collection" parameter
+* Choose <i class="fa fa-fw fa-archive"></i> "Tutorial sample exome (3229739b505d2b878b62aed09895a55a+142)":{{site.arvados_workbench_host}}/collections/3229739b505d2b878b62aed09895a55a+142 for the "sample" parameter
-notextile. <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">chmod +x run-md5sum.py</span></code></pre>
+For more information and examples for writing pipelines, see the "pipeline template reference":{{site.baseurl}}/api/methods/pipeline_templates.html
-Next, use Git to stage the file, commit, and push:
+h2. Re-using your pipeline run
-<notextile>
-<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">git add run-md5sum.py</span>
-~/<b>you</b>/crunch_scripts$ <span class="userinput">git commit -m"run external md5sum program"</span>
-~/<b>you</b>/crunch_scripts$ <span class="userinput">git push origin master</span>
-</code></pre>
-</notextile>
+Arvados allows users to re-use jobs that have the same inputs in order to save computing time and resources. Users are able to change a job downstream without re-computing earlier jobs. This section shows which version control parameters should be tuned to make sure Arvados will not re-compute your jobs.
-You should now be able to run your new script using Crunch, with @"script"@ referring to our new @run-md5sum.py@ script.
+Note: Job reuse can only happen if all input collections do not change.
+
+* @"arvados_sdk_version"@ : The arvados_sdk_version parameter is used to download the specific version of the Arvados sdk into the docker image. The latest version can be found in the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*. Make sure you set this to the same version as the previous run that you are trying to reuse.
+* @"script_version"@ : The script_version is the commit hash of the git branch that the crunch script resides in. This information can be found in your git repository by using the following command:
<notextile>
-<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cat >~/the_pipeline <<EOF
-{
- "name":"Run external md5sum program",
- "components":{
- "do_hash":{
- "script":"run-md5sum.py",
- "script_parameters":{
- "input":{
- "required": true,
- "dataclass": "Collection"
- }
- },
- "repository":"$USER",
- "script_version":"master"
- }
- }
-}
-EOF
-</span>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat ~/the_pipeline)"</span>
-</code></pre>
+<pre><code>~$ <span class="userinput">git rev-parse HEAD</span></code></pre>
</notextile>
-(Your shell should automatically fill in @$USER@ with your login name. The JSON that gets saved should have @"repository"@ pointed at your personal Git repository.)
-
-Your new pipeline template will appear on the Workbench "Compute %(rarr)→% Pipeline templates":https://{{ site.arvados_workbench_host }}/pipeline_templates page. You can run the "pipeline using Workbench":tutorial-pipeline-workbench.html.
+* @"docker_image"@ : This specifies the "Docker":https://www.docker.com/ runtime environment where jobs run their scripts. Docker version control is similar to git, and you can commit and push changes to your images. You must re-use the docker image hash from the previous run to use the same image. It can be found on the "Collection page":https://playground.arvados.org/collections/qr1hi-4zz18-dov6im679g3jr1n as the *Content address* or the *docker_image_locator* in a job's metadata.
projects / arvados.git / blobdiff