X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/715869b9a22e22ac68a7dbefa96f27150017f75d..3101fea587bc08e0f4f00a583d1d9e2c953417c1:/doc/user/tutorials/running-external-program.html.textile.liquid
diff --git a/doc/user/tutorials/running-external-program.html.textile.liquid b/doc/user/tutorials/running-external-program.html.textile.liquid
index e286013247..3ed6ea645b 100644
--- a/doc/user/tutorials/running-external-program.html.textile.liquid
+++ b/doc/user/tutorials/running-external-program.html.textile.liquid
@@ -1,67 +1,53 @@
---
layout: default
navsection: userguide
-navmenu: Tutorials
-title: "Running external programs"
-
+title: "Writing a pipeline template"
...
-h1. Running external programs
-
-This tutorial demonstrates how to use Crunch to run an external program by writting a wrapper using the Python SDK.
-
-*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.basedoc}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.basedoc}}/user/getting_started/check-environment.html*
+This tutorial demonstrates how to construct a two stage pipeline template that uses the "bwa mem":http://bio-bwa.sourceforge.net/ tool to produce a "Sequence Alignment/Map (SAM)":https://samtools.github.io/ file, then uses the "Picard SortSam tool":http://picard.sourceforge.net/command-line-overview.shtml#SortSam to produce a BAM (Binary Alignment/Map) file.
-In this tutorial, you will use the external program @md5sum@ to compute hashes instead of the built-in Python library used in earlier tutorials.
+{% include 'tutorial_expectations' %}
-Start by entering the @crunch_scripts@ directory of your git repository:
+Use the following command to create an empty template using @arv create pipeline_template@:
+$ cd you/crunch_scripts
-
~$ arv create pipeline_template
{% include 'run_md5sum_py' %}
+This will open the template record in an interactive text editor (as specified by $EDITOR or $VISUAL, otherwise defaults to @nano@). Now, update the contents of the editor with the following content:
-Make the file executable:
+$ chmod +x run-md5sum.py
+* @"name"@ is a human-readable name for the pipeline.
+* @"components"@ is a set of scripts or commands that make up the pipeline. Each component is given an identifier (@"bwa-mem"@ and @"SortSam"@) in this example).
+** Each entry in components @"components"@ is an Arvados job submission. For more information about individual jobs, see the "job object reference":{{site.baseurl}}/api/schema/Job.html and "job create method.":{{site.baseurl}}/api/methods/jobs.html#create
+* @"repository"@, @"script_version"@, and @"script"@ indicate that we intend to use the external @"run-command"@ tool wrapper that is part of the Arvados. These parameters are described in more detail in "Writing a script":tutorial-firstscript.html.
+* @"runtime_constraints"@ describes runtime resource requirements for the component.
+** @"docker_image"@ specifies the "Docker":https://www.docker.com/ runtime environment in which to run the job. The Docker image @"bcosc/arv-base-java"@ supplied here has the Java runtime environment, bwa, and samtools installed.
+** @"arvados_sdk_version"@ specifies a version of the Arvados SDK to load alongside the job's script.
+* @"script_parameters"@ describes the component parameters.
+** @"command"@ is the actual command line to invoke the @bwa@ and then @SortSam@. The notation @$()@ denotes macro substitution commands evaluated by the run-command tool wrapper.
+** @"task.stdout"@ indicates that the output of this command should be captured to a file.
+** @$(node.cores)@ evaluates to the number of cores available on the compute node at time the command is run.
+** @$(tmpdir)@ evaluates to the local path for temporary directory the command should use for scratch data.
+** @$(reference_collection)@ evaluates to the script_parameter @"reference_collection"@
+** @$(dir $(...))@ constructs a local path to a directory representing the supplied Arvados collection.
+** @$(file $(...))@ constructs a local path to a given file within the supplied Arvados collection.
+** @$(glob $(...))@ searches the specified path based on a file glob pattern and evalutes to the first result.
+** @$(basename $(...))@ evaluates to the supplied path with leading path portion and trailing filename extensions stripped
+* @"output_of"@ indicates that the @output@ of the @bwa-mem@ component should be used as the @"input"@ script parameter of @SortSam@. Arvados uses these dependencies between components to automatically determine the correct order to run them.
-Next, add the file to @git@ staging, commit and push:
+When using @run-command@, the tool should write its output to the current working directory. The output will be automatically uploaded to Keep when the job completes.
-$ git add run-md5sum.py
-$ git commit -m"run external md5sum program"
-$ git push origin master
-
-$ cat >the_job <<EOF
-{
- "script": "run-md5sum.py",
- "script_version": "you:master",
- "script_parameters":
- {
- "input": "c1bad4b39ca5a924e481008009d94e32+210"
- }
-}
-EOF
-$ arv -h job create --job "$(cat the_job)"
-{
- ...
- "uuid":"qr1hi-xxxxx-xxxxxxxxxxxxxxx"
- ...
-}
-$ arv -h job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx
-{
- ...
- "output":"4d164b1658c261b9afc6b479130016a3+54",
- ...
-}
-
-