---
layout: default
navsection: userguide
navmenu: Tutorials
title: "Writing a pipeline"
...

In this tutorial, we will write the "hash" script demonstrated in the first tutorial.

*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html*

This tutorial uses *@you@* to denote your username.  Replace *@you@* with your user name in all the following examples.

h2. Setting up Git

As discussed in the previous tutorial, all Crunch scripts are managed through the @git@ revision control system.

First, you should do some basic configuration for git (you only need to do this the first time):

<notextile>
<pre><code>~$ <span class="userinput">git config --global user.name "Your Name"</span>
~$ <span class="userinput">git config --global user.email <b>you</b>@example.com</span></code></pre>
</notextile>

On the Arvados Workbench, navigate to "Compute %(rarr)&rarr;% Code repositories":http://{{site.arvados_workbench_host}}/repositiories .  You should see a repository with your user name listed in the *name* column.  Next to *name* is the column *push_url*.  Copy the *push_url* value associated with your repository.  This should look like <notextile><code>git@git.{{ site.arvados_api_host }}:<b>you</b>.git</code></notextile>.

Next, on the Arvados virtual machine, clone your git repository:

<notextile>
<pre><code>~$ <span class="userinput">git clone git@git.{{ site.arvados_api_host }}:<b>you</b>.git</span>
Cloning into '<b>you</b>'...</code></pre>
</notextile>

This will create an git checkout in the directory called *@you@*.

{% include 'notebox_begin' %}
For more information about using @git@, try

notextile. <pre><code>$ <span class="userinput">man gittutorial</span></code></pre>

or <b>"click here to search Google for git tutorials":http://google.com/#q=git+tutorial</b>
{% include 'notebox_end' %}

h2. Creating a Crunch script

Start by entering the *@you@* directory created by @git clone@.  Next create a subdirectory called @crunch_scripts@ and change to that directory:

<notextile>
<pre><code>~$ <span class="userinput">cd <b>you</b></span>
~/<b>you</b>$ <span class="userinput">mkdir crunch_scripts</span>
~/<b>you</b>$ <span class="userinput">cd crunch_scripts</span></code></pre>
</notextile>

Next, using @nano@ or your favorite Unix text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory.

notextile. <pre>~/<b>you</b>/crunch_scripts$ <code class="userinput">nano hash.py</code></pre>

Add the following code to compute the md5 hash of each file in a collection:

<notextile> {% code 'tutorial_hash_script_py' as python %} </notextile>

Make the file executable:

notextile. <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">chmod +x hash.py</span></code></pre>

{% include 'notebox_begin' %}
The steps below describe how to execute the script after committing changes to git. To test the script locally, please see the "debugging a crunch script":{{site.baseurl}}/user/topics/tutorial-job-debug.html page.

{% include 'notebox_end' %}

Next, add the file to @git@ staging.  This tells @git@ that the file should be included on the next commit.

notextile. <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">git add hash.py</span></code></pre>

Next, commit your changes to git.  All staged changes are recorded into the local @git@ repository:

<notextile>
<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">git commit -m"my first script"</span>
[master (root-commit) 27fd88b] my first script
 1 file changed, 33 insertions(+)
 create mode 100755 crunch_scripts/hash.py</code></pre>
</notextile>

Finally, upload your changes to the Arvados server:

<notextile>
<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">git push origin master</span>
Counting objects: 4, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 682 bytes, done.
Total 4 (delta 0), reused 0 (delta 0)
To git@git.qr1hi.arvadosapi.com:you.git
 * [new branch]      master -> master</code></pre>
</notextile>

h2. Create a pipeline template

Next, create a file that contains the pipeline definition:

<notextile>
<pre><code>$ <span class="userinput">cat &gt;the_pipeline &lt;&lt;EOF
{
  "name":"My first pipeline",
  "components":{
    "do_hash":{
      "script":"hash.py",
      "script_parameters":{
        "input":{
          "required": true,
          "dataclass": "Collection"
        }
      },
      "script_version":"you:master"
    }
  }
}
EOF
</span></code></pre>
</notextile>

* @cat@ is a standard Unix utility that simply copies standard input to standard output
* @<<EOF@ tells the shell to direct the following lines into the standard input for @cat@ up until it sees the line @EOF@
* @>the_job@ redirects standard output to a file called @the_job@
* @"name"@ is a human-readable name for the pipeline
* @"components"@ is a set of scripts that make up the pipeline
* The component is listed with a human-readable name (@"do_hash"@ in this example)
* @"script"@ specifies the name of the script to run.  The script is searched for in the "crunch_scripts/" subdirectory of the @git@ checkout specified by @"script_version"@.
* @"script_version"@ specifies the version of the script that you wish to run.  This can be in the form of an explicit @git@ revision hash, or in the form "repository:branch" (in which case it will take the HEAD of the specified branch).  Arvados logs the script version that was used in the run, enabling you to go back and re-run any past job with the guarantee that the exact same code will be used as was used in the previous run.  You can access a list of available @git@ repositories on the Arvados workbench under "Compute %(rarr)&rarr;% Code repositories":http://{{site.arvados_workbench_host}}/repositiories .
* @"script_parameters"@ describes the parameters for the script.  In this example, there is one parameter called @input@ which is @required@ and is a @Collection@.

Now, use @arv pipeline_template create@ tell Arvados about your pipeline template:

<notextile>
<pre><code>$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat the_pipeline)"</span>
</code></pre>
</notextile>

Your new pipeline template will appear on the "Workbench %(rarr)&rarr;% Compute %(rarr)&rarr;% Pipeline&nbsp;templates":http://{{ site.arvados_workbench_host }}/pipeline_instances page.  You can run the "pipeline using workbench":tutorial-pipeline-workbench.html