4 title: "Writing a multi-step pipeline"
7 A pipeline in Arvados is a collection of crunch scripts, in which the output from one script may be used as the input to another script.
9 *This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html*
11 This tutorial uses *@you@* to denote your username. Replace *@you@* with your user name in all the following examples.
13 h2. Create a new script
15 Our second script will filter the output of @hash.py@ and only include hashes that start with 0. Create a new script in <notextile><code>~/<b>you</b>/crunch_scripts/</code></notextile> called @0-filter.py@:
17 <notextile> {% code '0_filter_py' as python %} </notextile>
22 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">chmod +x 0-filter.py</span>
23 ~/<b>you</b>/crunch_scripts$ <span class="userinput">git add 0-filter.py</span>
24 ~/<b>you</b>/crunch_scripts$ <span class="userinput">git commit -m"zero filter"</span>
25 ~/<b>you</b>/crunch_scripts$ <span class="userinput">git push origin master</span>
29 h2. Create a pipeline template
31 Next, create a file that contains the pipeline definition:
34 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cat >~/the_pipeline <<EOF
36 "name":"Filter md5 hash values",
43 "dataclass": "Collection"
46 "script_version":"<b>you</b>:master",
47 "output_is_persistent":false
50 "script":"0-filter.py",
56 "script_version":"<b>you</b>:master",
57 "output_is_persistent":true
65 * @"output_of"@ indicates that the @output@ of the @do_hash@ component should be used as the @"input"@ parameter for the @filter@ component. Arvados determines the correct order to run the jobs when such dependencies are present.
67 Now, use @arv pipeline_template create@ tell Arvados about your pipeline template:
70 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv pipeline_template create --pipeline-template "$(cat ~/the_pipeline)"</span>
74 Your new pipeline template will appear on the "Workbench %(rarr)→% Compute %(rarr)→% Pipeline templates":http://{{ site.arvados_workbench_host }}/pipeline_templates page.