X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/4d3a1c5346c99da360af58f1932b8f7a2ba87723..4b2ab09e3ee91cb63ae42a21d0efb004c053af8a:/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid diff --git a/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid b/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid index a4bad5820d..efcc8757fd 100644 --- a/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid +++ b/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid @@ -6,7 +6,7 @@ title: "Writing a multi-step pipeline" A pipeline in Arvados is a collection of crunch scripts, in which the output from one script may be used as the input to another script. -*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html* +{% include 'tutorial_expectations' %} This tutorial uses *@you@* to denote your username. Replace *@you@* with your user name in all the following examples. @@ -16,7 +16,7 @@ Our second script will filter the output of @hash.py@ and only include hashes th {% code '0_filter_py' as python %} -Now add it to git: +Now add it to your repository:
~/you/crunch_scripts$ chmod +x 0-filter.py
@@ -33,7 +33,7 @@ Next, create a file that contains the pipeline definition:
 
 
~/you/crunch_scripts$ cat >~/the_pipeline <<EOF
 {
-  "name":"Filter md5 hash values",
+  "name":"Filter MD5 hash values",
   "components":{
     "do_hash":{
       "script":"hash.py",
@@ -43,18 +43,20 @@ Next, create a file that contains the pipeline definition:
           "dataclass": "Collection"
         }
       },
-      "repository":"you",
-      "script_version":"master"
+      "repository":"$USER",
+      "script_version":"master",
+      "output_is_persistent":false
     },
-    "filter":{
+    "do_filter":{
       "script":"0-filter.py",
       "script_parameters":{
         "input":{
           "output_of":"do_hash"
         }
       },
-      "repository":"you",
-      "script_version":"master"
+      "repository":"$USER",
+      "script_version":"master",
+      "output_is_persistent":true
     }
   }
 }
@@ -62,14 +64,17 @@ EOF
 
-* @"output_of"@ indicates that the @input@ of the @do_hash@ component is connected to the @output@ of @filter@. This is a _dependency_. Arvados uses the dependencies between jobs to automatically determine the correct order to run the jobs. +* @"output_of"@ indicates that the @output@ of the @do_hash@ component should be used as the @"input"@ of @do_filter@. Arvados uses these dependencies between jobs to automatically determine the correct order to run them. -Now, use @arv pipeline_template create@ tell Arvados about your pipeline template: +(Your shell should automatically fill in @$USER@ with your login name. The JSON that gets saved should have @"repository"@ pointed at your personal Git repository.) + +Now, use @arv pipeline_template create@ to register your pipeline template in Arvados:
~/you/crunch_scripts$ arv pipeline_template create --pipeline-template "$(cat ~/the_pipeline)"
 
-Your new pipeline template will appear on the "Workbench %(rarr)→% Compute %(rarr)→% Pipeline templates":https://{{ site.arvados_workbench_host }}/pipeline_instances page. +Your new pipeline template will appear on the Workbench "Compute %(rarr)→% Pipeline templates":https://{{ site.arvados_workbench_host }}/pipeline_templates page. +For more information and examples for writing pipelines, see the "pipeline template reference":{{site.baseurl}}/api/schema/PipelineTemplate.html