X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/e4f96fc61cd9c85f91bdb0020bc365f2f4825ffb..77f1129ec53edffb5ed5a859106675cf262977e8:/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid diff --git a/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid b/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid index b09e624473..83303ad7cc 100644 --- a/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid +++ b/doc/user/tutorials/tutorial-new-pipeline.html.textile.liquid @@ -6,68 +6,22 @@ title: "Writing a multi-step pipeline" A pipeline in Arvados is a collection of crunch scripts, in which the output from one script may be used as the input to another script. -*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html* +{% include 'tutorial_expectations' %} -This tutorial uses *@you@* to denote your username. Replace *@you@* with your user name in all the following examples. -h2. Create a new script - -Our second script will filter the output of @hash.py@ and only include hashes that start with 0. Create a new script in ~/you/crunch_scripts/ called @0-filter.py@: - - {% code '0_filter_py' as python %} - -Now add it to git: - - -
~/you/crunch_scripts$ chmod +x 0-filter.py
-~/you/crunch_scripts$ git add 0-filter.py
-~/you/crunch_scripts$ git commit -m"zero filter"
-~/you/crunch_scripts$ git push origin master
-
-
- -h2. Create a pipeline template - -Next, create a file that contains the pipeline definition: +First, use @arv pipeline_template create@ to create a new empty template. The @--format=uuid@ option will print out the unique identifier for the new template: -
~/you/crunch_scripts$ cat >~/the_pipeline <<EOF
-{
-  "name":"Filter md5 hash values",
-  "components":{
-    "do_hash":{
-      "script":"hash.py",
-      "script_parameters":{
-        "input":{
-          "required": true,
-          "dataclass": "Collection"
-        }
-      },
-      "script_version":"you:master"
-    },
-    "filter":{
-      "script":"0-filter.py",
-      "script_parameters":{
-        "input":{
-          "output_of":"do_hash"
-        }
-      },
-      "script_version":"you:master"
-    }
-  }
-}
-EOF
+
~$ arv --format=uuid pipeline_template create --pipeline-template '{}'
+qr1hi-p5p6p-wt1vdhkezgx7g2k
 
-* @"output_of"@ indicates that the @input@ of the @do_hash@ component is connected to the @output@ of @filter@. This is a _dependency_. Arvados uses the dependencies between jobs to automatically determine the correct order to run the jobs. +Next, use @arv edit@ to edit the template. This will open the template record in an interactive text editor (as specified by $EDITOR or $VISUAL, otherwise defaults to @nano@). Replace the empty fields with the following content: -Now, use @arv pipeline_template create@ tell Arvados about your pipeline template: +{% code 'tutorial_bwa_pipeline' as javascript %} - -
~/you/crunch_scripts$ arv pipeline_template create --pipeline-template "$(cat ~/the_pipeline)"
-
-
-Your new pipeline template will appear on the "Workbench %(rarr)→% Compute %(rarr)→% Pipeline templates":http://{{ site.arvados_workbench_host }}/pipeline_instances page. +Your new pipeline template will appear on the Workbench "Pipeline templates":https://{{ site.arvados_workbench_host }}/pipeline_templates page. +For more information and examples for writing pipelines, see the "pipeline template reference":{{site.baseurl}}/api/schema/PipelineTemplate.html