--- layout: default navsection: userguide navmenu: Tutorials title: "Running on an Arvados cluster" ... This tutorial demonstrates how to create a pipeline to run your crunch script on an Arvados cluster. Cluster jobs can scale out to multiple nodes, and use @git@ and @docker@ to store the complete system snapshot required to achieve reproducibilty. {% include 'tutorial_expectations' %} This tutorial uses @$USER@ to denote your username. Replace @$USER@ with your user name in all the following examples. h2. Setting up Git All Crunch scripts are managed through the Git revision control system. Before you start using Git, you should do some basic configuration (you only need to do this the first time):
~$ git config --global user.name "Your Name"
~$ git config --global user.email $USER@example.com
On the Arvados Workbench, navigate to "Code repositories":https://{{site.arvados_workbench_host}}/repositories. You should see a repository with your user name listed in the *name* column. Next to *name* is the column *push_url*. Copy the *push_url* value associated with your repository. This should look like git@git.{{ site.arvados_api_host }}:$USER.git. Next, on the Arvados virtual machine, clone your Git repository:
~$ cd $HOME # (or wherever you want to install)
~$ git clone git@git.{{ site.arvados_api_host }}:$USER.git
Cloning into '$USER'...
This will create a Git repository in the directory called @$USER@ in your home directory. Say yes when prompted to continue with connection. Ignore any warning that you are cloning an empty repository. {% include 'notebox_begin' %} For more information about using Git, try notextile.
$ man gittutorial
or *"search Google for Git tutorials":http://google.com/#q=git+tutorial*. {% include 'notebox_end' %} h2. Creating a Crunch script Start by entering the @$USER@ directory created by @git clone@. Next create a subdirectory called @crunch_scripts@ and change to that directory:
~$ cd $USER
~/$USER$ mkdir crunch_scripts
~/$USER$ cd crunch_scripts
Next, using @nano@ or your favorite Unix text editor, create a new file called @hash.py@ in the @crunch_scripts@ directory. notextile.
~/$USER/crunch_scripts$ nano hash.py
Add the following code to compute the MD5 hash of each file in a collection (if you already completed "Writing a Crunch script":tutorial-firstscript.html you can just copy the @hash.py@ file you created previously.) {% code 'tutorial_hash_script_py' as python %} Make the file executable: notextile.
~/$USER/crunch_scripts$ chmod +x hash.py
Next, add the file to the staging area. This tells @git@ that the file should be included on the next commit. notextile.
~/$USER/crunch_scripts$ git add hash.py
Next, commit your changes. All staged changes are recorded into the local git repository:
~/$USER/crunch_scripts$ git commit -m"my first script"
[master (root-commit) 27fd88b] my first script
 1 file changed, 45 insertions(+)
 create mode 100755 crunch_scripts/hash.py
Finally, upload your changes to the Arvados server:
~/$USER/crunch_scripts$ git push origin master
Counting objects: 4, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 682 bytes, done.
Total 4 (delta 0), reused 0 (delta 0)
To git@git.qr1hi.arvadosapi.com:$USER.git
 * [new branch]      master -> master
h2. Create a pipeline template Next, create a file that contains the pipeline definition:
~/$USER/crunch_scripts$ cd ~
~$ cat >the_pipeline <<EOF
{
  "name":"My md5 pipeline",
  "components":{
    "do_hash":{
      "script":"hash.py",
      "script_parameters":{
        "input":{
          "required": true,
          "dataclass": "Collection"
        }
      },
      "repository":"$USER",
      "script_version":"master",
      "runtime_constraints":{
        "docker_image":"arvados/jobs-java-bwa-samtools"
      }
    }
  }
}
EOF
* @"repository"@ is the name of a git repository to search for the script version. You can access a list of available git repositories on the Arvados Workbench under "Code repositories":https://{{site.arvados_workbench_host}}/repositories. * @"script_version"@ specifies the version of the script that you wish to run. This can be in the form of an explicit Git revision hash, a tag, or a branch (in which case it will use the HEAD of the specified branch). Arvados logs the script version that was used in the run, enabling you to go back and re-run any past job with the guarantee that the exact same code will be used as was used in the previous run. * @"script"@ specifies the filename of the script to run. Crunch expects to find this in the @crunch_scripts/@ subdirectory of the Git repository. Now, use @arv pipeline_template create@ to register your pipeline template in Arvados:
~$ arv pipeline_template create --pipeline-template "$(cat the_pipeline)"
h2. Running your pipeline Your new pipeline template should appear at the top of the Workbench "pipeline templates":https://{{ site.arvados_workbench_host }}/pipeline_templates page. You can run your pipeline "using Workbench":tutorial-pipeline-workbench.html or the "command line.":{{site.baseurl}}/user/topics/running-pipeline-command-line.html For more information and examples for writing pipelines, see the "pipeline template reference":{{site.baseurl}}/api/schema/PipelineTemplate.html