---
layout: default
navsection: userguide
title: "Debug a Crunch script"
navorder: 114
---

h1. Tutorial: Debug a Crunch script

To test changes to a script by running a job, the change must be pushed into @git@, the job queued asynchronously, and the actual execution may be run on any compute server.  As a result, debugging a script can be difficult and time consuming.  This tutorial demonstrates using @arv-crunch-job@ to run your job in your local VM.  This avoids the job queue and allows you to execute the script from your uncomitted git tree.

*This tutorial assumes that you are "logged into an Arvados VM instance":ssh-access.html#login, and have a "working environment.":check-environment.html*

This tutorial uses _you_ to denote your username.  Replace _you_ with your user name in all the following examples.

h2. Create a new script

Change to your git directory and create a new script in "crunch_scripts/".

<notextile>
<pre><code>$ <span class="userinput">cd you/crunch_scripts</span>
$ <span class="userinput">cat &gt;hello-world.py &lt;&lt;EOF
#!/usr/bin/env python

print "hello world"
EOF</span>
$ <span class="userinput">chmod +x hello-world.py</span>
</code></pre>
</notextile>

h2. Using arv-crunch-job to run the job in your VM

Instead of a git commit hash, we provide the path to the directory in the "script_version" parameter.  The script specified in "script" will actually be searched for in the "crunch_scripts/" subdirectory of the directory specified "script_version".  Although we are running the script locally, the script still requires access to the Arvados API server and Keep storage service, and the job will be recorded on then Arvados Workbench job history.

<notextile>
<pre><code>$ <span class="userinput">cat &gt;the_job &lt;&lt;EOF
{
 "script":"hello-world.py",
 "script_version":"/home/you/you",
 "script_parameters":{}
}
EOF</span>
$ <span class="userinput">arv-crunch-job --job "$(cat the_job)"</span>
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  check slurm allocation
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  node localhost - 1 slots
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  start
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  script hello-world.py
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  script_version /home/you/you
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  script_parameters {}
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  runtime_constraints {"max_tasks_per_node":0}
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  start level 0
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  status: 0 done, 0 running, 1 todo
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 job_task qr1hi-ot0gb-4zdajby8cjmlguh
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 child 29834 started on localhost.1
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827  status: 0 done, 1 running, 0 todo
2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 stderr hello world
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 child 29834 on localhost.1 exit 0 signal 0 success=
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 failure (#1, permanent) after 0 seconds
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 output 
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827  Every node has failed -- giving up on this round
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827  wait for last 0 children to finish
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827  status: 0 done, 0 running, 0 todo
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827  Freeze not implemented
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827  collate
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827  output d41d8cd98f00b204e9800998ecf8427e+0
2013-12-12_21:36:44 qr1hi-8i9sb-okzukfzkpbrnhst 29827  meta key is c00bfbd58e6f58ce3bebdd47f745a70f+1857+K@qr1hi
</code></pre>
</notextile>

This is the line of interest:

bc. 2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 stderr hello world

The script's printout is captured in the log, which is useful for print statement debugging.  However, although it the script returned a status code of 0 (success), the job failed.  Why?  For a job to complete successfully scripts must explicitly add their output to Keep, and then tell Arvados about it.  Here is a second try:

<notextile>
<pre><code>$ <span class="userinput">cat &gt;hello-world.py &lt;&lt;EOF
#!/usr/bin/env python

import arvados

# Create a new collection
out = arvados.CollectionWriter()

# Set the name of the file in the collection to write to
out.set_current_file_name('hello.txt')

# Actually output our text
out.write('hello world')

# Commit the collection to Keep
out_collection = out.finish()

# Tell Arvados which Keep object is our output
arvados.current_task().set_output(out_collection)

# Done!
EOF</span>
$ <span class="userinput">chmod +x hello-world-fixed.py</span>
$ <span class="userinput">cat &gt;the_job &lt;&lt;EOF
{
 "script":"hello-world-fixed.py",
 "script_version":"/home/you/you",
 "script_parameters":{}
}
EOF</span>
$ <span class="userinput">arv-crunch-job --job "$(cat the_job)"</span>
2013-12-12_21:56:59 qr1hi-8i9sb-79260ykfew5trzl 31578  check slurm allocation
2013-12-12_21:56:59 qr1hi-8i9sb-79260ykfew5trzl 31578  node localhost - 1 slots
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  start
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  script hello-world.py
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  script_version /home/you/you
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  script_parameters {}
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  runtime_constraints {"max_tasks_per_node":0}
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  start level 0
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  status: 0 done, 0 running, 1 todo
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578 0 job_task qr1hi-ot0gb-u8g594ct0wt7f3f
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578 0 child 31585 started on localhost.1
2013-12-12_21:57:00 qr1hi-8i9sb-79260ykfew5trzl 31578  status: 0 done, 1 running, 0 todo
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 0 child 31585 on localhost.1 exit 0 signal 0 success=true
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 0 success in 1 seconds
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 0 output 576c44d762ba241b0a674aa43152b52a+53
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578  wait for last 0 children to finish
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578  status: 1 done, 0 running, 0 todo
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578  Freeze not implemented
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578  collate
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578  output 576c44d762ba241b0a674aa43152b52a+53
2013-12-12_21:57:03 qr1hi-8i9sb-79260ykfew5trzl 31578  finish
2013-12-12_21:57:04 qr1hi-8i9sb-79260ykfew5trzl 31578  meta key is 9f937693334d0c9234ccc1f808ee7117+1761+K@qr1hi
</code></pre>
</notextile>

The job succeeded, with output in Keep object @576c44d762ba241b0a674aa43152b52a+53@.  Let's look at our output:

<notextile>
<pre><code>$ <span class="userinput">arv keep get 576c44d762ba241b0a674aa43152b52a+53/hello.txt</span>
hello world
</code></pre>
</notextile>

Next, "parallel tasks.":tutorial-parallel.html