---
layout: default
navsection: userguide
-title: "Debug a Crunch script"
-navorder: 114
----
+navmenu: Tutorials
+title: "Debugging a Crunch script"
+navorder: 14
+...
-h1. Tutorial: Debug a Crunch script
+h1. Debugging a Crunch script
To test changes to a script by running a job, the change must be pushed into @git@, the job queued asynchronously, and the actual execution may be run on any compute server. As a result, debugging a script can be difficult and time consuming. This tutorial demonstrates using @arv-crunch-job@ to run your job in your local VM. This avoids the job queue and allows you to execute the script from your uncomitted git tree.
-*This tutorial assumes that you are "logged into an Arvados VM instance":ssh-access.html#login, and have a "working environment.":check-environment.html*
+*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.basedoc}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.basedoc}}/user/getting_started/check-environment.html*
This tutorial uses _you_ to denote your username. Replace _you_ with your user name in all the following examples.
h2. Using arv-crunch-job to run the job in your VM
-Instead of a git commit hash, we provide the path to the directory in the "script_version" parameter. The script specified in "script" will actually be searched for in the "crunch_scripts/" subdirectory of the directory specified "script_version". Although we are running the script locally, the script still requires access to the Arvados API server and Keep storage service, and the job will be recorded on then Arvados Workbench job history.
+Instead of a git commit hash, we provide the path to the directory in the "script_version" parameter. The script specified in "script" will actually be searched for in the "crunch_scripts/" subdirectory of the directory specified "script_version". Although we are running the script locally, the script still requires access to the Arvados API server and Keep storage service. The job will be recorded in the Arvados job history, and visible in Workbench.
<notextile>
<pre><code>$ <span class="userinput">cat >the_job <<EOF
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 Freeze not implemented
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 collate
2013-12-12_21:36:43 qr1hi-8i9sb-okzukfzkpbrnhst 29827 output d41d8cd98f00b204e9800998ecf8427e+0
-2013-12-12_21:36:44 qr1hi-8i9sb-okzukfzkpbrnhst 29827 meta key is c00bfbd58e6f58ce3bebdd47f745a70f+1857+K@qr1hi
+2013-12-12_21:36:44 qr1hi-8i9sb-okzukfzkpbrnhst 29827 meta key is c00bfbd58e6f58ce3bebdd47f745a70f+1857
</code></pre>
</notextile>
bc. 2013-12-12_21:36:42 qr1hi-8i9sb-okzukfzkpbrnhst 29827 0 stderr hello world
-The script's printout is captured in the log, which is useful for print statement debugging. However, although it the script returned a status code of 0 (success), the job failed. Why? For a job to complete successfully scripts must explicitly add their output to Keep, and then tell Arvados about it. Here is a second try:
+The script's output is captured in the log, which is useful for print statement debugging. However, although this script returned a status code of 0 (success), the job failed. Why? For a job to complete successfully scripts must explicitly add their output to Keep, and then tell Arvados about it. Here is a second try:
<notextile>
<pre><code>$ <span class="userinput">cat >hello-world.py <<EOF
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 collate
2013-12-12_21:57:02 qr1hi-8i9sb-79260ykfew5trzl 31578 output 576c44d762ba241b0a674aa43152b52a+53
2013-12-12_21:57:03 qr1hi-8i9sb-79260ykfew5trzl 31578 finish
-2013-12-12_21:57:04 qr1hi-8i9sb-79260ykfew5trzl 31578 meta key is 9f937693334d0c9234ccc1f808ee7117+1761+K@qr1hi
+2013-12-12_21:57:04 qr1hi-8i9sb-79260ykfew5trzl 31578 meta key is 9f937693334d0c9234ccc1f808ee7117+1761
</code></pre>
</notextile>
</code></pre>
</notextile>
+h3. Location of temporary files
+
+Crunch job tasks are supplied with @TASK_WORK@ and @JOB_WORK@ environment variables, to be used as scratch space. When running in local development mode using @arv-crunch-job@, Crunch sets these variables to point to directory called @crunch-job-{USERID}@ in @TMPDIR@ (or @/tmp@ if @TMPDIR@ is not set).
+
+* Set @TMPDIR@ to @/scratch@ to make Crunch use a directory like @/scratch/crunch-job-{USERID}/@ for temporary space.
+
+* Set @CRUNCH_TMP@ to @/scratch/foo@ to make Crunch use @/scratch/foo/@ for temporary space (omitting the default @crunch-job-{USERID}@ leaf name)
+
+h3. Testing job scripts without SDKs and Keep access
+
+Read and write data to @/tmp/@ instead of Keep. This only works with the Python SDK.
+
+notextile. <pre><code>$ <span class="userinput">export KEEP_LOCAL_STORE=/tmp</span></code></pre>
+
Next, "parallel tasks.":tutorial-parallel.html
projects / arvados.git / blobdiff