doc/user/topics/tutorial-parallel.html.textile.liquid

   1 ---
   2 layout: default
   3 navsection: userguide
   4 title: "Parallel Crunch tasks"
   5 ...
   6
   7 In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically parallelize our jobs by creating a separate task per file.  For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file.  In this this tutorial will demonstrate how to create Crunch tasks directly.
   8
   9 Start by entering the @crunch_scripts@ directory of your git repository:
  10
  11 <notextile>
  12 <pre><code>~$ <span class="userinput">cd <b>you</b>/crunch_scripts</span>
  13 </code></pre>
  14 </notextile>
  15
  16 Next, using @nano@ or your favorite Unix text editor, create a new file called @parallel-hash.py@ in the @crunch_scripts@ directory.
  17
  18 notextile. <pre>~/<b>you</b>/crunch_scripts$ <code class="userinput">nano parallel-hash.py</code></pre>
  19
  20 Add the following code to compute the md5 hash of each file in a
  21
  22 <notextile> {% code 'parallel_hash_script_py' as python %} </notextile>
  23
  24 Make the file executable:
  25
  26 notextile. <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">chmod +x parallel-hash.py</span></code></pre>
  27
  28 Next, add the file to @git@ staging, commit and push:
  29
  30 <notextile>
  31 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">git add parallel-hash.py</span>
  32 ~/<b>you</b>/crunch_scripts$ <span class="userinput">git commit -m"parallel hash"</span>
  33 ~/<b>you</b>/crunch_scripts$ <span class="userinput">git push origin master</span>
  34 </code></pre>
  35 </notextile>
  36
  37 You should now be able to run your new script using Crunch, with "script" referring to our new "parallel-hash.py" script.  We will use a different input from our previous examples.  We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":{{site.baseurl}}/user/tutorials/tutorial-keep.html#dir).
  38
  39 <notextile>
  40 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cat &gt;~/the_job &lt;&lt;EOF
  41 {
  42  "script": "parallel-hash.py",
  43  "repository": "<b>you</b>",
  44  "script_version": "master",
  45  "script_parameters":
  46  {
  47   "input": "887cd41e9c613463eab2f0d885c6dd96+83"
  48  }
  49 }
  50 EOF</span>
  51 ~/<b>you</b>/crunch_scripts$ <span class="userinput">arv job create --job "$(cat ~/the_job)"</span>
  52 {
  53  ...
  54  "uuid":"qr1hi-xxxxx-xxxxxxxxxxxxxxx"
  55  ...
  56 }
  57 ~/<b>you</b>/crunch_scripts$ <span class="userinput">arv job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx</span>
  58 {
  59  ...
  60  "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162",
  61  ...
  62 }
  63 </code></pre>
  64 </notextile>
  65
  66 Because the job ran in parallel, each instance of parallel-hash creates a separate @md5sum.txt@ as output.  Arvados automatically collates theses files into a single collection, which is the output of the job:
  67
  68 <notextile>
  69 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
  70 md5sum.txt
  71 md5sum.txt
  72 md5sum.txt
  73 ~/<b>you</b>/crunch_scripts$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
  74 0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
  75 504938460ef369cd275e4ef58994cffe bob.txt
  76 8f3b36aff310e06f3c5b9e95678ff77a carol.txt
  77 </code></pre>
  78 </notextile>
  79
  80