doc/user/topics/tutorial-parallel.html.textile.liquid

   1 ---
   2 layout: default
   3 navsection: userguide
   4 title: "Concurrent Crunch tasks"
   5 ...
   6 {% comment %}
   7 Copyright (C) The Arvados Authors. All rights reserved.
   8
   9 SPDX-License-Identifier: CC-BY-SA-3.0
  10 {% endcomment %}
  11
  12 {% include 'pipeline_deprecation_notice' %}
  13
  14 In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically create concurrent jobs by creating a separate task per file.  For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file.  This tutorial will demonstrate how to create Crunch tasks directly.
  15
  16 Start by entering the @crunch_scripts@ directory of your Git repository:
  17
  18 <notextile>
  19 <pre><code>~$ <span class="userinput">cd $USER/crunch_scripts</span>
  20 </code></pre>
  21 </notextile>
  22
  23 Next, using @nano@ or your favorite Unix text editor, create a new file called @concurrent-hash.py@ in the @crunch_scripts@ directory.
  24
  25 notextile. <pre>~/$USER/crunch_scripts$ <code class="userinput">nano concurrent-hash.py</code></pre>
  26
  27 Add the following code to compute the MD5 hash of each file in a collection:
  28
  29 <notextile> {% code 'concurrent_hash_script_py' as python %} </notextile>
  30
  31 Make the file executable:
  32
  33 notextile. <pre><code>~/$USER/crunch_scripts$ <span class="userinput">chmod +x concurrent-hash.py</span></code></pre>
  34
  35 Add the file to the Git staging area, commit, and push:
  36
  37 <notextile>
  38 <pre><code>~/$USER/crunch_scripts$ <span class="userinput">git add concurrent-hash.py</span>
  39 ~/$USER/crunch_scripts$ <span class="userinput">git commit -m"concurrent hash"</span>
  40 ~/$USER/crunch_scripts$ <span class="userinput">git push origin master</span>
  41 </code></pre>
  42 </notextile>
  43
  44 You should now be able to run your new script using Crunch, with "script" referring to our new "concurrent-hash.py" script.  We will use a different input from our previous examples.  We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":{{site.baseurl}}/user/tutorials/tutorial-keep.html#dir).
  45
  46 <notextile>
  47 <pre><code>~/$USER/crunch_scripts$ <span class="userinput">cat &gt;~/the_job &lt;&lt;EOF
  48 {
  49  "script": "concurrent-hash.py",
  50  "repository": "$USER/$USER",
  51  "script_version": "master",
  52  "script_parameters":
  53  {
  54   "input": "887cd41e9c613463eab2f0d885c6dd96+83"
  55  }
  56 }
  57 EOF</span>
  58 ~/$USER/crunch_scripts$ <span class="userinput">arv job create --job "$(cat ~/the_job)"</span>
  59 {
  60  ...
  61  "uuid":"qr1hi-xxxxx-xxxxxxxxxxxxxxx"
  62  ...
  63 }
  64 ~/$USER/crunch_scripts$ <span class="userinput">arv job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx</span>
  65 {
  66  ...
  67  "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162",
  68  ...
  69 }
  70 </code></pre>
  71 </notextile>
  72
  73 (Your shell should automatically fill in @$USER@ with your login name.  The job JSON that gets saved should have @"repository"@ pointed at your personal Git repository.)
  74
  75 Because the job ran in concurrent, each instance of concurrent-hash creates a separate @md5sum.txt@ as output.  Arvados automatically collates theses files into a single collection, which is the output of the job:
  76
  77 <notextile>
  78 <pre><code>~/$USER/crunch_scripts$ <span class="userinput">arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
  79 ./md5sum.txt
  80 ~/$USER/crunch_scripts$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
  81 0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
  82 504938460ef369cd275e4ef58994cffe bob.txt
  83 8f3b36aff310e06f3c5b9e95678ff77a carol.txt
  84 </code></pre>
  85 </notextile>