Merge branch 'master' into 3219-further-docker-improvements

[arvados.git] / doc / user / topics / tutorial-parallel.html.textile.liquid

diff --git a/doc/user/topics/tutorial-parallel.html.textile.liquid b/doc/user/topics/tutorial-parallel.html.textile.liquid
index 23b7cfbecdc3d782e71c3e6e108e583c0482f3e1..d7a093a245085bf650cd6ee9b0ae1896e7313939 100644 (file)
--- a/doc/user/topics/tutorial-parallel.html.textile.liquid
+++ b/doc/user/topics/tutorial-parallel.html.textile.liquid
@@ -1,16 +1,12 @@
 ---
 layout: default
 navsection: userguide
-navmenu: Tutorials
 title: "Parallel Crunch tasks"
-
 ...
 
-h1. Parallel Crunch tasks
-
-In the tutorial "writing a crunch script,":tutorial-firstscript.html our script used a "for" loop to compute the md5 hashes for each file in sequence.  This approach, while simple, is not able to take advantage of the compute cluster with multiple nodes and cores to speed up computation by running tasks in parallel.  This tutorial will demonstrate how to create parallel Crunch tasks.
+In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically parallelize our jobs by creating a separate task per file.  For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file.  In this this tutorial will demonstrate how to create Crunch tasks directly.
 
-Start by entering the @crunch_scripts@ directory of your git repository:
+Start by entering the @crunch_scripts@ directory of your Git repository:
 
 <notextile>
 <pre><code>~$ <span class="userinput">cd <b>you</b>/crunch_scripts</span>
@@ -21,7 +17,7 @@ Next, using @nano@ or your favorite Unix text editor, create a new file called @
 
 notextile. <pre>~/<b>you</b>/crunch_scripts$ <code class="userinput">nano parallel-hash.py</code></pre>
 
-Add the following code to compute the md5 hash of each file in a 
+Add the following code to compute the MD5 hash of each file in a collection:
 
 <notextile> {% code 'parallel_hash_script_py' as python %} </notextile>
 
@@ -29,7 +25,7 @@ Make the file executable:
 
 notextile. <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">chmod +x parallel-hash.py</span></code></pre>
 
-Next, add the file to @git@ staging, commit and push:
+Add the file to the Git staging area, commit, and push:
 
 <notextile>
 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">git add parallel-hash.py</span>
@@ -38,13 +34,14 @@ Next, add the file to @git@ staging, commit and push:
 </code></pre>
 </notextile>
 
-You should now be able to run your new script using Crunch, with "script" referring to our new "parallel-hash.py" script.  We will use a different input from our previous examples.  We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":tutorial-keep.html).
+You should now be able to run your new script using Crunch, with "script" referring to our new "parallel-hash.py" script.  We will use a different input from our previous examples.  We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":{{site.baseurl}}/user/tutorials/tutorial-keep.html#dir).
 
 <notextile>
 <pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">cat &gt;~/the_job &lt;&lt;EOF
 {
  "script": "parallel-hash.py",
- "script_version": "you:master",
+ "repository": "$USER",
+ "script_version": "master",
  "script_parameters":
  {
   "input": "887cd41e9c613463eab2f0d885c6dd96+83"
@@ -66,13 +63,13 @@ EOF</span>
 </code></pre>
 </notextile>
 
+(Your shell should automatically fill in @$USER@ with your login name.  The job JSON that gets saved should have @"repository"@ pointed at your personal Git repository.)
+
 Because the job ran in parallel, each instance of parallel-hash creates a separate @md5sum.txt@ as output.  Arvados automatically collates theses files into a single collection, which is the output of the job:
 
 <notextile>
-<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
-md5sum.txt
-md5sum.txt
-md5sum.txt
+<pre><code>~/<b>you</b>/crunch_scripts$ <span class="userinput">arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162</span>
+./md5sum.txt
 ~/<b>you</b>/crunch_scripts$ <span class="userinput">arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt</span>
 0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
 504938460ef369cd275e4ef58994cffe bob.txt
@@ -80,9 +77,4 @@ md5sum.txt
 </code></pre>
 </notextile>
 
-h2. The one job per file pattern
-
-This example demonstrates how to schedule a new task per file.  Because this is a common pattern, the Crunch Python API contains a convenience function to "queue a task for each input file":{{site.baseurl}}/sdk/python/crunch-utility-libraries.html#one_task_per_input which reduces the amount of boilerplate code required to handle parallel jobs.
-
-Next, "Constructing a Crunch pipeline":tutorial-new-pipeline.html