X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/2183113c4c357e07719251854e3d249cdcd394dd..a4eed7a6cc62e05553796a7cee0a7713e735a726:/doc/user/tutorials/tutorial-parallel.textile diff --git a/doc/user/tutorials/tutorial-parallel.textile b/doc/user/tutorials/tutorial-parallel.textile index 88785a8dc3..b76d1a5238 100644 --- a/doc/user/tutorials/tutorial-parallel.textile +++ b/doc/user/tutorials/tutorial-parallel.textile @@ -1,11 +1,12 @@ --- layout: default navsection: userguide -title: "Parallel tasks" -navorder: 115 +navmenu: Tutorials +title: "Parallel Crunch tasks" +navorder: 15 --- -h1. Tutorial: Parallel tasks +h1. Tutorial: Parallel Crunch tasks In the tutorial "writing a crunch script,":tutorial-firstscript.html our script used a "for" loop to compute the md5 hashes for each file in sequence. This approach, while simple, is not able to take advantage of the compute cluster with multiple nodes and cores to speed up computation by running tasks in parallel. This tutorial will demonstrate how to create parallel Crunch tasks. @@ -33,7 +34,7 @@ $ git push origin master -You should now be able to run your new script using Crunch, with "script" referring to our new "parallel-hash.py" script. We will use a different input from our previous examples. We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":intro-keep.html). +You should now be able to run your new script using Crunch, with "script" referring to our new "parallel-hash.py" script. We will use a different input from our previous examples. We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":tutorial-keep.html).
$ cat >the_job <<EOF
@@ -55,7 +56,7 @@ $ arv -h job create --job "$(cat the_job)"
 $ arv -h job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx
 {
  ...
- "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162+K@qr1hi",
+ "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162",
  ...
 }
 
@@ -64,11 +65,11 @@ $ arv -h job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx -
$ arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162+K@qr1hi
+
$ arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162
 md5sum.txt
 md5sum.txt
 md5sum.txt
-$ arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162+K@qr1hi/md5sum.txt
+$ arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt
 0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
 504938460ef369cd275e4ef58994cffe bob.txt
 8f3b36aff310e06f3c5b9e95678ff77a carol.txt
@@ -77,4 +78,4 @@ $ arv keep get e2ccd204bca37c77c0ba59fc470cd0f7+162+K@qr
 
 h2. The one job per file pattern
 
-This example demonstrates how to schedule a new task per file.  Because this is a common pattern, the Crunch Python API contains a convenience function to "queue a task for each input file":crunch-utility-libraries.html#one_task_per_input which reduces the amount of boilerplate code required to handle parallel jobs.
+This example demonstrates how to schedule a new task per file.  Because this is a common pattern, the Crunch Python API contains a convenience function to "queue a task for each input file":{{site.basedoc}}/user/reference/crunch-utility-libraries.html#one_task_per_input which reduces the amount of boilerplate code required to handle parallel jobs.