X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/d0930bfe4226ba361a5b1087b01946130f8282fd..0812bc1c717e5fed57d420b177f6ca9d41e81032:/doc/user/topics/tutorial-parallel.html.textile.liquid diff --git a/doc/user/topics/tutorial-parallel.html.textile.liquid b/doc/user/topics/tutorial-parallel.html.textile.liquid index 0cbceda619..9be610358b 100644 --- a/doc/user/topics/tutorial-parallel.html.textile.liquid +++ b/doc/user/topics/tutorial-parallel.html.textile.liquid @@ -1,10 +1,10 @@ --- layout: default navsection: userguide -title: "Parallel Crunch tasks" +title: "Concurrent Crunch tasks" ... -In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically parallelize our jobs by creating a separate task per file. For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file. In this this tutorial will demonstrate how to create Crunch tasks directly. +In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically create concurrent jobs by creating a separate task per file. For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file. In this this tutorial will demonstrate how to create Crunch tasks directly. Start by entering the @crunch_scripts@ directory of your Git repository: @@ -13,33 +13,33 @@ Start by entering the @crunch_scripts@ directory of your Git repository: -Next, using @nano@ or your favorite Unix text editor, create a new file called @parallel-hash.py@ in the @crunch_scripts@ directory. +Next, using @nano@ or your favorite Unix text editor, create a new file called @concurrent-hash.py@ in the @crunch_scripts@ directory. -notextile.
~/$USER/crunch_scripts$ nano parallel-hash.py
+notextile. ~/$USER/crunch_scripts$ nano concurrent-hash.py
Add the following code to compute the MD5 hash of each file in a collection:
-~/$USER/crunch_scripts$ chmod +x parallel-hash.py
+notextile. ~/$USER/crunch_scripts$ chmod +x concurrent-hash.py
Add the file to the Git staging area, commit, and push:
~/$USER/crunch_scripts$ git add parallel-hash.py
-~/$USER/crunch_scripts$ git commit -m"parallel hash"
+~/$USER/crunch_scripts$ git add concurrent-hash.py
+~/$USER/crunch_scripts$ git commit -m"concurrent hash"
~/$USER/crunch_scripts$ git push origin master
~/$USER/crunch_scripts$ cat >~/the_job <<EOF
{
- "script": "parallel-hash.py",
+ "script": "concurrent-hash.py",
"repository": "$USER",
"script_version": "master",
"script_parameters":
@@ -65,7 +65,7 @@ EOF
(Your shell should automatically fill in @$USER@ with your login name. The job JSON that gets saved should have @"repository"@ pointed at your personal Git repository.)
-Because the job ran in parallel, each instance of parallel-hash creates a separate @md5sum.txt@ as output. Arvados automatically collates theses files into a single collection, which is the output of the job:
+Because the job ran in concurrent, each instance of concurrent-hash creates a separate @md5sum.txt@ as output. Arvados automatically collates theses files into a single collection, which is the output of the job:
~/$USER/crunch_scripts$ arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162