--- layout: default navsection: userguide title: "Concurrent Crunch tasks" ... {% comment %} Copyright (C) The Arvados Authors. All rights reserved. SPDX-License-Identifier: CC-BY-SA-3.0 {% endcomment %} {% include 'pipeline_deprecation_notice' %} In the previous tutorials, we used @arvados.job_setup.one_task_per_input_file()@ to automatically create concurrent jobs by creating a separate task per file. For some types of jobs, you may need to split the work up differently, for example creating tasks to process different segments of a single large file. This tutorial will demonstrate how to create Crunch tasks directly. Start by entering the @crunch_scripts@ directory of your Git repository:
~$ cd $USER/crunch_scripts
Next, using @nano@ or your favorite Unix text editor, create a new file called @concurrent-hash.py@ in the @crunch_scripts@ directory. notextile.
~/$USER/crunch_scripts$ nano concurrent-hash.py
Add the following code to compute the MD5 hash of each file in a collection: {% code 'concurrent_hash_script_py' as python %} Make the file executable: notextile.
~/$USER/crunch_scripts$ chmod +x concurrent-hash.py
Add the file to the Git staging area, commit, and push:
~/$USER/crunch_scripts$ git add concurrent-hash.py
~/$USER/crunch_scripts$ git commit -m"concurrent hash"
~/$USER/crunch_scripts$ git push origin master
You should now be able to run your new script using Crunch, with "script" referring to our new "concurrent-hash.py" script. We will use a different input from our previous examples. We will use @887cd41e9c613463eab2f0d885c6dd96+83@ which consists of three files, "alice.txt", "bob.txt" and "carol.txt" (the example collection used previously in "fetching data from Arvados using Keep":{{site.baseurl}}/user/tutorials/tutorial-keep.html#dir).
~/$USER/crunch_scripts$ cat >~/the_job <<EOF
{
 "script": "concurrent-hash.py",
 "repository": "$USER/$USER",
 "script_version": "master",
 "script_parameters":
 {
  "input": "887cd41e9c613463eab2f0d885c6dd96+83"
 }
}
EOF
~/$USER/crunch_scripts$ arv job create --job "$(cat ~/the_job)"
{
 ...
 "uuid":"qr1hi-xxxxx-xxxxxxxxxxxxxxx"
 ...
}
~/$USER/crunch_scripts$ arv job get --uuid qr1hi-xxxxx-xxxxxxxxxxxxxxx
{
 ...
 "output":"e2ccd204bca37c77c0ba59fc470cd0f7+162",
 ...
}
(Your shell should automatically fill in @$USER@ with your login name. The job JSON that gets saved should have @"repository"@ pointed at your personal Git repository.) Because the job ran in concurrent, each instance of concurrent-hash creates a separate @md5sum.txt@ as output. Arvados automatically collates theses files into a single collection, which is the output of the job:
~/$USER/crunch_scripts$ arv keep ls e2ccd204bca37c77c0ba59fc470cd0f7+162
./md5sum.txt
~/$USER/crunch_scripts$ arv-get e2ccd204bca37c77c0ba59fc470cd0f7+162/md5sum.txt
0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt
504938460ef369cd275e4ef58994cffe bob.txt
8f3b36aff310e06f3c5b9e95678ff77a carol.txt