navorder: 3
---
-{% include alert-stub.html %}
-
h1. Crunch setup
h4. Perl SDK dependencies
* @apt-get install libjson-perl libwww-perl libio-socket-ssl-perl libipc-system-simple-perl@
+h4. Python SDK dependencies
+
+On controller and all compute nodes:
+
+* @apt-get install python-pip@
+* @pip install --upgrade google-api-python-client@
+
+h4. Likely crunch job dependencies
+
+On compute nodes:
+
+* @pip install --upgrade pyvcf@
+
h4. Repositories
Crunch scripts must be in git repositories in @/var/cache/git/*/.git@ (or whatever is configured in @services/api/config/environments/production.rb@).
while sleep 60
do
cd /path/to/arvados/services/api
- setuidgid www-data env RAILS_ENV=production bundle exec ./script/import_commits.rb
+ setuidgid www-data env RAILS_ENV=production /usr/local/rvm/bin/rvm 2.0.0 do bundle exec ./script/import_commits.rb 2>&1
done
</pre>
Once you have imported some commits, you should be able to create a new job:
<pre>
-read newjob <<EOF; arv job create --job "$newjob"
+read -rd "\000" newjob <<EOF; arv job create --job "$newjob"
{"script_parameters":{"input":"f815ec01d5d2f11cb12874ab2ed50daa"},
"script_version":"master",
"script":"hash"}
* @services/api/script/crunch-dispatch.rb@ must be running.
* @crunch-dispatch.rb@ needs @services/crunch/crunch-job@ in its @PATH@.
* @crunch-job@ needs @sdk/perl/lib@ and @warehouse-apps/libwarehouse-perl/lib@ in its @PERLLIB@
+* @crunch-job@ needs @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@)
Example @/var/service/arvados_crunch_dispatch/run@ script:
<pre>
#!/bin/sh
set -e
-PATH=$PATH:/path/to/arvados/services/crunch
+export PATH="$PATH":/path/to/arvados/services/crunch
export PERLLIB=/path/to/arvados/sdk/perl/lib:/path/to/warehouse-apps/libwarehouse-perl/lib
+export ARVADOS_API_HOST=xyzzy.arvadosapi.com
+
+## Only if your SSL cert is unverifiable:
+# export ARVADOS_API_HOST_INSECURE=yes
+
cd /path/to/arvados/services/api
-RAILS_ENV=production bundle exec ./script/crunch-dispatch.rb
+RAILS_ENV=production /usr/local/rvm/bin/rvm 2.0.0 do bundle exec ./script/crunch-dispatch.rb 2>&1
</pre>