X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/35336cd73e444534cb2eda20e3730464cc4e6553..48dd255814cfc90a095132b6f621af13430267e0:/doc/install/install-crunch-dispatch.html.textile.liquid
diff --git a/doc/install/install-crunch-dispatch.html.textile.liquid b/doc/install/install-crunch-dispatch.html.textile.liquid
index 9b0e9b82a1..4c276a4e4f 100644
--- a/doc/install/install-crunch-dispatch.html.textile.liquid
+++ b/doc/install/install-crunch-dispatch.html.textile.liquid
@@ -5,74 +5,142 @@ title: Install the Crunch dispatcher
...
+The dispatcher normally runs on the same host/VM as the API server.
+h2. Perl SDK dependencies
-The dispatcher normally runs on the same host/VM as the API server.
+Install the Perl SDK on the controller.
-h4. Perl SDK dependencies
+* See "Perl SDK":{{site.baseurl}}/sdk/perl/index.html page for details.
-* @apt-get install libjson-perl libwww-perl libio-socket-ssl-perl libipc-system-simple-perl@
+h2. Python SDK dependencies
-Add this to @/etc/apt/sources.list@
+Install the Python SDK and CLI tools on controller and all compute nodes.
-@deb http://git.oxf.freelogy.org/apt wheezy main contrib@
+* See "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html page for details.
-Then
+h2(#slurm). Set up SLURM
-@apt-get install libwarehouse-perl@
+On the API server, install SLURM and munge, and generate a munge key.
-h4. Python SDK dependencies
+On Debian-based systems:
-On controller and all compute nodes:
+
+~$ sudo /usr/bin/apt-get install slurm-llnl munge
+~$ sudo /usr/sbin/create-munge-key
+
+ControlMachine=uuid_prefix.your.domain
+SlurmctldPort=6817
+SlurmdPort=6818
+AuthType=auth/munge
+StateSaveLocation=/tmp
+SlurmdSpoolDir=/tmp/slurmd
+SwitchType=switch/none
+MpiDefault=none
+SlurmctldPidFile=/var/run/slurmctld.pid
+SlurmdPidFile=/var/run/slurmd.pid
+ProctrackType=proctrack/pgid
+CacheGroups=0
+ReturnToService=2
+TaskPlugin=task/affinity
+#
+# TIMERS
+SlurmctldTimeout=300
+SlurmdTimeout=300
+InactiveLimit=0
+MinJobAge=300
+KillWait=30
+Waittime=0
+#
+# SCHEDULING
+SchedulerType=sched/backfill
+SchedulerPort=7321
+SelectType=select/cons_res
+SelectTypeParameters=CR_CPU_Memory
+FastSchedule=1
+#
+# LOGGING
+SlurmctldDebug=3
+#SlurmctldLogFile=
+SlurmdDebug=3
+#SlurmdLogFile=
+JobCompType=jobcomp/none
+#JobCompLoc=
+JobAcctGatherType=jobacct_gather/none
+#
+# COMPUTE NODES
+NodeName=DEFAULT
+PartitionName=DEFAULT MaxTime=INFINITE State=UP
+PartitionName=compute Default=YES Shared=yes
+
+NodeName=compute[0-255]
+PartitionName=compute Nodes=compute[0-255]
+
+assign_node_hostname: worker1-%
+* In @slurm.conf@: NodeName=worker1-[0000-0255]
-h4. Importing commits
+If your worker hostnames are already assigned by other means, and the full set of names is known in advance, have your worker node bootstrapping script (see "Installing a compute node":install-compute-node.html) send its current hostname, rather than expect Arvados to assign one.
+* In @application.yml@: assign_node_hostname: false
+* In @slurm.conf@: NodeName=alice,bob,clay,darlene
-@services/api/script/import_commits.rb production@ must run periodically. Example @/var/service/arvados_import_commits/run@ script for daemontools or runit:
+If your worker hostnames are already assigned by other means, but the full set of names is _not_ known in advance, you can use the @slurm.conf@ and @application.yml@ settings in the previous example, but you must also update @slurm.conf@ (both on the controller and on all worker nodes) and run @sudo scontrol reconfigure@ whenever a new node comes online.
-
-#!/bin/sh -set -e -while sleep 60 -do - cd /path/to/arvados/services/api - setuidgid www-data env RAILS_ENV=production /usr/local/rvm/bin/rvm-exec 2.0.0 bundle exec ./script/import_commits.rb 2>&1 -done -+h2. Enable SLURM job dispatch -Once you have imported some commits, you should be able to create a new job: +In your API server's @application.yml@ configuration file, add the line @crunch_job_wrapper: :slurm_immediate@ under the appropriate section. (The second colon is not a typo. It denotes a Ruby symbol.) + +h2. Crunch user account + +Run @sudo adduser crunch@. The crunch user should have the same UID, GID, and home directory on all compute nodes and on the dispatcher (API server). + +h2. Git Repositories + +Crunch scripts must be in Git repositories in the directory configured as @git_repositories_dir@/*.git (see the "API server installation":install-api-server.html#git_repositories_dir). + +Once you have a repository with commits -- and you have read access to the repository -- you should be able to create a new job:
read -rd $'\000' newjob <-h4. Running jobs +h2. Running jobs * @services/api/script/crunch-dispatch.rb@ must be running. * @crunch-dispatch.rb@ needs @services/crunch/crunch-job@ in its @PATH@. -* @crunch-job@ needs @sdk/perl/lib@ and @warehouse-apps/libwarehouse-perl/lib@ in its @PERLLIB@ -* @crunch-job@ needs @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@) +* @crunch-job@ needs the installation path of the Perl SDK in its @PERLLIB@. +* @crunch-job@ needs the @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@) environment variable set. Example @/var/service/arvados_crunch_dispatch/run@ script:@@ -82,23 +150,31 @@ Without getting this error: ArgumentError: Specified script_version does not resolve to a commit
#!/bin/sh set -e + +rvmexec="" +## uncomment this line if you use rvm: +#rvmexec="/usr/local/rvm/bin/rvm-exec 2.1.1" + export PATH="$PATH":/path/to/arvados/services/crunch -export PERLLIB=/path/to/arvados/sdk/perl/lib:/path/to/warehouse-apps/libwarehouse-perl/lib export ARVADOS_API_HOST={{ site.arvados_api_host }} export CRUNCH_DISPATCH_LOCKFILE=/var/lock/crunch-dispatch +# This is the path to docker on your compute nodes. You might need to +# change it to "docker", "/opt/bin/docker", etc. +export CRUNCH_JOB_DOCKER_BIN=docker.io + fuser -TERM -k $CRUNCH_DISPATCH_LOCKFILE || true ## Only if your SSL cert is unverifiable: @@ -106,5 +182,5 @@ fuser -TERM -k $CRUNCH_DISPATCH_LOCKFILE || true cd /path/to/arvados/services/api export RAILS_ENV=production -exec /usr/local/rvm/bin/rvm-exec 2.0.0 bundle exec ./script/crunch-dispatch.rb 2>&1 +exec $rvmexec bundle exec ./script/crunch-dispatch.rb 2>&1