X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/0f125dd51c7dc13047672dee5362866f31885e0a..e02c4e1b1ed61d57d86f98b9372813d913d5b375:/doc/install/install-crunch-dispatch.html.textile.liquid
diff --git a/doc/install/install-crunch-dispatch.html.textile.liquid b/doc/install/install-crunch-dispatch.html.textile.liquid
index 4a695ca199..e0ed147694 100644
--- a/doc/install/install-crunch-dispatch.html.textile.liquid
+++ b/doc/install/install-crunch-dispatch.html.textile.liquid
@@ -4,26 +4,58 @@ navsection: installguide
title: Install the Crunch dispatcher
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
-
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
The dispatcher normally runs on the same host/VM as the API server.
-h4. Perl SDK dependencies
+h2. Test the Arvados job queue
+
+Crunch dispatches work from the job queue on the Arvados API server. Before you start installing the Crunch dispatcher, now's a good time to check that the API server and Git server can coordinate to create job records. Run these commands *on your shell server* to create a collection, and a job to calculate the MD5 checksum of every file in it:
+
+
+~$ echo 'Hello, Crunch!' | arv-put --portable-data-hash -
+…
+d40c7f35d80da669afb9db1896e760ad+49
+~$ read -rd $'\000' newjob <<EOF; arv job create --job "$newjob"
+{"script_parameters":{"input":"d40c7f35d80da669afb9db1896e760ad+49"},
+ "script_version":"0988acb472849dc0",
+ "script":"hash",
+ "repository":"arvados"}
+EOF
+
+ArgumentError: Specified script_version does not resolve to a commit ++ +it often means that the API server can't read the specified repository—either because it doesn't exist, or because the user running the API server doesn't have permission to read the repository files. Check the API server's log (@/var/www/arvados-api/current/log/production.log@) for details, and double-check the instructions in the "Git server installation guide":install-arv-git-httpd.html. + +If everything goes well, the API server should create a job record, and your @arv@ command will output the JSON for that record. It should have state @Queued@ and script_version @0988acb472849dc08d576ee40493e70bde2132ca@. If the job JSON includes those fields, you can proceed to install the Crunch dispatcher and a compute node. This job will remain queued until you install those services. + +h2. Perl SDK dependencies Install the Perl SDK on the controller. * See "Perl SDK":{{site.baseurl}}/sdk/perl/index.html page for details. -h4. Python SDK dependencies +h2. Python SDK dependencies Install the Python SDK and CLI tools on controller and all compute nodes. * See "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html page for details. -h4. Slurm +h2(#slurm). Set up SLURM + +On the API server, install SLURM and munge, and generate a munge key. -On the API server, install slurm and munge, and generate a munge key: +On Debian-based systems:
~$ sudo /usr/bin/apt-get install slurm-llnl munge
@@ -31,7 +63,14 @@ On the API server, install slurm and munge, and generate a munge key:
~$ sudo yum install slurm munge slurm-munge
+
+@@ -61,9 +100,8 @@ Waittime=0 # SCHEDULING SchedulerType=sched/backfill SchedulerPort=7321 -SelectType=select/cons_res -SelectTypeParameters=CR_CPU_Memory -FastSchedule=1 +SelectType=select/linear +FastSchedule=0 # # LOGGING SlurmctldDebug=3 @@ -77,70 +115,93 @@ JobAcctGatherType=jobacct_gather/none # COMPUTE NODES NodeName=DEFAULT PartitionName=DEFAULT MaxTime=INFINITE State=UP -PartitionName=compute Default=YES Shared=yes NodeName=compute[0-255] - -PartitionName=compute Nodes=compute[0-255] +PartitionName=compute Nodes=compute[0-255] Default=YES Shared=YES
-read -rd $'\000' newjob <+If your worker node bootstrapping script (see "Installing a compute node":install-compute-node.html) does not send the worker's current hostname, the API server will choose an unused hostname from the set given in @application.yml@, which defaults to @compute[0-255]@. -Without getting this error: +If it is not feasible to give your compute nodes hostnames like compute0, compute1, etc., you can accommodate other naming schemes with a bit of extra configuration. - -ArgumentError: Specified script_version does not resolve to a commit -+If you want Arvados to assign names to your nodes with a different consecutive numeric series like @{worker1-0000, worker1-0001, worker1-0002}@, add an entry to @application.yml@; see @/var/www/arvados-api/current/config/application.default.yml@ for details. Example: +* In @application.yml@:assign_node_hostname: worker1-%
+* In @slurm.conf@:04d NodeName=worker1-[0000-0255]
-h4. Running jobs +If your worker hostnames are already assigned by other means, and the full set of names is known in advance, have your worker node bootstrapping script (see "Installing a compute node":install-compute-node.html) send its current hostname, rather than expect Arvados to assign one. +* In @application.yml@:assign_node_hostname: false
+* In @slurm.conf@:NodeName=alice,bob,clay,darlene
-* @services/api/script/crunch-dispatch.rb@ must be running. -* @crunch-dispatch.rb@ needs @services/crunch/crunch-job@ in its @PATH@. -* @crunch-job@ needs @sdk/perl/lib@ and @warehouse-apps/libwarehouse-perl/lib@ in its @PERLLIB@ -* @crunch-job@ needs @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@) +If your worker hostnames are already assigned by other means, but the full set of names is _not_ known in advance, you can use the @slurm.conf@ and @application.yml@ settings in the previous example, but you must also update @slurm.conf@ (both on the controller and on all worker nodes) and run @sudo scontrol reconfigure@ whenever a new node comes online. -Example @/var/service/arvados_crunch_dispatch/run@ script: +h2. Enable SLURM job dispatch --#!/bin/sh +In your API server's @application.yml@ configuration file, add the line @crunch_job_wrapper: :slurm_immediate@ under the appropriate section. (The second colon is not a typo. It denotes a Ruby symbol.) + +h2. Crunch user account + +Run @sudo adduser crunch@. The crunch user should have the same UID, GID, and home directory on all compute nodes and on the dispatcher (API server). + +h2. Run the Crunch dispatcher service + +To dispatch Arvados jobs: + +* The API server script @crunch-dispatch.rb@ must be running. +* @crunch-job@ needs the installation path of the Perl SDK in its @PERLLIB@. +* @crunch-job@ needs the @ARVADOS_API_HOST@ (and, if necessary, @ARVADOS_API_HOST_INSECURE@) environment variable set. + +Install runit to monitor the Crunch dispatch daemon. {% include 'install_runit' %} + +Install the script below as the run script for the Crunch dispatch service, modifying it as directed by the comments. + +++ +#!/bin/sh set -e rvmexec="" -## uncomment this line if you use rvm: -#rvmexec="/usr/local/rvm/bin/rvm-exec 2.1.1" +## Uncomment this line if you use RVM: +#rvmexec="/usr/local/rvm/bin/rvm-exec default" -export PATH="$PATH":/path/to/arvados/services/crunch -export ARVADOS_API_HOST={{ site.arvados_api_host }} +export ARVADOS_API_HOST=uuid_prefix.your.domain export CRUNCH_DISPATCH_LOCKFILE=/var/lock/crunch-dispatch +export HOME=$(pwd) +export RAILS_ENV=production -fuser -TERM -k $CRUNCH_DISPATCH_LOCKFILE || true +## Uncomment and edit this line if your compute nodes have cgroup info +## somewhere other than /sys/fs/cgroup (e.g., "/cgroup" for CentOS 7) +#export CRUNCH_CGROUP_ROOT="/sys/fs/cgroup" -## Only if your SSL cert is unverifiable: -# export ARVADOS_API_HOST_INSECURE=yes +## Uncomment this line if your cluster uses self-signed SSL certificates: +#export ARVADOS_API_HOST_INSECURE=yes -cd /path/to/arvados/services/api -export RAILS_ENV=production +# This is the path to docker on your compute nodes. You might need to +# change it to "docker", "/opt/bin/docker", etc. +export CRUNCH_JOB_DOCKER_BIN=docker.io + +fuser -TERM -k $CRUNCH_DISPATCH_LOCKFILE || true +cd /var/www/arvados-api/current exec $rvmexec bundle exec ./script/crunch-dispatch.rb 2>&1 -