SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-The SLURM dispatcher can run on any node that can submit requests to both the Arvados API server and the SLURM controller. It is not resource-intensive, so you can run it on the API server node.
-
-h2. Install the dispatcher
+{% include 'notebox_begin_warning' %}
+crunch-dispatch-slurm is only relevant for on premise clusters that will spool jobs to Slurm. Skip this section if you are installing a cloud cluster.
+{% include 'notebox_end' %}
-First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
+# "Introduction":#introduction
+# "Update config.yml":#update-config
+# "Install crunch-dispatch-slurm":#install-packages
+# "Start the service":#start-service
+# "Restart the API server and controller":#restart-api
-On Red Hat-based systems:
+h2(#introduction). Introduction
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install crunch-dispatch-slurm</span>
-~$ <span class="userinput">sudo systemctl enable crunch-dispatch-slurm</span>
-</code></pre>
-</notextile>
+This assumes you already have a SLURM cluster, and have "set up all of your compute nodes":install-compute-node.html . For information on installing SLURM, see "this install guide":https://slurm.schedmd.com/quickstart_admin.html
-On Debian-based systems:
+The Arvados SLURM dispatcher can run on any node that can submit requests to both the Arvados API server and the SLURM controller (via @sbatch@). It is not resource-intensive, so you can run it on the API server node.
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install crunch-dispatch-slurm</span>
-</code></pre>
-</notextile>
+h2(#update-config). Update config.yml (optional)
-h2. Configure the dispatcher (optional)
+Crunch-dispatch-slurm reads the common configuration file at @config.yml@.
-Crunch-dispatch-slurm reads the common configuration file at @/etc/arvados/config.yml@. The essential configuration parameters will already be set by previous install steps, so no additional configuration is required. The following sections describe optional configuration parameters.
+The following configuration parameters are optional.
h3(#PollPeriod). Containers.PollInterval
crunch-dispatch-slurm polls the API server periodically for new containers to run. The @PollInterval@ option controls how often this poll happens. Set this to a string of numbers suffixed with one of the time units @ns@, @us@, @ms@, @s@, @m@, or @h@. For example:
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
<code class="userinput">PollInterval: <b>3m30s</b>
</code></pre>
</notextile>
Supports suffixes @KB@, @KiB@, @MB@, @MiB@, @GB@, @GiB@, @TB@, @TiB@, @PB@, @PiB@, @EB@, @EiB@ (where @KB@ is 10[^3^], @KiB@ is 2[^10^], @MB@ is 10[^6^], @MiB@ is 2[^20^] and so forth).
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
<code class="userinput">ReserveExtraRAM: <b>256MiB</b></code>
</pre>
</notextile>
If SLURM is unable to run a container, the dispatcher will submit it again after the next PollPeriod. If PollPeriod is very short, this can be excessive. If MinRetryPeriod is set, the dispatcher will avoid submitting the same container to SLURM more than once in the given time span.
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
<code class="userinput">MinRetryPeriod: <b>30s</b></code>
</pre>
</notextile>
Some Arvados installations run a local keepstore on each compute node to handle all Keep traffic. To override Keep service discovery and access the local keep server instead of the global servers, set ARVADOS_KEEP_SERVICES in SbatchEnvironmentVariables:
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
SLURM:
<span class="userinput">SbatchEnvironmentVariables:
- "http://127.0.0.1:25107": {}</span>
+ ARVADOS_KEEP_SERVICES: "http://127.0.0.1:25107"</span>
</code></pre>
</notextile>
The smallest usable value is @1@. The default value of @10@ is used if this option is zero or negative. Example:
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
SLURM:
<code class="userinput">PrioritySpread: <b>1000</b></code></pre>
</notextile>
When crunch-dispatch-slurm invokes @sbatch@, you can add arguments to the command by specifying @SbatchArguments@. You can use this to send the jobs to specific cluster partitions or add resource requests. Set @SbatchArguments@ to an array of strings. For example:
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
SLURM:
<code class="userinput">SbatchArgumentsList:
- <b>"--partition=PartitionName"</b></code>
If your SLURM cluster uses the @task/cgroup@ TaskPlugin, you can configure Crunch's Docker containers to be dispatched inside SLURM's cgroups. This provides consistent enforcement of resource constraints. To do this, use a crunch-dispatch-slurm configuration like the following:
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
<code class="userinput">CrunchRunArgumentsList:
- <b>"-cgroup-parent-subsystem=memory"</b></code>
</pre>
Older Linux kernels (prior to 3.18) have bugs in network namespace handling which can lead to compute node lockups. This by is indicated by blocked kernel tasks in "Workqueue: netns cleanup_net". If you are experiencing this problem, as a workaround you can disable use of network namespaces by Docker across the cluster. Be aware this reduces container isolation, which may be a security risk.
<notextile>
-<pre>
-Clusters:
- zzzzz:
- Containers:
+<pre> Containers:
<code class="userinput">CrunchRunArgumentsList:
- <b>"-container-enable-networking=always"</b>
- <b>"-container-network-mode=host"</b></code>
</pre>
</notextile>
-h2. Restart the dispatcher
+{% assign arvados_component = 'crunch-dispatch-slurm' %}
-{% include 'notebox_begin' %}
-
-The crunch-dispatch-slurm package includes configuration files for systemd. If you're using a different init system, you'll need to configure a service to start and stop a @crunch-dispatch-slurm@ process as desired. The process should run from a directory where the @crunch@ user has write permission on all compute nodes, such as its home directory or @/tmp@. You do not need to specify any additional switches or environment variables.
-
-{% include 'notebox_end' %}
+{% include 'install_packages' %}
-Restart the dispatcher to run with your new configuration:
+{% include 'start_service' %}
-<notextile>
-<pre><code>~$ <span class="userinput">sudo systemctl restart crunch-dispatch-slurm</span>
-</code></pre>
-</notextile>
+{% include 'restart_api' %}
projects / arvados.git / blobdiff