arvados-dispatch-lsf is only relevant for on premises clusters that will spool jobs to LSF. Skip this section if you are installing a cloud cluster.
{% include 'notebox_end' %}
-Containers can be dispatched to an LSF cluster. The dispatcher sends work to the cluster using LSF's @bsub@ command, so it works in a variety of LSF configurations.
-
-*LSF support is currently considered experimental.*
+h2(#overview). Overview
-Limitations include:
-* Arvados container priority is not propagated to LSF job priority. This can cause inefficient use of compute resources, and even deadlock if there are fewer compute nodes than concurrent Arvados workflows.
-* Combining LSF with docker may not work, depending on LSF configuration and user/group IDs (if LSF only sets up the configured user's primary group ID when executing the crunch-run process on a compute node, it may not have permission to connect to the docker daemon).
+Containers can be dispatched to an LSF cluster. The dispatcher sends work to the cluster using LSF's @bsub@ command, so it works in a variety of LSF configurations.
In order to run containers, you must choose a user that has permission to set up FUSE mounts and run Singularity/Docker containers on each compute node. This install guide refers to this user as the @crunch@ user. We recommend you create this user on each compute node with the same UID and GID, and add it to the @fuse@ and @docker@ system groups to grant it the necessary permissions. However, you can run the dispatcher under any account with sufficient permissions across the cluster.
Set up all of your compute nodes "as you would for a SLURM cluster":../crunch2-slurm/install-compute-node.html.
+*Current limitations*:
+* Arvados container priority is not propagated to LSF job priority. This can cause inefficient use of compute resources, and even deadlock if there are fewer compute nodes than concurrent Arvados workflows.
+* Combining LSF with docker may not work, depending on LSF configuration and user/group IDs (if LSF only sets up the configured user's primary group ID when executing the crunch-run process on a compute node, it may not have permission to connect to the docker daemon).
h2(#update-config). Update config.yml
Review the following configuration parameters and adjust as needed.
-
h3(#BsubSudoUser). Containers.LSF.BsubSudoUser
arvados-dispatch-lsf uses @sudo@ to execute @bsub@, for example @sudo -E -u crunch bsub [...]@. This means the @crunch@ account must exist on the hosts where LSF jobs run ("execution hosts"), as well as on the host where you are installing the Arvados LSF dispatcher (the "submission host"). To use a user account other than @crunch@, configure @BsubSudoUser@:
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+h2(#overview). Overview
+
Arvados can be configured to use "Singularity":https://sylabs.io/singularity/ instead of Docker to execute containers on cloud nodes or a SLURM/LSF cluster. Singularity may be preferable due to its simpler installation and lack of long-running daemon process and special system users/groups.
-Please note:
-* *Singularity support is currently considered experimental.*
+*Current limitations*:
* Even when using the singularity runtime, users' container images are expected to be saved in Docker format using @arv keep docker@. Arvados converts the Docker image to Singularity format (@.sif@) at runtime as needed. Specifying a @.sif@ file as an image when submitting a container request is not yet supported.
* Singularity does not limit the amount of memory available in a container. Each container will have access to all memory on the host where it runs, unless memory use is restricted by SLURM/LSF.
* Programs running in containers may behave differently due to differences between Singularity and Docker.
** The root (image) filesystem is read-only in a Singularity container. Programs that attempt to write outside a designated output or temporary directory are likely to fail.
** The Docker ENTRYPOINT instruction is ignored.
-* Arvados is currently tested with Singularity version 3.7.4.
+* Arvados is tested with Singularity version 3.7.4. Other versions may not work.
+
+h2(#configuration). Configuration
To use singularity, first make sure "Singularity is installed":https://sylabs.io/guides/3.7/user-guide/quick_start.html on your cloud worker image or SLURM/LSF compute nodes as applicable. Note @squashfs-tools@ is required.
Then update @Containers.RuntimeEngine@ in your cluster configuration:
<notextile>
-<pre><code> # Container runtime: "docker" (default) or "singularity" (experimental)
+<pre><code> # Container runtime: "docker" (default) or "singularity"
RuntimeEngine: singularity
</code></pre>
</notextile>
AcceptAccessTokenScope: ""
PAM:
- # (Experimental) Use PAM to authenticate users.
+ # Use PAM to authenticate users.
Enable: false
# PAM service name. PAM will apply the policy in the
UsePreemptibleInstances: false
# PEM encoded SSH key (RSA, DSA, or ECDSA) used by the
- # (experimental) cloud dispatcher for executing containers on
- # worker VMs. Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
+ # cloud dispatcher for executing containers on worker VMs.
+ # Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
# and ends with "\n-----END RSA PRIVATE KEY-----\n".
DispatchPrivateKey: ""
# Minimum time between two attempts to run the same container
MinRetryPeriod: 0s
- # Container runtime: "docker" (default) or "singularity" (experimental)
+ # Container runtime: "docker" (default) or "singularity"
RuntimeEngine: docker
Logging:
GitInternalDir: /var/lib/arvados/internal.git
CloudVMs:
- # Enable the cloud scheduler (experimental).
+ # Enable the cloud scheduler.
Enable: false
# Name/number of port where workers' SSH services listen.
AcceptAccessTokenScope: ""
PAM:
- # (Experimental) Use PAM to authenticate users.
+ # Use PAM to authenticate users.
Enable: false
# PAM service name. PAM will apply the policy in the
UsePreemptibleInstances: false
# PEM encoded SSH key (RSA, DSA, or ECDSA) used by the
- # (experimental) cloud dispatcher for executing containers on
- # worker VMs. Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
+ # cloud dispatcher for executing containers on worker VMs.
+ # Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
# and ends with "\n-----END RSA PRIVATE KEY-----\n".
DispatchPrivateKey: ""
# Minimum time between two attempts to run the same container
MinRetryPeriod: 0s
- # Container runtime: "docker" (default) or "singularity" (experimental)
+ # Container runtime: "docker" (default) or "singularity"
RuntimeEngine: docker
Logging:
GitInternalDir: /var/lib/arvados/internal.git
CloudVMs:
- # Enable the cloud scheduler (experimental).
+ # Enable the cloud scheduler.
Enable: false
# Name/number of port where workers' SSH services listen.
projects / arvados.git / commitdiff