From f9ffb421cddf3a1bfcb6d79b31958b4d54ed5906 Mon Sep 17 00:00:00 2001 From: Brett Smith Date: Mon, 8 Aug 2016 10:35:55 -0400 Subject: [PATCH] 9705: Refactor out partials from compute node install guide. These will be reused in the Crunch2 install guide. --- doc/_includes/_install_compute_docker.liquid | 45 ++++++++++ doc/_includes/_install_compute_fuse.liquid | 17 ++++ doc/_includes/_install_docker_cleaner.liquid | 39 +++++++++ .../install-compute-node.html.textile.liquid | 82 +------------------ 4 files changed, 104 insertions(+), 79 deletions(-) create mode 100644 doc/_includes/_install_compute_docker.liquid create mode 100644 doc/_includes/_install_compute_fuse.liquid create mode 100644 doc/_includes/_install_docker_cleaner.liquid diff --git a/doc/_includes/_install_compute_docker.liquid b/doc/_includes/_install_compute_docker.liquid new file mode 100644 index 0000000000..915db02465 --- /dev/null +++ b/doc/_includes/_install_compute_docker.liquid @@ -0,0 +1,45 @@ +h2. Install Docker + +Compute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution. + +For Debian-based systems, the Arvados package repository includes a backported @docker.io@ package with a known-good version you can install. + +h2. Configure the Docker daemon + +Crunch runs Docker containers with relatively little configuration. You may need to start the Docker daemon with specific options to make sure these jobs run smoothly in your environment. This section highlights options that are useful to most installations. Refer to the "Docker daemon reference":https://docs.docker.com/reference/commandline/daemon/ for complete information about all available options. + +The best way to configure these options varies by distribution. + +* If you're using our backported @docker.io@ package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker.io@. +* If you're using another Debian-based package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker@. +* On Red Hat-based distributions, you can list these options in the @other_args@ setting in @/etc/sysconfig/docker@. + +h3. Default ulimits + +Docker containers inherit ulimits from the Docker daemon. However, the ulimits for a single Unix daemon may not accommodate a long-running Crunch job. You may want to increase default limits for compute containers by passing @--default-ulimit@ options to the Docker daemon. For example, to allow containers to open 10,000 files, set @--default-ulimit nofile=10000:10000@. + +h3. DNS + +Your containers must be able to resolve the hostname of your API server and any hostnames returned in Keep service records. If these names are not in public DNS records, you may need to specify a DNS resolver for the containers by setting the @--dns@ address to an IP address of an appropriate nameserver. You may specify this option more than once to use multiple nameservers. + +h2. Configure Linux cgroups accounting + +Linux can report what compute resources are used by processes in a specific cgroup or Docker container. Crunch can use these reports to share that information with users running compute work. This can help pipeline authors debug and optimize their workflows. + +To enable cgroups accounting, you must boot Linux with the command line parameters @cgroup_enable=memory swapaccount=1@. + +On Debian-based systems, open the file @/etc/default/grub@ in an editor. Find where the string @GRUB_CMDLINE_LINUX@ is set. Add @cgroup_enable=memory swapaccount=1@ to that string. Save the file and exit the editor. Then run: + + +
$ sudo update-grub
+
+ + +On Red Hat-based systems, run: + + +
$ sudo grubby --update-kernel=ALL --args='cgroup_enable=memory swapaccount=1'
+
+ + +Finally, reboot the system to make these changes effective. diff --git a/doc/_includes/_install_compute_fuse.liquid b/doc/_includes/_install_compute_fuse.liquid new file mode 100644 index 0000000000..2bf315207e --- /dev/null +++ b/doc/_includes/_install_compute_fuse.liquid @@ -0,0 +1,17 @@ +h2. Configure FUSE + +FUSE must be configured with the @user_allow_other@ option enabled for Crunch to set up Keep mounts that are readable by containers. Install this file as @/etc/fuse.conf@: + + +
+# Set the maximum number of FUSE mounts allowed to non-root users.
+# The default is 1000.
+#
+#mount_max = 1000
+
+# Allow non-root users to specify the 'allow_other' or 'allow_root'
+# mount options.
+#
+user_allow_other
+
+ diff --git a/doc/_includes/_install_docker_cleaner.liquid b/doc/_includes/_install_docker_cleaner.liquid new file mode 100644 index 0000000000..e26b2be586 --- /dev/null +++ b/doc/_includes/_install_docker_cleaner.liquid @@ -0,0 +1,39 @@ +h2. Configure the Docker cleaner + +The arvados-docker-cleaner program removes least recently used Docker images as needed to keep disk usage below a configured limit. + +{% include 'notebox_begin' %} +This also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or run it with @--remove-stopped-containers never@. +{% include 'notebox_end' %} + +Install runit to supervise the Docker cleaner daemon. {% include 'install_runit' %} + +Configure runit to run the image cleaner using a suitable quota for your compute nodes and workload: + + +
~$ sudo mkdir -p /etc/sv
+~$ cd /etc/sv
+/etc/sv$ sudo mkdir arvados-docker-cleaner; cd arvados-docker-cleaner
+/etc/sv/arvados-docker-cleaner$ sudo mkdir log log/main
+/etc/sv/arvados-docker-cleaner$ sudo sh -c 'cat >log/run' <<'EOF'
+#!/bin/sh
+exec svlogd -tt main
+EOF
+/etc/sv/arvados-docker-cleaner$ sudo sh -c 'cat >run' <<'EOF'
+#!/bin/sh
+if [ -d /opt/rh/python33 ]; then
+  source scl_source enable python33
+fi
+exec python3 -m arvados_docker.cleaner --quota 50G
+EOF
+/etc/sv/arvados-docker-cleaner$ sudo chmod +x run log/run
+/etc/sv/arvados-docker-cleaner$ sudo ln -s "$(pwd)" /etc/service/
+
+ + +If you are using a different daemon supervisor, or if you want to test the daemon in a terminal window, an equivalent shell command to run arvados-docker-cleaner is: + + +
python3 -m arvados_docker.cleaner --quota 50G
+
+ diff --git a/doc/install/install-compute-node.html.textile.liquid b/doc/install/install-compute-node.html.textile.liquid index f55bceb561..b4d0d596f2 100644 --- a/doc/install/install-compute-node.html.textile.liquid +++ b/doc/install/install-compute-node.html.textile.liquid @@ -32,29 +32,7 @@ On Debian-based systems: -h2. Install Docker - -Compute nodes must have Docker installed to run jobs inside containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution. - -For Debian-based systems, the Arvados package repository includes a backported @docker.io@ package with a known-good version you can install. - -h2. Configure Docker - -Crunch runs jobs in Docker containers with relatively little configuration. You may need to start the Docker daemon with specific options to make sure these jobs run smoothly in your environment. This section highlights options that are useful to most installations. Refer to the "Docker daemon reference":https://docs.docker.com/reference/commandline/daemon/ for complete information about all available options. - -The best way to configure these options varies by distribution. - -* If you're using our backported @docker.io@ package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker.io@. -* If you're using another Debian-based package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker@. -* On Red Hat-based distributions, you can list these options in the @other_args@ setting in @/etc/sysconfig/docker@. - -h3. Default ulimits - -Docker containers inherit ulimits from the Docker daemon. However, the ulimits for a single Unix daemon may not accommodate a long-running Crunch job. You may want to increase default limits for compute jobs by passing @--default-ulimit@ options to the Docker daemon. For example, to allow jobs to open 10,000 files, set @--default-ulimit nofile=10000:10000@. - -h3. DNS - -Your containers must be able to resolve the hostname in the ARVADOS_API_HOST environment variable (provided by the Crunch dispatcher) and any hostnames returned in Keep service records. If these names are not in public DNS records, you may need to set a DNS resolver for the containers by specifying the @--dns@ address with the IP address of an appropriate nameserver. You may specify this option more than once to use multiple nameservers. +{% include 'install_compute_docker' %} h2. Set up SLURM @@ -64,63 +42,9 @@ h2. Copy configuration files from the dispatcher (API server) The @slurm.conf@ and @/etc/munge/munge.key@ files need to be identical across the dispatcher and all compute nodes. Copy the files you created in the "Install the Crunch dispatcher":install-crunch-dispatch.html step to this compute node. -h2. Configure FUSE - -Install this file as @/etc/fuse.conf@: - - -
-# Set the maximum number of FUSE mounts allowed to non-root users.
-# The default is 1000.
-#
-#mount_max = 1000
-
-# Allow non-root users to specify the 'allow_other' or 'allow_root'
-# mount options.
-#
-user_allow_other
-
- +{% include 'install_compute_fuse' %} -h2. Configure the Docker cleaner - -The arvados-docker-cleaner program removes least recently used docker images as needed to keep disk usage below a configured limit. - -{% include 'notebox_begin' %} -This also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or run it with @--remove-stopped-containers never@. -{% include 'notebox_end' %} - -Install runit to supervise the Docker cleaner daemon. {% include 'install_runit' %} - -Configure runit to run the image cleaner using a suitable quota for your compute nodes and workload: - - -
~$ sudo mkdir -p /etc/sv
-~$ cd /etc/sv
-/etc/sv$ sudo mkdir arvados-docker-cleaner; cd arvados-docker-cleaner
-/etc/sv/arvados-docker-cleaner$ sudo mkdir log log/main
-/etc/sv/arvados-docker-cleaner$ sudo sh -c 'cat >log/run' <<'EOF'
-#!/bin/sh
-exec svlogd -tt main
-EOF
-/etc/sv/arvados-docker-cleaner$ sudo sh -c 'cat >run' <<'EOF'
-#!/bin/sh
-if [ -d /opt/rh/python33 ]; then
-  source scl_source enable python33
-fi
-exec python3 -m arvados_docker.cleaner --quota 50G
-EOF
-/etc/sv/arvados-docker-cleaner$ sudo chmod +x run log/run
-/etc/sv/arvados-docker-cleaner$ sudo ln -s "$(pwd)" /etc/service/
-
- - -If you are using a different daemon supervisor, or if you want to test the daemon in a terminal window, an equivalent shell command to run arvados-docker-cleaner is: - - -
python3 -m arvados_docker.cleaner --quota 50G
-
- +{% include 'install_docker_cleaner' %} h2. Add a Crunch user account -- 2.30.2