Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
- Additional services:
- install/install-ws.html.textile.liquid
- install/install-arv-git-httpd.html.textile.liquid
- Additional services:
- install/install-ws.html.textile.liquid
- install/install-arv-git-httpd.html.textile.liquid
- - install/install-shell-server.html.textile.liquid
+ - install/install-shell-server.html.textile.liquid
+ - Containers API support on cloud:
+ - install/install-dispatch-cloud.html.textile.liquid
- Containers API support on SLURM:
- install/crunch2-slurm/install-prerequisites.html.textile.liquid
- install/crunch2-slurm/install-slurm.html.textile.liquid
- install/crunch2-slurm/install-compute-node.html.textile.liquid
- install/crunch2-slurm/install-dispatch.html.textile.liquid
- install/crunch2-slurm/install-test.html.textile.liquid
- Containers API support on SLURM:
- install/crunch2-slurm/install-prerequisites.html.textile.liquid
- install/crunch2-slurm/install-slurm.html.textile.liquid
- install/crunch2-slurm/install-compute-node.html.textile.liquid
- install/crunch2-slurm/install-dispatch.html.textile.liquid
- install/crunch2-slurm/install-test.html.textile.liquid
- - install/install-nodemanager.html.textile.liquid
- - install/install-compute-ping.html.textile.liquid
- - Containers API support on cloud (beta):
- - install/install-dispatch-cloud.html.textile.liquid
- External dependencies:
- install/install-postgresql.html.textile.liquid
- install/ruby.html.textile.liquid
- External dependencies:
- install/install-postgresql.html.textile.liquid
- install/ruby.html.textile.liquid
Compute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution.
Compute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution.
-For Debian-based systems, the Arvados package repository includes a backported @docker.io@ package with a known-good version you can install.
-
h2(#configure_docker_daemon). Configure the Docker daemon
Crunch runs Docker containers with relatively little configuration. You may need to start the Docker daemon with specific options to make sure these jobs run smoothly in your environment. This section highlights options that are useful to most installations. Refer to the "Docker daemon reference":https://docs.docker.com/reference/commandline/daemon/ for complete information about all available options.
The best way to configure these options varies by distribution.
h2(#configure_docker_daemon). Configure the Docker daemon
Crunch runs Docker containers with relatively little configuration. You may need to start the Docker daemon with specific options to make sure these jobs run smoothly in your environment. This section highlights options that are useful to most installations. Refer to the "Docker daemon reference":https://docs.docker.com/reference/commandline/daemon/ for complete information about all available options.
The best way to configure these options varies by distribution.
-* If you're using our backported @docker.io@ package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker.io@.
-* If you're using another Debian-based package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker@.
+* If you're using the Debian package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker@.
* On Red Hat-based distributions, you can list these options in the @other_args@ setting in @/etc/sysconfig/docker@.
h3. Default ulimits
* On Red Hat-based distributions, you can list these options in the @other_args@ setting in @/etc/sysconfig/docker@.
h3. Default ulimits
-# Set the maximum number of FUSE mounts allowed to non-root users.
-# The default is 1000.
-#
-#mount_max = 1000
-
# Allow non-root users to specify the 'allow_other' or 'allow_root'
# mount options.
#
# Allow non-root users to specify the 'allow_other' or 'allow_root'
# mount options.
#
h2. Configure the Docker cleaner
h2. Configure the Docker cleaner
-The arvados-docker-cleaner program removes least recently used Docker images as needed to keep disk usage below a configured limit.
+The @arvados-docker-cleaner@ program removes least recently used Docker images as needed to keep disk usage below a configured limit.
{% include 'notebox_begin' %}
This also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with @"RemoveStoppedContainers":"never"@.
{% include 'notebox_begin' %}
This also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with @"RemoveStoppedContainers":"never"@.
-First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
{% include 'note_python_sc' %}
{% include 'note_python_sc' %}
-On Red Hat-based systems:
-
<notextile>
<pre><code>~$ <span class="userinput">echo 'exclude=python2-llfuse' | sudo tee -a /etc/yum.conf</span>
~$ <span class="userinput">sudo yum install python-arvados-fuse crunch-run arvados-docker-cleaner</span>
</code></pre>
</notextile>
<notextile>
<pre><code>~$ <span class="userinput">echo 'exclude=python2-llfuse' | sudo tee -a /etc/yum.conf</span>
~$ <span class="userinput">sudo yum install python-arvados-fuse crunch-run arvados-docker-cleaner</span>
</code></pre>
</notextile>
-On Debian-based systems:
<notextile>
<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-fuse crunch-run arvados-docker-cleaner</span>
<notextile>
<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-fuse crunch-run arvados-docker-cleaner</span>
layout: default
navsection: installguide
title: Install the cloud dispatcher
layout: default
navsection: installguide
title: Install the cloud dispatcher
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-The cloud dispatch service is an *experimental* service for running containers on cloud VMs. It eliminates the need for SLURM, Node Manager, and SLURM dispatcher. It works with Microsoft Azure and Amazon EC2; future versions will also support Google Compute Engine.
+# "Update config.yml":#update-config
+# "Install arvados-dispatch-cloud":#install-packages
+# "Confirm working installation":#confirm-working
+
+The cloud dispatch service is for running containers on cloud VMs. It works with Microsoft Azure and Amazon EC2; future versions will also support Google Compute Engine.
The cloud dispatch service can run on any node that can connect to the Arvados API service, the cloud provider's API, and the SSH service on cloud VMs. It is not resource-intensive, so you can run it on the API server node.
The cloud dispatch service can run on any node that can connect to the Arvados API service, the cloud provider's API, and the SSH service on cloud VMs. It is not resource-intensive, so you can run it on the API server node.
</code></pre>
</notextile>
</code></pre>
</notextile>
-h2. Create a dispatcher token
-
-If you haven't already done so, create an Arvados superuser token to use as SystemRootToken in your cluster config file.
-
-{% include 'create_superuser_token' %}
+h2(#update-config). Update config.yml
-h2. Create a private key
+h3. Create a private key
Generate an SSH private key with no passphrase. Save it in the cluster configuration file (see @PrivateKey@ in the example below).
Generate an SSH private key with no passphrase. Save it in the cluster configuration file (see @PrivateKey@ in the example below).
</code></pre>
</notextile>
</code></pre>
</notextile>
-h2. Configure the dispatcher
Add or update the following portions of your cluster configuration file, @/etc/arvados/config.yml@. Refer to "config.defaults.yml":{{site.baseurl}}/admin/config.html for information about additional configuration options.
<notextile>
Add or update the following portions of your cluster configuration file, @/etc/arvados/config.yml@. Refer to "config.defaults.yml":{{site.baseurl}}/admin/config.html for information about additional configuration options.
<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- ManagementToken: xyzzy
- SystemRootToken: <span class="userinput">zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz</span>
- Services:
- Controller:
- ExternalURL: "https://<span class="userinput">uuid_prefix.arvadosapi.com</span>"
DispatchCloud:
InternalURLs:
"http://localhost:9006": {}
DispatchCloud:
InternalURLs:
"http://localhost:9006": {}
Minimal configuration example for Amazon EC2:
<notextile>
Minimal configuration example for Amazon EC2:
<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- Containers:
CloudVMs:
ImageID: ami-01234567890abcdef
Driver: ec2
CloudVMs:
ImageID: ami-01234567890abcdef
Driver: ec2
Minimal configuration example for Azure:
<notextile>
Minimal configuration example for Azure:
<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- Containers:
CloudVMs:
ImageID: "https://zzzzzzzz.blob.core.windows.net/system/Microsoft.Compute/Images/images/zzzzz-compute-osDisk.55555555-5555-5555-5555-555555555555.vhd"
Driver: azure
CloudVMs:
ImageID: "https://zzzzzzzz.blob.core.windows.net/system/Microsoft.Compute/Images/images/zzzzz-compute-osDisk.55555555-5555-5555-5555-555555555555.vhd"
Driver: azure
</code></pre>
</notextile>
</code></pre>
</notextile>
-h2. Test your configuration
-
-First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
-
-Next, install the arvados-server package.
-
-On Red Hat-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install arvados-server</span>
-</code></pre>
-</notextile>
-
-On Debian-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install arvados-server</span>
-</code></pre>
-</notextile>
+h3. Test your configuration
Run the @cloudtest@ tool to verify that your configuration works. This creates a new cloud VM, confirms that it boots correctly and accepts your configured SSH private key, and shuts it down.
Run the @cloudtest@ tool to verify that your configuration works. This creates a new cloud VM, confirms that it boots correctly and accepts your configured SSH private key, and shuts it down.
Refer to the "cloudtest tool documentation":../admin/cloudtest.html for more information.
Refer to the "cloudtest tool documentation":../admin/cloudtest.html for more information.
-h2. Install the dispatcher
+h2. Install arvados-dispatch-cloud
+
+{% include 'notebox_begin' %}
+
+The arvados-dispatch-cloud package includes configuration files for systemd. If you're using a different init system, configure a service to start and stop an @arvados-dispatch-cloud@ process as desired.
-On Red Hat-based systems:
+{% include 'notebox_end' %}
+
+h3. Centos 7
-<pre><code>~$ <span class="userinput">sudo yum install arvados-dispatch-cloud</span>
-~$ <span class="userinput">sudo systemctl enable arvados-dispatch-cloud</span>
+<pre><code># <span class="userinput">yum install arvados-dispatch-cloud</span>
</code></pre>
</notextile>
</code></pre>
</notextile>
-On Debian-based systems:
-<pre><code>~$ <span class="userinput">sudo apt-get install arvados-dispatch-cloud</span>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install arvados-dispatch-cloud</span>
</code></pre>
</notextile>
</code></pre>
</notextile>
-{% include 'notebox_begin' %}
-
-The arvados-dispatch-cloud package includes configuration files for systemd. If you're using a different init system, configure a service to start and stop an @arvados-dispatch-cloud@ process as desired.
-
-{% include 'notebox_end' %}
-
-h2. Verify the dispatcher is running
+h2(#confirm-working). Confirm working installation
-Use your ManagementToken to test the dispatcher's metrics endpoint.
+Use your @ManagementToken@ to test the dispatcher's metrics endpoint.
<notextile>
<pre><code>~$ <span class="userinput">token="xyzzy"</span>
<notextile>
<pre><code>~$ <span class="userinput">token="xyzzy"</span>