---
layout: default
navsection: installguide
-title: Prerequisites
+title: Planning and prerequisites
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-h2. Supported Cloud and HPC platforms
+Before attempting installation, you should begin by reviewing supported platforms, choosing backends for identity, storage, and scheduling, and decide how you will distribute Arvados services onto machines. You should also choose an Arvados Cluster ID, choose your hostnames, and aquire TLS certificates. It may be helpful to make notes as you go along.
-Arvados can run in a variety of configurations. For compute scheduling, Arvados supports HPC clusters using @slurm@, and supports elastic cloud computing on AWS, Google and Azure. For storage, Arvados can store blocks on regular file systems such as ext4 or xfs, on network file systems such as GPFS, or object storage such as Azure blob storage, Amazon S3, and other object storage that supports the S3 API including Google Cloud Storage and Ceph.
+The Arvados storage subsystem is called "keep". The compute subsystem is called "crunch".
-h2. Hardware (or virtual machines)
+# "Supported GNU/Linux distributions":#supportedlinux
+# "Identity provider":#identity
+# "Storage backend (Keep)":#storage
+# "Container compute scheduler (Crunch)":#scheduler
+# "Hardware or virtual machines":#machines
+# "Arvados Cluster ID":#clusterid
+# "DNS and TLS":#dnstls
-This guide assumes you have seven systems available in the same network subnet:
-
-<div class="offset1">
-table(table table-bordered table-condensed).
-|_. Function|_. Number of nodes|
-|Arvados API, Crunch dispatcher, Git, Websockets and Workbench|1|
-|Arvados Compute node|1|
-|Arvados Keepproxy and Keep-web server|1|
-|Arvados Keepstore servers|2|
-|Arvados Shell server|1|
-|Arvados SSO server|1|
-</div>
-
-The number of Keepstore, shell and compute nodes listed above is a minimum. In a real production installation, you will likely run many more of each of those types of nodes. In such a scenario, you would probably also want to dedicate a node to the Workbench server and Crunch dispatcher, respectively. For performance reasons, you may want to run the database server on a separate node as well.
-
-h2. Supported GNU/Linux distributions
+h2(#supportedlinux). Supported GNU/Linux distributions
table(table table-bordered table-condensed).
|_. Distribution|_. State|_. Last supported version|
|CentOS 7|Supported|Latest|
+|Debian 10 ("buster")|Supported|Latest|
|Debian 9 ("stretch")|Supported|Latest|
|Ubuntu 16.04 ("xenial")|Supported|Latest|
|Ubuntu 18.04 ("bionic")|Supported|Latest|
Arvados packages are published for current Debian releases (until the EOL date), current Ubuntu LTS releases (until the end of standard support), and the latest version of CentOS.
-h2(#repos). Arvados package repositories
+h2(#identity). Identity provider
-On any host where you install Arvados software, you'll need to set up an Arvados package repository. They're available for several popular distributions.
+Choose which backend you will use to authenticate users.
-h3. CentOS
+* Google login to authenticate users with a Google account. Note: if you only use Google, you can use @arvados-controller@ support for Google login, and do not need to install the Arvados Single Sign-On server (SSO).
+* LDAP login to authenticate users using the LDAP protocol, supported by many services such as OpenLDAP and Active Directory. Supports username/password authentication.
+* Standalone SSO server user database. Supports username/password authentication. Supports new user sign-up.
-Packages are available for CentOS 7. To install them with yum, save this configuration block in @/etc/yum.repos.d/arvados.repo@:
+h2(#storage). Storage backend
-<notextile>
-<pre><code>[arvados]
-name=Arvados
-baseurl=http://rpm.arvados.org/CentOS/$releasever/os/$basearch/
-gpgcheck=1
-gpgkey=http://rpm.arvados.org/CentOS/RPM-GPG-KEY-curoverse
-</code></pre>
-</notextile>
+Choose which backend you will use for storing and retrieving content-addressed Keep blocks.
-{% include 'install_redhat_key' %}
+* File systems storage, such as ext4 or xfs, or network file systems such as GPFS or Lustre
+* Amazon S3, or other object storage that supports the S3 API including Google Cloud Storage and Ceph.
+* Azure blob storage
-h3. Debian and Ubuntu
+You should also determine the desired replication factor for your data. A replication factor of 1 means only a single copy of a given data block is kept. With a conventional file system backend and a replication factor of 1, a hard drive failure is likely to lose data. For this reason the default replication factor is 2 (two copies are kept).
-Packages are available for Debian 9 ("stretch"), Ubuntu 16.04 ("xenial") and Ubuntu 18.04 ("bionic").
+A backend may have its own replication factor (such as durability guarantees of cloud buckets) and Arvados will take this into account when writing a new data block.
-First, register the Curoverse signing key in apt's database:
+h2(#scheduler). Container compute scheduler
-{% include 'install_debian_key' %}
+Choose which backend you will use to schedule computation.
-Configure apt to retrieve packages from the Arvados package repository. This command depends on your OS vendor and version:
+* On AWS EC2 and Azure, you probably want to use @crunch-dispatch-cloud@ to manage the full lifecycle of cloud compute nodes: starting up nodes sized to the container request, executing containers on those nodes, and shutting nodes down when no longer needed.
+* For on-premise HPC clusters using "slurm":https://slurm.schedmd.com/ use @crunch-dispatch-slurm@ to execute containers with slurm job submissions.
+* For single node demos, use @crunch-dispatch-local@ to execute containers directly.
-table(table table-bordered table-condensed).
-|_. OS version|_. Command|
-|Debian 9 ("stretch")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ stretch main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Ubuntu 16.04 ("xenial")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ xenial main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Ubuntu 18.04 ("bionic")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ bionic main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+h2(#machines). Hardware (or virtual machines)
-{% include 'notebox_begin' %}
-
-fn1. Arvados packages for Ubuntu may depend on third-party packages in Ubuntu's "universe" repository. If you're installing on Ubuntu, make sure you have the universe sources uncommented in @/etc/apt/sources.list@.
+For a production installation, here is a suggested configuration:
-{% include 'notebox_end' %}
+<div class="offset1">
+table(table table-bordered table-condensed).
+|_. Function|_. Number of nodes|_. Recommended specs|
+|Postgres database, Arvados API server, Arvados controller, Git, Websockets|1|16+ GiB RAM, 4+ cores, fast disk for database|
+|Single Sign-On (SSO) server ^1^|1|2 GiB RAM|
+|Workbench, Keepproxy, Keep-web, Keep-balance|1|8 GiB RAM, 2+ cores|
+|Keepstore servers ^2^|2+|4 GiB RAM|
+|Compute scheduler|1|2 GiB RAM|
+|Compute worker nodes ^2^|2+ |Depends on workload|
+|User shell nodes ^3^|0+|Depends on workload|
+</div>
-Retrieve the package list:
+^1^ May be omitted when using Google login support in @arvados-controller@
+^2^ Should be scaled up as needed
+^3^ Refers to shell nodes managed by Arvados, that provide ssh access for users to interact with Arvados at the command line. Optional.
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get update</span>
-</code></pre>
-</notextile>
+{% include 'notebox_begin' %}
+For a small demo installation, it is possible to run all the Arvados services on a single node. Special considerations for single-node installs will be noted in boxes like this.
+{% include 'notebox_end' %}
-h2. A unique identifier
+h2(#clusterid). Arvados Cluster ID
-Each Arvados installation should have a globally unique identifier, which is a unique 5-character lowercase alphanumeric string. For testing purposes, here is one way to make a random 5-character string:
+Each Arvados installation should have a cluster identifier, which is a unique 5-character lowercase alphanumeric string. Here is one way to make a random 5-character string:
<notextile>
<pre><code>~$ <span class="userinput">tr -dc 0-9a-z </dev/urandom | head -c5; echo</span>
</code></pre>
</notextile>
-You may also use a different method to pick the unique identifier. The unique identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @uuid_prefix@.
-
-
-h2. SSL certificates
-
-There are six public-facing services that require an SSL certificate. If you do not have official SSL certificates, you can use self-signed certificates.
-
-{% include 'notebox_begin' %}
-
-Most Arvados clients and services will accept self-signed certificates when the @ARVADOS_API_HOST_INSECURE@ environment variable is set to @true@. However, web browsers generally do not make it easy for users to accept self-signed certificates from Web sites.
-
-Users who log in through Workbench will visit at least three sites: the SSO server, the API server, and Workbench itself. When a browser visits each of these sites, it will warn the user if the site uses a self-signed certificate, and the user must accept it before continuing. This procedure usually only needs to be done once in a browser.
+You may also use a different method to pick the cluster identifier. The cluster identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @ClusterID@.
-After that's done, Workbench includes JavaScript clients for other Arvados services. Users are usually not warned if these client connections are refused because the server uses a self-signed certificate, and it is especially difficult to accept those cerficiates:
+h2(#dnstls). DNS entries and TLS certificates
-* JavaScript connects to the Websockets server to provide incremental page updates and view logs from running jobs.
-* JavaScript connects to the API and Keepproxy servers to upload local files to collections.
-* JavaScript connects to the Keep-web server to download log files.
-
-In sum, Workbench will be much less pleasant to use in a cluster that uses self-signed certificates. You should avoid using self-signed certificates unless you plan to deploy a cluster without Workbench; you are deploying only to evaluate Arvados as an individual system administrator; or you can push configuration to users' browsers to trust your self-signed certificates.
-
-{% include 'notebox_end' %}
-
-By convention, we use the following hostname pattern:
+The following services are normally public-facing and require DNS entries and corresponding TLS certificates. Get certificates from your preferred TLS certificate provider. We recommend using "Let's Encrypt":https://letsencrypt.org/ . You can run several services on same node, but each distinct hostname requires its own TLS certificate. A later part of this guide will describe how to set up Nginx virtual hosts.
<div class="offset1">
table(table table-bordered table-condensed).
|_. Function|_. Hostname|
-|Arvados API|@uuid_prefix@.your.domain|
-|Arvados Git server|git.@uuid_prefix@.your.domain|
-|Arvados Keepproxy server|keep.@uuid_prefix@.your.domain|
-|Arvados Keep-web server|download.@uuid_prefix@.your.domain
-_and_
-*.collections.@uuid_prefix@.your.domain or
-*<notextile>--</notextile>collections.@uuid_prefix@.your.domain or
-collections.@uuid_prefix@.your.domain (see the "keep-web install docs":install-keep-web.html)|
+|Arvados API|@ClusterID@.your.domain|
+|Arvados Git server|git.@ClusterID@.your.domain|
+|Arvados Websockets endpoint|ws.@ClusterID@.your.domain|
|Arvados SSO Server|auth.your.domain|
-|Arvados Websockets endpoint|ws.@uuid_prefix@.your.domain|
-|Arvados Workbench|workbench.@uuid_prefix@.your.domain|
+|Arvados Workbench|workbench.@ClusterID@.your.domain|
+|Arvados Workbench 2|workbench2.@ClusterID@.your.domain|
+|Arvados Keepproxy server|keep.@ClusterID@.your.domain|
+|Arvados Keep-web server|download.@ClusterID@.your.domain
+_and_
+*.collections.@ClusterID@.your.domain or
+*<notextile>--</notextile>collections.@ClusterID@.your.domain or
+collections.@ClusterID@.your.domain (see the "keep-web install docs":install-keep-web.html)|
</div>
+
+{% include 'notebox_begin' %}
+It is also possible to create your own certificate authority, issue server certificates, and install a custom root certificate in the browser. This is out of scope for this guide.
+{% include 'notebox_end' %}
projects / arvados.git / commitdiff