# "Limitations of the single host install":#limitations
# "Prerequisites and planning":#prerequisites
# "Download the installer":#download
-# "Edit local.params":#localparams
+# "Edit local.params* files":#localparams
# "Choose the SSL configuration":#certificates
## "Using a self-signed certificate":#self-signed
## "Using a Let's Encrypt certificate":#lets-encrypt
# "Configure your authentication provider":#authentication
# "Further customization of the installation":#further_customization
# "Begin installation":#installation
-# "Confirm the cluster is working":#test-install
# "Install the CA root certificate":#ca_root_certificate
+# "Confirm the cluster is working":#test-install
# "Initial user and login":#initial_user
# "After the installation":#post_install
*NOTE: The single host installation is a good choice for evaluating Arvados, but it is not recommended for production use.*
-Using the default configuration, has a number of limitations:
+Using the default configuration, the single host install has scaling limitations compared to a production multi-host install:
-* All services run on the same machine, and they will compete for resources. This includes any compute jobs.
-* It uses the local machine disk for Keep storage (under the @/tmp@ directory). There may not be a lot of space available.
-* It installs the @crunch-dispatch-local@ dispatcher, which has a limit of eight concurrent jobs. These jobs will be executed on the same machine that runs all the Arvados services may compete for resources.
+* It uses the local disk for Keep storage (under the @/var/lib/arvados@ directory).
+* It uses the @crunch-dispatch-local@ dispatcher, which has a limit of eight concurrent jobs.
+* Because jobs and Arvados services all run on the same machine, they will compete for CPU/RAM resources.
h2(#prerequisites). Prerequisites and planning
h3. Cluster ID and base domain
-Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or ${CLUSTER}, you should substitute the cluster id you have selected.
+Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or @${CLUSTER}@, you should substitute the cluster id you have selected.
Determine if you will use a single hostname, or multiple hostnames.
* Single hostname is simpler to set up and can even be used without a hostname at all, just a bare IP address.
* Multiple hostnames is more similar to the recommended production configuration may make it easier to migrate to a multi-host production configuration in the future, but is more complicated as it requires adding a number of DNS entries.
-If you are using multiple hostnames, determine the base domain for the cluster. This will be referred to as ${DOMAIN}
+If you are using multiple hostnames, determine the base domain for the cluster. This will be referred to as @${DOMAIN}@.
-For example, if CLUSTER is "xarv1" and DOMAIN is "example.com", then "controller.${CLUSTER}.${DOMAIN}" means "controller.xargv1.example.com".
+For example, if CLUSTER is @xarv1@ and DOMAIN is @example.com@, then @controller.${CLUSTER}.${DOMAIN}@ means @controller.xarv1.example.com@.
h3. Machine specification
Note: if you want to try out Arvados inside a Docker container, use "Arvbox":arvbox.html. The package-based install method uses @systemd@ to manage services; lightweight container images generally lack an init system and other tools that the installer requires.
The single host install stores user data in a PostgreSQL database (usually found under @/var/lib/postgresql@) and as Keep blocks that are stored as files under @/var/lib/arvados/@.
-Arvados logs are also kept in @/var/log@ and @/var/www/arvados-api/shared/log@. Accordingly, you should ensure that the disk partition containing @/var@ has adequate storage for your planned usage. We suggest starting with 50GiB of free space.
+Arvados logs are also kept in @/var/log@ and @/var/www/arvados-api/shared/log@. Accordingly, you should ensure that the disk partition containing @/var@ has adequate storage for your planned usage. We suggest starting with at least 50GiB of free space.
h3(#DNS). DNS hostnames for each service (multi-hostname only)
# @workbench2.${CLUSTER}.${DOMAIN}@
# @webshell.${CLUSTER}.${DOMAIN}@
# @shell.${CLUSTER}.${DOMAIN}@
+# @prometheus.${CLUSTER}.${DOMAIN}@
+# @grafana.${CLUSTER}.${DOMAIN}@
This is described in more detail in "DNS entries and TLS certificates":install-manual-prerequisites.html#dnstls.
# root or passwordless @sudo@ access on the account where you are doing the install
this usually means adding the account to the @sudo@ group and having a rule like this in @/etc/sudoers.d/arvados_passwordless@ that allows members of group @sudo@ to execute any command without entering a password.
-<pre>%sudo all=(all:all) nopasswd:all</pre>
+<pre>%sudo ALL=(ALL:ALL) NOPASSWD:ALL</pre>
# @git@ installed on the machine
# Port 443 reachable by clients
# For the single-host install, ports 8800-8805 also need to be reachable from your client (configurable in @local.params@, see below)
If you are using multiple hostname configuration, substitute 'multiple_hostnames' where it says 'single_hostname' in the command above.
-h2(#localparams). Edit @local.params@
+h2(#localparams). Edit @local.params*@ files
+
+The cluster configuration parameters are included in two files: @local.params@ and @local.params.secrets@. These files can be found wherever you choose to initialize the installation files (e.g., @~/setup-arvados-xarv1@ in these examples).
-This can be found wherever you choose to initialize the install files (@~/setup-arvados-xarv1@ in these examples).
+The @local.params.secrets@ file is intended to store security-sensitive data such as passwords, private keys, tokens, etc. Depending on the security requirements of the cluster deployment, you may wish to store this file in a secrets store like AWS Secrets Manager or Jenkins credentials.
+
+h3. Parameters from @local.params@:
# Set @CLUSTER@ to the 5-character cluster identifier (e.g "xarv1")
# Set @DOMAIN@ to the base DNS domain of the environment, e.g. "example.com"
# Single hostname only: set @IP_INT@ to the host's IP address.
# Single hostname only: set @HOSTNAME_EXT@ to the hostname that users will use to connect.
# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.
+
+h3. Parameters from @local.params.secrets@:
+
# Set each @KEY@ / @TOKEN@ to a random string
Here's an easy way to create five random tokens:
<pre><code>for i in 1 2 3 4 5; do
For example, if the password is @Lq&MZ<V']d?j@
With backslash quoting the special characters it should appear like this in local.params:
<pre><code>DATABASE_PASSWORD="Lq\&MZ\<V\'\]d\?j"</code></pre>
-
+# Set @DISPATCHER_SSH_PRIVKEY@ to @"no"@, as it isn't needed.
{% include 'ssl_config_single' %}
h2(#authentication). Configure your authentication provider (optional, recommended)
./installer.sh deploy
</pre>
+h2(#ca_root_certificate). Install the CA root certificate (SSL_MODE=self-signed only)
+
+*If you are not using self-signed certificates (you selected SSL_MODE=lets-encrypt or SSL_MODE=bring-your-own), skip this section.*
+
+Arvados uses SSL to encrypt communications. The web interface uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the installer has the option to create its own a root certificate to authorize Arvados services. The installer script will leave a copy of the generated CA's certificate (something like @xarv1.example.com-arvados-snakeoil-ca.crt@) in the script's directory so you can add it to your workstation.
+
+{% assign ca_cert_name = 'xarv1.example.com-arvados-snakeoil-ca.crt' %}
+
+{% include 'install_ca_cert' %}
+
h2(#test-install). Confirm the cluster is working
-When everything has finished, you can run the diagnostics.
+When everything has finished, you can run the diagnostics. This requires the `arvados-client` package:
+
+<pre>
+apt-get install arvados-client
+</pre>
Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.
Most service logs go to @/var/log/syslog@.
-The logs for Rails API server and for Workbench can be found in
-
-@/var/www/arvados-api/current/log/production.log@
-and
-@/var/www/arvados-workbench/current/log/production.log@
-
-on the appropriate instances.
+The logs for Rails API server can be found in @/var/www/arvados-api/current/log/production.log@ on the appropriate instance.
Workbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in "Tools → Developer Tools").
<pre>dpkg-reconfigure arvados-api-server</pre>
# Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.
-h2(#ca_root_certificate). Install the CA root certificate (SSL_MODE=self-signed only)
+h2(#initial_user). Initial user and login
-*If you are not using self-signed certificates (you selected SSL_MODE=lets-encrypt or SSL_MODE=bring-your-own), skip this section.*
+At this point you should be able to log into the Arvados cluster. The initial URL for the single hostname install will use the hostname or IP address you put in @HOSTNAME_EXT@:
-Arvados uses SSL to encrypt communications. The web interface uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+https://${HOSTNAME_EXT}
-For this reason, the installer has the option to create its own a root certificate to authorize Arvados services. The installer script will leave a copy of the generated CA's certificate (something like @xarv1.example.com-arvados-snakeoil-ca.crt@) in the script's directory so you can add it to your workstation.
+For the multi-hostname install, it will be:
-{% assign ca_cert_name = 'xarv1.example.com-arvados-snakeoil-ca.crt' %}
+https://workbench.@${CLUSTER}.${DOMAIN}@
-{% include 'install_ca_cert' %}
+If you did *not* "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
-h2(#initial_user). Initial user and login
+If you *did* configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
-At this point you should be able to log into the Arvados cluster. The initial URL for the single hostname install will use the hostname or IP address you put in @HOSTNAME_EXT@:
+h2(#monitoring). Monitoring and Metrics
-https://${HOSTNAME_EXT}
+You can monitor the health and performance of the system using the admin dashboard.
For the multi-hostname install, it will be:
-https://workbench.${CLUSTER}.${DOMAIN}
+https://grafana.@${CLUSTER}.${DOMAIN}@
-If you did *not* "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
+To log in, use username "admin" and @${INITIAL_USER_PASSWORD}@ from @local.conf@.
-If you *did* configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
+Once logged in, you will want to add the dashboards to the front page.
+
+# On the left icon bar, click on "Browse"
+# If the check box next to "Starred" is selected, click on it to de-select it
+# You should see a folder with "Arvados cluster overview", "Node exporter" and "Postgres exporter"
+# You can visit each dashboard and click on the star next to the title to "Mark as favorite"
+# They should now be linked on the front page.
h2(#post_install). After the installation
projects / arvados.git / blobdiff