SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-# "Install Saltstack":#saltstack
-# "Single host install using the provision.sh script":#single_host
-# "Local testing Arvados in a Vagrant box":#vagrant
-# "DNS configuration":#final_steps
+# "Limitations of the single host install":#limitations
+# "Prerequisites and planning":#prerequisites
+# "Download the installer":#download
+# "Edit local.params":#localparams
+# "Choose the SSL configuration":#certificates
+## "Using a self-signed certificate":#self-signed
+## "Using a Let's Encrypt certificate":#lets-encrypt
+## "Bring your own certificate":#bring-your-own
+# "Configure your authentication provider":#authentication
+# "Further customization of the installation":#further_customization
+# "Begin installation":#installation
+# "Confirm the cluster is working":#test-install
+# "Install the CA root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
+# "After the installation":#post_install
-h2(#saltstack). Install Saltstack
+h2(#limitations). Limitations of the single host install
-If you already have a Saltstack environment you can skip this section.
+*NOTE: The single host installation is a good choice for evaluating Arvados, but it is not recommended for production use.*
-The simplest way to get Salt up and running on a node is to use the bootstrap script they provide:
+Using the default configuration, has a number of limitations:
-<notextile>
-<pre><code>curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
-sudo sh /tmp/bootstrap_salt.sh -XUdfP -x python3
-</code></pre>
-</notextile>
+* All services run on the same machine, and they will compete for resources. This includes any compute jobs.
+* It uses the local machine disk for Keep storage (under the @/tmp@ directory). There may not be a lot of space available.
+* It installs the @crunch-dispatch-local@ dispatcher, which has a limit of eight concurrent jobs. These jobs will be executed on the same machine that runs all the Arvados services may compete for resources.
-For more information check "Saltstack's documentation":https://docs.saltstack.com/en/latest/topics/installation/index.html
+h2(#prerequisites). Prerequisites and planning
-h2(#single_host). Single host install using the provision.sh script
+h3. Cluster ID and base domain
-This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+Choose a 5-character cluster identifier that will represent the cluster. Here are "guidelines on choosing a cluster identifier":../architecture/federation.html#cluster_id . Only lowercase letters and digits 0-9 are allowed. Examples will use @xarv1@ or ${CLUSTER}, you should substitute the cluster id you have selected.
-Use the @provision.sh@ script to deploy Arvados, which is implemented with the @arvados-formula@ in a Saltstack master-less setup:
+Determine if you will use a single hostname, or multiple hostnames.
-* edit the variables at the very beginning of the file,
-* run the script as root
-* wait for it to finish
+* Single hostname is simpler to set up and can even be used without a hostname at all, just a bare IP address.
+* Multiple hostnames is more similar to the recommended production configuration may make it easier to migrate to a multi-host production configuration in the future, but is more complicated as it requires adding a number of DNS entries.
-This will install all the main Arvados components to get you up and running. The whole installation procedure takes somewhere between 15 to 60 minutes, depending on the host and your network bandwidth. On a virtual machine with 1 core and 1 GB RAM, it takes ~25 minutes to do the initial install.
+If you are using multiple hostnames, determine the base domain for the cluster. This will be referred to as ${DOMAIN}
-If everything goes OK, you'll get some final lines stating something like:
+For example, if CLUSTER is "xarv1" and DOMAIN is "example.com", then "controller.${CLUSTER}.${DOMAIN}" means "controller.xargv1.example.com".
-<notextile>
-<pre><code>arvados: Succeeded: 109 (changed=9)
-arvados: Failed: 0
-</code></pre>
-</notextile>
+h3. Machine specification
+
+You will need a dedicated (virtual) machine for your Arvados server with at least 2 cores and 8 GiB of RAM (4+ cores / 16+ GiB recommended if you are running workflows) running a supported Linux distribution:
+
+{% include 'supportedlinux' %}
+
+The single host install stores user data in a PostgreSQL database (usually found under @/var/lib/postgresql@) and as Keep blocks that are stored as files under @/var/lib/arvados/@.
+Arvados logs are also kept in @/var/log@ and @/var/www/arvados-api/shared/log@. Accordingly, you should ensure that the disk partition containing @/var@ has adequate storage for your planned usage. We suggest starting with 50GiB of free space.
+
+h3(#DNS). DNS hostnames for each service (multi-hostname only)
+
+If you are using a single hostname for all services (they will be distingushed by listening port), you can skip this section.
+
+If you are using the multi-hostname configuration, you will need a DNS entry for each service. If you are using "bring-your-own" TLS certificates, your certificate will need to include all of these hostnames.
+
+In the default configuration these are:
+
+# @controller.${CLUSTER}.${DOMAIN}@
+# @ws.${CLUSTER}.${DOMAIN}@
+# @keep0.${CLUSTER}.${DOMAIN}@
+# @keep1.${CLUSTER}.${DOMAIN}@
+# @keep.${CLUSTER}.${DOMAIN}@
+# @download.${CLUSTER}.${DOMAIN}@
+# @*.collections.${CLUSTER}.${DOMAIN}@ -- important note, this must be a wildcard DNS, resolving to the @keepweb@ service
+# @workbench.${CLUSTER}.${DOMAIN}@
+# @workbench2.${CLUSTER}.${DOMAIN}@
+# @webshell.${CLUSTER}.${DOMAIN}@
+# @shell.${CLUSTER}.${DOMAIN}@
+
+This is described in more detail in "DNS entries and TLS certificates":install-manual-prerequisites.html#dnstls.
+
+h3. Additional prerequisites
+
+# root or passwordless @sudo@ access on the account where you are doing the install
+this usually means adding the account to the @sudo@ group and having a rule like this in @/etc/sudoers.d/arvados_passwordless@ that allows members of group @sudo@ to execute any command without entering a password.
+<pre>%sudo all=(all:all) nopasswd:all</pre>
+# @git@ installed on the machine
+# Port 443 reachable by clients
+# For the single-host install, ports 8800-8805 also need to be reachable from your client (configurable in @local.params@, see below)
+# When using "Let's Encrypt":#lets-encrypt port 80 needs to be reachable from everywhere on the internet
+# When using "bring your own certificate":#bring-your-own you need TLS certificate(s) covering the hostname(s) used by Arvados
+
+h2(#download). Download the installer
-h2(#final_steps). DNS configuration
+{% assign local_params_src = 'single_host_single_hostname' %}
+{% assign config_examples_src = 'single_host/single_hostname' %}
+{% include 'download_installer' %}
-After the setup is done, you need to set up your DNS to be able to access the cluster.
+If you are using multiple hostname configuration, substitute 'multiple_hostnames' where it says 'single_hostname' in the command above.
-The simplest way to do this is to edit your @/etc/hosts@ file (as root):
+h2(#localparams). Edit @local.params@
-<notextile>
-<pre><code>export CLUSTER="arva2"
-export DOMAIN="arv.local"
-export HOST_IP="127.0.0.2" # This is valid either if installing in your computer directly
- # or in a Vagrant VM. If you're installing it on a remote host
- # just change the IP to match that of the host.
-echo "${HOST_IP} api keep keep0 collections download ws workbench workbench2 ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} keep.${CLUSTER}.${DOMAIN} keep0.${CLUSTER}.${DOMAIN} collections.${CLUSTER}.${DOMAIN} download.${CLUSTER}.${DOMAIN} ws.${CLUSTER}.${DOMAIN} workbench.${CLUSTER}.${DOMAIN} workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+This can be found wherever you choose to initialize the install files (@~/setup-arvados-xarv1@ in these examples).
+
+# Set @CLUSTER@ to the 5-character cluster identifier (e.g "xarv1")
+# Set @DOMAIN@ to the base DNS domain of the environment, e.g. "example.com"
+# Single hostname only: set @IP_INT@ to the host's IP address.
+# Single hostname only: set @HOSTNAME_EXT@ to the hostname that users will use to connect.
+# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.
+# Set each @KEY@ / @TOKEN@ to a random string
+ Here's an easy way to create five random tokens:
+<pre><code>for i in 1 2 3 4 5; do
+ tr -dc A-Za-z0-9 </dev/urandom | head -c 32 ; echo ''
+done
</code></pre>
-</notextile>
+# Set @DATABASE_PASSWORD@ to a random string
+ Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.
+ For example, if the password is @Lq&MZ<V']d?j@
+ With backslash quoting the special characters it should appear like this in local.params:
+<pre><code>DATABASE_PASSWORD="Lq\&MZ\<V\'\]d\?j"</code></pre>
+
+{% include 'ssl_config_single' %}
+
+h2(#authentication). Configure your authentication provider (optional, recommended)
+
+By default, the installer will use the "Test" provider, which is a list of usernames and cleartext passwords stored in the Arvados config file. *This is low security configuration and you are strongly advised to configure one of the other "supported authentication methods":setup-login.html* .
+
+h2(#further_customization). Further customization of the installation (optional)
+
+If you want to customize the behavior of Arvados, this may require editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ contains the template (in the @arvados.cluster@ section) used to produce the Arvados configuration file. Consult the "Configuration reference":config.html for a comprehensive list of configuration keys.
+
+Any extra Salt "state" files you add under @local_config_dir/states@ will be added to the Salt run and applied to the hosts.
+
+h2(#installation). Begin installation
+
+At this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.
+
+Run this in the @~/arvados-setup-xarv1@ directory:
+
+<pre>
+./installer.sh deploy
+</pre>
+
+h2(#test-install). Confirm the cluster is working
+
+When everything has finished, you can run the diagnostics.
+
+Depending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.
+
+If you are running the diagnostics on the same machine where you installed Arvados, you want @-internal-client@ .
+
+You are an "external client" if you running the diagnostics from your workstation outside of the private network.
+
+<pre>
+./installer.sh diagnostics (-internal-client|-external-client)
+</pre>
+
+h3(#debugging). Debugging issues
+
+The installer records log files for each deployment.
+
+Most service logs go to @/var/log/syslog@.
+
+The logs for Rails API server and for Workbench can be found in
+
+@/var/www/arvados-api/current/log/production.log@
+and
+@/var/www/arvados-workbench/current/log/production.log@
+
+on the appropriate instances.
+
+Workbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in "Tools → Developer Tools").
+
+h3(#iterating). Iterating on config changes
+
+You can iterate on the config and maintain the cluster by making changes to @local.params@ and @local_config_dir@ and running @installer.sh deploy@ again.
+
+h3(#common-problems). Common problems and solutions
+
+h4. PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
+
+The arvados-api-server package sets up the database as a post-install script. If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.
+
+This will manifest as an error like this:
+
+<pre>
+#<ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation \"api_clients\" does not exist
+</pre>
+
+If this happens, you need to
+
+# correct the database information
+# run @./installer.sh deploy@ to update the configuration
+# Log in to the server, then run this command to re-run the post-install script, which will set up the database:
+<pre>dpkg-reconfigure arvados-api-server</pre>
+# Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.
+
+h2(#ca_root_certificate). Install the CA root certificate (SSL_MODE=self-signed only)
+
+*If you are not using self-signed certificates (you selected SSL_MODE=lets-encrypt or SSL_MODE=bring-your-own), skip this section.*
+
+Arvados uses SSL to encrypt communications. The web interface uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the installer has the option to create its own a root certificate to authorize Arvados services. The installer script will leave a copy of the generated CA's certificate (something like @xarv1.example.com-arvados-snakeoil-ca.crt@) in the script's directory so you can add it to your workstation.
+
+{% assign ca_cert_name = 'xarv1.example.com-arvados-snakeoil-ca.crt' %}
+
+{% include 'install_ca_cert' %}
h2(#initial_user). Initial user and login
-At this point you should be able to log into the Arvados cluster.
+At this point you should be able to log into the Arvados cluster. The initial URL for the single hostname install will use the hostname or IP address you put in @HOSTNAME_EXT@:
+
+https://${HOSTNAME_EXT}
+
+For the multi-hostname install, it will be:
+
+https://workbench.${CLUSTER}.${DOMAIN}
-If you changed nothing in the @provision.sh@ script, the initial URL will be:
+If you did *not* "configure a different authentication provider":#authentication you will be using the "Test" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster. It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ the @local.params@ file.
-* https://workbench.arva2.arv.local
+If you *did* configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.
-or, in general, the url format will be:
+h2(#post_install). After the installation
-* https://workbench.@<cluster>.<domain>@
+As part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates. You should retain this repository but be aware that it contains sensitive information (passwords and tokens used by the Arvados services).
-By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+As described in "Iterating on config changes":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades. However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.
-Assuming you didn't change these values in the @provision.sh@ script, the initial credentials are:
+When deploying upgrades, consult the "Arvados upgrade notes":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@. To specify the version to upgrade to, set the @VERSION@ parameter in @local.params@.
-* User: 'admin'
-* Password: 'password'
-* Email: 'admin@arva2.arv.local'
+See also "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
projects / arvados.git / blobdiff