- Docker quick start:
- install/arvbox.html.textile.liquid
- Installation with Salt:
- - install/salt.html.textile.liquid
- - install/salt-vagrant.html.textile.liquid
- install/salt-single-host.html.textile.liquid
- install/salt-multi-host.html.textile.liquid
- Arvados on Kubernetes:
--- /dev/null
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+{% include 'branchname' %}
+
+This procedure will install all the main Arvados components to get you up and running in a single host.
+
+This is a package-based installation method, however the installation script is currently distributed in source form via @git@. We recommend checking out the git tree on your local workstation, not directly on the target(s) where you want to install and run Arvados.
+
+<notextile>
+<pre><code>git clone https://git.arvados.org/arvados.git
+git checkout {{ branchname }}
+cd arvados/tools/salt-install
+</code></pre>
+</notextile>
+
+The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located in the "arvados-formula git repository":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
--- /dev/null
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+h2(#ca_root_certificate). Install the CA root certificate (SSL_MODE=self-signed only)
+
+Arvados uses SSL to encrypt communications. The web interface uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so you can add it to your workstation.
+
+Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
+
+# Go to the certificate manager in your browser.
+#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
+#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
+# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
+
+The certificate will be added under the "Arvados Formula".
+
+To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
+
+* On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+* On CentOS:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
--- /dev/null
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+h2(#certificates). Choose the SSL configuration (SSL_MODE)
+
+Arvados requires an SSL certificate to work correctly. This installer supports these options:
+
+* @self-signed@: let the installer create self-signed certificates
+* @lets-encrypt@: automatically obtain and install an SSL certificates for your hostnames
+* @bring-your-own@: supply your own certificates in the `certs` directory
+
+h3(#self-signed). Using self-signed certificates
+
+To make the installer use self-signed certificates, change the configuration like this:
+
+<notextile>
+<pre><code>SSL_MODE="self-signed"
+</code></pre>
+</notextile>
+
+When connecting to the Arvados web interface for the first time, you will need to accept the self-signed certificates as trusted to bypass the browser warnings. This can be a little tricky to do. Alternatively, you can also install the self-signed root certificate in your browser, see <a href="#ca_root_certificate">below</a>.
+
+h3(#lets-encrypt). Using a Let's Encrypt certificate
+
+In the default configuration, this installer gets a valid certificate via Let's Encrypt. If you have the <b>CLUSTER.DOMAIN</b> domain in a route53 zone, you can set <b>USE_LETSENCRYPT_ROUTE53</b> to <b>YES</b> and supply appropriate credentials so that Let's Encrypt can use dns-01 validation to get the appropriate certificates.
+
+<notextile>
+<pre><code>SSL_MODE="lets-encrypt"
+USE_LETSENCRYPT_ROUTE53="yes"
+LE_AWS_REGION="us-east-1"
+LE_AWS_ACCESS_KEY_ID="AKIABCDEFGHIJKLMNOPQ"
+LE_AWS_SECRET_ACCESS_KEY="thisistherandomstringthatisyoursecretkey"
+</code></pre>
+</notextile>
+
+Please note that when using AWS, EC2 instances can have a default hostname that ends with <i>amazonaws.com</i>. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the <i>amazonaws.com</i> domain, which means the default hostname can not be used to get a certificate from Let's Encrypt.
+
+h3(#bring-your-own). Bring your own certificates
+
+To supply your own certificates, change the configuration like this:
+
+<notextile>
+<pre><code>SSL_MODE="bring-your-own"
+</code></pre>
+</notextile>
+
+{% include 'multi_host_install_custom_certificates' %}
+
+All certificate files will be used by nginx. You may need to include intermediate certificates in your certificate files. See "the nginx documentation":http://nginx.org/en/docs/http/configuring_https_servers.html#chains for more details.
--- /dev/null
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+h2(#certificates). Choose the SSL configuration (SSL_MODE)
+
+Arvados requires an SSL certificate to work correctly. This installer supports these options:
+
+* @self-signed@: let the installer create a self-signed certificate
+* @lets-encrypt@: automatically obtain and install an SSL certificate for your hostname
+* @bring-your-own@: supply your own certificate in the `certs` directory
+
+h3(#self-signed). Using a self-signed certificate
+
+In the default configuration, this installer uses self-signed certificate(s):
+
+<notextile>
+<pre><code>SSL_MODE="self-signed"
+</code></pre>
+</notextile>
+
+When connecting to the Arvados web interface for the first time, you will need to accept the self-signed certificate as trusted to bypass the browser warnings. This can be a little tricky to do. Alternatively, you can also install the self-signed root certificate in your browser, see <a href="#ca_root_certificate">below</a>.
+
+h3(#lets-encrypt). Using a Let's Encrypt certificate
+
+To automatically get a valid certificate via Let's Encrypt, change the configuration like this:
+
+<notextile>
+<pre><code>SSL_MODE="lets-encrypt"
+</code></pre>
+</notextile>
+
+The hostname for your Arvados cluster must be defined in @HOSTNAME_EXT@ and resolve to the public IP address of your Arvados instance, so that Let's Encrypt can validate the domainname ownership and issue the certificate.
+
+When using AWS, EC2 instances can have a default hostname that ends with <i>amazonaws.com</i>. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the <i>amazonaws.com</i> domain, which means the default hostname can not be used to get a certificate from Let's Encrypt.
+
+h3(#bring-your-own). Bring your own certificate
+
+To supply your own certificate, change the configuration like this:
+
+<notextile>
+<pre><code>SSL_MODE="bring-your-own"
+</code></pre>
+</notextile>
+
+Copy your certificate files to the directory specified with the variable @CUSTOM_CERTS_DIR@. The provision script will find it there. The certificate and its key need to be copied to a file named after @HOSTNAME_EXT@. For example, if @HOSTNAME_EXT@ is defined as @my-arvados.example.net@, the script will look for
+
+<notextile>
+<pre><code>${CUSTOM_CERTS_DIR}/my-arvados.example.net.crt
+${CUSTOM_CERTS_DIR}/my-arvados.example.net.key
+</code></pre>
+</notextile>
+
+All certificate files will be used by nginx. You may need to include intermediate certificates in your certificate file. See "the nginx documentation":http://nginx.org/en/docs/http/configuring_https_servers.html#chains for more details.
{% endcomment %}
# "Introduction":#introduction
-# "Hosts preparation":#hosts_preparation
-## "Create a compute image":#create_a_compute_image
-# "Multi host install using the provision.sh script":#multi_host
-# "Choose the desired configuration":#choose_configuration
-## "Multiple hosts / multiple hostnames":#multi_host_multi_hostnames
-## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
+# "Prerequisites and planning":#prerequisites
+# "Download the installer":#download
+# "Copy and customize the configuration files":#copy_config
+# "Choose the SSL configuration":#certificates
+## "Using a self-signed certificates":#self-signed
+## "Using a Let's Encrypt certificates":#lets-encrypt
+## "Bring your own certificates":#bring-your-own
+# "Create a compute image":#create_a_compute_image
+# "Further customization of the installation (modifying the salt pillars and states)":#further_customization
# "Installation order":#installation_order
# "Run the provision.sh script":#run_provision_script
+# "Install the CA root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
# "After the installation":#post_install
h2(#introduction). Introduction
-Arvados components can be installed in a distributed infrastructure, whether it is an "on-prem" with physical or virtual hosts, or a cloud environment.
+This multi host installer is an AWS specific example that is generally useful, but will likely need to be adapted for your environment. The installer is highly configurable.
-As infrastructures vary a great deal from site to site, these instructions should be considered more as 'guidelines' than fixed steps to follow.
+h2(#prerequisites). Prerequisites and planning
-We provide an "installer script":salt.html that can help you deploy the different Arvados components. At the time of writing, the provided examples are suitable to install Arvados on AWS.
+Prerequisites:
+* git
+* a number of (virtual) machines for your Arvados cluster with at least 2 cores and 8 GiB of RAM, running a "supported Arvados distribution":{{site.baseurl}}/install/install-manual-prerequisites.html#supportedlinux
+* a number of DNS hostnames that resolve to the IP addresses of your Arvados hosts
+* ports 443 need to be reachable from your client (configurable in @local.params@, see below)
+* port 80 needs to be reachable from everywhere on the internet (only when using "Let's Encrypt":#lets-encrypt without Route53 integration)
+* SSL certificatse matching the hostnames in use (only when using "bring your own certificates":#bring-your-own)
-
-h2(#hosts_preparation). Hosts preparation
-
-In order to run Arvados on a multi-host installation, there are a few requirements that your infrastructure has to fulfill.
-
-These instructions explain how to setup a multi-host environment that is suitable for production use of Arvados.
+Planning:
We suggest distributing the Arvados components in the following way, creating at least 6 hosts:
# KEEPPROXY node:
## arvados keepproxy
## arvados keepweb
-# KEEPSTOREs (at least 2)
+# KEEPSTORE nodes (at least 2)
## arvados keepstore
# SHELL node (optional):
## arvados shell
-Note that these hosts can be virtual machines in your infrastructure and they don't need to be physical machines.
-
-Again, if your infrastructure differs from the setup proposed above (ie, using RDS or an existing DB server), remember that you will need to edit the configuration files for the scripts so they work with your infrastructure.
-
-h2(#multi_host). Multi host install using the provision.sh script
-
-{% include 'branchname' %}
-
-This is a package-based installation method. Start with the @provision.sh@ script which is available by cloning the @{{ branchname }}@ branch from "https://git.arvados.org/arvados.git":https://git.arvados.org/arvados.git . The @provision.sh@ script and its supporting files can be found in the "arvados/tools/salt-install":https://git.arvados.org/arvados.git/tree/refs/heads/{{ branchname }}:/tools/salt-install directory in the Arvados git repository.
-
-This procedure will install all the main Arvados components to get you up and running in a multi-host environment.
+If your infrastructure differs from the setup proposed above (ie, using RDS or an existing DB server), remember that you will need to edit the configuration files for the scripts so they work with your infrastructure.
-The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located at "arvados-formula":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
+h2(#download). Download the installer
-After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
+{% include 'download_installer' %}
-h3(#create_a_compute_image). Create a compute image
+h2(#copy_config). Copy and customize the configuration files
-In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/main/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image ID in the Arvados configuration in the next steps.
-
-h2(#choose_configuration). Choose the desired configuration
-
-For documentation's sake, we will use the cluster name <i>arva2</i> and the domain <i>arv.local</i>. If you don't change them as required in the next steps, installation won't proceed.
-
-We will try to provide a few Arvados' multi host installation configurations examples for different infrastructure providers. Currently only AWS is available but they can be used with almost any provider with little changes.
-
-You need to copy one of the example configuration files and directory, and edit them to suit your needs.
-
-h3(#multi_host_multi_hostnames). Multiple hosts / multiple hostnames
<notextile>
<pre><code>cp local.params.example.multiple_hosts local.params
cp -r config_examples/multi_host/aws local_config_dir
</code></pre>
</notextile>
-Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_INT_IP, *_TOKEN</b> and <b>*KEY</b> variables. Those variables will be used to do a search and replace on the <i>pillars/*</i> in place of any matching __VARIABLE__.
+Edit the variables in the <i>local.params</i> file. Pay attention to the <notextile><b>*_INT_IP, *_TOKEN</b> and <b>*_KEY</b></notextile> variables. The *SSL_MODE* variable is discussed in the next section.
+
+{% include 'ssl_config_multi' %}
-The <i>multi_host</i> example includes Let's Encrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
+h2(#create_a_compute_image). Create a compute image
-{% include 'multi_host_install_custom_certificates' %}
+{% include 'branchname' %}
-If you want to use valid certificates provided by Let's Encrypt, set the variable <i>SSL_MODE=lets-encrypt</i> and make sure that all the FQDNs that you will use for the public-facing applications (API/controller, Workbench, Keepproxy/Keepweb) are reachable.
+In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/install/crunch2-cloud/install-compute-node.html. Once you have that image created, you will need to update the <i>pillars/arvados.sls</i> file with the AMI ID and the private ssh key for the dispatcher.
-h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
+h2(#further_customization). Further customization of the installation (modifying the salt pillars and states)
You will need further customization to suit your environment, which can be done editing the Saltstack pillars and states files. Pay particular attention to the <i>pillars/arvados.sls</i> file, where you will need to provide some information that describes your environment.
</code></pre>
</notextile>
+{% include 'install_ca_cert' %}
+
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster. The initial URL will be:
# "Limitations of the single host install":#limitations
# "Prerequisites":#prerequisites
-# "Download the installer":#single_host
+# "Download the installer":#download
+# "Copy the configuration files":#copy_config
# "Choose the SSL configuration":#certificates
## "Using a self-signed certificate":#self-signed
## "Using a Let's Encrypt certificate":#lets-encrypt
* all services run on the same machine, and they will compete for resources. This includes any compute jobs.
* it uses the local machine disk for Keep storage (under the @/tmp@ directory). There may not be a lot of space available.
-* it installs the @crunch-dispatch-local@ dispatcher, which can run just eight concurrent CWL jobs. These jobs will be executed on the same machine that runs all the Arvados services and may well starve them of resources.
+* it installs the @crunch-dispatch-local@ dispatcher, which has a limit of eight concurrent jobs. These jobs will be executed on the same machine that runs all the Arvados services and may well starve them of resources.
It is possible to start with the single host installation method and modify the Arvados configuration file later to address these limitations. E.g. switch to a "different storage volume setup":{{site.baseurl}}/install/configure-s3-object-storage.html for Keep, and switch to "the cloud dispatcher":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html to provision compute resources dynamically.
-h2(#prerequisites). Prerequisites and planning
-
-Prerequisites:
+h2(#prerequisites). Prerequisites
* git
* a dedicated (virtual) machine for your Arvados server with at least 2 cores and 8 GiB of RAM, running a "supported Arvados distribution":{{site.baseurl}}/install/install-manual-prerequisites.html#supportedlinux
* port 80 needs to be reachable from everywhere on the internet (only when using "Let's Encrypt":#lets-encrypt)
* an SSL certificate matching the hostname in use (only when using "bring your own certificate":#bring-your-own)
-h2(#single_host). Download the installer
-
-{% include 'branchname' %}
-
-This procedure will install all the main Arvados components to get you up and running in a single host.
-
-This is a package-based installation method, however the installation script is currently distributed in source form via @git@:
-
-<notextile>
-<pre><code>git clone https://git.arvados.org/arvados.git
-git checkout {{ branchname }}
-cd arvados/tools/salt-install
-</code></pre>
-</notextile>
+h2(#download). Download the installer
-The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located in the "arvados-formula git repository":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
+{% include 'download_installer' %}
-First, copy the configuration files:
+h2(#copy_config). Copy the configuration files
<notextile>
<pre><code>cp local.params.example.single_host_single_hostname local.params
</code></pre>
</notextile>
-Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_PORT, *_TOKEN</b> and <b>*KEY</b> variables. The *SSL_MODE* variable is discussed in the next section.
-
-h2(#certificates). Choose the SSL configuration (SSL_MODE)
-
-Arvados requires an SSL certificate to work correctly. This installer supports these options:
-
-* @self-signed@: let the installer create a self-signed certificate
-* @lets-encrypt@: automatically obtain and install an SSL certificate for your hostname
-* @bring-your-own@: supply your own certificate in the `certs` directory
-
-h3(#self-signed). Using a self-signed certificate
-
-In the default configuration, this installer uses self-signed certificate(s):
-
-<notextile>
-<pre><code>SSL_MODE="self-signed"
-</code></pre>
-</notextile>
-
-When connecting to the Arvados web interface for the first time, you will need to accept the self-signed certificate as trusted to bypass the browser warnings.
-
-h3(#lets-encrypt). Using a Let's Encrypt certificate
-
-To automatically get a valid certificate via Let's Encrypt, change the configuration like this:
-
-<notextile>
-<pre><code>SSL_MODE="lets-encrypt"
-</code></pre>
-</notextile>
-
-The hostname for your Arvados cluster must be defined in @HOSTNAME_EXT@ and resolve to the public IP address of your Arvados instance, so that Let's Encrypt can validate the domainname ownership and issue the certificate.
-
-When using AWS, EC2 instances can have a default hostname that ends with <i>amazonaws.com</i>. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the <i>amazonaws.com</i> domain, which means the default hostname can not be used to get a certificate from Let's Encrypt.
-
-h3(#bring-your-own). Bring your own certificate
+Edit the variables in the <i>local.params</i> file. Pay attention to the <notextile><b>*_PORT, *_TOKEN</b> and <b>*_KEY</b></notextile> variables. The *SSL_MODE* variable is discussed in the next section.
-To supply your own certificate, change the configuration like this:
-
-<notextile>
-<pre><code>SSL_MODE="bring-your-own"
-</code></pre>
-</notextile>
-
-Copy your certificate files to the directory specified with the variable @CUSTOM_CERTS_DIR@. The provision script will find it there. The certificate and its key need to be copied to a file named after @HOSTNAME_EXT@. For example, if @HOSTNAME_EXT@ is defined as @my-arvados.example.net@, the script will look for
-
-<notextile>
-<pre><code>${CUSTOM_CERTS_DIR}/my-arvados.example.net.crt
-${CUSTOM_CERTS_DIR}/my-arvados.example.net.key
-</code></pre>
-</notextile>
-
-All certificate files will be used by nginx. You may need to include intermediate certificates in your certificate file. See "the nginx documentation":http://nginx.org/en/docs/http/configuring_https_servers.html#chains for more details.
+{% include 'ssl_config_single' %}
h2(#further_customization). Further customization of the installation (modifying the salt pillars and states)
h2(#run_provision_script). Run the provision.sh script
-When you finished customizing the configuration, you are ready to copy the files to the host (if needed) and run the @provision.sh@ script:
+When you finished customizing the configuration, you are ready to copy the files to the target host where Arvados will be installed, and run the @provision.sh@ script there:
<notextile>
<pre><code>scp -r provision.sh local* tests user@host:
</code></pre>
</notextile>
-and wait for it to finish. The script will need 5 to 10 minutes to install and configure everything.
+Wait for it to finish. The script will need 5 to 10 minutes to install and configure everything.
If everything goes OK, you'll get final output that looks similar to this:
</code></pre>
</notextile>
-h2(#ca_root_certificate). Install the CA root certificate (SSL_MODE=self-signed only)
-
-Arvados uses SSL to encrypt communications. The web interface uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
-
-For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so you can add it to your workstation.
-
-Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
-
-# Go to the certificate manager in your browser.
-#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
-#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
-# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
-
-The certificate will be added under the "Arvados Formula".
-
-To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
-
-* On Debian/Ubuntu:
-
-<notextile>
-<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
-/usr/sbin/update-ca-certificates
-</code></pre>
-</notextile>
-
-* On CentOS:
-
-<notextile>
-<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
-/usr/bin/update-ca-trust
-</code></pre>
-</notextile>
+{% include 'install_ca_cert' %}
h2(#initial_user). Initial user and login
+++ /dev/null
----
-layout: default
-navsection: installguide
-title: Planning and prerequisites
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-# "Introduction":#introduction
-# "Provisioning Arvados with Saltstack":#provisioning_arvados
-# "The provisioning tool files and directories":#provisioning_tool_files and directories
-# "Choose an Arvados installation configuration":#choose_configuration
-## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
-# "Dump the configuration files created with the provision script":#dump_provision_config
-# "Add the Arvados formula to your Saltstack infrastructure":#add_formula_to_saltstack
-
-h2(#introduction). Introduction
-
-{% include 'branchname' %}
-
-To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://git.arvados.org/arvados-formula.git which can help you get an Arvados cluster up and running.
-
-Saltstack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. It can be used in a _master/minion_ setup (where a master node orchestrates and coordinates the configuration of nodes in an infrastructure) or <i>master-less</i>, where Saltstack is run locally in a node, with no communication with a master node.
-
-Similar to other configuration management tools like Puppet, Ansible or Chef, Saltstack uses files named <i>states</i> to describe the tasks that will be performed on a node to take it to a desired state, and <i>pillars</i> to configure variables passed to the states, adding flexibility to the tool.
-
-You don't need to be running a Saltstack infrastructure to install Arvados: we wrote a provisioning script that will take care of setting up Saltstack in the node/s where you want to install Arvados and run a <i>master-less installer</i>. Once Arvados is installed, you can either uninstall Saltstack and its files or you can keep them, to modify/maintain your Arvados installation in the future.
-
-This is a package-based installation method.
-
-h2(#provisioning_arvados). Provisioning Arvados with Saltstack
-
-The "tools/salt-install":https://git.arvados.org/arvados.git/tree/{{ branchname }}:/tools/salt-install directory in the Arvados git repository contains a script that you can run in the node/s where you want to install Arvados' components (the @provision.sh@ script) and a few configuration examples for different setups, that you can use to customize your installation.
-
-The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located at "arvados-formula":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
-
-After setting up a few variables in a config file and copying a directory from the examples (see below), you'll be ready to run it and get Arvados deployed.
-
-
-
-h2(#provisioning_tool_files and directories). The provisioning tool files and directories
-
-The "tools/salt-install":https://git.arvados.org/arvados.git/tree/{{ branchname }}:/tools/salt-install directory contains the following elements:
-
-* The @provision.sh@ script itself. You don't need to modify it.
-* A few @local.params.*@ example files. You will need to copy one of these files to a file named @local.params@, which is the main configuration file for the @provision.sh@ script.
-* A few @config_examples/*@ directories, with pillars and states templates. You need to copy one of these to a @local_config_dir@ directory, which will be used by the @provision.sh@ script to setup your nodes.
-* A @tests@ directory, with a simple workflow and arvados CLI commands you can run to tests your cluster is capable of running a CWL workflow, upload files and create a user.
-
-Once you decide on an Arvados architecture you want to apply, you need to copy one of the example configuration files and directory, and edit them to suit your needs.
-
-Ie., for a multi-hosts / multi-hostnames in AWS, you need to do this:
-<notextile>
-<pre><code>cp local.params.example.multiple_hosts local.params
-cp -r config_examples/multi_host/aws local_config_dir
-</code></pre>
-</notextile>
-
-These local files will be preserved if you upgrade the repository.
-
-
-
-h2(#choose_configuration). Choose an Arvados installation configuration
-
-The configuration examples provided with this installer are suitable to install Arvados with the following distribution of hosts/roles:
-
-* All roles on a single host, which can be done in two fashions:
-** Using a single hostname, assigning <i>a different port (other than 443) for each user-facing service</i>: This choice is easier to setup, but the user will need to know the port/s for the different services she wants to connect to. See "Single host install using the provision.sh script":salt-single-host.html for more details.
-** Using multiple hostnames on the same IP: this setup involves a few extra steps but each service will have a meaningful hostname so it will make easier to access them later. See "Single host install using the provision.sh script":salt-single-host.html for more details.
-* Roles distributed over multiple AWS instances, using multiple hostnames. This example can be adapted to use on-prem too. See "Multiple hosts installation":salt-multi-host.html for more details.
-
-Once you decide which of these choices you prefer, copy one of the example configuration files and directory, and edit them to suit your needs.
-
-Ie, if you decide to install Arvados on a single host using multiple hostnames:
-<notextile>
-<pre><code>cp local.params.example.single_host_multiple_hostnames local.params
-cp -r config_examples/single_host/multiple_hostnames local_config_dir
-</code></pre>
-</notextile>
-
-Edit the variables in the <i>local.params</i> file.
-
-
-
-h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
-
-If you want or need further customization, you can edit the Saltstack pillars and states files. Pay particular attention to the <i>pillars/arvados.sls</i> one. Any extra <i>state</i> file you add under <i>local_config_dir/states</i> will be added to the salt run and applied to the host.
-
-
-
-h2(#dump_provision_config). Dump the configuration files created with the provision script
-
-As mentioned above, the @provision.sh@ script helps you create a set of configuration files to be used by the Saltstack @arvados-formula@ and other helper formulas.
-
-Is it possible you want to inspect these files before deploying them or use them within your existing Saltstack environment. In order to get a rendered version of these files, the @provision.sh@ script has a option, @--dump-config@, which takes a directory as mandatory parameter. When this option it used, the script will create the specified directory and write the pillars, states and tests files so you can inspect them.
-
-Ie.
-<notextile>
-<pre><code>./provision.sh --dump-config ./config_dump --role workbench
-</code></pre>
-</notextile>
-
-will dump the configuration files used to install a workbench node under the @config_dump@ directory.
-
-These files are also suitable to be used in your existing Saltstack environment (see below).
-
-
-
-h2.(#add_formula_to_saltstack). Add the Arvados formula to your Saltstack infrastructure
-
-If you already have a Saltstack environment you can add the arvados-formula to your Saltstack master and apply the corresponding states and pillars to the nodes on your infrastructure that will be used to run Arvados.
-
-The @--dump-config@ option described above writes a @pillars/top.sls@ and @salt/top.sls@ files that you can use as a guide to configure your infrastructure.
echo >&2 " -h, --help Display this help and exit"
echo >&2 " --dump-config <dest_dir> Dumps the pillars and states to a directory"
echo >&2 " This parameter does not perform any installation at all. It's"
- echo >&2 " intended to give you a parsed sot of configuration files so"
+ echo >&2 " intended to give you a parsed set of configuration files so"
echo >&2 " you can inspect them or use them in you Saltstack infrastructure."
echo >&2 " It"
echo >&2 " - parses the pillar and states templates,"
echo >&2 " - downloads the helper formulas with their desired versions,"
echo >&2 " - prepares the 'top.sls' files both for pillars and states"
- echo >&2 " for the selected role/s"
+ echo >&2 " for the selected role(s)"
echo >&2 " - writes the resulting files into <dest_dir>"
echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
echo >&2 " --development Run in dev mode, using snakeoil certs"
projects / arvados.git / commitdiff