SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+# "Introduction":#introduction
# "Hosts preparation":#hosts_preparation
-## "Hosts setup using terraform (experimental)":#hosts_setup_using_terraform
## "Create a compute image":#create_a_compute_image
# "Multi host install using the provision.sh script":#multi_host
# "Choose the desired configuration":#choose_configuration
# "Run the provision.sh script":#run_provision_script
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
+# "After the installation":#post_install
+
+h2(#introduction). Introduction
+
+Arvados components can be installed in a distributed infrastructure, whether it is an "on-prem" with physical or virtual hosts, or a cloud environment.
+
+As infrastructures vary a great deal from site to site, these instructions should be considered more as 'guidelines' than fixed steps to follow.
+
+We provide an "installer script":salt.html that can help you deploy the different Arvados components. At the time of writing, the provided examples are suitable to install Arvados on AWS.
+
+
h2(#hosts_preparation). Hosts preparation
## arvados controller
## arvados websocket
## arvados cloud dispatcher
+## arvados keepbalance
# WORKBENCH node:
## arvados workbench
## arvados workbench2
+## arvados webshell
# KEEPPROXY node:
## arvados keepproxy
## arvados keepweb
Note that these hosts can be virtual machines in your infrastructure and they don't need to be physical machines.
-h3(#hosts_setup_using_terraform). Hosts setup using terraform (experimental)
-
-We added a few "terraform":https://terraform.io/ scripts (https://github.com/arvados/arvados/tree/master/tools/terraform) to let you create these instances easier.
-Check "the Arvados terraform documentation":/doc/install/terraform.html for more details.
+Again, if your infrastructure differs from the setup proposed above (ie, using RDS or an existing DB server), remember that you will need to edit the configuration files for the scripts so they work with your infrastructure.
h2(#multi_host). Multi host install using the provision.sh script
-This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+{% include 'branchname' %}
+
+This is a package-based installation method. Start with the @provision.sh@ script which is available by cloning the @{{ branchname }}@ branch from "https://git.arvados.org/arvados.git":https://git.arvados.org/arvados.git . The @provision.sh@ script and its supporting files can be found in the "arvados/tools/salt-install":https://git.arvados.org/arvados.git/tree/refs/heads/{{ branchname }}:/tools/salt-install directory in the Arvados git repository.
-This procedure will install all the main Arvados components to get you up and running in a multi host environment.
+This procedure will install all the main Arvados components to get you up and running in a multi-host environment.
-We suggest you to use the @provision.sh@ script to deploy Arvados, which is implemented with the @arvados-formula@ in a Saltstack master-less setup. After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
+The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located at "arvados-formula":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
+
+After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
h3(#create_a_compute_image). Create a compute image
-In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/master/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image reference in the Arvados configuration in the next steps.
+In a multi-host installation, containers are dispatched in docker daemons running in the <i>compute instances</i>, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/main/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image ID in the Arvados configuration in the next steps.
h2(#choose_configuration). Choose the desired configuration
Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_INT_IP, *_TOKEN</b> and <b>*KEY</b> variables. Those variables will be used to do a search and replace on the <i>pillars/*</i> in place of any matching __VARIABLE__.
-The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API, Workbench) so it will need the hostnames to be reachable from the Internet. If this cluster will not be the case, please set the variable <i>USE_LETSENCRYPT=no</i>.
+The <i>multi_host</i> example includes Let's Encrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
-## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
+{% include 'multi_host_install_custom_certificates' %}
+
+If you want to use valid certificates provided by Let's Encrypt, set the variable <i>SSL_MODE=lets-encrypt</i> and make sure that all the FQDNs that you will use for the public-facing applications (API/controller, Workbench, Keepproxy/Keepweb) are reachable.
+
+h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
-You will need further customization to suit your environment, which can be done editing the Saltstack pillars and states files. Pay particular attention to the <i>pillars/arvados.sls</i> file, where you will need to provide some information that can be retrieved as output of the terraform run.
+You will need further customization to suit your environment, which can be done editing the Saltstack pillars and states files. Pay particular attention to the <i>pillars/arvados.sls</i> file, where you will need to provide some information that describes your environment.
Any extra <i>state</i> file you add under <i>local_config_dir/states</i> will be added to the salt run and applied to the hosts.
A few Arvados nodes need to be installed in certain order. The required order is
-#. Database
-#. API server
-#. The other nodes can be installed in any order after the two above
+* Database
+* API server
+* The other nodes can be installed in any order after the two above
h2(#run_provision_script). Run the provision.sh script
<notextile>
<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --roles comma,separated,list,of,roles,to,apply
+ssh user@host sudo ./provision.sh --roles comma,separated,list,of,roles,to,apply
</code></pre>
</notextile>
The distribution of role as described above can be applied running these commands:
-#. Database
+h4. Database
<notextile>
<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --config local.params --roles database
+ssh user@host sudo ./provision.sh --config local.params --roles database
</code></pre>
</notextile>
-#. API
+h4. API
<notextile>
<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --config local.params --roles api,controller,websocket,dispatcher
+ssh user@host sudo ./provision.sh --config local.params --roles api,controller,websocket,dispatcher,keepbalance
</code></pre>
</notextile>
-#. Keepstore/s
+h4. Keepstore(s)
<notextile>
<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --config local.params --roles keepstore
+ssh user@host sudo ./provision.sh --config local.params --roles keepstore
</code></pre>
</notextile>
-#. Workbench
+h4. Workbench
<notextile>
<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --config local.params --roles workbench,workbench2
+ssh user@host sudo ./provision.sh --config local.params --roles workbench,workbench2,webshell
</code></pre>
</notextile>
-#. Keepproxy / Keepweb
+h4. Keepproxy / Keepweb
<notextile>
<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --config local.params --roles keepproxy,keepweb
+ssh user@host sudo ./provision.sh --config local.params --roles keepproxy,keepweb
</code></pre>
</notextile>
-#. Shell
+h4. Shell (here we copy the CLI test workflow too)
<notextile>
-<pre><code>scp -r provision.sh local* user@host:
-ssh user@host sudo provision.sh --config local.params --roles shell
+<pre><code>scp -r provision.sh local* tests user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles shell
</code></pre>
</notextile>
-h2(#initial_user). Initial user and login
+h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster. The initial URL will be:
h2(#test_install). Test the installed cluster running a simple workflow
-The @provision.sh@ script saves a simple example test workflow in the @/tmp/cluster_tests@ directory in the node. If you want to run it, just ssh to the node, change to that directory and run:
+If you followed the instructions above, the @provision.sh@ script saves a simple example test workflow in the @/tmp/cluster_tests@ directory in the @shell@ node. If you want to run it, just ssh to the node, change to that directory and run:
<notextile>
<pre><code>cd /tmp/cluster_tests
-./run-test.sh
+sudo /run-test.sh
</code></pre>
</notextile>
INFO Final process status is success
</code></pre>
</notextile>
+
+h2(#post_install). After the installation
+
+Once the installation is complete, it is recommended to keep a copy of your local configuration files. Committing them to version control is a good idea.
+
+Re-running the Salt-based installer is not recommended for maintaining and upgrading Arvados, please see "Maintenance and upgrading":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.
projects / arvados.git / blobdiff