From 8e2d796199b51ea5148c77be1a763fee96947a88 Mon Sep 17 00:00:00 2001 From: Ward Vandewege Date: Thu, 2 Jun 2022 16:58:26 -0400 Subject: [PATCH] More Salt installer doc refactoring. refs #19175 Arvados-DCO-1.1-Signed-off-by: Ward Vandewege --- doc/_config.yml | 2 - doc/_includes/_download_installer.liquid | 2 +- .../salt-multi-host.html.textile.liquid | 12 +- .../salt-single-host.html.textile.liquid | 8 +- doc/install/salt.html.textile.liquid | 116 ------------------ 5 files changed, 12 insertions(+), 128 deletions(-) delete mode 100644 doc/install/salt.html.textile.liquid diff --git a/doc/_config.yml b/doc/_config.yml index f3ab1a5af0..7c5e6d986e 100644 --- a/doc/_config.yml +++ b/doc/_config.yml @@ -209,8 +209,6 @@ navbar: - Docker quick start: - install/arvbox.html.textile.liquid - Installation with Salt: - - install/salt.html.textile.liquid - - install/salt-vagrant.html.textile.liquid - install/salt-single-host.html.textile.liquid - install/salt-multi-host.html.textile.liquid - Arvados on Kubernetes: diff --git a/doc/_includes/_download_installer.liquid b/doc/_includes/_download_installer.liquid index 5690820086..5dfcd089e7 100644 --- a/doc/_includes/_download_installer.liquid +++ b/doc/_includes/_download_installer.liquid @@ -8,7 +8,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0 This procedure will install all the main Arvados components to get you up and running in a single host. -This is a package-based installation method, however the installation script is currently distributed in source form via @git@: +This is a package-based installation method, however the installation script is currently distributed in source form via @git@. We recommend checking out the git tree on your local workstation, not directly on the target(s) where you want to install and run Arvados. 
git clone https://git.arvados.org/arvados.git
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 640883baab..5145d433ba 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -12,7 +12,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
 # "Introduction":#introduction
 # "Prerequisites and planning":#prerequisites
 # "Download the installer":#download
-# "Copy the configuration files":#copy_config
+# "Copy and customize the configuration files":#copy_config
 # "Choose the SSL configuration":#certificates
 ## "Using a self-signed certificates":#self-signed
 ## "Using a Let's Encrypt certificates":#lets-encrypt
@@ -71,7 +71,7 @@ h2(#download). Download the installer
 
 {% include 'download_installer' %}
 
-h2(#copy_config). Copy the configuration files
+h2(#copy_config). Copy and customize the configuration files
 
 
 
cp local.params.example.multiple_hosts local.params
@@ -83,11 +83,13 @@ Edit the variables in the local.params file. Pay attention to the compute instances, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/main/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html. Once you have that image created, you will need to update the pillars/arvados.sls file with the AMI ID and the private ssh key for the dispatcher.
+{% include 'branchname' %}
 
-h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
+In a multi-host installation, containers are dispatched in docker daemons running in the compute instances, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/{{ branchname }}/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/install/crunch2-cloud/install-compute-node.html. Once you have that image created, you will need to update the pillars/arvados.sls file with the AMI ID and the private ssh key for the dispatcher.
+
+h2(#further_customization). Further customization of the installation (modifying the salt pillars and states)
 
 You will need further customization to suit your environment, which can be done editing the Saltstack pillars and states files. Pay particular attention to the pillars/arvados.sls file, where you will need to provide some information that describes your environment.
 
diff --git a/doc/install/salt-single-host.html.textile.liquid b/doc/install/salt-single-host.html.textile.liquid
index d436e5cdd7..ef1633b340 100644
--- a/doc/install/salt-single-host.html.textile.liquid
+++ b/doc/install/salt-single-host.html.textile.liquid
@@ -32,7 +32,7 @@ Using the default configuration, this installation method has a number of limita
 
 * all services run on the same machine, and they will compete for resources. This includes any compute jobs.
 * it uses the local machine disk for Keep storage (under the @/tmp@ directory). There may not be a lot of space available.
-* it installs the @crunch-dispatch-local@ dispatcher, which can run just eight concurrent CWL jobs. These jobs will be executed on the same machine that runs all the Arvados services and may well starve them of resources.
+* it installs the @crunch-dispatch-local@ dispatcher, which has a limit of eight concurrent jobs. These jobs will be executed on the same machine that runs all the Arvados services and may well starve them of resources.
 
 It is possible to start with the single host installation method and modify the Arvados configuration file later to address these limitations. E.g. switch to a "different storage volume setup":{{site.baseurl}}/install/configure-s3-object-storage.html for Keep, and switch to "the cloud dispatcher":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html to provision compute resources dynamically.
 
@@ -59,7 +59,7 @@ cp -r config_examples/single_host/single_hostname local_config_dir
 
 Edit the variables in the local.params file. Pay attention to the *_PORT, *_TOKEN and *_KEY variables. The *SSL_MODE* variable is discussed in the next section.
 
-{% include 'ssl_config' %}
+{% include 'ssl_config_single' %}
 
 h2(#further_customization). Further customization of the installation (modifying the salt pillars and states)
 
@@ -67,7 +67,7 @@ If you want or need further customization, you can edit the Saltstack pillars an
 
 h2(#run_provision_script). Run the provision.sh script
 
-When you finished customizing the configuration, you are ready to copy the files to the host (if needed) and run the @provision.sh@ script:
+When you finished customizing the configuration, you are ready to copy the files to the target host where Arvados will be installed, and run the @provision.sh@ script there:
 
 
 
scp -r provision.sh local* tests user@host:
@@ -75,7 +75,7 @@ ssh user@host sudo ./provision.sh
 

 
 
-and wait for it to finish. The script will need 5 to 10 minutes to install and configure everything.
+Wait for it to finish. The script will need 5 to 10 minutes to install and configure everything.
 
 If everything goes OK, you'll get final output that looks similar to this:
 
diff --git a/doc/install/salt.html.textile.liquid b/doc/install/salt.html.textile.liquid
deleted file mode 100644
index 29a6eacf3b..0000000000
--- a/doc/install/salt.html.textile.liquid
+++ /dev/null
@@ -1,116 +0,0 @@
----
-layout: default
-navsection: installguide
-title: Planning and prerequisites
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-# "Introduction":#introduction
-# "Provisioning Arvados with Saltstack":#provisioning_arvados
-# "The provisioning tool files and directories":#provisioning_tool_files and directories
-# "Choose an Arvados installation configuration":#choose_configuration
-## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
-# "Dump the configuration files created with the provision script":#dump_provision_config
-# "Add the Arvados formula to your Saltstack infrastructure":#add_formula_to_saltstack
-
-h2(#introduction). Introduction
-
-{% include 'branchname' %}
-
-To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://git.arvados.org/arvados-formula.git which can help you get an Arvados cluster up and running.
-
-Saltstack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. It can be used in a _master/minion_ setup (where a master node orchestrates and coordinates the configuration of nodes in an infrastructure) or master-less, where Saltstack is run locally in a node, with no communication with a master node.
-
-Similar to other configuration management tools like Puppet, Ansible or Chef, Saltstack uses files named states to describe the tasks that will be performed on a node to take it to a desired state, and pillars to configure variables passed to the states, adding flexibility to the tool.
-
-You don't need to be running a Saltstack infrastructure to install Arvados: we wrote a provisioning script that will take care of setting up Saltstack in the node/s where you want to install Arvados and run a master-less installer. Once Arvados is installed, you can either uninstall Saltstack and its files or you can keep them, to modify/maintain your Arvados installation in the future.
-
-This is a package-based installation method.
-
-h2(#provisioning_arvados). Provisioning Arvados with Saltstack
-
-The "tools/salt-install":https://git.arvados.org/arvados.git/tree/{{ branchname }}:/tools/salt-install directory in the Arvados git repository contains a script that you can run in the node/s where you want to install Arvados' components (the @provision.sh@ script) and a few configuration examples for different setups, that you can use to customize your installation.
-
-The @provision.sh@ script will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located at "arvados-formula":https://git.arvados.org/arvados-formula.git/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using "Saltstack":https://saltproject.io/ and @provision.sh@ performs the install using master-less mode.
-
-After setting up a few variables in a config file and copying a directory from the examples (see below), you'll be ready to run it and get Arvados deployed.
-
-
-
-h2(#provisioning_tool_files and directories). The provisioning tool files and directories
-
-The "tools/salt-install":https://git.arvados.org/arvados.git/tree/{{ branchname }}:/tools/salt-install directory contains the following elements:
-
-* The @provision.sh@ script itself. You don't need to modify it.
-* A few @local.params.*@ example files. You will need to copy one of these files to a file named @local.params@, which is the main configuration file for the @provision.sh@ script.
-* A few @config_examples/*@ directories, with pillars and states templates. You need to copy one of these to a @local_config_dir@ directory, which will be used by the @provision.sh@ script to setup your nodes.
-* A @tests@ directory, with a simple workflow and arvados CLI commands you can run to tests your cluster is capable of running a CWL workflow, upload files and create a user.
-
-Once you decide on an Arvados architecture you want to apply, you need to copy one of the example configuration files and directory, and edit them to suit your needs.
-
-Ie., for a multi-hosts / multi-hostnames in AWS, you need to do this:
-
-
cp local.params.example.multiple_hosts local.params
-cp -r config_examples/multi_host/aws local_config_dir
-

-
-
-These local files will be preserved if you upgrade the repository.
-
-
-
-h2(#choose_configuration). Choose an Arvados installation configuration
-
-The configuration examples provided with this installer are suitable to install Arvados with the following distribution of hosts/roles:
-
-* All roles on a single host, which can be done in two fashions:
-** Using a single hostname, assigning a different port (other than 443) for each user-facing service: This choice is easier to setup, but the user will need to know the port/s for the different services she wants to connect to. See "Single host install using the provision.sh script":salt-single-host.html for more details.
-** Using multiple hostnames on the same IP: this setup involves a few extra steps but each service will have a meaningful hostname so it will make easier to access them later. See "Single host install using the provision.sh script":salt-single-host.html for more details.
-* Roles distributed over multiple AWS instances, using multiple hostnames. This example can be adapted to use on-prem too. See "Multiple hosts installation":salt-multi-host.html for more details.
-
-Once you decide which of these choices you prefer, copy one of the example configuration files and directory, and edit them to suit your needs.
-
-Ie, if you decide to install Arvados on a single host using multiple hostnames:
-
-
cp local.params.example.single_host_multiple_hostnames local.params
-cp -r config_examples/single_host/multiple_hostnames local_config_dir
-

-
-
-Edit the variables in the local.params file.
-
-
-
-h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
-
-If you want or need further customization, you can edit the Saltstack pillars and states files. Pay particular attention to the pillars/arvados.sls one. Any extra state file you add under local_config_dir/states will be added to the salt run and applied to the host.
-
-
-
-h2(#dump_provision_config). Dump the configuration files created with the provision script
-
-As mentioned above, the @provision.sh@ script helps you create a set of configuration files to be used by the Saltstack @arvados-formula@ and other helper formulas.
-
-Is it possible you want to inspect these files before deploying them or use them within your existing Saltstack environment. In order to get a rendered version of these files, the @provision.sh@ script has a option, @--dump-config@, which takes a directory as mandatory parameter. When this option it used, the script will create the specified directory and write the pillars, states and tests files so you can inspect them.
-
-Ie.
-
-
./provision.sh --dump-config ./config_dump --role workbench
-

-
-
-will dump the configuration files used to install a workbench node under the @config_dump@ directory.
-
-These files are also suitable to be used in your existing Saltstack environment (see below).
-
-
-
-h2.(#add_formula_to_saltstack). Add the Arvados formula to your Saltstack infrastructure
-
-If you already have a Saltstack environment you can add the arvados-formula to your Saltstack master and apply the corresponding states and pillars to the nodes on your infrastructure that will be used to run Arvados.
-
-The @--dump-config@ option described above writes a @pillars/top.sls@ and @salt/top.sls@ files that you can use as a guide to configure your infrastructure.
-- 
2.30.2