X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/397981dadc145225c691c8643b10527c9710f1fb..35bf3afb8b312dcc30fe3b61042779d33bf1db10:/doc/install/salt-multi-host.html.textile.liquid
diff --git a/doc/install/salt-multi-host.html.textile.liquid b/doc/install/salt-multi-host.html.textile.liquid
index 4ba153faf9..04ef9e8684 100644
--- a/doc/install/salt-multi-host.html.textile.liquid
+++ b/doc/install/salt-multi-host.html.textile.liquid
@@ -9,91 +9,164 @@ Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-# "Install Saltstack":#saltstack
-# "Install dependencies":#dependencies
-# "Install Arvados using Saltstack":#saltstack
-# "DNS configuration":#final_steps
+# "Hosts preparation":#hosts_preparation
+## "Hosts setup using terraform (experimental)":#hosts_setup_using_terraform
+## "Create a compute image":#create_a_compute_image
+# "Multi host install using the provision.sh script":#multi_host
+# "Choose the desired configuration":#choose_configuration
+## "Multiple hosts / multiple hostnames":#multi_host_multi_hostnames
+## "Further customization of the installation (modifying the salt pillars and states)":#further_customization
+# "Installation order":#installation_order
+# "Run the provision.sh script":#run_provision_script
# "Initial user and login":#initial_user
+# "Test the installed cluster running a simple workflow":#test_install
-h2(#saltstack). Install Saltstack
+h2(#hosts_preparation). Hosts preparation
-If you already have a Saltstack environment you can skip this section.
+In order to run Arvados on a multi-host installation, there are a few requirements that your infrastructure has to fulfill.
-The simplest way to get Salt up and running on a node is to use the bootstrap script they provide:
+These instructions explain how to setup a multi-host environment that is suitable for production use of Arvados.
+We suggest distributing the Arvados components in the following way, creating at least 6 hosts:
+
+# Database server:
+## postgresql server
+# API node:
+## arvados api server
+## arvados controller
+## arvados websocket
+## arvados cloud dispatcher
+# WORKBENCH node:
+## arvados workbench
+## arvados workbench2
+## arvados webshell
+# KEEPPROXY node:
+## arvados keepproxy
+## arvados keepweb
+# KEEPSTOREs (at least 2)
+## arvados keepstore
+# SHELL node (optional):
+## arvados shell
+
+Note that these hosts can be virtual machines in your infrastructure and they don't need to be physical machines.
+
+h3(#hosts_setup_using_terraform). Hosts setup using terraform (experimental)
+
+We added a few "terraform":https://terraform.io/ scripts (https://github.com/arvados/arvados/tree/master/tools/terraform) to let you create these instances easier.
+Check "the Arvados terraform documentation":/doc/install/terraform.html for more details.
+
+h2(#multi_host). Multi host install using the provision.sh script
+
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+
+This procedure will install all the main Arvados components to get you up and running in a multi host environment.
+
+We suggest you to use the @provision.sh@ script to deploy Arvados, which is implemented with the @arvados-formula@ in a Saltstack master-less setup. After setting up a few variables in a config file (next step), you'll be ready to run it and get Arvados deployed.
+
+h3(#create_a_compute_image). Create a compute image
+
+In a multi-host installation, containers are dispatched in docker daemons running in the compute instances, which need some special setup. We provide a "compute image builder script":https://github.com/arvados/arvados/tree/master/tools/compute-images that you can use to build a template image following "these instructions":https://doc.arvados.org/main/install/crunch2-cloud/install-compute-node.html . Once you have that image created, you can use the image reference in the Arvados configuration in the next steps.
+
+h2(#choose_configuration). Choose the desired configuration
+
+For documentation's sake, we will use the cluster name arva2 and the domain arv.local. If you don't change them as required in the next steps, installation won't proceed.
+
+We will try to provide a few Arvados' multi host installation configurations examples for different infrastructure providers. Currently only AWS is available but they can be used with almost any provider with little changes.
+
+You need to copy one of the example configuration files and directory, and edit them to suit your needs.
+
+h3(#multi_host_multi_hostnames). Multiple hosts / multiple hostnames
-curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
-sudo sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+cp local.params.example.multiple_hosts local.params
+cp -r config_examples/multi_host/aws local_config_dir
-For more information check "Saltstack's documentation":https://docs.saltstack.com/en/latest/topics/installation/index.html
+Edit the variables in the local.params file. Pay attention to the *_INT_IP, *_TOKEN and *KEY variables. Those variables will be used to do a search and replace on the pillars/* in place of any matching __VARIABLE__.
-h2(#dependencies). Install dependencies
+The multi_host include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API, Workbench) so it will need the hostnames to be reachable from the Internet. If this cluster will not be the case, please set the variable USE_LETSENCRYPT=no.
-Arvados depends in a few applications and packages (postgresql, nginx+passenger, ruby) that can also be installed using their respective Saltstack formulas.
+h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
-The formulas we use are:
+You will need further customization to suit your environment, which can be done editing the Saltstack pillars and states files. Pay particular attention to the pillars/arvados.sls file, where you will need to provide some information that can be retrieved as output of the terraform run.
-* "postgres":https://github.com/saltstack-formulas/postgres-formula.git
-* "nginx":https://github.com/saltstack-formulas/nginx-formula.git
-* "docker":https://github.com/saltstack-formulas/docker-formula.git
-* "locale":https://github.com/saltstack-formulas/locale-formula.git
+Any extra state file you add under local_config_dir/states will be added to the salt run and applied to the hosts.
-There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/saltstack-formulas/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+h2(#installation_order). Installation order
-h2(#saltstack). Install Arvados using Saltstack
-
-This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+A few Arvados nodes need to be installed in certain order. The required order is
-The Arvados formula we maintain is located in the Saltstack's community repository of formulas:
+#. Database
+#. API server
+#. The other nodes can be installed in any order after the two above
-* "arvados-formula":https://github.com/saltstack-formulas/arvados-formula.git
+h2(#run_provision_script). Run the provision.sh script
-The @development@ version lives in our own repository
+When you finished customizing the configuration, you are ready to copy the files to the hosts and run the @provision.sh@ script. The script allows you to specify the role/s a node will have and it will install only the Arvados components required for such role. The general format of the command is:
-* "arvados-formula development":https://github.com/arvados/arvados-formula.git
-
-This last one might break from time to time, as we try and add new features. Use with caution.
+
+scp -r provision.sh local* user@host:
+ssh user@host sudo ./provision.sh --roles comma,separated,list,of,roles,to,apply
+
+
-As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados.
+and wait for it to finish.
-For those familiar with Saltstack, the process to get it deployed is similar to any other formula:
+If everything goes OK, you'll get some final lines stating something like:
-1. Fork/copy the formula to your Salt master host.
-2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
-3. Run a @state.apply@ to get it deployed.
+
+arvados: Succeeded: 109 (changed=9)
+arvados: Failed: 0
+
+
-h2(#final_steps). DNS configuration
+The distribution of role as described above can be applied running these commands:
-After the setup is done, you need to set up your DNS to be able to access the cluster's nodes.
+#. Database
+
+scp -r provision.sh local* user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles database
+
+
-The simplest way to do this is to add entries in the @/etc/hosts@ file of every host:
+#. API
+
+scp -r provision.sh local* user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles api,controller,websocket,dispatcher
+
+
+#. Keepstore/s
-export CLUSTER="arva2"
-export DOMAIN="arv.local"
-
-echo A.B.C.a api ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.b keep keep.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.c keep0 keep0.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.d collections collections.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.e download download.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.f ws ws.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.g workbench workbench.${CLUSTER}.${DOMAIN} >> /etc/hosts
-echo A.B.C.h workbench2 workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+scp -r provision.sh local* user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles keepstore
-Replacing in each case de @A.B.C.x@ IP with the corresponding IP of the node.
+#. Workbench
+
+scp -r provision.sh local* user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles workbench,workbench2,webshell
+
+
-If your infrastructure uses another DNS service setup, add the corresponding entries accordingly.
+#. Keepproxy / Keepweb
+
+scp -r provision.sh local* user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles keepproxy,keepweb
+
+
-h2(#initial_user). Initial user and login
+#. Shell (here we copy the CLI test workflow too)
+
+scp -r provision.sh local* tests user@host:
+ssh user@host sudo ./provision.sh --config local.params --roles shell
+
+
-At this point you should be able to log into the Arvados cluster.
+h2(#initial_user). Initial user and login
-If you did not change the defaults, the initial URL will be:
+At this point you should be able to log into the Arvados cluster. The initial URL will be:
* https://workbench.arva2.arv.local
@@ -103,8 +176,100 @@ or, in general, the url format will be:
By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
-Assuming you didn't change the defaults, the initial credentials are:
+Assuming you didn't change these values in the @local.params@ file, the initial credentials are:
* User: 'admin'
* Password: 'password'
* Email: 'admin@arva2.arv.local'
+
+h2(#test_install). Test the installed cluster running a simple workflow
+
+If you followed the instructions above, the @provision.sh@ script saves a simple example test workflow in the @/tmp/cluster_tests@ directory in the @shell@ node. If you want to run it, just ssh to the node, change to that directory and run:
+
+
+cd /tmp/cluster_tests
+sudo /run-test.sh
+
+
+
+It will create a test user (by default, the same one as the admin user), upload a small workflow and run it. If everything goes OK, the output should similar to this (some output was shortened for clarity):
+
+
+Creating Arvados Standard Docker Images project
+Arvados project uuid is 'arva2-j7d0g-0prd8cjlk6kfl7y'
+{
+ ...
+ "uuid":"arva2-o0j2j-n4zu4cak5iifq2a",
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+}
+Uploading arvados/jobs' docker image to the project
+2.1.1: Pulling from arvados/jobs
+8559a31e96f4: Pulling fs layer
+...
+Status: Downloaded newer image for arvados/jobs:2.1.1
+docker.io/arvados/jobs:2.1.1
+2020-11-23 21:43:39 arvados.arv_put[32678] INFO: Creating new cache file at /home/vagrant/.cache/arvados/arv-put/c59256eda1829281424c80f588c7cc4d
+2020-11-23 21:43:46 arvados.arv_put[32678] INFO: Collection saved as 'Docker image arvados jobs:2.1.1 sha256:0dd50'
+arva2-4zz18-1u5pvbld7cvxuy2
+Creating initial user ('admin')
+Setting up user ('admin')
+{
+ "items":[
+ {
+ ...
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+ "uuid":"arva2-o0j2j-1ownrdne0ok9iox"
+ },
+ {
+ ...
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+ "uuid":"arva2-o0j2j-1zbeyhcwxc1tvb7"
+ },
+ {
+ ...
+ "email":"admin@arva2.arv.local",
+ ...
+ "owner_uuid":"arva2-tpzed-000000000000000",
+ ...
+ "username":"admin",
+ "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
+ ...
+ }
+ ],
+ "kind":"arvados#HashList"
+}
+Activating user 'admin'
+{
+ ...
+ "email":"admin@arva2.arv.local",
+ ...
+ "username":"admin",
+ "uuid":"arva2-tpzed-3wrm93zmzpshrq2",
+ ...
+}
+Running test CWL workflow
+INFO /usr/bin/cwl-runner 2.1.1, arvados-python-client 2.1.1, cwltool 3.0.20200807132242
+INFO Resolved 'hasher-workflow.cwl' to 'file:///tmp/cluster_tests/hasher-workflow.cwl'
+...
+INFO Using cluster arva2 (https://arva2.arv.local:8443/)
+INFO Upload local files: "test.txt"
+INFO Uploaded to ea34d971b71d5536b4f6b7d6c69dc7f6+50 (arva2-4zz18-c8uvwqdry4r8jao)
+INFO Using collection cache size 256 MiB
+INFO [container hasher-workflow.cwl] submitted container_request arva2-xvhdp-v1bkywd58gyocwm
+INFO [container hasher-workflow.cwl] arva2-xvhdp-v1bkywd58gyocwm is Final
+INFO Overall process status is success
+INFO Final output collection d6c69a88147dde9d52a418d50ef788df+123
+{
+ "hasher_out": {
+ "basename": "hasher3.md5sum.txt",
+ "class": "File",
+ "location": "keep:d6c69a88147dde9d52a418d50ef788df+123/hasher3.md5sum.txt",
+ "size": 95
+ }
+}
+INFO Final process status is success
+
+