X-Git-Url: https://git.arvados.org/arvados-formula.git/blobdiff_plain/0fd79b30ae5f9b55490e701782bcaba992951eb6..HEAD:/docs/README.rst diff --git a/docs/README.rst b/docs/README.rst index 7e252f8..b756aaa 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,21 +1,14 @@ .. _readme: -TEMPLATE-formula -================ - -|img_travis| |img_sr| +.. + # Copyright (C) The Arvados Authors. All rights reserved. + # + # SPDX-License-Identifier: Apache-2.0 -.. |img_travis| image:: https://travis-ci.com/saltstack-formulas/TEMPLATE-formula.svg?branch=master - :alt: Travis CI Build Status - :scale: 100% - :target: https://travis-ci.com/saltstack-formulas/TEMPLATE-formula -.. |img_sr| image:: https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg - :alt: Semantic Release - :scale: 100% - :target: https://github.com/semantic-release/semantic-release +arvados-formula +================ -A SaltStack formula that is empty. It has dummy content to help with a quick -start on a new formula and it serves as a style guide. +A SaltStack formula to install and configure an `Arvados cluster `_. .. contents:: **Table of Contents** @@ -25,14 +18,6 @@ General notes See the full `SaltStack Formulas installation and usage instructions `_. -If you are interested in writing or contributing to formulas, please pay attention to the `Writing Formula Section -`_. - -If you want to use this formula, please pay attention to the ``FORMULA`` file and/or ``git tag``, -which contains the currently released version. This formula is versioned according to `Semantic Versioning `_. - -See `Formula Versioning Section `_ for more details. - If you need (non-default) configuration, please pay attention to the ``pillar.example`` file and/or `Special notes`_ section. Contributing to this repo @@ -45,114 +30,179 @@ Please see `How to contribute `_ you can find `a provision script `_ +to deploy a single-node, all-in-one Arvados cluster (The script uses this formula to get a cluster up and running in Saltstack's master-less mode). + +The `single-node` install does not include SLURM: it is intended for an `all-in-one-host` installation, +so it uses `crunch-dispatch-local` to run containers in the same instance. + +The provision script can be run anywhere, so you can run it in an AWS instance and you'll get a `single-node` Arvados cluster there. + +The Arvados formula allows you to `install any dispatcher available `_, +provided you configure the pillars the way you need them. + +Arvados currently has three dispatchers: + +* **crunch-dispatch-local** (for single node installations and testing purposes, not suitable for production), +* **arvados-dispatch-cloud** (for dynamic compute on AWS or Azure) and +* **crunch-dispatch-slurm** (for SLURM integration). -Using this template -^^^^^^^^^^^^^^^^^^^ +Requisites +---------- -The easiest way to use this template formula as a base for a new formula is to use GitHub's **Use this template** button to create a new repository. For consistency with the rest of the formula ecosystem, name your formula repository following the pattern ``-formula``, where ```` consists of lower-case alphabetic characters and numbers. +Arvados **requires** a Postgres database for its API server and SSL for communications. If you don't satisfy these two requirements, things +won't work. It also uses an Nginx server as a redirector but probably almost any other webserver/redirector can be used instead. -In the rest of this example we'll use ``example`` as the ````. +We suggest you use the `postgres-formula `_, +the `nginx-formula `_ and the +`letsencrypt-formula `_ to satisfy these dependencies. +In the **test/salt/pillar/examples/** directory there are example pillar YAMLs to set up these packages, using the mentioned formulas +as Arvados needs them.a -Follow these steps to complete the conversion from ``template-formula`` to ``example-formula``. :: +In the **test/salt/states/examples/** directory there are some example helper states to set up a few requirements for single-node +(all-in-one) Arvados host. - $ git clone git@github.com:YOUR-USERNAME/example-formula.git - $ cd example-formula/ - $ bin/convert-formula.sh example - $ git push --force +Usage +----- -Alternatively, it's possible to clone ``template-formula`` into a new repository and perform the conversion there. For example:: +As Arvados is a *suite* of tools that can be installed in different hosts and configured to interact, this formula is split in +those components, which can be installed or removed independently of the other components. This means that you'll get flexibility +to install your cluster as you prefer at the expense of having to take care on some steps: - $ git clone https://github.com/saltstack-formulas/template-formula example-formula - $ cd example-formula/ - $ bin/convert-formula.sh example +The formula has the following components/submodules available: -To take advantage of `semantic-release `_ for automated changelog generation and release tagging, you will need a GitHub `Personal Access Token `_ with at least the **public_repo** scope. +* `api `_: installs the Arvados API server packages. Requires a running + Postgres database and an Nginx+Passenger server. +* `config `_: creates and deploys a valid Arvados config file. This state is automatically + include in all the components that require it (at the moment, all but `shell`), so you will rarely need to invoke this state manually. +* `controller `_: installs the Arvados API controller. +* `keepproxy `_: installs and configures the Arvados Keepproxy gateway + to the Keep storages. +* `keepstore `_: installs and configures an Arvados Keep storages. +* `keepweb `_: installs and configures the WebDAV access to the Keep storages. +* `repo `_: configures the repositories to install arvados. It's enabled by default. +* `shell `_: installs the user CLI apps to communicate with the cluster. +* `websocket `_: installs the websocket notifcations gateway. +* `workbench `_: installs the webUI to communicate with the cluster. +* `workbench2 `_: installs the next generation webUI for Arvados. -In the Travis repository settings for your new repository, create an `environment variable `_ named ``GH_TOKEN`` with the personal access token as value, restricted to the ``master`` branch for security. +If you just use the `arvados` meta-state, it will install all the components in a single host. -.. REMOVEME> +Also, please note that the individual subcomponents' `clean` states **won't remove the config file**: as the config is common to all the suite +components and they can be installed in the same host, removing it with a subcomponent might break others. + +If you want to remove the config in a host where you're removing a subcomponent, use the `arvados.config.clean` state after the +`arvados..clean` state. + +Finally, the `arvados.clean` meta-state will remove everything, config included, and can be used in any host to remove all of arvados files. Available states ---------------- +For each of the components, there are *meta-states* named after the component that will include other states in the component subdir +that perform the actual work. + +For example, using *arvados.keepstore* will include, in order: + +* arvados.keepstore.package.install +* arvados.config.file +* arvados.keepstore.service.running + +while using *arvados.keepstore.clean* will include, in order: + +* arvados.keepstore.service.clean +* arvados.keepstore.package.clean + +Or you can use individual states, like + +* arvados.keepstore.package.install +* arvados.keepstore.service.clean + +to get the *keepstore* package installed with the service stopped. + +The generic description for the states is + .. contents:: :local: -``TEMPLATE`` +``arvados`` ^^^^^^^^^^^^ *Meta-state (This is a state that includes other states)*. -This installs the TEMPLATE package, -manages the TEMPLATE configuration file and then -starts the associated TEMPLATE service. +This installs the *WHOLE* arvados suite in a single host, +manages the arvados configuration file and then +starts the associated arvados services. -``TEMPLATE.package`` -^^^^^^^^^^^^^^^^^^^^ +``arvados.clean`` +^^^^^^^^^^^^^^^^^ -This state will install the TEMPLATE package only. +*Meta-state (This is a state that includes other states)*. -``TEMPLATE.config`` -^^^^^^^^^^^^^^^^^^^ +This state will undo everything performed in the ``arvados`` meta-state in reverse order, i.e. +stops the services, removes the configuration file and then uninstalls the packages. -This state will configure the TEMPLATE service and has a dependency on ``TEMPLATE.install`` -via include list. -``TEMPLATE.service`` -^^^^^^^^^^^^^^^^^^^^ +``arvados.config`` +^^^^^^^^^^^^^^^^^^ -This state will start the TEMPLATE service and has a dependency on ``TEMPLATE.config`` -via include list. +This state will configure the arvados cluster. As all the arvados components use the same config +file, any of the following components will include this state and you will rarely need to call it +independently. You can still do, ie, to get a parsed config file to use somewhere else. -``TEMPLATE.clean`` -^^^^^^^^^^^^^^^^^^ +``arvados.config.clean`` +^^^^^^^^^^^^^^^^^^^^^^^^^ -*Meta-state (This is a state that includes other states)*. +This state will remove the configuration of the arvados node. -this state will undo everything performed in the ``TEMPLATE`` meta-state in reverse order, i.e. -stops the service, -removes the configuration file and -then uninstalls the package. +``arvados.repo`` +^^^^^^^^^^^^^^^^ -``TEMPLATE.service.clean`` -^^^^^^^^^^^^^^^^^^^^^^^^^^ +This state will configure the arvados repository. -This state will stop the TEMPLATE service and disable it at boot time. +``arvados.repo.clean`` +^^^^^^^^^^^^^^^^^^^^^^ -``TEMPLATE.config.clean`` -^^^^^^^^^^^^^^^^^^^^^^^^^ +This state will remove the arvados repository configuration. -This state will remove the configuration of the TEMPLATE service and has a -dependency on ``TEMPLATE.service.clean`` via include list. -``TEMPLATE.package.clean`` -^^^^^^^^^^^^^^^^^^^^^^^^^^ +``arvados.`` +^^^^^^^^^^^^^^^^^^^^^^^ -This state will remove the TEMPLATE package and has a depency on -``TEMPLATE.config.clean`` via include list. +*Meta-state (This is a state that includes other states)*. -``TEMPLATE.subcomponent`` -^^^^^^^^^^^^^^^^^^^^^^^^^ +This state will install the package, configure the component (if applicable) and start the service (if applicable). + +``arvados..clean`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *Meta-state (This is a state that includes other states)*. -This state installs a subcomponent configuration file before -configuring and starting the TEMPLATE service. +This state will undo everything performed in the ``arvados.`` meta-state in reverse order, i.e. +stop the service and uninstall the package/s. + +``arvados..package`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``TEMPLATE.subcomponent.config`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +This state will install the arvados package/s only. + +``arvados..package.clean`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This state will remove the packages of the arvados node and has a depency on +``arvados..service.clean`` via include list (if applicable). + +``arvados..service`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This state will start the arvados service and has a dependency on ``arvados.config`` +via include list. -This state will configure the TEMPLATE subcomponent and has a -dependency on ``TEMPLATE.config`` via include list. +``arvados..service.clean`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``TEMPLATE.subcomponent.config.clean`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +This state will stop the arvados service and disable it at boot time. -This state will remove the configuration of the TEMPLATE subcomponent -and reload the TEMPLATE service by a dependency on -``TEMPLATE.service.running`` via include list and ``watch_in`` -requisite. Testing ------- @@ -172,12 +222,12 @@ Requirements $ bin/kitchen test [platform] Where ``[platform]`` is the platform name defined in ``kitchen.yml``, -e.g. ``debian-9-2019-2-py3``. +e.g. ``debian-10-3000-1-py3``. ``bin/kitchen converge`` ^^^^^^^^^^^^^^^^^^^^^^^^ -Creates the docker instance and runs the ``TEMPLATE`` main state, ready for testing. +Creates the docker instance and runs the ``arvados`` main state, ready for testing. ``bin/kitchen verify`` ^^^^^^^^^^^^^^^^^^^^^^