.. _readme:
-TEMPLATE-formula
+arvados-formula
================
|img_travis| |img_sr|
-.. |img_travis| image:: https://travis-ci.com/saltstack-formulas/TEMPLATE-formula.svg?branch=master
+.. |img_travis| image:: https://travis-ci.com/saltstack-formulas/arvados-formula.svg?branch=master
:alt: Travis CI Build Status
:scale: 100%
- :target: https://travis-ci.com/saltstack-formulas/TEMPLATE-formula
+ :target: https://travis-ci.com/saltstack-formulas/arvados-formula
.. |img_sr| image:: https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg
:alt: Semantic Release
:scale: 100%
:target: https://github.com/semantic-release/semantic-release
-A SaltStack formula that is empty. It has dummy content to help with a quick
-start on a new formula and it serves as a style guide.
+A SaltStack formula to install and configure an `Arvados cluster <https://arvados.org>`_.
.. contents:: **Table of Contents**
Special notes
-------------
-.. <REMOVEME
+In the `Arvados repository <https://github.com/arvados/arvados/>`_ you can find `a provision script <https://github.com/arvados/arvados/tree/master/tools/salt-install>`_
+to deploy a single-node, all-in-one Arvados cluster (The script uses this formula to get a cluster up and running in Saltstack's master-less mode).
-Using this template
-^^^^^^^^^^^^^^^^^^^
+The `single-node` install does not include SLURM: it is intended for an `all-in-one-host` installation,
+so it uses `crunch-dispatch-local` to run containers in the same instance.
-The easiest way to use this template formula as a base for a new formula is to use GitHub's **Use this template** button to create a new repository. For consistency with the rest of the formula ecosystem, name your formula repository following the pattern ``<formula theme>-formula``, where ``<formula theme>`` consists of lower-case alphabetic characters and numbers.
+The provision script can be run anywhere, so you can run it in an AWS instance and you'll get a `single-node` Arvados cluster there.
-In the rest of this example we'll use ``example`` as the ``<formula theme>``.
+The Arvados formula allows you to `install any dispatcher available <https://github.com/saltstack-formulas/arvados-formula/blob/master/pillar.example#L182-L191>`_,
+provided you configure the pillars the way you need them.
-Follow these steps to complete the conversion from ``template-formula`` to ``example-formula``. ::
+Arvados currently has three dispatchers:
- $ git clone git@github.com:YOUR-USERNAME/example-formula.git
- $ cd example-formula/
- $ bin/convert-formula.sh example
- $ git push --force
+* **crunch-dispatch-local** (for single node installations),
+* **arvados-dispatch-cloud** (for dynamic compute on AWS or Azure) and
+* **crunch-dispatch-slurm** (for SLURM integration).
-Alternatively, it's possible to clone ``template-formula`` into a new repository and perform the conversion there. For example::
+Requisites
+----------
- $ git clone https://github.com/saltstack-formulas/template-formula example-formula
- $ cd example-formula/
- $ bin/convert-formula.sh example
+Arvados **requires** a Postgres database for its API server and SSL for communications. If you don't satisfy these two requirements, things
+won't work. It also uses an Nginx server as a redirector but probably almost any other webserver/redirector can be used instead.
-To take advantage of `semantic-release <https://github.com/semantic-release/semantic-release>`_ for automated changelog generation and release tagging, you will need a GitHub `Personal Access Token <https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line>`_ with at least the **public_repo** scope.
+We suggest you use the `postgres-formula <https://github.com/saltstack-formulas/postgres-formula/>`_,
+the `nginx-formula <https://github.com/saltstack-formulas/nginx-formula/>`_ and the
+`letsencrypt-formula <https://github.com/saltstack-formulas/letsencrypt-formula/>`_ to satisfy these dependencies.
+In the **test/salt/pillar/examples/** directory there are example pillar YAMLs to set up these packages, using the mentioned formulas
+as Arvados needs them.a
-In the Travis repository settings for your new repository, create an `environment variable <https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings>`_ named ``GH_TOKEN`` with the personal access token as value, restricted to the ``master`` branch for security.
+In the **test/salt/states/examples/** directory there are some example helper states to set up a few requirements for single-node
+(all-in-one) Arvados host.
-.. REMOVEME>
+Usage
+-----
+
+As Arvados is a *suite* of tools that can be installed in different hosts and configured to interact, this formula is split in
+those components, which can be installed or removed independently of the other components. This means that you'll get flexibility
+to install your cluster as you prefer at the expense of having to take care on some steps:
+
+The formula has the following components/submodules available:
+
+* `api <https://doc.arvados.org/install/install-api-server.html>`_: installs the Arvados API server packages. Requires a running
+ Postgres database and an Nginx+Passenger server.
+* `config <https://doc.arvados.org/v2.0/admin/config.html>`_: creates and deploys a valid Arvados config file. This state is automatically
+ include in all the components that require it (at the moment, all but `shell`), so you will rarely need to invoke this state manually.
+* `controller <https://doc.arvados.org/v2.0/install/install-api-server.html>`_: installs the Arvados API controller.
+* `keepproxy <https://doc.arvados.org/v2.0/install/install-keepproxy.html>`_: installs and configures the Arvados Keepproxy gateway
+ to the Keep storages.
+* `keepstore <https://doc.arvados.org/v2.0/install/install-keepstore.html>`_: installs and configures an Arvados Keep storages.
+* `keepweb <https://doc.arvados.org/v2.0/install/install-keep-web.html>`_: installs and configures the WebDAV access to the Keep storages.
+* `repo <https://doc.arvados.org/v2.0/install/packages.html>`_: configures the repositories to install arvados. It's enabled by default.
+* `shell <https://doc.arvados.org/v2.0/install/install-shell-server.html>`_: installs the user CLI apps to communicate with the cluster.
+* `websocket <https://doc.arvados.org/v2.0/install/install-ws.html>`_: installs the websocket notifcations gateway.
+* `workbench <https://doc.arvados.org/v2.0/install/install-workbench-app.html>`_: installs the webUI to communicate with the cluster.
+* `workbench2 <https://doc.arvados.org/v2.0/install/install-workbench2-app.html>`_: installs the next generation webUI for Arvados.
+
+If you just use the `arvados` meta-state, it will install all the components in a single host.
+
+Also, please note that the individual subcomponents' `clean` states **won't remove the config file**: as the config is common to all the suite
+components and they can be installed in the same host, removing it with a subcomponent might break others.
+
+If you want to remove the config in a host where you're removing a subcomponent, use the `arvados.config.clean` state after the
+`arvados.<subcomponent>.clean` state.
+
+Finally, the `arvados.clean` meta-state will remove everything, config included, and can be used in any host to remove all of arvados files.
Available states
----------------
+For each of the components, there are *meta-states* named after the component that will include other states in the component subdir
+that perform the actual work.
+
+For example, using *arvados.keepstore* will include, in order:
+
+* arvados.keepstore.package.install
+* arvados.config.file
+* arvados.keepstore.service.running
+
+while using *arvados.keepstore.clean* will include, in order:
+
+* arvados.keepstore.service.clean
+* arvados.keepstore.package.clean
+
+Or you can use individual states, like
+
+* arvados.keepstore.package.install
+* arvados.keepstore.service.clean
+
+to get the *keepstore* package installed with the service stopped.
+
+The generic description for the states is
+
.. contents::
:local:
-``TEMPLATE``
+``arvados``
^^^^^^^^^^^^
*Meta-state (This is a state that includes other states)*.
-This installs the TEMPLATE package,
-manages the TEMPLATE configuration file and then
-starts the associated TEMPLATE service.
+This installs the *WHOLE* arvados suite in a single host,
+manages the arvados configuration file and then
+starts the associated arvados services.
-``TEMPLATE.package``
-^^^^^^^^^^^^^^^^^^^^
+``arvados.clean``
+^^^^^^^^^^^^^^^^^
-This state will install the TEMPLATE package only.
+*Meta-state (This is a state that includes other states)*.
-``TEMPLATE.config``
-^^^^^^^^^^^^^^^^^^^
+This state will undo everything performed in the ``arvados`` meta-state in reverse order, i.e.
+stops the services, removes the configuration file and then uninstalls the packages.
-This state will configure the TEMPLATE service and has a dependency on ``TEMPLATE.install``
-via include list.
-``TEMPLATE.service``
-^^^^^^^^^^^^^^^^^^^^
+``arvados.config``
+^^^^^^^^^^^^^^^^^^
-This state will start the TEMPLATE service and has a dependency on ``TEMPLATE.config``
-via include list.
+This state will configure the arvados cluster. As all the arvados components use the same config
+file, any of the following components will include this state and you will rarely need to call it
+independently. You can still do, ie, to get a parsed config file to use somewhere else.
-``TEMPLATE.clean``
-^^^^^^^^^^^^^^^^^^
+``arvados.config.clean``
+^^^^^^^^^^^^^^^^^^^^^^^^^
-*Meta-state (This is a state that includes other states)*.
+This state will remove the configuration of the arvados node.
+
+``arvados.repo``
+^^^^^^^^^^^^^^^^
-this state will undo everything performed in the ``TEMPLATE`` meta-state in reverse order, i.e.
-stops the service,
-removes the configuration file and
-then uninstalls the package.
+This state will configure the arvados repository.
-``TEMPLATE.service.clean``
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+``arvados.repo.clean``
+^^^^^^^^^^^^^^^^^^^^^^
-This state will stop the TEMPLATE service and disable it at boot time.
+This state will remove the arvados repository configuration.
-``TEMPLATE.config.clean``
-^^^^^^^^^^^^^^^^^^^^^^^^^
-This state will remove the configuration of the TEMPLATE service and has a
-dependency on ``TEMPLATE.service.clean`` via include list.
+``arvados.<component>``
+^^^^^^^^^^^^^^^^^^^^^^^
-``TEMPLATE.package.clean``
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+*Meta-state (This is a state that includes other states)*.
-This state will remove the TEMPLATE package and has a depency on
-``TEMPLATE.config.clean`` via include list.
+This state will install the package, configure the component (if applicable) and start the service (if applicable).
-``TEMPLATE.subcomponent``
-^^^^^^^^^^^^^^^^^^^^^^^^^
+``arvados.<component>.clean``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*Meta-state (This is a state that includes other states)*.
-This state installs a subcomponent configuration file before
-configuring and starting the TEMPLATE service.
+This state will undo everything performed in the ``arvados.<component>`` meta-state in reverse order, i.e.
+stop the service and uninstall the package/s.
+
+``arvados.<component>.package``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This state will install the arvados <component> package/s only.
-``TEMPLATE.subcomponent.config``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+``arvados.<component>.package.clean``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This state will remove the packages of the arvados <component> node and has a depency on
+``arvados.<component>.service.clean`` via include list (if applicable).
+
+``arvados.<component>.service``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This state will start the arvados service and has a dependency on ``arvados.config``
+via include list.
-This state will configure the TEMPLATE subcomponent and has a
-dependency on ``TEMPLATE.config`` via include list.
+``arvados.<component>.service.clean``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-``TEMPLATE.subcomponent.config.clean``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This state will stop the arvados service and disable it at boot time.
-This state will remove the configuration of the TEMPLATE subcomponent
-and reload the TEMPLATE service by a dependency on
-``TEMPLATE.service.running`` via include list and ``watch_in``
-requisite.
Testing
-------
$ bin/kitchen test [platform]
Where ``[platform]`` is the platform name defined in ``kitchen.yml``,
-e.g. ``debian-9-2019-2-py3``.
+e.g. ``debian-10-3000-1-py3``.
``bin/kitchen converge``
^^^^^^^^^^^^^^^^^^^^^^^^
-Creates the docker instance and runs the ``TEMPLATE`` main state, ready for testing.
+Creates the docker instance and runs the ``arvados`` main state, ready for testing.
``bin/kitchen verify``
^^^^^^^^^^^^^^^^^^^^^^
projects / arvados-formula.git / blobdiff