- install/install-shell-server.html.textile.liquid
- install/install-webshell.html.textile.liquid
- install/install-arv-git-httpd.html.textile.liquid
- - Containers API:
+ - Containers API (cloud):
- install/install-jobs-image.html.textile.liquid
- install/crunch2-cloud/install-compute-node.html.textile.liquid
- install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid
- - install/crunch2-slurm/install-compute-node.html.textile.liquid
+ - Containers API (slurm):
- install/crunch2-slurm/install-dispatch.html.textile.liquid
+ - install/crunch2-slurm/configure-slurm.html.textile.liquid
+ - install/crunch2-slurm/install-compute-node.html.textile.liquid
- install/crunch2-slurm/install-test.html.textile.liquid
- External dependencies:
- install/install-postgresql.html.textile.liquid
+++ /dev/null
-user-management.html.textile.liquid
\ No newline at end of file
+++ /dev/null
-logging.html.textile.liquid
\ No newline at end of file
+++ /dev/null
-../admin/user-management-cli.html.textile.liquid
\ No newline at end of file
+++ /dev/null
----
-layout: default
-navsection: installguide
-title: Install client libraries
-
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-The "SDK Reference":{{site.baseurl}}/sdk/index.html page has installation instructions for each of the SDKs.
-
---
layout: default
navsection: installguide
-title: Set up Slurm
+title: Configure Slurm
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+{% include 'notebox_begin_warning' %}
+crunch-dispatch-slurm is only relevant for on premises clusters that will spool jobs to Slurm. Skip this section if you are installing a cloud cluster.
+{% include 'notebox_end' %}
+
Containers can be dispatched to a Slurm cluster. The dispatcher sends work to the cluster using Slurm's @sbatch@ command, so it works in a variety of SLURM configurations.
In order to run containers, you must run the dispatcher as a user that has permission to set up FUSE mounts and run Docker containers on each compute node. This install guide refers to this user as the @crunch@ user. We recommend you create this user on each compute node with the same UID and GID, and add it to the @fuse@ and @docker@ system groups to grant it the necessary permissions. However, you can run the dispatcher under any account with sufficient permissions across the cluster.
+We will assume that you have Slurm and munge running.
-On the API server, install Slurm and munge, and generate a munge key.
-
-On Debian-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get install slurm-llnl munge</span>
-~$ <span class="userinput">sudo /usr/sbin/create-munge-key</span>
-</code></pre>
-</notextile>
-
-On Red Hat-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install slurm munge slurm-munge</span>
-</code></pre>
-</notextile>
+h3. Sample Slurm configuration file
-Now we need to give Slurm a configuration file. On Debian-based systems, this is installed at @/etc/slurm-llnl/slurm.conf@. On Red Hat-based systems, this is installed at @/etc/slurm/slurm.conf@. Here's an example @slurm.conf@:
+Here's an example @slurm.conf@ for use with Arvados:
<notextile>
<pre><code>
+++ /dev/null
----
-layout: default
-navsection: installguide
-title: Containers API Slurm prerequisites
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
+++ /dev/null
----
-layout: default
-navsection: installguide
-title: Overview
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-{% comment %}
-Obsolete page, no longer in nav.
-{% endcomment %}
-
-<notextile><script>
-window.location = "install-manual-prerequisites.html";
-</script></notextile>
-
-Please proceed to "Prerequisites":install-manual-prerequisites.html.
+++ /dev/null
----
-layout: default
-navsection: installguide
-title: Install pre-built Docker images
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-This method is intended for evaluation and development on a local workstation. It is not suitable for production use in a cluster deployment.
-
-{% include 'notebox_begin' %}
-* The automatic network configuration allows you to log in to Workbench from a browser _running on the same host as Docker_. Connecting from other hosts requires additional configuration (not covered here).
-* Your data will be stored inside the Docker containers. You may stop and restart the containers without loss, but if you delete the container, your data will be gone.
-* Updating the Arvados software inside the Docker containers is not supported. You may download updated Docker images, but migrating data to updated containers is not yet supported.
-{% include 'notebox_end' %}
-
-h2. Prerequisites
-
-# A GNU/Linux x64 (virtual) machine
-# A working Docker installation (see "Installing Docker":https://docs.docker.com/installation/)
-# curl
-
-h2. Verify prerequisites
-
-Make sure that @curl@ and @docker@ are installed on your system, and that you are in the docker group (see "Installing Docker":https://docs.docker.com/installation/).
-
-<notextile><pre><code>~$ <span class="userinput">which curl</span>
-/usr/bin/curl
-~$ <span class="userinput">docker.io --version</span>
-Docker version 1.2.0-dev, build dc243c8
-~$ <span class="userinput">groups</span>
-yourusername sudo fuse docker
-</code></pre></notextile>
-
-h2. Download and install Arvados.
-
-<notextile>
-<pre><code>~$ <span class="userinput">\curl -sSL get.arvados.org | bash</span>
-</code></pre></notextile>
-
-This command will download the latest build of the Arvados docker images. It also gets the @arvdock@ command and saves it in the current working directory. It then uses @arvdock@ to spin up Arvados. Note that the Arvados Docker images are large and may take a while to download.
-
-If you prefer, you can also download and inspect the installation script before running it. @get.arvados.org@ redirects to "https://raw.githubusercontent.com/curoverse/arvados-dev/master/install/easy-docker-install.sh":https://raw.githubusercontent.com/curoverse/arvados-dev/master/install/easy-docker-install.sh, which is the installation script.
-
-The @arvdock@ command usage is listed here:
-
-<pre>
-usage: ./arvdock (start|stop|restart|reset|test) [options]
-
-start run new or restart stopped arvados containers
-stop stop arvados containers
-restart stop and then start arvados containers
-reset stop and delete containers WARNING: this will delete the data inside Arvados!
-test run tests
-
-./arvdock start/stop/restart options:
- -d[port], --doc[=port] Documentation server (default port 9898)
- -w[port], --workbench[=port] Workbench server (default port 9899)
- -s[port], --sso[=port] SSO server (default port 9901)
- -a[port], --api[=port] API server (default port 9900)
- -c, --compute Compute nodes (starts 2)
- -v, --vm Shell server
- -n, --nameserver Nameserver
- -k, --keep Keep servers
- -p, --keepproxy Keepproxy server
- -h, --help Display this help and exit
-
- If no options are given, the action is applied to all servers.
-
-./arvdock test [testname] [testname] ...
- By default, all tests are run.
-</pre>
+++ /dev/null
----
-layout: default
-navsection: userguide
-title: Create a Workflow with Composer
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-The Arvados Workflow Composer is a graphical interface for building Common Workflow Language (CWL) workflows to run on Arvados.
-
-This tutorial will demonstrate:
-
-# Creating a new git repository through Arvados to store the workflow
-# Creating CommandLineTools for "sort" and "uniq"
-# Creating a Workflow which uses "sort" and "uniq" to remove duplicate lines from a text file
-# Submitting the Workflow to run on Arvados
-
-h3. 1. Access from workbench
-
-!(screenshot)c1.png!
-
-h3. 2. Composer starting page
-
-!(screenshot)c2.png!
-
-h3. 3. Manage git repositories (opens Workbench in new tab)
-
-!(screenshot)c2b.png!
-
-h3. 4. Add a new repository
-
-!(screenshot)c4.png!
-
-!(screenshot)c3.png!
-
-h3. 5. Return to Composer. Use refresh button to discover new repository (may take a few moments to show up).
-
-!(screenshot)c2c.png!
-
-h3. 6. Create a new Command Line Tool
-
-!(screenshot)c5.png!
-
-!(screenshot)c20.png!
-
-h3. 7. Set Docker image, base command, and input port for "sort" tool
-
-The "Docker Repository" is the name:tag of a "Docker image uploaded Arvados.":{{site.baseurl}}/user/topics/arv-docker.html (Use @arv-keepdocker --pull debian:10@) You can also find prepackaged bioinformatics tools on various sites, such as http://dockstore.org and http://biocontainers.pro/ .
-
-!(screenshot)c6.png!
-
-h3. 8. Redirect stdout to a file
-
-!(screenshot)c7.png!
-
-h3. 9. Capture output file
-
-!(screenshot)c8.png!
-
-h3. 10. Save Command Line Tool
-
-!(screenshot)c22.png!
-
-h3. 11. Repeat steps 6-10 for "uniq" tool
-
-Create a new tool with a "base command" of "uniq".
-
-h3. 12. Switch back to "Home" tab and create workflow
-
-!(screenshot)c24.png!
-
-!(screenshot)c9.png!
-
-!(screenshot)c10.png!
-
-h3. 13. Drag and drop tools into Workflow
-
-!(screenshot)c11.png!
-
-h3. 14. Drag from input port of "sort" to empty space to create workflow input
-
-!(screenshot)c21.png!
-
-h3. 15. Drag from output port of "sort" to input port of "uniq"
-
-!(screenshot)c13.png!
-
-h3. 16. Drag from output port of "uniq" to empty space to create workflow output
-
-!(screenshot)c14.png!
-
-h3. 17. Save Workflow
-
-!(screenshot)c23.png!
-
-h3. 18. Click on "Test" tab then click "Run"
-
-!(screenshot)c15.png!
-
-h3. 19. Choose input file
-
-You may need to "upload an input file":{{site.baseurl}}/user/tutorials/tutorial-keep.html
-
-!(screenshot)c16.png!
-
-h3. 20. Run the workflow
-
-!(screenshot)c17.png!
-
-h3. 21. Monitor progress (may take several minutes)
-
-!(screenshot)c18.png!
-
-h3. 22. Get workflow output
-
-!(screenshot)c19.png!
+++ /dev/null
----
-layout: default
-navsection: userguide
-title: Introduction to Crunch
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-The Arvados "Crunch" framework is designed to support processing very large data batches (gigabytes to terabytes) efficiently, and provides the following benefits:
-* Increase concurrency by running tasks asynchronously, using many CPUs and network interfaces at once (especially beneficial for CPU-bound and I/O-bound tasks respectively).
-* Track inputs, outputs, and settings so you can verify that the inputs, settings, and sequence of programs you used to arrive at an output is really what you think it was.
-* Ensure that your programs and workflows are repeatable with different versions of your code, OS updates, etc.
-* Interrupt and resume long-running jobs consisting of many short tasks.
-* Maintain timing statistics automatically, so they're there when you want them.
-
-h2. Prerequisites
-
-To get the most value out of this section, you should be comfortable with the following:
-
-# Using a secure shell client such as SSH or PuTTY to log on to a remote server
-# Using the Unix command line shell, Bash
-# Viewing and editing files using a unix text editor such as vi, Emacs, or nano
-# Revision control using Git
-
-We also recommend you read the "Arvados Platform Overview":https://dev.arvados.org/projects/arvados/wiki#Platform-Overview for an introduction and background information about Arvados.