--- /dev/null
+<notextile>
+<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
+</code></pre>
+</notextile>
--- /dev/null
+<notextile>
+<pre><code>~$ <span class="userinput">gpg --keyserver pool.sks-keyservers.net --recv-keys 1078ECD7</span>
+~$ <span class="userinput">gpg --armor --export 1078ECD7 >/tmp/curoverse.key</span>
+~$ <span class="userinput">sudo rpm --import /tmp/curoverse.key</span>
+</code></pre>
+</notextile>
--- /dev/null
+{% include 'notebox_begin' %}
+
+On older Red Hat-based systems, these packages require the "python27 Software Collection":https://www.softwarecollections.org/en/scls/rhscl/python27/.
+
+{% include 'notebox_end' %}
title: Install Git server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
The arv-git-httpd server provides HTTP access to hosted git repositories, using Arvados authentication tokens instead of passwords. It is intended to be installed on the system where your git repositories are stored, and accessed through a web proxy that provides SSL support.
By convention, we use the following hostname for the git service:
h2. Install arv-git-httpd
-First add the Arvados apt repository, and then install the arv-git-httpd package.
+On Debian-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install git arv-git-httpd</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install arv-git-httpd</span>
+<pre><code>~$ <span class="userinput">sudo yum install git arv-git-httpd</span>
</code></pre>
</notextile>
</code></pre>
</notextile>
-We recommend running @arv-git-httpd@ under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+We recommend running @arv-git-httpd@ under "runit":http://smarden.org/runit/ or something similar.
Your @run@ script should look something like this:
h3. Tell the API server about the arv-git-httpd service
-In your API server's config/application.yml file, add the following entry:
+In your API server's @config/application.yml@ file, add the following entry:
<notextile>
<pre><code>git_http_base: git.<span class="userinput">uuid_prefix</span>.your.domain
title: Install a compute node
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
h2. Install dependencies
-First add the Arvados apt repository, and then install a number of packages.
+First, "add the appropriate package repository for your distribution":/install/install-manual-prerequisites.html#repos.
+
+On Debian-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install python-pip python-pyvcf python-gflags python-google-api-python-client python-virtualenv libattr1-dev libfuse-dev python-dev python-llfuse fuse crunchstat python-arvados-fuse iptables ca-certificates lxc apt-transport-https docker.io</span>
+<pre><code>~$ <span class="userinput">sudo apt-get install perl python-virtualenv fuse python-arvados-python-client python-arvados-fuse crunchstat iptables ca-certificates</span>
</code></pre>
</notextile>
...
-
-
The dispatcher normally runs on the same host/VM as the API server.
-h4. Perl SDK dependencies
+h2. Perl SDK dependencies
Install the Perl SDK on the controller.
* See "Perl SDK":{{site.baseurl}}/sdk/perl/index.html page for details.
-h4. Python SDK dependencies
+h2. Python SDK dependencies
Install the Python SDK and CLI tools on controller and all compute nodes.
* See "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html page for details.
-h4. Slurm
+h2(#slurm). Set up SLURM
+
+On the API server, install SLURM and munge, and generate a munge key.
-On the API server, install slurm and munge, and generate a munge key:
+On Debian-based systems:
<notextile>
<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get install slurm-llnl munge</span>
</code></pre>
</notextile>
-Now we need to give slurm a configuration file in @/etc/slurm-llnl/slurm.conf@. Here's an example:
+On Red Hat-based systems, "install SLURM and munge from source following their installation guide":https://computing.llnl.gov/linux/slurm/quickstart_admin.html.
+
+Now we need to give SLURM a configuration file in @/etc/slurm-llnl/slurm.conf@. Here's an example:
<notextile>
<pre>
</pre>
</notextile>
-Please make sure to update the value of the @ControlMachine@ parameter to the hostname of your dispatcher (api server).
+Please make sure to update the value of the @ControlMachine@ parameter to the hostname of your dispatcher (API server).
+
+h2. Enable SLURM job dispatch
h4. Crunch user account
ArgumentError: Specified script_version does not resolve to a commit
</pre>
-h4. Running jobs
+h2. Running jobs
* @services/api/script/crunch-dispatch.rb@ must be running.
* @crunch-dispatch.rb@ needs @services/crunch/crunch-job@ in its @PATH@.
-* @crunch-job@ needs @sdk/perl/lib@ and @warehouse-apps/libwarehouse-perl/lib@ in its @PERLLIB@
-* @crunch-job@ needs @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@)
+* @crunch-job@ needs the installation path of the Perl SDK in its @PERLLIB@.
+* @crunch-job@ needs the @ARVADOS_API_HOST@ (and, if necessary in a development environment, @ARVADOS_API_HOST_INSECURE@) environment variable set.
Example @/var/service/arvados_crunch_dispatch/run@ script:
title: Install Keepproxy server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
The Keepproxy server is a gateway into your Keep storage. Unlike the Keepstore servers, which are only accessible on the local LAN, Keepproxy is designed to provide secure access into Keep from anywhere on the internet.
By convention, we use the following hostname for the Keepproxy:
h2. Install Keepproxy
-First add the Arvados apt repository, and then install the Keepproxy package.
+On Debian-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install keepproxy</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install keepproxy</span>
+<pre><code>~$ <span class="userinput">sudo yum install keepproxy</span>
</code></pre>
</notextile>
</code></pre>
</notextile>
-It's recommended to run Keepproxy under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+It's recommended to run Keepproxy under "runit":http://smarden.org/runit/ or something similar.
h3. Create an API token for the Keepproxy server
title: Install Keepstore servers
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
We are going to install two Keepstore servers. By convention, we use the following hostname pattern:
<div class="offset1">
h2. Install Keepstore
-First add the Arvados apt repository, and then install the Keepstore package.
+On Debian-based systems:
<notextile>
-<pre><code>~$ <span class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</span>
-~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
-~$ <span class="userinput">sudo /usr/bin/apt-get install keepstore</span>
+<pre><code>~$ <span class="userinput">sudo apt-get install keepstore</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install keepstore</span>
</code></pre>
</notextile>
2015/05/08 13:44:26 keepstore starting, pid 2765
2015/05/08 13:44:26 Using volume [UnixVolume /mnt/keep] (writable=true)
2015/05/08 13:44:26 listening at :25107
-
</code></pre>
</notextile>
-It's recommended to run Keepstore under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+It's recommended to run Keepstore under "runit":http://smarden.org/runit/ or something similar.
Repeat this section for each Keepstore server you are setting up.
title: Install Ruby and bundler
...
-We recommend Ruby >= 2.1.
+We recommend Ruby >= 2.1. We currently require Bundler 1.9 (not 1.10 or above) because of a limitation in some of our gem dependencies.
h2(#rvm). Option 1: Install with rvm
<notextile>
<pre><code>~$ <span class="userinput">gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3</span>
~$ <span class="userinput">\curl -sSL https://get.rvm.io | bash -s stable --ruby=2.1</span>
-~$ <span class="userinput">gem install bundler
+~$ <span class="userinput">gem install -v 1.9.9 bundler
</span></code></pre></notextile>
h2(#fromsource). Option 2: Install from source
make
sudo make install
-sudo gem install bundler</span>
+sudo gem install -v 1.9.9 bundler</span>
</code></pre></notextile>
The number of Keepstore, shell and compute nodes listed above is a minimum. In a real production installation, you will likely run many more of each of those types of nodes. In such a scenario, you would probably also want to dedicate a node to the Workbench server and Crunch dispatcher, respectively. For performance reasons, you may want to run the database server on a separate node as well.
+h2(#repos). Arvados package repositories
+
+On any host where you install Arvados software, you'll need to set up an Arvados package repository. They're available for several popular distributions.
+
+h3. CentOS
+
+Packages are available for CentOS 6. First, register the Curoverse signing key in RPM's database:
+
+{% include 'install_redhat_key' %}
+
+Then save this configuration block in @/etc/yum.repos.d/arvados.repo@:
+
+<notextile>
+<pre><code>[arvados]
+name=Arvados
+baseurl=http://rpm.arvados.org/CentOS/$releasever/os/$basearch/
+</code></pre>
+</notextile>
+
+h3. Debian
+
+Packages are available for Debian 7 ("wheezy"). First, register the Curoverse signing key in apt's database:
+
+{% include 'install_debian_key' %}
+
+Then save the configuration line listed for your version of Debian in @/etc/apt/sources.list.d/arvados.list@:
+
+table(table table-bordered table-condensed).
+|*Debian version*|*@/etc/apt/sources.list.d/arvados.list@*|
+|7 ("wheezy")|@deb http://apt.arvados.org/ wheezy main@|
+
+Finally, add this new repository to apt's database:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-get update</span>
+</code></pre>
+</notextile>
+
+h3. Ubuntu
+
+Packages are available for Ubuntu 12.04 ("precise"). First, register the Curoverse signing key in apt's database:
+
+{% include 'install_debian_key' %}
+
+Then save the configuration line listed for your version of Ubuntu in @/etc/apt/sources.list.d/arvados.list@:
+
+table(table table-bordered table-condensed).
+|*Ubuntu version*|*@/etc/apt/sources.list.d/arvados.list@*|
+|12.04 ("precise")|@deb http://apt.arvados.org/ precise main@|
+
h2. A unique identifier
Each Arvados installation should have a globally unique identifier, which is a unique 5-character alphanumeric string. Here is a snippet of ruby that generates such a string based on the hostname of your computer:
Digest::MD5.hexdigest(`hostname`).to_i(16).to_s(36)[0..4]
</pre>
-You may also use a different method to pick the unique identifier. The unique identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @uuid_prefix@.
+You may also use a different method to pick the unique identifier. The unique identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @uuid_prefix@.
h2. SSL certificates
title: Install a shell server
...
-This installation guide assumes you are on a 64 bit Debian or Ubuntu system.
-
-There is nothing inherently special about an Arvados shell server. It is just a GNU/Linux machine with the Arvados SDKs installed. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster, but that is not required.
+There is nothing inherently special about an Arvados shell server. It is just a GNU/Linux machine with Arvados utilites and SDKs installed. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster, but that is not required.
h2. Install API tokens
Please follow the "API token guide":{{site.baseurl}}/user/reference/api-tokens.html to get API tokens for your user and install them on your shell server. We will use those tokens to test the SDKs as we install them.
-h2. Install the SDKs
+h2. Install the Ruby SDK and utilities
+
+If you're using RVM:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo rvm-exec gem install arvados-cli</span>
+</code></pre>
+</notextile>
+
+If you're not using RVM:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo gem install arvados-cli</span>
+</code></pre>
+</notextile>
+
+h2. Install the Python SDK and utilities
+
+On Debian-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-python-client python-arvados-fuse</span>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install python27-python-arvados-python-client python27-python-arvados-fuse</span>
+</code></pre>
+</notextile>
-Install the "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html and the "Command line SDK":{{site.baseurl}}/sdk/cli/install.html
+{% include 'note_python27_sc' %}
h3. Installation
+h4. Option 1: Install from distribution packages
+
+First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
+
+On Debian-based systems:
+
<notextile>
-<pre>
-$ <code class="userinput">sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl</code>
-$ <code class="userinput">git clone https://github.com/curoverse/arvados.git</code>
-$ <code class="userinput">cd arvados/sdk/perl</code>
-$ <code class="userinput">perl Makefile.PL</code>
-$ <code class="userinput">sudo make install</code>
-</pre>
+<pre><code>~$ <span class="userinput">sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl libarvados-perl</code>
+</code></pre>
+</notextile>
+
+On Red Hat-based systems:
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo yum install perl-ExtUtils-MakeMaker perl-JSON perl-IO-Socket-SSL perl-WWW-Curl libarvados-perl</code>
+</code></pre>
+</notextile>
+
+h4. Option 2: Install from source
+
+First, install dependencies from your distribution. Refer to the package lists above, but don't install @libarvados-perl@.
+
+Then run the following:
+
+<notextile>
+<pre><code>~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
+~$ <span class="userinput">cd arvados/sdk/perl</span>
+~$ <span class="userinput">perl Makefile.PL</span>
+~$ <span class="userinput">sudo make install</span>
+</code></pre>
</notextile>
-h4. Test installation
+h3. Test installation
If the SDK is installed, @perl -MArvados -e ''@ should produce no errors.
If your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ environment variables are set up correctly (see "api-tokens":{{site.baseurl}}/user/reference/api-tokens.html for details), the following test script should work:
<notextile>
-<pre>$ <code class="userinput">perl <<'EOF'
+<pre>~$ <code class="userinput">perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
If you are logged in to an Arvados VM, the Python SDK should be installed.
-To use the Python SDK elsewhere, you can either install the Python SDK via PyPI or build and install the package using the arvados source tree.
+To use the Python SDK elsewhere, you can install from a distribution package, PyPI, or source.
{% include 'notebox_begin' %}
-The Python SDK requires Python 2.7
+The Python SDK requires Python 2.7.
{% include 'notebox_end' %}
-h4. Option 1: install with PyPI
+h4. Option 1: Install from distribution packages
+
+First, "add the appropriate package repository for your distribution":{{ site.baseurl }}/install/install-manual-prerequisites.html#repos.
+
+On Debian-based systems:
<notextile>
-<pre>
-$ <code class="userinput">sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml</code>
-$ <code class="userinput">sudo pip install arvados-python-client</code>
-</pre>
+<pre><code>~$ <span class="userinput">sudo apt-get install python-arvados-python-client</code>
+</code></pre>
</notextile>
-_If your version of @pip@ is 1.4 or newer, the @pip install@ command might give an error: "Could not find a version that satisfies the requirement arvados-python-client". If this happens, fix it by adding a @--pre@ flag:_
+On Red Hat-based systems:
<notextile>
-<pre>
-$ <code class="userinput">sudo pip install --pre arvados-python-client</code>
-</pre>
+<pre><code>~$ <span class="userinput">sudo yum install python27-python-arvados-python-client</code>
+</code></pre>
</notextile>
-h4. Option 2: install from distribution packages (Debian/Ubuntu only)
+{% include 'note_python27_sc' %}
-First add @http://apt.arvados.org@ to your list of apt repositories:
+h4. Option 2: Install with pip
-<notextile>
-<pre>
-$ <code class="userinput">echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list</code>
-</pre>
-</notextile>
+Run @pip-2.7 install arvados-python-client@ in an appropriate installation environment, such as a virtualenv.
-Then install the package:
+If your version of @pip@ is 1.4 or newer, the @pip install@ command might give an error: "Could not find a version that satisfies the requirement arvados-python-client". If this happens, try @pip-2.7 install --pre arvados-python-client@.
-<notextile>
-<pre>
-$ <code class="userinput">sudo apt-get update</code>
-$ <code class="userinput">sudo apt-get install python-arvados-python-client</code>
-</pre>
-</notextile>
+h4. Option 3: Install from source
-h4. Option 3: build and install from source
+Install the @python-setuptools@ package from your distribution. Then run the following:
<notextile>
-<pre>
-~$ <code class="userinput">sudo apt-get install python-dev libattr1-dev libfuse-dev pkg-config</code>
-~$ <code class="userinput">git clone https://github.com/curoverse/arvados.git</code>
-~$ <code class="userinput">cd arvados/sdk/python</code>
-~/arvados/sdk/python$ <code class="userinput">sudo python setup.py install</code>
-</pre>
+<pre><code>~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
+~$ <span class="userinput">cd arvados/sdk/python</span>
+~$ <span class="userinput">python2.7 setup.py install</span>
+</code></pre>
</notextile>
+You may optionally run the final installation command in a virtualenv, or with the @--user@ option.
+
h4. Test installation
If the SDK is installed and your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ environment variables are set up correctly (see "api-tokens":{{site.baseurl}}/user/reference/api-tokens.html for details), @import arvados@ should produce no errors:
<notextile>
-<pre>$ <code class="userinput">python</code>
-Python 2.7.4 (default, Sep 26 2013, 03:20:26)
+<pre>~$ <code class="userinput">python2.7</code>
+Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> <code class="userinput">import arvados</code>
</notextile>
The SDK retrieves the list of API methods from the server at run time. Therefore, the set of available methods is determined by the server version rather than the SDK version.
-
projects / arvados.git / commitdiff