SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+
+
<notextile>
-<pre><code>~$ <span class="userinput">sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install gnupg</span>
+# <span class="userinput">/usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7</span>
</code></pre>
</notextile>
<ol class=>
<li>Start a shell for the postgres user:
-<notextile><pre>~$ <span class="userinput">sudo -u postgres bash</span></pre></notextile>
+<notextile><pre># <span class="userinput">su postgres</span></pre></notextile>
</li>
<li>Generate a new database password:
-<notextile><pre>$ <span class="userinput"><span class="userinput">tr -dc 0-9a-zA-Z </dev/urandom | head -c25; echo</span>
+<notextile><pre>postgres$ <span class="userinput"><span class="userinput">tr -dc 0-9a-zA-Z </dev/urandom | head -c25; echo</span>
yourgeneratedpassword
</pre></notextile> Record this. You'll need it when you set up the Rails server later.
</li>
<li>Create a database user with the password you generated:
- <notextile><pre><code>$ <span class="userinput">createuser --encrypted -R -S --pwprompt {{service_role}}</span>
+ <notextile><pre><code>postgres$ <span class="userinput">createuser --encrypted --no-createrole --no-superuser --pwprompt {{service_role}}</span>
Enter password for new role: <span class="userinput">yourgeneratedpassword</span>
Enter it again: <span class="userinput">yourgeneratedpassword</span></code></pre></notextile>
</li>
<li>Create a database owned by the new user:
- <notextile><pre><code>$ <span class="userinput">createdb {{service_database}} -T template0 -E UTF8 -O {{service_role}}</span></code></pre></notextile>
+ <notextile><pre><code>postgres$ <span class="userinput">createdb {{service_database}} -T template0 -E UTF8 -O {{service_role}}</span></code></pre></notextile>
</li>
{% if use_contrib %}
<li>Enable the pg_trgm extension
- <notextile><pre>$ <span class="userinput">psql {{service_database}} -c "CREATE EXTENSION IF NOT EXISTS pg_trgm"</span></pre></notextile>
+ <notextile><pre>postgres$ <span class="userinput">psql {{service_database}} -c "CREATE EXTENSION IF NOT EXISTS pg_trgm"</span></pre></notextile>
</li>
{% endif %}
<li>Exit the postgres user shell:
- <notextile><pre>$ <span class="userinput">exit</span></pre></notextile>
+ <notextile><pre>postgres$ <span class="userinput">exit</span></pre></notextile>
</li>
</ol>
h3. Debian and Ubuntu
-<pre>
-apt-get --no-install-recommends install bundler
-</pre>
+<notextile>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install ruby-dev bundler</span></code></pre>
+</notextile>
h2(#rvm). Option 2: Install with RVM
h3. Install RVM
<notextile>
-<pre><code># <span class="userinput">gpg --keyserver keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
+<pre><code># <span class="userinput">gpg --keyserver pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
\curl -sSL https://get.rvm.io | bash -s stable --ruby=2.5
</span></code></pre></notextile>
h3(#empty). Create empty configuration file
-<pre>
-export ClusterID=xxxxx
-mkdir -p /etc/arvados
-cat > /etc/arvados/config.yml <<EOF
+<notextile>
+<pre><code># <span class="userinput">export ClusterID=xxxxx</span>
+# <span class="userinput">mkdir -p /etc/arvados</span>
+# <span class="userinput">cat > /etc/arvados/config.yml <<EOF
Clusters:
${ClusterID}:
-EOF
-</pre>
+EOF</span></code></pre>
+</notextile>
h2. Nginx configuration
{% endcomment %}
# "Introduction":#introduction
+# "Install dependencies":#dependencies
# "Set up database":#database-setup
# "Update config.yml":#update-config
# "Update nginx configuration":#update-nginx
-# "Install packages":#install-packages
+# "Install arvados-api-server and arvados-controller":#install-packages
+# "Confirm working installation":#confirm-working
h2(#introduction). Introduction
!(full-width){{site.baseurl}}/images/proxy-chain.svg!
-h2(#database-setup). Set up database
+h2(#dependencies). Install dependencies
+
+# "Install PostgreSQL":install-postgresql.html
+# "Install Ruby and Bundler":ruby.html
+# "Install nginx":nginx.html
+# "Install Phusion Passenger":https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/install_passenger_main.html
-"On the node that will host the database, install PostgreSQL":install-postgresql.html .
+h2(#database-setup). Set up database
{% assign service_role = "arvados" %}
{% assign service_database = "arvados_production" %}
ManagementToken: <span class="userinput">"$management_token"</span>
API:
RailsSessionSecretToken: <span class="userinput">"$rails_secret_token"</span>
+ Collections:
+ BlobSigningKey: <span class="userinput">"blob_signing_key"</span>
</code></pre>
</notextile>
@API.RailsSessionSecretToken@ is required by the API server.
+@Collections.BlobSigningKey@ is used to control access to Keep blocks.
+
You can generate a random token for each of these items at the command line like this:
<notextile>
h2(#update-nginx). Update nginx configuration
-Create a new file @/etc/nginx/conf.d/arvados-api-and-controller.conf@ . Configuration options that need attention are marked with "TODO".
+Use a text editor to create a new file @/etc/nginx/conf.d/arvados-api-and-controller.conf@ with the following configuration. Options that need attention are marked with "TODO".
<notextile>
-<pre><code>
-proxy_http_version 1.1;
+<pre><code>proxy_http_version 1.1;
# When Keep clients request a list of Keep services from the API
# server, use the origin IP address to determine if the request came
index index.html index.htm index.php;
passenger_enabled on;
- # If you're using RVM, uncomment the line below.
+
+ # TODO: If you are using RVM, uncomment the line below.
+ # If you're using system ruby, leave it commented out.
#passenger_ruby /usr/local/rvm/wrappers/default/ruby;
# This value effectively limits the size of API objects users can
</code></pre>
</notextile>
-h2(#install-packages). Install packages
-
-Step 1: "Install Ruby and Bundler":ruby.html
+h2(#install-packages). Install arvados-api-server and arvados-controller
-Step 2: "Install Nginx and Phusion Passenger":https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/install_passenger_main.html.
-
-Step 3: Install arvados-api-server and arvados-controller
-
-h3. Debian and Ubuntu
+h3. Centos 7
<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install bison build-essential libcurl4-openssl-dev git arvados-api-server arvados-controller</span>
+<pre><code># <span class="userinput">yum install bison make automake gcc gcc-c++ libcurl-devel git arvados-api-server arvados-controller</span>
</code></pre>
</notextile>
-h3. Centos 7
+h3. Debian and Ubuntu
<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install bison make automake gcc gcc-c++ libcurl-devel git arvados-api-server arvados-controller</span>
+<pre><code># <span class="userinput">apt-get --no-install-recommends install bison build-essential libcurl4-openssl-dev git zlib1g-dev arvados-api-server arvados-controller</span>
</code></pre>
</notextile>
-h2. Confirm working installation
+h2(#confirm-working). Confirm working installation
Confirm working controller:
$ curl -H "Authorization: Bearer $system_root_token" https://xxxxx.example.com/arvados/v1/users/current
</pre>
-h2. Troubleshooting
+h3. Troubleshooting
See the admin page on "Logging":{{site.baseurl}}/admin/logging.html .
Keepstore provides access to underlying storage for reading and writing content-addressed blocks, with enforcement of Arvados permissions. Keepstore supports a variety of cloud object storage and POSIX filesystems for its backing store.
-h3. Plan your storage layout
+h2. Plan your storage layout
In the steps below, you will configure a number of backend storage volumes (like local filesystems and S3 buckets) and specify which keepstore servers have read-only and read-write access to which volumes.
<div class="offset1">
table(table table-bordered table-condensed).
|_Hostname_|
-|keep0.@uuid_prefix@.your.domain|
-|keep1.@uuid_prefix@.your.domain|
+|keep0.@ClusterID@.example.com|
+|keep1.@ClusterID@.example.com|
</div>
Keepstore servers should not be directly accessible from the Internet (they are accessed via "keepproxy":install-keepproxy.html), so the hostnames only need to resolve on the private network.
-h2. Install Keepstore
+h2. Update cluster config
-On Debian-based systems:
+h3. List services
-<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get install keepstore</span>
-</code></pre>
-</notextile>
-
-On Red Hat-based systems:
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo yum install keepstore</span>
-</code></pre>
-</notextile>
-
-Verify that Keepstore is functional:
+Add each keepstore server to @/etc/arvados/config.yml@ .
<notextile>
-<pre><code>~$ <span class="userinput">keepstore --version</span>
-</code></pre>
-</notextile>
-
-h3. Create a superuser token
-
-If you haven't already done so, create a superuser token.
-
-{% include 'create_superuser_token' %}
-
-h3. Update cluster config file
-
-Add or update the following sections of @/etc/arvados/config.yml@ as needed. Refer to the examples and comments in the "default config.yml file":{{site.baseurl}}/admin/config.html for more information.
-
-<notextile>
-<pre><code>Clusters:
- <span class="userinput">uuid_prefix</span>:
- SystemRootToken: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
- Services:
+<pre><code> Services:
Keepstore:
+ # No ExternalURL because they are only accessed by the internal subnet.
InternalURLs:
- "http://<span class="userinput">keep0.uuid_prefix.example.com</span>:25107/": {}
- API:
- MaxKeepBlobBuffers: 128
+ "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107/": {}
+ "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107/": {}
+ # and so forth
</code></pre>
</notextile>
-h3. Note on storage management
-
-On its own, a keepstore server never deletes data. Instead, the keep-balance service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash. Please see the "Balancing Keep servers":{{site.baseurl}}/admin/keep-balance.html for more details.
-
h3. Configure storage volumes
Available storage volume types include POSIX filesystems and cloud object storage.
* If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions on "S3 Object Storage":configure-s3-object-storage.html
* If you are using Azure Blob Storage, follow the setup instructions on "Azure Blob Storage":configure-azure-blob-storage.html
-h2. Run keepstore as a supervised service
+h2. Install keepstore package
-h3. Start the service (option 1: systemd)
+On each host that will run keepstore, install the @keepstore@ package.
-If your system does not use systemd, skip this section and follow the "runit instructions":#runit instead.
-
-If your system uses systemd, the keepstore service should already be set up. Restart it to read the updated configuration, and check its status:
+h3. Centos 7
<notextile>
-<pre><code>~$ <span class="userinput">sudo systemctl restart keepstore</span>
-~$ <span class="userinput">sudo systemctl status keepstore</span>
-● keepstore.service - Arvados Keep Storage Daemon
- Loaded: loaded (/etc/systemd/system/keepstore.service; enabled; vendor preset: enabled)
- Active: active (running) since Tue 2019-09-10 14:16:29 UTC; 1s ago
- Docs: https://doc.arvados.org/
- Main PID: 25465 (keepstore)
- Tasks: 9 (limit: 4915)
- CGroup: /system.slice/keepstore.service
- └─25465 /usr/bin/keepstore
-[...]
+<pre><code># <span class="userinput">yum install keepstore</span>
</code></pre>
</notextile>
-h3(#runit). Start the service (option 2: runit)
-
-Install runit to supervise the keepstore daemon. {% include 'install_runit' %}
-
-Install this script as the run script @/etc/sv/keepstore/run@ for the keepstore service:
+h3. Debian and Ubuntu
<notextile>
-<pre><code>#!/bin/sh
-
-exec 2>&1
-GOGC=10 exec keepstore
+<pre><code># <span class="userinput">apt-get install keepstore</span>
</code></pre>
</notextile>
-h2. Set up additional servers
-
-Repeat the above sections to prepare volumes and bring up supervised services on each Keepstore server you are setting up.
-
h2. Restart the API server and controller
After adding all of your keepstore servers to the Services section, make sure the cluster config file is up to date on the API server host, and restart the API server and controller processes to ensure the changes are applied.
-<pre>
-sudo systemctl restart nginx arvados-controller
-</pre>
+<notextile>
+<pre><code># <span class="userinput">systemctl restart nginx arvados-controller</span>
+</code></pre>
+</notextile>
h2(#testing). Testing keep
$ arv-get 59389a8f9ee9d399be35462a0f92541c+53/hello.txt
hello world!
</pre>
+
+h3. Note on storage management
+
+On its own, a keepstore server never deletes data. Instead, the keep-balance service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash. Please see the "Balancing Keep servers":{{site.baseurl}}/admin/keep-balance.html for more details.
{% assign rh_version = "7" %}
{% include 'note_python_sc' %}
-# Install PostgreSQL:
- <notextile><pre>~$ <span class="userinput">sudo yum install rh-postgresql95 rh-postgresql95-postgresql-contrib</span>
+# Install PostgreSQL
+ <notextile><pre># <span class="userinput">yum install rh-postgresql95 rh-postgresql95-postgresql-contrib</span>
~$ <span class="userinput">scl enable rh-postgresql95 bash</span></pre></notextile>
-# Initialize the database:
- <notextile><pre>~$ <span class="userinput">sudo postgresql-setup initdb</span></pre></notextile>
-# Configure the database to accept password connections:
- <notextile><pre><code>~$ <span class="userinput">sudo sed -ri -e 's/^(host +all +all +(127\.0\.0\.1\/32|::1\/128) +)ident$/\1md5/' /var/lib/pgsql/data/pg_hba.conf</span></code></pre></notextile>
-# Configure the database to launch at boot:
- <notextile><pre>~$ <span class="userinput">sudo systemctl enable rh-postgresql95-postgresql</span></pre></notextile>
-# Start the database:
- <notextile><pre>~$ <span class="userinput">sudo systemctl start rh-postgresql95-postgresql</span></pre></notextile>
+# Initialize the database
+ <notextile><pre># <span class="userinput">postgresql-setup initdb</span></pre></notextile>
+# Configure the database to accept password connections
+ <notextile><pre><code># <span class="userinput">sed -ri -e 's/^(host +all +all +(127\.0\.0\.1\/32|::1\/128) +)ident$/\1md5/' /var/lib/pgsql/data/pg_hba.conf</span></code></pre></notextile>
+# Configure the database to launch at boot
+ <notextile><pre># <span class="userinput">systemctl enable rh-postgresql95-postgresql</span></pre></notextile>
+# Start the database
+ <notextile><pre># <span class="userinput">systemctl start rh-postgresql95-postgresql</span></pre></notextile>
h3(#debian). Debian or Ubuntu
Ubuntu 14.04 (Trusty) requires an updated PostgreSQL version, see "the PostgreSQL ubuntu repository":https://www.postgresql.org/download/linux/ubuntu/
-# Install PostgreSQL:
- <notextile><pre>~$ <span class="userinput">sudo apt-get install postgresql postgresql-contrib</span></pre></notextile>
+# Install PostgreSQL
+ <notextile><pre># <span class="userinput">apt-get --no-install-recommends install postgresql postgresql-contrib</span></pre></notextile>
+# Configure the database to launch at boot
+ <notextile><pre># <span class="userinput">systemctl enable postgresql</span></pre></notextile>
+# Start PostgreSQL
+ <notextile><pre># <span class="userinput">systemctl start postgresql</span></pre></notextile>
+{% comment %}
h2(#sso). Set up SSO server credentials and database
{% assign service_role = "arvados_sso" %}
{% assign service_database = "arvados_sso_production" %}
{% assign use_contrib = false %}
{% include 'install_postgres_database' %}
+
+{% endcomment %}
{% include 'install_debian_key' %}
-Configure apt to retrieve packages from the Arvados package repository. This command depends on your OS vendor and version:
+As root, add the Arvados package repository to your sources. This command depends on your OS vendor and version:
table(table table-bordered table-condensed).
|_. OS version|_. Command|
-|Debian 10 ("buster")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ buster main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Debian 9 ("stretch")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ stretch main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Ubuntu 18.04 ("bionic")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ bionic main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Ubuntu 16.04 ("xenial")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ xenial main" | sudo tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Debian 10 ("buster")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ buster main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Debian 9 ("stretch")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ stretch main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Ubuntu 18.04 ("bionic")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ bionic main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Ubuntu 16.04 ("xenial")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ xenial main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
{% include 'notebox_begin' %}
Retrieve the package list:
<notextile>
-<pre><code>~$ <span class="userinput">sudo apt-get update</span>
+<pre><code># <span class="userinput">apt-get update</span>
</code></pre>
</notextile>
projects / arvados.git / commitdiff