---
layout: default
navsection: installguide
title: Install Keep-balance
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.

SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}

Keep-balance deletes unreferenced and overreplicated blocks from Keep servers, makes additional copies of underreplicated blocks, and moves blocks into optimal locations as needed (e.g., after adding new servers).

{% include 'notebox_begin' %}

If you are installing keep-balance on an existing system with valuable data, you can run keep-balance in "dry run" mode first and review its logs as a precaution. To do this, edit your keep-balance startup script to use the flags @-commit-pulls=false -commit-trash=false@.

{% include 'notebox_end' %}

h2. Install keep-balance

Keep-balance can be installed anywhere with network access to Keep services. Typically it runs on the same host as keepproxy.

*A cluster should have only one keep-balance process running at a time.*

On Debian-based systems:

<notextile>
<pre><code>~$ <span class="userinput">sudo apt-get install keep-balance</span>
</code></pre>
</notextile>

On Red Hat-based systems:

<notextile>
<pre><code>~$ <span class="userinput">sudo yum install keep-balance</span>
</code></pre>
</notextile>

Verify that @keep-balance@ is functional:

<notextile>
<pre><code>~$ <span class="userinput">keep-balance -h</span>
...
Usage: keep-balance [options]

Options:
  -commit-pulls
        send pull requests (make more replicas of blocks that are underreplicated or are not in optimal rendezvous probe order)
  -commit-trash
        send trash requests (delete unreferenced old blocks, and excess replicas of overreplicated blocks)
...
</code></pre>
</notextile>

h3. Create a keep-balance token

Create an Arvados superuser token for use by keep-balance. *On the API server*, run:

<notextile>
<pre><code>apiserver:~$ <span class="userinput">cd /var/www/arvados-api/current</span>
apiserver:/var/www/arvados-api/current$ <span class="userinput">sudo -u <b>webserver-user</b> RAILS_ENV=production bundle exec script/create_superuser_token.rb</span>
zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
</code></pre>
</notextile>

h3. Update keepstore configuration files

On each node that runs keepstore, save the token you generated in the previous step in a text file like @/etc/arvados/keepstore/system-auth-token.txt@ and then create or update @/etc/arvados/keepstore/keepstore.yml@ with the following key:

<notextile>
<pre><code>SystemAuthTokenFile: /etc/arvados/keepstore/system-auth-token.txt
</code></pre>
</notextile>

Restart all keepstore services to apply the updated configuration.

h3. Create a keep-balance configuration file

On the host running keep-balance, create @/etc/arvados/keep-balance/keep-balance.yml@ using the token you generated above.  Follow this YAML format:

<notextile>
<pre><code>Client:
  APIHost: <span class="userinput">uuid_prefix.your.domain</span>:443
  AuthToken: zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
KeepServiceTypes:
  - disk
RunPeriod: 10m
CollectionBatchSize: 100000
CollectionBuffers: 1000
</code></pre>
</notextile>

If your API server's SSL certificate is not signed by a recognized CA, add the @Insecure@ option to the @Client@ section:

<notextile>
<pre><code>Client:
  <span class="userinput">Insecure: true</span>
  APIHost: ...
</code></pre>
</notextile>

h3. Start the service (option 1: systemd)

If your system does not use systemd, skip this section and follow the "runit instructions":#runit instead.

If your system uses systemd, the keep-balance service should already be set up. Start it and check its status:

<notextile>
<pre><code>~$ <span class="userinput">sudo systemctl restart keep-balance</span>
~$ <span class="userinput">sudo systemctl status keep-balance</span>
&#x25cf; keep-balance.service - Arvados Keep Balance
   Loaded: loaded (/lib/systemd/system/keep-balance.service; enabled)
   Active: active (running) since Sat 2017-02-14 18:46:01 UTC; 3 days ago
     Docs: https://doc.arvados.org/
 Main PID: 541 (keep-balance)
   CGroup: /system.slice/keep-balance.service
           └─541 /usr/bin/keep-balance -commit-pulls -commit-trash

Feb 14 18:46:01 zzzzz.arvadosapi.com keep-balance[541]: 2017/02/14 18:46:01 starting up: will scan every 10m0s and on SIGUSR1
Feb 14 18:56:01 zzzzz.arvadosapi.com keep-balance[541]: 2017/02/14 18:56:01 Run: start
Feb 14 18:56:01 zzzzz.arvadosapi.com keep-balance[541]: 2017/02/14 18:56:01 skipping zzzzz-bi6l4-rbtrws2jxul6i4t with service type "proxy"
Feb 14 18:56:01 zzzzz.arvadosapi.com keep-balance[541]: 2017/02/14 18:56:01 clearing existing trash lists, in case the new rendezvous order differs from previous run
</code></pre>
</notextile>

h3(#runit). Start the service (option 2: runit)

Install runit to supervise the keep-balance daemon.  {% include 'install_runit' %}

Create a supervised service.

<notextile>
<pre><code>~$ <span class="userinput">sudo mkdir /etc/service/keep-balance</span>
~$ <span class="userinput">cd /etc/service/keep-balance</span>
~$ <span class="userinput">sudo mkdir log log/main</span>
~$ <span class="userinput">printf '#!/bin/sh\nexec keep-balance -commit-pulls -commit-trash 2>&1\n' | sudo tee run</span>
~$ <span class="userinput">printf '#!/bin/sh\nexec svlogd main\n' | sudo tee log/run</span>
~$ <span class="userinput">sudo chmod +x run log/run</span>
~$ <span class="userinput">sudo sv exit .</span>
~$ <span class="userinput">cd -</span>
</code></pre>
</notextile>

Use @sv stat@ and check the log file to verify the service is running.

<notextile>
<pre><code>~$ <span class="userinput">sudo sv stat /etc/service/keep-balance</span>
run: /etc/service/keep-balance: (pid 12520) 2s; run: log: (pid 12519) 2s
~$ <span class="userinput">tail /etc/service/keep-balance/log/main/current</span>
2017/02/14 18:46:01 starting up: will scan every 10m0s and on SIGUSR1
2017/02/14 18:56:01 Run: start
2017/02/14 18:56:01 skipping zzzzz-bi6l4-rbtrws2jxul6i4t with service type "proxy"
2017/02/14 18:56:01 clearing existing trash lists, in case the new rendezvous order differs from previous run
</code></pre>
</notextile>

h2. Enable delete operations on keepstore volumes

Ensure your keepstore services have the "delete" operation enabled. If it is disabled (which is the default), unneeded blocks will be identified by keep-balance, but will never be deleted from the underlying storage devices.

Add the @-never-delete=false@ command line flag to your keepstore run script:

<notextile>
<pre><code>keepstore <span class="userinput">-never-delete=false</span> -volume=...
</code></pre>
</notextile>

{% comment %}
// To replace the above section when the keepstore page recommends YAML...

Use the @EnableDelete@ flag in your YAML configuration file @/etc/arvados/keepstore/keepstore.yml@:

<notextile>
<pre><code>...
BlobSigningKeyFile: /etc/keepstore/blob-signing.key
<span class="userinput">EnableDelete: true</span>
Listen: :25107
...
</code></pre>
</notextile>
{% endcomment %}