3 navsection: installguide
4 title: Install Keepstore servers
7 Copyright (C) The Arvados Authors. All rights reserved.
9 SPDX-License-Identifier: CC-BY-SA-3.0
12 Keepstore provides access to underlying storage for reading and writing content-addressed blocks, with enforcement of Arvados permissions. Keepstore supports a variety of cloud object storage and POSIX filesystems for its backing store.
14 We recommend installing at least two Keepstore servers. By convention, we use the following hostname pattern:
17 table(table table-bordered table-condensed).
19 |keep0.@uuid_prefix@.your.domain|
20 |keep1.@uuid_prefix@.your.domain|
23 Keepstore servers should not be directly accessible from the Internet (they are accessed via "keepproxy":install-keepproxy.html), so the hostnames only need to resolve on the private network.
27 On Debian-based systems:
30 <pre><code>~$ <span class="userinput">sudo apt-get install keepstore</span>
34 On Red Hat-based systems:
37 <pre><code>~$ <span class="userinput">sudo yum install keepstore</span>
41 Verify that Keepstore is functional:
44 <pre><code>~$ <span class="userinput">keepstore --version</span>
48 h3. Create config file
50 By default, keepstore will look for its configuration file at @/etc/arvados/keepstore/keepstore.yml@
52 You can override the configuration file location using the @-config@ command line option to keepstore.
54 The following is a sample configuration file:
57 # Duration for which new permission signatures (returned in PUT
58 # responses) will be valid. This should be equal to the API
59 # server's blob_signature_ttl configuration entry.
60 BlobSignatureTTL: 336h0m0s
62 # Local file containing the secret blob signing key (used to generate
63 # and verify blob signatures). The contents of the key file must be
64 # identical to the API server's blob_signing_key configuration entry.
65 BlobSigningKeyFile: ""
67 # Print extra debug logging
70 # Maximum number of concurrent block deletion operations (per
71 # volume) when emptying trash. Default is 1.
74 # Enable trash and delete features. If false, trash lists will be
75 # accepted but blocks will not be trashed or deleted.
76 # Keepstore does not delete data on its own. The keep-balance
77 # service determines which blocks are candidates for deletion
78 # and instructs the keepstore to move those blocks to the trash.
81 # Local port to listen on. Can be 'address:port' or ':port', where
82 # 'address' is a host IP address or name and 'port' is a port number
86 # Format of request/response and error logs: "json" or "text".
90 # Maximum RAM to use for data buffers, given in multiples of block
91 # size (64 MiB). When this limit is reached, HTTP requests requiring
92 # buffers (like GET and PUT) will wait for buffer space to be
95 # It should be set such that MaxBuffers * 64MiB + 10% fits
96 # comfortably in memory. On a host dedicated to running keepstore,
97 # divide total memory by 88MiB to suggest a suitable value. For example,
98 # if grep MemTotal /proc/meminfo reports MemTotal: 7125440 kB,
99 # compute 7125440 / (88 * 1024)=79 and configure MaxBuffers: 79
102 # Maximum concurrent requests. When this limit is reached, new
103 # requests will receive 503 responses. Note: this limit does not
104 # include idle connections from clients using HTTP keepalive, so it
105 # does not strictly limit the number of concurrent connections. If
106 # omitted or zero, the default is 2 * MaxBuffers.
109 # Path to write PID file during startup. This file is kept open and
110 # locked with LOCK_EX until keepstore exits, so "fuser -k pidfile" is
111 # one way to shut down. Exit immediately if there is an error
112 # opening, locking, or writing the PID file.
115 # Maximum number of concurrent pull operations. Default is 1, i.e.,
116 # pull lists are processed serially.
119 # Honor read requests only if a valid signature is provided. This
120 # should be true, except for development use and when migrating from
121 # a very old version.
122 RequireSignatures: true
124 # Local file containing the Arvados API token used by keep-balance
125 # or data manager. Delete, trash, and index requests are honored
126 # only for this token.
127 SystemAuthTokenFile: ""
129 # Path to server certificate file in X509 format. Enables TLS mode.
131 # Example: /var/lib/acme/live/keep0.example.com/fullchain
132 TLSCertificateFile: ""
134 # Path to server key file in X509 format. Enables TLS mode.
136 # The key pair is read from disk during startup, and whenever SIGHUP
139 # Example: /var/lib/acme/live/keep0.example.com/privkey
142 # How often to check for (and delete) trashed blocks whose
143 # TrashLifetime has expired.
144 TrashCheckInterval: 24h0m0s
146 # Time duration after a block is trashed during which it can be
147 # recovered using an /untrash request.
148 TrashLifetime: 336h0m0s
150 # Maximum number of concurrent trash operations. Default is 1, i.e.,
151 # trash lists are processed serially.
155 h3. Notes on storage management
157 On its own, a keepstore server never deletes data. The "keep-balance":install-keep-balance.html service service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash.
159 When a block is newly written, it is protected from deletion for the duration in @BlobSignatureTTL@. During this time, it cannot be trashed.
161 If keep-balance instructs keepstore to trash a block which is older than @BlobSignatureTTL@, and @EnableDelete@ is true, the block will be moved to "trash".
163 h3. Configure storage volumes
165 Available storage volume types include cloud object storage and POSIX filesystems.
167 If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions "S3 Object Storage":configure-s3-object-storage.html page instead and then "Run keepstore as a supervised service.":#keepstoreservice
169 If you are using Azure Blob Storage, follow the setup instructions "Azure Blob Storage":configure-azure-blob-storage.html and then proceed to "Run keepstore as a supervised service.":#keepstoreservice
171 To use a POSIX filesystem, including both local filesystems (ext4, xfs) and network file system such as GPFS or Lustre, continue reading this section.
173 h4. Setting up filesystem mounts
175 Volumes are configured in the @Volumes@ section of the configuration
176 file. You may provide multiple volumes for a single keepstore process
177 to manage multiple disks. Keepstore distributes blocks among volumes
178 in round-robin fashion.
182 - # The volume type, indicates this is a filesystem directory.
185 # The actual directory that will be used as the backing store.
186 Root: /mnt/local-disk
188 # How much replication is performed by the underlying filesystem.
189 # (for example, a network filesystem may provide its own replication).
190 # This is used to inform replication decisions at the Keep layer.
191 DirectoryReplication: 1
193 # If true, do not accept write or trash operations, only reads.
196 # When true, read and write operations (for whole 64MiB blocks) on
197 # an individual volume will queued and issued sequentially. When
198 # false, read and write operations will be issued concurrently as
201 # When using spinning disks where storage partitions map 1:1 to
202 # physical disks that are dedicated to Keepstore, enabling this may
203 # reduce contention and improve throughput by minimizing seeks.
205 # When using SSDs, RAID, or a parallel network filesystem, you probably
209 # Storage classes to associate with this volume. See "Configuring
210 # storage classes" in the "Admin" section of doc.arvados.org.
213 # Example of a second volume section
214 - DirectoryReplication: 2
216 Root: /mnt/network-disk
222 h3(#keepstoreservice). Run keepstore as a supervised service
224 Install runit to supervise the keepstore daemon. {% include 'install_runit' %}
226 Install this script as the run script @/etc/sv/keepstore/run@ for the keepstore service:
232 GOGC=10 exec keepstore -config /etc/arvados/keepstore/keepstore.yml
236 h3. Set up additional servers
238 Repeat the above sections to prepare volumes and bring up supervised services on each Keepstore server you are setting up.
240 h3. Tell the API server about the Keepstore servers
242 The API server needs to be informed about the presence of your Keepstore servers. For each of the Keepstore servers you have created, please execute the following commands on your <strong>shell server</strong>.
244 Make sure to update the @service_host@ value to match each of your Keepstore servers.
247 <pre><code>~$ <span class="userinput">prefix=`arv --format=uuid user current | cut -d- -f1`</span>
248 ~$ <span class="userinput">echo "Site prefix is '$prefix'"</span>
249 ~$ <span class="userinput">read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice"</span>
250 <span class="userinput">{
251 "service_host":"<strong>keep0.$prefix.your.domain</strong>",
252 "service_port":25107,
253 "service_ssl_flag":false,
254 "service_type":"disk"
257 </code></pre></notextile>