X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/682dd5b6cc23a455766a7651e3e841257660b31c..e8e78685ed7d893433e1ebe799a66084e39a0345:/doc/install/install-keepstore.html.textile.liquid diff --git a/doc/install/install-keepstore.html.textile.liquid b/doc/install/install-keepstore.html.textile.liquid index eb537836f5..13dfaf6725 100644 --- a/doc/install/install-keepstore.html.textile.liquid +++ b/doc/install/install-keepstore.html.textile.liquid @@ -4,8 +4,6 @@ navsection: installguide title: Install Keepstore servers ... -This installation guide assumes you are on a 64 bit Debian or Ubuntu system. - We are going to install two Keepstore servers. By convention, we use the following hostname pattern:
@@ -19,13 +17,17 @@ Because the Keepstore servers are not directly accessible from the internet, the h2. Install Keepstore -First add the Arvados apt repository, and then install the Keepstore package. +On Debian-based systems: -
~$ echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
-~$ sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
-~$ sudo /usr/bin/apt-get update
-~$ sudo /usr/bin/apt-get install keepstore
+
~$ sudo apt-get install keepstore
+

+
+
+On Red Hat-based systems:
+
+
+
~$ sudo yum install keepstore
 

 
 
@@ -35,6 +37,10 @@ Verify that Keepstore is functional:
 
~$ keepstore -h
 2015/05/08 13:41:16 keepstore starting, pid 2565
 Usage of ./keepstore:
+  -azure-storage-account-key-file="": File containing the account key used for subsequent --azure-storage-container-volume arguments.
+  -azure-storage-account-name="": Azure storage account name used for subsequent --azure-storage-container-volume arguments.
+  -azure-storage-container-volume=[]: Use the given container as a storage volume. Can be given multiple times.
+  -azure-storage-replication=3: Replication level to report to clients when data is stored in an Azure container.
   -blob-signature-ttl=1209600: Lifetime of blob permission signatures. See services/api/config/application.default.yml.
   -blob-signing-key-file="": File containing the secret key for generating and verifying blob permission signatures.
   -data-manager-token-file="": File with the API token used by the Data Manager. All DELETE requests or GET /index requests must carry this token.
@@ -52,24 +58,71 @@ Usage of ./keepstore:
 

 
 
-If you want access control on your Keepstore server(s), you must specify the @-enforce-permissions@ flag and provide a signing key. The @-blob-signing-key-file@ argument should be a file containing a long random alphanumeric string with no internal line breaks (it is also possible to use a socket or FIFO: keepstore reads it only once, at startup). This key must be the same as the @blob_signing_key@ configured in the "API server":install-api-server.html config/application.yml file.
+h3. Prepare storage volumes
+
+{% include 'notebox_begin' %}
+This section uses a local filesystem as a backing store. If you are using Azure Storage, follow the setup instructions on the "Azure Blob Storage":configure-azure-blob-storage.html page instead.
+{% include 'notebox_end' %}
 
-The @-max-buffers@ argument can be used to restrict keepstore's memory use. By default, keepstore will allocate no more than 128 blocks (8 GiB) worth of data buffers at a time. Normally this should be set as high as possible without risking swapping.
+There are two ways to specify a set of local directories where keepstore should store its data files.
+# Implicitly, by creating a directory called @keep@ at the top level of each filesystem you intend to use, and omitting @-volume@ arguments.
+# Explicitly, by providing a @-volume@ argument for each directory.
+
+For example, if there are filesystems mounted at @/mnt@ and @/mnt2@:
+
+
+
~$ mkdir /mnt/keep /mnt2/keep
+~$ keepstore
+2015/05/08 13:44:26 keepstore starting, pid 2765
+2015/05/08 13:44:26 Using volume [UnixVolume /mnt/keep] (writable=true)
+2015/05/08 13:44:26 Using volume [UnixVolume /mnt2/keep] (writable=true)
+2015/05/08 13:44:26 listening at :25107
+

+
 
-Prepare one or more volumes for Keepstore to use. Simply create a /keep directory on all the partitions you would like Keepstore to use, and then start Keepstore. For example, using 2 tmpfs volumes:
+Equivalently:
 
 
-
~$ keepstore -blob-signing-key-file=./blob-signing-key
+
~$ mkdir /mnt/keep /mnt2/keep
+~$ keepstore -volume=/mnt/keep -volume=/mnt2/keep
 2015/05/08 13:44:26 keepstore starting, pid 2765
 2015/05/08 13:44:26 Using volume [UnixVolume /mnt/keep] (writable=true)
+2015/05/08 13:44:26 Using volume [UnixVolume /mnt2/keep] (writable=true)
 2015/05/08 13:44:26 listening at :25107
+

+
+
+h3. Run keepstore as a supervised service
 
+Install runit to supervise the keepstore daemon.  {% include 'install_runit' %}
+
+Install this script as the run script for the keepstore service, modifying it as directed below.
+
+
+
#!/bin/sh
+
+exec 2>&1
+exec GOGC=10 GOMAXPROCS=4 keepstore \
+ -enforce-permissions=true \
+ -blob-signing-key-file=/etc/keepstore/blob-signing.key \
+ -max-buffers=100 \
+ -serialize=true \
+ -volume=/mnt/keep \
+ -volume=/mnt2/keep
 

 
 
-It's recommended to run Keepstore under "runit":https://packages.debian.org/search?keywords=runit or something similar.
+The @GOMAXPROCS@ environment variable determines the maximum number of concurrent threads, and should normally be set to the number of CPU cores present.
 
-Repeat this section for each Keepstore server you are setting up.
+The @-max-buffers@ argument limits keepstore's memory usage. It should be set such that @max-buffers * 64MiB + 10%@ fits comfortably in memory. For example, @-max-buffers=100@ is suitable for a host with 8 GiB RAM.
+
+If you want access control on your Keepstore server(s), you must specify the @-enforce-permissions@ flag and provide a signing key. The @-blob-signing-key-file@ argument should be a file containing a long random alphanumeric string with no internal line breaks (it is also possible to use a socket or FIFO: keepstore reads it only once, at startup). This key must be the same as the @blob_signing_key@ configured in the "API server's":install-api-server.html configuration file, @/etc/arvados/api/application.yml@.
+
+The @-serialize=true@ (default: @false@) argument limits keepstore to one reader/writer process per storage partition. This avoids thrashing by allowing the storage device underneath the storage partition to do read/write operations sequentially. Enabling @-serialize@ can improve Keepstore performance if the storage partitions map 1:1 to physical disks that are dedicated to Keepstore, particularly so for mechanical disks. In some cloud environments, enabling @-serialize@ has also also proven to be beneficial for performance, but YMMV. If your storage partition(s) are backed by network or RAID storage that can handle many simultaneous reader/writer processes without thrashing, you probably do not want to set @-serialize@.
+
+h3. Set up additional servers
+
+Repeat the above sections to prepare volumes and bring up supervised services on each Keepstore server you are setting up.
 
 h3. Tell the API server about the Keepstore servers
 
@@ -82,13 +135,10 @@ Make sure to update the @service_host@ value to match each of your Keepstore ser
 ~$ echo "Site prefix is '$prefix'"
 ~$ read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice"
 {
- "service_host":"keep0.$prefix.your.domain",
+ "service_host":"keep0.$prefix.your.domain",
  "service_port":25107,
  "service_ssl_flag":false,
  "service_type":"disk"
 }
 EOF
 

-
-
-