Keepstore can store data in object storage compatible with the S3 API, such as Amazon S3, Google Cloud Storage, or Ceph RADOS.
-h2. Configure keepstore
-
-Copy the "access key" and "secret key" to files where they will be accessible to keepstore at startup time.
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo sh -c 'cat >/etc/arvados/keepstore/aws_s3_access_key.txt <<EOF'
-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz==
-EOF</span>
-~$ <span class="userinput">sudo chmod 0400 /etc/arvados/keepstore/aws_s3_access_key.txt</span>
-</code></pre>
-</notextile>
-
-Next, edit the @Volumes@ section of the @keepstore.yml@ config file.
-
-h3. Example config for Amazon S3
-
-<pre>
-Volumes:
-- # The volume type, this indicates object storage compatible with the S3 API
- Type: S3
-
- # Storage provider. If blank, uses Amazon S3 by default.
- # See below for example alternate configuration for Google cloud
- # storage.
- Endpoint: ""
-
- # The bucket to use for the backing store.
- Bucket: example-bucket-name
-
- # The region where the bucket is located.
- Region: us-east-1
-
- # The credentials to use to access the bucket.
- AccessKeyFile: /etc/arvados/keepstore/aws_s3_access_key.txt
- SecretKeyFile: /etc/arvados/keepstore/aws_s3_secret_key.txt
-
- # Maximum time to wait making the initial connection to the backend before
- # failing the request.
- ConnectTimeout: 1m0s
-
- # Page size for s3 "list bucket contents" requests
- IndexPageSize: 1000
-
- # True if the region requires a LocationConstraint declaration
- LocationConstraint: false
-
- # Maximum eventual consistency latency
- RaceWindow: 24h0m0s
-
- # If true, do not accept write or trash operations, only reads.
- ReadOnly: false
-
- # Maximum time to wait for a complete response from the backend before
- # failing the request.
- ReadTimeout: 2m0s
-
- # How much replication is performed by the underlying bucket.
- # This is used to inform replication decisions at the Keep layer.
- S3Replication: 2
-
- # Storage classes to associate with this volume. See
- # "Storage classes" in the "Admin" section of doc.arvados.org.
- StorageClasses: null
-
- # Enable deletion (garbage collection) even when TrashLifetime is
- # zero. WARNING: eventual consistency may result in race conditions
- # that can cause data loss. Do not enable this unless you know what
- # you are doing.
- UnsafeDelete: false
-</pre>
-
-Start (or restart) keepstore, and check its log file to confirm it is using the new configuration.
-
-h3. Example config for Google cloud storage
-
-See previous section for documentation of configuration fields.
-
-<pre>
-Volumes:
-- # Example configuration using alternate storage provider
- # Configuration for Google cloud storage
- Endpoint: https://storage.googleapis.com
- Region: ""
-
- AccessKeyFile: /etc/arvados/keepstore/gce_s3_access_key.txt
- SecretKeyFile: /etc/arvados/keepstore/gce_s3_secret_key.txt
- Bucket: example-bucket-name
- ConnectTimeout: 1m0s
- IndexPageSize: 1000
- LocationConstraint: false
- RaceWindow: 24h0m0s
- ReadOnly: false
- ReadTimeout: 2m0s
- S3Replication: 2
- StorageClasses: null
- UnsafeDelete: false
-</pre>
-
-Start (or restart) keepstore, and check its log file to confirm it is using the new configuration.
+Volumes are configured in the @Volumes@ section of the cluster configuration file.
+
+{% include 'assign_volume_uuid' %}
+
+<notextile><pre><code> Volumes:
+ <span class="userinput">ClusterID</span>-nyw5e-<span class="userinput">000000000000000</span>:
+ AccessViaHosts:
+ # This section determines which keepstore servers access the
+ # volume. In this example, keep0 has read/write access, and
+ # keep1 has read-only access.
+ #
+ # If the AccessViaHosts section is empty or omitted, all
+ # keepstore servers will have read/write access to the
+ # volume.
+ "http://<span class="userinput">keep0.ClusterID.example.com</span>:25107": {}
+ "http://<span class="userinput">keep1.ClusterID.example.com</span>:25107": {ReadOnly: true}
+
+ Driver: <span class="userinput">S3</span>
+ DriverParameters:
+ # Bucket name.
+ Bucket: <span class="userinput">example-bucket-name</span>
+
+ # IAM role name to use when retrieving credentials from
+ # instance metadata. It can be omitted, in which case the
+ # role name itself will be retrieved from instance metadata
+ # -- but setting it explicitly may protect you from using
+ # the wrong credentials in the event of an
+ # installation/configuration error.
+ IAMRole: <span class="userinput">""</span>
+
+ # If you are not using an IAM role for authentication,
+ # specify access credentials here instead.
+ AccessKey: <span class="userinput">""</span>
+ SecretKey: <span class="userinput">""</span>
+
+ # Storage provider region. For Google Cloud Storage, use ""
+ # or omit.
+ Region: <span class="userinput">us-east-1a</span>
+
+ # Storage provider endpoint. For Amazon S3, use "" or
+ # omit. For Google Cloud Storage, use
+ # "https://storage.googleapis.com".
+ Endpoint: ""
+
+ # Change to true if the region requires a LocationConstraint
+ # declaration.
+ LocationConstraint: false
+
+ # Use V2 signatures instead of the default V4. Amazon S3
+ # supports V4 signatures in all regions, but this option
+ # might be needed for other S3-compatible services.
+ V2Signature: false
+
+ # Requested page size for "list bucket contents" requests.
+ IndexPageSize: 1000
+
+ # Maximum time to wait while making the initial connection
+ # to the backend before failing the request.
+ ConnectTimeout: 1m
+
+ # Maximum time to wait for a complete response from the
+ # backend before failing the request.
+ ReadTimeout: 2m
+
+ # Maximum eventual consistency latency
+ RaceWindow: 24h
+
+ # How much replication is provided by the underlying bucket.
+ # This is used to inform replication decisions at the Keep
+ # layer.
+ Replication: 2
+
+ # If true, do not accept write or trash operations, even if
+ # AccessViaHosts.*.ReadOnly is false.
+ #
+ # If false or omitted, enable write access (subject to
+ # AccessViaHosts.*.ReadOnly, where applicable).
+ ReadOnly: false
+
+ # Storage classes to associate with this volume. See "Storage
+ # classes" in the "Admin" section of doc.arvados.org.
+ StorageClasses: null
+</code></pre></notextile>
projects / arvados.git / blobdiff