1 // A Volume is an interface representing a Keep back-end storage unit:
2 // for example, a single mounted disk, a RAID array, an Amazon S3 volume,
13 type Volume interface {
14 // Get a block. IFF the returned error is nil, the caller must
15 // put the returned slice back into the buffer pool when it's
18 // loc is guaranteed to consist of 32 or more lowercase hex
21 // Get should not verify the integrity of the returned data:
22 // it should just return whatever was found in its backing
23 // store. (Integrity checking is the caller's responsibility.)
25 // If an error is encountered that prevents it from
26 // retrieving the data, that error should be returned so the
27 // caller can log (and send to the client) a more useful
30 // If the error is "not found", and there's no particular
31 // reason to expect the block to be found (other than that a
32 // caller is asking for it), the returned error should satisfy
33 // os.IsNotExist(err): this is a normal condition and will not
34 // be logged as an error (except that a 404 will appear in the
35 // access log if the block is not found on any other volumes
38 // If the data in the backing store is bigger than BLOCKSIZE,
39 // Get is permitted to return an error without reading any of
41 Get(loc string) ([]byte, error)
43 // Put writes a block to an underlying storage device.
45 // loc is as described in Get.
47 // len(block) is guaranteed to be between 0 and BLOCKSIZE.
49 // If a block is already stored under the same name (loc) with
50 // different content, Put must either overwrite the existing
51 // data with the new data or return a non-nil error.
53 // Put also sets the timestamp for the given locator to the
56 // Put must return a non-nil error unless it can guarantee
57 // that the entire block has been written and flushed to
58 // persistent storage, and that its timestamp is current. Of
59 // course, this guarantee is only as good as the underlying
60 // storage device, but it is Put's responsibility to at least
61 // get whatever guarantee is offered by the storage device.
63 // Put should not verify that loc==hash(block): this is the
64 // caller's responsibility.
65 Put(loc string, block []byte) error
67 // Touch sets the timestamp for the given locator to the
70 // loc is as described in Get.
72 // If invoked at time t0, Touch must guarantee that a
73 // subsequent call to Mtime will return a timestamp no older
74 // than {t0 minus one second}. For example, if Touch is called
75 // at 2015-07-07T01:23:45.67890123Z, it is acceptable for a
76 // subsequent Mtime to return any of the following:
78 // - 2015-07-07T01:23:45.00000000Z
79 // - 2015-07-07T01:23:45.67890123Z
80 // - 2015-07-07T01:23:46.67890123Z
81 // - 2015-07-08T00:00:00.00000000Z
83 // It is not acceptable for a subsequente Mtime to return
84 // either of the following:
86 // - 2015-07-07T00:00:00.00000000Z -- ERROR
87 // - 2015-07-07T01:23:44.00000000Z -- ERROR
89 // Touch must return a non-nil error if the timestamp cannot
91 Touch(loc string) error
93 // Mtime returns the stored timestamp for the given locator.
95 // loc is as described in Get.
97 // Mtime must return a non-nil error if the given block is not
98 // found or the timestamp could not be retrieved.
99 Mtime(loc string) (time.Time, error)
101 // IndexTo writes a complete list of locators with the given
102 // prefix for which Get() can retrieve data.
104 // prefix consists of zero or more lowercase hexadecimal
107 // Each locator must be written to the given writer using the
110 // loc "+" size " " timestamp "\n"
114 // - size is the number of bytes of content, given as a
115 // decimal number with one or more digits
117 // - timestamp is the timestamp stored for the locator,
118 // given as a decimal number of seconds after January 1,
121 // IndexTo must not write any other data to writer: for
122 // example, it must not write any blank lines.
124 // If an error makes it impossible to provide a complete
125 // index, IndexTo must return a non-nil error. It is
126 // acceptable to return a non-nil error after writing a
127 // partial index to writer.
129 // The resulting index is not expected to be sorted in any
131 IndexTo(prefix string, writer io.Writer) error
133 // Delete deletes the block data from the underlying storage
136 // loc is as described in Get.
138 // If the timestamp for the given locator is newer than
139 // blob_signature_ttl, Delete must not delete the data.
141 // If a Delete operation overlaps with any Touch or Put
142 // operations on the same locator, the implementation must
143 // ensure one of the following outcomes:
145 // - Touch and Put return a non-nil error, or
146 // - Delete does not delete the block, or
147 // - Both of the above.
149 // If it is possible for the storage device to be accessed by
150 // a different process or host, the synchronization mechanism
151 // should also guard against races with other processes and
152 // hosts. If such a mechanism is not available, there must be
153 // a mechanism for detecting unsafe configurations, alerting
154 // the operator, and aborting or falling back to a read-only
155 // state. In other words, running multiple keepstore processes
156 // with the same underlying storage device must either work
157 // reliably or fail outright.
159 // Corollary: A successful Touch or Put guarantees a block
160 // will not be deleted for at least blob_signature_ttl
162 Delete(loc string) error
164 // Status returns a *VolumeStatus representing the current
165 // in-use and available storage capacity and an
166 // implementation-specific volume identifier (e.g., "mount
167 // point" for a UnixVolume).
168 Status() *VolumeStatus
170 // String returns an identifying label for this volume,
171 // suitable for including in log messages. It should contain
172 // enough information to uniquely identify the underlying
173 // storage device, but should not contain any credentials or
177 // Writable returns false if all future Put, Mtime, and Delete
178 // calls are expected to fail.
180 // If the volume is only temporarily unwritable -- or if Put
181 // will fail because it is full, but Mtime or Delete can
182 // succeed -- then Writable should return false.
186 // A VolumeManager tells callers which volumes can read, which volumes
187 // can write, and on which volume the next write should be attempted.
188 type VolumeManager interface {
189 // AllReadable returns all volumes.
190 AllReadable() []Volume
192 // AllWritable returns all volumes that aren't known to be in
193 // a read-only state. (There is no guarantee that a write to
194 // one will succeed, though.)
195 AllWritable() []Volume
197 // NextWritable returns the volume where the next new block
198 // should be written. A VolumeManager can select a volume in
199 // order to distribute activity across spindles, fill up disks
200 // with more free space, etc.
201 NextWritable() Volume
203 // Close shuts down the volume manager cleanly.
207 // RRVolumeManager is a round-robin VolumeManager: the Nth call to
208 // NextWritable returns the (N % len(writables))th writable Volume
209 // (where writables are all Volumes v where v.Writable()==true).
210 type RRVolumeManager struct {
216 func MakeRRVolumeManager(volumes []Volume) *RRVolumeManager {
217 vm := &RRVolumeManager{}
218 for _, v := range volumes {
219 vm.readables = append(vm.readables, v)
221 vm.writables = append(vm.writables, v)
227 func (vm *RRVolumeManager) AllReadable() []Volume {
231 func (vm *RRVolumeManager) AllWritable() []Volume {
235 func (vm *RRVolumeManager) NextWritable() Volume {
236 if len(vm.writables) == 0 {
239 i := atomic.AddUint32(&vm.counter, 1)
240 return vm.writables[i%uint32(len(vm.writables))]
243 func (vm *RRVolumeManager) Close() {