10 Keep is a content-addressable storage system. Its semantics are
11 inherently different from the POSIX-like file systems you're used to.
13 Using Keep looks like this:
17 3. Use locator to retrieve data.
18 4. Tag the locator with a symbolic name.
20 By contrast, using POSIX looks like this:
22 1. Choose locator (*i.e.*, filename).
23 2. Write data to locator.
24 3. Use locator to retrieve data.
26 Content addressing provides various benefits, including:
28 * Reduction of unnecessary data duplication
29 * Prevention of race conditions (a given locator always references the same data)
30 * Systematic client- and server-side verification of data integrity
31 * Provenance reporting (when combined with Arvados MapReduce jobs)
35 Keep arranges data into **collections** and **data blocks**.
37 A collection is analogous to a directory tree in a POSIX
38 filesystem. It contains subdirectories and filenames, and indicates
39 where to find the data blocks which comprise the files. It is encoded
42 A data block contains between 1 byte and 64 MiB of data. Its locator
43 is the MD5 checksum of the data, followed by a plus sign and its size
44 in bytes (encoded as a decimal number).
46 `acbd18db4cc2f85cedef654fccc4a4d8+3`
48 Keep distributes data blocks among the available disks. It also stores
49 multiple copies of each block, so a single disk or node failure does
50 not cause any data to become unreachable.
54 One of the side effects of the Keep write semantics is the lack of a
55 "delete" operation. Instead, Keep relies on garbage collection to
56 delete unneeded data blocks.
58 ### Tagging valuable data
60 Valuable data must be marked explicitly by creating a Collection in
61 Arvados. Otherwise, the data blocks will be deleted during garbage
64 Use the arv(1) program to create a collection. For example:
66 arv collections create --uuid "acbd18db4cc2f85cedef654fccc4a4d8+3"
70 Write three bytes of data to Keep.
76 acbd18db4cc2f85cedef654fccc4a4d8+3
80 whget acbd18db4cc2f85cedef654fccc4a4d8+3
87 {% include alert-stub.html %}
89 ### Writing a collection
91 ### Reading a file from a collection
93 ### Adding a collection to Arvados
95 ### Tagging a collection
97 ### Mounting Keep as a read-only POSIX filesystem
99 ### Mounting a single collection as a POSIX filesystem