|output_path|string|Path to a directory or file inside the container that should be preserved as container's output when it finishes. This path must be, or be inside, one of the mount targets. For best performance, point output_path to a writable collection mount.|Required.|
|priority|integer|Higher value means spend more resources on this container_request, i.e., go ahead of other queued containers, bring up more nodes etc.|Priority 0 means a container should not be run on behalf of this request. Clients are expected to submit ContainerRequests with zero priority in order to prevew the container that will be used to satisfy it. Priority can be null if and only if state!="Committed".|
|expires_at|datetime|After this time, priority is considered to be zero.|Not yet implemented.|
+|use_existing|boolean|If possible, use an existing (non-failed) container to satisfy the request instead of creating a new one.|Default is true|
+@true@
+@false@|
|filters|string|Additional constraints for satisfying the container_request, given in the same form as the filters parameter accepted by the container_requests.list API.||
h2(#mount_types). {% include 'mount_types' %}
* The oldest "Locked" container with the highest priority, i.e., the container that is most likely to start first
* The oldest "Queued" container with the highest priority, i.e, the container that is most likely to start first
-{% include 'notebox_begin' %}
-Currently, container reuse is the default behavior and a mechanism to skip reuse is not supported.
-{% include 'notebox_end' %}
-
h2(#cancel_container). Canceling a ContainerRequest
A ContainerRequest may be canceled by setting it's priority to 0, using an update call.
*GET /blocklocator+size+A@token*
-Fetch the data block, if the token is valid.
+Fetch the data block. Response returns block contents. If permission checking is enabled, requires a valid token hint.
*PUT /blocklocator*
-Returns the md5 sum of the data along with the signed token.
+Body: the block contents. Responds the block locator consisting of MD5 sum of the data, block size, and signed token hint.
+
+*POST /*
+
+Body: the block contents. Responds the block locator consisting of MD5 sum of the data, block size, and signed token hint.
h2. Rendezvous hashing
Each @keep_service@ resource has an assigned uuid. To determine priority assignments of blocks to servers, for each keep service compute the MD5 sum of the string concatenation of the block locator (hex-coded hash part only) and service uuid, then sort this list in descending order. Blocks are preferentially placed on servers at the beginning of the list.
-h2. Keep locator format
+h2(#locator). Keep locator format
BNF notation for a valid Keep locator string (with hints). For example @d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709@53bed294@
sign-timestamp ::= <8 lowercase hexadecimal digits>
</pre>
+h3. Token signatures
+
+A token signature (sign-hint) provides proof-of-access for a data block. It is computed by taking a SHA1 HMAC of the blob signing token (a shared secret between the API server and keep servers), block digest, current API token, expiration timestamp, and blob signature TTL.
+
+When communicating with the Keep store to fetch a block, or the API server to create or update a collection, the service computes the expected token signature for each block and compares it to the token signature that was presented by the client. Keep clients receive valid block signatures when uploading a block to a keep store (getting back a signed token as proof of knowledge) or, from the API server, getting the manifest text of a collection on which the user has read permission.
+
+Security of a token signature is derived from the following characteristics:
+
+# Valid signatures can only be generated by entities that know the shared secret (the "blob signing token")
+# A signature can only be used by an entity that also know the API token that was used to generate it.
+# It expires after a set date (the expiration time, based on the "blob signature time-to-live (TTL)")
+
h3. Regular expression to validate locator
<pre>
table(table table-bordered table-condensed).
|@d41d8cd98f00b204e9800998ecf8427e+0@|
|@d41d8cd98f00b204e9800998ecf8427e+0+Z@|
-|@d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709@53bed294@|
+|<code>d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709@53bed294</code>|
h3. Invalid locators
A manifest is utf-8 encoded text, consisting of zero or more newline-terminated streams.
-Each stream consists of three or more space-delimited tokens:
-* The first token is a stream name, consisting of one or more path components, delimited by @"/"@.
+<pre>
+manifest ::= stream*
+stream ::= stream-name (" " locator)+ (" " file-segment)+ "\n"
+stream-name ::= "." ("/" path-component)*
+path-component ::= <printable ASCII - (whitespace, "/")>+
+file-segment ::= position ":" size ":" filename
+position ::= [0-9]+
+size ::= [0-9]+
+filename ::= path-component ("/" path-component)*
+</pre>
+
+Notes:
+
+* The first token is the stream name, consisting of one or more path components, delimited by @"/"@.
** The first path component is always @"."@.
** No path component is empty.
-** No path component is "." or "..".
+** No path component following the first one can be "." or "..".
** The stream name never begins or ends with @"/"@.
-* The second token is a keep locator, described above
-* ...possibly followed by more data blob locators...
-* The first token that is not a block locator, and all subsequent tokens, are file tokens.
+* The next N tokens are "keep locators":#locator
+** These describe the "data stream". By logically concatenating the blocks in the order that they appear, we can refer to "positions" in the data stream.
+* File tokens come after the sequence of keep locators.
** A file token has three parts, delimited by @":"@: position, size, filename.
-** Position and size are given in decimal, and are counted from the beginning of the first data blob.
+** Position and size are given in decimal
+** The position is the position in the data stream
+** The size is the count of bytes following the position in the data stream. A file size may cross multiple blocks in the data stream.
** Filename may contain @"/"@ characters, but must not start or end with @"/"@, and must not contain @"//"@.
** Filename components (delimited by @"/"@) must not be @"."@ or @".."@.
+** There may be multiple file tokens.
-A manifest contains no TAB characters, nor other ASCII whitespace characters other than the spaces or newline delimiters specified above.
+It is legal to have multiple file tokens in the manifest (possible across different streams) with the same combined path name @stream name + "/" + filename@. This must be interpreted as a concatenation of file content, in the order that the file tokens appear in the manifest.
-A manifest always ends with a newline -- except the empty (zero-length) string, which is a valid manifest.
+Spaces are represented by the escape sequence @\040@. Spaces in stream names and filenames must be translated when reading and writing manifests. A manifest may not contain TAB characters, nor other ASCII whitespace characters or control codes other than the spaces or newlines used as delimiters specified above. A manifest always ends with a newline -- except the empty (zero-length) string, which is a valid manifest.
-h2. Example manifests
+h3. Normalized manifest v1
-A signed manifest with four files in two directories:
+A normalized manifest is a manifest that meets the following additional restrictions:
-<pre>
-. 930625b054ce894ac40596c3f5a0d947+33+A1f27a35dd9af37191d63ad8eb8985624451e7b79@5835c8bc 0:0:a 0:0:b 0:33:output.txt
-./c d41d8cd98f00b204e9800998ecf8427e+0+A27117dcd30c013a6e85d6d74c9a50179a1446efa@5835c8bc 0:0:d
-</pre>
+* Streams are in alphanumeric order.
+* Each stream name is unique within the manifest.
+* Files within a stream are listed in alphanumeric order.
+* Blocks within a stream are ordered based on order of file tokens of the stream. A given block is listed at most once in a stream.
+* Filename must not contain @"/"@ (the stream name represents the path prefix)
+
+h3. Example manifests
-The same manifest without block signatures:
+A manifest with four files in two directories:
<pre>
. 930625b054ce894ac40596c3f5a0d947+33 0:0:a 0:0:b 0:33:output.txt
./c d41d8cd98f00b204e9800998ecf8427e+0 0:0:d
</pre>
-A manifest containing a large file consisting of multiple blocks:
+The same manifest with permission signatures on each block:
<pre>
-. c449ed86671e4a34a8b8b9430850beba+67108864 74376cb0283a3d9cc1b613f624600745+67108864 cef3ba09db65559a671deff0a9f4edd9+67108864+A6e4832dd8d6df80cef080225a8d74e7a1bab2726@5835c92c f660cf00b5f23544f330572224600a99+67108864+A9de655fb05f387821503be7e6f1562dc3185167d@5835c92c 36331288ca1387eb7aa5684505d59eae+67108864+A4bf1bbdbf66fa43005712cb5bdcd88a7963a1ed8@5835c92c 7ce9fd55a8b39036bd1fa7b4645ef582+67108864+A07fcb24b256a6f5e5234055f12ccdb4b5c241e0a@5835c92c b8c32af7dd2d00ae3958dacb1cabdaa8+67108864+A7493b72649b8de4f8c754cb38a4ec0b9617f6129@5835c92c 1ec3958e4e80f6cbdb69920c6921df90+67108864+Ac49d991d55041626ece414881ce5d967670dca8c@5835c92c 09fcfea01c3a141b89dd0dcfa1b7768e+22534144+Aaca28860ddc16d7ba144909f54168376c90b224a@5835c92c 0:559405056:393fb89a2ac7757aca9902dc1286049eecab7b9afd89a9bb6d5b9402e350b280.tar
+. 930625b054ce894ac40596c3f5a0d947+33+A1f27a35dd9af37191d63ad8eb8985624451e7b79@5835c8bc 0:0:a 0:0:b 0:33:output.txt
+./c d41d8cd98f00b204e9800998ecf8427e+0+A27117dcd30c013a6e85d6d74c9a50179a1446efa@5835c8bc 0:0:d
</pre>
-h2. Normalized manifest v1
-
-A normalized manifest has the following additional restrictions.
+A manifest containing a file consisting of multiple blocks and a space in the file name:
-* Streams are in alphanumeric order.
-* Each stream name is unique within the manifest.
-* Files within a stream are in alphanumeric order.
-* Blocks within a stream are ordered based on first appearence in the list of file segments, a given block is listed at most once in a stream.
-* Filename must not contain @"/"@.
+<pre>
+. c449ed86671e4a34a8b8b9430850beba+67108864 09fcfea01c3a141b89dd0dcfa1b7768e+22534144 0:89643008:Docker\040image.tar
+</pre>
projects / arvados.git / commitdiff
