Merge branch '21764-project-picker-crash' into main. Closes #21764

[arvados.git] / doc / api / methods / container_requests.html.textile.liquid

---
layout: default
navsection: api
navmenu: API Methods
title: "container_requests"

...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.

SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}

API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/container_requests@

Object type: @xvhdp@

Example UUID: @zzzzz-xvhdp-0123456789abcde@

h2. Resource

A container request is a request for the Arvados cluster to perform some computational work.  See "computing with Crunch":{{site.baseurl}}/api/execution.html for details.

Each ContainerRequest offers the following attributes, in addition to the "Common resource fields":{{site.baseurl}}/api/resources.html:

All attributes are optional, unless otherwise marked as required.

table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|_. Notes|
|name|string|The name of the container_request.||
|description|string|The description of the container_request.||
|properties|hash|User-defined metadata that does not affect how the container is run.  May be used in queries using "subproperty filters":{{site.baseurl}}/api/methods.html#subpropertyfilters||
|state|string|The allowed states are "Uncommitted", "Committed", and "Final".|Once a request is Committed, the only attributes that can be modified are priority, container_uuid, and container_count_max. A request in the "Final" state cannot have any of its functional parts modified (i.e., only name, description, and properties fields can be modified).|
|requesting_container_uuid|string|The uuid of the parent container that created this container_request, if any. Represents a process tree.|The priority of this container_request is inherited from the parent container, if the parent container is cancelled, this container_request will be cancelled as well.|
|container_uuid|string|The uuid of the container that satisfies this container_request. The system may return a preexisting Container that matches the container request criteria. See "Container reuse":#container_reuse for more details.|Container reuse is the default behavior, but may be disabled with @use_existing: false@ to always create a new container.|
|container_count_max|integer|Maximum number of containers to start, i.e., the maximum number of "attempts" to be made.||
|mounts|hash|Objects to attach to the container's filesystem and stdin/stdout.|See "Mount types":#mount_types for more details.|
|secret_mounts|hash|Objects to attach to the container's filesystem.  Only "json" or "text" mount types allowed.|Not returned in API responses. Reset to empty when state is "Complete" or "Cancelled".|
|runtime_constraints|hash|Restrict the container's access to compute resources and the outside world.|Required when in "Committed" state. e.g.,<pre><code>{
  "ram":12000000000,
  "vcpus":2,
  "API":true
}</code></pre>See "Runtime constraints":#runtime_constraints for more details.|
|scheduling_parameters|hash|Parameters to be passed to the container scheduler when running this container.|e.g.,<pre><code>{
"partitions":["fastcpu","vfastcpu"]
}</code></pre>See "Scheduling parameters":#scheduling_parameters for more details.|
|container_image|string|Portable data hash of a collection containing the docker image to run the container.|Required.|
|environment|hash|Environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exist) overrides environment variables given in the image's Dockerfile.||
|cwd|string|Initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile.|Required.|
|command|array of strings|Command to execute in the container.|Required. e.g., @["echo","hello"]@|
|output_path|string|Path to a directory or file inside the container that should be preserved as container's output when it finishes. This path must be one of the mount targets. For best performance, point output_path to a writable collection mount.  See "Pre-populate output using Mount points":#pre-populate-output for details regarding optional output pre-population using mount points and "Symlinks in output":#symlinks-in-output for additional details.|Required.|
|output_glob|array of strings|Glob patterns determining which files (of those present in the output directory when the container finishes) will be included in the output collection. If multiple patterns are given, files that match any pattern are included. If null or empty, all files will be included.|e.g., @["**/*.vcf", "**/*.vcf.gz"]@
See "Glob patterns":#glob_patterns for more details.|
|output_name|string|Desired name for the output collection. If null or empty, a name will be assigned automatically.||
|output_ttl|integer|Desired lifetime for the output collection, in seconds. If zero, the output collection will not be deleted automatically.||
|priority|integer|Range 0-1000.  Indicate scheduling order preference.|Clients are expected to submit container requests with zero priority in order to preview the container that will be used to satisfy it. Priority can be null if and only if state!="Committed".  See "below for more details":#priority .|
|expires_at|datetime|After this time, priority is considered to be zero.|Not yet implemented.|
|use_existing|boolean|If possible, use an existing (non-failed) container to satisfy the request instead of creating a new one.|Default is true|
|log_uuid|string|Log collection containing log messages provided by the scheduler and crunch processes.|Null if the container has not yet started running.
To retrieve logs in real time while the container is running, use the log API (see below).|
|output_uuid|string|Output collection created when the container finished successfully.|Null if the container has failed or not yet completed.|
|filters|string|Additional constraints for satisfying the container_request, given in the same form as the filters parameter accepted by the container_requests.list API.|
|runtime_token|string|A v2 token to be passed into the container itself, used to access Keep-backed mounts, etc.  |Not returned in API responses.  Reset to null when state is "Complete" or "Cancelled".|
|runtime_user_uuid|string|The user permission that will be granted to this container.||
|runtime_auth_scopes|array of string|The scopes associated with the auth token used to run this container.||
|output_storage_classes|array of strings|The storage classes that will be used for the log and output collections of this container request|default is ["default"]|
|output_properties|hash|User metadata properties to set on the output collection.  The output collection will also have default properties "type" ("intermediate" or "output") and "container_request" (the uuid of container request that produced the collection).|
|cumulative_cost|number|Estimated cost of the cloud VMs used to satisfy the request, including retried attempts and completed subrequests, but not including reused containers.|0 if container was reused or VM price information was not available.|

h2(#lifecycle). Container request lifecycle

A container request may be created in the Committed state, or created in the Uncommitted state and then moved into the Committed state.

Once a request is in the Committed state, Arvados locates a suitable existing container or schedules a new one. When the assigned container finishes, the request state changes to Final.

A client may cancel a committed request early (before the assigned container finishes) by setting the request priority to zero.

!{max-width:60em;}{{site.baseurl}}/api/methods/container_request_lifecycle.svg!
{% comment %}
# svg generated using `graphviz -Tsvg -O`
digraph {
    graph [nojustify=true] [labeljust=l]

    invisiblestart [label = ""] [color=white] [group=lifecycle];
    node [color=black] [fillcolor=white] [style=filled] [shape=box] [nojustify=true];
    uncommitted [label = "container request:\l   state=Uncommitted\l"] [fillcolor=lightgrey] [group=lifecycle];
    {
        rank=same;
        committed [label = "container request:\l   state=Committed\l   priority>0\l"] [group=lifecycle];
        reused [label = "container request:\l   state=Final\lcontainer:\l   state=Complete\l(reused existing container)\l"] [fillcolor=lightblue] [group=endstate];
    }
    invisiblestart -> uncommitted [label = "   user creates container request\l"] [color=navy] [fontcolor=navy];
    uncommitted -> committed [label = "   user updates to\l      state=Committed, priority>0\l"] [color=navy] [fontcolor=navy];
    queued [label = "container request:\l   state=Committed\l   priority>0\lcontainer:\l   state=Queued\l"] [group=lifecycle];
    committed -> queued [label = "   Arvados creates a new container\l"];
    {
        rank=same;
        locked [label = "container request:\l   state=Committed\l   priority>0\lcontainer:\l   state=Locked\l"] [group=lifecycle];
        latecancelled [label = "container request:\l   state=Final\lcontainer:\l   state=Cancelled\l"] [fillcolor=lightblue] [group=endstate];
    }
    queued -> locked [label = "   Arvados is ready to dispatch the container\l"];
    {
        rank=same;
        running [label = "container request:\l   state=Committed\l   priority>0\lcontainer:\l   state=Running\l"] [group=lifecycle];
        containerfailed [label = "container request:\l   state=Final\lcontainer:\l   state=Complete\l   exit_code≠0\l"] [fillcolor=lightblue] [group=endstate];
    }
    locked -> running [label = "   Arvados starts the container process\l"];
    containerfinished [label = "container request:\l   state=Final\lcontainer:\l   state=Complete\l   exit_code=0\l"] [fillcolor=lightblue] [group=lifecycle];

    committed -> reused [label = "Arvados selects an existing container"] [constraint=false] [labeldistance=0.5];
    queued -> latecancelled [label = "user updates to priority=0"] [color=navy] [fontcolor=navy];
    locked -> latecancelled [label = "user updates to priority=0"] [color=navy] [fontcolor=navy] [constraint=false];
    running -> latecancelled [label = "user updates to priority=0"] [color=navy] [fontcolor=navy] [constraint=false];
    running -> containerfailed [label = "container process fails"];
    running -> containerfinished [label = "   container process succeeds\l"];

    # layout hacks
    reused -> latecancelled [style=invis];
    latecancelled -> containerfailed [style=invis];
}
{% endcomment %}

h2(#priority). Priority

The @priority@ field has a range of 0-1000.

Priority 0 means no container should run on behalf of this request, and containers already running will be terminated (setting container priority to 0 is the cancel operation.)

Priority 1 is the lowest priority.

Priority 1000 is the highest priority.

The actual order that containers execute is determined by the underlying scheduling software (e.g. Slurm) and may be based on a combination of container priority, submission time, available resources, and other factors.

In the current implementation, the magnitude of difference in priority between two containers affects the weight of priority vs age in determining scheduling order.  If two containers have only a small difference in priority (for example, 500 and 501) and the lower priority container has a longer queue time, the lower priority container may be scheduled before the higher priority container.  Use a greater magnitude difference (for example, 500 and 600) to give higher weight to priority over queue time.

h2(#mount_types). {% include 'mount_types' %}

h2(#runtime_constraints). {% include 'container_runtime_constraints' %}

h2(#scheduling_parameters). {% include 'container_scheduling_parameters' %}

h2(#glob_patterns). {% include 'container_glob_patterns' %}

h2(#container_reuse). Container reuse

When a container request is "Committed", the system will try to find and reuse an existing Container with the same command, cwd, environment, output_path, container_image, mounts, secret_mounts, runtime_constraints, runtime_user_uuid, and runtime_auth_scopes being requested.

* The serialized fields environment, mounts, and runtime_constraints are normalized when searching.
* The system will also search for containers with minor variations in the keep_cache_disk and keep_cache_ram runtime_constraints that should not affect the result. This searches for other common values for those constraints, so a container that used a non-default value for these constraints may not be reused by later container requests that use a different value.

In order of preference, the system will use:

* The first matching container to have finished successfully (i.e., reached state "Complete" with an exit_code of 0) whose log and output collections are still available.
* The oldest matching "Running" container with the highest progress, i.e., the container that is most likely to finish first.
* The oldest matching "Locked" container with the highest priority, i.e., the container that is most likely to start first.
* The oldest matching "Queued" container with the highest priority, i.e,, the container that is most likely to start first.
* A new container.

h2(#cancel_container). Canceling a container request

A container request may be canceled by setting its priority to 0, using an update call.

When a container request is canceled, it will still reflect the state of the Container it is associated with via the container_uuid attribute. If that Container is being reused by any other container_requests that are still active, i.e., not yet canceled, that Container may continue to run or be scheduled to run by the system in future. However, if no other container_requests are using that Container, then the Container will get canceled as well.

h2. Methods

See "Common resource methods":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.

Required arguments are displayed in %{background:#ccffcc}green%.

Supports federated @create@, @delete@, @get@, @list@, and @update@.

h2(#create). create

Create a new container request.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|container_request|object|Container request resource.|request body||
|cluster_id|string|The federated cluster to submit the container request.|query||

The request body must include the required attributes command, container_image, cwd, and output_path. It can also inlcude other attributes such as environment, mounts, and runtime_constraints.

h3. delete

Delete an existing container request.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path||

h3. get

Get a container request's metadata by UUID.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path||

h3. list

List container requests.

See "common resource list method.":{{site.baseurl}}/api/methods.html#index

The @filters@ argument can also filter on attributes of the container referenced by @container_uuid@. For example, @[["container.state", "=", "Running"]]@ will match any container request whose container is running now.

h3. update

Update attributes of an existing container request.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path||
|container_request|object||query||

{% include 'notebox_begin' %}
Setting the priority of a committed container_request to 0 may cancel a running container assigned for it.
See "Canceling a container request":{{site.baseurl}}/api/methods/container_requests.html#cancel_container for further details.
{% include 'notebox_end' %}

h3(#container_status). container_status

Get container status.

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |
{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path|

Example request: @GET /arvados/v1/container_requests/zzzzz-xvdhp-0123456789abcde/container_status@

Response attributes:

table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|_. Examples|
|uuid|string|The UUID of the container assigned to this request.||
|state|string|The state of the container assigned to this request (see "container resource attributes":containers.html).||
|scheduling_status|string|A brief explanation of the container's status in the dispatch queue, or an empty string if scheduling is not applicable, e.g., the container is running or finished.|@waiting for cloud resources: queue position 3@
@creating new instance@
@preparing runtime environment@|

h3(#log). log

Get container log data using WebDAV methods.

This API retrieves data from the container request's log collection. It can be used at any time in the container request lifecycle.
* Before a container has been assigned (the request is @Uncommitted@) it returns an empty directory.
* While the container is @Queued@ or @Locked@, it returns an empty directory.
* While the container is @Running@, @.../log/{container_uuid}/@ returns real-time logging data.
* While the container is @Complete@ or @Cancelled@, @.../log/{container_uuid}/@ returns the final log collection.

If a request results in multiple containers being run (see @container_count_max@ above), the logs from prior attempts remain available at @.../log/{old_container_uuid}/@.

Currently, this API has a limitation that a directory listing at the top level @/arvados/v1/container_requests/{uuid}/log/@ does not reveal the per-container subdirectories. Instead, clients should look up the container request record and use the @container_uuid@ attribute to request files and directory listings under the per-container directory, as in the examples below.

This API supports the @Range@ request header, so it can be used to poll for and retrieve logs incrementally while the container is running.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|method|string|Read-only WebDAV method|HTTP method|@GET@, @OPTIONS@, @PROPFIND@|
{background:#ccffcc}.|uuid|string|The UUID of the container request.|path|zzzzz-xvdhp-0123456789abcde|
{background:#ccffcc}.|path|string|Path to a file in the log collection.|path|@/zzzzz-dz642-0123456789abcde/stderr.txt@|

Examples:
* @GET /arvados/v1/container_requests/zzzzz-xvdhp-0123456789abcde/log/zzzzz-dz642-0123456789abcde/stderr.txt@
* @PROPFIND /arvados/v1/container_requests/zzzzz-xvdhp-0123456789abcde/log/zzzzz-dz642-0123456789abcde/@