10008: Merge branch 'master' into 10008-flaky-token-test

[arvados.git] / doc / api / methods / jobs.html.textile.liquid

---
layout: default
navsection: api
navmenu: API Methods
title: "jobs"

...

See "REST methods for working with Arvados resources":{{site.baseurl}}/api/methods.html

API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/jobs@

Required arguments are displayed in %{background:#ccffcc}green%.

h2. cancel

Cancel a job that is queued or running.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string||path||

h2(#create). create

Create a new Job.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|job|object|See "Job resource":{{site.baseurl}}/api/schema/Job.html|request body||
|minimum_script_version |string     |Git branch, tag, or commit hash specifying the minimum acceptable script version (earliest ancestor) to consider when deciding whether to re-use a past job.[1]|query|@"c3e86c9"@|
|exclude_script_versions|array of strings|Git commit branches, tags, or hashes to exclude when deciding whether to re-use a past job.|query|@["8f03c71","8f03c71"]@
@["badtag1","badtag2"]@|
|filters|array of arrays|Conditions to find Jobs to reuse.|query||
|find_or_create         |boolean    |Before creating, look for an existing job that has identical script, script_version, and script_parameters to those in the present job, has nondeterministic=false, and did not fail (it could be queued, running, or completed). If such a job exists, respond with the existing job instead of submitting a new one.|query|@false@|

When a job is submitted to the queue using the **create** method, the @script_version@ attribute is updated to a full 40-character Git commit hash based on the current content of the specified repository. If @script_version@ cannot be resolved, the job submission is rejected.

fn1. See the "note about specifying Git commits on the Job resource page":{{site.baseurl}}/api/schema/Job.html#script_version for more detail.

h3. Specialized filters

Special filter operations are available for specific Job columns.

* @script_version@ @in git@ @REFSPEC@, @arvados_sdk_version@ @in git@ @REFSPEC@<br>Resolve @REFSPEC@ to a list of Git commits, and match jobs with a @script_version@ or @arvados_sdk_version@ in that list.  When creating a job and filtering @script_version@, the search will find commits between @REFSPEC@ and the submitted job's @script_version@; all other searches will find commits between @REFSPEC@ and HEAD.  This list may include parallel branches if there is more than one path between @REFSPEC@ and the end commit in the graph.  Use @not in@ or @not in git@ filters (below) to blacklist specific commits.

* @script_version@ @not in git@ @REFSPEC@, @arvados_sdk_version@ @not in git@ @REFSPEC@<br>Resolve @REFSPEC@ to a list of Git commits, and match jobs with a @script_version@ or @arvados_sdk_version@ not in that list.

* @docker_image_locator@ @in docker@ @SEARCH@<br>@SEARCH@ can be a Docker image hash, a repository name, or a repository name and tag separated by a colon (@:@).  The server will find collections that contain a Docker image that match that search criteria, then match jobs with a @docker_image_locator@ in that list.

* @docker_image_locator@ @not in docker@ @SEARCH@<br>Negate the @in docker@ filter.

h3. Reusing jobs

Because Arvados records the exact version of the script, input parameters, and runtime environment that was used to run the job, if the script is deterministic (meaning that the same code version is guaranteed to produce the same outputs from the same inputs) then it is possible to re-use the results of past jobs, and avoid re-running the computation to save time.  Arvados uses the following algorithm to determine if a past job can be re-used:

notextile. <div class="spaced-out">

# If @find_or_create@ is false or omitted, create a new job and skip the rest of these steps.
# If @filters@ are specified, find jobs that match those filters. If any filters are given, there must be at least one filter on the @repository@ attribute and one on the @script@ attribute: otherwise an error is returned.
# If @filters@ are not specified, find jobs with the same @repository@ and @script@, with a @script_version@ between @minimum_script_version@ and @script_version@ inclusively (excluding @excluded_script_versions@), and a @docker_image_locator@ with the latest Collection that matches the submitted job's @docker_image@ constraint.  If the submitted job includes an @arvados_sdk_version@ constraint, jobs must have an @arvados_sdk_version@ between that refspec and HEAD to be found. *This form is deprecated: use filters instead.*
# If the found jobs include a completed job, and all found completed jobs have consistent output, return one of them.  Which specific job is returned is undefined.
# If the found jobs only include incomplete jobs, return one of them.  Which specific job is returned is undefined.
# If no job has been returned so far, create and return a new job.

</div>

h3. Examples

Run the script "crunch_scripts/hash.py" in the repository "you" using the "master" commit.  Arvados should re-use a previous job if the script_version of the previous job is the same as the current "master" commit. This works irrespective of whether the previous job was submitted using the name "master", a different branch name or tag indicating the same commit, a SHA-1 commit hash, etc.

<notextile><pre>
{
  "job": {
    "script": "hash.py",
    "repository": "<b>you</b>/<b>you</b>",
    "script_version": "master",
    "script_parameters": {
      "input": "c1bad4b39ca5a924e481008009d94e32+210"
    }
  },
  "find_or_create": true
}
</pre></notextile>

Run using exactly the version "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5". Arvados should re-use a previous job if the "script_version" of that job is also "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5".

<notextile><pre>
{
  "job": {
    "script": "hash.py",
    "repository": "<b>you</b>/<b>you</b>",
    "script_version": "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5",
    "script_parameters": {
      "input": "c1bad4b39ca5a924e481008009d94e32+210"
    }
  },
  "find_or_create": true
}
</pre></notextile>

Arvados should re-use a previous job if the "script_version" of the previous job is between "earlier_version_tag" and the "master" commit (inclusive), but not the commit indicated by "blacklisted_version_tag". If there are no previous jobs matching these criteria, run the job using the "master" commit.

<notextile><pre>
{
  "job": {
    "script": "hash.py",
    "repository": "<b>you</b>/<b>you</b>",
    "script_version": "master",
    "script_parameters": {
      "input": "c1bad4b39ca5a924e481008009d94e32+210"
    }
  },
  "minimum_script_version": "earlier_version_tag",
  "exclude_script_versions": ["blacklisted_version_tag"],
  "find_or_create": true
}
</pre></notextile>

The same behavior, using filters:

<notextile><pre>
{
  "job": {
    "script": "hash.py",
    "repository": "<b>you</b>/<b>you</b>",
    "script_version": "master",
    "script_parameters": {
      "input": "c1bad4b39ca5a924e481008009d94e32+210"
    }
  },
  "filters": [["script", "=", "hash.py"],
              ["repository", "=", "<b>you</b>/<b>you</b>"],
              ["script_version", "in git", "earlier_version_tag"],
              ["script_version", "not in git", "blacklisted_version_tag"]],
  "find_or_create": true
}
</pre></notextile>

Run the script "crunch_scripts/monte-carlo.py" in the repository "you/you" using the current "master" commit. Because it is marked as "nondeterministic", this job will not be considered as a suitable candidate for future job submissions that use the "find_or_create" feature.

<notextile><pre>
{
  "job": {
    "script": "monte-carlo.py",
    "repository": "<b>you</b>/<b>you</b>",
    "script_version": "master",
    "nondeterministic": true,
    "script_parameters": {
      "input": "c1bad4b39ca5a924e481008009d94e32+210"
    }
  }
}
</pre></notextile>

h2. delete

Delete an existing Job.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string|The UUID of the Job in question.|path||

h2. get

Gets a Job's metadata by UUID.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string|The UUID of the Job in question.|path||

h2. list

List jobs.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
|limit|integer (default 100)|Maximum number of jobs to return.|query||
|order|string|Order in which to return matching jobs.|query||
|filters|array|Conditions for filtering jobs.|query||

See the create method documentation for more information about Job-specific filters.

h2. log_tail_follow

log_tail_follow jobs

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string||path||
|buffer_size|integer (default 8192)||query||

h2. queue

Get the current job queue.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
|order|string||query||
|filters|array||query||

This method is equivalent to the "list method":#list, except that the results are restricted to queued jobs (i.e., jobs that have not yet been started or cancelled) and order defaults to queue priority.

h2. update

Update attributes of an existing Job.

Arguments:

table(table table-bordered table-condensed).
|_. Argument |_. Type |_. Description |_. Location |_. Example |
{background:#ccffcc}.|uuid|string|The UUID of the Job in question.|path||
|job|object||query||