...
-
-
Applications submit compute jobs when:
* Provenance is important, i.e., it is worth recording how the output was produced; or
* Computation time is significant; or
h2. Methods
-See "REST methods for working with Arvados resources":{{site.baseurl}}/api/methods.html
-
-API endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/jobs@
-
-h3. Additional parameters for "Create" method
-
-table(table table-bordered table-condensed).
-|_. Attribute |_. Type|_. Accepted values |_. Required|_. Description|
-|script |string |filename |yes |The actual script that will be run by crunch. Must be the name of an executable file in the crunch_scripts/ directory at the git revision specified by script_version.|
-|script_version |string |git branch, tag, or version hash |yes |The code version to run, which is available in the specified repository. May be a git hash or tag to specify an exact version, or a branch. If it is a branch, use the branch head.|
-|repository |string |name of git repository hosted by Arvados |yes |The repository to search for script_version.|
-|script_parameters |object |any JSON object |yes |The input parameters for the job, with the parameter names as keys mapping to parameter values.|
-|minimum_script_version |string |git branch, tag, or version hash |no |The minimum acceptable script version when deciding whether to re-use a past job.|
-|exclude_script_versions|array of strings|git branch, tag, or version hash|no |Script versions to exclude when deciding whether to re-use a past job.|
-|nondeterministic |boolean | |no |If true, never re-use a past job, and flag this job so it will never be considered for re-use.|
-|no_reuse |boolean | |no |If true, do not re-use a past job, but this job may be re-used.|
-
-See the "job and pipeline reference":{{site.baseurl}}/user/reference/job-and-pipeline-reference.html for more information.
-
-h3. Queue
-
-<pre>
-GET https://{{ site.arvados_api_host }}/arvados/v1/jobs/queue
-
-POST https://{{ site.arvados_api_host }}/arvados/v1/jobs/queue
-_method=GET
-where[owner_uuid]=xyzzy-tpzed-a4lcehql0dv2u25
-</pre>
-
-→ Job resource list
-
-This method is equivalent to the "index method":{{site.baseurl}}/api/methods.html#index, except that the results are restricted to queued jobs (i.e., jobs that have not yet been started or cancelled) and order defaults to queue priority.
+See "jobs":{{site.baseurl}}/api/methods/jobs.html
h2. Resource
table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|_. Notes|
-|submit_id|string|Unique ID provided by client when job was submitted|Optional. Can be used by client to ensure idempotence|
-|priority|string|||
-|runtime_constraints{}|list|Constraints that must be satisfied by the job/task scheduler in order to run the job.|See below.|
-|script|string|Name of crunch script in @/crunch_scripts/@||
-|script_parameters{}|list|Parameters passed to MapReduce script||
-|script_version|string|The git ref of the the git commit used to run the job.| When the job starts, Arvados updates this field to the precise git commit hash used by the job.|
-|repository |string |The repository from which the script_version will be fetched.|
+|script|string|The filename of the job script.|This program will be invoked by Crunch for each job task. It is given as a path to an executable file, relative to the @/crunch_scripts@ directory in the Git tree specified by the _repository_ and _script_version_ attributes.|
+|script_parameters|hash|The input parameters for the job.|Conventionally, one of the parameters is called @"input"@. Typically, some parameter values are collection UUIDs. Ultimately, though, the significance of parameters is left entirely up to the script itself.|
+|repository|string|Git repository|Given as the name of a locally hosted git repository.|
+|script_version|string|Git commit|During a **create** transaction, this is the Git branch, tag, or hash supplied by the client. Before the job starts, Arvados updates it to the full 40-character SHA-1 hash of the commit used by the job.
+See "Script versions":#script_version below for more detail about acceptable ways to specify a commit.|
|cancelled_by_client_uuid|string|API client ID|Is null if job has not been cancelled|
|cancelled_by_user_uuid|string|Authenticated user ID|Is null if job has not been cancelled|
|cancelled_at|datetime|When job was cancelled|Is null if job has not been cancelled|
|running|boolean|Whether the job is running||
|success|boolean|Whether the job indicated successful completion|Is null if job has not finished|
|is_locked_by_uuid|string|UUID of the user who has locked this job|Is null if job is not locked. The system user locks the job when starting the job, in order to prevent job attributes from being altered.|
-|log|string|||
-|tasks_summary|Hash|||
-|output|string|||
-|nondeterministic |boolean |Indicates whether this job is a candidate for re-use|If true, this job will never be considered for re-use.|
+|node_uuids|array|List of UUID strings for node objects that have been assigned to this job||
+|log|string|Collection UUID|Is null if the job has not finished. After the job runs, the given collection contains a text file with log messages provided by the @arv-crunch-job@ task scheduler as well as the standard error streams provided by the task processes.|
+|tasks_summary|hash|Summary of task completion states.|Example: @{"done":0,"running":4,"todo":2,"failed":0}@|
+|output|string|Collection UUID|Is null if the job has not finished.|
+|nondeterministic|boolean|The job is expected to produce different results if run more than once.|If true, this job will not be considered as a candidate for automatic re-use when submitting subsequent identical jobs.|
+|submit_id|string|Unique ID provided by client when job was submitted|Optional. This can be used by a client to make the "jobs.create":{{site.baseurl}}/api/methods/jobs.html#create method idempotent.|
+|priority|string|||
+|runtime_constraints|hash|Constraints that must be satisfied by the job/task scheduler in order to run the job.|See below.|
+
+h3(#script_version). Script versions
+
+The @script_version@ attribute is typically given as a branch, tag, or commit hash, but there are many more ways to specify a Git commit. The "specifying revisions" section of the "gitrevisions manual page":http://git-scm.com/docs/gitrevisions.html has a definitive list. Arvados accepts @script_version@ in any format listed there that names a single commit (not a tree, a blob, or a range of commits). However, some kinds of names can be expected to resolve differently in Arvados than they do in your local repository. For example, <code>HEAD@{1}</code> refers to the local reflog, and @origin/master@ typically refers to a remote branch: neither is likely to work as desired if given as a @script_version@.
h3. Runtime constraints
table(table table-bordered table-condensed).
|_. Key|_. Type|_. Description|_. Implemented|
-|dependencies{}|list|[
- {
- "name": "freebayes",
- "package_type": "git",
- "origin": "https://github.com/ekg/freebayes.git",
- "version": "011561f4a96619125d4388b66b2e82b173f3de7a"
- },
- ...
-]||
+|docker_image|string|The Docker image that this Job needs to run. If specified, Crunch will create a Docker container from this image, and run the Job's script inside that. The Keep mount and work directories will be available as volumes inside this container. The image must be uploaded to Arvados using @arv keep docker@. You may specify the image in any format that Docker accepts, such as @arvados/jobs@, @debian:latest@, or the Docker image id. Alternatively, you may specify the UUID of the image Collection, returned by @arv keep docker@.|✓|
|min_nodes|integer||✓|
|max_nodes|integer|||
+|min_cores_per_node|integer|Require that each node assigned to this Job have the specified number of CPU cores|✓|
+|min_ram_mb_per_node|integer|Require that each node assigned to this Job have the specified amount of real memory (in MiB)|✓|
+|min_scratch_mb_per_node|integer|Require that each node assigned to this Job have the specified amount of scratch storage available (in MiB)|✓|
|max_tasks_per_node|integer|Maximum simultaneous tasks on a single node|✓|
|min_ram_per_task|integer|Minimum real memory (KiB) per task||
-|min_ram_per_node|integer|Minimum real memory (KiB) per node||
projects / arvados.git / blobdiff