|_. Attribute |_. Type|_. Accepted values |_. Required|_. Description|
|script |string |filename |yes |The actual script that will be run by crunch. Must be the name of an executable file in the crunch_scripts/ directory at the git revision specified by script_version.|
|script_version |string |git branch, tag, or version hash |yes |The code version to run, which is available in the specified repository. May be a git hash or tag to specify an exact version, or a branch. If it is a branch, use the branch head.|
-|repository |string |name of git repository hosted by arvados |yes |The repository to search for script_version.|
+|repository |string |name of git repository hosted by Arvados |yes |The repository to search for script_version.|
|script_parameters |object |any JSON object |yes |The input parameters for the job, with the parameter names as keys mapping to parameter values.|
|minimum_script_version |string |git branch, tag, or version hash |no |The minimum acceptable script version when deciding whether to re-use a past job.|
|exclude_script_versions|array of strings|git branch, tag, or version hash|no |Script versions to exclude when deciding whether to re-use a past job.|
# If 'nondeterministic' or 'no_reuse' are true, always create a new job.
# Find a list of acceptable values for 'script_version'. If 'minimum_script_version' is specified, this is the set of all revisions in the git commit graph between 'minimum_script_version' and 'script_version' (inclusive) [2]. If 'minimum_script_version' is not specified, only 'script_version' is added to the list. If 'exclude_script_versions' is specified, the listed versions are excluded from the list.
-# Select jobs have the same 'script' and 'script_parameters' attributes, and where the 'script_version' attribute is in the list of acceptable versions. Exclude failed jobs or where 'nondeterministic' is true.
+# Select jobs have the same 'script' and 'script_parameters' attributes, and where the 'script_version' attribute is in the list of acceptable versions. Exclude jobs that failed or set 'nondeterministic' to true.
# If there is more than one candidate job, check that all selected past jobs actually did produce the same output.
# If everything passed, re-use one of the selected past jobs (if there is more than one match, which job will be returned is undefined). Otherwise create a new job.
h3. Examples
-Run the script "crunch_scripts/hash.py" in the repository "you" using the "master" branch head. Arvados is allowed to re-use a previous job if the script_version of the past job is the same as the "master" branch head (i.e. there have not been any subsequent commits to "master").
+Run the script "crunch_scripts/hash.py" in the repository "you" using the "master" branch head. Arvados is allowed to re-use a previous job if the script_version of the past job is the same as the "master" branch head (i.e., there have not been any subsequent commits to "master").
-<pre>
+<notextile><pre>
{
"script": "hash.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": {
"input": "c1bad4b39ca5a924e481008009d94e32+210"
}
}
-</pre>
+</pre></notextile>
-Run using exactly the version "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5". Arvados is allowed to re-use a previous job if the script_version of that job is also "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5".
+Run using exactly the version "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5". Arvados is allowed to re-use a previous job if the "script_version" of that job is also "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5".
-<pre>
+<notextile><pre>
{
"script": "hash.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "d00220fb38d4b85ca8fc28a8151702a2b9d1dec5",
"script_parameters": {
"input": "c1bad4b39ca5a924e481008009d94e32+210"
}
}
-</pre>
+</pre></notextile>
-Arvados is allowed to re-use a previous job if the script_version of the past job is between "earlier_version_tag" and the head of the "master" branch (inclusive), but not "blacklisted_version_tag". If there are no previous jobs, run the job using the head of the "master" branch as specified in "script_version".
+Arvados is allowed to re-use a previous job if the "script_version" of the past job is between "earlier_version_tag" and the head of the "master" branch (inclusive), but not "blacklisted_version_tag". If there are no previous jobs, run the job using the head of the "master" branch as specified in "script_version".
-<pre>
+<notextile><pre>
{
"script": "hash.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"minimum_script_version": "earlier_version_tag",
"script_version": "master",
- "exclude_script_versions", ["blacklisted_version_tag"],
+ "exclude_script_versions": ["blacklisted_version_tag"],
"script_parameters": {
"input": "c1bad4b39ca5a924e481008009d94e32+210"
}
}
-</pre>
+</pre></notextile>
Run the script "crunch_scripts/monte-carlo.py" in the repository "you" using the "master" branch head. Because it is marked as "nondeterministic", never re-use previous jobs, and never re-use this job.
-<pre>
+<notextile><pre>
{
"script": "monte-carlo.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"nondeterministic": true,
"script_parameters": {
"input": "c1bad4b39ca5a924e481008009d94e32+210"
}
}
-</pre>
+</pre></notextile>
h2. Pipelines
h3. Examples
-This a pipeline named "Filter md5 hash values" with two components, "do_hash" and "filter". The "input" script parameter of the "do_hash" component is required to be filled in by the user, and the expected data type is "Collection". This also specifies that the "input" script parameter of the "filter" component is the output of "do_hash", so "filter" will not run until "do_hash" completes successfully. When the pipeline runs, past jobs that meet the criteria described above may be substituted for either or both components to avoid redundant computation.
+This is a pipeline named "Filter md5 hash values" with two components, "do_hash" and "filter". The "input" script parameter of the "do_hash" component is required to be filled in by the user, and the expected data type is "Collection". This also specifies that the "input" script parameter of the "filter" component is the output of "do_hash", so "filter" will not run until "do_hash" completes successfully. When the pipeline runs, past jobs that meet the criteria described above may be substituted for either or both components to avoid redundant computation.
-<pre>
+<notextile><pre>
{
"name": "Filter md5 hash values",
"components": {
"do_hash": {
"script": "hash.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": {
"input": {
},
"filter": {
"script": "0-filter.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": {
"input": {
}
}
}
-</pre>
+</pre></notextile>
This pipeline consists of three components. The components "thing1" and "thing2" both depend on "cat_in_the_hat". Once the "cat_in_the_hat" job is complete, both "thing1" and "thing2" can run in parallel, because they do not depend on each other.
-<pre>
+<notextile><pre>
{
"name": "Wreck the house",
"components": {
"cat_in_the_hat": {
"script": "cat.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": { }
},
"thing1": {
"script": "thing1.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": {
"input": {
},
"thing2": {
"script": "thing2.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": {
"input": {
},
}
}
-</pre>
+</pre></notextile>
This pipeline consists of three components. The component "cleanup" depends on "thing1" and "thing2". Both "thing1" and "thing2" are started immediately and can run in parallel, because they do not depend on each other, but "cleanup" cannot begin until both "thing1" and "thing2" have completed.
-<pre>
+<notextile><pre>
{
"name": "Clean the house",
"components": {
"thing1": {
"script": "thing1.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": { }
},
"thing2": {
"script": "thing2.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": { }
},
"cleanup": {
"script": "cleanup.py",
- "repository": "you",
+ "repository": "<b>you</b>",
"script_version": "master",
"script_parameters": {
"mess1": {
}
}
}
-</pre>
+</pre></notextile>
projects / arvados.git / blobdiff