In the following examples, you can use "dry run mode" to determine the command line that @run-command@ will use without actually running the command. For example:
<notextile>
-<pre><code>~$ <span class="userinput">./run-command --dry-run --script-parameters '{
+<pre><code>~$ <span class="userinput">cd $HOME/arvados/crunch_scripts</span>
+~$ <span class="userinput">./run-command --dry-run --script-parameters '{
"command": ["echo", "hello world"]
}'</span>
run-command: echo hello world
}
</pre>
+Finally, if "command" is a list of lists, it specifies a Unix pipeline where the standard output of the previous command is piped into the standard input of the next command. The following example describes the Unix pipeline @cat foo | grep bar@:
+
+<pre>
+{
+ "command": [["cat", "foo"], ["grep", "bar"]]
+}
+</pre>
+
h2. Parameter substitution
The "command" list can include parameter substitutions. Substitutions are enclosed in "$(...)" and may contain the name of a user-defined parameter. In the following example, the value of "a" is "hello world"; so when "command" is evaluated, it will substitute "hello world" for "$(a)":
<pre>
{
- "command": ["echo", "$(file $(a))"],
- "a": "c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2"
+ "a": "c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2",
+ "command": ["echo", "$(file $(a))"]
}
</pre>
|$(basename ...) | Strip leading directory and trailing file extension from the path provided. For example, $(basename /foo/bar.baz.txt) will evaluate to "bar.baz".|
|$(glob ...) | Take a Unix shell path pattern (supports @*@ @?@ and @[]@) and search the local filesystem, returning the first match found. Use together with $(dir ...) to get a local filesystem path for Arvados collections. For example: $(glob $(dir $(mycollection)/*.bam)) will find the first .bam file in the collection specified by the user parameter "mycollection". If there is more than one match, which one is returned is undefined. Will raise an error if no matches are found.|
+h3. Escape sequences
+
+If your command includes a @$()@ sequence that shouldn't be interpreted by run-command—for example, because you're writing shell code that calls a subcommand—you can prevent run-command from interpreting it by placing a backslash in front of the @$@ character. Note that JSON also uses backslash to escape characters, so you'll need to write two backslashes for run-command to see one after parsing the parameter. This example uppercases all alphabetic characters in the "pattern" parameter before using it as a regular expression in grep:
+
+<pre>{"command": ["bash", "-c", "grep \\$(echo '$(pattern)' | tr a-z A-Z) '$(input)'"]}</pre>
+
+You can put a literal backslash in your command by escaping it with another backslash. Ultimately this means that where the primary Unix command includes a single backslash, you'll need to write four backslashes: double the backslashes for run-command escaping, then double them again for JSON escaping.
+
+<pre>{"command": ["grep", "\\\\bword\\\\b", "$(input)"]}</pre>
+
h2. List context
-When a parameter is evaluated in a list context, that means its value should evaluate to a list instead of a string. Parameter values can be a static list (as demonstrated above), a path to a file, a path to a directory, or a JSON object describing a list context function.
+Where specified by the documentation, parameters may be evaluated in a "list context". That means the value will evaluate to a list instead of a string. Parameter values can be a static list, a path to a file, a path to a directory, or a JSON object describing a list context function.
-If the value is a static list, it will evaluate the list items for parameter substation and list functions.
+If the value is a string, it is interpreted as a path. If the path specifies a regular file, that file will be opened as a text file and produce a list with one item for each line in the file (end-of-line characters will be stripped). If the path specifies a directory, produce a list containing all of the entries in the directory. Note that parameter expansion is not performed on list items produced this way.
-If the value is a string, it is interpreted as a path. If the path specifies a regular file, that file will be opened as a text file and produce a list with one item for each line in the file (end-of-line characters will be stripped). If the path specifies a directory, produce a list containing all of the entries in the directory. Note that parameter expansion is not performed on lists produced this way.
+If the value is a static list, it will evaluate each item and return the expanded list. Each item may be a string (evaluated for parameter substitution), a list (recursively evaluated), or a JSON object (indicating a list function, described below).
If the value is a JSON object, it is evaluated as a list function described below.
h2. List functions
-When @run-command@ is evaluating a list (such as "command"), in addition to string parameter substitution, you can use list item functions. In the following functions, you can either specify the name of a user parameter to act on or provide list value directly in line, for example, the following two fragments yield the same result:
+When @run-command@ is evaluating a list (such as "command"), in addition to string parameter substitution, you can use list item functions. In the following functions, you specify the name of a user parameter to act on (@"$(a)"@ in the first example); the value of that user parameter will be evaluated in a list context (as described above) to get the list value. Alternately, you can provide list value directly in line. As an example, the following two fragments yield the same result:
<pre>
{
- "command": ["echo", {"foreach": "$(a)", "command": ["--something", "$(a)"]}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"foreach": "$(a)",
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
</pre>
<pre>
{
- "command": ["echo", {"foreach": ["alice", "bob"], "var":"a", "command": ["--something", "$(a)"]}],
+ "command": ["echo", {"foreach": ["alice", "bob"],
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
</pre>
Note: when you provide the list inline with "foreach" or "index", you must include the "var" parameter to specify the substitution variable name to use when evaluating the command fragment.
-You can also nest functions:
+You can also nest functions. This filters @["alice", "bob", "betty"]@ on the regular expression @"b.*"@ to get the list @["bob", "betty"]@, assigns @a_var@ to each value of the list, then expands @"command"@ to get @["--something", "bob", "--something", "betty"]@.
<pre>
{
- "command": ["echo", {"foreach": {"filter": ["alice", "bob"], "regex": "b.*"},
- "var":"a",
- "command": ["--something", "$(a)"]}]
+ "command": ["echo", {"foreach": {"filter": ["alice", "bob", "betty"],
+ "regex": "b.*"},
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
</pre>
<pre>
{
- "command": ["echo", {"foreach": "$(a)", "command": ["--something", "$(a)"]}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"foreach": "$(a)",
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
</pre>
<pre>
{
- "command": ["echo", {"list": "$(a)", "var":"a", "index": 1, "command": ["--something", "$(a)"]}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"list": "$(a)",
+ "var": "a_var",
+ "index": 1,
+ "command": ["--something", "$(a_var)"]}]
}
</pre>
<pre>
{
- "command": ["echo", {"filter": "$(a)", "regex": "b.*"}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"filter": "$(a)",
+ "regex": "b.*"}]
}
</pre>
h3. group
-Generate a list of lists, where items are grouped on common subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "--group", "alice", "carol", "dave", "--group", "bob"]@:
+Generate a list of lists, where items are grouped on common subexpression match. Items which don't match the regular expression are excluded. In the following example, the subexpression is @(a?)@, resulting in two groups, strings that contain the letter 'a' and strings that do not. The following example evaluates to @["echo", "--group", "alice", "carol", "dave", "--group", "bob", "betty"]@:
<pre>
{
- "command": ["echo", {"foreach": "$(b)", "command":["--group", {"foreach": "b", "command":"$(b)"}]}],
- "a": ["alice", "bob", "carol", "dave"],
- "b": {"group": "a", "regex": "[^a]*(a?).*"}
+ "a": ["alice", "bob", "betty", "carol", "dave"],
+ "b": {"group": "$(a)",
+ "regex": "[^a]*(a?).*"},
+ "command": ["echo", {"foreach": "$(b)",
+ "var": "b_var",
+ "command": ["--group", "$(b_var)"]}]
}
</pre>
h3. extract
-Generate a list of lists, where items are split by subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "c", "a", "rol", "d", "a", "ve"]@:
+Generate a list of lists, where items are split by subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "--something", "c", "a", "rol", "--something", "d", "a", "ve"]@:
<pre>
{
- "command": ["echo", {"foreach": "$(b)", "command":[{"foreach": "$(b)", "command":"$(b)"}]}],
"a": ["alice", "bob", "carol", "dave"],
- "b": {"extract": "a", "regex": "(.+)(a)(.*)"}
+ "b": {"extract": "$(a)",
+ "regex": "(.+)(a)(.*)"},
+ "command": ["echo", {"foreach": "$(b)",
+ "var": "b_var",
+ "command": ["--something", "$(b_var)"]}]
}
</pre>
h3. batch
-Generate a list of lists, where items are split into batch size. If the list does not divide evenly into batch sizes, the last batch will be short. The following example evaluates to @["echo", "--something", "alice", "bob", "--something", "carol", "dave"]@
+Generate a list of lists, where items are split into a batch size. If the list does not divide evenly into batch sizes, the last batch will be short. The following example evaluates to @["echo", "--something", "alice", "bob", "--something", "carol", "dave"]@
<pre>
{
- "command": ["echo", {"foreach":{"batch": "$(a)", "size": 2}, "var":"a", "command":["--something", "$(a)"]}],
- "a": ["alice", "bob", "carol", "dave"]
+ "a": ["alice", "bob", "carol", "dave"],
+ "command": ["echo", {"foreach":{"batch": "$(a)",
+ "size": 2},
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
</pre>
h3. task.cwd
-This directive sets the initial current working directory that your command will run in. If @task.cwd@ is not specified, the default current working directory is @task.outdir@.
+This directive sets the initial current working directory in which your command will run. If @task.cwd@ is not specified, the default current working directory is @task.outdir@.
+
+h3. task.ignore_rcode
+
+By Unix convention a task which exits with a non-zero return code is considered failed. However, some programs (such as @grep@) return non-zero codes for conditions that should not be considered fatal errors. Set @"task.ignore_rcode": true@ to indicate the task should always be considered a success regardless of the return code.
h3. task.stdin and task.stdout
Provide standard input and standard output redirection.
-@task.stdin@ must evaluate to a path to a file to be bound to the command's standard input stream.
+@task.stdin@ must evaluate to a path to a file to be bound to the standard input stream of the command. When command describes a Unix pipeline, this goes into the first command.
-@task.stdout@ specifies the desired file name in the output directory to save the content of standard output.
+@task.stdout@ specifies the desired file name in the output directory to save the content of standard output. When command describes a Unix pipeline, this captures the output of the last command.
h3. task.vwd
<pre>
{
- "command": ["echo", "$(a)", "$(b)"],
- "task.foreach": ["a", "b"],
"a": ["alice", "bob"],
- "b": ["carol", "dave"]
+ "b": ["carol", "dave"],
+ "task.foreach": ["a", "b"],
+ "command": ["echo", "$(a)", "$(b)"]
}
</pre>
projects / arvados.git / blobdiff