X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/b7f67c80916c2efa0c234ab5f4e92c24d47223c5..d3cb9460f0a30f9602508bed7507c906b8a6d91d:/doc/user/topics/run-command.html.textile.liquid
diff --git a/doc/user/topics/run-command.html.textile.liquid b/doc/user/topics/run-command.html.textile.liquid
index 6d3e87ba29..78f1c84aee 100644
--- a/doc/user/topics/run-command.html.textile.liquid
+++ b/doc/user/topics/run-command.html.textile.liquid
@@ -47,14 +47,22 @@ Items in the "command" list may include lists and objects in addition to strings
}
+Finally, if "command" is a list of lists, it specifies a Unix pipeline where the standard output of the previous command is piped into the standard input of the next command. The following example describes the Unix pipeline @cat foo | grep bar@:
+
+
+{
+ "command": [["cat", "foo"], ["grep", "bar"]]
+}
+
+
h2. Parameter substitution
The "command" list can include parameter substitutions. Substitutions are enclosed in "$(...)" and may contain the name of a user-defined parameter. In the following example, the value of "a" is "hello world"; so when "command" is evaluated, it will substitute "hello world" for "$(a)":
{
- "command": ["echo", "$(file $(a))"],
- "a": "c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2"
+ "a": "c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2",
+ "command": ["echo", "$(file $(a))"]
}
@@ -67,17 +75,47 @@ table(table table-bordered table-condensed).
h2. List context
-When a parameter is evaluated in a list context, that means its value should evaluate to a list instead of a string. Parameter values can be a static list (as demonstrated above), a path to a file, a path to a directory, or a JSON object describing a list context function.
+Where specified by the documentation, parameters may be evaluated in a "list context". That means the value will evaluate to a list instead of a string. Parameter values can be a static list, a path to a file, a path to a directory, or a JSON object describing a list context function.
-If the value is a static list, it will evaluate the list items for parameter substation and list functions.
+If the value is a string, it is interpreted as a path. If the path specifies a regular file, that file will be opened as a text file and produce a list with one item for each line in the file (end-of-line characters will be stripped). If the path specifies a directory, produce a list containing all of the entries in the directory. Note that parameter expansion is not performed on list items produced this way.
-If the value is a string, it is interpreted as a path. If the path specifies a regular file, that file will be opened as a text file and produce a list with one item for each line in the file (end-of-line characters will be stripped). If the path specifies a directory, produce a list containing all of the entries in the directory. Note that parameter expansion is not performed on lists produced this way.
+If the value is a static list, it will evaluate each item and return the expanded list. Each item may be a string (evaluated for parameter substitution), a list (recursively evaluated), or a JSON object (indicating a list function, described below).
If the value is a JSON object, it is evaluated as a list function described below.
h2. List functions
-When @run-command@ is evaluating a list (such as "command"), in addition to string parameter substitution, you can use list item functions. Note: in the following functions, you specify the name of a user parameter to act on; you cannot provide the list value directly in line.
+When @run-command@ is evaluating a list (such as "command"), in addition to string parameter substitution, you can use list item functions. In the following functions, you specify the name of a user parameter to act on (@"$(a)"@ in the first example); the value of that user parameter will be evaluated in a list context (as described above) to get the list value. Alternately, you can provide list value directly in line. As an example, the following two fragments yield the same result:
+
+
+{
+ "a": ["alice", "bob"],
+ "command": ["echo", {"foreach": "$(a)",
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
+}
+
+
+
+{
+ "command": ["echo", {"foreach": ["alice", "bob"],
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
+}
+
+
+Note: when you provide the list inline with "foreach" or "index", you must include the "var" parameter to specify the substitution variable name to use when evaluating the command fragment.
+
+You can also nest functions. This filters @["alice", "bob", "betty"]@ on the regular expression @"b.*"@ to get the list @["bob", "betty"]@, assigns @a_var@ to each value of the list, then expands @"command"@ to get @["--something", "bob", "--something", "betty"]@.
+
+
+{
+ "command": ["echo", {"foreach": {"filter": ["alice", "bob", "betty"],
+ "regex": "b.*"},
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
+}
+
h3. foreach
@@ -85,8 +123,10 @@ The @foreach@ list item function (not to be confused with the @task.foreach@ dir
{
- "command": ["echo", {"foreach": "a", "command": ["--something", "$(a)"]}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"foreach": "$(a)",
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
@@ -96,8 +136,11 @@ This function extracts a single item from a list. The value of @index@ is zero-
{
- "command": ["echo", {"list": "a", "index": 1, "command": ["--something", "$(a)"]}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"list": "$(a)",
+ "var": "a_var",
+ "index": 1,
+ "command": ["--something", "$(a_var)"]}]
}
@@ -107,32 +150,53 @@ Filter the list so that it only includes items that match a regular expression.
{
- "command": ["echo", {"filter": "a", "regex": "b.*"}],
- "a": ["alice", "bob"]
+ "a": ["alice", "bob"],
+ "command": ["echo", {"filter": "$(a)",
+ "regex": "b.*"}]
}
h3. group
-Generate a list of lists, where items are grouped on common subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "--group", "alice", "carol", "dave", "--group", "bob"]@:
+Generate a list of lists, where items are grouped on common subexpression match. Items which don't match the regular expression are excluded. In the following example, the subexpression is @(a?)@, resulting in two groups, strings that contain the letter 'a' and strings that do not. The following example evaluates to @["echo", "--group", "alice", "carol", "dave", "--group", "bob", "betty"]@:
{
- "command": ["echo", {"foreach": "b", "command":["--group", {"foreach": "b", "command":"$(b)"}]}],
- "a": ["alice", "bob", "carol", "dave"],
- "b": {"group": "a", "regex": "[^a]*(a?).*"}
+ "a": ["alice", "bob", "betty", "carol", "dave"],
+ "b": {"group": "$(a)",
+ "regex": "[^a]*(a?).*"},
+ "command": ["echo", {"foreach": "$(b)",
+ "var": "b_var",
+ "command": ["--group", "$(b_var)"]}]
}
h3. extract
-Generate a list of lists, where items are split by subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "c", "a", "rol", "d", "a", "ve"]@:
+Generate a list of lists, where items are split by subexpression match. Items which don't match the regular expression are excluded. The following example evaluates to @["echo", "--something", "c", "a", "rol", "--something", "d", "a", "ve"]@:
+
+
+{
+ "a": ["alice", "bob", "carol", "dave"],
+ "b": {"extract": "$(a)",
+ "regex": "(.+)(a)(.*)"},
+ "command": ["echo", {"foreach": "$(b)",
+ "var": "b_var",
+ "command": ["--something", "$(b_var)"]}]
+}
+
+
+h3. batch
+
+Generate a list of lists, where items are split into a batch size. If the list does not divide evenly into batch sizes, the last batch will be short. The following example evaluates to @["echo", "--something", "alice", "bob", "--something", "carol", "dave"]@
{
- "command": ["echo", {"foreach": "b", "command":[{"foreach": "b", "command":"$(b)"}]}],
"a": ["alice", "bob", "carol", "dave"],
- "b": {"extract": "a", "regex": "(.+)(a)(.*)"}
+ "command": ["echo", {"foreach":{"batch": "$(a)",
+ "size": 2},
+ "var": "a_var",
+ "command": ["--something", "$(a_var)"]}]
}
@@ -142,15 +206,19 @@ Directives alter the behavior of run-command. All directives are optional.
h3. task.cwd
-This directive sets the initial current working directory that your command will run in. If @task.cwd@ is not specified, the default current working directory is @task.outdir@.
+This directive sets the initial current working directory in which your command will run. If @task.cwd@ is not specified, the default current working directory is @task.outdir@.
+
+h3. task.ignore_rcode
+
+By Unix convention a task which exits with a non-zero return code is considered failed. However, some programs (such as @grep@) return non-zero codes for conditions that should not be considered fatal errors. Set @"task.ignore_rcode": true@ to indicate the task should always be considered a success regardless of the return code.
h3. task.stdin and task.stdout
Provide standard input and standard output redirection.
-@task.stdin@ must evaluate to a path to a file to be bound to the command's standard input stream.
+@task.stdin@ must evaluate to a path to a file to be bound to the standard input stream of the command. When command describes a Unix pipeline, this goes into the first command.
-@task.stdout@ specifies the desired file name in the output directory to save the content of standard output.
+@task.stdout@ specifies the desired file name in the output directory to save the content of standard output. When command describes a Unix pipeline, this captures the output of the last command.
h3. task.vwd
@@ -185,10 +253,10 @@ You can also specify multiple parameters:
{
- "command": ["echo", "$(a)", "$(b)"],
- "task.foreach": ["a", "b"],
"a": ["alice", "bob"],
- "b": ["carol", "dave"]
+ "b": ["carol", "dave"],
+ "task.foreach": ["a", "b"],
+ "command": ["echo", "$(a)", "$(b)"]
}