5 title: "arv subcommands"
9 _In order to use the @arv@ command, make sure that you have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html_
11 h3(#arv-create). arv create
13 @arv create@ can be used to create Arvados objects from the command line. Arv create opens up the editor of your choice (set the EDITOR environment variable) and allows you to type or paste a json or yaml description. When saved the object will be created on the API server, if it passes validation.
17 $ <code class="userinput">arv create --help</code>
19 --project-uuid, -p <s>: Project uuid in which to create the object
20 --help, -h: Show this message
26 @arv get@ can be used to get a textual representation of Arvados objects from the command line. The output can be limited to a subset of the object's fields. This command can be used with only the knowledge of an object's UUID.
30 $ <code class="userinput">arv arv get --help</code>
31 Usage: arv get [uuid] [fields...]
32 Fetch the specified Arvados object, select the specified fields,
33 and print a text representation (json or yaml, use --format).
36 *Note*: the 'format' flag is used by @arv@ (not @arv get@), as detailed on the "arv CLI overview page":{{site.baseurl}}/sdk/cli/index.html.
38 This command can be used instead of the previously required:
42 $ <code class="userinput">EDITOR=cat arv edit [uuid]</code>
46 h3(#arv-edit). arv edit
48 @arv edit@ can be used to edit Arvados objects from the command line. Arv edit opens up the editor of your choice (set the EDITOR environment variable) with the json or yaml description of the object. Saving the file will update the Arvados object on the API server, if it passes validation.
52 $ <code class="userinput">arv edit --help</code>
53 Arvados command line client
54 Usage: arv edit [uuid] [fields...]
56 Fetch the specified Arvados object, select the specified fields,
57 open an interactive text editor on a text representation (json or
58 yaml, use --format) and then update the object. Will use 'nano'
59 by default, customize with the EDITOR or VISUAL environment variable.
63 h3(#arv-copy). arv copy
65 @arv copy@ can be used to copy a pipeline instance, template or collection from one Arvados instance to another. It takes care of copying the object and all its dependencies.
69 $ <code class="userinput">arv copy --help</code>
70 usage: arv-copy [-h] [-v] [--progress] [--no-progress] [-f] --src
71 SOURCE_ARVADOS --dst DESTINATION_ARVADOS [--recursive]
72 [--no-recursive] [--dst-git-repo DST_GIT_REPO]
73 [--project-uuid PROJECT_UUID] [--retries RETRIES]
76 Copy a pipeline instance, template or collection from one Arvados instance to
80 object_uuid The UUID of the object to be copied.
83 -h, --help show this help message and exit
84 -v, --verbose Verbose output.
85 --progress Report progress on copying collections. (default)
86 --no-progress Do not report progress on copying collections.
87 -f, --force Perform copy even if the object appears to exist at
88 the remote destination.
89 --src SOURCE_ARVADOS The name of the source Arvados instance (required).
90 May be either a pathname to a config file, or the
92 $HOME/.config/arvados/instance_name.conf.
93 --dst DESTINATION_ARVADOS
94 The name of the destination Arvados instance
95 (required). May be either a pathname to a config file,
96 or the basename of a file in
97 $HOME/.config/arvados/instance_name.conf.
98 --recursive Recursively copy any dependencies for this object.
100 --no-recursive Do not copy any dependencies. NOTE: if this option is
101 given, the copied object will need to be updated
102 manually in order to be functional.
103 --dst-git-repo DST_GIT_REPO
104 The name of the destination git repository. Required
105 when copying a pipeline recursively.
106 --project-uuid PROJECT_UUID
107 The UUID of the project at the destination to which
108 the pipeline should be copied.
109 --retries RETRIES Maximum number of times to retry server requests that
110 encounter temporary failures (e.g., server down).
115 h3(#arv-tag). arv tag
117 @arv tag@ is used to tag Arvados objects.
121 $ <code class="userinput">arv tag --help</code>
124 arv tag add tag1 [tag2 ...] --object object_uuid1 [object_uuid2...]
125 arv tag remove tag1 [tag2 ...] --object object_uuid1 [object_uuid2...]
128 --dry-run, -n: Don't actually do anything
129 --verbose, -v: Print some things on stderr
130 --uuid, -u: Return the UUIDs of the objects in the response, one per
132 --json, -j: Return the entire response received from the API server, as
134 --human, -h: Return the response received from the API server, as a JSON
135 object with whitespace added for human consumption
136 --pretty, -p: Synonym of --human
137 --yaml, -y: Return the response received from the API server, in YAML
139 --help, -e: Show this message
146 @arv ws@ provides access to the websockets event stream.
150 $ <code class="userinput">arv ws --help</code>
151 usage: arv-ws [-h] [-u UUID] [-f FILTERS]
152 [--poll-interval POLL_INTERVAL | --no-poll]
153 [-p PIPELINE | -j JOB]
156 -h, --help show this help message and exit
157 -u UUID, --uuid UUID Filter events on object_uuid
158 -f FILTERS, --filters FILTERS
159 Arvados query filter to apply to log events (JSON
161 --poll-interval POLL_INTERVAL
162 If websockets is not available, specify the polling
163 interval, default is every 15 seconds
164 --no-poll Do not poll if websockets are not available, just fail
165 -p PIPELINE, --pipeline PIPELINE
166 Supply pipeline uuid, print log output from pipeline
168 -j JOB, --job JOB Supply job uuid, print log output from jobs
172 h3(#arv-keep). arv keep
174 @arv keep@ provides access to the Keep storage service.
178 $ <code class="userinput">arv keep --help</code>
179 Usage: arv keep [method] [--parameters]
180 Use 'arv keep [method] --help' to get more information about specific methods.
182 Available methods: ls, get, put, less, check, docker
186 h3(#arv-keep-ls). arv keep ls
190 $ <code class="userinput">arv keep ls --help</code>
191 usage: arv-ls [-h] [--retries RETRIES] [-s] locator
193 List contents of a manifest
195 positional arguments:
196 locator Collection UUID or locator
199 -h, --help show this help message and exit
200 --retries RETRIES Maximum number of times to retry server requests that
201 encounter temporary failures (e.g., server down). Default
203 -s List file sizes, in KiB.
207 h3(#arv-keep-get). arv keep get
211 $ <code class="userinput">arv keep get --help</code>
212 usage: arv-get [-h] [--retries RETRIES]
213 [--progress | --no-progress | --batch-progress]
214 [--hash HASH | --md5sum] [-n] [-r] [-f | --skip-existing]
215 locator [destination]
217 Copy data from Keep to a local file or pipe.
219 positional arguments:
220 locator Collection locator, optionally with a file path or
222 destination Local file or directory where the data is to be written.
223 Default: /dev/stdout.
226 -h, --help show this help message and exit
227 --retries RETRIES Maximum number of times to retry server requests that
228 encounter temporary failures (e.g., server down). Default
230 --progress Display human-readable progress on stderr (bytes and, if
231 possible, percentage of total data size). This is the
232 default behavior when it is not expected to interfere
233 with the output: specifically, stderr is a tty _and_
234 either stdout is not a tty, or output is being written to
235 named files rather than stdout.
236 --no-progress Do not display human-readable progress on stderr.
237 --batch-progress Display machine-readable progress on stderr (bytes and,
238 if known, total data size).
239 --hash HASH Display the hash of each file as it is read from Keep,
240 using the given hash algorithm. Supported algorithms
241 include md5, sha1, sha224, sha256, sha384, and sha512.
242 --md5sum Display the MD5 hash of each file as it is read from
244 -n Do not write any data -- just read from Keep, and report
245 md5sums if requested.
246 -r Retrieve all files in the specified collection/prefix.
247 This is the default behavior if the "locator" argument
248 ends with a forward slash.
249 -f Overwrite existing files while writing. The default
250 behavior is to refuse to write *anything* if any of the
251 output files already exist. As a special case, -f is not
252 needed to write to /dev/stdout.
253 --skip-existing Skip files that already exist. The default behavior is to
254 refuse to write *anything* if any files exist that would
255 have to be overwritten. This option causes even devices,
256 sockets, and fifos to be skipped.
260 h3(#arv-keep-put). arv keep put
264 $ <code class="userinput">arv keep put --help</code>
265 usage: arv-put [-h] [--max-manifest-depth N | --normalize]
266 [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]
267 [--use-filename FILENAME] [--filename FILENAME]
268 [--portable-data-hash] [--replication N]
269 [--project-uuid UUID] [--name NAME]
270 [--progress | --no-progress | --batch-progress]
271 [--resume | --no-resume] [--retries RETRIES]
274 Copy data from the local filesystem to Keep.
276 positional arguments:
277 path Local file or directory. Default: read from standard
281 -h, --help show this help message and exit
282 --max-manifest-depth N
283 Maximum depth of directory tree to represent in the
284 manifest structure. A directory structure deeper than
285 this will be represented as a single stream in the
286 manifest. If N=0, the manifest will contain a single
287 stream. Default: -1 (unlimited), i.e., exactly one
288 manifest stream per filesystem directory that contains
290 --normalize Normalize the manifest by re-ordering files and
291 streams after writing data.
292 --as-stream Synonym for --stream.
293 --stream Store the file content and display the resulting
294 manifest on stdout. Do not write the manifest to Keep
295 or save a Collection object in Arvados.
296 --as-manifest Synonym for --manifest.
297 --in-manifest Synonym for --manifest.
298 --manifest Store the file data and resulting manifest in Keep,
299 save a Collection object in Arvados, and display the
300 manifest locator (Collection uuid) on stdout. This is
301 the default behavior.
302 --as-raw Synonym for --raw.
303 --raw Store the file content and display the data block
304 locators on stdout, separated by commas, with a
305 trailing newline. Do not store a manifest.
306 --use-filename FILENAME
307 Synonym for --filename.
308 --filename FILENAME Use the given filename in the manifest, instead of the
309 name of the local file. This is useful when "-" or
310 "/dev/stdin" is given as an input file. It can be used
311 only if there is exactly one path given and it is not
312 a directory. Implies --manifest.
313 --portable-data-hash Print the portable data hash instead of the Arvados
314 UUID for the collection created by the upload.
315 --replication N Set the replication level for the new collection: how
316 many different physical storage devices (e.g., disks)
317 should have a copy of each data block. Default is to
318 use the server-provided default (if any) or 2.
319 --project-uuid UUID Store the collection in the specified project, instead
320 of your Home project.
321 --name NAME Save the collection with the specified name.
322 --progress Display human-readable progress on stderr (bytes and,
323 if possible, percentage of total data size). This is
324 the default behavior when stderr is a tty.
325 --no-progress Do not display human-readable progress on stderr, even
327 --batch-progress Display machine-readable progress on stderr (bytes
328 and, if known, total data size).
329 --resume Continue interrupted uploads from cached state
331 --no-resume Do not continue interrupted uploads from cached state.
332 --retries RETRIES Maximum number of times to retry server requests that
333 encounter temporary failures (e.g., server down).
339 h3(#arv-pipeline-run). arv pipeline run
341 @arv pipeline run@ can be used to start a pipeline run from the command line.
343 The User Guide has a page with a bit more information on "using arv pipeline run":{{site.baseurl}}/user/topics/running-pipeline-command-line.html.
347 $ <code class="userinput">arv pipeline run --help</code>
349 --dry-run, -n: Do not start any new jobs or wait for existing jobs to
350 finish. Just find out whether jobs are finished,
351 queued, or running for each component.
352 --status-text <s>: Store plain text status in given file. (Default:
354 --status-json <s>: Store json-formatted pipeline in given file. (Default:
356 --no-wait: Do not wait for jobs to finish. Just look up status,
357 submit new jobs if needed, and exit.
358 --no-reuse: Do not reuse existing jobs to satisfy pipeline
359 components. Submit a new job for every component.
360 --debug, -d: Print extra debugging information on stderr.
361 --debug-level <i>: Set debug verbosity level.
362 --template <s>: UUID of pipeline template, or path to local pipeline
364 --instance <s>: UUID of pipeline instance.
365 --submit: Submit the pipeline instance to the server, and exit.
366 Let the Crunch dispatch service satisfy the components
367 by finding/running jobs.
368 --run-pipeline-here: Manage the pipeline instance in-process. Submit jobs
369 to Crunch as needed. Do not exit until the pipeline
371 --run-jobs-here: Run jobs in the local terminal session instead of
372 submitting them to Crunch. Implies
373 --run-pipeline-here. Note: this results in a
374 significantly different job execution environment, and
375 some Crunch features are not supported. It can be
376 necessary to modify a pipeline in order to make it run
378 --run-here: Synonym for --run-jobs-here.
379 --description <s>: Description for the pipeline instance.
380 --version, -v: Print version and exit
381 --help, -h: Show this message
385 h3(#arv-run). arv run
387 The @arv-run@ command creates Arvados pipelines at the command line that fan out to multiple concurrent tasks across Arvados compute nodes.
389 The User Guide has a page on "using arv-run":{{site.baseurl}}/user/topics/arv-run.html.
393 $ <code class="userinput">arv run --help</code>
394 usage: arv-run [-h] [--retries RETRIES] [--dry-run] [--local]
395 [--docker-image DOCKER_IMAGE] [--ignore-rcode] [--no-reuse]
396 [--no-wait] [--project-uuid PROJECT_UUID] [--git-dir GIT_DIR]
397 [--repository REPOSITORY] [--script-version SCRIPT_VERSION]
400 positional arguments:
404 -h, --help show this help message and exit
405 --retries RETRIES Maximum number of times to retry server requests that
406 encounter temporary failures (e.g., server down).
408 --dry-run Print out the pipeline that would be submitted and
410 --local Run locally using arv-run-pipeline-instance
411 --docker-image DOCKER_IMAGE
412 Docker image to use, otherwise use instance default.
413 --ignore-rcode Commands that return non-zero return codes should not
414 be considered failed.
415 --no-reuse Do not reuse past jobs.
416 --no-wait Do not wait and display logs after submitting command,
418 --project-uuid PROJECT_UUID
419 Parent project of the pipeline
420 --git-dir GIT_DIR Git repository passed to arv-crunch-job when using
422 --repository REPOSITORY
423 repository field of component, default 'arvados'
424 --script-version SCRIPT_VERSION
425 script_version field of component, default 'master'