$ arv copy --help
-usage: arv-copy [-h] [-v] [--progress] [--no-progress] [-f] --src
- SOURCE_ARVADOS --dst DESTINATION_ARVADOS [--recursive]
- [--no-recursive] [--dst-git-repo DST_GIT_REPO]
- [--project-uuid PROJECT_UUID] [--retries RETRIES]
- object_uuid
+usage: arv_copy.py [-h] [-v] [--progress] [--no-progress] [-f] --src
+ SOURCE_ARVADOS --dst DESTINATION_ARVADOS [--recursive]
+ [--no-recursive] [--dst-git-repo DST_GIT_REPO]
+ [--project-uuid PROJECT_UUID] [--retries RETRIES]
+ object_uuid
Copy a pipeline instance, template or collection from one Arvados instance to
another.
@@ -64,15 +83,15 @@ optional arguments:
--no-progress Do not report progress on copying collections.
-f, --force Perform copy even if the object appears to exist at
the remote destination.
- --src SOURCE_ARVADOS The name of the source Arvados instance (required).
- May be either a pathname to a config file, or the
- basename of a file in
- $HOME/.config/arvados/instance_name.conf.
+ --src SOURCE_ARVADOS The name of the source Arvados instance (required) -
+ points at an Arvados config file. May be either a
+ pathname to a config file, or (for example) "foo" as
+ shorthand for $HOME/.config/arvados/foo.conf.
--dst DESTINATION_ARVADOS
The name of the destination Arvados instance
- (required). May be either a pathname to a config file,
- or the basename of a file in
- $HOME/.config/arvados/instance_name.conf.
+ (required) - points at an Arvados config file. May be
+ either a pathname to a config file, or (for example)
+ "foo" as shorthand for $HOME/.config/arvados/foo.conf.
--recursive Recursively copy any dependencies for this object.
(default)
--no-recursive Do not copy any dependencies. NOTE: if this option is
@@ -121,6 +140,8 @@ arv tag remove --all
h3(#arv-ws). arv ws
+This is a frontend to @arv-ws@.
+
@arv ws@ provides access to the websockets event stream.
@@ -149,7 +170,7 @@ optional arguments:
h3(#arv-keep). arv keep
-@arv keep@ provides access to the Keep storage service.
+@arv keep@ commands for accessing the Keep storage service.
@@ -157,96 +178,164 @@ $ arv keep --help
Usage: arv keep [method] [--parameters]
Use 'arv keep [method] --help' to get more information about specific methods.
-Available methods: ls, get, put, less, check, docker
+Available methods: ls, get, put, docker
-h3(#arv-pipeline-run). arv pipeline run
-
-@arv pipeline run@ can be used to start a pipeline run from the command line.
+h3(#arv-keep-ls). arv keep ls
-The User Guide has a page with a bit more information on "using arv pipeline run":{{site.baseurl}}/user/topics/running-pipeline-command-line.html.
+This is a frontend to @arv-ls@.
-$ arv pipeline run --help
-Options:
- --dry-run, -n: Do not start any new jobs or wait for existing jobs to
- finish. Just find out whether jobs are finished,
- queued, or running for each component.
- --status-text <s>: Store plain text status in given file. (Default:
- /dev/stdout)
- --status-json <s>: Store json-formatted pipeline in given file. (Default:
- /dev/null)
- --no-wait: Do not wait for jobs to finish. Just look up status,
- submit new jobs if needed, and exit.
- --no-reuse: Do not reuse existing jobs to satisfy pipeline
- components. Submit a new job for every component.
- --debug, -d: Print extra debugging information on stderr.
- --debug-level <i>: Set debug verbosity level.
- --template <s>: UUID of pipeline template, or path to local pipeline
- template file.
- --instance <s>: UUID of pipeline instance.
- --submit: Submit the pipeline instance to the server, and exit.
- Let the Crunch dispatch service satisfy the components
- by finding/running jobs.
- --run-pipeline-here: Manage the pipeline instance in-process. Submit jobs
- to Crunch as needed. Do not exit until the pipeline
- finishes (or fails).
- --run-jobs-here: Run jobs in the local terminal session instead of
- submitting them to Crunch. Implies
- --run-pipeline-here. Note: this results in a
- significantly different job execution environment, and
- some Crunch features are not supported. It can be
- necessary to modify a pipeline in order to make it run
- this way.
- --run-here: Synonym for --run-jobs-here.
- --description <s>: Description for the pipeline instance.
- --version, -v: Print version and exit
- --help, -h: Show this message
+$ arv keep ls --help
+usage: arv-ls [-h] [--retries RETRIES] [-s] locator
+
+List contents of a manifest
+
+positional arguments:
+ locator Collection UUID or locator
+
+optional arguments:
+ -h, --help show this help message and exit
+ --retries RETRIES Maximum number of times to retry server requests that
+ encounter temporary failures (e.g., server down). Default
+ 3.
+ -s List file sizes, in KiB.
-h3(#arv-run). arv run
+h3(#arv-keep-get). arv keep get
+
+This is a frontend to @arv-get@.
-The @arv-run@ command creates Arvados pipelines at the command line that fan out to multiple concurrent tasks across Arvado compute nodes.
+
+
+$ arv keep get --help
+usage: arv-get [-h] [--retries RETRIES]
+ [--progress | --no-progress | --batch-progress]
+ [--hash HASH | --md5sum] [-n] [-r] [-f | --skip-existing]
+ locator [destination]
-The User Guide has a page on "using arv-run":{{site.baseurl}}/user/topics/arv-run.html.
+Copy data from Keep to a local file or pipe.
+
+positional arguments:
+ locator Collection locator, optionally with a file path or
+ prefix.
+ destination Local file or directory where the data is to be written.
+ Default: /dev/stdout.
+
+optional arguments:
+ -h, --help show this help message and exit
+ --retries RETRIES Maximum number of times to retry server requests that
+ encounter temporary failures (e.g., server down). Default
+ 3.
+ --progress Display human-readable progress on stderr (bytes and, if
+ possible, percentage of total data size). This is the
+ default behavior when it is not expected to interfere
+ with the output: specifically, stderr is a tty _and_
+ either stdout is not a tty, or output is being written to
+ named files rather than stdout.
+ --no-progress Do not display human-readable progress on stderr.
+ --batch-progress Display machine-readable progress on stderr (bytes and,
+ if known, total data size).
+ --hash HASH Display the hash of each file as it is read from Keep,
+ using the given hash algorithm. Supported algorithms
+ include md5, sha1, sha224, sha256, sha384, and sha512.
+ --md5sum Display the MD5 hash of each file as it is read from
+ Keep.
+ -n Do not write any data -- just read from Keep, and report
+ md5sums if requested.
+ -r Retrieve all files in the specified collection/prefix.
+ This is the default behavior if the "locator" argument
+ ends with a forward slash.
+ -f Overwrite existing files while writing. The default
+ behavior is to refuse to write *anything* if any of the
+ output files already exist. As a special case, -f is not
+ needed to write to /dev/stdout.
+ --skip-existing Skip files that already exist. The default behavior is to
+ refuse to write *anything* if any files exist that would
+ have to be overwritten. This option causes even devices,
+ sockets, and fifos to be skipped.
+
+
+
+h3(#arv-keep-put). arv keep put
+
+This is a frontend to @arv-put@.
-$ arv run --help
-usage: arv-run [-h] [--retries RETRIES] [--dry-run] [--local]
- [--docker-image DOCKER_IMAGE] [--ignore-rcode] [--no-reuse]
- [--no-wait] [--project-uuid PROJECT_UUID] [--git-dir GIT_DIR]
- [--repository REPOSITORY] [--script-version SCRIPT_VERSION]
- ...
+$ arv keep put --help
+usage: arv-put [-h] [--max-manifest-depth N | --normalize]
+ [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]
+ [--use-filename FILENAME] [--filename FILENAME]
+ [--portable-data-hash] [--replication N]
+ [--project-uuid UUID] [--name NAME]
+ [--progress | --no-progress | --batch-progress]
+ [--resume | --no-resume] [--retries RETRIES]
+ [path [path ...]]
+
+Copy data from the local filesystem to Keep.
positional arguments:
- args
+ path Local file or directory. Default: read from standard
+ input.
optional arguments:
-h, --help show this help message and exit
+ --max-manifest-depth N
+ Maximum depth of directory tree to represent in the
+ manifest structure. A directory structure deeper than
+ this will be represented as a single stream in the
+ manifest. If N=0, the manifest will contain a single
+ stream. Default: -1 (unlimited), i.e., exactly one
+ manifest stream per filesystem directory that contains
+ files.
+ --normalize Normalize the manifest by re-ordering files and
+ streams after writing data.
+ --as-stream Synonym for --stream.
+ --stream Store the file content and display the resulting
+ manifest on stdout. Do not write the manifest to Keep
+ or save a Collection object in Arvados.
+ --as-manifest Synonym for --manifest.
+ --in-manifest Synonym for --manifest.
+ --manifest Store the file data and resulting manifest in Keep,
+ save a Collection object in Arvados, and display the
+ manifest locator (Collection uuid) on stdout. This is
+ the default behavior.
+ --as-raw Synonym for --raw.
+ --raw Store the file content and display the data block
+ locators on stdout, separated by commas, with a
+ trailing newline. Do not store a manifest.
+ --use-filename FILENAME
+ Synonym for --filename.
+ --filename FILENAME Use the given filename in the manifest, instead of the
+ name of the local file. This is useful when "-" or
+ "/dev/stdin" is given as an input file. It can be used
+ only if there is exactly one path given and it is not
+ a directory. Implies --manifest.
+ --portable-data-hash Print the portable data hash instead of the Arvados
+ UUID for the collection created by the upload.
+ --replication N Set the replication level for the new collection: how
+ many different physical storage devices (e.g., disks)
+ should have a copy of each data block. Default is to
+ use the server-provided default (if any) or 2.
+ --project-uuid UUID Store the collection in the specified project, instead
+ of your Home project.
+ --name NAME Save the collection with the specified name.
+ --progress Display human-readable progress on stderr (bytes and,
+ if possible, percentage of total data size). This is
+ the default behavior when stderr is a tty.
+ --no-progress Do not display human-readable progress on stderr, even
+ if stderr is a tty.
+ --batch-progress Display machine-readable progress on stderr (bytes
+ and, if known, total data size).
+ --resume Continue interrupted uploads from cached state
+ (default).
+ --no-resume Do not continue interrupted uploads from cached state.
--retries RETRIES Maximum number of times to retry server requests that
encounter temporary failures (e.g., server down).
Default 3.
- --dry-run Print out the pipeline that would be submitted and
- exit
- --local Run locally using arv-run-pipeline-instance
- --docker-image DOCKER_IMAGE
- Docker image to use, default arvados/jobs
- --ignore-rcode Commands that return non-zero return codes should not
- be considered failed.
- --no-reuse Do not reuse past jobs.
- --no-wait Do not wait and display logs after submitting command,
- just exit.
- --project-uuid PROJECT_UUID
- Parent project of the pipeline
- --git-dir GIT_DIR Git repository passed to arv-crunch-job when using
- --local
- --repository REPOSITORY
- repository field of component, default 'arvados'
- --script-version SCRIPT_VERSION
- script_version field of component, default 'master'