---
layout: default
navsection: userguide
-title: "How arv-copy works"
+title: "Using arv-copy"
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
This tutorial describes how to copy Arvados objects from one cluster to another by using @arv-copy@.
{% include 'tutorial_expectations' %}
-Also note that they must work from a machine with an SSH key that is allowed to access both clusters (required to copy git repositories).
+h2. arv-copy
-h2. Arv-copy
+@arv-copy@ allows users to copy collections and workflows from one cluster to another. By default, @arv-copy@ will recursively go through the workflow and copy all dependencies associated with the object.
-@arv-copy@ allows users to copy collections, pipeline templates, and pipeline instances, including all their dependencies from one cluster to another.
+For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *pirca*, to *dstcl*. The names *pirca* and *dstcl* are interchangable with any cluster id. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *zzzzz*-4zz18-tci4vn4fa95w0zx, the cluster name is *zzzzz* .
-Suppose we want to copy from one cloud cluster to another. For example cluster "one" to cluster "two". The names "one" and "two" are interchangable with any cluster name.
+In order to communicate with both clusters, you must create custom configuration files for each cluster. In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*. Copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files in @~/.config/arvados@, one for each cluster. The names of the files must have the format of *ClusterID.conf*. Navigate to the *Current token* page on each of *pirca* and *dstcl* to get the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@.
-First we have to make configuration files for both clusters.
+!{display: block;margin-left: 25px;margin-right: auto;}{{ site.baseurl }}/images/api-token-host.png!
-<notextile>
-<pre><code>~$ <span class="userinput">cd ~/.config/arvados</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=one.arvadosapi.com" >> one.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=123456789abcdefghijkl" >> one.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=two.arvadosapi.com" >> two.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=987654321lkjihgfedcba" >> two.conf</span>
-</code></pre>
-</notextile>
+The config file consists of two lines, one for ARVADOS_API_HOST and one for ARVADOS_API_TOKEN:
-Your ARVADOS_API_TOKEN can be found in the manage accounts page in Workbench under Current Token.
+<pre>
+ARVADOS_API_HOST=zzzzz.arvadosapi.com
+ARVADOS_API_TOKEN=v2/zzzzz-gj3su-xxxxxxxxxxxxxxx/123456789abcdefghijkl
+</pre>
-Once you have your config files ready, you can start copying!
+Copy your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ into the config files as shown below in the shell account from which you are executing the commands. In our example, you need two files, @~/.config/arvados/pirca.conf@ and @~/.config/arvados/dstcl.conf@.
-Here are some examples:
+Now you're ready to copy between *pirca* and *dstcl*!
h3. How to copy a collection
-First, copy the uuid of the collection you want to copy from the cluster you want to copy from. The uuid can be found inside of a collection in the top left box, or from the URL bar (the part after @collections/...@)
+First, determine the uuid or portable data hash of the collection you want to copy from the source cluster. The uuid can be found in the collection display page in the collection summary area (top left box), or from the URL bar (the part after @collections/...@)
-Now copy the collection from cluster one to cluster two. We will use the uuid one-4zz18-1234567890abcde as an example.
+Now copy the collection from *pirca* to *dstcl*. We will use the uuid @jutro-4zz18-tv416l321i4r01e@ as an example. You can find this collection on <a href="https://playground.arvados.org/collections/jutro-4zz18-tv416l321i4r01e">playground.arvados.org</a>.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src one --dst two one-4zz18-1234567890abcde</span>
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl jutro-4zz18-tv416l321i4r01e</span>
+jutro-4zz18-tv416l321i4r01e: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-The output should be like this:
+You can also copy by content address:
<notextile>
-<pre><code>
-one-4zz18-1234567890abcde: 100M / 100M 100.0%
-arvados.arv-copy[1234] INFO: Success: created copy with uuid two-4zz18-8765943210cdbae
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl 2463fa9efeb75e099685528b3b9071e0+438</span>
+2463fa9efeb75e099685528b3b9071e0+438: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-The collection will be placed in your home project in the destination cluster. If you want to place your collection inside of a pre-created project, you can specify the project you want it to be in using the tag @--project-uuid@
+The output of arv-copy displays the uuid of the collection generated in the destination cluster. By default, the output is placed in your home project in the destination cluster. If you want to place your collection in an existing project, you can specify the project you want it to be in using the tag @--project-uuid@ followed by the project uuid.
-h3. How to copy a pipeline template or pipeline instance
+For example, this will copy the collection to project dstcl-j7d0g-a894213ukjhal12 in the destination cluster.
-When copying a pipeline template or pipeline instance recursively (copying all dependencies in the template/instance) you must have a git repository in the destination cluster.
+<notextile> <pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --project-uuid dstcl-j7d0g-a894213ukjhal12 jutro-4zz18-tv416l321i4r01e
+</code></pre>
+</notextile>
-If you do not have a repository created, you can follow the documentation here: (Point to #6014)
+h3. How to copy a workflow
-Once you have your repository created, you can copy your pipeline template/instance. We will use the uuid one-p5p6p-abcd123efghi45jkl as an example, and samplegitrepo/samplename.git as an example git repo.
+We will use the uuid @jutro-7fd4e-mkmmq53m1ze6apx@ as an example workflow.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src one --dst two --dst-git-repo samplegitrepo/samplename.git one-p5p6p-abcd123efghi45jkl</span>
+<pre><code>~$ <span class="userinput">arv-copy --src jutro --dst pirca --project-uuid pirca-j7d0g-ecak8knpefz8ere jutro-7fd4e-mkmmq53m1ze6apx</span>
+ae480c5099b81e17267b7445e35b4bc7+180: 23M / 23M 100.0%
+2463fa9efeb75e099685528b3b9071e0+438: 156M / 156M 100.0%
+jutro-4zz18-vvvqlops0a0kpdl: 94M / 94M 100.0%
+2020-08-19 17:04:13 arvados.arv-copy[4789] INFO:
+2020-08-19 17:04:13 arvados.arv-copy[4789] INFO: Success: created copy with uuid pirca-7fd4e-s0tw9rfbkpo2fmx
</code></pre>
</notextile>
-New branches in the destination git repo will be created for each branch used in the pipeline template. For example, if your source branch was named branch_name, your new branch will be named git_git_one_arvadosapi_com_reponame_git_branch_name.
+The name, description, and workflow definition from the original workflow will be used for the destination copy. In addition, any *collections* and *docker images* referenced in the source workflow definition will also be copied to the destination.
+
+If you would like to copy the object without dependencies, you can use the @--no-recursive@ flag.
projects / arvados.git / blobdiff