---
layout: default
navsection: userguide
-title: "Using Arv-copy"
+title: "Using arv-copy"
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
This tutorial describes how to copy Arvados objects from one cluster to another by using @arv-copy@.
{% include 'tutorial_expectations' %}
-h2. Arv-copy
+h2. arv-copy
-@arv-copy@ allows users to copy collections, pipeline templates, and pipeline instances, including all their dependencies from one cluster to another.
+@arv-copy@ allows users to copy collections, workflow definitions and projects from one cluster to another.
-For example, lets copy from *cluster1* to *cluster2*. The names *cluster1* and *cluster2* are interchangable with any cluster name.
+For projects, @arv-copy@ will copy all the collections workflow definitions owned by the project, and recursively copy subprojects.
-In order for the clusters to be able to communicate with each other, you must create custom configuration files for both clusters. First, go to your Manage Account page in Workbench and copy the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ in both of your clusters. Then, create two configuration files, one for each cluster. The names of the files is very important, they must match the cluster names.
+For workflow definitions, @arv-copy@ will recursively go through the workflow and copy all associated dependencies (input collections and Docker images).
+For example, let's copy from the <a href="https://playground.arvados.org/">Arvados playground</a>, also known as *pirca*, to *dstcl*. The names *pirca* and *dstcl* are interchangable with any cluster id. You can find the cluster name from the prefix of the uuid of the object you want to copy. For example, in *zzzzz*-4zz18-tci4vn4fa95w0zx, the cluster name is *zzzzz* .
+
+In order to communicate with both clusters, you must create custom configuration files for each cluster. The "Getting an API token":{{site.baseurl}}/user/reference/api-tokens.html page describes how to get a token and create a configuration file. However, instead of "settings.conf" in @~/.config/arvados@ you need two configuration files, one for each cluster, with filenames in the format of *ClusterID.conf*.
+
+In this example, navigate to the *Current token* page on each of *pirca* and *dstcl* to get the @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@.
+
+The config file consists of two lines, one for ARVADOS_API_HOST and one for ARVADOS_API_TOKEN:
+
+<pre>
+ARVADOS_API_HOST=zzzzz.arvadosapi.com
+ARVADOS_API_TOKEN=v2/zzzzz-gj3su-xxxxxxxxxxxxxxx/123456789abcdefghijkl
+</pre>
+
+Copy your @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ into the config files as shown below in the shell account from which you are executing the commands. In our example, you need two files, @~/.config/arvados/pirca.conf@ and @~/.config/arvados/dstcl.conf@.
+
+Now you're ready to copy between *pirca* and *dstcl*!
+
+h3. How to copy a collection
+
+First, determine the uuid or portable data hash of the collection you want to copy from the source cluster. The uuid can be found in the collection display page in the collection summary area (top left box), or from the URL bar (the part after @collections/...@)
+
+Now copy the collection from *pirca* to *dstcl*. We will use the uuid @jutro-4zz18-tv416l321i4r01e@ as an example. You can find this collection on <a href="https://playground.arvados.org/collections/jutro-4zz18-tv416l321i4r01e">playground.arvados.org</a>.
<notextile>
-<pre><code>~$ <span class="userinput">cd ~/.config/arvados</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=cluster1.arvadosapi.com" >> cluster1.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=123456789abcdefghijkl" >> cluster1.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_HOST=cluster2.arvadosapi.com" >> cluster2.conf</span>
-~$ <span class="userinput">echo "ARVADOS_API_TOKEN=987654321lkjihgfedcba" >> cluster2.conf</span>
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl jutro-4zz18-tv416l321i4r01e</span>
+jutro-4zz18-tv416l321i4r01e: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-h3. How to copy a collection
+You can also copy by content address:
-First, select the uuid of the collection you want to copy from the source cluster. The uuid can be found inside of a collection in the top left box, or from the URL bar (the part after @collections/...@)
+<notextile>
+<pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl 2463fa9efeb75e099685528b3b9071e0+438</span>
+2463fa9efeb75e099685528b3b9071e0+438: 6.1M / 6.1M 100.0%
+arvados.arv-copy[1234] INFO: Success: created copy with uuid dstcl-4zz18-xxxxxxxxxxxxxxx
+</code></pre>
+</notextile>
+
+The output of arv-copy displays the uuid of the collection generated in the destination cluster. By default, the output is placed in your home project in the destination cluster. If you want to place your collection in an existing project, you can specify the project you want it to be in using the tag @--project-uuid@ followed by the project uuid.
+
+For example, this will copy the collection to project @dstcl-j7d0g-a894213ukjhal12@ in the destination cluster.
+
+<notextile> <pre><code>~$ <span class="userinput">arv-copy --src pirca --dst dstcl --project-uuid dstcl-j7d0g-a894213ukjhal12 jutro-4zz18-tv416l321i4r01e
+</code></pre>
+</notextile>
+
+Additionally, if you need to specify the storage classes where to save the copied data on the destination cluster, you can do that by using the @--storage-classes LIST@ argument, where @LIST@ is a comma-separated list of storage class names.
+
+h3. How to copy a workflow
+
+We will use the uuid @jutro-7fd4e-mkmmq53m1ze6apx@ as an example workflow.
-Now copy the collection from *cluster1* to *cluster2*. We will use the uuid @cluster1-4zz18-1234567890abcde@ as an example.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src cluster1 --dst cluster2 cluster1-4zz18-1234567890abcde</span>
-cluster1-4zz18-1234567890abcde: 100M / 100M 100.0%
-arvados.arv-copy[1234] INFO: Success: created copy with uuid cluster2-4zz18-8765943210cdbae
+<pre><code>~$ <span class="userinput">arv-copy --src jutro --dst pirca --project-uuid pirca-j7d0g-ecak8knpefz8ere jutro-7fd4e-mkmmq53m1ze6apx</span>
+ae480c5099b81e17267b7445e35b4bc7+180: 23M / 23M 100.0%
+2463fa9efeb75e099685528b3b9071e0+438: 156M / 156M 100.0%
+jutro-4zz18-vvvqlops0a0kpdl: 94M / 94M 100.0%
+2020-08-19 17:04:13 arvados.arv-copy[4789] INFO:
+2020-08-19 17:04:13 arvados.arv-copy[4789] INFO: Success: created copy with uuid pirca-7fd4e-s0tw9rfbkpo2fmx
</code></pre>
</notextile>
-The collection will be placed in your home project in the destination cluster. If you want to place your collection inside of a pre-created project, you can specify the project you want it to be in using the tag @--project-uuid@ followed by the project uuid.
+The name, description, and workflow definition from the original workflow will be used for the destination copy. In addition, any *collections* and *docker images* referenced in the source workflow definition will also be copied to the destination.
-h3. How to copy a pipeline template or pipeline instance
+If you would like to copy the object without dependencies, you can use the @--no-recursive@ flag.
-{% include 'arv_copy_expectations' %}
+h3. How to copy a project
-We will use the uuid @cluster1-p5p6p-abcd123efghi45jkl@ as an example pipeline template or instance, and @samplegitrepo/samplename.git@ as an example git repo.
+We will use the uuid @jutro-j7d0g-xj19djofle3aryq@ as an example project.
<notextile>
-<pre><code>~$ <span class="userinput">arv-copy --src cluster1 --dst cluster2 --dst-git-repo samplegitrepo/samplename cluster1-p5p6p-abcd123efghi45jkl</span>
+<pre><code>~$ <span class="userinput">peteramstutz@shell:~$ arv-copy --project-uuid pirca-j7d0g-lr8sq3tx3ovn68k jutro-j7d0g-xj19djofle3aryq
+2021-09-08 21:29:32 arvados.arv-copy[6377] INFO:
+2021-09-08 21:29:32 arvados.arv-copy[6377] INFO: Success: created copy with uuid pirca-j7d0g-ig9gvu5piznducp
</code></pre>
</notextile>
-New branches in the destination git repo will be created for each branch used in the pipeline template. For example, if your source branch was named branch_name, your new branch will be named @git_git_cluster1_arvadosapi_com_reponame_git_branch_name@.
+The name and description of the original project will be used for the destination copy. If a project already exists with the same name, collections and workflow definitions will be copied into the project with the same name.
+
+If you would like to copy the project but not its subproject, you can use the @--no-recursive@ flag.
projects / arvados.git / blobdiff