---
layout: default
navsection: userguide
-title: "Storing and Retrieving data using Keep"
+title: "Uploading data"
...
-This tutorial introduces you to the Arvados file storage system.
+This tutorial describes how to to upload new Arvados data collections using the command line tool @arv-put@. This example uses a freely available TSV file containing variant annotations from "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905.":https://my.pgp-hms.org/profile/hu599905
+notextile. <div class="spaced-out">
-*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html*
-
-The Arvados distributed file system is called *Keep*. Keep is a content-addressable file system. This means that files are managed using special unique identifiers derived from the _contents_ of the file, rather than human-assigned file names (specifically, the md5 hash). This has a number of advantages:
-* Files can be stored and replicated across a cluster of servers without requiring a central name server.
-* Systematic validation of data integrity by both server and client because the checksum is built into the identifier.
-* Minimizes data duplication (two files with the same contents will result in the same identifier, and will not be stored twice.)
-* Avoids data race conditions (an identifier always points to the same data.)
-
-h1. Putting Data into Keep
-
-We will start with downloading a freely available VCF file from the "Personal Genome Project (PGP)":http://www.personalgenomes.org subject "hu599905":https://my.personalgenomes.org/profile/hu599905 to a staging directory on the VM, and then add it to Keep.
-
-In the following tutorials, replace <b><code>you</code></b> with your user id.
-
-First, log into the Arvados VM instance and set up the staging area:
-
-notextile. <pre><code>~$ <span class="userinput">mkdir /scratch/<b>you</b></span></code></pre>
-
-Next, download the file:
-
+# Begin by installing the "Arvados Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on the system from which you will upload the data (such as your workstation, or a server containing data from your sequencer). This will install the Arvados file upload tool, @arv-put@. Alternately, you can log into an Arvados VM (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login).
+# On system from which you will upload data, configure the environment with the Arvados instance host name and authentication token as decribed in "Getting an API token.":{{site.baseurl}}/user/reference/api-tokens.html (If you are logged into an Arvados VM, you can skip this step.)
+# Download the following example file. (If you are uploading your own data, you can skip this step.)
<notextile>
-<pre><code>~$ <span class="userinput">cd /scratch/<b>you</b></span>
-/scratch/<b>you</b>$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.personalgenomes.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
+<pre><code>~$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 216M 100 216M 0 0 10.0M 0 0:00:21 0:00:21 --:--:-- 9361k
</code></pre>
</notextile>
-
-{% include 'notebox_begin' %}
-
-If you have your own data, for example @MyData.vcf@, you can use @scp@ or @rsync@ to copy from your local workstation to the shell VM (run this on your local workstation):
-
-notextile. <pre><code>~$ <span class="userinput">scp MyData.vcf <b>you@shell</b>.arvados:/scratch/<b>you</b>/MyData.vcf</span></code></pre>
-
-{% include 'notebox_end' %}
-
-Now use @arv keep put@ to add your VCF data to Keep, then delete the local copy of the file:
-
+# Now upload the file to Keep using @arv-put@:
<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
+<pre><code>~$ <span class="userinput">arv-put var-GS000016015-ASM.tsv.bz2</span>
+216M / 216M 100.0%
c1bad4b39ca5a924e481008009d94e32+210
-/scratch/<b>you</b>$ <span class="userinput">rm var-GS000016015-ASM.tsv.bz2</span>
</code></pre>
</notextile>
-The output value @c1bad4b39ca5a924e481008009d94e32+210@ from @arv keep put@ is the Keep locator. This enables you to access the file you just uploaded, and is explained in the next section.
+* The output value @c1bad4b39ca5a924e481008009d94e32+210@ is the Arvados collection locator that uniquely describes this file.
-h2(#dir). Putting a directory
+Now go to the workbench collections page: <a href="https://{{ site.arvados_workbench_host }}/collections" target="_blank">https://{{ site.arvados_workbench_host }}/collections</a>. Your newly uploaded collection should appear near the top, with the value in the *uuid* column matching the Arvados collection locator that was printed by @arv-put@. Click on the *<i class="fa fa-fw fa-archive"></i> Show* button to go to the workbench page for your collection. Alternately, you can paste the Arvados collection locator into the *Search* box of the collections page to find your collection.
-You can also use @arv keep put@ to add an entire directory:
+The show collection page allows you to view the contents of the collection, download files from the collection, and set sharing options. To put your collection into a project, click on <span class="btn btn-xs btn-primary" ><i class="fa fa-fw fa-folder"></i> Add to project...</span>. This will open a modal dialog allowing you to select a destination project for your collection.
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">mkdir tmp</span>
-/scratch/<b>you</b>$ <span class="userinput">echo "hello alice" > tmp/alice.txt</span>
-/scratch/<b>you</b>$ <span class="userinput">echo "hello bob" > tmp/bob.txt</span>
-/scratch/<b>you</b>$ <span class="userinput">echo "hello carol" > tmp/carol.txt</span>
-/scratch/<b>you</b>$ <span class="userinput">arv keep put tmp</span>
-0M / 0M 100.0%
-887cd41e9c613463eab2f0d885c6dd96+83
-</code></pre>
-</notextile>
-
-The locator @887cd41e9c613463eab2f0d885c6dd96+83@ represents a collection with multiple files.
-
-h1. Getting Data from Keep
-
-h2. Using Workbench
-
-You may access collections through the "Collections section of Arvados Workbench":https://{{ site.arvados_workbench_host }}/collections located at "https://{{ site.arvados_workbench_host }}/collections":https://{{ site.arvados_workbench_host }}/collections . You can also access individual collections and individual files within a collection. Some examples:
-
-* "https://{{ site.arvados_workbench_host }}/collections/c1bad4b39ca5a924e481008009d94e32+210":https://{{ site.arvados_workbench_host }}/collections/c1bad4b39ca5a924e481008009d94e32+210
-* "https://{{ site.arvados_workbench_host }}/collections/887cd41e9c613463eab2f0d885c6dd96+83/alice.txt":https://{{ site.arvados_workbench_host }}/collections/887cd41e9c613463eab2f0d885c6dd96+83/alice.txt
+notextile. </div>
-h2(#arv-get). Using arv-get
-
-You can view the contents of a collection using @arv keep ls@:
-
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep ls c1bad4b39ca5a924e481008009d94e32+210</span>
-var-GS000016015-ASM.tsv.bz2
-</code></pre>
-
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep ls 887cd41e9c613463eab2f0d885c6dd96+83</span>
-alice.txt
-bob.txt
-carol.txt
-</code></pre>
-</notextile>
-
-Use @-s@ to print file sizes rounded up to the nearest kilobyte:
-
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep ls -s c1bad4b39ca5a924e481008009d94e32+210</span>
-221887 var-GS000016015-ASM.tsv.bz2
-</code></pre>
-</notextile>
-
-Use @arv keep get@ to download the contents of a collection and place it in the directory specified in the second argument (in this example, @.@ for the current directory):
-
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210/ .</span>
-</code></pre>
-</notextile>
-
-You can also download individual files:
-
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get 887cd41e9c613463eab2f0d885c6dd96+83/alice.txt .</span>
-</code></pre>
-</notextile>
-
-With a local copy of the file, we can do some computation, for example computing the md5 hash of the complete file:
-
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">md5sum var-GS000016015-ASM.tsv.bz2</span>
-44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
-</code></pre>
-</notextile>
-
-h2. Using arv-mount
-
-Use @arv-mount@ to take advantage of the "File System in User Space / FUSE":http://fuse.sourceforge.net/ feature of the Linux kernel to mount a Keep collection as if it were a regular directory tree.
-
-<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">mkdir mnt</span>
-/scratch/<b>you</b>$ <span class="userinput">arv-mount --collection c1bad4b39ca5a924e481008009d94e32+210 mnt &</span>
-/scratch/<b>you</b>$ <span class="userinput">cd mnt</span>
-/scratch/<b>you</b>/mnt$ <span class="userinput">ls</span>
-var-GS000016015-ASM.tsv.bz2
-/scratch/<b>you</b>/mnt$ <span class="userinput">md5sum var-GS000016015-ASM.tsv.bz2</span>
-44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
-/scratch/<b>you</b>/mnt$ <span class="userinput">cd ..</span>
-/scratch/<b>you</b>$ <span class="userinput">fusermount -u mnt</span>
-</code></pre>
-</notextile>
+h2(#dir). Putting a directory
-You can also mount the entire Keep namespace in "magic directory" mode:
+If you give @arv-put@ a directory, it will recursively upload the entire directory:
<notextile>
-<pre><code>/scratch/<b>you</b>$ <span class="userinput">mkdir mnt</span>
-/scratch/<b>you</b>$ <span class="userinput">arv-mount mnt &</span>
-/scratch/<b>you</b>$ <span class="userinput">cd mnt/c1bad4b39ca5a924e481008009d94e32+210</span>
-/scratch/<b>you</b>/mnt/c1bad4b39ca5a924e481008009d94e32+210$ <span class="userinput">ls</span>
-var-GS000016015-ASM.tsv.bz2
-/scratch/<b>you</b>/mnt/c1bad4b39ca5a924e481008009d94e32+210$ <span class="userinput">md5sum var-GS000016015-ASM.tsv.bz2</span>
-44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
-/scratch/<b>you</b>/mnt/c1bad4b39ca5a924e481008009d94e32+210$ <span class="userinput">cd ../..</span>
-/scratch/<b>you</b>$ <span class="userinput">fusermount -u mnt</span>
+<pre><code>~$ <span class="userinput">mkdir tmp</span>
+~$ <span class="userinput">echo "hello alice" > tmp/alice.txt</span>
+~$ <span class="userinput">echo "hello bob" > tmp/bob.txt</span>
+~$ <span class="userinput">echo "hello carol" > tmp/carol.txt</span>
+~$ <span class="userinput">arv-put tmp</span>
+0M / 0M 100.0%
+887cd41e9c613463eab2f0d885c6dd96+83
</code></pre>
</notextile>
-
-Using @arv-mount@ has several significant benefits:
-
-* You can browse, open and read Keep entries as if they are regular files.
-* It is easy for existing tools to access files in Keep.
-* Data is downloaded on demand, it is not necessary to download an entire file or collection to start processing
projects / arvados.git / blobdiff