X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/430d4da65c609d1fde6e3cd428e7591b58cfd9e9..f2060a005a084ff544d9479ea50bb8a02fe6a31d:/doc/user/tutorials/tutorial-keep.html.textile.liquid diff --git a/doc/user/tutorials/tutorial-keep.html.textile.liquid b/doc/user/tutorials/tutorial-keep.html.textile.liquid index e02082388f..53cdfe4a08 100644 --- a/doc/user/tutorials/tutorial-keep.html.textile.liquid +++ b/doc/user/tutorials/tutorial-keep.html.textile.liquid @@ -1,143 +1,85 @@ --- layout: default navsection: userguide -navmenu: Tutorials -title: "Storing and Retrieving data using Arvados Keep" - +title: "Uploading data" ... +{% comment %} +Copyright (C) The Arvados Authors. All rights reserved. -h1. Storing and Retrieving data using Arvados Keep - -This tutorial introduces you to the Arvados file storage system. - - -*This tutorial assumes that you are "logged into an Arvados VM instance":{{site.basedoc}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.basedoc}}/user/getting_started/check-environment.html* +SPDX-License-Identifier: CC-BY-SA-3.0 +{% endcomment %} -The Arvados distributed file system is called *Keep*. Keep is a content-addressable file system. This means that files are managed using special unique identifiers derived from the _contents_ of the file, rather than human-assigned file names (specifically, the md5 hash). This has a number of advantages: -* Files can be stored and replicated across a cluster of servers without requiring a central name server. -* Systematic validation of data integrity by both server and client because the checksum is built into the identifier. -* Minimizes data duplication (two files with the same contents will result in the same identifier, and will not be stored twice.) -* Avoids data race conditions (an identifier always points to the same data.) +Arvados Data collections can be uploaded using either the @arv keep put@ command line tool or using Workbench. -h1. Putting Data into Keep +# "*Upload using command line tool*":#upload-using-command +# "*Upload using Workbench*":#upload-using-workbench -We will start with downloading a freely available VCF file from the "Personal Genome Project (PGP)":http://www.personalgenomes.org subject "hu599905":https://my.personalgenomes.org/profile/hu599905 to a staging directory on the VM, and then add it to Keep. +notextile.
you
with your user id.
+h2(#upload-using-command). Upload using command line tool
-First, log into the Arvados VM instance and set up the staging area:
-
-notextile. ~$ mkdir /scratch/you
-
-Next, download the file:
+{% include 'tutorial_expectations' %}
+To upload a file to Keep using @arv keep put@:
~$ cd /scratch/you
-/scratch/you$ curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.personalgenomes.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'
- % Total % Received % Xferd Average Speed Time Time Time Current
- Dload Upload Total Spent Left Speed
-100 216M 100 216M 0 0 10.0M 0 0:00:21 0:00:21 --:--:-- 9361k
+~$ arv keep put var-GS000016015-ASM.tsv.bz2
+216M / 216M 100.0%
+Collection saved as ...
+qr1hi-4zz18-xxxxxxxxxxxxxxx
~$ scp MyData.vcf you@shell.arvados:/scratch/you/MyData.vcf
-{% include 'notebox_end' %}
+The output value @qr1hi-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
-Now use @arv keep put@ to add your VCF data to Keep, then delete the local copy of the file:
+Note: The file used in this example is a freely available TSV file containing variant annotations from the "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905":https://my.pgp-hms.org/profile/hu599905), downloadable "here":https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2. Alternatively, you can replace @var-GS000016015-ASM.tsv.bz2@ with the name of any file you have locally, or you could get the TSV file by "downloading it from Keep.":{{site.baseurl}}/user/tutorials/tutorial-keep-get.html
-/scratch/you$ arv keep put var-GS000016015-ASM.tsv.bz2
-c1bad4b39ca5a924e481008009d94e32+210
-/scratch/you$ rm var-GS000016015-ASM.tsv.bz2
-
-/scratch/you$ mkdir tmp
-/scratch/you$ echo "hello alice" > tmp/alice.txt
-/scratch/you$ echo "hello bob" > tmp/bob.txt
-/scratch/you$ echo "hello carol" > tmp/carol.txt
-/scratch/you$ arv keep put tmp
-0M / 0M 100.0%
-887cd41e9c613463eab2f0d885c6dd96+83
+~$ mkdir tmp
+~$ echo "hello alice" > tmp/alice.txt
+~$ echo "hello bob" > tmp/bob.txt
+~$ echo "hello carol" > tmp/carol.txt
+~$ arv keep put tmp
+0M / 0M 100.0%
+Collection saved as ...
+qr1hi-4zz18-yyyyyyyyyyyyyyy
/scratch/you$ arv keep get c1bad4b39ca5a924e481008009d94e32+210
-. 204e43b8a1185621ca55a94839582e6f+67108864 b9677abbac956bd3e86b1deb28dfac03+67108864 fc15aff2a762b13f521baf042140acec+67108864 323d2a3ce20370c4ca1d3462a344f8fd+25885655 0:227212247:var-GS000016015-ASM.tsv.bz2
-
-204e43b8a1185621ca55a94839582e6f+67108864
, b9677abbac956bd3e86b1deb28dfac03+67108864
, fc15aff2a762b13f521baf042140acec+67108864
, 323d2a3ce20370c4ca1d3462a344f8fd+25885655
.
+h3. Locate your collection in Workbench
-Let's use @arv keep get@ to download the first datablock:
+Visit the Workbench *Dashboard*. Click on *Projects* dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection name printed by @arv keep put@ will appear under the *name* column.
-notextile. /scratch/you$ arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1
+To move the collection to a different project, check the box at the left of the collection row. Pull down the *Selection...* menu near the top of the page tab, and select *Move selected...* button. This will open a dialog box where you can select a destination project for the collection. Click a project, then finally the Move button.
-Let's look at the size and compute the md5 hash of @block1@:
-
-/scratch/you$ ls -l block1
--rw-r--r-- 1 you group 67108864 Dec 9 20:14 block1
-/scratch/you$ md5sum block1
-204e43b8a1185621ca55a94839582e6f block1
-
-204e43b8a1185621ca55a94839582e6f+67108864
of:
-* the md5 hash @204e43b8a1185621ca55a94839582e6f@ which matches the md5 hash of @block1@
-* a size hint @67108864@ which matches the size of @block1@
+Click on the * Show* button next to the collection's listing on a project page to go to the Workbench page for your collection. On this page, you can see the collection's contents, download individual files, and set sharing options.
-Next, let's use @arv keep get@ to download and reassemble @var-GS000016015-ASM.tsv.bz2@ using the following command:
+notextile. /scratch/you$ arv keep get c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2 .
-
+h2(#upload-using-workbench). Upload using Workbench
-This downloads the file @var-GS000016015-ASM.tsv.bz2@ described by collection @c1bad4b39ca5a924e481008009d94e32+210@ from Keep and places it into the local directory. Now that we have the file, we can compute the md5 hash of the complete file:
+To upload using Workbench, visit the Workbench *Dashboard*. Click on *Projects* dropdown menu in the top navigation menu and select your *Home* project or any other project of your choosing. You will see the *Data collections* tab for this project, which lists the collections in this project.
-/scratch/you$ md5sum var-GS000016015-ASM.tsv.bz2
-44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
-
-/scratch/you$ arv keep ls c1bad4b39ca5a924e481008009d94e32+210
-var-GS000016015-ASM.tsv.bz2
-/scratch/you$ arv keep ls -s c1bad4b39ca5a924e481008009d94e32+210
-221887 var-GS000016015-ASM.tsv.bz2
-
-