We will start with downloading a freely available VCF file from the "Personal Genome Project (PGP)":http://www.personalgenomes.org subject "hu599905":https://my.personalgenomes.org/profile/hu599905 to a staging directory on the VM, and then add it to Keep.
+In the following tutorials, replace <b><code>you</code></b> with your user id.
+
First, log into the Arvados VM instance and set up the staging area:
-notextile. <pre><code>$ <span class="userinput">mkdir /scratch/<b>you</b></span></code></pre>
+notextile. <pre><code>~$ <span class="userinput">mkdir /scratch/<b>you</b></span></code></pre>
Next, download the file:
<notextile>
-<pre><code>$ <span class="userinput">mkdir /scratch/<b>you</b></span>
-$ <span class="userinput">cd /scratch/<b>you</b></span>
-$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.personalgenomes.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
+<pre><code>~$ <span class="userinput">cd /scratch/<b>you</b></span>
+/scratch/<b>you</b>$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.personalgenomes.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 216M 100 216M 0 0 10.0M 0 0:00:21 0:00:21 --:--:-- 9361k
</code></pre>
</notextile>
-Alternately, if you have your own data, for example @MyData.vcf@, you can use @scp@ or @rsync@ to copy from your local workstation to the shell VM (run this on your local workstation):
+{% include 'notebox_begin' %}
+
+If you have your own data, for example @MyData.vcf@, you can use @scp@ or @rsync@ to copy from your local workstation to the shell VM (run this on your local workstation):
+
+notextile. <pre><code>~$ <span class="userinput">scp MyData.vcf <b>you@shell</b>.arvados:/scratch/<b>you</b>/MyData.vcf</span></code></pre>
-notextile. <pre><code>$ <span class="userinput">scp MyData.vcf <b>you@shell</b>.arvados:/scratch/<b>you</b>/MyData.vcf</span></code></pre>
+{% include 'notebox_end' %}
-Now use @arv keep put@ to add your VCF data to Keep:
+Now use @arv keep put@ to add your VCF data to Keep, then delete the local copy of the file:
<notextile>
-<pre><code>$ <span class="userinput">cd /scratch/<b>you</b></span>
-$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep put var-GS000016015-ASM.tsv.bz2</span>
c1bad4b39ca5a924e481008009d94e32+210
+/scratch/<b>you</b>$ <span class="userinput">rm var-GS000016015-ASM.tsv.bz2</span>
</code></pre>
</notextile>
-The output value @c1bad4b39ca5a924e481008009d94e32+210@ is the Keep locator. This enables you to access the file you just uploaded, and is explained in the next section.
+The output value @c1bad4b39ca5a924e481008009d94e32+210@ from @arv keep put@ is the Keep locator. This enables you to access the file you just uploaded, and is explained in the next section.
h2. Putting a directory
You can also use @arv keep put@ to add an entire directory:
<notextile>
-<pre><code>$ <span class="userinput">mkdir tmp</span>
-$ <span class="userinput">echo "hello alice" > tmp/alice.txt</span>
-$ <span class="userinput">echo "hello bob" > tmp/bob.txt</span>
-$ <span class="userinput">echo "hello carol" > tmp/carol.txt</span>
-$ <span class="userinput">arv keep put tmp</span>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">mkdir tmp</span>
+/scratch/<b>you</b>$ <span class="userinput">echo "hello alice" > tmp/alice.txt</span>
+/scratch/<b>you</b>$ <span class="userinput">echo "hello bob" > tmp/bob.txt</span>
+/scratch/<b>you</b>$ <span class="userinput">echo "hello carol" > tmp/carol.txt</span>
+/scratch/<b>you</b>$ <span class="userinput">arv keep put tmp</span>
0M / 0M 100.0%
887cd41e9c613463eab2f0d885c6dd96+83
</code></pre>
In this example we will use @c1bad4b39ca5a924e481008009d94e32+210@ which we added to keep in the previous section. First let us examine the contents of this collection using @arv keep get@:
<notextile>
-<pre><code>$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210</span>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210</span>
. 204e43b8a1185621ca55a94839582e6f+67108864 b9677abbac956bd3e86b1deb28dfac03+67108864 fc15aff2a762b13f521baf042140acec+67108864 323d2a3ce20370c4ca1d3462a344f8fd+25885655 0:227212247:var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
-@arv keep get@ fetches the contents of the locator @c1bad4b39ca5a924e481008009d94e32+210@. This is a locator for a collection data block, so it fetches the contents of the collection. In this example, this collection consists of a single file @var-GS000016015-ASM.tsv.bz2@ which is 227212247 bytes long, and is stored using four sequential data blocks, <code>204e43b8a1185621ca55a94839582e6f+67108864</code>, <code>b9677abbac956bd3e86b1deb28dfac03+67108864</code>, <code>fc15aff2a762b13f521baf042140acec+67108864</code>, <code>323d2a3ce20370c4ca1d3462a344f8fd+25885655</code>.
+The command @arv keep get@ fetches the contents of the locator @c1bad4b39ca5a924e481008009d94e32+210@. This is a locator for a collection data block, so it fetches the contents of the collection. In this example, this collection consists of a single file @var-GS000016015-ASM.tsv.bz2@ which is 227212247 bytes long, and is stored using four sequential data blocks, <code>204e43b8a1185621ca55a94839582e6f+67108864</code>, <code>b9677abbac956bd3e86b1deb28dfac03+67108864</code>, <code>fc15aff2a762b13f521baf042140acec+67108864</code>, <code>323d2a3ce20370c4ca1d3462a344f8fd+25885655</code>.
Let's use @arv keep get@ to download the first datablock:
-notextile. <pre><code>$ <span class="userinput">arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
+notextile. <pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get 204e43b8a1185621ca55a94839582e6f+67108864 > block1</span></code></pre>
Let's look at the size and compute the md5 hash of @block1@:
<notextile>
-<pre><code>$ <span class="userinput">ls -l block1</span>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">ls -l block1</span>
-rw-r--r-- 1 you group 67108864 Dec 9 20:14 block1
-$ <span class="userinput">md5sum block1</span>
+/scratch/<b>you</b>$ <span class="userinput">md5sum block1</span>
204e43b8a1185621ca55a94839582e6f block1
</code></pre>
</notextile>
Next, let's use @arv keep get@ to download and reassemble @var-GS000016015-ASM.tsv.bz2@ using the following command:
-notextile. <pre><code>$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2 .</span></code></pre>
+<notextile>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep get c1bad4b39ca5a924e481008009d94e32+210/var-GS000016015-ASM.tsv.bz2 .</span>
+</code></pre>
This downloads the file @var-GS000016015-ASM.tsv.bz2@ described by collection @c1bad4b39ca5a924e481008009d94e32+210@ from Keep and places it into the local directory. Now that we have the file, we can compute the md5 hash of the complete file:
<notextile>
-<pre><code>$ <span class="userinput">md5sum var-GS000016015-ASM.tsv.bz2</span>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">md5sum var-GS000016015-ASM.tsv.bz2</span>
44b8ae3fde7a8a88d2f7ebd237625b4f var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
There are a couple of other ways to access a collection. You may view the contents of a collection using @arv keep ls@:
<notextile>
-<pre><code>$ <span class="userinput">arv keep ls c1bad4b39ca5a924e481008009d94e32+210</span>
+<pre><code>/scratch/<b>you</b>$ <span class="userinput">arv keep ls c1bad4b39ca5a924e481008009d94e32+210</span>
var-GS000016015-ASM.tsv.bz2
-$ <span class="userinput">arv keep ls -s c1bad4b39ca5a924e481008009d94e32+210</span>
+/scratch/<b>you</b>$ <span class="userinput">arv keep ls -s c1bad4b39ca5a924e481008009d94e32+210</span>
221887 var-GS000016015-ASM.tsv.bz2
</code></pre>
</notextile>
projects / arvados.git / blobdiff