navsection: userguide
title: "Uploading data"
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
-This tutorial describes how to to upload new Arvados data collections using the command line tool @arv-put@. This example uses a freely available TSV file containing variant annotations from "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905.":https://my.pgp-hms.org/profile/hu599905
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+Arvados Data collections can be uploaded using either Workbench or the @arv-put@ command line tool.
+
+# "*Upload using Workbench*":#upload-using-workbench
+# "*Upload using command line tool*":#upload-using-command
+
+h2(#upload-using-workbench). Upload using Workbench
+
+To upload using Workbench, first identify the project to upload the files into. This is done by browsing your projects in the navigation menu on the left, or to search for the project using the search field on the top.
+
+Having navigated to the project, click on the <span class="btn btn-sm btn-primary">+ NEW</span> button in the top-left corner. In the pop-up menu, select the item *New collection*.
+
+<figure> !{width: 80%;}{{ site.baseurl }}/images/add-new-collection-wb2.png! <figcaption> _Creating a new collection in the project "WGS Processing Tutorial"_ </figcaption></figure>
+
+In the dialog box that follows, you will be prompted to create a new collection in your chosen project. Here, the *Collection Name* field is required. After entering the name for this new collection (and optionally other fields), you have the choice to create it with new file updates -- by drag-and-drop into the *Files* area or with the traditional file-upload dialog opened by your browser.
+
+<figure>!{width: 100%;}{{ site.baseurl }}/images/new-collection-modal-wb2.png!<figcaption>_Providing the new collection with a name (required). Optionally, you can upload files in this step._</figcaption></figure>
+
+You can then click on the <span class="btn btn-sm btn-primary">CREATE A COLLECTION</span> button and proceed to the newly-created collection's page. If you don't upload any data when creating the collection, the new collection will be empty, and you can upload files into it later.
+
+<figure>!{width: 100%;}{{ site.baseurl }}/images/newly-created-collection-empty-wb2.png!<figcaption>_The newly-created collection without any files yet._</figcaption></figure>
+
+In the <span class="btn btn-sm btn-primary">FILES</span> panel, there is a button labeled <span class="btn btn-sm btn-primary">UPLOAD DATA</span>. Click on it, and you will be prompted to upload files by drag-and-drop or the file-selection dialog opened by your browser.
+
+The files you choose to upload will then be displayed, and you can review them before clicking on the <span class="btn btn-sm btn-primary">UPLOAD DATA</span> button to initiate the actual file transfer.
+
+<figure>!{width: 100%;}{{ site.baseurl }}/images/upload-data-prompt-with-files-wb2.png!<figcaption>_Selecting the files to upload_</figcaption></figure>
+
+Once the file upload completes, you will be notified by a message, and the files will appear under the <span class="btn btn-sm btn-primary">FILES</span> panel shortly.
+
+<figure>!{width: 100%;}{{ site.baseurl }}/images/upload-data-progress-wb2.png!<figcaption>_Upload status being displayed, with the files to appear shortly_</figcaption></figure>
+
+*Note:* If you leave the collection page during the upload, the upload process will be aborted and you will need to upload the files again.
+
+*Note:* You can also use the <span class="btn btn-sm btn-primary">UPLOAD DATA</span> button to add additional files to an existing collection.
notextile. <div class="spaced-out">
-# Begin by installing the "Arvados Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html on the system from which you will upload the data (such as your workstation, or a server containing data from your sequencer). This will install the Arvados file upload tool, @arv-put@. Alternately, you can log into an Arvados VM (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login).
-# On system from which you will upload data, configure the environment with the Arvados instance host name and authentication token as decribed in "Getting an API token.":{{site.baseurl}}/user/reference/api-tokens.html (If you are logged into an Arvados VM, you can skip this step.)
-# Download the following example file. (If you are uploading your own data, you can skip this step.)
-<notextile>
-<pre><code>~$ <span class="userinput">curl -o var-GS000016015-ASM.tsv.bz2 'https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2'</span>
- % Total % Received % Xferd Average Speed Time Time Time Current
- Dload Upload Total Spent Left Speed
-100 216M 100 216M 0 0 10.0M 0 0:00:21 0:00:21 --:--:-- 9361k
-</code></pre>
-</notextile>
-# Now upload the file to Keep using @arv-put@:
+
+h2(#upload-using-command). Upload using command line tool
+
+{% include 'tutorial_expectations' %}
+
+To upload a file to Keep using @arv-put@:
<notextile>
<pre><code>~$ <span class="userinput">arv-put var-GS000016015-ASM.tsv.bz2</span>
216M / 216M 100.0%
Collection saved as ...
-qr1hi-4zz18-xxxxxxxxxxxxxxx
+zzzzz-4zz18-xxxxxxxxxxxxxxx
</code></pre>
</notextile>
-* The output value @qr1hi-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
-
-Now visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection locator printed by @arv-put@ will appear under the *name* column.
-To move the collection to a different project, check the box at the left of the collection row. Pull down the *Selection...*<span class="caret"></span> menu near the top of the page tab, and select *Move selected*. This will open a dialog box where you can select a destination project for the collection. Click a project, then finally the <span class="btn btn-sm btn-primary">Move</span> button.
-
-!{{ site.baseurl }}/images/workbench-move-selected.png!
-
-Click on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection's listing on a project page to go to the Workbench page for your collection. On this page, you can see the collection's contents, download individual files, and set sharing options.
-
-notextile. </div>
+The output value @zzzzz-4zz18-xxxxxxxxxxxxxxx@ is the uuid of the Arvados collection created.
-h2(#dir). Putting a directory
+Note: The file used in this example is a freely available TSV file containing variant annotations from the "Personal Genome Project (PGP)":http://www.pgp-hms.org participant "hu599905":https://my.pgp-hms.org/profile/hu599905), downloadable "here":https://warehouse.pgp-hms.org/warehouse/f815ec01d5d2f11cb12874ab2ed50daa+234+K@ant/var-GS000016015-ASM.tsv.bz2. Alternatively, you can replace @var-GS000016015-ASM.tsv.bz2@ with the name of any file you have locally, or you could get the TSV file by "downloading it from Keep.":{{site.baseurl}}/user/tutorials/tutorial-keep-get.html
-If you give @arv-put@ a directory, it will recursively upload the entire directory:
+<notextile><a name="dir"></a></notextile>It is also possible to upload an entire directory with @arv-put@:
<notextile>
<pre><code>~$ <span class="userinput">mkdir tmp</span>
~$ <span class="userinput">arv-put tmp</span>
0M / 0M 100.0%
Collection saved as ...
-qr1hi-4zz18-yyyyyyyyyyyyyyy
+zzzzz-4zz18-yyyyyyyyyyyyyyy
</code></pre>
</notextile>
+
+In both examples, the @arv-put@ command created a collection. The first collection contains the single uploaded file. The second collection contains the entire uploaded directory.
+
+@arv-put@ accepts quite a few optional command line arguments, which are described on the "arv subcommands":{{site.baseurl}}/sdk/cli/subcommands.html#arv-keep-put page.
+
+h3. Locate your collection in Workbench
+
+Visit the Workbench and go to your <i class="fa fa-fw fa-folder"></i> *Home Projects*. Your newly uploaded collection should appear in the main panel. The collection name printed by @arv-put@ will appear under the *Name* column, and its *Type* will be "_Data collection_".
+
+Click on the collection's name will lead you to its Workbench page, where you can see the collection's contents and download individual files.
+
+To move the collection to a different project, locate the collection and right-click on it. This will bring up a context menu with *Move to*. Click on this item, and you will see a dialog box where you can select the target project to move this collection to, by search or navigation. This context menu is also available from the triple-dot button in the project/collection listing or the collection details page.
+
+<figure>!{width: 80%;}{{ site.baseurl }}/images/workbench-move-wb2.png!<figcaption> _Context menu with the_ *Move to* _item_ </figcaption></figure>
+
+notextile. </div>
projects / arvados.git / blobdiff