specs:
RedCloth (4.3.2)
coderay (1.1.3)
- colorize (0.8.1)
+ colorize (1.0.5)
commonjs (0.2.7)
kramdown (1.17.0)
less (2.6.0)
commonjs (~> 0.2.7)
- liquid (4.0.3)
+ liquid (4.0.4)
makerakeworkwell (1.0.4)
rake (>= 0.9.2, < 15)
- rake (13.0.1)
- zenweb (3.10.4)
+ rake (13.0.6)
+ zenweb (3.10.6)
coderay (~> 1.0)
kramdown (~> 1.4)
less (~> 2.0)
end
file "sdk/python/arvados/index.html" do |t|
- if ENV['NO_SDK'] || File.exists?("no-sdk")
+ if ENV['NO_SDK'] || File.exist?("no-sdk")
next
end
`which pdoc`
end
file "sdk/R/arvados/index.html" do |t|
- if ENV['NO_SDK'] || File.exists?("no-sdk")
+ if ENV['NO_SDK'] || File.exist?("no-sdk")
next
end
`which R`
end
file "sdk/java-v2/javadoc/index.html" do |t|
- if ENV['NO_SDK'] || File.exists?("no-sdk")
+ if ENV['NO_SDK'] || File.exist?("no-sdk")
next
end
`which java`
- user/index.html.textile.liquid
- user/getting_started/community.html.textile.liquid
- Walkthough:
- - user/tutorials/wgs-tutorial.html.textile.liquid
- - Using Workbench:
- user/getting_started/workbench.html.textile.liquid
- - user/tutorials/tutorial-workflow-workbench.html.textile.liquid
- - user/topics/workbench-migration.html.textile.liquid
+ - user/tutorials/wgs-tutorial.html.textile.liquid
- Working at the Command Line:
- user/getting_started/setup-cli.html.textile.liquid
- user/reference/api-tokens.html.textile.liquid
- user/getting_started/check-environment.html.textile.liquid
- - Access an Arvados virtual machine:
- - user/getting_started/vm-login-with-webshell.html.textile.liquid
- - user/getting_started/ssh-access-unix.html.textile.liquid
- - user/getting_started/ssh-access-windows.html.textile.liquid
- Working with data sets:
+ - user/tutorials/tutorial-projects.html.textile.liquid
- user/tutorials/tutorial-keep.html.textile.liquid
- user/tutorials/tutorial-keep-get.html.textile.liquid
+ - user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
+ - user/topics/arv-copy.html.textile.liquid
- user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid
- user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid
- user/tutorials/tutorial-keep-mount-windows.html.textile.liquid
- - user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid
- - user/topics/arv-copy.html.textile.liquid
- user/topics/collection-versioning.html.textile.liquid
- user/topics/storage-classes.html.textile.liquid
- Data Analysis with Workflows:
- - user/cwl/arvados-vscode-training.html.md.liquid
- - user/cwl/rnaseq-cwl-training.html.textile.liquid
+ - user/tutorials/tutorial-workflow-workbench.html.textile.liquid
- user/cwl/cwl-runner.html.textile.liquid
- user/cwl/cwl-run-options.html.textile.liquid
- - user/tutorials/writing-cwl-workflow.html.textile.liquid
+ - user/cwl/crunchstat-summary.html.textile.liquid
+ - user/debugging/container-shell-access.html.textile.liquid
+ - user/cwl/costanalyzer.html.textile.liquid
+ - user/cwl/federated-workflows.html.textile.liquid
+ - Common Workflow Language:
+ - user/cwl/rnaseq-cwl-training.html.textile.liquid
+ - user/cwl/arvados-vscode-training.html.md.liquid
- user/topics/arv-docker.html.textile.liquid
- user/cwl/cwl-style.html.textile.liquid
+ - user/tutorials/writing-cwl-workflow.html.textile.liquid
- user/cwl/cwl-extensions.html.textile.liquid
- - user/cwl/federated-workflows.html.textile.liquid
- user/cwl/cwl-versions.html.textile.liquid
- - user/cwl/crunchstat-summary.html.textile.liquid
- - user/cwl/costanalyzer.html.textile.liquid
- - user/debugging/container-shell-access.html.textile.liquid
- - Working with git repositories:
- - user/tutorials/add-new-repository.html.textile.liquid
- - user/tutorials/git-arvados-guide.html.textile.liquid
+ - Access an Arvados virtual machine:
+ - user/getting_started/vm-login-with-webshell.html.textile.liquid
+ - user/getting_started/ssh-access-unix.html.textile.liquid
+ - user/getting_started/ssh-access-windows.html.textile.liquid
- Reference:
+ - user/topics/workbench-migration.html.textile.liquid
- user/topics/link-accounts.html.textile.liquid
- user/reference/cookbook.html.textile.liquid
- Arvados License:
h1(#workbench). Adding your key to Arvados Workbench
-h3. From the Workbench dashboard
+In the Workbench top navigation menu, click on the dropdown menu icon <i class="fa fa-lg fa-user"></i> to access the Account Management menu. Then, click on the menu item *Ssh keys* to go to the *SSH keys* page. Click on the <span class="btn btn-primary">+ ADD NEW SSH KEY</span> button in the upper-right on that page. You will see a popup as shown in this screenshot:
-In the Workbench top navigation menu, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> to access the user settings menu and click on the menu item *SSH keys* to go to the *SSH keys* page. Click on the <span class="btn btn-primary">*+* Add new SSH key</span> button in this page. This will open a popup as shown in this screenshot:
+!{width: 100%;}{{ site.baseurl }}/images/ssh-adding-public-key.png!
-!{{ site.baseurl }}/images/ssh-adding-public-key.png!
-Paste your public key into the text area labeled *Public Key*, and click on the <span class="btn btn-primary">Submit</span> button. You are now ready to "log into an Arvados VM":#login.
+Paste your _public_ key into the text area labeled *Public Key*, and click on the <span class="btn btn-sm btn-primary">ADD NEW SSH KEY</span> button in lower-right. You are now ready to "log into an Arvados VM":#login.
h1(#login). Using SSH to log into an Arvados VM
-To see a list of virtual machines that you have access to, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, then click on the menu item *Virtual machines* to go to the Virtual machines page.
+To see a list of virtual machines that you have access to, click on the dropdown menu icon <i class="fa fa-lg fa-user"></i> in the upper right corner of the top navigation menu to access the Account Management menu. Then, click on the menu item *Virtual Machines*.
-This page lists the virtual machines you can access. The *Host name* column lists the name of each available VM. The *Login name* column lists your login name on that VM. The *Command line* column provides a sample @ssh@ command line.
+You will then see a page that lists the virtual machines you can access. The *Host name* column lists the name of each available VM. The *Login name* column lists your login name on that VM. The *Command line* column provides a sample @ssh@ command.
+
+!{width: 100%;}{{ site.baseurl }}/images/vm-access-with-webshell.png!
At the bottom of the page there may be additional instructions for connecting your specific Arvados instance. If so, follow your site-specific instructions. If there are no site-specific instructions, you can probably connect directly with @ssh@.
-The following are generic instructions. In the examples the login will be *_you_* and the hostname will be *_shell.ClusterID.example.com_* and . Replace these with your login name and hostname as appropriate.
+The following are generic instructions. In these examples, the login name will be *_you_* and the host domain will be *_ClusterID.example.com_*. Replace these with your login name and hostname as appropriate.
h2(#registering). Registering a workflow to use in Workbench
-Use @--create-workflow@ to register a CWL workflow with Arvados. Use @--project-uuid@ to upload the workflow to a specific project, along with its dependencies. You can share the workflow with other Arvados users by sharing that project. You can run the workflow by clicking the <span class="btn btn-sm btn-primary"><i class="fa fa-fw fa-gear"></i> Run a process...</span> button on the Workbench Dashboard, and on the command line by UUID.
+Use @--create-workflow@ to register a CWL workflow with Arvados. Use @--project-uuid@ to upload the workflow to a specific project, along with its dependencies. You can share the workflow with other Arvados users by sharing that project. You can run the workflow by clicking the <span class="btn btn-sm btn-primary">+ NEW</span> → <i class="fa fa-fw fa-gear"></i> *Run a process* menu items on the Workbench, and on the command line by UUID.
<notextile>
<pre><code>~/arvados/doc/user/cwl/bwa-mem$ <span class="userinput">arvados-cwl-runner --project-uuid zzzzz-j7d0g-p32bi47ogkjke11 --create-workflow bwa-mem.cwl</span>
* @-t@ specifies the key type (must be "rsa")
* @-C@ specifies a comment (to remember which account the key is associated with)
-We strongly recommend that you protect your key with a passphrase. This means that when the key is used, you will be required to enter the passphrase. However, unlike logging into remote system using a password, the passphrase is never sent over the network, it is only used to decrypt your private key.
+We strongly recommend that you protect your key with a passphrase. This means that when the key is used, you will be required to enter the passphrase. However, unlike logging into remote system using a password, the passphrase is never sent over the network; it is only used to decrypt your private key locally.
-Display the contents of @~/.ssh/id_rsa.pub@ (this is your public key) using @cat@ and then copy it onto the clipboard:
+Display the contents of @~/.ssh/id_rsa.pub@ (this is your public key) using @cat@, and then copy it onto the clipboard. The content of the public key may look similar to the following example:
<notextile>
<pre><code>$ <span class="userinput">cat ~/.ssh/id_rsa.pub</span>
</code></pre>
</notextile>
+* The above is a specimen that cannot be used as a valid public key.
+
Now you can set up @ssh-agent@ (next) or proceed with "adding your key to the Arvados Workbench.":#workbench
h3. Set up ssh-agent (optional)
notextile. <pre><code>$ <span class="userinput">ssh-add -l</span></code></pre>
-If you get the error "Could not open a connection to your authentication agent" you will need to run @ssh-agent@ with the following command:
+If you get the error "_Could not open a connection to your authentication agent_", you will need to run @ssh-agent@ with the following command:
-notextile. <pre><code>$ <span class="userinput">eval $(ssh-agent -s)</span></code></pre>
+notextile. <pre><code>$ <span class="userinput">eval "$(ssh-agent -s)"</span></code></pre>
-@ssh-agent -s@ prints out values for environment variables SSH_AUTH_SOCK and SSH_AGENT_PID and then runs in the background. Using "eval" on the output as shown here causes those variables to be set in the current shell environment so that subsequent calls to SSH can discover how to access the agent process.
+@ssh-agent -s@ runs an agent process in the background to hold your SSH credentials, and it prints out the values of environment variables @SSH_AUTH_SOCK@ and @SSH_AGENT_PID@. By applying the shell builtin @eval@ to this output, as we show here using the shell command-substitution syntax, we set those variables in the current shell environment. In this way, subsequent invocations of @ssh@ in this shell session will be able to access the agent process for the credentials without asking you each time.
-After running @ssh-agent@, or if @ssh-add -l@ prints "The agent has no identities", add your key using the following command. The passphrase to decrypt the key is the same used to protect the key when it was created with @ssh-keygen@:
+After running @ssh-agent@, or if @ssh-add -l@ prints "_The agent has no identities_", add your private key to the SSH agent using the following command. The passphrase to decrypt the key is the same one used to protect the key when it was created with @ssh-keygen@:
<notextile>
<pre><code>$ <span class="userinput">ssh-add</span>
</code></pre>
</notextile>
-When everything is set up, @ssh-add -l@ should yield output that looks something like this:
+When everything is set up, @ssh-add -l@ should yield output that looks like this:
<notextile>
<pre><code>$ <span class="userinput">ssh-add -l</span>
h3. Connecting directly
-If the VM is available on the public Internet (or you are on the same private network as the VM) you can connect directly with @ssh@. You can probably copy-and-paste the text from *Command line* column directly into a terminal.
+If the VM is available on the public Internet (or you are on the same private network as the VM), you can connect directly with @ssh@. You can copy-and-paste the text from the *Command line* column (see the screenshot above) directly into a shell session.
-Use the following example command to connect as _you_ to the _shell.ClusterID.example.com_ VM instance. Replace *<code>you@shell.ClusterID.example.com</code>* at the end of the following command with your *login* and *hostname* from Workbench.
+Use the following example command to connect, as the user "_you_" to the VM instance at the hostname "_shell.ClusterID.example.com_". Replace *<code>you@shell.ClusterID.example.com</code>* at the end of the following command with your actual *login* and *hostname* from Workbench.
notextile. <pre><code>$ <span class="userinput">ssh <b>you@shell.ClusterID.example.com</b></span></code></pre>
h3. Connecting through switchyard
-Some Arvados installations use "switchyard" to isolate shell VMs from the public Internet.
+Some Arvados installations use "switchyard" to isolate shell VMs from the public Internet. In such cases, you cannot log in directly to virtual machines over the public Internet. Instead, you log into a "switchyard" server and then tell the switchyard which virtual machine you want to connect to.
Use the following example command to connect to the _shell_ VM instance as _you_. Replace *<code>you@shell</code>* at the end of the following command with your *login* and *hostname* from Workbench:
notextile. <pre><code>$ <span class="userinput">ssh -o "ProxyCommand ssh -p2222 turnout@switchyard.ClusterID.example.com -x -a <b>shell</b>" -x <b>you@shell</b></span></code></pre>
-This command does several things at once. You usually cannot log in directly to virtual machines over the public Internet. Instead, you log into a "switchyard" server and then tell the switchyard which virtual machine you want to connect to.
+This command does several things at once.
* @-o "ProxyCommand ..."@ configures SSH to run the specified command to create a proxy and route your connection through it.
* @-p2222@ specifies that the switchyard is running on non-standard port 2222.
* <code>turnout@switchyard.{{ site.arvados_api_host }}</code> specifies the user (@turnout@) and hostname (@switchyard.{{ site.arvados_api_host }}@) of the switchyard server that will proxy our connection to the VM.
* @-x@ tells SSH not to forward your X session to the switchyard.
* @-a@ tells SSH not to forward your ssh-agent credentials to the switchyard.
-* *@shell@* is the name of the VM that we want to connect to. This is sent to the switchyard server as if it were an SSH command, and the switchyard server connects to the VM on our behalf.
-* After the ProxyCommand section, we repeat @-x@ to disable X session forwarding to the virtual machine.
+* *@shell@* is the host name of the VM that we want to connect to. In summary, the string inside the quotation marks is sent to the switchyard server, as if it were an SSH command, and the switchyard server connects to the VM on our behalf.
+* After the @ProxyCommand@ section, we repeat @-x@ to disable X session forwarding to the virtual machine.
* Finally, *<code>you@shell</code>* specifies your login name and repeats the hostname of the VM. The username can be found in the *logins* column in the VMs Workbench page, discussed in the previous section.
You should now be able to log into the Arvados VM and "check your environment.":check-environment.html
Some Arvados clusters may not have webshell set up. If you do not see a "Log in" button or "web shell" column, you will have to follow the "Unix":ssh-access-unix.html or "Windows":ssh-access-windows.html @ssh@ instructions.
{% include 'notebox_end' %}
-In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Virtual machines* to see the list of virtual machines you can access. If you do not have access to any virtual machines, please click on <span class="btn btn-sm btn-primary">Send request for shell access</span> (if present) or contact your system administrator. For the Arvados Playground, this is "info@curii.com":mailto:info@curii.com .
+In the Arvados Workbench, click on the dropdown menu icon <i class="fa fa-lg fa-user"></i> in the upper right corner of the top navigation menu to access the _Account Management_ menu, and click on the menu item *Virtual machines* to see the list of virtual machines you can access. If you do not have access to any virtual machines, please click on <span class="btn btn-sm btn-primary">SEND REQUEST FOR SHELL ACCESS</span> (if present) or contact your system administrator. For the Arvados Playground, this is "info@curii.com":mailto:info@curii.com .
-Each row in the Virtual Machines panel lists the hostname of the VM, along with a <code>Log in as *you*</code> button under the column "Web shell". Clicking on this button will open up a webshell terminal for you in a new browser tab and log you in.
+Each row in the Virtual Machines panel lists the hostname of the VM, along with a <span class="btn btn-sm btn-default" style="background-color: #e0e0e0">Log in as [your name]</span> button under the column "*Web shell*". Clicking on this button will open up a webshell terminal for you in a new browser tab and log you in.
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/vm-access-with-webshell.png!
+!{width: 100%;}{{ site.baseurl }}/images/vm-access-with-webshell.png!
You are now ready to work in your Arvados VM.
{% endcomment %}
{% include 'notebox_begin' %}
-This guide covers the classic Arvados Workbench web application, sometimes referred to as "Workbench 1". There is also a new Workbench web application under development called "Workbench 2". This guide will be updated to cover "Workbench 2" in the future. See "Workbench 2 migration":{{site.baseurl}}/user/topics/workbench-migration.html for more information.
+This guide covers modern Arvados Workbench web application, which may be referred to as "Workbench 2" to distinguish it from the previous Arvados Workbench web application ("Workbench 1"). Documentation for the classic Workbench can be found in "older versions of the user guide":https://doc.arvados.org/v2.6/user/getting_started/workbench.html . See also "Workbench 2 migration":{{site.baseurl}}/user/topics/workbench-migration.html for more information.
{% include 'notebox_end' %}
You can access the Arvados Workbench used in this guide using this link:
You will be asked to log in. Arvados uses only your name and email address for identification, and will never access any personal information. If you are accessing Arvados for the first time, the Workbench may indicate your account status is *New / inactive*. If this is the case, contact the administrator of the Arvados instance to request activation of your account.
-Once your account is active, logging in to the Workbench will present you with the Dashboard. This gives a summary of your projects and recent activity in the Arvados instance. You are now ready to "upload data":{{ site.baseurl }}/user/tutorials/tutorial-keep.html or "run your first workflow.":{{ site.baseurl }}/user/tutorials/tutorial-workflow-workbench.html
+Once your account is active, logging in to the Workbench will present you with an overview of your Home Projects. You are now ready to "upload data":{{ site.baseurl }}/user/tutorials/tutorial-keep.html or "run your first workflow.":{{ site.baseurl }}/user/tutorials/tutorial-workflow-workbench.html
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/workbench-dashboard.png!
+!{width: 100%;}{{ site.baseurl }}/images/workbench-first-page.png!
Open a shell on the system where you want to use the Arvados client. This may be your local workstation, or an Arvados virtual machine accessed with "Webshell":{{site.baseurl}}/user/getting_started/vm-login-with-webshell.html or SSH (instructions for "Unix":{{site.baseurl}}/user/getting_started/ssh-access-unix.html#login or "Windows":{{site.baseurl}}/user/getting_started/ssh-access-windows.html#login).
-In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Current token*, which lists your current token and instructions to set up your environment.
+In the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> in the upper right corner of the top navigation menu to access the _Account Management_ menu. Then, in the pop-up menu, click on the menu item *Get API token*. This will open a dialog box that lists your current token and the instructions for setting up your environment.
h2. Setting environment variables
-The *Current token* page, accessed using the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu, includes a command you may copy and paste directly into the shell. It will look something as the following.
+In the dialog box opened after clicking on the *Get API token* menu item, there is a sequence of commands you may copy and paste directly into the shell. It will look something as the following.
bc. HISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'
export ARVADOS_API_TOKEN=2jv9346o396exampledonotuseexampledonotuseexes7j1ld
</code></pre>
</notextile>
+* The output-redirection operator @>@ in the first command will cause the target file @~/.config/arvados/settings.conf@ to be created anew, wiping out the content of any existing file at that path.
+* The @>>@ operator in the second command appends to the target file.
+
h2. .bashrc
-Alternately, you may add the declarations of @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ to the @~/.bashrc@ file on the system on which you intend to use the Arvados client. If you have already put the variables into the environment following the instructions above, you can use these commands to append the environment variables to your @~/.bashrc@:
+Alternately, you may add the definitions of @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ to the @~/.bashrc@ file on the system where you intend to use the Arvados client. If you have already put the variables into the environment following the instructions above, you can use the commands below to append to your @~/.bashrc@, which tells Bash to export them as environment variables in newly-started interactive shell sessions:
<notextile>
<pre><code>$ <span class="userinput">echo "export ARVADOS_API_HOST=$ARVADOS_API_HOST" >> ~/.bashrc</span>
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Arvados is in the process of migrating from the classic web application, referred to as "Workbench 1", to a completely new web application, referred to as "Workbench 2".
+Beginning in version 2.7, Arvados now defaults to a new web application, referred to as "Workbench 2". This is a major step in the migration from the classic web application, referred to as "Workbench 1". Workbench 1 should be considered deprecated and suppport for the Workbench 1 application will be dropped in a future Arvados release.
!{width: 90%}{{ site.baseurl }}/images/wb2-example.png!
-Workbench 2 is the new Workbench web application that will, over time, replace Workbench 1. Workbench 2 is being built based on user feedback, and it is approaching feature parity with Workbench 1. Workbench 2 has a modern look and feel and offers several advanced features and performance enhancements. Arvados clusters typically have both Workbench applications installed and have a dropdown menu option in the user menu to switch between versions.
+Workbench 2 is the new Workbench web application that replaces Workbench 1. Workbench 2 is being built based on user feedback, and has feature parity with Workbench 1. Workbench 2 has a modern look and feel and offers many advanced features and performance enhancements over the previous Workbench application.
+
+To facilitate migration, Arvados may have both Workbench applications installed and have a dropdown menu option in the user menu to switch between versions.
!{{ site.baseurl }}/images/switch-to-wb2.png!
-Workbench 2 is stable and recommended for general use, but still lacks some features available in the classic Workbench 1 application. When necessary, you can easily switch back:
+Workbench 2 is stable and recommended for general use. However, if you find that you need something in Workbench 1, you can easily switch back:
!{{ site.baseurl }}/images/switch-to-wb1.png!
h2. Add "tutorial" repository
-On the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Repositories*.
+On the Arvados Workbench, click on the dropdown menu icon <i class="fa fa-lg fa-user"></i> (Account Management) in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Repositories*.
-In the *Repositories* page, you will see the *Add new repository* button.
+In the *Repositories* page, you will see the <span class="btn btn-sm btn-primary">+ NEW REPOSITORY</span> button.
-!{display: block;margin-left: 25px;margin-right: auto;}{{ site.baseurl }}/images/repositories-panel.png!
+!{width: 100%;}{{ site.baseurl }}/images/repositories-panel.png!
-Click the *Add new Repository* button to open the popup to add a new arvados repository. You will see a text box where you can enter the name of the repository. Enter *tutorial* in this text box and click on *Create*.
+Click the <span class="btn btn-sm btn-primary">+ NEW REPOSITORY</span> button to open the popup to add a new Arvados repository. You will see a text box where you can enter the name of the repository. Enter *tutorial* in this text box and click on *Create*.
{% include 'notebox_begin' %}
The name you enter here must begin with a letter and can only contain alphanumeric characters.
{% include 'notebox_end' %}
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/add-new-repository.png!
+!{width: 100%;}{{ site.baseurl }}/images/add-new-repository.png!
This will create a new repository with the name @$USER/tutorial@. It can be accessed using the URL <notextile><code>https://git.{{ site.arvados_api_host }}/$USER/tutorial.git</code></notextile> or <notextile><code>git@git.{{ site.arvados_api_host }}:$USER/tutorial.git</code></notextile>
~$ <span class="userinput">git config --global user.email $USER@example.com</span></code></pre>
</notextile>
-On the Arvados Workbench, click on the dropdown menu icon <span class="fa fa-lg fa-user"></span> <span class="caret"></span> in the upper right corner of the top navigation menu to access the user settings menu, and click on the menu item *Repositories*. In the *Repositories* page, you should see the @$USER/tutorial@ repository listed in the *name* column. Next to *name* is the column *URL*. Copy the *URL* value associated with your repository. This should look like <notextile><code>https://git.{{ site.arvados_api_host }}/$USER/tutorial.git</code></notextile>. Alternatively, you can use <notextile><code>git@git.{{ site.arvados_api_host }}:$USER/tutorial.git</code></notextile>
+On the Arvados Workbench, click on the dropdown menu icon <i class="fa fa-lg fa-user"></i> in the upper right corner of the top navigation menu to access the Account Management menu, and click on the menu item *Repositories*. In the *Repositories* page, you should see the @$USER/tutorial@ repository listed in the *name* column. Next to *name* is the column *URL*. Copy the *URL* value associated with your repository. This should look like <notextile><code>https://git.{{ site.arvados_api_host }}/$USER/tutorial.git</code></notextile>. Alternatively, you can use <notextile><code>git@git.{{ site.arvados_api_host }}:$USER/tutorial.git</code></notextile>
Next, on the Arvados virtual machine, clone your Git repository:
h3. Untrashing a collection using workbench
-To untrash a collection using workbench, go to trash page on workbench by clicking on the "Trash" icon in the top navigation in workbench and use the recycle icon or selection dropdown option.
+To untrash a collection using Workbench, go to the *Trash* page from the right-side panel, locate the trashed collection, and click on the "clock" or _Restore_ button at the end of the row. The *Restore* button is also available from the right-click context menu.
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/trash-button-topnav.png!
+!{width: 80%}{{ site.baseurl }}/images/trash-buttons.png!
h3. Untrashing a collection using arv command line tool
Arvados Data collections can be downloaded using either the arv commands or using Workbench.
# "*Download using Workbench*":#download-using-workbench
-# "*Sharing collections*":#download-shared-collection
+# "*Creating a special download URL for a collection*":#download-shared-collection
# "*Download using command line tools*":#download-using-arv
h2(#download-using-workbench). Download using Workbench
You can also download Arvados data collections using the Workbench.
-Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. You will see the *Data collections* tab, which lists the collections in this project.
+When you visit a project in Workbench (for instance, the <i class="fa fa-fw fa-folder"></i> *Home Projects* or any projects under it), the collections will show up on the project details page, with "_Data collection_" in the *Type* column.
-You can access the contents of a collection by clicking on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection. This will take you to the collection's page. Using this page you can see the collection's contents, and download individual files.
+Clicking on a collection will bring you to its details page. There, the lower panel acts like a file manager where you can navigate to or search for files, select them for actions, and download them.
-You can now download the collection files by clicking on the <span class="btn btn-sm btn-info"><i class="fa fa-download"></i></span> button(s).
+To download a file, simply click on the file, or bring up the context menu using right-click or the triple-dot button on its row, and then select the menu item *Download*.
-h2(#download-shared-collection). Sharing collections
+h2(#download-shared-collection). Creating a special download URL for a collection
-h3. Sharing with other Arvados users
+To share a collection with users that do not have an account on your Arvados cluster, locate the collection and then go to the *Sharing settings* dialog box as described above. There, select the *SHARING URLS* tab.
-Collections can be shared with other users on the Arvados cluster by sharing the parent project. Navigate to the parent project using the "breadcrumbs" bar, then click on the *Sharing* tab. From the sharing tab, you can choose which users or groups to share with, and their level of access.
+You can then generate a new sharing URL using the <span class="btn btn-sm btn-primary">CREATE SHARING URL</span> button, with the option to set an expiration time for the URL. You can then copy the URL to the clipboard for sharing with others. To revoke (that is, delete) a sharing URL, click on the cross icon beside it.
-h3. Creating a special download URL
+<figure>!{width: 80%}{{ site.baseurl }}/images/sharing-collection-url.png!<figcaption>_The_ *SHARING URLS* _tab in the_ *Sharing settings* _dialog box, showing the created URL with an expiration time_</figcaption></figure>
-To share a collection with users that do not have an account on your Arvados cluster, visit the collection page using Workbench as described in the above section. Once on this page, click on the <span class="btn btn-sm btn-primary" >Create sharing link</span> button.
+Any user with the sharing URL can download this collection by simply accessing this URL using browser. It will present a downloadable version of the collection as shown below.
-This will create a sharing link for the collection as shown below. You can copy the sharing link in this page and share it with other users.
-
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/shared-collection.png!
+!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/download-shared-collection.png!
-A user with this url can download this collection by simply accessing this url using browser. It will present a downloadable version of the collection as shown below.
+When a collection is being shared by URL, in the *WITH USERS/GROUS* tab of *Sharing settings*, the following message will appear if *General access* is Private: _Although there aren't specific permissions set, this is publicly accessible via Sharing URL(s)._
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/download-shared-collection.png!
+* *Note:* Sharing by URL is specific to collections. Projects or individual files cannot be shared in this way.
h2(#download-using-arv). Download using command line tools
Arvados Data collections can be uploaded using either Workbench or the @arv-put@ command line tool.
# "*Upload using Workbench*":#upload-using-workbench
-# "*Creating projects*":#creating-projects
# "*Upload using command line tool*":#upload-using-command
h2(#upload-using-workbench). Upload using Workbench
-To upload using Workbench, visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu and select your *Home* project or any other project of your choosing. You will see the *Data collections* tab for this project, which lists the collections in this project.
+To upload using Workbench, first identify the project to upload the files into. This is done by browsing your projects in the navigation menu on the left, or to search for the project using the search field on the top.
-To upload files into a new collection, click on *Add data*<span class="caret"></span> dropdown menu and select *Upload files from my computer*.
+Having navigated to the project, click on the <span class="btn btn-sm btn-primary">+ NEW</span> button in the top-left corner. In the pop-up menu, select the item *New collection*.
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/upload-using-workbench.png!
+<figure> !{width: 80%;}{{ site.baseurl }}/images/add-new-collection-wb2.png! <figcaption> _Creating a new collection in the project "WGS Processing Tutorial"_ </figcaption></figure>
-<br/>This will create a new empty collection in your chosen project and will take you to the *Upload* tab for that collection.
+In the dialog box that follows, you will be prompted to create a new collection in your chosen project. Here, the *Collection Name* field is required. After entering the name for this new collection (and optionally other fields), you have the choice to create it with new file updates -- by drag-and-drop into the *Files* area or with the traditional file-upload dialog opened by your browser.
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/upload-tab-in-new-collection.png!
+<figure>!{width: 100%;}{{ site.baseurl }}/images/new-collection-modal-wb2.png!<figcaption>_Providing the new collection with a name (required). Optionally, you can upload files in this step._</figcaption></figure>
-Click on the *Browse...* button and select the files you would like to upload. Selected files will be added to a list of files to be uploaded. After you are done selecting files to upload, click on the *<i class="fa fa-fw fa-play"></i> Start* button to start upload. This will start uploading files to Arvados and Workbench will show you the progress bar. When upload is completed, you will see an indication to that effect.
+You can then click on the <span class="btn btn-sm btn-primary">CREATE A COLLECTION</span> button and proceed to the newly-created collection's page. If you don't upload any data when creating the collection, the new collection will be empty, and you can upload files into it later.
-!{display: block;margin-left: 25px;margin-right: auto;border:1px solid lightgray;}{{ site.baseurl }}/images/files-uploaded.png!
+<figure>!{width: 100%;}{{ site.baseurl }}/images/newly-created-collection-empty-wb2.png!<figcaption>_The newly-created collection without any files yet._</figcaption></figure>
-*Note:* If you leave the collection page during the upload, the upload process will be aborted and you will need to upload the files again.
+In the <span class="btn btn-sm btn-primary">FILES</span> panel, there is a button labeled <span class="btn btn-sm btn-primary">UPLOAD DATA</span>. Click on it, and you will be prompted to upload files by drag-and-drop or the file-selection dialog opened by your browser.
-*Note:* You can also use the Upload tab to add additional files to an existing collection.
+The files you choose to upload will then be displayed, and you can review them before clicking on the <span class="btn btn-sm btn-primary">UPLOAD DATA</span> button to initiate the actual file transfer.
-notextile. <div class="spaced-out">
+<figure>!{width: 100%;}{{ site.baseurl }}/images/upload-data-prompt-with-files-wb2.png!<figcaption>_Selecting the files to upload_</figcaption></figure>
-h2(#creating-projects). Creating projects
+Once the file upload completes, you will be notified by a message, and the files will appear under the <span class="btn btn-sm btn-primary">FILES</span> panel shortly.
-Files are organized into Collections, and Collections are organized by Projects.
+<figure>!{width: 100%;}{{ site.baseurl }}/images/upload-data-progress-wb2.png!<figcaption>_Upload status being displayed, with the files to appear shortly_</figcaption></figure>
-Click on *Projects*<span class="caret"></span> <span class="rarr">→</span> <i class="fa fa-fw fa-plus"></i>*Add a new project* to add a top level project.
+*Note:* If you leave the collection page during the upload, the upload process will be aborted and you will need to upload the files again.
-To create a subproject, navigate to the parent project, and click on <i class="fa fa-fw fa-plus"></i>*Add a subproject*.
+*Note:* You can also use the <span class="btn btn-sm btn-primary">UPLOAD DATA</span> button to add additional files to an existing collection.
+
+notextile. <div class="spaced-out">
-See "Sharing collections":tutorial-keep-get.html#download-shared-collection for information about sharing projects and collections with other users.
h2(#upload-using-command). Upload using command line tool
h3. Locate your collection in Workbench
-Visit the Workbench *Dashboard*. Click on *Projects*<span class="caret"></span> dropdown menu in the top navigation menu, select your *Home* project. Your newly uploaded collection should appear near the top of the *Data collections* tab. The collection name printed by @arv-put@ will appear under the *name* column.
+Visit the Workbench and go to your <i class="fa fa-fw fa-folder"></i> *Home Projects*. Your newly uploaded collection should appear in the main panel. The collection name printed by @arv-put@ will appear under the *Name* column, and its *Type* will be "_Data collection_".
-To move the collection to a different project, check the box at the left of the collection row. Pull down the *Selection...*<span class="caret"></span> menu near the top of the page tab, and select *Move selected...* button. This will open a dialog box where you can select a destination project for the collection. Click a project, then finally the <span class="btn btn-sm btn-primary">Move</span> button.
+Click on the collection's name will lead you to its Workbench page, where you can see the collection's contents and download individual files.
-!{display: block;margin-left: 25px;margin-right: auto;}{{ site.baseurl }}/images/workbench-move-selected.png!
+To move the collection to a different project, locate the collection and right-click on it. This will bring up a context menu with *Move to*. Click on this item, and you will see a dialog box where you can select the target project to move this collection to, by search or navigation. This context menu is also available from the triple-dot button in the project/collection listing or the collection details page.
-Click on the *<i class="fa fa-fw fa-archive"></i> Show* button next to the collection's listing on a project page to go to the Workbench page for your collection. On this page, you can see the collection's contents, download individual files, and set sharing options.
+<figure>!{width: 80%;}{{ site.baseurl }}/images/workbench-move-wb2.png!<figcaption> _Context menu with the_ *Move to* _item_ </figcaption></figure>
notextile. </div>
--- /dev/null
+---
+layout: default
+navsection: userguide
+title: "Organizating data"
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+h2. Projects and Collections
+
+In Arvados, files are organized into "collections", and collections are organized by "project".
+
+Only collections can contain files. A collection is a distinct database record identified by a universal unique id (UUID). Arvados maintains a history of changes to the collection. Every collection version has an immutable identifier called a "portable data hash" which is computed from the file content of the collection. This can be used to refer to the immutable file content independently of the collection UUID. If two collections have the same portable data hash, they have the same file content.
+
+Projects contain collections, workflows and workflow runs, and other projects (subprojects). Both collections and projects can have user-provided metadata.
+
+Projects are the main unit of organization and sharing. See "Sharing collections":#sharing-projects for information about sharing projects and collections with other users.
+
+h2(#creating-projects). Creating a project
+
+When you have navigated to any existing project, clicking on <span class="btn btn-sm btn-primary">+ NEW</span> <span class="rarr">→</span> <i class="fa fa-fw fa-folder"></i> *New project* will prompt you to create a new subproject under the current project.
+
+If you're at the top-level <i class="fa fa-fw fa-folder"></i> *Home Projects*, a new top-level project will be created.
+
+Alternatively, you can right-click on the link to an existing project to bring up a context menu, and select *New project*.
+
+h2(#sharing-projects). Sharing projects
+
+Projects can be shared with other users on the Arvados cluster. First, locate the collection or project using any available means (for instance, by manually navigating in the Workbench, or using the Search bar). Then right-click on its link in a listing, or click on the triple-dot button in the details page. You will find the menu item *Share*, which opens the dialog box *Sharing settings*.
+
+To share with other Arvados users, select the *WITH USERS/GROUPS* tab in the *Sharing settings* dialog box. Under *Add people and groups*, in the input field you can search for the user or group names. Select one you will be sharing with, choose the *Authorization* level (Read/Write/Manage) in the drop-down menu, and click on the plus sign (+) on the right. This can be repeated for other users or groups, each with their own *Authorization* level. The selected ones will appear under *People with access*. You can revisit the *Sharing settings* dialog box to modify the users or their access levels at a later time.
+
+The *General access* drop-down menu controls the default sharing setting, with the following choices:
+
+* *Private*: This is the initial state when no users or groups have been selected for sharing. At any time, by setting *General access* to private, the current sharing setting will be cleared, and any users or groups formerly with access will lose that access.
+* *Public*: This means the list of *People with access* will include _Anonymous users_, even if they are not users of the current cluster. You can further set their access level in the *Authorization* level.
+* *All users*: This means sharing with other users who are logged in on the current cluster.
+* *Shared*: When you choose to share with specific people or groups, *General access* will be set to *Shared*. From this state, you can further specify the default sharing settings for *Public* and *All users*.
notextile. <div class="spaced-out">
-# Start from the *Workbench Dashboard*. You can access the Dashboard by clicking on *<i class="fa fa-lg fa-fw fa-dashboard"></i> Dashboard* in the upper left corner of any Workbench page.
-# Click on the <span class="btn btn-sm btn-primary"><i class="fa fa-fw fa-gear"></i> Run a process...</span> button. This will open a dialog box titled *Choose a pipeline or workflow to run*.
-# In the search box, type in *bwa-mem.cwl*.
-# Select *<i class="fa fa-fw fa-gear"></i> bwa-mem.cwl* and click the <span class="btn btn-sm btn-primary" >Next: choose inputs <i class="fa fa-fw fa-arrow-circle-right"></i></span> button. This will create a new process in your *Home* project and will open it. You can now supply the inputs for the process. Please note that all required inputs are populated with default values and you can change them if you prefer.
-# For example, let's see how to set read pair *read_p1* and *read_p2* for this workflow. Click the <span class="btn btn-sm btn-primary">Choose</span> button beneath the *read_p1* header. This will open a dialog box titled *Choose a file*.
-# In the file dialog, click on *Home <span class="caret"></span>* menu and then select *All Projects*.
-# Enter *HWI-ST1027* into the search box. You will see one or more collections. Click on *<i class="fa fa-fw fa-archive"></i> HWI-ST1027_129_D0THKACXX for CWL tutorial*
-# The right hand panel will list two files. Click on the first one ending in "_1" and click the <span class="btn btn-sm btn-primary" >OK</span> button.
-# Repeat the steps 5-8 to set the *read_p2* except selecting the second file ending in "_2"
-# Scroll to the bottom of the "Inputs" panel and click on the <span class="btn btn-sm btn-primary" >Run <i class="fa fa-fw fa-play"></i></span> button. The page updates to show you that the process has been submitted to run on the Arvados cluster.
-# Once the process starts running, you can track the progress by watching log messages from the component(s). This page refreshes automatically. You will see a <span class="label label-success">complete</span> label when the process completes successfully.
-# Click on the *Output* link to see the results of the process. This will load a new page listing the output files from this process. You'll see the output SAM file from the alignment tool under the *Files* tab.
-# Click on the <span class="btn btn-sm btn-info"><i class="fa fa-download"></i></span> download button to the right of the SAM file to download your results.
+# Click on the <span class="btn btn-sm btn-primary">+ NEW</span> button in the top-left.
+# In the pop-up menu, select *<i class="fa fa-fw fa-gear"></i> Run a workflow*. This will open the _Run Process_ panel in the Workbench.
+# In the search field under *Choose a workflow*, type in _bwa-mem.cwl_.
+# Select *bwa-mem.cwl* in the search results, and click the <span class="btn btn-sm btn-primary" >NEXT</span> button. This will create a new process in one of your Home Projects and will open it. To specify the project for the workflow run, click on the input line below "*Project where the workflow will run*", and in the pop-up dialog box, choose a project under your Home Projects.
+# You can now supply the inputs for the process. Please note that all required inputs are populated with default values and you can change them if you prefer.
+# For example, let's see how to set read pair *read_p1* and *read_p2* for this workflow. Click on the input line under the *read_p1* header. This will open a dialog box titled *Choose a file*.
+# Enter the search terms _user guide resources_ into the *Search for a Project* field on the left. You will see one or more collections in the search results appearing below and, among them, the one with the exact title *<i class="fa fa-fw fa-folder"></i> User guide resources*. Your goal is to locate the file _HWI-ST1027_129_D0THKACXX.1_1.fastq_.
+# You may either locate the file manually, by clicking on the triangles ▶ to the left of each item to expand them (projects and the collections under it) until you find the file, or by filtering the search results using the *Filter Collections list in Projects* field, for example, with a term like "_HWI-ST1027_".
+# Either way, you will find the file <i class="fa fa-fw fa-file"></i> *HWI-ST1027_129_D0THKACXX.1_1.fastq* in the search results. Click on it, and then the <span class="btn btn-sm btn-primary">OK</span> button in the bottom-right.
+# Repeat the steps 7--9 to set the value for *read_p2*, except selecting the file ending in "_2"
+# Scroll to the bottom of the "Inputs" panel and click on the <span class="btn btn-sm btn-primary" >RUN WORKFLOW</span> button. The page updates to show you that the process has been queued to run on the Arvados cluster.
+# Once the process starts running, you can track the progress by watching the log messages from the component(s) (scroll down to the *Logs* panel). This page refreshes automatically, and you can also click on the <span class="btn btn-sm btn-primary">REFRESH</span> button on the top of the page. You will see a <span class="label label-success">Completed</span> label when the process completes successfully.
+# The output of the workflow can be found by following the link "Output from bwa-mem.cwl" under the heading *Output collection* in the main or <span class="btn btn-sm btn-primary">DETAILS</span> panel, or in the <span class="btn btn-sm btn-primary">OUTPUTS</span> panel further down. Click on the *Output from bwa-mem.cwl* link to see the detailed results from the workflow run. This will lead you to a page that lists the metadata of the outputs, and you'll see the output SAM file there, in the <span class="btn btn-sm btn-primary">FILES</span> panel.
+# To download your results, simply click on the SAM file name.
notextile. </div>
projects / arvados.git / commitdiff
