installation. It now lives at dev.arvados.org.
refs #7187
Here's how to build the HTML pages locally so you can preview your updates before you commit and push.
-Additional information is available on the "'Documentation' page on the Arvados wiki":https://arvados.org/projects/arvados/wiki/Documentation.
+Additional information is available on the "'Documentation' page on the Arvados wiki":https://dev.arvados.org/projects/arvados/wiki/Documentation.
h2. Install dependencies
<div class="alert alert-block alert-info">
<button type="button" class="close" data-dismiss="alert">×</button>
<h4>Hi!</h4>
- <P>This section is incomplete. Please be patient with us as we fill in the blanks — or <A href="https://arvados.org/projects/arvados/wiki/Documentation#Contributing">contribute to the documentation project.</A></P>
+ <P>This section is incomplete. Please be patient with us as we fill in the blanks — or <A href="https://dev.arvados.org/projects/arvados/wiki/Documentation#Contributing">contribute to the documentation project.</A></P>
</div>
<div class="alert alert-block alert-info">
<button type="button" class="close" data-dismiss="alert">×</button>
<h4>Hi!</h4>
- <p>This section is incomplete. Please be patient with us as we fill in the blanks — or <A href="https://arvados.org/projects/arvados/wiki/Documentation#Contributing">contribute to the documentation project.</A></p>
+ <p>This section is incomplete. Please be patient with us as we fill in the blanks — or <A href="https://dev.arvados.org/projects/arvados/wiki/Documentation#Contributing">contribute to the documentation project.</A></p>
</div>
<li {% if page.navsection == 'sdk' %} class="active" {% endif %}><a href="{{ site.baseurl }}/sdk/index.html">SDKs</a></li>
<li {% if page.navsection == 'api' %} class="active" {% endif %}><a href="{{ site.baseurl }}/api/index.html">API</a></li>
<li {% if page.navsection == 'installguide' %} class="active" {% endif %}><a href="{{ site.baseurl }}/install/index.html">Install</a></li>
- <li><a href="https://arvados.org/projects/arvados/" style="padding-left: 2em">arvados.org »</a></li>
+ <li><a href="https://arvados.org" style="padding-left: 2em">arvados.org »</a></li>
</ul>
<div class="pull-right" style="padding-top: 6px">
<p><a href="https://arvados.org/">Arvados</a> enables you to quickly begin using cloud computing resources in your data science work. It allows you to track your methods and datasets, share them securely, and easily re-run analyses.
</p>
<p><strong>Communications</strong>
- <p>Read our <a href="https://arvados.org/projects/arvados/blogs">blog updates</a> or look through our <a href="https://arvados.org/projects/arvados/activity">recent developer activity</a>.
+ <p>Read our <a href="https://dev.arvados.org/projects/arvados/blogs">blog updates</a> or look through our <a href="https://dev.arvados.org/projects/arvados/activity">recent developer activity</a>.
</p>
<p>Questions? Email <a href="http://lists.arvados.org/mailman/listinfo/arvados">the mailing list</a>, or chat with us on IRC: <a href="irc://irc.oftc.net:6667/#arvados">#arvados</a> @ OFTC (you can <a href="https://webchat.oftc.net/?channels=arvados">join in your browser</a>).
</p>
<p><strong>Want to contribute?</strong></p>
- <p>Check out our <a href="https://arvados.org/projects/arvados">developer site</a>. We're open source, check out our code on <a href="https://github.com/curoverse/arvados">github</a>.
+ <p>Check out our <a href="https://dev.arvados.org/projects/arvados">developer site</a>. We're open source, check out our code on <a href="https://github.com/curoverse/arvados">github</a>.
</p>
<p><strong>License</strong></p>
<p>Arvados is under the copyleft <a href="{{ site.baseurl }}/user/copying/agpl-3.0.html">GNU AGPL v3</a>, with our SDKs under <a href="{{ site.baseurl }}/user/copying/LICENSE-2.0.html">Apache License 2.0</a> (so that you can incorporate proprietary toolchains into your pipelines).
<div class="col-sm-6" style="border-left: solid; border-width: 1px">
<p><strong>Quickstart</strong>
<p>
- Try any pipeline from the <a href="https://arvados.org/projects/arvados/wiki/Public_Pipelines_and_Datasets">list of public pipelines</a>. For instance, the <a href="http://curover.se/pathomap">Pathomap Pipeline</a> links to these <a href="https://arvados.org/projects/arvados/wiki/pathomap_tutorial/">step-by-step instructions</a> for trying Arvados out right in your browser using Curoverse's <a href="http://lp.curoverse.com/beta-signup/">public Arvados instance</a>.
+ Try any pipeline from the <a href="https://dev.arvados.org/projects/arvados/wiki/Public_Pipelines_and_Datasets">list of public pipelines</a>. For instance, the <a href="http://curover.se/pathomap">Pathomap Pipeline</a> links to these <a href="https://dev.arvados.org/projects/arvados/wiki/pathomap_tutorial/">step-by-step instructions</a> for trying Arvados out right in your browser using Curoverse's <a href="http://lp.curoverse.com/beta-signup/">public Arvados instance</a>.
</p>
<!--<p>-->
<!--<ol>-->
Pipeline Developer Quickstart
</strong></p>
<p>
- Want to port your pipeline to Arvados? Check out the step-by-step <a href="https://arvados.org/projects/arvados/wiki/Port_a_Pipeline">Port-a-Pipeline</a> guide on the Arvados wiki.
+ Want to port your pipeline to Arvados? Check out the step-by-step <a href="https://dev.arvados.org/projects/arvados/wiki/Port_a_Pipeline">Port-a-Pipeline</a> guide on the Arvados wiki.
</p>
<p><strong>More in-depth guides
</strong></p>
Edit @/etc/arvados/api/application.yml@ to configure the settings described in the following sections. The deployment script will consistently deploy this to the API server's configuration directory. The API server reads both @application.yml@ and its own @config/application.default.yml@ file. The settings in @application.yml@ take precedence over the defaults that are defined in @config/application.default.yml@. The @config/application.yml.example@ file is not read by the API server and is provided as a starting template only.
-@config/application.default.yml@ documents additional configuration settings not listed here. You can "view the current source version":https://arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml for reference.
+@config/application.default.yml@ documents additional configuration settings not listed here. You can "view the current source version":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/services/api/config/application.default.yml for reference.
Only put local configuration in @application.yml@. Do not edit @application.default.yml@.
~$ <span class="userinput">git clone https://github.com/curoverse/arvados.git</span>
</code></pre></notextile>
-See also: "Downloading the source code":https://arvados.org/projects/arvados/wiki/Download on the Arvados wiki.
+See also: "Downloading the source code":https://dev.arvados.org/projects/arvados/wiki/Download on the Arvados wiki.
h2. Building the Arvados Docker containers
<notextile>{% code 'example_sdk_go_imports' as go %}</notextile>
-If you need pre-release client code, you can use the latest version from the repo by following "these instructions.":https://arvados.org/projects/arvados/wiki/Go#Using-Go-with-Arvados
+If you need pre-release client code, you can use the latest version from the repo by following "these instructions.":https://dev.arvados.org/projects/arvados/wiki/Go#Using-Go-with-Arvados
h3. Example
<notextile>{% code 'example_sdk_go' as go %}</notextile>
-A few more usage examples can be found in the "services/keepproxy":https://arvados.org/projects/arvados/repository/revisions/master/show/services/keepproxy and "sdk/go/keepclient":https://arvados.org/projects/arvados/repository/revisions/master/show/sdk/go/keepclient directories in the arvados source tree.
+A few more usage examples can be found in the "services/keepproxy":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/services/keepproxy and "sdk/go/keepclient":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/go/keepclient directories in the arvados source tree.
In this quickstart guide, we'll run an existing pipeline with pre-existing data. Step-by-step instructions are shown below. You can follow along using your own local install or by using Curoverse's <a href="http://lp.curoverse.com/beta-signup/">hosted version of Arvados</a> (in public beta, any Google account can be used to login).
-(For more information about this pipeline, see our <a href="https://arvados.org/projects/arvados/wiki/LobSTR_tutorial">detailed lobSTR guide</a>).
+(For more information about this pipeline, see our <a href="https://dev.arvados.org/projects/arvados/wiki/LobSTR_tutorial">detailed lobSTR guide</a>).
<div id="carousel-firstpipe" class="carousel slide" data-interval="false">
<!-- Indicators -->
title: "Scripts provided by Arvados"
...
-Several crunch scripts are included with Arvados in the "/crunch_scripts directory":https://arvados.org/projects/arvados/repository/revisions/master/show/crunch_scripts. They are intended to provide examples and starting points for writing your own scripts.
+Several crunch scripts are included with Arvados in the "/crunch_scripts directory":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/crunch_scripts. They are intended to provide examples and starting points for writing your own scripts.
h4. bwa-aln
-Run the bwa aligner on a set of paired-end fastq files, producing a BAM file for each pair. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/bwa-aln
+Run the bwa aligner on a set of paired-end fastq files, producing a BAM file for each pair. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/bwa-aln
<div class="offset1">
table(table table-bordered table-condensed).
h4. bwa-index
-Generate an index of a fasta reference genome suitable for use by bwa-aln. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/bwa-index
+Generate an index of a fasta reference genome suitable for use by bwa-aln. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/bwa-index
<div class="offset1">
table(table table-bordered table-condensed).
h4. picard-gatk2-prep
-Using the FixMateInformation, SortSam, ReorderSam, AddOrReplaceReadGroups, and BuildBamIndex modules from picard, prepare a BAM file for use with the GATK2 tools. Additionally, run picard's CollectAlignmentSummaryMetrics module to produce a @*.casm.tsv@ statistics file for each BAM file. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/picard-gatk2-prep
+Using the FixMateInformation, SortSam, ReorderSam, AddOrReplaceReadGroups, and BuildBamIndex modules from picard, prepare a BAM file for use with the GATK2 tools. Additionally, run picard's CollectAlignmentSummaryMetrics module to produce a @*.casm.tsv@ statistics file for each BAM file. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/picard-gatk2-prep
<div class="offset1">
table(table table-bordered table-condensed).
h4. GATK2-realign
-Run GATK's RealignerTargetCreator and IndelRealigner modules on a set of BAM files. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-realign
+Run GATK's RealignerTargetCreator and IndelRealigner modules on a set of BAM files. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-realign
<div class="offset1">
table(table table-bordered table-condensed).
h4. GATK2-bqsr
-Run GATK's BaseQualityScoreRecalibration module on a set of BAM files. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-bqsr
+Run GATK's BaseQualityScoreRecalibration module on a set of BAM files. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-bqsr
<div class="offset1">
table(table table-bordered table-condensed).
h4. GATK2-merge-call
-Merge a set of BAM files using picard, and run GATK's UnifiedGenotyper module on the merged set to produce a VCF file. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-merge-call
+Merge a set of BAM files using picard, and run GATK's UnifiedGenotyper module on the merged set to produce a VCF file. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-merge-call
<div class="offset1">
table(table table-bordered table-condensed).
h4. file-select
-Pass through the named files from input to output collection, and ignore the rest. "View source.":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/file-select
+Pass through the named files from input to output collection, and ignore the rest. "View source.":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/file-select
<div class="offset1">
table(table table-bordered table-condensed).
h2. On the web
-The Arvados Free Sofware project page is located at "http://arvados.org":http://arvados.org . The "Arvados Wiki":https://arvados.org/projects/arvados/wiki is a collaborative site for documenting Arvados and provides an overview of the Arvados Platform and Components. The "Arvados blog":https://arvados.org/projects/arvados/blogs posts articles of interest about Arvados.
+The Arvados Free Sofware project page is located at "https://arvados.org":https://arvados.org . The "Arvados Wiki":https://dev.arvados.org/projects/arvados/wiki is a collaborative site for documenting Arvados and provides an overview of the Arvados Platform and Components. The "Arvados blog":https://dev.arvados.org/projects/arvados/blogs posts articles of interest about Arvados.
h2. Mailing lists
h2. Bug tracking
-If you think you have found a bug, or would like to make a feature request, check the "Arvados issue tracker":https://arvados.org/projects/arvados/issues to see if has already been reported or "add a new issue.":https://arvados.org/projects/arvados/issues/new
+If you think you have found a bug, or would like to make a feature request, check the "Arvados issue tracker":https://dev.arvados.org/projects/arvados/issues to see if has already been reported or "add a new issue.":https://dev.arvados.org/projects/arvados/issues/new
h2. Submit a GATK job
-The Arvados distribution includes an example crunch script ("crunch_scripts/GATK2-VariantFiltration":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-VariantFiltration) that runs the GATK VariantFiltration tool with some default settings.
+The Arvados distribution includes an example crunch script ("crunch_scripts/GATK2-VariantFiltration":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-VariantFiltration) that runs the GATK VariantFiltration tool with some default settings.
<notextile>
<pre><code>~$ <span class="userinput">src_version=76588bfc57f33ea1b36b82ca7187f465b73b4ca4</span>
# Programming in Python
# Revision control using Git
-We also recommend you read the "Arvados Platform Overview":https://arvados.org/projects/arvados/wiki#Platform-Overview for an introduction and background information about Arvados.
+We also recommend you read the "Arvados Platform Overview":https://dev.arvados.org/projects/arvados/wiki#Platform-Overview for an introduction and background information about Arvados.
* @"repository"@, @"script_version"@, and @"script"@ indicate that we intend to use the external @"run-command"@ tool wrapper that is part of the Arvados. These parameters are described in more detail in "Writing a script":tutorial-firstscript.html.
* @"runtime_constraints"@ describes runtime resource requirements for the component.
** @"docker_image"@ specifies the "Docker":https://www.docker.com/ runtime environment in which to run the job. The Docker image @"bcosc/arv-base-java"@ supplied here has the Java runtime environment, bwa, and samtools installed.
-** @"arvados_sdk_version"@ specifies a version of the Arvados SDK to load alongside the job's script. The example uses 'master'. If you would like to use a specific version of the sdk, you can find it in the "Arvados Python sdk repository":https://arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
+** @"arvados_sdk_version"@ specifies a version of the Arvados SDK to load alongside the job's script. The example uses 'master'. If you would like to use a specific version of the sdk, you can find it in the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
* @"script_parameters"@ describes the component parameters.
** @"command"@ is the actual command line to invoke the @bwa@ and then @SortSam@. The notation @$()@ denotes macro substitution commands evaluated by the run-command tool wrapper.
** @"task.stdout"@ indicates that the output of this command should be captured to a file.
See the "run-command reference":{{site.baseurl}}/user/topics/run-command.html for more information about using @run-command@.
*Note:* When trying to get job reproducibility without re-computation, you need to set these parameters to their specific hashes. Using a version such as master in @"arvados_sdk_version"@ will grab the latest version hash, which will allow Arvados to re-compute your job if the sdk gets updated.
-* @"arvados_sdk_version"@ : The latest version can be found on the "Arvados Python sdk repository":https://arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
+* @"arvados_sdk_version"@ : The latest version can be found on the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*.
* @"script_version"@ : The current version of your script in your git repository can be found by using the following command:
<notextile>
Note: Job reuse can only happen if all input collections do not change.
-* @"arvados_sdk_version"@ : The arvados_sdk_version parameter is used to download the specific version of the Arvados sdk into the docker image. The latest version can be found in the "Arvados Python sdk repository":https://arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*. Make sure you set this to the same version as the previous run that you are trying to reuse.
+* @"arvados_sdk_version"@ : The arvados_sdk_version parameter is used to download the specific version of the Arvados sdk into the docker image. The latest version can be found in the "Arvados Python sdk repository":https://dev.arvados.org/projects/arvados/repository/revisions/master/show/sdk/python under *Latest revisions*. Make sure you set this to the same version as the previous run that you are trying to reuse.
* @"script_version"@ : The script_version is the commit hash of the git branch that the crunch script resides in. This information can be found in your git repository by using the following command:
<notextile>