Add "crunch examples" and "crunch utility libraries" pages

in doc/user/.

refs #1513
refs #1656

diff --git a/doc/user/crunch-examples.textile b/doc/user/crunch-examples.textile
new file mode 100644 (file)
index 0000000..fbf3cf9
--- /dev/null
+++ b/doc/user/crunch-examples.textile
@@ -0,0 +1,35 @@
+---
+layout: default
+navsection: userguide
+title: "Crunch examples"
+navorder: 30
+---
+
+h1. Crunch examples
+
+Several crunch scripts are included with Arvados in the @/crunch_scripts@ directory. They are intended to provide examples and starting points for writing your own scripts.
+
+h4. bwa-aln
+
+Run the bwa aligner on a set of paired-end fastq files, producing a BAM file for each pair.
+
+h4. bwa-index
+
+Generate an index of a fasta reference genome suitable for use by bwa-aln.
+
+h4. picard-gatk2-prep
+
+Using the FixMateInformation, SortSam, ReorderSam, AddOrReplaceReadGroups, and BuildBamIndex modules from picard, prepare a BAM file for use with the GATK2 tools. Additionally, run picard's CollectAlignmentSummaryMetrics module to produce a @*.casm.tsv@ statistics file for each BAM file.
+
+h4. GATK2-bqsr
+
+Run GATK's BaseQualityScoreRecalibration module on a set of BAM files.
+
+h4. GATK2-merge-call
+
+Merge a set of BAM files using picard, and run GATK's UnifiedGenotyper module on the merged set to produce a VCF file.
+
+h4. GATK2-realign
+
+Run GATK's RealignerTargetCreator and IndelRealigner modules on a set of BAM files.
+

diff --git a/doc/user/crunch-utility-libraries.textile b/doc/user/crunch-utility-libraries.textile
new file mode 100644 (file)
index 0000000..42ffc43
--- /dev/null
+++ b/doc/user/crunch-utility-libraries.textile
@@ -0,0 +1,240 @@
+---
+layout: default
+navsection: userguide
+title: "Crunch utility libraries"
+navorder: 31
+---
+
+h1. Crunch utility libraries
+
+Several utility libraries are included with Arvados. They are intended to make it quicker and easier to write your own crunch scripts.
+
+h4. Python SDK extras
+
+The Python SDK adds some convenience features that are particularly useful in crunch scripts, in addition to the standard set of API calls.
+
+<div class="offset1">
+
+In a crunch job, the environment variables @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ will be set up so the job has the privileges of the user who submitted the job.
+
+<pre>
+import arvados
+
+my_user = arvados.api().users().current().execute()
+my_uuid = my_user['uuid']
+</pre>
+
+h4. Get the current job and task parameters
+
+@arvados.current_job()@ and @arvados.current_task()@ are convenient ways to retrieve the current Job and Task, using the @JOB_UUID@ and @TASK_UUID@ environment variables provided to each crunch task process.
+
+<pre>
+this_job = arvados.current_job()
+this_task = arvados.current_task()
+this_job_input = this_job['script_parameters']['input']
+this_task_input = this_task['parameters']['input']
+</pre>
+
+h4. Queue a task for each input file
+
+A common pattern for a crunch job is to run one task to scan the input, and one task per input file to do the work.
+
+The @one_task_per_input_file()@ function implements this pattern. Pseudocode:
+
+<pre>
+if this is the job's first (default) task:
+    for each file in the 'input' collection:
+        queue a new task, with parameters['input'] = file
+    exit
+else:
+    return
+</pre>
+
+Usage:
+
+<pre>
+import arvados
+arvados.job_setup.one_task_per_input_file(if_sequence=0, and_end_task=True)
+
+# Now do the work on a single file
+my_input = this_task['parameters']['input']
+</pre>
+
+h4. Set the current task's output and success flag
+
+Each task in a crunch job must make an API call to record its output and set its @success@ attribute to True. The object returned by @current_task()@ has a @set_output()@ method to make the process more succinct.
+
+<pre>
+arvados.current_task().set_output(my_output_locator)
+</pre>
+
+</div>
+
+h4. arvados_ipc.py
+
+Manage child processes and FIFOs (pipes).
+
+<div class="offset1">
+
+This module makes it easier to check the exit status of every child process you start, and close the unused end of each FIFO at the appropriate time.
+
+<pre>
+from arvados_ipc import *
+
+children = {}
+pipes = {}
+
+pipe_setup(pipes, 'hellopipe')
+if 0 == named_fork(children, 'child_a'):
+    pipe_closeallbut(pipes, ('hellopipe', 'w'))
+    os.write(pipes['hellopipe', 'w'], "Hello, parent.")
+    os._exit(0)
+
+pipe_closeallbut(pipes, ('hellopipe', 'r'))
+with os.fdopen(pipes['hellopipe', 'r'], 'rb') as f:
+    message = f.read()
+    sys.stderr.write("Child says: " + message + "\n")
+
+if not waitpid_and_check_children(children):
+    raise Exception("Child process exited non-zero.")
+</pre>
+
+The "crunch scripts" included with Arvados include some more examples of using the arvados_ipc module.
+
+</div>
+
+h3. Toolkit wrappers
+
+The following *arvados-&lowast;.py* modules provide "extract, build, run" helpers to make it easy to incorporate common analysis tools in your crunch scripts.
+
+h4. arvados_bwa.py
+
+Build and run the "bwa":http://bio-bwa.sourceforge.net/bwa.shtml program.
+
+<div class="offset1">
+
+The module retrieves the bwa source code from Keep, using the job's @bwa_tbz@ parameter.
+
+<pre>
+import arvados_bwa
+arvados_bwa.run('aln', [ref_basename, '-'],
+                stdin=open(fastq_filename,'rb'),
+                stdout=open(aln_filename,'wb'))
+</pre>
+
+On qr1hi.arvadosapi.com, the source distribution @bwa-0.7.5a.tar.bz2@ is available in the collection @8b6e2c4916133e1d859c9e812861ce13+70@.
+
+<pre>
+{
+ "script_parameters":{
+  "bwa_tbz":"8b6e2c4916133e1d859c9e812861ce13+70",
+  ...
+ },
+ ...
+}
+</pre>
+
+</div>
+
+h4. arvados_gatk2.py
+
+Extract and run the "Genome Analysis Toolkit":http://www.broadinstitute.org/gatk/ programs.
+
+<div class="offset1">
+
+The module retrieves the binary distribution tarball from Keep, using the job's @gatk_tbz@ parameter.
+
+<pre>
+arvados_gatk2.run(
+    args=[
+        '-nct', 8,
+        '-T', 'BaseRecalibrator',
+        '-R', ref_fasta_files[0],
+        '-I', input_bam_files[0],
+        '-o', recal_file,
+        ])
+</pre>
+
+On qr1hi.arvadosapi.com, the binary distribution @GenomeAnalysisTK-2.6-4.tar.bz2@ is available in the collection @5790482512cf6d5d6dfd50b7fd61e1d1+86@.
+
+The GATK data bundle is available in the collection @d237a90bae3870b3b033aea1e99de4a9+10820@.
+
+<pre>
+{
+ "script_parameters":{
+  "gatk_tbz":"5790482512cf6d5d6dfd50b7fd61e1d1+86",
+  "gatk_bundle":"d237a90bae3870b3b033aea1e99de4a9+10820",
+  ...
+ },
+ ...
+}
+</pre>
+
+</div>
+
+h4. arvados_samtools.py
+
+Build and run the "samtools":http://samtools.sourceforge.net/samtools.shtml program.
+
+<div class="offset1">
+
+The module retrieves the samtools source code from Keep, using the job's @samtools_tgz@ parameter.
+
+<pre>
+import arvados_samtools
+arvados_samtools.run('view', ['-S', '-b', '-'],
+                     stdin=open(sam_filename,'rb'),
+                     stdout=open(bam_filename,'wb'))
+</pre>
+
+On qr1hi.arvadosapi.com, the source distribution @samtools-0.1.19.tar.gz@ is available in the collection @c777e23cf13e5d5906abfdc08d84bfdb+74@.
+
+<pre>
+{
+ "script_parameters":{
+  "samtools_tgz":"c777e23cf13e5d5906abfdc08d84bfdb+74",
+  ...
+ },
+ ...
+}
+</pre>
+
+</div>
+
+h4. arvados_picard.py
+
+Build and run the "picard":http://picard.sourceforge.net/command-line-overview.shtml program.
+
+<div class="offset1">
+
+The module retrieves the picard distribution from Keep, using the job's @picard_zip@ parameter.
+
+<pre>
+import arvados_picard
+arvados_picard.run(
+    'FixMateInformation',
+    params={
+        'i': input_bam_path,
+        'o': '/dev/stdout',
+        'quiet': 'true',
+        'so': 'coordinate',
+        'validation_stringency': 'LENIENT',
+        'compression_level': 0
+        },
+    stdout=open('out.bam','wb'))
+</pre>
+
+On qr1hi.arvadosapi.com, the source distribution @picard-tools-1.82.zip@ is available in the collection @687f74675c6a0e925dec619cc2bec25f+77@.
+
+<pre>
+{
+ "script_parameters":{
+  "picard_zip":"687f74675c6a0e925dec619cc2bec25f+77",
+  ...
+ },
+ ...
+}
+</pre>
+
+</div>
+