4 title: "Using GATK with Arvados"
7 Copyright (C) The Arvados Authors. All rights reserved.
9 SPDX-License-Identifier: CC-BY-SA-3.0
12 This tutorial demonstrates how to use the Genome Analysis Toolkit (GATK) with Arvados. In this example we will install GATK and then create a VariantFiltration job to assign pass/fail scores to variants in a VCF file.
14 {% include 'tutorial_expectations' %}
18 Download the GATK binary tarball[1] -- e.g., @GenomeAnalysisTK-2.6-4.tar.bz2@ -- and "copy it to your Arvados VM":{{site.baseurl}}/user/tutorials/tutorial-keep.html.
21 <pre><code>~$ <span class="userinput">arv-put GenomeAnalysisTK-2.6-4.tar.bz2</span>
22 c905c8d8443a9c44274d98b7c6cfaa32+94
26 Next, you need the GATK Resource Bundle[2]. This may already be available in Arvados. If not, you will need to download the files listed below and put them into Keep.
29 <pre><code>~$ <span class="userinput">arv keep ls -s d237a90bae3870b3b033aea1e99de4a9+10820</span>
30 50342 1000G_omni2.5.b37.vcf.gz
31 1 1000G_omni2.5.b37.vcf.gz.md5
32 464 1000G_omni2.5.b37.vcf.idx.gz
33 1 1000G_omni2.5.b37.vcf.idx.gz.md5
34 43981 1000G_phase1.indels.b37.vcf.gz
35 1 1000G_phase1.indels.b37.vcf.gz.md5
36 326 1000G_phase1.indels.b37.vcf.idx.gz
37 1 1000G_phase1.indels.b37.vcf.idx.gz.md5
38 537210 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.gz
39 1 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.gz.md5
40 3473 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.idx.gz
41 1 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.idx.gz.md5
42 19403 Mills_and_1000G_gold_standard.indels.b37.vcf.gz
43 1 Mills_and_1000G_gold_standard.indels.b37.vcf.gz.md5
44 536 Mills_and_1000G_gold_standard.indels.b37.vcf.idx.gz
45 1 Mills_and_1000G_gold_standard.indels.b37.vcf.idx.gz.md5
46 29291 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.gz
47 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.gz.md5
48 565 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.idx.gz
49 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.idx.gz.md5
50 37930 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.gz
51 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.gz.md5
52 592 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.idx.gz
53 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.idx.gz.md5
54 5898484 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam
55 112 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.bai.gz
56 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.bai.gz.md5
57 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.md5
58 3837 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.gz
59 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.gz.md5
60 65 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.idx.gz
61 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.idx.gz.md5
62 275757 dbsnp_137.b37.excluding_sites_after_129.vcf.gz
63 1 dbsnp_137.b37.excluding_sites_after_129.vcf.gz.md5
64 3735 dbsnp_137.b37.excluding_sites_after_129.vcf.idx.gz
65 1 dbsnp_137.b37.excluding_sites_after_129.vcf.idx.gz.md5
66 998153 dbsnp_137.b37.vcf.gz
67 1 dbsnp_137.b37.vcf.gz.md5
68 3890 dbsnp_137.b37.vcf.idx.gz
69 1 dbsnp_137.b37.vcf.idx.gz.md5
70 58418 hapmap_3.3.b37.vcf.gz
71 1 hapmap_3.3.b37.vcf.gz.md5
72 999 hapmap_3.3.b37.vcf.idx.gz
73 1 hapmap_3.3.b37.vcf.idx.gz.md5
74 3 human_g1k_v37.dict.gz
75 1 human_g1k_v37.dict.gz.md5
76 2 human_g1k_v37.fasta.fai.gz
77 1 human_g1k_v37.fasta.fai.gz.md5
78 849537 human_g1k_v37.fasta.gz
79 1 human_g1k_v37.fasta.gz.md5
80 1 human_g1k_v37.stats.gz
81 1 human_g1k_v37.stats.gz.md5
82 3 human_g1k_v37_decoy.dict.gz
83 1 human_g1k_v37_decoy.dict.gz.md5
84 2 human_g1k_v37_decoy.fasta.fai.gz
85 1 human_g1k_v37_decoy.fasta.fai.gz.md5
86 858592 human_g1k_v37_decoy.fasta.gz
87 1 human_g1k_v37_decoy.fasta.gz.md5
88 1 human_g1k_v37_decoy.stats.gz
89 1 human_g1k_v37_decoy.stats.gz.md5
95 The Arvados distribution includes an example crunch script ("crunch_scripts/GATK2-VariantFiltration":https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-VariantFiltration) that runs the GATK VariantFiltration tool with some default settings.
98 <pre><code>~$ <span class="userinput">src_version=76588bfc57f33ea1b36b82ca7187f465b73b4ca4</span>
99 ~$ <span class="userinput">vcf_input=5ee633fe2569d2a42dd81b07490d5d13+82</span>
100 ~$ <span class="userinput">gatk_binary=c905c8d8443a9c44274d98b7c6cfaa32+94</span>
101 ~$ <span class="userinput">gatk_bundle=d237a90bae3870b3b033aea1e99de4a9+10820</span>
102 ~$ <span class="userinput">cat >the_job <<EOF
104 "script":"GATK2-VariantFiltration",
105 "repository":"arvados",
106 "script_version":"$src_version",
109 "input":"$vcf_input",
110 "gatk_binary_tarball":"$gatk_binary",
111 "gatk_bundle":"$gatk_bundle"
118 * @"input"@ is collection containing the source VCF data. Here we are using an exome report from PGP participant hu34D5B9.
119 * @"gatk_binary_tarball"@ is a Keep collection containing the GATK 2 binary distribution[1] tar file.
120 * @"gatk_bundle"@ is a Keep collection containing the GATK resource bundle[2].
125 <pre><code>~$ <span class="userinput">arv job create --job "$(cat the_job)"</span>
127 "href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-n9k7qyp7bs5b9d4",
128 "kind":"arvados#job",
129 "etag":"9j99n1feoxw3az448f8ises12",
130 "uuid":"qr1hi-8i9sb-n9k7qyp7bs5b9d4",
131 "owner_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
132 "created_at":"2013-12-17T19:02:15Z",
133 "modified_by_client_uuid":"qr1hi-ozdt8-obw7foaks3qjyej",
134 "modified_by_user_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
135 "modified_at":"2013-12-17T19:02:15Z",
136 "updated_at":"2013-12-17T19:02:15Z",
139 "script":"GATK2-VariantFiltration",
140 "script_parameters":{
141 "input":"5ee633fe2569d2a42dd81b07490d5d13+82",
142 "gatk_binary_tarball":"c905c8d8443a9c44274d98b7c6cfaa32+94",
143 "gatk_bundle":"d237a90bae3870b3b033aea1e99de4a9+10820"
145 "script_version":"76588bfc57f33ea1b36b82ca7187f465b73b4ca4",
147 "cancelled_by_client_uuid":null,
148 "cancelled_by_user_uuid":null,
154 "is_locked_by_uuid":null,
156 "runtime_constraints":{},
162 Once the job completes, the output can be found in hu34D5B9-exome-filtered.vcf:
164 <notextile><pre><code>~$ <span class="userinput">arv keep ls bedd6ff56b3ae9f90d873b1fcb72f9a3+91</span>
165 hu34D5B9-exome-filtered.vcf
171 fn1. "Download the GATK tools":http://www.broadinstitute.org/gatk/download
173 fn2. "Information about the GATK resource bundle":http://gatkforums.broadinstitute.org/discussion/1213/whats-in-the-resource-bundle-and-how-can-i-get-it and "direct download link":ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/2.5/b37/ (if prompted, submit an empty password)