4 title: "Using GATK with Arvados"
7 h1. Using GATK with Arvados
9 This tutorial demonstrates how to use the Genome Analysis Toolkit (GATK) with Arvados. In this example we will install GATK and then create a VariantFiltration job to assign pass/fail scores to variants in a VCF file.
11 *This tutorial assumes that you are "logged into an Arvados VM instance":{{site.baseurl}}/user/getting_started/ssh-access.html#login, and have a "working environment.":{{site.baseurl}}/user/getting_started/check-environment.html*
15 Download the GATK binary tarball[1] -- e.g., @GenomeAnalysisTK-2.6-4.tar.bz2@ -- and "copy it to your Arvados VM":tutorial-keep.html.
18 <pre><code>~$ <span class="userinput">arv keep put GenomeAnalysisTK-2.6-4.tar.bz2</span>
19 c905c8d8443a9c44274d98b7c6cfaa32+94
23 Next, you need the GATK Resource Bundle[2]. This may already be available in Arvados. If not, you will need to download the files listed below and put them into Keep.
26 <pre><code>~$ <span class="userinput">arv keep ls -s d237a90bae3870b3b033aea1e99de4a9+10820</span>
27 50342 1000G_omni2.5.b37.vcf.gz
28 1 1000G_omni2.5.b37.vcf.gz.md5
29 464 1000G_omni2.5.b37.vcf.idx.gz
30 1 1000G_omni2.5.b37.vcf.idx.gz.md5
31 43981 1000G_phase1.indels.b37.vcf.gz
32 1 1000G_phase1.indels.b37.vcf.gz.md5
33 326 1000G_phase1.indels.b37.vcf.idx.gz
34 1 1000G_phase1.indels.b37.vcf.idx.gz.md5
35 537210 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.gz
36 1 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.gz.md5
37 3473 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.idx.gz
38 1 CEUTrio.HiSeq.WGS.b37.bestPractices.phased.b37.vcf.idx.gz.md5
39 19403 Mills_and_1000G_gold_standard.indels.b37.vcf.gz
40 1 Mills_and_1000G_gold_standard.indels.b37.vcf.gz.md5
41 536 Mills_and_1000G_gold_standard.indels.b37.vcf.idx.gz
42 1 Mills_and_1000G_gold_standard.indels.b37.vcf.idx.gz.md5
43 29291 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.gz
44 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.gz.md5
45 565 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.idx.gz
46 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.sites.vcf.idx.gz.md5
47 37930 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.gz
48 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.gz.md5
49 592 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.idx.gz
50 1 NA12878.HiSeq.WGS.bwa.cleaned.raw.subset.b37.vcf.idx.gz.md5
51 5898484 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam
52 112 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.bai.gz
53 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.bai.gz.md5
54 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.bam.md5
55 3837 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.gz
56 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.gz.md5
57 65 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.idx.gz
58 1 NA12878.HiSeq.WGS.bwa.cleaned.recal.b37.20.vcf.idx.gz.md5
59 275757 dbsnp_137.b37.excluding_sites_after_129.vcf.gz
60 1 dbsnp_137.b37.excluding_sites_after_129.vcf.gz.md5
61 3735 dbsnp_137.b37.excluding_sites_after_129.vcf.idx.gz
62 1 dbsnp_137.b37.excluding_sites_after_129.vcf.idx.gz.md5
63 998153 dbsnp_137.b37.vcf.gz
64 1 dbsnp_137.b37.vcf.gz.md5
65 3890 dbsnp_137.b37.vcf.idx.gz
66 1 dbsnp_137.b37.vcf.idx.gz.md5
67 58418 hapmap_3.3.b37.vcf.gz
68 1 hapmap_3.3.b37.vcf.gz.md5
69 999 hapmap_3.3.b37.vcf.idx.gz
70 1 hapmap_3.3.b37.vcf.idx.gz.md5
71 3 human_g1k_v37.dict.gz
72 1 human_g1k_v37.dict.gz.md5
73 2 human_g1k_v37.fasta.fai.gz
74 1 human_g1k_v37.fasta.fai.gz.md5
75 849537 human_g1k_v37.fasta.gz
76 1 human_g1k_v37.fasta.gz.md5
77 1 human_g1k_v37.stats.gz
78 1 human_g1k_v37.stats.gz.md5
79 3 human_g1k_v37_decoy.dict.gz
80 1 human_g1k_v37_decoy.dict.gz.md5
81 2 human_g1k_v37_decoy.fasta.fai.gz
82 1 human_g1k_v37_decoy.fasta.fai.gz.md5
83 858592 human_g1k_v37_decoy.fasta.gz
84 1 human_g1k_v37_decoy.fasta.gz.md5
85 1 human_g1k_v37_decoy.stats.gz
86 1 human_g1k_v37_decoy.stats.gz.md5
92 The Arvados distribution includes an example crunch script ("crunch_scripts/GATK2-VariantFiltration":https://arvados.org/projects/arvados/repository/revisions/master/entry/crunch_scripts/GATK2-VariantFiltration) that runs the GATK VariantFiltration tool with some default settings.
95 <pre><code>~$ <span class="userinput">src_version=76588bfc57f33ea1b36b82ca7187f465b73b4ca4</span>
96 ~$ <span class="userinput">vcf_input=5ee633fe2569d2a42dd81b07490d5d13+82</span>
97 ~$ <span class="userinput">gatk_binary=c905c8d8443a9c44274d98b7c6cfaa32+94</span>
98 ~$ <span class="userinput">gatk_bundle=d237a90bae3870b3b033aea1e99de4a9+10820</span>
99 ~$ <span class="userinput">cat >the_job <<EOF
101 "script":"GATK2-VariantFiltration",
102 "script_version":"$src_version",
105 "input":"$vcf_input",
106 "gatk_binary_tarball":"$gatk_binary",
107 "gatk_bundle":"$gatk_bundle"
114 * @"input"@ is collection containing the source VCF data. Here we are using an exome report from PGP participant hu34D5B9.
115 * @"gatk_binary_tarball"@ is a Keep collection containing the GATK 2 binary distribution[1] tar file.
116 * @"gatk_bundle"@ is a Keep collection containing the GATK resource bundle[2].
121 <pre><code>~$ <span class="userinput">arv job create --job "$(cat the_job)"</span>
123 "href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-n9k7qyp7bs5b9d4",
124 "kind":"arvados#job",
125 "etag":"9j99n1feoxw3az448f8ises12",
126 "uuid":"qr1hi-8i9sb-n9k7qyp7bs5b9d4",
127 "owner_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
128 "created_at":"2013-12-17T19:02:15Z",
129 "modified_by_client_uuid":"qr1hi-ozdt8-obw7foaks3qjyej",
130 "modified_by_user_uuid":"qr1hi-tpzed-9zdpkpni2yddge6",
131 "modified_at":"2013-12-17T19:02:15Z",
132 "updated_at":"2013-12-17T19:02:15Z",
135 "script":"GATK2-VariantFiltration",
136 "script_parameters":{
137 "input":"5ee633fe2569d2a42dd81b07490d5d13+82",
138 "gatk_binary_tarball":"c905c8d8443a9c44274d98b7c6cfaa32+94",
139 "gatk_bundle":"d237a90bae3870b3b033aea1e99de4a9+10820"
141 "script_version":"76588bfc57f33ea1b36b82ca7187f465b73b4ca4",
143 "cancelled_by_client_uuid":null,
144 "cancelled_by_user_uuid":null,
150 "is_locked_by_uuid":null,
152 "runtime_constraints":{},
155 "5ee633fe2569d2a42dd81b07490d5d13+82",
156 "c905c8d8443a9c44274d98b7c6cfaa32+94",
157 "d237a90bae3870b3b033aea1e99de4a9+10820"
159 "log_stream_href":"https://qr1hi.arvadosapi.com/arvados/v1/jobs/qr1hi-8i9sb-n9k7qyp7bs5b9d4/log_tail_follow"
161 ~$ <span class="userinput">arv job log_tail_follow --uuid qr1hi-8i9sb-n9k7qyp7bs5b9d4</span>
162 Tue Dec 17 19:02:16 2013 salloc: Granted job allocation 1251
163 Tue Dec 17 19:02:17 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 check slurm allocation
164 Tue Dec 17 19:02:17 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 node compute13 - 8 slots
165 Tue Dec 17 19:02:17 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 start
166 Tue Dec 17 19:02:17 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 Install revision 76588bfc57f33ea1b36b82ca7187f465b73b4ca4
167 Tue Dec 17 19:02:18 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 Clean-work-dir exited 0
168 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 Install exited 0
169 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 script GATK2-VariantFiltration
170 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 script_version 76588bfc57f33ea1b36b82ca7187f465b73b4ca4
171 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 script_parameters {"input":"5ee633fe2569d2a42dd81b07490d5d13+82","gatk_bundle":"d237a90bae3870b3b033aea1e99de4a9+10820","gatk_binary_tarball":"c905c8d8443a9c44274d98b7c6cfaa32+94"}
172 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 runtime_constraints {"max_tasks_per_node":0}
173 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 start level 0
174 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 0 done, 0 running, 1 todo
175 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 0 job_task qr1hi-ot0gb-d3sjxerucfbvyev
176 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 0 child 4946 started on compute13.1
177 Tue Dec 17 19:02:19 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 0 done, 1 running, 0 todo
178 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 0 child 4946 on compute13.1 exit 0 signal 0 success=true
179 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 0 success in 1 seconds
180 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 0 output
181 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 wait for last 0 children to finish
182 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 1 done, 0 running, 1 todo
183 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 start level 1
184 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 1 done, 0 running, 1 todo
185 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 job_task qr1hi-ot0gb-w8ujbnulxjaamxf
186 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 child 4984 started on compute13.1
187 Tue Dec 17 19:02:20 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 1 done, 1 running, 0 todo
188 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 child 4984 on compute13.1 exit 0 signal 0 success=true
189 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 success in 110 seconds
190 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 1 output bedd6ff56b3ae9f90d873b1fcb72f9a3+91
191 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 wait for last 0 children to finish
192 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 status: 2 done, 0 running, 0 todo
193 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 release job allocation
194 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 Freeze not implemented
195 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 collate
196 Tue Dec 17 19:04:10 2013 salloc: Job allocation 1251 has been revoked.
197 Tue Dec 17 19:04:10 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 output bedd6ff56b3ae9f90d873b1fcb72f9a3+91
198 Tue Dec 17 19:04:11 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 finish
199 Tue Dec 17 19:04:12 2013 qr1hi-8i9sb-n9k7qyp7bs5b9d4 4867 log manifest is 1e77aaceee2df499e14dc5dde5c3d328+91
203 Once the job completes, the output can be found in hu34D5B9-exome-filtered.vcf:
205 <notextile><pre><code>~$ <span class="userinput">arv keep ls bedd6ff56b3ae9f90d873b1fcb72f9a3+91</span>
206 hu34D5B9-exome-filtered.vcf
212 fn1. "Download the GATK tools":http://www.broadinstitute.org/gatk/download
214 fn2. "Information about the GATK resource bundle":http://gatkforums.broadinstitute.org/discussion/1213/whats-in-the-resource-bundle-and-how-can-i-get-it and "direct download link":ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/2.5/b37/ (if prompted, submit an empty password)