Tom Clegg [Tue, 13 Jul 2021 15:25:00 +0000 (11:25 -0400)]
Compress export output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Jul 2021 14:09:16 +0000 (10:09 -0400)]
Fix up VCF format.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jul 2021 16:07:31 +0000 (12:07 -0400)]
Fix recomputing diffs N times.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jul 2021 14:20:11 +0000 (10:20 -0400)]
Separate pvcf/vcf output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jul 2021 17:04:42 +0000 (13:04 -0400)]
Progress indicator for exportSeq.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jul 2021 14:32:46 +0000 (10:32 -0400)]
Improve concurrency in export-diff.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jul 2021 13:13:03 +0000 (09:13 -0400)]
Include tagset in reference-genome files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 23:06:07 +0000 (19:06 -0400)]
Fix missing output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 20:48:14 +0000 (16:48 -0400)]
Write ref seqs to their own files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 20:48:04 +0000 (16:48 -0400)]
Use more cpus.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 20:47:35 +0000 (16:47 -0400)]
Fix use of nil writer.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 15:25:43 +0000 (11:25 -0400)]
Enable network access to get port forwarding.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 14:24:44 +0000 (10:24 -0400)]
Option to export one vcf/csv file per chromosome.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 18 Jun 2021 13:26:31 +0000 (09:26 -0400)]
Fix race.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 21:17:26 +0000 (17:17 -0400)]
Improve concurrency more.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 20:14:48 +0000 (16:14 -0400)]
Improve loading concurrency.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 19:23:37 +0000 (15:23 -0400)]
Split sequence data into 64K independently locked partitions.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 18:37:57 +0000 (14:37 -0400)]
Fix exportnumpy test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:25:56 +0000 (11:25 -0400)]
Add concurrent load to exportnumpy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:25:44 +0000 (11:25 -0400)]
Skip expensive Tracef.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:15:32 +0000 (11:15 -0400)]
Up load throttle to gomaxprocs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:13:13 +0000 (11:13 -0400)]
Move alloc/copy out of lock.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:03:47 +0000 (11:03 -0400)]
Limit load goroutines to gomaxprocs/2.
Otherwise we bottleneck early on lots of IO that we can't process that
quickly.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 14:22:15 +0000 (10:22 -0400)]
Distribute genomes across output files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 14:02:25 +0000 (10:02 -0400)]
Fix lock held during getRef.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Jun 2021 15:02:33 +0000 (11:02 -0400)]
Fixup multiple-file reading.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Jun 2021 13:03:23 +0000 (09:03 -0400)]
Fix "gob: encoder: message too big"
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Jun 2021 04:47:35 +0000 (00:47 -0400)]
Add "flake" command (read, tidy, write).
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Jun 2021 15:12:58 +0000 (11:12 -0400)]
Add concurrency in Tidy remap phase.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Jun 2021 15:10:54 +0000 (11:10 -0400)]
Adjust memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Jun 2021 03:09:54 +0000 (23:09 -0400)]
Write hgvs-based numpy matrix.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 3 Jun 2021 20:27:44 +0000 (16:27 -0400)]
Fix another category of misspelled hgvs diff.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 25 May 2021 13:40:21 +0000 (09:40 -0400)]
Bring memory back down.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 24 May 2021 14:54:42 +0000 (10:54 -0400)]
Limit tile size in export.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 21 May 2021 14:30:28 +0000 (10:30 -0400)]
Save runtime profile data periodically.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 20 May 2021 06:01:14 +0000 (02:01 -0400)]
Don't try to buffer/sort export.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 19 May 2021 00:45:35 +0000 (20:45 -0400)]
More memory, but release buffers.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 18 May 2021 20:51:07 +0000 (16:51 -0400)]
Manage export memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 18 May 2021 20:50:46 +0000 (16:50 -0400)]
Add test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 11 May 2021 05:41:09 +0000 (01:41 -0400)]
Avoid buffering entire output in memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 7 May 2021 13:47:18 +0000 (09:47 -0400)]
Export hgvs-onehot.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 26 Apr 2021 15:32:13 +0000 (11:32 -0400)]
Bump exportnumpy memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 23 Apr 2021 17:05:32 +0000 (13:05 -0400)]
Bump merge memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 23 Apr 2021 17:05:24 +0000 (13:05 -0400)]
Error out early.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Apr 2021 20:33:21 +0000 (16:33 -0400)]
Bump exportnumpy memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Apr 2021 13:19:45 +0000 (09:19 -0400)]
Bump merge memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Apr 2021 20:04:05 +0000 (16:04 -0400)]
Fix HGVS diff: GGAA>AAAA is GG>AA, not delGG,=AA,insAA.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 1 Apr 2021 17:42:32 +0000 (13:42 -0400)]
Add -expand-regions flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 24 Mar 2021 18:07:05 +0000 (14:07 -0400)]
Export selected regions.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 15 Mar 2021 14:50:10 +0000 (10:50 -0400)]
Test numpy annotations.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 11 Mar 2021 15:54:42 +0000 (10:54 -0500)]
Option to write numpy in chunks (same rows, fewer columns).
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 4 Mar 2021 01:31:30 +0000 (20:31 -0500)]
Bump export RAM.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 21:30:31 +0000 (16:30 -0500)]
Bump import RAM.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 20:53:44 +0000 (15:53 -0500)]
Ask for preemptible instances.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 20:47:45 +0000 (15:47 -0500)]
Reduce import RAM.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 20:35:31 +0000 (15:35 -0500)]
Throttle goroutines and unflushed bufs to NCPUs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 19 Feb 2021 15:54:16 +0000 (10:54 -0500)]
Upgrade to debian bullseye to get bcftools 1.11.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 8 Feb 2021 21:11:13 +0000 (16:11 -0500)]
Add missing file for batchArgs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 8 Feb 2021 16:17:53 +0000 (11:17 -0500)]
Move command line tool to subdir.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 29 Jan 2021 16:23:56 +0000 (11:23 -0500)]
Fake a complete_genomics_pass_all preset.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 29 Jan 2021 01:34:07 +0000 (20:34 -0500)]
Log # low quality tile variants during import.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 29 Jan 2021 01:33:19 +0000 (20:33 -0500)]
Log exported numpy shape.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 18 Jan 2021 18:36:37 +0000 (13:36 -0500)]
Add dumpgob command.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 14 Jan 2021 20:36:10 +0000 (15:36 -0500)]
Fix log message.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 18 Dec 2020 16:00:44 +0000 (11:00 -0500)]
Share a single sitefs for multiple arvados files.
Avoid needlessly re-fetching the manifest when reading multiple files
from one collection.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 18 Dec 2020 15:52:57 +0000 (10:52 -0500)]
Bump import memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Dec 2020 15:36:08 +0000 (10:36 -0500)]
Bump exportnumpy memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 16 Dec 2020 19:49:55 +0000 (14:49 -0500)]
Sort rows by label shown in csv, not full file path.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Dec 2020 14:31:30 +0000 (09:31 -0500)]
Bump merge memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Dec 2020 14:31:13 +0000 (09:31 -0500)]
Load library with pgzip.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Dec 2020 21:00:37 +0000 (16:00 -0500)]
Faster merge output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 8 Dec 2020 15:35:01 +0000 (10:35 -0500)]
Bigger output buffer for annotate.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Tue, 8 Dec 2020 15:34:07 +0000 (10:34 -0500)]
Save git version in lightning binary collection name and properties.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Fri, 4 Dec 2020 15:47:13 +0000 (10:47 -0500)]
Include numpy matrix filename in labels csv.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Fri, 4 Dec 2020 15:46:42 +0000 (10:46 -0500)]
More memory + direct Keep access for merge and exportnumpy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 2 Dec 2020 20:49:20 +0000 (15:49 -0500)]
Update example.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 2 Dec 2020 20:48:01 +0000 (15:48 -0500)]
Don't pass -gvcf-type="" to gvcf_regions.py.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 2 Dec 2020 20:47:48 +0000 (15:47 -0500)]
Fix deadlock on error.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 2 Dec 2020 20:47:44 +0000 (15:47 -0500)]
Improve import and vcf2fasta performance.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Sun, 29 Nov 2020 17:53:04 +0000 (12:53 -0500)]
Increase concurrency, reduce allocs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Sun, 29 Nov 2020 17:52:29 +0000 (12:52 -0500)]
Reduce lock contention.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 25 Nov 2020 21:25:14 +0000 (16:25 -0500)]
Use smaller machines for small batches.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 25 Nov 2020 20:50:16 +0000 (15:50 -0500)]
Use buffered writer to avoid overwhelming arv-mount.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 25 Nov 2020 14:24:25 +0000 (09:24 -0500)]
Share CR refresh throttle when running multiple containers.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Wed, 25 Nov 2020 06:07:46 +0000 (01:07 -0500)]
Export labels.csv with numpy array.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Tue, 24 Nov 2020 20:10:29 +0000 (15:10 -0500)]
Concurrent-batches mode for vcf2fasta and import.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Mon, 23 Nov 2020 02:43:41 +0000 (21:43 -0500)]
More memory for pca.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Mon, 23 Nov 2020 02:43:15 +0000 (21:43 -0500)]
Propagate -match-chromosome arg to container.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Sun, 22 Nov 2020 17:07:14 +0000 (12:07 -0500)]
Pass gvcf-type to gvcf_regions.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Sun, 22 Nov 2020 08:21:19 +0000 (03:21 -0500)]
Fix fasta sequence names.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Sun, 22 Nov 2020 05:50:40 +0000 (00:50 -0500)]
Accept filter args in pca-go.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Sun, 22 Nov 2020 05:48:57 +0000 (00:48 -0500)]
Let caller set container output name.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Thu, 19 Nov 2020 05:38:25 +0000 (00:38 -0500)]
Fix divide by zero.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Thu, 19 Nov 2020 01:39:46 +0000 (20:39 -0500)]
Configurable chromosome name pattern.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Thu, 19 Nov 2020 01:24:15 +0000 (20:24 -0500)]
Fix wrong container name.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Thu, 19 Nov 2020 01:22:43 +0000 (20:22 -0500)]
Indicate low quality tile variants with -1 in numpy array.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Fri, 13 Nov 2020 07:43:37 +0000 (02:43 -0500)]
Gzip gob files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Fri, 13 Nov 2020 07:19:29 +0000 (02:19 -0500)]
Update example.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Thu, 12 Nov 2020 23:56:48 +0000 (18:56 -0500)]
When not saving incomplete tilevars, still save hashes/indexes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Thu, 5 Nov 2020 18:29:49 +0000 (13:29 -0500)]
Add numpy-common-variants sanity check.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
projects / lightning.git / log