Tom Clegg [Tue, 14 Sep 2021 20:15:35 +0000 (16:15 -0400)]
Fix mem usage, improve logging.
refs #17996
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 13 Sep 2021 15:23:23 +0000 (11:23 -0400)]
Generate annotations for slices.
refs #17996
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 13 Sep 2021 14:01:49 +0000 (10:01 -0400)]
Generate numpy matrices from slices.
refs #17996
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Sun, 12 Sep 2021 19:24:45 +0000 (15:24 -0400)]
Always log version at startup.
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Sun, 12 Sep 2021 19:24:20 +0000 (15:24 -0400)]
Slice imported data by tag#.
refs #17996
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 9 Sep 2021 15:28:08 +0000 (11:28 -0400)]
Use callbacks in struct instead of args to Load*().
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 1 Sep 2021 17:44:57 +0000 (13:44 -0400)]
Fix missed TranslatePaths and unreported error.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 31 Aug 2021 18:56:04 +0000 (14:56 -0400)]
Add -p-value and -cases options for exporting hgvs numpy.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 31 Aug 2021 18:55:56 +0000 (14:55 -0400)]
Add chi square func.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 12 Aug 2021 13:56:50 +0000 (09:56 -0400)]
Fix unchecked error.
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 12 Aug 2021 03:16:36 +0000 (23:16 -0400)]
Fix array index out of bounds.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 11 Aug 2021 22:10:26 +0000 (18:10 -0400)]
Add -match-genome=regexp filter.
refs #17939
refs #17922
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 11 Aug 2021 21:31:23 +0000 (17:31 -0400)]
Export hgvs one-hot numpy: -1 for missing / low quality tiles.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 10 Aug 2021 19:26:37 +0000 (15:26 -0400)]
Add filter options to export cmd.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 10 Aug 2021 13:18:10 +0000 (09:18 -0400)]
Add .licenseignore.
No issue #
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 10 Aug 2021 13:16:58 +0000 (09:16 -0400)]
Memory tweaks.
refs #17562
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 6 Aug 2021 16:49:21 +0000 (12:49 -0400)]
Add copyright headers.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 6 Aug 2021 16:11:32 +0000 (12:11 -0400)]
Reduce memory use, Limit goroutines when exporting numpy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 3 Aug 2021 17:18:14 +0000 (13:18 -0400)]
Reduce lock contention.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 3 Aug 2021 15:27:08 +0000 (11:27 -0400)]
Write annotations through to conserve memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 2 Aug 2021 19:09:51 +0000 (15:09 -0400)]
Export hgvs one-hot numpy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 30 Jul 2021 13:20:00 +0000 (09:20 -0400)]
Use tsv filename when using tab separator.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 29 Jul 2021 20:49:24 +0000 (16:49 -0400)]
Track which tile variants each hgvs.Variant appeared in.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 15 Jul 2021 15:07:08 +0000 (11:07 -0400)]
Skip diffs on long ref seqs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Jul 2021 19:37:38 +0000 (15:37 -0400)]
Fix some entries skipped by WriteDir.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Jul 2021 15:25:00 +0000 (11:25 -0400)]
Compress export output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 13 Jul 2021 14:09:16 +0000 (10:09 -0400)]
Fix up VCF format.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jul 2021 16:07:31 +0000 (12:07 -0400)]
Fix recomputing diffs N times.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Jul 2021 14:20:11 +0000 (10:20 -0400)]
Separate pvcf/vcf output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jul 2021 17:04:42 +0000 (13:04 -0400)]
Progress indicator for exportSeq.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jul 2021 14:32:46 +0000 (10:32 -0400)]
Improve concurrency in export-diff.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 9 Jul 2021 13:13:03 +0000 (09:13 -0400)]
Include tagset in reference-genome files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 23:06:07 +0000 (19:06 -0400)]
Fix missing output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 20:48:14 +0000 (16:48 -0400)]
Write ref seqs to their own files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 20:48:04 +0000 (16:48 -0400)]
Use more cpus.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 20:47:35 +0000 (16:47 -0400)]
Fix use of nil writer.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 15:25:43 +0000 (11:25 -0400)]
Enable network access to get port forwarding.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Jul 2021 14:24:44 +0000 (10:24 -0400)]
Option to export one vcf/csv file per chromosome.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 18 Jun 2021 13:26:31 +0000 (09:26 -0400)]
Fix race.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 21:17:26 +0000 (17:17 -0400)]
Improve concurrency more.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 20:14:48 +0000 (16:14 -0400)]
Improve loading concurrency.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 19:23:37 +0000 (15:23 -0400)]
Split sequence data into 64K independently locked partitions.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 18:37:57 +0000 (14:37 -0400)]
Fix exportnumpy test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:25:56 +0000 (11:25 -0400)]
Add concurrent load to exportnumpy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:25:44 +0000 (11:25 -0400)]
Skip expensive Tracef.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:15:32 +0000 (11:15 -0400)]
Up load throttle to gomaxprocs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:13:13 +0000 (11:13 -0400)]
Move alloc/copy out of lock.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 15:03:47 +0000 (11:03 -0400)]
Limit load goroutines to gomaxprocs/2.
Otherwise we bottleneck early on lots of IO that we can't process that
quickly.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 14:22:15 +0000 (10:22 -0400)]
Distribute genomes across output files.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Jun 2021 14:02:25 +0000 (10:02 -0400)]
Fix lock held during getRef.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Jun 2021 15:02:33 +0000 (11:02 -0400)]
Fixup multiple-file reading.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Jun 2021 13:03:23 +0000 (09:03 -0400)]
Fix "gob: encoder: message too big"
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Jun 2021 04:47:35 +0000 (00:47 -0400)]
Add "flake" command (read, tidy, write).
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Jun 2021 15:12:58 +0000 (11:12 -0400)]
Add concurrency in Tidy remap phase.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Jun 2021 15:10:54 +0000 (11:10 -0400)]
Adjust memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Jun 2021 03:09:54 +0000 (23:09 -0400)]
Write hgvs-based numpy matrix.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 3 Jun 2021 20:27:44 +0000 (16:27 -0400)]
Fix another category of misspelled hgvs diff.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 25 May 2021 13:40:21 +0000 (09:40 -0400)]
Bring memory back down.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 24 May 2021 14:54:42 +0000 (10:54 -0400)]
Limit tile size in export.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 21 May 2021 14:30:28 +0000 (10:30 -0400)]
Save runtime profile data periodically.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 20 May 2021 06:01:14 +0000 (02:01 -0400)]
Don't try to buffer/sort export.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 19 May 2021 00:45:35 +0000 (20:45 -0400)]
More memory, but release buffers.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 18 May 2021 20:51:07 +0000 (16:51 -0400)]
Manage export memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 18 May 2021 20:50:46 +0000 (16:50 -0400)]
Add test.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 11 May 2021 05:41:09 +0000 (01:41 -0400)]
Avoid buffering entire output in memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 7 May 2021 13:47:18 +0000 (09:47 -0400)]
Export hgvs-onehot.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 26 Apr 2021 15:32:13 +0000 (11:32 -0400)]
Bump exportnumpy memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 23 Apr 2021 17:05:32 +0000 (13:05 -0400)]
Bump merge memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 23 Apr 2021 17:05:24 +0000 (13:05 -0400)]
Error out early.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Apr 2021 20:33:21 +0000 (16:33 -0400)]
Bump exportnumpy memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 12 Apr 2021 13:19:45 +0000 (09:19 -0400)]
Bump merge memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 8 Apr 2021 20:04:05 +0000 (16:04 -0400)]
Fix HGVS diff: GGAA>AAAA is GG>AA, not delGG,=AA,insAA.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 1 Apr 2021 17:42:32 +0000 (13:42 -0400)]
Add -expand-regions flag.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 24 Mar 2021 18:07:05 +0000 (14:07 -0400)]
Export selected regions.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 15 Mar 2021 14:50:10 +0000 (10:50 -0400)]
Test numpy annotations.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 11 Mar 2021 15:54:42 +0000 (10:54 -0500)]
Option to write numpy in chunks (same rows, fewer columns).
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 4 Mar 2021 01:31:30 +0000 (20:31 -0500)]
Bump export RAM.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 21:30:31 +0000 (16:30 -0500)]
Bump import RAM.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 20:53:44 +0000 (15:53 -0500)]
Ask for preemptible instances.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 20:47:45 +0000 (15:47 -0500)]
Reduce import RAM.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 1 Mar 2021 20:35:31 +0000 (15:35 -0500)]
Throttle goroutines and unflushed bufs to NCPUs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 19 Feb 2021 15:54:16 +0000 (10:54 -0500)]
Upgrade to debian bullseye to get bcftools 1.11.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 8 Feb 2021 21:11:13 +0000 (16:11 -0500)]
Add missing file for batchArgs.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 8 Feb 2021 16:17:53 +0000 (11:17 -0500)]
Move command line tool to subdir.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 29 Jan 2021 16:23:56 +0000 (11:23 -0500)]
Fake a complete_genomics_pass_all preset.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 29 Jan 2021 01:34:07 +0000 (20:34 -0500)]
Log # low quality tile variants during import.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 29 Jan 2021 01:33:19 +0000 (20:33 -0500)]
Log exported numpy shape.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 18 Jan 2021 18:36:37 +0000 (13:36 -0500)]
Add dumpgob command.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 14 Jan 2021 20:36:10 +0000 (15:36 -0500)]
Fix log message.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 18 Dec 2020 16:00:44 +0000 (11:00 -0500)]
Share a single sitefs for multiple arvados files.
Avoid needlessly re-fetching the manifest when reading multiple files
from one collection.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Fri, 18 Dec 2020 15:52:57 +0000 (10:52 -0500)]
Bump import memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Thu, 17 Dec 2020 15:36:08 +0000 (10:36 -0500)]
Bump exportnumpy memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Wed, 16 Dec 2020 19:49:55 +0000 (14:49 -0500)]
Sort rows by label shown in csv, not full file path.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Dec 2020 14:31:30 +0000 (09:31 -0500)]
Bump merge memory.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 15 Dec 2020 14:31:13 +0000 (09:31 -0500)]
Load library with pgzip.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Mon, 14 Dec 2020 21:00:37 +0000 (16:00 -0500)]
Faster merge output.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>
Tom Clegg [Tue, 8 Dec 2020 15:35:01 +0000 (10:35 -0500)]
Bigger output buffer for annotate.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Tue, 8 Dec 2020 15:34:07 +0000 (10:34 -0500)]
Save git version in lightning binary collection name and properties.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Fri, 4 Dec 2020 15:47:13 +0000 (10:47 -0500)]
Include numpy matrix filename in labels csv.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
Tom Clegg [Fri, 4 Dec 2020 15:46:42 +0000 (10:46 -0500)]
More memory + direct Keep access for merge and exportnumpy.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>
projects / lightning.git / log