lightning.git
4 years agoOption to skip/ignore out-of-order tags.
Tom Clegg [Mon, 6 Apr 2020 14:09:46 +0000 (10:09 -0400)]
Option to skip/ignore out-of-order tags.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFixup tagset collection in example script.
Tom Clegg [Mon, 6 Apr 2020 14:08:44 +0000 (10:08 -0400)]
Fixup tagset collection in example script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRequest more memory when using mask.
Tom Clegg [Fri, 3 Apr 2020 16:36:53 +0000 (12:36 -0400)]
Request more memory when using mask.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRead contig sizes from vcf.
Tom Clegg [Fri, 3 Apr 2020 00:14:30 +0000 (20:14 -0400)]
Read contig sizes from vcf.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoKill other children and exit early if one child process fails.
Tom Clegg [Thu, 2 Apr 2020 20:14:12 +0000 (16:14 -0400)]
Kill other children and exit early if one child process fails.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoSort genome file same way as vcf regions.
Tom Clegg [Thu, 2 Apr 2020 20:11:08 +0000 (16:11 -0400)]
Sort genome file same way as vcf regions.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agos/python/python2/
Tom Clegg [Thu, 12 Mar 2020 03:26:11 +0000 (23:26 -0400)]
s/python/python2/

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd python2 to docker image.
Tom Clegg [Thu, 12 Mar 2020 01:45:30 +0000 (21:45 -0400)]
Add python2 to docker image.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix example script.
Tom Clegg [Thu, 12 Mar 2020 01:06:19 +0000 (21:06 -0400)]
Fix example script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoPropagate genome arg.
Tom Clegg [Thu, 12 Mar 2020 01:06:09 +0000 (21:06 -0400)]
Propagate genome arg.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRefactor websocket listener.
Tom Clegg [Wed, 11 Mar 2020 20:43:37 +0000 (16:43 -0400)]
Refactor websocket listener.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoDon't pipe gvcf_regions to bedtools. Avoid short-read race bug.
Tom Clegg [Wed, 11 Mar 2020 19:34:52 +0000 (15:34 -0400)]
Don't pipe gvcf_regions to bedtools. Avoid short-read race bug.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUse named pipe for bcftools --mask data.
Tom Clegg [Wed, 11 Mar 2020 18:53:49 +0000 (14:53 -0400)]
Use named pipe for bcftools --mask data.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoShow file path in ref2genome -local=false.
Tom Clegg [Wed, 11 Mar 2020 14:57:22 +0000 (10:57 -0400)]
Show file path in ref2genome -local=false.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd -priority to example script.
Tom Clegg [Wed, 11 Mar 2020 14:20:33 +0000 (10:20 -0400)]
Add -priority to example script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix missing VCPUs.
Tom Clegg [Wed, 11 Mar 2020 14:09:10 +0000 (10:09 -0400)]
Fix missing VCPUs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix unchecked error.
Tom Clegg [Wed, 11 Mar 2020 14:08:17 +0000 (10:08 -0400)]
Fix unchecked error.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoConnect stdin/stdout in docker containers.
Tom Clegg [Wed, 11 Mar 2020 14:02:55 +0000 (10:02 -0400)]
Connect stdin/stdout in docker containers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoNo O_EXCL.
Tom Clegg [Wed, 11 Mar 2020 13:52:49 +0000 (09:52 -0400)]
No O_EXCL.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoError reporting.
Tom Clegg [Wed, 11 Mar 2020 05:49:28 +0000 (01:49 -0400)]
Error reporting.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUpdate example script.
Tom Clegg [Tue, 10 Mar 2020 21:11:25 +0000 (17:11 -0400)]
Update example script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix docker mounts.
Tom Clegg [Tue, 10 Mar 2020 21:11:17 +0000 (17:11 -0400)]
Fix docker mounts.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoGenerate genome file from fasta.
Tom Clegg [Tue, 10 Mar 2020 21:01:00 +0000 (17:01 -0400)]
Generate genome file from fasta.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix example collections.
Tom Clegg [Tue, 10 Mar 2020 20:15:36 +0000 (16:15 -0400)]
Fix example collections.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoInstall bedtools.
Tom Clegg [Tue, 10 Mar 2020 20:15:18 +0000 (16:15 -0400)]
Install bedtools.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoShow stderr from children.
Tom Clegg [Tue, 10 Mar 2020 20:11:55 +0000 (16:11 -0400)]
Show stderr from children.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoDon't try to pass fd 3 to docker.
Tom Clegg [Tue, 10 Mar 2020 20:08:50 +0000 (16:08 -0400)]
Don't try to pass fd 3 to docker.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agovcf2fasta -mask cont'd.
Tom Clegg [Tue, 10 Mar 2020 19:55:17 +0000 (15:55 -0400)]
vcf2fasta -mask cont'd.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoPass gvcf_regions.py via mount.
Tom Clegg [Tue, 10 Mar 2020 14:21:48 +0000 (10:21 -0400)]
Pass gvcf_regions.py via mount.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoContainer request priority flag.
Tom Clegg [Tue, 10 Mar 2020 13:33:27 +0000 (09:33 -0400)]
Container request priority flag.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd vcf2fasta -mask option.
Tom Clegg [Mon, 9 Mar 2020 19:28:31 +0000 (15:28 -0400)]
Add vcf2fasta -mask option.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd vcf2fasta command.
Tom Clegg [Mon, 9 Mar 2020 14:34:28 +0000 (10:34 -0400)]
Add vcf2fasta command.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRemove debug bit.
Tom Clegg [Sat, 7 Mar 2020 07:06:20 +0000 (02:06 -0500)]
Remove debug bit.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUse more tiling workers than NumCPU.
Tom Clegg [Thu, 5 Mar 2020 19:18:08 +0000 (14:18 -0500)]
Use more tiling workers than NumCPU.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFilter logs by container on client side of ws connection.
Tom Clegg [Thu, 5 Mar 2020 18:55:03 +0000 (13:55 -0500)]
Filter logs by container on client side of ws connection.

(arvados-ws currently ignores object_uuid filters.)

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoLess verbose logging.
Tom Clegg [Thu, 5 Mar 2020 18:52:10 +0000 (13:52 -0500)]
Less verbose logging.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUpgrade deps.
Tom Clegg [Thu, 5 Mar 2020 18:24:42 +0000 (13:24 -0500)]
Upgrade deps.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix up log/event handling.
Tom Clegg [Thu, 5 Mar 2020 15:19:14 +0000 (10:19 -0500)]
Fix up log/event handling.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoI/O pipeline, show arvados container logs.
Tom Clegg [Thu, 5 Mar 2020 03:18:26 +0000 (22:18 -0500)]
I/O pipeline, show arvados container logs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd example script.
Tom Clegg [Wed, 4 Mar 2020 20:31:43 +0000 (15:31 -0500)]
Add example script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd plot subcommand.
Tom Clegg [Wed, 4 Mar 2020 20:19:39 +0000 (15:19 -0500)]
Add plot subcommand.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoTidy log messages.
Tom Clegg [Wed, 4 Mar 2020 01:57:28 +0000 (20:57 -0500)]
Tidy log messages.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix input arg in export.
Tom Clegg [Wed, 4 Mar 2020 01:54:03 +0000 (20:54 -0500)]
Fix input arg in export.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd pca subcommand.
Tom Clegg [Tue, 3 Mar 2020 20:30:01 +0000 (15:30 -0500)]
Add pca subcommand.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRun "filter" and "export" in arvados containers.
Tom Clegg [Tue, 3 Mar 2020 16:41:23 +0000 (11:41 -0500)]
Run "filter" and "export" in arvados containers.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRun "import" in arvados container.
Tom Clegg [Mon, 2 Mar 2020 21:49:48 +0000 (16:49 -0500)]
Run "import" in arvados container.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd import -o argument.
Tom Clegg [Mon, 2 Mar 2020 14:53:12 +0000 (09:53 -0500)]
Add import -o argument.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoDon't require reference data when all inputs are fasta.
Tom Clegg [Mon, 2 Mar 2020 14:36:03 +0000 (09:36 -0500)]
Don't require reference data when all inputs are fasta.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUpdate go.sum.
Tom Clegg [Fri, 7 Feb 2020 18:28:14 +0000 (13:28 -0500)]
Update go.sum.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix typo.
Tom Clegg [Fri, 7 Feb 2020 17:17:48 +0000 (12:17 -0500)]
Fix typo.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix zeroing everything if max-variants not specified.
Tom Clegg [Thu, 6 Feb 2020 20:40:13 +0000 (15:40 -0500)]
Fix zeroing everything if max-variants not specified.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix copy-paste error in color map.
Tom Clegg [Thu, 6 Feb 2020 15:01:39 +0000 (10:01 -0500)]
Fix copy-paste error in color map.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd filter subcommand.
Tom Clegg [Wed, 5 Feb 2020 21:05:55 +0000 (16:05 -0500)]
Add filter subcommand.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoSplit gvcf2numpy command into import and export-numpy.
Tom Clegg [Tue, 4 Feb 2020 21:28:51 +0000 (16:28 -0500)]
Split gvcf2numpy command into import and export-numpy.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoColor PCA plot.
Tom Clegg [Mon, 3 Feb 2020 15:25:20 +0000 (10:25 -0500)]
Color PCA plot.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoCompact on the fly to reduce memory use.
Tom Clegg [Fri, 31 Jan 2020 18:20:00 +0000 (13:20 -0500)]
Compact on the fly to reduce memory use.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoSkip chr*_*.
Tom Clegg [Fri, 31 Jan 2020 07:39:02 +0000 (02:39 -0500)]
Skip chr*_*.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoReduce allocs while tiling.
Tom Clegg [Fri, 31 Jan 2020 07:28:23 +0000 (02:28 -0500)]
Reduce allocs while tiling.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAccount for WIP in progress report.
Tom Clegg [Thu, 30 Jan 2020 23:05:26 +0000 (18:05 -0500)]
Account for WIP in progress report.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoRemove spurious else.
Tom Clegg [Thu, 30 Jan 2020 23:05:10 +0000 (18:05 -0500)]
Remove spurious else.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUse 2bit-packed uint64 as tagmapKey.
Tom Clegg [Thu, 30 Jan 2020 22:53:23 +0000 (17:53 -0500)]
Use 2bit-packed uint64 as tagmapKey.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix file mode.
Tom Clegg [Thu, 30 Jan 2020 21:04:49 +0000 (16:04 -0500)]
Fix file mode.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd -pprof flag.
Tom Clegg [Thu, 30 Jan 2020 21:04:02 +0000 (16:04 -0500)]
Add -pprof flag.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix writing all variant numbers to npy row 0.
Tom Clegg [Thu, 30 Jan 2020 15:01:57 +0000 (10:01 -0500)]
Fix writing all variant numbers to npy row 0.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoSkip extra assignment.
Tom Clegg [Thu, 30 Jan 2020 14:55:20 +0000 (09:55 -0500)]
Skip extra assignment.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAdd example PCA script.
Tom Clegg [Thu, 30 Jan 2020 07:08:10 +0000 (02:08 -0500)]
Add example PCA script.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix shadowed var captured in goroutine.
Tom Clegg [Thu, 30 Jan 2020 07:03:22 +0000 (02:03 -0500)]
Fix shadowed var captured in goroutine.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoTest fasta2numpy.
Tom Clegg [Mon, 20 Jan 2020 15:40:09 +0000 (10:40 -0500)]
Test fasta2numpy.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoInclude ending tag in tile sequence.
Tom Clegg [Mon, 20 Jan 2020 14:54:26 +0000 (09:54 -0500)]
Include ending tag in tile sequence.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoUse blake256 sum.
Tom Clegg [Sun, 19 Jan 2020 02:54:07 +0000 (21:54 -0500)]
Use blake256 sum.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAccept fasta inputs.
Tom Clegg [Sun, 19 Jan 2020 02:53:59 +0000 (21:53 -0500)]
Accept fasta inputs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoDon't try to match overlapping tags.
Tom Clegg [Sun, 19 Jan 2020 02:53:34 +0000 (21:53 -0500)]
Don't try to match overlapping tags.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoDrop tiles with no-calls.
Tom Clegg [Fri, 17 Jan 2020 19:03:29 +0000 (14:03 -0500)]
Drop tiles with no-calls.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix deadlock from unclosed chan.
Tom Clegg [Fri, 17 Jan 2020 18:54:24 +0000 (13:54 -0500)]
Fix deadlock from unclosed chan.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoLog tile library size periodically.
Tom Clegg [Fri, 17 Jan 2020 18:20:03 +0000 (13:20 -0500)]
Log tile library size periodically.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoShow progress / ETA.
Tom Clegg [Thu, 16 Jan 2020 16:40:08 +0000 (11:40 -0500)]
Show progress / ETA.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoint32 -> uint16 in numpy output.
Tom Clegg [Thu, 16 Jan 2020 16:24:47 +0000 (11:24 -0500)]
int32 -> uint16 in numpy output.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix errs channel race.
Tom Clegg [Thu, 16 Jan 2020 15:03:33 +0000 (10:03 -0500)]
Fix errs channel race.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoWrite numpy array.
Tom Clegg [Thu, 16 Jan 2020 00:16:14 +0000 (19:16 -0500)]
Write numpy array.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agomd5 -> blake2b
Tom Clegg [Wed, 15 Jan 2020 19:37:43 +0000 (14:37 -0500)]
md5 -> blake2b

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoAccept directories as vcf source args.
Tom Clegg [Wed, 15 Jan 2020 16:45:10 +0000 (11:45 -0500)]
Accept directories as vcf source args.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agolimit concurrency
Tom Clegg [Tue, 14 Jan 2020 17:20:33 +0000 (12:20 -0500)]
limit concurrency

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoaccept csi or tbi index
Tom Clegg [Tue, 14 Jan 2020 16:40:18 +0000 (11:40 -0500)]
accept csi or tbi index

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoFix docker buffering everything.
Tom Clegg [Tue, 14 Jan 2020 16:19:31 +0000 (11:19 -0500)]
Fix docker buffering everything.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoLimit concurrency.
Tom Clegg [Tue, 14 Jan 2020 16:18:30 +0000 (11:18 -0500)]
Limit concurrency.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoOutput tile variant arrays.
Tom Clegg [Tue, 14 Jan 2020 16:16:41 +0000 (11:16 -0500)]
Output tile variant arrays.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>

4 years agoInitial tiling code.
Tom Clegg [Mon, 13 Jan 2020 17:17:33 +0000 (12:17 -0500)]
Initial tiling code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>