Linkage Disequilibrium never completing

[Mary LaPorte]

Hello! I am having an issue where Linkage Disequilibrium is never running past 10% and I'm not sure how to move forward.

The hapmap file is 53.84 GB and contains 92,790,713 sites after filtering. It is running on 94 cores with 5GB/core (470GB). I let it run for about 24 hours before terminating the run (I can schedule it for 96 hours max). Is there a more efficient way to run LD in TASSEL? For context, GWAS using this hapmap worked like a charm. 

Some other options I have considered: 

• Selecting every 100th marker and running LD on this (not sure if it makes sense/is rigorous, not sure how to do it yet)
• running it on a 32GB core (would wait a while in the queue, and not sure it would help)

Could you advise on what to do in this situation, and what may be causing the "hanging" at 10%? Thank you very much for your help! 

TASSEL output:

$ run_pipeline.pl -Xmx460g -debug -h /xdisk/group/user/temp.hmp.txt -ld -export /xdisk/group/user/ld_out.txt

/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/trove-3.0.3.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jackson-annotations-2.13.2.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/phg.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/error_prone_annotations-2.19.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/biojava-genome-6.0.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/kotlin-stdlib-1.6.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/ahocorasick-0.2.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jhdf5-14.12.5.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/protobuf-java-util-3.23.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/gs-core-1.3.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/slf4j-simple-1.7.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/json-simple-1.1.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/sshj-0.32.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/biojava-alignment-6.0.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jackson-core-2.13.2.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/commons-codec-1.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jfreesvg-3.2.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/forester-1.039.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/kotlin-stdlib-jdk8-1.6.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jackson-module-kotlin-2.13.2.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/postgresql-42.6.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/protobuf-kotlin-3.23.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jfreechart-1.0.19.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/javax.json-1.0.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/commons-math3-3.4.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/colt-1.2.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/log4j-core-2.21.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/guava-22.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/protobuf-java-3.23.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jackson-databind-2.13.2.2.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/slf4j-api-1.7.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/gs-ui-1.3.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/jcommon-1.0.23.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/ini4j-0.5.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/kotlinx-coroutines-core-jvm-1.6.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/junit-4.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/snappy-java-1.1.8.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/sqlite-jdbc-3.39.2.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/log4j-api-2.21.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/ejml-core-0.41.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/ejml-ddense-0.41.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/scala-library-2.10.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/kotlin-reflect-1.6.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/kotlin-stdlib-jdk7-1.6.10.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/fastutil-8.2.2.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/htsjdk-2.24.1.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/itextpdf-5.1.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/mail-1.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/biojava-core-6.0.4.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/commons-io-2.11.0.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/lib/biojava-phylo-4.2.12.jar:/opt/ohpc/pub/apps/tassel/5.2.96/bin/sTASSEL.jar
Memory Settings: -Xms512m -Xmx460g
Tassel Pipeline Arguments: -debug -h /xdisk/group/user/temp.hmp.txt -ld -export /xdisk/group/user/ld_out.txt
[main] INFO net.maizegenetics.tassel.TasselLogging - Tassel Version: 5.2.96  Date: March 31, 2025
[main] INFO net.maizegenetics.tassel.TasselLogging - Max Available Memory Reported by JVM: 418702 MB
[main] INFO net.maizegenetics.tassel.TasselLogging - Java Version: 1.8.0_452
[main] INFO net.maizegenetics.tassel.TasselLogging - OS: Linux
[main] INFO net.maizegenetics.tassel.TasselLogging - Number of Processors: 94
[main] INFO net.maizegenetics.tassel.TasselLogging - Tassel Citation: Bradbury PJ, Zhang Z, Kroon DE, Casstevens TM, Ramdoss Y, Buckler ES. (2007) TASSEL: Software for association mapping of complex traits in diverse samples. Bioinformatics 23:2633-2635.
[main] INFO net.maizegenetics.tassel.TasselLogging -
[main] INFO net.maizegenetics.tassel.TasselLogging - Tassel Using Library: Practical Haplotype Graph (PHG): Version: 1.10 Date: August 20, 2024
[main] INFO net.maizegenetics.tassel.TasselLogging - PHG Citation: Bradbury PJ, Casstevens T, Jensen SE, Johnson LC, Miller ZR, Monier B, Romay MC, Song B, Buckler ES. The Practical Haplotype Graph, a platform for storing and using pangenomes for imputation. Bioinformatics. 2022 Aug 2;38(15):3698-3702. doi: 10.1093/bioinformatics/btac410. PMID: 35748708; PMCID: PMC9344836.
[main] INFO net.maizegenetics.pipeline.TasselPipeline - Tassel Pipeline Arguments: [-fork1, -h, /xdisk/group/user/temp.hmp.txt, -ld, -export, /xdisk/group/user/ld_out.txt, -runfork1]
net.maizegenetics.analysis.data.FileLoadPlugin
   net.maizegenetics.analysis.popgen.LinkageDisequilibriumPlugin
      net.maizegenetics.analysis.data.ExportMultiplePlugin
[pool-2-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Starting net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:51:31
[pool-2-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin -
FileLoadPlugin Parameters
format: Hapmap
sortPositions: false
keepDepth: false

[pool-2-thread-1] INFO net.maizegenetics.analysis.data.FileLoadPlugin - Start Loading File: /xdisk/group/user/temp.hmp.txt time: Jul 21, 2025 10:51:31
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:26: progress: 0%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:28: progress: 10%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:30: progress: 20%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:32: progress: 30%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:34: progress: 40%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:36: progress: 50%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:38: progress: 60%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:40: progress: 70%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:42: progress: 80%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:54:44: progress: 90%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 10:55:1: progress: 100%
[pool-2-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - net.maizegenetics.analysis.data.FileLoadPlugin  Citation: Bradbury PJ, Zhang Z, Kroon DE, Casstevens TM, Ramdoss Y, Buckler ES. (2007) TASSEL: Software for association mapping of complex traits in diverse samples. Bioinformatics 23:2633-2635.
[pool-2-thread-1] INFO net.maizegenetics.analysis.data.FileLoadPlugin - Finished Loading File: /xdisk/group/user/temp.hmp.txt time: Jul 21, 2025 11:00:28
Genotype Table Name: temp
Number of Taxa: 256
Number of Sites: 92790713
Sites x Taxa: 23754422528
Chromosomes...
NC_007578.1: start site: 0 (1704) last site: 151 (148393) total: 152
NC_042756.1: start site: 152 (35830) last site: 1649 (362549) total: 1498
NC_056623.2: start site: 1650 (2858) last site: 10307871 (252569385) total: 10306222
NC_056624.2: start site: 10307872 (8416) last site: 20067427 (236377555) total: 9759556
NC_056625.2: start site: 20067428 (11353) last site: 31351525 (324655322) total: 11284098
NC_056626.2: start site: 31351526 (21723) last site: 46667277 (410279543) total: 15315752
NC_056627.2: start site: 46667278 (5151) last site: 59783373 (371825205) total: 13116096
NC_056628.2: start site: 59783374 (10423) last site: 66553094 (206649905) total: 6769721
NC_056629.2: start site: 66553095 (10117) last site: 73196181 (209884597) total: 6643087
NC_056630.2: start site: 73196182 (7970) last site: 85215915 (343479930) total: 12019734
NC_056631.2: start site: 85215916 (13513) last site: 92786607 (229402556) total: 7570692
NW_026440193.1: start site: 92786608 (15921) last site: 92786764 (635921) total: 157
NW_026440194.1: start site: 92786765 (3489) last site: 92786959 (389962) total: 195
NW_026440195.1: start site: 92786960 (19840) last site: 92787014 (359666) total: 55
NW_026440196.1: start site: 92787015 (19839) last site: 92787094 (108454) total: 80
NW_026440197.1: start site: 92787095 (320) last site: 92787443 (146153) total: 349
NW_026440198.1: start site: 92787444 (60655) last site: 92787458 (96137) total: 15
NW_026440199.1: start site: 92787459 (643) last site: 92788589 (140840) total: 1131
NW_026440200.1: start site: 92788590 (46656) last site: 92788603 (133629) total: 14
NW_026440201.1: start site: 92788604 (51735) last site: 92788605 (84164) total: 2
NW_026440202.1: start site: 92788606 (7923) last site: 92788608 (8093) total: 3
NW_026440207.1: start site: 92788609 (1220) last site: 92788706 (67581) total: 98
NW_026440208.1: start site: 92788707 (24477) last site: 92788710 (45007) total: 4
NW_026440209.1: start site: 92788711 (639) last site: 92788717 (40963) total: 7
NW_026440211.1: start site: 92788718 (9580) last site: 92788718 (9580) total: 1
NW_026440212.1: start site: 92788719 (20072) last site: 92788736 (26160) total: 18
NW_026440213.1: start site: 92788737 (37885) last site: 92788737 (37885) total: 1
NW_026440215.1: start site: 92788738 (35112) last site: 92788739 (35113) total: 2
NW_026440222.1: start site: 92788740 (1543) last site: 92788740 (1543) total: 1
NW_026440228.1: start site: 92788741 (27550) last site: 92788756 (27710) total: 16
NW_026440229.1: start site: 92788757 (3113) last site: 92788995 (29282) total: 239
NW_026440230.1: start site: 92788996 (489) last site: 92789374 (31447) total: 379
NW_026440233.1: start site: 92789375 (7621) last site: 92789383 (7942) total: 9
NW_026440237.1: start site: 92789384 (4029) last site: 92789393 (10637) total: 10
NW_026440238.1: start site: 92789394 (2242) last site: 92789400 (2420) total: 7
NW_026440239.1: start site: 92789401 (25868) last site: 92789401 (25868) total: 1
NW_026440242.1: start site: 92789402 (255) last site: 92790617 (25210) total: 1216
NW_026440244.1: start site: 92790618 (12658) last site: 92790638 (23813) total: 21
NW_026440250.1: start site: 92790639 (9138) last site: 92790658 (9416) total: 20
NW_026440253.1: start site: 92790659 (7312) last site: 92790659 (7312) total: 1
NW_026440255.1: start site: 92790660 (10256) last site: 92790670 (14197) total: 11
NW_026440260.1: start site: 92790671 (4729) last site: 92790690 (5212) total: 20
NW_026440263.1: start site: 92790691 (4114) last site: 92790712 (4230) total: 22

[pool-2-thread-1] INFO net.maizegenetics.plugindef.AbstractPlugin - Finished net.maizegenetics.analysis.data.FileLoadPlugin: time: Jul 21, 2025 11:00:28
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.popgen.LinkageDisequilibriumPlugin: time: Jul 21, 2025 11:00:29: progress: 0%
[pool-2-thread-1] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.popgen.LinkageDisequilibriumPlugin: time: Jul 21, 2025 12:40:11: progress: 10%

[cut off by user the next day around 12:00:00]

Other variations I have tried: 

run_pipeline.pl -Xmx460g -debug -h /xdisk/group/user/temp.hmp.txt -ld -ldRapidAnalysis true -ldWinSize 100000 -export /xdisk/group/user/ld_out.txt

run_pipeline.pl -Xmx460g -debug -h /xdisk/group/user/temp.hmp.txt -ld -ldRapidAnalysis true -ldWinSize 50 -export /xdisk/group/user/ld_out.txt

run_pipeline.pl -Xmx400g -h ./xdisk/group/user/temp.hmp.txt -ld -ldType SiteByAll -export /xdisk/group/user/ld_out.txt

[Muhammad Atif Wahid]

It depends on type of genetic markers...if it's SSRs I can tell you some methods of using tassel 2.1

--
You received this message because you are subscribed to the Google Groups "TASSEL - Trait Analysis by Association, Evolution and Linkage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tassel+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/tassel/b97386a9-b92c-43d5-9483-4ce0f387636fn%40googlegroups.com.

[Mary LaPorte]

Yes, SSRs! Thank you very much