Juicer2 mega.sh - Error fetching block null and java.util.zip.DataFormatException: incorrect header check

[Vanessa Roy]

Hi all,

I am having some trouble with the mega.sh script after running juicer2 with the 'early exit' flag, so I can run mega.sh to create a mega map.

The issue I get is this recurring error that seems to be a javascript error when I try running juicer2 mega.sh, but I don't understand how to fix it. When I run mega.sh to merge replicates (SLURM), I get the following error message (from the hic1-xyz.err file in the debug folder) that I have shortened here (the java.util.zip.DataFormatException: incorrect header check error repeats over and over, creating a file that is 1.5T large):

head -n 1000 hic1-4328.err

real 5m36.796s
user 5m28.833s
sys 0m6.736s
Picked up _JAVA_OPTIONS: -Xms150000m -Xmx150000m
Unable to find All-All  or  All-All
Unable to find 1-GL456219.1  or  GL456219.1-1
Unable to find 1-GL456359.1  or  GL456359.1-1
Unable to find 1-GL456366.1  or  GL456366.1-1
Unable to find 1-JH584298.1  or  JH584298.1-1
Unable to find 1-JH584300.1  or  JH584300.1-1
Unable to find 1-JH584302.1  or  JH584302.1-1
Unable to find 1-JH584303.1  or  JH584303.1-1
Unable to find 2-GL456219.1  or  GL456219.1-2
Unable to find 2-GL456359.1  or  GL456359.1-2
Unable to find 2-GL456366.1  or  GL456366.1-2
Unable to find 2-GL456379.1  or  GL456379.1-2
Unable to find 2-JH584295.1  or  JH584295.1-2
Unable to find 2-JH584298.1  or  JH584298.1-2
Unable to find 2-JH584300.1  or  JH584300.1-2
Unable to find 2-JH584301.1  or  JH584301.1-2
Unable to find 2-JH584302.1  or  JH584302.1-2
Unable to find 2-JH584303.1  or  JH584303.1-2
Unable to find 3-GL456219.1  or  GL456219.1-3
Unable to find 3-GL456359.1  or  GL456359.1-3

.

.

.

real 1252m23.048s
user 322m11.410s
sys 312m2.015s
Picked up _JAVA_OPTIONS: -Xms150000m -Xmx150000m
Error fetching block null
Error fetching block null
Error fetching block null
Error fetching block null
Error fetching block null
Error fetching block null
Error fetching block null
Error fetching block null
Error fetching block null
java.util.zip.DataFormatException: incorrect header check
at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base/java.util.zip.Inflater.inflate(Inflater.java:378)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:87)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:56)
at juicebox.data.DatasetReaderV2.decompress(DatasetReaderV2.java:1032)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:980)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:922)
at juicebox.data.iterator.ContactRecordIterator.hasNext(ContactRecordIterator.java:85)
at juicebox.data.iterator.CoupledIteratorAndOffset.hasNext(CoupledIteratorAndOffset.java:44)
at juicebox.data.iterator.BigContactRecordList.populateListOfListsFromSingleIterator(BigContactRecordList.java:41)
at juicebox.data.iterator.ListOfListGenerator.lambda$populateListOfLists$0(ListOfListGenerator.java:91)
at juicebox.tools.dev.ParallelizedJuicerTools$1.run(ParallelizedJuicerTools.java:42)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
java.util.zip.DataFormatException: incorrect header check
at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base/java.util.zip.Inflater.inflate(Inflater.java:378)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:87)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:56)
at juicebox.data.DatasetReaderV2.decompress(DatasetReaderV2.java:1032)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:980)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:922)
at juicebox.data.iterator.ContactRecordIterator.hasNext(ContactRecordIterator.java:85)
at juicebox.data.iterator.CoupledIteratorAndOffset.hasNext(CoupledIteratorAndOffset.java:44)
at juicebox.data.iterator.BigContactRecordList.populateListOfListsFromSingleIterator(BigContactRecordList.java:41)
at juicebox.data.iterator.ListOfListGenerator.lambda$populateListOfLists$0(ListOfListGenerator.java:91)
at juicebox.tools.dev.ParallelizedJuicerTools$1.run(ParallelizedJuicerTools.java:42)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
java.util.zip.DataFormatException: incorrect header check
at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base/java.util.zip.Inflater.inflate(Inflater.java:378)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:87)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:56)
at juicebox.data.DatasetReaderV2.decompress(DatasetReaderV2.java:1032)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:980)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:922)
at juicebox.data.iterator.ContactRecordIterator.hasNext(ContactRecordIterator.java:85)
at juicebox.data.iterator.CoupledIteratorAndOffset.hasNext(CoupledIteratorAndOffset.java:44)
at juicebox.data.iterator.BigContactRecordList.populateListOfListsFromSingleIterator(BigContactRecordList.java:41)
at juicebox.data.iterator.ListOfListGenerator.lambda$populateListOfLists$0(ListOfListGenerator.java:91)
at juicebox.tools.dev.ParallelizedJuicerTools$1.run(ParallelizedJuicerTools.java:42)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

.

.

.

---

I found these below posts on this error message, but I'm still not sure how to fix this.

https://stackoverflow.com/questions/11399350/gzinflate-in-java/11401785#11401785

https://github.com/rest-assured/rest-assured/issues/482

The inter.hic and inter_30.hic maps are still created, however, I cannot run Arrowhead or HiCCUPs on the files (getting 0 domains and either the same java.util.zip.DataFormatException: incorrect header check error or Data not available for XYZ at ABC resolution).

I am using juicer2 (latest version) and juicer_tools_1.22.01.jar.

I greatly appreciate your help on this!

Best wishes, and many thanks!

Vanessa