addNorm Exception msg at higher resolution

[Saumya]

Dear all,

I am running addNorm on a hic file generated using mega.sh script. I am able to add normalization at lower resolution but as soon as it reaches 100KB it starts throwing following exception. I am not sure why I am getting this message.  If i run addNorm on individual hic files that were used to generate the combine hic files then I don't get any message and script manage to finish the run.

java -Xmx50000m   -jar ./juicer_tools.2.20.00.jar addNorm -k VC -r 100000 -j 20 inter_30.hic

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARN [2023-03-05T10:24:57,574]  [Globals.java:138] [main]  Development mode is enabled
Using 20 CPU thread(s) for primary task

Calculating norms for zoom BP_2500000
Calculating norms for zoom BP_1000000
Calculating norms for zoom BP_500000
Calculating norms for zoom BP_250000
Calculating norms for zoom BP_100000java.util.zip.DataFormatException: incorrect header check
at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base/java.util.zip.Inflater.inflate(Inflater.java:385)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:87)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:56)
at juicebox.data.DatasetReaderV2.decompress(DatasetReaderV2.java:1032)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:980)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:922)
at juicebox.data.iterator.ContactRecordIterator.hasNext(ContactRecordIterator.java:85)
at juicebox.data.iterator.BigContactRecordList.populateListOfListsFromSingleIterator(BigContactRecordList.java:41)
at juicebox.data.iterator.ListOfListGenerator.populateListOfLists(ListOfListGenerator.java:101)
at juicebox.data.iterator.ListOfListGenerator.tryToCreateIteratorInRAM(ListOfListGenerator.java:64)
at juicebox.data.iterator.ListOfListGenerator.createFromZD(ListOfListGenerator.java:42)
at juicebox.data.MatrixZoomData.getIteratorContainer(MatrixZoomData.java:1242)
at juicebox.tools.utils.norm.NormalizationVectorUpdater.updateHicFile(NormalizationVectorUpdater.java:184)
at juicebox.tools.clt.old.AddNorm.launch(AddNorm.java:83)
at juicebox.tools.clt.old.AddNorm.run(AddNorm.java:137)
at juicebox.tools.HiCTools.main(HiCTools.java:97)

Any advice from the forum will be very helpful,

Many thanks in advance,

Saumya

[Saumya]

Dear all,

Just wanted to share the solution for the above problem, in case any one run into same trouble:

1) I sorted the individual merged_30.txt file (-k2,2d -k6,6d -k3,3n -k7,7n) as described in the previous post

https://groups.google.com/g/3d-genomics/c/axgvyaDbcLg/m/HpZ1RHyIDAAJ

2) Then merged the sorted files

sort --parallel=40 -T ${tmpdir} -m -k2,2d -k6,6d -k3,3n -k7,7n  ${merged_names30} > ${outputdir}/merged30.txt

3) Run pre on individual merged file

java -Xmx10000m   -jar ./juicer_tools.2.20.00.jar  pre   -g ./inter_30_hists.m -q 30 -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000,2000,1000 -f ./juicer-main/restriction_sites/hg38_Arima.txt  -k VC,VC_SQRT,KR --threads 10   ./merged30.txt  test.hic  hg38.chrom.sizes

Now, I am able to generate the merged .hic files without error.

Still not able to solve the issue with mega.sh

Best,

Saumya