Issue with dumping KR matrix

[Holden Leung]

Hello,

I used the code below to dump observed KR matrix from .hic files, however, some .hic files don't have KR-normalized matrixes with a certain resolution. For example, one .hic file has KR normalization matrixes for chr11_25kb, chr11_50kb, chr11_100kb, chr12_25kb, chr12_100kb, but don't have one for chr12_50kb. Although it worked perfectly with VC, we need KR-normalized matrix for our pipeline.

java -jar juicer_tools.jar dump observed KR sample.hic 12 12 BP 50000

The only message I got after running this code is as below:

WARN [2021-01-05T19:16:10,229]  [Globals.java:138] [main]  Development mode is enabled

No error message or any message. This is the only message that popped out.

I searched in this group and found this post (https://groups.google.com/g/3d-genomics/c/vsUgDt21CF4/m/-lIByy3HBgAJ). It says it may because the KR normalization didn't converge, so no output was generated. And it suggests us to increase the iteration time for function computeKR in juicebox.tools.utils.original.NormalizationCalculations.java., so that the matrix can converge, following by running addNorm function to re-create KR vectors for the .hic file.

However, when I ran the code below, I lost all normalized matrices under different normalization (KR, VC, VC_SQRT), different resolution, and different chromosomes (I tested 1-22 and chr1-chr22). Although the size of the .hic file is the same after running addNorm.

java -jar juicer_tools.jar addNorm sample.hic -j 10

java -jar juicer_tools.jar addNorm sample.hic -j 10 -w 500

This is the version of Juicer I'm using right now: Juicer Tools Version 1.22.01

So there are two main issues happening: 

1. Some KR-normalized matrices are missing at certain resolution for certain chromosomes.

2. After running addNorm, all normalization vectors are missing in the .hic file.

I will greatly appreciate it if you guys can give us some suggestions to fix those issues!

Thank you ahead!

Holden

[Holden Leung]

Just in case, below is the output from running addNorm:

WARN [2021-01-05T14:23:44,185]  [Globals.java:138] [main]  Development mode is enabled

Using 10 CPU thread(s)

Calculating norms for zoom BP_2500000

Calculating norms for zoom BP_1000000

Calculating norms for zoom BP_500000

Calculating norms for zoom BP_250000

Calculating norms for zoom BP_100000

Calculating norms for zoom BP_50000

Calculating norms for zoom BP_25000

Calculating norms for zoom BP_10000

Calculating norms for zoom BP_5000

Calculating norms for zoom BP_1000

Calculating norms for zoom FRAG_500

Calculating norms for zoom FRAG_200

Calculating norms for zoom FRAG_100

Calculating norms for zoom FRAG_50

Calculating norms for zoom FRAG_20

Calculating norms for zoom FRAG_5

Calculating norms for zoom FRAG_2

Calculating norms for zoom FRAG_1

Finished writing norms

[Muhammad Saad Shamim]

Hi Holden,

AddNorm does indeed overwrite and recalculate norms when it's not being used for appending a custom norm. You can create a github issue for a request to change this. 

I'm not sure what exactly you're trying to do regarding KR though. Did you change the number of iterations in the java code? (the w flag has nothing to do with iterations). You may want to use the latest jar, which builds the SCALE normalization (and is on par/better than KR, and more likely to converge).

Best,

- Muhammad Saad Shamim

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/38583de5-0ea4-4c66-8131-dbfcbb53492dn%40googlegroups.com.

[Holden Leung]

Hello Muhammad,

Thanks for the quick response! I will create a GitHub issue for the addNorm.

For KR, I haven't change the java code yet. Because to re-generate the KR normalization vectors, I need to change the number of iterations, and then run addNorm to the existed .hic files. So I tested the addNorm function first. However, the addNorm function is not working as expected. Thanks for the reminder, I know the -w flag is not related to the iteration, I just want to show that with or without the -w flag, the addNorm function is not working either way. 

Do you mind elaborate a little bit more about the SCALE normalization? What do you mean by it's better than KR?

Thank you again! I appreciate your help!

Holden

[Shawn Bai]

Hi, I am generating a hic file with 1.7B reads to 50bp resolution, but the last step alwasy failed: How can it be resolved? Thanks!

Calculating norms for zoom BP_200Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
        at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
        at org.broad.igv.tdf.BufferedByteWriter.putLong(BufferedByteWriter.java:134)
        at org.broad.igv.tdf.BufferedByteWriter.putDouble(BufferedByteWriter.java:115)
        at juicebox.tools.utils.original.NormalizationVectorUpdater.writeNormalizationVector(NormalizationVectorUpdater.java:859)
        at juicebox.tools.utils.original.NormalizationVectorUpdater.updateHicFile(NormalizationVectorUpdater.java:174)
        at juicebox.tools.clt.old.AddNorm.run(AddNorm.java:84)
        at juicebox.tools.HiCTools.main(HiCTools.java:96)

[Moshe Olshansky]

Hello,

You should use juicer_tools pre without normalization and then add norm manually with addNorm avoiding KR at high (say above 1k) resolution. You can add SCALE norm at 50 bp resolution. It is similar to KR.