Merging un-normalized matrices vs. merging data before alignment

[Debbie Chasman]

Hello,

I'm wondering if these two orders of operations should give equivalent results. 

Say we have two different technical replicates of the same experiment and want to pool them to increase the sequencing depth.

Option 1: Use HiCPro to align and quantify each replicate separately in as two different "samples". Then, add up the raw/unnormalized matrices from the two samples. Finally, apply ice normalization to the single combined unnormalized matrix.

Option 2: Align and quantify both replicates together by putting all fastq files within the same sample data directory. Use HiC-Pro as usual to quantify and then ice normalize one matrix.

My intuition is that the two final matrices should be equivalent, because alignment and presumably QC is done independently per read pair. But perhaps I'm missing something. I'd really appreciate your insight.

Many thanks,

Best,

Deborah

[nservant]

Hi Debbie,

Yes, the two options should give exactly the same results.
On my side, I usually used Option 1 ... it allows to check the replicates quality independantly and to further merge them to reach higher resolution.
Best
Nicolas

[Debbie Chasman]

Okay, great. Thank you very much for your response! 

[Ashley S Doane]

Hi Nicolas,

can you or someone who has done this advise on pooling replicate samples?  Ideally I could cat the validPairs file for replicate 1 and replicate 2, and sort.  Or something similar.  Can you advise?

thanks,

Ashley

[nservant]

Hi Ashley

yes, this is exactly the best way to go.

Note that in practice, it should be exactly the same than simply adding the two raw contact maps.

Best

N

[Moshe Olshansky]

Hi Nicolas,

Won't more duplicated pairs of reads be potentially removed when using Option 2?

[nservant]

Option 2: Align and quantify both replicates together by putting all fastq files within the same sample data directory. Use HiC-Pro as usual to quantify and then ice normalize one matrix.

Yes absolutely !!

That's why I recommand to put one sample per folder first. So that each sample is run independantly (including duplicates removal).

Then create another input folder with link to the validPairs (in the same folder now). Set rm_dup=FALSE, et re-run HIC-Pro in stepwise mode, just to generate the maps.

Does it make sense for you ?

Nicolas

[Moshe Olshansky]

Yes, definitely.

Thank you.

[Daniel Bsteh]

Okay but I am confused how I can proceed after I merged the replicates as you described above from the .validPairs files... I want to visualize the data in juicebox or higlass? The input files seems to require the allvalidpairs files? Just wanna make sure the duplicates are handled correctly... Is the hicpro2juicebox utility actually removing the duplicates?

Thanks!

[Moshe Olshansky]

hicpro2juicebox does not remove duplicates and it is good. Replicates have been removed from each sample during the creation of allValidPairs. So now you can sort-merge all individual validPairs into one validPairs file and then run hicpro2juicebox on it.

[Daniel Bsteh]

Okay thanks! So to put this straight the .validPairs contain duplicates but the .allvalidPairs don't? So I can't set the configure file to rm_dup=0 with .validPairs but only with the .allValidPairs files? And you merge and sort the allvalidpairs files outside of HiCPro with like cat?

[Moshe Olshansky]

Normally duplicated reads for each sample will be removed at some stage before allValidPairs are produced and so you should not remove them again. As to merging the files from several samples, look at Unix sort command with -m flag.