Failing read pairing step

[Shamith Samarajiwa]

Dear Nicholas,

 I've run the test data set on a new installation of HiC-Pro and the pipeline worked. HiC-Pro fails at the read pairing step when using our data. I've included the HIC-Pro log file and mergeBAM.py log file. The fastq headers look ok. Any help in troubleshooting this is much appreciated.

kind regards,

 Shamith

Run HiC-Pro 2.7.8

--------------------------------------------

Thu 29 Sep 20:05:01 BST 2016

Bowtie2 alignment step1 ...

/home/SS861/src/HiC-Pro_2.7.8/scripts/bowtie_wrap.sh -c /home/SS861/src/HiC-Pro_2.7.8/config-hicpro.txt -u >> hicpro.log

--------------------------------------------

Fri 30 Sep 03:18:59 BST 2016

Bowtie2 alignment step2 ...

/home/SS861/src/HiC-Pro_2.7.8/scripts/bowtie_wrap.sh -c /home/SS861/src/HiC-Pro_2.7.8/config-hicpro.txt -l >> hicpro.log

--------------------------------------------

Fri 30 Sep 08:02:34 BST 2016

Combine both alignment ...

/home/SS861/src/HiC-Pro_2.7.8/scripts/bowtie_combine.sh -c /home/SS861/src/HiC-Pro_2.7.8/config-hicpro.txt >> hicpro.log

/home/SS861/src/HiC-Pro_2.7.8/scripts/hic.inc.sh: line 91: 22946 Killed                  /home/SS861/src/bin/samtools-1.1/samtools sort -@ 54 -n bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_R2_hg19.bwt2merged.bam -T tmp/SLX-10691.A006.H7YGCBBXX.s_1_R2_hg19 -o bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_R2_hg19.bwt2merged.sorted.bam

[bam_sort_core] merging from 162 files...

[bam_sort_core] merging from 108 files...

/home/SS861/src/HiC-Pro_2.7.8/scripts/hic.inc.sh: line 91: 23929 Killed                  /home/SS861/src/bin/samtools-1.1/samtools sort -@ 54 -n bowtie_results/bwt2/A12/SLX-10691.A012.H7YGCBBXX.s_1.r_R1_hg19.bwt2merged.bam -T tmp/SLX-10691.A012.H7YGCBBXX.s_1.r_R1_hg19 -o bowtie_results/bwt2/A12/SLX-10691.A012.H7YGCBBXX.s_1.r_R1_hg19.bwt2merged.sorted.bam

[bam_sort_core] merging from 108 files...

--------------------------------------------

Fri 30 Sep 09:23:56 BST 2016

Bowtie2 mapping statistics for R1 and R2 tags ...

/home/SS861/src/HiC-Pro_2.7.8/scripts/mapping_stat.sh -c /home/SS861/src/HiC-Pro_2.7.8/config-hicpro.txt >> hicpro.log

--------------------------------------------

Fri 30 Sep 10:12:44 BST 2016

Pairing of R1 and R2 tags ...

/home/SS861/src/HiC-Pro_2.7.8/scripts/bowtie_pairing.sh -c /home/SS861/src/HiC-Pro_2.7.8/config-hicpro.txt >> hicpro.log

/home/SS861/src/HiC-Pro_2.7.8/bin/../scripts//Makefile:136: recipe for target 'bowtie_pairing' failed

make: *** [bowtie_pairing] Error 1

## mergeBAM.py

## forward= bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_R1_hg19.bwt2merged.bam

## reverse= bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_R2_hg19.bwt2merged.bam

## output= bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_hg19.bwt2pairs.bam

## min mapq= 0

## report_single= False

## report_multi= False

## verbose= True

## Merging forward and reverse tags ...

Forward and reverse reads not paired. Check that BAM files have the same read names and are sorted.

### fastq R1:

@K00252:29:H7YGCBBXX:1:1101:1316:1560 1:N:0:GCCAAT

NGAAGTTAAAAGGCATAGTGCAGGTGAGCATGACTAATTTTTGCTCAGTAAGCCAAGCTAGCGTTATAGCCATGTACCTCCACCCTTTGACGGAGTGGGGAAAGGGAAAAAGGAGGGAAGTAAGGTCCCGGCACTCATGCTTCAAGGCCA

+

#AAAFF<F-FJAFFJJJJJAJJFJJAF--<-7-FJJ-FJJJJJJJ----<J<F-7F7FA-<A-<<JJ-<FFFJ-FJAJJJJFJJJFJF7AJA7AAF-A777<JAFFJFFJ-<<7<FAAF-FAJ7F-FAJJ7JFJ7FFJ-7-J7-77<F7-

@K00252:29:H7YGCBBXX:1:1101:1580:1560 1:N:0:GCCAAT

NTAATTGTGTTGTTTAGTTGTTAAAAAACTAAAATAGGTGTGCCTATATCTCACCAGTACAAGTATCTGGATTTTGTAAACTATTGAATAAAAATAATATTTCAATTAAAATAAATTATAATTTATTTTCAAAATCAATTAATGATGATT

+

#AAFFJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJJJAFJJJJJJJJJJJJJJJJJJJJJJF7JFJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJFJJJJJJJJJJJJJJJJJFF7FJJ

@K00252:29:H7YGCBBXX:1:1101:1742:1560 1:N:0:GCCAAT

ATTAGAAAAGGGCACCGTTCTTGGCTGACACAGCCCCTCAGACTGGCATGTGAGTGGAAAGGGGGCTGGAATGCAGCCAGCTCCTAGCAGCCCAGGGACCAGGCACTCAGGTGGAAGTCCTACCTCTAACCTCTCCCTACCTTACCTGGG

####fastq R2

@K00252:29:H7YGCBBXX:1:1101:1316:1560 2:N:0:GCCAAT

CCACCTCTGAGGTTTGCAGCGTGACAGCCACCCTCCCCAGACAGATCTCCACAATGCAGTTCTTGGCCTTGAAGCATGAGTGCCGGGACCTTAATTCCCTCCTTTTTCCCTTTCCCCACTCAGTCAAAGGGCGGAGCTCAAGCGCTGGAA

+

-AA----77<F-7FJFJJ<FFJJJJJ<A-A-AF-FA-7A<A--7F-7<--<7FJ<-<--<AAFJ-J---77FJ-<AAFJ<A-7FA-A-FJJJJJ--7-FA-AJA7--7-7AJAJAFJ7-)F-A-FA<F-AA))-)))))----7)7))-<

@K00252:29:H7YGCBBXX:1:1101:1580:1560 2:N:0:GCCAAT

ACTTATGCATTTACCTGAATCTAGAAATCAGAAAAAGGAGTATGCTAGAGTTTTTGTAAGCTAGCTTTACAGTTCATTTAAATAACCTGGTTTCAGTTTGTTAATCATCATTAATTGATTTTGAAAATAAATTATAAGTTATTTTAATTG

+

AA<FFJJJJJJJJJJJJJJJJJJJJJJJJJJJFJJJJJJJFJJFJ7FJFJFFF<<-AJFAJJJ7AAFFJJFJJJFFJJJJJJJJJJF-<JFFA7<7JFJ7FJJFJFJ<FFJAAJJJJFFJJJJFFJJJAFJJ-AAAA-FA<<F-<<--<-

@K00252:29:H7YGCBBXX:1:1101:1742:1560 2:N:0:GCCAAT

AGGCCTCTCACTGTGGGGTGGCATGCTTAGTCAAGCTAGCCTCCACTTCCAATGTCTAAGAATTCTTTGTTCAATTGATGAGTGAACAAACATCAATTGGATATATCAAGTTATGCATTTCTTTCTTTCAGTCTGTCCCTGTACCAGACC

[nservant]

Hi Shamith,

The pairing step expects that the reads name in R1 and R2 file were exactly the same.

R1 :

@K00252:29:H7YGCBBXX:1:1101:1316:1560 1:N:0:GCCAAT

R2 :

@K00252:29:H7YGCBBXX:1:1101:1316:1560 2:N:0:GCCAAT

The read names are not exaclty the same (1:N, 2:N) which explain the error.
You will need to remove this part from the fastq files and rerun HiC-Pro.
Actually, I do not really now if there is a convention in naming the R1/R2 read names.
I do not know if this type of name is common but you are not the first one who reports this issue ...
Let me know if it works
Best
Nicolas

[Sameet Mehta]

Hi Nicolas,

This is a fairly common convention.  Would it be possible to automatically consider only the first part of the header.  If I have to do this on say 2.5 Billion reads, it will take a lot of time.

Sameet

[Sameet Mehta]

A follow-up question here is, what exactly is the merging doing?  If it looked at the bam files, then it should not have mattered.  The bowtie2 output stops at the first space.  This error should not have occured.

Sameet

[nservant]

Hi Shamith,

I double checked this question of read names.
And actually, based on my test it seems that even if your fastq files have the field "1:N:0:GCCAAT" or "2:N:0:GCCAAT", your BAM files do not !
So the read names of your two BAM files should be the name.
Could you please check that by looking at the first lines of the files ;
bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_R1_hg19.bwt2merged.bam
bowtie_results/bwt2/A06/SLX-10691.A006.H7YGCBBXX.s_1_R2_hg19.bwt2merged.bam

Another hypothesis would be that the sort of these files crashed (for instance for memory issue). In this case, the BAM files to merge, will not be sorted in the same order, and it will launch an error ...
Thank you for your feedbacks

Best
Nicolas


On Friday, 30 September 2016 20:54:29 UTC+2, Shamith Samarajiwa wrote:

[Victoria]

Hi Nicolas,

I was hoping that you could help me with this. I am trying to run HiC-Pro with my own CHiC data, and I get the bowtie_pairing error message as well. 

## mergeBAM.py

## forward= bowtie_results/bwt2/Sample_CHiC_AdipoPE_Brd4_Cre_D2/CHiC_AdipoPE_Brd4_Cre_D2_S3_L002_R1_mm9.bwt2merged.bam

## reverse= bowtie_results/bwt2/Sample_CHiC_AdipoPE_Brd4_Cre_D2/CHiC_AdipoPE_Brd4_Cre_D2_S3_L002_R2_mm9.bwt2merged.bam

## output= bowtie_results/bwt2/Sample_CHiC_AdipoPE_Brd4_Cre_D2/CHiC_AdipoPE_Brd4_Cre_D2_S3_L002_mm9.bwt2pairs.bam

## min mapq= 0

## report_single= False

## report_multi= False

## verbose= True

## Merging forward and reverse tags ...

Forward and reverse reads not paired. Check that BAM files have the same read names and are sorted.

I did go ahead and checked my BAM files: 

/bowtie_results/bwt2/Sample_CHiC_AdipoPE_Brd4_Cre_D2/CHiC_AdipoPE_Brd4_Cre_D2_S3_L002_R1_mm9.bwt2merged.bam

SN398:581:HVFCJBCXY:2:1101:1155:1996 16 chr5 81800358 42 51M * 0 0 CATCTTTCACTGAAAAATTCAGTAGATTTTTCATGCCAACTGTTTAAGGAA IHHIIIIIIIIIIIIIIIIIHHHHIIIIIIIIHIHIIIIHFIHHHGDDDDD AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:51 YT:Z:UU RG:Z:BMG

/bowtie_results/bwt2/Sample_CHiC_AdipoPE_Brd4_Cre_D2/CHiC_AdipoPE_Brd4_Cre_D2_S3_L002_R2_mm9.bwt2merged.bam

SN398:581:HVFCJBCXY:2:1101:1155:1996 0 chr5 77028533 42 51M * 0 0 GTTTTTAAAGTACTTTGAAAATTATTTGGATTAGAAAGGAGTTAAAGGAAA DDDDBHIIIGHHHGHH?FHHIF?HE11<<@11D1FHIH1FC11CHIC@HIH AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:51 YT:Z:UU RG:Z:BMG

If I am understanding correctly, the bolded part of the BAM files should be the same? Do you mind sharing your thoughts on this? Also, we receive sorted.bam and sorted.bam.bai files per sample from the core facility, do you think I can use this sorted.bam file to run HiC-Pro? I would appreciate your help on this! 

Best regards,

Victoria   

[Ajaz Wani]

Hi Nicholas,

I am running HiC-Pro and get stuck at pairing step.

Pairing of R1 and R2 tags ...

Logs: logs/triplemutant/mergeSAM.log

make: *** [bowtie_pairing] Error 1

I have checked the read names and made sure that they are exactly same between two fastq files.

Following are comments from log files.

mapping_comb.log----------

usr/local/bin/samtools merge -@ 2 -n -f bowtie_results/bwt2/triplemutant/tm_R1_sacCer3.bwt2merged.bam bowtie_results/bwt2_global/triplemutant/tm_R1_sacCer3.bwt2glob.bam bowtie_results/bwt2_local/triplemutant/tm_R1_sacCer3.bwt2glob.unmap_bwt2loc.bam

/usr/local/bin/samtools merge -@ 2 -n -f bowtie_results/bwt2/triplemutant/tm_R2_sacCer3.bwt2merged.bam bowtie_results/bwt2_global/triplemutant/tm_R2_sacCer3.bwt2glob.bam bowtie_results/bwt2_local/triplemutant/tm_R2_sacCer3.bwt2glob.unmap_bwt2loc.bam

/usr/local/bin/samtools sort -@ 2 -m 768M -n -T mp/tm_R1_sacCer3 -o bowtie_results/bwt2/triplemutant/tm_R1_sacCer3.bwt2merged.sorted.bam bowtie_results/bwt2/triplemutant/tm_R1_sacCer3.bwt2merged.bam

[E::hts_open_format] Failed to open file "mp/tm_R1_sacCer3.0000.bam" : No such file or directory

[E::hts_open_format] Failed to open file "mp/tm_R1_sacCer3.0001.bam" : No such file or directorysamtools sort: failed to create temporary file "mp/tm_R1_sacCer3.0000.bam": No such file or directory

samtools sort: failed to create temporary file "mp/tm_R1_sacCer3.0001.bam": No such file or directory

/usr/local/bin/samtools sort -@ 2 -m 768M -n -T mp/tm_R2_sacCer3 -o bowtie_results/bwt2/triplemutant/tm_R2_sacCer3.bwt2merged.sorted.bam bowtie_results/bwt2/triplemutant/tm_R2_sacCer3.bwt2merged.bam

[E::hts_open_format] Failed to open file "mp/tm_R2_sacCer3.0001.bam" : No such file or directory

[E::hts_open_format] Failed to open file "mp/tm_R2_sacCer3.0000.bam" : No such file or directory

samtools sort: failed to create temporary file "mp/tm_R2_sacCer3.0000.bam": No such file or directory

samtools sort: failed to create temporary file "mp/tm_R2_sacCer3.0001.bam": No such file or directory

mergeSAM.log-----------

/Library/Frameworks/Python.framework/Versions/2.7/bin/python /usr/local/bin/HiC-Pro_2.11.4/scripts/mergeSAM.py -q 0 -t -v -f bowtie_results/bwt2/triplemutant/tm_R1_sacCer3.bwt2merged.bam -r bowtie_results/bwt2/triplemutant/tm_R2_sacCer3.bwt2merged.bam -o bowtie_results/bwt2/triplemutant/tm_sacCer3.bwt2pairs.bam

## mergeBAM.py

## forward= bowtie_results/bwt2/triplemutant/tm_R1_sacCer3.bwt2merged.bam

## reverse= bowtie_results/bwt2/triplemutant/tm_R2_sacCer3.bwt2merged.bam

## output= bowtie_results/bwt2/triplemutant/tm_sacCer3.bwt2pairs.bam

## min mapq= 0

## report_single= False

## report_multi= False

## verbose= True

## Merging forward and reverse tags ...

Forward and reverse reads not paired. Check that BAM files have the same read names and are sorted.

Can you please help me with this.

Best

Ajaz

[nservant]

Hi

I did not see this error for a long time, and actually I never really understood what's going wrong :(

Your `samtools sort` command failed.

```

/samtools sort -@ 2 -m 768M -n -T mp/tm_R1_sacCer3

```

In theroy, the command should be 

```

-T /tmp/tm_R1_sacCer3

```

I never understood why the `tmp` become a `mp` !!!

Can you try to set up `TMP_DIR = mp` in the config file please ?

N