bowtie_pairing error but everything is produced

245 views
Skip to first unread message

Claudia Giambartolomei

unread,
Jan 30, 2020, 3:47:22 AM1/30/20
to hic...@googlegroups.com

Hi Nicolas,

I get an error 

HiCpro_s1_hic.e830665.1:make: *** [/home2/bin/HiC-Pro_2.11.1/scripts/Makefile:144: bowtie_pairing] Error 1


But everything else seems fine: all the files are produced and I could not find an error in my log files. However in the mapping log file I found this: 


mapping_stats.log (same error in mapping_combine.log)


/apps/bio/samtools/1.9/samtools-1.9/samtools view -c bowtie_results/bwt2/LNK27/LNK27_R2_val_2_hg19.bwt2merged.bam

/apps/bio/samtools/1.9/samtools-1.9/samtools view -c bowtie_results/bwt2/LNK27/LNK27_R1_val_1_hg19.bwt2merged.bam

[W::bam_hdr_read] EOF marker is absent. The input is probably truncated

[E::bgzf_read] Read block operation failed with error 4 after 0 of 4 bytes

[main_samview] truncated file.


samtools Version: 1.9 (using htslib 1.9)

Bowtie 2 version 2.3.5.1 


But then the program continues so it all seems OK otherwise.
Do you think there is a problem? I tried re-installing bowtie2 as well but the error remained. 

Many thanks for your advice as always.
All Best,
Claudia

nservant

unread,
Jan 30, 2020, 4:08:34 AM1/30/20
to HiC-Pro
Hi
I'm afraid the pairing of the two BAM files was not complete. 
Could you check the file bwt2pairs.bam. In theory, you should have exactly the same number of reads in the R1.bwt2merged.bam, R2.bwt2merged.bam
and the bwt2pairs.bam.
Best

Claudia Giambartolomei

unread,
Jan 30, 2020, 5:06:11 AM1/30/20
to nservant, HiC-Pro
Hi Nicolas,

Thank you for the quick reply!
Nope I confirm the pairing did not work then...

All different rows in:

 wc -l rep_hg19.bwt2pairs.bam 

11447389 rep_hg19.bwt2pairs.bam


wc -l rep_R1_val_1_hg19.bwt2merged.bam

9551287 rep_R1_val_1_hg19.bwt2merged.bam


wc -l rep_R2_val_2_hg19.bwt2merged.bam

10778496 rep_R2_val_2_hg19.bwt2merged.bam


Maybe there is a problem with my package versions (I had problems before also with bx-python installation to work properly). 
I am on a shared cluster system where I do not have permissions so I am using an environment. 
What would you think is the safest way to go about re-installing everything? Below is what I plan to re-do from the start... If you have any suggestions please do let me know. Thank you again!

1) download HiC-Pro
wget https://github.com/nservant/HiC-Pro/archive/v2.11.1.tar.gz
tar xvzf V2.11.1
cd HiC-Pro

2) install dependencies:
conda create -n hicpro 
conda activate hicpro

# check python version
python --version
# Python 2.7.16 :: Anaconda, Inc.
# otherwise do: conda install -y python=2.7.XX
conda install -y -c anaconda scipy
conda install -y -c anaconda numpy
conda install -y -c bcbio bx-python
conda install -y -c bioconda pysam
conda install -c bioconda iced
conda update readline
conda install -c r r-base
conda install -c r r-ggplot2=2.2.1
conda install -c r r-rcolorbrewer
conda install -c r r-gridbase

3) install main
make configure




--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/hic-pro/662ed909-c0f7-407b-b235-510df9d009dd%40googlegroups.com.


--
Research Fellow 
MINDED and Marie Skłodowska-Curie Action
Istituto Italiano di Tecnologia, Central RNA Lab, Genova, Italy

nservant

unread,
Jan 30, 2020, 5:18:57 AM1/30/20
to HiC-Pro
So the point which is weird is that you do not have the same number of reads in your two bwt2merged.bam files.

wc -l rep_R1_val_1_hg19.bwt2merged.bam

9551287 rep_R1_val_1_hg19.bwt2merged.bam


wc -l rep_R2_val_2_hg19.bwt2merged.bam

10778496 rep_R2_val_2_hg19.bwt2merged.bam




First, could you pleae check if you do have the same number of reads in your original R1/R2 files ?
Then, if there is an error, you should have something in the mergeSAM.log files. Could you also check them ?
As you mentionned, it can be due to an installation issue, but in this case, you should have clear error messages in the logs
N


On Thursday, 30 January 2020 11:06:11 UTC+1, Claudia Giambartolomei wrote:
Hi Nicolas,

To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+unsubscribe@googlegroups.com.


--
Research Fellow 
MINDED and Marie Skłodowska-Curie Action
Istituto Italiano di Tecnologia, Central RNA Lab, Genova, Italy

Claudia Giambartolomei

unread,
Jan 30, 2020, 6:07:41 AM1/30/20
to nservant, HiC-Pro
Hi Nicolas,

Yes, thank you for your help. 

First, could you pleae check if you do have the same number of reads in your original R1/R2 files ?

gunzip -c rep_R1_val_1.fastq.gz | wc -l

100752436

gunzip -c rep_R2_val_2.fastq.gz | wc -l

100752436



Then, if there is an error, you should have something in the mergeSAM.log files. Could you also check them ?
There is no error or warning here and everything seems OK... I copied the log file in any case below...

Thanks,
Claudia

# mergeSAM.log

/home2/anaconda2/bin/python /home2/bin/HiC-Pro_2.11.1/scripts/mergeSAM.py -q 20 -t -v -f bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam -r bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam -o bowtie_results/bwt2/rep/rep_hg19.bwt2pairs.bam

## mergeBAM.py

## forward= bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam

## reverse= bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam

## output= bowtie_results/bwt2/rep/rep_hg19.bwt2pairs.bam

## min mapq= 20

## report_single= False

## report_multi= False

## verbose= True

## Merging forward and reverse tags ...

## 1000000

## 2000000

## 3000000

## 4000000

## 5000000

## 6000000

## 7000000

## 8000000

## 9000000

## 10000000

## 11000000

## 12000000

## 13000000

## 14000000

## 15000000

## 16000000

## 17000000

## 18000000

## 19000000

## 20000000

## 21000000

## 22000000

## 23000000

## 24000000

## 25000000

~ 


To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+u...@googlegroups.com.


--
Research Fellow 
MINDED and Marie Skłodowska-Curie Action
Istituto Italiano di Tecnologia, Central RNA Lab, Genova, Italy

--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/hic-pro/d25ca8a8-4037-4e25-a966-4dcaafa9139f%40googlegroups.com.

nservant

unread,
Jan 30, 2020, 6:22:35 AM1/30/20
to HiC-Pro
Sorry, I was thinking about the mapping_combine.log file please ?


On Thursday, 30 January 2020 12:07:41 UTC+1, Claudia Giambartolomei wrote:
Hi Nicolas,

To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+unsubscribe@googlegroups.com.


--
Research Fellow 
MINDED and Marie Skłodowska-Curie Action
Istituto Italiano di Tecnologia, Central RNA Lab, Genova, Italy

--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+unsubscribe@googlegroups.com.

Claudia Giambartolomei

unread,
Jan 30, 2020, 6:43:54 AM1/30/20
to nservant, HiC-Pro
Ah yes... this file definitely has errors:

/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R1_val_1_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R1_val_1_hg19.bwt2glob.unmap_bwt2loc.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R2_val_2_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam

[W::bam_hdr_read] EOF marker is absent. The input is probably truncated
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated
/apps/bio/samtools/1.9/samtools-1.9/samtools sort -@ 4 -n -T tmp/rep_R1_val_1_hg19 -o bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam

[E::bgzf_read] Read block operation failed with error 4 after 0 of 4 bytes
samtools merge: "bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam" is truncated
[bam_sort_core] merging from 8 files and 4 in-memory blocks...
mv bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R1_val_1_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R1_val_1_hg19.bwt2glob.unmap_bwt2loc.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R2_val_2_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools sort -@ 4 -n -T tmp/rep_R1_val_1_hg19 -o bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools sort -@ 4 -n -T tmp/rep_R2_val_2_hg19 -o bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam
[bam_sort_core] merging from 8 files and 4 in-memory blocks...
[bam_sort_core] merging from 8 files and 4 in-memory blocks...
mv bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam
mv bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R2_val_2_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R1_val_1_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R1_val_1_hg19.bwt2glob.unmap_bwt2loc.bam
[W::bam_hdr_read] [W::bam_hdr_read] EOF marker is absent. The input is probably truncated

EOF marker is absent. The input is probably truncated
[E::bgzf_read] Read block operation failed with error 4 after 0 of 4 bytes
samtools merge: "bowtie_results/bwt2_local/rep/rep_R1_val_1_hg19.bwt2glob.unmap_bwt2loc.bam" is truncated

[E::bgzf_read] Read block operation failed with error 4 after 0 of 4 bytes
samtools merge: "bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam" is truncated
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R1_val_1_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R1_val_1_hg19.bwt2glob.unmap_bwt2loc.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R2_val_2_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools sort -@ 4 -n -T tmp/rep_R1_val_1_hg19 -o bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam
/apps/bio/samtools/1.9/samtools-1.9/samtools sort -@ 4 -n -T tmp/rep_R2_val_2_hg19 -o bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam
[bam_sort_core] merging from 8 files and 4 in-memory blocks...
[bam_sort_core] merging from 8 files and 4 in-memory blocks...
mv bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R1_val_1_hg19.bwt2merged.bam
mv bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.sorted.bam bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam

To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+u...@googlegroups.com.


--
Research Fellow 
MINDED and Marie Skłodowska-Curie Action
Istituto Italiano di Tecnologia, Central RNA Lab, Genova, Italy

--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/hic-pro/d82f1270-d62d-4cb3-801a-c31645daed7d%40googlegroups.com.

nservant

unread,
Jan 30, 2020, 6:48:21 AM1/30/20
to HiC-Pro
You can see that the first `samtools merge` has an error ... not sure why ... memory issue ? disk space issue ? difficult to say.
Maybe you can try to run this command by yourself to see if it works. Or you can still try to double check the two bam files, and check if bowtie has finished without any error ...

/apps/bio/samtools/1.9/samtools-1.9/samtools merge -@ 4 -n -f bowtie_results/bwt2/rep/rep_R2_val_2_hg19.bwt2merged.bam bowtie_results/bwt2_global/rep/rep_R2_val_2_hg19.bwt2glob.bam bowtie_results/bwt2_local/rep/rep_R2_val_2_hg19.bwt2glob.unmap_bwt2loc.bam
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated

To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+unsubscribe@googlegroups.com.


--
Research Fellow 
MINDED and Marie Skłodowska-Curie Action
Istituto Italiano di Tecnologia, Central RNA Lab, Genova, Italy

--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "HiC-Pro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hic-pro+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages