HiC Pro error (make: *** No targets. Stop.) even when using Singularity image

[Eric Kofman]

Hi there,

I see that somebody else had a similar issue but without a clear resolution. In my case, I am attempting to run the HiC-Pro pipeline from the pre-made singularity image available on the GitHub page (https://zerkalo.curie.fr/partage/HiC-Pro/singularity_images/hicpro_latest_ubuntu.img). 

I successfully pulled the image, conda activated the specified HiC-Pro conda environment, and then tried running using the command: 

(HiC-Pro_v3.1.0)$ singularity run hicpro_latest_ubuntu.img HiC-Pro -h
usage : HiC-Pro -i INPUT -o OUTPUT -c CONFIG [-s ANALYSIS_STEP] [-p] [-h] [-v]
Use option -h|--help for more information
Error - Installation - config system not detected. Please (re) install HiC-Pro !

Not sure what this means, given that I thought the program should be ready to go within the Singularity imagge. However when I try running from within singularity, e.g. 

singularity shell hicpro_latest_ubuntu.img

Then:

Singularity> HiC-Pro -i /projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/ -o /projects/ps-yeolab4/ekofman/Inge_PLAC/output_folder/ -c configPLACseq_hg19_myeloid_project_MG_Macrophages-JS.txt -p

I get

make: *** No targets.  Stop.

I am unsure why this would not be working out of the box with a singularity image... any ideas what's going on or how to get this to run? Thank you!

-Eric

[nservant]

Hi,

Could you try this one please ;

https://zerkalo.curie.fr/partage/HiC-Pro/singularity_images/hicpro_3.1.0_ubuntu.img

N

[Eric Kofman]

Thanks for the quick response. Looks like that is a slight improvement, in that the --help command now works:

(HiC-Pro_v3.1.0) $ singularity run hicpro_3.1.0_ubuntu.img HiC-Pro -h

usage : HiC-Pro -i INPUT -o OUTPUT -c CONFIG [-s ANALYSIS_STEP] [-p] [-h] [-v]
Use option -h|--help for more information

HiC-Pro 3.1.0
---------------
OPTIONS

   -i|--input INPUT : input data folder; Must contains a folder per sample with input files
....etc...
   [-h|--help]: help
   [-v|--version]: version

However the actual command is still strangely failing in the same way:

(HiC-Pro_v3.1.0) $ singularity run hicpro_3.1.0_ubuntu.img HiC-Pro -i /projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/ -o /projects/ps-yeolab4/ekofman/Inge_PLAC/output_folder/ -c configPLACseq_hg19_myeloid_project_MG_Macrophages-JS.txt -p

make: *** No targets.  Stop.

[nservant]

Could you try without the -p option ?

[Eric Kofman]

Same issue unfortunately even without -p (make: *** No targets.  Stop.)

[nservant]

I do not think this is related to the singularity image. Could you show me the content of your input folder and your configuration file please ?

[Eric Kofman]

the input folder is structured as such:

(HiC-Pro_v3.1.0) [ekofman@tscc-1-50 Inge_PLAC]$ ls -lh /projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/*
/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/RH_145:
total 22G
-rw-r--r-- 1 ekofman glass-group 10G Dec  8 18:39 RH_145_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 12G Dec  8 18:54 RH_145_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/RH_86:
total 25G
-rw-r--r-- 1 ekofman glass-group 12G Dec  8 18:38 RH_86_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 14G Dec  8 18:57 RH_86_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_Human_Microglia_PLAC_RH375_JOS_l20180507_TTAGGC_S1_L003:
total 12G
-rw-r--r-- 1 ekofman glass-group 5.4G Dec  8 18:59 human_Human_Microglia_PLAC_RH375_JOS_l20180507_TTAGGC_S1_L003_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 6.4G Dec  8 18:42 human_Human_Microglia_PLAC_RH375_JOS_l20180507_TTAGGC_S1_L003_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_Human_Microglia_PLAC_RH375_JOS_l20180507_TTAGGC_S1_L008:
total 12G
-rw-r--r-- 1 ekofman glass-group 5.3G Dec  8 18:58 human_Human_Microglia_PLAC_RH375_JOS_l20180507_TTAGGC_S1_L008_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 6.0G Dec  8 18:45 human_Human_Microglia_PLAC_RH375_JOS_l20180507_TTAGGC_S1_L008_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_Microglia_PLACseq_H3K4me3_RH145_MAH_l20171213_CGATGT_S5_L008:
total 4.4G
-rw-r--r-- 1 ekofman glass-group 2.0G Dec  8 18:47 human_Microglia_PLACseq_H3K4me3_RH145_MAH_l20171213_CGATGT_S5_L008_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 2.4G Dec  8 18:49 human_Microglia_PLACseq_H3K4me3_RH145_MAH_l20171213_CGATGT_S5_L008_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_P35_Microglia_PLAC_RH145_rep3_AN_CGATGT_S2_L006:
total 12G
-rw-r--r-- 1 ekofman glass-group 5.7G Dec  8 18:49 human_P35_Microglia_PLAC_RH145_rep3_AN_CGATGT_S2_L006_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 6.3G Dec  8 18:52 human_P35_Microglia_PLAC_RH145_rep3_AN_CGATGT_S2_L006_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_P35_Microglia_PLAC_RH145_rep4_AN_CGATGT_S5_L007:
total 12G
-rw-r--r-- 1 ekofman glass-group 5.7G Dec  8 18:50 human_P35_Microglia_PLAC_RH145_rep4_AN_CGATGT_S5_L007_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 6.4G Dec  8 18:48 human_P35_Microglia_PLAC_RH145_rep4_AN_CGATGT_S5_L007_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_P41_Microglia_PLAC_RH86_rep3_AN_ACTGAT_S1_L006:
total 15G
-rw-r--r-- 1 ekofman glass-group 7.0G Dec  8 18:40 human_P41_Microglia_PLAC_RH86_rep3_AN_ACTGAT_S1_L006_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 7.6G Dec  8 19:00 human_P41_Microglia_PLAC_RH86_rep3_AN_ACTGAT_S1_L006_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_P41_Microglia_PLAC_RH86_rep4_AN_ACTGAT_S4_L007:
total 15G
-rw-r--r-- 1 ekofman glass-group 7.0G Dec  8 18:36 human_P41_Microglia_PLAC_RH86_rep4_AN_ACTGAT_S4_L007_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 7.6G Dec  8 18:44 human_P41_Microglia_PLAC_RH86_rep4_AN_ACTGAT_S4_L007_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_P8_Microglia_PLAC_RH375_rep2_AN_TTAGGC_S3_L006:
total 12G
-rw-r--r-- 1 ekofman glass-group 5.5G Dec  8 18:51 human_P8_Microglia_PLAC_RH375_rep2_AN_TTAGGC_S3_L006_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 6.0G Dec  8 18:45 human_P8_Microglia_PLAC_RH375_rep2_AN_TTAGGC_S3_L006_R2.fastq.gz

/projects/ps-yeolab4/ekofman/Inge_PLAC/Microglia/human_P8_Microglia_PLAC_RH375_rep3_AN_TTAGGC_S6_L007:
total 12G
-rw-r--r-- 1 ekofman glass-group 5.5G Dec  8 18:46 human_P8_Microglia_PLAC_RH375_rep3_AN_TTAGGC_S6_L007_R1.fastq.gz
-rw-r--r-- 1 ekofman glass-group 6.0G Dec  8 18:41 human_P8_Microglia_PLAC_RH375_rep3_AN_TTAGGC_S6_L007_R2.fastq.gz

this is the config:

# Please change the variable settings below if necessary

#########################################################################
## Paths and Settings  - Do not edit !
#########################################################################

TMP_DIR = tmp
LOGS_DIR = logs
BOWTIE2_OUTPUT_DIR = bowtie_results
MAPC_OUTPUT = hic_results
RAW_DIR = rawdata

#######################################################################
## SYSTEM - PBS - Start Editing Here !!
#######################################################################
N_CPU = 15
LOGFILE = hicpro.log

JOB_NAME = PLACseq_12_9_2021
JOB_MEM = 15gb
JOB_WALLTIME = 6:00:00
JOB_QUEUE = batch
JOB_MAIL = ******@****.edu

#########################################################################
## Data
#########################################################################

PAIR1_EXT = _R1
PAIR2_EXT = _R2

#######################################################################
## Alignment options
#######################################################################

FORMAT = phred33
MIN_MAPQ = 0

BOWTIE2_IDX_PATH = /projects/ps-yeolab4/ekofman/Inge_PLAC/hg19_bowtie
BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder
BOWTIE2_LOCAL_OPTIONS =  --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################
## Annotation files
#######################################################################

REFERENCE_GENOME = genome
GENOME_SIZE = /projects/ps-yeolab4/ekofman/Inge_PLAC/chrom_hg19.sizes
#CAPTURE_TARGET (is new in 2.9)

#######################################################################
## Allele specific
#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################
## Digestion Hi-C
#######################################################################

GENOME_FRAGMENT = /projects/ps-yeolab4/ekofman/Inge_PLAC/IGenomes_hg19_wholeGenome_FASTA_restriction.^GATC.bed
LIGATION_SITE = CTAG
MIN_FRAG_SIZE = 100 ##DEFAULT
MAX_FRAG_SIZE = 100000 #DEFAULT
MIN_INSERT_SIZE = 100 #DEFAULT
MAX_INSERT_SIZE = 1500 #DEFAULT is 600, voorheen 1000

#######################################################################
## Hi-C processing
#######################################################################

MIN_CIS_DIST =
GET_ALL_INTERACTION_CLASSES = 1
GET_PROCESS_SAM = 1
RM_SINGLETON = 1
RM_MULTI = 1
RM_DUP = 1

#######################################################################
## Contact Maps
#######################################################################

BIN_SIZE =  5000 10000 20000 50000 100000  
MATRIX_FORMAT = upper

#######################################################################
## ICE Normalization
#######################################################################
MAX_ITER = 100
FILTER_LOW_COUNT_PERC = 0.02
FILTER_HIGH_COUNT_PERC = 0
EPS = 0.1


#######################################################################
## HiC-Pro - System settings -- van dat andere bestand gekopieerd!
#######################################################################
#######################################################################
## Required Software - Specified the DIRECTORY name of the executables
## If not specified, the program will try to locate the executable
## using the 'which' command
#######################################################################
BOWTIE2_PATH = /home/ekofman/new_anaconda3/envs/HiC-Pro_v3.1.0/bin/bowtie2
SAMTOOLS_PATH = /home/ekofman/new_anaconda3/envs/HiC-Pro_v3.1.0/bin/samtools
R_PATH = /home/ekofman/new_anaconda3/envs/HiC-Pro_v3.1.0/bin/R
PYTHON_PATH = /home/ekofman/new_anaconda3/envs/HiC-Pro_v3.1.0/bin/python
CLUSTER_SYS = TORQUE

[nservant]

Everything looks good :(

Could you try with the test data please ?

https://zerkalo.curie.fr/partage/HiC-Pro/HiCPro_testdata.tar.gz

[nservant]

oh ... I'm wondering if this could be an issue with Singularity which cannot access your data ?

Could you please add the `--bind /projects/ps-yeolab4/ekofman/Inge_PLAC/` option ?

[Eric Kofman]

Hmm you are indeed right that adding the --bind was necessary, because the example seems to be working with the following command:

singularity run --bind /projects/ps-yeolab4/ekofman hicpro_3.1.0_ubuntu.img HiC-Pro -c config_test_latest.txt -i test_data -o hicpro_latest_test

So it seems there must just be something wrong with my config. I will try to see what in the config might be improperly specified and will report as soon as I diagnose it.

[Eric Kofman]

I am really not sure what the fix was, but there was something wrong with the old config that was breaking things. Perhaps an extra space, missing newline at end of file, or just improper character due to copying and pasting potentially. For future users who run into this issue, just start from the test config file and work from there instead. Thank you nservant for your quick responses and help!

This one now works:

# Please change the variable settings below if necessary

#########################################################################
## Paths and Settings  - Do not edit !
#########################################################################

TMP_DIR = tmp
LOGS_DIR = logs
BOWTIE2_OUTPUT_DIR = bowtie_results
MAPC_OUTPUT = hic_results
RAW_DIR = rawdata

#######################################################################
## SYSTEM - PBS - Start Editing Here !!
#######################################################################

N_CPU = 2
LOGFILE = hicpro.log

JOB_NAME = Inge_split
JOB_MEM = 10gb
JOB_WALLTIME = 10:00:00
JOB_QUEUE = batch
JOB_MAIL = #####@####.edu

#########################################################################
## Data
#########################################################################

PAIR1_EXT = _R1
PAIR2_EXT = _R2

#######################################################################
## Alignment options
#######################################################################

FORMAT = phred33
MIN_MAPQ = 0

BOWTIE2_IDX_PATH = /projects/ps-yeolab4/ekofman/Inge_PLAC/hg19_bowtie
BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder
BOWTIE2_LOCAL_OPTIONS =  --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder


#######################################################################
## Annotation files
#######################################################################

REFERENCE_GENOME = hg19
GENOME_SIZE = /projects/ps-yeolab4/ekofman/Inge_PLAC/chrom_hg19.sizes

BIN_SIZE = 500000 1000000

MATRIX_FORMAT = upper

#######################################################################
## ICE Normalization
#######################################################################
MAX_ITER = 100
FILTER_LOW_COUNT_PERC = 0.02
FILTER_HIGH_COUNT_PERC = 0
EPS = 0.1

[Eric Kofman]

Only one last thing -- once I have the two scripts generated for running on a cluster, what is the best approach to run them using the singularity install?

[nservant]

Good question ! I think the easiest would be to modified the scripts and to add a `singularity run` prefix before the command. I guess it should run.

However, running HiC-Pro in parallele mode with Singularity has not been really tested because in the meantime we developed the nf-core-hic pipeline (https://github.com/nf-core/hic)

which i/ give the same results than HiC-Pro ii/ is natively supporting parallele computing and containers like docker or singularity.

FInally, if you really want to run HiC-Pro in parallel on a cluster with Singularity, I think the best strategy would be to run the nf-core-hic pipeline !