std::bad_alloc or Exit_status=137 with populations

58 views

Skip to first unread message

M Noon

unread,

Jul 8, 2015, 1:27:53 PM7/8/15

to stacks...@googlegroups.com

Hi Stack users,

I know people have reported a similar error before and my problem seems to be memory related as well. Here is the command I used:

populations -b 1 -P /rsgrps2/sjmiller/Galbraith_Pistachio/bcMM2_adapt/denovo_correction_3_smp -M /rsgrps2/sjmiller/Galbraith_Pistachio/popmap_95 -t 12 -r 0.75 --fstats -s --vcf --vcf_haplotypes --genepop --structure --phase --plink --phylip --phylip_var

Basically I have 95 samples, some of which has replicates and most are unknown. I am trying to find their relatedness by running structure and/or phylip next. Since I dont know anything about unknown samples, I considered them as distinct populations. Thats why I have about 77 populations reported in the population file.

First error I received was with exit_status 137 which indicates time limit exceeded and job terminated in an hour even though wall time had many hours left. I increased the wall time and re submitted the job.

Second error was bad_alloc, I requested more memory on nodes and re-submitted the job. it got terminated in about 8 hours.

Third time I got exit_status 137 again, even though I have 80 wall time hours.

Below is my memory and time request parameters.

#PBS -l select=4:ncpus=16:mem=63Gb => its about 252Gb with four nodes

#PBS -l pvmem=1000gb

#PBS -l cput=864:0:0

#PBS -l walltime=80:0:0

I dont understand why it complains about time/memory when I have requested ample memory and time. what do you think is happening here? how much more memory should I request to complete this job. I have tried using filters with -P 77 and -r 0.5 parameters. The job completes within 10-15 minutes without any error but I end up with very small number of loci.

my questions are:

1. why I encounter these memory or time limit errors and how much more time and memory do I need to complete my job without using filters?

2. how much would it affect the output if I used filters ( -P 77 and -r 0.5 ). Would it impact the final answer I am trying to get?

3. I want to know if I am using a right approach to figure out relatedness between known and unknown samples?

Any help/feedback is appreciated. Thanks,

Muhammad

here is my full code that includes all assembly steps with denovomap, correction with rxStacks and remapping.

## Run the pipeline as usual, turn off database interaction for now.

cd $PBS_O_WORKDIR/bcMM2_adapt

mkdir denovo_pipeline_3

denovo_map.pl -T 12 -m 3 -M 3 -n 2 -b 1 -t -S -D "batch_1" -O /rsgrps2/sjmiller/Galbraith_Pistachio/popmap_95 -o denovo_pipeline_3 \

-s 1.fq -s 2.fq -s 3.fq -s 4.fq -s 5.fq -s 6.fq -s 7.fq -s 8.fq -s 9.fq -s 10.fq \

-s 11.fq -s 12.fq -s 13.fq -s 14.fq -s 15.fq -s 16.fq -s 17.fq -s 18.fq -s 19.fq -s 20.fq \

-s 21.fq -s 22.fq -s 23.fq -s 24.fq -s 25.fq -s 26.fq -s 27.fq -s 28.fq -s 29.fq -s 30.fq \

-s 31.fq -s 32.fq -s 33.fq -s 34.fq -s 35.fq -s 36.fq -s 37.fq -s 38.fq -s 39.fq -s 40.fq \

-s 41.fq -s 42.fq -s 43.fq -s 44.fq -s 45.fq -s 46.fq -s 47.fq -s 48.fq -s 49.fq -s 50.fq \

-s 51.fq -s 52.fq -s 53.fq -s 54.fq -s 55.fq -s 56.fq -s 57.fq -s 58.fq -s 59.fq -s 60.fq \

-s 61.fq -s 62.fq -s 63.fq -s 64.fq -s 65.fq -s 66.fq -s 67.fq -s 68.fq -s 69.fq -s 70.fq \

-s 71.fq -s 72.fq -s 73.fq -s 74.fq -s 75.fq -s 76.fq -s 77.fq -s 78.fq -s 79.fq -s 80.fq \

-s 81.fq -s 82.fq -s 83.fq -s 84.fq -s 85.fq -s 86.fq -s 87.fq -s 88.fq -s 89.fq -s 90.fq \

-s 91.fq -s 92.fq -s 93.fq -s 94.fq -s 95.fq

## Make population-based corrections.

cd $PBS_O_WORKDIR/bcMM2_adapt

mkdir denovo_correction_3

rxstacks -b 1 -P denovo_pipeline_3 -o denovo_correction_3 \

--conf_lim 0.25 --prune_haplo \

--model_type bounded --bound_high 0.05 \

--lnl_lim -8.0 --lnl_dist -t 8 --verbose

## Rebuild the catalog.

cd $PBS_O_WORKDIR/bcMM2_adapt/denovo_correction_3

mkdir rebulid_catalog_2

cstacks -b 1 -n 2 -p 36 -o rebulid_catalog_2 \

-s 1 -s 2 -s 3 -s 4 -s 5 -s 6 -s 7 -s 8 -s 9 -s 10 \

-s 11 -s 12 -s 13 -s 14 -s 15 -s 16 -s 17 -s 18 -s 19 -s 20 \

-s 21 -s 22 -s 23 -s 24 -s 25 -s 26 -s 27 -s 28 -s 29 -s 30 \

-s 31 -s 32 -s 33 -s 34 -s 35 -s 36 -s 37 -s 38 -s 39 -s 40 \

-s 41 -s 42 -s 43 -s 44 -s 45 -s 46 -s 47 -s 48 -s 49 -s 50 \

-s 51 -s 52 -s 53 -s 54 -s 55 -s 56 -s 57 -s 58 -s 59 -s 60 \

-s 61 -s 62 -s 63 -s 64 -s 65 -s 66 -s 67 -s 68 -s 69 -s 70 \

-s 71 -s 72 -s 73 -s 74 -s 75 -s 76 -s 77 -s 78 -s 79 -s 80 \

-s 81 -s 82 -s 83 -s 84 -s 85 -s 86 -s 87 -s 88 -s 89 -s 90 \

-s 91 -s 92 -s 93 -s 94 -s 95

## Rematch against the catalog.

cd $PBS_O_WORKDIR/bcMM2_adapt/denovo_correction_3/rebulid_catalog_2

mkdir match_catalog

sstacks -b 1 -c batch_1 -s ../1 -o match_catalog

sstacks -b 1 -c batch_1 -s ../2 -o match_catalog

sstacks -b 1 -c batch_1 -s ../3 -o match_catalog

sstacks -b 1 -c batch_1 -s ../4 -o match_catalog

sstacks -b 1 -c batch_1 -s ../5 -o match_catalog

sstacks -b 1 -c batch_1 -s ../6 -o match_catalog

sstacks -b 1 -c batch_1 -s ../7 -o match_catalog

sstacks -b 1 -c batch_1 -s ../8 -o match_catalog

sstacks -b 1 -c batch_1 -s ../9 -o match_catalog

sstacks -b 1 -c batch_1 -s ../10 -o match_catalog

sstacks -b 1 -c batch_1 -s ../11 -o match_catalog

sstacks -b 1 -c batch_1 -s ../12 -o match_catalog

sstacks -b 1 -c batch_1 -s ../13 -o match_catalog

sstacks -b 1 -c batch_1 -s ../14 -o match_catalog

sstacks -b 1 -c batch_1 -s ../15 -o match_catalog

sstacks -b 1 -c batch_1 -s ../16 -o match_catalog

sstacks -b 1 -c batch_1 -s ../17 -o match_catalog

sstacks -b 1 -c batch_1 -s ../18 -o match_catalog

sstacks -b 1 -c batch_1 -s ../19 -o match_catalog

sstacks -b 1 -c batch_1 -s ../20 -o match_catalog

sstacks -b 1 -c batch_1 -s ../21 -o match_catalog

sstacks -b 1 -c batch_1 -s ../22 -o match_catalog

sstacks -b 1 -c batch_1 -s ../23 -o match_catalog

sstacks -b 1 -c batch_1 -s ../24 -o match_catalog

sstacks -b 1 -c batch_1 -s ../25 -o match_catalog

sstacks -b 1 -c batch_1 -s ../26 -o match_catalog

sstacks -b 1 -c batch_1 -s ../27 -o match_catalog

sstacks -b 1 -c batch_1 -s ../28 -o match_catalog

sstacks -b 1 -c batch_1 -s ../29 -o match_catalog

sstacks -b 1 -c batch_1 -s ../30 -o match_catalog

sstacks -b 1 -c batch_1 -s ../31 -o match_catalog

sstacks -b 1 -c batch_1 -s ../32 -o match_catalog

sstacks -b 1 -c batch_1 -s ../33 -o match_catalog

sstacks -b 1 -c batch_1 -s ../34 -o match_catalog

sstacks -b 1 -c batch_1 -s ../35 -o match_catalog

sstacks -b 1 -c batch_1 -s ../36 -o match_catalog

sstacks -b 1 -c batch_1 -s ../37 -o match_catalog

sstacks -b 1 -c batch_1 -s ../38 -o match_catalog

sstacks -b 1 -c batch_1 -s ../39 -o match_catalog

sstacks -b 1 -c batch_1 -s ../40 -o match_catalog

sstacks -b 1 -c batch_1 -s ../41 -o match_catalog

sstacks -b 1 -c batch_1 -s ../42 -o match_catalog

sstacks -b 1 -c batch_1 -s ../43 -o match_catalog

sstacks -b 1 -c batch_1 -s ../44 -o match_catalog

sstacks -b 1 -c batch_1 -s ../45 -o match_catalog

sstacks -b 1 -c batch_1 -s ../46 -o match_catalog

sstacks -b 1 -c batch_1 -s ../47 -o match_catalog

sstacks -b 1 -c batch_1 -s ../48 -o match_catalog

sstacks -b 1 -c batch_1 -s ../49 -o match_catalog

sstacks -b 1 -c batch_1 -s ../50 -o match_catalog

sstacks -b 1 -c batch_1 -s ../51 -o match_catalog

sstacks -b 1 -c batch_1 -s ../52 -o match_catalog

sstacks -b 1 -c batch_1 -s ../53 -o match_catalog

sstacks -b 1 -c batch_1 -s ../54 -o match_catalog

sstacks -b 1 -c batch_1 -s ../55 -o match_catalog

sstacks -b 1 -c batch_1 -s ../56 -o match_catalog

sstacks -b 1 -c batch_1 -s ../57 -o match_catalog

sstacks -b 1 -c batch_1 -s ../58 -o match_catalog

sstacks -b 1 -c batch_1 -s ../59 -o match_catalog

sstacks -b 1 -c batch_1 -s ../60 -o match_catalog

sstacks -b 1 -c batch_1 -s ../61 -o match_catalog

sstacks -b 1 -c batch_1 -s ../62 -o match_catalog

sstacks -b 1 -c batch_1 -s ../63 -o match_catalog

sstacks -b 1 -c batch_1 -s ../64 -o match_catalog

sstacks -b 1 -c batch_1 -s ../65 -o match_catalog

sstacks -b 1 -c batch_1 -s ../66 -o match_catalog

sstacks -b 1 -c batch_1 -s ../67 -o match_catalog

sstacks -b 1 -c batch_1 -s ../68 -o match_catalog

sstacks -b 1 -c batch_1 -s ../69 -o match_catalog

sstacks -b 1 -c batch_1 -s ../70 -o match_catalog

sstacks -b 1 -c batch_1 -s ../71 -o match_catalog

sstacks -b 1 -c batch_1 -s ../72 -o match_catalog

sstacks -b 1 -c batch_1 -s ../73 -o match_catalog

sstacks -b 1 -c batch_1 -s ../74 -o match_catalog

sstacks -b 1 -c batch_1 -s ../75 -o match_catalog

sstacks -b 1 -c batch_1 -s ../76 -o match_catalog

sstacks -b 1 -c batch_1 -s ../77 -o match_catalog

sstacks -b 1 -c batch_1 -s ../78 -o match_catalog

sstacks -b 1 -c batch_1 -s ../79 -o match_catalog

sstacks -b 1 -c batch_1 -s ../80 -o match_catalog

sstacks -b 1 -c batch_1 -s ../81 -o match_catalog

sstacks -b 1 -c batch_1 -s ../82 -o match_catalog

sstacks -b 1 -c batch_1 -s ../83 -o match_catalog

sstacks -b 1 -c batch_1 -s ../84 -o match_catalog

sstacks -b 1 -c batch_1 -s ../85 -o match_catalog

sstacks -b 1 -c batch_1 -s ../86 -o match_catalog

sstacks -b 1 -c batch_1 -s ../87 -o match_catalog

sstacks -b 1 -c batch_1 -s ../88 -o match_catalog

sstacks -b 1 -c batch_1 -s ../89 -o match_catalog

sstacks -b 1 -c batch_1 -s ../90 -o match_catalog

sstacks -b 1 -c batch_1 -s ../91 -o match_catalog

sstacks -b 1 -c batch_1 -s ../92 -o match_catalog

sstacks -b 1 -c batch_1 -s ../93 -o match_catalog

sstacks -b 1 -c batch_1 -s ../94 -o match_catalog

sstacks -b 1 -c batch_1 -s ../95 -o match_catalog

## Calculate population statistics.

Reply all

Reply to author

Forward

0 new messages