excessive RAM use after splitting partitions

[jeremy le luyer]

Hello,

I tested EVM to compile different genes prediction, however even after partitioning I end up with a very large memory usage (>200Go), which looks excessive. 

"""

partition_EVM_inputs.pl --genome "$GENOME" \

 --gene_predictions denovo.test.gff3 --protein_alignments homology.test.gff3 \

 --transcript_alignments transcript.test.gff3 \

--segmentSize 100000 --overlapSize 10000 --partition_listing partitions_list.out

write_EVM_commands.pl --genome "$GENOME" --weights "$WEIGHT" \

 --gene_predictions denovo.test.gff3 --protein_alignments homology.test.gff3 \

--transcript_alignments transcript.test.gff3 \

  --output_file_name "$OUTPUT" --partitions partitions_list.out >commands.list

"""

I tried running a single command to check how much RAM would be necessary:

"""

evidence_modeler.pl --genome sspace.final.scaffolds.fasta --gene_predictions denovo.gff3 --weights weight.txt --transcript_alignments transcript.gff3  --protein_alignments homology.gff3  --exec_dir scaffold1size567472/scaffold1size567472_450001-550000 > scaffold1size567472/scaffold1size567472_450001-550000/EVM_output.txt 2> scaffold1size567472/scaffold1size567472_450001-550000/EVM_output.txt.lo

"""

"""

=>> PBS: job killed: mem 211812812kb exceeded limit 209715200kb

"""

the files included in this partition have a reasonable size:

3.8 Feb 28 23:33 scaffold1size567472/scaffold1size567472_450001-550000/denovo.test.gff3

0 Mar  1 01:29 scaffold1size567472/scaffold1size567472_450001-550000/EVM_output.txt

171 Mar  1 01:29 scaffold1size567472/scaffold1size567472_450001-550000/EVM_output.txt.log

0 Feb 28 23:33 scaffold1size567472/scaffold1size567472_450001-550000/homology.test.gff3

100K Feb 28 23:33 scaffold1size567472/scaffold1size567472_450001-550000/sspace.final.scaffolds.fasta

17K Feb 28 23:33 scaffold1size567472/scaffold1size567472_450001-550000/transcript.test.gff3

Do you have any idea of what I might be doing wrong here ?

Thank you

J.

[Brian Haas]

Hi,

Can you send me the inputs that are causing this?   I'll figure out what the issue is.

bh...@broadinstitute.org

best,

~brian

--
You received this message because you are subscribed to the Google Groups "EVidenceModeler-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evidencemodeler-...@googlegroups.com.
To post to this group, send email to evidencemo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/evidencemodeler-users/ca71bb37-3291-4831-a57f-59be82462577%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[Brian Haas]

Hi Jeremy,

Thanks for sending one of the examples.   I ran it like so on my laptop:

~/GITHUB/EVidenceModeler/evidence_modeler.pl --genome sspace.final.scaffolds.fasta --gene_predictions denovo.test.gff3 --weights weight.txt --transcript_alignments transcript.test.gff3 > evm.out

and it ran through in a few seconds w/o any trouble.

If you do the same, does it work for you?

It could be that this is not the small partition that's causing trouble. If there's a way to see which inputs are driving the RAM up, that's the one that we want to tackle.

best,

~brian

[linz li]

Dear Brian, 

I meet a same problem when I run EVM, I split each file into very small file. but it showed >300Gb RAM, and got out of memory error. Did you guys fixed it?

Best

Linzhou

[Brian Haas]

Hi,

Do you have a small example you can share with me that reproduces the problem?  My guess is that there's something off with the coordinates.  If I can nail it down, I can patch the code to prevent it in the future.

You can send files to:  bh...@broadinstitute.org

thx

--
You received this message because you are subscribed to the Google Groups "EVidenceModeler-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evidencemodeler-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/evidencemodeler-users/6fcdf3d3-8eae-4a8e-8dd4-64ec815c1869o%40googlegroups.com.