Error: cons memory exhausted

[Ellen McMullen]

Hello Dr. Broman,

I have been using RQTL to analyze a very large dataset. I started off testing everything with a small subset of the markers, then used an RStudio AMI on AWS to run the same analysis on a larger subset of the markers on an instance with  more RAM, so I know the analysis works on this AWS instance type.

When I increased the subset of markers again, the analysis failed during scantwo (the initial analysis, not the permutations). It gave a memory error, despite the instance monitoring showing plenty of available RAM (~85% free). I had already set the R session settings so that R could access nearly all of the RAM.

This is the exact error message printed:
Error: cons memory exhausted (limit reached?)
Error during wrapup: cons memory exhausted (limit reached?)
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
It prints maybe 40 times in a row and then exits the function.

I am a little stumped on what could be causing this issue. I know some of the RQTL functions go down into C - is there any configuration I need to do to allow C to access enough RAM? Or do you know what else could be causing this memory error?

Thanks,
Ellen McMullen

[Karl Broman]

What is the number of markers, and how much RAM is there?

I'm not aware of any need to enable access to memory; R handles it automatically.

My guess is that it's asking for more than is available.

karl

[Ellen McMullen]

I am using 3000 markers, and have 2Tb of RAM. My memory logging shows only ~220Gb was used when it failed.

I will also mention that it is not failing at the beginning of the function. It completes the scan of (2,2) and a decent portion of (2,3) before it fails.

--
You received this message because you are subscribed to a topic in the Google Groups "R/qtl discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rqtl-disc/6Wn9XweltSs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rqtl-disc+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqtl-disc/d3339b60-1401-4cb6-b21d-bf7507d22443n%40googlegroups.com.

[Karl Broman]

Maybe try using assumeCondIndep=TRUE in the call to scantwo().

scantwo with 3000 markers shouldn't use very much memory. 3000x3000 doubles is only like 69 Mb.

But for the 2d-scan within a chromosome, it calculates the joint genotype probabilities for all pairs of positions; maybe that object is becoming really large?

karl

[Ellen McMullen]

Yes, I was not expecting to have any memory issues until I got to the scantwo permutations.

Thanks, I will try that out! Does assumeCondIndep work for both the scan and the permutations?

To view this discussion on the web visit https://groups.google.com/d/msgid/rqtl-disc/976dc27a-f086-4096-bedd-4afbc35e9095n%40googlegroups.com.

[Karl Broman]

Yes the assumeCondIndep can also be used in permutations.

And if you’re using Haley-Knott regression, it’s  best to use scantwopermhk for the permutations.

karl