Problems with inla.collect.results(results.dir,...)

[Maria Rodrigo]

Dear INLA team,

I am using INLA to investigate several thousands of genes and I can see that at some point it stops running and I just get an error message again and again:

Error in inla.collect.results(results.dir, control.results = cont.results,  :
  This is not a directory:  /tmp/RtmpRcTmIn/file69ea6bf38f1a/results.files

The folder name ("file........") changes for each gene that is not analysed. When I look at the output of inla() for those genes, it is empty, though it looks like it took the time to process them

Call:
c("inla(formula = DEU.formula, family = \"gaussian\", data = gene_dataframe, ",  "    lincomb = Delta.gamma, control.inla = list(lincomb.derived.correlation.matrix = TRUE), ",  "    control.fixed = list(correlation.matrix = TRUE))")

Time used:
 Pre-processing    Running inla Post-processing           Total
   0.6322844028    0.0006976128    0.0011129379    0.6340949535

Integration Strategy: Model contains  hyperparameters
The model has no fixed effects and no intercept

Likelihood model: gaussian

The model has no random effects

I imagine there is a problem with the temporary folders that it creates, but I cannot figure out why it works for the first thousands of genes and then it stops. I have considered chopping the dataset into smaller chunks and analysing them separately but

(a) That would be a very poor fix

(b) The index of the gene at which it stops working changes a lot from time to time.

Any explanations/help would be greatly appreciated.

I am working on a Linux machine:

> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-pc-linux-gnu (64-bit)

Best regards.
Maria Rodrigo-Domingo

[INLA help]

Hi

its hard to tell what is the problem, but if you send me stuff so I can
rerun it here, I would check it. there isn't anything in what you sent
which can be used to locate the problem; the seemingly random directory
name is because it is random.

I guess you're using the latest testing version otherwise, if you can
try that one it would be great.

Best
H

> --
> You received this message because you are subscribed to the Google
> Groups "R-inla discussion group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to r-inla-discussion...@googlegroups.com.
> To post to this group, send an email to
> r-inla-disc...@googlegroups.com.
> Visit this group at
> http://groups.google.com/group/r-inla-discussion-group.
> For more options, visit https://groups.google.com/groups/opt_out.


--
Håvard Rue
he...@r-inla.org

[Maria Rodrigo]

Dear Håvard,

Thank you very much for your reply. I have been making some tests and I believe the problem is the memory usage. I think this might explain why it works fine in Windows but not in Linux. 

I can see (using top) that the memory used keeps growing and growing, and that it is not released to the operating system even when I delete all the variables, the only solution is to re-start R and I'm not too happy about that...  Would you know of any way to limit the memory usage or release the memory back to the operating system?

I have been able to reproduce the error using your Epil dataset thousands of times, I attach the code. In my 32Gb server, it stops after about 16000 iterations.

I am using the latest (stable) INLA version, I don't assume using the testing version would help with this, right? The output of sessionInfo() is the same as last time.

> inla.version()

        INLA build date .........: Sun May 26 16:17:26 CEST 2013

        INLA hgid ...............: hgid: cc6741ca2667  date: Sun May 26 15:07:01 2013 +0100

        INLA-program hgid .......: hgid: cc6741ca2667  date: Sun May 26 15:07:01 2013 +0100

        Maintainers .............: Havard Rue <hr...@math.ntnu.no>

                                 : Finn Lindgren <finn.l...@gmail.com>

                                 : Daniel Simpson <dp.si...@gmail.com>

                                 : Andrea Riebler <andrea....@math.ntnu.no>

        Web-page ................: http://www.r-inla.org

        Email support ...........: he...@r-inla.org

                                 : r-inla-disc...@googlegroups.com

        Source-code .............: http://inla.googlecode.com

Thank you very much for your help.

Best regards,

Maria

[Finn Lindgren]

On 16 Sep 2013, at 21:42, Maria Rodrigo <mro...@gmail.com> wrote:
> Thank you very much for your reply. I have been making some tests and I believe the problem is the memory usage. I think this might explain why it works fine in Windows but not in Linux.
>
> I can see (using top) that the memory used keeps growing and growing, and that it is not released to the operating system even when I delete all the variables, the only solution is to re-start R and I'm not too happy about that...

That indicates that the memory handling of R itself is to blame. I know R 3.0.0 had memory issues, but I thought they fixed that for R 3.0.1 (they may have had more issues than they fixed...)

Finn

[INLA help]

On Mon, 2013-09-16 at 22:42 +0200, Maria Rodrigo wrote:
> Dear Håvard,
>
>
> Thank you very much for your reply. I have been making some tests and
> I believe the problem is the memory usage. I think this might explain
> why it works fine in Windows but not in Linux.
>
>
> I can see (using top) that the memory used keeps growing and growing,
> and that it is not released to the operating system even when I delete
> all the variables, the only solution is to re-start R and I'm not too
> happy about that... Would you know of any way to limit the memory
> usage or release the memory back to the operating system?
>
>
> I have been able to reproduce the error using your Epil dataset
> thousands of times, I attach the code. In my 32Gb server, it stops
> after about 16000 iterations.
>
>
> I am using the latest (stable) INLA version, I don't assume using the
> testing version would help with this, right? The output of
> sessionInfo() is the same as last time.

Hi

there is a new stable version out, but this would not fix this issue.

using R as you do in the example, you'll need to store all the results
in the 'apply' call, which will just use more and more mem. you can
follow the progress also with

gc()

which will also 'clean up'.

do you really need to store all the 'result' or can you just extract
from them what you need and return only that?


Best
H




--
Håvard Rue
he...@r-inla.org