Confusing map output of INLA when predicting large dataset split in chunks

[Chengbi Liu]

Hi,

I have been using INLA to do spatial binomial modelling and prediction over small spatial entities and everything was fine. However, recently I tried on a big dataset with ~20,000 sample points, 30 explanatory variables and ~3.8 million grids (India at 1km2 resolution) for prediction and the modelling crashed due to memory issue (I think, the error reads "Segmentation fault (core dumped)"). I then tried to keep the same mesh at national level but split the prediction grids into 8 chunks, each one consisting 500,000 grids, did prediction one at a time, and mosaic the predicted results in the end. It worked fine but the output map is rather interesting. As you can see in the map, there are roughly 8 horizontal lines/areas which mark high predicted values across India, which I can only attribute to the fact that I did prediction 8 times separately and mosaic them together. 

My question is: Is this expected behavior when doing prediction over split data chunks? I should note that because I understand INLA does modelling and prediction the same time, each time I did prediction I effectively rebuilt the inla stack and model, only changing the "A.pred" matrix from the mesh based on the new 500,000 grids in the prediction stack. Should I change the way to proceed?

Thanks so much in advance. I'm happy to provide any detail in data/code if necessary.

Best,

Chengbi

[Finn Lindgren]

Hi,

That looks like strange results if you ran the full domain of the model each time, and only altered the prediction part. But one can see horizontal streaks across the whole domain; it’s just more visible at the high-level places.

If it’s really a numerical issue, please try the new inla.mode=“experimental” setting for inla(), that handles this better than the original method.

But also check what happens if you split the domain differently (e.g. vertically) as well for comparison to rule out possible indexing issues in the interpretation/mosaic in GOF the output!

Finn

On 1 Sep 2021, at 22:39, Chengbi Liu <chengbi...@gmail.com> wrote:

Hi,

--
You received this message because you are subscribed to the Google Groups "R-inla discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to r-inla-discussion...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/r-inla-discussion-group/4d06933c-3154-4222-9e6b-3dcacef291f2n%40googlegroups.com.

<Inla map.png>

[Helpdesk]

On Wed, 2021-09-01 at 22:55 +0100, Finn Lindgren wrote:
> If it’s really a numerical issue, please try the new
> inla.mode=“experimental” setting for inla(), that handles this better
> than the original method.
>

then you need R-4.1 and the most recent testing version. Linux or Mac
will help and you'll have to enable PARDISO; see inla.pardiso()

--
Håvard Rue
he...@r-inla.org

[Chengbi Liu]

Thanks for your prompt response. I'll try the experimental mode and let you know how it goes. 

[Chengbi Liu]

Hi, I've upgraded the INLA version and installed pardiso successfully. However, when I tried to set the inla.mode parameter there was an error message saying "unused argument", please see my attached screenshot. Could you pls help? Thank you so much!

Best,

Chengbi

[INLA help]

I guess you run R-4.0.   Can you upgrade to R-4.1 as the new testing version is only for 4.1 

—

Haavard Rue

HelpDesk 

help@r-inla. org

--
You received this message because you are subscribed to the Google Groups "R-inla discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to r-inla-discussion...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/r-inla-discussion-group/5c5dcb6c-91a5-4192-8793-ba8410a004fcn%40googlegroups.com.

[Chengbi Liu]

Thanks, but I am running R 4.1.0. Maybe 4.1.1 is required?