Issue Running inla() with mclapply on Cluster (Newton-Raphson Optimizer Did Not Converge)

[Florine Kempf]

Hi R-INLA group,

I need your help to solve a problem I'm facing.

I'm encountering an issue when running the inla() function in parallel using mclapply (from the parallel package) on a computing cluster. More precisely, I'm using mclapply to run multiple inla() calls in parallel on different datasets. INLA itself also uses internal parallelism, so I reserved half the cores for that purpose. The same inla() model, runned serially (i.e., outside of mclapply), works fine on my local machine but seems to fail on the cluster (even if I execute it serially). However, this issue doesn't occur for all datasets. For some dataframes, the model fails both locally and on the cluster, while others work as expected.

Here are some details:
- Local setup: R version 4.4.3 (2024.12.1) on Windows
- Cluster setup: R version 4.5 (2025.05.0)
- INLA installation command (used both locally and on the cluster): install.packages("INLA", repos = c(getOption("repos"), INLA = "https://inla.r-inla-download.org/R/stable"), dep = TRUE)

I'm fitting a random error spatial joint model with a Poisson likelihood. The model formula is as follows:
formula <- y ~ -1 + Intercept +
  f(sp.1, model = "bym2", scale.model = TRUE, graph = graph) +
  f(sp.2, copy = "sp.idx1", hyper = list(beta = list(fixed = TRUE))) +
  f(s.2, model = "iid")
inla(formula, data = df, family = "poisson",
     control.predictor = list(compute = TRUE), E = E,
     control.compute = list(config = TRUE, waic = TRUE), verbose = T)

When running this on the cluster using mclapply and allocating half of the available cores for INLA, I receive the following error:
GitId: b2abbedea53ff456149dbe37d7be47fdd0cfdf9f - Wed Dec 11 22:31:59 2024 +0300
Error:12 Reason: The Newton-Raphson optimizer did not converge
Message: Condition ‘lambda < 1.0 / lambda_lim’ is not TRUE
Line:1026 Function: GMRFLib_init_GMRF_approximation_store__intern

I get the same error for certain datasets even when running locally. I’ve attached one such dataframe that reproduces the issue both on the cluster and on my machine.

Has anyone experienced similar behavior when using INLA with mclapply on a cluster? Could this be related to characteristics of specific datasets, or a version/environment mismatch? Any suggestions for debugging or resolving this would be greatly appreciated.

I could provide supplementary code and datasets if needed.

Thanks in advance!

Best regards,
Florine

[Helpdesk (Haavard Rue)]

can you provide an rda that contains what's needed to rerun ?

> --
> You received this message because you are subscribed to the Google Groups "R-
> inla discussion group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to r-inla-discussion...@googlegroups.com.
> To view this discussion, visit
> https://groups.google.com/d/msgid/r-inla-discussion-group/c1dd9524-f159-4844-924a-29141d28c020n%40googlegroups.com
> .

--
Håvard Rue
he...@r-inla.org

[Florine Kempf]

I realized I made a mistake in the model formula I previously shared — the response variable is not y, but O ...

I’ve attached an .rda file that contains everything needed to rerun the model. 

The following code also needs to be executed beforehand to properly set up the graph and formula:

nb2INLA(here("data", "graph"), W.nc.nb)
graph <- here("data", "graph")

formula <- O ~ -1 + Intercept +
    f(sp.idx1, model = "bym2", scale.model = T, graph = nc.adj) +
    f(sp.idx2, copy = "sp.idx1", hyper = list(beta = list(fixed = T))) +

    f(s.2, model = "iid")
inla(formula, data = df, family = "poisson",

     control.predictor = list(compute = T), E = E,
     control.compute = list(config = T, waic = T), verbose = T)

Thanks !

Florine

[Helpdesk (Haavard Rue)]

Hi,

there is no issues with the current example on my side. the package develop all
the time, so the default answer is always to upgrade.

I would suggest you try the following

- upgrade to R-4.5 everywhere. we only build for 4.5 hence you'll be stuck in
old versions without this

- use the testing version (just use 'testing' instead of 'stable'

install.packages("INLA", repos = c(getOption("repos"), INLA =

"https://inla.r-inla-download.org/R/testing"), dep = TRUE)

- with such small data, there is no need to use mclapply. you'll just make it
slower. or, you need to set number of threads the inla call to be small-ish,
like

inla.setOption(num.threads="4:1")

and then you could try to mclapply beyond that

- the time of each run, is basically to compute the prior used in 'bym2', as the
inla-program runs in 0.15 seconds on my laptop.

- on the side-note, I do not understand your formula, well, this part

f(sp.idx1, model = "bym2", graph = graph) +

f(sp.idx2, copy = "sp.idx1", hyper = list(beta = list(fixed = T))) +

as idx1 is c(1:n,rep(NA,n)), and idx2 is c(rep(NA,n),1:n). with these values
(and 'beta=1', fixed), then you might say the same using

f(sp.idx1, model = "bym2", graph = graph) +

and set sp.idx1 = c(1:n, 1:n) . it can be that this is just a snapshot of a
larger setup, but I just wanted to say



Best wishes
Havard

[Florine Kempf]

Thanks again for your response — it helped me a lot!

The model now runs correctly on my local R installation, but I'm still encountering issues on the cluster. I updated my local setup, so both my local environment and the cluster are now using the same R version: R 4.5. The RStudio version is slightly different: 2025.05.1 locally and 2025.05.0 on the cluster.

I really want to get this working on the cluster, as I’m conducting a large simulation study. I'm not just running the model on a single dataset — I’m running it on over 30,000 varied datasets, so using mclapply() for parallelization is crucial to speed up computation by fitting several models at once.

Please let me know if there’s anything else I should try to resolve the cluster-side issue — I’m happy to test suggestions or provide additional details.

Best regards,
Florine


PS : To clarify the structure of my model: I want to compare two spatial distributions, so I’ve structured the model with sp.idx1 and a copy of it as sp.idx2 to explicitly separate the components, even though they share the same underlying structure.

[Helpdesk (Haavard Rue)]

On Thu, 2025-06-12 at 05:17 -0700, Florine Kempf wrote:
> Thanks again for your response — it helped me a lot!
>
> The model now runs correctly on my local R installation, but I'm still
> encountering issues on the cluster. I updated my local setup, so both my local
> environment and the cluster are now using the same R version: R 4.5. The
> RStudio version is slightly different: 2025.05.1 locally and 2025.05.0 on the
> cluster.

as long as you have R-4.5 you're good

>
> I really want to get this working on the cluster, as I’m conducting a large
> simulation study. I'm not just running the model on a single dataset — I’m
> running it on over 30,000 varied datasets, so using mclapply() for
> parallelization is crucial to speed up computation by fitting several models
> at once.

R is pretty bad with parallel computations, and I always do this from the
outside, doing parallel calls to R instead. it all depends on whats your
knowledge in linux as you have to loop on the outside, not inside R. it is not
hard to do, but if you havn't done this before then you might seek some help of
some with more linux-skills.

I attach a small example on how you could pass arguments from the shell into R,
and then you can do one-simulation example, and write the output. the '&' will
do this job in the background,

so

./do-job

will run all 3^3 jobs in parallel

[Helpdesk (Haavard Rue)]

If you're doing this trick, you'll be better off running each job in serial,
just add

inla(..., num.threads="1:1")