INLA breaking at very large dataset sizes

[Imantha Gunasekera]

Hi, 
I am running INLA on a very large dataset. It contains 632 regions over time span of 10 years. The issue I believe is the number of records which is quite large, ~7.5 million. I have been testing on a portion of this dataset (1 year - ~700k records) but the model still fails. To get the model working, I had to reduce this test dataset even further, size of around ~100k. This was done by aggregating the data from daily to weekly level. At this dataset size the model works but it isn't ideal to have the data at weekly level (ideally require daily). 

I am writing to figure out two things
- Is the dataset size the reason why INLA is failing, and if so why ? (A quick check from GPT mentioned that INLA creates a large number of latent variables but wanted to confirm whether thats correct)
- The other question is, is there a possibility for INLA handle such large datasets and if so how ?

The compute I have is : 256GB RAM, 16 CPU cores @ 2.90 GHZ 

Code for running model on 700k records

```
formula_bym2 <- hospitalisation_cum_lead7 ~ age_band3 + sex + offset(log(population_interp)) +
  f(region_idx, model = "bym2", graph = g, scale.model = TRUE) #offset needs to be added

# Fit INLA
res <- INLA::inla(
  formula_bym2,
  family = "poisson",
  data = df_concise,
  control.predictor = list(compute = TRUE),
  verbose = TRUE
)

# Dataframe summary
> str(df_concise)
tibble [695,796 x 11] (S3: tbl_df/tbl/data.frame)
 $ publicholiday            : Factor w/ 2 levels "0","1": 1 2 1 1 1 1 1 1 1 1 ...
 $ dow                      : Factor w/ 3 levels "Weekday","Saturday",..: 3 1 1 1 1 1 2 3 1 1 ...
 $ date                     : Date[1:695796], format: "2023-10-01" "2023-10-02" "2023-10-03" "2023-10-04" ...
 $ hw_mean_ehf_severity     : num [1:695796] 0 0 0 0 0 0 0 0 0 0 ...
 $ age_band3                : Ord.factor w/ 3 levels "0-17"<"18-64"<..: 1 1 1 1 1 1 1 1 1 1 ...
 $ sex                      : Factor w/ 2 levels "male","female": 1 1 1 1 1 1 1 1 1 1 ...
 $ sa2_2021_code            : chr [1:695796] "101021007" "101021007" "101021007" "101021007" ...
 $ population_interp        : num [1:695796] 507 507 507 507 507 507 507 507 507 507 ...
 $ hospitalisation_cum_lead7: int [1:695796] 0 0 0 0 0 0 0 0 0 1 ...
 $ region_idx               : int [1:695796] 1 1 1 1 1 1 1 1 1 1 ...
 $ time_idx                 : int [1:695796] 1 2 3 4 5 6 7 8 9 10 ...

# Error
 *** inla.core.safe:  The inla program failed, but will rerun in case better initial values may help. try=1/1


INLA ERROR CAUGHT:
<simpleError in inla.core.safe(formula = formula, family = family, contrasts = contrasts,     data = data, quantiles = quantiles, E = E, offset = offset,     scale = scale, weights = weights, Ntrials = Ntrials, strata = strata,     lp.scale = lp.scale, link.covariates = link.covariates, verbose = verbose,     lincomb = lincomb, selection = selection, control.compute = control.compute,     control.predictor = control.predictor, control.family = control.family,     control.inla = control.inla, control.fixed = control.fixed,     control.mode = control.mode, control.expert = control.expert,     control.hazard = control.hazard, control.lincomb = control.lincomb,     control.update = control.update, control.lp.scale = control.lp.scale,     control.pardiso = control.pardiso, control.stiles = control.stiles,     control.taucs = control.taucs, control.numa = control.numa,     only.hyperparam = only.hyperparam, inla.call = inla.call,     inla.arg = inla.arg, num.threads = num.threads, keep = keep,     working.directory = working.directory, silent = silent, inla.mode = inla.mode,     safe = FALSE, debug = debug, .parent.frame = .parent.frame): The inla-program exited with an error. Unless you interupted it yourself, please rerun with verbose=TRUE and check the output carefully.
  If this does not help, please contact the developers at <he...@r-inla.org>.
The inla program failed and the maximum number of tries has been reached.>
```