Inla on large datasets

[surf]

Hi, 

I am trying to apply the inla library to a time series datasets to model spatial temporal dynamics. 

It needs about 15 GB memory for the Cmatrix only. Suppose I have enough memory, can the precompiled

Inla library handle this, or the Inla only use 4G of RAM since the FAQ said it compiled based on 32 bit CPU?  Any suggestion will be very welcome. Thank you. 

Jeff

[Finn Lindgren]

INLA can use more than 4G RAM (I've run sessions with >80 GB), but 15 GB
for a single matrix sounds a bit extravagant (and I'm not sure what the
limit on single variables are, or if there are differences across
platforms). What is the size of your model (number of timesteps and
spatial locations), and what kind of Markov structure do you have in
space and time?

Finn

[surf]

Thanks for the responses, Dr. Lindgren.  We have 88 spatial units for monthly data of 40 years (480 steps), 

and currently use the first order adjacency structure. 15G is my guess by following the spatial dynamics 

tutorial codes you have. Thank you,

Jeff

[Finn Lindgren]

On 16/05/13 23:23, surf wrote:
> Thanks for the responses, Dr. Lindgren. We have 88 spatial units for
> monthly data of 40 years (480 steps),
> and currently use the first order adjacency structure. 15G is my guess
> by following the spatial dynamics
> tutorial codes you have. Thank you,

That sound like an extreme overestimate. What spatial model do you use,
and how many nodes are there in that? Which specific tutorial are you
referring to? (If you have a high-resolution spatial model, then you
might be correct, but you shouldn't need that when you only have 88
locations.)

Finn

[surf]

Hi, I was following the 6th example at http://www.math.ntnu.no/inla/r-inla.org/papers/DLM_examples_11112010.R.

Please check the size of st.cmat.  Isn't the size of C will be 88*88*480*480?  If every entry take 8 bytes, the memory footprint 

goes to 13.6G.

Thanks,

Jeff

[surf]

sorry, not the size of C, but the size of st.cmat in that tutorial codes.

[Elias Krainski]

Both C (spatial) and the diagonal
matrix (for time) are sparse.
Sorry because in that codes I don't take
advantage of it... The best way is to use
C = Matrix(diag(d) - nb2mat(nc.nb, style="B"))
because it takes a very less memory size!

[Finn Lindgren]

Yes, that is the issue with that code. Unfortunately, nb2mat() itself
appears not to take advantage of sparse matrix storage, which makes it
unsuitable for large models (strange, since it ), so even the above fix
will only help for small problems.

Jeff, your space-time matrix with only first order neighbours should
have about 88*480*7*3 nonzero elements (if it's a CAR(1) in space and
kronecker with AR(1) for time), which is considerably less than (88*480)^2.

Finn

[Elias Teixeira Krainski]

> Yes, that is the issue with that code. Unfortunately, nb2mat() itself
> appears not to take advantage of sparse matrix storage, which makes it
> unsuitable for large models (strange, since it ), so even the above fix
> will only help for small problems.

So, is better to use
d <- card(nb)
n <- length(d)
C <- Diagonal(n,d) - sparseMatrix(rep(1:n, d), unlist(nb))

[surf]

Yes, Dr. Lindgren, it is a very sparse matrix, if using the current code.

Thank you, Elias, I will give it a try.

Best,
Jeff

[surf]

Hi, I tried these changes, and the memory footprint has been significantly reduced. 

However, it still gave me the following errors (with 4G RAM available). Any thoughts?

Could the pre-compiled binary use more than 4G memory? Thank you.

gsl: init_source.c:46: ERROR: failed to allocate space for block data

Default GSL error handler invoked.

Aborted (core dumped)

Error in inla.inlaprogram.has.crashed() : 

  The inla-program exited with an error; please rerun with verbose=TRUE and check the output carefully.

  If this does help; please contact the developers at he...@r-inla.org.

On Friday, May 17, 2013 6:36:16 AM UTC-5, Elias T. Krainski wrote: