What is the best way to create and store a multi-dimensional matrix in Julia?

[yi lu]

In scientific computing, we need multi-dimensional arrays to store time and space grid points for computing and for plotting. So, what is best recommended to use in Julia for such requirement? What I mean by best is fastest, cheapest and so on.

Thanks in advance.

Yi

[Kevin Squire]

Hello Yi,

Your question is rather open-ended.  Can you give an example of the kind of multi-dimensional data you're interested in, and what you want to visualize about it?

Cheers,

   Kevin

[yi lu]

Hi Kevin,

Let me explain it with an example.

For example, one-dimensional heat equation.
u_t-u_xx=f

If I discrete time and space in t_0, t_1,...,t_m, and x_0,x_1,...,x_n respectively. We want to store these points. Then it is a matrix u(t_i, x_j).

What I am concern is how to store such a big matrix. When we solve this equation, it is often the case that we get at first the u(t_0, *)s, and then u(t_1, *)s, and then u(t_2, *), and so on and so forth.

For two or three dimensional heat equation, we want a multi-dimensional array e.g. u(t_i, x_j, y_k). We may sometimes use u(t_i, z_k) instead, in which k = 1,2,...,m*n, that is to say, we use a long array to replace the last two sub-indexes in matrix u(t_i, x_j, y_k).

So what is a good strategy to create and store such "matrices"?

Yi

[Fabian Gans]

Hi Yi,

in this case I would suggest to use the HDF5 package https://github.com/timholy/HDF5.jl. It supports reading and writing julia data file (.jld) if you want to write whole variables. If you want something more portable that you can read from other applications you can use plain HDF5 files. It is possible to read and write slices of arrays to h5 files, so for your heat equation example you would have a 1D array in your julia memory that you iteratively update and write consecutive slices to the h5 file.

Fabian

[yi lu]

I don't quite understand Fabian. For the h5 files part, do you mean I can interact with h5 files in the disk while the program is running?

Yi

[Tim Holy]

On Monday, May 26, 2014 07:22:16 PM yi lu wrote:
> I don't quite understand *Fabian*. For the *h5 files* part, do you mean I

> can interact with h5 files in the disk while the program is running?

Yes, you can. But in your question, you left it ambiguous as to whether you
want to store in memory or store to disk. If you only need to store to memory
(if you have enough RAM for your big arrays), then HDF5 won't be relevant---
just use an Array. Please do read the documentation about multidimensional
arrays, they already do everything you're asking about, including your
u[t_i,z_k] example.

If you do need to store them to disk, then yes, HDF5 is a good choice. See the
documentation:
https://github.com/timholy/HDF5.jl
https://github.com/timholy/HDF5.jl/blob/master/doc/hdf5.md
https://github.com/timholy/HDF5.jl/blob/master/doc/jld.md
and pay particular attention to the parts about incremental writes and memory
mapping.

--Tim

[yi lu]

OK, I understand. I think storing data into memory will be fine for me.

I do read the Multi-dimensional Arrays part, and find  the assignment part not so familiar. This is because I use MATLAB before and I often do some code like
a=[a,b]

which may be very slow in fact(I am not so sure).

Yi

[Tim Holy]

On Monday, May 26, 2014 08:13:21 PM yi lu wrote:
> OK, I understand. I think storing data into memory will be fine for me.
>

> I do read the *Multi-dimensional Arrays *part*, *and find the assignment
> part not so familiar.

Aside from the use of [] rather than (), assignment is almost exactly like in
Matlab. The main practical difference is that in Julia it's very efficient to
write loops and directly address elements as u[xi, yi, ti].

> This is because I use MATLAB before and I often do
> some code like
> a=[a,b]
> which may be very slow in fact(I am not so sure).

Right. While in Matlab there isn't a great alternative, for growing 1d arrays
Julia has push! and append! which are vastly more efficient. For various
(important) reasons, that doesn't work with a multidimensional array. However,
even in 1d the fastest of all is to pre-allocate your storage and address each
element directly, and that also works in 3d.

[yi lu]

Thanks. Your answers are really helpful.

Yi