Large SparseArray in Mathematica 9 and the predictive interface

jmm...@gmail.com

unread,

Dec 30, 2012, 8:49:26 PM12/30/12

to

Hi there,

I was generating a banded sparse array with the following code
in the new Mathematica 9.0:

gmat[K_, l_, m_] :=
Module[{bandU, bandL, bandD, res},

bandU = Table[l, {K - 1}];
bandL = Table[m, {K - 1}];
bandD = Table[-l - m, {K}];
bandD[[1]] = -l;
bandD[[K]] = -m;

res = DiagonalMatrix[SparseArray[bandU], 1]
+ DiagonalMatrix[SparseArray[bandD]]
+ DiagonalMatrix[SparseArray[bandL], -1];

Return[res];
];

then I used the function with the following parameters (without semicolon):

gmat[100, 0.8, 1]

...and everything was ok! a 100x100 sparse matrix was created and the
"predictive interface bar" appeared.

Nevertheless, when I typed (without semicolon):

gmat[10000, 0.8, 1]

...everything went not so good, a 10000x10000 sparse matrix was created
(what was ok), but the kernel went nuts with the memory use, some gigabytes
for a small sparsearray (You can try with greater parameters, but starting
with K=5000 things go weird)

I think the problem has to do with the "predictive interface" that tries to
generate the normal form of the matrix, so it uses a lot of memory (and even
produces a kernel crash).

Has anyone had the same problem with SparseArray? or have any
explanation to what actually happens? or it is just me?

Many thanks in advance for any help or advice with this issue.

Regards,

Jose M. Martinez
Ph.D. Student
Department of Electronic Engineering
Universidad Tecnica Federico Santa Maria
Valparaiso, Chile

Bill Rowe

unread,

Dec 31, 2012, 7:45:07 PM12/31/12

to

On 12/30/12 at 8:50 PM, jmm...@gmail.com wrote:

>I was generating a banded sparse array with the following code in
>the new Mathematica 9.0:

>gmat[K_, l_, m_] :=
>Module[{bandU, bandL, bandD, res},

>bandU = Table[l, {K - 1}]; bandL = Table[m, {K - 1}]; bandD = Table[-l - m, {K}]; bandD[[1]] = -l; bandD[[K]] = -m;

>res = DiagonalMatrix[SparseArray[bandU], 1] +
>DiagonalMatrix[SparseArray[bandD]] +
>DiagonalMatrix[SparseArray[bandL], -1];

>Return[res];
>];

>when I typed (without semicolon):

>gmat[10000, 0.8, 1]

>...everything went not so good, a 10000x10000 sparse matrix was
>created (what was ok), but the kernel went nuts with the memory use,
>some gigabytes for a small sparsearray (You can try with greater
>parameters, but starting with K=5000 things go weird)

>I think the problem has to do with the "predictive interface" that
>tries to generate the normal form of the matrix, so it uses a lot of
>memory (and even produces a kernel crash).

A few comments:

First, I cannot reproduce your results. That using your code
slightly cleaned up, I don't see any problems. By slightly
cleaned up I mean replacing K (which as a built-in meaning) with
k and eliminating the local variable res which isn't needed to get:

gmat[k_, l_, m_] := Module[{bandU, bandL, bandD},
bandU = Table[l, {k - 1}];
bandL = Table[m, {k - 1}];
bandD = Table[-l - m, {k}];
bandD[[1]] = -l;
bandD[[k]] = -m;

DiagonalMatrix[SparseArray[bandU], 1] +
DiagonalMatrix[SparseArray[bandD]] +

DiagonalMatrix[SparseArray[bandL], -1]]

And this code can be further simplified to:

gMat[k_, l_, m_] :=
SparseArray[{Band[{1, 1}] -> -m - l, Band[{1, 2}] -> l,
Band[{2, 1}] -> m}, {k, k}] +
SparseArray[{{1, 1} -> m, {k, k} -> l}]

But I do see a difference between version 8 and 9 with respect
to the amount of memory used. That is:

In[1]:= gMat[k_, l_, m_] :=
SparseArray[{Band[{1, 1}] -> -m - l, Band[{1, 2}] -> l,
Band[{2, 1}] -> m}, {k, k}] +
SparseArray[{{1, 1} -> m, {k, k} -> l}]

In[2]:= a = MemoryInUse[];
gMat[10000, .8, 1]; MemoryInUse[] - a

Out[3]= 1003920

In[4]:= $Version

Out[4]= 9.0 for Mac OS X x86 (64-bit) (November 20, 2012)

and

In[1]:= gMat[k_, l_, m_] :=
SparseArray[{Band[{1, 1}] -> -m - l, Band[{1, 2}] -> l,
Band[{2, 1}] -> m}, {k, k}] +
SparseArray[{{1, 1} -> m, {k, k} -> l}]

In[2]:= a = MemoryInUse[];
gMat[10000, .8, 1]; MemoryInUse[] - a

Out[3]= 493024

In[4]:= $Version

Out[4]= 8.0 for Mac OS X x86 (64-bit) (October 5, 2011)

In version 9 on my system, the array requires a bit more than
twice the memory that version 8 requires. So, there may be
something to your speculation about the predictive interface.

One other thing. I should point out I have 16GB of RAM installed
which may be why I am not seeing problems with your code.

Bob Hanlon

unread,

Dec 31, 2012, 7:45:18 PM12/31/12

to

Turn off the predictive interface by unchecking: Mathematica |
Preferences... | Show Suggestions Bar after last input.

Also, your Module does not need an explicit Return nor a definition
for the local variable res.

gmat[K_,l_,m_]:=Module[
{bandU,bandL,bandD},
bandU=Table[l,{K-1}];
bandL=Table[m,{K-1}];
bandD=Table[-l-m,{K}];
bandD[[1]]=-l;
bandD[[K]]=-m;
DiagonalMatrix[SparseArray[bandU],1]+
DiagonalMatrix[SparseArray[bandD]]+
DiagonalMatrix[SparseArray[bandL],-1]];

g1=gmat[100,0.8,1]

SparseArray[<298>,{100,100}]

Unitize[g1 // Flatten] // Total

298

g2=gmat[10000,0.8,1]

SparseArray[<29998>,{10000,10000}]

Unitize[g2 // Flatten] // Total

29998

Bob Hanlon

On Sun, Dec 30, 2012 at 8:50 PM, <jmm...@gmail.com> wrote:
> Hi there,

>
> I was generating a banded sparse array with the following code
> in the new Mathematica 9.0:
>
> gmat[K_, l_, m_] :=
> Module[{bandU, bandL, bandD, res},
>
> bandU = Table[l, {K - 1}];
> bandL = Table[m, {K - 1}];
> bandD = Table[-l - m, {K}];
> bandD[[1]] = -l;
> bandD[[K]] = -m;
>
> res = DiagonalMatrix[SparseArray[bandU], 1]
> + DiagonalMatrix[SparseArray[bandD]]
> + DiagonalMatrix[SparseArray[bandL], -1];
>
> Return[res];
> ];
>
>

> then I used the function with the following parameters (without semicolon):
>
>
> gmat[100, 0.8, 1]
>
>
> ...and everything was ok! a 100x100 sparse matrix was created and the
> "predictive interface bar" appeared.
>

> Nevertheless, when I typed (without semicolon):

>
>
> gmat[10000, 0.8, 1]
>
>
> ...everything went not so good, a 10000x10000 sparse matrix was created
> (what was ok), but the kernel went nuts with the memory use, some gigabytes
> for a small sparsearray (You can try with greater parameters, but starting
> with K=5000 things go weird)
>
> I think the problem has to do with the "predictive interface" that tries to
> generate the normal form of the matrix, so it uses a lot of memory (and even
> produces a kernel crash).
>

David Reiss

unread,

Dec 31, 2012, 7:45:34 PM12/31/12

to

of course you should report this as a bug to Wolfram Support....
--David

On Dec 30, 8:49 pm, jmm...@gmail.com wrote:
> Hi there,
>
> I was generating a banded sparse array with the following code
> in the new Mathematica 9.0:
>
> gmat[K_, l_, m_] :=
> Module[{bandU, bandL, bandD, res},
>
> bandU = Table[l, {K - 1}];
> bandL = Table[m, {K - 1}];
> bandD = Table[-l - m, {K}];
> bandD[[1]] = -l;
> bandD[[K]] = -m;
>
> res = DiagonalMatrix[SparseArray[bandU], 1]
> + DiagonalMatrix[SparseArray[bandD]]
> + DiagonalMatrix[SparseArray[bandL], -1];
>
> Return[res];
> ];
>

> then I used the function with the following parameters (without semicolon:

jmm...@gmail.com

unread,

Jan 2, 2013, 9:14:10 PM1/2/13

to

Hi again,

You could even try the following (which is not a big sparse array):

DiagonalMatrix[SparseArray@Table[1.0, {100000}]]

...(without the semicolon and the "Suggestion bar" activated) and
the kernel will crash.
But, if you deactivate the "Suggestion Bar" everything goes smoothly.

I use to monitor the processes and memory by using the Sysinternals suite,
and in particular, the "process explorer" that comes with it.

My machine is a 4Gb Windows 7 Pro notebook with an Intel i5.

many thanks for the several answers given,
regards,

Jose M. Martinez
Ph.D. Student
Department of Electronic Engineering
Universidad Tecnica Federico Santa Maria
Valparaiso, Chile