Sparse Matrix specification

[Abhishek Joshi]

I need to use sparse matrix for page rank calculations. I have the following questions to ask

1) Can I make a 1600000*1600000 matrix which is actually sparse?

2) What is the correct format of the MatrixMarket file , is the following correct?

%%MatrixMarket matrix coordinate real general(Do I actually need these?)

% rows columns non-zero-values

4 4 2

% row column value

1 2 1.0

2 3 1.0

3) What needs to be used for generating a sparse matrix from the above file, should it be CCS matrix class or the CRS matrix class?

I would be grateful if you could provide me a running code for generating a sparse matrix from MatrixMarket File.

Thanks

[Vladimir Kostyukov]

Hi Abhishek,

I will try to help you.

1) Yes you can. The la4j has limit for matrix size: (Integer.MAX_VALUE - 1) by (Integer.MAX_VALUE - 1). So, you can create you 1600000x1600000 sparse matrix. But, there in an issue in la4j-0.4.5 that doesn't allow you to creat such matrix, due to integer overflow. It was fixed a week ago by this issue. So, I would suggest you to update your library instalation manually.

There is still another issue with big sparse matrices that is almost fixed now by this PR. The patch author is currently in the middle of improving the fix. It should be merged with mainstream in couple of days. So, I don't know whether it's possible to wait for this or not. Please, let me know if you need the fixed version of the la4j right now. I will patch the library with my own fix and send the binaries to you. Anyway, I'm still planing to wait for the original fix from author who actually found the issue. This is fair.

2) Your version of the MM file is corrent. You only need the first line - it's a MatrixMarket specified header. So, you can remove the rest comments:

%%MatrixMarket matrix coordinate real general

4 4 2

1 2 1.0

2 3 1.0

3) It can be either CRSMatrix or CCSMatrix. By default (using org.la4j.io.MatrixMarketStream) CRSMatrix will be created as sparse instance. But you can specify what kind of matrix do you want. Like this (using MatrixSource provider):

Matrix a = new CCSMatrix(Matrices.asMatrixMarketSource(new FileInputStream("matrix.mm")));

Anyway, I would recomend you to use CRSMatrix. It is a good choise for most of the cases.

--
Thanks,

Vladimir Kostyukov

[Abhishek Joshi]

Thanks for a quick reply. I have some more issues while I run the code
it gives me illegalStateException

Exception in thread "main" java.lang.IllegalStateException: java.io.IOException
at org.la4j.matrix.source.StreamMatrixSource.rows(StreamMatrixSource.java:66)
at org.la4j.matrix.sparse.CCSMatrix.<init>(CCSMatrix.java:69)
at test.main(test.java:13)
Caused by: java.io.IOException
at org.la4j.io.MatrixMarketStream.ensureNext(MatrixMarketStream.java:327)
at org.la4j.io.MatrixMarketStream.parseMatrix(MatrixMarketStream.java:260)
at org.la4j.io.MatrixMarketStream.readMatrix(MatrixMarketStream.java:131)
at org.la4j.io.MatrixMarketStream.readMatrix(MatrixMarketStream.java:139)
at org.la4j.matrix.source.StreamMatrixSource.<init>(StreamMatrixSource.java:36)
at org.la4j.matrix.Matrices.asMatrixMarketSource(Matrices.java:1257)
... 1 more

The trace looks something like the above with the MatrixMarket
specification I don't know what is going wrong. Can you look into this
I am attaching the file test.java that is trying to generate the
sparse matrix, matrix.mm file is the same as I have mentioned in the
earlier mail.

Once again thanks a lot.

[Vladimir Kostyukov]

Right. I've check it. Please, remove all the unnecessary comments (leave only the header). It should looks like following:

%%MatrixMarket matrix coordinate real general

4 4 2

1 2 1.0

2 3 1.0

Looks like there is an issue in MatrixMarketStream. I've filed a ticket about it. 

--
Thanks,

Vladimir Kostyukov