IO-Performance of Mathematica 4.1

Marko Kastens

unread,

Nov 20, 2003, 3:21:31 AM11/20/03

to

Hi!

I'am searching for a good (fast) way to import big datasets in Mathematica 4.1.

What I've done so far:
- The read-in from the _orignal_ (ASCII)file needs 97 sec. (a 5xxxx x 2
matrix)
- Exporting my matrix using Export[] with "List" or "Table" needs
(nearly) infinity of time :-(
- Exporting the matrix with the HDF (Hierarchy Data Format): 7 seconds
:-) Great, but...
- Importing the matrix with HDF: fast but wrong: the HDF seems to
supports only Real64. My first column is a 10-digit-Integer representing
the MM-time-format. By converting this to yyExx the last digits get lost
and my time is doing funny jumps...

- using the Experimental-Functions like this:

<< Experimental`
t0 = AbsoluteTime[];
strm = OpenWrite["tmp0.bin", DOSTextFormat -> False];
BinaryExport[strm, Length[data], "Integer32"]
For[i = 1, i < Length[data] + 1,
{
BinaryExport[strm, data[[i]], {"Integer64", "Real64", "Integer64"}]
}; i++]
Close[strm];
Print[AbsoluteTime[] - t0];

it takes 415 seconds to export and 247 seconds to import. Even worser
than importing the original.

Only the good old ASCII-way is better:

t0 = AbsoluteTime[];
strm = OpenWrite["tmp1.dat"];
Write[strm, Length[data]]
For[i = 1, i < Length[data] + 1,
{
Write[strm, data[[i, 1]]],
Write[strm, data[[i, 2]]],
Write[strm, data[[i, 3]]]
}; i++]
Close[strm];
Print[AbsoluteTime[] - t0];

It takes 32 seconds to export and 16 seconds to import.

Hmm, normaly binary-import/export is faster than ascii-import/export.
Does anybody know a proper and fast method? Thanks a lot.

marko

Marko Kastens

unread,

Nov 21, 2003, 5:21:47 AM11/21/03

to

Frank Iannarilli

unread,

Nov 21, 2003, 5:49:18 AM11/21/03

to

Marko Kastens <Kas...@Hamburg.baw.de> wrote in message news:<bphtib$1lu$1...@smc.vnet.net>...

>
> Hmm, normaly binary-import/export is faster than ascii-import/export.
> Does anybody know a proper and fast method? Thanks a lot.

Marko, check out:
http://library.wolfram.com/infocenter/MathSource/4870/
and
http://library.wolfram.com/infocenter/MathSource/4918/

Benedetto Bongiorno

unread,

Nov 24, 2003, 12:20:18 AM11/24/03

to

I too have speed problems with importing large data sets. After many years
of experimenting with various methods, I have concluded:
1 - Import & Export commands work best on *.csv 2 dimensional matrix files.
2 - Mathematica is not a speed demon on I/O functions.

Hi!

Hmm, normaly binary-import/export is faster than ascii-import/export.
Does anybody know a proper and fast method? Thanks a lot.

marko

Christos Argyropoulos M.D.

unread,

Nov 27, 2003, 11:45:58 AM11/27/03

to

Hello,
Ran into the same problem with really big datasets (Iam using Mathematica to
analyze measurements from "DNA-chips", which tend to be huge files).
One can speed mathematica's I/O by Importing numerical data as "Table" and
setting ConversiongOptions->{DataBlockErrorChecking" ->False}
Christos Argyropoulos MD
Patras Greece

----- Original Message -----
From: "Marko Kastens" <Kas...@Hamburg.baw.de>
Subject: IO-Performance of Mathematica 4.1

Christos Argyropoulos M.D.

unread,

Nov 27, 2003, 11:52:04 AM11/27/03

to

Hello,
Ran into the same problem with really big datasets (Iam using Mathematica to
analyze measurements from "DNA-chips", which tend to be huge files).
One can speed mathematica's I/O by Importing numerical data as "Table" and
setting ConversiongOptions->{DataBlockErrorChecking" ->False}
Christos Argyropoulos MD
Patras Greece

----- Original Message -----
From: "Marko Kastens" <Kas...@Hamburg.baw.de>
Subject: IO-Performance of Mathematica 4.1