Importing large file into table and calculating takes a long time. How to improve efficiency?

[Gangerolv]

(First my disclaimer, I'm new to mathematica)

I'm importing a file with three values, x,y,z in hexadecimal.
Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc", 27, 98}, {0., 29, 95},...

This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, 153}, {220, 39, 152}, {0., 41, 149},...

A final vector is calculated using: g==sqrt(x^2 + y^2 + z^2) for each data set.

The data is then plotted.

All of this is good an works great for a small file (2000 data sets). But when I try to import and calculate a larger file (over 100k data sets), it seems to take forever. Either my methods are not efficient (use of Table), or I'm not using correct settings for importing of the data. The file is only 2Mb so I know mathematica should be able to handle it.

Here's what I'm doing:

==============================================================
dataHex ==
Import["C:\\Projects\\Mathematica\\test.csv"]
points == Length[dataHex]
dataDec == Table[
ToExpression["16^^" <> #] & /@ {
ToString[dataHex[[i, 1]]],
ToString[dataHex[[i, 2]]],
ToString[dataHex[[i, 3]]]},
{i, 1, points}]
dataDecCompl == Table[{
If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]] - 65536, dataDec[[i, 1]]],
If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 2]]],
If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 3]]]},
{i, 1, points}];

dataG == MovingAverage[Table[
Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N,
{i, 1, points - filter}],
3];
plotG == ListLinePlot[{dataG}, PlotRange -> All];
===========================================================================

Any suggestions would be highly appreciated. Using M 8.0.4 on Win7.

Cheers!

jan

[Šerých Jakub]

I haven't studied your code, but definitely the first error I can see is, that normal asignment is "=" not "==" in Mathematica.


Jakub

> -----Original Message-----
> From: Gangerolv [mailto:ips...@yahoo.com]
> Sent: Tuesday, May 01, 2012 11:24 AM
> To: math...@smc.vnet.net
> Subject: Importing large file into table and calculating takes a
> long time. How to improve efficiency?
>
> (First my disclaimer, I'm new to mathematica)
>
> I'm importing a file with three values, x,y,z in hexadecimal.
> Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc",
> 27, 98}, {0., 29, 95},...
>
> This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, 153}, {220,
> 39, 152}, {0., 41, 149},...
>
> A final vector is calculated using: g==sqrt(x^2 + y^2 + z^2) for each data set.
>
> The data is then plotted.
>
> All of this is good an works great for a small file (2000 data sets). But when I
> try to import and calculate a larger file (over 100k data sets), it seems to take
> forever. Either my methods are not efficient (use of Table), or I'm not using
> correct settings for importing of the data. The file is only 2Mb so I know
> mathematica should be able to handle it.
>
> Here's what I'm doing:
>

> =========================================================================

> dataHex ==
> Import["C:\\Projects\\Mathematica\\test.csv"]
> points == Length[dataHex]
> dataDec == Table[
> ToExpression["16^^" <> #] & /@ {
> ToString[dataHex[[i, 1]]],
> ToString[dataHex[[i, 2]]],
> ToString[dataHex[[i, 3]]]},
> {i, 1, points}]
> dataDecCompl == Table[{
> If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]] - 65536, dataDec[[i, 1]]],
> If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 2]]],
> If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 3]]]},
> {i, 1, points}];
>
> dataG == MovingAverage[Table[
> Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N,
> {i, 1, points - filter}],
> 3];
> plotG == ListLinePlot[{dataG}, PlotRange -> All];
> =========================================================================
>

[Bob Hanlon]

data = RandomInteger[100000, {10, 3}];

dataHex = IntegerString[data, 16];

dataDec = Map[FromDigits[#, 16] &, dataHex, {2}];

dataDecCompl = Map[If[# > 32768, # - 65536, #] &, dataDec, {2}];

You did not define dataX, dataY, dataZ, and filter. The following
assumes that dataX, dataY, and dataZ are the components of elements of
dataDecCompl and that filter =0. If not, modify as required.

Since

Norm[{x, y, z}]

Sqrt[Abs[x]^2 + Abs[y]^2 +
Abs[z]^2]

then

dataG = MovingAverage[Norm /@ dataDecCompl // N, 3];

plotG = ListLinePlot[dataG, PlotRange -> All]


Bob Hanlon