data array for many series

[manut...@gmail.com]

Hello
Fist, thanks for your amazing work.

I will use for the research in EPFL (solar cell, Graetzel technologie)

In this page (made for the tests):
http://mbtserver.net/uc/testedy

I have something in this form:

var intensity_iv_0=
[
    [
        803683,798669,793705,788681,783683,778664,
        -1998007,-1844589,-1701054,-1564902,-1437480
    ],
    [
        798617,793958,789107,784283,
        -2015058,-1872539,-1734238,-1604843
    ]
];


Then, I have to programme with JS something to produce this :


[-1437480,778664,],
[-1564902,783683,],
[-1604843,,784283],
...

And the need of memory is so big that the web browser can't calculate for too much curves.
This is because the x values are not shared

Is it any way to use directly the data in the initial form?
Thanks

[Robert Konigsberg]

Hi there. Not really, no. Eventually data gets reduced into an array form. What generates the initial array? If it's done on the server side, then perhaps the server can generate a friendlier format?

--
You received this message because you are subscribed to the Google Groups "dygraphs-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dygraphs-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Robert Konigsberg

[Dan Vanderkam]

You could also downsample your data

[manut...@gmail.com]

Hello,
In the final version of the system (that is actually in use in a lab), The "server" is a microcontroller (pic32MX795F512H) with a TCP/IP library from microchip that can hold some little HTML page. To dinamically write the data in the web page, I use C code instead of PHP.
The total memory is 512 kb flash and 128 kb ram!
This is why the web browser is doing the job for calculate the data.
In the futur, I will made a real server with database and PHP that collect the data (if I find the time)

The measurement card:
http://www.mbtserver.net/ucwiki/en/ls/8_cells
and the server:
http://www.mbtserver.net/ucwiki/fr/uc/net
unfortunately, the project description is in french (http://www.mbtserver.net/ucwiki/fr/ls)

A funny project : I think it is the first time that your library run on microcontroller!
Anyway, it work fine with a reasonable quantity of curves.
Best

[manut...@gmail.com]

Thrue, I can downsample the data, but it is important for the user to zoom in the data for analysis.

[manut...@gmail.com]

A way that I tryed is to group the x values that are similar.
For example instead of this:

[1.0,12,],
[1.2.,,21],
[1.3,16,],
[1.9.,,28],
[2.1,17,],
[2.2.,,29]

then this:

[1.0,12,21],
[1.3,16,],
[1.9,17,28],
[2.1,,29]


It is made with the variable "tolerance_x" in my code, but I don't us it beacaus the systeme is used for scientific analysis.
however, I indicate this solution to complete this topic.

[Dan Vanderkam]

It is possible to dynamically load data after the user zooms, see https://groups.google.com/d/msg/dygraphs-users/kgdNdt1A0dk/xnddpFp8lAIJ

This is something I'd eventually like to make simpler by incorporating it into the library.

--

[manut...@gmail.com]

I forgot to say:
Apparently, the quantity of memory for the array in javascript to sort the data is equal to:
size of a curve multiply by the square of the number of curves. (when all the x values are different)
This is why it is easy to crash the web browser when computing.

[manut...@gmail.com]

Thanks for your comment danvk.
The way for a good system will be to use a real server with a database to allows to sort the data with PHP or to dynamically load data as you mention.
Best

[Dan Vanderkam]

Nothing should be O(N_series^2). Are you talking about dygraphs code or your code?

--

[manut...@gmail.com]

I am tolking about my code. This is beacause if you have three curves:
Let say:

[1.0,10],
[1.5.,20],
[2.0,30]
and
[1.1,11],
[1.6.,21],
[2.1,31]
and
[1.2,12],
[1.7.,22],
[2.2,32]

The resulting arraw:

[1.0,10,,],
[1.1,,11,],
[1.2,,,12],
[1.5.,20,,],
[1.6.,,21,],
[1.7.,,,22],
[2.0,30,,],
[2.1,,31,],
[2.2,,,32]

totalise the same number of lines. But we should consider that this:
[1.0,10,,],
should take two time more memory than
[1.0,10],
because null values take space in the RAM.

Then, if for one curve:
[1.0,10],
[1.5.,20],
[2.0,30]
The memory taken is 6 float (3 for the x axis, 3 for the data),
for three curves :
3x n dots(for the x values)+3x n curves (the number of lines)+3x n curves (the number values per lines including the null or white space )

[manut...@gmail.com]

Sorry I wanted to say

(n curves x n dots per curves(for the x values))+(n dots x n curves (the number of lines per curves)x n curves (the number values per lines including the null or white space ))
Then (without comments)
(n curves x n dots per curves)+(n dots x n curves x n curves)
then
(n curves x n dots per curves)+(n dots x (n curves square))

[manut...@gmail.com]

In my application, I hade up 64 curves of 512 dots.(32 solar cells measured with 4 intensity of light)
The total number of dot is then 128x512=64k dots, this an acceptable number.
But the need in the memory is the same than a unique curve of about 128x128x512=8 Million dots.

512 dots per cells is not so hight, then downsample isn't really possible.

This is why, for cloud of dot mad with many series, no way if the number of series is to big.

For some case where the number of series is too big, I plan to use this one:
http://jsgraph.org/ (made by a colleague of me)
which allow to add many series one by one in a graph
But is is actually maybe a little bit less complet.

[Dan Vanderkam]

I'm not sure what you mean by the memory blowup. If you have 64k data points, then you should have 64k data points in memory, unless you're using an insanely inefficient sparse format.

--

[manut...@gmail.com]

Maybe you are right: I should try another way for the data format! Like the URL method (or CSV)
In my exemple, I have something like this:

var data_for_my_graph=
[

[1.0,10,,],
[1.1,,11,],
[1.2,,,12],
[1.5.,20,,],
[1.6.,,21,],
[1.7.,,,22],
[2.0,30,,],
[2.1,,31,],
[2.2,,,32]

];
Then, the data are in a variable in the RAM.
If I use the URL method (or CSV), then, I will not longer have the problem.
But then, my poor microcontroller will not be able to sort the data.
For sure, I will have to program something with PHP to made the bridge between my microcontroller and the web browser.
Thanks