can this tool visualize files of 10 million records per each file generated every 5 minutes

116 views
Skip to first unread message

yesyesd...@gmail.com

unread,
Jul 28, 2016, 3:01:51 AM7/28/16
to Google Visualization API

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.

We need to visualize these records  in a chart or graph or histogram.

Could you please tell me if this can be possible with this tool ?

Daniel LaLiberte

unread,
Jul 28, 2016, 9:53:19 AM7/28/16
to Google Visualization API
There are no limits on the size of the data, other than what the browser supports.  But be aware that it may take a long time to generate a chart with 10 million rows.  And there may still be memory leaks that accumulate over time, with your update every 5 minutes.  Otherwise, no problem. :)

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.
To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-visualization-api/69719e5f-8263-4ca5-a19d-1e6d0ffd9c78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

yesyesd...@gmail.com

unread,
Jul 28, 2016, 10:18:56 AM7/28/16
to Google Visualization API
Hi Daniel,

Thanks for the reply. I have one more small question.

What type of data input is required? Can this tool read from Comma Separated Value CSV files and generate line charts?.

Thanks
shashi.

On Thursday, July 28, 2016 at 7:23:19 PM UTC+5:30, Daniel LaLiberte wrote:
There are no limits on the size of the data, other than what the browser supports.  But be aware that it may take a long time to generate a chart with 10 million rows.  And there may still be memory leaks that accumulate over time, with your update every 5 minutes.  Otherwise, no problem. :)
On Thu, Jul 28, 2016 at 3:01 AM, <yesyesd...@gmail.com> wrote:

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.

We need to visualize these records  in a chart or graph or histogram.

Could you please tell me if this can be possible with this tool ?

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.



--

Daniel LaLiberte

unread,
Jul 28, 2016, 11:54:05 AM7/28/16
to Google Visualization API
There is no support in Google Charts for csv input, but it wouldn't be hard to add your own csv converter.  

The expected input is a DataTable in one of a couple different forms.  See: https://developers.google.com/chart/interactive/docs/datatables_dataviews
Also, the data can be retrieved from servers in a couple different ways:  https://developers.google.com/chart/interactive/docs/queries

On Thu, Jul 28, 2016 at 10:18 AM, <yesyesd...@gmail.com> wrote:
Hi Daniel,

Thanks for the reply. I have one more small question.

What type of data input is required? Can this tool read from Comma Separated Value CSV files and generate line charts?.

Thanks
shashi.

On Thursday, July 28, 2016 at 7:23:19 PM UTC+5:30, Daniel LaLiberte wrote:
There are no limits on the size of the data, other than what the browser supports.  But be aware that it may take a long time to generate a chart with 10 million rows.  And there may still be memory leaks that accumulate over time, with your update every 5 minutes.  Otherwise, no problem. :)
On Thu, Jul 28, 2016 at 3:01 AM, <yesyesd...@gmail.com> wrote:

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.

We need to visualize these records  in a chart or graph or histogram.

Could you please tell me if this can be possible with this tool ?

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-visualization-api/048e89f9-7f1e-45b1-b288-420b7904ac21%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

yesyesd...@gmail.com

unread,
Jul 29, 2016, 2:44:34 AM7/29/16
to Google Visualization API
Hi Daniel,

Thanks for ur reply. Our requirements have slightly changed and i am giving them below :

 

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.If we divide 10 million/5 minutes(300 seconds),  its about 33000 records or data points for a  second.

 

Can we show these 33000 datapoints per second and the graph is flowing from  right  to left going on for 5 minutes and then it keeps on repeating for  next file that comes after 5 minutes  to show this data from another file( 10 million records generated every  5 minutes in each file). We have data stored in CSV comma separated value files.


Thanks



On Thursday, July 28, 2016 at 9:24:05 PM UTC+5:30, Daniel LaLiberte wrote:
There is no support in Google Charts for csv input, but it wouldn't be hard to add your own csv converter.  

The expected input is a DataTable in one of a couple different forms.  See: https://developers.google.com/chart/interactive/docs/datatables_dataviews
Also, the data can be retrieved from servers in a couple different ways:  https://developers.google.com/chart/interactive/docs/queries
On Thu, Jul 28, 2016 at 10:18 AM, <yesyesd...@gmail.com> wrote:
Hi Daniel,

Thanks for the reply. I have one more small question.

What type of data input is required? Can this tool read from Comma Separated Value CSV files and generate line charts?.

Thanks
shashi.

On Thursday, July 28, 2016 at 7:23:19 PM UTC+5:30, Daniel LaLiberte wrote:
There are no limits on the size of the data, other than what the browser supports.  But be aware that it may take a long time to generate a chart with 10 million rows.  And there may still be memory leaks that accumulate over time, with your update every 5 minutes.  Otherwise, no problem. :)
On Thu, Jul 28, 2016 at 3:01 AM, <yesyesd...@gmail.com> wrote:

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.

We need to visualize these records  in a chart or graph or histogram.

Could you please tell me if this can be possible with this tool ?

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

Daniel LaLiberte

unread,
Jul 29, 2016, 6:57:21 AM7/29/16
to Google Visualization API
When you put it that way, drawing 33000 rows per second, it does sound like Google Charts would have a problem keeping up.  It is not currently designed to handle to large amounts of data quickly, but how much data you can show per unit time also depends a lot on what type of chart you are drawing, which options you enable, and how many columns are in each row.     

Also, how much data do you want to be visible at one time.  Given pixel limitations, you won't be able to see more than, say, 1000 points at a time, so another way to look at this is can you redraw 33 times per second, 1000 at a time?  That might be feasible. You'll need to do some experimenting to find out.


On Fri, Jul 29, 2016 at 2:44 AM, <yesyesd...@gmail.com> wrote:
Hi Daniel,

Thanks for ur reply. Our requirements have slightly changed and i am giving them below :

 

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.If we divide 10 million/5 minutes(300 seconds),  its about 33000 records or data points for a  second.

 

Can we show these 33000 datapoints per second and the graph is flowing from  right  to left going on for 5 minutes and then it keeps on repeating for  next file that comes after 5 minutes  to show this data from another file( 10 million records generated every  5 minutes in each file). We have data stored in CSV comma separated value files.


Thanks



On Thursday, July 28, 2016 at 9:24:05 PM UTC+5:30, Daniel LaLiberte wrote:
There is no support in Google Charts for csv input, but it wouldn't be hard to add your own csv converter.  

The expected input is a DataTable in one of a couple different forms.  See: https://developers.google.com/chart/interactive/docs/datatables_dataviews
Also, the data can be retrieved from servers in a couple different ways:  https://developers.google.com/chart/interactive/docs/queries
On Thu, Jul 28, 2016 at 10:18 AM, <yesyesd...@gmail.com> wrote:
Hi Daniel,

Thanks for the reply. I have one more small question.

What type of data input is required? Can this tool read from Comma Separated Value CSV files and generate line charts?.

Thanks
shashi.

On Thursday, July 28, 2016 at 7:23:19 PM UTC+5:30, Daniel LaLiberte wrote:
There are no limits on the size of the data, other than what the browser supports.  But be aware that it may take a long time to generate a chart with 10 million rows.  And there may still be memory leaks that accumulate over time, with your update every 5 minutes.  Otherwise, no problem. :)
On Thu, Jul 28, 2016 at 3:01 AM, <yesyesd...@gmail.com> wrote:

We have 10 million records of data  stored in a Comma Separated CSV file and files like these are generated every 5 minutes.

We need to visualize these records  in a chart or graph or histogram.

Could you please tell me if this can be possible with this tool ?

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-visualization-api/e80d9d42-77a4-405e-93f2-72822c3c637f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

yesyesd...@gmail.com

unread,
Jul 29, 2016, 7:36:23 AM7/29/16
to Google Visualization API
Hi Daniel,

Thanks for your reply. I am planning to use a line chart with timestamp with micro seconds on x-axis and latency on y-axis. I need to show 33000 data points per second. Is this feasible with Google charts. If not how many data points can you show per second. You initially agreed saying that u can process 10 million records in 5 minutes. what about that?

Thanks
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

Daniel LaLiberte

unread,
Jul 29, 2016, 10:11:28 AM7/29/16
to Google Visualization API
I initially said that there is no limit on the size of the data, but there are practical limits, both in how much the browser can handle, within the memory space of your machine, and moreover, how long it will take.   Since you require a certain amount of data to be displayed within a certain time, then your machine speed is also a factor.  

Otherwise, given the variables, I can't really tell you how many data points you can plot with Google Charts in a second.  I think you should do a few experiments to find out how well it will work for you.  It shouldn't take long to write a simple program to try it out.

To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-visualization-api/1bdf10ba-73f2-4839-8952-8d27c650d7d3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

yesyesd...@gmail.com

unread,
Jul 29, 2016, 12:28:50 PM7/29/16
to Google Visualization API
Hi Daniel,

Thanks for ur reply. could you please answer below:
1. Suppose i take average of every 1000 records( we have 10 milion records in 5 minutes) and plot in a chart, that would be 10000 datapoints in 5 minutes, is that feasible with your tool?
Then refresh the chart after every 5 minutes.Is that possible?

2.  Can I show 33 datapoints per second ( 10000  in 5 minutes) and the graph flows from right to left for 5 minutes showing a total of 10000 records? Is this possible with your tool?

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualization-api+unsub...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

Daniel LaLiberte

unread,
Jul 29, 2016, 1:28:23 PM7/29/16
to Google Visualization API
Sampling and averaging is a good way to effectively draw more data in less time, but I can't tell you any more about how well it will perform.  You'll have to do some experimenting to find out how well it works.

To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.



--

--
You received this message because you are subscribed to the Google Groups "Google Visualization API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-visualizati...@googlegroups.com.

To post to this group, send email to google-visua...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-visualization-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-visualization-api/3deeb0cf-2742-4efd-90ea-9ddc8cd032a7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages