reading data from CSV file and displaying real time plot

[Alex Strickland]

Hi there,

I'm pretty new to python (am an embedded C programmer by trade), but I have a need to be able to plot data being written to a CSV file and show the plot in pseudo-real time in pyqtgraph.

I have built the GUI with the plots and am happy with it plotting static data and rolling round (it reads data from a CSV file once using pandas) but I'm having trouble getting pandas to work for real time data that is being written to the CSV file by another process once a second.

I am wondering if pandas are the right way to go for this? Once a second my CSV file will be written to, it has a variable data rate of either 1,16,32 or 128Hz and each sample (on a new line) contains a timestamp and three vectors (it's telemetry data from a magnetometer).

 

So depending on the data rate I need to read each CSV file once a second and pull out either 1, 16,32 or 128 lines and then put the vectors into a list and then update the plot moving the data along at the data rate speed.

I wonder if there are better solutions than pandas for this? As in my update function I am forced to re-read the CSV file and pull the whole file out each time and tail the necessary lines into a copy then using the tolist() method

But as the file grows this is probably fighting a losing battle to be efficient and could even slow down the drawing of the plots?

Any advice appreciated! Thanks, Alex

[Grant R]

I agree that pandas is probably not the right approach here, especially as the file grows. However, probably worth doing a quick test at seeing how fast a large file is and maybe it is fine. Probably the best is to go to the end of the file and then read in reverse, something like https://stackoverflow.com/questions/10933838/how-to-read-a-csv-file-in-reverse-order-in-python seems like would be a good solution. I'm sure that is very fast as well. Since it's a csv, you can use the built-in csv reader or pass the resulting buffer into pandas and do it that way.

[Robert J. Budzyński]

Here's something you may find useful: 
https://github.com/pbs/python-ditto/blob/master/tail.py

--- 

RJB

On Fri, Feb 8, 2019 at 11:21 AM Alex Strickland <danced...@gmail.com> wrote:

Hi there,

I'm pretty new to python (am an embedded C programmer by trade), but I have a need to be able to plot data being written to a CSV file and show the plot in pseudo-real time in pyqtgraph.

I have built the GUI with the plots and am happy with it plotting static data and rolling round (it reads data from a CSV file once using pandas) but I'm having trouble getting pandas to work for real time data that is being written to the CSV file by another process once a second.

I am wondering if pandas are the right way to go for this? Once a second my CSV file will be written to, it has a variable data rate of either 1,16,32 or 128Hz and each sample (on a new line) contains a timestamp and three vectors (it's telemetry data from a magnetometer).

 

--

RJ Budzyński

Ośrodek Komputerowy Wydziału Fizyki UW

[Alex Strickland]

Thanks for the responses.

I have it working ok with pandas, using 'skiprows' and only reading out the number of rows I need via 'nrows'

This seems to help as the file gets bigger, but I need to test it over the course of a few hours and see what effect this has on performance.