Improving Performance of pyqtgraph real-time plotting

[Paul Gross]

Hi, I am working on displaying real-time telemetry data using pyqtgraph. I am quite pleased with the visual results however I am having issues with the frame rate dropping as more data is plotted. I am receiving about 100 data points per second. At the beginning plotting is quite fast but the frame right dives rapidly as more data is being displayed.

I have fixed the size of the plots and auto-ranging is disabled, I have tried down sampling and it helps, but not quite enough. I am looking for the fastest way to plot a large amount of data points in real-time, as I receive them.

I have found other posts about speed and some of them reference ‘arrayToQPath’, but I am not sure if that is the best way to approach my issue. My biggest concern is with plotting the data points as quickly as possible as to not slow down the rest of the event loop. The current way that I am plotting data in the ‘PlotWidget’ is setting the data of a ‘PlotDataItem’ in the following manner, where the arguments are either a numpy array or python list: ‘plot_data_item.curve.setData(data['time'][self.x_min:], data[data_type][self.x_min:])’. I have tried plotting a new curve each time, but that seemed to be quite slow.  In an ideal world I would just be able to append points without having to connect all of the previous points to each other.  At some point I want to reset the plot to an empty plot, but that really isn’t an issue because this is a rare occurrence in my program. I simply want to display in real time several minutes of data with multiple different plots, each of fixed size with no scrolling, panning, or auto-resizing.

I would not be opposed to threading some of the work out if that is an option on top of any suggestions you have. Something I am not entirely clear on is if the GIL is an issue when it comes to QThreads since they are C++? Will I get any performance boost by using QThreads in the update function? I am reading from a data queue in the update function, so ideally that would be taken out of the loop.  That however isn’t my main concern because if I reset the plots (change the range of data that is taken from my data list/array the frame right jumps right back up).

I am using the Dev Branch because my application requires PYQt5 and python 3.5.

Any suggestions you have would be greatly appreciated!

[vas...@gmail.com]

My way is to calculate data in one thread, and everything about pyqtgraph (and qt) to process in other one.

--
You received this message because you are subscribed to the Google Groups "pyqtgraph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyqtgraph+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/4e08f9bc-4380-4d08-9148-755463f2e976%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vas...@gmail.com]

# If you look at the today code I reply to tukom (https://groups.google.com/d/msg/pyqtgraph/nkPWgbHEwdE/dLzwCdFhAQAJ), for a bit improved 'threaded' code could be:

import numpy as np
import pyqtgraph as pg
import threading

class FuncThread(threading.Thread):
    def __init__(self,t,*a):
        self._t=t
        self._a=a
        threading.Thread.__init__(self)
    def run(self):
        self._t(*self._a)

class MyApp():
    def __init__(self):
        self.number_of_data_steps = 100000
        self.qapp = pg.mkQApp()
        self.win = pg.GraphicsWindow()
        plot1 = self.win.addPlot()
        curve = plot1.plot()
        curve.setPen('r')       
        x = np.linspace(0, self.number_of_data_steps, self.number_of_data_steps)
        self.y = np.linspace(-5, 5, self.number_of_data_steps)
       
        calculating_thread = FuncThread(self.calculate)
        calculating_thread.start()           

        while self.win.isVisible():  
            curve.setData(x,self.y)
            self.qapp.processEvents()

    def calculate(self):
        while self.win.isVisible():  
            self.y += np.random.random(self.number_of_data_steps) - 0.5
                   
if __name__=='__main__': 
    m = MyApp()

[Paul Gross]

Thanks for the responses, sorry I am so late in responding.

 I definitely want to thread out all of the data queue operations and the calculations, but that doesn't really tackle the the problems I am having with having to paint all of the data points every iteration. Ideally I would simply be able to append new points to the graph as quickly as possible.

What would the recommend method of doing this, in real time, be?

Bottom line I need to display a lot of data points (O(100/second) ) without the frame rating dropping drastically as the graph nears its end.

Thanks,

Paul

[Terekhov, Mikhail]

The key point is to separate data acquisition from data visualization. If you design your application
for human consumption, then I guess you will have hard time finding a person who will be able to
distinguish plots updating 100 fps from those updating say 50 fps. Even if you will manage to find
one, then you will need to find out what he would do with such fast changing data.

My advise is - collect data as fast as you need/can and think about how you would transform it
for visualization so that pure humans could analyze and act upon them :)

Regards,
Mikhail

--

You received this message because you are subscribed to the Google Groups "pyqtgraph" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pyqtgraph+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/750f1594-d03f-43a8-be00-89b8ae83dab2%40googlegroups.com.

[Mark]

Paul - 

Pyqtgraph has real performance limitations is you are using anything but the standard plot. And even then, it cannot handle the load you are generating in your use-case.

I have a completely separate application generating and storing data.

Another process then loads it into a pandas dataframe

And then I take the data from Pandas and pass into a numpy array for use within pyqtgraph.

Luke has talked about migrating OPENGL type functions as used in VISPY into pyqtgraph - and until that is done, I don't think pyqtgraph would be suitable for your needs.

I am going to try bokeh in lieu of anything else.

Mark

[Paul Gross]

Mark,

Thanks for the response, I greatly appreciate it.

Unfortunately at this point in time switching to something else would be a huge headache and frankly I don't have the time.

If I had to stick with it, what would you recommend in terms of getting the most performance out of it as a I can.

Here is the exact scenario I am dealing with:

I have two windows: a main window and a second window. Both windows have multiple graphs and tables displaying real-time data (coming from a queue from another process that is doing all the calculations). The main window also has two GLViewWidgets that are displaying GLMeshItems that are being translated (This is not an issue as it is a constant cost per frame and no matter what, they only moves once per frame).

Here are my questions:

1. Would it be better to have each window in its own process?

2. Are there any under the hood ways I can use the QPath's to make appending data faster? I have tried doing some things similar to this http://stackoverflow.com/questions/17103698/plotting-large-arrays-in-pyqtgraph

3. I have noticed an append function in the source for the 'CurveDataItem', but its simply an empty function. Was there/is there a plan for implementing that and if so how would that be done?

Thanks,

Paul

On Tuesday, February 28, 2017 at 1:17:38 PM UTC-6, Paul Gross wrote: