Speed issue -- is it possible to add a single X, Y point to an existing graph without plotting the entire data set each time?

[CSPlusC]

Hi all, first, my thanks to our host for some good software--I found pyqtgraph yesterday while looking for something more lightweight than matplotlib for basic graphing, and I've been impressed with the examples and my first few attempts at using the code.

I'm using it to do real-time graphing of sine, triangle, and square waves for a GUI project.  I added a PlotWidget object through QT Designer, and have had no problem using said PlotWidget in my code.

The graphing of waves requires me to take samples every 20 milliseconds, and I need to graph the wave by time.  So, I'm adding 50 X, Y points a second to a pair of Python lists, and passing those to the PlotWidget using the plot() method like so:

# xPoints, yPoints are both standard Python lists, and the plotWidget is a pyqtgraph PlotWidget object
self.plotWidget.plot(x=XPoints, y=rollYPoints)

The problem I'm running into is that those two lists--rollXPoints, rollYPoints--are growing rapidly, and even if I only call the plot() function a few times a second, it will inevitably reach a point where drawing all the points bogs down the code so much that the GUI gets unresponsive and other commands get excessively delayed.

Passing the entire set of points every time I call plot() seems inefficient, is there a way that I can simply update the existing graph with a single X, Y pair at a time?  Or, barring that is there a way to speed up the plotting while still maintaining all the data points?  I've dug through the example code and much of the documentation without finding a way, and would greatly appreciate any input.

[Luke Campagnola]

On Tue, May 8, 2012 at 5:34 PM, CSPlusC <alexand...@gmail.com> wrote:

The graphing of waves requires me to take samples every 20 milliseconds, and I need to graph the wave by time.  So, I'm adding 50 X, Y points a second to a pair of Python lists, and passing those to the PlotWidget using the plot() method like so:

# xPoints, yPoints are both standard Python lists, and the plotWidget is a pyqtgraph PlotWidget object
self.plotWidget.plot(x=XPoints, y=rollYPoints)

The problem I'm running into is that those two lists--rollXPoints, rollYPoints--are growing rapidly, and even if I only call the plot() function a few times a second, it will inevitably reach a point where drawing all the points bogs down the code so much that the GUI gets unresponsive and other commands get excessively delayed.

Passing the entire set of points every time I call plot() seems inefficient, is there a way that I can simply update the existing graph with a single X, Y pair at a time?  Or, barring that is there a way to speed up the plotting while still maintaining all the data points?  I've dug through the example code and much of the documentation without finding a way, and would greatly appreciate any input.

I've thought about automating a feature like this, but it's not written in yet. 

Here are my recommendations:

1. It sounds like (correct me if I misunderstood) you are appending values onto the end of XPoints and YPoints, then replotting the whole set. Since the PlotWidget keeps track of everything that has previously been plotted, it is actually replotting the same data exponentially as you make calls to plot(). To fix this, you can either call self.plotWidget.clear() to remove the previous plot or you can change the plot call to look like self.plotWidget.plot(x=XPoints, y=rollYPoints, clear=True). This should help a lot.

2. If you need more speed yet (or if you're already correctly clearing the plot), then I'd recommend just plotting the last 50 segments every time you make an update:

  self.plotWidget.plot(x=XPoints[:-51], y=rollYPoints[:-51])

This will create a large number of independent PlotDataItems, but it turns out that Qt's GraphicsView is usually very good at efficiently handling this type of situation.

Let me know if this doesn't work out..

Luke

  

 

[CSPlusC]

Mich obliged for the fast response.  I didn't realize that the way I was plotting was exponentially increasing the number of dots.  I tried out both solutions, and found a dramatic increase in speed from #1:  by setting the clear=True flag in plot(), the GUI handled a long test run without problems.

Solution #2 intuitively makes sense, though surprisingly ran slower.  I didn't have the same exponential drop in speed from fast-to-dead in about 40 seconds, but it definitely slowed down noticeably over time.

Perhaps my implementation is off though, I will keep on it.  In the meantime it looks like this will work, thanks again!

[Fabio Varesano]

Hello guys, I'm reopening this since I'm having exactly the same problem as CSPlusC.. huge dataset updated frequently.. I also have the addition of being within a Qt application which I don't think it helps much.. Basically after 1 or 2 minutes everything becomes unresponsive.. I tried the suggested workaround above but, even if they provided improvements, they are not enough.

I think I'd like to just do addPoints or something like that without causing the whole graph to be redrawn.. any suggestion?

[Luke Campagnola]

Hi Fabio,

On Mon, Sep 17, 2012 at 12:26 PM, Fabio Varesano <fabio.v...@gmail.com> wrote:

Hello guys, I'm reopening this since I'm having exactly the same problem as CSPlusC.. huge dataset updated frequently.. I also have the addition of being within a Qt application which I don't think it helps much..

Doing plotting as part of a larger application is not necessarily bad--this will only be a problem if the rest of the application is using CPU time independently of the plot.

 

Basically after 1 or 2 minutes everything becomes unresponsive.. I tried the suggested workaround above but, even if they provided improvements, they are not enough.

We need to know exactly why it is slowing down before deciding how to fix it. 

- Are you displaying points, or just lines? (points slow things down quite a lot)

- Can you tell me roughly how many data points you are plotting when it starts to slow down? 

- If you create a single plot with that many points, do you get roughly the same speed when panning/zooming the plot?

import pyqtgraph as pg

import numpy as np

pg.plot(np.random.normal(size=number_of_points))

I think I'd like to just do addPoints or something like that without causing the whole graph to be redrawn.. any suggestion?

For very large data sets, the best approach is to partition the data into chunks and plot them separately. This way, when you update the very end of a plot, it is not necessary to redraw the entire previously-generated curve. (but note that if your view range is changing, then _all_ items will be redrawn)

Another approach is to downsample the data before plotting it (PlotItem actually has some built-in downsampling capabilities, but the API around that feature is not quite complete)

[Fabio Varesano]

Hi Luke, thanks for your suggestions. I tried to set a range to my graphs so that they would have a predefined range, so that a redraw wouldn't be necessary.
Unfortunately I'm still having performance issues... I have my code on http://bazaar.launchpad.net/~fabio-varesano/freeimu/trunk/files/head:/libraries/FreeIMU/calibration/freeimu_cal_gui/
Maybe you can have a look at it and see if you see any obvious performance issue?

Thanks

[Luke Campagnola]

On Wednesday, September 19, 2012 3:53:54 AM UTC-4, Fabio Varesano wrote:

Hi Luke, thanks for your suggestions. I tried to set a range to my graphs so that they would have a predefined range, so that a redraw wouldn't be necessary.
Unfortunately I'm still having performance issues... I have my code on http://bazaar.launchpad.net/~fabio-varesano/freeimu/trunk/files/head:/libraries/FreeIMU/calibration/freeimu_cal_gui/
Maybe you can have a look at it and see if you see any obvious performance issue?

I see a few places in this code that affect performance. 

1) The data coming out of SerialWorker.run() grows over time because buff is not reset within the while-loop at line 111. Perhaps this was intentional, but a consequence is that self.acc_data grows exponentially over time (well, parabolically).

2) The plot code creates a new plot object with every iteration and the amount of data in each new plot grows with each iteration, so the amount of redundant plotted data is growing exponentially over time (even faster than acc_data). The easiest solution to this is to add clear=True as an argument to the plot() calls to avoid plotting redundant data:

    self.accXY.plot(x = self.acc_data[0][:-25], y = self.acc_data[1][:-25], clear=True)
    self.accYZ.plot(x = self.acc_data[1][:-25], y = self.acc_data[2][:-25], clear=True)
    self.accZX.plot(x = self.acc_data[2][:-25], y = self.acc_data[0][:-25], clear=True)

In this case, performance doesn't degrade much until the number of samples gets to around 30k or so (as measured by my aging computer). Also, since the entire data set is being replotted at every iteration, there is no need to set a fixed range on the plot.

3) I'm not sure what you were trying to accomplish by indexing acc_data with [:-25] when plotting. This plots all data except the last 25 samples..  I guess you might have intended [-25:] to plot only the last 25 samples, but even this would have caused data to be plotted redundantly, since the data set is expanded by only 5 samples at every iteration. So it would be possible to get the same results (but faster) by changing the index to [-6:] and removing the clear=True argument I suggested in 2). 

Note that in this case, there is very little performance benefit to breaking up the plot into segments because the lines are largely overlapping. This means that for every update, all line segments are likely to be redrawn anyway. However if you draw points instead of lines, each new point will be mostly non-overlapping with previous points so there is some performance benefit in that case (although this is still much slower than drawing lines).

Luke

 

[Fabio Varesano]

Right. I guess I should have blamed by sloppy code instead of your nice library! :-)
Btw, I implemented your suggested changes, made some other optimizations and now the code works flawlessly.

Thanks for your help.

Fabio

[Su-Shin Ang]

Hi,

Is is possible for you to give me a quick example of how to use the plot method after you have promoted the GraphicsView widget in QT designer?

At the moment, I am only able to get it to work in Ui_MainWindow method.

Any help will be appreciated.

Thanks

Ed

[Nicolas M]

Hello !

I have the same problem, i have to plot a huge amount of data. 

The biggest case is to plot 2 curves every 2/5 ms during 15 s. So as you can know this is a little bit slow.

But as if that was not enough, I have to write at the same frequency the same data in a text file.

Here is my Update function :

  def update(self,my_values):

        value = self.real_nbr(my_values)

        self.append_to_log_file(value)

        for i in range (self.curve_number):

            if len(self.data[i]) < int(self.max_scope):

                self.data[i].append(int(value[i]))

            else:

                del self.data[i][0]

                self.data[i].append(int(value[i]))

            self.xdata =  np.array(self.data[i], dtype='int')

            #if self.cpt_update == 40:

            self.curve[i].setData(self.xdata)

I read your response :

For very large data sets, the best approach is to partition the data into chunks and plot them separately. This way, when you update the very end of a plot, it is not necessary to redraw the entire previously-generated curve. (but note that if your view range is changing, then _all_ items will be redrawn)

i guess it's the best choice to me, could you tell me more about it please ?

Best regards

[Nicolas M]

Hi 

I wrote some test to know what's slow in my code,

 I think this is the curves number who is the problem, i mean the way i display them. 

Test 2 : 4 Curves : frequencies: 20ms , X = 3000, some lags 

Test 3 : 2 curves : frequencies: 10 ms, X = 3000 : No problem

Test 4 : 3 curves frequencies: 10 ms, X = 3000 : some lags 

Test 5 : 2 curves frequencies: 5 ms , X = 3000 : No problem

Test 6 : 3 curves frequencies: 5 ms, X = 3000 : some lags

Test 7 : 2 curves frequencies: 1ms, X = 3000 : No problem

So if i have the same result with 2 curves at 1ms and 10 ms, and if i try to draw another curve at 10ms, it become slow, i guess the problem is how i handle my curve  

[Luke Campagnola]

Can you post a working code example to demonstrate what you are seeing? Otherwise it will be very difficult to determine how to improve performance.

[Nicolas M]

Here is my complet code 

to call update and get my value i use Qthread and signal 

Thx for your time !

#!/usr/bin/python

# -*- coding: utf-8 -*-

#import initExample

import PySide

import os.path

import imp

imp.reload(PySide)

toto = PySide

from pyqtgraph.Qt import QtGui, QtCore, USE_PYSIDE

import serial

from Lib.Processing import Processing

from Lib.Processing import Conf

import numpy as np

import pyqtgraph as pg

from pyqtgraph.ptime import time

from Thread_manager import *

from Lib.Access import Access

from Thread_manager import ComMonitorThread

from datetime import datetime

if USE_PYSIDE:

    from ui_TEST_WINDOWS_PYSIDE import Ui_Form

    print ("Using Pyside")

else:

    print ("Using nothing, let's crash !")

    pass

class Graphic_plot():

    def __init__(self,value_to_display,value_to_log):

        self.initialize_var(value_to_display,value_to_log)

    def initialize_var(self,value_to_display,value_to_log):

        self.connection = Access()

        c_processing_json_file = processing_db_access_conf()

        data_json = c_processing_json_file.open_json_file_read()

        com_port =data_json["com"]["port_com"]

        self.one_conf_path = data_json["conf_ventil"]["out"]

        self.data = []

        self.data_4 = [[],[],[],[]]

        self.data_3 = [[],[],[]]

        self.data_2 = [[],[]]

        self.data_1 = [[]]

        self.curve =  []

        self.cpt_update = 0

        self.can_be_clean = False

        self.value_to_log_ = value_to_log

        self.value_to_display_ = None

        self.value_to_display_ = value_to_display

        self.count_value_to_display = 0

        self.curve_number = 0

        self.max_scope = 0

        self.translated_value = []

        self.xdata = np.array([])

        self.connection_var = self.connection.Connect(com_port,"serial")

        self.run_graph()

    def enumerate_colors(self,number_of_curves):

        """Limited to 5 curves 1 : Red ,2 : Green, 3 : White"""

        my_colors = np.array(['r','g','w','y'])

        return my_colors[:number_of_curves]

    def run_graph(self):

        pw = pg.PlotWidget()

        pw.show()

        pw.setWindowTitle('Db_Access Graphic')

        p = pw.plotItem

        p.setLabels(left='axis 1')

        self.l = pg.LegendItem((100,60), offset=(70,30))

        self.l.setParentItem(p.graphicsItem())

        p.showGrid(True,True,251)

        p.showButtons()

        

        c_processing_json_file = processing_db_access_conf()

        data_json = c_processing_json_file.open_json_file_read()

        self.max_scope = data_json["graph_parameters"]["axis_scope_max"]

        min_scope = data_json["graph_parameters"]["axis_scope_min"]

        

        p.setRange(rect=None, xRange=[int(min_scope),int(self.max_scope)], yRange=[1300,-300], padding=0.09, update=True, disableAutoRange=True)

        

        self.curve_number = len(self.value_to_display_)/2

        self.curve_number = int(self.curve_number)

        

        if self.curve_number == 1:

            self.data = self.data_1

        elif self.curve_number == 2:

            self.data = self.data_2

        elif self.curve_number == 3:

            self.data = self.data_3

        elif self.curve_number == 4:

            self.data = self.data_4

        colors_to_use = self.enumerate_colors(self.curve_number)

        legend_item_to_display = self.get_translated_text()

        for i in colors_to_use:

            self.curve.append(p.plot(pen= i))

        if len(legend_item_to_display) == 1:

            self.l.addItem(p.plot(pen = colors_to_use[0]),legend_item_to_display[0])

        elif len(legend_item_to_display) == 2:

            self.l.addItem(p.plot(pen = colors_to_use[0]),legend_item_to_display[0])

            self.l.addItem(p.plot(pen = colors_to_use[1]),legend_item_to_display[1])

        elif len(legend_item_to_display) == 3:

            self.l.addItem(p.plot(pen = colors_to_use[0]),legend_item_to_display[0])

            self.l.addItem(p.plot(pen = colors_to_use[1]),legend_item_to_display[1])

            self.l.addItem(p.plot(pen = colors_to_use[2]),legend_item_to_display[2])

        elif len(legend_item_to_display) == 4:

            self.l.addItem(p.plot(pen = colors_to_use[0]),legend_item_to_display[0])

            self.l.addItem(p.plot(pen = colors_to_use[1]),legend_item_to_display[1])

            self.l.addItem(p.plot(pen = colors_to_use[2]),legend_item_to_display[2])

            self.l.addItem(p.plot(pen = colors_to_use[3]),legend_item_to_display[3])

        self.connection_var.ask_streaming_start(self.value_to_display_,self.value_to_log_,False)

    def real_nbr(self,nbr):

        my_list = []

        for item in nbr:

            if item > 32767:

                my_list.append(item-65536)

            else :

                my_list.append(item)

        return my_list

        

    def grouped(self, iterable, n):

        "s -> (s0,s1,s2,...sn-1), (sn,sn+1,sn+2,...s2n-1), (s2n,s2n+1,s2n+2,...s3n-1), ..."

        return zip(*[iter(iterable)]*n)

    def get_translated_text(self):

        text_node = []

        text_value = self.grouped(self.value_to_display_,2)

        c_process = Processing(self.one_conf_path)

        for item in text_value:

            self.translated_value.append(c_process.get_text_node(item))

        for item in self.translated_value:

            text_node.append(item[2:])

        return text_node

    def append_to_log_file(self,value_to_log):

        if self.can_be_clean == False:

            self.my_log_file = 'log_file_graph.txt'

            text_node = []

            text_node = self.get_translated_text()

            my_file = open(self.my_log_file,'w')

            self.count_value_to_display= len(self.value_to_log_)/2

            my_file.write("[Time - H-M-S-MS] "+ str(text_node[:int(self.count_value_to_display)]) + '\n')

            self.my_file_2 = open(self.my_log_file,'a')

            self.can_be_clean = True

        self.my_file_2.write( "[" + str(datetime.now().time()) + " ]  : " + str(value_to_log[:int(self.count_value_to_display)]) + '\n')

    def update(self,my_values):

        value = self.real_nbr(my_values)

        if len(self.value_to_log_) > 1:

            self.append_to_log_file(value)

        for i in range (self.curve_number):

            if len(self.data[i]) < int(self.max_scope):

                self.data[i].append(value[i])

            else:

                del self.data[i][0]

                self.data[i].append(int(value[i]))

            self.xdata =  np.array(self.data[i], dtype='int')

            self.curve[i].setData(self.xdata)

if __name__ == '__main__':

    app = QtGui.QApplication([])

    aa = Graphic_plot()