Plotting many (thousands of) lines with pyqtgraph

[louie D]

Hi there,

I'm trying to plot a series of particle paths using pyqtgraph as it's performing much better than the packages I've used before so thank you for all the work that has gone into it so far. 

Unfortunately I want to plot many many lines and have noticed that as I add more lines to the plot there is an exponential performance decrease that brings the program to its knees before 100,000 lines (although it can cope with 10,000). I'm relatively new to python but I'm assuming this is due to some sort of append/pushback function requiring a complete data copy every time I add a new line.

currently I'm creating a plot & window and then using:

    curve = pg.PlotCurveItem(x,y)

    plt.addItem(curve)

for each different line I want to add to the plot

My question is, is there anyway to add a set of lines in one go (without looping over .additem or .plot)? So that I have to call pyqtgraph less often and hopefully reducing the effect of this slow down?

I may have misunderstood the cause for the performance issues and if that's the case, any advice that would allow me to plot around 100,000 lines (with around 100 points each) would be much appreciated, I don't need to do any re-scaling or pull any data from the graph, just plot it out to look at so things can be very static.

I realise the scope of plot is quite large so if its something you do not think is reasonable with pyqtgraph I understand.

Thanks,

[Luke Campagnola]

On Tue, Nov 12, 2013 at 9:48 AM, louie D <dloui...@gmail.com> wrote:

Hi there,

I'm trying to plot a series of particle paths using pyqtgraph as it's performing much better than the packages I've used before so thank you for all the work that has gone into it so far. 

Unfortunately I want to plot many many lines and have noticed that as I add more lines to the plot there is an exponential performance decrease that brings the program to its knees before 100,000 lines (although it can cope with 10,000). I'm relatively new to python but I'm assuming this is due to some sort of append/pushback function requiring a complete data copy every time I add a new line.

Please clarify: do you want to improve performance in the initial plotting, or in interactive pan/zooming, or both? Also: are your lines largely overlapping or non-overlapping? Both of these can affect the best strategy for improving performance. It would be most helpful if you can provide a simple example benchmark so we can be sure we are trying to optimize the same thing.

 

currently I'm creating a plot & window and then using:

    curve = pg.PlotCurveItem(x,y)

    plt.addItem(curve)

for each different line I want to add to the plot

My first question is: have you tried disabling auto-range in the plot? This can significantly reduce the amount of time taken by plt.addItem(). 

My question is, is there anyway to add a set of lines in one go (without looping over .additem or .plot)? So that I have to call pyqtgraph less often and hopefully reducing the effect of this slow down?

One way is to put all lines into a single item. Here's an example:

import pyqtgraph as pg

import numpy as np

plt = pg.plot()

lines = 10000

points = 100

x = np.empty((lines, points))

x[:] = np.arange(points)

y = np.random.normal(size=(lines, points))

connect = np.ones((lines, points), dtype=np.ubyte)

connect[:,-1] = 0  #  disconnect segment between lines

path = pg.arrayToQPath(x.reshape(lines*points), y.reshape(lines*points), connect.reshape(lines*points))

item = pg.QtGui.QGraphicsPathItem(path)

item.setPen(pg.mkPen('w'))

plt.addItem(item)

This works reasonably well with 1,000 lines, but gets slow around 10,000. At 100,000, my machine starts swapping and eventually Qt crashes after a memory allocation issue. Most of the memory usage appears immediately after the item is added to the plot, so I suspect this memory is being used directly by Qt.. 

I may have misunderstood the cause for the performance issues and if that's the case, any advice that would allow me to plot around 100,000 lines (with around 100 points each) would be much appreciated, I don't need to do any re-scaling or pull any data from the graph, just plot it out to look at so things can be very static.

My suspicion is that 100k lines * 100 samples will be out of reach for pyqtgraph (at least until vispy is integrated), but I am open to seeing if we can get there. A better bet is to try out some of the OpenGL visualization libraries (galry, visvis, glumpy). 

Luke

 

[Jochen Schröder]

Hi,

if you want to continue using pyqtgraph, you can use my 2d opengl
plotting additions to pyqtgraph. I can easily plot 100k+ spots+lines.
The github directory is here:

https://github.com/cycomanic/pyqtgraph/

use the glcamera_attribute branch.

Cheers
Jochen

> --
> -- [ You are subscribed to pyqt...@googlegroups.com. To unsubscribe,
> send email to pyqtgraph+...@googlegroups.com ]
> ---
> You received this message because you are subscribed to the Google
> Groups "pyqtgraph" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pyqtgraph+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[louie D]

Firstly thank you both for your responses, they have both been very useful.

Luke:

I think the easiest way to show you what I'm trying to do is to upload a picture. This is a run with 1000 lines. 

I don't need to be able to interact with the graph at all (no pan/zoom or axis scaling). I just want to plot and show all the lines with some sort of reasonable update rate (around every few seconds) and then save the final image to a file(which I can already do). As you can see the lines largely overlap so could we optimize with removing covered lines or doing a image save/reload? 

I would upload a benchmark however its quite a long model to generate the lines so its probably not worth picking through just for the plotting.

Currently it just runs with this initialization:

if plot_on==1 or plot_on==2:

   pg.setConfigOption('background', 'w')

   pg.setConfigOption('line','k')

   win=pg.GraphicsWindow()

   plt=win.addPlot()

   plt.enableAutoRange(False)

   plt.setYRange(-1000,0.00001)

   plt.setXRange(-600,600)

and then interatively calls:

     curve = pg.PlotCurveItem(xdat,ydat,pen='k')

     plt.addItem(curve)

as new paths are completed.

where xdat and ydat are two lists of data that vary from 10 to 1000 points long but normally around 100.

Thanks for the info on how to load lines in bulk, I received similar results on memory issues with 100,000 on my machine which I guess may limit any further attempts to get this working if QT can't hold the data.

Finally on OpenGL options, thank you for suggesting them, I'll have a look at them now and see if one fits, I'm guessing pyqtgraph might not be suited due to my lack of need to interact with the data, just plot an image of it.

Jochen:

That sounds great, can I install this alongside normal PyQtGraph or do I need to uninstall/reinstall? (Sorry if this is obvious, I'm fairly new to python modules)

Also would you mind posting the code you use to generate the plots you mentioned? Do I have to change anything in my pyqtgraph code to use the OpenGL version or just install it and go?

Thank you so much for your help!

[Luke Campagnola]

On Wed, Nov 13, 2013 at 2:24 PM, louie D <dloui...@gmail.com> wrote:

I don't need to be able to interact with the graph at all (no pan/zoom or axis scaling). I just want to plot and show all the lines with some sort of reasonable update rate (around every few seconds) and then save the final image to a file(which I can already do). As you can see the lines largely overlap so could we optimize with removing covered lines or doing a image save/reload? 

With high overlap and no interactivity, probably stacking many lines into a single QGraphicsPathItem is the best option. I did another test, and it turns out that although you can't efficiently get 10M points into a single path item, it is possible to do 100 paths with 100k points each. It's slow at about 10 seconds per update, but it works:

import pyqtgraph as pg

import numpy as np

plt = pg.plot()

lines = 100000

pointsPerLine = 100

linesPerItem = 1000

numItems = lines/linesPerItem

pointsPerItem = linesPerItem * pointsPerLine

x = np.empty((linesPerItem, pointsPerLine))

x[:] = np.arange(pointsPerLine)

x = x.reshape(pointsPerItem)

connect = np.ones(pointsPerItem, dtype=np.ubyte)

connect[pointsPerLine-1::pointsPerLine] = 0  #  disconnect segment between lines

y = np.random.normal(size=(lines*pointsPerLine))

for i in range(numItems):    

    path = pg.arrayToQPath(x, y[pointsPerItem*i:pointsPerItem*(i+1)], connect)

    item = pg.QtGui.QGraphicsPathItem(path)

    item.setPen(pg.mkPen('w'))

    plt.addItem(item)

 I would also agree that you can probably achieve nearly identical results with much less hassle if you can work out a way to cull some of the highly-overlapping lines in the center. 

Luke

[mallamma Reddy]

Hi there,

please i request you to have some patience to listen my  problem statement and I am new on this kind of projects and also i also  hope not be asking dump questions.i have 8 sensors in my project and that data i stored in Excel and it will be updated for 1ms and updated data i am taking it in text file (read-write ) and then i am plotting same updated text(Aw1.txt) in graph.i have created  PYQT5 GUI Application(AW.txt) and then i wrote code for reading and writing the data in python.please can anyone tell me how to plot the graph in qt designer and how to all 8 sensor data .i dont no how to select the check box and how to click the plot and pause button for plotting the graph..please can anyone help me.

Thanks in Advance 

mallu 

 

Thanks,