Thoughts on QThreads, mutex, Python threads and multiprocessing module?

[Jaykumar Vaidya]

Hi everyone,

I recently started using pyqt with matplotlib (due to quality work) and wanted to know your suggestions on going to one of the two ways for creating a real time plot widget. I am talking about using threads or using multiprocessing module. 

Previously, I have used python threads for Tkinter GUIs and that worked well but not the best. as using matplotlib is not thread safe and all I could do is separate plot process in different threads but not speed up the plotting for real time in the case of sampling every 10 ms. 

Now thatI am using PyQT, I am wondering whether I should use QThreads, Python Threads or multiprocessing module and wanted to know people's opinions about the same. I am sticking with matplotlib due to it's quality work, unlike pyqtgraph, but I am open to change to any other library which would be as good as matplotlib as well as is thread safe. After all, I just need to have 10 or so figures running paralley, with other parts of the GUI, with also communicating changes in data amongst them. 

Thanks in advance!

[Martin Chase]

Heya,

There isn't any performance difference between QThreads and python Threads; that's mostly a difference of how you want to integrate with Qt (QThreads fit into the rest of the Qt lifecycle better). Otherwise, the question that I ask to determine thread v. multiprocess is: will my worker mostly be sitting around waiting? If that's the case (e.g. waiting on i/o), then using a thread is great. Otherwise, if the worker needs to do a bunch of heavy work before the data is ready to display (e.g. it's the output of a difficult analysis), put that work on a different core.

Only matplotlib is as good at plotting as matplotlib; it's the best python plotting library. If that's your minimum bar, I agree that you'll be writing everything yourself from scratch. PyQtGraph is built for performance, though, so you're going to need the best of both libraries if that's your use-case.

Good luck,

 - Martin

[Edmondo Giovannozzi]

Personally I want speed in a GUI application. So I always use pyqtgraph, even sometime modifying the pyqtgraph source in order to have what I need. It is well written, so modifying it is easy. 

If a calculation is slowing me down I can transform it in a Fortran or C code and use ctypes to link it to Python. 

--
You received this message because you are subscribed to the Google Groups "pyqtgraph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyqtgraph+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CAD_p8v3668bj82zYmWi%2Bns_DfHK4JHAbwvb8oSmitU-RkArFpA%40mail.gmail.com.

[Ognyan Moore]

I just want to add that the real advantage of QThreads (when talking about pyqtgraph) is that you can do CPU-heavy processing in a non-GUI thread, so the GUI doesn't (temporarily) freeze/lag as number crunching is happening.  I use a QThread to do some log-based calculations here, and I use the Signal/Slot mechanism to relay the information I want pyqtgraph to plot: 

https://github.com/j9ac9k/barney/blob/main/src/barney/controllers/SubPlotControllers/SpectrogramController.py

The calculation is based on an interactive element in the GUI, so when I first had this setup, when the user would adjust a linear region item, the GUI would temporarily freeze while I was doing a log calculation on a ~500x2000 numpy array.  Moving to a QThread meant the resulting plot was a little late to update vs. the others, but the GUI didn't lag or freeze.

Adding QThreads does add complexity, and I hope to roll an example in the pyqtgraph example library on how to do this; but the general advice is don't go down that route unless you have an actual need.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CACwj3kUb7toyty5sAE4vP6mV3YsRxQ%2BjGMxRh-hecLFvy9w2DA%40mail.gmail.com.

[Jay Vaidya]

Thank you all for your thoughts and responses. I think I will use QThreads (instead of threading.thread for Python) if I choose threads over multiprocessing. Now, after looking at some of the posts, I see that memory is not shared in multiprocessing, whereas threads have a shared memory. I have like 10 plots to be continuously plotting real time data, and that data would be read from a file that is being written by another Python script on the disk. Yet, whether or not I should switch to pyqtgraph is still a question as I was able to do real time plotting of a million full data points by threading with Tkinter when I last used matplotlib.And that is what I need now as well, however, using PyQt gives me advantage of possibly putting in QThreads to call out pyqtsignals which would be otherwise complex with just Python threads. Since I am rethinking my approach while using PyQt, I am also thinking whether I should use threads or not and just use multiprocessing (which is not something I am experienced with). I know Martin already answered about multiprocessing vs threading but I am still confused whether what I did earlier with Tkinter GUIs is correct or should I use multiprocessing now for PyQt GUIs. My software needs to get the data read continuously from a file on disk, and plot it continuously (real time) with 10 plot figures. I understand that multiprocessing is for heavy number crunching, but why would it not be a good candidate for data fetching from the disk (which could be one process) parallely to other 10 plot processes? My old approach consisted of data fetching as a thread and plotting as another thread different from the main thread. But, I didn't know multiprocessing capabilities then and that it can distribute the processing load to multiple cores. So, I am wondering if I can deploy multiprocessing for data fetching processes and data plotting processes, and there may be additional processes added later that have some mathematical calculations but not complicated ones (meaning scaling data vector or addition and subtraction of data vectors), OR if that is not correct use of multiprocessing and that I should stick with threading instead?

Another reason why I am procrastinating to try one of the two, and research first, is because I am making a software that should run real time forever until interrupted, and that plots should be able to plot data of let's say 3 days ago where a day full data is like 10 million plot points (unlike some real time GUIs which would work as real time but for sometime only).Of Course I can read the data from disk if user needs to plot a 3 day or a week old data, but I don't know how plausible that is, as that comes with a overhead of read time for data, so I instead store data in a numpy array of length 10 million (beyond which software really struggles). At the same time, having all the week-long data in memory (using numpy or whatever) isn't possible or maybe also inefficient. Any thoughts on this? Your thoughts would be greatly appreciated. Thanks again!

To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CA%2BnduTFi5wBSuqB9_g43HC_yYhwVSunEf7qz4Km4fjUMLnLbXw%40mail.gmail.com.

[Ognyan Moore]

General advice with this sort of thing is, start w/ the simplest approach (no multiprocessing or multiple threads) and have a go at it, and optimize from there. PyQtGraph can handle 10 plots without any multi-threading capability.  When it comes to that many points, multiprocessing won't help you.

A million data points is a lot of data, on my laptop, it takes almost 200 ms to go from a call to setData to the time the QEventLoop is finished.

Sounds like you're in the preliminary stages, it's hard to say if pyqtgraph will work for your need with certainty or not.  I can tell you if you're trying to render 1,000,000 points in a line plot in real-time, I think you're in for a bad time regardless of what library you're using.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CAKhctBFBTGF5%3DUmYiPD9trwZDh-mPkykjG4ZpJP1%3D1Rk8MFzNw%40mail.gmail.com.

[Erik Johansson]

Hi everyone,

I know I am late to the party, but I wanted to share my experience using both pyqtgraph and matplotlib embedded in a PyQt application and the use of QtThreads for various tasks. It is actually four different applications, each one is a status screen with live updating images and plots based on telemetry inputs from instrumentation, but the same approach is used in each status screen.

First of all, I used PyQt5 as it was a much easier environment in which to work than Qt, which is C++ :-). I used Qt Creator and Designer to develop the overall application display windows. For real-time image display, I used two different approaches:

1) I created a generic image display widget using pyqtgraph's GraphicsLayoutWidget, subclassing it to add methods to update the image data and add an overlay grid.

2) I created a generic surface plot widget using pyqtgraph's GLViewWidget, subclassing it to add methods for updating the display data and for changing the amplitude scale of the surface plot.

The generic image display widget was used to display images of varying sizes, all the way up to a 2k x 2k image updating at 10 Hz. On a couple of the images I also used pyqtgraph's HistogramLUTWidget to allow for real-time adjustment of the display to maximize contrast based on the current image conditions (autoscaling is not always your friend in this regard).

I used matplotlib for graphs and plots, as they all had fairly slow update requirements and matplotlib was fast enough to handle the load. If matplotlib had not been fast enough I would have used pyqtgraph for the plots. I developed a few simple QtWidgets with matplotlib plots embedded in them. All of the image display and plotting was done by the main Qt application's event loop, as that way Qt handles all of the multiple graphics tasks that were happening simultaneously.

I used QThreads to handle the receipt of the telemetry data, which came from several socket subscriptions to the various data sources on other servers. The nice thing about the QThreads was that it allowed me to separate the receipt of new data from the display application. Moreover, QThreads support the Qt infrastructure, so I was able to use signals and slots to communicate the new data and values directly to the Qt widgets that displayed them. Basically I had one QThread per data source, with the run method subscribing to the data source, then repeatedly calling a data acquisition method until a shutdown command occurs, and finally, unsubscribing from the data source. The data acquisition method performs blocking reads on the data queue associated with that thread, performs minimal preprocessing to unpack the data and then calls the Qt signal methods to pass the data to the respective widgets. This design works surprisingly well and is able to support a high volume of image data and simultaneous plots updating in real time.

Hope this is helpful.

Regards,
Erik

To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CA%2BnduTFi5wBSuqB9_g43HC_yYhwVSunEf7qz4Km4fjUMLnLbXw%40mail.gmail.com.

[edmondo.g...@gmail.com]

I have used multiprocessing for reading data because the library that I used was not thread safe so it couldn't be used inside a thread (they are experimental data, not data written on a disk, so I needed an external compiled library to access them). To share the read data between the processes I used shared memory. I used multiprocessing RawArray given the Python version I had. There may be newer alternatives, but if you want to share big array between multiple process you need or shared memory or MPI or some other thing. The standard Python way may be too slow.

[Jay Vaidya]

Thank you everyone for your input. I have tried threads now and it seems to not work well as I have 6 threads apart from the main thread and each thread has a matplotlib graph window with a context menu that allows the user to select data to plot. Data is generally 1 million to 500k x 50 dict or sometimes 1 million x 50 dict. Each thread has a mutex lock with a QWaitCondition so it only runs when I call wakeAll(). Despite that, my app crashes anytime randomly and sometimes it never crashes for the same data and all the steps. Crashing varies depending on how many apps are open on my laptop at a given time and I can verify that it is taking too much of my memory when it is just about to crash using task manager. I also ensured that there are no variables or pointers that I am keeping undefined or any other reasons which can lead to the app crashing, and it seems to be my RAM limitations. My RAM is 8 GB which I thought should work but apparently not for 6 matplotlib graphs each with 500k x 50 data size. I hope using a pandas data frame doesn't result in a crash as I know that pandas are slower than a dict and I use that for some parts of my code. I am trying to get rid of that and just have a NumPy array. Each thread receives a copy of the big data from the main thread by means of a signal. 

It seems that having all 6 plot windows as different processes would be better and so I guess I need to use the multiprocessing module... Though I don't know if I would be able to integrate all the plots (which are in different processes) still in the main GUI with dockWidgets, as I am currently doing. 

Your thoughts would be greatly appreciated, thanks in advance!

To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/e3b41dea-5854-4dc7-8174-ddf3b67d32den%40googlegroups.com.

[Ed Goodson]

Try looking into PGLive Its Thread safe and easy to implement.

From the site.

By default, pyqtgraph doesn't support live plotting. Aim of this package is to provide easy implementation of Line, Scatter and Bar Live plot.

Every plot is connected with it's DataConnector, which sole purpose is to consume data points and manage data re-plotting.

DataConnector interface provides Pause and Resume method, update rate and maximum number of plotted points.

Each time data point is collected, call DataConnector.cb_set_data or DataConnector.cb_append_data_point callback.

That's all You need to update plot with new data. Callbacks are Thread safe, so it works nicely in applications with multiple data collection Threads.