Unresponsive GUI during file read and writes

[Thomas Pope]

I've been battling this for about 2 weeks off and on and haven't had much luck.

I have a kivy GUI that works great. My problem is that part of the script pulls temperature data from DS18B20 temp sensors (reading files), displays it as labels, and also saves the data to a spreadsheet. Technically everything works, but whenever there is any reading or writing going on the GUI becomes unresponsive. I've tried reading and writing with subprocess and that did improve things a little. Is there any way to handle this that won't cause the lag I'm experiencing? Thanks in advance!

Tom

[Oon-Ee Ng]

You don't mention whether this is running on an ARM processor
(android, for example) or on a laptop.

Also, what is 'subprocess'? Are you using the multi-processing module
(implying you're on a laptop)?

If your GUI becomes unresponsive its normally because one of your kivy
functions is blocking on your I/O call. So for example if your button
click calls function foo() and foo calls a long-running I/O process,
your GUI will hang until foo() completes. The solution here is for
foo() to send a signal to a thread/process rather than directly call a
function.

> --
> You received this message because you are subscribed to the Google Groups
> "Kivy users support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kivy-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Steve B]

Sorry with out knowing which platform you are using I'm guessing its ARM in Android or Pi. 

If its kivy in a Raspberry Pi you in luck as this polled at system levelat /sys/bus/w1/devices and doesn't require you to write your own drive code; see 

https://learn.adafruit.com/adafruits-raspberry-pi-lesson-11-ds18b20-temperature-sensing/ds18b20

For Thread (multiprocess) check out https://pymotw.com/2/threading/ which stop your GUI hanging on any platform

[Thomas Pope]

Sorry, I'm working with on a raspberry pi.

I'm pretty familiar with the workings of the sensors and am getting great data, but even though it is system level it is still causing lag because each sensor check is basically opening a file and reading the lines. Each "read" takes about half a second and I'm running 5 sensors, then I need to save the data to a spreadsheet and that takes another half second or so. I want to update the sensor readings as often as possible, but as it is I'm maxed at about once every three seconds just due to the time associated with each process and this leaves my GUI completely shut down.

I've been doing a ton of reading and it sounds like threading is the way to go, but I haven't found any examples that are passing data in multiple directions and running in a specific sequence (and I'm not experienced enough with it to just put it all together myself yet). I need read sensors -> store globally -> display on GUI -> store in log file -> repeat without ever interrupting the GUI. I realize its going to take some CPU, but thats fine as nothing else is running while this is going.

Thanks again!

[Steve B]

Okay, for Raspberry-Pi kivy users heading this way... time.sleep() will hang up your GUI :( To run smoothly you need to grasp Event interrupts and multiprocess like the Threading module to keep everything ticking over without wait loops. In short your hardware code needs to run in the background to Kivy.

Check out the Kivy Clock.schedule module, this will help you setup time delays and events without using wait loops and time module functions. You can also pass some basic variable to an instance using partial... 

from functools import partial

Clock.schedule_once(partial(self.assylist, var1, var2) 0.1)     # Call self._assylist(var1, var2) after 0.1 secs

For Threading, look at the link https://pymotw.com/2/threading/. Seriously, you need to grasp this to run any basic parallel process like... read sensor and do GUI and something else. It is no possible to run one routine after the other scooping up results as you go, the GUI will miss vital background tasks causing it hang or crash. Sorry there is no example cut and paste routine for this, the script is built around keeping the the GUI running and polling parallel process results.   

I think this Link will help you, it shows how to flash two timed LEDs while doing something else...

http://raspberrypi.stackexchange.com/questions/28984/how-to-blink-leds-on-off-continually-while-continuing-execution-of-a-script

and...

http://raspberrypi.stackexchange.com/questions/42807/threading-with-gpio-and-buttons

I hope that helps you get back on track and lets your Kivy run. BTW On the Adafruit link there is a code example for uploading the temp results to GDrive. 

[Thomas Pope]

I'm running the clock module and have completely avoided any use of sleep. I have intervals scheduled to read the sensor data, then update the gui and finally, write it to a file. The schedule intervals work beautifully, but while the read and write processes are happening the gui hangs. From the reading I've done it looks like I need to run a second (or third) thread to deal with the sensor data.

The flashing LED example looks like it may be exactly what I needed. Thanks!

P.S. I tried uploading to drive first and it was unbelievably slow. The security handshake that Google requires was taking 2-4 seconds per data dump. It was awesome to have the data available on any platform, anywhere, but it completely killed all functionality of my setup. Running it in a thread may solve part of the problem, but it would still mean that the data logger's frequency is pretty limited.

[Steve B]

I've cobbled together a very simple thread example for polling a DHT22 sensor and saving the results to a file. I is not a kivy example nor practical. It only demonstrates the fundamentals of very simple use of thread while doing a count....

  

#!/usr/bin/python

# 

import sys

from Adafruit_CharLCD import Adafruit_CharLCD

import Adafruit_DHT

import time

import threading

def readSensor():

    ''' Read DHT sensor and return formated result string '''

    humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT22, 4)

    if humidity is not None and temperature is not None:

        str = ('Temp: {0:0.1f}*c#Humidity: {1:0.1f}%'.format(temperature, humidity))

        return str.split('#')

    else:

        print('Failed to get reading. Try again!')

        return None, None

   

def saveData(mydata):

    ''' Save my results append to file '''

    fo = open('myresults.txt', 'a')

    fo.write(mydata)

    fo.close() 

def myLoop():    

    '''Main thread Loop, replace while and sleep with kivvy Clock calls from within your script'''

    while True:    

        t, h = readSensor()  

        lcd.clear()

        lcd.message( t + "\n" + h)

        saveData(t + " : " + h + "\r\n")

        time.sleep(1)    

def runSensors():           

    ''' Starting a loop using threading '''

    proc = threading.Thread( name = "Run Loop", target  = myLoop )

    proc.daemon = True

    proc.start()       

    

if __name__ == '__main__':

    

    lcd = Adafruit_CharLCD.Adafruit_CharLCD()

    lcd.clear()

    

    runSensors()

    

    count = 0

    

    while True:                             # A loop to keep the thread alive

        count +=1

        print "Just counting... ", count

        time.sleep(0.5)

All it does is call a thread to parallel process 'runSensors()' which, reads the sensor, prints it to an LCD, saves the results and does it all again after a second. Meanwhile a main while loop prints a count output. If you observe the time of the loop and the process loading you will see glitches during the peak loading of 1-wire and file save latency. Trick is to time the polling of your parallel routines to give other routines and in this case the GUI essential housekeeping time from the CPU. 

In this example, to split off the sensor as a separate thread and buffer the results is a better way to go as the filing is slower than the sensor. And the display is display is slower than the sensor, but faster than the filing routine. Oon-Ee Ng touched on this with signals, but I'm trying to keep it simple. Using the kivy clock you can poll the threaded routines at intervals appropriate to the task. and you need to move on to the next poll if the result is not ready. Read the python docs on thread for more control over your threads like, join(), and isalive, and so on.

You need to change or delete the LCD and sensor arrangement to suit your needs, but its something to play with to get your head around time sharing. File routines are a bit gutsy for the Pi and there is limitation to the CPU power available with the overhead of a real time GUI. Displaying a graph of results and saving results from ram on demand is a better way to go. I hope this helps.  

[Thomas Pope]

Thanks so much for your help! Even after implementing a separate thread (again using only clock schedules rather than waits) I'm getting some lag on the GUI. I need to do some more reading and lean out my code, but for now I think what I have will work for the time being. The reality is that I'm asking a lot of for a raspberry pi and I may not be able to get everything running seamlessly.

[Damien Moore]

It could just be the limitations of the Pi, but bear in mind that python uses a global interpreter lock and so your UI can become unresponsive if the worker thread doesn't release the lock often enough. Python threads work best when the worker thread is calling routines that call into C code and release the the lock before doing so. (Reading a python file into memory should fall into that category, but I'm not sure.)

[Thomas Pope]

Unfortunately running two threads isn't speeding things up. The GUI is basically dead any time a sensor is read even though the sensor reading is being handled by a second thread and is scheduled with the kivy clock. Is there a way to force threads to run on separate cores of the cpu - truly independent of each other using global variables so the information is easily grabbed or populated by both threads?

I'm not seeing more that about 25% CPU usage so I know the pi can handle more if I can get python to handle processes better.

[Oon-Ee Ng]

I wouldn't be so sure about the 'pi can handle more' bit, I have a
couple of Pis and they're not awesome at multi-tasking really.

Try multi-process rather than multi-thread, as the GIL would still
block you if its engaged. More complex, but your best bet at this
time. Except that your code would then be undeployable to android (and
iOs I think).

[Bill Janssen]

Use the Python multiprocessing module.  You can use a Manager to host the multiprocessing versions of global variables.