threading

[Charlie]

I use threading module for concurrent SQL queries. Using a simple python script, all threads run normaly but when i use the same code in a WSGI app, the threads never start

Thanks for your help

Charlie

[Graham Dumpleton]

Would need to see an example of the code you are talking about to understand better what you are talking about. In particular, how are you creating the threads and where? How are you waiting on the threads to complete, etc?

Also, are you using manually configure Apache/mod_wsgi or are you using mod_wsgi-express? If manually configure Apache/mod_wsgi, how are you configuring mod_wsgi?

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/b2e6594a-0477-4b4e-a2b6-bba162922296n%40googlegroups.com.

[Charlie]

My code is :

from threading import Thread

class SQLThread(Thread):

        def __init__(self, id):

                 print(' SQLThread ')
                Thread.__init__(self)
                self.id = id
                self._fullname = None

        def run(self):
                try:
                        print('run')
                        connection = MySQLdb.connect(MYSQL_SERVER, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DB, connect_timeout=2)
                        cursor = connection.cursor()
                        self._fullname = get_user_fullname(cursor, self.id)
                        connection.close()
                except MySQLdb.Error as e:
                        print('SQLThread: %s' % e)

Sorry, i thought the threads never run because 'run' is not written in the apache log file whereas ' SQLThread' is written. Do you know why ?

[Graham Dumpleton]

You never actually start the thread by calling start() on the thread object.

Do be aware that creating a background thread per request like this to do a database operation is generally a bad idea. If hit with a large number of requests you could very easily overwhelm the number of allowed database connections. You also aren't cleaning up the state of the thread by calling join() on it afterwards either.

Is there a reason you aren't doing this within the context of the request rather than in a background thread?

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/899fb259-674c-4b02-bded-906274f18321n%40googlegroups.com.

[Charlie]

The threads are started in the following function :

def get_users_fullname(site, ids):
        users = {}
        threads = []
        for id in ids:
                t =  SQLThread (id)
                threads.append(t)
                t.start()
        for t in threads:
                t.join()
                users[str(t.id)] = t._fullname
        return users

I have to collect a lot of data from 5 sql servers. I tried without multithreading and i was very long to process, then i use threading module and the process time was reduced by 10.

I tried to give the sql cursor to the thread to avoid reconnecting to the sql server but it didn't work, maybe it didn't use the good method

[Graham Dumpleton]

How many user IDs are you looking up? It would be much more efficient to use the SQL IN operator and pass the set of IDs to the database.

https://www.w3schools.com/sql/sql_in.asp

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/3418ff45-33ed-480f-801c-e58cdbf4ab96n%40googlegroups.com.