Software design question

[akhenakh]

Hi

I'm actually working on a radius accounting server using the very last gevent DatagramServer, everything is working like a charm.

I decode accounting tickets then insert them into a Mysql server, I need a very fast throughput on the UDP socket.

So I've created a Queue and a Thread pool to consume the Queue and insert into slow mysql server, I'm not using monkey patching at all and everything is working really fast.

q = Queue.Queue()

for x in range(4):
        mysqlThread = MysqlTread(q, config, x)
        mysqlThread.start()

RadiusServer('127.0.0.1:1813', queue=q).serve_forever()

But what is the recommended gevent way to do it ?

Is it safe to mix async gevent with real heavy threads ?

Is it usefull to move from real thread to greenlet for my pool in this case ?

Can I simply monkey patch a full python mysql driver like mysql.connector, remove my pool, insert inside RadiusServer and voila everything will work async ?

Thanks !

AkH

[Sean Talts]

If I were you, I'd use a project called "amysql" for gevent compatible
async mysql connections:

https://github.com/esnme/amysql

It has hooks for gevent and seems to be used in real production
systems. The "downside" is that the interface is pretty different
from the DBAPI standard.

-Sean

[akhenakh]

Hi Sean

thanks for your answer but my questions was more general topic, mysql or whatever.

Thanks anyway !

[Sean Talts]

You cannot monkey patch most mysql drivers because they are written in
C. Are you just asking about threads vs. greenlets? It depends on
your situation. Threads in Python are not usually super helpful
because they all compete for the GIL. However, this sounds like a
case where your mysql server is the bottleneck and adding more threads
or greenlets won't help too much, anyway. That said, greenlets are
probably a good choice but only if you can use an async mysql driver
like amysql. Otherwise, they will actually be worse than threads in
this situation, because a single greenlet will block the entire gevent
loop.

[GUI]

I think the fastest db operating is make the sql execution in a same
transaction, not to make the db connections as much as it can.

So, if you separate the db operating to a single process, and make the
process collect the insert operations to execute them in a same
transaction group by group, it will make the insert speed maximumly.

Thanks!
Please excuse my pool English.