Is snakeMQ a fit for this use case?

[roger.d...@gmail.com]

I run a web cam with the motion package on a Raspberry Pi. Whenever there's movement, motion saves an image. motion has a function where, after saving an image, it can call an external command with the path to the image as argument. I'm using this to run wput to push the new image to an FTP server and then delete the local copy. The problem with this is that the FTP upload is relatively slow and motion runs the external commands in separate processeses as fast as the images are saved. So, when a series of images is captured, a bunch of concurrent wput processes are launched, all fighting for bandwidth.

Enter snakeMQ. I was thinking that I could write a small script that motion calls instead of wput, which just pushes the image path onto a queue and exits. And then have a listener running continuously that pops the paths off the queue and uses wput to upload them one by one in queued order.

So the system must support multiple processes (launched by motion) possibly trying to push to the queue concurrently and the processes launched by motion should be able to exit right away, regardless of how many messages are in the queue.

I did a simple test where I created a listener that simulated FTP upload time by having a sleep() in on_recv() but the sender would hang while the listener was in the sleep(), so there was no actual queue, just one message handled at a time. In the sender, I have a sys.exit() in on_sent(), so that it exits out once the message has been sent.

Thanks for any info.

[David Siroky]

Hi!

If you put sleep() in the Link.on_recv() then it is the same as if you
bend a garden hose. The flow will stop. You are blocking the
communication channel. Queuing in sMQ is in a higher layer.

You need to implement a very simple resident sender. It must be running
the whole time, otherwise you loose the queuing. No quitting after a
single message in the on_sent(). Pass images to this resident sender.

David

[roger.d...@gmail.com]

Thank you for the reply and for snakeMQ.

It seems that, if I was to implement a resident sender, I might as well queue the messages there and do the uploads directly, circumventing a third party mq altogether. So, for this particular project, I went with beanstalkd and the beanstalkc wrapper for Python, which let me implement it like I first outlined.

I'll keep snakeMQ in mind for any future mq needs. Thanks again.

[David Siroky]

As far as I know beanstalk does not provide a persistency and the client
does not automatically reconnect on a connection failure. You will have
to deal with those things on your own. SnakeMQ does this inherently.

D