Consuming a Queue in production

[James]

Hi,

What's the recommended way of consuming a queue in a production environment?

The example in the documentation show you looping over the items in the consume method, but how do I do that in my production environment?

Thanks for the great tool!

[richardhenry]

Without knowing exactly what you want to do I can't give you specific advice, but I can make up an example. Let's say you have a page in a Python web app, and when that page is loaded, you want to log the fact that the page was viewed to a database. But the database is slow, and you don't want to block the page load on writing to the database. You could write to HotQueue instead, which would be faster.

You would want to add some code to the view controller that renders that page. Make up a queue name (in this case "pageview_queue", that you can refer to later). This code is pretty simple, and might look like this:

    from hotqueue import HotQueue

    queue = HotQueue("pageview_queue", host="localhost", port=6379, db=0)

    queue.put({'ip': request.ip, 'time': datetime.utcnow()})

Then, you would create a new Python file called `pageview_queue_worker.py` with the following code:

    from hotqueue import HotQueue

    import MySQLdb

    queue = HotQueue("pageview_queue", host="localhost", port=6379, db=0)

    @queue.worker

    def pageview_queue_worker(data):

        try:

            conn = MySQLdb.connect("localhost", "myuser", "mypass", "pageview_db");

            cursor = conn.cursor()

            cursor.execute("INSERT INTO pageviews (ip, time) VALUES ('%s', '%s')" %

                (data['ip'], data['time']))

        except Exception:

            queue.put(data) # Basic error handling to retry later

        finally:

            if conn:

                conn.close()

    if __name__ == "__main__":

        pageview_queue_worker()

Then you just want to run python pageview_queue_worker.py in the background on your server to keep working the queue. You can use a tool like hotwatch to monitor the length of the queue to see if your database is keeping with the queue over time. You can run a second instance of python pageview_queue_worker.py to parallelize insertions — there's probably a magic number of queue workers where you're not overloading your database with simultaneous connections. If the workers run out of work (nobody is viewing the page) they will just wait until more work is available and do nothing.

Hopefully this example helped, let me know if you have any questions.

Richard

[James]

Thanks Richard.

I ended up going with a similar solution for my project.