Re: Redis + Python for a Producer/Consumer Application [should I?]

[Brandon Adams]

> Details:
>
> It's a job scheduler for a small cluster of software testing servers. My
> users need to schedule test jobs that are distributed over the machines in
> the cluster, so I need a daemon running on each computing node accepting
> jobs and a command line client that my users will call to schedule their
> jobs across the cluster. I need the message acks because the client-side app
> needs to know when a job failed to run, so it can be re-scheduled to another
> machine - and also to provide a progress feedback for those jobs to the user
> running it.

Why do you need message acks? Consider this:

daemon receives a job. The daemon publishes the job id to a pub connection
or pushes to a list. After a certain (probably short) time, the client
checks for
the job id in the received queue/list. If it doesn't see one, it
retries (likely to
a different node).

I'm not saying you should necessarily do this, but from your requirements, I'm
not explicitly seeing a need for an ACK bit on the queue itself. A second queue
of ACKs should do just fine.

[Josiah Carlson]

Celery uses what is known as a "tombstone", which is the result of the
call. This tombstone usually stores state about the result, whether it
had raised an exception, etc.

The op can implement that himself with SETEX at the tail end.

It is also one of the features that is on my list of things to do for
rpqueue. I didn't bother before because I figured that people could do
it themselves, but having that available in a library can simplify a
few other common use-cases.

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>

[Carlo Pires]

Hello Gustavo,

If you dont need persistence, than all you need to do is a round robin pub with nodes as subscribers. I see zeromq sockets as a good fit for such project.

--
  Carlo Pires

Em sexta-feira, 19 de outubro de 2012 15h10min18s UTC-3, Gustavo Campos escreveu:

Hi guys;

I'm developing a Python application (more details at the end) using a Producer/Consumer model. I started off using Pika and RabbitMQ, but then moved on to Celery/RabbitMQ since Pika looks like abandoned. I'm not liking the celery way very much, so a friend suggested me to go the python-redis + Redis way.

I was reading about it around and really enjoying everything, but read somewhere (SO I think) that Redis doesn't support message acks. Is there a (easy) way for me to simulate that so I can use Redis in my application? Is Redis recommended to do that, or should I stay with AMQP?

Details:

It's a job scheduler for a small cluster of software testing servers. My users need to schedule test jobs that are distributed over the machines in the cluster, so I need a daemon running on each computing node accepting jobs and a command line client that my users will call to schedule their jobs across the cluster. I need the message acks because the client-side app needs to know when a job failed to run, so it can be re-scheduled to another machine - and also to provide a progress feedback for those jobs to the user running it.

I don't need any persistence aside from that: I don't need a history of past ran jobs, anything like that. If a client-side app dies, it would be great if I had some easy way of killing it's jobs, but that's not a priority item.