Benefits of Celery

[Quite Lost]

Hello,

We are trying to implement Celery for distributed task execution. Right now we are in locked horns between team members trying to reason on whether to use Celery (with Redis / SQS) or use SQS directly using its AWS SDK.

It seems there are some problems with Celery + SQS when running in Production. There are Github issues on that - https://github.com/celery/celery/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+sqs+label%3A%22Issue+Type%3A+Bug+Report%22+

So, I just wanted to know what benefits can we expect to get by using Celery (with Redis / SQS) rather than SQS directly. Also, we are not willing to use RabbitMQ.

In a way, what does Celery do differently or provide, that would be difficult to implement when using a system like SQS directly (mainly on the worker side / worker management)?

We are not using most of the advanced features provided by Celery anyways (Canvas, Beat, etc.)

When it comes to monitoring, is it true that Flower cannot be used in production with heavy workloads? If true, can someone point me in the right direction.

Other than Flower, I don't see any other of-the-shelve monitoring solution.

I already know what Celery + SQS does not support:

• Recurring jobs / Scheduling are not supported.
• SQS doesn’t support worker remote control commands.
• SQS doesn’t support events.

The last point specifically would make it difficult to have complete Flower support with SQS. Also, we need some of the guarantees that SQS as a queue service provides, not available in Redis inherently.

Regards.

[Gal Cohen]

I would say it depends

We migrated our project from Redis to SQS recently but we did it because of regulation issues - we could not afford loosing jobs (look for redis eviction policy

It was only a configuration change and solved the problem for us

If its a brand new product - I would avoid using Celery with SQS, just use boto  and implement a polling mechanism by yourself. 

Things to consider:

1. Do you want the tasks to be acked before or after the execution? (celery default - before execution

2. Are you planning on using DLQ? 

3. Do you need redrive policy? what is the visibility timeout you are looking for? 

4. Celery SQS requires extensive IAM permissions in order to manage the queues by itself

Regarding scheduling - it is supported using celery beat of course 

[Ing. Josue Balandrano Coronel]

In general I always recommend to use a proper AMQP implementation  with Celery and not Redis. If you're not willing to use something like RabbitMQ then that's less points for Celery. That is less point because SQS lacks some for the key components that makes Celery an interesting alternative to implement stuff by yourself.

Celery it's just a task queue framework that uses an AMQP library written specifically for task queues (kombu) and a connection pooling library (billiards). It implements a basic structure for workers and producers so you don't have to worry about it.

So, if you are going to have some heavy loads on production Redis is out of the question. You're only left with "should we use Celery with SQS". And you are not going to do anything complicated like workflows. The only thing Celery is bringing to the table is:

1. An abstraction layer that will manage SQS queues for you, which is good since your code can be migrated to, say, RabbitMQ in the future without a hassle. There's other options like ZeroQ and others...

2. Basic worker implementation with re retries, throttling, class based workers, etc... Some of this is already bedded in SQS like retries, tho.

3. The biggest thing Celery brings to the table, I would say, is a result backend implementation. You can save the tasks results or just metadata to implement monitoring or anything like that.

Flower is OK for monitoring but it's not 100% perfect and it might fall short in production (we need more support on this side). But, with Celery's abstraction layer it's easier to implement some custom monitoring, e.g. using class based workers.

I can't tell you weather to use Celery or not because I don't have enough info on your project but I hope this sheds some more light into the issue. Anything that's in Celery can be easily developed in-house, mainly since we're not talking about the more advanced features. I think the tie breaker would be to do a proof of concepts of both and try to decide which is going to take less time to get running.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/celery-users/7751dbd9-447a-4bd5-8012-ab9b4c937f1b%40googlegroups.com.

[Gal Cohen]

SQS is not supporting results backend..

"Warning
Don’t use the amqp result backend with SQS.

It will create one queue for every task, and the queues will not be collected. This could cost you money that would be better spent contributing an AWS result store backend back to Celery"

[Ing. Josue Balandrano Coronel]

SQS shouldn't be used to store results, no. Redis should be used to store results.

Quote on the previous message refers to using the AMQP protocol and SQS as backend Ro store the results. This is something used with RabbitMQ as broker; it works in some setups because the result is stored as a message in a queue and it can be used to trigger other stuff or just because reading a message in an AMQP queue destroys it so it can only be read once.

So, when configuring a results backend, regardless of the broker, use a proper DB as Redis or something like that or RabbitMQ.

But since you can't use RabbitMQ I would go for Redis.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/celery-users/59d6c706-3652-4b26-9204-8fe09e19c428%40googlegroups.com.

[Quite Lost]

Hi,

Thank you for the responses! We are not going to use the result store as well. We are directly making changes to DB from within the task logic.

Regards.

[Ing. Josue Balandrano Coronel]

So then there's really only two big things Celery brings to the table and this is mainly because of the SQS limitations.

I would try to.impmement a proof of concept with and without Celery and evaluate.

--

You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/celery-users/c01735ea-5b41-43a3-aa95-687047078fc4%40googlegroups.com.