Can you use SaltStack as alternative to Celery?

[Thomas Güttler]

Hi,

Can you use SaltStack as alternative to Celery?

Up to now we don't use SaltStack, I read the docs and know that SaltStack is not only
useful for configuration.

What misses SaltStack that Celery gives me?

Up to now we use Celery in a small project. We use it to do some
stuff in parallel to speed up a simple batch job.

Thomas Güttler

[John Spray]

That's kind of an interesting question, I hadn't thought about using salt for parallelising tasks, but I guess you could... the main thing you'd be missing from celery is the persistent queuing aspect: celery will let you push jobs onto a persistent rabbitmq queue and then (more or less) reliably execute them eventually, whereas salt isn't really build to manage a long queue of tasks for you.

John

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Glenn E. Bailey III]

Curious as to why you wouldn't consider Salt for parallel execution as that was one of it's design goals? If you asynchronously submit a job into Salt you receive a JID back which can then be used to track the entire life cycle of the job. I'm not that familiar with Celery so maybe I'm missing something? 

http://salt.readthedocs.org/en/latest/topics/index.html#parallel-execution

--

glenn - like tears in rain... Time to die.

[John Spray]

Salt's parallel execution is built for doing the same thing to a number of hosts, whereas a work queue + worker pool like celery is built for doing lots of different jobs, shared among a number of hosts.

I'm not saying you couldn't achieve some of the same things with both, especially if you've got a single scatter/gather phase in a batch job rather than having a queue of tasks, but celery & salt are pretty fundamentally different tools designed for different purposes.

For example, celery let's you say "here are 100 units of work, execute them on this pool of 10 workers".  If you were trying to accomplish the same thing with salt, you'd have to go write your own code for scheduling the tasks across the workers.

John

[Stephen Wood]

They are indeed very different tasks. I would not use saltstack as an amqp replacement. Not that it maybe couldn't do this type of work, but this task is better left up to dedicated messaging software for a few reasons:

1. Persistency & scalability of data
2. Ease of management
   
3. Better APIs for interaction

Stephen Wood

www.heystephenwood.com

[Glenn E. Bailey III]

I think I see what y'all are saying. I'm currently evaluating Salt to basically do remote execution with dab on config management on top. But I will have a separate scheduler system that would be injecting jobs into Salt via the API. So not purely message queuing but "Do X on Y node right now and gimme an ID to track by". 

[Thomas S Hatch]

Just fyi, we are working on a salt runner interface that accepts jobs on the master and queues them out to minions for work. This is because we have a growing user and customer base that want to do exactly what is being discussed here as well a large interest in supercomputing for this specific feature. It may make the Helium release

Thomas S. Hatch  |  Founder, CTO

2825 E. Cottonwood Parkway, Suite 360 | Salt Lake City, UT 84121

tha...@saltstack.com | www.saltstack.com

[Chris Overend]

Is this implemented?

On Tuesday, April 8, 2014 at 2:21:24 PM UTC-4, Thomas Hatch wrote:

salt runner interface that accepts jobs on the master and queues them out to minions for work.

[Dimitri Maziuk]

On 04/05/2017 04:34 PM, Chris Overend wrote:
> Is this implemented?

Yes, it's called htcondor.

> On Tuesday, April 8, 2014 at 2:21:24 PM UTC-4, Thomas Hatch wrote:
>>
>> salt runner interface that accepts jobs on the master and queues them out
>> to minions for work.

--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

[Seth House]

I believe this thread is referencing the queue runner module:

https://docs.saltstack.com/en/latest/ref/runners/all/salt.runners.queue.html

Which is an interface to Salt's queue modules:

https://docs.saltstack.com/en/latest/ref/queues/all/

[Thomas Güttler]

Am Mittwoch, 5. April 2017 23:58:06 UTC+2 schrieb dmaziuk:

On 04/05/2017 04:34 PM, Chris Overend wrote:
> Is this implemented?

Yes, it's called htcondor.

According to wikipedia htcondor is:

HTCondor is an open-source high-throughput computing software framework for coarse-grained distributed parallelization of computationally intensive tasks.

This question (which celebrates its third birthday the day after tomorrow) is about how you can use the salt infrastructure as alternative to celery.

Yes, htcondor is an alternative to celery.

AFAIK htcondor is not related to saltstack. AFAIK saltstack is not based on htcondor.

BTW, thank you to all involved in this thread and for reactivating after three years of  long Sleeping Beauty slumber :-)

Regards,
  Thomas Güttler