Taskspooler excellent for server-side compute queues

[da...@agi.io]

I just wanted to share my experiences using taskspooler (ts) for server side compute jobs. 

In a web application, HTTP requests should return quickly meaning that lengthy calculations should be performed asynchronously. 

I have used taskspooler a couple of times for machine learning jobs to manage a queue 

The alternative is something like AWS Simple Queue Service (SQS), but to be honest if you're not doing auto-scaling to meet a constant time target, taskspooler is perfectly adequate.

Typically, I get the web-application (any language, Python or e.g. Rails) to issue a system call which runs a script to enqueue a job with taskspooler. The job itself is then run by taskspooler and then does a local HTTP request to PUT or POST the results back to the web app, also acting as a notification that the job has finished.

Anyway I thought someone might find this a useful way to run small machine learning jobs in a queue as part of a webapp.

[Lluís Batlle i Rossell]

Thank you!

> --
> You received this message because you are subscribed to the Google Groups "taskspooler" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to taskspooler...@googlegroups.com.
> To post to this group, send an email to tasks...@googlegroups.com.
> Visit this group at https://groups.google.com/group/taskspooler.
> For more options, visit https://groups.google.com/d/optout.


--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/