sidekiq for elixir/phoenix

[micha...@gmail.com]

I'm coming from the Rails world.  What is the current community accepted method to do background jobs with Elixir/Phoenix?

Currently for ruby/rails the community uses resque / sidekiq / rabbitmq for queue

I have seen a talk by Jose Valim, and realized elixir's nature for being distributed and concurrent nature may have so many more possibilities than PHP, Ruby, Python, especially with supervisors.

For the most part, although Ruby has threads and parallel concurrent threads is possible with jruby, most "average" programmers still avoid using threads.

Some smarter folks have been using celluloid with actor pattern but still for true parallelism you still need jruby.

However people for the most part still use the forked ruby process where processes cannot talk to each other.  So people end up sharing state between the processes using redis/ sidekiq.

What do you guys do if lets say I have a queue of work where each work takes 3 min to process?

[Barruum Rex]

You should check out Verk. Coming from a ruby/rails background, that will feel particularly familiar to you. 

Generally people use genservers for anything that needs to persist and perform tasks asynchronously. If you just need to trigger actions off of events, you can also use tasks. Everything is based off of processes and message passing. You've got options. Try them out and see what works for your use case. 

--
You received this message because you are subscribed to the Google Groups "phoenix-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phoenix-talk...@googlegroups.com.
To post to this group, send email to phoeni...@googlegroups.com.
Visit this group at https://groups.google.com/group/phoenix-talk.
To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/634a61af-b5ed-46bf-85e0-e4c02d5be3f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Hamilton]

I will also bring up an often overlooked approach for people coming from ruby/rails: do the work in the foreground.

Under Rails, you have a finite number of workers, each with significant memory requirements.Tying up a worker with a long running request limits the number of workers available for other requests.  Your goal is to return a response as quickly as possible so you can take on another request. With these constraints, moving any expensive or blocking work to the background is desirable.

Under Phoenix, you have a practically unbounded number of workers serving web requests. Each is very lightweight and the preemptive scheduler in BEAM is great at keeping live processes executing. If you need to block or do something expensive in one of the handlers, it's not a big deal.

There's definitely a case for backgrounding in Elixir, but the threshold for doing so can be much higher than in Ruby. Just something to think about.

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/CAKshaqVSUPLHCXrJMACsHjTr7JT0mg2B1PxX-xgEFFDK3v%2BVzA%40mail.gmail.com.

[micha...@gmail.com]

Yes, this is what I'm interested in.

With Elixir/Erlang, I am able to to communicate with any light weight process not only in same machine, but in other machines, all abstracted.  I don't have to worry about garbage collection overhead since the processes are isolated.  

in the Rails way, the solution was to put the work into a queue and have other rail processes pick work off the queue.

with Elixir, is there some sort of distributed solution, such as...

any worker just gets work possibly triggered directly from an http request,  if it isn't too busy, just do the work on the same machine with some other process.  if it is busy, is there a way it can hand off the work directly to another machine without going to some central controller?  basically a distributed solution vs centralized solution

Is there a library that abstracts this already?

[Peter Hamilton]

What are you trying to solve? Is this a CPU bound situation? Is it a blocking task?

What you are describing sounds like overkill. It would solve problems you aren't likely to have.

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/67f755eb-04a1-4caf-a213-ab46256f8806%40googlegroups.com.

[micha...@gmail.com]

i'm just brainstorming, elixir is opening my mind on what is possible, I figure we don't always have to do things the same way as always.  

Maybe elixir has a new and better way to just solve a group of these types of problem all at once, sorta like how git replaced SVN by being completely distributed vs centralized.

[Imran Ismail]

I also would like to see something similar but using technologies that are baked into Erlang like amnesia for fault tolerant background task.

- Schedule job to be run ahead of time

- Retries (Supervised task)

And so on.

[Christian Kruse]

Hi,

Peter Hamilton <petergh...@gmail.com> writes:

> I will also bring up an often overlooked approach for people coming from
> ruby/rails: do the work in the foreground.

Doing the work in the foreground might have negative implications on the
response time for the user. E.g. sending a mail or resizing images can
take several seconds, and when doing it in the foreground it may lead to
very long response times. Not a very good UX.

Best regards,
--
Christian Kruse
https://wwwtech.de/about

[Steve Domin]

I agree with Christian, I don't think we should advocate for that approach, it's going to start with emails but then people are going to start doing more and more things and response time will go through the roof.

Dynamically supervised async task is the first step in my opinion, if you don't care about persistence and don't need any guarantee about whether a job is done or not.

Eduardo has done a fantastic job on verk, and there are all the other resque/sidekick alternatives (exq, etc.) but there are still all based on Redis and it's a bit of shame imo. I've been thinking a lot about this lately but didn't come up with a satisfying solution yet. Very interested in what the community has to say on that subject.

--
You received this message because you are subscribed to the Google Groups "phoenix-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phoenix-talk...@googlegroups.com.
To post to this group, send email to phoeni...@googlegroups.com.
Visit this group at https://groups.google.com/group/phoenix-talk.

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/87a8jer0t0.fsf%40sunshine.fritz.box.

[micha...@gmail.com]

Can't a elixir process just receive the request, create another lightweight process that is on its own, that MAY be even on another machine, and just respond immediately?  

the problem with php, ruby, python etc is, the language wasn't fault tolerant so if you fork a process to do some work like emails, if it crashed you could lose the email, but now with elixir, if it is truly fault tolerant, maybe don't need redis to persist the work?  

[Christian Kruse]

Hi,

micha...@gmail.com writes:

> the problem with php, ruby, python etc is, the language wasn't fault
> tolerant so if you fork a process to do some work like emails, if it
> crashed you could lose the email, but now with elixir, if it is truly fault
> tolerant, maybe don't need redis to persist the work?

I had similiar thoughts. I see several problems with this:

- No control. We can’t e.g. monitor background jobs, with solutions like
Verk monitoring is a first-class citizen.
- What about failing jobs? Just ignoring or retrying n times might not
be enough, depending on the job itself.
- What about load balancing? Rate limiting might also with Erlang/Elixir
be a necessity, e.g. when you want to limit I/O.

I think a persistent solution with a queue and monitoring might really
be a better idea. IMHO it doesn’t have to be redis (yet another
dependency), it can also be a solution based on mnesia or something
similiar.

[Steve Domin]

That's exactly what the Task module (http://elixir-lang.org/docs/stable/elixir/Task.html) is for. And more or less what I'm suggesting in my previous message, sorry I should have been clearer.

Elixir is indeed fault-tolerant, it means that if something crashes it can restart in a known good state, it doesn't mean you can't lose state.

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/98ba6651-b83e-448d-824c-45bb9c8b2a25%40googlegroups.com.

[Peter Hamilton]

My suggestion wasn't to do everything in the foreground. It was merely that the technical reasons for doing it in the background are less prominent. UX should still be a primary driver of the decision, but in my experience there's some cargo culting around background work that isn't necessary.

HTTP is a wonderful synchronous protocol. There are massive benefits in maintaining that synchronous behavior. It provides back pressure to the client as well as greater guarantees on confirmation of execution. These are important aspects of UX that often have complicated solutions due to technical limitations.

BEAM is sufficiently robust and powerful as to allow is to model systems and processes in intuitive ways. For example, a call center might choose to put you on hold for a short period of time or they might offer to call you back. There are valid reasons for each approach that are deeply rooting in the caller's experience. Being able to model an interaction like that somewhat directly in software is exactly what erlang is built for. (Cue a Joe Armstrong rant about "breaking the laws of physics").

Anyway, that's the crux of my advice here: don't let technical constraints on other platforms skew your requirements. Step back and focus on the actual interaction you are modelling and in my experience a simpler approach exists that was otherwise unavailable due to technical limitations.

- Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/CAB-Ba_Y5fYeu0O%2B-eif12Ezndmwe%2BZ1SYZyZvMj7ev2afrC%2Bng%40mail.gmail.com.

[Chris McCord]

I agree with Peter's point. Obviously UX has to be taken into account, but the concept instant offloading of anything remotely blocking a request can be safely ignored in Elixir land. So for something like sending an email, I'd just block personally. 

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/CAOMhEnx8u44zwupm3SNoRW6FKPGtQP7iXbikmVkeo04%2B5%2Bd90w%40mail.gmail.com.

[Alexander]

I'd like to see any kind out-of-box solution.

Points already mentioned in this thread:

* Monitoring! some kind of UI to control what's going on there

* per-"queue" rate limiting

* configurable auto-retries

* any kind of reporting for failed/retried "jobs"

It's shame the only solution(s) are using redis

Atm using exq+exq_ui, feels little unpolished at edges

[Chris Keathley]

IMO one of the best things about Elixir is that it gives you all of the primitives that you need to replace a traditional worker queue that you might find in the Rails world. In our Elixir apps Supervised Tasks or GenServers are typically all we need and give us more flexibility when it comes to handling errors and retries than a traditional worker queue like redis or sidekiq. If we need to store job state somewhere then its typically in a GenServer, ETS, or the DB. All of this implementation gets wrapped up behind its own module or application in an umbrella app. In general we try to discourage bringing in other dependencies (like Redis) unless we absolutely need them (aka. we deployed to heroku).

That said, one scenario that I don't think has been brought up yet is that sometimes you'll want a queue or worker pool in order to rate limit outbound connections. 

For instance lets say when a user makes a request there is a call to a 3rd party service. Depending on the service its entirely possible that you'll run over your concurrent request limit given enough users. In those scenarios you might need to use something like Poolboy to rate limit the number of outbound connections.

[Michał Muskała]

When you have a HTML rendering website, the latency related to sending
email or doing any long-running tasks during requests is generally
unacceptable.
On the other hand when you're working with an API based SPA, it may be
completely fine to have a long-running request. The UI is still fully
responsive and by the time request finishes you can tell your user if
sending the email succeeded or failed instead of just saying "yeah,
we'll try to send you the email sometime soon" - this is a really
important advantage.

> https://groups.google.com/d/msgid/phoenix-talk/e5950b98-013d-4158-a27a-b12fde237e43%40googlegroups.com.

[micha...@gmail.com]

that is true,  we should be using a different example rather than emails such as doing a sleep for 3 minutes (basically something that takes a long time)

[Michał Muskała]

It's still completely fine to do a long-running request that is not
sending an email with an API if you have a meaningful response at the
end.
Instead of client doing two (or more) requests - to enqueue and check
status, you need just one.

> https://groups.google.com/d/msgid/phoenix-talk/df9fa382-6877-47f4-a700-b26aaf0d6856%40googlegroups.com.

[Stephen Schor]

I've been worked professionally on Rails applications for 8 years and have just started learning

Phoenix (as a gateway to Elixir) so please correct me if misspeak.

A lot of the appeal of Ruby's background/delay libraries is their immediate usefulness.

Sidekiq users aside - I don't think you'll find many people boasting about their jobs' parallelism.

But I also don't encounter other Ruby developers bemoaning the introduction of Redis.

Every use case is different - but I've found value in having a datastore (including DelayedJob's use of the DB table)

as a place, not just to pop items off a list, but to store state about the tasks....state that the popular libraries

make use of to govern priority and backoff policies. To say nothing of persistence in the face of a (OS) process dying during

deployments or machine reboots.

I'm curious if this perspective is a non-issue in Elixir.

I'm sure it's possible to store state in objects that represent tasks...or simply send code out to a managed background with

intelligence about retrying - but how much visibility do you have about the whole queue/(s) and deal with persistence.

Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/CAGAFNp%3D0g_W9UL3dDcJJbimOUpxmKkiT5ORAg%2B_Kq9vrcncCPQ%40mail.gmail.com.

[Peter Hamilton]

I feel like some in depth examples will be more useful to everyone. I'm going to put together a blog post showing how to do many standard (and non standard) tasks purely in Elixir with OTP. Stay tuned!

To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/CAHTvwu0DuAW9tSFy38fGdF0Mkdq2vh1MCT_hTAzW2SV6_K9SOA%40mail.gmail.com.

[Scott Ribe]

On May 25, 2016, at 2:50 PM, Michał Muskała <mic...@muskala.eu> wrote:
>
> It's still completely fine to do a long-running request that is not
> sending an email with an API if you have a meaningful response at the
> end.

Except that if you let an HTTP/S connection be idle for more than 30 seconds or so, you can run into all sorts of problems with gateways that terminate idle connections.

--
Scott Ribe
scott...@elevated-dev.com
http://www.elevated-dev.com/
https://www.linkedin.com/in/scottribe/
(303) 722-0567 voice