How to get the result of a finished Minion job back to the app

[Per Carlson]

Hi.

I'm thinking of using Minion as a job queue in an existing Mojolicious application. The concept of handing off a job to Minion is straight forward, but how to get the result back?

Here's a small (not working) example application:

#!/usr/bin/env perl

use Mojolicious::Lite;
use Net::Ping;

plugin Minion => { File => "minion.db" };
 
app->minion->add_task(ping => sub {
  my ($job, $host) = @_;

  my $p = Net::Ping->new;
  $p->port_number(22);

  if ($p->ping($host)) {
    $job->finish("Host $host is alive");
  } else {
    $job->fail("Host $host doesn't respond");
  }
  return;
});

get '/ping/(:host)' => sub {
  my $c = shift;

  my $jobid = $c->minion->enqueue(
    ping => [ $c->param('host') ]
  );
  $c->render_later;

  # Register event handlers to get the result/error

  # from Minion

  my $job = $c->minion->job($jobid);

  $job->on(error => sub {
    my ($job, $err) = @_;
    $c->render(status => 500, text => $err);
  });

  $job->on(finish => sub {
    my ($job, $result) = @_;
    $c->render(text => $result);
  });
};

app->start;

​The above doesn't work at all. I suspect that's because the worker is running in another process than the app, and doesn't share the same event scopes.

Is there another (working) way to retrieve the result in the same route​?

--

Pelle

Research is what I'm doing when I don't know what I'm doing.
- Wernher von Braun

[Glen]

The main intent of minion is to execute long running, blocking tasks that would take longer than the life cycle of a request, so return values aren't a feature.

Think of it as a fire and forget in terms of the initiating request, and you can check the status of it in subsequent requests.

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[sri]

​The above doesn't work at all. I suspect that's because the worker is running in another process than the app, and doesn't share the same event scopes.

I've updated the event descriptions to make the behavior more obvious.

    https://github.com/kraih/minion/commit/3f10630d154b7b7c914719900c9758d4591a36fc

--

sebastian 

[sri]

The main intent of minion is to execute long running, blocking tasks that would take longer than the life cycle of a request, so return values aren't a feature.

 

They actually have become a (minor) feature recently, since it was convenient to turn the old error field into a generic result field. But we do indeed not support async notifications for job state transitions, for the reason you mentioned.

--

sebastian

[Per Carlson]

Hi Glen and Sebastian.

Thanks for clarifying this in text and POD.

I'm more or less exploring the possibilities of Minion. If it were possible to get async notifications back, this would be a magnificent way to bridge the sync and async worlds!

For most of my intended uses of Minion, it's perfectly fine to enqueue a job, keep track of the jobid and return from the route. But there are also some shorter blocking operations that I was curious if it could handle.

--

You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[sri]

I'm more or less exploring the possibilities of Minion. If it were possible to get async notifications back, this would be a magnificent way to bridge the sync and async worlds!

It used to be supported in the beginning, but didn't fit in very well.

    https://github.com/kraih/minion/commit/2399f7b94be5a2e611b17df2c6b8e3b5eba040cb#diff-b30d9d6fe78ad2d38daed2446ba66d92L321

For most backends it would have to be done with a recurring timer trying to pull the result in regular intervals anyway. Which you can of course implement yourself as a helper in just a few lines too.

--

sebastian

[John]

On 01/27/2015 05:23 AM, sri wrote:

For most backends it would have to be done with a recurring timer trying to pull the result in regular intervals anyway. Which you can of course implement yourself as a helper in just a few lines too.

Maybe I did this in an especially ugly way.  With Websockets I want real time updates to client:

    https://github.com/john-/rtl_fm-scanner/blob/audio_manager/bin/rtl_fm-scanner#L180

Ideally it would be nice for the route to receive a message when something interesting happens (determined by logic associated with the long running process).

John

[sri]

For most backends it would have to be done with a recurring timer trying to pull the result in regular intervals anyway.

This is actually the case for both backends included with Minion, DBM::Deep has no notification mechanism, and PostgreSQL is not good enough at high concurrency yet. You'd need a specialized backend for push notifications, like Redis.

--

sebastian 

[John]

On 01/27/2015 05:23 AM, sri wrote:

The first two mojolicious applications I have written now have the same pattern:  Web Socket client that should get instant notification when something managed by that same application generates events of interest to the client.

It would seem some kind of publish/subscribe type thing would be used here.  Can a "third party" module be grafted in or is there some fundamental reason why these kinds of applications aren't cleanly implemented?

John

[sri]

It would seem some kind of publish/subscribe type thing would be used here.  Can a "third party" module be grafted in or is there some fundamental reason why these kinds of applications aren't cleanly implemented?

Of course, in fact i'm considering the addition of a much easier to use pub/sub system to Mojo::Pg for example.

    https://gist.github.com/anonymous/9d6b6d9f049a8ce12931

--

sebastian

[sri]

    https://gist.github.com/anonymous/9d6b6d9f049a8ce12931

There's a pubsub branch for Mojo::Pg too now.

    https://github.com/kraih/mojo-pg/compare/pubsub

I wouldn't hold my breath though, as long as reconnects are this fragile there is not much of a chance for it to get merged.

--

sebastian