Subprocesses

289 views
Skip to first unread message

sri

unread,
Aug 29, 2016, 10:14:37 AM8/29/16
to Mojolicious
We've just released Mojolicious 7.04, and it contains a rather big new experimental feature.


Basically, this addresses a very common problem, and you can now use subprocesses to perform
computationally expensive operations without blocking or leaving the Mojolicious event loop. This
is also a pretty complicated feature, and we'd very much appreciate it getting tested in many
different environments.

--
sebastian

Stefan Adams

unread,
Aug 29, 2016, 11:51:20 AM8/29/16
to mojolicious

On Mon, Aug 29, 2016 at 9:14 AM, sri <kra...@gmail.com> wrote:
Basically, this addresses a very common problem, and you can now use subprocesses to perform
computationally expensive operations without blocking or leaving the Mojolicious event loop. This
is also a pretty complicated feature, and we'd very much appreciate it getting tested in many
different environments.

Woo hoo!!  GREAT!!

Alexey Stavrov

unread,
Aug 29, 2016, 12:32:03 PM8/29/16
to Mojolicious
What about to add pool of subprocesses? Pool would allow to use subprocesses safely in controllers.

понедельник, 29 августа 2016 г., 19:14:37 UTC+5 пользователь sri написал:

sri

unread,
Aug 29, 2016, 12:40:02 PM8/29/16
to Mojolicious
What about to add pool of subprocesses? Pool would allow to use subprocesses safely in controllers.

Please elaborate, what exactly makes the current approach unsafe?

--
sebastian 

Alexey Stavrov

unread,
Aug 29, 2016, 1:45:00 PM8/29/16
to Mojolicious
For example, server use subpress in controller:

#!/usr/bin/perl
use Mojo::Base -strict;

use Mojolicious::Lite;
use Mojo::IOLoop;

any
'/' => sub {
 
my $c = shift->render_later;
 
Mojo::IOLoop->subprocess(
   
sub {
      sleep
2;
   
},
   
sub {
      $c
->rendered(200);
   
});
};

app
->start;


Client create many connections to server and on each connection server make fork:

#!/usr/bin/perl
use Mojo::Base -strict;

use Mojo::IOLoop;
use Mojo::UserAgent;

my $d = Mojo::IOLoop->delay(sub {  say 'finish' });

my @uas;

for (0 .. 800) {
  push
@uas, my $ua = Mojo::UserAgent->new;
  $ua
->get('http://127.0.0.1:3000' => $d->begin);
}

$d
->wait;


So. server have many heavy processes.
If there was a pool of connections, then this would not be a problem.

понедельник, 29 августа 2016 г., 21:40:02 UTC+5 пользователь sri написал:

Alexey Stavrov

unread,
Aug 29, 2016, 1:48:41 PM8/29/16
to Mojolicious
Above I made a typo.
Not a pool of connections. I mean pool of subprocesses.

понедельник, 29 августа 2016 г., 22:45:00 UTC+5 пользователь Alexey Stavrov написал:

sri

unread,
Aug 29, 2016, 3:17:43 PM8/29/16
to Mojolicious
Client create many connections to server and on each connection server make fork...

What you're asking for is not a process pool but a process limit.

--
sebastian

sri

unread,
Aug 29, 2016, 3:24:23 PM8/29/16
to Mojolicious
What you're asking for is not a process pool but a process limit.

And i think for now we won't do that, but we'll keep an eye on it and maybe add it later once the approach has proven itself.

--
sebastian 

Tobias Oetiker

unread,
Aug 30, 2016, 2:37:12 AM8/30/16
to Mojolicious
Hi Sebastian,

I just looked at the source ... wow ... so simple ! very cool ...

I am itching to add some features to allow communication with the
subprocess while it is running ... and for the log messages from the sub process to
flow automatically into the masters log ...

would you be interessted in a PR, or do you already have plans for this ...

cheers
tobi

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch to...@oetiker.ch +41 62 775 9902

Alexey Stavrov

unread,
Aug 30, 2016, 7:04:58 AM8/30/16
to Mojolicious
Oh, yes, really. I think about process limit.
BTW, if you implement process pool then code will perform faster because not need always to create processes to perform operation (feature).

вторник, 30 августа 2016 г., 0:17:43 UTC+5 пользователь sri написал:

sri

unread,
Aug 30, 2016, 8:58:00 AM8/30/16
to Mojolicious
I am itching to add some features to allow communication with the
subprocess while it is running ...

For now that might be incompatible with the project goals, which are to make subprocesses as simple
and solid as possible. But i'd be interested to see how this could be implemented, especially
considering that there's no event loop running in the forked process.
 
and for the log messages from the sub process to
flow automatically into the masters log ...

This should not be a problem, Mojo::Log is already being used with Mojo::Server::Prefork.

--
sebastian 

Tobias Oetiker

unread,
Aug 30, 2016, 9:07:30 AM8/30/16
to Mojolicious
I will create a PR and then you can shred it ... :)

cheers
tobi

sri

unread,
Aug 30, 2016, 9:23:04 AM8/30/16
to Mojolicious
I will create a PR and then you can shred it ... :)

Or better yet, share your code on IRC so we can discuss it. ;)

--
sebastian 

David Elkins

unread,
Sep 7, 2016, 2:46:53 AM9/7/16
to mojol...@googlegroups.com
This is just fantastic!

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscribe@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

sri

unread,
Dec 18, 2016, 12:02:39 PM12/18/16
to Mojolicious
The subprocess feature is no longer experimental.


And there's now a recipe in the cookbook.


--
sebastian

Alexis Allot

unread,
Jan 9, 2017, 10:08:44 AM1/9/17
to Mojolicious
I'm worried about the comment from Alexey Stavrov. Does it mean that anyone who uses subprocesses exposes not only his app but also the server machine to easy DOS attacks, by allowing anyone to generate thousands of processes on the server ? Is there a way to prevent this in the current implementation ?

sri

unread,
Jan 9, 2017, 10:35:23 AM1/9/17
to Mojolicious
I'm worried about the comment from Alexey Stavrov. Does it mean that anyone who uses subprocesses exposes not only his app but also the server machine to easy DOS attacks, by allowing anyone to generate thousands of processes on the server ? Is there a way to prevent this in the current implementation ?

Your question sounds a bit hyperbolic, but i'll answer anyway. No, by using subprocesses you will not necessarily make your
server easy to DOS. It all depends on what your subprocesses are actually doing and how your server is configured. By
default the Mojolicious web servers will only accept up to 1000 connections per worker process, and Perl processes tend to
be pretty small thanks to copy on write.

But of course it is always good to be careful. We welcome anyone who is sincerely concerned about subprocess limits to
make a real proposal we can put up for a vote.

--
sebastian
Reply all
Reply to author
Forward
0 new messages