announcing the new EV module, a modern redesign of Event
Marc Lehmann
I used Event for many, many years, and, while its easily the best event
loop out there, I had a number of issues with it (mostly scalability and
bugs that the not-very-active maintainer doesn't fix, but also performance
problems and specific limitations for my uses).
Recently I set out to write an interface to libevent, but since it had the
same kind of artificial limitations and bugs as Tk (which has about the
worst event loop possible :), I created my own replacement for libevent
called libev, and based my perl module on it (it also beats libevent
hands-down, see http://libev.schmorp.de/bench.html).
The perl interface to libev is called EV, is on CPAN, and is of course
not yet as mature as Event, but is used in production already in some
commercial apps, the deliantra game server (and the underlying libev
library is used in the gnu virtual private ethernet and rxvt-unicode
packages). It supports select, poll, epoll, kqueue (on the few platforms
where it works, i.e. "netbsd only") and solaris ports as backends.
Documentation for the perl module: http://cvs.schmorp.de/EV/README
Documentation for libev library: http://cvs.schmorp.de/libev/ev.html
All this is something I wanted to do for many years, but the effort
required kept me from doing it (originally I wanted to take Event and
improve it, it being the best event loop after all, but this didn't turn
out to be reasonable, for example, changing watcher lifetime semantics
would obvioously not be reasonable).
Here is a partial list of (perceived or real) problems or lacks of
features that EV improves upon over Event:
- watcher creation is very fast (about 25 times(!) faster than with
Event), making it feasible to create and tear down watchers quickly.
I measured creating and destroying a timer with EV, AnyEvent+EV, Event
and Glib, and also compared to just using $ev_timer->set+start (EV_set).
AnyEvent: 3 wallclock secs ( 3.09 usr + 0.00 sys = 3.09 CPU) @ 323624.60/s (n=1000000)
EV: 0 wallclock secs ( 0.95 usr + 0.00 sys = 0.95 CPU) @ 1052631.58/s (n=1000000)
EV_set: 0 wallclock secs ( 0.36 usr + 0.00 sys = 0.36 CPU) @ 2777777.78/s (n=1000000)
Event: 40 wallclock secs (39.61 usr + 0.01 sys = 39.62 CPU) @ 25239.78/s (n=1000000)
Glib: 3 wallclock secs ( 3.66 usr + 0.01 sys = 3.67 CPU) @ 272479.56/s (n=1000000)
- EV watchers are normal perl objects, no need for explicit destruction
(this is slightly inconvinient for unrealistically simple programs, but
much simpler for real-world cases).
- it copes with timejumps, both forward and backward (how well it does
that depends on the availability of a monotonic clock, but it
copes even in its absence).
- it has usable "hard realtime" timers (the hard realtime timers in Event
will go into an almost endless loop on timejumps and other delays and
make them completely useless, as you have to implement your own
scheduling anyways to make them usable).
- it separates timers into timers based on wallclock time ("periodics")
and timers based on real time. The former is good for cron-like timers,
the latter is good for delays and timeouts. The fact that you cannot do
either with Event always made me uneasy, and also ruled out Event usage
in some very important use cases, as freezes due to ntp clock resets are
unacceptable sometimes.
- all operations are either O(1) or O(log n), not O(n) (n = #watchers)
or worse as in Event. If you have a lot of timers or I/O watchers this
makes an extreme difference, as Event walks its timer and I/O list on
every poll call, while in libev, this operation is O(1).
- It uses *much* less memory per watcher than Event (honestly, I never
saw that as a problem with Event, but in big programs, it made a very
noticable difference in ps output).
- there cannot be endless resource overflow (the event loop in Event
can grow endlessly without ever having a chance to reduce its size,
freezing the program on certain conditions). This is a real big
problem with Event, which basically makes its priority system useless.
- the main loop is reentrant w.r.t. coroutines.
- it has support for child process status change events.
- EV is much more parsimonous of system calls.
- events have timestamps (at no additional cost. This is useful for more
exact scheduling of timeouts for example, or simply to see by how much
event processing is delayed).
- the design is still open to some discussion and feature requests,
so feel free to ask for things and I might be able to provide them.
It also has a full-featured C API (which Event also has, but some of the
quirks that made it hard to use are gone), delivering the full libev API
to any extensions.
AnyEvent is fully supported (ok, this is not surprising, being from the
same author), so modules using AnyEvent can take advantage of it without
thinking a second.
I also tried to preserve all the good things about Event (which after all,
was my Event module of choice for almost a decade), and decided that sharing
EV with everybody else might be helpful to some. I hope thats true.
(Some of the good things in Event are purely ingenious, for example, the
check and prepare hooks, which allow easy integration of many C and Perl
libraries into Event, something which I have not yet seen in other event
loop implementations, but which I think are absolutely essential. Event
has a number of those concepts not found elsewhere that make it very
mature and general).
EV is available on CPAN, its homepage is linked from http://libev.schmorp.de.
To give you an impression of how the API looks like, here are some examples
from the manpage:
# TIMERS
my $w = EV::timer 2, 0, sub {
warn "is called after 2s";
};
my $w = EV::timer 2, 2, sub {
warn "is called roughly every 2s (repeat = 2)";
};
undef $w; # destroy event watcher again
my $w = EV::periodic 0, 60, 0, sub {
warn "is called every minute, on the minute, exactly";
};
# IO
my $w = EV::io *STDIN, EV::READ, sub {
my ($w, $revents) = @_; # all callbacks receive the watcher and event mask
warn "stdin is readable, you entered: ", <STDIN>;
};
# SIGNALS
my $w = EV::signal ’QUIT’, sub {
warn "sigquit received\n";
};
# CHILD/PID STATUS CHANGES
my $w = EV::child 666, sub {
my ($w, $revents) = @_;
my $status = $w->rstatus;
};
# MAINLOOP
EV::loop; # loop until EV::unloop is called or all watchers stop
EV::loop EV::LOOP_ONESHOT; # block until at least one event could be handled
EV::loop EV::LOOP_NONBLOCK; # try to handle same events, but do not block
And here is a rough example of how to integrate e.g. Net::SNMP, by
registering a prepare (called before blocking) and check (called after
blocking) watcher to create and tear down watchers for the sockets
Net::SNMP uses as well as the timeout it needs:
our @snmp_watcher;
our $snmp_prepare = EV::prepare sub {
# do nothing unless active
$dispatcher->{_event_queue_h}
or return;
# make the dispatcher handle any outstanding stuff
... not shown
# create an IO watcher for each and every socket
@snmp_watcher = (
(map { EV::io $_, EV::READ, sub { } }
keys %{ $dispatcher->{_descriptors} }),
EV::timer +($event->[Net::SNMP::Dispatcher::_ACTIVE]
? $event->[Net::SNMP::Dispatcher::_TIME] - EV::now : 0),
0, sub { },
);
};
our $snmp_check = EV::check sub {
# destroy all watchers
@snmp_watcher = ();
# make the dispatcher handle any new stuff
... not shown
};
Thanks for your consideration.
--
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / p...@goof.com
-=====/_/_//_/\_,_/ /_/\_\
Jochen Stenzel
thank you for EV and your detailed intro. I think it's great that you
are engaged in the improvement of event loops as you have a detailed
knowledge of related libraries and system calls and a great experience
in using the various loops. Although I did not have too many problems
with Event and really like to use it, I think it's very worth to have a
look at EV, too.
As I am curious I just started to play with EV. I think I just scratched
its surface, but nevertheles possibly my first impressions might be helpful.
* I think it's great to have Windows support from the beginning, so
programs using EV can be expected to be portable between UNIX and
Windows, similar to the current version of Event.
* For the interface, I like the named parameters interface of Event.
Of course this is a matter of taste (and performance), so this is
just a personal note.
* I found that the timers are not restricted to full second
intervals but support higher resolution. I suggest to mention this
in the docs which currently use to speak of "seconds" (which is
correct but does not necessarily imply that fractions will work as
well).
* Speaking about the docs I find it a bit confusing to mix the
descriptions of methods and watcher constructors. I would suggest
to have a section for the constructors, and another one for the
methods.
* The example provided for data() causes a warning (if run with
PxPerl (a perl 5.8.7) under Windows XP SP 2). This happens because
data() supplies an undefined value, so it seems either setting the
data value or retrieving it does not work. This was tried with
timers, periodic and idle watchers.
* A signal watcher for the "INT" signal causes a Windows exception,
so Windows displays a dialog reporting that perl caused an error
and has to be terminated. Interestingly, the event loop continues
to work while the dialog is presented, so as long as one defers to
press "OK" the callbacks for timers and periodic watchers continue
to be called.
* An io watcher for STDIN does not work (in the mentioned Windows
environment): terminal input does not reach the program, instead
it is propagated to the command like processor after program
termination.
* The STDIN io watcher causes a very high load (100%).
Here is my short test script:
- snip --
# pragmata
use strict;
use warnings;
# modules
use EV;
# install watchers
my $wTimer=EV::timer_ns(10, 10, \&timerCallback);
$wTimer->data('timer');
my $wPeriodic=EV::periodic_ns(4, 4, undef, \&periodicCallback);
$wPeriodic->data('periodic');
my $wSignal=EV::signal_ns('INT', \&signalCallback);
$wSignal->data('signal');
my $wIO=EV::io(\*STDIN, EV::READ, \&ioCallback);
$wIO->data('io');
#my $wIdle=EV::idle(\&idleCallback);
#$wIdle->data('signal');
# loop
EV::loop();
# callbacks
sub timerCallback
{print 'Timer watcher data: ', $_[0]->data(), "\n";}
sub periodicCallback
{print 'Periodoc watcher data: ', $_[0]->data(), "\n";}
sub signalCallback
{print 'Signal watcher data: ', $_[0]->data(), "\n";}
sub ioCallback
{
my $userData=<STDIN>;
print 'IO watcher data: ', $_[0]->data(), "\nRead: $userData";
}
sub idleCallback
{print 'Idle watcher data: ', $_[0]->data(), "\n";}
- snip --
Regards
Jochen
Marc Lehmann
> * I think it's great to have Windows support from the beginning, so
> programs using EV can be expected to be portable between UNIX and
> Windows, similar to the current version of Event.
It was a side effect of the portability requireemnts for libev :)
> * For the interface, I like the named parameters interface of Event.
> Of course this is a matter of taste (and performance), so this is
> just a personal note.
Oh, you can have the same interface, much higher portability (and slightly
reduced functionality) by using the AnyEvent module. That provides an
interface very similar to Event but transparently works on Glib, Tk,
Event, EV, Coro and even provides a portable perl event loop.
As you can see form the benchmarks, while AnyEvent is extremely slow
compred to native EV, it is still much faster than Event, and still
quite a bit faster than e.g. Glib, so efficiency should be of secondary
consideration.
> * I found that the timers are not restricted to full second
I will.
> * Speaking about the docs I find it a bit confusing to mix the
I will look into this.
> * The example provided for data() causes a warning (if run with
Oh right, I forgot to implement that (I don't use this kind of interface
myself :) Well spotted!
> * A signal watcher for the "INT" signal causes a Windows exception,
While some effort has been made to be able to catch (some) signals under
windows, microsoft claims that windows doesn't support signals, so I guess
the first step would be to make windows support it?
In any case, I canot really reproduce this with my activestate perl, so if
you know of a way to catch signals reliably on windows (according to msn
there isn't), then you should speak up and probably implement it :)
signals are a unix/posix thing, and while EV tries, of course this has to
be supported in the OS somehow.
> * An io watcher for STDIN does not work (in the mentioned Windows
> environment): terminal input does not reach the program, instead
> it is propagated to the command like processor after program
> termination.
Windows doesn't support this operation, select (both the perl one and the
windows one) works only on sockets. Since the select backend is used, and
libev happily uses perls select, you would have to first teach windows
about supporting select on the console:
Windows itself provides no alternative mechanism to do readyness
notifications (its I/O model if there were any, does not support
non-blocking operations, but is centered around asynchronous operations,
and the various WaitFor functions are not generic enough).
Either you use sockets, or you lose, and no event library can do anything
about it (this becomes more complicated as windows doesn't even support
the notion of a shared handle space, such as fds, so the fact that perl
provides fileno and other unix-stuff on windows cannot hide the fact that
this is part of a unix emulation, and it breaks down here and in many
other places).
Also, ignoring the fact that libev is also a C library, in case of EV, it
has to handle perl fds. And it does handle perl fds just like perl, which
IMHO is the right thing to do. Perl fds are not neccssarily a concept
known to windows (at best perl uses the msvcrt).
If you need good posix support, you have to use a posix operating system,
or an emulation layer like cygwin, where select actually works on stdin.
> * The STDIN io watcher causes a very high load (100%).
Your callback doesn't check for errors (if windows provides any you will
get an revents with EV::ERROR set). libev tries to detect bad fds, but
this hasn't been tested very well on windows, either. In any case, EV
cannot do what the operating system itself does not support (or, in this
case, perl, as that is what is being used).
But this is probably OK, I cannot imagine anybody trying to do serious
development under windows, especially not with the half-baked unix
emulation perl provides. It simply doesn't work well to force one model
that works everywhere else onto windows where it doesn't. EV shares the
same limitations as perl and other programs suffer from, and there is
little one can do.
EV actually should work for sockets, the only part of windows where the
event model maps onto the EV/Event etc. model. Thats the only part that
can reasonably work.
Jochen Stenzel
> Here is my short test script:
>
methods. Of course intially these were the starting versions.
Jochen
Uri Guttman
>> * An io watcher for STDIN does not work (in the mentioned Windows
>> environment): terminal input does not reach the program, instead
>> it is propagated to the command like processor after program
>> termination.
ML> Windows doesn't support this operation, select (both the perl one and the
ML> windows one) works only on sockets. Since the select backend is used, and
ML> libev happily uses perls select, you would have to first teach windows
ML> about supporting select on the console:
ML> Windows itself provides no alternative mechanism to do readyness
ML> notifications (its I/O model if there were any, does not support
ML> non-blocking operations, but is centered around asynchronous operations,
ML> and the various WaitFor functions are not generic enough).
ML> Either you use sockets, or you lose, and no event library can do anything
ML> about it (this becomes more complicated as windows doesn't even support
ML> the notion of a shared handle space, such as fds, so the fact that perl
ML> provides fileno and other unix-stuff on windows cannot hide the fact that
ML> this is part of a unix emulation, and it breaks down here and in many
ML> other places).
i ran into this problem a while back when trying to make an async
console (stdio) for stem with event.pm. i also learn that redmond in
their infinite stupidity did not allow it. my solution was to 'fork'
reader and writer processes (of course this is faked with threads) and a
socketpair to each process shared in the main process. then select or
event will work on those sockets just fine. i did get it to work but i
think it has been broken for a while (i don't to winblows dev if i can
help it). so an idea for EV would be to write these short processes to
allow an async console. they are pretty much just a read/write loop with
a socket on one side and stdin/out on the other. no brains, just good
old blocking i/o with no size limits. then you can either trap a
watching of STDIN/OUT on winblows and fork these off or some other way
to allow them. this way we have a properly portable way to do consoles.
i will have to explore EV as soon as i can and wrap stem around it.
another event loop that needs proper wrapping is wx. i found that it
does NOT HAVE a way to add an external socket to its event loop. only
sockets created via wx can be added. both stem and poe resorted to using
polled timers to check sockets which sucks horribly. there needs to be a
wx (and wxperl wrapper) that allows sockets to be added. i create my own
socket connections for various reasons and didn't want to use wx's
api. i don't know c++ or wx's event guts (very confusing code down
there) enough to code this. if marc can't do it, maybe someone else can
help. it doesn't even need to be a true wx interface as long as there is
EV support so we can use its common API.
thanx,
uri
--
Uri Guttman ------ u...@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
Marc Lehmann
> their infinite stupidity did not allow it. my solution was to 'fork'
> reader and writer processes (of course this is faked with threads) and a
> socketpair to each process shared in the main process. then select or
> event will work on those sockets just fine.
Yeah, note that windows also has neither local sockets nor socketpair, but
fortunately, perl emulates socketpair by creating a tcp connection (which
is of course pretty slow and a scarce resource).
> help it). so an idea for EV would be to write these short processes to
> allow an async console. they are pretty much just a read/write loop with
> a socket on one side and stdin/out on the other. no brains, just good
> old blocking i/o with no size limits. then you can either trap a
> watching of STDIN/OUT on winblows and fork these off or some other way
> to allow them. this way we have a properly portable way to do consoles.
I don't think this belongs in EV for these reasons:
- its a pretty drastic semantics change
- perls pseudo-threads(windows fork emulation create a copy
of the whole interpreter, thats large resource wastage that shouldn't
be done lightly.
- despite your description of "no brains" (ehe :), it is far more
complicated: EV would, from C create and tear down those threads or, if its
perl threads, it would have to tear it down together with the main program,
as otherwise the program won't exit (and I found eof-detection in select
flaky on windows).
- it is an extremely rare case, and the much more prominent cases of breakage
(for example, windows can't do this on pipes either) wouldn't be helped by
that.
This would be extremely invasive, and would also require a lot of code
(where EV already has quite a lot of code specific to win32 to work
around its limitations and missing funcitonality.
So to me it clearly fails the is-it-reasonable test, especially as far
more important cases such as pipes are not covered by this and will still
not work.
> another event loop that needs proper wrapping is wx. i found that it
Well, this sounds more like a case of AnyEvent. The diea behind AnyEvent is:
1. you use AnyEvent instead of any other event toolkit in your module
2. it will adapt automatically to the event loop used
Of course, if there is no way to add your own sockets, this sucks
majorly. I can look into it, but I would of course prefer not to have xs
code in AnyEvent so its always easily installable.
Of course, the wx interface could also provide its own anyevent API
(rxvt-unicode does the same, AnyEvent doesn't support rxvt-unicode itself
but rxvt-unicode uses its hook-in mechanism to make it work), and all
problems would be solved.
> does NOT HAVE a way to add an external socket to its event loop.
While glib for example can do this, its not obvious either (the glib I/O
model is somehow modelled after the most common denominator of win32 and
posix, which is win32, and thus a bit confusing to posix users).
(Interestingly enough, one can wrap Glib around EV by simply loading the
Glib::EV module, which solves one way, but its not that efficient).
> api. i don't know c++ or wx's event guts (very confusing code down
> there) enough to code this. if marc can't do it, maybe someone else can
> help. it doesn't even need to be a true wx interface as long as there is
> EV support so we can use its common API.
You mean AnyEvent here. EV is as stubborn as wx or other event loops: it
needs to be in control. While it would actually be possible to use other
event loops by providing libev backends for it (after all, kqueue/epoll
etc. are also very simple "event loops"), I don't think thats the right
solution, although the idea is interesting.
If I find time, I'll have a look at the wx situation.
Marc Lehmann
I had a relatively cursory look at Wx. Indeed, not only can't I see any
way to add ones own fds, Wx apperently equates "socket" with "tcp v4
socket", as nothing else seems to be supported.
I don't know the rationale for this, but here's my take at it: Wx, like
many other programs, want to support unix and unfortunately have to
support windows, too.
As windows doesn't have the generic I/O model of unix, the smallest common
denominator is indeed the windows way, where each I/O channel has different
functions, which lends itself to a socket class that wraps around
the OS functions.
That way, you can support sockets in a platform-independent way, but of
course you have no generality at all.
Perl itself goes a slightly different route, with basically the same
result (but at least it could be improved): Perl emulates the generic unix
model (usually by using the fd model provided by msvcrt), wraps select
etc. but then "fails" to support anything besides sockets.
Both models support sockets only, but the perl model of course is
theoretically extendable: one *could* provide wrappers around pipes and other
sources, but that gets very slow (as seen in cygwin), and tedious, especially
if the OS itself doesn't really support the operation and you have to work
around.
For example, many I/O channels have no notion of readyness under windows, so
the while "tell me when I can read data" model falls down.
Glib for example also works around this by providing a windows-like model
without readyness events (which can be efficiently emulated under unix,
fortunately), doing its own I/O buffering and then faking its own readyness
events on top of it, again providing something like the unix model, by
totally reimplementing the windows I/O. And it also fails for many event
sources.
So, with Wx, it looks bleak: Wx doesn't even seem to have the notion of
file descriptors, so you lost.
Jochen Stenzel
two short notes, in a hurry: today I tried EV on Solaris 8 and found
that -lresolv needs to be added to the linker options in order to make
inet_aton() available. Perhaps this can be added to the default
configuration.
Then I run a simple loop in the debugger (on the same system), with just
one periodic watcher. Surprisingly, once started I could not interrupt
the loop via CTRL-C. This is different on Windows where I just retried
it (Windows, I know ;-) ) (and different to what I expected as typically
an uncatched signal is handled as usual). I cannot say if this is caused
by EV, Solaris or even the Solaris version.
Besides this, it worked well and with a very low load.
Greetings
Jochen
Jochen Stenzel
Hi Marc,
thank you for your long and detailed answer. Unfortunately I am in a
hurry these days, so sorry for the delay.
>> * For the interface, I like the named parameters interface of Event.
>> Of course this is a matter of taste (and performance), so this is
>> just a personal note.
>
> Oh, you can have the same interface, much higher portability (and slightly
> reduced functionality) by using the AnyEvent module. That provides an
> interface very similar to Event but transparently works on Glib, Tk,
> Event, EV, Coro and even provides a portable perl event loop.
Ok, I wasn't aware of this specific fact, thank you for the hint.
>> * Speaking about the docs [...]
>
> I will look into this.
Thanks a lot, I had a look at the docs in version 1.5 and could follow
the structure much easier.
>> * A signal watcher for the "INT" signal causes a Windows exception,
>
> While some effort has been made to be able to catch (some) signals under
> windows, microsoft claims that windows doesn't support signals, so I guess
> the first step would be to make windows support it?
>
> In any case, I canot really reproduce this with my activestate perl, so if
> you know of a way to catch signals reliably on windows (according to msn
> there isn't), then you should speak up and probably implement it :)
>
> signals are a unix/posix thing, and while EV tries, of course this has to
> be supported in the OS somehow.
OK, I will check if this runs on Strawberry Perl. While it was a very
handy distribution PxPerl is no longer maintained, so if the failure is
specific to that compilation there is no urgent need to fix it.
But perhaps ... could there be be something like a "portability" section
in the docs? Just as an idea, as without such notes a reader without
special Windows API insight could expect things to work exactly as
described, so such a section could be helpful.
For the general signal handling on Windows, I did not use the Windows
API for a few years now and of course I trust you for all the details,
but I am a bit irritated to hear it should not work in general as from a
users point of view I know that %SIG worked under DOS and Windows, at
least for SIGINT, so I suppose a standard Perl user would just "expect"
things to work to this degree, or again I suggest to add a compatibility
note. But the failure might be very specific to PxPerl ...
>
>> * An io watcher for STDIN does not work (in the mentioned Windows
>> environment): terminal input does not reach the program, instead
>> it is propagated to the command like processor after program
>> termination.
>
> Windows doesn't support this operation, [detailed explanation]
Thanks for this description, I wasn't aware if this.
>> * The STDIN io watcher causes a very high load (100%).
>
> Your callback doesn't check for errors (if windows provides any you will
> get an revents with EV::ERROR set).
Yes, it was a very simple callback indeed.
For the error checking, sorry to mention the docs once more but could it
... ? ;-)
> libev tries to detect bad fds, but
> this hasn't been tested very well on windows, either. In any case, EV
> cannot do what the operating system itself does not support (or, in this
> case, perl, as that is what is being used).
>
> But this is probably OK, I cannot imagine anybody trying to do serious
> development under windows, especially not with the half-baked unix
> emulation perl provides. It simply doesn't work well to force one model
> that works everywhere else onto windows where it doesn't. EV shares the
> same limitations as perl and other programs suffer from, and there is
> little one can do.
Ok, thanks again for pointing this out. To sum it up, I really think
these system dependencies are worth to be mentioned in the manual, in
order to make it clear to future users what they can expect on a certain
system.
Greetings
Jochen
Marc Lehmann
> two short notes, in a hurry: today I tried EV on Solaris 8 and found
> that -lresolv needs to be added to the linker options in order to make
> inet_aton() available. Perhaps this can be added to the default
Its actually quite rare to not have -lresolv in LIBS already as perl
itself usually adds it. In any case, its a common enough problem for
solaris admins so they know it.
> Then I run a simple loop in the debugger (on the same system), with just
> one periodic watcher. Surprisingly, once started I could not interrupt
> the loop via CTRL-C.
That is interesting. I am not aware that solaris event ports would block
any signals, and in fatc, I cannot reproduce this. Are you sure that your
tty settings actually do anything with CTRL-C (such as sending an INT)?
> it (Windows, I know ;-) ) (and different to what I expected as typically
> an uncatched signal is handled as usual). I cannot say if this is caused
> by EV, Solaris or even the Solaris version.
I'd say check your intr setting first to see if ^C really sends an INT in
the first place.
> Besides this, it worked well and with a very low load.
Thats good (although the event ports aren't really that well
designed. They require much more cpu power for programs not using a lot of
inactive fds, compared to e.g. poll, and they tend to geenrate spurious
readyness events, further adding load).
Thanks anyways! And please investigate the SIGINT issue, as neither EV
nor perl should do anything with it, and event ports should not have any
effects either.
Marc Lehmann
> OK, I will check if this runs on Strawberry Perl. While it was a very
> handy distribution PxPerl is no longer maintained, so if the failure is
> specific to that compilation there is no urgent need to fix it.
>
> But perhaps ... could there be be something like a "portability" section
> in the docs? Just as an idea, as without such notes a reader without
> special Windows API insight could expect things to work exactly as
> described, so such a section could be helpful.
The problem isn't the windows API, which EV doesn't use, but the perl
API, which EV has to use on windows, because windows doesn't have a file
descriptor concept. The perl API is a problem because it doesn't exist
in a stable form. I can only support the single perl I sometimes test with
(which is activestate perl).
I cannot say anything else about other perls, thats not really my job and not
really in my abilities.
If you want to use the POSIX model under windows, you will necessarily
have to learn a lot about the limitations of your particular perl,
but that doesn't belong in the EV documentation, but in the perl
documentation, because thats where it is implemented.
> but I am a bit irritated to hear it should not work in general as from a
You have to take that up to microsoft, not me, however, as I am not
responsible for that (check up on msdn for details).
> > Windows doesn't support this operation, [detailed explanation]
>
> Thanks for this description, I wasn't aware if this.
In fact, windows *does* support this operation, as console handles
are special. It does not, however, work for anything else (pipes
etc.). Windows also has the limitation of a maximum of 64 objects that you
can wait on at the same time (this seems to be a kernel limitation even in
current versions of windows).
All EV (and perl) can do is provide some very crude and limited emulation for
something windows ultimately doesn't provide.
> >> * The STDIN io watcher causes a very high load (100%).
> >
> > Your callback doesn't check for errors (if windows provides any you will
> > get an revents with EV::ERROR set).
>
> Yes, it was a very simple callback indeed.
>
> For the error checking, sorry to mention the docs once more but could it
> ... ? ;-)
I think it is clearly documented in the libev documentation (which also
has a lot of portability hints). The EV module only describes the perl
interface.
> Ok, thanks again for pointing this out. To sum it up, I really think
> these system dependencies are worth to be mentioned in the manual,
The problem is that with "system" you mean "all the various flavours of perl
under windows", and I think this clearly belongs into the documentation for
that relevant flavour.
(libev also has a native win32 for use outside perl, but it is not used in
EV because it has to be compatible to perl itself).
> order to make it clear to future users what they can expect on a certain
> system.
I think thats already done, the libev manual is quite saturated with
portability hints and backend characteristics.
--
The choice of a
-----==- _GNU_
----==-- _ generation Marc Lehmann
---==---(_)__ __ ____ __ p...@goof.com
--==---/ / _ \/ // /\ \/ / http://schmorp.de/
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE
Jochen Stenzel
Hello Marc,
>> Then I run a simple loop in the debugger (on the same system), with just
>> one periodic watcher. Surprisingly, once started I could not interrupt
>> the loop via CTRL-C.
>
> That is interesting. I am not aware that solaris event ports would block
> any signals, and in fatc, I cannot reproduce this. Are you sure that your
> tty settings actually do anything with CTRL-C (such as sending an INT)?
I am - I retried it today, other programs receive the signal and the
effect is the same if I send the signal via kill from another terminal.
I know the following is an incomplete report as it does not contain
exact system information (in general, Solaris 5.8) and perl version (in
general, 5.8.6 without thread support), but today the picture was a bit
different:
* I *could* stop the loop in the debugger today, but it took a while:
DB<2> sub d {print "===> TIME: ", time, "\n";}
DB<3> d
===> TIME: 1196849803
DB<4> use EV
DB<5> $p=EV::periodic(0, 10, 0, sub {print "Periodic call at ", time,
".\n"})
DB<6> d(); EV::loop
===> TIME: 1196849839
Periodic call at 1196849839.
Periodic call at 1196849840.
Periodic call at 1196849850.
Periodic call at 1196849860.
Periodic call at 1196849870.
^Cd
Periodic call at 1196849880.
EV::loop((eval 32)[.../perl5db.pl:628]:2):
2:
DB<<7>> d
===> TIME: 1196849890
* so, CTRL-C worked this time ...
* ... but not immediately (as usual, instead it took until the next
watcher event, plus the time to the next event check (as it seems)).
This was verified with longer intervals and it was reproducable.
* In contrast, with an additional signal watcher the signal is
handled instantly.
* By the way, there *were* cases today when the signal was ignored
completely, this time in a script where I had to add an explicit
signal watcher to make it interruptable at all. I hope to shrink
this down to a short demo script to send.
Another interesting effect you might be interested in was this:
DB<1> use EV
DB<2> $p=EV::periodic(0, 2, 0, sub {print "Periodic call with short
intervall at ", time, ".\n"})
DB<3> EV::loop
Periodic call with short intervall at 1196850121.
Periodic call with short intervall at 1196850122.
Periodic call with short intervall at 1196850124.
Periodic call with short intervall at 1196850126.
^CPeriodic call with short intervall at 1196850128.
EV::loop((eval 23)[.../perl5db.pl:628]:2):
2:
DB<<4>> $p=EV::periodic(0, 20, 0, sub {print "Periodic call with
longer intervall at ", time,
".\n"})
DB<<5>> EV::loop
Periodic call with longer intervall at 1196850145.
Periodic call with short intervall at 1196850145.
Periodic call with short intervall at 1196850146.
Periodic call with short intervall at 1196850148.
Periodic call with short intervall at 1196850150.
Periodic call with short intervall at 1196850152.
Periodic call with short intervall at 1196850154.
Periodic call with short intervall at 1196850156.
Periodic call with short intervall at 1196850158.
Periodic call with longer intervall at 1196850160.
Periodic call with short intervall at 1196850160.
^CPeriodic call with short intervall at 1196850162.
So, a periodic watcher with a short interval was assigned to a variable,
then the loop was started and interrupted via CTRL-C. After that a new
watcher was installed and assigned to the same variable, which - as I
understood the docs - should uninstall the first watcher as now there
was no further reference to the first watcher object. But when running
the loop again *both* watchers were present.
Finally, in both traces one sees a first callback invocation that does
not fit into the periodic scheme.
I am not sure when I wil have time to send a demo script for the
mentioned case of an ignored interrupt but I hope the traces give you a
first impression.
Jochen
Rusty Conover
the I/O timers don't have a nice way to schedule both a timeout and a
I/O callback at the same time into the same callback.
Such as setting the timeout parameter to the IO watcher.
Is this just my oversight or a design decision?
Thanks,
Rusty
Marc Lehmann
> I'm looking into using EV rather then Event for my needs but it seems
> the I/O timers don't have a nice way to schedule both a timeout and a
> I/O callback at the same time into the same callback.
Thats a feature, and compared to Event, actually a bugfix.
> Such as setting the timeout parameter to the IO watcher.
>
> Is this just my oversight or a design decision?
Its a design decision. Combining timers with io watchers is unnatural,
because:
- you often don't need both, so shouldn't be forced to pay for them
- you can't choose the type of timeout (event only has absolute timers
anyways, though, which are unsuitable for many tasks such as idle timeouts
for i/o, where you need relative timeouts).
- you have to manage two timeouts or weirder stuff when you have
two separate i/o watchers, or use a third watcher (paying even more).
- it is often unclear on which events exactly the timeout gets reset
(if at all), and even if not, there is no flexibility (how do you
chose between an idle timeout and an overall timeout
for completion with Event? you cannot).
As it is trivial to create a separate timeout (and much faster than with
Event, too), there is no drawback to this design.
It also usually leads to smaller and simpler code, once one understands
ones own needs...
Marc Lehmann
> > any signals, and in fatc, I cannot reproduce this. Are you sure that your
> > tty settings actually do anything with CTRL-C (such as sending an INT)?
>
> I am - I retried it today, other programs receive the signal and the
> effect is the same if I send the signal via kill from another terminal.
Here is what happens on windows btw: at least activestate perl overrides
the system console handler and thus disables those signals. As it doesn't
provide signal emulation for other programs you cnanot catch those signals
anymore, and there is nothing one can do about it. Or to put it in other
words, perl actually emulates those signals in perl without emulating them
in the C level - nothing an event loop can do about that, thats a limitation
of at leats activestate perl.
> I know the following is an incomplete report as it does not contain
> exact system information (in general, Solaris 5.8) and perl version (in
Since there is no solaris 5.8 version in existance, I guess you mean
solaris 8 (that could make a difference as I only test on solaris 10).
> * By the way, there *were* cases today when the signal was ignored
> completely, this time in a script where I had to add an explicit
> signal watcher to make it interruptable at all. I hope to shrink
> this down to a short demo script to send.
Indeed, that would be helpful.
However, are you sure you don't provide your own signal handlers via %SIG
and you really have a signal handler installed for SIGINT? Either would
actually perfectly explain the behaviour you are seeing. %SIG cannot work
because in newer perls this is only handled when the perl interpreter gets
control (which cannot be the case unless you run some watchers). The whole
point of an event loop is to shield you from asynchronous signals, you
have to embed them into the event loop (thats not specific to EV, other
event loops work the same way).
If you cannot install signal handlers properly you could try to install a
check watcher, which runs at every loop iteration, and hope that whoever
installs its own signal handler doesn't make syscalls restartable, in
which case you have lost anyways.
> DB<2> $p=EV::periodic(0, 2, 0, sub {print "Periodic call with short intervall at ", time, ".\n"})
Here you store one reference into $p, the other is kept by the debugger.
> DB<<4>> $p=EV::periodic(0, 20, 0, sub {print "Periodic call with > longer intervall at ", time, ".\n"})
> Periodic call with longer intervall at 1196850145.
> Periodic call with short intervall at 1196850145.
Since you only overwrote one reference here, both watchers are still
active.
Watchers only get destroyed when _all_ refererences are broken, as with
any other perl object (the same is true with Event, except that you
additionally need an additional explicit cancel call with Event).
> was no further reference to the first watcher object. But when running
> the loop again *both* watchers were present.
Yup, because both are alive.
Try outside the debugger, that will be less confusing (also, it would help
if you wouldn't wrap codelines in your emails :).
Try e.g.:
use EV;
$p=EV::periodic(0, 2, 0, sub {print "Periodic call with short intervall at ", EV::now, ".\n"});
$t = EV::timer 3,0,sub { EV::unloop };
EV::loop;
$p = EV::periodic(0, 20, 0, sub {print "Periodic call with longer intervall at ", EV::now, ".\n"});
EV::loop;
Here it is easy to see when values get destroyed.
As an additional hint, you should give EV::now a try: it is faster then
time (no syscall) and more accurate in the sense that it returns the event
timestamp (while time returns the current time, which might or might not
be what you are interested in).
> Finally, in both traces one sees a first callback invocation that does
> not fit into the periodic scheme.
That "one" must have rather magical abilities to see anything when you
round your time and do not use the event time but some other time :->
Try using EV::now and you will see that the timestamps are actually
correct. Here are a few sample runs with both EV::now and time to
elaborate this:
cerebro /tmp# perl x.pl
Periodic call with short intervall at 1196906694.00098 1196906693.
Periodic call with longer intervall at 1196906700.00108 1196906699.
cerebro /tmp# perl x.pl
Periodic call with short intervall at 1196906706.00513 1196906706.
Periodic call with short intervall at 1196906708.00117 1196906707.
Periodic call with longer intervall at 1196906720.00134 1196906719.
Periodic call with longer intervall at 1196906740.0056 1196906739.
You could alternatively use Time::Hires, which also doesn't round:
cerebro /tmp# perl x.pl
Periodic call with short intervall at 1196906782.00217 1196906782.00224.
Periodic call with short intervall at 1196906784.00219 1196906784.00222.
cerebro /tmp# perl x.pl
Periodic call with short intervall at 1196906796.00233 1196906796.00241.
Periodic call with longer intervall at 1196906800.00238 1196906800.00242.
Periodic call with longer intervall at 1196906820.00265 1196906820.00268.
In any case, you would see the same behaviour with Event, as it uses
Time::HiRes internally and is therefore also more accurate than time().
> I am not sure when I wil have time to send a demo script for the
> mentioned case of an ignored interrupt but I hope the traces give you a
> first impression.
So far, it seems everything works right. While EV is young and certainly
has some bugs, it is used in a lot of projects, and really can't be
_totally_ broken.
So, when you see totally off behaviour like above, first give your test
script a bit of thinking, and try to find out wether your test actually
can give accurate results (a good example where you should is the time
call and your conclusion that its resolution must be enough to actually
check the scheduling accuracy of EV, while forgetting that its resolution
is very bad indeed).
That keeps adrenaline levels down on all sides :)
Marc Lehmann
> * I *could* stop the loop in the debugger today, but it took a while:
Although the behaviour is consistent with any other extension module for
perl, I tried to document this in the next EV release:
=head1 PERL SIGNALS
While Perl signal handling (C<%SIG>) is not affected by EV, the behaviour
with EV is as with any other C library: signals will only be handled when
Perl runs, which means your signal handler might be invoked only the next
time an event gets handles.
The solution is to use the the EV signal handlers (see C<EV::signal>). If
you cannot do this for whatever reason, you can also force a watcher to be
called on every iteration by installing a C<EV::check> watcher:
my $async_check = EV::check sub { };
This ensures that perl shortly gets into control for a short time.
Note, however, that this does not work for the perl debugger as the perl
debugger can only stop at the next perl statement to be executed - this is
a limitation of perl (or more precisely the debugger).
As this is also the exact same behaviour as with Event (or other event
loops), so I find it rather surprising that it needs mentioning, but
I hope the additional documentation is not wasted and actually proves
helpful to somebody.
(But then, signal handling does require knowledge of the underlying
operating system handling of signals).
--
The choice of a
Uri Guttman
ML> Its a design decision. Combining timers with io watchers is unnatural,
ML> because:
ML> - you often don't need both, so shouldn't be forced to pay for them
ML> - you can't choose the type of timeout (event only has absolute timers
ML> anyways, though, which are unsuitable for many tasks such as idle timeouts
ML> for i/o, where you need relative timeouts).
ML> - you have to manage two timeouts or weirder stuff when you have
ML> two separate i/o watchers, or use a third watcher (paying even more).
ML> - it is often unclear on which events exactly the timeout gets reset
ML> (if at all), and even if not, there is no flexibility (how do you
ML> chose between an idle timeout and an overall timeout
ML> for completion with Event? you cannot).
ML> As it is trivial to create a separate timeout (and much faster than with
ML> Event, too), there is no drawback to this design.
i do that in stem's event layer too. i/o watchers are created and an
optional timer is added if desired (a separate event with a different
method to callback into the object). this means all the event layer
needs is simple i/o watchers and a simple timer. my code builds up the
combined i/o with timer as needed. much cleaner and easier to manage at
all levels.
as marc said it is also simpler in the lower levels to code up. and i
bet most i/o watchers don't need timers which saves on the storage and
coding.
Marc Lehmann
As it might not be too obvious, here is (in pseudocode) an example
of an idle timeout, i.e. a timeout that triggers afetr some time of
inactivity. It uses two watchers (which is not required but customary):
# 60 seconds idle timeout
my $to = EV::timer_ns 0, 60, sub {
close $fh; ... log error;
};
$fh = ... new socket;
$to->again;
my $wr = EV::io $fh, EV::READ, sub {
$to->again; # reset timer
# do stuff
};
my $ww = EV::io $fh, EV::WRITE, sub {
$to->again; # reset timer
# do stuff
};
The read and write watchers can easily be started and stopped
independently of each other this way, and also independent of the timeout
($to->stop ... $to->again is a valid sequence to temporarily disable a
timeout, e.g. when the connection is not dpoing any work and can be kept
open).
If you try to do this in event and re-use one watcher's timeout, you will
need to stop/start it each time you get activity, which is rather costly
(and costs a lot when one uses e.g. the epoll backend). Of course one can
use a separate watcher with Event, too.
If an overall timeout is required (e.g. a single request must be done
afetr 90 seconds, no matter what), the code becomes simpler (basiclaly its
then using a simple timer and never resets it):
# 90 seconds idle timeout
$fh = ... new socket;
my $to = EV::timer_ns 90, 0, sub {
close $fh; ... log error;
};
my $wr = EV::io $fh, EV::READ, sub {
# do stuff
};
my $ww = EV::io $fh, EV::WRITE, sub {
# do stuff
};
Jochen Stenzel
Hi Marc,
> So far, it seems everything works right. While EV is young and certainly
> has some bugs, it is used in a lot of projects, and really can't be
> _totally_ broken.
I never thought it was broken. If there was an impression as if I wanted
to express something like that, or even "totally broken", I'm very
sorry. I was curious, I played a bit, I hoped to use EV in a project
(and actually started to do so now), some things didn't seem to work
perfectly and so I made some notes, and yes given that I was under heavy
time pressure I agree it was unwise to quickly send first impressions in
between spontaneously.
Time pressure continues and so it could take a while until I can return
to the test scripts and your detailed answer (thanks for that), but I
just want to make clear now that there was absolutely no intention to
blame you or your work. *If* there are bugs they can probably be fixed.
There was no adrenaline involved here ;-) As I said before: I consider
it great news you wrote and maintain EV, and I think you are the right
man for such a project, according to my impressions. Sorry for any other
impression.
Greetings
Jochen
Marc Lehmann
> > So far, it seems everything works right. While EV is young and certainly
> > has some bugs, it is used in a lot of projects, and really can't be
> > _totally_ broken.
>
> I never thought it was broken. If there was an impression as if I wanted
> to express something like that, or even "totally broken", I'm very
> sorry.
Nono, I didn't get that impression at all. "totally broken" were only my
words :)
> I was curious, I played a bit, I hoped to use EV in a project
> (and actually started to do so now), some things didn't seem to work
> perfectly and so I made some notes, and yes given that I was under heavy
> time pressure I agree it was unwise to quickly send first impressions in
> between spontaneously.
Good. Happens to me, too (the reporting-too-early). I was actually
surprised to see the debugger take a reference btw....
Its better this way than knowing about a bug and being too shy to report
it :)
> There was no adrenaline involved here ;-) As I said before: I consider
Your mail was worded quite neutrally, if that helps :) I said that mostly
for the benefit of other people following the discussion.