[erlang-questions] erlang:send_after/3 questions

[Dmitry Demeshchuk]

Hello, list.

It's time to reveal some erlang:send_after/3 mysteries I think :)

1. When is the message term stored? It's obviously not the callee process, and it doesn't seem to be the target process (I don't see a process part that could store it). Is it dangerous to pass large messages through it?

2. What happens when the target process dies before send_after/3 occurs? The returned ref makes me think that we actually spawn some sort of a monitor here, and we just stop waiting if the process died by whatever reason. Is that correct?

3. Is spawning a lot of send_after's or timers a dangerous idea? In case of using timer module, we'll just lose some started timers due to port restart. But something tells me that crash of erlang internal timers is a somewhat more dangerous thing that might even crash the VM.

Any links or facts would be very appreciated.

--
Best regards,
Dmitry Demeshchuk

[fr...@circlewave.net]

Hi,

My understanding, but I'm not ERTS internals expert:

On Sat, Sep 22, 2012 at 12:38:02AM +0400, Dmitry Demeshchuk wrote:
> It's time to reveal some erlang:send_after/3 mysteries I think :)
>
> 1. When is the message term stored? It's obviously not the callee process,
> and it doesn't seem to be the target process (I don't see a process part
> that could store it). Is it dangerous to pass large messages through it?

It's copied to a heap fragment -- roughly the same kind of C struct is used
as for regular message passing. Except unlike regular message passing the
copying seems to be done in a way that blocks invoking scheduler thread.

So passing large structures this way is probably a bad idea (and, in my
opinion, quite an exotic thing to want to do). But you could always use
term_to_binary/1 first and cheat around this limitation.

> 2. What happens when the target process dies before send_after/3 occurs?
> The returned ref makes me think that we actually spawn some sort of a
> monitor here, and we just stop waiting if the process died by whatever
> reason. Is that correct?

The returned Ref is precisely the same thing as if make_ref/1 were called.
To each successfuly started timer there corresponds a small C structure.

Timers are inserted into a list hanging off receiving process' structure
if Pid is known and alive at the time send_after/3 is called. This list
enables receiving process to release all pending timers on exit. Timers
are additionally inserted into a global table of some sort.

Otherwise, when receiving process is given by name, timer will presumably
happily live in this global table and will be released upon expiry (the
erlang(3) manpage agrees with this speculation). They can't be released
any sooner than that, registered name may refer to different Pids over
time and by passing name instead of a real Pid you're expressing the
wish to deliver to the latest instance possible.

If receiving process is a Pid but dead already when send_after/3 is
called, send_after/3 degenerates into a sligtly slower variant of
make_ref/1. ;-)

> 3. Is spawning a lot of send_after's or timers a dangerous idea? In case of
> using timer module, we'll just lose some started timers due to port
> restart. But something tells me that crash of erlang internal timers is a
> somewhat more dangerous thing that might even crash the VM.

Not sure I understand you here.

These timers are a facility built directly into ERTS. If there were a
bug somewhere in the implementation, that could be as horribly nasty
as any other bug in ERTS. Other than that, they are dead pieces of
data interpreted in some conventional way by a bunch of C code, so
as such they can't really "crash" in any obvious meaning of the
word.

> Any links or facts would be very appreciated.

Just link to the implementation:

https://github.com/erlang/otp/blob/maint/erts/emulator/beam/erl_bif_timer.c#L495

BR,
-- Jachym
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Michael Truog]

On 09/21/2012 03:15 PM, fr...@circlewave.net wrote:
>> 3. Is spawning a lot of send_after's or timers a dangerous idea? In case of
>> using timer module, we'll just lose some started timers due to port
>> restart. But something tells me that crash of erlang internal timers is a
>> somewhat more dangerous thing that might even crash the VM.
> Not sure I understand you here.
>
> These timers are a facility built directly into ERTS. If there were a
> bug somewhere in the implementation, that could be as horribly nasty
> as any other bug in ERTS. Other than that, they are dead pieces of
> data interpreted in some conventional way by a bunch of C code, so
> as such they can't really "crash" in any obvious meaning of the
> word.

I remember reading about problems using the timer module with a large number of timers because of excessive memory consumption, such that erlang:send_after/3 or erlang:start_timer/3 is generally the preferred approach. I don't have a link to the mailing list message(s) for instances of that problem, but I assume it is still an issue today, because of extra storage used by the timer module.

[Mahesh Paolini-Subramanya]

The timer module is basically a gen_server that is using timeouts to invoke a timer and an ordered_set ETS to store the timers

The ordered_set is probably the cause of the memory issues you are referring to, and the gen_server/timeout implementation is, well, not exactly the most scalable way to do this...

Cheers

Mahesh Paolini-Subramanya

That Tall Bald Indian Guy...

Blog   |   Twitter   |   Google+

[Rapsey]

3. Is spawning a lot of send_after's or timers a dangerous idea? In case of using timer module, we'll just lose some started timers due to port restart. But something tells me that crash of erlang internal timers is a somewhat more dangerous thing that might even crash the VM.

How much is a lot of send_afters? My server runs a lot of timers, but very rarely in more than one per process. But there can easily be thousands of timers running in different processes.

I would advise against using timer module. Using the timer module in a few places turned out to be a big mistake. It sometimes just stops working completely. It may be a mistake on my part, but erlang:send_after generally does everything you need and is completely reliable.

Sergej 

[Gleb Peregud]

In my experience, using excessive timers once bit me hard due to lock contention on global timers lock in Erts. Roughly it was about 3-5 timers created/expired for each of around 200k-500k processes at roughly the same moment (a big broadcast to all processes). Fortunately lcnt showed the problem which was slowing down the broadcast tenfold - lock conflicts was at 95% level on timers lock.

[Bryan]

Hi Gleb,

Hope you don’t mind me digging up this thread, but I was wondering if you could ask you a few questions based on your experience as we are looking at a similar situation where we have 500k (or more) processes active on a single node. Each process kicks off a couple of long running timers using erlang:send_after with dest being self(). We haven’t hit any slowdowns yet, but I am curious to understand your experience better. 

Were you using erlang:send_after or the timer module?

How did you eventually resolve your problem with the locking? Did you have to redesign how you were using the 3 to 5 timers per process so the total number of timers was reduced, or did you do something else, such as stagger the timers so they were not all firing at the same time?

Thanks in advance!

Cheers,

Bryan

----

Bryan Hughes

 Go Factory

http://www.go-factory.net

[Imants Cekusins]

Maybe a timer pool is needed.

It could handle timed calls for a few subscribers and stagger them if it's ok for subscribers.

Is there an app which does this already?