[erlang-questions] Bounded Buffer Problems/Solutions

[Eric Newhuis]

What are the generally accepted solutions for when producers need to be throttled? gen_server as consumer or otherwise?

Are there any good solutions or philosophy for the bounded buffer problem in Erlang? ..flow control? Freaky out-of-band communication at a distance to limit messages sent? Use the idea of rope: Message in reverse to a producer shim that he is allowed to send N more messages before he needs to ask for the right to send more? Other?

Are there common Erlang idioms for monitoring execution time at the gen_server level so that an API can respond upward with a policy of "the system is currently busier than a one legged man in a kicking contest?"

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Michael Truog]

On 10/10/2012 12:37 PM, Eric Newhuis wrote:
> What are the generally accepted solutions for when producers need to be throttled? gen_server as consumer or otherwise?
>
> Are there any good solutions or philosophy for the bounded buffer problem in Erlang? ..flow control? Freaky out-of-band communication at a distance to limit messages sent? Use the idea of rope: Message in reverse to a producer shim that he is allowed to send N more messages before he needs to ask for the right to send more? Other?
>
> Are there common Erlang idioms for monitoring execution time at the gen_server level so that an API can respond upward with a policy of "the system is currently busier than a one legged man in a kicking contest?"

I haven't seen a generic chunk of Erlang code that would handle all throttling needs because I think it depends on the situation, so the source code is generally ad-hoc. However, I do think that in the situation that many Erlang process act as producers with 1 or more Erlang process acting as a consumer, you can use the process dictionary in a valid way (that most people wouldn't be too angry about). If you use a unique atom as the process dictionary key in each producing process that contains a timestamp/count tuple, you are able to do a simple rate limiter that is non-intrusive to the other source code. I have done this for async logging, to prevent flooding processes here:
https://github.com/okeuday/CloudI/blob/master/src/lib/cloudi/src/cloudi_logger.erl#L362

Using the process dictionary helps to avoid checking with a message, in the consuming process. Also, checking in the producing process could be erroneous if all producing cases are not handled properly. Throttling generally wants temporary storage which makes it a problem better suited for the process dictionary than ets (and you can avoid global locking on ets tables).

[Joe Armstrong]

On Wed, Oct 10, 2012 at 9:37 PM, Eric Newhuis <enew...@gmail.com> wrote:
> What are the generally accepted solutions for when producers need to be throttled? gen_server as consumer or otherwise?
>
> Are there any good solutions or philosophy for the bounded buffer problem in Erlang? ..flow control? Freaky out-of-band communication at a distance to limit messages sent? Use the idea of rope: Message in reverse to a producer shim that he is allowed to send N more messages before he needs to ask for the right to send more? Other?

Actually no - we noticed years ago (round 1990) that if producers ran
faster than consumers
you got a problem. So it this case we made everything synchronous.
Send a message, wait for a reply
ad nauseam. This is used in gen_servers and virtually everywhere. It
doesn't seem to matter
I guess if the CPU is always busy we're happy bunnies. Ok so you wait
a bit longer, bit something else gets done while you're waiting.

Now you could (say) require an ack after every ten messages - and send up to ten
messages without requiring an ack or something but things actually go
surprisingly well
without this ...

>
> Are there common Erlang idioms for monitoring execution time at the gen_server level so that an API can respond upward with a policy of "the system is currently busier than a one legged man in a kicking contest?"

No - easy to implement though.

My approach has always been take the simplest solution that works -
then measure and
only fix things if they are broken.

So we've built massive systems with synchronous client server RPCs all
over the place
then we tune them - most often it's the time to parse inputs or the
database that's the problem.

Getting data in and out of the system and computation seem to be the
problem areas
the interprocess message passing is very fast compared to this - so
complex buffering seems
not to be necessary.

I'm not saying you never need it - just that I've never seen it myself.

Cheers

/Joe