Inter-process multicast?

385 views
Skip to first unread message

Ed B.

unread,
Jan 23, 2011, 4:42:58 PM1/23/11
to erlangcamp
Anyone have a fix on the best way to do a one-to-many message cast?

I should be clear, I'm not necessarily looking to send a message to
multiple clients. Rather, I'm looking to send a message to the first
of multiple workers that is available.

I was reading in Go that you can have multiple goroutines subscribe to
a channel, and it will be random which one picks up a message. When a
goroutine is avail for work, it grabs the next message off the
channel. If there aren't any, it blocks. It's a very easy way to
workshare.

I guess the idiomatic Erlang way is to spawn a new worker for each
job. That also potentially avoids garbage collection in the workers,
depending on the size of the job. I don't have a fix on the time
overhead for spawning, but everyone implies it's very small.

But what if you're using multiple nodes? If you spawn a new worker
each time, how do you know which node to use? If you maintained a
central message repository (ie a channel), you could pub/sub to it,
and free processes could automatically load-balance.

Two options seem to present themselves:

1. Create a proxy process, which worker processes query for work.
This is obviously a bottleneck. It also creates a lot of extra
message copying and traffic.

2. Create an mnesia table to stow & replicate the messages, which
workers then remove in a transaction. Obviously the transaction has a
significant cost, as does the constant querying of the table.
Finally, there's nothing in the VM to wake or create a worker process
when there's work waiting, so either you have processes waking
themselves intermittently, or a supervisor process still has to keep
track of workers, loading, etc.

I guess you could reduce some of the overhead of (2) by doing dirty
reads on the table, assuming it's okay for more than one process to
execute a job.

I'm sure all of this has been addressed before. Any ideas/established
solutions? Have I not done enough homework? I saw one post where Wil
Larson wrote about calling erlang:process_info and checking
message_queue_length or stack_size:

http://lethain.com/entry/2009/sep/12/load-balancing-across-erlang-process-groups/

That's an intriguing idea, but it also seems like it would have a lot
of overhead. For each job cast (or at least each batch), I would have
to query all the workers.

Thx,
Ed

Martin Logan

unread,
Jan 23, 2011, 6:06:02 PM1/23/11
to erlan...@googlegroups.com
You should indeed spawn a new process each time. Pooling processes
would be an optimization that you may employ way later in the game. If
you need speed prior to pooling the processes you may look to rewrite
them as non OTP processes to avoid the overhead of process startup
with proc_lib that comes with the gen family of processes.

If you are using multiple nodes you would have those nodes operate as
independent services, and you would not spawn anything directly on
them typically. As far as which node to use, with any scheme, you can
either come up with an straightforward algorithmic load balancing
scheme like round robin, you track load and push intelligently, or
have a pull model where workers pull when they are ready. If the work
required for these messages is rather uniform across the set of
messages then the round-robin approach works nicely. If not, then you
need something more complex, either pull or push.

Not sure if this helps.

Cheers,
Martin

--
Martin Logan
Erlang & OTP in Action (Manning) http://manning.com/logan
http://twitter.com/martinjlogan
http://erlware.org

Martin Logan

unread,
Jan 23, 2011, 6:06:55 PM1/23/11
to erlan...@googlegroups.com
Oh, I forgot, you could also push to all workers and have them come to
a consensus on who does the work.

ll...@writersglen.com

unread,
Jan 23, 2011, 10:42:27 PM1/23/11
to erlan...@googlegroups.com
Hi,

What is the difference between io:fwrite and io:format? When would I use one over the other?

Thanks,

LRP

Martin Logan

unread,
Jan 23, 2011, 11:15:33 PM1/23/11
to erlan...@googlegroups.com
Lloyd, there is absolutely no difference. Here is the code for fwrite
for your edification:

--io.erl--

fwrite(Format) ->
format(Format).

fwrite(Format, Args) ->
format(Format, Args).

--

ll...@writersglen.com

unread,
Jan 23, 2011, 11:20:42 PM1/23/11
to erlan...@googlegroups.com
Thanks, Martin,

I never think to look at source. I'll have to develop the habit.

All the best,

Lloyd

Martin Logan

unread,
Jan 23, 2011, 11:42:34 PM1/23/11
to erlan...@googlegroups.com
No problem Lloyd. Looking at code in some languages can be a bit of a
waste of time for small issues due to the time it takes to get context
and understand what you are looking at. With Erlang it is often quite
valuable to jump into the code - referential transparency is to thank
for that. What you see is often what you get. This holds true for any
functional language that eschews side effects as well of course.

Cheers,
Martin

ll...@writersglen.com

unread,
Jan 26, 2011, 1:40:28 AM1/26/11
to erlan...@googlegroups.com
Hi,

I'm trying to convert Erlang DateTime format: {Year, Month, Day}{Hour, Minute, Second} to RFC 2616 DateString format: Sun, 06 Nov 1994 08:49:37

I'd like to be able to convert the DateString back to Erlang DateTime using httpd_util:convert_request_date(DateString).

But note if day-of-month, hour, minute, or second are single digit, they need to be of form "0n"; e.g. 01, 02, etc.

So, I have a single digit integer, say 5. But I can't figure out how to display it as "05" in io:format.

~2.1.0w will do it, but fails if the integer has two digits, say 22.

And at this point I'm lost.

Why am I doing this? I want DateString to be easily readable by humans. But I want to be able to sort by DateTimes.

Can anyone help me light the darkness?

Or maybe there's a better way to solve the problem altogether?

Many thanks,

LRP

Barry Nicholson

unread,
Jan 26, 2011, 12:38:35 PM1/26/11
to erlan...@googlegroups.com
In cases like this I would do the following (the following is from a
shell session).

5> io:format("~2.10.0b~n",[3]).
03
ok
6> io:format("~2.10.0b~n",[23]).
23
ok
7> io:format("~2.10.0b~n",[233]).
**
ok

That help?

Barry Nicholson

ll...@writersglen.com

unread,
Jan 26, 2011, 1:45:05 PM1/26/11
to erlan...@googlegroups.com
Exactly what I needed!

Thank you, Barry

bill robertson

unread,
Jan 26, 2011, 2:49:35 PM1/26/11
to erlan...@googlegroups.com

I had a similar need awhile ago.  I borrowed the implementation in yaws.

Sent from my Android phone.

On Jan 26, 2011 1:45 PM, <ll...@writersglen.com> wrote:

Exactly what I needed!

Thank you, Barry


-----Original Message-----
From: "Barry Nicholson" <b.nic...@niceng.com>

Sent: Wednesday, Januar...

ll...@writersglen.com

unread,
Jan 26, 2011, 3:44:13 PM1/26/11
to erlan...@googlegroups.com
Hello again,

Well, it took your help and an hour or so of flailing, but I finally got this to work:

I have this function in the books.erl:

this_second() ->
% Generate date string in RFC 2616 format
{Today, This_Minute} = calendar:universal_time(),
{Year, Month, DOM} = Today,
{Hour, Minute, Second} = This_Minute,
DOW = calendar:day_of_the_week(Year, Month, DOM),
Day = httpd_util:day(DOW),
M = httpd_util:month(Month),
lists:flatten(io_lib:format("~s, ~2.10.0b ~s ~p ~2.10.0b:~2.10.0b:~2.10.0b", [Day,DOM,M,Year,Hour,Minute,Second])).

books:this_second() gives me a human readable date string:

3> books:this_second().
"Wed, 26 Jan 2011 20:37:48"

And I can convert the date string back to erlang DateTime thus:

4> httpd_util:convert_request_date(books:this_second()).
{{2011,1,26},{20,26,53}}

Given the Erlang DataTime format, I can compare dates:

5> A = httpd_util:convert_request_date(books:this_second()).
{{2011,1,26},{20,36,12}}
6> B = httpd_util:convert_request_date(books:this_second()).
{{2011,1,26},{20,36,46}}
7> A < B.
true
8> A == B.
false
9> A > B.
false

Hope this function is useful to others.

So, thanks again, Barry.

LRP


-----Original Message-----
From: "Barry Nicholson" <b.nic...@niceng.com>

Sent: Wednesday, January 26, 2011 12:38pm
To: erlan...@googlegroups.com
Subject: Re: Converting integers to strings

ll...@writersglen.com

unread,
Feb 20, 2011, 10:41:12 PM2/20/11
to erlan...@googlegroups.com
Hello,

I'd like to run eDocs on Zotonic source code. If, working from the Zotonic directory /home/zotonic/zotonic/src, I pattern my call on the example on page 130 in Erlang and OTP in Action, e.g.:

edoc:application(zotonic, ".", [])

I get a page full of errors.

Is it likely that these errors result from:

1) The way I'm calling the function?
2) The Zotonic source files?
3) The directory structure of the Zotonic application?

If I try to run edoc:files/2 based on a single Zotonic file I get:

1> edoc:files(["z_string.erl"],[]).
./z_string.erl, in module header: at line 3: warning: tag @date not recognized.
./z_string.erl, function replace/3: at line 591: tag @copyright not allowed here.
edoc: skipping source file './z_string.erl': {'EXIT',error}.
edoc: error in doclet 'edoc_doclet': {'EXIT',error}.
** exception exit: error
in function edoc_lib:run_plugin/5

Here, again, I'm not sure if the error is due to the way I'm calling edoc:files/2 or in the source file itself.

In short, edoc is rather baffling to me due to the lack of clear confidence-building examples.

Can anyone help me climb over this speed bump in my learning curve?

Many thanks,

LRP

*********************************************
My books: Available through Amazon
or by request from your favorite bookstore

Freein' Pancho
Aya Takeo

Websites:

freeinpancho.com
ayatakeo.com
vol1.ayatakeo.com
**********************************************


Martin Logan

unread,
Feb 21, 2011, 2:23:19 PM2/21/11
to erlan...@googlegroups.com, ll...@writersglen.com
It is zotonic, their edocs don't compile.

--

Reply all
Reply to author
Forward
0 new messages