[erlang-questions] UDP client/server performance

[Ronny Meeus]

Hello

I have created a simple UDP based client/server application.
The code is available on bitbucket
(https://bitbucket.org/meeusr/erl-examples) in the networking
directory.

The server is just an echo server running in 1 process.
The client can behave like a sync client (sending 1 message and
waiting for the reply) or like an async client (sending X messages
before waiting for all the replies).
I see that my PC (Intel(R) Quad code i7 CPU 860 @ 2.80GHz) is able to
process something like 7K - 10K messages per second. Both the sync and
the async are in the same order of magnitude. Please note that I use
the loopback interface.

$ erl +sbt db
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
1> c(gen_udp_test).
gen_udp_test.erl:47: Warning: variable 'Data' is unused
gen_udp_test.erl:47: Warning: variable 'Msg' is unused
gen_udp_test.erl:62: Warning: variable 'Data' is unused
gen_udp_test.erl:62: Warning: variable 'Msg' is unused
{ok,gen_udp_test}
2> gen_udp_test:start_server().
true
...
12> gen_udp_test:client_async(5000).
stop : 841482
ok
13> gen_udp_test:client_async(6000).
stop : 1082679
ok

The timings are the number of usec for the processing of the X number
of messages (round-trip).

If I implement the same application in C (client and server are 2
applications), I see that the application performs an order of
magnitude better for the sync solution (86Kpkts/sec) and 100 times
better (700Kpkts/sec) for the async version.
The code for this is also available in the repo on bitbucket.

I know there this test is only looking to a part of the application
(transport of the data), but I find the difference rather big.
Is this normal or is there something wrong in my Erlang code?

Thanks.
Regards,
Ronny
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[JD Bothma]

I'm curious:

What are the numbers like (c versus erlang) when it's between two
physical machines? Or between two (vmware or virtualbox, not xen)
virtual machines?

Or said otherwise: does the fact that it's the same machine, network
stack and loopback interface affect the comparison between c and
erlang?

JD

[Mike Oxford]

Erlang will be slower and will consume more memory than C.  That's pretty much par for the course due to type munging and layers between you and the hardware, including the VM itself.  Some are using libevent through an Erlang port to get around this.

You can also try "when I receive a packet, spawn an unlinked process to do the reply" which may unblock things.  (Not sure, haven't profiled it myself.)

Writing C directly to low-level structures is known to be much faster than the Erlang VM.  You can also see this in action at http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3

-mox

[Ronny Meeus]

Hello

I did a quick check.
- The values between 2 machines are comparable for Erlang.
- For the C code there is a difference: the async version drops from
700Kpks/sec to 400Kpks/sec while the sync version drops from
86Kpkts/sec to 10Kpkts/sec.

[Ronny Meeus]

Is there any code available that implements a NIF to have access to UDP sockets?
I have found the procket implementation but it looks like it is using
a separate process to communicate with the socket. This sounds like
overkill to me for this test.

Ronny

[Michael Santos]

On Tue, Aug 14, 2012 at 08:25:23AM +0200, Ronny Meeus wrote:
> Is there any code available that implements a NIF to have access to UDP sockets?
> I have found the procket implementation but it looks like it is using
> a separate process to communicate with the socket. This sounds like
> overkill to me for this test.

You can get the socket directly using procket:socket/3. See
procket:sendto/4 and procket:recvfrom/4 as well for retrieving the
struct sockaddr for the peer.

-module(udp_srv).
-export([s/0]).

-define(UINT16(N), N:2/native-unsigned-integer-unit:8).
-define(PF_INET, 2).
-define(PORT, 6001).

s() ->
{ok,FD} = procket:socket(inet, dgram, 0),
% sockaddr for Linux
Sockaddr = <<?UINT16(?PF_INET), ?PORT:16, 0:32, 0:64>>,
ok = procket:bind(FD, Sockaddr),
Ref = erlang:open_port({fd, FD, FD}, [stream, binary]),
loop(Ref).

loop(Ref) ->
receive
{Ref,{data,Data}} ->
error_logger:info_report(Data),
loop(Ref)
end.

[Ronny Meeus]

Hello

thanks for the info.
I'm currently playing with the procket and it is working well.

I have 2 questions:

- I need to run the erl as root (or use sudo to start it) or I get:

2> {ok, FD} = procket:open(53, [{protocol, udp},{type, dgram},{family, inet}]).
** exception error: no match of right hand side value {error,eperm}

- How do I use recvfrom/4 in the server? If I call recvfrom/4 when
there is no message present in the socket, it returns an error since
the socket is put in async mode. Does this mean that the socket needs
to be polled by the Erlang process or is it possible to get a
notification (msg) when there is data present in the socket so that
the recvfrom/4 can be called at that moment?

Thanks.
Ronny

[Michael Santos]

On Tue, Aug 14, 2012 at 10:10:57PM +0200, Ronny Meeus wrote:
> Hello
>
> thanks for the info.
> I'm currently playing with the procket and it is working well.
>
> I have 2 questions:
>
> - I need to run the erl as root (or use sudo to start it) or I get:
>
> 2> {ok, FD} = procket:open(53, [{protocol, udp},{type, dgram},{family, inet}]).
> ** exception error: no match of right hand side value {error,eperm}

Make priv/procket setuid or allow your user to call priv/procket using
sudo.

> - How do I use recvfrom/4 in the server? If I call recvfrom/4 when
> there is no message present in the socket, it returns an error since
> the socket is put in async mode.

Yes, the socket has to be non-blocking or it will block the scheduler.

> Does this mean that the socket needs
> to be polled by the Erlang process or is it possible to get a
> notification (msg) when there is data present in the socket so that
> the recvfrom/4 can be called at that moment?

You could plug the fd back into gen_udp:open(Port, [{fd, FD}]) ;)

For a UDP socket, I can't think of any options other than the ones
you've mentioned: either spin on EAGAIN or use another NIF to notify you
when the socket is ready. There are a few ports to libev and libuv around.

[Ronny Meeus]

On Wed, Aug 15, 2012 at 2:53 AM, Michael Santos
<michael...@gmail.com> wrote:
> On Tue, Aug 14, 2012 at 10:10:57PM +0200, Ronny Meeus wrote:
>> Hello
>>
>> thanks for the info.
>> I'm currently playing with the procket and it is working well.
>>
>> I have 2 questions:
>>
>> - I need to run the erl as root (or use sudo to start it) or I get:
>>
>> 2> {ok, FD} = procket:open(53, [{protocol, udp},{type, dgram},{family, inet}]).
>> ** exception error: no match of right hand side value {error,eperm}
>
> Make priv/procket setuid or allow your user to call priv/procket using
> sudo.

I tried all options mentioned in your readme already before but non of
them seem to work on Ubuntu. I think there is an issue with the sudo
stuff on this distribution since no matter what I do, it always ask
for a password when the procket command is executed.

Now I succeeded by explicitly specifying the name of the program to use:
procket:open(53, [{progname,"/usr/local/bin/procket"},{protocol,
udp},{type,dgram}]).

>
>> - How do I use recvfrom/4 in the server? If I call recvfrom/4 when
>> there is no message present in the socket, it returns an error since
>> the socket is put in async mode.
>
> Yes, the socket has to be non-blocking or it will block the scheduler.
>
>> Does this mean that the socket needs
>> to be polled by the Erlang process or is it possible to get a
>> notification (msg) when there is data present in the socket so that
>> the recvfrom/4 can be called at that moment?
>
> You could plug the fd back into gen_udp:open(Port, [{fd, FD}]) ;)
>
> For a UDP socket, I can't think of any options other than the ones
> you've mentioned: either spin on EAGAIN or use another NIF to notify you
> when the socket is ready. There are a few ports to libev and libuv around.

OK I will look around.
Thanks.

Yesterday I did a quick test and created an application that once it
receives a message, it starts to send this back in a loop.
It looks like it is able to send almost 200Kpkts/sec on a low end PC
while the gen_udp solution only was able to send something like
10Kpks/sec on a much more powerful PC. So it looks promising ...

[Kenneth Lundin]

Hi Ronny,

I think you are a little bit fast in drawing the conclusion that you
must look at other solutions
than just using the standard gen_udp module.

If you have found some real bottle neck in performance we should of
course try to find out why the difference between C and Erlang is so
big, I don't think it should be that big.

In case there is something unnecessarily inefficient going on when
using gen_udp we should
try to address that instead of looking at more complex and not so
general alternatives

A deeper analysis of what your test is doing would be good, is it
comparable with the C case?
Can the difference be explained etc.

I am not convinced that you will get better speed the alternative ways
you have been recommended. The socket code is written in C even when
you use gen_udp so I don't
really see the big difference. I also suppose you in reality want to
do something with the
data on the Erlang side and that would probably change the
characteristics. Another thing is if there is multiple cores involved
etc.

Kenneth , Erlang/OTP Ericsson

[Max Lapshin]

I've met difference between erlang and C code only once.

I had to receive MPEG-TS UDP packets, about 100 mbits of them, pack
them into large packets and send via http.

One MPEG-TS UDP packet is about 1300 bytes (7 blocks of 188 bytes), so
there is about 70 000 of messages per second.

I've rewritten UDP handling code in C (with using all erl_driver
capabilities) and made packaging inside C driver.

Now erlang code received about 700 messages per second and CPU got
down from 90% to 10%. So, my ehnancement was in lowering amount of
messages and preallocating big buffers inside C level.

[Ronny Meeus]

Hello

my final goal is to implement a statsd like application completely in
Erlang (as a exercise to learn more about the various aspects of
Erlang). The first step for me is to implement a UDP echo server to
have a feeling about the performance.
Since this was very low in my opinion (the server handled only 5000
messages per second), I wrote the same in C. While running the etop I
also observed that the system was using a lot of memory while sending
the messages.

Basically both applications do the same thing: the client is sending a
string "TEST" and the server echos it back.

The bad performance of the Erlang solution triggered me to send a mail
to the community, maybe I was doing something wrong.
So it might be good to have a look at the code I have created and
check whether it is using the gen_udp in a correct and most performant
way.

I agree that it is better to improve the gen_udp code so that it is
beneficial for all applications, but I'm afraid that my skills are not
good enough for that.

Ronny

[Michael Santos]

On Wed, Aug 15, 2012 at 09:59:22AM +0200, Ronny Meeus wrote:
> On Wed, Aug 15, 2012 at 2:53 AM, Michael Santos
> <michael...@gmail.com> wrote:
> > On Tue, Aug 14, 2012 at 10:10:57PM +0200, Ronny Meeus wrote:
> >> Hello
> >>
> >> thanks for the info.
> >> I'm currently playing with the procket and it is working well.
> >>
> >> I have 2 questions:
> >>
> >> - I need to run the erl as root (or use sudo to start it) or I get:
> >>
> >> 2> {ok, FD} = procket:open(53, [{protocol, udp},{type, dgram},{family, inet}]).
> >> ** exception error: no match of right hand side value {error,eperm}
> >
> > Make priv/procket setuid or allow your user to call priv/procket using
> > sudo.
>
> I tried all options mentioned in your readme already before but non of
> them seem to work on Ubuntu. I think there is an issue with the sudo
> stuff on this distribution since no matter what I do, it always ask
> for a password when the procket command is executed.
>
> Now I succeeded by explicitly specifying the name of the program to use:
> procket:open(53, [{progname,"/usr/local/bin/procket"},{protocol,
> udp},{type,dgram}]).

procket uses priv/procket as the default path. Sorry, README should be
clearer.

> >> - How do I use recvfrom/4 in the server? If I call recvfrom/4 when
> >> there is no message present in the socket, it returns an error since
> >> the socket is put in async mode.
> >
> > Yes, the socket has to be non-blocking or it will block the scheduler.
> >
> >> Does this mean that the socket needs
> >> to be polled by the Erlang process or is it possible to get a
> >> notification (msg) when there is data present in the socket so that
> >> the recvfrom/4 can be called at that moment?
> >
> > You could plug the fd back into gen_udp:open(Port, [{fd, FD}]) ;)
> >
> > For a UDP socket, I can't think of any options other than the ones
> > you've mentioned: either spin on EAGAIN or use another NIF to notify you
> > when the socket is ready. There are a few ports to libev and libuv around.
>
> OK I will look around.
> Thanks.
>
> Yesterday I did a quick test and created an application that once it
> receives a message, it starts to send this back in a loop.
> It looks like it is able to send almost 200Kpkts/sec on a low end PC
> while the gen_udp solution only was able to send something like
> 10Kpks/sec on a much more powerful PC. So it looks promising ...

That's cool! But remember, the C and the Erlang examples you posted are
doing different things. The C code is single tasking. It doesn't do any
error handling either. The Erlang code is multitasking, switching
between processes, monitoring processes for failure, ....

If you run C NIF code in a tight loop, you'll get something closer to
the single tasking C code at the expense of concurrency. Maybe that is
fine for your needs though. Anyway, good luck in hitting your numbers!

[Dmitry Kolesnikov]

Hello,

I would slightly disagree with your evaluation approach on UDP performance. It is obvious that Erlang code is slower to compare with C code. Therefore, any single threaded benchmark would always show a highest performance of C code. You have to push it in multi client env... I've re run you test cases and got a bit different results with vanilla gen_udp.

So, the lowest latency of inter packet arrival time on Erlang is about 16-17 usec. This is the time used by underlying SW stack to process packet and deliver it to application process. I did not managed to get any lower latency on my hw (Mac Lion Server).

Async case: pure C solution handled about 341K packet/sec, erlang server runs 256K packet/sec. One of the biggest problem for async is high packet drop rate in both C & Erlang cases.
Sync case: pure C solution handled about 160K packet/sec, erlang server runs 45-50K packet/sec.

Regards, Dmitry

[Ronny Meeus]

Hello

Thanks for taking the time to run the examples.
I have run the tests also on a different machine and there is see also
better results so it looks like there is an issue with the beam
running on the first machine. Maybe it is compiled with different
options ...
Anyhow, I will continue the development of my application based on
Erlang since I like the mechanisms it offers (threading, messaging,
error handling etc) a lot. I initially wrote the mail because of the
huge difference in performance I saw.

Ronny