[erlang-questions] Improving performance on a thrift to protobuf proxy

78 vistas
Ir al primer mensaje no leído

Anthony Molinaro

no leída,
14 jun 2010, 2:52:59 p.m.14/6/10
para erlang-questions
Hi,

I have a thrift server (using the generated thrift server from the thrift
distribution) which contains a single method. The method takes two thrift
objects (a context, and a list of places to call). The thrift method handler
spawns a process for each place to call. Each of those processes, takes the
context, combines it with some meta info particular to the place to call,
and serializes the request as a protobuf. It then makes an http call using
httpc with a timeout, and calls the place to call passing the protobuf as the
body of an HTTP GET request.

For each incoming request I end up with 6-10 outgoing requests. Right now
the server runs pretty well, but it seems to use a lot of CPU, also even
though I set a pretty tight timeout on my httpc call (120 msecs), the
server often doesn't respond within 200 msecs to the caller.

I did a little bit of profiling in isolation (using eprof), but that didn't
really show any low hanging fruit (however, I'm not entirely sure I used the
right profiling). So I'm looking for pointers to help improve throughput
and lower CPU usage.

Specifically,

- what is the best way to profile this sort of application? using tracing?
fprof? eprof? on the live server or try to test in isolation (I profiled
in isolation before, but as I said no real low hanging fruit).
- is httpc the ideal http client? are there better http client libraries?
should I roll my own using event driven non-blocking techniques? does
the timeout in httpc really work, or should I do something with a timer?
- this is really just sort of a glorified web crawler, has anyone out there
written a crawler and have pointers?

Any other hints or tips would be greatly appreciated.

Thanks,

-Anthony

--
------------------------------------------------------------------------
Anthony Molinaro <anth...@alumni.caltech.edu>

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questio...@erlang.org

Anthony Molinaro

no leída,
14 jul 2010, 4:24:56 p.m.14/7/10
para erlang-questions
Hi,

While I haven't yet answered all my own questions I did discover 2 things
which might help future erlang programmers.

1. use ibrowse as an http client, not httpc. httpc leaks memory and is buggy
in R13B04
2. R13B04 is better than R12B05, much better. It used half the CPU (or at
least did after switching from httpc to ibrowse).

If I discover more I will send more.

-Anthony

Responder a todos
Responder al autor
Reenviar
0 mensajes nuevos