stress test of mochiweb for a comet server

[pablo platt]

Hi

I'm using MochiWeb for a long-polling comet server.
I'm running an app on one machine that spawns N gen_servers, each one login and send an http get request.
Mochiweb on another machine waits 30 secs and reply. The gen_server sends a new request...
I'm creating the clients and start sending requests immediately without setting a delay.

When I spawn 100 long-polling clients it works fine and they keep running.
When I'm increasing the number to 120 I see errors both on the client and the server machines.
On the mochiweb side I see {tcp_closed, Socket} messages.
On the client side I'm getting 404 responses:
{{"HTTP/1.1",400,"Bad Request"},
                [{"date","Wed, 18 May 2011 21:33:21 GMT"},
                 {"server","MochiWeb/1.0 (Any of you quaids got a smint?)"},
                 {"content-length","0"}],
                <<>>}

I'm using R140B on ubuntu 11.04. I didn't change any linux/erlang settings.

I'm starting mochiweb with the default settings:
mochiweb_http:start([{name, ?MODULE}, {loop, Loop}, {ip, IP}, {port, Port}]).

The waiting loop for long-polling:
wait(MochiReq) ->
    % put the socket in active once mode so we can receive the tcp_closed message.
    Socket = MochiReq:get(socket),
    inet:setopts(Socket, [{active, once}]), 
    receive
        {tcp_closed, Socket} ->
            {error, socket_closed};
        Res ->
            Res
    after
        ?WAIT ->
            timeout
    end.

Are there any system/mochiweb settings I need to use?
Could it be that I'm reaching some limits? How can I find out what?

Thanks

[pablo platt]

I was using the default httpc profile for all clients and somehow the request of client2 was received by the request pid of client1.
I don't understand how it could happen but now I'm using a separate profile for each client and I'm able to have more than 600 clients.
With a separate profile per client I'm running out of ets tables so I have to understand how to use a single profile.

[Bob Ippolito]

httpc probably isn't a good choice for this, I would expect the
problem to be the on the client side given how you've described it.
You might have better luck with another client such as lhttpc or
ibrowse.

> --
> You received this message because you are subscribed to the Google Groups
> "MochiWeb" group.
> To post to this group, send email to moch...@googlegroups.com.
> To unsubscribe from this group, send email to
> mochiweb+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mochiweb?hl=en.
>

[pablo platt]

I've switched to ibrowse and still having the same problem.
when setting the socket of the request to {active, once} before the receive loop, it sometimes gets the following message, which a request of a different client:
{tcp,#Port<0.4011>,<<"GET /test/?user=12 HTTP/1.1\r\nHost: host.test:8000\r\nContent-Length: 0\r\n\r\n">>}
When I'm removing the line inet:setopts(Socket, [{active, once}]) or when I'm using inets with a separate profile for each client, the problem goes away.

Is it possible that mochiweb gets confused when sending several requests from the same profile?
What's the difference between using the default profile or a separate profile for each test client?

The waiting loop of a request:

wait(MochiReq) ->
    % put the socket in active once mode so we can receive the tcp_closed message.
    Socket = MochiReq:get(socket),

    inet:setopts(Socket, [{active, once}]),  % when commenting this line I'm not getting the message but maybe it just hides the problem.

    receive
        {tcp_closed, Socket} ->
            {error, socket_closed};
        Res ->
            Res
    after
        ?WAIT ->
            timeout
    end.

[Bob Ippolito]

I don't know what your problem is and can't investigate without full
source code that reproduces this problem. It's unlikely that mochiweb
is getting "confused" in this way, there could be some other problem
with the code that isn't present in the snippets you've sent so far.

[pablo platt]

Several clients were using the same sockets.
The following settings force each request to use a separate socket:
httpc:set_options([{max_pipeline_length, 0}, {max_sessions, NUsers*2}, {max_keep_alive_length, 0}]).