[erlang-questions] Fwd: phantom ports in erlang:ports()

[Tim Watson]

Hi guys - can anyone answer this (below) please?

Begin forwarded message:

> From: Matthias Radestock <matt...@rabbitmq.com>
> Subject: phantom ports in erlang:ports()
> Date: 12 November 2012 18:37:47 GMT
> To: erlang-q...@erlang.org
>
> I've got a customer system, running R15B02, which encountered system_limit errors when accepting tcp connections.
>
> 'lsof' indicates that there are just a few hundred file descriptors in use. However, erlang:ports() returns 1010 entries. For the majority of these ports:
> - erlang:port_info(Port) returns 'undefined'
> - port_close(Port) fails with 'badarg'
> - exit(Port, die) doesn't make them go away
>
> These ports have been in that state for days.
>
> Does anybody have any clue what's going on here?
>
> Is there any way to find out more about these ports? Looks like ctrl-c'ing in the emulator shell would allow me to get more info with the 'o' option. Unfortunately the customer's emulator is running detached, so that won't work.
>
> Regards,
>
> Matthias.

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Patrik Nyblom]

Sounds like a bug allright.

Please send all relevant information, including platform, any special
conditions (nif's, drivers whatnot) and any relevant logs etc to
erlang_bugs or directly to me.

Cheers,
Patrik

[Björn-Egil Dahlberg]

2012/11/13 Patrik Nyblom <p...@erlang.org>

Sounds like a bug allright.

Sounds like it yes ..

A couple of things. Ports needs to flush there io-queues before terminating. If there is something in the queue when closing, the driver callback for flush will be called.

If no such callback exists there might be trouble. In the case of tcp and inets there exists a tcp_inet_flush and this is callback is invoked if the queue is not empty.

So what drivers are used that doesn't have flush callbacks?

One exception to the flush mechanism should be exit(Port, kill). If the exit signal 'kill' is received by the port, it should die no matter what. (If I read the code correctly that is =)

// Björn-Egil

[Valentin Micic]

I am wondering if this may be attributed to CLOSE_WAIT/TIME_WAIT states sockets may be in during the connection shutdown?
By the some token, if the server in question has a higher frequency of short-lived connections, this may also explain discrepancy between number of file descriptors as reported by lsof and number of active ports reported by emulator.

Just speculating, of course.

V/

[Jesper Louis Andersen]

On Nov 16, 2012, at 7:29 AM, Valentin Micic <vale...@pixie.co.za> wrote:

> I am wondering if this may be attributed to CLOSE_WAIT/TIME_WAIT states sockets may be in during the connection shutdown?
> By the some token, if the server in question has a higher frequency of short-lived connections, this may also explain discrepancy between number of file descriptors as reported by lsof and number of active ports reported by emulator.
>

TIME_WAIT is not a problem. It is just the kernel keeping track of the connection if packets get sent still by the other end. CLOSE_WAIT on the other hand sounds bad. This state should happen when the other end has sent a FIN and started to close down the socket, but the application (i.e., Erlang?) has not closed the socket down yet.

If you have lingering CLOSE_WAITs that are there for a long time it is usually a warning sign that something may be bad.

Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen

[Valentin Micic]

I guess, a more interesting questions is weather or not sockets in this state may be seen as "phantom" ports?

V/

[Tim Watson]

I can't get to the issue tracker right this moment, but these sockets are stuck in CLOSE_WAIT for a long time. The customer reports normally having circa 80±20 live connections, most of them listeners that do not close the sockets. Under normal conditions, there are no CLOSE_WAIT sockets showing up at all.

The customer also reports that even when the CLOSE_WAIT connections drop off that no further connections can be established, and we see some rather delightful errors coming back from these 'phantom' ports:

{inet_async,#Port<0.5808>,25398,{error,system_limit}}

but the system limits are *not* reached by any stretch of the imagination:

{file_descriptors,
[{total_limit,924},
{total_used,126},
{sockets_limit,829},
{sockets_used,56}]},
{processes,[{limit,1048576},{used,693}]}...


So this is more than just sockets stuck in CLOSE_WAIT by the looks of things.