Increasing TCP connections at CLOSE_WAIT state

1,613 views
Skip to first unread message

Yasin Tamer

unread,
Mar 19, 2014, 5:51:35 PM3/19/14
to haze...@googlegroups.com
Hi all,

When we deploy our rest application with Hazelcast, 
Number of TCP connections at CLOSE_WAIT state increase immediately.

At first we didnt limit some linux system parameter so our Tomcat throws 
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
        at java.lang.Thread.run(Thread.java:744)


Then we limit these parameter as seen above;
net.ipv4.tcp_fin_timeout=30 
#Fine time_out bekleme suresini dusur
net.ipv4.tcp_keepalive_intvl=30 
#Aktif baglantilar arasinda bekleme suresi
net.ipv4.tcp_keepalive_probes=5
# Zaman asimi olusmadan probe larin sayisi
net.ipv4.tcp_keepalive_time=1800 
# Baglantiyi istek olmadiginda aktif etme suresi
net.ipv4.tcp_max_syn_backlog=2048 
#Ack gelmemesis SYN paketlerinin kuyrugu

After set these parameters number of TCP connections at CLOSE_WAIT state still increase but system kill them. 

Currently we run our application on only one machine.

Part of Output; netstat -anp | grep 7308 command
tcp        0      0 :::8010                     :::*                        LISTEN      7308/java           
tcp        0      0 :::6701                     :::*                        LISTEN      7308/java           
tcp        0      0 ::ffff:127.0.0.1:8016       :::*                        LISTEN      7308/java           
tcp        0      0 :::8081                     :::*                        LISTEN      7308/java           
tcp        0      0 ::ffff:127.0.0.1:34221      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java
tcp        1      0 ::ffff:127.0.0.1:39426      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:35949      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:39748      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        0      0 ::ffff:127.0.0.1:35086      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:39933      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:37821      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:36117      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:38060      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:34413      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:38919      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:35961      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        0      0 ::ffff:127.0.0.1:34806      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:34466      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java           
tcp        1      0 ::ffff:127.0.0.1:40085      ::ffff:127.0.0.1:8080       CLOSE_WAIT  7308/java


Part of Output; lsof -p 7308
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
java    7308 root   57r  IPv6             471182      0t0    TCP localhost.localdomain:33542->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   58r  IPv6             471189      0t0    TCP localhost.localdomain:33546->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   59r  IPv6             471193      0t0    TCP localhost.localdomain:33548->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   60r  IPv6             471195      0t0    TCP localhost.localdomain:33549->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   61r  IPv6             471199      0t0    TCP localhost.localdomain:33551->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   62r  IPv6             474940      0t0    TCP localhost.localdomain:35468->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   63r  IPv6             471205      0t0    TCP localhost.localdomain:33554->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   64r  IPv6             471211      0t0    TCP localhost.localdomain:33558->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   65r  IPv6             471216      0t0    TCP localhost.localdomain:33560->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   66r  IPv6             471218      0t0    TCP localhost.localdomain:33561->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   67r  IPv6             471227      0t0    TCP localhost.localdomain:33565->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   68r  IPv6             471233      0t0    TCP localhost.localdomain:33568->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   69r  IPv6             471237      0t0    TCP localhost.localdomain:33570->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   70r  IPv6             471241      0t0    TCP localhost.localdomain:33573->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   71r  IPv6             471245      0t0    TCP localhost.localdomain:33575->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root  652u  IPv6             473955      0t0    TCP localhost.localdomain:profinet-rtm->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   41u  unix 0xffff8803b8d3c980      0t0 408254 socket
java    7308 root   42u  IPv6             408256      0t0    TCP *:tproxy (LISTEN)
java    7308 root   43u  IPv6             408257      0t0    TCP *:8010 (LISTEN)
java    7308 root   44w   REG             202,64   290262 657041 /usr/tomcat_test/logs/localhost_access_log.2014-03-19.txt
java    7308 root   45r   REG             202,64  4197306  10838 /root/mylogfile.log
java    7308 root   46r  IPv6             412768      0t0    TCP *:kti-icad-srvr (LISTEN)
java    7308 root   47r  IPv6             471171      0t0    TCP localhost.localdomain:33536->localhost.localdomain:webcache (CLOSE_WAIT)
java    7308 root   48r  FIFO                0,8      0t0 412772 pipe
java    7308 root   49r  FIFO                0,8      0t0 412772 pipe
java    7308 root   50r   REG                0,9        0   3672 anon_inode
java    7308 root  206u  IPv6             409173      0t0    TCP localhost.localdomain:8016 (LISTEN)


Our Enviroment
Hazelcast 3.1.6
Apache Tomcat/7.0.47
JVM 1.7.0_45-b18
CentOS release 6.4

Hazelcast Configuration File
hz.group.password 123456
hz.managementcenter.enabled true
hz.managementcenter.url http://localhost:8080/mancenter-3.1.6 
hz.network.port 6701
hz.network.portautoincrement true
hz.multicast.enabled false
hz.multicast.group 224.2.2.4
hz.multicast.port 54327
hz.tcpip.enabled true

<hz:hazelcast id="hzInstance">
    <hz:config>
     <hz:group name="${hz.group.name}" password="${hz.group.password}"/>
    <hz:management-center enabled="${hz.managementcenter.enabled}" url="${hz.managementcenter.url}"/>
        <hz:network port="${hz.network.port}" port-auto-increment="${hz.network.portautoincrement}">
            <hz:join>
                <hz:multicast enabled="${hz.multicast.enabled}"
                              multicast-group="${hz.multicast.group}"
                              multicast-port="${hz.multicast.port}"/>
                <hz:tcp-ip enabled="${hz.tcpip.enabled}"/>
            </hz:join>
        </hz:network>
    <hz:map name="accessToken" max-size="1000"
           eviction-percentage="5" eviction-policy="LRU"
           merge-policy="LATEST_UPDATE" max-idle-seconds="1800"/>
    </hz:config>
</hz:hazelcast>

Noctarius

unread,
Mar 19, 2014, 6:02:31 PM3/19/14
to haze...@googlegroups.com
Thanks Yasin,

I guess this is already reported and fixed. There was a problem closing the Management Center connections after uploading data.
As a workaround for now you can disable the Management Center to see if it is still happening.

Chris

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/5d496133-42b4-4718-92f3-66d0bc704e6e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Yasin Tamer

unread,
Mar 20, 2014, 4:13:46 AM3/20/14
to haze...@googlegroups.com, noctar...@googlemail.com

Yes u r right, i closed man-center then this problem not occured....

Emin Demirci

unread,
Mar 20, 2014, 4:18:18 AM3/20/14
to haze...@googlegroups.com, noctar...@googlemail.com
Can you try with the 3.1.7-SNAPSHOT ? 

-- 
Emin Demirci
Hazelcast | Open source in-memory data grid

Sent with Sparrow

Reply all
Reply to author
Forward
0 new messages