vert.x memory leaks?

[Sergey Nekoval]

I tried experimenting with vert.x giving it some production load. We have steadily reproducable OutOfMemory situation.

I noticed that OldGeneration size always keeps increasing (no decrease) until it all dies, which makes me think about connection leaks.
The application that we run is very simple EventBus which uses bus.publish(...) for sending updates to all connected Javascript clients.

The graph below shows my attempts to increase heap size from 750m to (finally) 1.5g.

So far I'm a bit confused with the following:

• Why OldGeneration never decreases? I'm sure lots of JavaScript clients disconnect (moving away from the page), shouldn't these connections be cleared on timeout or smth?
• It could that I ACTUALLY NEED large amount of memory to keep all connections alive. This is ok, but how could I possibly calculate it? there's no way to know how many connections I have, and also what memory usage per connection is expected.
• Does SockJS server-side have any timeout handling/heartbeats?

Regards,

Serge

[Pid]

On 19/07/2012 08:30, Sergey Nekoval wrote:
> I tried experimenting with vert.x giving it some production load. We
> have steadily reproducable OutOfMemory situation.
>
> I noticed that OldGeneration size always keeps increasing (no decrease)
> until it all dies, which makes me think about connection leaks.
> The application that we run is very simple EventBus which uses
> bus.publish(...) for sending updates to all connected Javascript clients.
>
> The graph below shows my attempts to increase heap size from 750m to
> (finally) 1.5g.
>

> <https://lh6.googleusercontent.com/-kYgfFKQ4TLM/UAe1BSwVRJI/AAAAAAAAAmI/PfM1txlS5O8/s1600/vertx-oom.png>

>
> So far I'm a bit confused with the following:
>

> * Why OldGeneration never decreases? I'm sure lots of JavaScript

> clients disconnect (moving away from the page), shouldn't these
> connections be cleared on timeout or smth?

> * It could that I ACTUALLY NEED large amount of memory to keep all

> connections alive. This is ok, but how could I possibly calculate
> it? there's no way to know how many connections I have, and also
> what memory usage per connection is expected.

> * Does SockJS server-side have any timeout handling/heartbeats?
>
> Regards,

Can you post some code that reproduces the issue? It's pretty hard to
diagnose it without an example.


p

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit
> https://groups.google.com/d/msg/vertx/-/3m1cFGb7dBMJ.
> To post to this group, send an email to ve...@googlegroups.com.
> To unsubscribe from this group, send email to
> vertx+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/vertx?hl=en-GB.


--

[key:62590808]

[Sergey Nekoval]

Here is a Groovy verticle (pretty much a bit modified bridge example):

def server = vertx.createHttpServer()

// Serve the static resources
server.requestHandler { req ->
  if (req.uri == '/') req.response.sendFile('my-live/index.html')
  if (req.uri == '/vertxbus.js') req.response.sendFile('my-live/vertxbus.js')
}

def inboundPermitted = []
def outboundPermitted = [[:]]

vertx.createSockJSServer(server).bridge(prefix: '/eventbus', inboundPermitted, outboundPermitted)

server.listen(80)

// Run a private message server using 8080
def eb = vertx.eventBus
vertx.createHttpServer().requestHandler { request ->

    if (request.uri == '/publish/movie' && request.method == 'POST') {
        def body = new Buffer(0)

        request.dataHandler { buffer -> body << buffer }

        request.endHandler {
            println "[${body as String}]"
            JsonObject json = new JsonObject(body as String)
            eb.publish("new.movie", json)
            request.response.with {
                statusCode = 201
                statusMessage = "Movie accepted"
                end()
            }
        }

    } else if (request.uri == '/') {
        request.response.sendFile('my-live/index.html')
    } else if (request.uri == '/vertxbus.js') {
        request.response.sendFile('my-live/vertxbus.js')
    } else {
        request.response.with {
            statusCode = 404
            statusMessage = 'Unavailable'
            end()
        }
    }
}.listen(8080)

[Sergey Nekoval]

The client is a very simple Javascript listener which counts all incoming events:

<script type="text/javascript">
  var eb = null;

  function subscribe(address) {
    if (eb) {
      eb.registerHandler(address, function(msg, replyTo) {
        var cnt = jQuery("#newMoviesCount").text();
        cnt++;
        jQuery("#newMoviesCount").text(cnt);
      });
    }
  }

  function closeConn() {
    if (eb) {
      eb.close();
    }
  }

  function openConn() {
    if (!eb) {
      eb = new vertx.EventBus("http://live.server/eventbus");

      eb.onopen = function() {
        subscribe("new.movie");
      };
 
      eb.onclose = function() {
        // disconnected
        // reconnect after some timeout
        eb = null;
        setTimeout(openConn, 5000);
      };
    }
  }

openConn();

</script>

[Tim Fox]

What version are you using?

Does this happen with master?

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/P7Xfa4lSAzQJ.

> To post to this group, send an email to ve...@googlegroups.com.
> To unsubscribe from this group, send email to
> vertx+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/vertx?hl=en-GB.


--

Tim Fox

Vert.x - effortless polyglot asynchronous application development
http://vertx.io
twitter:@timfox

[Sergey Nekoval]

right now I'm using 1.1.0 final

[Tim Fox]

Yes, you configure SockJS timeouts using the SockJS config that you pass in when calling bridge or installApp on the SockJS server.

[Tim Fox]

This client will only open one connection (unless it fails).

How are you creating all the connections? Do you have many instances of this client running?

Ideally.. can you provide a client which actually opens many connections, so we can replicate the issue?

[Tim Fox]

http://vertx.io/core_manual_java.html#sockjs

See session timeout and heartbeat period

[Sergey Nekoval]

Well yes. The client (browser) opens a connection from the webpage (per user).
However, users are free to navigate away from the page (and I think they do it very frequently) so in theory users' behavior generates a lot of "stale" connections.
Let me think if I can simulate it somehow.

Serge

[Tim Fox]

Maybe create and close connections in a loop from some JS (with a delay) ?

[Sergey Nekoval]

Thanks for the info, unfortunately when I try to adjust timeouts like this
vertx.createSockJSServer(server).bridge([prefix: '/eventbus', heartbeat_period: 10000L, session_timeout: 10000L], inboundPermitted, outboundPermitted)

I get these:
[New I/O  worker #2] 21:32:36,295 SEVERE [deployment-7bf4d6c1-6631-423c-b568-e0fb9082c9ce-0]  Exception in Groovy verticle
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
        at org.vertx.java.core.sockjs.impl.XhrTransport$2.handle(XhrTransport.java:92)
        at org.vertx.java.core.sockjs.impl.XhrTransport$2.handle(XhrTransport.java:88)
        at org.vertx.java.core.http.RouteMatcher.route(RouteMatcher.java:344)
        at org.vertx.java.core.http.RouteMatcher.handle(RouteMatcher.java:74)
        at org.vertx.java.core.http.RouteMatcher.handle(RouteMatcher.java:49)
        at org.vertx.java.core.http.impl.ServerConnection.handleRequest(ServerConnection.java:123)
        at org.vertx.java.core.http.impl.ServerConnection.processMessage(ServerConnection.java:246)
        at org.vertx.java.core.http.impl.ServerConnection.handleMessage(ServerConnection.java:95)
        at org.vertx.java.core.http.impl.DefaultHttpServer$ServerHandler.messageReceived(DefaultHttpServer.java:543)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
......................

On Thursday, July 19, 2012 3:41:48 PM UTC+4, Tim Fox wrote:

http://vertx.io/core_manual_java.html#sockjs

[Tim Fox]

What language are you using?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/jxw1x04HMdoJ.

[Sergey Nekoval]

That's Groovy. I also tried casting to java.lang.Long with no luck...

[Tim Fox]

That would imply the thing in the map is an Integer not a Long. Is
10000L the correct way to specify a long literal in Groovy?

You can try creating the map first, verifying you have a Long in it,
then passing that to the bridge method.

> <http://vertx.io/core_manual_java.html#sockjs>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit
> https://groups.google.com/d/msg/vertx/-/jxw1x04HMdoJ.
> To post to this group, send an email to ve...@googlegroups.com.
> To unsubscribe from this group, send email to
> vertx+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/vertx?hl=en-GB.

[Guillaume Laforge]

10000L will indeed force a long.

long n = 1000 will also create a long.

To unsubscribe from this group, send email to vertx+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/vertx?hl=en-GB.

--
Tim Fox

Vert.x - effortless polyglot asynchronous application development
http://vertx.io
twitter:@timfox

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To post to this group, send an email to ve...@googlegroups.com.

To unsubscribe from this group, send email to vertx+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/vertx?hl=en-GB.

--
Guillaume Laforge
Groovy Project Manager
SpringSource, a division of VMware

Blog: http://glaforge.appspot.com/

Social: @glaforge / Google+

[Sergey Nekoval]

Even this very explicit snippet:
Map config = [:]
config['prefix'] = '/eventbus'
config['heartbeat_period'] = 10000L as Long
config['session_timeout'] = 10000L as Long

vertx.createSockJSServer(server).bridge(config, inboundPermitted, outboundPermitted)

causes exceptions everywhere.

[Tim Fox]

Please try again from master. Should be fixed now.

[Sergey Nekoval]

Ok. It took me a while to build it.
The result is now tons of java.io.IOException: Connection reset by peer errors when starting server. hmm...

[Tim Fox]

On 20/07/12 09:53, Sergey Nekoval wrote:
> Ok. It took me a while to build it.

Interesting, what was the issue with building? Should be very quick.

> The result is now tons of java.io.IOException: Connection reset by
> peer errors when starting server. hmm...

This is normal. If a connection is closed from the closed from the
client side the server will get this exception. Previously these were
supressed, but we dropped the supression because people requested that.

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/zm2Ud1nfkMoJ.

[Sergey Nekoval]

On Friday, July 20, 2012 1:21:34 PM UTC+4, Tim Fox wrote:

On 20/07/12 09:53, Sergey Nekoval wrote:
> Ok. It took me a while to build it.

Interesting, what was the issue with building? Should be very quick.

I'm building on Windows, maybe that's the problem.
Some dependencies were not resolved (I've added repositories) and some JRuby failures which I don't understand so I've commented them out.
 

This is normal. If a connection is closed from the closed from the
client side the server will get this exception. Previously these were
supressed, but we dropped the supression because people requested that.

Then I would vote to make it configurable, since I'm getting several exceptions per second.

[Tim Fox]

Can you show me what you are doing on the client side? I would like to
replicate what you are doing.

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/STfngtpalcIJ.

[Tim Fox]

On 20/07/12 09:53, Sergey Nekoval wrote:

> Ok. It took me a while to build it.
> The result is now tons of java.io.IOException: Connection reset by
> peer errors when starting server.

Can you please post the stack trace?

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/zm2Ud1nfkMoJ.

[Tim Fox]

Actually don't worry... I've just removed an errant t.printStackTrace() from the code ;)

Can you try again from master?

> vertx+unsubscribe@googlegroups.com.

[Tim Fox]

Sergey,

Just wondering if you're still seeing the memory leak?

[Sergey Nekoval]

Hi,

Yes I still have the problem. After playing with timeouts for some time (1.2.0-snapshot) memory usage increases indefinitely (takes several hours to crash).
Wondering how to make test of this...

Sergey,

> vertx+un...@googlegroups.com.

[Tim Yates]

Did you try 1.2.3.final instead of 1.2.0-snapshot?

To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/7IwkdCSu5H4J.

[Sergey Nekoval]

Nope, have to try the latest one in production... this will take a while...

[Tim Fox]

On Friday, August 17, 2012 11:40:41 AM UTC+1, Sergey Nekoval wrote:

Hi,

Yes I still have the problem. After playing with timeouts for some time (1.2.0-snapshot) memory usage increases indefinitely (takes several hours to crash).
Wondering how to make test of this...

If you can figure out what your clients are doing, hopefully you can write some client side JS which emulates that... 

[Ron Yang]

Have you examined your heap dump in, e.g., Eclipse MAT?

[Sergey Nekoval]

Hi,

I tried to do the simplest thing and used jmap to generate heap histogram of the running vertx instance, here is the top of the list:

 num     #instances         #bytes  class name
----------------------------------------------
   1:       2741077      278351304  [C
   2:       3547074      113506368  java.lang.String
   3:         32768       79609856  [Lorg.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry;
   4:       1783756       57080192  java.util.concurrent.ConcurrentHashMap$HashEntry
   5:        591274       47301584  [Ljava.util.concurrent.ConcurrentHashMap$Segment;
   6:       1145601       45824040  java.util.concurrent.ConcurrentHashMap$Segment
   7:        407657       39135072  org.vertx.java.core.sockjs.impl.Session
   8:       1178525       37712800  java.util.concurrent.locks.ReentrantLock$NonfairSync
   9:        504895       37481968  [Ljava.util.HashMap$Entry;
  10:       1145601       36096936  [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
  11:        897964       35918560  org.jboss.netty.handler.codec.http.HttpHeaders$Entry
  12:        910125       29124000  java.util.LinkedList
  13:        591274       28381152  java.util.concurrent.ConcurrentHashMap
  14:        504836       24232128  java.util.HashMap
  15:        723975       23167200  java.util.HashMap$Entry
  16:        639092       20450944  org.jboss.netty.buffer.BigEndianHeapChannelBuffer
  17:        488788       19048072  [B
  18:        516562       16529984  java.util.concurrent.LinkedTransferQueue$Node
  19:        381998       15279920  org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext
  20:        591311       14191464  org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerInfo
  21:        591311       14191464  org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerHolder
  22:        591226       14189424  org.vertx.java.core.eventbus.impl.DefaultEventBus$Handlers
  23:        865607       13849712  java.lang.Object
  24:         95499       11459880  org.jboss.netty.channel.socket.nio.NioAcceptedSocketChannel
  25:         95499       10695888  sun.nio.ch.SocketChannelImpl
  26:        407657        9783768  org.vertx.java.core.sockjs.impl.EventBusBridge$1
  27:        407657        9783768  org.vertx.java.core.sockjs.impl.EventBusBridge$2
  28:        235517        9420680  org.jboss.netty.channel.DefaultChannelFuture
  29:        235294        9411760  org.jboss.netty.buffer.DynamicChannelBuffer
  30:        381170        9148080  java.net.InetSocketAddress
  31:         92501        8140088  [Lorg.jboss.netty.handler.codec.http.HttpHeaders$Entry;
  32:        328708        7888992  org.vertx.java.core.sockjs.impl.EventBusBridge$2$1
  33:        235398        7532736  org.jboss.netty.channel.DownstreamMessageEvent
  34:         91784        6608448  org.vertx.java.core.http.impl.DefaultWebSocket
  35:        407657        6522512  org.vertx.java.core.sockjs.SockJSSocket$1
  36:        407555        6520880  java.util.HashMap$EntrySet
  37:        191000        6112000  java.net.Inet4Address
  38:         94753        6064192  org.vertx.java.core.http.impl.ServerConnection
  39:        123008        5904384  org.jboss.netty.buffer.CompositeChannelBuffer

hmm.. is having 400K instances of SockJS Session okay?

Serge

[jad younan]

I'm getting OOM errors too, the GC doesn't seem to collect the heap memory. Clients connect using sockjs javascript:  new SockJS('http://'+this.host+':'+this.port+'/'+this.path)

The server is very basic, registering socket handles, and pushing updates.

        JsonObject cfg = new JsonObject();

        cfg.putString("prefix", "/chat");

        cfg.putNumber("heartbeat_period", 10000L);

        cfg.putNumber("session_timeout", 10000L);

        sockServer.installApp(cfg, new Handler<SockJSSocket>() {

            @Override

            public void handle(final SockJSSocket sock) {

                final String socketHandle = (sock.writeHandlerID);

 

                sock.dataHandler(new Handler<Buffer>() {

                    @Override

                    public void handle(Buffer data) {

                        try { 

                            JSONObject obj = new JSONObject(new String(data.getBytes()));

 

                            incoming(socketHandle, obj);

                        } catch (Exception ex) {

                            ex.printStackTrace();

                        }

                    }

                });

            }

        });

void incoming(..){

[Dmitry Sklyut]

I was getting OOMs even with 4-8G heaps.  Always in the Jackson conversion code.  In my case I do have few hundred thousand messages peak at one point.

After some soul searching and a healthy dose of Laphroiag, following jvm configuration helped me:

-XX:+UseG1GC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods

Out of the three, first one is the clear winner.  G1 gc collector made a world of difference.  Instead of bombing out with OOMs with 8G heap I am happily sitting at 2G with same load.

Hope it helps

Dmitry

[Tim Fox]

Do you have a client I can run which replicates the issue?

> <https://lh6.googleusercontent.com/-kYgfFKQ4TLM/UAe1BSwVRJI/AAAAAAAAAmI/PfM1txlS5O8/s1600/vertx-oom.png>

>
>
>
> So far I'm a bit confused with the following:
>

> * Why OldGeneration never decreases? I'm sure lots of JavaScript

> clients disconnect (moving away from the page), shouldn't
> these connections be cleared on timeout or smth?

> * It could that I ACTUALLY NEED large amount of memory to keep

> all connections alive. This is ok, but how could I possibly
> calculate it? there's no way to know how many connections I
> have, and also what memory usage per connection is expected.

> * Does SockJS server-side have any timeout handling/heartbeats?
>
> Regards,
>
> Serge
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/VyNwfhuc478J.

[jad younan]

G1GC did not help much. I wonder if the garbage collector can make a difference with memory leaks, as sockjs objects still have references in the main program.

[Sam Pullara]

While your service is running and filling the heap, get a histogram of the objects in the live heap:

jmap -histo:live [process pid]

That will give you a good idea what is hanging around.

Sam

[jad younan]

Here's a printout of the top objects. The number of org.vertx.* objects is ridiculous compared to the number of opened connections ( 200 concurrent) and the 1G heap fills up in 4-5 hours.

 num     #instances         #bytes  class name

----------------------------------------------

   1:       1151015 131636096  [C

   2:       1773090  56738880  java.lang.String

   3:        688236  27529440  org.jboss.netty.handler.codec.http.HttpHeaders$Entry

   4:        654082  20930624  java.util.concurrent.ConcurrentHashMap$HashEntry

   5:        216274  17301584  [Ljava.util.concurrent.ConcurrentHashMap$Segment;

   6:        419296  16771840  java.util.concurrent.ConcurrentHashMap$Segment

   7:        452207  14470624  java.util.concurrent.locks.ReentrantLock$NonfairSync

   8:        419296  14308200  [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;

   9:        737701  11803216  java.lang.Object

  10:        266777  10671080  org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext

  11:        216274  10381152  java.util.concurrent.ConcurrentHashMap

  12:        273779        8760928  java.util.HashMap$Entry

  13:         83485        8014560  org.vertx.java.core.sockjs.impl.Session

  14:         66694        8003280  org.jboss.netty.channel.socket.nio.NioAcceptedSocketChannel

  15:         66920        7495040  java.net.SocksSocketImpl

  16:         66859        7488208  sun.nio.ch.SocketChannelImpl

  17:        233642        7476544  java.util.LinkedList

  18:        267108        6410592  java.net.InetSocketAddress

  19:         66339        5837832  [Lorg.jboss.netty.handler.codec.http.HttpHeaders$Entry;

  20:        172219        5511008  java.util.concurrent.LinkedTransferQueue$Node

  21:        215813        5179512  org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerInfo

  22:        215813        5179512  org.vertx.java.core.eventbus.impl.DefaultEventBus$Handlers

  23:        215813        5179512  org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerHolder

  24:         66164        4763808  org.vertx.java.core.http.impl.DefaultWebSocket

  25:        133746        4279872  java.net.Inet4Address

  26:         66645        4265280  org.vertx.java.core.http.impl.ServerConnection

  27:         28516        4003056  <constMethodKlass>

  28:         28516        3889936  <methodKlass>

  29:         67825        3347240  [Ljava.util.HashMap$Entry;

  30:        204511        3272176  java.util.concurrent.atomic.AtomicInteger

  31:         67781        3253488  java.util.HashMap

  32:        133400        3201600  java.util.concurrent.LinkedTransferQueue

  33:         66695        3201360  org.jboss.netty.channel.AbstractChannel$ChannelCloseFuture

  34:         66694        3201312  org.jboss.netty.channel.socket.nio.DefaultNioSocketChannelConfig

  35:         72571        3039960  [B

  36:         63201        3033648  org.vertx.java.core.http.impl.ws.hybi08.WebSocketFrameDecoder08

  37:         66987        2679480  java.lang.ref.Finalizer

  38:         66859        2674360  sun.nio.ch.SocketAdaptor

  39:         66252        2650080  org.jboss.netty.handler.codec.http.DefaultHttpRequest

  40:          2313        2622832  <constantPoolKlass>

  41:         66860        2139520  [Ljava.nio.channels.SelectionKey;

  42:        133396        2134336  java.util.concurrent.atomic.AtomicBoolean

  43:         66695        2134240  org.jboss.netty.channel.DefaultChannelPipeline

  44:         66694        2134208  org.jboss.netty.handler.stream.ChunkedWriteHandler

[Tim Fox]

The way SockJS works (if you're using a polling transport) is the session will stay open on the server until timeout (default 5 mins) - with polling transports it's a new connection each time so the server can't just close the session on the server when a polling connection closes.

So, if you have a lot of clients or you're creating and throwing away (rather than reusing) a lot of sockjs sockets in a short space of time then you can end up with a lot of sockjs sessions on the server waiting to timeout.

In your case there are are about 84k sessions, which is probably what is holding on to the memory.

You can try reducing the timeout, or refactor your clients if possible to reuse sockjs sockets.

I've said this a few times before but if someone could replicate the issue in a simple client, or give some clue as to what the clients are doing, that's critical to fixing this issue. Without that it's guesswork.

[Tim Fox]

Can you show the client code, and give an indication of the number of clients, and how often the client code is executed?

[jad younan]

Tim- the client encapsulates the sockjs connection to add manual ping/heartbeat. 

(function() {
	var Socket = function(host, port, path) {
	var ref = this;
	this.host = host;
	this.port = port;
	this.path = path;
	this.debug = 0;
	if (typeof console === undefined) {
	this.debug = 0;
	}
	};
	Socket.prototype.doConnect = function() {
	var ref = this;
	try {
	if (typeof this.sock !== 'undefined') {
	this.sock.close();
	}
	} catch(e) {
	if (ref.debug === 1) {
	console.log(e);
	}
	}
	this.sock = new SockJS('http://'+this.host+':'+this.port+'/'+this.path);
	this.lastSeen = new Date();
	this.sock.onopen = function() {
	var obj;
	ref.onopen();
	ref.sock.onmessage = function(e) {
	if (ref.debug === 1) {
	var blah = JSON.parse(e.data);
	if (blah.payload.action) console.log('<~~~ ', blah.payload.action, blah.payload);
	else console.log('<~~~ ', blah.payload);
	}
	ref.lastSeen = new Date();
	obj = JSON.parse(e.data.toString());
	if (obj.system === '1') {
	ref.sock.send(
	JSON.stringify({
	system:'1', payload:obj.payload
	})
	);
	} else {
	ref.onmessage(obj.payload);
	}
	};
	};
	setInterval(function() {
	if (new Date()-ref.lastSeen>10*1000) {
	if (ref.debug === 1) {
	console.log('not connected, re-connecting...'+(new Date()-ref.lastSeen));
	}
	setTimeout(function() {
	ref.doConnect();
	}, 4000+Math.random()*4000);
	}
	}, 10*1000);
	};
	Socket.prototype.onopen = function() {
	};
	Socket.prototype.send = function(obj) {
	if (this.sock.readyState === 1) { //WebSocket.OPEN
	if (this.debug === 1) {
	console.log('****~>', JSON.stringify({payload:obj}));
	}
	this.sock.send(JSON.stringify({system:'0', payload:obj}));
	} else {
	if (this.debug === 1) {
	console.log("The socket is not open.");
	}
	}
	};
	Socket.prototype.onmessage = function(obj) {
	};
	var chat = function() {
	this.sock = new Socket('chost2', 80, 'chat');
	this.sock.doConnect();
	};
	})();

[Tim Fox]

Can you send me something runnable? Your code snippet doesn't tell me how you're using it. Also, can you answer my previous questions about number of clients etc?

--

You received this message because you are subscribed to the Google Groups "vert.x" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/M-X7GL0leNAJ.

[jad younan]

Sure thing:

try this link:

http://ch.woopra.com/web/chat.html?cookie=HYO09CBESW3WNSG003I664VLBY84FDP2&host=jadyounan.com&username=jad&title=CTO&chat_name=Jad%20Y.&chat_avatar=%2F%2Fstatic.woopra.com%2Favatars%2F1f98d254fcfc24ee525c8b5f5f2aa2e4.jpg&chat_color=1d2642&imsg=Test&vid=391&w=1&chat_biz=true&ra=W9IK4C3LTMVLZN1H

It should open a chat window. There are around a thousand concurrent connection in peak times, data transfer is relatively low ( less than 10kbps ). 

[Tim Fox]

This doesn't help me much either. I'm looking for something small and self contained that I can run to replicate the issue.

I'd also like to understand the usage patterns of your clients. I'm interested in how and when you create SockJS connections (is there just one per client application, or are you creating them every time you send messages.

If you can pretend I'm an alien (not hard to do), and you're trying to explain to me how your app works, that would be a start ;)

To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/D4BijG6IsMIJ.

[Tim Fox]

Ok... so I took a look at your code.

Your hand-rolled time out stuff doesn't look right to me: every time you call doConnect you call setInterval to fire every 10 seconds- when this fires you set a random timeout to close and reconnect. However I can't see anywhere where you cancel the setInteval. 

So even with just one connection, after a minutes you're going to have many period timers firing all the time and opening and closing the connection.

I'm not sure what you're trying to achieve with this code, but I'd recommend removing that timer stuff, running it again and seeing if you have the same issues.

To unsubscribe from this group, send email to vertx+unsubscribe@googlegroups.com.

[Tim Fox]

BTW, i've spent a bit of time looking at this today, and I've found a bug in the SockJS server side code which would create a memory leak when using some of the non-websockets transports.

I'll work on this a bit more today and tomorrow and see if I can get a fix in master.

On Thursday, July 19, 2012 8:49:54 AM UTC+1, Sergey Nekoval wrote:

Here is a Groovy verticle (pretty much a bit modified bridge example):

def server = vertx.createHttpServer()

// Serve the static resources
server.requestHandler { req ->
  if (req.uri == '/') req.response.sendFile('my-live/index.html')
  if (req.uri == '/vertxbus.js') req.response.sendFile('my-live/vertxbus.js')
}

def inboundPermitted = []
def outboundPermitted = [[:]]

vertx.createSockJSServer(server).bridge(prefix: '/eventbus', inboundPermitted, outboundPermitted)

server.listen(80)

// Run a private message server using 8080
def eb = vertx.eventBus
vertx.createHttpServer().requestHandler { request ->

    if (request.uri == '/publish/movie' && request.method == 'POST') {
        def body = new Buffer(0)

        request.dataHandler { buffer -> body << buffer }

        request.endHandler {
            println "[${body as String}]"
            JsonObject json = new JsonObject(body as String)
            eb.publish("new.movie", json)
            request.response.with {
                statusCode = 201
                statusMessage = "Movie accepted"
                end()
            }
        }

    } else if (request.uri == '/') {
        request.response.sendFile('my-live/index.html')
    } else if (request.uri == '/vertxbus.js') {
        request.response.sendFile('my-live/vertxbus.js')
    } else {
        request.response.with {
            statusCode = 404
            statusMessage = 'Unavailable'
            end()
        }
    }
}.listen(8080)

[Tim Fox]

I have completed a fix for this in master, if anyone wants to try it out. I also reduced the default session_timeout to 5 seconds which brings into line with the value used in sockjs-node

[jad younan]

Tim- I just used the master code, still facing the same bug. However, after browsing in the code I believe (Actually I'm pretty sure) that websockets are causing the problem. Removing the websocket transport does not cause the OOM anymore, the process has been up for over 24 hours vs 4 hours with websockets.

Here's my java config code:

 JsonObject config = new JsonObject();

 config.putString("prefix", "/chat");

 config.putNumber("heartbeat_period", 10000L);

 config.putNumber("session_timeout", 10000L);

 JsonArray disabledTransports = new JsonArray();

 disabledTransports.add(Transport.WEBSOCKET.toString());

 config.putArray("disabled_transports", disabledTransports);

Can anyone try it and confirm?

[Tim Fox]

What client code are you using?

This is the missing piece of the puzzle for me.

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/chhhONdg6DUJ.

> To post to this group, send an email to ve...@googlegroups.com.
> To unsubscribe from this group, send email to
> vertx+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/vertx?hl=en-GB.


--

[jad younan]

the client code can be downloaded here:  http://198.58.99.196/web/web.zip

[Tim Fox]

Doesn't work for me. I tried to download using wget, and it's not a
valid zip file.

(Doesn't work using a browser either)

> <https://groups.google.com/d/msg/vertx/-/chhhONdg6DUJ>.

> > To post to this group, send an email to ve...@googlegroups.com

> <javascript:>.

> > To unsubscribe from this group, send email to

> > vertx+un...@googlegroups.com <javascript:>.

> > For more options, visit this group at
> > http://groups.google.com/group/vertx?hl=en-GB

> <http://groups.google.com/group/vertx?hl=en-GB>.

>
>
> --
> Tim Fox
>
> Vert.x - effortless polyglot asynchronous application development
> http://vertx.io
> twitter:@timfox
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/c112Uf6MQQQJ.

[jad younan]

[dev@localhost ~]$ wget  http://198.58.99.196/web/web.zip

--2012-10-11 04:46:24--  http://198.58.99.196/web/web.zip

Connecting to 198.58.99.196:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 34566 (34K) [application/octet-stream]

Saving to: `web.zip'

100%[======================================>] 34,566      63.9K/s   in 0.5s    

2012-10-11 04:46:25 (63.9 KB/s) - `web.zip' saved [34566/34566]

[dev@localhost ~]$ 

It appears valid to me, can you try again?

[jad younan]

Attached. 

[Tim Fox]

You just saved *a file* doesn't mean it's a valid zip file. Try
unzipping it and you will see what I mean.

On 11/10/12 12:47, jad younan wrote:
>
> [dev@localhost ~]$ wget http://198.58.99.196/web/web.zip
> --2012-10-11 04:46:24-- http://198.58.99.196/web/web.zip
> Connecting to 198.58.99.196:80... connected.
> HTTP request sent, awaiting response... 200 OK

> *Length: 34566 (34K) [application/octet-stream]*

> Saving to: `web.zip'
>
> 100%[======================================>] 34,566 63.9K/s in 0.5s
>
> 2012-10-11 04:46:25 (63.9 KB/s) - `web.zip' saved [34566/34566]
>
> [dev@localhost ~]$
>
> It appears valid to me, can you try again?
>
> On Thursday, October 11, 2012 4:43:19 AM UTC-7, Tim Fox wrote:
>
> Doesn't work for me. I tried to download using wget, and it's not a
> valid zip file.
>
> (Doesn't work using a browser either)
>
> On 11/10/12 12:40, jad younan wrote:
> > the client code can be downloaded here:

> http://198.58.99.196/web/web.zip <http://198.58.99.196/web/web.zip>

> <https://groups.google.com/d/msg/vertx/-/c112Uf6MQQQJ>.

> > To post to this group, send an email to ve...@googlegroups.com
> <javascript:>.
> > To unsubscribe from this group, send email to
> > vertx+un...@googlegroups.com <javascript:>.
> > For more options, visit this group at
> > http://groups.google.com/group/vertx?hl=en-GB
> <http://groups.google.com/group/vertx?hl=en-GB>.
>
>
> --
> Tim Fox
>
> Vert.x - effortless polyglot asynchronous application development
> http://vertx.io
> twitter:@timfox
>
> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/MWRTbKWmFHEJ.

[jad younan]

[T.J. Crowder]

On Thursday, 11 October 2012 12:47:01 UTC+1, jad younan wrote:

[dev@localhost ~]$ wget  http://198.58.99.196/web/web.zip

--2012-10-11 04:46:24--  http://198.58.99.196/web/web.zip

Connecting to 198.58.99.196:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 34566 (34K) [application/octet-stream]

Saving to: `web.zip'

100%[======================================>] 34,566      63.9K/s   in 0.5s    

2012-10-11 04:46:25 (63.9 KB/s) - `web.zip' saved [34566/34566]

[dev@localhost ~]$ 

It appears valid to me, can you try again?

Tim didn't say he couldn't download it, he said it wasn't a valid zip file.

I can confirm that. I downloaded it both from  http://198.58.99.196/web/web.zip and from your attachment to your on-list message just now. In both cases, I get a file that's 34.6k in size (34566 bytes, the same as your quoted wget above). When I try to unzip it, I get this error:

Archive:  /home/tjc/Downloads/web.zip

Zip file size: 34566 bytes, number of entries: 10

error [/home/tjc/Downloads/web.zip]:  missing 3282843907 bytes in zipfile

  (attempting to process anyway)

error [/home/tjc/Downloads/web.zip]:  attempt to seek before beginning of zipfile

  (please check that you have transferred or created the zipfile in the

  appropriate BINARY mode and that you have compiled UnZip properly)

-- T.J.

[Tim Fox]

Ok, this is just the complete code from your application that you sent a
link to the other day.

It's normal practice in just about any project I know when making bug
reports to make some attempt to replicate the issue in a small self
contained piece of code, not to just dump your entire application and
say "it leaks" ;)

I also notice that you're using the same piece of code with what looked
like buggy reconnect logic, that I pointed out in another reply.

On 11/10/12 12:52, jad younan wrote:
>
>
> On Thursday, July 19, 2012 12:30:00 AM UTC-7, Sergey Nekoval wrote:
>
> I tried experimenting with vert.x giving it some production load.
> We have steadily reproducable OutOfMemory situation.
>
> I noticed that OldGeneration size always keeps increasing (no
> decrease) until it all dies, which makes me think about connection
> leaks.
> The application that we run is very simple EventBus which uses
> bus.publish(...) for sending updates to all connected Javascript
> clients.
>
> The graph below shows my attempts to increase heap size from 750m
> to (finally) 1.5g.
>

> <https://lh6.googleusercontent.com/-kYgfFKQ4TLM/UAe1BSwVRJI/AAAAAAAAAmI/PfM1txlS5O8/s1600/vertx-oom.png>

>
>
>
> So far I'm a bit confused with the following:
>

> * Why OldGeneration never decreases? I'm sure lots of JavaScript

> clients disconnect (moving away from the page), shouldn't
> these connections be cleared on timeout or smth?

> * It could that I ACTUALLY NEED large amount of memory to keep

> all connections alive. This is ok, but how could I possibly
> calculate it? there's no way to know how many connections I
> have, and also what memory usage per connection is expected.

> * Does SockJS server-side have any timeout handling/heartbeats?
>
> Regards,
>
> Serge
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/KW3fczZEDP8J.

[jad younan]

I apologize for that, i re-uploaded it.

[Tim Fox]

Ok .. (puts on constructive hat). Your app looks pretty simple. I'm going to run it here against your server (can you post your server code too please?) And try and figure out what is going on.

> vertx+unsubscribe@googlegroups.com.

[jad younan]

Tim-

I will refactor the code into a single contained file, though the socket logic is all in a single file and very similar to the example sockjs example code in the vertx downloadable. That said, we have been successfully running a version without websockets for over 2 days now without any issue. 

[Tim Fox]

Np. Feel free to send me the code directly if you don't want to post it
on the group.

> <https://groups.google.com/d/msg/vertx/-/KW3fczZEDP8J>.

> > To post to this group, send an email to ve...@googlegroups.com

> <javascript:>.

> > To unsubscribe from this group, send email to

> > vertx+un...@googlegroups.com <javascript:>.

> > For more options, visit this group at
> > http://groups.google.com/group/vertx?hl=en-GB

> <http://groups.google.com/group/vertx?hl=en-GB>.

>
>
> --
> Tim Fox
>
> Vert.x - effortless polyglot asynchronous application development
> http://vertx.io
> twitter:@timfox
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/NJRebQJdUgwJ.

[Tim Fox]

Are you talking about the client or server code? I have some free time now to look at this, but I don't have your _server_ code. Can you post that?

[Tim Fox]

I've got your chat app working with a simple echo server, let's hope I can replicate the issue with that

[Tim Fox]

That doesn't make a lot of sense to me, you wouldn't have any Session objects in your heap dump unless you were using a non websockets transports since websockets don't use Session.

[Tim Fox]

Here are my new results:

Using current vert.x from master, I Created a simple echo SockJS server and ran your client side application against that.

I adapted your client side app to create 200 instances of the Chat() object - which is the number of concurrent connections you said you had.

I modified the SockJS server to display how many sessions it has every second.

With this setup there is no heartbeat the 10 second client timeout kicks in quickly for each of the clients, this causes a random 4 second delay to kick after which reconnection is attempted. As I pointed out in an earlier post, your client code is buggy as it does not cancel the timer created in the call to setInterval().

The end results of this is, after a few minutes of doing nothing, soon many many timers are firing all over the place and connections are being opened and closed very rapidly. At this point I can see hundreds of sessions on the server (event though there are only 200 chat instances).

After a few more minutes the entire virtual machine I am running in has to hard reset (!)

I fixed your client code, to cancel the timer, as so (Note the call to clearInterval):

var id = setInterval(function() {

            console.log("timeout fired");

            if (new Date()-ref.lastSeen>10*1000) {

                console.log('not connected, re-connecting...'+(new Date()-ref.lastSeen));

                clearInterval(id);

                setTimeout(function() {

                    ref.doConnect();

                }, 4000+Math.random()*4000);

            }

        }, 10*1000);

Then ran it again. Now it runs without problem and I no longer see more than 200 sessions on the server.

That's the only issue I have seen so far.

[Tim Fox]

BTW, that was with xhr_polling.

I've tried the same with websockets, and don't see any evidence of memory leaks either.

If you can provide an example which I can run which demonstrates any memory leaks, I will happily look at this again.

[phront]

Hi, all

i have just started using vertx.io and have  faced a memory leak problem too.

here is jmap histogram http://pastebin.com/ztwyVJDA   

can anyone tell me where is the problem in my code  http://pastebin.com/9VDx28F9 ?

thanx in advance.

[Tim Fox]

Are you using master? I recently fixed a memory leak there.

Also please post your client code - can't tell by just looking at the
server stuff.

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/k6mngfjzZnUJ.

[phront]

thanx for your reply

no, 1.2.3 final (our vertx runs inproc in a play2framework application, that still uses netty 3.5.0)

the client code is here http://pastebin.com/333Nn6ZJ

we tried to apply https://github.com/vert-x/vert.x/commit/70d44d5774ca58bf0830e6455dea7c797178e2c6 in testing environment, but with no effect  

пятница, 19 октября 2012 г., 22:33:52 UTC+10 пользователь Tim Fox написал:

[phront]

also  jhat shows tons of org.vertx.java.core.sockjs.impl.Session objects 

http://pastebin.com/RshZztG9

пятница, 19 октября 2012 г., 22:33:52 UTC+10 пользователь Tim Fox написал:

Are you using master? I recently fixed a memory leak there.

[Tim Fox]

Just because you have lots of sessions doesn't mean you have a memory leak. This has been discussed a few times, but the way SockJS works is that every you create a new connection from the client you'll get a new session on the server.

This session will hang around for the session timeout. So if you have a lot of clients, or you're creating connections rapidly from a particular client you'll see a lot of sessions.

[phront]

can a session object hang around  alive after timeout ? please, look at this url http://pastebin.com/RshZztG9

corresponding org.vertx.java.core.sockjs.impl.WebSocketTransport is closed.

1. timeout (J) : -1
2. timeoutTimerID (J) : -1
   
   

пятница, 19 октября 2012 г., 23:33:51 UTC+10 пользователь Tim Fox написал:

[Tim Fox]

According to that, the session (it's a websocket session) is being prevented from garbage collection since it's held by the field "sock" of an instance of a class ru.ykt.notifier.Notifier-  I guess this is one of your own classes.

ru.ykt.notifier.Notifier$2$2@0x78426a828 (32 bytes) : field val$sock

[phront]

Notifier$2$2 is this

       sockServer.installApp(config, new Handler<SockJSSocket>() {

            public void handle(final SockJSSocket sock) {

.....

                sock.dataHandler(new Handler<Buffer>() {

                    public void handle(Buffer data) {

                        JsonObject object = new JsonObject(data.toString());

                        String type = object.getString("type", "empty");

пятница, 19 октября 2012 г., 23:51:36 UTC+10 пользователь Tim Fox написал:

[Tim Fox]

Is what? I don't understand what you mean.

On 19/10/12 15:10, phront wrote:
>
> Notifier$2$2 is this
> |
> sockServer.installApp(config, new Handler<SockJSSocket>() {
> public void handle(final SockJSSocket sock) {
>
> .....
> sock.dataHandler(new Handler<Buffer>() {
> public void handle(Buffer data) {
>
> JsonObject object = new JsonObject(data.toString());
> String type = object.getString("type", "empty");
>
> |
>
>
>

> О©╫О©╫О©╫О©╫О©╫О©╫О©╫, 19 О©╫О©╫О©╫О©╫О©╫О©╫О©╫ 2012 О©╫., 23:51:36 UTC+10 О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫ Tim Fox
> О©╫О©╫О©╫О©╫О©╫О©╫О©╫:

>
> According to that, the session (it's a websocket session) is being
> prevented from garbage collection since it's held by the field
> "sock" of an instance of a class ru.ykt.notifier.Notifier- I
> guess this is one of your own classes.
>
> ru.ykt.notifier.Notifier$2$2@0x78426a828 (32 bytes) : field val$sock
>
>
>
> On Friday, October 19, 2012 2:43:27 PM UTC+1, phront wrote:
>
> can a session object hang around alive after timeout ?
> please, look at this url http://pastebin.com/RshZztG9
> <http://pastebin.com/RshZztG9>
>
> corresponding org.vertx.java.core.sockjs.impl.WebSocketTransport
> is closed.
>

> 1.
> timeout (J) : -1
> 2.
> timeoutTimerID (J) : -1
>
>
> О©╫О©╫О©╫О©╫О©╫О©╫О©╫, 19 О©╫О©╫О©╫О©╫О©╫О©╫О©╫ 2012 О©╫., 23:33:51 UTC+10 О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫ Tim
> Fox О©╫О©╫О©╫О©╫О©╫О©╫О©╫:

>
> Just because you have lots of sessions doesn't mean you
> have a memory leak. This has been discussed a few times,
> but the way SockJS works is that every you create a new
> connection from the client you'll get a new session on the
> server.
>
> This session will hang around for the session timeout. So
> if you have a lot of clients, or you're creating
> connections rapidly from a particular client you'll see a
> lot of sessions.
>
> On Friday, October 19, 2012 2:14:50 PM UTC+1, phront wrote:
>
>
> thanx for your reply
>
> no, 1.2.3 final (our vertx runs inproc in a
> play2framework application, that still uses netty 3.5.0)
> the client code is here http://pastebin.com/333Nn6ZJ
> <http://pastebin.com/333Nn6ZJ>
>
> we tried to apply
> https://github.com/vert-x/vert.x/commit/70d44d5774ca58bf0830e6455dea7c797178e2c6
> <https://github.com/vert-x/vert.x/commit/70d44d5774ca58bf0830e6455dea7c797178e2c6> in
> testing environment, but with no effect
>

> О©╫О©╫О©╫О©╫О©╫О©╫О©╫, 19 О©╫О©╫О©╫О©╫О©╫О©╫О©╫ 2012 О©╫., 22:33:52 UTC+10
> О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫ Tim Fox О©╫О©╫О©╫О©╫О©╫О©╫О©╫:

>
> Are you using master? I recently fixed a memory
> leak there.
>
> Also please post your client code - can't tell by
> just looking at the
> server stuff.
>
> On 19/10/12 13:08, phront wrote:
> >
> >
> >
> > Hi, all
> >
> > i have just started using vertx.io

> <http://vertx.io> and have faced a memory leak

> > problem too.
> > here is jmap histogram http://pastebin.com/ztwyVJDA
> >
> > can anyone tell me where is the problem in my code
> > http://pastebin.com/9VDx28F9 ?
> >
> > thanx in advance.
> >
> > --
> > You received this message because you are
> subscribed to the Google
> > Groups "vert.x" group.
> > To view this discussion on the web, visit
> >
> https://groups.google.com/d/msg/vertx/-/k6mngfjzZnUJ

> <https://groups.google.com/d/msg/vertx/-/k6mngfjzZnUJ>.

>
> > To post to this group, send an email to
> ve...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > vertx+un...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/vertx?hl=en-GB

> <http://groups.google.com/group/vertx?hl=en-GB>.

>
>
> --
> Tim Fox
>
> Vert.x - effortless polyglot asynchronous
> application development
> http://vertx.io
> twitter:@timfox
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/UfTJPCSLmkIJ.

[Tim Fox]

If you post your entire server side code showing the class ru.ykt.notifier.Notifier it would be a lot easier than posting snippets.

> vertx+unsubscribe@googlegroups.com.

[Tim Fox]

Also can you post your DbWorker class? The Runnable that you're adding to the queue will hold a reference to the SockJS socket and therefore to the Session too. If you're not removing the Runnable from the queue then you'll get a leak. I can't see anywhere in the code you posted where you're doing that.

I would recommend using a static class to implement your Runnable and initialise that with the values from your socket. That way you won't be unnecessarily holding on to everything and the kitchen sink after the socket has closed.

[phront]

it seems to me that ru.ykt.notifier.Notifier$2$2@0x78426a828 (32 bytes) : field val$sock cannot prevent the session object to be collected because it  is a  Handler object of installApp and   sock  is a final parameter . 

also jhat shows this  static references for  the Session object.  

Java Static References

Static reference from ru.ykt.notifier.Notifier.server (from class ru.ykt.notifier.Notifier) :

--> org.vertx.java.core.http.impl.DefaultHttpServer@0x782d441c8 (121 bytes) (field ctx:)
--> org.vertx.java.core.impl.EventLoopContext@0x783431d68 (56 bytes) (field closeHooks:)
--> java.util.HashMap@0x783a73a60 (64 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0x78370a0e0 (144 bytes) (Element 11 of [Ljava.util.HashMap$Entry;@0x78370a0e0:)
--> java.util.HashMap$Entry@0x783f5f258 (44 bytes) (field value:)
--> org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerCloseHook@0x783f58af0 (32 bytes) (field entries:)
--> java.util.HashSet@0x783f58b08 (24 bytes) (field map:)
--> java.util.HashMap@0x783f76c38 (64 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0x78515eb20 (262160 bytes) (Element 5199 of [Ljava.util.HashMap$Entry;@0x78515eb20:)
--> java.util.HashMap$Entry@0x784274188 (44 bytes) (field key:)
--> org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerEntry@0x78426bcb0 (40 bytes) (field handler:)
--> org.vertx.java.core.sockjs.SockJSSocket$1@0x78426a810 (24 bytes) (field this$0:)
--> org.vertx.java.core.sockjs.impl.Session@0x78426e6c8 (164 bytes) 

[Tim Fox]

Your runnable object (that you add to the DbWorker with
dbWorker.addTask) maintains a reference to the sock. You add this
Runnable to your queue. So unless you remove it from the queue, you have
a leak there. That's why I asked to see your DBWorker class, and
recommended using a static class instead for your Runnable.

On 19/10/12 15:55, phront wrote:
>
>
> it seems to me that ru.ykt.notifier.Notifier$2$2@0x78426a828 (32
> bytes) : field val$sock cannot prevent the session object to be
> collected because it is a Handler object of installApp and
> sock is a final parameter .
>
>
>
> also jhat shows this static references for the Session object.
>

That's probably just unclosed Sessions

>
>
> Java Static References
>
>
> Static reference from ru.ykt.notifier.Notifier.server (from
> class ru.ykt.notifier.Notifier)

> <http://tucanalocal.ykt.ru:7000/object/0x783d69bf0> :
>
> --> org.vertx.java.core.http.impl.DefaultHttpServer@0x782d441c8 (121
> bytes) <http://tucanalocal.ykt.ru:7000/object/0x782d441c8> (field ctx:)
> --> org.vertx.java.core.impl.EventLoopContext@0x783431d68 (56 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x783431d68> (field closeHooks:)
> --> java.util.HashMap@0x783a73a60 (64 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x783a73a60> (field table:)

> --> [Ljava.util.HashMap$Entry;@0x78370a0e0 (144 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x78370a0e0> (Element 11 of

> [Ljava.util.HashMap$Entry;@0x78370a0e0:)
> --> java.util.HashMap$Entry@0x783f5f258 (44 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x783f5f258> (field value:)
> -->
> org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerCloseHook@0x783f58af0
> (32 bytes) <http://tucanalocal.ykt.ru:7000/object/0x783f58af0> (field

> entries:)
> --> java.util.HashSet@0x783f58b08 (24 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x783f58b08> (field map:)
> --> java.util.HashMap@0x783f76c38 (64 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x783f76c38> (field table:)

> --> [Ljava.util.HashMap$Entry;@0x78515eb20 (262160 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x78515eb20> (Element 5199 of

> [Ljava.util.HashMap$Entry;@0x78515eb20:)
> --> java.util.HashMap$Entry@0x784274188 (44 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x784274188> (field key:)
> -->
> org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerEntry@0x78426bcb0
> (40 bytes) <http://tucanalocal.ykt.ru:7000/object/0x78426bcb0> (field

> handler:)
> --> org.vertx.java.core.sockjs.SockJSSocket$1@0x78426a810 (24 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x78426a810> (field this$0:)
> --> org.vertx.java.core.sockjs.impl.Session@0x78426e6c8 (164 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x78426e6c8>

>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/R-RF6T8CD9gJ.

[Tim Fox]

On 19/10/12 15:55, phront wrote:
>
>

> it seems to me that ru.ykt.notifier.Notifier$2$2@0x78426a828 (32
> bytes) : field val$sock cannot prevent the session object to be
> collected because it is a Handler object of installApp and
> sock is a final parameter .
>
>
>
> also jhat shows this static references for the Session object.
>

I can see a potential memory leak here due to Session not calling
super.close. I'll take a closer look at this over the next few days.

>
>
> Java Static References
>
>
> Static reference from ru.ykt.notifier.Notifier.server (from
> class ru.ykt.notifier.Notifier)

> <http://tucanalocal.ykt.ru:7000/object/0x783d69bf0> :
>
> --> org.vertx.java.core.http.impl.DefaultHttpServer@0x782d441c8 (121

> bytes) <http://tucanalocal.ykt.ru:7000/object/0x782d441c8> (field ctx:)
> --> org.vertx.java.core.impl.EventLoopContext@0x783431d68 (56 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x783431d68> (field closeHooks:)
> --> java.util.HashMap@0x783a73a60 (64 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x783a73a60> (field table:)

> --> [Ljava.util.HashMap$Entry;@0x78370a0e0 (144 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x78370a0e0> (Element 11 of

> [Ljava.util.HashMap$Entry;@0x78370a0e0:)
> --> java.util.HashMap$Entry@0x783f5f258 (44 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x783f5f258> (field value:)
> -->
> org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerCloseHook@0x783f58af0
> (32 bytes) <http://tucanalocal.ykt.ru:7000/object/0x783f58af0> (field

> entries:)
> --> java.util.HashSet@0x783f58b08 (24 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x783f58b08> (field map:)
> --> java.util.HashMap@0x783f76c38 (64 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x783f76c38> (field table:)

> --> [Ljava.util.HashMap$Entry;@0x78515eb20 (262160 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x78515eb20> (Element 5199 of

> [Ljava.util.HashMap$Entry;@0x78515eb20:)
> --> java.util.HashMap$Entry@0x784274188 (44 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x784274188> (field key:)
> -->
> org.vertx.java.core.eventbus.impl.DefaultEventBus$HandlerEntry@0x78426bcb0
> (40 bytes) <http://tucanalocal.ykt.ru:7000/object/0x78426bcb0> (field

> handler:)
> --> org.vertx.java.core.sockjs.SockJSSocket$1@0x78426a810 (24 bytes)

> <http://tucanalocal.ykt.ru:7000/object/0x78426a810> (field this$0:)
> --> org.vertx.java.core.sockjs.impl.Session@0x78426e6c8 (164 bytes)
> <http://tucanalocal.ykt.ru:7000/object/0x78426e6c8>

>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/R-RF6T8CD9gJ.

[phront]

I really appreciate your help.

here is a listing with the vertx server module  http://pastebin.com/7bfed7M9

also i will refactor my code to cut off unnessary depencies, but it seems to me that  i misuse the vertx system.

 

[phront]

thank you very much.

i will wait for results.