Relevant settings for memory utilization of vertx http

[Przemo T]

Hello guys,

I'm writing a vertx based HTTP proxy. One of my requirements is to use as low memory as possible. Right now I've setup JVM with 32M heap and I have test case for proxying 20 simultaneous connections of POST/PUT request (2MB payload size). I'm getting java.lang.OutOfMemoryError: Java heap space error.

at io.netty.buffer.UnpooledUnsafeHeapByteBuf.<init>(UnpooledUnsafeHeapByteBuf.java:29)

        at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160)9)

        at io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1648)ator.java:151)

        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:191)va:400)

        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:735)

        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:50))

        at io.vertx.core.streams.impl.PumpImpl$$Lambda$37/8381481.handle(Unknown Source)

        at io.vertx.core.http.impl.ServerConnection.handleChunk(ServerConnection.java:294)java:373)

        at io.vertx.core.http.impl.ServerConnection.handleMessage(ServerConnection.java:139))

        at io.vertx.core.http.impl.HttpServerImpl$ServerHandler.doMessageReceived(HttpServerImpl.java:522)

        at io.vertx.core.http.impl.VertxHttpHandler$$Lambda$30/682343505.run(Unknown Source)va:71)

        at io.vertx.core.impl.ContextImpl$$Lambda$9/1958411219.run(Unknown Source)

        at io.vertx.core.http.impl.VertxHttpHandler.channelRead(VertxHttpHandler.java:71)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:326)

Can you suggest whether there are any vertx/netty settings that I could try to make my use case working without increasing heap size?

About my app:

I'm deploying N instances of verticle that is my proxy. Every instance creates its own HttpClient with default M pool.

If I understand properly when I make 20 requests - N*M of them can be serviced immediately and the rest of them is enqued to wait for a free connection in pool.

I tried to modify N/M values but always getting OOM (even for 1/1).

I don't buffer full payload. When my server receives request, I create new request with HttpClient and pipe. My HttpClient can not use chunked transfer. But content-length is already known (from original request) so I hope vertx/netty does not buffer whole payload when I run pump. Would be nice if someone could confirm that.

PS: do you know if netty and vertx are reusing the same threads/event loops or does netty has own pool not related to vertx pool?

Any help will be appreciated.

[Jochen Mader]

Hard to tell without seeing your code. Provide us with a reproducer.

A general debugging advice for such scenarios is to enable -XX:+HeapDumpOnOutOfMemoryError and take a look at the generated heapdump (visualvm, Eclipse MAT ...)

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a7ea90c0-fb72-49b0-9ef3-936a38d1175d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jochen Mader | Lead IT Consultant

codecentric AG | Elsenheimerstr. 55a | 80687 München | Deutschland
tel: +49 89 215486633 | fax: +49 89 215486699 | mobil: +49 152 51862390
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de | www.more4fi.de

Sitz der Gesellschaft: Düsseldorf | HRB 63043 | Amtsgericht Düsseldorf
Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz

[ad...@cs.miami.edu]

Hi,

> at io.netty.buffer.UnpooledUnsafeHeapByteBuf.<init>(UnpooledUnsafeHeapByteBuf.java:29)

This first line indicates to me that the actual problem might not me the heap size.  An "Unsafe" memory buffer is taken from direct memory (off-heap).  But then again the OOM error said heap space error?  So, I don't know.  I imagine that you are not using a JVM option to limit or set he direct memory?  Is this box very memory constrained when this error is thrown?  Perhaps it tried to allocate off-heap and there simply was not enough, and it threw a general OOM error?   Have you seen what happens if you just increase the heap?  Even if you do not want to do this ultimately, what happens if you double it?  Does it never crash?

> do you know if netty and vertx are reusing the same threads/event loops or does netty has own pool not related to vertx pool?

I don't know, but a profiler should be able to let you know this info.  If not, just periodically print to log the names of all the threads, and see how many there are. The name of Vertx threads should be obvious when looking at them.

-Adam

[Julien Viet]

Hi,

here is a try of explanation of what is happening, blind guess according to what you say, as you don’t share code.

I’m assuming also you are running this locally so your client is able to send the request body very fast with very low latency.

Your proxy creates http connection to a backend with an initial empty connection pool, these connections to the backend take time to create (even locally, compared to the already established connection between the client and the proxy and also that 2MB on localhost is peanut).

The pump works as described in the documentation, there is no cheating on that. However when you create the request to the backend when you get a proxy request, it is likely that at this moment it connects to the backend at the same time. The pump pauses the incoming http requests when the connection buffer is full, however as this is not yet connected, the buffers are accumulated in the HttpClientRequest until request to the backend connects. So the pump does not kick in yet.

I suspect that during the time it takes to connect to the backend, it accumulates quite some data there, more than it should if it were already connected, let’s say it takes 1 meg, if you multiply that by 20, it makes more than half of the heap you want to dedicate.

Normally this should not be a problem, but in your case you have very low memory footprint requirement, also this happen because you are likely to run this on localhost or a LAN which has a very low latency compared to the internet.

If you need such requirements, my suggestion is that when you receive an HttpServerRequest in the proxy:

1/ you immediately pause() the HttpServerRequest

2/ you create the HttpClientRequest and you set the method + headers on it

3/ you call clientRequest#setWriteQueueMaxSize(maxBufferSize) with a low value depending on your requirement

4/ you call clientRequest.sendHead(v -> { // lambda }) , the lambda will be called when head has been sent, so at this time you know that you are connected and you sent the headers - at this moment you resume() the HttpServerRequest and you start the pump

Julien

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Przemo T]

Hi guys,

It's been a while but getting back to the proxy. Looks like "request pausing" and waiting for the client connection establishment indeed improved things. Thank you!

So my proxy is implemented as standard verticle. According to docs - standard verticle instance is always assigned to one event loop thread. Is there any point of deploying more instances of the same verticle then event loops count (which in turn is calculated as cpu * 2)? Like having lets say 4 event loops but 20 instances of the same verticle?

Przemek

[Julien Viet]

no there is no real interest.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f6cbf5fe-2b4c-4372-8ac3-efbbe35846ef%40googlegroups.com.

[Przemo T]

Hi Julien,

Thank you for your input.

Now I'm wondering about proxy exception handling...

1) Server exception

The case that you make calls to proxy and at some point you close the client (like terminate). In such case I would expect to get notification on server request/response.

Looking at the API I can see the following handlers:

HttpServerRequest.connection().closeHandler

HttpServerRequest.connection().exceptionHandler

HttpServerRequest.exceptionHandler

HttpServerResponse.exceptionHandler

Which handlers should I implement?

Currently I'm using HttpServerRequest.connection().exceptionHandler were I call something like: proxyRequest.connection().close()

I hoped that this way I can release proxyRequest related resources immediately. Not sure however if this is the proper way.

Do I need additional checks when applying pause/sendHead/resume pattern? Should Pump be stopped as part of exception handling?

2) The second case is related to situation when at some point you get exception on proxy side (proxyRequest/proxyResponse). Also not sure what happens when exception is called in the middle of Pump.

Does the following code make sense for handling proxyRequest/proxyResponse exception:

if (!serverResponse.closed() && !serverResponse.ended()) {

if (!serverResponse.headWritten()) {

serverResponse.setStatusCode(500).end();

} else {

serverResponse.end();

}

}

[Przemo T]

How about something like below (note: I have not tested this code yet but maybe you can point any issues knowing better what is inside Vertx):

package com.wolterskluwer.blobs.handlers;

import io.vertx.core.AbstractVerticle;

import io.vertx.core.Handler;

import io.vertx.core.http.HttpClient;

import io.vertx.core.http.HttpClientRequest;

import io.vertx.core.http.HttpServerRequest;

import io.vertx.core.http.HttpServerResponse;

import io.vertx.core.streams.Pump;

import io.vertx.ext.web.RoutingContext;

public class ProxyHandler implements Handler<RoutingContext> {

private HttpClient httpClient;

public ProxyHandler(AbstractVerticle ctx) {

this.httpClient = ctx.getVertx().createHttpClient();

}

@Override

public void handle(RoutingContext routingContext) {

HttpServerRequest request = routingContext.request();

// you want to pause until you are connected to proxy

request.pause();

HttpServerResponse response = routingContext.response();

response.setChunked(true);

Handler<Throwable> basicRequestHandler = t -> {

try {

if (!response.closed()) {

if (!response.headWritten()) {

response.setStatusCode(500).end();

} else if (!response.ended()) {

response.end();

}

}

} finally {

// make sure request is not left as paused

request.resume();

}

};

request.exceptionHandler(basicRequestHandler);

// Note: to complete reading request handling switch to proxyRequest.sendHead

// and get back here for response

HttpClientRequest proxyRequest = httpClient.request(request.method(), "proxy_host",

routingContext.request().path(), proxyResponse -> {

response.setStatusCode(proxyResponse.statusCode());

response.setStatusMessage(proxyResponse.statusMessage());

// start the pump

proxyResponse.endHandler(Void -> {

response.end();

});

Pump responsePump = Pump.pump(proxyResponse, response).start();

Handler<Throwable> responseHandler = t -> {

responsePump.stop();

// try to end response

if (!response.closed()) {

if (!response.headWritten()) {

response.setStatusCode(500).end();

} else if (!response.ended()) {

response.end();

}

}

// question: how to discard proxyResponse??

// set empty handler to simply consume the payload...

proxyResponse.handler(b -> {

});

proxyResponse.endHandler(Void -> {

});

// make sure it is not left paused

proxyResponse.resume();

};

response.exceptionHandler(responseHandler);

proxyResponse.exceptionHandler(responseHandler);

});

// before we send anything to proxy make sure we set handler

proxyRequest.exceptionHandler(basicRequestHandler);

proxyRequest.sendHead(version -> {

// now we are connected to proxy

// we can resume request

request.resume();

// and start pump

request.endHandler(Void -> {

proxyRequest.end();

});

Pump requestPump = Pump.pump(request, proxyRequest).start();

// but lets modify request exception handler to also close the pump and proxy

Handler<Throwable> proxyConnectedHandler = t -> {

try {

requestPump.stop();

proxyRequest.end(); // I guess we may endup sending to the

// proxy something incomplete

} finally {

basicRequestHandler.handle(t);

}

};

request.exceptionHandler(proxyConnectedHandler);

proxyRequest.exceptionHandler(proxyConnectedHandler);

});

}

}