Service Discovery best practice

[Giacomo Zanotti]

Hi all,

I have developed a reverse proxy for my application, using service discovery. There is something I cannot explain myself and I do not know if I am doing it right or wrong.

Let me explain. 

Service A would call service B by using my reverse proxy.

The reverse proxy would query vert.x ServiceDiscovery object to find service B and proxying the request.

In my first implementation, I create a service object each time service A would call service B via the proxy.

I deployed the fat jar in a docker container, and deployed that container on EC2 instance (t2.micro. Because it is free).

I found out that the docker memory would increase a bit everytime the reverse proxy would proxy the request from A to B. Until it breaks.

Of course as the docs tells to do, I  close the serivce discovery instance

In my second implementation, there is only one single servicedsicovery instance.

I never close it, because if I close it when another request is being proxied the request would fail. Still, I always release the servicereference

The memory usage by the docker container is way, way less than before.

I guess the service object is pretty heavy, but I would like a second opinion. 

[Thomas SEGISMONT]

Do you mean that anytime you get a request you do:

ServiceDiscovery discovery = ServiceDiscovery.create(vertx);
// Do something...
discovery.close();

It is not necessary to create an instance for each request. Yet it should just produce more garbage and not leak anything.

If you can create a small reproducer we can take a look.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/572e87cf-6959-4ddb-9ef5-a14af23996cfn%40googlegroups.com.

[Giacomo Zanotti]

Hi,

seems I missed your response. I am deeply sorry about that. I think I solved the problem in the question by myself. Heap does not max out even if i make thousands and thousands of request per seconds (with caching policies, and also kind of expensive operations also!), cpu is very low and deployed in a t2.micro aws instance. Vert.x is just great, I must find the courage to make a presentation at work cause javaEE is boring as hell.

What I am curious about is the method that helps retrieve the webclient from a service record:

HttpEndpoint.getClient(discovery, new JsonObject().put("name", "some-http-service"), ar -> {
  if (ar.succeeded()) {
    HttpClient client = ar.result();

    // You need to path the complete path
    client.request(HttpMethod.GET, "/api/persons").compose(request ->
      request
        .send()
        .compose(HttpClientResponse::body))
      .onComplete(ar2 -> {
        // Dont' forget to release the service
        ServiceDiscovery.releaseServiceObject(discovery, client);
      });
  }
});

I was using this solution offered by the docs, retrieving directly the webclient from the service record. This was causing memory leaks and heap was maxing out, because I was creating a webclient object every time I would make a request to my proxy.

I knew what i was doing, but i guess calling ServiceDiscovery.realease wasn't enough. I guess there might be two reasons:

1) My bad code (more likely): I was very cautios and checked every possibility, and I'm pretty sure i was releasing the webclient each time. Anyway, it could be that i was missing some ServiceDiscovery.release calls.

2) It is a bad practice to do so, and it should follow a service discovery.close()

Now a snippet from memory-leaky code:

private void rerouteToService(RoutingContext routingContext, String root, String uri) {
    LOGGER.info("handling request " + routingContext.request().method().name() + " "
            + routingContext.request().absoluteURI());
    HttpEndpoint.getWebClient(discovery, record -> record.getLocation().getString("root").equals(root),
            webClientAsyncResult -> {
                if (webClientAsyncResult.succeeded()) {
                    WebClient webClient = webClientAsyncResult.result();
                    Buffer body = routingContext.getBody();
                    HttpServerRequest httpServerRequest = routingContext.request();
                    HttpServerResponse httpServerResponse = routingContext.response();
                    HttpMethod method = httpServerRequest.method();

                    HttpRequest<Buffer> request = webClient.request(method, uri);
                    request.timeout(1000);
                    request.putHeaders(httpServerRequest.headers());

                    // put, post with body
                    if (body != null && !body.toString().isEmpty()) {
                        // send json object does not work...
                        LOGGER.info("Payload " + body.toString());
                        request.sendBuffer(body, httpResponseAsyncResult -> handleHttpResponse(uri,
                                httpServerResponse, httpResponseAsyncResult, webClient, discovery));
                    } else if (method == HttpMethod.GET) {
                        handleCachingGetRequest(uri, request, httpServerResponse, routingContext, webClient,
                                discovery);
                    } else {
                        request.send(httpResponseAsyncResult -> handleHttpResponse(uri, httpServerResponse,
                                httpResponseAsyncResult, webClient, discovery));
                    }
                } else {
                    BadRequest("service with root " + root + " was not found", routingContext);

                }

            });
}

now a snippet from my non-memory-leaky code:

In this case I just retrieve the service record, not calliing release and the client is instantiated with keepAlive(false) on start

private void rerouteToService(RoutingContext routingContext, String root, String uri) {
    LOGGER.info("handling request " + routingContext.request().method().name() + " "
            + routingContext.request().absoluteURI());
    discovery.getRecord(record -> record.getLocation().getString("root").equals(root), recordAsync -> {
        if (recordAsync.succeeded() && recordAsync.result()!=null) {
            Record serviceRecord = recordAsync.result();
            int port = serviceRecord.getLocation().getInteger("port");
            String host = serviceRecord.getLocation().getString("host");
            Buffer body = routingContext.getBody();
            HttpServerRequest httpServerRequest = routingContext.request();
            HttpServerResponse httpServerResponse = routingContext.response();
            HttpMethod method = httpServerRequest.method();

            HttpRequest<Buffer> request = client.request(method, port, host, uri);
            request.timeout(1000);
            request.putHeaders(httpServerRequest.headers());
            if (body != null && !body.toString().isEmpty()) {
                request.sendBuffer(body, httpResponseAsyncResult -> {
                    handleHttpResponse(uri, httpServerResponse, httpResponseAsyncResult, discovery);
                    // breakerPromise.complete();
                });
            } else if (method == HttpMethod.GET) {
                handleCachingGetRequest(uri, request, httpServerResponse, routingContext, discovery);
            } else {
                request.send(httpResponseAsyncResult -> {
                    handleHttpResponse(uri, httpServerResponse, httpResponseAsyncResult, discovery);
                    // breakerPromise.complete();
                });
            }

        } else {
            BadRequest("service with root " + root + " was not found", routingContext);

        }

    });

You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/ZLZ1EYTvW-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CACiEr_R9n77oxS9ejWets337Cdg2DSLaG_QeRF0CQ3aULL97iw%40mail.gmail.com.

[Thomas SEGISMONT]

Hi,

I'm glad you could fix the problem and are happy with Vert.x in production.

If you can put together a simple reproducer for the memory-leaking code, it would be awesome.

It's difficult to find the cause with the snippets only.

Regards

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CA%2Bj7p5jiFGG3DBim949t-kj_835B2Dk0kMKmm48eoLJm1kzXUg%40mail.gmail.com.