Thread block while using circuit breaker

[Jyoti Patel]

Hi All,

I am facing this issue from quite a long time and not able to figure out exactly where am I messing up with the thread.

Details:

Version

io.vertx vertx-circuit-breaker 4.3.1

Context

I am getting high memory usage with the circuit breaker. I am just using httpbin.org to get all success responses. For individual requests, it works fine. While running a load test the JVM old gen utilization is spiking up.

Reproducer

I have the main verticle code, pasting it here itself:

public class CleanServer extends AbstractVerticle {
    Logger logger = Logger.getLogger(CleanServer.class.getName());

    @Override
    public void start(Promise<Void> startPromise) throws Exception {
        Router router = Router.router(vertx);
        CircuitBreakerCache cbc = new CircuitBreakerCache(vertx);
        router.route(HttpMethod.GET, "/get").handler(context -> {
            List<String> domains = context.queryParam("user");
            String domain = domains.get(0);
            CircuitBreaker cb = cbc.getCircuitBreaker(domain + context.request().path());
            HttpServerResponse serverResponse =
                    context.response().setChunked(true);
            cb.executeWithFallback(promise -> {
                WebClientOptions options = new WebClientOptions().setTryUseCompression(true).setTcpNoDelay(true).setTcpCork(true).setReceiveBufferSize(128).setConnectTimeout(400);
                WebClient client = WebClient.create(vertx, options);
                client.get(80, "httpbin.org", "/status/200")
                        .timeout(2000)
                        .send(ar -> {
                            if (ar.succeeded()) {
                                HttpResponse<Buffer> response = ar.result();
                                int statusCode = response.statusCode();
                                if (statusCode != 200) {
                                    promise.fail(response.statusMessage());
                                } else {
                                    serverResponse.end("Hello!!");
                                    promise.complete();
                                }
                            } else {
                                promise.fail(ar.cause().getMessage());
                            }
                        });
            }, v -> {
                // Executed when the circuit is opened
                logger.log(Level.INFO, domain + " Failed " + cb.state().toString() + " Error: Circuit open");
                serverResponse.setStatusCode(200).setStatusMessage("Circuit Open").end("Circuit Open");
                return context;
            });
        });

        // Create the HTTP server
        vertx.createHttpServer(new HttpServerOptions().setMaxInitialLineLength(10000))
                // Handle every request using the router
                .requestHandler(router)
                // Start listening
                .listen(8080)
                // Print the port
                .onSuccess(server ->
                        System.out.println(
                                "HTTP server started on port " + server.actualPort()
                        )
                );


    }
}

Circuit breaker options:
CircuitBreakerOptions() .setMaxFailures(50) .setTimeout(5000) .setFallbackOnFailure(true) .setResetTimeout(10000)));

Steps to reproduce

1. API used: http://localhost:8080/get?user=abc
2. When I am hitting the above API at 50 QPS for 30 minutes. The java heap is getting filled up.

Extra

<vertx.version>4.2.6</vertx.version>

JVM params used:
-XX:+UseG1GC -Xms4g -Xmx4g -XX:InitiatingHeapOccupancyPercent=70 -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5

JVM memory with the load test.
Image attached.

Error:
WARNING: Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked for 3050 ms, time limit is 2000 ms

I think I am blocking the thread somewhere but not sure where exactly as the code seems pretty simple as given in the documentation.

[Thomas SEGISMONT]

Hi,

We've noticed the issue you file on GH https://github.com/vert-x3/vertx-circuit-breaker/issues/58 and will continue the conversation there.

Thank you

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8221b2ec-6e3f-45ef-b763-473e310be0e0n%40googlegroups.com.