There is a query still being run here - race -> false

[Mumuney Abdlquadri]

Hi guys,

I am seeing this errors in my usage of MySQL / PostgreSQL service.

My usage is such that I close the connection after each query and reopen it afterwards.

com.github.mauricio.async.db.exceptions.ConnectionStillRunningQueryException: [2] - There is a query still being run here - race -> false

        at com.github.mauricio.async.db.mysql.MySQLConnection.validateIsReadyForQuery(MySQLConnection.scala:250)

        at com.github.mauricio.async.db.mysql.MySQLConnection.sendPreparedStatement(MySQLConnection.scala:232)

        at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl$$anonfun$queryWithParams$1.apply(AsyncSQLConnectionImpl.scala:58)

        at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl$$anonfun$queryWithParams$1.apply(AsyncSQLConnectionImpl.scala:57)

        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)

        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249)

        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)

        at io.vertx.ext.asyncsql.impl.pool.SimpleExecutionContext$$anon$1.execute(SimpleExecutionContext.scala:12)

        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)

        at scala.concurrent.impl.Promise$KeptPromise.onComplete(Promise.scala:333)

        at scala.concurrent.Future$class.flatMap(Future.scala:249)

        at scala.concurrent.impl.Promise$KeptPromise.flatMap(Promise.scala:322)

        at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.queryWithParams(AsyncSQLConnectionImpl.scala:57)

How to reproduce:

Deploy this Verticle https://gist.github.com/abdlquadri/96e8b6b4f90fcff32b4a with a number of instances (e.g. 4)

Using wrk, send load: wrk -t1 -c5 -d30s http://127.0.0.1:8282/profile/873798982

However if you reduce connections to number of verticle instances,no error:

wrk -t1 -c4 -d30s http://127.0.0.1:8282/profile/873798982

Is there something I am doing wrong?

Regards.

[Joern Bernhardt]

Hi,

from the code, it looks like you're opening the connection when starting the verticle and then reusing that for each request. I guess that's the root cause for the "seeing race conditions here": You are scheduling commands on a connection that is already doing something.

It would be better to just open a connection when there is a request. The MySQL/PostgreSQL client should automatically take one that's available in the pool or create a new one if the connection timed out. And don't forget to close it after you're done with it. That means just open/close the connection on a request basis. Doing so should solve the issue described in your other thread, too.

HTH,

Joern

[Mumuney Abdlquadri]

Thanks,

I will try this when I get to work.

Regards

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mumuney Abdlquadri]

Hi Joern,

I am now seeing this:

May 12, 2015 10:05:54 AM io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool

INFO: creating a connection went wrong: com.github.mauricio.async.db.mysql.exceptions.MySQLException: Error 1040 - Too ma - ny connections

Please Note that during the load test:

If you reduce connections to number of verticle instances,no error:

wrk -t1 -c4 -d30s http://127.0.0.1:8282/profile/873798982

Regards.

[Joern Bernhardt]

Can you update your gist with the newest code?
If the pool is used correctly, it should let you wait until a connection is free, actually...

You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/yCQbhUA9HmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

[Mumuney Abdlquadri]

Thanks,

I have been able to get rid of the error by making sure the mySqlService.getConnection is called on each request as you suggested.

Regards.

[shailender arya]

Hi Joern. 

I am facing this same issue using postgressClient. 

The way am doing it is that PostgresVerticle is listening on the messageBus and whenever a message is received am creating a connection. 

Can you please help. PFA my code

[Joern Bernhardt]

What error do you see exactly?

In what lines is the breaking problem?

If it is somewhere around "handler.handle(connection, message);", you need to provide more code. The best would be a GitHub project for me to checkout ;)

[shailender arya]

Hi Joern. 

Please find all the relevant files. 

Exception that am getting is  on line 38 on OrderByIdRDSHandler.java. 

It works fine for sequential request but when i bombard it with 1200 req/sec on a single server then around 3% of the requests fail with these exceptions.

vertx version that am using is 3.2.1. 

following is the stack trace 

com.github.mauricio.async.db.exceptions.ConnectionStillRunningQueryException: [7] - There is a query still being run here - race -> false

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.notReadyForQueryError(PostgreSQLConnection.scala:271)

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.validateIfItIsReadyForQuery(PostgreSQLConnection.scala:279)

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.validateQuery(PostgreSQLConnection.scala:282)

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.sendPreparedStatement(PostgreSQLConnection.scala:114)

    at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.lambda$queryWithParams$11(AsyncSQLConnectionImpl.java:126)

    at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.beginTransactionIfNeeded(AsyncSQLConnectionImpl.java:240)

    at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.queryWithParams(AsyncSQLConnectionImpl.java:125)

    at com.delhivery.rds.handler.OrderByIdRDSHandler.handle(OrderByIdRDSHandler.java:38)

    at com.delhivery.vertex.verticles.PostgresVerticle.executeWithHandler(PostgresVerticle.java:72)

    at com.delhivery.vertex.verticles.PostgresVerticle.lambda$null$0(PostgresVerticle.java:51)

    at io.vertx.ext.asyncsql.impl.BaseSQLClient.lambda$getConnection$2(BaseSQLClient.java:67)

    at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.take(AsyncConnectionPool.java:89)

    at io.vertx.ext.asyncsql.impl.BaseSQLClient.getConnection(BaseSQLClient.java:63)

    at io.vertx.ext.asyncsql.impl.AsyncSQLClientImpl.getConnection(AsyncSQLClientImpl.java:53)

    at io.vertx.ext.asyncsql.impl.ClientWrapper.getConnection(ClientWrapper.java:51)

    at com.delhivery.vertex.verticles.PostgresVerticle.lambda$registerMessageHandlers$1(PostgresVerticle.java:50)

    at io.vertx.core.eventbus.impl.HandlerRegistration.handleMessage(HandlerRegistration.java:207)

    at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:201)

[Joern Bernhardt]

Hi Shailender,

line 38 is just "}" in the OrderByIdRDSHandler.java you've sent (and line 72 in PostgresVerticle is "try{" btw). Are you sure you've sent the correct files here...?

From here, I don't really see an error. We wait for the result of closing the connection usually, but I doubt that you'd have to do that actually and it's okay to just use "close()". Otherwise this may hint to a bug in the pool implementation - maybe you could check out the project and create a reproducer for that case, if you cannot post the real code that generates the errors for you?

Thanks,

Joern

[shailender arya]

my bad I will update it by tonight in a public git hub project

[Paulo Lopes]

Just my 2cts,

This could be a underlying driver issue, I've noticed that you cannot "batch" queries. For example say that you want to insert 10 rows you cannot call:

insert into T values (...);
insert into T values (...);
insert into T values (...);
insert into T values (...);
insert into T values (...);
insert into T values (...);
insert into T values (...);
insert into T values (...);
insert into T values (...);

You'll see that error, in order for it to work, I've seen that i need to have a handler to notify me that the previous query has run so i can issue the second...

This was during 3.2.1, the driver has been updated in between so i'm not sure if this is still the error...

Cheers,
Paulo

[shailender arya]

Hi Joern

Github public repo containing the code is https://github.com/arya-shailender/vertx

Exception trace is [11 May 16 12:16:05] arya.postgresPerformance.verticle.PostgresVerticle DEBUG  - Exception occured while fetching from DB

com.github.mauricio.async.db.exceptions.ConnectionStillRunningQueryException: [5] - There is a query still being run here - race -> false

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.notReadyForQueryError(PostgreSQLConnection.scala:271)

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.validateIfItIsReadyForQuery(PostgreSQLConnection.scala:279)

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.validateQuery(PostgreSQLConnection.scala:282)

    at com.github.mauricio.async.db.postgresql.PostgreSQLConnection.sendPreparedStatement(PostgreSQLConnection.scala:114)

    at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.lambda$queryWithParams$11(AsyncSQLConnectionImpl.java:126)

    at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.beginTransactionIfNeeded(AsyncSQLConnectionImpl.java:240)

    at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.queryWithParams(AsyncSQLConnectionImpl.java:125)

    at arya.postgresPerformance.rds.OrderByIdRDSHandler.handle(OrderByIdRDSHandler.java:33)

    at arya.postgresPerformance.verticle.PostgresVerticle.executeWithHandler(PostgresVerticle.java:73)

    at arya.postgresPerformance.verticle.PostgresVerticle.lambda$null$0(PostgresVerticle.java:52)

    at io.vertx.ext.asyncsql.impl.BaseSQLClient.lambda$getConnection$2(BaseSQLClient.java:67)

    at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.take(AsyncConnectionPool.java:89)

    at io.vertx.ext.asyncsql.impl.BaseSQLClient.getConnection(BaseSQLClient.java:63)

    at io.vertx.ext.asyncsql.impl.AsyncSQLClientImpl.getConnection(AsyncSQLClientImpl.java:53)

    at io.vertx.ext.asyncsql.impl.ClientWrapper.getConnection(ClientWrapper.java:51)

    at arya.postgresPerformance.verticle.PostgresVerticle.lambda$registerMessageHandlers$1(PostgresVerticle.java:51)

    at io.vertx.core.eventbus.impl.HandlerRegistration.handleMessage(HandlerRegistration.java:207)

    at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:201)

    at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$127(EventBusImpl.java:498)

README.md has description on how to start running the project. 

Looking forward for your reply

[shailender arya]

Hi Guys

I got the error. It was my mistake only. 

I was closing the connection in the executeWithHandler whereas I should have closed it in the connection.queryWithParams responseHandler.

After doing that it is working properly

[Joern Bernhardt]

Top! :)

[Brian Roberts]

I had this error too. I think my solution below solved it. But I don't have access to high throughput to test it right now... but the errors have all but disappeared.

It turns out you need to close the connection as soon as its query returns. I was instead closing the connection when the enclosing body AROUND the query method and its returning result body. Problem solved! Thanks for the comments, guys.

this is how it was: 

            postgreSQLClient.getConnection(res5 -> {
                if (res5.succeeded()) {
                    System.out.println("res succeeded");
                    SQLConnection connection = res5.result();
                    try {

                        connection.query("SELECT * FROM " +
                                Constants.LOCATIONS_TABLE_NAME
                                 + ";", res33 -> {

                                future.complete();
                            }
                        });
                    } catch (Exception e) {
                        e.printStackTrace();
                    } finally {

                     connection.close();
                    }

this is it, fixed: notice the connection.close();

            postgreSQLClient.getConnection(res5 -> {
                if (res5.succeeded()) {
                    System.out.println("res succeeded");
                    SQLConnection connection = res5.result();
                    try {

                        connection.query("SELECT * FROM " +
                                Constants.LOCATIONS_TABLE_NAME
                                 + ";", res33 -> {

                                connection.close();

                                future.complete();
                            }
                        });
                    } catch (Exception e) {
                        e.printStackTrace();
                    } finally {

            //empty now
                    } etc

[Brian Roberts]

As per my reply above,

The error has not been resolved. Does anyone have any input? BTW the postgresSQLClient object is an instance of io.vertx.ext.asyncsql.AsyncSQLClient;