connection.close() does not seem to work

[Fabrizio Fortino]

Hi there,

I am experiencing a pretty critical problem with version 1.5.1.

All the requests in my application are handled in the following way:

ODatabaseDocument conn = null;

try {
   conn = new ODatabaseDocumentTx(dbName).open(username, password);
   
   // do stuff

} finally{
   conn.close();
}

As you can see no database pools are used. After a while the application is not able to process any request. The following Exception is shown in the client application log:

java.lang.OutOfMemoryError: unable to create new native thread

        at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]

        at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]

        at com.orientechnologies.orient.enterprise.channel.binary.OAsynchChannelServiceThread.<init>(OAsynchChannelServiceThread.java:40)

        at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryClient.<init>(OChannelBinaryClient.java:89)

        at com.orientechnologies.orient.client.remote.OStorageRemote.createNetworkConnection(OStorageRemote.java:1712)

        at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1886)

        at com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1572)

        at com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:176)

        at com.orientechnologies.orient.client.remote.OStorageRemoteThread.open(OStorageRemoteThread.java:70)

        at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:105)

        at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:49)

        at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:119)

        at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:49)

Moreover, the Server Management panel in Orient Studio shows a long list of binary active connections but the Database field is not specified.

If I restart the client application the active connections disappeared and the application returns to work properly.

Since I am pretty sure to close all the connections (the close() call is in the finally block) I think there is a problem with the OrientDB close() method.

Is there any workaround to sort it out?

Thanks,

Fabrizio 

[Andrey Lomakin]

Hi,

It is quite serious I will look on this issue https://github.com/orientechnologies/orientdb/issues/1686 tomorrow.

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[Fabrizio Fortino]

Hi Andrey,

The problem seems to be reproducible only in a multi-threaded environment. I have updated the issue with a test case.

Regards,

Fabrizio

[Andrey Lomakin]

Cool, but I  reproduced your problem and you are right )) , I will provide fix tomorrow.

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Fabrizio Fortino]

Hi,

I am trying to figure out a workaround to avoid this kind of problems in short time.

At the moment I am restarting my application every couple of hours but this is pretty nasty :-(

I am now testing some alternative way to sort it out. 

The first method is a script that get all connections (using the rest call http://orientdbhost:2480/server) and kill the inactive ones (using the rest call http://orientdbhost:2480/connection/kill/<id>). Even if the connections disappear from the Server Management view the number of threads owned by the Orient server process does not decrease.

The second method is the following snippet executed in my application every 10 minutes:

    @Schedule(minute="*/10", hour="*")
    public void automaticRestart() {
        logger.info("Restarting orient client connections");
        Orient.instance().shutdown();
        Orient.instance().startup();
    }

With this approach the connections are successfully reset and the threads decrease as expected. The only drawback is that during the restart operation any ongoing request using a connection fails. Could you please tell me if there is a way to execute the restart only when all the connections are inactive?

Thanks,

Fabrizio

[Fabrizio Fortino]

I think I have found a more elegant workaround. Basically, my class that returns the Orient connections (OrientConnectionFactory) now implements ODatabaseListener. The getConnection() method register itself as listener for every opened connection:

synchronized (this) {
  db0 = new ODatabaseDocumentTx(dbName).open(username, password);
}
db0.registerListener(this);

When the connection get closed, the listener onClose() method is called. My implementation check if the storage has more than one user. If not, the storage is closed with force flag set to true.

@Override
public void onClose(ODatabase iDatabase) {
  synchronized (this) {
    OStorage storage = iDatabase.getStorage();
    if (storage.getUsers() <= 1) {
      LOGGER.debug("Closing database storage {}", storage);
      storage.close(true);
    }
  }
}

Connections are successfully reset and threads number remain stable :-)

Cheers,

Fabrizio

[Andrey Lomakin]

Yes, but closing the storage you flush all caches and as result your performance is bounded by disk performance.

[Fabrizio Fortino]

I know. I will use it as a temporary workaround until the issue get fixed.

Thanks,

Fabrizio