async, non blocking java API

[santo...@gmail.com]

Hi,

Do you provide an async, non blocking java API for OrientDB and if not, are there any plans to provide it in the near future ?
The reason I ask this is because I'm writing a java web application based on the Vert.x platform.
In Vert.x it's required that you don't have any blocking code and therefore I'm interested in a database with which my application can interact in an async, non blocking way.

regards,

Santo

[Andrey Lomakin]

Hi,

We have async API which is based on listeners, but listeners are executed in the same thread.

When you mean non-blocking does it mean that you need async API which will be run in seprate thread, right ?

Could you define non-blocking ?

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[santo...@gmail.com]

Hi Andrey,

With non-blocking I mean that the call should return immediately, probably with a handler which is called whenever the database operation is ready.

The reason for this is that Vert.x is an event-driven framework using so-called eventloops on which multiple handlers will be registered.
If one of those handlers performs some blocking code (a sleep, a wait, a blocking database call, whatever), the whole eventloop gets blocked, meaning the other handlers on the loop can't be served.

So a Vert.x application consists of one or more modules, where each module consists of one or more handlers that are registered on the eventloop.
One of the modules in the application can be e.g. a database module that handles all database related operations for the application.
Because such a module actually consists of one or more handlers that are registered on the eventloop, it is of uttermost importance that it can perform all its operations in an async, non-blocking way.
Hence the request for an async, non-blocking api for OrientDB which can be used for any db related operation (crud, document/graph, geo related, it doesn't matter, it should all be available)

I hope this clarifies what I meant.

regards,

Santo

Op woensdag 19 maart 2014 18:59:13 UTC+1 schreef Andrey Lomakin:

[ste...@activitystream.com]

Hi Andrey,

The Play framework is another example of a none-blocking io. I mention them because they have made some database drivers/orm extensions available that respect this need.

You might find more information there.

This would be the same as is java Futures where used to deliver query results.

Regards,

  - Stefán

[ste...@activitystream.com]

make that non-blocking :)

[Andrey Lomakin]

Hi, yes, it clarifies what you mean.

Guys what do you think about making Aynch requests non blocking it is relatively easy to do ?

[ste...@activitystream.com]

Hi,

Can you please consider this along with non-blocking reads/queries (read locking).

These two things go well together :) in the non-blocking world.

Regards,

  -Stefán

[sANTo L]

Hi Andrey,

do you have any updates on this ?

regards,

Santo

Op vrijdag 21 maart 2014 10:30:21 UTC+1 schreef Andrey Lomakin:

[Andrey Lomakin]

Artem, Luca ,

So what do we think ?

[Luca Garulli]

Hi all,

sorry for the delay.

If I understand well, we already have it: https://github.com/orientechnologies/orientdb/wiki/Document-Database#asynchronous-query

Lvc@

[ste...@activitystream.com]

it's not the same :)

[Andrey Lomakin]

The difference is following listener lives in the same thread but should be called in background thread and after that thread should be shut down.

Luca WDYT if we will change asynchronous queries in this way ?

[sANTo L]

Hi,

Please note that it is not only for queries, but also for other db related operations: crud, graph traversal, etc ;-)

regards,

Santo

Op donderdag 27 maart 2014 16:05:58 UTC+1 schreef Andrey Lomakin:

[Artem Orobets]

Hi all,

Sorry for delay. Sounds good. +1 from me.

Best regards,

Artem Orobets

Orient Technologies

the Company behind OrientDB

[Luca Garulli]

+1, could anybody create a new issue?

We could have an executor pool for this purpose.

Lvc@

[sANTo L]

Ok, I created an issue for it:
https://github.com/orientechnologies/orientdb/issues/2173

thanks,

Santo

Op vrijdag 28 maart 2014 14:42:09 UTC+1 schreef Lvc@:

[Jakub Liska]

I'd also appreciate this feature. Non-blocking api is essential for a lot of frameworks now. Event loop (reactor pattern) based frameworks cannot even used it since it blocks a thread. 

It would be a good marketing tool too.

[Leng Sheng Hong]

this will be extremely useful as I am too using vertx async framework. +1

[Sebastien Deleuze]

Hi all,

An executor with a dedicated thread pool will be still blocking, the only advantage is that the thread is in a separate thread pool.

The whole purpose of a real asynchronous non blocking implementation is to have no thread waiting for the database response.

All systems can handle a limited number of simultaneous threads, so this kind of model greatly improves scalability.

Since implementation is not obvious, most http or persistence libraries use Netty or equivalent to implement it.

You can have a look at Mongo Java Driver 3.0-SNAPSHOT async-driver as an example of NoSQL driver implementing this kind of needs.

https://github.com/mongodb/mongo-java-driver/tree/3.0.x

Historically this kind of architecture has been more popular in Scala ecosystem, because of their nice Promise API.

Hopefully, Java 8 now comes with lambdas (great for specifying callbacks) and the great CompletableFuture.

CompletableFuture is non blocking compliant since it allows to specify a callback that will be called later,

unlike Java 5 Future which can only produce async blocking API.

Since most NoSql drivers do not depend on Java 8, most of them implement their own CompletableFuture equivalent.

For example https://github.com/mongodb/mongo-java-driver/blob/3.0.x/driver/src/main/org/mongodb/MongoFuture.java

These equivalents allow to use lambdas since they implement a functional interface (interface with one method).

@Luca, as I told you a few months ago, my initial plan was to use OrientDB for OpenSnap implementation

(a SnapChat clone based on Java 8, Spring 4 and AngularDart) but I had to fallback on MongoDB since it 

provides such support. I would be really happy to migrate it to OrientDB when it will be available.

You can have a look to OpenSnap source code (https://github.com/sdeleuze/opensnap) in order to have a look

to what a full asynchronous non blocking app based on Spring looks like.

Such feature would be a huge enabler to switch on OrientDB !!!

Regards,

Sébastien

[Carlo Pradissitto]

Hi all,

since our architecture is heavily based on non-blocking services, both at the nginx level and (especially) at the vert.x level, we think that using the blocking ODB APIs might lead to a scalability bottleneck.

As Jackub Liska has underlined, nowadays many frameworks are based on event-loop rather than a more traditional thread pool logic.

We'd appreciate a lot to know whether this issue is going to be assigned to a milestone or rejected, so that we can arrange our development schedule accordingly.

Regards,

Carlo

[Mohammad Naghavi]

so I want to ask more details on this issue. As I have seen on documentation as well as on the github issue, there is a non-blocking API for document API planned, but is there something similar planned for the graph API? does it actually make sense?

The reason I ask is that I'm using graph API and we specially benefit from a lot of automatic work done in background when we use it. Like bidirectional links, nested transactions and so on. so it will be a hard and very time consuming change to switch to document API to improve performance with non-blocking document API.

[Luigi Dell'Aquila]

Hi Mohammad,

What do you exactly mean? Non blocking queries can be used to query both documents and graphs, non blocking commands can be executed on both ODatabaseDocument and OrientGraph and you can execute graph queries out of the box on both.

In addition, you can always convert documents do vertices/edges with graph.getVertex(doc)

Luigi

--

[Mohammad Naghavi]

Hi Luigi,

I'm using the Graph API for Java mostly the Tinkerpop compliant parts due to some advantages it has for me. My question was, is there a way to make the parts of that API that need communication with DB as non-blocking or should I switch all my DB interaction code to use SQL instead?

regards,

Mohammad

[Luigi Dell'Aquila]

Hi Mohammad,

TinkerPop API is designed for synchronous communication, and OrientDB API is compliant with this behavior.

If you need asynchronous communication I suggest you to use SQL and commands

Regards

Luigi

[Mohammad Naghavi]

Hi again,

ok thanks, that seems clear now.

Regards,

You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/MBQJmUNV7bA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Jose Antonio Illescas del Olmo]

Please, anyone can submit an example of async call to orientdb from verticle (vert.x)

Thank you!!

El sábado, 30 de mayo de 2015, 10:38:34 (UTC+2), Mohammad Naghavi escribió:

Hi again,

ok thanks, that seems clear now.

Regards,

[Thom]

I would also like to know how to do async non blocking queries in orientdb (for vert.x)
I tried using OSQLNonBlockingQuery but i get different errors when actually using the returned odocument  (see below).
Should I make a github issue or am I just using orientdb the wrong way?
If I dont do a graph.shutdown() it will print the fieldnames and not throw the last exception but then how do I release my resources?
Last question, in the event I get the non-blocking query working is it possible to return results in batches (I suspect otherwise I will get into trouble with latency given Im using a remote connection)?

        OrientGraph graph = graphFactory.getTx();           
        try {
           
            OSQLNonBlockingQuery<Object> test = new OSQLNonBlockingQuery<Object>("select * from Prod where pid = '421'", new OCommandResultListener() {
                  int resultCount = 0;
                  @Override
                  public boolean result(Object iRecord) {
                    resultCount++;
                  
                    ODocument odoc = ((ODocument)iRecord);

                    for(String name : odoc.fieldNames()) {     //  <----------- PROBLEM
                        System.out.println("Name:"+name);
                    }
                   
                    System.out.println("callback "+resultCount+" invoked");
                    return resultCount > 20 ? false : true;
                  }

                  @Override
                  public void end() {
                  }
                });
           
            Future future = graph.command(test).execute();
                System.out.println("query executed");
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
           
            if (graph != null) {
                graph.shutdown(); // <------- If i remove this bit it wont throw the last exception but how do I then release resources
            }
        }

java.lang.UnsupportedOperationException: Trying to iterate over a non-blocking query result
    at com.orientechnologies.orient.core.sql.query.OSQLNonBlockingQuery$ONonBlockingQueryFuture.iterator(OSQLNonBlockingQuery.java:102)
    at com.tinkerpop.blueprints.impls.orient.OrientGraphCommand.execute(OrientGraphCommand.java:51)
    at com.myapp.Endpoint.main(Endpoint.java:57)

May 17, 2016 11:14:40 AM com.orientechnologies.common.log.OLogManager log
WARNING: Error deserializing record with id #38:235 send this data for debugging: ABpEYXRhX1Byb2R1Y3RzRwAAANdNAAAA20kAAADvSwAAASVFAAAC1kMAAALZFGVnZW5za2FiZXIAAAMaCgxncm91cHMAAAMcChRhbnZlbmRlbHNlAAADUgoYcHJvZHVjdF9uYW1lAAADVAcGdXJsAAADaAcMSU1QT1JUAAADmQcSZG93bmxvYWRzAAADuQoQcmVscGFnZXMAAASTChRyZWxwcm9kdWN0AAAElQoOc2VnbWVudAAABPIHGHByb2R1Y3Rncm91cAAABPcHTwAABP9RAAAFAAAGNDIxJlRyw6RsaW0gUFUgTGp1cyA0MjFqVmF0dGVuZmFzdCBENC1saW0gZsO2ciB0csOkLCBtZXRhbGwsIGjDpXJkIHBsYXN0IG0ubS7eBjxwIGNsYXNzPSJib2R5dGV4dCI+VHLDpGxpbSBQVSBManVzIDQyMSDDpHIgZXR0IDEta29tcG9uZW50IGZ1a3Row6RyZGFuZGUgcG9seXVyZXRhbmxpbSwgc3BlY2llbGx0IGzDpG1wbGlndCBmw7ZyIGxpbW5pbmcgYXYgdHLDpCBtZWQgaMO2ZyBmdWt0aGFsdCBzYW10IGbDtnIgbGltbmluZyBhdiBtZXRhbGwsIGjDpXJkIHBsYXN0LCBleHBhbmRlcmFkIHBvbHlzdHlyZW4gKGNlbGxwbGFzdCksIGtlcmFtaXNrYSBtYXRlcmlhbCBtLm0uIDwvcD48cCBjbGFzcz0iYm9keXRleHQiPlRyw6RsaW0gUFUgTGp1cyA0MjEgYW52w6RuZHMgZMOkciBkZXQgc3TDpGxscyBzw6Ryc2tpbHQgaMO2Z2Ega3JhdiBww6UgdmFudHRlbmZhc3RoZXQgb2NoIHN0eXJrYS4gTGltbWV0IHVwcGZ5bGxlciBrcmF2ZW4gdGlsbCBrbGFzcyBENCwgZW5sLiBFTiAyMDQvMjA1LiA8L3A+BHNlBAcKdXJsLWwAAALwBwcGdXJsAAADBQcoL3BpY3MvMGEzNWE1ZTMyYy5qcGcoL3BpY3MvNGU4ZjZmNGY1Yy5qcGcAFwgXBxxwcm9kdWt0a2F0YWxvZwcIYnlnZwcOdHJhZWxpbQcmdHJhZWxpbS1wdS1sanVzLTQyMQAXJnRyYWVsaW0tcHUtbGp1cy00MjFgcHJvZHVrdGthdGFsb2cvYnlnZy90cmFlbGltL3RyYWVsaW0tcHUtbGp1cy00MjEvPkRBTkFMSU1fSU1QT1JUXzIwMTYtMjgtMTIgMTE6MjgGFwwEBwZhbHQAAAPRBwcGdXJsAAAD5AckUHJvZHVrdGluZm9ybWF0aW9uPC9wcm9kdWt0aW5mb3JtYXRpb24vc2UvNDIxLnBkZgwEBwZhbHQAAAQZBwcGdXJsAAAELAckU8Oka2VyaGV0c2RhdGFibGFkPC9zaWtrZXJoZWRzZGF0YWJsYWQvc2UvNDIxLnBkZgwEBwZhbHQAAARhBwcGdXJsAAAEdQcmQnlnZ3ZhcnVkZWtsYXJhdGlvbjovYnlnZ3ZhcnVkZWtsYXRpb24vc2UvNDIxLnBkZgAXAhcMBAcEdjEAAASrBwcEdjIAAATdB2Jwcm9kdWt0a2F0YWxvZy9ieWdnL3RyYWVsaW0vdHJhZWxpbS1wdS1tb2Vyay00MjIvKFRyw6RsaW0gUFUgTcO2cmsgNDIyCGJ5Z2cOdHJhZWxpbQAA
Exception in thread "Thread-3" com.orientechnologies.orient.core.exception.ODatabaseException: Database 'remote:orientdb:2424/mydb' is closed
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.checkOpeness(ODatabaseDocumentTx.java:2920)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.getMetadata(ODatabaseDocumentTx.java:793)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.getMetadata(ODatabaseDocumentTx.java:121)
    at com.orientechnologies.orient.core.record.impl.ODocument.setClassNameIfExists(ODocument.java:1917)
    at com.orientechnologies.orient.core.record.impl.ODocument.fillClassIfNeed(ODocument.java:2180)
    at com.orientechnologies.orient.core.record.impl.ODocumentInternal.fillClassNameIfNeeded(ODocumentInternal.java:64)
    at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.deserialize(ORecordSerializerBinaryV0.java:159)
    at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinary.fromStream(ORecordSerializerBinary.java:74)
    at com.orientechnologies.orient.core.record.impl.ODocument.deserializeFields(ODocument.java:1817)
    at com.orientechnologies.orient.core.record.impl.ODocument.checkForFields(ODocument.java:2416)
    at com.orientechnologies.orient.core.record.impl.ODocument.fieldNames(ODocument.java:736)
    at com.myapp.Endpoint$1.result(Endpoint.java:42)
    at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1214)
    at com.orientechnologies.orient.client.remote.OStorageRemoteThread.command(OStorageRemoteThread.java:453)
    at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:72)
    at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
    at com.orientechnologies.orient.core.sql.query.OSQLNonBlockingQuery.access$001(OSQLNonBlockingQuery.java:47)
    at com.orientechnologies.orient.core.sql.query.OSQLNonBlockingQuery$1.run(OSQLNonBlockingQuery.java:258)
    at java.lang.Thread.run(Thread.java:745)