elasticsearch in vertx

[Ryan Chazen]

It was mentioned that elasticsearch works better than mongodb and has a nice async java driver. After looking at elasticsearch, it really does seem like a better mongodb (better indexing/searching, better clustering with quorums, versioning of documents for keeping data consistent, etc) and the async driver is nicely made. However, it looks like the async driver runs its own thread pool. Is this alright or would it be a good idea to somehow replace the elasticsearch threadpool for a vertx worker pool somehow?

Assuming the elasticsearch threadpool is OK (it seems to run fine), is this the correct way to handle the responses? I wrapped the response using runOnContext to make sure the result returns onto the correct vertx thread...

        final HttpServerRequest event = ...

        Node node = NodeBuilder.nodeBuilder().clusterName("test").build();
        Client client = node.client();
        client.prepareIndex().setSource("").execute(new ActionListener<IndexResponse>() {
           
            @Override
            public void onResponse(IndexResponse response) {
                vertx.runOnContext(new Handler<Void>() {
                   
                    @Override
                    public void handle(Void v) {
                        event.response().end("success");
                    }
                });
               
            }
           
            @Override
            public void onFailure(Throwable e) {
                // TODO Auto-generated method stub                
            }
        });
        client.close();

Also, redeploying the module a few times caused a permgen error, likely because elasticsearch has so many classes. To solve this, I was thinking of putting elasticsearch into its own module with resident set to true so that it doesn't get reloaded. http://vertx.io/mods_manual.html#resident
Is this the right way to handle it? I can't think of any other way around the permgen issue.

[Ryan Chazen]

After more investigation, Elasticsearch doesn't seem to be a good fit for vert.x. While the driver is async, it handles this by creating threadpools... and there doesn't seem to be any way to replace them. The callback function generally gets called on the thread of one of the threadpools and so vertx.runOnContext doesn't work as there is no context. You can bypass it a bit by storing the context:

final Context context = vertx.currentContext();
and then inside the callback
context.runOnContext(new Handler<Void>() {

This works, but I'm assuming vert.x is not meant to be used this way and that you could get errors as multiple threads hit the non-thread safe context. Right ?

So it looks like using elasticsearch as a new async database driver on vert.x is not a good idea after all.

[Tim Fox]

On 08/01/14 11:58, Ryan Chazen wrote:

It was mentioned that elasticsearch works better than mongodb and has a nice async java driver. After looking at elasticsearch, it really does seem like a better mongodb (better indexing/searching, better clustering with quorums, versioning of documents for keeping data consistent, etc) and the async driver is nicely made. However, it looks like the async driver runs its own thread pool. Is this alright or would it be a good idea to somehow replace the elasticsearch threadpool for a vertx worker pool somehow?

Assuming the elasticsearch threadpool is OK (it seems to run fine), is this the correct way to handle the responses? I wrapped the response using runOnContext to make sure the result returns onto the correct vertx thread...

That's almost right. What you want to do is to use Context ctx = vertx.currentContext() at the beginning of your verticle to get the verticle context, then use ctx.runOnContext() (not vertx.runOnContext())

        final HttpServerRequest event = ...

        Node node = NodeBuilder.nodeBuilder().clusterName("test").build();
        Client client = node.client();
        client.prepareIndex().setSource("").execute(new ActionListener<IndexResponse>() {
           
            @Override
            public void onResponse(IndexResponse response) {
                vertx.runOnContext(new Handler<Void>() {
                   
                    @Override
                    public void handle(Void v) {
                        event.response().end("success");
                    }
                });
               
            }
           
            @Override
            public void onFailure(Throwable e) {
                // TODO Auto-generated method stub                
            }
        });
        client.close();

Also, redeploying the module a few times caused a permgen error, likely because elasticsearch has so many classes. To solve this, I was thinking of putting elasticsearch into its own module with resident set to true so that it doesn't get reloaded. http://vertx.io/mods_manual.html#resident
Is this the right way to handle it? I can't think of any other way around the permgen issue.

That makes sense.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Tim Fox]

On 10/01/14 09:20, Ryan Chazen wrote:

After more investigation, Elasticsearch doesn't seem to be a good fit for vert.x. While the driver is async, it handles this by creating threadpools... and there doesn't seem to be any way to replace them. The callback function generally gets called on the thread of one of the threadpools and so vertx.runOnContext doesn't work as there is no context.

See my last reply

You can bypass it a bit by storing the context:

final Context context = vertx.currentContext();
and then inside the callback
context.runOnContext(new Handler<Void>() {

This works, but I'm assuming vert.x is not meant to be used this way

No, this is the exact use case this was designed for.

and that you could get errors as multiple threads hit the non-thread safe context. Right ?

No

So it looks like using elasticsearch as a new async database driver on vert.x is not a good idea after all.

It's not ideal to use drivers which have their own thread pools, but they should work.

Does elastic search expose their wire protocol somewhere? If so, someone could write a 100% vert.x client for it.

On Wednesday, January 8, 2014 1:58:47 PM UTC+2, Ryan Chazen wrote:

It was mentioned that elasticsearch works better than mongodb and has a nice async java driver. After looking at elasticsearch, it really does seem like a better mongodb (better indexing/searching, better clustering with quorums, versioning of documents for keeping data consistent, etc) and the async driver is nicely made. However, it looks like the async driver runs its own thread pool. Is this alright or would it be a good idea to somehow replace the elasticsearch threadpool for a vertx worker pool somehow?

Assuming the elasticsearch threadpool is OK (it seems to run fine), is this the correct way to handle the responses? I wrapped the response using runOnContext to make sure the result returns onto the correct vertx thread...

        final HttpServerRequest event = ...

        Node node = NodeBuilder.nodeBuilder().clusterName("test").build();
        Client client = node.client();
        client.prepareIndex().setSource("").execute(new ActionListener<IndexResponse>() {
           
            @Override
            public void onResponse(IndexResponse response) {
                vertx.runOnContext(new Handler<Void>() {
                   
                    @Override
                    public void handle(Void v) {
                        event.response().end("success");
                    }
                });
               
            }
           
            @Override
            public void onFailure(Throwable e) {
                // TODO Auto-generated method stub                
            }
        });
        client.close();

Also, redeploying the module a few times caused a permgen error, likely because elasticsearch has so many classes. To solve this, I was thinking of putting elasticsearch into its own module with resident set to true so that it doesn't get reloaded. http://vertx.io/mods_manual.html#resident
Is this the right way to handle it? I can't think of any other way around the permgen issue.

[Ryan Chazen]

Ah excellent - I thought doing ctx.runOnContext() from a non-vertx thread was a bad idea. I see now that vertx-mod-elasticsearch does this also, although it uses the eventbus from the non-vertx thread (message.reply).

Elasticsearch does have their own wire protocol, but you lose a lot of nice benefits from it. If you use the java client directly, then in a multiple server environment where your data may be on different servers, the java client will know which server to contact directly instead of having to be redirected around by other servers.

I made a module with resident:true and a simple static cache of an elasticsearch node (es is thread safe) and included it in my verticle and it is working very well with context.runOnContext. I tested redeploying the verticle a bunch of times which gave me the permgen error before and now it doesn't have a problem.

I'm playing around with joining it with a simple ORM also and the result seems very nice so far - and very speedy. I'll post up a module to the repo later (when I work out how) since it might be useful to someone - it let me put together a simple RESTful test server for a few different objects very easily, type safe and in a couple lines.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/kVtP9qNJz_Y/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to vertx+un...@googlegroups.com.

[Norman Maurer]

Sounds creat… Keep us posted!