Vert.x 3.0 draft plan

[Tim Fox]

I've started putting together a draft plan for Vert.x 3.0 https://github.com/eclipse/vert.x/wiki/Vert.x-3.0-plan

Please, feel free to comment, or make any suggestions.

[Krzysztof Makowski]

Looks great. How much effort is needed? Is it possible to estimate it? Would be great to have it done in six months :)

[Jari Louvem]

Tim!

It looks awesome. Is there a way that I can help with this plan?

[Frank Reiter]

Looks *very* exciting!

Any chance of one of the modules used to avoid nested callbacks becoming official?  I am thinking of when, yoke, rx,  etc.  Or will this issue be solved with the move to Java 8?

I think this is important to make core because practically all the code one writes will be dependant on it.  If a third party stopped maintaining your asynchronicity manager of choice you would be in a little trouble. 

Frank.

On Apr 30, 2014 5:23 AM, "Tim Fox" <timv...@gmail.com> wrote:

I've started putting together a draft plan for Vert.x 3.0 https://github.com/eclipse/vert.x/wiki/Vert.x-3.0-plan

Please, feel free to comment, or make any suggestions.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

On 30/04/14 16:57, Frank Reiter wrote:

Looks *very* exciting!

Any chance of one of the modules used to avoid nested callbacks becoming official?  I am thinking of when, yoke, rx,  etc.  Or will this issue be solved with the move to Java 8?

I think this is important to make core because practically all the code one writes will be dependant on it.  If a third party stopped maintaining your asynchronicity manager of choice you would be in a little trouble. 

I am hoping that when the proverbial magical code generator gets written, we can generate RxJava versions of the core APIs automatically as part of the build process (and maybe CompletableFuture too), then ship them as part of the core.

Of course I have to write said magical generator first, and it might be more difficult than I think.

[Paulo Lopes]

Could we have a round of discussion on the JSON objects, rework the API, add missing features e.g.: 

* add array elements to a specific index

* make the jsonelement a bit more functional e.g.: allow using get/set no mater the implementation

* why method like: addString(String, String) and addNumber and not just add(String, String) and add(String, Number)...

maybe this would deserve a own thread... 

[Tim Fox]

Agreed - let's take the opportunity to improve the API.

Yes, let's start a new thread. Hopefully Rick can help here too as he wrote Boon so has some experience in this area :)

--

[Rick Hight]

Sounds good.

[Rick Hight]

I like the direction. Some of the bullets are pretty high level so I am not sure if they mean what I think they mean, but it they do...Cool.

RE: New Gradle template
RE: New, vastly simplified Gradle template

I started messing around with Gradle and Vertx. I might be able to help here. I still have a lot to learn about Gradle. I'd like to see some specific templates for specific features and langauges as well as a kitchen sink template.

Ditto Maven. Although I get hives when I work on Maven. I understand the direction towards it.

http://www.stringtemplate.org/ code gen looks very interesting, maybe I can help on the Groovy shims.

RE: Remove Vert.x platform
RE: Migrate HA to Vert.x core
RE: Migrate Classloader isolation to Vert.x core
RE: Migrate auto-redeploy to verticles
RE: Provide new SPI for new language support

This mostly sounds like our move away from Vertx modules and towards NPM, Maven based modules. Right?

Official modules...

I am interested in helping with these:

RE: JDBC, Postgres/MySQL

Mainly because of techempower benchmarks. What can I say... I am a sucker for benchmarks, and I want to see vertx dominate.
Benchmarks and graphs are a personal weakness of mine.
I don't know why vertx is not in more db stuff and it looks like the winners in this benchmarks are the ones who are using MySQL (we use MongoDB last time I looked).

RE: Refactor docs
RE: Refactor examples

I enjoy this type of stuff so this is an area I can possibly help in.
I like that the docs are terse, I think we need a less terse tutorials area.
I am not good at terse so I am more of a tutorial/examples guy, and less of a docs guy.

RE: Worker Queues

I am also interested in this one and messaging in general.
I wrote a key/value store abstraction over hazelcast, mysql, leveldb, rocksdb, lmdb (and mongodb is in progress) for persistent jobs.
https://github.com/RichardHightower/slumberdb

NEW TOPIC

EventBus Routing:

It seems now (and I could be wrong) that in order to use the event bus you need to be on the same cluster (or be a javascript client using SockJS)

I did not see a SockJSClient.java only a SockJSServer.java so...

I was wondering if we could create an EventBus Router. It is just a repeater to another cluster ( one day another opaque message system (Redis, JMS, Rabbit.... whatever)).

I would just use it to bridge messages between clusters.

So let's say hypothetically I have a highspeed cluster A that is handling 2,000,000 requests per second, and I have a middleware cluster B that is integrating with outside systems but at a much more relaxed pace.

My knee jerk reaction is to put them on independent clusters and maybe even put the hardware network setup around them differently. They have to communicate. B has to send A updates. They don't happen often (batch jobs, integration, middleware stuff).

But it seems (and I could be wrong) that in order to use the event bus, we would need to bridge A and B since the event bus seems to only work on one cluster.

So it seems like it would be nice for the event bus to have some adapter/bridges so other clusters, languages, platforms could participate in it all.

This probably already exists but it would more or less be generic lib that looks like the event bus lib. It will have a verticle and a client API. You just write the message to it, and it will repeat it to the event bus. It is just a pass through. This makes the event bus accessible to anyone who can use websocket or HTTP or whatever. The API will be the same too. Please let me know if this already exists because I actually have to write this for work if it does not already exist.

[Jordan Halterman]

That actually reminds me of something that was proposed I think during an IRC meeting last year. And I think this is what Richard was getting at. Refactor the event bus protocol (if it's necessary) to support event bus client libraries. Right now, I'm using a simple delimited protocol through a NetServer to forward json messages to the event bus from PHP scripts. It would be nice to take the server out of the equation :-) But with the event bus being a peer-to-peer system that uses Hazelcast maps for address registries, I'm not sure how difficult it would be to write a proper client. Maybe Vert.x nodes would just have to expose an interface internally.

Sent from my iPhone

[Nick Scavelli]

On Wednesday, April 30, 2014 8:56:25 PM UTC-4, Rick Hight wrote:

NEW TOPIC

EventBus Routing:

It seems now (and I could be wrong) that in order to use the event bus you need to be on the same cluster (or be a javascript client using SockJS)

I did not see a SockJSClient.java only a SockJSServer.java so...

I actually started one awhile ago here https://github.com/nscavell/vert.x/tree/sockjs-client but that got pushed aside. For 3.0 we probably just want to create a wire protocol for the event bus and not depend on sockjs & websockets.

[Rick Hightower]

wire protocol + http + websocket.

so make it a REST/JSON wire protocol over http / web socket then any client (can send a message on the bus if we configure it)

So js has sockjs so lets call that done., 

Java, Python, Ruby, PHP, Go. etc. all have HTTP/Websocket. The nice thing about Websocket/HTTP/JSON is every programming language can get involved because it is all part of the core for those languages. 

Then the forwarder verticle just reads the payload, extract the body, and sends the message on the event bus. It is just a forwarder verticle. The wire protocol can be as simple as ["address", payload].

(That was a JSON array)

Or if you want...

["publish","address", payload]. //p

["request","address", messageID, payload].

["reply","address", messageID, payload].

["error","address", messageID, payload].

where messageID is a number, and payload is any damn think you want.

And address is the event bus address.

or how about this

["publish",{}, "address", payload]. //p

["request",{}, "address", messageID, payload].

["reply",{}, "address", messageID, payload].

["error",{}, "address", messageID, payload].

If the second argument is a map, then there is a header (for auth or additional routing).

Ok leave that out...it is staring to look like JMS

["publish","address", payload]. //p

["request","address", messageID, payload].

["reply","address", messageID, payload].

["error","address", messageID, payload].

There ... there is a wire protocol. Now every client in the world can publish a message on our bus over REST and web socket. Tim/Norman becomes super rich and famous, and we all retire in the south of France.

It seems like it would take ten minutes to write. :)

The setup would be similar to the worker queue verticle/module. :)

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/DHL9cAg4SYo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Rick Hightower
(415) 968-9037
Profile 

[Tim Fox]

On 01/05/14 18:28, Rick Hightower wrote:

wire protocol + http + websocket.

so make it a REST/JSON wire protocol over http / web socket then any client (can send a message on the bus if we configure it)

This already exists and is how the eventbusbridge works.

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Arno Schulz]

Not 10 minutes, but this is starting to sound like atmosphere -- and there is a vertx module for it https://github.com/Atmosphere/atmosphere-vertx

--

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Rick Hightower]

Ok.. 

eventbusbridge

I thought the eventbusbridge was sockjs or are you saying sockjs already allows REST, web socket calls and you could just use it?

Going through the online docs it seems like the bridge is only for JS clients.

There are no examples. I have Java and Python clients, who need to post things on the bus. So... 

But is that where I should look?

Ok.. I that is enough of a hint.

[Rick Hightower]

It is amazing what google can find.

https://groups.google.com/forum/#!topic/vertx/sLtXoIQu6-8

"""

3/2/13

I realized that I can use vertx http client t o connect to the SockJS server.
Vertx vertx = Vertx.newVertx();
    HttpClient client = vertx.createHttpClient().setHost("127.0.0.1").setPort(8080);;
    client.connectWebsocket("/testapp",WebSocketVersion.RFC6455,new Handler<WebSocket>() { .....

"""

[Rick Hightower]

So I found it on the web, but this would be good to add to the docs. I will submit an issue.

"""Thanks to clarification from Tim I'm now using a less complex way with raw websockets:

static void vertxWrite(type, address, body) {

    VertxFactory.newVertx().createHttpClient().setHost('localhost').setPort(8888)

        .connectWebsocket("/eventbus/websocket", new Handler<WebSocket>() {

        public void handle(WebSocket ws) {

            def envelope = [type:type, address:address, body:body]

            ws.writeTextFrame(JsonOutput.toJson(envelope))

            ws.close()

        }

    }).close()

}

"""

[Rick Hightower]

Arno:

I thought it sounded more like JAMP. :)

http://hessian.caucho.com/jamp/index.xtp
https://github.com/RichardHightower/dart-jamp-examples

http://json-amp.github.io/

https://github.com/json-amp/json-amp.github.com/wiki/Intro-to-JAMP

http://java.dzone.com/announcements/geekfest-w-richard-hightower

But... seems like I need to learn more about Atmosphere. :)

[Arno Schulz]

Maybe I'm missing something here. I meant atmosphere would allow to connect via various protocols (websockets, sockjs) to the event bus.

It also has a java/scala client library https://github.com/Atmosphere/wasync

I was playing with it to see if it could make a hipache replacement based on vert.x

This was also discussed a while back on replacing the vert.x internal Sock.js
https://groups.google.com/forum/#!topic/vertx/jbW3SxyMwW0

[Rick Hightower]

I am against "just using atmosphere" for vertx core.

To me sockjs, eventbusbridge and the eventbus is core to vertx.

I just think vertx needs a nicer non-JS api for it.

A good Java client API would be nice.

(I am not against atmosphere modules or verticles, but vertx is how I do websockets/sock.js)

[Arno Schulz]

It depends what one sees as the core, to me the eventloop is the central tenant of Vert.x (ditto for Node.js).

Other elements can be thought of as modular to encourage experimentation -- mod-rxvertx is a great example of this.

For the Java websocket/sockjs client the question becomes is it preferable to have something part of the core or use wasync (directly or as a module).

[Tim Fox]

My question would be: What are the problems you see with the current SockJS/websockets implementation that make you want to replace it? I.e. what is the business case for atmosphere?

[ddimensia]

I would like to see all the Vert.x Default* classes to be modified so that they can be extended/overridden.  As things are currently it's impossible to get access to lower level constructs in the codebase to provide more detailed metrics or logging.  It's also impossible to provide alternative implementations for things like hostname resolution, flow control, etc where the default implementations don't suit an applications needs.

-ddimensia

[Arno Schulz]

There is no business case for atmosphere vs the current implementation.

The current implementation for websockets/sockjs works and modules such a atmosphere-vertx already exist for anyone to use (ditto for using wasync as a Java/Scala client)

[Rick Hightower]

API.

As part of Vertx 3, Can we have an API like the EventBus API that works from cluster to cluster using the event bridge?

If so, I will volunteer fork/PR it and as it so happens, I have to do this for my day job so.....

So server config looks like:

{
  address : 'demo.persistor',
  match : {
    action : 'find',
    collection : 'albums'
  },
  requires_auth: true
}

(above From docs)

Client (client cluster), looks like this:

CONFIG:

bridgeClient.json
{

    {

       "name" : "videoVendorServiceCluster"

        "servers" : [ipaddress1, ipadress2, ipaddress3]

    }
}

The client lib....

CODE:

EventBus eb = vertx.eventBus("videoVendorServiceCluster");

Handler<Message> myHandler = new Handler<Message>() {
    public void handle(Message message) {
        System.out.println("I received a message " + message.body);
    }
};

eb.registerHandler("test.address", myHandler);

[bytor99999]

Looks Awesome. Thanks Tim

Mark

On Wednesday, April 30, 2014 5:23:41 AM UTC-7, Tim Fox wrote:

[Rick Hightower]

Impossible? No such thing. aspectJ?

opening up all classes that implement interfaces already right?

If you need tooling for metrics there are lots of tools that do this without turning the core into spaghetti or JavaObjectFactoryFactoryFactory cruft.

Wouldn't that negate the cohesiveness of the core if you opened up everything?

RE:  It's also impossible to provide alternative implementations for things like hostname resolution

Really? Can you give an example of a similar lib or platform that opens up everything?

--

You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/DHL9cAg4SYo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ddimensia]

I'm not asking to open up every aspect of the default implementations just to make it so that 'Default' doesn't mean 'Only' as it does today.  There are plenty of systems that are far more extensible that Vert.x when it comes to hooking into or controlling default implementations of the platform.  Jackson is a prime example, you can tweak just about anything in the system by implementing against a known interface and providing your own implementation.  RestEasy is another example of an extensible framework that allows for hooks at various levels to control the execution and flow of requests.  

The current situation in Vert.x is that I have to hope that the system does the right thing for my application.  In cases where it doesn't the options are added setters/getters in the existing implementation to control behavior and hope the changes are excepted or work around it.  It'd be far better if there was an abstract base class the default implementation is based on that others could extend and the ability to pass a factory into the Platform or Vertx implementations to allow overriding of a default factory.   So, instead of having a DefaultHttpClient I could build a LoadBalancingHttpClient that uses SRV dns records to and round-robin over a list of IPs.  

I guess if the consensus is that Vert.x is perfect and made all the right decisions for all reasonable situations then there's nothing more to discuss.  I'd prefer to see a better more extensible framework built that can be adapted to use cases not originally considered.

-ddimensia

[Rick Hightower]

My opinion is certainly not a consensus. I am not on the team. I am a mere groupie. 

That said... To me Vertx seems a bit lower level than that. 

I think the answer is that Vertx is a core lib and you would build something like you described on top of Vertx, and in Vertx 2, you would make this a module. http://modulereg.vertx.io/

In vertx 3, you would make into a thing-formerly-known-as-a-module or just a java lib aka jar if in fact modules go away at some level at which I a not 100% sure about so.... 

So there is the core / platform and you build stuff on top of the core. 

The core is small. 

The vertx core and platform is actually quite small. 60K lines of code. I think Jackson is much much bigger for example.

So stuff like you described belongs in a module.

[Jordan Halterman]

Maybe extensibility is in the future of Vert.x, but simplicity is certainly one of the features that attracted a lot of users to Vert.x. The Java ecosystem is notoriously wrought with configurability and complexity to the point where frameworks become almost useless. That's not to say that extensibility is a bad thing, I'm just saying there's a balance that needs to be sought.

Jordan Halterman

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Arno Schulz]

How about a core (event loop, eventbus and what not) vs "reference" implementations (sockjs, http server) reference may not be the right name though.

[Rick Hightower]

Vertx Core/Platform = event loop, eventbus, clustering, sharedmaps et al, websocket, http, TCP/IP, UDP, host, async file I/O, and sockjs.

Why sockjs? Realistically you have to support sockjs to support websocket because you need a standard way to fallback to long polling, etc.

So sockjs because only real way to use websocket ATM and be compatible with as many browsers as possible.
 

[Arno Schulz]

I get what the current core is, it's tight we'll written and uses netty at its core and as you mentioned all the elements are required in one way or the other.

To me And I can be wrong I visualize vert.x/node.js in terms of layers

- there's the netty API

- there's the Vertx API

- there's the networking primitives (TCP/IP, UDP, SSL)

- there's the network (HTTP, webbsocket , sockjs, clientDNS)

Let's take an example a CRON scheduler module in the implementation I could go either through the Vert.X periodics API (the easy way), or go against the Netty API. in the end it produces the same module that can take a json configuration message and sends back a message when reaching it's scheduled time.

If we take a look at node.js, the default API also has the same layering, but most sites will go using express.js.

Facilitating this type of extensibility is what I think off, does this make sense.

[Tim Fox]

On 01/05/14 21:41, ddimensia wrote:

I'm not asking to open up every aspect of the default implementations just to make it so that 'Default' doesn't mean 'Only' as it does today.  There are plenty of systems that are far more extensible that Vert.x when it comes to hooking into or controlling default implementations of the platform.  Jackson is a prime example, you can tweak just about anything in the system by implementing against a known interface and providing your own implementation.  RestEasy is another example of an extensible framework that allows for hooks at various levels to control the execution and flow of requests.  

The current situation in Vert.x is that I have to hope that the system does the right thing for my application.  In cases where it doesn't the options are added setters/getters in the existing implementation to control behavior and hope the changes are excepted or work around it.  It'd be far better if there was an abstract base class the default implementation is based on that others could extend and the ability to pass a factory into the Platform or Vertx implementations to allow overriding of a default factory.   So, instead of having a DefaultHttpClient I could build a LoadBalancingHttpClient that uses SRV dns records to and round-robin over a list of IPs. 

I don't see why you have to remove the DefaultHttpClient in order to build your LoadBalancingHttpClient....

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[dexter31]

How about a visual vert.x interface over simple tcp with the core i/o's publishing? Just as a debugger functionality and to better understand how the core works with jvm?

[Tim Fox]

Can you elaborate on what you mean by a "visual vert.x interface"? (This page is just to discuss core API changes)

--

[Tim Fox]

Ah sorry! I mistakenly thought you were replying to the post on Vert.x 3.0 core API changes which I posted just minutes before..... 

On Friday, 2 May 2014 13:49:00 UTC+1, Tim Fox wrote:

Can you elaborate on what you mean by a "visual vert.x interface"? (This page is just to discuss core API changes)

On 02/05/14 13:40, dexter31 wrote:

How about a visual vert.x interface over simple tcp with the core i/o's publishing? Just as a debugger functionality and to better understand how the core works with jvm?

On Wednesday, April 30, 2014 8:23:41 PM UTC+8, Tim Fox wrote:

I've started putting together a draft plan for Vert.x 3.0 https://github.com/eclipse/vert.x/wiki/Vert.x-3.0-plan

Please, feel free to comment, or make any suggestions.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

[dexter31]

I was talking abt a visual way for developers to understand how the event bus context works within Vertx. The simple reason being, not even very good developers seem to understand chaining on events within their programmatic context. Ive been through this a lot on node and angular js - on large scale projects, where chaining is a need rather than a nice to have.

My thoughts were based on - if they visualize what happens within the bus, they know what to do. Again, one of my assumptions that this is easier on a VM.

[Thomas Cataldo]

Hi guys,

Some comments :

The negative ones first :

Our app(s) is osgi/eclipse runtime based, sad to see modules and their
classloader will just move to core and not just disappear. APIs that
take a classname and defines classloaders are hard to deal with for
us, as it messes with what osgi does for us.

Nothing for NetSocket. For our vertx based IMAP proxy we had to expose
ConnectionBase::queueForWrite & use reflection to be able to call
channel::flush() at the right times. NetSocket.write() doing a
writeAndFlush was a major headache for us.

Buffer memory management & copy management. ByteBuf.slice() is not
exposed, so the first thing we do is buffer.getByteBuf() so we can use
slice & composite buffers to avoid some copies. Then we activated
netty PARANOID leak detector and got reports we didn't understand. My
understanding is that vertx marks its buffers as unreleasable and
let's the garbage collector do its magic but I may be very wrong.
While reviewing that stuff we realised that this part of the app would
be better as pure netty4 stuff, but I'm a fan of the eventbus
send/reply stuff to push a task to another threadpool.

The positive ones :

Your dispatcher proposition for multiple worker queues.

rxjava & reactive streams stuff. Say goodbye to some/most nested callbacks hell.

Not raised stuff:

allow eventBus.send with a java.lang.Object, or require anything with
an Immutable annotation / inheritance, at least to a local verticle.
Paying serdes cost to just switch a task to another threadpool is
hard.


Regards,
Tom.

[dexter31]

As well, I had to highlight one thing - programming was simple at some point, before which the new paradigm came by. Eventing is scalable, but not easy to adopt. Im just thinking of old times and why we changed from one to the other:)

[Jordan Halterman]

A UI is being developed for Vertigo which does what I think you're describing - displays relationships between verticles and message flow and what not. I think it may be more relevant in that context where Vertigo explicitly defines relationships between verticles whereas the event bus is inherently agnostic about who communicates with who.

[Tim Fox]

That sounds awesome.

Is there any possibility that it can be made generic enough so it can work with raw Vert.x event bus, as well as Vertigo?

[Tim Fox]

On 02/05/14 20:26, Thomas Cataldo wrote:
> Hi guys,
>
> Some comments :
>
> The negative ones first :
>
> Our app(s) is osgi/eclipse runtime based, sad to see modules and their
> classloader will just move to core and not just disappear. APIs that
> take a classname and defines classloaders are hard to deal with for
> us, as it messes with what osgi does for us.

The proposal is that modules will disappear. The isolation proposal is
that you can use an api to isolate a verticle deployment, e.g.

vertx.deployVerticle(verticleClassName, isolationGroup);

But this is entirely optional, so you can just not use it.

>
> Nothing for NetSocket. For our vertx based IMAP proxy we had to expose
> ConnectionBase::queueForWrite & use reflection to be able to call
> channel::flush() at the right times. NetSocket.write() doing a
> writeAndFlush was a major headache for us.

Can you elaborate on this? I am not aware of this issue.

>
> Buffer memory management & copy management. ByteBuf.slice() is not
> exposed, so the first thing we do is buffer.getByteBuf() so we can use
> slice & composite buffers to avoid some copies. Then we activated
> netty PARANOID leak detector and got reports we didn't understand. My
> understanding is that vertx marks its buffers as unreleasable and
> let's the garbage collector do its magic but I may be very wrong.
> While reviewing that stuff we realised that this part of the app would
> be better as pure netty4 stuff, but I'm a fan of the eventbus
> send/reply stuff to push a task to another threadpool.

We could consider adding slice etc to the Buffer....

[Jordan Halterman]

I'm not developing it (that's Oliver Rolle) and we're just now beginning to get into the details of it all (though there is a prototype based on the older Vertigo version) but I will definitely see what we can do to abstract away enough Vertigo details to potentially make the UI usable to graph general Vert.x verticle event bus relationships. That would be pretty amazing! But Vertigo has the unique benefit of having predefined relationships through configuration objects, whereas the event bus doesn't have such inherent knowledge of where messages should be coming from or going to. I suppose it would require some sort of hook or API to extract the relevant information from the event bus (where handlers are registered and where messages are being sent to) to generate a graph, but obviously with enough useful information it should be possible.

The UI is modeled on connections between named in/out ports, but maybe those can just as easily be represented as event bus addresses.

I'll try to make sure it's well designed with event bus compatibility in mind in case it can be of use when it's more complete :-)

[Thomas Cataldo]

On Sat, May 3, 2014 at 10:28 AM, Tim Fox <timv...@gmail.com> wrote:
> On 02/05/14 20:26, Thomas Cataldo wrote:
>>
>> Hi guys,
>>
>> Some comments :
>>
>> The negative ones first :
>>
>> Our app(s) is osgi/eclipse runtime based, sad to see modules and their
>> classloader will just move to core and not just disappear. APIs that
>> take a classname and defines classloaders are hard to deal with for
>> us, as it messes with what osgi does for us.
>
>
> The proposal is that modules will disappear. The isolation proposal is that
> you can use an api to isolate a verticle deployment, e.g.
>
> vertx.deployVerticle(verticleClassName, isolationGroup);
>
> But this is entirely optional, so you can just not use it.
>
>
>>
>> Nothing for NetSocket. For our vertx based IMAP proxy we had to expose
>> ConnectionBase::queueForWrite & use reflection to be able to call
>> channel::flush() at the right times. NetSocket.write() doing a
>> writeAndFlush was a major headache for us.
>
>
> Can you elaborate on this? I am not aware of this issue.

Sure, here is a short version of what we do. Our dataHandler to the
upstream imap server puts everything into a RecordParser (slightly
modified as our delimiter is always \r\n we changed it to use a
ByteBufProcessor to chunk the delimited stuff). Then the RecordParser
calls our "handleRecord" to send the pieces back to the client socket.

A version without flush exposed looks like this :
https://gist.github.com/anonymous/abfd4c43c4fcd13f08ac
A version fetching the Channel from ConnectionBase looks like this :
https://gist.github.com/anonymous/e5acc4b20b595ca702f4

The we run a unit test that forces the upstream imap server to send a
lot of stuff to us, lots of splittable lines.
The test runs in 16 seconds when we control the flush call. The test
runs in 50sec when we do plain NetSocket.write(Buffer).

I attached screenshots of how the two performs from a yourkit point of view.

[Tim Fox]

I'm still not following - I see Netty classes in your example... ?

[Thomas Cataldo]

Well yes. The first thing we do in our Handler<Buffer> is
getByteBuf(). Our record parser is modified to act on ByteBuf so it
can use slice, create composites instead of appending and use a
ByteBufProcessor instead of lots of getByte(int) to find delimiters.

The point is if we use plain NetSocket.write() our hotspot is calls to
the pipeline.writeAndFlush whereas when we instruct our code when to
flush, the tests becomes dominated by reading and parsing which looks
more normal given what the server sends to us.

[Tim Fox]

I don't have a good picture of what you're doing here and what this
handler is for, perhaps you can explain in more detail?

[Thomas Cataldo]

The architecture is pretty simple :

imap client -> NetSocket
The NetSocket has a dataHandler that does
pimpedRecordParser.handle(buffer.getByteBuf())

The pimped record parser calls handleRecord for each splitted incoming
imap line.

Until the login phase is complete (which is needed to know which
upstream server the user data is on) we handle all the imap dialog
like a normal imap server.
Once the login phase is complete, we create a second NetSocket to the
upstream server with a similar setup.

Now we go into proxy mode : handleRecord from the client NetSocket
writes to the upstream NetSocket.
handleRecord from the upstream NetSocket writes its data chunks back
to the client NetSocket.

Of course we don't parse the imap chunks both ways "for fun",
otherwise nginx as an imap proxy works like a charm. When we identify
a command starting, we send a message on the event bus to remember it
is in progress.

On the other side when the server sends us data back, we analyse to
find command completions. For example when your mail client finishes
to append an email in your Sent folder, an event on the bus will
trigger elasticsearch indexing of your new item.

So on one way we have :
imap client -> netsocket.datahandler.handle -> recordparser ->
handlerecord -> upstreamsocket.write
And when data flows back:
imap server -> netsocket.datahandler.handle -> recordparser ->
handlerecord -> clientsocket.write

I did not provide a link to the real files in the first place (but
it's opensource) as you're already doing a wonderfull job and you're
not here to review every users application. But for the record here is
the link to the modified RecordParser that shows lots of netty classes
in yourkit : http://git.blue-mind.net/gitlist/bluemind/blob/master/plugins/net.bluemind.ips.vertx/src/net/bluemind/ips/vertx/impl/ByteBufProcParser.java

[dexter31]

Echo Tim there. Im a relative newb here. Is this the one you are referring to? https://groups.google.com/forum/#!topic/vertx-vertigo/IdPk2A6q_Yc

[Arno Schulz]

That's it 

if you want just to see what it looks like you can see an example on the noflo project page http://app.flowhub.io/#example/6699161

[Arno Schulz]

And also there the project page for the ui verticleflow http://verticleflow.partypeer.de/

And the github page https://github.com/orolle/vertigo-ui

[Tim Fox]

I see.. Another possibility to avoid going to Netty might be to buffer
your writes yourself, i.e. just append to a buffer, and when it gets to
a certain size write the whole thing with a netsocket.write().

We could also look at exposing the Netty flush() functionality through
the netSocket interface.

[Norman Maurer]

Expose the flush() operation sounds like something easy to do and should not harm at all. 

Opened a PR for it:

https://github.com/eclipse/vert.x/pull/830

-- 
Norman Maurer

[Steven Holmes]

Instead of a gradle template have you considered doing a lazybones template?  I think it's a better overall user experience than cloning a github repo.  You could also build seperate templates for different use cases.

[Norman Maurer]

Could you give a bit more details? I never heard of this..

--

[dexter31]

Was abt to post an example. Was just playing around 'Play' framework today. And they had something along the same lines on actor models and DLQ's. But took a look at the flowhub example posted. Looks pretty cool, and from a developer standpoint - something really useful. Tks.

[JC]

Re templating - Yoke covers it pretty fine for server-side generation (not only web apps). But moving Yoke to "core" would imply imho some (heavy) trimming I'm afraid, as some of its features would overlap with other core modules' way of working...

Re exposing simple wire protocol - do you mean low level access to it, or also the possibility to replace it? What if I think for my kind of data, Thrift would be better suited? (that would help also C++ connecting to the eventbus). Or that around the (upcoming) eventbus bridge, one side uses JSON the other side something else and the bridge does translate? Is combining these up a good idea at all?

S

[Steven Holmes]

Lazybones is a project templating tool.  If you use GVM it could be super easy for an installation:

gvm install vertx
gvm install lazybones
lazybones create vertx-webapp my-app/

This would create the directory structure, build.gradle, and any other necessary configuration files.

[Galder Zamarreño]

Re: "Generate other language shims (e.g. JavaScript, Groovy, Scala, etc) for our core APIs. This will enable us to concentrate on the core APIs and not spend a lot of effort manually keeping other language APIs up to date)."

^ I know it's an example, but as agreed, please leave Scala out of the code generation until we have a true Scala API that makes the most of the language. This is something we'll be working on for mod-lang-scala 2.0 which will target Vert.x 3.0

Cheers,

[Rick Hightower]

heard of lazybones. what is main advantage of it over maven archetypes?

so don't know why.. but saw Tim say they are switching to maven for build from gradle...

so why back to maven? (I just learned gradle well enough to work on vertx), and if maven is the direction ......

so does lazybones work with maven?

I've created maven archetypes and... well.. they are ok... but... since you are advocating lazybones, i'd like to know more....

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/DHL9cAg4SYo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Rick Hightower
(415) 968-9037
Profile 

[Steven Holmes]

Why switching to Maven?  Is there something it does that Gradle does not?

[Tim Fox]

It's the principle of least surprise - more devs tend to use Maven than Gradle.

--

[Rick Hightower]

I thought that Spring and Hibernate moved to Gradle.

I guess I got used to Gradle now. 

I don't know it well, but the ability to do stuff with it seems to be there, and maven seems a bit harder to do random bits with (I can see both the good and bad in this).

Gradle:

Hibernate (confirmed)

Spring (confirmed)

Project reactor (confirmed Gradle)

Anyway... I can't find many new projects that are using maven. And some of the old stalwarts have switched.

Boon sadly uses maven, and I am unlikely to switch. :)

I "learned" (I don't know it that well yet) gradle to work on groovy and vertx. 

I think that moving to maven is a mistake at this point. 

But it is easy to work with either.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/DHL9cAg4SYo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arno Schulz]

One plus for Maven is that it easy to integrate gitflow (atlassian -- https://bitbucket.org/atlassian/jgit-flow/wiki/Home)

[Steven Holmes]

Maven definitely is the standard bearer today but Gradle is making leaps and bounds in terms of adoption.  I'd bet a dollar that Gradle will match Maven's adoption in the next year or two.  Using maven (mvn build) is just as easy as Gradle but I've found Gradle to be so much more flexible as soon as your tread beyond the standard Java Project world.  Especially if you're using loads of alternative JVM languages, and possibly non-JVM languages in testing (C++ for the streaming), Gradle is going to handle that fine whereas Maven will fall on it's face.  Gradle is far less opinionated.

Steve Holmes

"It is better to keep your mouth closed and let people think you are a fool than to open it and remove all doubt."
-Mark Twain (or maybe Abraham Lincoln, I've heard both)

[Tim Fox]

On 20/05/14 17:42, Rick Hightower wrote:

I thought that Spring and Hibernate moved to Gradle.

I guess I got used to Gradle now. 

I don't know it well, but the ability to do stuff with it seems to be there, and maven seems a bit harder to do random bits with (I can see both the good and bad in this).

Gradle:

Hibernate (confirmed)

Spring (confirmed)

Project reactor (confirmed Gradle)

I know that most JBoss projects use Maven, and Spring and Reactor are Pivotal who are very Groovy/Gradle oriented.

[Tim Fox]

I know that whatever build tool we chose there will be people who are unhappy with the choice. Personally I'm not a huge fan of either Maven or Gradle, but Maven has significantly more adoption than Gradle which makes it a lower barrier entry to more people.

[Steven Holmes]

Honestly I don't care from a core point of view.  I hope extensions (modules?) can use whatever build system they want.  And vertx projects can use any build tool they are inclined.

Don't forget that Gradle is the Android build tool now.  So there are hundreds of thousands of Android developers using Gradle.  Although it's often hidden in the IDE...

[Tim Fox]

On 20/05/14 18:32, Steven Holmes wrote:

Honestly I don't care from a core point of view.  I hope extensions (modules?) can use whatever build system they want.  And vertx projects can use any build tool they are inclined.

Absolutely. We're only talking about the build tool used for the core Vert.x build itself. The majority of Vert.x users will never even run that, and they can use whatever they want in their own projects :)

[Rick Hightower]

Good point. It really does not matter. The thing I like about maven builds is they are more "standard" in that people don't go buck wild with them. 

[Tim Yates]

Java 6 probably has more adoption than Java 7 ;-) 😝

[Steven Holmes]

So...who's penned in to do the conversion?

[Nick Scavelli]

I share the same concerns, but in the end I'm not sure it matters that much. Currently the build is pretty simple, which works well with Maven. But as the build becomes more complicated I believe gradle has it's advantages for sure. So when we talk about bringing in modules into the core project, how complicated does the build get and would gradle be better suited for this scenario especially if any modules we include are not written in Java ? Not sure, but I don't think it's a decision we're forced to live with forever. Worst case scenario we realize it's a mistake and convert back to gradle :)

[Nick Scavelli]

Initial work has been pushed to vertx_3.0_master branch.

[Jakob Adala]

When i use maven i am not surprised getting problems ;-) Maven is only good when you use it as intended, but breaking out of this often causes problems. So i would be glad to stay with gradle.