New version coming?

[Anton Hughes]

Hi all

Now that Vertx 3.0 is released, will there be a new version of Vertigo coming soon?

Thanks and regards

[Oliver Rolle]

yes, Jordan gave me write access to his vertigo repo so I can continue developing on vertigo.

I plan a 2 month time box to release vertigo 1.0. The idea is to port vertigo 0.7 (based on vertx 2) to vertigo 1.0 (based on vertx 3). Jordan did some architectural changes to be able to do that.
@Jordan: Maybe we both can do a skype talk so that I can really understand the new concepts for v1.0?

Initially I will break down the work for 1.0 into small tasks, so that we have one truth about the development progress of vertigo. On the other hand through the tasks, like a small feature or a bug fix, developers can contribute code in a coordinate way to the project. I will try to get some professional project management tool to be able to do that. (I hope some Microsoft guys are reading this? They'll should write me an email.)

My vision is to have a easy-to-extend, easy-to-integrate, distributed stream processing framework for a vertx environment. I build / know a couple of extension-prototypes for vertigo:
* Hadoop YARN compatible
* Event Processing Language (EPL); SQL for event processing; It should be easy to add new functionalities to the EPL (like machine learning).
* Actor-like Business Process Engine
* Self-learning actors based on Neural Networks
* Jordans CopyCat

Do you have your own vertigo extensions? We should add your experience in using vertigo to build a really neat API.

best regards
Oli

[Suminda Dharmasena]

See https://taiga.io/ for project management.

Looking forward to this. Please avoid JSon to communicate and be GC friendly perhaps using: https://github.com/kuujo/alleycat

[Anton Hughes]

On Thursday, July 16, 2015 at 1:46:07 PM UTC+2, Suminda Dharmasena wrote:

 Please avoid JSon to communicate

Why avoid json? 

[Suminda Dharmasena]

Need to parse hence CPU cycles are wasted and not good for low latency and throughput. Having string wastes memory. On how it might be implemented may not be GC efficient.

[Suminda Dharmasena]

I am saying avoid Json for internal message transport, rpc, etc. and provide support there it might be needed.

[Suminda Dharmasena]

Also look at integration with http://www.cascading.org/projects/cascalog/ and try to get the dataflow casees in: https://embedded.eecs.berkeley.edu/caltrop/ in addition to general http://noflojs.org/. I definitely will be a early adopter of this when it comes out.

[Anton Hughes]

Have you had a look at https://samoa.incubator.apache.org/ ?

It is basically a higher abstraction of event processing. Currently without support for vertx/vertigo - but could be good to align with this.

On Thu, Jul 16, 2015 at 6:05 PM, Suminda Dharmasena <sirina...@gmail.com> wrote:

Also look at integration with http://www.cascading.org/projects/cascalog/ and try to get the dataflow casees in: https://embedded.eecs.berkeley.edu/caltrop/ in addition to general http://noflojs.org/. I definitely will be a early adopter of this when it comes out.

--
You received this message because you are subscribed to a topic in the Google Groups "vertigo" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx-vertigo/T5rpF2RNiBM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.
To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/262917f0-0505-439e-8090-70bfd3f0fd64%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Suminda Dharmasena]

Best to have seamless integration with all similar platforms. 

[Suminda Dharmasena]

Something interesting: https://twitter.com/BenjaminMalley/status/614518387746541568

[Oliver Rolle]

thanks for the great interest and suggestions!

I revisited vertigo 0.7 source and tried to figure out a way how to switch easily from vertx2 to vertx3. I think I understand why Jordan developed copycat & alleycat - before doing distributed event processing, you need a way to distribute data efficiently, deal with failures and handle states of components throughout the cluster. Vertigo has a big code base, because it mixes distributed data management with distributed event processing. Because of java 7 it has lots of json objects to "glue" vertigos building blocks over the event bus together. My major worry is that I cannot decide, which building blocks of vertigo I have to replace with copycat api calls - its looks very interrelated! My second worry is, that a huge amount of json objects are constructed so that vertigo building blocks are glued via event bus together.

I investigated a way to build vertigo on top of a distributed data management (copycat), prevent serializing to JSON (as Suminda pointed out) and avoid building messages manually (less glue messages). I came up with a totally new architecture. Major advantage is to use serialized java 8 anonymous functions, which are serializable to byte[] and minimize manual message building to nearly 0. You like to iterate over the data in the cluster, just define an anonymous function and send it to the cluster nodes - no translation in json messages necessary. This make development of distributed algorithms a lot easier!

You can take a look at the prototype - pull it, build it & run it: https://github.com/orolle/reactive-vertigo/ (main() in rvertigo.main.Starter)

But this means to start vertigos event processing at square 1 again. As first step, I want to port java 8 streaming api to vertigo. First, I think it is a really useful feature when developing vertx apps (its just a map + reduce to calculate trending topics and stuff). Second, it will allow to use java 8 stream functions on vertigo ports later. Who likes to support me in porting the javas steaming api to vertigo?

[Suminda Dharmasena]

Please look at:

• http://javaslang.com/
  
• http://www.functionaljava.org/
  
• https://github.com/aol/simple-react
  
• https://github.com/aol/cyclops
  
• http://totallylazy.com/
  
• https://github.com/nurkiewicz/LazySeq
  
• https://github.com/jOOQ/jOOL
  
• https://github.com/johnlcox/motif
  
• http://reactivex.io/
  
• http://nodered.org/
  
• http://noflojs.org/noflo-ui/
  

These might give you some ideas.

Any way try to achieve the following also:

• On startup you can create GCable objects
• On shutdown you can ccrate GCable objects
• During the running of the program do not created any GCable objects

[Suminda Dharmasena]

Also another thing about communication is try hard to achive:

• Same JVM - no serialisation
• Same machine - memory mapped files
• different machine - using SBE, FlatBuffer, Cap'n Proto

[Suminda Dharmasena]

Also please have a look at: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala.html?_ga=1.166116350.1170681472.1433145789

[Suminda Dharmasena]

Study the DSLs. Also other projects which unrelated but can give you ideas are:

• https://embedded.eecs.berkeley.edu/caltrop/language.html, http://orcc.sourceforge.net/, https://en.wikipedia.org/wiki/CAL_Actor_Language
  
• https://en.wikipedia.org/wiki/Dataflow_programming
  
• https://en.wikipedia.org/wiki/Functional_reactive_programming, http://julialang.org/Reactive.jl/
• https://en.wikipedia.org/wiki/Incremental_computing
  
• https://en.wikipedia.org/wiki/Stream_processing
  
• https://en.wikipedia.org/wiki/Algorithmic_skeleton
  
• https://github.com/GPars/GPars
  

[Suminda Dharmasena]

Some more resources that might be inspiring and some which you can use:

• http://reactivesocket.io/
  
• https://github.com/reactive-ipc
  
• https://github.com/reactor/reactor
  
• http://www.paralleluniverse.co/
  
• https://github.com/RuedigerMoeller/kontraktor/
  
• http://tinkerpop.incubator.apache.org/