What's New?

[Suminda Dharmasena]

What is happening with Vertigo? Is it dead? Can this be revived?

[Carter Youngblood]

Still interest here as well. I have seen the amazing work done with http://atomix.io, and I can see how you can use it to make vertigo what it can be. Programming in the vertigo framework has been fun and very useful!

Please revive :)

[Suminda Dharmasena]

Maybe this can also be an atomix project? (Reactive & Data flow Grid / Cluster computing)

[jordan.h...@gmail.com]

I agree! I could definitely see that happening. There is *definitely* intention to put a processing framework on top of Atomix 100%. That's the most underdeveloped aspect of it right now, and I think working with Vert.x/Vertigo makes complete sense in that respect as I'm still very fond of the ability to support multiple languages.

In fact, I was just thinking about this today. But my time is so monopolized by Copycat and Atomix releases right now that a revival of Vertigo will necessarily have to be an extension of Atomix, but that sounds awesome! Still, it will be some time before something like that happens. If Copycat/Atomix/Vertigo were my full time job, I'm sure we would have everything we wanted :-)

On Nov 5, 2015, at 2:06 AM, Suminda Dharmasena <sirina...@gmail.com> wrote:

Maybe this can also be an atomix project? (Reactive & Data flow Grid / Cluster computing)

--
You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.
To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/b39a5c4f-26dc-4a83-8390-eb9827d1f2f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Suminda Dharmasena]

Why don't you make this part off your job like perhaps introduce it to your work place and have it in their products. Your office can leverage from the effort that is going on to this project in your spare time also which is and win win situation.

--
You received this message because you are subscribed to a topic in the Google Groups "vertigo" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx-vertigo/858YGq8hS18/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx-vertig...@googlegroups.com.

To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/D053F81E-5E37-46C9-B4FB-2C22AB5CBACD%40gmail.com.

[jordan.h...@gmail.com]

I obviously agree.

I just suppose we'll have to wait and see what happens ;-)

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/CAAfFNYHFzdgZym3cULBYR7ArBXSAO0J7nPvuwdtuVaCZStaPWA%40mail.gmail.com.

[Suminda Dharmasena]

Next step would be to convince them to have a couple of developers working on it.

For this to happen you have to show off and promote its capabilities at work. 

[Suminda Dharmasena]

Is it possible to put out an Alpha quality version soon. Something you can code against.

Also have a look at: http://tensorflow.org/

[jordan.h...@gmail.com]

Not solely from me, but hopefully from Oliver (hint hint) :-) I'd be happy to commit to handling the task of making a well-designed graph processing framework distributed, reliable, and awesome. But what I can't focus on now is designing that framework. My open source schedule is far too full at the moment to take it on by myself. I'm trying not to take on any more projects until full versions of Copycat and Atomix are released, and that will be after at least a couple more development iterations. I'm also working on an extensive paper describing and specifying Copycat's extensions to the Raft consensus algorithm, and I still maintain several projects for the Vert.x community itself. I expect those things to consume most of my free time for at least the next month. After that, I'll likely begin playing with some variation of this concept, and knowing myself will probably put a lot of time and effort into creating something good, I just can't think about it too much yet.

Tensorflow's pretty awesome BTW!

On Nov 9, 2015, at 7:21 AM, Suminda Dharmasena <sirina...@gmail.com> wrote:

Is it possible to put out an Alpha quality version soon. Something you can code against.

Also have a look at: http://tensorflow.org/

--

You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/aa512709-5567-4f6f-8575-45be9f177469%40googlegroups.com.

[Suminda Dharmasena]

I understand. 

Also try to convince your work place to make better use of you.

[Magnus Koch]

Hi,

Wanted to voice interest in this as well. We are currently planning to use a vertigo on vertx3 for a new platform we are using, where it will play an important part in wiring together loosely coupled components. 

We have some minor work done on the latest 1.0 version in a fork at https://github.com/englishtown/vertigo/tree/vertx3-ef. So far it's mostly wiring together existing and implied bits of functionality that were not complete, and adding some unit and integration tests, but we may be able to put in more serious work if needed. 

So we're quite interested in knowing more about the direction this is going in. What we really like about the current version is the separation between network configuration and component implementations. The way we're using it is to allow "tenants" to get independently built components to work together by configuration. For that the current network definition (json or class structure) and port/connection abstractions are almost perfect. The prototype in Oliver's repo seems quite different with more of a reactive stream focus, and at this point it's hard to see how that compares.

Have you guys had discussions on the overall design? Can you share some thoughts?

Best,

Magnus

[Oliver Rolle]

I am committed to vertigo. Just my time is the big issue. The company I work for is rapidly expanding and I am in charge of building our data scientist and business analytic team. Also I just moved and will move in 2 weeks again, so even less free time. Maybe I am at the CES in Vegas in January, but its not sure atm.

I had 2 point I suspended development of vertigo for vertx3. The first point is that Jordan already put a lot of effort into stuff like distributed state management, distributed collections, serializing, etc. into atomix which is pretty much the foundation I need for stream processing and if I implemented the same functionality for vertx3 it would be huge obstacle to verify its correctness (and still has the pain points of hazelcast). As atomix was not released on maven central at that time, I could not estimate the effort what to do to make it usable, when it will be usable and was scared about frequent API changes of yet-to-be-released atomix. The second point is that I will not work alone on any project because I hate it to work alone.

I already talked to my boss / investor to support vertigo. But he does not see any business value in open sourcing software. Its always the question "how is the money made with free software?". I argued to build the open standard / platform for distributed reactive applications in the cloud and sell the brand, enterprise version and support, but he was not convinced. (@Jordan: You should think about to create a brand around atomix - I love the name! Like "Intel inside" logo on Dell machines, "powered by atomix" on distributed reactive applications is a huge marketing advantage! If only one successful product is launched with "powered by atomix" on the back page of the product manual than the inferior competitors will do the copycat (pun intended), but they do not have the know-how upfront - so they will pay for it.)

PS: One side note to TensorFlow. In my opinion it is a downgraded version of Googles only-in-the-cloud solution of DataFlow. TF only supports 3 data types. Googles strategy is to implement the learning process of TensorFlow locally and do the heavy calculations in Google Cloud. I just wrote an article about TF and Microsofts DMTK on my blog http://posidev.com/blog/2015/11/14/googles-tensorflow-and-microsofts-dmtk-goes-open-source/

[Oliver Rolle]

Am Mittwoch, 25. November 2015 17:59:20 UTC+1 schrieb Magnus Koch

So we're quite interested in knowing more about the direction this is going in. What we really like about the current version is the separation between network configuration and component implementations. The way we're using it is to allow "tenants" to get independently built components to work together by configuration. For that the current network definition (json or class structure) and port/connection abstractions are almost perfect. The prototype in Oliver's repo seems quite different with more of a reactive stream focus, and at this point it's hard to see how that compares.

Yes the concept is pretty nice. The name of this concept is flow-based programming. The current vertigo implementation supports "flat" networks. The networks support only the same abstraction level. But to really exploit the concepts to its full potential vertigo has to support hierarchical networks (different abstraction levels). To give it a try I implemented an operator which is able to connect to or nest networks in networks based on conditions. Its like a SQL join which connects networks. As networks are functions, the join operation joins functions - wicked idea imo. As I implemented it in vertigo I got a single point of failure and performance bottleneck problem with vertigo in vertx. I ask Jordan if vertigo could support state recovery and scaling. After some black magic was applied to that problem copycat appeared.

A related problem to the state issue is how components interact with each other. As the component instances (& nested networks) are distributed in a cluster you might not be able to communicate with any network but only with components / nested networks sub-set defined by the join conditions. As these components might be state full, its important to communicate with these components (via its ports) not "randomly" but send messages to its specific ports in a specific order to work probably. To ensure that state full components are communicated the right way, I use a method to model a communication protocol for a component, which is based on finite state machine modelling (google "state machine compiler"). Now this stuff gets really crazy.

All component and network instances are is identified by a condition. A component state is secured via copycat. All data the component gets is received via ports and the component state and reaction is finite (state machine). Through all these preparations, I was able to train a component instead of implementing its functionality. I got a 3 in a row IA component which was able to beat random player in 2/3 of the cases. As the logic is within the components state I could reuse the trained component in other 3 in a row network instances (3 in a row plays).

I hope I could explain the issues and potential of vertigo and I wish I could convince you to work together on it.

[Suminda Dharmasena]

When designing please keep in mind:

1. Ultra low latency
2. Hard real time SLA
3. High throughput
4. Low overhead
5. Event sourcing and low over head state persistence for HA, recovery and testability

[Cosmic Interloper]

Hey Oliver / Jordan : 

  Is there a plan/design/strategy that that could be translated into issues, or outlined into interfaces that volunteer developers could implement, to help move the project forward?

--
You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/c3059373-5c25-4d33-9a40-7171bac6818c%40googlegroups.com.

[Oliver Rolle]

Hi Cosmic,

thanks for your voluntary work! I will provide a design & issues the coming days.

[Oliver Rolle]

The basic strategy to implement vertigo with vertx and copycat is discussed here: https://groups.google.com/forum/#!topic/copycat/_a5-OImFkrs

In short Jordans suggestion is the following:

1. Embed AtomixReplicas or clients on each node in the cluster

2. For each shard, create a DistributedLeaderElection resource called "shard.n" or whatever

3. When a node is elected leader, start listening for messages for the shard that was elected leader (this could be done with any external API like a Catalyst Transport, HTTP, Vert.x, Netty, or with DistributedMessageBus)

I need to take a deeper look at copycat and the DistributedLeaderElection to define clean interfaces.

High level concept
We need a vertx verticle which contains the at least 1 AtomixReplica, which then runs the leader election and if its a leader, then listens on the vertx event bus for incoming events for processing.
For intelligent message routing need some kind of logic, so that networks deliver to the right component instance. I have an DHT / DistributedTree implementation on top of the event bus which is able to do conditional broadcasting.
For vertigo components we need a copycat StateMachine implementation. I believe components can have a quiet complex behaviour, so maybe we need some kind of modelling tool. This tool might be even useful for modelling routing and vertigo management stuff in a robust way.

[Suminda Dharmasena]

My suggestion is to keep everything lean and mean for maximum throughput and lowest latency. Features which are there in the Axomic stack for completeness but may in some cases can done faster in Vertigo should be done in the most optimal way always considering throughput and latency implications.

--
You received this message because you are subscribed to a topic in the Google Groups "vertigo" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx-vertigo/858YGq8hS18/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx-vertig...@googlegroups.com.

To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/a19cf434-581b-4ed4-ae25-4d8b5537f56a%40googlegroups.com.

[jordan.h...@gmail.com]

This is why Atomix should only be used for safety critical things. As I mentioned in that thread, what Kafka does is use ZooKeeper for leader election and coordination, but it exposes a separate interface for handling actual writes. Atomix will ensure that all of the processes see the same view of the cluster at the same (logical) time. 

For example, if we follow the Kafka model for sharding streams, [1] each partition will elect a leader (or alternatively you could [2] elect a single leader and have it coordinate a membership group). 

In the case of leader elections, Atomix will guarantee that no two processes perceive themselves to be the leader at any given time. This is critical since if we were to follow the first model, if multiple nodes believe themselves to be the leader them multiple nodes will accept messages and those messages have to then either be discarded or resolved in some way, and that's a complex problem. 

If we were to follow the second model, both leader election and group membership consistency is at play. Atomix again guarantees that only one node will perceive itself to be the leader, and this is critical because in this case if two nodes perceived themselves to be the leader, then two nodes could potentially attempt to make changes to the way shards are balanced within the membership group. So, the strong consistency guarantees make the system much easier to reason about from a coordination perspective.

For most of the Atomix coordination utilities, the overhead of state changes is relatively significant, but the overhead of periods of inactivity is minimal, and that's the most common case. A leader will only be elected when another leader crashes or is partitioned. A membership group will only change when a node joins or leaves the group or crashes. So, the overhead of coordination is ultimately minimal but turns out to be a huge advantage.

Once you've integrated Atomix and have linearizability guarantees across the cluster, that greatly simplifies coordinating your own custom high-availability, fast replication algorithm. What Atomix does is allows a single process to control the entire cluster as if it were a single unit. Absent consensus algorithms, network partitions *will* result in "split brain," and that means you must either accept data loss or implement additional measures to resolve conflicts.

This is why Atomix is called a "cluster coordination framework."

By the way, there are also some other great uses for storing state in Atomix. In addition to electing leaders and managing group (cluster) membership, many systems also use CP data stores to store stream state. Kafka consumer offsets are periodically stored in ZooKeeper so the entire log doesn't have to be replayed when a consumer disconnects and reconnects. Kafka benefits from the consistency model of ZooKeeper here since all consumer processes can retrieve the most recent state of the stream when rebalanced, and that may not be true of eventually consistent stores.

But you still have to take care not to write too much to Atomix. Each write requires a majority of the Atomix core cluster to respond, and writes are usually flushed to disk. So, storing every acknowledged offset of a stream in Atomix would likely significantly slow down your system or overload Atomix, which doesn't currently have good back pressure mechanisms. Similarly, Kafka only stores every n number of offsets in ZooKeeper. Last time I tested, Atomix could handle tens of thousands of writes per second, but that can still be plenty insufficient at scale. Vert.x alone can destroy that speed on a single event bus connection. If Vertigo can be made to scale linearly, management of things like acknowledgements should be thought of carefully.

So, I guess that's all to say yes: use Atomix to guarantee that the system's state is what you want it to be. Use it for safety, and use something else for performance. In Vertigo, that would mean some mechanism that controls the cluster understands what networks and components are running at any given (logical) time, what resources are available to those networks, and how networks and components are distributed among them. Messaging should be done via other mechanisms but the flow of messages should be managed via Atomix. If done right, Atomix can allow you to write a single process that controls all this information for the cluster, and it will do a ton to ensure coordination continues to progress in the face of failures and there are no inconsistencies that may result in lost data or other bugs.

You received this message because you are subscribed to the Google Groups "vertigo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.

To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/CAAfFNYF7q9zPv8zB5rhUBnDC9UAK4rM99u8X5Aq34-J8GS-BAQ%40mail.gmail.com.

[Magnus Koch]

On Saturday, November 28, 2015 at 1:47:27 AM UTC+1, Oliver Rolle wrote:

The current vertigo implementation supports "flat" networks. The networks support only the same abstraction level. But to really exploit the concepts to its full potential vertigo has to support hierarchical networks (different abstraction levels). To give it a try I implemented an operator which is able to connect to or nest networks in networks based on conditions. Its like a SQL join which connects networks. As networks are functions, the join operation joins functions - wicked idea imo. As I implemented it in vertigo I got a single point of failure and performance bottleneck problem with vertigo in vertx. I ask Jordan if vertigo could support state recovery and scaling. After some black magic was applied to that problem copycat appeared.

A related problem to the state issue is how components interact with each other. As the component instances (& nested networks) are distributed in a cluster you might not be able to communicate with any network but only with components / nested networks sub-set defined by the join conditions. As these components might be state full, its important to communicate with these components (via its ports) not "randomly" but send messages to its specific ports in a specific order to work probably. To ensure that state full components are communicated the right way, I use a method to model a communication protocol for a component, which is based on finite state machine modelling (google "state machine compiler"). Now this stuff gets really crazy.

Not sure I follow here, please bear with me if I'm missing something. A network is a predefined set of linked components, which form a particular flow to handle a stream of events, right? It is not a function in a traditional sense - there is no return value and no useful output at the network level, because there is no "final" component - the networks can be branched to parallel flows inside the same network, or components can be recursively connected to form a loop. The closest you get to network level output would be wiring into a specific component's output port. By joining and nesting networks, do you mean that a component output port from one network is connected to a component input port in a different network (with conditions applied to that connection, and/or syntactic sugar to make it easier)?

[Suminda Dharmasena]

Sounds great. Just make sure you can maintain sub micro second latencies.

[Suminda Dharmasena]

Also perhaps you can look into using Graal (http://openjdk.java.net/projects/graal/) / Truffle (https://wiki.openjdk.java.net/display/Graal/Truffle+FAQ+and+Guidelines) to further optimise some of the functioality. This perhaps can be used in all atomix projects and also be a sub project on its own right.