Architectural Changes and Road Map for Vertigo 1.0

[Jordan Halterman]

Hey community. For many months, we’ve had a plan to target the development and release of Vertigo 1.0 to coincide with Vert.x 3. As the Vert.x 3 API seems to be nearing stability, this post is intended to kick off the development of Vertigo 1.0 with a description of the planned changes in the upcoming version. I strongly encourage users to contribute their experiences and requests to this discussion as well as challenge any of the proposed plans to follow.

First, there are many components that contributed to the decision to develop Vertigo 1.0 on Vert.x 3, and here are a few of them:

• Vert.x 3 supports automatic language API generation for core and contributed libraries. Previously, we’ve had to maintain several implementations of the Vertigo API for various languages. But it’s difficult to keep multiple mirrored APIs up-to-date, and supporting all the Vert.x supported languages requires contributors who know each of those languages. Vert.x 3 will allow Vertigo to support all Vert.x languages by simply generating language specific APIs from the core Java API.
  
• Vert.x 3 supports event bus proxies which significantly simplify interacting with remote APIs over the event bus. Currently, Vertigo provides event bus wrappers to model remote APIs. But with Vert.x 3, much of that responsibility can be delegated back to Vert.x core.
  
• Similarly, Vert.x 3 provides cluster-wide shared data structures. Again, Vertigo currently provides these types of data structures in an event bus wrapper API, but much of that code can be delegated to Vert.x core with Vert.x 3.
  

Now on to the proposed changes.

In the last major version, we focused on refactoring and solidifying the messaging model to follow, to some extent, the FBP model. I think we are happy with these changes, and there are certainly no intentions to change the model again, so it’s likely that much of the public APIs will be directly portable to Vert.x 3.

One component that I’ve long felt was lacking in Vertigo is support for fault-tolerant stageful components. Many users (including me) model Vertigo components as state machines. However, doing so implies a risk that if a node dies and a component fails, that state will be lost. Currently, a user would have to take steps to persist component state to ensure that state in failed components is recoverable. However, since state is obviously such an essential component of software and distributed systems, a primary focus of Vertigo 1.0 will be to provide facilities for managing state.

To that end, I have been working on state management and replication algorithms for quite a long time (about a year), and I think I have some ideas about how stageful components in Vertigo can work.

The concept is that Vertigo 1.0 will model components as finite state machines. Users will be afforded some control over how state is managed and replicated. The way this works is:

• When a message is received on an input port, the message is immediately logged to an in-memory or file-based append-only log (this is configurable, and the current no-log configuration will certainly be supported). File-based logging can be done using a memory-mapped log that I’ve benchmarked at rates that far outpace the Vert.x event bus, so I don’t anticipate logging being a significant bottleneck.
  
• Messages will be logged with additional metadata such as on which port a message was received. This allows messages to potentially be replayed if a node dies, but more on that later.
  
• If replication is enabled (also configurable), the message is synchronously or asynchronously (also configurable) replicated according to the replication factor for the component (also configurable). The replication algorithm used is similar to the algorithm implemented for Kafka 0.8 with some key differences. As with Kafka, each component (equivalent to Kafka’s topics) will elect a leader for the component set. This leader will contain the active Vertigo component to which messages are sent. When the leader receives a message, it will log and replicate the message to followers if necessary. However, in contrast to Kafka which maintains an “in-sync” replica (ISR) set and requires synchronous replication to all ISRs, Vertigo’s implementation will only require synchronous replication to the fastest quorum of nodes in the replica set. And as with Kafka, users can control whether replicas log events synchronously or asynchronously.
  
• Once the message has been applied to the replicated (and very configurable) log, the appropriate input handler for the port is called as usual.
  
• The input handler potentially alters the state machine’s state and produces one or more output messages.
  
• As is currently the case, output ports maintain communication with remote input ports to ensure that messages are delivered reliably and in order. Each output connection receives acks from remote input connections indicating whether messages have been successfully received and logged. Vertigo currently does this by using a local counter within each output connection. Each time a message is sent on the output connection, the counter is incremented and sent with the message. If the remote input connection receives a message out of order, it immediately responds with the index of the last message it received in order, allowing the source output connection to begin resending messages from that point. Additionally, the remote input connection periodically sends an ack (for every 1000 messages, for instance) indicating the last message received. This is a simplistic description of the current algorithm, but my benchmarks indicate that it’s about twice as fast as the Storm-like algorithm that was originally implemented in Vertigo.
  
• Vertigo will periodically take a snapshot of the component’s state and log the snapshot to the component’s log. Then Vertigo will remove old events from the component’s log up to the last event seen by all components connected via output ports. This allows Vertigo to rebuild component state and output after failures.
  
• When a component fails, if a replica is alive, Vertigo will automatically fail over to the node on which the replica resides. When failover occurs, Vertigo will create a new instance of the component, apply the last snapshot, and replay all events in the log.
  
• Since each component keeps track of which output events have been processed by remote components, Vertigo can replay events to rebuild component state while ignoring deterministic output from those components.
  
• Note that this model implies that components must necessarily be deterministic, meaning mutable state from external sources cannot be used to contribute to component state or output.
  

This statefulness aspect is my first task on the Vertigo 1.0 new features todo list. However, there are still a lot of additional tasks on the todo list and issues to figure out. Here’s a brief, incomplete task list:

• Port the existing network, component, and communication interfaces to Vert.x 3.
  
• Refactor core interfaces to use Java 8 features wherever necessary such as default/static interface methods.
  
• Create a cluster manager verticle for Vertigo 1.0. First, the existing cluster-wide data structures need to be removed from the existing cluster manager. Second, new event logging/replication features will need to be integrated into the cluster manager.
  
• Implement component message logging.
  
• Implement component message replication.
  
• Determine how to package and distribute components remotely. This is perhaps one of the most important/difficult aspects of development. Vert.x 3 does not have a module system, so Vertigo will likely need to use Vert.x 3’s services or a similar pluggable framework to allow users to provide component metadata and resources which can be distributed across a cluster. Perhaps the network API should just support generic resources being added to the cluster, e.g. addResource(“some.jar”) or addResource(“some.zip”)
  

This is most certainly an incomplete list, so feel free to contribute to it. I’m really excited about getting started on Vertigo 1.0; I think it will be a huge step forward if we can do it right.

I’ve already implemented the replication algorithm, it’s now just a matter of integrating it with the Vertigo messaging model and building APIs for stateful components. I anticipate it will be a few weeks before we can even get a realistic idea of how much work it will take to get to a release. There are simply too many unknowns right now, but at the same time there is a lot we’ve already figured out in earlier versions.

Like I said, feel free to contribute your own experiences and requests!

[Jordan Halterman]

By the way, what I'm going to do first is design the project using interfaces. I'm wiping all implementations from Vertigo to design the interface. I'll commit and push the interfaces and then start working on the implementations.

[Oliver Rolle]

Wow, this was fast!
Cool state management!

Can a network be composed with managed stateful and non stateful components? Do you think non deterministic stateless components are possible? Like a rest api

Here is 2am. I will continue tomorrow.

[Jordan Halterman]

These are the types of things we'll have to think about. I think it can be done, but perhaps with some limitations and a clear understanding and explanation of the side effects. It's possible that it could require a limitation on the structure of networks that use stateful, deterministic components with stateless non-deterministic components. Or there could be no enforced limitations, but rather a clear explanation of the potential side effects of such a configuration.

Of course, in practice, it just depends on the use case. Problems would only arise if a stateful, deterministic component were feeding a non-stateful, non-deterministic component and the stateful, deterministic component failed, restarted, and Vertigo resends messages from that component. But even in this case, since in the model as I described it components keep track of which messages they've processed, the non-deterministic component should ignore messages replayed by the deterministic component. So, the only limitation would be that non-deterministic components cannot (or should not) support replay of events. So, deterministic, stateful components use log replay to rebuild state after failures, and stateless components are simply responsible for managing their own state and recovery.

I would certainly like to provide as much flexibility in configuration as possible as long as there are no hidden risks to doing things like this. I would definitely prefer that network configurations remain as flexible as possible, supporting variable logging, replication, and statefulness configurations on a per-component basis.

> --
> You received this message because you are subscribed to the Google Groups "vertigo" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
> To post to this group, send email to vertx-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/vertx-vertigo.
> To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/692db7f8-a67f-45ad-bcea-51d53541e67c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jordan Halterman]

So, I've begun implementing interfaces for Vertigo 1.0. I'll probably push the initial API in the next couple days. Like I said, most of the messaging API will remain the same until we decide otherwise. I don't see any issue with it now, and I don't think anyone else does either. It will fit the stateful model just fine. Most of the interface changes are simply intended to mirror the style changes in Vert.x 3 and support code generation. Also, some additional interfaces are being added for logging, replication, and component management. I've removed the active network interfaces for now, but I do think that runtime network configuration changes can still be supported even with stateful components, it just requires careful coordination. Live configuration changes are a requirement that I've had in some of my own use cases, so I'd like them to remain a part of the API if possible. I just don't want configuration changes to limit the development of statefulness features for the time being.

I think tomorrow I'll work a bit on formalizing the planned logging/replication/state management model and perhaps begin developing a proof. This will include descriptions of the log, the leader election algorithm, the replication algorithm, checkpointing, log compaction, and log replay/recovery. I'll post any documents here as the algorithm is codified and give people an opportunity to challenge the concepts and proof. The algorithm, though, is largely based on a mixture of two well tested algorithms for leader election and fast, consistent replication, and I've already implemented a proof of concept, so I'm confident it will be successful.

[Oliver Rolle]

I thought about the failure recovery... it is really difficult! We should not over engineer because its very depended on the use case.
At first, we provide failure-recovery only for stateful deterministic components and manual recovery for the rest. Later we might add other recovery when we see a demand.

Am Mittwoch, 29. Oktober 2014 02:51:43 UTC+1 schrieb Jordan Halterman:

I would certainly like to provide as much flexibility in configuration as possible as long as there are no hidden risks to doing things like this. I would definitely prefer that network configurations remain as flexible as possible, supporting variable logging, replication, and statefulness configurations on a per-component basis.

I agree that everything should be able to be configured in the network. I think we should have a practical default config for the network so that it is easy to create a network that works (convention over configuration).
Should statefulness be defined by the component itself? Maybe the network config can overwrite the statefulness of the component?
 

[Oliver Rolle]

If you have anything just post it / email me, I will look over it and / or help you in the development.

[Jordan Halterman]

Agree. I think there are very different needs/requirements for different types of components, so I suspect a lot of discussion and refactoring will need to be done to get it right. It's true that these types of behaviors (failure recovery) could definitely cause unexpected side effects depending on how components are implemented, and we certainly don't want to limit the use cases for Vertigo. I think it's just important that there be some transparency without too much complexity. I think we'll get more clarity once things get started.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/7217799f-6236-482f-bd14-c156b147d918%40googlegroups.com.

[Jordan Halterman]

By the way, while I'm still documenting the state stuff, I think these are the steps to take in terms of development:

1. Remove all implementations and port existing interfaces to Vert.x 3. This involves adding codegen annotations and updating some APIs to adhere to some new Vert.x standards.

2. Remove all cluster management utilities (the net.kuujo.vertigo.cluster package). I think cluster management is something that can be handled near the end of the development cycle. It's an area that requires a better understanding of how clustering will be implemented in Vert.x 3 and, more specifically, how resources can be distributed across a cluster. I think trying to tackle cluster management and state management at the same time will simply slow development.

3. Refactor the configuration API for Vert.x 3. Vert.x 3 uses options classes rather than many overloaded methods, so I'll probably update the configuration APIs to follow that convention.

4. Perform whatever refactoring is necessary to support deployment of networks within a single Vert.x instance.

5. Rebuild the messaging framework with optional logging. This will be a really simple implementation of event logging that will serve as a prototype intended to prove that logging is viable in terms of performance. No state management features will be implemented at this point.

6. Implement state management on top of event logging in the messaging framework. This should support periodic checkpointing and log compaction. The goal here is to prove that component state can be reliably managed via the log. Deterministic components should be able to be stopped and restarted without losing state by replaying logs. Obviously, this may require some discussion of how to handle the distinction between deterministic and non-deterministic components.

7. Implement leader election and log replication on top of the existing event log. This stage will require the most refactoring and testing, but I already know the algorithm works. This will involve adding replication configuration options, adding state to component instances (follower, leader, etc), and altering the behavior of the messaging framework according to instance state. It could be wise to release an alpha version with the preceding features prior to starting development on fault tolerance. That way, if it proves to be more difficult or less stable than expected, perhaps fault tolerance features can be delayed until a later release. I think there will need to be a lot of discussion regarding how failures are handled at this point, as you mentioned.

8. Rebuild the cluster manager to properly distribute resources, deploy networks, and support replication and fault tolerance features.

9. Ensure Vertigo bindings can be generated for all Vert.x supported languages (I have a feeling this may require some contribution to the codegen project which is currently largely oriented to generating Vert.x core APIs only)

I think this order of operations will help reduce the risk involved in this undertaking. I spent a lot of time in YourKit working on performance in Vertigo 0.7, so the goal here is to ensure that performance is not lost to over engineering.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/448a09e9-1f03-4af5-8f42-e3615b5b3e6c%40googlegroups.com.

[Jordan Halterman]

I'll post it soon. It's probably gonna take a couple of days. There is a lot of detail including how components and messaging are controlled, how leaders are elected, how messages are logged and replicated, how messages are ordered and tracked between connections, how checkpoints are persisted, how component state is recovered from checkpoints, how connection state is recovered, how logs are replayed while still guaranteeing exactly-once processing even on connected components, etc. It's difficult to organize all this in a logical and easy to understand manner. But I definitely look forward to sharing it soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/448a09e9-1f03-4af5-8f42-e3615b5b3e6c%40googlegroups.com.

[Jordan Halterman]

Actually, I should put towards the beginning of that: port all implementations to Vert.x 3. I'll get Vertigo 1.0 running on Vert.x 3 this weekend and then begin refactoring from there. It shouldn't be too difficult to update all the event bus and deployment code.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/7217799f-6236-482f-bd14-c156b147d918%40googlegroups.com.

[Jordan Halterman]

Alright, so while I'm still working on the formalized proposal for state management, I've ported the current messaging system to Vert.x 3 and pushed the 1.0 branch:

https://github.com/kuujo/vertigo/tree/1.0

This is a "bare bones" implementation of all basic interfaces and the messaging system. I removed hooks, configuration changes, and other features that sit on top of the messaging system for the time being. This simply implements the standard port API with counter-based ordering/batching of messages/acks. But it doesn't run right now. I need to finish updating the immutable configurations before it will work.

I think this weekend I should be able to make good progress on getting Vertigo 1.0 in a working state. That means being able to create and deploy networks within a single Vert.x instance. Once that's done I'll move on to the next stages - baseline performance testing, adding logging, comparative performance testing, adding state, comparative performance and state testing, adding a component controller, adding leader election and log replication, and finally more performance testing. Then I'll add back in the important existing features of the current Vertigo version like hooks, cluster management, configuration changes, etc.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/448a09e9-1f03-4af5-8f42-e3615b5b3e6c%40googlegroups.com.

[Oliver Rolle]

if you have vertigo in a working state in the repo, can you post a short how-to to get it to work?

I like to get a feeling how to use vertx3 + vertigo 1.0, try some sbpm model to source code generation (with yoke integration) and hands on vertigos messaging with the intention to implement ScopeSelector / instantiation of nested networks.

[Oliver Rolle]

we might consider to write a scientific paper about the message acking and component state management?

[Jordan Halterman]

Alright,

So what I have done thus far is:

- Removed clustering features

- Removed runtime network configuration changes

- Removed other high level features like hooks

- Rebuilt the metadata interfaces and implementations (used to be *Context, currently is *Info, e.g. ComponentInfo, InstanceInfo, ConnectionInfo, etc - naming is definitely flexible)

- Added codegen annotations to all APIs

- Rebuilt the messaging system using the basic acking algorithm already implemented in Vertigo 0.7

What I have to do to get it in a workable state is:

- Replace the deployment system to compile network metadata and deploy components

For now, networks will be presumably static, but I intend to make them dynamic once the messaging stuff is complete. Thus far, the only significant changes to the API have been for adhering to Vert.x 3 standards (such as using interfaces and options classes for code generation). I think pretty soon here we'll have to discuss how to package and deploy components with dependencies. Fortunately, Athe new VerticleFactory API is really flexible, and Tim implemented a VerticleFactory (service) that is very similar to modules. My tentative plan is to provide a similar SPI (call it ComponentFactory) which allows for pluggable component/cluster management, but I'm not too focused on that quite yet.

I think I'll keep moving forward on getting it in a plain but working state so we can start figuring out things from there. I'll keep this updated as I get things working.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/e549aa65-0836-442a-8ae6-188c80045dfd%40googlegroups.com.

[Jordan Halterman]

In addition to working on getting the basic framework for Vertigo 1.0 working, I'm also working on a prototype of the state management framework. There are a lot of issues to consider such as how deterministic components can talk to non-deterministic components (basically, how to manage idempotency between them). Hopefully, the behavior of non-deterministic components will remain the same as it is currently, but stateful, deterministic components can be provided with additional tools for managing state, and Vertigo will ensure that components whose state is managed by the framework will not cause unexpected behavior in vanilla components and vice versa. Also, looking at other things like how to manage checkpoints and replay. I've been taking notes that I'll share here pretty soon. I'm going to try to get through the prototype tonight.

I would definitely like to turn this into a paper once I've compiled the notes. I think it could be very important in getting input on the algorithm to ensure that it is correct. My suggested process would be something like:

- Write and document the prototype

- Discuss and debate the correctness

- Revise etc

- Complete and document the implementation

- Write extensive paper proving correctness (much of the state management is taken from existing, proven algorithms, so I don't think this should be an issue)

I will definitely try to get a working version of Vertigo in Vert.x 3, a prototype of the state management, and notes on the algorithm out ASAP.

Sent from my iPhone

On Nov 2, 2014, at 7:53 AM, Oliver Rolle <oliver...@googlemail.com> wrote:

we might consider to write a scientific paper about the message acking and component state management?

--

You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/1a5e79de-29d2-466c-9388-e307178e8fb9%40googlegroups.com.

[Jordan Halterman]

Change that. Looks like Tim added a Context type for Vert.x, and it's similar to Vertigo's contexts, so I think I'll revert that API to *Context

To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertigo+unsubscribe@googlegroups.com.

[Jordan Halterman]

I've written a lot of code on the 1.0 branch this week, but I haven't pushed it yet. I'll push it soon, but I thought I'd share an update and some thoughts on what I've run in to.

One of the great things about the module system for Vertigo was the ability to get static metadata about a component and locate and zip dependencies for distribution across the cluster. Vert.x 3 does away with the module system, but it also makes it really easy to replace it with a custom system. To that end, I added a system similar to the module system which allows components to be defined via a {component}.json, {component}.properties or {component}.conf file. The system uses Typesafe's excellent configuration library to support JSON, properties, and a custom JSON-like syntax.

https://github.com/typesafehub/config

The Typesafe configuration syntax supports some really interesting features like includes of other *.conf files. I think we could do some really interesting things with this.

The "component" verticle type locates the component configuration which defines standard verticle settings (main, worker, multi-threaded, etc) and static information about ports and various optional strategies (for instance, state related options). For now, the idea is to use this like the module system is used in Vert.x 2: Assuming all dependencies are contained within a component directory, Vertigo can zip and distribute the module to other modes in the cluster. Of course, I'm wide open to other ideas for distributing code to other nodes. Note, though, that it has to be language agnostic which is why the module system has worked so well for this purpose.

One interesting feature which I'm in the process of adding right now is component references. The idea here is that once a network is deployed, the user can get a ComponentReference pointing to a component in the network, and using that reference send messages to an input port or subscribe to messages on an output port. This makes things like DRPC as simple as deploying a network and then sending a message to it, something which had to be accommodated by the event bus previously.

Another idea I've been playing with is the idea of adding a DSL for building networks. The question is, what should the DSL look like?

More on the state management portion soon...

On Nov 2, 2014, at 7:53 AM, Oliver Rolle <oliver...@googlemail.com> wrote:

we might consider to write a scientific paper about the message acking and component state management?

--

You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/1a5e79de-29d2-466c-9388-e307178e8fb9%40googlegroups.com.

[Oliver Rolle]

I will write a longer answer Saturday or Sunday. Here my short answer

DSL: I think about this issue a longer time Maybe FBP for network definition. see http://noflojs.org/documentation/fbp/

[Jordan Halterman]

Alright, so I just pushed what I've been working on for 1.0. Right now it's a major mess and nothing is well documented, but I'd like people to take a look at what's going on while it's going on.

https://github.com/kuujo/vertigo/tree/1.0

Most of the work right now is centered on good design principals. So, you'll see a lot of design patterns all over the place. I'm trying to work towards separating the project into many modules. So far you'll see three modules - core, cluster, and config. The core module has all the core configuration, deployment, and messaging code. The cluster module will have the remote cluster manager implementation. The config module has the Typesafe Config based configuration support (core only supports code based configuration, but the config module adds the ability to define networks and components in configuration files). I'm not sure this will remain separated this way. Everything is purely experimental at this stage.

Especially take a look at the configuration/builder APIs. I'm still wiring all this together simply because I've been focusing so much on the underlying design to start even thinking about working on messaging and deployment (though the messaging system is a direct port of the Vertigo 0.7 messaging system right now).

[Jordan Halterman]

And some more work. Like I said, up until now most of the work has been in interfaces and design patterns. Tonight I started adding more implementations. Some of this involves porting Vertigo 0.7 code, but most of this amounts to a complete rewriting of Vertigo. I'm really exciting to see where it's going, though.

I was also able to completely separate out clustering from Vertigo core. So, presumably clustering will be an optional add-on to Vertigo code which can be used by loading the vertigo-cluster project. I separated the cluster module by simply putting a pluggable Vertigo network deployment interface in Vertigo core which by default uses the local Vertx.deployVerticle method.

I think I'm getting really close to being able to run basic networks again, which is exciting, but I doubt it will happen before the weekend. Then on to the more interesting things.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/c7581c62-d474-47eb-a8d9-1e86f38161fa%40googlegroups.com.

[Jordan Halterman]

So, I'm considering removing "instances" from Vertigo all together, and I want to see what people think about this. The reason I want to do this is because I think it gives more visibility into how Vertigo actually handles components (it does not actually deploy multiple instances using the Vert.x instances option) and I think it's just simpler to understand. We could just provide utilities for accomplishing the same types of multi-instance components and routing in a higher level API.

What do you think? We could also add a separate set of components like a "router" to accomplish some of the same types of routing behaviors as before.

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/76a5c704-bb49-4c02-ae53-7a254d545440%40googlegroups.com.

[Jordan Halterman]

By the way, I think I should be able to implement the entire state replication algorithm using a normal component. So, what I'll do is write a StatefulComponent decorator that does logging, snapshotting, and replication via normal ports, and it's up to the user to configure the network for replication. I think it will be an interested test of the framework and a great demonstration of what can be accomplished with it.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/76a5c704-bb49-4c02-ae53-7a254d545440%40googlegroups.com.

[Oliver Rolle]

Hi Jordan,

sorry that I did not answer earlier. I got ill over the weekend and still cannot concentrate clearly. I cannot say when I can work / think productively on vertigo (and related stuff).

I like the idea of removing instances. Maybe it can be replaced by some load balancing which leverages the failure recovery for state replication? I think we should be very concentrated which features should be in 1.0 and which should come later, which should be in core or as extension?

[Jordan Halterman]

It's all good! Feel better!

It seems like the trend is and has been towards stripping stuff down and making core low level, which is fine with me. I think that could mean that 1.0 can have a much earlier alpha/beta release than the other newer features. That is really preferable to me as it would give me a lot more time to make sure the messaging and configuration stuff is stable, code generation works, and I can do a lot of profiling to get the best performance out of it before it's released.

I think basically what I want to do is remove instances and continue rewriting the messaging deployment systems. Vertigo core will basically just become just the local version of what Vertigo is now. Ideally, the core library will just be developed with extensions in mind. That is, it will not provide these advanced features natively but it will make it possible to support clustering, state management, messaging strategies, etc on top of the core API. I think all of this can be accomplished just by good design and by using the existing messaging model (state management, load balancing, and other interesting features can be done by adding on top of components, and a pluggable deployment system which I've already created can be used to allow users to switch out local deployment for clustered deployment).

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/213a5ce8-3b5b-498b-9d1c-84d802862375%40googlegroups.com.

[Jordan Halterman]

Still chugging along. Trying to figure out the API, but I think I'm pretty happy with the direction. A lot of the configuration API is now extensible, so modules can add new ways to configure networks, components, and ports (via JSON files, for instance). Similarly, the deployment system is pluggable, so all the clustered deployment stuff can be moved to a separate module. I haven't yet figured out how to make the messaging system extensible, though, simply because in that case extensibility means adding API methods and the Vert.x code generation is sensitive to how code is written and extended.

I'll try to document some of this stuff this weekend so people can look and critique.

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/213a5ce8-3b5b-498b-9d1c-84d802862375%40googlegroups.com.

[Suminda Dharmasena]

Developments in Akka Streams maybe of some interest. (http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html?_ga=1.268769137.1234628154.1418042628) Especilly having a flow graph and partial from graphs which can b combined.

[Oliver Rolle]

Hi Suminda,

I not surprised of this development. Akka is used in a couple of Big Data projects (e.g. https://spark.apache.org/, https://flink.apache.org/) which uses directed acyclic graphs for data processing implemented on top of akka. The reactive steams (RS) consortium pushes reactive streams on akka (which reinvents a similar concept of the 1970s flow based programming (FBP)) so that they standardize the stream processing API for these big data projects - less duplicate code basis, easier for upcoming big data projects to use. In comparison to FBP RS has some advantages in a distributed execution env (back pressure on streams, failure handling) but lacks the refined logic which is better in FBP (named ports, messages in groups, component-based).

[Jordan Halterman]

Vert.x 3 is also adopting some support for reactive streams:

https://github.com/vert-x3/vertx-reactive-streams

It's certainly an interesting idea, but in my view it's more about promoting interoperability between technologies like Akka and Vert.x than standardizing stream processing in general. It will be really interesting to see how it's received and what sort of benefits can be derived from reactive streams.

--

You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/8fc691f4-aed3-4246-98d4-6ec2b450df88%40googlegroups.com.

[Oliver Rolle]

It's certainly an interesting idea, but in my view it's more about promoting interoperability between technologies like Akka and Vert.x than standardizing stream processing in general. It will be really interesting to see how it's received and what sort of benefits can be derived from reactive streams.

you are probably right!

[Oliver Rolle]

2 minutes ago I stumbled over nifi. Nifi open sources the NSAs scalable dataflow system.
Maybe we can get some inspiration for future features and how we relate to nifi?

http://nifi.incubator.apache.org/index.html

http://wiki.apache.org/incubator/NiFiProposal

[Jordan Halterman]

Wow! That's awesome! I'm definitely going to dig through that tonight!

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/fe75086a-f0e8-4268-971f-54cebde9dbed%40googlegroups.com.

[Oliver Rolle]

Googles dataflow runs on top of apache spark https://github.com/cloudera/spark-dataflow

Interesting emerging technologies :)

[Jordan Halterman]

Wow! That's an interesting idea! I've actually been writing Spark jobs the past couple of days at work. That would be another interesting bit of code to dig through. I'm curious about how they integrate the two.

On Jan 20, 2015, at 8:50 AM, Oliver Rolle <oliver...@googlemail.com> wrote:

Googles dataflow runs on top of apache spark https://github.com/cloudera/spark-dataflow

Interesting emerging technologies :)

--
You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/71e91ae6-fbb7-4e9c-82f6-8f30bf4fb238%40googlegroups.com.

[Oliver Rolle]

Is pretty simple imo. Spark streams data in buckets / mini batches through a directed acyclic graph. As DAGs allow generalized operators implementing wrapper classes to accomplish interoperability with dataflow should be straight forward.

--
Diese Nachricht wurde von meinem Mobiltelefon mit Kaiten Mail gesendet.

[Jordan Halterman]

Indeed it is.

Sent from my iPhone

> To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/99cf9ffb-47b5-401e-b257-6fb42e332cc5%40email.android.com.

[Suminda Dharmasena]

When will Vertigo start moving forward? Hardly any commits lately.

[Jordan Halterman]

It is always moving forward! But I know what you mean. 

Actually, a lot of the underlying infrastructure for new Vertigo features (fault tolerance, state machines, cluster management) are being developed separately here:

http://github.com/kuujo/copycat

Copycat is my implementation of the Raft consensus algorithm. But it's more than just a Raft implementation; Copycat has a pluggable protocol (do it can be used with the Vert.x event bus) and supports a variety of cluster coordination use cases built on a Raft replicated log. More importantly for Vertigo's performance, it supports partitioned resources and is designed to be embedded in other frameworks. Vertigo will use it for cluster management, replicated state machines, persistent messaging, and some other use cases. Of course, all of this will be optional/configurable, and not much will change in terms of Vertigo's architecture. These are just features to prevent data loss in Vertigo streams (via persistent messaging), support fault tolerant stateful components (via replicated state machines), and prevent malformed Vertigo networks and duplicate deployments due to split brain (the existing Vertigo release does not handle network partitions well). But I digress (that's my new favorite word!)

As you can see, we have been putting a ton of work into Copycat (I've been working on it for over a year now). It is finally yfeature complete and nearing beta-level stability (I'm running performance tests tonight). I started work on porting Vertigo to Vert.x 3 some time back and adding the new features (request-reply ports, state machines, persistent messaging). I anticipate I'll continue the work on the Vertigo side towards the end of this week. Fortunately, porting Vertigo should be significantly less challenging than implementing Raft and gossip protocols and architecting Copycat's resource management. The two primary challenges are rewriting the messaging layer with these new features and managing networks and their components.

On Jan 24, 2015, at 9:29 AM, Suminda Dharmasena <sirina...@gmail.com> wrote:

When will Vertigo start moving forward? Hardly any commits lately.

--

You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/df5ae6c5-a3d0-444a-aced-17ddd66e40e6%40googlegroups.com.