Microservices etc...

[Arno Schulz]

Just as an FYI

There's been a lot of talk about microservices, and out of it I read about netkernel which sounds akin to what vertigo (+ ui) is trying to achieve (see the blog for some interesting posts as well)

Most notably some interesting elements of netkernel:

• A request visualizer to help debug the system
• Fine grained measurements (such as a heatmap like tool)
  

[Oliver Rolle]

Did you try out netkernel?

[Oliver Rolle]

Am Dienstag, 13. Mai 2014 15:06:08 UTC+2 schrieb Arno Schulz:

Most notably some interesting elements of netkernel:

• A request visualizer to help debug the system

Do you more info about the debug?

• Fine grained measurements (such as a heatmap like tool)
  

Heatmap is added to my features list :-)

[Arno Schulz]

For the debug they wanted to replicate what we're used to with java debuggers. All the messages are sent to a cache of sorts. When an error occurs you can use the caching mechanism to find out where it came from, and as everything is cached you can go back in time (a bit like JCAPS http://en.wikipedia.org/wiki/Java_Caps -- I used this at work a while, it's still being used as no real alternative exists as of yet).

I was thinking that this mechanism could be duplicated somehow by having the messages sent to a specific instance of elasticsearch (not elegant but functional) and perhaps figure out a way to associate the vertigo ui with the elasticsearch data (such as click on a node and see the last 10 messages -- and elasticsearch then provides the additional functonality to go back in time or to see the flow of a particular message -- one could even replay the message from a given node).

Anyhow just speculating right now I'm working my way up the docker entrails

--
You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/50546427-cb5d-425a-8196-078640056a7d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jordan Halterman]

Yeah that sounds really interesting!

All very good ideas. No doubt there is endless work that can be done both to Vertigo and the UI. Thankfully the priorities are pretty obvious.

[Oliver Rolle]

Am Dienstag, 13. Mai 2014 17:37:26 UTC+2 schrieb Arno Schulz:

For the debug they wanted to replicate what we're used to with java debuggers. All the messages are sent to a cache of sorts. When an error occurs you can use the caching mechanism to find out where it came from, and as everything is cached you can go back in time (a bit like JCAPS http://en.wikipedia.org/wiki/Java_Caps -- I used this at work a while, it's still being used as no real alternative exists as of yet).

I thought about netkernel. It seems they scale on cores of one cpu, but not on whole clusters. JCAPS: I like the idea of duplicating  a proven solution.

I was thinking that this mechanism could be duplicated somehow by having the messages sent to a specific instance of elasticsearch (not elegant but functional) and perhaps figure out a way to associate the vertigo ui with the elasticsearch data (such as click on a node and see the last 10 messages -- and elasticsearch then provides the additional functonality to go back in time or to see the flow of a particular message -- one could even replay the message from a given node).

This might be interesting for small (local) deployments but for large this is a problematic solution. First we can use an logging module which is deployed like a component but keeps a connection to the ui. For real debugging we might need something more intelligent (based on Hooks and levering the acking mechanism) to track down failures.
 

[Oliver Rolle]

Am Mittwoch, 14. Mai 2014 08:49:53 UTC+2 schrieb Jordan Halterman:

On May 13, 2014, at 8:25 AM, Oliver Rolle <oliver...@googlemail.com> wrote:

Am Dienstag, 13. Mai 2014 15:06:08 UTC+2 schrieb Arno Schulz:

Most notably some interesting elements of netkernel:

• A request visualizer to help debug the system

Do you more info about the debug?

• Fine grained measurements (such as a heatmap like tool)
  

Heatmap is added to my features list :-)

Yeah that sounds really interesting!

All very good ideas. No doubt there is endless work that can be done both to Vertigo and the UI. Thankfully the priorities are pretty obvious.

I agree, first things first!
 

[Arno Schulz]

Is there an ideas page in the wiki where we could keep these when someone brings them up?

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/784a368a-3751-4bac-bc8d-240a86ec9dfd%40googlegroups.com.

[Jordan Halterman]

Good idea.

I proposed the first idea on the ideas page be a page for ideas!

I'm workin on docs a lot lately. I'll be moving em back to the wiki and reorganizing it so this is definitely on the list. Would be good to have some sort of road map.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/CA%2BJY%2B9bHqkw5202iyKXiB4YdDX7jx_vbAJzmaDWFHNL5frLj_g%40mail.gmail.com.

[Arno Schulz]

Road map the word I was looking for the whole day.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/6FD9758C-CAD3-4761-8A64-CF71B3C4D0BA%40gmail.com.

[Arno Schulz]

PS whatever solution to get the messages coming through a system eelasticsearch is really proving itself to be a backend.

If you check out http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html stackoverflow uses two eelasticsearch instances to power their search tags, etc...

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/5e2ee948-08e5-44bc-8846-337666f5e4fd%40googlegroups.com.

[Oliver Rolle]

oh yeah, good idea!

Am Donnerstag, 15. Mai 2014 01:53:20 UTC+2 schrieb Arno Schulz:

Road map the word I was looking for the whole day.

On Wednesday, May 14, 2014, Jordan Halterman <jordan.h...@gmail.com> wrote:

Good idea.

I proposed the first idea on the ideas page be a page for ideas!

I'm workin on docs a lot lately. I'll be moving em back to the wiki and reorganizing it so this is definitely on the list. Would be good to have some sort of road map.

On May 14, 2014, at 6:10 AM, Arno Schulz  wrote:

Is there an ideas page in the wiki where we could keep these when someone brings them up?

On Wed, May 14, 2014 at 9:09 AM, Oliver Rolle  wrote:

Am Mittwoch, 14. Mai 2014 08:49:53 UTC+2 schrieb Jordan Halterman:

On May 13, 2014, at 8:25 AM, Oliver Rolle  wrote:

Am Dienstag, 13. Mai 2014 15:06:08 UTC+2 schrieb Arno Schulz:

Most notably some interesting elements of netkernel:

• A request visualizer to help debug the system

Do you more info about the debug?

• Fine grained measurements (such as a heatmap like tool)
  

Heatmap is added to my features list :-)

Yeah that sounds really interesting!

All very good ideas. No doubt there is endless work that can be done both to Vertigo and the UI. Thankfully the priorities are pretty obvious.

I agree, first things first!
 

--
You received this message because you are subscribed to the Google Groups "vertigo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.

Visit this group at http://groups.google.com/group/vertx-vertigo.
To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/50546427-cb5d-425a-8196-078640056a7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vertigo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertigo+unsubscribe@googlegroups.com.

To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/784a368a-3751-4bac-bc8d-240a86ec9dfd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vertigo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertigo+unsubscribe@googlegroups.com.

To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.
To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/CA%2BJY%2B9bHqkw5202iyKXiB4YdDX7jx_vbAJzmaDWFHNL5frLj_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vertigo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertigo+unsubscribe@googlegroups.com.

[Oliver Rolle]

Am Donnerstag, 15. Mai 2014 03:36:02 UTC+2 schrieb Arno Schulz:

PS whatever solution to get the messages coming through a system eelasticsearch is really proving itself to be a backend.

If you check out http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html stackoverflow uses two eelasticsearch instances to power their search tags, etc...

Its interesting! But this makes distribution of the ui hard. Much to install and configure before it is usable.

I would like to experiment with a graph database because vertigos messaging is based on a graph.

 

[Arno Schulz]

I get what you're thinking, and I'll do my best to explain my (planned) setup.

Once I start deploying vertigo networks each network will be deployed in their own docker container, the nodes will dump their logs straight into one log file (per network, across multiple networs -- I'm thinking using log4j 2.0 + disruptor to do this asynchronously see http://www.javacodegeeks.com/2013/07/log4j-2-performance-close-to-insane.html)

What is special is where the logs are written (and this gets tricky as logging is one of the remaining tricky topics with docker) instead of being written in a volume of their own container it will be written to a common container (1 log container per host) on which I'm going to run logstash-forwarder (see https://denibertovic.com/post/docker-and-logstash-smarter-log-management-for-your-containers/) to send the results to elasticsearch.

The idea is not to overburden the vertigo network to do any kind of fancy logging rather dump the logs with metadata that allows to trace back a log message to a vertigo node at a point in time for a specific message. The front-end then has to do a bit of magic, but again elasticsearch kills on that.

In this manner you completely decouple your logs.

I would do the same for any kind of metrics just dump it into a logfile and let elasticsearch deal with sorting and filtering

PS: I also realize this contradicts something I mentioned earlier about having a node that collects all the error messages as per no-flo.js example. But I think dumping everything into a single file (errors, metrics, etc...) makes the rest easier to digest for a centralized logger.

What do you guys think?

--

You received this message because you are subscribed to the Google Groups "vertigo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx-vertig...@googlegroups.com.
To post to this group, send email to vertx-...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx-vertigo.

To view this discussion on the web visit https://groups.google.com/d/msgid/vertx-vertigo/06bb3c7e-875e-4bc3-81ae-0a0b4e3a0258%40googlegroups.com.

[Tom Mueck]

NetKernel scales out and up. About performance tests on two 8-core servers see http://1060research.com/netkernel/scaling/; FYI other topics are covered in this evernote https://www.evernote.com/l/AQYmfUrg1ANA0KZSNxMvdZ3t0_t6pwwu7Ac or simply check out PragPub Magazine Dec 2014 issue.