How to correlate log events in distributed Vertx system

[James Adams]

Hello Vertx Google Group,

I would like to aggregate log events from our distributed Vertx application and be able to correlate as much logging output as possible based on an eventId we generate when an initial event (HTTP request, etc) enters the system and send along with every EventBus message as metadata.

The main problem I am finding is that I need to pass around the eventId, or a logger instance created from it to every part of my code which outputs any logging which is cumbersome and impossible in many places.

Typical MDC mechanisms don't work because they are based on Thread context, but in a Vertx application a request may be processed by multiple threads, and a single thread may process multiple requests concurrently. As far as I can see there is no such thing as a "vertx processing context" in which I can store data. Ideally I would like to be able to put data into such a context when a Message is received, and have any code executing as a result of that message have access to this state.

This feels like it must be a common problem but I am struggling to find a decent solution, I welcome any suggestions and would be happy to provide more context/info but am not sure what else would help at the present.

Kind regards,

James

[Julien Viet]

Somebody recently has shown a set of modifications to Vert.x that allow this kind of scenario and the modifications looked good and quite minimalist (although not tested), their solution is based on Zipkin. Clement and myself were keen on having that in Vert.x Core to allows such scenarii.

Perhaps you want to dig a bit the archive and have a loook ?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4f6a1b26-0ecb-43a5-9d52-26454e373f79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clement Escoffier]

Hi,

Exactly. These modifications would allow providing some kind of MDC / Processing context support to Vert.x. Obviously there are case to check such as [event loop context -> worker context -> event loop context], but most of the logic is there. The discussion is on this thread: https://groups.google.com/d/msgid/vertx/49acc8d1-e5d6-4d9c-9192-8789dc384951%40googlegroups.com.

Clement 

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/0FE6BB6D-142A-4DD7-BB79-E5CA14945307%40julienviet.com.

[Emmanuel Idi]

Yeah, the logging MDC propagation across execution contexts will be possible with the change I was proposing. When I was experimenting with the zipkin module[1] I was able to put the trace ID in the MDC and get log record correlation from -- in my use-case -- the incoming HTTP request through the various verticle executions. 

I should get some cycles over the US holiday next week to spend time getting the changes updated for vertx `master` and tested. I'll also try to get the `vertx-zipkin` module cleaned up and made suitable for consumption by others.

Thanks for the feedback.

-ei

[1] https://github.com/emmanuelidi/vertx-zipkin

[Julien Viet]

great, we will need tests to ensure the behavior of the listener also (I don’t remember seing any in the original diff you exhibited)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/898fdd7f-24fc-4278-9da7-2dee4d8813fe%40googlegroups.com.

[Julien Viet]

Hi Emmanuel,

are you still looking forward to contribute to Vert.x Core the changes necessary for making ZipKin integration in Vert.x ?

This thread [1] on openzipkin/brave repository mentions your effort that could be integrate but to make it so, we need Vert.x Core to have these changes.

Julien

[1] https://github.com/openzipkin/brave/issues/307#issuecomment-268924421

On Nov 18, 2016, at 5:56 PM, Emmanuel Idi <emmanu...@gmail.com> wrote:

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/898fdd7f-24fc-4278-9da7-2dee4d8813fe%40googlegroups.com.

[Emmanuel Idi]

Hi Julien,

Yes, I believe I have the changes completed[1]. I wanted to finish doing some testing with the zipkin module prior to opening a PR. Is there a deadline by which to get this into 3.4.0?

- Emmanuel

[1] https://github.com/eclipse/vert.x/compare/master...emmanuelidi:master+tracing

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

[Julien Viet]

It would be good to have it early January so it can be properly reviewed for the 3.4.0 beta.

It might have an impact on performances and one of my concerns is to check that there is no overhead without distributed tracing.

Julien

[Clement Escoffier]

Hi,

I’ve started looking at the PR, and found a couple of things that need to be discussed early January. Typically, the model would not easily work on multi-threaded worker verticles (yes, I know, these verticles should not be used, but they do exist).

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5E258ED7-B8A7-43DE-91BD-4121063FE48F%40julienviet.com.

[Pradp]

Hi,

Is this feature available now? At least in beta?

Pradeep

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/898fdd7f-24fc-4278-9da7-2dee4d8813fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicolas Ocquidant]

I would also be interested in this feature!

[hyginous neto]

Is vertx zipkin ready for use?

We would like to use it to pass along correlationId across Vertx verticles. Currently, do not see any solution.

[Nirmit Goyal]

Hi,
Is this problem solved now(either using ZipKin) ?

[infor...@gmail.com]

Pardon yet another ping.  What is the latest on this?  I tried to follow the other threads but got lost in the process.  Zipkin, from what I've gathered, is more about metrics, rather than ensuring that each log entry has a correlation id.  Or is the latter a prerequisite to the tracing capability?  Will read up more but I would appreciate pointers regarding the specific topic of replacing MDC with an async-compatible solution for logging correlation ids, especially from those who have since tried something and succeeded.

Thanks,

Iggy

[Дмитрий Марачев]

Hi guys. Any updates regarding this tracking pocibiity?

вторник, 15 ноября 2016 г., 10:40:37 UTC-8 пользователь James Adams написал:

[shailender arya]

Is this available now ?