Hello,
I run a distributed system and quite often need to see how a request goes through each component of it. I've thought of adding a flag to each passed message, indicating that some information must be logged at that point to a central location, so I can see how the request is processed in real time. It seems to me that such a system would be useful enough for other people to already exist in some way, but I couldn't find it. The closest I got was Tracelytics but it doesn't seem to do what I need exactly. I believe I have three options:
- Develop a web app that takes these messages through http or rabbitmq (which I already use everywhere) and show them in realtime on a page.
- Develop a plugin for sentry that intercepts log messages but instead of storing them as usual shows them on the realtime page (I belive using redis would work for that). These messages would be sent using a custom log level and have a key on the "extra" parameter, such as {'sentry_tracing_id': 'uuid_string'}. I know this might seem very hackish and isn't even supported by the current plugin API, but It makes sense to me that such a tracing tool would be a feature of sentry and not a separate system. Also I'd get the transport for those messages for free, without the need to write a client.
- Keep looking and find such a system.
What do you think? Any tips on an existing system that does what I need?
Thank you.