Mutiny useful stacktraces and debugging

[Den]

Hello,

in  Spring there are some tools to make debugging of reactive code a lot easier:

https://spring.io/blog/2019/03/28/reactor-debugging-experience

Basically, you can turn on more useful stack traces.

Is there something similar or better in mutiny or any plans to implement it?

Thanks!

[clement escoffier]

Hello,

We have been looking into the stack frames rewrite, but no great progress so far (@jponge, maybe you can add a comment).

Quarkus does not need BlockHound. It has a different way to detect when the I/O threads are blocked and if a thread can be blocked.

Clement

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/4bf73a69-31b2-4278-88fd-1acb195f5010n%40googlegroups.com.

[Loïc MATHIEU]

Hi,

A BlockingHood integration should be pretty easy to do as described in the integration documentation: https://github.com/reactor/BlockHound/blob/master/docs/custom_integrations.md 

You just need to use the right thread name pattern and it should work.

Regards,

Loïc

Le mer. 9 févr. 2022 à 16:01, Den <denni...@gmail.com> a écrit :

[Julien Ponge]

I had done some explorations on that front, see https://github.com/smallrye/smallrye-mutiny/issues/177#issuecomment-783626041 but that wasn’t conclusive with a non-instrumenting solution.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAKW6fif92WDcgMA9xQnnDCNB3nCvmC233bwoz4OCdht8A032VQ%40mail.gmail.com.

[Den]

right, it's more about stack traces agent, as quarkus already detects blocked threads

[Julien Ponge]

I think Den was talking about rewriting stack-traces so you’d get a pointer to the operator rather than the actual execution stack-trace.

BlockHound is interesting for detecting blocking operations but we already have very simple and effective checks in place in Mutiny so that it will report a call to, say, multi.await() while on an event-loop. Unless we have more complex blocking issues to spot we won’t need any BlockHound-style instrumentation.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVE0ER%3Drx%2BRMPiDeF9c4eZB-8%2BC_a2FQ8qDzR951Dd-ZxA%40mail.gmail.com.

[Den]

Spring reactive has a concept of checkpoints https://projectreactor.io/docs/core/release/reference/#checkpoint-alternative

and the ReactorDebugAgint.init() instruments the code by inserting a checkpoint on each operator, as far as I understand.

[Loïc MATHIEU]

Hi Julien,

As far as I know, we have something to detect "long blocking call" and "hibernate reactive wrong usage" but not a general system to detect blocking calls inside a non-blocking construct. 

BlockHound is more low level as it detects I/O usage at the JDK level so if you open a file or make a call to a remote server (like via a rest client, a database client, an elasticsearch client, ...) it will generate an exception immediately.

But I agree that we already have some checks available that covers some bad usage but if someone use a library not integrated as a Quarkus extension there is more risk that it's usage block an event loop.

Regards,

Loïc

[Julien Ponge]

Hi Loic,

Yes you are right, what we have right now from the Mutiny side is that Mutiny blocking calls (e.g., not code in lambdas doing, say blocking I/O) will result in a failure when run on the event-loop.

Cheers

[Julien Ponge]

Hi,

BlockHound is more low level as it detects I/O usage at the JDK level so if you open a file or make a call to a remote server (like via a rest client, a database client, an elasticsearch client, ...) it will generate an exception immediately.

So I’ve played a bit with BlockHound last night, then in Quarkus this morning.

• You have to configure it (a lot), else it quickly turns your code into a tight loop of java.lang.IllegalMonitorStateException stack traces, for instance because by default it complains if you block from a ServiceExecutor. So you get to exclude a few methods so service executors work fine, but then it’s so coarse-grained that it becomes complicated if any possible to make a distinction between some executors that can block and some that shouldn’t block.
• Configuration is a bit counter-intuitive:
• you can specify what methods block, but not which types to ignore,
• you can mention what threads are non-blocking, but not easily what threads to just ignore (e.g., Vert.x worker pools are made for blocking)
• It doesn’t play nice in Quarkus, because of classloaders, AESH overrides, etc
• On Java 13+ you need to add flags for the bytecode instrumentation to work (nothing BlockHound can do, but it requires care).

So I’m sure it’s a fine tool for the stack it was written for, but not something that would fit nicely in Quarkus (or at least without significant changes).

Perhaps we could have a custom-made extension that inserts probes to check what thread methods get called on in tests and dev mode, that would possibly work better. However such kind of engineering is easy for the 80%, and the remaining 20% are just horribly complex corner cases :-)

My 2 cents