Weird count of HTTP server active requests

[Joel Vimenet]

Hello, 

we are using Kamon 2.5.12 with Play 2.8.18 and sbt-kanela-runner-play 2.0.14. We are experiencing a weird behavior : our play application makes HTTP calls to a backend. Under certain circumstances, this backend ends up being very slow, so the response time exceeds the configured play timeout (play.server.akka.requestTimeout) of the application. When this happens, the number of active requests increases and never decreases. Other metrics don't reflect this behavior : RAM, number of open connections, active threads, executor queue size, etc... This is the only metric that looks strange.

After having done this analysis, we ended up with two options : either play leaks some active requests or there is a problem with the count done by Kamon.

I did not find any open bug with this (the closer being this one : https://github.com/kamon-io/Kamon/issues/992), so we are wondering if anyone experienced this.

I had a look to the code, and I was wondering if it could not come from the case were onUpdatreamFinish or onDownstreamFinish is called (https://github.com/kamon-io/Kamon/blob/master/instrumentation/kamon-akka-http/src/main/scala/kamon/instrumentation/akka/http/ServerFlowWrapper.scala#L172). In this case, if we passed in the requestIn handler but not yet in the responseIn handler, the count could be erroneous. However, I don't know akka HTTP graph stage API well enough to be sure this theory is relevant or not, and I don't know if it's a bug or not. That's why I send this email.

Best regards

Joël

Disclaimer

The information contained in this email may be confidential. It has been sent for the sole use of the intended recipient(s). If the reader of this email is not an intended recipient, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this email in error, please notify the sender immediately and destroy all copies of the message.

For more information on how we process personal data please see www.meltwater.com/privacy. Please note that neither the Company nor the sender accepts any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments.