Heap memory usage of org.cometd.server.ServerSessionImpl instances

[Cosimo]

Hi,

I'm investigating a case of OOM caused by heap exhaustion. Our servers normally run for months uninterrupted without issues. We had a few cases of servers stopping due to hard OOM errors, as in:

Heap dump file created [4620244762 bytes in 239.607 secs]
Exception in VM (AttachListener::init) :
java.lang.OutOfMemoryError: Java heap space
2025-06-24 01:52:20
Full thread dump OpenJDK 64-Bit Server VM (21.0.7+6-Ubuntu-0ubuntu124.04 mixed mode, sharing):
...

No Java expert here. I've been looking into the heap dump with MAT, and found that the 3GB of heap were all (99.9%) held up by org.cometd.server.ServerSessionImpl instances. When diving into the dump, I can see that each of these objects was storing a long queue of messages, presumably not acknowledged or in some other way "pending". Other evidence suggests this is related to some sort of malicious/spammy client traffic.

Before I investigate more about this, and how to reproduce it consistently, would there be any circumstances where the ServerSessionImpl objects retain `ServerMessageImpl` instances for an abnormal amount of time? Would there be a way to limit this behavior?

In other words: is it possible to cause a CometD server to blow up by connecting a lot of clients to it, and then somehow stall reception of messages, so they would be retained at the server level?

This sort of malfunction is rare, but it does cause OOM and thus complete server breakdown, hence I'm looking into what's possible to do to prevent it.

Thanks,

--
Cosimo

[Cosimo]

On Wed, Jun 25, 2025, at 11:46, Cosimo wrote:
> Hi,
>
> I'm investigating a case of OOM caused by heap exhaustion. Our servers
> normally run for months uninterrupted without issues. We had a few
> cases of servers stopping due to hard OOM errors, as in:

Forgot to mention that we're running Java CometD server 8.0.6 on Jetty 12.0.15,
JVM is OpenJDK 21 on Ubuntu 24.04.

--
Cosimo

[Simone Bordet]

Cosimo,

do you mind moving this to CometD Discussions
(https://github.com/cometd/cometd/discussions)?

I'll reply there.

> --
> --
> You received this message because you are subscribed to the Google Groups "cometd-users" group.
> To post to this group, send email to cometd...@googlegroups.com
> To unsubscribe from this group, send email to cometd-users...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/cometd-users
>
> Visit the CometD website at http://www.cometd.org
>
> ---
> You received this message because you are subscribed to the Google Groups "cometd-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cometd-users...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cometd-users/96569b35-f314-48d6-a952-91e9c78344ed%40app.fastmail.com.



--
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless. Victoria Livschitz

[Cosimo]

On Wed, Jun 25, 2025, at 15:35, Simone Bordet wrote:
> Cosimo,
>
> do you mind moving this to CometD Discussions
> (https://github.com/cometd/cometd/discussions)?

Of course.
Here it is: https://github.com/cometd/cometd/discussions/2116

--
Cosimo