Topic channels are not being cleaned even after last member of topic leaves

[Dilshan Sandhu]

Hi,
I did some tests with different topic queue limits (hub.go:run():case join). Check scenario 1(og) and scenario2(modified). A gatling test was ran against both these conditions(and everything else same). At the start and end of each test(all ws closed), we noted the Resident Set (as given by htop)(thiese values remained stable after 5min also).
     Before    After

A  417M      1.71G
B  419M      2.09G

This indicates that the channels are not being cleaned for a topic which is not active. This shouldn't happen.

Scenario1(A)

clientMsg: make(chan *ClientComMessage, 192),

serverMsg: make(chan *ServerComMessage, 64),

reg: make(chan *ClientComMessage, 256),

unreg: make(chan *ClientComMessage, 256),

meta: make(chan *ClientComMessage, 64),

Scenario2(B)

clientMsg: make(chan *ClientComMessage, 4096),

serverMsg: make(chan *ServerComMessage, 4096),

reg: make(chan *ClientComMessage, 4096),

unreg: make(chan *ClientComMessage, 4096),

meta: make(chan *ClientComMessage, 128),

At the start df

[Dilshan Sandhu]

Also if you think they are being closed in code somewhere. Please point me there.

[Gene]

Closing a channel is optional:

https://stackoverflow.com/a/8593986/6692196

https://groups.google.com/g/golang-nuts/c/pZwdYRGxCIk/m/qpbHxRRPJdUJ

A still-open channel will be garbage-collected unless there are live references to it. And I don't think there are live references once the topic is shut down.

I would guess that what you are observing is memory not being returned to the OS because there is no memory pressure.

[Dilshan Sandhu]

We have seen container(with 32GB ram and GOMEMLIMIT=25GiB) OOM killed. According to this blog https://www.glean.com/blog/how-we-analyzed-and-fixed-a-golang-memory-leak, they are saying this issue of OS not demanding memory back has been fixed in 1.16. We are using 1.20.6 version to build binary.
More insights would be helpful.

[Gene]

I would suggest to gather more information and post to golang-nuts https://groups.google.com/g/golang-nuts/

[Dilshan Sandhu]

Is it possible that if channel is closed abruptly and the data is in this channel which is in the above example, *ClientCommMessage is a reference to pointer, can become dangling and is taking memory?

[Gene Sokolov]

Only if there is a bug in the language:

A channel that the garbage collector determines to be unreachable will have its resources reclaimed whether or not it is closed.

“The information contained in this message is intended for the addressee only and may contain classified information. If you are not the addressee, please delete this message and notify the sender; you should not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the individual(s) and not necessarily of the Docquity. No reliance may be placed on this message without written confirmation from an authorised representative of its contents.”

--
You received this message because you are subscribed to the Google Groups "Tinode General" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tinode+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tinode/01cd9e93-a078-4ad2-926c-4cc0b9a875aan%40googlegroups.com.

[Dilshan Sandhu]

Ok, so could you please clarify if this is happening with your server as well.
Thanks

[Gene S]

The prod has been up for ~10 days. It apparently did crash then, but the logs are already gone. I'll keep an eye on it for the next week.

Our monitoring covers 48 hours. There is no meaningful change in memory usage in the last 48 hours.

To view this discussion on the web visit https://groups.google.com/d/msgid/tinode/09f948ef-9b0a-4dec-ac0f-f0ec0e02c880n%40googlegroups.com.

[Gene]

I've been watching memory usage for almost 3 weeks now. It's been pretty consistent. I'm sure there are no memory leaks in production under our normal usage.

[Gene]

BTW, I did see a large spike in memory usage:

 

I'll investigate the cause.

[Gene]

On Saturday, August 26, 2023 at 9:07:35 AM UTC+3 Gene wrote:

BTW, I did see a large spike in memory usage:

 

I'll investigate the cause.

Strangely, there is nothing in the logs. It must be a garbage collection process: either an accounts GC, or unused images.