Create a 1GB ballast but it takes up RSS and pages are Dirty?

[Kn (Kn)]

Hi, guys, I have a question about memory management.

I created a ballast in my go application, I didn't create the ballast in the very beginning like initializing a global variable or do this in package level init functions. I initialize it later after some setup process.

Now the problem begins. I expect the ballast like `ballast := make([]byte, 1<<30)` shouldn't take up any physical memory because there's no any writing to it. While when I use `top` command or `pmap` command to check the RSS field, it takes up the physical memory, also `pmap` reports RSS size is equal to DIRTY size.

This makes me confused. After some thinking and testing, I find this is relevant with go Garbage Collector and Memory Allocator. I use 2 manners to verify this:

- 1st manner, turn off the GC. I run the application with GOGC=off, then ballast takes up no physical space.

- 2nd manner, turn off the GC in package level init function using `debug.SetGCPercent(-1)`, then after the necessary setup, explicitly trigger GC `debug.FreeOSMemory()`, then I create the ballast and reset the GC `ballast := make([]byte, 1<<30); debug.SetGCPercent(100)`.

During these two tests, ballast works as expected. Actually If we initialize the ballast  as a global variable, it will work as expected, too.

These tests make me think the ballast takes up RSS is relevant with go Garbage Collector and Memory Allocator. After some time digging into go source code, I think it maybe caused like this:

- during package initialization, maybe there're heap space allocation and recycle by using mmap and madvise. Maybe some pages allocated to create mcentral/mspan to prepare space for small objects. The pages are written, so they're dirty pages. Then after objects recycled, the pages maybe put into p.mcache or mcentral or page cache heap.

- then we create a ballast with 1GB virtual space, it's a large object, go runtime allocate it directly by heap. It may use mmap and advise OS to provide some hugepages (actually on my OS hugepage is configured to [never]).

I know the dirty size is incremented by pagesize when a specific mapped page is written. But I notice the dirty size is about 1GB (which is the same as RSS and ballast size). Who write the space? I didn't write it explicity even once. I just work around this problem using solutions mentioned above. But I am still confused what happened.

That's interesting. 

[Jan Mercl]

On Sun, Nov 6, 2022 at 12:54 PM Kn (Kn) <hit.zh...@gmail.com> wrote:

> Now the problem begins. I expect the ballast like `ballast := make([]byte, 1<<30)` shouldn't take up any physical memory because there's no any writing to it.

The backing array is specified to be zeroed, so we cannot say there's
no writing to it. Depending on the size of the backing array and the
operating system it may not get written as an optimization if backed
by memory the OS can guarantee to be zero filled. Only then it may
remain not yet bound to physical memory.

A simple implementation will just zero it, meaning the opposite
happens - every byte of the backing array gets written and backing
pages for it get allocated.

[Kn (Kn)]

Before, I think the memory is allocated by mmap anon, which Linux OS guarantees reading will return zero value and no physical pages allocated. When writing happens, the physical pages will be allocated. Then I think the create a byte slice maybe the same.

Your idea is clear. I agree with it. 

Just now, I use gdb to dump the ballast anon memory and use hexdump to check its dirty content, all data is zero (Maybe zeroing happens).

But after turning GC off, it works as expected (no RSS is taken, no DIRTY). 

I think there must be something I didn't get it.

[tapi...@gmail.com]

I ever also found this problem: global ballast doesn't work.

So I always use local ballast instead now.

[Kn (Kn)]

Hi, guys, I know what happened.

When we write `ballast := make([]byte, 1<<30)`, it will call makeslice to create a new slice. It's a large object. The memory will be allocated directly via allocLarge() function from heap.

Actually, after allocating a mspan for it, it will check the address range whether it should be zeroed. Please check function `func allocNeedsZero(base, npage uintptr) bool`. When this function returns true, it means the slice underlying memory will be written to zero, otherwise it won't write it to zero.

When we make a ballast as mentioned before, maybe it succeed (no RSS taken up) or fail (RSS taken up), it's relevant with the return value of function allocNeedsZero(...). And this function return true or false is relevant with the arena's state which is affected by previous object allocation and recycle.

[Michael Knyszek]

That's correct. The runtime has a simple heuristic for avoiding zeroing but it's far from perfect. As a result, a ballast is inherently always going to be a little risky. This is especially true on some platforms, such as Windows, since there's no way to avoid marking the memory as committed (Windows is free to use demand paging for memory in the range, so overall system memory pressure may increase, but you can't avoid it being counted as committed for a particular process).

Taking a step back: why a ballast? What about your application makes a ballast a better idea than, for example, setting GOMEMLIMIT=<something> and GOGC=off?

(For additional context, back when the memory limit was proposed, so was a memory target (https://go.googlesource.com/proposal/+/master/design/44309-user-configurable-memory-target.md) which more directly replaces a ballast. I found very little interest in this feature.)

[Kn (Kn)]

We used go1.16.5 before go1.19 released. Occassionally we found ballast takes up RSS :)

About go1.19 GOMEMLIMIT, I tested and have some thinking:

- GOMEMLIMIT is a soft limit, if we deploy by per container per service, GOGC=off+GOMEMLIMIT=70%*totalMemory, it may works as expected. And it may replace the ballast directly.

- If we deploy multiple services in the same host (not per container per service), and if we specify GOGC=off+GOMEMLIMIT=<?>, we may calculate how much memory (and max memory) a service should use.

I think per container per service will be better and easier to deploy to use GOMEMLIMIT.

[Jie Zhang]

Actually, we used go1.16.5 before go1.19 released. Now we're consider using go1.19 GOMEMLIMIT instead.

But GOMEMLIMIT is a soft limit, if you deploy multiple services in the same host (not per container per service), there will be problems.

The GC is delayed, and if combined with GOGC=off, the memory usage will be higher.

Maybe manually trigger forced GC, and combine GOMEMLIMIT will be better. I'm testing...

[Michael Knyszek]

On Wed, Nov 16, 2022 at 10:19 AM Jie Zhang <tx.z...@gmail.com> wrote:

Actually, we used go1.16.5 before go1.19 released. Now we're consider using go1.19 GOMEMLIMIT instead.

But GOMEMLIMIT is a soft limit, if you deploy multiple services in the same host (not per container per service), there will be problems.

The GC is delayed, and if combined with GOGC=off, the memory usage will be higher.

In that scenario, I might suggest setting GOGC higher than 100 (not off) but with some memory limit on each process instead.

Maybe manually trigger forced GC, and combine GOMEMLIMIT will be better. I'm testing...

Manually triggering GCs has a relatively high risk of hurting performance. Even if it doesn't hurt performance today, it might in the next release, unless used very carefully. I don't recommend it outside of testing.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/66d0cItfkjY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/858a4a4d-d049-40d7-9efe-ee6cfae0fa2dn%40googlegroups.com.