[erlang-questions] How can I release beam process memory?

[Jack Tang]

Hello list,

I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.

```

KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers

KiB Swap:        0 total,        0 used,        0 free. 18684864 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp

```

I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:

```

lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)).

```

and erlang:memory() decreases from 32G to 23G.

```

[{total,22982011544},

 {processes,12884182336},

 {processes_used,12884170336},

 {system,10097829208},

 {atom,13828705},

 {atom_used,13796692},

 {binary,170530288},

 {code,16450626},

 {ets,9637717576}]

```

However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks

BR

-Jack

[Dániel Szoboszlay]

Hi Jack,

I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed).

You can check with recon_alloc how much unused memory is lost due to fragmentation in the various allocators.

The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator.

However, if the memory is wasted in the eheap_alloc, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all.

Cheers,

Daniel

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Frank Muller]

I'm very interested in that too. Last time I asked for help only @Max Lapshin was kind enough to share some feedbacks (today's Daniel).

I would appreciate if the OTP team can teach us how to debug VM's mem issues, and  how pick the right allocator.

I'm aware of the excellent recon module. But to be honest, I was never be able to successfully use it to track my mem issue. Maybe not using it correctly. 

Thanks in advance.

Best,

/Frank

[Jack Tang]

Hi Daniel

Thank you for the clue and excellent recon toolkits! I would like dig it deeper  :)

BR

-Jack

--

Jack Tang


http://www.linkedin.com/in/jacktang

[Lukas Larsson]

Hello Frank,

On Sun, Jan 15, 2017 at 12:03 AM, Frank Muller <frank.mu...@gmail.com> wrote:

I would appreciate if the OTP team can teach us how to debug VM's mem issues, and  how pick the right allocator.

The right allocator is the default allocator. It is only in very rare circumstances that you would want to change it. Sure you can optimize them to be more perfect, but in most cases the defaults are great.

If you suspect that you have issues that tuning erts_alloc will solve the best starting points is to read and understand Fred Hebert's excellent blogs/manuals about using erts_alloc to combat memory fragmentation. I cannot stress enough how great a resource his blog post (https://blog.heroku.com/logplex-down-the-rabbit-hole) and manual (https://www.erlang-in-anger.com/) are. 

Then you may also want to watch mine and Rickard Green's/Patrik Nyblom's talks on the subject, they basically cover the same things that Fred covers, but some parts in greater detail. There is also some internal documentation in https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc that explain some of the features that you can tune in erts_alloc. There are also quite a few mail conversations on this mailing list about these types of problems.

When it comes to allocator tuning there is no magic formula. We attempted to build something that automatically tuned the settings in the instrument module (http://erlang.org/doc/man/instrument.html), but it didn't turn out very useful. The problems that to pop up with erts_alloc seem to be very different in both symptoms and solutions. 

If you explain what it is that you have issues with (backing the claims with recon_alloc statistics) and what solutions you have tried then we'll very likely answer.

Lukas

[Jack Tang]

Hello,

In the memory issue investigation, I find something interesting. As I mentioned, 

we usually adopted remote Erlang shell to interactive with nodes in production. 

Sometime we use mnesia:select(...) to return some query result, and it may contain 

billions of rows. And we leave the remote shell alive. I find the query results do 

consume the memory of remote node, since the return value is referenced. In the 

shell, we still can access it using `v(N)` command. I tried `f()` command, and it only 

unbinds the variables. So what's the mechanism of Erlang shell gc? Thanks.

BR

[Jack Tang]

After setting up erlang memory visualization, we find etc allocator does not release the memory during some historical datum are remove from mnesia tables. Can I release the memory on the fly rather than restart the mnesia application? Thanks!

BR

On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <dszob...@gmail.com> wrote:

[Dániel Szoboszlay]

Hi Jack,

tl;dr; try starting Erlang with '+MEas bf'

Long explanation:

If the memory is used by the ets_alloc allocator, it can be either actively used by ETS tables (such as Mnesia disc_copies tables) or it can be lost due to memory fragmentation. I guess this is memory fragmentation in your case, but let's quickly rule out the other possibility first. (By the way, in either case, garbage collecting processes won't reclaim the memory. GC operates on eheap_alloc allocated memory.)

So if erlang:memory(ets) is close to what recon reports as allocated, the memory is in use. Your system wrote a lot of data to ETS/Mnesia, and didn't remove it afterwards. Inspect the contents of the tables and figure out what was left there that shouldn't be there, which part of the application should have removed that data and why didn't it do its job properly.

The other, more likely option is fragmentation. I also experienced that the default memory allocation strategy (aoffcbf = address order first fit carrier best fit) can perform poorly when you use a lot of memory. All address order first fit strategies will use heavily the multiblock carriers with the lowest memory addresses, and if you have many carriers, those placed higher in memory will have less and less chance to be used. In my particular case ets_alloc created a total of 150k multiblock carriers for storing ~1.2TB data in ETS tables. This resulted in ~100 GB unused memory being wasted in the high address carriers. You have a much smaller system, but depending on the usage patterns of your ETS data you can end up in a similar situation.

You can check the number of carriers with erlang:system_info({allocator, ets_alloc}). It will print something like this:

[{instance,0,

           [{versions,"0.9","3.0"},

            {options,[...]},

            {mbcs,[...

                   {carriers,1,1,1},  %% <- number of multi block carriers = 1

                   ...]},

            {sbcs,[...

                   {carriers,0,0,0},  %% <- number of single block carriers = 0

                   ...]},

            {calls,[...]}]},

 {instance,1,...

Check theaw numbera across all your allocator instances. You will typically have very few single block carriers (unless you store huge records in ETS). In my experience, fragmentation correlates well with the number of carriers an allocator handles, and can become quite significant above ~10 carriers.

So, If you have the same problem I described, I have some bad news and some good news. The bad news is that I don't know a way of forcing the VM to defragment memory and get rid of the waste. The good news is that the bf (best fit) allocation strategy (which used to be the default up to R16) performs much better when you have many carriers. You need to pass the '+MEas bf' command line argument to the VM to switch ets_alloc to bf strategy.

Hope it helps,

Daniel

[Jack Tang]

Hi Daniel

Thank you for the detailed explanation and sharing the experiences. I focused on the performance of GC heavily before and ignored the allocation strategy. 

One more question, as you mentioned memory on eheap_alloc can be released by calling GC on every process. Right now I setup periodic gc process  to free eheap. However can I set some process flag to make GC automatically?

Thanks again!

BR

-Jack

[Vans S]

Just use the instrument module to track the actual terms held then reference the PID that created the term, set some hooks in beams allocator to track the pid references to the term (since instrument does not do this).  Read beams memory to check what the actual term is.

All these statistics and reporting we currently have for erlang memory issues is simply poking at random things in the dark.

[Michael Truog]

Hi Jack,

The simplest way to make an Erlang process automatically GC is to use a temporary Erlang process, since the death of the Erlang process will cause GC to occur soon after.  It is common to avoid long-lived Erlang processes that create lots of temporary data quickly due to excessive memory consumption, and to refactor the long-lived process a temporary process is used to create the temporary data within (this matters at high request rates, though you may not see the problem with low request rates).  You may also tune an Erlang process with fullsweep_after to cause GC to occur more frequently, but I see that as only performance tuning, not fixing a problem that should be fixed with a temporary Erlang process.  There also is the erlang:garbage_collect function, though you shouldn't use that in production source code (source code that needs to be dependable as its expectations change).

Best Regards,
Michael