I'm trying to write something that will account for memory use on a RabbitMQ system. To do this I'm trying to match up the numbers I get out of erlang:memory(ets) and ets:info(T, memory), and erlang:memory(processes) and erlang:process_info/2.
And it doesn't make a huge amount of sense to me. I don't expect things to match up perfectly, of course there will be some memory unaccounted for, but the ETS numbers in particular can be quite off:
1> erlang:memory(ets).
2120291872
2> lists:sum([ets:info(T, memory) || T <- ets:all()]) * erlang:system_info(wordsize).
1874292776
So I have ~2GB of ETS memory, and ~230MB unaccounted for. Should I expect these to match up closer?
And then...
3> erlang:memory(processes).
1519425156
4> lists:sum([element(2, process_info(P, memory)) || P <- processes()]).
1552852008
So for processes there is less in total than if you add them all up. Is some memory shared? (Are we counting binaries and atoms in there as well?)
Is there a way to get this sort of information in a more consistent manner?
On Fri, Sep 21, 2012 at 8:50 PM, Simon MacMullen <si...@rabbitmq.com> wrote:
> I'm trying to write something that will account for memory use on a RabbitMQ
> system. To do this I'm trying to match up the numbers I get out of
> erlang:memory(ets) and ets:info(T, memory), and erlang:memory(processes) and
> erlang:process_info/2.
> And it doesn't make a huge amount of sense to me. I don't expect things to
> match up perfectly, of course there will be some memory unaccounted for, but
> the ETS numbers in particular can be quite off:
On Sep 21, 2012, at 6:50 PM, Simon MacMullen wrote:
> I'm trying to write something that will account for memory use on a RabbitMQ system. To do this I'm trying to match up the numbers I get out of erlang:memory(ets) and ets:info(T, memory), and erlang:memory(processes) and erlang:process_info/2.
> And it doesn't make a huge amount of sense to me. I don't expect things to match up perfectly, of course there will be some memory unaccounted for, but the ETS numbers in particular can be quite off:
> So for processes there is less in total than if you add them all up. Is some memory shared? (Are we counting binaries and atoms in there as well?)
This can happen when there are a lot of outstanding messages.
> Is there a way to get this sort of information in a more consistent manner?
The ets case above will be fixed. In the process case I haven't decided what to do, yet, but something will be done.
erlang:memory/[0,1] use a very different (and cheaper) strategy than ets:info() and process_info(). The results will more or less always differ, and should since there are memory allocated that isn't for any specific process or ets-table in the erlang:memory() case.
If you want to know memory used for all processes or all ets-tables use erlang:memory(). Much cheaper (both from the callers perspective, and especially from an overall performance perspective), as well as generally a more true value.
Summing up results of process_info(), or ets:info() will, of course, not give you a consistent snapshot of the memory usage at one point in time. Note that the same is true in the erlang:memory() case. The result is the sum of memory usage in different allocator instances that have been read at different (but close) points in time.
erlang:memory() and 'top' reports information about two different things.
erlang:memory() reports the amount of memory that the runtime system has dynamically requested from the erts internal memory allocators. Memory allocators will satisfy these request by placing memory blocks in larger memory areas which we call carriers. There will both be memory used for allocator internal data structures as well as free unused memory in these carriers which is not accounted for by erlang:memory(), and shouldn't be accounted for by erlang:memory().
'top' reports all memory mapped by the emulator process. This includes the carriers used by the erts internal memory allocators, but also statically allocated memory, code of the emulator, etc...
When the emulator has grown large, memory carriers will normally (but not necessarily always) make up the major part of the memory reported by 'top' and other similar tools. I've attached a module, called memory, that can be used in order to sum up the sizes of all used carriers. The memory allocators may however keep unused carriers a while in order to try to reuse them. These unused carriers are not accounted for by the memory-module. Also note that this is a sum of sub results acquired at different (but close) points in time.
It should be possible to use the memory-module on R10B and upwards. Note that it is more or less untested. I've only smoke tested it on R15B02.
Regards,
Rickard Green, Erlang/OTP, Ericsson AB.
On Sep 22, 2012, at 10:18 AM, Anton Lebedevich wrote:
> Total in memory() output is about 100Mb but real memory usage of
> erlang vm (R14B04) is ~600Mb
> On Fri, Sep 21, 2012 at 8:50 PM, Simon MacMullen <si...@rabbitmq.com> wrote:
>> I'm trying to write something that will account for memory use on a RabbitMQ
>> system. To do this I'm trying to match up the numbers I get out of
>> erlang:memory(ets) and ets:info(T, memory), and erlang:memory(processes) and
>> erlang:process_info/2.
>> And it doesn't make a huge amount of sense to me. I don't expect things to
>> match up perfectly, of course there will be some memory unaccounted for, but
>> the ETS numbers in particular can be quite off:
>> So for processes there is less in total than if you add them all up. Is some
>> memory shared? (Are we counting binaries and atoms in there as well?)
>> Is there a way to get this sort of information in a more consistent manner?
> erlang:memory() and 'top' reports information about two different things.
> erlang:memory() reports the amount of memory that the runtime system has dynamically requested from the erts internal memory allocators. Memory allocators will satisfy these request by placing memory blocks in larger memory areas which we call carriers. There will both be memory used for allocator internal data structures as well as free unused memory in these carriers which is not accounted for by erlang:memory(), and shouldn't be accounted for by erlang:memory().
> 'top' reports all memory mapped by the emulator process. This includes the carriers used by the erts internal memory allocators, but also statically allocated memory, code of the emulator, etc...
> When the emulator has grown large, memory carriers will normally (but not necessarily always) make up the major part of the memory reported by 'top' and other similar tools. I've attached a module, called memory, that can be used in order to sum up the sizes of all used carriers. The memory allocators may however keep unused carriers a while in order to try to reuse them. These unused carriers are not accounted for by the memory-module. Also note that this is a sum of sub results acquired at different (but close) points in time.
> It should be possible to use the memory-module on R10B and upwards. Note that it is more or less untested. I've only smoke tested it on R15B02.
> Regards,
> Rickard Green, Erlang/OTP, Ericsson AB.
> On Sep 22, 2012, at 10:18 AM, Anton Lebedevich wrote:
>> There could be even more inconsistensies between memory consumed by
>> erlang vm and statistics reported by memory().
>> top output
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 15863 root 20 0 721m 559m 3316 S 0 14.1 3:47.58 beam.smp
>> Total in memory() output is about 100Mb but real memory usage of
>> erlang vm (R14B04) is ~600Mb
>> On Fri, Sep 21, 2012 at 8:50 PM, Simon MacMullen <si...@rabbitmq.com> wrote:
>>> I'm trying to write something that will account for memory use on a RabbitMQ
>>> system. To do this I'm trying to match up the numbers I get out of
>>> erlang:memory(ets) and ets:info(T, memory), and erlang:memory(processes) and
>>> erlang:process_info/2.
>>> And it doesn't make a huge amount of sense to me. I don't expect things to
>>> match up perfectly, of course there will be some memory unaccounted for, but
>>> the ETS numbers in particular can be quite off:
>>> So for processes there is less in total than if you add them all up. Is some
>>> memory shared? (Are we counting binaries and atoms in there as well?)
>>> Is there a way to get this sort of information in a more consistent manner?
>> So for processes there is less in total than if you add them all
>> up. Is some memory shared? (Are we counting binaries and atoms in
>> there as well?)
> This can happen when there are a lot of outstanding messages.
Not that I'm anything like as bothered about this case but for the
record my system was near-idle when I invoked those...
> The ets case above will be fixed.
That's cool, that was my main worry. Thank you.
> In the process case I haven't decided what to do, yet, but something
> will be done.
It's not too much of a bother for me, I just wanted to understand what was going on.
> erlang:memory/[0,1] use a very different (and cheaper) strategy than
> ets:info() and process_info(). The results will more or less always
> differ, and should since there are memory allocated that isn't for
> any specific process or ets-table in the erlang:memory() case.
Oh, sure. I am not trying to get things to add up precisely, but to let my users get a plausible idea of what the biggest blocks of memory are. I'm only concerned when the discrepancies are hundreds of megabytes.
