[erlang-questions] gen_server state is not released for gc

[Volodymyr Kyrychenko]

Hi. Here is simple gen_server that holds binary in its state and never
releases it.

http://ideone.com/U4f9X

1> X = myserver:start_link("/tmp/1.cache").
{ok,<0.33.0>}
2> myserver:info().
{binary,[{140408119021616,401681797,2}]}
3> myserver:release().
ok
4> myserver:info().
{binary,[{140408119021616,401681797,2}]}
7> memory(binary).
402043632

How come there are 2 references on it and what to do to free this binary?

--
Best Regards,
Volodymyr Kyrychenko
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Björn-Egil Dahlberg]

On 2012-01-13 18:33, Volodymyr Kyrychenko wrote:
> Hi. Here is simple gen_server that holds binary in its state and never
> releases it.
>
> http://ideone.com/U4f9X
>
> 1> X = myserver:start_link("/tmp/1.cache").
> {ok,<0.33.0>}
> 2> myserver:info().
> {binary,[{140408119021616,401681797,2}]}
> 3> myserver:release().
> ok
> 4> myserver:info().
> {binary,[{140408119021616,401681797,2}]}
> 7> memory(binary).
> 402043632
>
> How come there are 2 references on it and what to do to free this binary?

it is not gc:ed yet.
Try erlang:garbage_collect/0 info:

handle_call(info, _From, State) ->

erlang:garbage_collect(),

{reply, process_info(self(), binary), State};

// Björn-Egil

[Max Lapshin]

Try to return hibernate flag after your release call and check what will happen

[Björn-Egil Dahlberg]

On 2012-01-13 18:50, Björn-Egil Dahlberg wrote:
> On 2012-01-13 18:33, Volodymyr Kyrychenko wrote:
>> Hi. Here is simple gen_server that holds binary in its state and never
>> releases it.
>>
>> http://ideone.com/U4f9X
>>
>> 1> X = myserver:start_link("/tmp/1.cache").
>> {ok,<0.33.0>}
>> 2> myserver:info().
>> {binary,[{140408119021616,401681797,2}]}
>> 3> myserver:release().
>> ok
>> 4> myserver:info().
>> {binary,[{140408119021616,401681797,2}]}
>> 7> memory(binary).
>> 402043632
>>
>> How come there are 2 references on it and what to do to free this
>> binary?
> it is not gc:ed yet.

To be a little more specific. The references (Procbins) to your offheap
binary are still on the process heap even though nothing in the rootset
is referencing the procbins. A gc will be triggered when the current
heap is full, binary vheap is "full" or an explicit garbage_collect is
called for, which ever comes first. At that moment the procbins will be
removed and the offheap binary will be deallocated (if nothing else is
referencing it).

// Björn-Egil

[Max Lapshin]

I really thought that hibernating should make full cleanup of process state.

[Björn-Egil Dahlberg]

Den 13 januari 2012 21:51 skrev Max Lapshin <max.l...@gmail.com>:

I really thought that hibernating should make full cleanup of process state.

I am certain that hibernate will achieve the same intended result, i.e. no lingering procbins:

handle_call(release, _From, _State) ->

  {reply, ok, nothing, hibernate}.

Hibernate is a lot more aggressive, see doc for details, then a gc  but is also useful in scenarios when you want to shrink idle workers to a minimum.

Of course, you normally don't need to trigger gc:s or a hibernate unless your application scenario really requires it. Large lingering binaries are a bit of a problem if the space is tight. In those cases an well placed explicit gc is well worth the trade off.

// Björn-Egil

[Max Lapshin]

I thought, that calling gc will be a problem:

A = crypto:rand_bytes(1024*1024*1024),
erlang:garbagecollect(),
{reply, ok, State}

Will gc free A? I think that no, because A is still used in the place,
where gc is called. Am I right?

[Björn-Egil Dahlberg]

Den 13 januari 2012 23:11 skrev Max Lapshin <max.l...@gmail.com>:

I thought, that calling gc will be a problem:


A = crypto:rand_bytes(1024*1024*1024),
erlang:garbagecollect(),
{reply, ok, State}


Will gc free A? I think that no, because A is still used in the place,
where gc is called. Am I right?

 

Correct.

To be a little technical (and don't take this as an absolute truth in the details),

What we see here is "A is referencing a large binary", which is true. What happens internally is that Eterm A is a boxed pointer from the stack or a register which points to a ProcBin on the heap. The ProcBin in turn holds an increment of a Reference counted binary off heap. Both the stack and the registers will become part of the root set during the garbage collection hence letting the binary be reachable and therefore "live".  I believe that you need leave the function (release the stack) in order to free your binary. 

A gc without the A in the stack will not let the ProcBin survive during the copy phase of the gc. When the ProcBin is removed the RefcBinary will decrement and if it reaches zero, i.e. the last ProcBin has died, that too will be removed.

// Björn-Egil

[Jon Watte]

Is this an artifact of the current implementation?

Semantically, I don't see why a sufficiently smart VM wouldn't be able to collect the binary at that point.

Sincerely,

jw

--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.

2012/1/13 Björn-Egil Dahlberg <wallentin...@gmail.com>

[Björn-Egil Dahlberg]

Den 16 januari 2012 18:24 skrev Jon Watte <jwa...@gmail.com>:

Is this an artifact of the current implementation?

Semantically, I don't see why a sufficiently smart VM wouldn't be able to collect the binary at that point.

Hmm, yes, I did not answer correctly to the example. (Bad teacher) "A" isn't used in the above example so it will be garbed. Let me show you by example what I mean.

-module(test).

-compile([export_all]).

go1() ->

    {ok, A} = file:read_file("test.erl"),

    go1(A).

go1(A) ->

    ok = nothing(A),

    erlang:garbage_collect(),

    io:format("~p~n", [erlang:process_info(self(), binary)]),

    ok.

go2() ->

    {ok, A} = file:read_file("test.erl"),

    go2(A).

go2(A) ->

    erlang:garbage_collect(),

    ok = nothing(A),

    io:format("~p~n", [erlang:process_info(self(), binary)]),

    ok.

nothing(_) -> ok.

What will the two functions go1 and go2 produce?

Eshell V5.9.1  (abort with ^G) 

1> test:go1().

{binary,[]}

ok

2> test:go2().

{binary,[{11960880,575,2}]}

ok

// Björn-Egil 

[Jon Watte]

Again, the question is whether a smart enough compiler or VM couldn't just get rid of this reference? Specifically, because nothing is called without the module specifier, the VM could inline it, and in turn drop the reference to the binary, right?

Whereas if the call to nothing was made with module name qualification, then that inline replacement could not take place, and the reference would have to be retained?

Sincerely,

jw

--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.

2012/1/16 Björn-Egil Dahlberg <wallentin...@gmail.com>