Excessive memory usage?
Peter Gibbs
i586),
and allocates 1.5 million headers - is this not somewhat excessive?
set I0, 10000000
loop:
set S0, "Hello"
dec I0
gt I0, 0, loop
interpinfo I1, 7
print "There are "
print I1
print " total Buffer structs\n"
end
--
Peter Gibbs
EmKel Systems
Leopold Toetsch
> The program below uses approximately 60MB on my system (linux 2.2.23 on
> i586),
> and allocates 1.5 million headers - is this not somewhat excessive?
Still searching the reason.
The only thing I can tell till now is: Turning off TRACE_SYSTEM_AREAS in
dod.c doesn't show this memory abuse.
But trace_system_stack->trace_mem_block only marks ~8 buffers live,
which shouldn't lead to that many live string headers.
leo
Peter Gibbs
The problem seems to be caused by the 'skip' logic in do_dod_run, combined
with the ever-increasing size of allocations. Deactivating the 'skip'
removes the
problem completely (but obviously does more DOD runs for everything). The
attached patch bypasses the allocation growth if the previous DOD was
skipped,
which improves the situation considerably (takes less than 5MB, and ends up
with
102608 buffers allocated).
Leopold Toetsch
> The program below uses approximately 60MB on my system (linux 2.2.23 on
> i586),
> and allocates 1.5 million headers - is this not somewhat excessive?
I have removed the (somewhat bogus) skip logic, which worked only with
TRACE_SYSTEM_AREAS turned off.
The excessive memory usage is gone, but still way too much headers are
active compared to the case, when no processor stack gets walked. There
are 250 or 449 active string_headers in more_tracable_objects after the
DOD run.
leo
Leopold Toetsch
> The problem seems to be caused by the 'skip' logic in do_dod_run,
Ha. 1 minute ago - and before reading this mail - I did check in dod.c
with removed skip logic.
But this is not the whole reason. The skip-logic worked fine, when
walking the processor stack was not done. With walking the processor
stack there are either 250 or 449 free string headers (after alternate
DOD runs) with ~700 total headers. The free headers count is slightly
above the replenish level, so the excessive usage is gone. But the real
problem is: We have much too much active string headers. And I don't
see, where they are coming from - only tracing the processor area is
somehow related to it.
PS I did disable the 2nd call to trace_system_areas - seems bogus too.
leo
Peter Gibbs
> But this is not the whole reason. The skip-logic worked fine, when
> walking the processor stack was not done. With walking the processor
> stack there are either 250 or 449 free string headers (after alternate
> DOD runs) with ~700 total headers. The free headers count is slightly
> above the replenish level, so the excessive usage is gone. But the real
> problem is: We have much too much active string headers. And I don't
> see, where they are coming from - only tracing the processor area is
> somehow related to it.
This was a tricky one to find. The sequence of events is:
1) make_COW_reference is called to clone the constant string
2) new_string_header is called, and has no free headers available;
do_dod_run is invoked
3) walking the system stack finds the source (constant) string header
and marks it as live
4) do_dod_run does not walk the constant string header pool, so the
live flag does not get cleared
5) copy_string_header duplicates the header, including the live flag
6) we now have a normal (non-constant) string header with the live
flag set, outside a DOD run
Now it is just a matter of deciding how best to fix it! As a simple test,
killing the destination in make_COW_reference (i.e. clear live flag)
seems to work.
Leopold Toetsch
> 3) walking the system stack finds the source (constant) string header
> and marks it as live
> 4) do_dod_run does not walk the constant string header pool, so the
> live flag does not get cleared
> 5) copy_string_header duplicates the header, including the live flag
Thanks, you are the best. I really couldn't imagine, how this does
happen, but now you did prove show the problem.
> Now it is just a matter of deciding how best to fix it! As a simple test,
> killing the destination in make_COW_reference (i.e. clear live flag)
> seems to work.
Or, don't set live flag on constant items?
leo
Brent Dax
# This was a tricky one to find. The sequence of events is:
...
# 6) we now have a normal (non-constant) string header with the live
# flag set, outside a DOD run
On a tangentially-related topic: In secure execution mode, would it make
sense to run through all the pools and unset the live flags explicitly
*before* starting a DOD run? I could see a denial-of-service exploit
being built around a bug like this.
--Brent Dax <bren...@cpan.org>
@roles=map {"Parrot $_"} qw(embedding regexen Configure)
>How do you "test" this 'God' to "prove" it is who it says it is?
"If you're God, you know exactly what it would take to convince me. Do
that."
--Marc Fleury on alt.atheism