ponie unwell without --gc=libc
Nicholas Clark
it fails to build Unicode::Normalize, and fails about 40 regression tests.
If parrot's garbage collector is changed from the default (compacting, IIRC)
to the either libc or malloc, then ponie only fails 6 tests. ie make this
change:
Index: Configure.pl
===================================================================
RCS file: /cvs/public/ponie/Configure.pl,v
retrieving revision 1.12
diff -r1.12 Configure.pl
47c47,48
< system($^X,'Configure.pl',"--ccflags= :add{ -I$dir/perl}") && die "error";
---
> system($^X,'Configure.pl', "--gc=libc",
> "--ccflags= :add{ -I$dir/perl}") && die "error";
As I understand it parrot's default garbage collector will move data blocks
owned by PMCs. However, all of the PMCs ponie generates do not have gc-owned
data attached to them, so there should be no difference. Chatting with Dan
on IRC we think that it has to be a bug in parrot's GC.
Unfortunately Arthur and I can't get it down to a simple test case. For
example on x86 Linux t/op/pat fails at test 345 with parrot's default gc,
but that whole test passes with gc=libc. Trying to cut down the regression
tests to the fail point makes them pass, which suggests that it's the
cumulative resource usage that is really the problem, not any particular
perl construction. [tests fail in different places on OS X. Not tried other
systems yet. Still fighting AIX]
We're not sure how to track this one down further - at the moment the best
plan seems to be to apply the above patch to ponie, and make a snapshot
release with it in.
I've attached the diff between TEST run for the two configurations.
Nicholas Clark
Leopold Toetsch
> If parrot's garbage collector is changed from the default (compacting, IIRC)
> to the either libc or malloc, then ponie only fails 6 tests.
> As I understand it parrot's default garbage collector will move data blocks
> owned by PMCs. However, all of the PMCs ponie generates do not have gc-owned
> data attached to them, so there should be no difference.
Sure?
No PerlHash, PerlArray, PerlString?
No pointers to string's data?
...
All PMCs are anchored properly?
Anyway, to sort out this kind of bugs please provide for ponie two
command line options with these equivalents in imcc/main.c:
" -G --no-gc\n"
" --gc-debug\n"
Please UTSL for details. The first turns off DOD & GC, the second enables
a switch GC_DEBUG (for which there is an envirnonment setting too:
if (is_env_var_set("PARROT_GC_DEBUG"))
Turning off DOD/GC normally shows, if the error is related to that.
Turning on GC_DEBUG does more DODs, e.g. in each string_compare that is
anywhere, where a hash is searched for example.
leo
Nicholas Clark
> Nicholas Clark <ni...@ccl4.org> wrote:
>
> > If parrot's garbage collector is changed from the default (compacting, IIRC)
> > to the either libc or malloc, then ponie only fails 6 tests.
>
> > As I understand it parrot's default garbage collector will move data blocks
> > owned by PMCs. However, all of the PMCs ponie generates do not have gc-owned
> > data attached to them, so there should be no difference.
>
> Sure?
> No PerlHash, PerlArray, PerlString?
> No pointers to string's data?
> ...
> All PMCs are anchored properly?
Yes. Arthur and I got it down to the appended test case, which is pure C
embedding and extending parrot.
> Anyway, to sort out this kind of bugs please provide for ponie two
> command line options with these equivalents in imcc/main.c:
>
> " -G --no-gc\n"
> " --gc-debug\n"
>
> Please UTSL for details. The first turns off DOD & GC, the second enables
> a switch GC_DEBUG (for which there is an envirnonment setting too:
>
> if (is_env_var_set("PARROT_GC_DEBUG"))
>
> Turning off DOD/GC normally shows, if the error is related to that.
> Turning on GC_DEBUG does more DODs, e.g. in each string_compare that is
> anywhere, where a hash is searched for example.
I hacked this into the ponie source directly for testing. With GC disabled
on the default GC ponie only fails 4 tests (two related to exit codes of ``)
However, the appended test program will segfault (by default) eg:
$ valgrind ./stress_parrot 5000
==9478== Memcheck, a memory error detector for x86-linux.
==9478== Copyright (C) 2002-2003, and GNU GPL'd, by Julian Seward.
==9478== Using valgrind-2.1.0, a program supervision framework for x86-linux.
==9478== Copyright (C) 2000-2003, and GNU GPL'd, by Julian Seward.
==9478== Estimated CPU clock rate is 2802 MHz
==9478== For more details, rerun with: -v
==9478==
==9478== warning: Valgrind's pthread_attr_destroy does nothing
==9478== your program may misbehave as a result
==9478== warning: Valgrind's pthread_attr_destroy does nothing
==9478== your program may misbehave as a result
Hello world
==9478== Conditional jump or move depends on uninitialised value(s)
==9478== at 0x805A17F: compact_pool (src/resources.c:301)
==9478== by 0x8059F64: mem_allocate (src/resources.c:149)
==9478== by 0x805A61E: Parrot_reallocate (src/resources.c:500)
==9478== by 0x807227B: expand_hash (src/hash.c:529)
==9478==
==9478== Conditional jump or move depends on uninitialised value(s)
==9478== at 0x805A17F: compact_pool (src/resources.c:301)
==9478== by 0x8059F64: mem_allocate (src/resources.c:149)
==9478== by 0x805A848: Parrot_allocate_string (src/resources.c:634)
==9478== by 0x806B694: string_make (src/string.c:379)
==9478==
==9478== Invalid read of size 4
==9478== at 0x80A2D8C: get_free_object (src/smallobject.c:226)
==9478== by 0x8050A43: get_free_buffer (src/headers.c:87)
==9478== by 0x8050E11: new_string_header (src/headers.c:330)
==9478== by 0x806B645: string_make (src/string.c:367)
==9478== Address 0x44DF0BEC is not stack'd, malloc'd or free'd
==9478==
==9478== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==9478== Address not mapped to object at address 0x44DF0BEC
==9478== at 0x80A2D8C: get_free_object (src/smallobject.c:226)
==9478== by 0x8050A43: get_free_buffer (src/headers.c:87)
==9478== by 0x8050E11: new_string_header (src/headers.c:330)
==9478== by 0x806B645: string_make (src/string.c:367)
==9478==
==9478== ERROR SUMMARY: 6177 errors from 3 contexts (suppressed: 0 from 0)
==9478== malloc/free: in use at exit: 4704588 bytes in 129 blocks.
==9478== malloc/free: 135 allocs, 6 frees, 5087348 bytes allocated.
==9478== For a detailed leak analysis, rerun with: --leak-check=yes
==9478== For counts of detected errors, rerun with: -v
Segmentation fault
It crashes sooner (need less PMCs) with GC debugging turned on (quelle
surprise), and doesn't crash with GC disabled.
Neither Arthur nor myself know where to start in debugging parrot's GC.
Nicholas Clark
/* Needed to turn off GC */
#if 0
#include "parrot/parrot.h"
#endif
#include "parrot/embed.h"
#include "parrot/extend.h"
#include <stdio.h>
#include <stdlib.h>
Parrot_PMC make_a_pmc(Parrot_Interp interpreter) {
Parrot_Int type = Parrot_PMC_typenum(interpreter, "Integer");
Parrot_PMC p;
p = Parrot_PMC_new(interpreter, type);
Parrot_register_pmc(interpreter, p);
return p;
}
int main (int argc, char**argv) {
int count;
Parrot_Interp interpreter = Parrot_new(0);
Parrot_init(interpreter);
#if 0
/* Turn off GC */
interpreter->DOD_block_level++;
interpreter->GC_block_level++;
#endif
if (argc > 1) {
count = atoi(argv[1]);
} else {
count = 1000000;
}
printf ("Hello world\n");
while (count--) {
make_a_pmc(interpreter);
}
printf ("Goodbye world\n");
return 0;
}
/* Compile with
gcc -Iinclude -Wall -o stress_parrot stress_parrot.c blib/lib/libparrot.a
-lm -ldl -lpthread # Don't need these on *BSD :-)
*/
Leopold Toetsch
> On Thu, Mar 11, 2004 at 10:33:24PM +0100, Leopold Toetsch wrote:
>> All PMCs are anchored properly?
> Yes. Arthur and I got it down to the appended test case, which is pure C
> embedding and extending parrot.
I already had mailed earlier with Arthur about that very problem. Parrot
needs a stack_limit (interprer->lo_var_ptr) for stack tracing. This
includes tracing processor registers which are placed on the stach in
trace_system_areas(). When this stack limit isn't set, stack walking can
not be done and all PMCs in hardware CPU registers and on the stack are
missed, which normally leads to ugly DOD bugs - they are really hard to
trace down.
So you have two possibilities to set the stack limit:
interpreter->lo_var_ptr = &interpreter; // a local in the outermost
// stack frame
or better, you run all your code through the wrapper:
- Parrot_run_native()
which enters a run loop after doing normal initialization (which
includes setting up a Parrot_exception which is used for exception
handling. The ops that get run are "enternative <yourcode>" ; end; "
Below is a working revision of your code.
(I know, that extend.c is missing some bits but that shouldn't be the
problem, we have just to add it)
leo
/* Needed to turn off GC */
#if 1
#include "parrot/parrot.h"
#endif
#include "parrot/embed.h"
#include "parrot/extend.h"
#include <stdio.h>
#include <stdlib.h>
Parrot_PMC make_a_pmc(Parrot_Interp interpreter) {
Parrot_Int type = Parrot_PMC_typenum(interpreter, "Integer");
Parrot_PMC p;
p = Parrot_PMC_new(interpreter, type);
Parrot_register_pmc(interpreter, p);
return p;
}
static opcode_t*
run( Parrot_Interp interpreter, opcode_t *cur_op, opcode_t *start)
{
int count = REG_INT(5); // fake argv passing - its normally P5
printf ("Hello world %d\n", count);
while (count--) {
if (! (count & 0xfff)) {
printf(".");
fflush(stdout);
}
make_a_pmc(interpreter);
}
printf ("Goodbye world\n");
return NULL;
}
int
main (int argc, char**argv) {
int count;
Parrot_Interp interpreter = Parrot_new(0);
Parrot_init(interpreter);
#if 0
/* Turn off GC */
interpreter->DOD_block_level++;
interpreter->GC_block_level++;
#endif
if (argc > 1) {
count = atoi(argv[1]);
} else {
count = 1000000;
}
REG_INT(5) = count ; // fake argv passing
Parrot_run_native(interpreter, run);
Parrot_exit(0);
Arthur Bergman
On 12 Mar 2004, at 19:26, Leopold Toetsch wrote:
> Nicholas Clark <ni...@ccl4.org> wrote:
>> On Thu, Mar 11, 2004 at 10:33:24PM +0100, Leopold Toetsch wrote:
>>> All PMCs are anchored properly?
>
>> Yes. Arthur and I got it down to the appended test case, which is
>> pure C
>> embedding and extending parrot.
>
> I already had mailed earlier with Arthur about that very problem.
> Parrot
> needs a stack_limit (interprer->lo_var_ptr) for stack tracing. This
> includes tracing processor registers which are placed on the stach in
> trace_system_areas(). When this stack limit isn't set, stack walking
> can
> not be done and all PMCs in hardware CPU registers and on the stack are
> missed, which normally leads to ugly DOD bugs - they are really hard to
> trace down.
>
I think this idea is flawed when it comes to embedding and extending.
Parrot should never walk the stack outside of itself (the embedding
applications stack should be off limit).
Also note that in this example, there never are any dead objects to
find.
> So you have two possibilities to set the stack limit:
>
> interpreter->lo_var_ptr = &interpreter; // a local in the outermost
> // stack frame
>
I can't do this from embedding space since the internals of interpreter
are not known.
> or better, you run all your code through the wrapper:
>
> - Parrot_run_native()
> which enters a run loop after doing normal initialization (which
> includes setting up a Parrot_exception which is used for exception
> handling. The ops that get run are "enternative <yourcode>" ; end; "
>
I am sorry, but all I can say is yuck, you can't expect embedders to
have to wrap their main in Parrot.
> Below is a working revision of your code.
>
> (I know, that extend.c is missing some bits but that shouldn't be the
> problem, we have just to add it)
>
> leo
>
>
> /* Needed to turn off GC */
> #if 1
> #include "parrot/parrot.h"
> #endif
>
This is the luxury I cannot do :(
In fact, as an embedder, I cannot be sure I always have a local
variable on a stack frame available to me, so I will end up having to
set lo_var_ptr every time I call into the parrot API, hence I suggest
this setting should be in extend.c and embed.c
Arthur
Leopold Toetsch
> On 12 Mar 2004, at 19:26, Leopold Toetsch wrote:
>> ... When this stack limit isn't set, stack walking can not be done
>> and all PMCs in hardware CPU registers and on the stack are missed,
>> which normally leads to ugly DOD bugs - they are really hard to trace
>> down.
> I think this idea is flawed when it comes to embedding and extending.
> Parrot should never walk the stack outside of itself (the embedding
> applications stack should be off limit).
If you are using Parrot_run_native(), Parrot sets the stack limit to the
level at it's run loop. (This was the final solution at the Parrot BOF
on Sat at YAPC EU - if you remember :) This is the "stack of itself".
But you are missing the point: If the limit is *not* set, Parrot doesn't
do any stack walking, which implies *no* walking of hardware CPU
registers.
> Also note that in this example, there never are any dead objects to
> find.
So your are sure, that your compiler never puts a local into a CPU
register, that in all intermediate function calls the returned PMC is
already anchored in the root set?
That's just not true. While there are no dead objects, there are
unanchored objects, which get happily destroyed, because Parrot doesn't
find them to be alive.
Your test program and mine are showing that your assumptions are wrong.
> I can't do this from embedding space since the internals of interpreter
> are not known.
>> or better, you run all your code through the wrapper:
>>
>> - Parrot_run_native()
>> which enters a run loop after doing normal initialization (which
>> includes setting up a Parrot_exception which is used for exception
>> handling. The ops that get run are "enternative <yourcode>" ; end; "
> I am sorry, but all I can say is yuck, you can't expect embedders to
> have to wrap their main in Parrot.
Take it or leave it. Its one function call. Its easy. It does the Right
Thing for setting up everything. What's the problem?
>> #if 1
>> #include "parrot/parrot.h"
>> #endif
>>
> This is the luxury I cannot do :(
You had snipped away my comment, that the extending interface needs some
extensions, *if* we finally agree, what are your needs.
> In fact, as an embedder, I cannot be sure I always have a local
> variable on a stack frame available to me, so I will end up having to
> set lo_var_ptr every time I call into the parrot API, hence I suggest
> this setting should be in extend.c and embed.c
Providing a function call that sets the stack limit ins't the problem.
Doing it right is the problem. First you should paint a picture of
Ponie's stack layout: Where is the interpreter constructed, where do you
call into Parrot ... The range from the stack limit down the call chain
to the actual call into Parrot API has to cover all possible auto
(stack) Buffer and PMC variables.
> Arthur
leo