Segmentation fault (teardown realated)

[Andreas Stenius]

Hi,

Have this strange error, that I'm trying to narrow down.

First, a nasty gdb bt:

#0 0x00000000 in ?? ()
#1 0x0040d14f in MemoryLeakDetector::reportFailure (this=0x8003ba10,
message=0x413e6c "Deallocating non-allocated memory\n",
allocFile=0x413e60 "<unknown>", allocLine=0, allocSize=0,
allocAllocator=0x41d9b0, freeFile=0x413e60 "<unknown>", freeLine=0,
freeAllocator=0x41d9ec) at src/CppUTest/MemoryLeakDetector.cpp:315
#2 0x0040d71e in MemoryLeakDetector::deallocMemory (this=0x8003ba10,
allocator=0x41d9ec, memory=0x41d9ec,
file=0x413e60 "<unknown>", line=0, allocatNodesSeperately=false)
at src/CppUTest/MemoryLeakDetector.cpp:424
#3 0x0040d7cb in MemoryLeakDetector::deallocMemory (this=0x8003ba10,
allocator=0x41d9ec, memory=0x41d9ec,
allocatNodesSeperately=false) at src/CppUTest/MemoryLeakDetector.cpp:435
#4 0x00404437 in mem_leak_operator_delete (mem=0x41d9ec) at
src/CppUTest/MemoryLeakWarningPlugin.cpp:78
#5 0x00404622 in operator delete (mem=0x41d9ec) at
src/CppUTest/MemoryLeakWarningPlugin.cpp:136
#6 0x8003ba10 in ?? ()
#7 0x00402890 in helperDoTestTeardown (data=0x8003cc80) at
src/CppUTest/Utest.cpp:90
#8 0x0040bc07 in PlatformSpecificSetJmp (function=0x402878
<helperDoTestTeardown(void*)>, data=0x8003cc80)
at src/Platforms/Gcc/UtestPlatform.cpp:58
#9 0x00403dc3 in Utest::run (this=0x8003cc80) at src/CppUTest/Utest.cpp:478
#10 0x00402bdd in UtestShell::runOneTest (this=0x41b01c,
plugin=0x28ab58, result=...) at src/CppUTest/Utest.cpp:198
#11 0x004028d8 in helperDoRunOneTest (data=0x28aa84) at
src/CppUTest/Utest.cpp:110
#12 0x0040bc07 in PlatformSpecificSetJmp (function=0x402892
<helperDoRunOneTest(void*)>, data=0x28aa84)
at src/Platforms/Gcc/UtestPlatform.cpp:58
#13 0x00402af0 in UtestShell::runOneTestWithPlugins (this=0x41b01c,
plugin=0x28ab58, result=...) at src/CppUTest/Utest.cpp:172
#14 0x0040acd2 in TestRegistry::runAllTests (this=0x41b0e0,
result=...) at src/CppUTest/TestRegistry.cpp:62
#15 0x0040265a in CommandLineTestRunner::runAllTests (this=0x28abb8)
at src/CppUTest/CommandLineTestRunner.cpp:103
#16 0x00402483 in CommandLineTestRunner::runAllTestsMain
(this=0x28abb8) at src/CppUTest/CommandLineTestRunner.cpp:80
#17 0x00402359 in CommandLineTestRunner::RunAllTests (ac=1,
av=0x28ac30) at src/CppUTest/CommandLineTestRunner.cpp:62
#18 0x00401190 in main (argc=1, argv=0x28ac30) at
../../../../../src/tests/run_tests.cpp:6
warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

At some point, I got the failure message out, but most of the time, it
just seg faults. And all this started happening when I added a
seemingly harmless call from the group teardown() function.
If I comment that call out, it passes just fine.
I've #if 0'ed all tests and added a simple TEST(Data, Foo) { CHECK(1);
} just to trigger the setup/teardown.

I have a bunch of local stuff in the test group class, which may be
relevant (I haven't commented all that out yet, hoping someone spots
something obvious for me here.... ;)

I understand there's a lot of details not covered here, but I'll keep
digging some more first, in case it isn't something too obvious
here...

//Andreas

[James Grenning]

I would immediately suspect trying to free or delete something that was not malloc's or new'd. Also it can be a bad pointer dereference.

Run your tests from the command line, with -v to confirm which test set faults.

The use -g GroupName and/or -n TestName command line options to see if it just one test, which tests, which test groups. That will give you more clues.

thanks, James

--------------------------------------------------------------------------------------------
James Grenning Author of TDD for Embedded C
www.renaissancesoftware.net http://pragprog.com/titles/jgade/
www.renaissancesoftware.net/blog
www.twitter.com/jwgrenning

[Ja...@grenning.net]

Is it a null pointer dereferance?

-----
Sent from my brain pal

[Andreas Stenius]

No, this is seriously weird.

I have created a new empty test group for this to get rid of all the extras.

I have:

bsp_tests.cpp:

extern "C"
{
#include "init.h"
}

#include "CppUTest/TestHarness.h"

TEST_GROUP(BSP)
{
TEST_SETUP()
{
}

TEST_TEARDOWN()
{
}
};

TEST(BSP, Init)
{
bsp_init();
bsp_shutdown(); // comment this line to avoid the seg fault
}

and in init.c:

static int initialized = 0;
void bsp_init(void)
{

if (initialized++)
return;


}

void bsp_shutdown(void)
{
initialized = 0;
}


I still have some libs linked in, that may muddle the waters... hmm, I
find this vary strange indeed.

//Andreas

2012/7/1 Andreas Stenius <andreas...@astekk.se>:

[Andreas Stenius]

No (not from my code at least).

Single stepping doesn't seem to give me more at the moment, will see
what I can dig up stepping asm after finishen the test body...

some gdb logging from my stripped test group described in a previous mail:

$ cat gdb.txt
Breakpoint 1 at 0x418e03: file ../../../../src/bsp/init.c, line 37.
Starting program:
/work/Mobitron/proj/fat90v2/appl/build/pc/src/bsp/tests/bsp_tests
[New Thread 4728.0xad4]
[New Thread 4728.0x1fec]

Breakpoint 1, bsp_init () at ../../../../src/bsp/init.c:37
37 if (initialized++)
#0 bsp_init () at ../../../../src/bsp/init.c:37
#1 0x004011ab in TEST_BSP_Init_Test::testBody (this=0x8003cc80) at
../../../../../src/bsp/tests/bsp_tests.cpp:21
#2 0x0040d8f6 in helperDoTestBody (data=0x8003cc80) at
src/CppUTest/Utest.cpp:85
#3 0x00416c87 in PlatformSpecificSetJmp (function=0x40d8de
<helperDoTestBody(void*)>, data=0x8003cc80)
at src/Platforms/Gcc/UtestPlatform.cpp:58
#4 0x0040ee30 in Utest::run (this=0x8003cc80) at src/CppUTest/Utest.cpp:469
#5 0x0040dc5d in UtestShell::runOneTest (this=0x42f01c,

plugin=0x28ab58, result=...) at src/CppUTest/Utest.cpp:198

#6 0x0040d958 in helperDoRunOneTest (data=0x28aa84) at
src/CppUTest/Utest.cpp:110
#7 0x00416c87 in PlatformSpecificSetJmp (function=0x40d912

<helperDoRunOneTest(void*)>, data=0x28aa84)
at src/Platforms/Gcc/UtestPlatform.cpp:58

#8 0x0040db70 in UtestShell::runOneTestWithPlugins (this=0x42f01c,

plugin=0x28ab58, result=...) at src/CppUTest/Utest.cpp:172

#9 0x0041626e in TestRegistry::runAllTests (this=0x4316e0,
result=...) at src/CppUTest/TestRegistry.cpp:62
#10 0x0040d6da in CommandLineTestRunner::runAllTests (this=0x28abb8)
at src/CppUTest/CommandLineTestRunner.cpp:103
#11 0x0040d503 in CommandLineTestRunner::runAllTestsMain
(this=0x28abb8) at src/CppUTest/CommandLineTestRunner.cpp:80
#12 0x0040d3d9 in CommandLineTestRunner::RunAllTests (ac=1,
av=0x28ac30) at src/CppUTest/CommandLineTestRunner.cpp:62
#13 0x00401190 in main (argc=1, argv=0x28ac30) at
../../../../../src/tests/run_tests.cpp:6
42 }
TEST_BSP_Init_Test::testBody (this=0x8003cc80) at
../../../../../src/bsp/tests/bsp_tests.cpp:22
22 bsp_shutdown();
bsp_shutdown () at ../../../../src/bsp/init.c:46
46 initialized = 0;
68 }
__static_initialization_and_destruction_0 (__initialize_p=-2147234688,
__priority=65535)
at ../../../../../src/bsp/tests/bsp_tests.cpp:23
23 }
19 TEST(BSP, Init)
23 }

Program received signal SIGSEGV, Segmentation fault.
0x8003c007 in ?? ()
A debugging session is active.

Inferior 1 [process 4728] will be killed.

Quit anyway? (y or n) error return
/netrel/src/gdb-7.3.50-3/gdb/windows-nat.c:1251 was 5



2012/7/1 Ja...@grenning.net <Ja...@grenning.net>:

[Andreas Stenius]

OK,

(Thanks for looking at it, James :)

It seems that the return stack gets corrupt, somehow, as it jumps off
into the weeds.

I have to tripple check my build, I guess, since I've emptied all
functions down to just referencing a static var and it still goes into
the bushes...

Here's my build as it is right now. But it will be some time before I
can get back to this as I'm off on vacation for three weeks now...

$ make V=1
g++ -DHAVE_CONFIG_H -I. -I../../../../../src/bsp/tests -I../../..
-I../../../../../deps/cpputest/include -I../../../../../src/tests
-DCPPUTEST_COMPILATION -DCPPUTEST_STD_CPP_LIB_DISABLED
-I../../../../../src/bsp/tests/.. -I../../../../../src -include
../../../../../deps/cpputest/include/CppUTest/MemoryLeakDetectorNewMacros.h
-include ../../../../../deps/cpputest/include/CppUTest/MemoryLeakDetectorMallocMacros.h
-g -O2 -MT foo-run_tests.o -MD -MP -MF .deps/foo-run_tests.Tpo -c -o
foo-run_tests.o `test -f '../../../../../src/tests/run_tests.cpp' ||
echo '../../../../../src/bsp/tests/'`../../../../../src/tests/run_tests.cpp
mv -f .deps/foo-run_tests.Tpo .deps/foo-run_tests.Po
g++ -DHAVE_CONFIG_H -I. -I../../../../../src/bsp/tests -I../../..
-I../../../../../deps/cpputest/include -I../../../../../src/tests
-DCPPUTEST_COMPILATION -DCPPUTEST_STD_CPP_LIB_DISABLED
-I../../../../../src/bsp/tests/.. -I../../../../../src -include
../../../../../deps/cpputest/include/CppUTest/MemoryLeakDetectorNewMacros.h
-include ../../../../../deps/cpputest/include/CppUTest/MemoryLeakDetectorMallocMacros.h
-g -O2 -MT foo-bsp_tests.o -MD -MP -MF .deps/foo-bsp_tests.Tpo -c -o
foo-bsp_tests.o `test -f 'bsp_tests.cpp' || echo
'../../../../../src/bsp/tests/'`bsp_tests.cpp
mv -f .deps/foo-bsp_tests.Tpo .deps/foo-bsp_tests.Po
g++ -include ../../../../../deps/cpputest/include/CppUTest/MemoryLeakDetectorNewMacros.h
-include ../../../../../deps/cpputest/include/CppUTest/MemoryLeakDetectorMallocMacros.h
-g -O2 -L../../../deps/cpputest/lib -L../../../src/tests/mocks
-L../../../src/tests -L./.. -Xlinker -Map=foo.exe.map -o foo.exe
foo-run_tests.o foo-bsp_tests.o -lErrnoTest -lCppUTest -lfat90_bsp


//Andreas


2012/7/1 Andreas Stenius <g...@astekk.se>:

[Ja...@grenning.net]

Sounds like you are making progress.

You are running tests in your target?  You may want to move bsp_init and bsp_shutdown into main, surrounding RunAllTests

[Bas Vodde]

Hi Andreas,

Yah these kinds of crashes often look like something wrong on the stack.

Do you think you could share me some code (in file) that crashes so I can run it on my machine?

Bas

ps. Have a good vacation!

[Andreas Stenius]

I'll try to produce a simple working test case, in the mean time
trying to figure out what is going on.

It seems that gcc is stumbling over it's feet here...

I look at the generated asm code for three different scenarios.

1, an empty test case, 2, a test case with a single call to a bsp_init
function, and 3, with an additional call to bsp_shutdown.

Here's the asm for the test case body (I keep a few lines extra at the
end to show some context there):

1 (the empty test case)

.globl __ZN18TEST_BSP_Init_Test8testBodyEv
.def __ZN18TEST_BSP_Init_Test8testBodyEv; .scl 2; .type 32; .endef
__ZN18TEST_BSP_Init_Test8testBodyEv:
LFB33:
.loc 1 20 0
LVL2:
pushl %ebp
LCFI6:
movl %esp, %ebp
LCFI7:
.loc 1 24 0
popl %ebp
LCFI8:
ret
LFE33:
.section .text$_ZN27TEST_GROUP_CppUTestGroupBSPD1Ev,"x"
.linkonce discard

2 (works OK, a single call)

.globl __ZN18TEST_BSP_Init_Test8testBodyEv
.def __ZN18TEST_BSP_Init_Test8testBodyEv; .scl 2; .type 32; .endef
__ZN18TEST_BSP_Init_Test8testBodyEv:
LFB33:
.loc 1 20 0
LVL15:
pushl %ebp
LCFI36:
movl %esp, %ebp
LCFI37:
subl $8, %esp
LCFI38:
.loc 1 23 0
leave
LCFI39:
.loc 1 21 0
jmp _bsp_init
LFE33:
.section .rdata,"dr"

3 (two calls, fails with a seg fault)

.globl __ZN18TEST_BSP_Init_Test8testBodyEv
.def __ZN18TEST_BSP_Init_Test8testBodyEv; .scl 2; .type 32; .endef
__ZN18TEST_BSP_Init_Test8testBodyEv:
LFB33:
.loc 1 20 0
LVL15:
pushl %ebp
LCFI36:
movl %esp, %ebp
LCFI37:
subl $8, %esp
LCFI38:
.loc 1 21 0
call _bsp_init
.loc 1 22 0
call _bsp_shutdown
LFE33:
.section .rdata,"dr"


My observations is that in case 1, we simply call ret, which is fine.
In case 2, we jmp to bsp_init, in effect being a tail call (the ret
from bsp_init lands in the frame calling the test body) and case 3, is
call'ing both functions, again, that is fine, what is not is that
there is no ret instruction in the test body!
So when bsp_shutdown returns, we land in a static destructor that
takes two int args (now with bogus values, naturally).

This has to be a gcc bug. I'll just see if I can work around it (or
look for a different version and see if it improves the situation).
I hope I'm wrong.

Thankful for any insight or ideas you may have :)

Cheers,
Andreas

2012/7/2 Bas Vodde <ba...@odd-e.com>:

[Bas Vodde]

Wow Andreas

If what you say is true, then this gcc bug would pop up everywhere, right?

It seems something else must have caused this…

Is it a normal gcc or a x-compiling one?

Let me know what you discover. Interesting.

Bas

[Andreas Stenius]

Heh,

I guess you missed my mail I sent 5 min after that one.

Just after pressing send on that mail, I noticed that I had marked my
bsp_shutdown with a noreturn attribute...
so it was all on me :/ (puh)

//Andreas

2012/8/1 Bas Vodde <ba...@odd-e.com>:

[Andreas Stenius]

Bah^3,

replying to myself wasn't what I meant, really.

//Andreas


---------- Forwarded message ----------
From: Andreas Stenius <g...@astekk.se>
Date: 2012/7/31
Subject: Re: Segmentation fault (teardown realated)
To: Andreas Stenius <g...@astekk.se>


BAH!!!

Sorry. I just took a look at the declaration for bsp_shutdown in the
.h file, and I spotted a __attribute__ ((noreturn)) in there....

That explains quite a lot.
And yes, commenting out that noreturn attribute, the produced asm uses
a jmp instead of a call to bsp_shutdown, and voilá, it works.

I'm glad that it wasn't a gcc bug. I can live with the embarrassment :p

//Andreas

Ps.
Wow, taking a really long break helped :)
Only spent an hour to solve it now when I got back to it.

2012/7/31 Andreas Stenius <g...@astekk.se>:

[Bas Vodde]

Ah, I did miss that one :)

Anyways, good you found it :P Be careful with your attributes ;P

Bas

[Bas Vodde]

No wonder I missed it, I can't read your email :)

Anyways, I'm glad it solved. I'll still get back to you related to the compare memory buffer…

Bas

[Andreas Stenius]

Uhmm.... what compare memory buffer?

I do recall me writing a memcmp check, but I can't recall having an
outstanding question regarding that... ?
https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=7a0b85e205a03a0e87d9c33ccf9d0ff1bfce5f20

It's a quick 'n dirty hack - feel free to take it if you like it :)
(but in that case, don't miss the improved feedback,
https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=3178e699147ab7a1e15f7012829cacada0c3b341

)

//Andreas

2012/8/1 Bas Vodde <ba...@odd-e.com>:
>

[Bas Vodde]

Yah, I meant that. I think CppUTest is missing a memory assert. It is common to need it when working on more embedded or low-level software.

Uhm.. but… did you also have tests for it?

Bas

[James Grenning]

Interesting find: __attribute__

Just curious:
I guess the embedded compiler supports __attribute__ too. Seems to be a gcc thing.
The bps_shutdown in your tests does return. Does your production bps_shutdown return?

[Andreas Stenius]

It's the same function (code) ... but it doesn't do much yet as it's
under development. And there will be differences, naturally.

And yes, the attributes work on the embedded target too (using a gcc-arm-eabi).

Although I'm using almost the same compiler on both host and target,
there are differences. Perhaps most notably in the libs,
but also architecture. More than once have I found a bug on the target
that worked just fine on the host.

I have a rather nice setup now (finally) where I simulate the hardware
on the host so I can run the whole app on the host with introspection
w regard to the hw parts.

Go to run, dinners ready. ;)

//Andreas

2012/8/1 James Grenning <ja...@grenning.net>:

[James Grenning]

Andreas

When you get a chance, tell us more about target bugs that worked on the host.

thanks, James

[Andreas Stenius]

OK,

Right now, I can only recall one specific that I still remember what
it was. And that made me add this comment:

/*
* BEWARE: arg pointers must point to int sized data!!
*/
int pic_request(pic_command_t cmd, ...)

And that has to do with how the va_list is used, naturally. But on the
host (a Windows box running cygwin) it worked when I passed a pointer
to a short (or was it to a char even),
but on the target, when dereferencing the pointer, the address was
slightly off target.


Ah, here was my change for that:

@@ -326,9 +329,15 @@

int pic_set_alarm(pic_alarm_t alarm, struct tm *tm, pic_alarm_mask_t *mask)
{
- int res;
+ int res, mask_rsp;
pic_command_t cmd;

+ if (!mask)
+ {
+ errno = EINVAL;
+ return -1;
+ }
+
switch (alarm)
{
case PIC_ALARM_0: cmd = PIC_SET_ALARM_0; break;
@@ -349,10 +358,14 @@
&tm->tm_min,
&tm->tm_hour,
&tm->tm_mday,
- mask
+ &mask_rsp
);

+ if (0 == res)
+ *mask = mask_rsp;
+
pic_time_to_tm(tm);
+
return res;
}

@@ -427,19 +440,26 @@
return res;
}


The alarm mask type is a simple enum:

typedef enum {
PIC_ALARM_MASK_DAY = 0, /* check day, hour, minute & second */
PIC_ALARM_MASK_HOUR = 1, /* check hour, minute & second */
PIC_ALARM_MASK_MIN = 3, /* check minute & second */
PIC_ALARM_MASK_SEC = 7 /* check second */
} pic_alarm_mask_t;

Not sure how many bytes is allocated for that type, but less than four for sure.

I've had a few other issues as well, but I can't really recall them
now. Probably similar in nature as this one...

[James Grenning]

Did the tests point out the portability problem, making it easy to find? Or was it still a pain to find?

thanks for the details.

I also found a bug in the standard library delivered with an embedded complier.

[Andreas Stenius]

It was rather spot on.

From the tests I could see that the value sent was not the value
received. Didn't take too long to isolate the error of my ways... ;)


2012/8/1 James Grenning <ja...@grenning.net>: