How do I debug freezes?
Vladimir Vysotsky
I'm pretty new to vxWorks and embedded programming (about two
months :), so, when something strange happens, I often don't
know what to do.
The particular problem I'm stuck with right now is the following.
I have an MPC8260, a 60x to PCI bridge (Tundra Powerspan) and
several DSPs on the PCI bus. The code I'm trying to debug initializes
one of the DSPs, loads some software into it and tries to run it.
It also prints a lot of debug output using printf, which goes to
the console. I run it from the shell on the same console.
Sometimes the code executes fine, but sometimes it causes global
freeze - shell, network, everything. Of course, wdb doesn't help
in such cases. Using visionICE I've been able to look at the
stacks of the tasks. Strangely, tShell is blocked on something like
dsp_init_function->printf->...->tyWrite->semTake->semBtake.
So, my question is not really "what happens" (although I'd be happy
if somebody could give a clue), but rather "how do I find out what
happens". Are there any vxWorks- or Motorola/PowerPC-specific
debugging tips and techniques applicable to my problem?
Vladimir
Hwa Jin Bae
1. Use logic analyser to see the last batch of PC and memory data.
2. Use ICE, or PromICE
3. Use WindView and Stethoscope
4. If your system is VME based, you can write your own post mortem debugger.
Another CPU on the bus can copy the memory of the dead board and analyze it
later. The way I did before, I had to modify GDB (write a new backend) so
that it can understand flat memory dump of the target board. There is some
code I wrote that does something like this in the VxHacks repository.
--
Hwa Jin Bae
mailto:h...@pso.com
http://www.pso.com
John
There are some general hints and tips I can share about debugging
freezes for PPC... I've seen enough of them! I cannot give much of a
clue about your current problem though, sorry...
1) The ICE/Probe should be able to tell you the current location of
the PC, irrespective of the current task. Freezes are not often caused
by a task (though they can be). More often they are an ISR or similar
"invisible" code. So, check the PC, but don't be fooled by the current
task ID. BTW, the shell waiting on the semaphore is not unusual, it is
just trying to display a message.
2) When using the 'printf' debug strategy there are a couple of tips:
(a) consider using logMsg() instead of printf() - handy in case of
stack problems in the current task; (b) add a taskDelay() call after
the printf/logMsg call to allow the output to be sent to the
(relatively) slow serial port.
3) If you suspect interrupt handlers, or the quantity of debug output
is so high that the timing is changing, try writing a position
indicator to a special location. Now, since the bootrom will be
running after the freeze, you need a safe location. I use locations
going down from 0xfc under VxWorks 5.4, and locations starting at
0x3000 for VxWorks AE 1.1. Why the difference? Well, the pages below
0x3000 are "invisible" under AE, even to ISRs.
4) Once you have it narrowed down a little, you can use the memory
trick in (3) to store variable values too so you get an idea of the
state just before the crash.
5) When starting your task from the shell, spawn it as its own task
rather than executing it directly from the shell. Also, use
taskSpawn() so you can set the stack size to be very large (I tend to
favour 1MB in these cases, but then I have plenty of memory, usually
;-).
6) Final tip, and it may be less useful for your case but is handy
when debugging initial board start up, use calls to reboot() with a
parameter of 1 (i.e. no auto-boot), to track down the exact location
of the failure. First place I start when looking at boot problems,
using a binary search strategy, is to determine if execution reaches
usrRoot().
Hope something there helps with your problem,
John...
"Vladimir Vysotsky" <tri...@noir.crocodile.org> wrote in message news:<3c93c...@nopics.sjc>...
Vladimir Vysotsky
> There are some general hints and tips I can share about debugging
> freezes for PPC... I've seen enough of them! I cannot give much of a
> clue about your current problem though, sorry...
John,
thanks a lot - all your suggestions may be very helpful in future.
I didn't realize that task stacks in PPC/vxWorks are normally that
small - something I'll definitely need to take into account.
By the way - after three days I've finally debugged the problem,
and it appeared to be caused by hardware, not by software.
PCI controller generated a signal (ARTRY~) during PCI
configuration cycles, which, according to Motorola errata
document, "confused" the comm processor in 8260 to the point of
completely stopping console and Ethernet I/O. Go figure...
Vladimir.