Less trouble with lost interrupts
AchimNohl
it is a true story. A brand new Linux based touch-screen operated
internet device (I cannot tell the name here) has shown some really
annoying behavior with regard to the touchscreen. Every now and then,
touchscreen events have been dropped. Every time the user was tapping
on the screen, he was not sure whether there is just a normal
processing delay or the device has again simply ignored him. This made
the device almost unusable resulting in too many taps, opening
unwanted applications etc. The company’s engineering team which was
developing the device finally used a hand crafted LEGO robot, a logic
analyzer and many printfs to try to spot the problem.
One of our interns at that time has been able to reproduce (and
report) the problem using a Virtual Platform very quickly. It boiled
down to the fact that under certain rare circumstances, touchscreen
interrupts have been wrongly cleared while they were masked and not
handled yet.
Using a Virtual Platform you can assert the operation of hardware and
the interaction with software. This assertion is done using the
ability of a VP to observe what is going on in hardware and software
without interfering with the system. Meaning, in a non-intrusive way
hardware and software states can be read and conclusions can be drawn
triggered by events such as breakpoints or watchpoints. Essentially,
you can automate what you would typically do using a debugger when
debugging a scenario. But, with automation you can check millions of
those scenarios
This way you can implement a script that basically checks whether a
peripheral is programmed according to the specification (e.g.
programmer manual) or if a suspicious condition is triggered that
would require the attention of a human-being. The assertions enabled
by virtual platforms go beyond what you can assert in software. First,
you can inspect each and every state in hardware and software in any
peripheral and on any core. Second you can perform the inspection,
triggered by events asynchronous/concurrent to software such as
interrupt signals coming from the external world.
I have created a small example that asserts whether interrupts are
ignored that you can use with the VP on the cloud. I am going to post
and explain it tomorrow.
Regards,
Achim
AchimNohl
I am a little late this time with the follow up as I have been at
DATE. I have participated in a panel about embedded software testing
and interestingly what we are talking about here has been one of the
topics in the panel “How to assert the interaction of hardware and
software”.
To recap, our goal was to identify when the interrupt controller is
“ignoring” interrupts. Meaning, we want to identify any point in time
where the interrupt controller receives an interrupt (e.g. from the
touchscreen) and masks it out. We want to then assert if this
interrupt is cleared before it is unmasked and signaled to the
software again. In this example the touchscreen interrupt would get
lost.
We will realize this assertion using two watchpoints. One watchpoint
is observing when interrupts are coming in, and another watchpoint is
observing when interrupts are going out (to the core). This allows us
to track each in-coming interrupt and check whether it ever reaches
the output signal. If not, it is dropped. Whether this drop has been
intentionally or not depends on what the programmer intended todo.
But, we want to give an alert.
Thus, we will maintain a 32bit status variable in our Virtual Platform
script. The variable, irq_pending, will keep all incoming interrupts
that have not been signalled to the core yet. The pseudo code looks as
follows:
Watchpoint and callback: Whenever a new interrupt comes in and the
32bit interrupt input changes:
- Read the active interrupts into irq_in_now (32bit word)
- Compute interrupts that changed status
o irq_changed = irq_in_now BITWISE-XOR irq_pending
- Compute new interrupts that just went from false to true
o irq_went_on = irq_changed & irq_pending
- Compute the pending interrupts that have been cleared and just went
from true to false
o irq_went_off = irq_changed & irq_now
- If irq_went_off NOT ZERO then
o A pending interrupt was cleared
o Stop the simulation
- Compute the new pending interrupts
o irq_pending = irq_pending BITWISE-OR irq_went_on
Watchpoint and callback: Whenever the interrupt controller output
signal to the processor changes:
- Read the interrupt status register (shows interrupts that are
signalled to the core)
o irq_out_now = IRQ_STATUS_REGISTER
- Remove the signalled interrupts from the pending interrupts
o irq_pending = (BITWISE NOT irq_out_now) & irq_pending
You can find the real script:
http://groups.google.com/group/virtual-platform-users/web/example_show_dropped_irqs.tcl
Just download it on the VP in the cloud and enable it using “source
example_show_dropped_irqs.tcl” in the VPA shell.
This is what the script shows during operation:
11038277505000 : Pending irqs : i:01000000.
11046923690000 : Ingoing irqs : i:01200000.
11046923690000 : |->just went on : i:00200000.
11046923690000 : Pending irqs : i:01200000.
11046923690000 : Pending irqs : i:01200000.
11046923715000 : Ingoing irqs : i:01000000.
11046923715000 : Ingnoring irqs (!): i:00200000.
Warning: Those irqs have been cleared without ever being signalled out
In have triggered the assertion using the fault injection capabilities
of a VP. I have just enabled port_21 (set to true) of the secondary
interrupt controller i_SIC. The software does not know about this
interrupt and just clears it.
In essence, you can assert the operation or interaction of any
software or hardware module using the Virtual Platform scripting. This
was just one example and I will describe some more soon.
Let me know what you think. Is there anything you want to get asserted
in your system?
Best regards,
Achim