[virtual-platform-users:56] Continuous integration and test of hardware dependent software

[AchimNohl]

Dear all,

Initially I wanted to follow up on my recently started topic "Virtual
Prototype enabled product conformance testing". However, I have spent
the quite some time debugging a defect in OS that has been working
forever until I had to package and ship it last week to somebody. I
did a quick test before the packaging and it did not do what it should
do. So what has changed?! Since the last time that I have tried it
out, I have been updating my compiler, and got a new revision of the
underlying platform and I remember that I did some experiments for a
colleague that I have reverted (as far as I remember). After a week of
debugging I found out that I have messed around with the timer
configuration resulting in the fact that the boot is so slow that I
though it is just doing nothing. If I would have been patient enough,
I would have seen in the context trace that timer interrupts are
coming in too slowly, but I did not:(

Of course I had no good tests available due to the usual excuse. "The
OS is not written in a way that it can be tested easily, so skip the
testing." This is not a good excuse and it costs a lot of time. But,
there is also some truth to it. How, do you test as a software
developer that the system is doing what it should do? Especially, an
OS is a complex piece of layered software interacting with
concurrently operating hardware. And, you have to live with the fact
that many software stacks have not been written in a way that you can
extract individual modules easily and test them independently.
However, to perform an efficient root cause analysis you have to break
down the testing on a fine granular level. But, you may need to-do the
testing in the system context. How can you create a test that checks
that the timer starts ticking after the software has configured it
within the context of the OS? There is no delay that you can wait in
the test, as the timer is not working.

A Virtual Platform offered me a different perspective. It allowed me
to create system level tests that observe and assert the operation of
the software and its interaction with the software. Thus, the tests
are not part of the embedded software. The tests are implemented using
the Virtual Platform scripting. The tests are based on functions basic
primitives such as program breakpoints, resource watchpoints,
callbacks, reading/evaluating platform resources.

In essence, it enables continuous integration and test during hardware
dependent software development in a very pragmatic and easy way!

I have created a small framework with useful helper functions that I
want to introduce. Those functions allow creating complex tests very
quickly. With the test framework I can assert the operation of
embedded software functions and I can assert the operation of the
hardware at the same time. Let me give you some examples:

Example (1): You want to create a test that gets trigger when
start_kernel is called.
program_test_trigger start_kernel
proc callback_start_kernel { } {
...
}

Example (2): In a test, you want to assert that the data-abort handler
at 0xffff0010 is not called before the kernel boot enters the user-
space where data-aborts are OK:
assert_address_not_called_until 0xffff0010 "Data
Abort" [address_barrier rest_init]

Example (3): In a test, you want to make sure that
versatile_timer_init is called before the user-space:
assert_address_called_before versatile_timer_init "Timer initialized
before user-space." [address_barrier rest_init ]

Example (4): In a test, you want to make sure that the timer is
enabled before calibrate_delay is called:
assert_peripheral_value_change_before i_TIMER_01/Count1/ControlReg
0xe0 "Timer 0 enabled before calibrate delay" [address_barrier
calibrate_delay]

Example (5): You want to trigger a test when the timer is enabled:
peripheral_value_test_trigger i_TIMER_01/Count1/ControlReg 0xe0
callback_timer_enabled

Example (6): In a test, you want to make sure that a timer interrupt
is handled after 100ms from when test tests gets triggered
proc callback_timer_enabled {} {
# make sure that our timer interrupts is handled witin 100ms from
now
assert_address_called_before versatile_timer_interrupt "Timer
interrupt first in 100ms " [time_barrier +100 ms]

}

These are just some examples and the primitives will need to be
extended. But, already those few primitives allow creating useful
hardware/software tests in a very easy way along with your software.
Having had this test available would have saved me a huge amount of
time. Considering, the large amount of driver, modules etc. in a
complex embedded software stack, the ability to test is key to manage
the risk appropriately.

I have posted the example and the status here:
http://groups.google.com/group/virtual-platform-users/web/example_vp_software_testing_linux.tcl

You can try it out in the cloud with e.g. the Android skin. In the
cloud, download the TCL file from the link above. Just launch the VP
and the skin and in the TCL shell enter:

source example_vp_software_testing_linux.tcl

I will follow up on this with some more explanations.

Best regards,
Achim

--
You received this message because you are subscribed to the Google
Groups "Virtual Platform-Users" group.
To unsubscribe from this group, send email to
virtual-platform-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/virtual-platform-users?hl=en