Scenario creation, simulation and assertion for full user level HW/SW validation

[AchimNohl]

Hi all,

In this discussion I want to start connecting the lines between
several topics that have been introduced in this group. My goal is to
apply our assertion framework in context of real world user scenarios
for system level HW/SW validation. In discussion [1] (see below), we
have introduced a solution how stimuli to the device can be recorded
and replayed to automate scenarios where user interaction (e.g.
touchscreen) or other environment stimuli (e.g. accelerometer, GPS,
modem) are required. The solution has been implemented based on
watchpoints on all necessary interfaces. Those watchpoints are
triggering some functions in the VP scripting framework that peek the
stimuli data and generate a replay script that pokes the data using
time-triggers back into the system. This way, complex user scenarios
can be easily recorded and repeated in a deterministic fashion for
regressions.

Now, we want to combine those scenarios with the assertion framework
that we have created and which has been discussed in [2] and which is
described in [3]. Our goal is to perform a full HW/SW system
validation. Some examples of things we want to test:
- When the user is not using the phone, then the LCD should dim after
N seconds and the CPU should not be loaded more than 2%
- When the user is touching the phone, then the LCD should go to
normal mode
- When the maps application is launched, then the GPS should be turned
on automatically
- When the maps application is terminated, then the GPS should be
turned off automatically
- When the phone goes to suspend mode, then all devices should go into
low(est) power mode

One could think of many, many more scenarios that need to be tested.
What is particular about the way of testing we are looking for? On top
of the unit test that hopefully happen for each layer in the SW stack
(driver, HAL, middleware, Android Framework, Application), we will
perform a full integration test. Our stimuli are performed at the
exact same level as the user would generate them. And, our assertions
are performed at the very end of the chain in the hardware models of
the system. This way we can complement the testing with full HSW/SW
integration tests performed at the user-level.

Let’s recap what we have been doing in our assertion framework [2]. As
everything else the framework is implemented using the VP scripting
capabilities. Based on software-, hardware- and time- breakpoints,
callbacks and the ability to peek/poke data from the HW and SW the
framework provides assertion primitives for:
- Assert that a specific hardware- or software-event does not occur
until user defined hardware-, software- or time-event
- Assert that a specific hardware- or software-event does occur before
a user defined hardware-, software- or time-event
- Cascading assertions
- Applying hardware conditions (e.g. pin/register/memory value) and
software conditions (e.g. process name)

The assertions are fully sufficient to assert the items presented
above. Now, we need to link the scenario creation with those
assertions. Therefore, I have extended the script for scenario
creation with the ability to add callbacks for e.g. assertions. E.g.
when the user performs a specific action such as turning the LCD to
bright, he can add a callback that triggers an assertion that checks
if the LCD HW state is bright or not.

I will continue on this and describe the details tomorrow. For now I
want to leave you with a small screencast that shows an example
scenario driven HW/SW validation:

http://screenr.com/LYm

What it does:
- We will record a scenario where the user is setting the LCD to
bright, medium, low
- In the scenario we will add callbacks to assertions that check the
LCD brightness state via the GPIO 0 output
- We will assert that LCD is set to bright, medium, low
- We will assert that LCD automatically set to low after 30 seconds

What happens is that two assertions do fail:
- The LCD is turned to low only after 60 seconds. This was intended to
show how it works
- The LCD cannot be not turned to low by the user. This is a bug in my
HW Android adaption layer of the liblights that just found!

To be continued…

Best regards,
Achim


References:
[1] Discussion, “Enabling deterministic/repeatable (Android) GUI
testing by record and replay of user input Optionen”
http://groups.google.de/group/virtual-platform-users/browse_thread/thread/3208b43c8318fe64

[2] Discussion, “Software Centric System Level Assertions”
http://groups.google.de/group/virtual-platform-users/browse_thread/thread/6a2f81889008a5b9

[3] Document, “Software Centric System Level Assertions”
http://groups.google.de/group/virtual-platform-users/web/software-centric-system-level-assertions

[AchimNohl]

Dear all,

I have uploaded all recent examples on system level assertions etc. to
the VP in the cloud. In terms of the scenario described above, I have
recorded it and put the script also in the cloud. If you want to
execute it yourself:

- Make sure Android is booted and
- The main screen in on
- The display is bright (no dim)
- If not, drive the phone into the above described state
- Suspend the simulation
- Choose menu "Script Examples->Assertions>Auto Scenario->LCD Power"
- Continue the simulation
- It will take ~5 seconds until something happens

When launching the script it gives you the name of the script that you
can have a look at.

Also, I have added the "Recording" capability to the VPA Script
Examples Menu. This way you can record your own scenarios. By
suspending the simulation during record, you can add callbacks to your
custom assertsions via the TCL command add_replay_callback.

Example:
- You launch the record script from the VPA
- Enable recording via command "start_recording"
- You run the simulation and operate the phone
- Whenever you want to assert something, you suspend the simulation
- And enter command "add_replay_callback my_special_assertion"
- You continue...and stop the recording at the end of the scenario via
"stop_recording"

What you will get is a script called "record.tcl" that can be used for
replay using "source replay.tcl".

Of course you need to implement your special callbacks such as
my_special_assertion. So the replay should look as follow:

proc my_special_assertion { } {
# Do you checks here...
}
source record.tcl

As said, an example of such a scenario assertion setup file can be
found in the cloud. The name of the script is shown when choosing menu
"Script Examples->Assertions>Auto Scenario->LCD Power".

Let me know if you have questions or ideas for improvements.

Best regards,
Achim