Enabling deterministic/repeatable (Android) GUI testing by record and replay of user input

[AchimNohl]

Hi all,

have a look at the screencast:

http://screenr.com/5Kh

The screencast shows how Virtual Platforms can enable deterministic
and repeatable GUI testing. The principle is very simple: On every
trigger to the keyboard and touchscreen interfaces, I record the input
data together with a (relative) time-stamp (simulated, not wall-clock
time) and generate a replay script. The replay script contains time-
breakpoints with callbacks that feed the same data into the system
that have been previously entered manually.

I will post the script and explanations tomorrow.

Best regards,
Achim

[AchimNohl]

Hi all,

I have just oploaded the record and replay script for keyboard and
touchscreen events that I have shown in the screencast:

http://screenr.com/5Kh

It should be useful for any kind of application testing/debugging that
requires user interaction. The nice thing about it is that the replay
will be always the same. In case you have triggered a tricky, rarely
appearing problem, you can be sure that you can reproduce it now. Of
course the script can be extended to also record activity on the UART
for e.g. GPS etc.

The script is located here:
http://virtual-platform-users.googlegroups.com/web/android_example_record_touchscreen_and_keyboard.tcl

How do you use it with the VP in the cloud ?
- Download the script from the web using a browser and put it e.g.
into ARM926EJS-CoWareVP-VersatilePB-QuadCore/skins/Android/sim
- Boot the Android skin and suspend the simulation when booted (F9)
- Source the script via: source
android_example_record_touchscreen_and_keyboard.tcl
- Start the recording via: start_recording
- Continue the simulation (F5), operate the device, suspend the
simulation (F9)*
- Stop the recording via: stop_recording
- Replay the recording via: replay and continue the simulation (F5)

* Make sure your recording end in the same GUI state as it was when
you started.

I will describe how the script works later.

Best regards,
Achim

[AchimNohl]

Hi all,

I still wanted to comment on the script that records and replay the
user input for e.g. GUI testing, debugging, or optimizations.

What we finally want to get is a "timed" stimuli provided to the
keyboard and mouse interfaces (KMI) that the hardware keys and
touchscreen connect to. The peripherals "i_KMI_0" and "i_KMI_1" get
their data trough the receive logic. The VP models of the KMI (as well
as UARTs etc.) provide a high level "push" command that abstract this
logic.

Example, push value 22 into the i_KMI_0:
i_KMI_0 push 22

Typically, the data is provided by the PS/2 keyboard and touchscreen
emulator that is part of the phone GUI and controlled by the user. For
the replay, we want to put the data automatically into the KMIs
without user interaction. Therefore, we need to time the stimuli. This
can be easily achived using time breakpoints. Those breakpoints allow
to interrupt the simulation or trigger a callback at absolute or
relative (+) time stamps. Here is an example of what we want to
execute to provide (replay) the timed stimuli:

[vpa::create_time_breakpoint +5151 ms] set_callback "i_KMI_0 push 1;"
[vpa::create_time_breakpoint +7701 ms] set_callback "i_KMI_0 push 1;"
[vpa::create_time_breakpoint +9282 ms] set_callback "i_KMI_1 push 11"
[vpa::create_time_breakpoint +9550 ms] set_callback "i_KMI_1 push
240;i_KMI_1 push 11;"

The time we are referring to here is simulation time. This is
important as it has a significant benefit. If we are going to stop the
simulation during the replay, the simulation time will not advance.
Meaning, we will not mess up the time-stamps. So we can take all the
time we need to debug at certain points and can be sure that the
simuli will still be provided in the right sequence at the right time.

Of course we do not want to hand-code this stimuli script, as e.g. we
do not know about which touchscreen event results in which PS2 input.
So, we are going to watch what is happening on the interface of the
KMI to the trouchscreen and we will generate a script as shown above.
What we need to watch is the signal "Recv" that is set by the model
when receing data through the high level commands such as push. And,
we attach a callback that generates an entry for the simuli script on
every change:

# Create watchpoint on receive signals...
[i_KMI_0/Recv create_watchpoint] set_callback "record i_KMI_0"

Within the record callback we need to output(to a file) e.g.:
[vpa::create_time_breakpoint +9282 ms] set_callback "i_KMI_0 push 11"
[vpa::create_time_breakpoint +9550 ms] set_callback "i_KMI_0 push 240"

Therefore, we need to identify not only the data provided but also the
timestamp of the current time. This can be retrieved via
"[vpa::get_current_time ]". The delta time can be computed by current
time minus the start time which we store in t0.

So, a simplified version of the callback that records one entry is
here:
proc record { kmi } {
# Get value of KMI receive signal
set value [$kmi/Recv get_value]
# Get current time
set t_now [vpa::get_current_time ]
# Compute delta time
set ::delta [expr $t_now-$::t0]
# Generate in the replay script: Create a time breakpoint at the
delta time
puts -nonewline $::recordfile "\[vpa::create_time_breakpoint +
$::delta\] set_callback \"$kmi push $value\""
return ""
}

The version in the TCL file I have posted also handles the fact that
multiple commands can have the same timestamp and attaches them to the
same time breakpoint.

I am writing all that as the concept of time breakpoints and callbacks
is a very powerful method to provide deterministic stimuli into a
system or probe data on a certain frequency.

Best regards,
Achim