simulate key-held-down behaviour in stbt test

[yvcd...@gmail.com]

I have recently been writing a memory soak-test for an application running on my set-top-box. The test has to simulate general user behaviour over a period of time while memory usage data is collected. I have been comparing this to the results from a similar manual test and found that the memory-usage varied between the two.

This occurred because, as a manual user I was actually pressing and holding down the RCU key to scroll through a carousel of content. The app proved to have a memory leak during this procedure. The RCU handles this by sending continuous key-signals to the device. the device then sends continuous key-down signals to the application running, which continually scrolls to the next cell in the carousel. Once the key-signals stop being sent to the device, either because the user lets go of the key or the RCU is no longer pointing directly at the device, then the platform sends a key-up signal to the application.

However, when the automated test scrolled through the same carousel, it is sending a single key-signal to the device every time the test calls key.press(). Between each key-press command in the test I am searching for a set of images before repeating. There is obviously too much time in between these signals for the device to consider the repeated key presses to be due to a user pressing and holding a button. This usage of the application avoids the memory leak so the automated test fails to expose the leak.

Is there a way to get a test to cover this behaviour in STBT?

If not, could I request it is added. I imagine something like key.press_down(key) and a key.stop_press_down(key) would help me complete this test.

many thanks,

Liam

[David Röthlisberger]

On 28 Nov 2013, at 09:25, yvcd...@gmail.com wrote:
> Is there a way to get a test to cover this behaviour in STBT?
>
> If not, could I request it is added. I imagine something like key.press_down(key) and a key.stop_press_down(key) would help me complete this test.

Hi Liam

I don't think this is currently supported; we'll need to add it.
I'll look into it tomorrow. We certainly don't want you to have to run
some tests manually just because of this limitation! :-)

It will depend on the capabilities of the particular infrared mechanism
you're using (I understand you're using LIRC, i.e. a USB IR blaster).
I don't know if it will be possible with all mechanisms, but we'll aim
to support it with LIRC and with the irNetBox.

Dave.

[Will Manley]

On Thu, Nov 28, 2013, at 9:25, yvcd...@gmail.com wrote:
> If not, could I request it is added. I imagine something like
> key.press_down(key) and a key.stop_press_down(key) would help me complete
> this test.

I've got a branch which adds a hold() method to the Remote classes which
I was using when simulating keyboard input and wanting to hold down
SHIFT while pressing a key to get a capital letter. The commit can be
found on github[1] on my remote-hold branch[2].

API example:

with remote.hold('KEY_SHIFT'):
remote.press('KEY_A')

will type a capital A.

The commit still needs a lot of work including tests, review, real world
testing and an implementation for the IRNetBox.

[1]:
https://github.com/wmanley/stb-tester/commit/3437b98bcbd5cfeb9055f60c913d97970073a211
[2]: https://github.com/wmanley/stb-tester/tree/remote-hold

Thanks

Will

[David Röthlisberger]

On 28 Nov 2013, at 10:31, Will Manley wrote:
> On Thu, Nov 28, 2013, at 9:25, yvcd...@gmail.com wrote:
>> If not, could I request it is added. I imagine something like
>> key.press_down(key) and a key.stop_press_down(key) would help me complete
>> this test.
>

> The commit still needs a lot of work including tests, review, real world
> testing and an implementation for the IRNetBox.

Re. the irNetBox, here is some technical information from RedRat:

> At the moment, I'm afraid that to simulate a long IR button press, it is
> necessary to modify the "repeat" value in the IR signal before
> downloading/outputting it.
>
> In what format do you store the IR signal data? At some point in the
> process you must have the block of data to download, see section 5.2.8 -
> message 17 in the comms doc (the data block is the same for sync and
> async messages). There is a "Number of Signal Repeats" value, so this
> should be adjusted to reach the time required.
>
> In our C# code, we specify a long repeat in two ways:
>
> 1) Using the number of repeats, which is easiest. A rule of thumb is
> that 10 repeats results in a key hold time of around one to two
> seconds.
>
> 2) Specify an actual time (e.g. 5s) and then calculate the number of
> repeats needed for that IR signal. This is a bit more complicated,
> and how we do this is calculate the length of the MAIN and REPEAT
> parts of the IR signal, and so the total time is given by:
>
> Time = Length(main signal part) +
> ((Length(repeat signal part) + Intra-signal-Pause )
> * number of repeats)
>
> One limitation is that the number of repeats value is a byte, but in
> most IR signals this is large enough to create a 30s button press if
> needed.
>
> Long button presses seem to be used more frequently now, I guess for
> things like switching the STB into engineering mode or something? A
> better way of supporting this would be to delegate it to the hardware to
> do, so we'll look into this.

Re. their last sentence, I've asked for the capability to send the
irNetBox 2 different messages for "start sending" and "stop sending".
In the meantime we should be able to support holding a button for a
specified amount of time.

[Will Manley]

Is there any way to find out if the sending has finished? If we can't I
imagine this could introduce race conditions in test scripts where the
user is surprised by keys still being pressed after the end of some test
block. We'd also have to be careful of key presses bleeding between
tests.

What happens with the netbox if you send:

press("KEY_OK", repeat=10)
press("KEY_OK", repeat=10)

in quick succession? Is the key repeated 20 times or does the second
call interrupt the first? Could we simulate hold() by calling this
rapidly?

Thanks

Will

[David Röthlisberger]

> In what format do you store the IR signal data? At some point in the
> process you must have the block of data to download, see section 5.2.8 -
> message 17 in the comms doc (the data block is the same for sync and
> async messages). There is a "Number of Signal Repeats" value, so this
> should be adjusted to reach the time required.

We read the IR signal data from the configuration file that we generate
using the RedRat Signal DB Utility. I suppose it will simply be a matter
of parsing that data as per §5.2.8 of the irNetBox spec† and modifying
the "number of signal repeats" field before sending.

† http://redrat.co.uk/products/IRNetBox_Comms-V3.X.pdf

> Is there any way to find out if the sending has finished?

Yes. Ever since 92164a9c `stbt.press` won't return until the irNetBox
has finished sending the signal (the irNetBox sends a "transmission
complete" message). This (92164a9c) was for reliability of the test
scripts and for consistency with the other remote-control
implementations.

> What happens with the netbox if you send:
>
> press("KEY_OK", repeat=10)
> press("KEY_OK", repeat=10)
>
> in quick succession? Is the key repeated 20 times or does the second
> call interrupt the first? Could we simulate hold() by calling this
> rapidly?

I haven't tested changing the repeat count, but I suppose that an
implementation of `stbt.hold` could send multiple irNetBox commands
without waiting for the response like `stbt.press` does. If you use
`stbt.hold` as a Python context manager, its `__exit__` method could
then wait for the irNetBox's "transmission complete" message
corresponding to the last command. Of course this is all hypothetical
until we actually test it. I'm going to focus on the LIRC implementation
first.

Note also that the "repeat" part of the signal can be different from
the initial "key pressed" part of the signal. It depends on the
particular protocol used by your remote control.

[David Röthlisberger]

On 28 Nov 2013, at 10:31, Will Manley wrote:

> On Thu, Nov 28, 2013, at 9:25, yvcd...@gmail.com wrote:
>> If not, could I request it is added. I imagine something like
>> key.press_down(key) and a key.stop_press_down(key) would help me complete
>> this test.
>
> I've got a branch which adds a hold() method to the Remote classes which
> I was using when simulating keyboard input and wanting to hold down
> SHIFT while pressing a key to get a capital letter. The commit can be
> found on github[1] on my remote-hold branch[2].

I have made some changes to Will's prototype, on branch
https://github.com/drothlis/stb-tester/commits/remote-hold
following some preliminary testing.

* The YouView-internal VirtualRemote.hold doesn't work very well,
so it might as well raise NotImplementedError (at least until
YouView fix their VirtualRemote listener).
https://github.com/drothlis/stb-tester/commit/2432c28b

* LircRemote.hold works, with a slight modification
https://github.com/drothlis/stb-tester/commit/5f040b73

But you can't use "stbt.press" to send a different signal while
you are "stbt.hold"ing. See also
https://github.com/drothlis/stb-tester/commit/b1e0d5dc

On my TODO list: More testing, write fake-lircd unit tests, irNetBox
implementation & testing.

Dave.

[David Röthlisberger]

On 3 Dec 2013, at 12:38, David Röthlisberger wrote:
> * LircRemote.hold works, with a slight modification
> https://github.com/drothlis/stb-tester/commit/5f040b73

It worked on a Fedora 17 PC (Linux kernel 3.9).

But on an Ubuntu 12.10 VirtualBox VM with Linux kernel 3.5, on a Mac
OS X host, LircRemote.hold only registers as a single keypress on the
set-top box (using identical IR blaster hardware and lircd.conf). Same
for "irsend send_start" (you still have to issue "send_stop" to allow
further irsend commands).

[David Röthlisberger]

`irsend send_start <remote> <key>` didn't work on a VM with kernel 3.11
either (the STB only registers a single keypress). So it seems that the
difference in behaviour is caused by running in a VM, rather than the
kernel version.

[David Röthlisberger]

On 28 Nov 2013, at 10:31, Will Manley wrote:

> On Thu, Nov 28, 2013, at 9:25, yvcd...@gmail.com wrote:
>> If not, could I request it is added. I imagine something like
>> key.press_down(key) and a key.stop_press_down(key) would help me complete
>> this test.
>
> I've got a branch which adds a hold() method to the Remote classes
>

> The commit still needs a lot of work including tests, review, real world
> testing and an implementation for the IRNetBox.

For what it's worth I've pushed a simple unit test + documentation for
`stbt.hold` to https://github.com/drothlis/stb-tester/pull/73
(only the LIRC control is implemented).

I'm not sure if this API for `stbt.hold` is a good idea, given that it
will be quite difficult to implement for the RedRat irNetBox unless
RedRat implement support for hold+release. I suppose we could launch a
separate thread that sends lots of individual keypresses and hope that's
good enough (it will only work for infrared protocols where the initial
& repeat signals are the same). We might even have to do the same for
the LIRC implementation, given that LIRC's repeat sending doesn't work
inside a VM, at least not for me.

Liam, can you please confirm whether or not a stbt script like this:

while True:
stbt.press("CURSOR_RIGHT") # or whatever your key-name is
# Important: Nothing else inside the while loop

can reproduce the memory leak in your application? If it can, then we'll
be able to implement `stbt.hold` by starting up a separate thread that
calls `stbt.press` repeatedly (while the main stbt thread can carry on
with `wait_for_match` or whatever else is in your script).

Thanks,
Dave.

[yvcd...@gmail.com]

Sorry for the delay. Happy new year. Back online now and will test this today.

[yvcd...@gmail.com]

I've tested this code and it has not have been able to reproduce the memory leak issue.

I've written a test client to expose the signals received. From this I can now see that the problem is that, when the RCU key is pressed and held down, the content application is receiving continual key-down signals, but will not receive a key-up signal until the button is released.

However, when the STBT test ran, it sent a key-up signal directly after every key-down signal.

Apologies - it is not actually the speed of the signals as suspected but is actually the order of the key-up signal with the key-down signal.

[David Röthlisberger]

On 7 Jan 2014, at 12:14, yvcd...@gmail.com wrote:
> I've written a test client to expose the signals received. From this I can now see that the problem is that, when the RCU key is pressed and held down, the content application is receiving continual key-down signals, but will not receive a key-up signal until the button is released.
>
> However, when the STBT test ran, it sent a key-up signal directly after every key-down signal.

Thanks for the follow-up. Please try one more thing on the command line
(not in a stbt script):

irsend send_start <name of remote control> <name of button>
sleep 10
irsend send_stop <name of remote control> <name of button>

where "<name of remote control>" is the remote name in your
/etc/lirc/lircd.conf file, and "<name of button>" is the same name as
you'd use with "stbt.press()".

If this behaves in the way you need, then we can use it to implement the
"stbt.hold" function proposed earlier in this thread (but only for
lirc-based controls, i.e. USB infrared emitters; it still wouldn't work
with the irNetBox which is controlled over the network, not lirc).

Let me know either way.

Thanks,
Dave.

[yvcd...@gmail.com]

I tried this test. However, the receiving app still received a key-up signal after almost every key-down signal, for the 10 second wait, until the send_stop command. There were also a few runs of 2-3 key-down signals before the key-up signal was received.

[David Röthlisberger]

On 9 Jan 2014, at 10:53, yvcd...@gmail.com wrote:
>> irsend send_start <name of remote control> <name of button>
>> sleep 10
>> irsend send_stop <name of remote control> <name of button>
>

> I tried this test. However, the receiving app still received a key-up signal after almost every key-down signal, for the 10 second wait, until the send_stop command. There were also a few runs of 2-3 key-down signals before the key-up signal was received.

That's a shame. It seems that we can't use stb-tester to test this
scenario, except with very specialised configurations. For example
changing your lircd.conf so that it sends very long "repeat" signals
might work; but then you wouldn't be able to use that same lircd.conf
to send normal (single) button presses.

We can also wait (hope) until RedRat add this "hold" functionality to
their irNetBox product.

We can also try using the YouView-specific TCP-based "virtual remote"
built into stb-tester; I'll contact you privately about that.

Thanks,
Dave.