Configuring/Running Bluetooth Autotest Test Suite

[Scott Conover]

After setting up a test image on a usb dongle and running it on a Samsung Chromebook, I started playing around with the autotest suites on Chromium OS. It was fascinating to watch the test 'login_LoginSuccess' run to completion, and the HTML job report was easy to browse through and understand.

Since I was interested in some of the bluetooth autotests, I started going through the source and seeing if I could follow a pattern of check adapter, run inquiry, pair/unpair, etc. I was unsure how to run a Dynamic Test Suite, so after doing some reading, I decided just to go ahead and try some of the bluetooth autotests. Browsing the source, I came across control.bluetooth, which noted in the doc description that "This suite runs the complete set of Bluetooth tests for our hardware and software, including the general Sanity tests, and the Bluetooth Qualification tests." I tried to run it as a straight command "run_remote_tests.sh --remote=$Selected_IP control.bluetooth --use_emerge" from my Ubuntu machine (test server). However, it only ran what appeared to be a pair of functional dummy scripts. A second search revealed that there was another file named bluetooth.control, which ran a simple dummy script test - "This is a helper test that can only run on bluetooth devices, and should succeed trivially."

After trying combinations such as test_suites/bluetooth.control with no success, I concluded that I did not understand how to run test suites for the autotest framework on Chromium OS. How do I properly call dynamic test suites for Autotest on CrOS, and how can I properly call the bluetooth Autotest test suite?

[Achuith Bhandarkar]

I believe we prefer test_that to run_remote_tests.sh:

http://goto.google.com/cros-test_that

so the command should just be (from your chroot):

test_that <host_ip> suite:bluetooth

Looks like there are 8 tests in this suite, and they all failed for me. It's possible the tests have rotted, or that there's some additional setup that needs to happen?

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en
 

[Scott James Remnant]

Our Bluetooth test suites are still a WIP.

There are two suites of tests:

bluetooth_sanity: this does the basic kinds of tests you were talking about, check the state of the adapter, inquiry scans, basic connections, pairing, etc.

bluetooth_qualification: this is a set of tests that match the SIG Qualification Test Requirements/Plan for our stack

You run them using test_that, e.g.:

  test_that -b $BOARD $HOST suite:bluetooth_sanity

where BOARD is the board you're building for, and $HOST is the hostname of your selected machine (not IP address, see below)

However you need a little more setup than you're probably expecting:

You need:

1) a machine to test ($HOST above, with a test image compiled for $BOARD installed on it) - stick $HOST in /etc/hosts

2) a second machine, so far we only support the Samsung Chromebox from a couple of years back (aka stumpy). This needs a test image compiled for it that was created using the testbed-ap profile (pass as --profile to setup_board, then build_packages and build_image test as normal)

3) entry for the second machine in /etc/hosts named $HOST-bluetooth  ... ie. if your DUT is called "test" you'd have to name this second machine "test-bluetooth"

4) your workstation (this acts as the "server" for the autotest) - this does not need a bluetooth adapter

The tests work by making the stumpy act as an external tester, tests are run between the DUT and the stumpy, rather than your workstation

On Tue, Dec 3, 2013 at 10:48 AM, Scott Conover <scottj...@yahoo.com> wrote:

--

[Scott Conover]

Scott,

So, basically my machine would be a master server sending commands to the slave controller ($HOST), which would then interface with the DUT by running the various bluetooth scripts. Is this correct?

Is there a reason that a Chromebox must be used for the testing, i.e. that a specific machine with a specific test bed build must be used for the testing? Most of the Chromium OS tests appear to be very flexible, so I imagine there is a specific reason here.

I do not have a Chromebox, and I this suite could be interesting once finished and adapted to other builds, so is there a way I could contribute to move this WIP testing forward? I would love to work on adapting the testing to other machines/builds, moving the testing to a more modular perspective (if possible), and eventually expanding the possible test cases.

-Scott

[Scott James Remnant]

Not quite, your machine acts as an Autotest server yes, and the tests are implemented as server_tests - but it sends commands to both the DUT ($HOST) and the "Tester" (${HOST}-bluetooth) - it does this by installing and starting XML-RPC servers on them

There's no specific reason a stumpy must be used, other than the testbed-ap profile being only created for that device so far. Any Chromium-capable machine with a supported Bluetooth adapter should work - you'd need to copy the testbed-ap profile into the appropriate overlay. We selected stumpy simply because it happens to be about the right size to fit into our radio isolation boxes, and it's easy to pry the lid off and attach things directly to the board inside

Any help expanding the test suite would be appreciated, take a look at:

• server/site_tests/bluetooth_Sanity_Discoverable - an example test
• server/cros/bluetooth/bluetooth_test.py - the base class for the tests
• server/cros/bluetooth/bluetooth_tester.py - the "test"-side interface for the Tester stumpy
• server/cros/bluetooth/bluetooth_device.py - the "test"-side interface for the DUT
• client/common_lib/cros/bluetooth/bluetooth_tester_xmlrpc_server.py - the XML-RPC server run on the Tester stumpy
• client/common_lib/cros/bluetooth/bluetooth_device_xmlrpc_server.py - the XML-RPC server run on the DUT
• client/common_lib/cros/bluetooth/bluetooth_socket.py - socket APIs for mgmt_ops, HCI control and HCI monitor (WIP)

Outside of autotest, take a look at src/platform/btsocket - this is a Python socket subclass that handles bind(), connect(), recvmsg() etc. for sockets of the AF_BLUETOOTH class

[Scott Conover]

Ah, ok. I see - so my machine would be a server controller, sending commands to both the DUT and the test partner (TP). Makes sense to me.

The Chromebox (Stumpy) decision is logical hardware-wise, especially for the board - for instance, the HP Chromebook is particularly tough to take apart sometimes - so I can see the rationale for Stumpy.

Where can I find the testbed-ap profile in the source? I have not worked with specific profiles for the boards on Chromium OS, but I can explore adding profiles once I take a glance at the code and surrounding directory.

Also, glancing through the Autotest documentation, it looks like pyauto was replaced at some point with Telemetry (some documentation still references pyauto, so I was unsure at first). I am sure running the tests will also make it clear, but are the bluetooth tests fully automated using the new framework, or are there semi-automated portions as well? I can work with the semi-automated portions to confirm overlays and then automate the manual parts thereafter.

-SJC

[Scott James Remnant]

Oh weird, the testbed-ap is currently defined in the "Chrome OS" overlay rather than the "Chromium OS" one... I'll have a chat with the other users of that profile and see if we can make it public

The basic difference from the bluetooth POV is that it's built with USE=-bluetooth (yes that minus is not a typo, the Tester doesn't have bluetoothd on it so it behaves in a more controllable manner) - you should be able to get by with that USE change easily enough

Bluetooth tests are intended to be fully-automatic, with the "option" of an Interactive mode that does in fact use Telemetry under the hood, but via the "interactive_xmlrpc_server" wrapper - the Discoverable test case has an example of this flow, with a "control.manual" that doesn't use the Tester but instead uses Telemetry to drive a Test UI on the DUT

The intent here was that for the bluetooth_qualification suite, we could use this manual mode in concert with the official Bluetooth PTS

[Scott Conover]

Ah, I see, you could mimic user input in order to demonstrate the particular use case(s) for BT quals. So, in "normal" (non-interactive) mode, I presume the bluetooth test cases are carried out programmatically between DUT/TP, based on controller commands to send/accept BT calls.

Ok, I am fairly new to the autotools framework model, and relatively new to Bluez, but basically the USE flag "-bluetooth" is carried out for the TP build, which leaves it without the bluetoothd (Bluetooth Daemon). I thought the bluetoothd module was necessary to interface with the bluetooth devices via the virtual D-bus. Are you basically sending commands to the card via the dbus on the TP from the DUT using the actual D-bus api logic? I don't want to delve too deeply here, but I think this part conceptually confuses me.

It would be great if the testbed-ap profile - or some stripped-out version of it - could be made public, so I am using the same toolset. But, otherwise, I can mimic the setup for building a TP with USE=-bluetooth flag, correct? 

Also, I have previously not set a profile-specific USE flag for a profile on an overlay. I looked through the readme at $chromiumos/src/overlays, and it looks like you can use named profiles as a parameter to the overlay when you build the image. Can I just generate a profile similar to the "asan" directory I see in several of the overlay directories and include a make.defaults entry with the desired USE=-bluetooth flag?

-SJC

[Scott James Remnant]

Right, the entire point of the Tester is that we can program the adapter directly via the kernel and socket interfaces - and that we don't have the "bluetoothd behavior" or the D-Bus interface

The DUT has bluetoothd and the D-bus interface

[Scott Conover]

Alright, I have undergone my first "test_that" test attempt, and I think I am almost there. So, I have a DUT(x86 USE=bluetooth (default)) and a TP(x86 USE=-bluetooth), and they look to be configured correctly. I went through the steps, and it appears to be almost correct. Using the steps delineated in the previous post, I configured the host file on my server computer in chromiumos/chroot/etc outside of the chroot using subl. I assumed that the host name was the same as the version name (version string enclosed), and used the format evident in hosts:

IP_ADDRESS_DUT<tab>HOST_NAME

IP_ADDRESS_TP<tab>HOST_NAME-bluetooth

Yet the tests do not run. I know I am doing something incorrect, I am unsure specifically what I am doing wrong. Re-reading your instructions, do I also need to set anything on the $HOST machine (DUT) for the $HOST-bluetooth machine (TP)?

Our Bluetooth test suites are still a WIP.
There are two suites of tests:
bluetooth_sanity: this does the basic kinds of tests you were talking about, check the state of the adapter, inquiry scans, basic connections, pairing, etc.
bluetooth_qualification: this is a set of tests that match the SIG Qualification Test Requirements/Plan for our stack

You run them using test_that, e.g.:
  test_that -b $BOARD $HOST suite:bluetooth_sanity
where BOARD is the board you're building for, and $HOST is the hostname of your selected machine (not IP address, see below)

However you need a little more setup than you're probably expecting:
You need:
1) a machine to test ($HOST above, with a test image compiled for $BOARD installed on it) - stick $HOST in /etc/hosts
2) a second machine, so far we only support the Samsung Chromebox from a couple of years back (aka stumpy). This needs a test image compiled for it that was created using the testbed-ap profile (pass as --profile to setup_board, then build_packages and build_image test as normal)
3) entry for the second machine in /etc/hosts named $HOST-bluetooth  ... ie. if your DUT is called "test" you'd have to name this second machine "test-bluetooth"
4) your workstation (this acts as the "server" for the autotest) - this does not need a bluetooth adapter

The tests work by making the stumpy act as an external tester, tests are run between the DUT and the stumpy, rather than your workstation

-SJC

[Scott James Remnant]

should work :)  what's the output from test_that ?

[Scott Conover]

Scott,

Evidently, I am very impatient :) After letting the test run fully for awhile, it finally completed. My first attempt seemed to be hanging to me, which is an assertion I should start checking/testing - rather than assuming - in the future. Nevertheless, the test suite does appear to have issues connecting and instantiating the server. I  am uncertain if this is inherent to the current test code, some particular arguments or settings, or if I setup something improperly.

The output of the command "test_that -b $BOARD $HOST suite:bluetooth_sanity" is attached.

-SJC

[Scott James Remnant]

Unsure, but it does at least look like the test framework is working - but you're getting a FAIL from two tests

[Scott Conover]

I have some questions:

1) Are the fails normal? Is the typical benchmark set a series of passes?

2) This message comes up quite a bit - "FAILED RPC CALL: get_hosts {'hostname': 'Device_Under_Test'}", and it seems like the script attempts to run several times. Is this issue unique to the bluetooth tests, and if so, what steps I can do to troubleshoot it in order to improve the tests? If not, is there a way to stabilize the setup/connect feature, or is the RPC call failure due to a WIP, typical behavior, or just an anomaly in this particular testing? Perhaps I should review the Autotest framework and its implementation on Chromium OS, but the iterative manner of attempted connection and activity seems unusual to me.

3) I would prefer to run a multiple run regression of say, 100 iterations, in order to determine if the fails are consistent, and to find flaws in the suite testing. Can you run multiple iterations with "test_that" as the base command? Or do you have to run a separate shell command with a loop/counter to autorun the additional tests?

4) I ran the suite:bluetooth_qualification, but the results indicated that there were no tests present in the suite. Am I running the wrong suite command for the bluetooth_qualification, or am I missing some file, setting or configuration? The output of "test_that -b $BOARD $HOST suite:bluetooth_qualification" is attached.

-SJC

[Scott James Remnant]

Fails aren't normal, those tests should ordinarily all pass

I don't know why RPC calls would be failing either, they don't fail here

Our test approach here has been that a single iteration should be sufficient - the Discover and Discoverable tests, for example, self-iterate on failure if they believe another attempt would try and record how many iterations it took to get the test to pass

And yes, there aren't any public tests for that second suite yet

[Scott Conover]

Well, having explained my setup in an earlier post and having followed the instructions evidently fully (i.e. building USE=-bluetooth for the TP at all stages and so on), that leaves me with just a few observations and questions. Ok, so from here:

Observation(s)
Unless my setup or approach is incorrect, there may exist some substantial differences between the testbed-ap profile and the typical setup besides the equivalent of the flag USE=-bluetooth. While obviously it is none of my business what goes into the magical (for me) Chrome OS set of events, perhaps it may be possible to generate an equivalent (but less functional/non-proprietary) profile or equivalent, assuming I did not flub it somehow. Unfortunately, I have no idea how to go forward with generating the additional information/setup required, so I could use some advice in this area - assuming, again, that I did it correctly.

Questions/Followup

1) How can I further debug the testing failures I am receiving? I could use some general advice, given my relative inexperience.

2) Is there a way to effectively debug RPC failures? I did not see such failures on other autotests I ran, but does not mean that I will not run into them in the future.

3) OK, so the tests effectively autorun until you receive a pass, which gives you a ratio of success/tests for each particular test, in essence. I will read through the Discover/Discoverable tests to see exactly how they iteratively plays out.

4) Ok, I likely cannot really do anything about that until the tests are judged/refurbished to be public, so that is a dead-end for now, I guess.

-SJC

[Scott James Remnant]

Run the tests individually; I often try and debug by running the xml-rpc server by hand on the DUT at the right point and seeing what's happening on its output

(Getting the output/logs from the XML-RPC Server back to the test server is on my TODO list somewhere :p)

[Scott Conover]

Ok, I can run the tests individually - since I have already done that before. That should be useful in debugging.

I did not see the command to run the xml-rpc server manually when I browsed the Autotest source; where I can find the documentation on that - or is that an Autotest feature I should learn to use from the Autotest documentation?

-SJC

[Scott James Remnant]

It's defined in constants.py, the constant named in the bluetooth_client.py or bluetooth_tester.py bit that actually starts it

(All that's doing is ssh'ing in and running a command from the autotest tree)

[Scott Conover]

Scott,

I ran the two problematic tests individual (Discover and Discoverable), and it appears that the request is not being passed to the bluetooth socket, "FAIL: Unhandled Fault: <Fault 1: "<class 'autotest_lib.client.common_lib.cros.bluetooth.bluetooth_socket.BluetoothSocketError'>:Short write on socket". The piped output for the two tests is attached. Do you know what might cause that or how I can debug deeper into the bluetooth socket behavior?

I will also explore the xml-rpc server manual setup, i.e. ssh login and run command and see if there is any difference in functional behavior. Does it make sense to just run the commands to set up the server, then run the individual tests, or are there additional steps I should consider in troubleshooting?

-SJC

[Scott Conover]

Scott,

I swapped out the Samsung chromebook I was using for another test partner - an HP Chromebook in this case. The test ran fine the first time, failed for two tests the second time, and passed the third time with no issue. The tests appear to work alright, especially in this case, where I paired an HP Chromebook to an HP Chromebook for the underlying hardware. The piped output for iterations 2 (2 failures) and 3 (all pass) are enclosed. The "FAILED RPC CALL: get_hosts {'hostname': 'Device_Under_Test'}" error still comes up, so I will go ahead and walk through the manual setup of the rpc servers and see what gives.

I think I will also explore the expansion of text cases, which will also give me an opportunity to holistically understand the autotest framework and implementation. I am puzzled on some pieces of the autotest implementation, but walking through the specific examples and FAQ in code should help me figure it out. Are you using standard linux test case build as an example for the bluetooth sanity checks (if such exists) an ad-hoc approach or some particular or specific test case scenario set, e.g. the base functionality outline of the profiles from bluetooth SIG? I would prefer to use whichever approach is considered best by Chromium OS dev.

-SJC

[Scott James Remnant]

I'm not sure I understand your question in this e-mail?

The Bluetooth Sanity tests are intended to be basic checks of basic functionality, no specific scenario other than the kinds of tests we've found that smoked out particular problems we've had in the past

[Scott Conover]

Scott,

Thanks, you've answered my question.

-SJC

[pitera shi]

hi, Scott,

  sorry to bother you...

  I'd like to know how to setup BT test environment

  1.Do I need a ChromeBox for Bluetooth test?

  2.Do you have any docs describing how to setup test environment and run test scripts?

  could you help me on this, I appreciate very much on your help

Pitera

Scott Conover於 2013年12月4日星期三 UTC+8上午2時48分21秒寫道：

[pitera shi]

hi, Achuith Bhandarkar,

 there were many link you posted no longer valid

 could you release me link or docs describing how to setup BT test environment and run autotest scripts

 ps. do I need ChromeBox for Bluetooth test such as wifi auto test?

 this would be great help for me with your support,thanks

pitera

Achuith Bhandarkar於 2013年12月4日星期三 UTC+8上午3時03分37秒寫道：