Thread Test Harness procedure with OpenThread based devices

470 views
Skip to first unread message

Shahabuddin Mohammed

unread,
Nov 14, 2016, 5:01:57 AM11/14/16
to openthread-users
Hi,

I have installed Thread Harness setup, it detects a FRDM-K64F board as DUT and proceeds with the testing.

However, I want to test with my DUT, which works on OT.
I know that OT developers are already using it.

Can you please share some information on the test procedure for OpenThread enabled devices?

FYI, I can see 3 types of DUTs listed as,
  1. ARM: NXP FRDM-K64F
  2. NXP: USB-KW24D512
  3. SiLabs: EM358x
Thanks,
Shahab


xiaom

unread,
Nov 14, 2016, 6:53:11 AM11/14/16
to openthread-users
Hi Shahab,

Please follow below README.md to setup the OT reference unit with the THCI, it works with Harness to enable OT DUT.

Hope this can help you, please also let us know if still any problem.

Thanks,
Xiao

Shahabuddin Mohammed

unread,
Nov 15, 2016, 12:24:27 AM11/15/16
to openthread-users
Hi Xiao,

Thanks for sharing the information.

I have made all the changes described in the link provided by you.
However, the Harness software doesn't detect my DUT, it says. "No Valid THCI Found".

Do I need to enable some define in OT for THCI to work?

FYI, the serial communication works, I verified with a serial application (TeraTerm).

Thanks,
Shahab

xiaom

unread,
Nov 15, 2016, 12:58:10 AM11/15/16
to openthread-users
Hi Shahab,

As far as I know, "No Valid THCI Found" indicates that OpenThread THCI script misses some new updated APIs (each Harness release may include some new updated APIs).

You can also see this kind of log from Harness terminal output.

May I know which Harness version you are using?

Thanks,
Xiao

Shahabuddin Mohammed

unread,
Nov 15, 2016, 1:59:12 AM11/15/16
to openthread-users
Hi Xiao,

Thanks for pointing that out.

Thread Harness Info file contains,
Version =  Thread v1.0 Release Candidate - [Build 0706].

Thanks,
Shahab

xiaom

unread,
Nov 15, 2016, 3:33:15 AM11/15/16
to openthread-users
Hi Shahab,

The version you are using is an old one (since new Harness supports more features, we update the THCI along with each new Harness release accordingly), would you mind to update your Harness?

You can download the latest published version "Harness 1.1 Alpha v2.0" from below link:

And then go through the same process mentioned on above README.

Please notice that you may encounter "No Valid THCI Found" as well because one parameter is added in Harness recently, you can fix it simply by commenting line 84(ARM.py):

# self.SED_Polling_Rate = ModuleHelper.Default_Harenss_SED_Polling_Rate

I verified this with v2.0 version and it works for me. Please give another try.

Thanks,
Xiao

Shahabuddin Mohammed

unread,
Nov 15, 2016, 5:00:28 AM11/15/16
to openthread-users
Hi Xiao,

I have downloaded the latest Harness package.
I had requested Tom from Thread group for the serial number, he asked me to wait for a day, as a new release is about to be made.

I will update after trying it.

Thanks,
Shahab

xiaom

unread,
Nov 15, 2016, 5:13:50 AM11/15/16
to openthread-users
Sounds good, the new release might not need the above workaround. Just applying for the latest THCI script should be ok.

Thanks,
Xiao

Shahabuddin Mohammed

unread,
Nov 17, 2016, 6:41:30 AM11/17/16
to openthread-users
Hi Xiao,

Can you please shed some light on how the harness actually works?
Does it use NCP or CLI mode of OT?

How is serial communication used between Harness and DUT?

Thanks,
Shahab

Shahabuddin Mohammed

unread,
Nov 17, 2016, 6:48:18 AM11/17/16
to openthread-users
Hi Xiao,

I have installed the Beta version of Harness and tried with DUT running OT.

Following the changes to ARM.py; I used ARM device in Configure Test Bed panel.
But it doesn't detect DUT.

Please help.

Thanks,
Shahab

xiaom

unread,
Nov 17, 2016, 10:15:50 AM11/17/16
to openthread-users
Hi Shahab,

Harness software is a kind of automation framework to help verify Thread Stack implementation functionality.

The whole of system comprises of  Harness SW running on Windows OS, sniffer, reference unit (golden device) and DUT (device under test). You can find more information about the framework in the specification--Thread-Test-Harenss-Quick-Start-Guide.pdf, that is located under the Harness installation directory. It's better to go through this spec to learn how to use Harness quickly.

Sniffer -- for capturing the radio packets exchanged in the air between reference unit and DUT. Then generate pcap file for specific packet analysis and validation (Harness use this to decide whether all checking criteria are passed or not)
Reference unit -- As reference hardware, Harness calls THCI API to control its behavior over serial port, to communicate with DUT. Currently there are 3 kinds of reference unit, as you noticed: NXP, ARM and SiLabs, 3 vendors.
DUT -- Your own hardware running Thread Stack (for example, your hardware based on OpenThread)

Above all 3 devices are connected to Harness via serial port (USB port). Harness calls THCI API to control the reference unit or DUT (if you enable AUTODUT feature). For example, set the channel, panid, networkname, master key and etc. Taking example of OpenThread, it's CLI mode (you might notice that THCI API just sends CLI command to hardware over serial port), not NCP.

Here, what I understand is that you want to have one hardware running OT as reference unit and one hardware running OT as DUT. If you just want to verify the functionality of DUT running OT, you do not need to go through the process mentioned in above README.md. What you need to do is just picking up one vendor's hardware as reference unit and program it with corresponding firmware (you can find the firmware under the harness installation directory). Then drag the device in Configure Test Bed panel.

The process mentioned in above README.md is for standalone testing (OT as reference unit VS OT as DUT), since Harness does not support OT as reference unit officially.

Is this explanation clearer?

Thanks,
Xiao

Shahabuddin Mohammed

unread,
Nov 17, 2016, 11:46:27 PM11/17/16
to openthread-users
Hi Xiao,

Thanks for a detailed explanation, it cleared all my doubts and concerns.

Regards,
Shahab

Andrei Emeltchenko

unread,
Feb 22, 2017, 10:21:32 AM2/22/17
to openthread-users
Hi Xiao,

Thanks a lot for a good explanation how harness-thci works.
Could you briefly explain how it is different from harness-automation?
Do I understand right that in harness-automation start.sh script calls all specified test cases automatically and controls DUT. Does it use THCI for that?

Regards,
Andrei

xiaom

unread,
Feb 22, 2017, 9:25:44 PM2/22/17
to openthread-users
Hi Andrei,

Harness-automation tool is implemented based on selenium framework to allow fully automated Thread Certification Harness test cases execution.

Benefit of this tool is that you do not need to run each test case manually any more(click the harness, choose the reference devices/DUT, select the specific test case, click run button...). Instead you just need to do some custom configuration for Harness-automation tool before the execution (e.g. reference device COM number, the save path of final result files, etc), then Harness-automation will go through selected test cases or all and save the result files as well as running log automatically.

Harness-automation just simulates people to operate the Harness web GUI, however, Harness still needs THCI to control the reference devices. The same Harness-automation used for that.

For more information on how to use Harness-automation tool with start.sh command line options and sample settings configuration, please refer to the below link:



Thanks,
Xiao
Reply all
Reply to author
Forward
0 new messages