[mbed] Host test autodetection (#900)

32 views
Skip to first unread message

Przemek Wirkus

unread,
Feb 10, 2015, 7:16:19 PM2/10/15
to mbedmicro/mbed

Description

In order to achieve leaner test suite functionality and in the future decouple functionality such as:

  • mbed-enabled board auto-detection - mbed-ls project, former mbed-lmtools.
  • Host tests in worskace_tools/host_tests - set of test supervising scripts.
  • Test suite CLI (command line interface)
  • Building mechanism such as build.py, make.py and singletest.py will be replaced in future with separate project build tool.

mbed-ls - cool mbed-enabled platform auto-detection on your host!

mbed-ls is Python module used to detect and list mbed-enabled devices connected to host computer. package will support Windows 7, Ubuntu and MacOS. It will be delivered as redistributable Python module (package) and command line tool.
Currently mbed-ls module is under development but already decoupled mbed-ls functionality is delivered to mbed SDK's test suite.

You will be able for example to use this tool from command line!

$ mbedls
+---------------------+-------------------+-------------------+--------------------------------+
|platform_name        |mount_point        |serial_port        |target_id                       |
+---------------------+-------------------+-------------------+--------------------------------+
|KL25Z                |I:                 |COM89              |02000203240881BBD9F47C43        |
|NUCLEO_F302R8        |E:                 |COM34              |07050200623B61125D5EF72A        |
+---------------------+-------------------+-------------------+--------------------------------+

New command line switch --auto and --tc are added to workspace_tools/singletest.py script and replace (not completely, you will be still able to use configuration files just because they are more powerful than mbed-ls) existing file based test configuration.

Currently you will execute your automated tests like this:

$ singletest.py -i test_spec.json -M muts_all.json

With mbed-ls tools you will be able to auto-detect all connected to your host devices and just decide which toolchains you want to include.
Below command is replacement for -i and -M combo. Tests will be built using default compiler for each detected mbed-enabled platform.
Note: Now mbed-ls tools will detect devices for you, pair serial ports and disks (mount points) and based on TargetID determine mbed-enabled target generic name.

$ singletest.py --auto

or just for example specify that you want to build tests using default toolchains (per platform) and GCC_ARM for all platforms.

$ singletest.py --auto --tc default,GCC_ARM

New --auto command is compatible with existing command line switches so you can for example:

  • Check your auto-detected configuration:
$ singletest.py --auto --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: I:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 1     |             | NUCLEO_F302R8 | E:   | COM34 |
| 2     |             | KL25Z         | I:   | COM89 |
+-------+-------------+---------------+------+-------+

Test specification in auto-detected:
+---------------+-----+------+
| mcu           | ARM | uARM |
+---------------+-----+------+
| KL25Z         | Yes | -    |
| NUCLEO_F302R8 | -   | Yes  |
+---------------+-----+------+
  • Check supported toolchains for each platform (with optional filter -f):
$ singletest.py -S -f F302
+---------------+-----------+---------+-----------+--------+--------+-----------+------------+---------------+
| Platform      |    ARM    |   uARM  |  GCC_ARM  | GCC_CS | GCC_CR |    IAR    | GCC_CW_EWL | GCC_CW_NEWLIB |
+---------------+-----------+---------+-----------+--------+--------+-----------+------------+---------------+
| NUCLEO_F302R8 | Supported | Default | Supported |   -    |   -    | Supported |     -      |       -       |
+---------------+-----------+---------+-----------+--------+--------+-----------+------------+---------------+
*Default - default on-line compiler
*Supported - supported off-line compiler

Total platforms: 1
Total permutations: 4
  • Test suite will check for you if you configured settings and your toolchain path(s):
$ singletest.py --auto --config --tc all
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: I:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 1     |             | NUCLEO_F302R8 | E:   | COM34 |
| 2     |             | KL25Z         | I:   | COM89 |
+-------+-------------+---------------+------+-------+

Test specification in auto-detected:
+---------------+-----+---------+------------+---------------+-----+------+
| mcu           | ARM | GCC_ARM | GCC_CW_EWL | GCC_CW_NEWLIB | IAR | uARM |
+---------------+-----+---------+------------+---------------+-----+------+
| KL25Z         | Yes | Yes     | Yes#       | Yes#          | Yes | -    |
| NUCLEO_F302R8 | Yes | Yes     | -#         | -#            | Yes | Yes  |
+---------------+-----+---------+------------+---------------+-----+------+
Toolchain conflicts:
        # Toolchain GCC_CW_EWL path not found: C:/Freescale/CW MCU v10.3/MCU/ARM_GCC_Support/ewl/lib
        # Toolchain GCC_CW_NEWLIB path not found: C:/Freescale/CW MCU v10.3/Cross_Tools/arm-none-eabi-gcc-4_6_2/bin
  • Use test verbose mode -V to see test printouts in real time, choose test(s) you want to execute with -n option and define your favorite toolchain(s) with --tc:
$ singletest.py --auto --tc GCC_ARM -V -n DTCT_1
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: E:
Building library CMSIS (NUCLEO_F302R8, GCC_ARM)
Building library MBED (NUCLEO_F302R8, GCC_ARM)
Building project DETECT (NUCLEO_F302R8, GCC_ARM)
Executing 'python host_test.py -d E: -f "C:\Work\mbed-przemek\build\test\NUCLEO_F302R8\GCC_ARM\DTCT_1\detect.bin" -p COM34 -t 10 -C 1.5 -m NUCLEO_F302R8'
Test::Output::Start
MBED: Instrumentation: "COM34" and disk: "E:"
HOST: Copy image onto target...
HOST: Initialize serial port...
HOST: Reset target...
HOST: Property 'timeout' = '10'
HOST: Property 'host_test_name' = 'detect_auto'
HOST: Property 'description' = 'Simple detect test'
HOST: Property 'test_id' = 'DTCT_1'
HOST: Start test...
{{start}}
HOST: Detecting target name...
MBED: Target 'NUCLEO_F302R8'
HOST: MUT Target name 'NUCLEO_F302R8', expected 'NUCLEO_F302R8'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID '04dab79c-a6c8-4bf0-8b1e-3771b4540bb8'

{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F302R8::GCC_ARM::DTCT_1::Simple detect test [OK] in 1.07 of 10 sec
Test summary:
+--------+---------------+-----------+---------+--------------------+--------------------+---------------+-------+
| Result | Target        | Toolchain | Test ID | Test Description   | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------------+-----------+---------+--------------------+--------------------+---------------+-------+
| OK     | NUCLEO_F302R8 | GCC_ARM   | DTCT_1  | Simple detect test |        1.07        |       10      |  1/1  |
+--------+---------------+-----------+---------+--------------------+--------------------+---------------+-------+
Result: 1 OK

Completed in 4.33 sec

Host test script auto-detection

Structure for host tests and C/C++ test source code changed. In the future tests will be deployed together with corresponding modules. For now mbed SDK tests have to improve. We want to remove dependency on workspace_tools/tests.py test meta-data and move test / test runner handshake inside test source code.

  • Example target test, look how new macros TEST_* are used to notify host test system about prefered host test handler. Now test will decide which host test should handle it, not extra meta-data.
#include "mbed.h"
#include "test_env.h"

int main() {
    TEST_TIMEOUT(10);
    TEST_HOSTTEST(detect_auto);
    TEST_DESCRIPTION(Simple detect test);
    TEST_START("DTCT_1");

    notify_start();
    printf("MBED: Target '%s'\r\n", TEST_SUITE_TARGET_NAME);
    printf("MBED: Test ID '%s'\r\n", TEST_SUITE_TEST_ID);
    printf("MBED: UUID '%s'\r\n", TEST_SUITE_UUID);
    TEST_RESULT(true);
}

Note, things like:

  • Test case timeout (real time of execution) is defined in macro TEST_TIMEOUT().
  • Test case description is encapsulated with macro TEST_DESCRIPTION().
  • Preferred host test handler is defined in macro TEST_HOSTTEST().
  • Test case result can be send to test suite environment using TEST_RESULT() macro - if applicable. Host test can decide about test case result in many cases.

mbed SDK is now in color!

Colorama is new package dependency added so CLI is now colorful (Hard to say if this is good or bad).
Below sample of new color format:
autodetection_1

Testing

Solution tested with 8 of mbed-enabled boards. Both test auto-detection for test_env and unit tests using CppUTest works well.
This solution will be tested with greater number of boards soon, so pelase wait for notification in this pool request.

You can view, comment on, or merge this pull request online at:

  https://github.com/mbedmicro/mbed/pull/900

Commit Summary

  • Merge pull request #3 from mbedmicro/master
  • Merge remote-tracking branch 'upstream/master'
  • Added initial 'host test auto-detection API'
  • Refactored generic tests with timer host test supervision
  • Refactored few more tests to support autodetection: basic, call_before_main, dev_null, rtc, stdio
  • Refactored echo at 115200 test for new model of test autodetection
  • Refactored detection test to use new autodetection of host test
  • Refactored tests for SD card peripheral
  • Refactored DIV and VTAB relocation tests
  • Refactored C++ test and added '+' sign as recognized character in auto-test description
  • Refactored Semihost test (valid for few LPC platforms)
  • Refactored few peripheral tests like DigitalInOut, InteruptIn
  • Refactored host_test.py script to filter for TargetIds in auto-detection flow
  • Refactored RTOS_1 test and did some small indent in tests.py
  • Added test case time measurement instead of whole test time measurement for 'Elapsed Time (sec)' report in singletest.py console reports
  • Refactored EEPROM and InterruptIn test cases to do auto-detection act
  • Refactored TCP and UPD client tests (with default host test instrumentation)
  • Refactored TCP echo server
  • Refactored UDP echo server
  • Added -f (filter) to configuration switch --config
  • Added HTTP and NTP test to autodetection pool
  • Added NET_4, NET_6 and NET_13 tests to autodetection pool
  • Autodetection: Added support for Unit Tests (CppUTest test runner)
  • Simple indent inside Python module init
  • Added autodetection fior tests including MMA8451Q accelerometer
  • Added timeout detection from aut-detection MUT printout
  • Added new switch --auto to force autodetection with mbed_lstools
  • Fixed bug in --tc switch
  • Fixed bug in --config and --auto switches
  • Removed extra dependency to TEST_LIB - GCC_ARM linking bug fix
  • Added coloram module import
  • Added colorama to toolchain notification function and basic test suite prompts

File Changes

Patch Links:

—
Reply to this email directly or view it on GitHub.

Martin Kojtal

unread,
Feb 11, 2015, 2:43:33 AM2/11/15
to mbedmicro/mbed

In libraries/tests/mbed/env/test_env.h:

> @@ -22,6 +22,36 @@ void notify_performance_coefficient(const char* measurement_name, const int valu
>  void notify_performance_coefficient(const char* measurement_name, const unsigned int value);
>  void notify_performance_coefficient(const char* measurement_name, const double value);
>  
> +// Host test auto-detection API
> +void notify_host_test_name(const char *host_test);
> +void notify_timeout(int timeout);
> +void notify_test_id(const char *test_id);
> +void notify_test_description(const char *description);
> +
> +// Host test auto-detection API
> +#define TEST_START(TESTID)      notify_test_id(TESTID); notify_start()

I would use prefix with the name of this test env, it's either part of mbed or any other. The other libraries can easily have similar macros TEST_DESCRIPTION or similar.

Martin Kojtal

unread,
Feb 11, 2015, 2:44:35 AM2/11/15
to mbedmicro/mbed

In libraries/tests/rtos/mbed/mutex/main.cpp:

> @@ -16,8 +16,7 @@
>      #define STACK_SIZE DEFAULT_STACK_SIZE
>  #endif
>  
> -void print_char(char c = '*')
> -{

why are you changing the style? It is not the one which is mostly used in files here

Martin Kojtal

unread,
Feb 11, 2015, 3:08:54 AM2/11/15
to mbedmicro/mbed

In libraries/tests/rtos/mbed/file/main.cpp:

>              for (int i = 0; i < SIZE; i++) {
>                  data_written[i] = rand() % 0xff;
>                  fprintf(f, "%c", data_written[i]);
>                  printf("%02X ", data_written[i]);
>                  if (i && ((i % 20) == 19))
> -                    printf("\r\n");
> +                    printf(NL);

Is NL as new line? Is it a macro? I can't find the macro definition.

Isn't this NL host specific? We should use a newline.

Przemek Wirkus

unread,
Feb 11, 2015, 3:58:30 AM2/11/15
to mbedmicro/mbed

In libraries/tests/rtos/mbed/file/main.cpp:

>              for (int i = 0; i < SIZE; i++) {
>                  data_written[i] = rand() % 0xff;
>                  fprintf(f, "%c", data_written[i]);
>                  printf("%02X ", data_written[i]);
>                  if (i && ((i % 20) == 19))
> -                    printf("\r\n");
> +                    printf(NL);

Yes, NL is a macro in test)_env.h file I presume. It is defined like this I think:

#define NL "\n"
#define RCNL "\r\n"

Przemek Wirkus

unread,
Feb 11, 2015, 3:59:03 AM2/11/15
to mbedmicro/mbed

In libraries/tests/mbed/env/test_env.h:

> @@ -22,6 +22,36 @@ void notify_performance_coefficient(const char* measurement_name, const int valu
>  void notify_performance_coefficient(const char* measurement_name, const unsigned int value);
>  void notify_performance_coefficient(const char* measurement_name, const double value);
>  
> +// Host test auto-detection API
> +void notify_host_test_name(const char *host_test);
> +void notify_timeout(int timeout);
> +void notify_test_id(const char *test_id);
> +void notify_test_description(const char *description);
> +
> +// Host test auto-detection API
> +#define TEST_START(TESTID)      notify_test_id(TESTID); notify_start()

Yes, you are right, I will change it to something more mbed specific / descriptive.

Przemek Wirkus

unread,
Feb 11, 2015, 4:15:52 AM2/11/15
to mbedmicro/mbed

In libraries/tests/rtos/mbed/mutex/main.cpp:

> @@ -16,8 +16,7 @@
>      #define STACK_SIZE DEFAULT_STACK_SIZE
>  #endif
>  
> -void print_char(char c = '*')
> -{

I do not understand, which style here is the most popular one?
I'm alligning all tests to the same style which is something like this:

void function(int param1, char param2 = '*') {
    /* FUNCTION BODY */
}

Martin Kojtal

unread,
Feb 12, 2015, 4:03:03 AM2/12/15
to mbedmicro/mbed

In libraries/tests/rtos/mbed/mutex/main.cpp:

> @@ -16,8 +16,7 @@
>      #define STACK_SIZE DEFAULT_STACK_SIZE
>  #endif
>  
> -void print_char(char c = '*')
> -{

Ok I see.

Martin Kojtal

unread,
Feb 16, 2015, 4:38:01 AM2/16/15
to mbedmicro/mbed

Merged #900.

Reply all
Reply to author
Forward
0 new messages