What is max number of mock objects?

[Vitaliy Bortsov]

Hello!

This code need very long time to work on CYGWIN GCC (32-bit). Why?
Is it possible to reduce run time?

In fact, I can't wait to end tests and destroy test process.

#include "CppUTest/TestHarness.h"
#include "CppUTestExt/MockSupport.h"

TEST_GROUP(MockFail)
{
    virtual void setup()
    {
    }
    virtual void teardown()
    {
        mock().checkExpectations();
        mock().clear();
    }
};

TEST(MockFail, veryLongTimeToEnd)
{
    mock().expectNCalls(30000, "MainDelay");
}

[A. Robert S.]

Expect fewer calls ;-)

 

On a more serious note, I have had similar tests. Since I new 29999 calls were needed, and something was supposed to happen during the 30000 call, I

- called `mock().disable()`

- wrote code to make the 29999 calls

- called `mock().enable()`

- wrote code to make the 30000th call

- check whether the 30000th call behaved as expected.

 

This would also work if you just want to make sure the mock was actually called 30000 times.

 

As a bonus, you don't get 30000 error messages if anything went awry XD

 

Robert

[Vitaliy Bortsov]

But I have to call mock().expectNCalls(...) with .andReturnValue(...)

And that code will not work, according to mocking manual http://cpputest.github.io/mocking_manual.html

float get()
{
    return mock().actualCall("ivexpget").returnValue().getDoubleValue();
}

Of course, there is another solution -- use fake instead of mock. More code to write.

I wonder for problem from scratch. Is it internal to CppUTest or C++ library? What is source of problem?
time to complete test run:
3 ms for zero number call
14 ms for 100 calls
46 ms for 200
98 ms for 300
187 ms for 400
336 ms for 500
2358 ms for 1000 calls
17309 ms for 2000 calls

It is even more than quadratic dependence with big numbers.

I use that code

#include "CppUTest/TestHarness.h"
#include "CppUTestExt/MockSupport.h"

void MainDelay()
{
    mock().actualCall("MainDelay");

}


TEST_GROUP(MockFail)
{
    virtual void setup()
    {
    }
    virtual void teardown()
    {
        mock().checkExpectations();
        mock().clear();
    }
};

TEST(MockFail, veryLongTimeToEnd)
{

    const int n = 500;
    mock().expectNCalls(n, "MainDelay");
    for (int i = 0; i < n; ++i) MainDelay();
}

вторник, 28 октября 2014 г., 12:43:49 UTC+5 пользователь A. Robert S. написал:

[Vitaliy Bortsov]

Sorry, fake is not suitable -- mocking function is called from production code many times and because of that cannot be programmed with fake.
May be, I can use programmed array of return values? With setups, teardowns and etc...

[A. Robert S.]

I am sure I have used mock().expectNCalls() with .andReturnValue(). I can't find any reference in the documentation that this shouldn't work.

However, you would return the same value N times. This may not be okay for your application.

 

As for the slowness, it looks to me like an issue you might want to open on Github, along with your measurements. I might be worth looking into...

 

Robert

[A. Robert S.]

If I were in your position, I might also consider a vector for input values and a for loop to call the mock 29999 times (with mocking disabled).

 

Robert

[James Grenning]

Hello Vitally

I just looked at expectNCalls, and as you may guess from the
performance, it creates a new MockFunctionCall 30,000 times.

If you really need it to run 30,000 times and return the same value, the
handcrafted fake is quite simple and runs fast. I prefer fakes over
mocks for clarity and simplicity.

Why does the unit test have to do this thing 30,000 times to know it
works? Having a unit test that goest through 30,000 times seems a
little odd to me. I'd consider injecting the loop counter so you could
control it.


------------------------------------------------------------------
James Grenning Author of TDD for Embedded C
www.wingman-sw.com http://pragprog.com/titles/jgade/
www.wingman-sw.com/#blog
www.twitter.com/jwgrenning

> --
> You received this message because you are subscribed to the Google
> Groups "cpputest" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cpputest+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ryan Hartlage]

The root cause of this is that each expected call is added to the end of a linked list.  Just adding 30000 items to the end of a linked list is going to take a long time.  When an actual call is occurring, this list can potentially be traversed multiple times (each time withParameter is invoked).

I think there are two solutions -- either expected calls are added to a list with a count (to support tremendously large numbers of duplicate expected calls), or find a different way to test.  I don't know, but suspect, that needing 30000 expected calls is a symptom of a different issue.

Ryan

--

[Vitaliy Bortsov]

I attach a part of my working project.

I want to discuss with TDD followers for some problems regarding that project.

1. File organization.
  That archive has two eclipse projects: for target and for PC (cygwin, gcc, in directory tests). Test project use production code and test code. Production code use RTOS (FreeRTOS) and for now consist of some base blocks irrelevant to RTOS. There are tests for those base blocks.
  But when I go writing tests for RTOS tasks then it is too uncomfortably to use production code, it is better to use mock instead of production code (break dependency). For one test project, it is impossible (one executable cannot use production code and mock functions with the same name). Do I need to create many test projects (one test project for one task) instead of one test project? What is best approach?

2. Problems with big number call of mock()
  File tests/testsrc/MeasureRValveExp_test.cpp has  IGNORE_TEST(MeasureRValveExp, shortCircuit) which is hangs up PC. As Ryan said, there are an issue with that test. Can you help me to detect an issue?

Best regards,
  Vitaliy Bortsov

[James Grenning]

On 28 Oct 2014, at 22:56, Vitaliy Bortsov wrote:

I attach a part of my working project.

I want to discuss with TDD followers for some problems regarding that

project.

1. File organization.

That archive has two eclipse projects: for target and for PC (cygwin,

gcc, in directory tests).

Test project use production code and test code.

Production code use RTOS (FreeRTOS) and for now consist of some base blocks

irrelevant to RTOS. There are tests for those base blocks.

But when I go writing tests for RTOS tasks then it is too uncomfortably

to use production code, it is better to use mock instead of production code

(break dependency).

Testing your RTOS independent of you product seems like a good idea. Are these contract tests, tests that tell you if the RTOS works as expected.

For one test project, it is impossible (one executable

cannot use production code and mock functions with the same name).

If you are doing link-time substitution, a suitable choice for swapping out the real RTOS, then you only get one global instance of any symbol.

Do I

need to create many test projects (one test project for one task) instead

of one test project? What is best approach?

You can just have different makefiles or make targets.

2. Problems with big number call of mock()

File tests/testsrc/MeasureRValveExp_test.cpp has

IGNORE_TEST(MeasureRValveExp, shortCircuit) which is hangs up PC.

If you are referring to the expectNCalls(30000, ...), for starters, don't use a mock. Use a hand crafted fake that counts the calls. Does something special happen on call 30001. Your own fake will eliminate the test execution time problem.

As Ryan

said, there are an issue with that test. Can you help me to detect an issue?

I think Ryan and I were suggesting that you should not need 30,000 loops and calls in a unit test. What are you testing? Is the code in your zip file?

It is more likely a problem with the structure of your code that leads to an inconvenient test.

James

Best regards,

Vitaliy Bortsov

--

You received this message because you are subscribed to the Google Groups "cpputest" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cpputest+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[project.zip]

[Vitaliy Bortsov]

Hello, James.
It seems a good idea to have some targets to build some executables and only one eclupse project. Thank to advice.
Only what I need is to move test files to two directories (for example, "base" and "tasks") and write other makefile.

For link-time substitution in tests for RTOS tasks, I want to change not only RTOS calls but some of base blocks from production code.

Regards,
  Vitaliy Bortsov

[James Grenning]

Vitaliy,

I misunderstood before that you had tests for your RTOS, but what I
think now is that you have tests for your RTOS tasks.

Presuming that is right, you should design and structure your code so
that code that knows about the RTOS is minimal. Looking at your
MeasCurrentEMVtask, and not digging too deeply

Things to stub or mock
- vPortTaskUsesFPU
- MainDelay

Seems that the pwm instance should be passed to MeasCurrentEMVtask
though void *p. The pwm object could have updateDuty as a virtual
function. That way you can substitute a mock pwm object and check that
the right duty cycle is set. Where are pwm::on and pwm::off being used?
I don't have the whole picture.

In general, the code I am looking at does not separate and minimize the
code dependent on hardware from the code that is using the hardware for
some application purpose. You should be able to separate these items.

What RTOS task would you like a look at? That was just the first one I
grabbed.

James

------------------------------------------------------------------
James Grenning Author of TDD for Embedded C
www.wingman-sw.com http://pragprog.com/titles/jgade/
www.wingman-sw.com/#blog
www.twitter.com/jwgrenning

[Bas Vodde]

Hi,

As others pointed out, this is CppUTest problem. The Mock implementation has not focused at all on “performance’ as it assumes that there won’t be hundreds or thousands of mocks in a unit test.

I’ve bumped to this a couple times and think it’ll need to be fixed. I believe there is a search over two lists leading to O2 which was the real problem. I had no concrete way of solving it but never spend more than 5 minutes on the problem :(

There is an open github issue for this already.

Bas

[James Grenning]

It seems that you could take advantage of the "N" in expectNCalls, by
adding a counter.

It does not deal with O2 directly, but makes these kind of problems,
based on N, go away.

[Bas Vodde]

Yup, agreed.

The composite is now implemented as purely doing N times something and it could be done different. The current implementation was the easiest I could think of at the time.

It *might* be the best solution as the only time the problem has happened was with ExpectN… but that solution would mean it is still slow with lots of mocks, which I guess is ok…

Bas

[A. Robert S.]

Hi again,

 

It is possible to do runtime substitution by using function pointers. You can then explicitly set the pointer to use your mock, or a handwritten stub, at any point in your test.

 

Robert

[A. Robert S.]

https://github.com/arstrube/FakeFunctions

[James Grenning]

Vitaliy is also using C++, so he can use virtual functions too.

[A. Robert S.]

Of course. Function pointers are more flexible, though. Then again, he could wrap them more elegantly ;-)

[Vitaliy Bortsov]

Thank you, Robert.
I forget about run-time substitution using function pointers (did not use it before). It is more elegantly than two executables and special tests file organization.
For now, I don't know about using virtual functions (not read to that in Jeff Langr book).

Regards,
  Vitaliy

четверг, 30 октября 2014 г., 18:31:07 UTC+5 пользователь A. Robert S. написал: