Recommendation for functional testing framework

[Andreas S]

We're starting to use Google C++ Testing for unit testing, but we also
need a framework for expressing functional tests of programs. In our
functional tests we give the program an input and checks to see if it
returns the expected output. The expected output can either be text or
an exit status ( pass/fail/may fail). We've looked at DejaGNU, but it
requires cygwin on Windows so it is not an ideal solution. Do you have
a functional testing framework to recommend?

Best,
Andreas

[Steve Jaffe]

You can do this with google test by writing a little routine that runs the
program and captures/loads the output and does the comparison with the expected.
I've been doing that and it works very well.

Steve

[Steve Jaffe]

Not really, what you want to do is make use of the "value-parameterized tests" (http://code.google.com/p/googletest/wiki/AdvancedGuide#Value_Parameterized_Tests) You read a configuration file to specify the tests to run, the inputs and expected outputs, then provide that to the cases using the INSTANTIATE_TEST_CASE_P macro with the "container" argument.

Having written that I realize it is perhaps not completely obvious :-) I actually spent a bit of time designing it. If you need more details let me know.

Regards,

Steve

---

From: Andreas Saebjoernsen <and...@digitalplaywright.com>
To: Steve Jaffe <steve...@yahoo.com>
Cc: Andreas S <andreas.sa...@gmail.com>; Google C++ Testing Framework <googletes...@googlegroups.com>
Sent: Tue, February 8, 2011 4:49:03 PM
Subject: Re: [googletest] Recommendation for functional testing framework

Steve,

That seems like a simple and good solution, but maybe there's a bit more to implementing it than what I can see now. When you integrated the functional tests with the unit test suite, did you encounter any unexpected challenges?

Best,

Andreas

[Andreas Saebjoernsen]

That is a very useful tip. I'd like to work on a solution that integrates well with the autotools and cmake toolchains since I use both in my projects.

Please let me know if you'd like to share your code on github or somewhere similar where I can contribute to it.

Best,

Andreas

[Andreas Saebjoernsen]

Lior,

I agree that using Cucumber step definitions in C++ is definitely an interesting idea that has the additional benefit of providing a testable workflow. However, my assumption has been that the BDD toolchain does not apply well to functional testing of C++ so if you have some experience I'd like to ask you some questions. 

Does BDD tools solve the problem of whole program testing of command-line programs 

  - where an expected return state can either be fail, may fail, or pass? 

and/or

  - where each test run is provided with an expected output file for an input file, where the actual output file is compared with the provided (correct) output file? 

I've looked into CukeBins as a way to use Cucumber step definitions in C++. CukeBins has a googletest backend, and it seems very promising:

   https://github.com/paoloambrosio/cukebins

If you have experience with BDD in C++, do you have any recommendations?

Best,

Andreas

2011/2/6 Lior Friedman <lfri...@gmail.com>

Does it have to be c++ based?
You might wish to consider any of the BDD framework out there.
Something like NBehave, Robot, Cocumber, fit,....


Lior Friedman
Blog  - http://imistaken.blogspot.com

-----Original Message-----
From: googletes...@googlegroups.com
[mailto:googletes...@googlegroups.com] On Behalf Of Andreas S
Sent: Sunday, February 06, 2011 12:43 AM
To: Google C++ Testing Framework

Subject: [googletest] Recommendation for functional testing framework