Testing with CTest : Testing the same executable with more than one ".prm" and ".output" combination

[Bruno Blais]

Hello all,

I am re-implementing some tests on our solvers (following https://www.dealii.org/developer/developers/testsuite.html#layoutaddtests)

Right now, it is really easy to test individual applications with a combination of executable + output + prm file.

We also test with various number of processors.

However, I was wondering if there was way to test a single executable with more than one .prm file and corresponding output files

Ex: test.1.prm test.1.output, test.2.prm, test.2.output where test would be the executable in question.

Is that something that is feasible in the current framework?

If not, what would be your advice on the way to proceed? I could make identical executables that instantiate the same class, but that increases linking time for no reasons.

Best

Bruno

[Wolfgang Bangerth]

Bruno,

We've generally put the test in a function in the .cc file where that
function takes a filename as argument. The main() function then just
calls the function with each of the .prm files we have, and the output
file will simply contain all of the output of all .prm files
concatenated. If you add a .prm file, you'd have to adjust the main()
function and the .output file.

This might not be the very most elegant solution, but it's worked
reasonably well so far. The idea of the same executable (several .cc
files all #includeing the same .h file that contains the test code) of
course also works -- in the big picture, compile time and link time for
tests are not negligible, but also not a terrible burden.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Matthias Maier]

On Wed, Aug 14, 2019, at 13:38 CDT, Bruno Blais <blais...@gmail.com> wrote:

> Hello all,
> I am re-implementing some tests on our solvers (following
> https://www.dealii.org/developer/developers/testsuite.html#layoutaddtests)
> Right now, it is really easy to test individual applications with a
> combination of executable + output + prm file.
> We also test with various number of processors.
>
> However, I was wondering if there was way to test a single executable with
> more than one .prm file and corresponding output files
> Ex: test.1.prm test.1.output, test.2.prm, test.2.output where test would be
> the executable in question.
>
> Is that something that is feasible in the current framework?

For example:

Having in .../tests

test_1.prm
test_1.output
test_2.prm
test_2.output

with the following tests/CMakeLists.txt file:

SET(TEST_TARGET my_executable)
DEAL_II_PICKUP_TESTS()

will define two tests in which the target "my_executable" is run with
the corresponding parameter file. For this two work, you will need to
have an executable target somewhere else, for example under src:

src/CMakeLists.txt:

ADD_EXECUTABLE(my_executable ....)
DEAL_II_SETUP_TARGET(my_executable)

If you want to run more than one executable, you will need multiple
subdirectories (one for each executable).

Best,
Matthias

[Matthias Maier]

On Wed, Aug 14, 2019, at 18:55 CDT, Matthias Maier <tam...@43-1.org> wrote:

> [...]

This is actually explained here:

https://www.dealii.org/current/users/testsuite.html

:-D

[Matthias Maier]

Wolfgang,

there was a little bit of CMake development behind your back *cough*
*cough* (Luca wanted to have the feature Bruno was asking about).

Best,
Matthias

[Bruno Blais]

Dear Matthias,

If I understand correctly, the only constraint of working this way is that all tests for a single executable must be grouped in a single folder, and that a single folder can test a single executable?

That's perfect. I will try that tomorrow.

If I manage to make that work, it is exactly what I was looking for.

Thanks!

Best

Bruno

[Matthias Maier]

AND I apparently created a demo repository showcasing all of that:

https://github.com/tamiko/dealii-demo

o_O

[Bruno Blais]

Hahaha, that is amazing!

So much documentation that sometimes documentation becomes its own enemy :P

Thanks for everything! That will definitely be incredibly helpful (not only to me, but to others)!

[Matthias Maier]

On Wed, Aug 14, 2019, at 18:58 CDT, Bruno Blais <blais...@gmail.com> wrote:

> Dear Matthias,
> If I understand correctly, the only constraint of working this way is that
> all tests for a single executable must be grouped in a single folder, and
> that a single folder can test a single executable?
> That's perfect. I will try that tomorrow.

Exactly.

> If I manage to make that work, it is exactly what I was looking for.

Perfect!

Best,
Matthias

[Bruno Blais]

Dear Matthias,

I follow your repository and it works perfectly.

Thanks, I managed to go down from 11 executables to 3... :)!

Best

Bruno

[Matthias Maier]

Very happy to hear that :-)

I'd also love to get some feedback (in case you encounter some missing
functionality - the user base of this particular feature has just
increased from about 2 to 3 ^^)

Best,
Matthias