How to run tests conditionally

[Nikolaus Rath]

Hello,

In order for my unit tests to work, they have to be called with root
privileges, or one of the built executables has to be made setuid.

What is the best way to integrate the tests with Meson/Ninja?

My first attempt was to simply declare the tests with test(), and have
them exit with status 77 if pre-requisites aren't met. However, this
means that Ninja will still attempt to run all the tests, and they will
all be skipped. So either I will get no error message at all, or I will
get an error message for every test.

My second attempt was to define the tests conditionally, i.e. execute
test() only if the user is root or if the executable is setuid. But here
the problem is that on the first build, the executable has not yet been
built when the meson.build file is interpreted.


Ideally, I would like to define one "pre-test" that is guaranteed to be
executed before all the others. This pre-test could check for
root/setuid, and (if neither is the case) fail (rather than skip) with a
helpful message and prevent the other tests from being run.

Is there a way to do that? Or is there yet another way?

Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Jussi Pakkanen]

On Fri, Jan 6, 2017 at 6:38 PM, Nikolaus Rath <Niko...@rath.org> wrote:

> My first attempt was to simply declare the tests with test(), and have
> them exit with status 77 if pre-requisites aren't met. However, this
> means that Ninja will still attempt to run all the tests, and they will
> all be skipped. So either I will get no error message at all, or I will
> get an error message for every test.

Why is this a bad thing? Skipped tests are presented in the statistics
in their own column so they won't get mistaken for failed tests. This
is also the approach taken by e.g. Wayland.

> Ideally, I would like to define one "pre-test" that is guaranteed to be
> executed before all the others. This pre-test could check for
> root/setuid, and (if neither is the case) fail (rather than skip) with a
> helpful message and prevent the other tests from being run.

It is recommended that you make all tests runnable as standalone
without any pretests needed. Meson exposes all test information so
people can run e.g. custom test runners or integrate a "run test X
under debugger" inside their IDE or other editor. Adding pre- (or
post-) conditions or steps complicates all tooling that works together
with Meson.

We are also currently working on a better way to tag test suites
(https://github.com/mesonbuild/meson/pull/1279), so hopefully in the
future you can, if you want, put all tests that need root rights to a
single suite and then do something like this:

mesontest --remove-suite needroot

[Nikolaus Rath]

On Jan 06 2017, Jussi Pakkanen <jpak...@gmail.com> wrote:
> On Fri, Jan 6, 2017 at 6:38 PM, Nikolaus Rath <Niko...@rath.org> wrote:
>
>> My first attempt was to simply declare the tests with test(), and have
>> them exit with status 77 if pre-requisites aren't met. However, this
>> means that Ninja will still attempt to run all the tests, and they will
>> all be skipped. So either I will get no error message at all, or I will
>> get an error message for every test.
>
> Why is this a bad thing? Skipped tests are presented in the statistics
> in their own column so they won't get mistaken for failed tests. This
> is also the approach taken by e.g. Wayland.

Mostly because *all* the tests would be skipped, so there'd be no point
in trying to run them. Furthermore, it wouldn't give the user any
indication of *why* everything has been skipped.


That said, I just realized it wouldn't be a good idea to declare the
unit tests individually at all. While they can run on their own, they
are designed to be called from a different test runner
(pytest). Starting them separately will incur a huge performance
penalty, the output will be confusing (when run standalone, each tests
considers itself to be a test suite with just one test), and I'm loosing
many of the pytest features.


At first I thought I'd just declare a single "run pytest" test for
Meson, but that's not a good idea either: there is no progress reporting
(since Meson or Ninja captures the test output in a file), and the
pytest output isn't colored anymore (because it's not connected to a
terminal).

So I think I'll just stick with the existing test suite and require
users to call 'python -m pytest' instead of 'ninja test'. The only
difficulty is that the tests are stored in the source directory, but the
binaries-to-be-tested are in the build directory. So I have to pass one
of them to the test runner explicitly...

[Jussi Pakkanen]

On Sat, Jan 7, 2017 at 1:47 AM, Nikolaus Rath <Niko...@rath.org> wrote:

> So I think I'll just stick with the existing test suite and require
> users to call 'python -m pytest' instead of 'ninja test'. The only
> difficulty is that the tests are stored in the source directory, but the
> binaries-to-be-tested are in the build directory. So I have to pass one
> of them to the test runner explicitly...

For this use case you might want to create a run_target with all the
necessary arguments so you can tell people to just run 'ninja
yourtesttarget'.

[Nikolaus Rath]

Good idea! This works nicely.

Is there a way to print something helpful if the user runs "ninja test"
instead? I wrote a pseudo test that fails with a message telling the
user to run the custom test target instead, but I have trouble declaring
it as a test:

# Provide something helpful when running 'ninja test'
not_here = custom_target('not_here.sh', input: fname, output: fname,
command: ['cp', '-fP', '--preserve=mode',
'@INPUT@', '@OUTPUT@'])
test('not_here', not_here)

gives:

Meson encountered an error in file test/meson.build, line 36, column 0:
Second argument must be executable.

[Jussi Pakkanen]

On Sat, Jan 7, 2017 at 2:33 AM, Nikolaus Rath <Niko...@rath.org> wrote:

> # Provide something helpful when running 'ninja test'
> not_here = custom_target('not_here.sh', input: fname, output: fname,
> command: ['cp', '-fP', '--preserve=mode',
> '@INPUT@', '@OUTPUT@'])
> test('not_here', not_here)
>
> gives:
>
> Meson encountered an error in file test/meson.build, line 36, column 0:
> Second argument must be executable.

test('not here', find_program('not_here.sh'))

[Nikolaus Rath]

On Jan 07 2017, Jussi Pakkanen <jpak...@gmail.com> wrote:

It seems a little odd to have Meson "find" the program when it is
supposed to be generated by the custom target. Is there some deeper
logic here that I am missing?

[Jussi Pakkanen]

On Sat, Jan 7, 2017 at 7:19 AM, Nikolaus Rath <Niko...@rath.org> wrote:

>> test('not here', find_program('not_here.sh'))
>
> It seems a little odd to have Meson "find" the program when it is
> supposed to be generated by the custom target. Is there some deeper
> logic here that I am missing?

From what I can tell your custom target just copies the file from
source to build dir. You don't need to do that. Find_program detects
the script from the source dir and then the test runs it from there.
No need to copy anything.

[Nikolaus Rath]

On Jan 07 2017, Jussi Pakkanen <jpak...@gmail.com> wrote:

Yeah, that's true. But I was thinking about the more general
case. Suppose the script is generated by a more complicated process. Why
do I have to instruct meson to "find" a target that it knows how to
generate?

[Jussi Pakkanen]

On Sat, Jan 7, 2017 at 10:41 PM, Nikolaus Rath <Niko...@rath.org> wrote:

> Yeah, that's true. But I was thinking about the more general
> case. Suppose the script is generated by a more complicated process. Why
> do I have to instruct meson to "find" a target that it knows how to
> generate?

When custom_target was originally created it was meant for generating
data. The idea that it could be used for generating runnable scripts
was not thought about. We have fixed a bunch of places where custom
targets are used as executables but not all of them (as evidenced
here).

[Nikolaus Rath]

Is there a way to make this work multiple times? At the moment, ninja
refuses to run the tests more than once because there is "no work to do".

[Jussi Pakkanen]

On Wed, Jan 11, 2017 at 12:39 AM, Nikolaus Rath <Niko...@rath.org> wrote:

> Is there a way to make this work multiple times? At the moment, ninja
> refuses to run the tests more than once because there is "no work to do".

Run targets can be run multiple times and it will always run it anew.
However there is a slight nag in the implementation of Ninja. If you
generate a file that has the same name as your run target, Ninja will
interpret that as being up to date. Which kind of makes sense because
if you ask it to generate X and it already exists, then it can
reasonably assume that nothing needs to be done.