rez test extensions

[Aloys Baillet]

Hi,

We're trying to upgrade most of our test infrastructure to rez test (it is currently mostly calling a custom cmake target such as `rez build -- -- all_tests`).

Rez test is much cleaner as it allows specifying custom requirements for each test target, whereas we used to have to cram all our test requirements in the private_build_requires section.

We have a few ideas on how to improve rez test though, and I wanted to share them to get feedback if these would be worth contributing back.

1. Running `rez test` in a folder with a package file should be able to pickup the package and run the tests right there. One issue is that if the rez package is different from the installed rez package we would rather use the current package file, not the installed one as it allows for quick iteration on the test command while authoring the package file.
2. Adding a function to run additional commands as part of the test. This is mainly to be able to set custom environment variables as part of the test target. For now we have to create an external script to configure test env vars. It could look like this:
   
   def test_commands(targetName):
       prependenv('MYCUSTOMENVVAR', 'MYCUSTOMVALUE')
       if targetName == 'maya':
           prependenv('MAYAPLUGINPATH', '{root}/testplugin')
   
   tests = {'vanilla': {'command': 'nose2 mymodule.tests',

                       'requires': ['nose2']},
   

            'maya': {'command': 'mayapy -m nose2.main mymodule.tests',

                    'requires': ['nose2', 'maya-2018']}

   }
3. A new `rez test --env TARGETNAME` option to open a rez environment configured the same way the rez test is run, this is for development purposes and would allow saving on a rez rezolve every time we need to run the tests.

I hope these make sense, happy to discuss further and to possibly contribute these back if deemed appropriate!

Cheers,

Aloys

[Brent Villalobos]

Excuse me for replying to an old thread, but I want to "upvote" that first feature request for rez-test which is to use the local package.py if one exists.  Right now, to run unit tests and other test suites on the source code of a package, we first have to do a noop build to get the package deployed in the packages path so that rez-test can see it.

I'm pondering writing a small python script that uses the get_developer_package() function to load the local package.py and then parse the "tests" dictionary to run a given test.  Or is there a better way?

[Allan Johns]

I've talked about some of this in https://github.com/nerdvegas/rez/pull/759 but will reiterate here.

1.

RE using local package.py. I think it can make sense to source `tests` from here, but I don't think that the package should ever be getting implicitly installed as that PR does. In terms of running tests on the non-built/installed package, this does make sense for some tests (such as linting), but not for others, so we would have to treat these as a separate case. For example, perhaps you would add a "local=True" field for those tests. They would then be runnable in some 'local' rez-test mode (eg `rez-test .`, when in same dir as package.py).

So we have a few different cases:

* rez-test run on an installed package. Currently working, no issues

* rez-test on an installed package, but using local package.py for 'tests' attribute. Ok, but needs to be obvious and explicit

* rez-test on a package that hasn't been built yet. Ok too (eg linting), but again needs to be explicit, and we would need to identify those tests that this is valid for.

So I'm not against these features, but I think we need to be clear about how it works, and we need to ensure that it isn't too confusing to the user, before we implement it.

Off the top of my head, perhaps it would work like this:

* `rez-test .` puts us into "local" testing mode. In this mode, 'tests' from local package.py is always used

* In local testing mode, if a test has "local=True", then it is run on the uninstalled package (ie the source). If not, it's run on the installed package.

* local=True tests are stripped from installed packages.

2.

I like the proposal for 'test_commands', I think that makes sense. It would also make sense to go a little further... since you can run "command" in a rex function also, it stands to reason that you may want to do this in your 'test_commands' instead (perhaps the command differs based on the env). So it would also make sense for "command" in the tests dict entry to be optional.

Thx

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rez-config/7906dc4a-06e7-4b40-b0d7-1d245a629b96%40googlegroups.com.