rez-test

[allan.johns]

Hi all,

I'm thinking about adding a 'rez-test' tool in the near future.

The idea is pretty simple. Oftentimes a Rez package cannot be directly tested with unittest etc because you first need to be in that package's resolved environment in order for its dependencies to be available. Furthermore, you need to know what command to run to test the package in the first place.

The rez-test tool would just act as a simple test harness. Packages would have two new attributes:

test_requires: Any extra packages required for testing. Analogous to build_requires for building.

test: Command to run to perform a test.


Having this in place will help simplify things for studios wanting to put together an automated testing framework, since it'll provide a standard interface for running test on a package, regardless of whether that package is using unittest, some other testing framework, or something custom.

It might also make sense to support multiple different tests for a single package (you might want a unit test, and an integration test). In that case the following attributes might work better:

tests:
    unit_test:
        requires:
        - unittest2
        command: python -m unittest discover -s {root}/tests
    integration_test:
        ....

A simpler form for a single test would probably also be supported:

test:
    requires:
    - unittest2
    command: python -m unittest discover -s {root}/tests


In this case, running rez-test on a package with multiple tests will print the tests available, in much the same way rez-help does if there are multiple help entries. An optional globbed arg could be supplied to run all the tests that match the glob. Eg:

# list tests / run a single test
]$ rez-test foo
# run all tests
]$ rez-test foo '*'
# run all unit tests
]$ rez-test foo 'unit*'

It would also be useful to set a test so that it gets run before a release, perhaps controlled via a package attribute 'on_release' or similar. If the test fails, the release is aborted.

Thoughts? Mark I remember you had talked about a rez-test tool for testing purposes a while back, hence the rez-selftest rename.

Cheers
Allan

[Mark Streatfield]

Thoughts? Mark I remember you had talked about a rez-test tool for testing purposes a while back, hence the rez-selftest rename.

Yep, this is the sort of thing I was thinking of.

We currently have a standard target created via CMake that we use to run tests as build time.  Developers can (for most packages) pretty reliably run 'rez build -i -- -- all_tests'.  At release time the 'all_tests' target in make is automagically added to the 'all' target ensuring they are always run.  The 'all_tests' target calls other targets (which can also be invoked manually) to execute different category of tests (e.g. c++ via cppunit, python via nose, mayapy via nose etc).  

We also force pylint and epydoc runs in a slightly similar manner.

Current issues we have are:

* Sometimes we want extra packages at test time only.  Although when we've talked this through we actually talking about integration tests which are actually testing a number of (in development packages) which can't be satisfied through the current package alone.

* We want some (but not necessarily all/the same) commands to be applied to the test environment.

* Sometimes the test framework is linked to a build target - e.g. cppunit is making an executable so must be a build rule as well as then added to the add_test target.  It kinda all works out but does mean they are coupled to the build process (so if you haven't built the test executable then 'rez tests' would fail anyway).

So it sounds like we are thinking along similar lines.  I'd be curious to see how this ties into the build process and system however.  If they really can be completely separate/arbitrary commands or there is some link there.  For example, if the build process creates the test executable, how does the 'rez test' command know where to find it?  Given my example above with 'all_tests' would my configuration become:

test:

    command:

        rez build -i -- -- all_tests

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[allan.johns]

On Tuesday, April 28, 2015 at 3:35:39 AM UTC-7, Mark Streatfield wrote:

Thoughts? Mark I remember you had talked about a rez-test tool for testing purposes a while back, hence the rez-selftest rename.

Yep, this is the sort of thing I was thinking of.

We currently have a standard target created via CMake that we use to run tests as build time.  Developers can (for most packages) pretty reliably run 'rez build -i -- -- all_tests'.  At release time the 'all_tests' target in make is automagically added to the 'all' target ensuring they are always run.  The 'all_tests' target calls other targets (which can also be invoked manually) to execute different category of tests (e.g. c++ via cppunit, python via nose, mayapy via nose etc).  

We also force pylint and epydoc runs in a slightly similar manner.

Would using the release_hook 'commands' plugin work here? I don't think the various paths (install path, source path) are exposed to this plugin right now, but that could be fixed (I think I have a ticket for that).
 

Current issues we have are:

* Sometimes we want extra packages at test time only.  Although when we've talked this through we actually talking about integration tests which are actually testing a number of (in development packages) which can't be satisfied through the current package alone.

* We want some (but not necessarily all/the same) commands to be applied to the test environment.

* Sometimes the test framework is linked to a build target - e.g. cppunit is making an executable so must be a build rule as well as then added to the add_test target.  It kinda all works out but does mean they are coupled to the build process (so if you haven't built the test executable then 'rez tests' would fail anyway).

So it sounds like we are thinking along similar lines.  I'd be curious to see how this ties into the build process and system however.  If they really can be completely separate/arbitrary commands or there is some link there.  For example, if the build process creates the test executable, how does the 'rez test' command know where to find it? 

I was thinking that they would be entirely separate. rez-test would be a tool you'd run on a package that's been installed/released. Similar to the 'building' variable in rex, we'd introduce a 'testing' variable. What you might typically do is install your test executable to say, {root}/test_bin, then do something like:

def commands():
    env.PATH.append("{this.root}/bin")
    if testing:
        env.PATH.append("{this.root}/test_bin")

Or maybe just do this:

test = "{this.root}/bin_test/mytestprogram"
 
If you don't want to build the test executable every time, it would just be up to the user to make sure the test executable is built before trying to run rez-test.

Given my example above with 'all_tests' would my configuration become:

test:

    command:

        rez build -i -- -- all_tests

No I don't think that would make sense, since it'd mean rez-test would only work if you're in the source directory of the package, which isn't the intention.

The question is, how do we support release-time tests, so that a failed test aborts the release? Using release_hook/cmake targets only gets you so far, since (a) your package isn't installed yet and (b) test-only dependencies aren't available.

What if we had some sort of 'test_build' entry in the package file? Eg:

test_build:
  build_args:
  sub_build_args: all_tests

Then rez-build could have a --test option. This would perform a build using the args in 'test_build', would install to a temp directory, and would then run rez-test on the temporary install. Rez-release would do much the same thing, but would do it by default as part of the release process. Although this would mean installing the package twice (once to a tempdir, once to install path) - I can't see a way around that though if you want to test the package in a fully configured environment before releasing.



 

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Mark Streatfield]

Hi Allan,

Sorry, I realise I didn't respond to the questions in your last post,  

We also force pylint and epydoc runs in a slightly similar manner.

 Would using the release_hook 'commands' plugin work here?

While it would probably work, I don't think it's the right place.  In my mind, pylint/epydoc/nose2 are intrinsically part of the build process and so I want them configured in the CMakeLists.txt file as much as possible.  Whereas I see the hooks as being more useful for the administration type work, such as sending emails.  But that's primarily personal preference I guess.

The question is, how do we support release-time tests, so that a failed test aborts the release?

At the moment, we do this as a target in cmake so if the tests fail, the build fails, and the release fails.  While it's taken some coercing to make the tests run without the package being installed, I think that is ok.  I think it's made the tests more loosely coupled and actual unit tests as opposed to the hybrid unit/integration/functional tests we've had before (which is where I think some of what you describe is more applicable).  

We've also used rpathing to make this possible in C++ land.  At build time everything is rpath'd and at install time (local or central) the rpath is stripped which means we still play nicely with rez overall.  CMake supports this natively and it's working pretty well.

(b) test-only dependencies aren't available

In our case these are private_build_requires; because we're doing the unit test at build time that works ok. 

All that said, we've been talking about this again recently, we're going to collect a few different use cases we have, and will see how they fit with what you have described and report back.

Mark.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[Daniel Flood]

Hi Mark,

I'm trying to remember how this was done when I was working with you guys in 2015.  

I have setup a cmake target which I can run like so:

rez-build -- -- all_tests

Which works fine if invoked manually.  However I'm trying to figure out how to have it halt the build if it fails.  Do you use

 if(CENTRAL)

To figure out if running from rez-release-git?

cheers,

Dan

[Mark Streatfield]

Hello!

Yes, we used the CENTRAL CMake variable combined with the special ALL target to have the tests always run when doing a final release. 

By adding it to ALL you ensure it runs when a target is not specified (which is the case on a final release). 

There is a newer variable you should use however as CENTRAL is a hangover from Rez 1. I don't have access to the code but I think it is BUILD_TYPE or something similar (still in CMake but also as an environment variable).

To fail the build (and therefore the release) you should only need to make sure your test command (which in our case was normally nose of a cppunit compiled executable) returned a failure exit code.

Hope it helps! 

Mark

You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/TnWW4Ne-7II/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.

Visit this group at https://groups.google.com/group/rez-config.

[Daniel Flood]

Brilliant, thanks Mark!

[Fede Naum]

Hey Daniel, 

It is as Mark described, but I think we only use `If CENTRAL` and  `BuidType` is only used inside the code, I don't remember being exposed as an env variable.

I'll do a search tomorrow, but I bet Mark's memory is better than mine.

F

On Wednesday, 4 May 2016 13:19:29 UTC+10, Daniel Flood wrote:

Brilliant, thanks Mark!

[Mark Streatfield]

We are both right.

It is not an environment variable, but passed explicitly to CMake as a -D argument.  It's handled here.

--

[Daniel Flood]

Hey Fede!

Thanks for all the info guys, I have the halt-release-on-failed-test workflow working nicely.  We're on rez 1 for now, but this is valuable info and will save me alot of head scratching when we move to rez 2. 

cheers,

Dan

[Fede Naum]

Thanks Mark!!

Glad to hear Daniel that you got  that working. make a push to go to rez2 it's worth the effort.

Cheers

F

[Daniel Flood]

Hey guys,

So we moved to rez2!  I tried printing out the CMAKE_BUILT_TYPE variable during rez-build and rez-release, but found it is always "Release".  So I'm not sure how to test whether a rez-release is happening or not.

message("cmake build type: ${CMAKE_BUILD_TYPE}")

any ideas?

cheers,
Dan

[Mark Streatfield]

Hello, 

CMAKE_BUILD_TYPE is a CMake variable that controls how executables and libraries are built. Generally it's always Release, however Debug and RelWithDebug are among some other values. Ultimately it sets some default options to be passed to the compiler and isn't affected by Rez directly.

I think the variable you are after is REZ_BUILD_TYPE or something very similar to that. I don't have access to the code however so unable to checking directly.

Mark

--

[Thorsten Kaufmann]

From rezconfig.py:

# If True, Rez will continue to generate the given CMake variables at build and
# release time, even though their use has been deprecated in Rez-2.  The
# variables in question, and their Rez-2 equivalent (if any) are:
#   REZ-1               REZ-2
#   -----               -----
#   CENTRAL             REZ_BUILD_TYPE
rez_1_cmake_variables = True

So yes it should be REZ_BUILD_TYPE

On Tuesday, July 12, 2016 at 11:43:50 AM UTC+2, Mark Streatfield wrote:

Hello, 

CMAKE_BUILD_TYPE is a CMake variable that controls how executables and libraries are built. Generally it's always Release, however Debug and RelWithDebug are among some other values. Ultimately it sets some default options to be passed to the compiler and isn't affected by Rez directly.

I think the variable you are after is REZ_BUILD_TYPE or something very similar to that. I don't have access to the code however so unable to checking directly.

Mark

On Tuesday, 12 July 2016, Daniel Flood <daniel....@gmail.com> wrote:

Hey guys,

So we moved to rez2!  I tried printing out the CMAKE_BUILT_TYPE variable during rez-build and rez-release, but found it is always "Release".  So I'm not sure how to test whether a rez-release is happening or not.

message("cmake build type: ${CMAKE_BUILD_TYPE}")

any ideas?

cheers,
Dan

--
You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/TnWW4Ne-7II/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rez-config+unsubscribe@googlegroups.com.

[Daniel Flood]

Thanks again guys, worked like a charm.

[Daniel Flood]

I ran into an interesting issue with rez2.  I'm doing something like this to halt the release if tests fail, although I get the same problem if I just halt any old build if tests fail:

method 1: halt for build failure:

add_custom_target(unit_tests ALL DEPENDS myTestRunner)

method 2: halt only on release:

if(REZ_BUILD_TYPE STREQUAL "central")
    add_custom_target(unit_tests ALL DEPENDS myTestRunner)
endif()

 
In both cases the release is halted if a test fails on release, but this happens *after* the new package directory has been created in the release location.  This will then cause a rez-env conflict because there is a released package with no package.py.

I never really paid attention to how this worked in rez-1, but I'm pretty sure the behaviour was different, i.e. if the release halted it never wrote out a versioned package directory in the release location.  I had a look through rezconfig.py and couldn't see anything that seemed relevant to this.  Looking at the rez documentation, it sounds like this is a bug?

http://nerdvegas.github.io/rez/#rez-release

If all variants build successfully, then each is installed centrally.

[Allan Johns]

Hey Daniel,

It's a design flaw more so than a bug, but it's one I've coincidentally just fixed! I'll have it merged into master soon, after a few days of testing at my work. In the meantime just manually delete the empty released directory to fix the problem.

Thanks,

A

--

You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[Daniel Flood]

Cheers Allan! 

Dan

[Daniel Flood]

Just curious if this made it into the official rez-2 release?

cheers,
Dan

[Allan Johns]

Hey Daniel,

It did not, but I'll be merging it in this week, probably Friday. 2.0.1 is coming!

A

--

[Daniel Flood]

Hey Allan,

Let us know if you get this fix merged, I'm happy to test out a dev branch if you can point me to one. 

cheers,
Dan

[Allan Johns]

Hey Daniel,

Next week is my last week of work in the USA before I move back to Australia! And part of that week will involve catching up on some Rez PRs and moving my studio up to the newest version. This fix will be part of that work. It's waiting on a branch internally. I'll respond on this thread again as soon as it's done. I'm also sticking to formal release notes on the github page, so you'll see it there!

A

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Daniel Flood]

Hey Allan,

Just wondering if you've had a chance to look into this issue,

cheers,
Daniel