Programmatically React to Test Runs

[Sebastian P.]

Hey guys,

I'm would like to react to the result of test executions in the ReSharper/NUnit test runner. For example, I'm interested in which tests were executed, how long they ran, which ones failed, etc... I saw that it is generally possible to write an addin for NUnit, but it seems that users would have to install that separately. Is there a way to access the NUnit extension manager from within my ReSharper plugin?

Any pointers are appreciated. Thanks!

best

Sebastian Proksch

[Matt Ellis]

Yes, you can wire up to the signals that show a test session (the tab in the runner window) opening + closing, and for each session, when tests are run and what the results are. You can see it being done here, as part of the Clippy extension: https://github.com/citizenmatt/resharper-clippy/blob/master/src/resharper-clippy/src/UnitTestAnimations.cs

It uses the IUnitTestSessionConductor interface to wire up to the SessionOpened and SessionClosed signals, and for each session, listens for changes to the session's Launch property, which is what describes the tests being run. When the Launch changes, it subscribes to the Launch's State property, which tells us when the session is running, aborting, stopping, idle, and so on. Once we know that the session completes, you can use the IUnitTestResultManager to get at the results themselves.

The code makes good use of the Lifetime construct to manage how long the subscriptions last. The main component gets a Lifetime that is scoped to the solution, and this is used to subscribe to the SessionOpened/SessionClosed signals. When a session is opened, a new Lifetime is created, as a child of the component's lifetime (so if the solution closes, this new child lifetime is also terminated). This new lifetime is stored in a dictionary, and terminated when SessionClosed fires. The Launch property uses SequentialLifetimes, which maintains a single lifetime and automatically terminates the previous one when creating the next new one.

Hope that makes sense!

Regards

Matt

[Sebastian P.]

That is exactly what I was looking for. Thanks a lot!

best

Sebastian

[Sebastian P.]

I just read the blog post about the ReSharper 2016.1 release and started wondering if the proposed solution will be affected by the switch to the xUnit test framework?

[Matt Ellis]

Nope. This solution reports on the test elements that are being run, but those elements can come from any provider, including nunit, xunit, mstest, js, etc.

[Chris Freeman]

I've got some code very similar to this which works in R# 2016.1. But while trying to port to R#2016.2, the SolutionComponent attribute is no longer defined in the SDK. How do I register my solution component under 2016.2?

[Matt Ellis]

SolutionComponent is definitely still in the SDK - have you restarted Visual Studio after updating the nuget packages? NuGet rewrites various imports, which are only evaluated when a project is reloaded.

Regards

Matt

[Sebastian P.]

Are you sure that you are not missing the right using directive? Did you try to re-open the solution to make your code compile?

We are also using [SolutionComponent] in our 2016.2 code (see [1]) without problems.

best

Sebastian

[1] https://github.com/stg-tud/kave/blob/master/feedback-generator/KaVE.VS.FeedbackGenerator/Generators/ReSharper/TestRunEventGenerator.cs

[Chris Freeman]

Thanks to both Sebastian and Matt - restarting VS two or three times worked.

[Sebastian P.]

I'm wondering if there is a similar solution to react to the various stages of builds. I found several assemblies that sound related to building (e.g., Jetbrains.Platform.Build*) and I searched for similar interfaces that I could request for injection (e.g., *Manager or *Conductor), but I did not found anything so far.

I don't want to intercept anything during the build, I am just interested in the mere fact that a build is running, preferable with the information about which targets are being built and whether the build was triggered automatically or by the user. Is it possible to get this information from the public API?

best

Sebastian

[Matt Ellis]

You can try the BuildIsRunning solution component. This is a simple class with two observable properties (IProperty, that you can subscribe to with a Lifetime) that fire when a build is running, or when it's cancelled. You don't get any other information. (ReSharper uses this to e.g. pause solution wide analysis while a build is running)

Or you can try ISolutionBuilder, which is where BuildIsRunning gets its information. It's a bit more complex to use - it exposes a IProperty<SolutionBuilderRequest>. You subscribe to this with a Lifetime (e.g. builder.RunningRequest.ForEachValue(…)), and get notified every time a new value is set. This is a running build, which you can look at for more information, such as subscribing to the State property to hear when a build starts or completes.

Regards

Matt

[Sebastian P.]

Cool, seems to work... if prefer the latter solution though. :)

You will find my complete "proof-of-concept" on pastebin [1], I have some short follow-up questions. Is going through the requested projects to capture the build state the right way to get notified about intermediate build events (l.24)? Also, what is the diff between ISolutionBuilder.Projects and ISolutionBuilder.RequestedProjects, they always seem to contain the exact same projects...

On a related note... to improve my example, shouldn't I somehow derive a Lifetime for the state listeners (l.25) that is ended after the build ends? For example, if somebody removes a projects, the lifetime I provide to the listener is not invalidated (because the solution is still active) so -without knowing your implementation- I would assume that the state listener is not disposed.

best

Sebastian

[1] http://pastebin.com/FXjhmSBq