Jenkins running unit tests MSTest 10.0

[Simon Whale]

Hi, 

I am wondering if someone can help.  I have setup Jenkins to run Unit Tests.  Our application currently has 289 tests.   

If I run it through Jenkins only 89 pass.  But If I run it through VS 2010 247 pass. 

Is there something that I should be doing in Jenkins? 

This is how I have configured the Execute Windows batch command in the build settings for the project. 

"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /resultsfile:"%WORKSPACE%\Brian\Results.trx" /testcontainer:"C:\Program Files\Jenkins\workspace\Brian - Freestyle\Brian\trunk\BrianTestProject\bin\x86\Debug\BrianTestProject.dll" /nologo

[Mark Waite]

If your Jenkins agent is running as a service, then that may cause different behaviors if your tests depend on interacting with a user interface (like Selenium tests, or web browser driven tests).

If your Jenkins agent is running as a different user, and your tests depend on a specific user, that could cause different behavior in Jenkins than in the user environment.

If your Jenkins agent is running on a different Windows version and your tests have Windows version requirements, that could cause failures.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c5fc26e9-a2d7-48d2-a8e3-b561e2524547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jer...@bodycad.com]

Many thing can cause this

• unreliable tests, where concurrency can occur or uninitialized values
  
• Windows permission
• invalid path inside the machine
• the user under which the salve run the tests (service under windows don't have access to the GUI part and you need to run the salve under a normal user account, OpenGL, DirectX, Canvas, Qt, etc)
• Make sure the number you get is valid with MsBuild command line and not Visual Studio (they aren't totally the same when it come to project behavior).
• Make sure you use the right visual studio/MsBuild version along the right architecture (x86 vs x64).
• Try to run your tests multiple times into random order if the test suite allow it (help catch some problems).
• Make sure you development env and jenkins slave have the Fault Tolerent Heap disabled (world worst idea ever was born on the day that damn thing was created)https://msdn.microsoft.com/en-us/library/dd744764(VS.85).aspx

A failed tests even if not always happening (once in a blue moon also apply) is always a bug somewhere.