Unable to load plugin: SpecRun - HELP!

[Arvind Virk]

Hello,

We are currently struggling to getting SpecRun up and running in our environment.

We have setup our app.config file as follows:

 <specFlow>
 <unitTestProvider name="SpecRun+MSTest" />
    <plugins>
      <add name="SpecRun" path="..\..\..\..\Temp" />
    </plugins>
</specFlow>

When we start a build we get the error:

 

##[error]   Class Initialization method UserInterfaceTests.Features.AdminCertificatesFeature.FeatureSetup threw exception. TechTalk.SpecFlow.SpecFlowException: TechTalk.SpecFlow.SpecFlowException: Unable to load plugin: SpecRun. Please check http://go.specflow.org/doc-plugins for details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'SpecRun.SpecFlowPlugin' or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF.

We have copied the plugin to the C:\ and even pointed set the path= attribute at it but it fails with the above error regardless. It seems to be ignoring the path we have set. We have read the documentation section "SpecFlow+ And TFS VSTS" all without success.

If we change the app.config UnitTestProvider to "SpecRun" we don't get the error above but instead it fails as it is not running it on the correct machine (ignoring the test agent we have specified)

My question are:

Do we need to keep it as SpecRun+MSTest so it respects test agents?

If yes, How do we get it to look in the right directory for the .dll?

If no, How do we get it to run the tests on the environment where the test agent is active.

Any advice on getting this up and running would be appreciated.

Thanks,

Arvind

[Andreas Willich]

Hi

Could you explain a litte bit more about your environment? It looks rather unusual.

The SpecFlow+Runner is installed via NuGet package and there SpecFlow searches the plugin automatically. You don't need to normally move it somewhere else.

You had probably a reason for this. Could you explain it?

The path has to the the path to the SpecRun.SpecFlowPlugin.dll.

Btw, which versions are you using?

Best regards

Andreas

--
You received this message because you are subscribed to the Google Groups "SpecFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to specflow+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arvind Virk]

Hello Andreas,

Thanks for your reply!

There has been a few developments since I made the original post so let me just try and give you an insight into our issue:

We currently have a TFS Build Server that kicks off the test run. 

We are testing SpecRun which doesn't seem to be interacting with the est current Test Agent/Controller infrastructure in the same way that MSTest did previously.

It actually appears it is trying to run the project on the BuildServer itself!

The SpecFlow+Runner is installed via NuGet package and there SpecFlow searches the plugin automatically. You don't need to normally move it somewhere else.
You had probably a reason for this. Could you explain it?

When we initially tried to use the app.config setting <unitTestProvider name="SpecRun+MSTest" /> we had an the "unable to load plugin..."  issue I mentioned in my initial post. We then looked at the SpecFlow+ documentation which mentioned we could set a path attribute and we assumed we could move the SpecRun.SpecFlowPlugin.dll file to an arbitrary folder and configure the app.config path argument to find it. 

This failed and we continually kept getting the above error.

Changing the "UnitTestProvider" name to "SpecRun" stopped the "unable to load plugin..." issue. 

NuGet shows us using

SpecRun.SpecFlow v.1.6.3

SpecRun.Runner v1.6.3

SpecRun.SpecFlow.2-2-0 v1.6.3

SpecFlow.2.2.0

Thanks,

Arvind

[Andreas Willich]

Looks more like a problem with your build than with your SpecFlow configuration.

Configure it in the app.config to SpecRun as unitTestProvider and remove the path.

After regenerating all feature files and rebuilding the project, you should see it in the test explorer in Visual Studio.

If there isn't anything, you won't get tests in your build.

Which tasks are you using in your TFS Build process? The normal VSTest task or the "Run functional test" Task?

If you are using the VSTest task, be sure to set the "Path to Custom Test Adapters" to the path of the nuget package folder when you build.

[Arvind Virk]

Hello Andreas,

We use a "VSTest" task in TFS Build which has a "Path to Custom Test Adapters" value set to the NuGet package folder.

I have followed your steps of setting the "Unit Test Provider" back to "SpecRun" and removing the path argument. I have also regenerated the feature files and can see all the tests in Test Explorer. 

The original error "Unable to load plugin SpecRun" has now been resolved but I still have an issue executing my tests. My issue now is that no tests are being executed on the TestAgent. In our [BeforeScenario] hook we check that we can see a directory created when the application has been installed. It is at this point that an error is now thrown.

When I kick off a BRANCH build in TFS Build it states that this directory does not exist although I am looking at it. I cannot see any tests fired at the TestAgent machine because I do not see any changes in the Test Agent Status window. I can however see TFS Build throwing errors as it is running the UI Tests.

I have temporarily removed the steps to check for this directory and then I get a complaint that "ChromeDriver has timed out after 60 seconds" when i start another build. I can see Build Server throwing errors but I cannot see any attempts to launch a ChromeDriver session in either the Task Manager of the TestAgent or in the Test Agent Status window.

It gives the impression that it is running the tests on the BuildServer directly and within the source folder. I am assuming this is the case as I cannot see any activity on the TestAgent machine.

Any ideas?

[Andreas Willich]

Ok, as now the tests are shown, it has probably something with the build tasks.

We are working on a documentation how to setup a build with the "run functional test" tasks, but here are screenshots of the configuration for a working build:

Visual Studio Test Agent Deployment Tasks:

Run functional test tasks:

During preparation for the example, I had problems to find the right combination of task versions and test agent version. Having everything set to the latest version and Visual Studio 2017 worked. If you had other versions of the test agent installed, you have to remove it manually. If I did it not, the test tasks was not able to execute the tests.

I hope this helps you.

BR

Andreas

[Arvind Virk]

Hello Andreas,

I am currently trying to change the our build so we can try and use "Run Functional Tests" rather than "VS Test Task". We are also on Visual Studio 2015 which will be add to the fun. Out of interest had we stuck to "VSTest Task", would simply setting the path to the test adapter had been enough?

Thanks,

Arvind

[Andreas Willich]

Yes, when you use the "VSTest Task" it should be enough to set the path to the test adapter.

[Arvind Virk]

Hello,

I attempted to use the "Run Functional Test" task but we are still unable to get tests to run.

We added a "Visual Studio Test Agent Deployment" step before the "Run Functional Tests" step. The "Agent" step appears to be running fine as we can see the files being copied correctly and even a dialog box opening when the "Run Functional Tests" step has started.

When the "Run Functional Tests" step has started we have the following output:

1. DistributedTests: test configuration mapping:
   2017-12-20T16:28:26.7436392Z DistributedTests: Test Run with Id 29644 Queued
   2017-12-20T16:28:26.8061205Z DistributedTests: Test run '29644' is in 'InProgress' state.
   2017-12-20T16:28:36.8374647Z DistributedTests: Test run '29644' is in 'Aborted' state.
   2017-12-20T16:28:46.8531665Z ##[warning]DistributedTests: Test run is aborted. Logging details of the run logs.
   2017-12-20T16:28:46.9000375Z ##[warning]DistributedTests: New test run created.
   2017-12-20T16:28:46.9000375Z ##[warning]Test Run queued for Project Collection Build Service (TEAM FOUNDATION).
   2017-12-20T16:28:46.9000375Z ##[warning]DistributedTests: Test discovery started.
   2017-12-20T16:28:46.9000375Z ##[warning]DistributedTests: Test Run Discovery Aborted . Test run id : 29644
   2017-12-20T16:28:46.9000375Z ##[warning]DistributedTests: UnExpected error occured during test execution. Try again.
   2017-12-20T16:28:46.9156785Z ##[warning]DistributedTests: Error : One or more errors occurred.--->Could not load file or assembly 'Microsoft.TeamFoundation.TestPlatform.Client, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
   2017-12-20T16:28:46.9156785Z ##[warning]DistributedTests: Test run aborted. Test run id: 29644

We have installed and re-installed Visual Studio on the "Agent" due to build complaints. We have executed the "Test Agent" it copies from the internet on the "Agent" also but cannot get past this error. I have tried including it in the build folder, in the references of my project and copying the .dll into the TestAgent folder the windows download file goes into.

Nothing seems to work.

We are using TFS 2013 and will plan on upgrading to the latest in the new year.

Has anyone encountered this?

[Arvind Virk]

Apolgise but we are also using Visual Studio 2015 so our options are slightly different to the screenshot Andreas has kindly included - although we believe we have enough functionality to try what he is suggesting.

[Arvind Virk]

Hello,

I managed to solve this error by copying the .dll to the necessary directories, manually repairing the test agent and I also changed the Visual Studio version that was installed.

I am now getting an error "No tests were discovered from the specified test sources". Do i need to do specify anything so it knows I am using SpecFlow Runner and SpecFlow?

Thanks,

Arvind

[Arvind Virk]

Hello All,

I have now downgraded the TestAgent and Visual Studio to 2013 on the TestAgent machine. I have tried to get past the output  "No tests were discovered from the specified test sources"  but I am unable to. The information I am getting when i try and use vstest.console.exe is:

Starting test execution, please wait...
Warning: No test is available in C:\Tests\UI Tests Hyper-V BRANCH Build\UserInte
rfaceTests.dll. Make sure that installed test discoverers & executors, platform
& framework version settings are appropriate and try again.

If i then try and use the SpecRun as the test adapter I get:

Starting test execution, please wait...
Information: SpecFlow+Runner execution started
Information: SpecRun Evaluation Mode: Please purchase at http://www.specflow.org
/plus to remove test execution delay.
Information: SpecRun: running tests in C:\Tests\UI Tests Hyper-V BRANCH Build\Us
erInterfaceTests.dll
Information: Profile:
Information: Evaluation mode: test execution starts in 5 seconds...
Information: (this delay is removed in the registered version)
Information: Discovering target: Default
Passed   @SpecRun: Execution delay for evaluation mode: 5 seconds
Information: Result: tests executed with warnings
Information:   Total: 0
  Succeeded: 0
  Ignored: 0
  Pending: 0
  Skipped: 0
  Failed: 0

I am aware that it says Evaluation Version but we are currently using a timebox in our sprint to determine if we use SpecRun is for us going forward for parallel testing. We have full intention on buying a licence provided we can get it working in our environment.

I can execute the tests on my local machine through visual studio using vstest.console.exe but not when it has been propogated to the test agent machine by TFS Build.

Can someone help me get around this issue?

Thanks,

Arvind 

 

[Arvind Virk]

Hello,

After a lot of struggling to get Spec Run working in our environment I need to ask... Does it work in test agent/controller environments?

For us, we can see it try to run the tests on the build server rather than the test agent machine. We have tried using a test settings file which has a name of the agent and controller, we have tried to use Run Functional Tests. We have also tried to use a Run Settings file that references the TestSettings file inside.

Can we please get further assistance on this? I am happy to provide you with as much information as necessary to solve this issue.

Thanks

Arvind