using SetupFixture attribute

[Chris Busuttil]

Hi all,

I have a class called ConfigSettings with SetupFixture attribute, this class should retrieve the settings to be used throughout the tests. I have 5 different TestFixtures which all inherit the class ConfigSettings. I'm noticing that the SetupFixture [setup] method is run for every test, so i used a flag 'HasRun' to avoid this. However when TestFixture1 is ready and the runner goes to TestFixture2 the HasRun will be 0 again since a new instance will be created. How can I have the SetupFixture attribute class run only once at the beginning of the TestSuite? I solution would be to make the HasRun property as static, however if i open a new instance of NUnit, the HasRun property will have the same value of the first instance. any ideas please?

Thanks.

[Simone Busoli]

Hi Chris,

You sound a bit confused. SetupFixture has its methods run once per overall run, but if you subclass it with test fixtures you'll experience an undocumented behavior.
Do not subclass it, you can rather expose static members from it which you could populate inside its setup methods. You will probably then realize that its static constructor will be just enough and you won't need a SetupFixture anymore. For sure you'll never need a HasRun property whatsoever. Just as well as you'll then realize that loading your settings in a normal TestFixtureSetup method 5 times per run won't affect the execution time to a great extent and will simplify your whole code.

Simone

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at http://groups.google.com/group/nunit-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Chris Busuttil]

Hi Simone,

Many thanks for your help, I have changed the class as you suggested. Exposing it's members as static, and removed inheritance of the SetupFixture class from the TestFixtures. The issue i'm encountering now is that, if nunit is running the tests....and a new instance of nunit is opened and i start running tests also the tests of the first instance will get the settings of the second instance. the settings are retrieved using a userId. So let's say with the first nunit the userid was 7009 and  in the second instance the userid is 8700, if both are running at the same time the two instances will start using the same userid since the members are static.

[Simone Busoli]

Then using a SetupFixture is not what you want, just move the initialization code where you read the user-specific settings to a TestFixtureSetup in your fixtures or a base class, as long as the same user is used by all tests in that fixture.

[Chris Busuttil]

That's the issue actually that there might be more than 1 user running the TestSuite. I previously was using the TestFixtureSetup attribute where i'm retrieving the settings and the behavior was something similar to this:-


User A Starts TestSuite ( having TestFixture1 and TestFixture2)

after a few seconds User B starts the same TestSuite.

While both are running the same TestFixture everything is fine, however then when User A goes to TestFixture2 it would start getting the settings of User B. Basically i'm retrieving the userId from an SQl Database, depending on the user who last started the test. probably i need to improve this and maybe add some sort of token, however i cannot understand how. I'm still trying to actually figure out if it's really possible to use NUnit using multiple users. if you can point me in the right direction i would be really greatful!

Thanks.

[Simone Busoli]

C

hris, I don't get the multiple user thing. What do you mean that multiple users are running the same tests, possibly at the same time? I'm starting to think you're actually trying to solve a problem of shared state which is not in NUnit but rather somewhere else (like a SQL database). NUnit cannot do much about that if your tests access and eventually modify data in the database, you will have to protect the data somehow by structuring your tests in a way which allows that, but NUnit-specific details will not solve that problem out of the box.

Simone

[Chris Busuttil]

The project is like this:-

We have Web Application where users can login, choose a TestSuite (lets call it SanityCheck), input different settings and upon clicking run, using the Nunit RemoteTestRunner we are executing the TestSuite dll on a remote machine. Another user can login, choose the same TestSuite (SanityCheck) input his settings (which will be used in the tests) and upon clicking run the same dll will be called.

When the dll containing the tests is called, i'm retrieving the settings from the Sql Database which were previously entered by the user. Management would like to have this centrialised web test runner where everyone can login and run tests etc. When there is 1 user running the tests everything works great. I hope i was able to give you a clear picture of what i'm trying to achieve, and maybe give me a hint if i might be able to achieve such functionality using nunit.

Thanks

[Chris Busuttil]

I forgot to mention that the tests just read the data from the Sql Database they do not modify it. and since we have staging/live environments, the database is designed so that there are settings for staging and settings for live environments. If User A is running TestSuiteA on staging, userB will not be alowed to run the same tests on the same environment, he will be notified and wont be allowed to change the settings. since that might cause the tests of user a to crash.

[Simone Busoli]

Hi Chris,

do you have any sample code to show how exactly you are running tests remotely?

[Simone Busoli]

..and how you exactly provide the user input to your tests, too.

[Chris Busuttil]

Hi Simone,

I will do my best in this post do show you some sample code. So basically in the Web Application where users can login, run tests are have test progress show in grids etc, I have a Users class. This users class has the following method:-

        private void StartTests(string fileName)

        {

            var testPackage = new TestPackage(fileName);

            Runner = new RemoteTestRunner();

            Runner.Load(testPackage);

            var nunitEventListener = new NUnitEventListener();

            nunitEventListener.CompletedRun += new EventHandler(nunitEventListener_CompletedRun);

            Runner.BeginRun(nunitEventListener, TestFilter.Empty, true, LoggingThreshold.All);  

        }

I have a WebService which i use to trigger the tests. So on clicking the Run button the below would be triggered:-

        [WebMethod(EnableSession = true)]

        public void startTests()

        {

SessionManager session = new SessionManager();

Users user = new Users();

user.StartTests(@"C:\\pathToTestsDll");

                user.UpdateSession(session.userID, 1, session.clientID, session.isLive, 1, DateTime.Now);

        }

The UpdateSession method above, updates a table in the SqlDatabase where i have a table called CurrentSessions, which basically logs the current logged. The first table that the Tests read from is infact this CurrentSessions table. I get settings by search for the userId which has the latest TestStarted timestamp. Below is a sample code:-

public class ConfigSettings

{

    private static string strConnectionString = @"connectionStringHere";

    public string userName { get; set; }

    public string userId { get; set; }

    public int clientId { get; set; }

    public int isLive { get; set; }

    etc... 

    public void GetSettings()

    {

            using (var db = new autodb(strConnectionString))

            {

                var currentSession = (from session in db.CurrentSessions

                                      where session.TestSuiteID == 1

                                      orderby session.TestStarted descending

                                      select session).FirstOrDefault();

                userId = currentSession.UserId.ToString();

                clientId = currentSession.ClientID;

                isLive = currentSession.IsLive;

                etc...

            }

        

    }

}

The above ConfigSettings is just part of the code since i had to remove some of it, basically from the clientid and islive i then have other linq queries which get other settings which are then used in tests. That's it basically. In my tests then i log info to the database using log4net which is then in turn shown in the web interface. Also In the webapplication i have a class which implements the NUnit.Core.EventListener Interface. So on TestFinished i log the test results to database, and then using jquery ajax i query the database to show the results to the user. 

Hope this gives your a clearer picture of what i'm trying to accomplish. I could take the easy way out, and make it in a way so that a TestSuite can only be run a 1 user only at the same time and that would fix the issue. 

Thanks 

Chris.

[Simone Busoli]

Hi Chris,

from what I understand your problem lies in the data which is stored into the CurrentSessions database table. I expect there is an intrinsic concurrency issue in there because two users may try to start running tests in a short timeframe, which means inserting/updating the table twice, in which case two test runs triggered by different users will end up loading the same settings. If my understanding is correct, what you're basically after is run the tests with different inputs (i.e. user ids) which currently NUnit does not support, not easily at least. In other words, you cannot easily start a test run but passing some input that tests will be able to read unless you come up with an artificial way of doing it.

I can suggest one, which involves some changes to your code. The basic idea is that as long as you use shared state there are little chances that you can get rid of a concurrency issue unless you start locking. The other solution is therefore to use non-shared state, which is where you would store your inputs for your tests to be able to read them later. 

As far as I can see from your code what you need is access to a bunch of properties describing the "user session", which you are currently storing in the DB. What if you store this data (or an identifier to this data) in a file which is then accessed exclusively by the tests ran by that specific user in that specific session? The problem is still how to tell the test to load that specific file.

Let's assume that the user sessions table has a unique id called SessionId. If this ID is accessible to your code in the Users.StartTests method you could generate a standard .NET configuration file looking approximately like this and save it with a unique name in the temp folder:

<configuration>

  <appSettings>

    <add key="SessionId" value="12345"/>

  </appSettings>

</configuration>

A prerequisite for this solution is that you execute this statement once in your AppDomain (for an ASP.NET application the Global Application_Start event handler should work just fine) because you'll switch from using a single app domain to run your tests to multiple ones, so that you can leverage this trick of using different configuration files for each of them:

ServiceManager.Services.AddService(new DomainManager());

Then, if you change some code in Users.StartTests to look something like this you should be able to make the session id accessible from your tests:

private static void StartTests(string fileName, int sessionId)

{

var configFilePath = CreateConfigurationFile(sessionId);

var testPackage = new TestPackage(fileName) { ConfigurationFile = configFilePath };

var runner = new TestDomain();

runner.Load(testPackage);

var nunitEventListener = new NUnitEventListener();

nunitEventListener.CompletedRun += new EventHandler(nunitEventListener_CompletedRun);

runner.Run(new NullListener(), TestFilter.Empty, true, LoggingThreshold.All);

File.Delete(configFilePath);

}

private static string CreateConfigurationFile(int sessionId)

{

var configFile = string.Format(@"<?xml version=""1.0"" encoding=""utf-8"" ?>

<configuration>

 <appSettings>

<add key=""SessionId"" value=""{0}""/>

 </appSettings>

</configuration>", sessionId);

var tempFileName = Path.GetTempFileName();

File.WriteAllText(tempFileName, configFile);

return tempFileName;

}

Now, changing the code in ConfigSettings.GetSettings() to load that specific session id from the DB rather than the latest you should have solved your concurrency problem:

 var currentSession = (from session in db.CurrentSessions

                       where session.TestSuiteID == 1

                       where session.Id = int.Parse(ConfigurationManager.AppSettings["SessionId"])

                       select session).FirstOrDefault();

You get the idea hopefully.

Simone

[Chris Busuttil]

Hi Simone,

First of all thanks a lot for the detailed reply! I really appreciate! This morning I implemented your suggestion, so in global.asax file under Application_Start event, i included:-

ServiceManager.Services.AddService(new DomainManager());

and i changed the Users.StartsTests method to what you suggested also. I also modified the ConfigSettings in the tests dll. and the tests where triggered successfully. However in Users.StartsTests i'm noticing that we are passing a new NullListenter to the Run method instead of using the instance of the NunitEventListener class. Infact tests where running however no data was being logged into the database since the NunitEventListener class is the class which writes back to the database.

Another question, the ConfigSettings.GetSettings() method, should be called in a [SetupFixture]'s [Setup] method or in a [TestFixtureSetup]? And should the properties be stored in static variables?

Thanks.

[Simone Busoli]

Hi, the NullListener should indeed be replaced with your custom listener, I forgot to update my code accordingly.
The GetSettings method can be called any time as the configuration from which you read the session id is always available during the test run, but when to call it I believe depends mainly on the lifecycle of the data you store in the database. If it does not matter to you because the data does not change during the whole test run then I would say do it as rarely as possible, possibly once during the whole run, as you are reading it from the DB and I assume doing it multiple times would slow down your tests.

[Chris Busuttil]

If i pass an instance of the NunitEventListener class to the Run method, on TestFinished i get an exception "The type initializer for WebApp.ProgressReporter threw an exception, ProgressReporter is the class which has the methods that write to the database. Also after i implmented what you had suggested i had to put the [Serializable()] attribute in both Users and NunitEventListener classes.

[Simone Busoli]

You'll have to figure out what the exception is, NUnit might require that some classes are serializable as tests will run in a different AppDomain. Not having your code at hand makes it difficult to tell exactly what is happening, but if after spending some time on it you cannot figure it out post here some more details and we can try to help you.

...

[Chris Busuttil]

Hi Simone,

Basically i had to remove any static members used, it seems that when you a different domain static variables will start giving trouble. So now tests are being executed, and the Eventlistener class is writing progress to database. Now what i still need to fix is tests should be logging to database steps in using log4net which stopped working will try to figure it out and test accordingly using 2 different users at the same time. Will let you know how it goes later on.

[Chris Busuttil]

Hi Simone,

There is an issue that i need to fix before i can verify if two concurrent tests are running smoothly and that is the logging from the tests. Since we introduced the new appdomain per test the tests are not logging to the database using log4net. Is there anything that i need to do when running tests in a different domain?

On Monday, 3 June 2013 13:55:40 UTC+2, Chris Busuttil wrote:

On Monday, 3 June 2013 13:55:40 UTC+2, Chris Busuttil wrote:

[Simone Busoli]

Difficult to tell without looking at the code. First thing I can think of, make sure you initialize log4net calling BasicConfigurator.Configure or XmlConfigurator.Configure in the appdomain running your tests. This is indeed something you could do in a SetupFixture's Setup method. 

Pay attention that if you load your log4net configuration from a configuration file you'll as well need to make sure that the configuration file is loaded by the new appdomain, which means that besides including the appSettings I showed before you'll need to include the log4net configuration too when dumping its contents to the file system and instructing your test package to use that as a configuration file.

Simone

[Chris Busuttil]

In the Web Application under Application_Start i have log4net.Config.XmlConfigurator.Configure();

        protected void Application_Start(object sender, EventArgs e)
        {
            ServiceManager.Services.AddService(new DomainManager());
            log4net.Config.XmlConfigurator.Configure();
        }

The Config file which log4net uses is the web.config file of the web application. So in web.config file there is all the configuration for log4net. In the tests then i have something like this:-

public readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

and then in test methods i access log.Info(), log.Warn() etc.

[Simone Busoli]

Chris,

your tests will be running in a different AppDomain, with a different configuration file. If your tests need log4net then that configuration file will need to contain the log4net configuration and you will need to call XmlConfigurator.Configure() once in that AppDomain before you use log4net.

This means that:

1) you will need to enrich the configuration file you are creating dynamically in the latest version of your code with the log4net configuration you currently have in the web application's configuration file

2) you must XmlConfigurator.Configure() in your tests assembly before any test is run, therefore calling it in a SetupFixture's setup method would be a good approach.

On an unrelated note, don't you have some more senior colleagues who can help you out with this? This is not NUnit-related, and trivial if you're somewhat familiar with .NET.

[Chris Busuttil]

Hi Simone,

 

Thanks a lot you have been of great help! From a first glance it seems that the process is working as it should! Will confirm again tomorrow. What I would like to ask you is what does ServiceManager.Services.AddService(new DomainManager()); do? It just allows nunit to switch from a single domain to multiple ones? The reason I'm asking since tomorrow I will debug an issue were after we implemented this ajax requests first response is taking lots of time. And i'm suspecting it might be some sort of issue with appdomain loading delay issue.

[Simone Busoli]

Hi Chris,

glad to help. That method just allows you to use the TestDomain runner, nothing more.
You should expect some performance overhead associated with running tests in different AppDomains, but not a much longer running time. I can't help you there, you'll have to debug your code to see where the slowness comes from.

Simone

[Charlie Poole]

NUnit uses a number of services internally, under control of a Service Manager. These services are not exposed to users of NUnit, normally.

However, when trying to use NUnit programmatically by creating your own runners, it's necessary to ensure that all services needed by the runner are initialized. This is usually done in some sort of main routine, called only once.

If you are able to run tests without initializing the DomainManager, that means you are not using any part of NUnit that requires the service.

To be quite frank, when using internal parts of NUnit to run your tests, the amount of help we can give you is quite limited. Since NUnit's internals are, by definition, intended only to be called by NUnit, using them in other ways requires you to learn a great deal about how NUnit works. As Simone suggested, the source code is the best way to gain such knowledge. You can see how NUnit initializes services normally by looking at the code for nunit-console.

Charlie

[Chris Busuttil]

Great thanks a lot, if let's say an exception happens in one TestDomain, and there is also another TestDomain which is currently executing tests, will one effect the other? It shouldn't be the case right?

[Simone Busoli]

If we're talking about unhandled exceptions bubbling up from NUnit they will be processed as unhandled exceptions by ASP.NET (which is the host of your application), therefore they should not interfere with other test runs, but I would double check that this is really the case.

Simone