Testing Umbraco

[Dan Diplo]

Just wondering has any thought been put into testing Umbraco, especially for new releases? I'm conscious that quite a lot of bugs creep into each new release and, worryingly, some of these are regressions of ones that had previously been fixed (is this indicative of a problem with merging changes?). There have also been quite a few undocumented breaking changes that have crept in. On top of this some of the bugs have been pretty major, such as the infamous broken preview that appeared with 4.7 (?).

All this makes me thing there is a problem with testing Umbraco. Given the amount of re-factoring taking place I can only imagine the problem will get worse.

I totally understand that the much of the current code-base isn't amenable to unit testing. I also realise that automated testing (with the likes of Selenium) is very difficult, especially in things like the back-end.

So, I guess my questions are:

• How is Umbraco currently tested before release? (Is it even tested?)
• How can this be improved?

One way I can think of to test releases is to have some kind of base-site that you can test releases on. OK, this would be a mainly manual process, but if you create a fairly comprehensive Umbraco site, upgrade it and then just check basic functions then this would be a start. Just having a test plan that could be worked through against a test site would be a help.

[Aaron Powell]

Testing usually involves pushing out a beta release which people can either create new sites with, drop into their existing environments or “play around” with. The 2^nd option there is the most preferred as it’s a way to have the most valid tests performed but it’s obviously got its own risks to end users.

 

While I’d love to see automated tests we have a major problem in that the HttpContext is really heavily embedded into much of the logic and the back office is very javascript heavy that trying to do stuff with Selenium would be very troublesome (I’m on a commercial project that’s javascript heavy but has selenium tests, they freak out quite a lot).

 

So yeah currently it’s a lot of manual testing

 

Aaron Powell
MVP - Internet Explorer (Development) | FunnelWeb Team Member

http://apowell.me | http://twitter.com/slace | Skype: aaron.l.powell | Github | BitBucket

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/umbraco-dev/-/t_6DNHT5yhsJ.
To post to this group, send email to umbra...@googlegroups.com.
To unsubscribe from this group, send email to umbraco-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/umbraco-dev?hl=en.

[Niels Hartvig]

Any help on a test plan would be ace. The way we've done it in the past (on v4 branches) is installing some of the most popular packages as well as starter kits and see how they behave, then ship it for community testing but it would be much better if a more detailed/decent plan is in place - especially as the few sites we can test on ourselves usually don't have any nasty hacks (read: pushing the core to THE MAX :)).

/n

[rasmu...@gmail.com]

Would it be an idea to test it on different server enviorments including the new setups (.net 4.5 and SQL 2012).

Just a quick idea.

[Dan Diplo]

On Thursday, 5 July 2012 05:52:03 UTC+1, Niels Hartvig wrote:

Any help on a test plan would be ace. 

One thing you can do is create blank install of Umbraco (any 4.7 version) using SQL Lite and configure admin user and basic settings. You can then create a test site, with popular packages etc. and then archive this by creating a copy (the beauty of using an embedded database is that it's so easy to back-up everything, preserving state). You can then upgrade this to the latest version and test. Next time you just create another copy of the "base" site and you are good to go.

[Scott Williams]

One thing you can do is create blank install of Umbraco (any 4.7 version) using SQL Lite and configure admin user and basic settings. You can then create a test site, with popular packages etc. and then archive this by creating a copy (the beauty of using an embedded database is that it's so easy to back-up everything, preserving state). You can then upgrade this to the latest version and test. Next time you just create another copy of the "base" site and you are good to go.

Your probably more likely going to use LocalDB for a test instance;

LocalDB has a few benefits, first it uses a full version of sql express engine and secondly its a local db file like sqlce so its more easily backed up and restored as a clean test instance.

I've not actually tried testing running Umbraco off of a LocalDB instance but I can't, theoretically, see any major issues with running it from it.

Scott. 

[Robert J. Foster]

Any testing done should ideally include all possible database configurations - there's been quite a few instances where SqlCE hasn't been fully supported or tested - although I think most of those have been discovered/ironed out now… there are still some packages that break on SqlCE, but that's another story…

/Rob

--
You received this message because you are subscribed to the Google Groups "Umbraco development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/umbraco-dev/-/-SEV-eZn9SYJ.

[TaoistTotty]

Could you as a set of users from Our to become 'official' beta testers, asking, or arranging for them to test on a number of different platforms etc., and specifically if they have raised a problem and ask them if they now consider it solved.

The testers could be given specific questions to answer - e.g. does X work they way you would expect it to, and does Y solve this problem.

[Lee Kelleher]

Luckily, I'm in a position to have access to many of my client's websites, of which I can take local copies of and test 4.8 upgrades against.  Unfortunately, many of these websites I am under NDA, and wouldn't be able to *share* with the community.  I am speaking with some of my clients (who share open-source values) to see if we can share their content/set-up.

Wondering if we could put out a Call to Action for other devs/clients to share their Umbraco sites?  Obviously we don't want to be inundated with sites, however the more *real world* the usage, the better we can test against.

For my *tests*, I will be focusing on new features introduced in 4.8 - predictably migrated uComponents, (e.g. if using old MNTP can it seamlessly switch over to new core version?), and the then upgraded dependencies (ClientDependency, Examine, XML-RPC and SharpZipLib)

Cheers,

- Lee

[Pete Duncanson]

Think I'd focus on trying to make it testable in the first place. Is it possible to strip out HttpContext in some of the areas? Every area you pull it out of can be easier to test. Breaking existing functionality down into more focused/testable methods is another goodie. The API stays the same, existing methods just call the new testable worker methods. 

public void RenderUrl () {

  var MyDoc = GetDoc( Request.Form["id"] );

  var url = MyDoc.some + clever + stuff + to +work + out + the + url

  // More clever stuff

  Response.Write(  url );

}

Becomes

// This is testable, it returns just a string built from the goodies we pass in, no external stuff is assumed

public string GenerateUrl( Document MyDoc ) {

  var url = some + clever + stuff + to +work + out + the + url

  // More clever stuff

  return url

}

// This original method looks the same from the outside, internally it is doing real simple stuff that needs no/minimal testing

public void RenderUrl() {

  var MyDoc = GetDoc( Request.Form["id"] );

  return GenerateUrl( MyDoc );  

}

We do a lot of work with legacy code and getting it into a testable state is a chore we've done a lot. One of the way's we've found useful to make methods testable without really breaking anything is to use "Method dependency injection" using the same method name but a different parameter siggy. Then we change the original method to call the new method passing in a default object that it would work on. Thats been another go way around it for us.

To pull out some of the HttpContext stuff though you are going to have to create some Interfaces which can get passed around, I know some have a bad taste in their mouth about Interfaces since v5 but I think by showing a little more restraint in what we create and giving it sensible names which we can all understand that they can be a life saver.

Cheers

Pete

[Niels Hartvig]

Pete,

While I agree that more test coverage is good (and heaps of room for improvement in v4), we learned from v5 that even with pretty high coverage it's a false comfort. So I see more test coverage as an added bonus on top of a decent test plan (read: tests run and passed before we even go into alpha/beta/rtm release mode) and should be kept outside of that scope.

So let's keep this thread on how we can test a version of Umbraco that we believe is ready for being shipped.

Thanks!

/n

[Pete Duncanson]

No problem ;)

[Morten Bock Sørensen]

I'll keep this short, as I'm on my phone at the bar, but one thing I thought might be helpful would be to do regression testing as well. So when you fix/report a bug, you would start by creating a page in the umbraco test instance that demonstrates the bug. Then have selenium run through all pages and check that the output is correct.

It's a bit like unit testing in the sense that it will document expected behavior, and the tests could probably be reused in documentation samples afterwards.

/Bock

[Chad Rosenthal]

It's be great if the test plan were published in the documentation on Github. Then in codeplex, you can post a link to the github documentation to remind people of the documentation. The beta page could then also mention specific areas to test in addition to what is in the documentation. (aka, if we refactored the xxx functionality, we'd want to put some emphasis on that).

-C

[Chris M]

I'm no tester by trade so take my suggestions with a grain of salt.  Also, in dotpoint form so I don't rant too much (hopefully):

• Testing is hard to do well.  It's almost impossible to do well when you don't know what you're trying to test (you're limited to random, system-wide smoketesting - which has its place but should be a final line of defence to pick up otherwise unanticipated side effects of changes), what you need to test on (now you're searching for random combinations of things not working against random systems and configurations), and what integration points may be broken (now you've got the before compounded with random packages).
• Specification is the key to reducing the set of things that you need to test to a number that's manageable.  Unfortunately, this is a little at ends with Umbraco's "a hundred ways to solve a problem" ethos, so in this case, I'd reduce the formal testing to a preselected variety of tests on a preselected variety of configurations.  These tests should be based on a) what the "most important" functionality is to Umbraco, b) what's used by the most people and c) what has changed in the release.  If everything on this list gets tested and there's still allocated time left over, you can either expand on the smoketests, or bring additional formal tests into the testing.  
• Split things up into essential and extended (and possibly other) support lists (ie systems, configurations, packages), and make sure that the essential stuff is solid for a release.  Provide this list (in some edited form) as "Umbraco best practices" and/or "How we do things" so that the majority of corporate customers etc will stick to eg Windows Server 2008 R2 with SQL Server 2008 R2 using IIS7.5 on a .NET4 app pool with content editors using the latest version of Firefox on Windows 7 to administer the site if they want the absolute most working configuration.  Leading devs down a golden path also helps (unless it's a serious revenue stream, I'd even consider making umbraco.tv more accessible).  These will reduce the number of people who are (unintentionally) doing funky things to achieve their goals (Microsoft sneakily employs this strategy with all of the code samples on MSDN).
• You should also specify environments and figure out a way to make each relevent combination of these happen.  I'm talking about servers on multiple versions of windows, on multiple versions of IIS against multiple databases.  Content editing should occur from multiple versions of windows (and mac, and possibly linux) and browsers with these based on userbase.
• Automation is your friend, and I'm not even talking about UI tests (selenium or otherwise), I'm talking about configuration and installation.  I've seen powershell-distaste on this list (in favour of .bat - eww :) but http://learn.iis.net/page.aspx/433/powershell-snap-in-creating-web-sites-web-applications-virtual-directories-and-application-pools/ is ridiculous-powerful.  Umbraco should IMO absolutely be installable without going through the wizard (otherwise you waste so much time setting up each environment).  This method of installation should run the same code as the actual installer (which will require some refactoring).  You can see that this shell-based configuration/management is the direction that Msft is travelling (and IMO it's a good one).
• Automation is really your friend.  What Pete said is IMO relevant to even system testing.  UI testing is brittle enough that it can easily become a huge hassle, but if the UI functionality is decoupled from the underlying operations, you can start to expose those operations.  While code coverage % on unit tests isn't a great indication of whether your software has bugs, imagine this pipe-dream:
• Have some fixed test environments
• Scripts exist (powershell or something equivalent) that install and configure new Umbraco instances of various configurations (ie mess around with umbracoSettings.config etc) on the machines
• Scripts exist that perform a long sequence of operations reflecting the creation and management of a site (including adding packages)
• (optionally) Some mechanism can be used to run code tests on these "real world"ish sites.
• The systems are configured to notify someone whenever an exception is raised (outside of the code tests, some of which are expected to throw exceptions).
• You potentially have some scripts that recreate reported issues that have been fixed like "if I move something to the recycle bin then restore it, the unpublish screen gives me an exception"
• You have a spider that confirms that the output HTML on each page matches a static version of the site that you keep somewhere (and update when you change or add content to the script)
• The above may be long-term or not-doable, but there are some things that can be done in there (automating creation of [large] websites via existing webservices or courier or something; changing installer [and possible content editor interface] to be decoupled from the underlying logic; capturing reported bugs in tests [either automated or manual] to ensure that regressions don't recur).  In the "ideal" case, you'd know that the system is solid without ever needing to touch your browser.  Having a list of tests (each of which is a solid use-case) that get performed on each browser to ensure that golden-path functionality works should deal with the browser side of things.
• Improvements to the bug process also help with test - having a fixed set of reproduction steps (multiple if the bug can be triggered multiple ways), as well as keeping track of what's been tested by (ab)use of comments in the issue itself (eg notes of "Tested OK win2k8r2 / sql2k8r2 / IIS7.5 / Umbraco 4.8.0.0").  This way if a regression occurs you can start to look at whether the reproduction steps aren't sufficient, whether there are dependencies on other code that caused the problem to come up again (identifying problem areas in the codebase), or whether the issue just didn't receive enough testing for whatever reason.  Alternatively, put all of this into its own test plan document and publish it somewhere.
• Give more specific release/change notes (especially on betas!) and try to guide people towards looking at and reporting if bugs actually are fixed in their environments.

Wow, much longer than I'd anticipated.  

TL;DR: Specify preferred configuration(s); Limit user/dev options (or guide users/devs); Define test machines/environments; Automate deployment and install; Automate testing or parts thereof; Update bugs with repro information / test steps, update bugs when tested; document.  Hopefully there's at least one or two useful things there...

--Chris

[Pete Duncanson]

I've been thinking about this comment the last few days:

" we learned from v5 that even with pretty high coverage it's a false comfort. "

I'm not sure its fair to write off more unit/behaviour tests simply because you had masses of them in v5. It depends what you test. I'm not one for testing everything. I much prefer to test what is "likely to break" and even prioritising that. Everyone is busy. Having 100% coverage is nothing. Good tests are what you want, they are intended to be a back stop to prevent you breaking anything when you make a change not just to get that "green light" feel good feeling or 100% test coverage.

I'd take automation over manual testing any day. Ones repeatable the other is error prone and erratic at best.

I think there would be a lot of mileage in having a few of us set loose on the core and allow to try to bash it into a most testable shape, little by little, I'm not expecting a complete test suite. Overtime we'd have a good chunk of robust tests which exercise "the dangerous bits". No point starting this though until we know what is likely to change in the next 6 months so we know where to focus but I'd chalk it up as one of the jobs to do once we have the roadmap.

Additionally some smoke tests can go a long way to stopping "silly" errors going out the door.

As for breaking packages, as long as the API's/extension points have not changed then packages should be ok. If a package has hacked the core or similar to get itself working then it deserves to break. That said it might highlight an area that needs work to potentially improve the extension points/api so the package can work without the need for hacking (obviously this does not mean that any old package can ask for any crazy mod just to get it working nice). So blanket package testing should not be required as long as the APIs etc. stay the same, test can help prove that those API's are working as expected.

Pete

On Thursday, 5 July 2012 10:38:37 UTC+1, Niels Hartvig wrote: