Thoughts on layout testing

[Cuker]

Just want to know what people using cucumber for a front end project do to test how a website looks. Cucumber scenarios for such projects are mainly functional. So if a link moved from top to bottom the test would still work functionally. Of course one can identify links in selenium using scopes but only for cases where it really matters and even in those cases, the scoping is not too strict. Most of my stakeholders ask QAs to write tests that would tell you if for example a heading was not present or the position of fields in a registration form moved up or down etc. Of course all of these things can be tested in a number of ways but writing tests to such level of detail (for text, images etc) could be an overkill. At the end of the day such requirement only come in from stakeholders because they would like to be alerted if something has changed and not because it is very important for customers in most cases. I know there are tools that would allow screenshots to be compared with baseline images etc. Then there's also an argument that one could test styes by accessing style attributes of the elements. But then again, this can't be done for everything and screenshot comparison doesn't seem a great solution. So, in short, how do people make sure that if Address line 1 moved below Address line 2 in a registration form for example?

 

[Andrew Premdas]

On 20 April 2015 at 19:30, Cuker <rahulsh...@gmail.com> wrote:

Just want to know what people using cucumber for a front end project do to test how a website looks. Cucumber scenarios for such projects are mainly functional. So if a link moved from top to bottom the test would still work functionally. Of course one can identify links in selenium using scopes but only for cases where it really matters and even in those cases, the scoping is not too strict. Most of my stakeholders ask QAs to write tests that would tell you if for example a heading was not present or the position of fields in a registration form moved up or down etc. Of course all of these things can be tested in a number of ways but writing tests to such level of detail (for text, images etc) could be an overkill. At the end of the day such requirement only come in from stakeholders because they would like to be alerted if something has changed and not because it is very important for customers in most cases. I know there are tools that would allow screenshots to be compared with baseline images etc. Then there's also an argument that one could test styes by accessing style attributes of the elements. But then again, this can't be done for everything and screenshot comparison doesn't seem a great solution. So, in short, how do people make sure that if Address line 1 moved below Address line 2 in a registration form for example?

Personally I don't, the cost of writing, and even more so, running such tests is just not worth it. Cucumber is best used for validating/assesing the functionality of a website, i.e. WHAT you can do, not HOW you do it.

Fundamentally if you want to ensure you have a pretty and consistent front end you need to have real people using that front end frequently. Some good people to do this are

1. Users, (of course you have to make it easy for them to give you feedback, and reward them for this)

2. Stakeholders, customer, product owner

3. Designers

4. Front end devs
...

A good way to ensure this happens is to deliver to production as frequently as possible.

 

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Roberto Lo Giacco]

Il giorno lunedì 20 aprile 2015 21:07:12 UTC+2, apremdas ha scritto:

On 20 April 2015 at 19:30, Cuker <rahulsh...@gmail.com> wrote:

Just want to know what people using cucumber for a front end project do to test how a website looks. Cucumber scenarios for such projects are mainly functional. So if a link moved from top to bottom the test would still work functionally. Of course one can identify links in selenium using scopes but only for cases where it really matters and even in those cases, the scoping is not too strict. Most of my stakeholders ask QAs to write tests that would tell you if for example a heading was not present or the position of fields in a registration form moved up or down etc. Of course all of these things can be tested in a number of ways but writing tests to such level of detail (for text, images etc) could be an overkill. At the end of the day such requirement only come in from stakeholders because they would like to be alerted if something has changed and not because it is very important for customers in most cases. I know there are tools that would allow screenshots to be compared with baseline images etc. Then there's also an argument that one could test styes by accessing style attributes of the elements. But then again, this can't be done for everything and screenshot comparison doesn't seem a great solution. So, in short, how do people make sure that if Address line 1 moved below Address line 2 in a registration form for example?

Personally I don't, the cost of writing, and even more so, running such tests is just not worth it. Cucumber is best used for validating/assesing the functionality of a website, i.e. WHAT you can do, not HOW you do it.

Fundamentally if you want to ensure you have a pretty and consistent front end you need to have real people using that front end frequently. Some good people to do this are

1. Users, (of course you have to make it easy for them to give you feedback, and reward them for this)

2. Stakeholders, customer, product owner

3. Designers

4. Front end devs
...

A good way to ensure this happens is to deliver to production as frequently as possible.

 I agree on all the line

[Sam Hatoum]

The ordering of elements is better stipulated in unit/iteration tests for the front end rather than using Cucumber, which is more suited for automating specifications (and these are typically end to end). You can easily use something like Jasmine / Mocha / other front end framework to ensure the order of elements is as expected and you can use selenium/webdriver too if you really want to. So now the question is more of a selenium/webdriver question than a Cucumber one.

If you are accessing front end elements using Selenium and want them to be less brittle to structural changes, then I would advise that you consider changing your locator strategy. The best way I've found to do this is to specify an element as a user would see them. Using IDs is usually the fastest way to locate elements on the DOM, but it's not how the user would call an element. For a "search button" for example, I would use a locator that finds a button that contains the word "search" or the button that has a label that contains "search". Now you're less dependent on the structure and element, but you are dependent on the copy! I personally prefer the latter as the copy should be the same as the domain language. You can also extract the copy and create a mapping file.

You touched on visual regression tools. These are a good QA practice to have as they streamline the process of finding visual bugs. Being able to capture feedback earlier than putting it out to users is better than users finding it. So I would put in these tools into your release pipeline if you are able to. Depending on how quickly you're able to deal with this feedback, you can decide whether you want this step to block your deployment pipeline to production or not.

And I too totally agree with apremdas answer