Code Coverage of Selenium
2,768 views
Skip to first unread message
Josef M.
Aug 27, 2012, 10:35:56 AM8/27/12
to seleniu...@googlegroups.com
Hey, i want to get the Code Coverage from the Selenium project..
'til it's build with crazy fun, i dunno how to get the information i'm looking for
i would prefer to use sonar with maven.. did anyone has experience with that?
'til it's build with crazy fun, i dunno how to get the information i'm looking for
i would prefer to use sonar with maven.. did anyone has experience with that?
Peter Gale
Aug 27, 2012, 10:48:18 AM8/27/12
to Selenium Users
If you could completely isolate an environment so nothing was running other than you Selenium tests, then you could run your full Selenium test suite and measure the server-side code coverage with the likes of Sonar.
But if you have 100% code coverage with unit testing, measuring the code coverage with Selenium in this way never quite seemed a worthwhile activity to us.
Date: Mon, 27 Aug 2012 07:35:56 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: [selenium-users] Code Coverage of Selenium
But if you have 100% code coverage with unit testing, measuring the code coverage with Selenium in this way never quite seemed a worthwhile activity to us.
Date: Mon, 27 Aug 2012 07:35:56 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: [selenium-users] Code Coverage of Selenium
Hey, i want to get the Code Coverage from the Selenium project..
'til it's build with crazy fun, i dunno how to get the information i'm looking for
i would prefer to use sonar with maven.. did anyone has experience with that?
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/BO27Jsl7rRAJ.
For more options, visit https://groups.google.com/groups/opt_out.
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/BO27Jsl7rRAJ.
For more options, visit https://groups.google.com/groups/opt_out.
Josef M.
Aug 27, 2012, 11:10:02 AM8/27/12
to seleniu...@googlegroups.com
Is there another possibility to measure the code coverage? maybe with a plugin for crazy fun or an external program?
Peter Gale
Aug 27, 2012, 11:17:22 AM8/27/12
to Selenium Users
Perhaps I can ask what you mean by "code coverage", Josef.
Which "code" are you referring to, exactly?
Date: Mon, 27 Aug 2012 08:10:02 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: Re: [selenium-users] Code Coverage of Selenium
Which "code" are you referring to, exactly?
Date: Mon, 27 Aug 2012 08:10:02 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: Re: [selenium-users] Code Coverage of Selenium
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/6RvW0E_NrZsJ.
Josef M.
Aug 27, 2012, 11:30:45 AM8/27/12
to seleniu...@googlegroups.com
Hi Peter,
I write a thesis about the Selenium Project. I use different metrics to measure several categories.
In one category i want to measure the quality of the product. For that i want to know how good is the code coverage of Selenium.
I write a thesis about the Selenium Project. I use different metrics to measure several categories.
In one category i want to measure the quality of the product. For that i want to know how good is the code coverage of Selenium.
Peter Gale
Aug 27, 2012, 11:34:14 AM8/27/12
to Selenium Users
Very good ... but you haven't answered my question: what do you mean by "code coverage"?
By any definition of the term "code coverage" that I can imagine, I can't see that that such a measure would be a metric about Selenium, but rather a metric of the tests that were written using Selenium.
Date: Mon, 27 Aug 2012 08:30:45 -0700
By any definition of the term "code coverage" that I can imagine, I can't see that that such a measure would be a metric about Selenium, but rather a metric of the tests that were written using Selenium.
Date: Mon, 27 Aug 2012 08:30:45 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: Re: [selenium-users] Code Coverage of Selenium
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/e5eE5G4akfEJ.
Josef M.
Aug 27, 2012, 11:48:45 AM8/27/12
to seleniu...@googlegroups.com
Oh, it's about the tests written to test the functionality of selenium, usally unit-tests.
here a short definition: "Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested."
here a short definition: "Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested."
Peter Gale
Aug 27, 2012, 11:50:56 AM8/27/12
to Selenium Users
And which program are you referring to?
Date: Mon, 27 Aug 2012 08:48:45 -0700
Date: Mon, 27 Aug 2012 08:48:45 -0700
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/mFQq8eGwFSQJ.
Thomas Sundberg
Aug 27, 2012, 12:04:02 PM8/27/12
to seleniu...@googlegroups.com
On 27 August 2012 17:48, Josef M. <sicl...@googlemail.com> wrote:
> Oh, it's about the tests written to test the functionality of selenium,
> usally unit-tests.
> here a short definition: "Code coverage is a measure used in software
> testing. It describes the degree to which the source code of a program has
> been tested."
You could annotate your classes using Cobertura.[1] Then package your
> Oh, it's about the tests written to test the functionality of selenium,
> usally unit-tests.
> here a short definition: "Code coverage is a measure used in software
> testing. It describes the degree to which the source code of a program has
> been tested."
annotated classes into a deployable archive, deploy it and execute
your test suite. This should create a file that records all
interactions with the code. If you then combine it with your source
code, you could see the lines of code that has been executed.
I have done something similar when I measured the test coverage for a
multi module Maven project.[2]
This approach would not involve Sonar, it would just involve Cobertura
(and Ant and Maven). But if you are writing a thesis, I assume that
fiddling a bit with Cobertura could be some additional stuff for your
thesis.
HTH
Thomas
[1] http://cobertura.sourceforge.net/
[2] http://thomassundberg.wordpress.com/2012/02/18/test-coverage-in-a-multi-module-maven-project/
--
Thomas Sundberg
M. Sc. in Computer Science
Mobile: +46 70 767 33 15
Blog: http://thomassundberg.wordpress.com/
Twitter: @thomassundberg
Better software through faster feedback
Josef M.
Aug 27, 2012, 12:05:34 PM8/27/12
to seleniu...@googlegroups.com
the selenium webdriver
Peter Gale
Aug 27, 2012, 12:07:06 PM8/27/12
to Selenium Users
Josef, why onearth would you want to measure the code coverage of the Selnium Webdriver?????
the selenium webdriver
Am Montag, 27. August 2012 17:50:56 UTC+2 schrieb PeterJef...@Hotmail.co.uk:
Am Montag, 27. August 2012 17:50:56 UTC+2 schrieb PeterJef...@Hotmail.co.uk:
And which program are you referring to?
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/o2datBP-kDcJ.
Josef M.
Aug 27, 2012, 12:10:04 PM8/27/12
to seleniu...@googlegroups.com
cause i want to measure the quality of selenium webdriver. i dont want to test "my" code coverage of a test-project. my thesis is about selenium, so i need to know how good is the code coverage of the webdriver.
Josef M.
Aug 27, 2012, 12:14:22 PM8/27/12
to seleniu...@googlegroups.com, t...@kth.se
Hi Thomas, thx for your response. But i dont want to measure my own coverage of a test-project :/ i want the code coverage of the developers of selenium webdriver :/
Jim Evans
Aug 27, 2012, 12:15:41 PM8/27/12
to seleniu...@googlegroups.com
You're going to find "code coverage of the code in the WebDriver project" challenging, I think. You'll need to strictly define what you mean by "WebDriver code" that you want to get code coverage of. Do you mean language bindings? If so, which ones? Java, .NET, Ruby, and Python are all supplied directly from the project, so you'll have to decide how you want to get code coverage metrics for each of those projects. Do you mean code that makes up the browser drivers themselves? The Firefox and Safari drivers are written primarily in JavaScript, the IE driver in C++. Would you be instrumenting those and looking for code coverage metrics for those? The code for the Chrome and Opera drivers aren't part of the Selenium project. How do you propose to generate code coverage metrics for those? These are just some of the questions you'll need to answer before you can even attempt to "measure the quality of selenium webdriver".
--Jim
--Jim
Peter Gale
Aug 27, 2012, 12:15:41 PM8/27/12
to Selenium Users
Sorry, you are not making any sense at all now, Joself.
I'm sure the Selenium developers have fully tested the Selenium Webdriver with unit tests and funcitonal tests. Unless you want to get involved with the Selenium projec, there should be no reason for you to try to run tests against Selenium Webdriver itself.
Indeed, as consumers of WebDriver, we may not even use anywhere near 100% of its functionality, so we may never cover 100% of its code .. but why should that matter?
Date: Mon, 27 Aug 2012 09:10:04 -0700
I'm sure the Selenium developers have fully tested the Selenium Webdriver with unit tests and funcitonal tests. Unless you want to get involved with the Selenium projec, there should be no reason for you to try to run tests against Selenium Webdriver itself.
Indeed, as consumers of WebDriver, we may not even use anywhere near 100% of its functionality, so we may never cover 100% of its code .. but why should that matter?
Date: Mon, 27 Aug 2012 09:10:04 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: Re: [selenium-users] Code Coverage of Selenium
cause i want to measure the quality of selenium webdriver. i dont want to test "my" code coverage of a test-project. my thesis is about selenium, so i need to know how good is the code coverage of the webdriver.
Am Montag, 27. August 2012 18:07:06 UTC+2 schrieb PeterJef...@Hotmail.co.uk:
Josef, why onearth would you want to measure the code coverage of the Selnium Webdriver?????
Date: Mon, 27 Aug 2012 09:05:34 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: Re: [selenium-users] Code Coverage of Selenium
the selenium webdriver
Am Montag, 27. August 2012 17:50:56 UTC+2 schrieb PeterJef...@Hotmail.co.uk:And which program are you referring to?
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/o2datBP-kDcJ.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/D7ZGJT9r_wcJ.
Thomas Sundberg
Aug 27, 2012, 12:16:20 PM8/27/12
to seleniu...@googlegroups.com
On 27 August 2012 18:14, Josef M. <sicl...@googlemail.com> wrote:
> Hi Thomas, thx for your response. But i dont want to measure my own coverage
> of a test-project :/ i want the code coverage of the developers of selenium
> webdriver :/
Ok, then do something similar for the source code of WebDriver. It is
> Hi Thomas, thx for your response. But i dont want to measure my own coverage
> of a test-project :/ i want the code coverage of the developers of selenium
> webdriver :/
an open source project. All you have to do is build it and execute all
of its tests after you have annotated the class files.
/Thomas
>
> Am Montag, 27. August 2012 18:04:02 UTC+2 schrieb Thomas Sundberg:
>>
>> On 27 August 2012 17:48, Josef M. <sicl...@googlemail.com> wrote:
>> > Oh, it's about the tests written to test the functionality of selenium,
>> > usally unit-tests.
>> > here a short definition: "Code coverage is a measure used in software
>> > testing. It describes the degree to which the source code of a program
>> > has
>> > been tested."
>>
>> You could annotate your classes using Cobertura.[1] Then package your
>> annotated classes into a deployable archive, deploy it and execute
>> your test suite. This should create a file that records all
>> interactions with the code. If you then combine it with your source
>> code, you could see the lines of code that has been executed.
>>
>> I have done something similar when I measured the test coverage for a
>> multi module Maven project.[2]
>>
>> This approach would not involve Sonar, it would just involve Cobertura
>> (and Ant and Maven). But if you are writing a thesis, I assume that
>> fiddling a bit with Cobertura could be some additional stuff for your
>> thesis.
>>
>> HTH
>> Thomas
>>
>> [1] http://cobertura.sourceforge.net/
>> [2]
>> http://thomassundberg.wordpress.com/2012/02/18/test-coverage-in-a-multi-module-maven-project/
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Selenium Users" group.
> To post to this group, send email to seleniu...@googlegroups.com.
> To unsubscribe from this group, send email to
> selenium-user...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/selenium-users/-/mAfSJ3xMd5IJ.
> You received this message because you are subscribed to the Google Groups
> "Selenium Users" group.
> To post to this group, send email to seleniu...@googlegroups.com.
> To unsubscribe from this group, send email to
> selenium-user...@googlegroups.com.
> To view this discussion on the web visit
Peter Gale
Aug 27, 2012, 12:21:16 PM8/27/12
to Selenium Users
I should add that "code coverage" as a percentage is not a measure of the quality of the code being tested (i.e. the quality of the product), it is just a measure of the extent to which we have tested it, and from one point of view only. So 100% code coverage from our unit tests does not been that we have 100% coverage of the functionality of the end user application.
From: peterjef...@hotmail.co.uk
To: seleniu...@googlegroups.com
Subject: RE: [selenium-users] Code Coverage of Selenium
Date: Mon, 27 Aug 2012 17:15:41 +0100
From: peterjef...@hotmail.co.uk
To: seleniu...@googlegroups.com
Subject: RE: [selenium-users] Code Coverage of Selenium
Date: Mon, 27 Aug 2012 17:15:41 +0100
Josef M.
Aug 27, 2012, 12:30:50 PM8/27/12
to seleniu...@googlegroups.com
@Peter
i dont want to run tests against selenium webdriver.
i want to know huch much percentage of code coverage the developers of selenium webdriver have done.
from a user's point of view is that not important, for my thesis it is!
@Thomas
Yeah, i know... "simply build" and "just" annotate the classes?
the selenium-server module has about 62.361 and 658 classes. That's why i asked if there is a possibility to run a tool like sonar to get the code coverage ;)
@Jim
i see. it's really complicated to define what exactly is the selenium webdriver. i analyze just the language java and want to get the code coverage of the java-bindings.
i think i'm looking for the maven selenium-java package code coverage.
i dont want to run tests against selenium webdriver.
i want to know huch much percentage of code coverage the developers of selenium webdriver have done.
from a user's point of view is that not important, for my thesis it is!
@Thomas
Yeah, i know... "simply build" and "just" annotate the classes?
the selenium-server module has about 62.361 and 658 classes. That's why i asked if there is a possibility to run a tool like sonar to get the code coverage ;)
@Jim
i see. it's really complicated to define what exactly is the selenium webdriver. i analyze just the language java and want to get the code coverage of the java-bindings.
i think i'm looking for the maven selenium-java package code coverage.
Josef M.
Aug 27, 2012, 12:33:37 PM8/27/12
to seleniu...@googlegroups.com
i would appreciate any other good metric to measure the quality of the written code of webdriver
Peter Gale
Aug 27, 2012, 12:46:45 PM8/27/12
to Selenium Users
You could downoad the sourcecode, as Thomas has suggested, and run tools such as "Find Bugs" on it, but I'd be pretty sure the developers do that regularly as part to their CI process.
Still, if you were to find any "coding" quality issues, I hope you would fed them back to the developers so that they coud address the probems.
As an end user, the quality of the coding isn't really of big concern to me, that's more a internal maintenance issue for the developers. I'd be more interested in measures of how effectively it delivered the functionality is prmised to.
Date: Mon, 27 Aug 2012 09:33:37 -0700
Still, if you were to find any "coding" quality issues, I hope you would fed them back to the developers so that they coud address the probems.
As an end user, the quality of the coding isn't really of big concern to me, that's more a internal maintenance issue for the developers. I'd be more interested in measures of how effectively it delivered the functionality is prmised to.
Date: Mon, 27 Aug 2012 09:33:37 -0700
From: sicl...@googlemail.com
To: seleniu...@googlegroups.com
Subject: Re: [selenium-users] Code Coverage of Selenium
i would appreciate any other good metric to measure the quality of the written code of webdriver
Am Montag, 27. August 2012 18:21:16 UTC+2 schrieb PeterJef...@Hotmail.co.uk:
Am Montag, 27. August 2012 18:21:16 UTC+2 schrieb PeterJef...@Hotmail.co.uk:
I should add that "code coverage" as a percentage is not a measure of the quality of the code being tested (i.e. the quality of the product), it is just a measure of the extent to which we have tested it, and from one point of view only. So 100% code coverage from our unit tests does not been that we have 100% coverage of the functionality of the end user application.
--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To post to this group, send email to seleniu...@googlegroups.com.
To unsubscribe from this group, send email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/fTkvNUt1Gp8J.
Josef M.
Aug 27, 2012, 1:00:28 PM8/27/12
to seleniu...@googlegroups.com
You're right. It's not really important for end user.
How would you measure the effectivity of the functioanlity? i can't do a study with probands and give them a list of tasks they have to do in a specific time period
How would you measure the effectivity of the functioanlity? i can't do a study with probands and give them a list of tasks they have to do in a specific time period
Peter Gale
Aug 27, 2012, 1:15:16 PM8/27/12
to Selenium Users
Given a specific application under test, and using a specifc browser and web driver driver implementation, you could change one variable in the chain, e.g. the Selenium Webdriver verison in use, and do comparative measures of very targeted functions between those.
But then why would anyone not go for the latest version of a particular flaviour of Selenium, given that it should have all the latest bug fixes and new functionality in?
You could try to do comparisons of Selnium vs other tools like QTP or Sikuli, but then the choice of which tool to use is not just about the absolutely efficiency of the way the tool does what it sets out to achieve, but the more subjective choice about which is better architecture to build your automation on.
But it all sounds like a very academic exercise.
Date: Mon, 27 Aug 2012 10:00:28 -0700
But then why would anyone not go for the latest version of a particular flaviour of Selenium, given that it should have all the latest bug fixes and new functionality in?
You could try to do comparisons of Selnium vs other tools like QTP or Sikuli, but then the choice of which tool to use is not just about the absolutely efficiency of the way the tool does what it sets out to achieve, but the more subjective choice about which is better architecture to build your automation on.
But it all sounds like a very academic exercise.
Date: Mon, 27 Aug 2012 10:00:28 -0700
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/Gf_bklugpNoJ.
Tanusree Mukherjee
Aug 27, 2012, 1:28:23 PM8/27/12
to seleniu...@googlegroups.com
U can use eCobertura. People use it for Java code coverage(It's a plug in for eclipse), honestly I'm not sure if it can be useful for your project or not.Regards.
Tanusree
Tanusree
Josef M.
Aug 27, 2012, 1:36:58 PM8/27/12
to seleniu...@googlegroups.com
@Peter
That's what i looking for my university thesis ;)
I compare Selenium WebDriver with Thucydides. Therefor i wanted to compare the code coverage of both frameworks ;)
And ur right, it's a little bit senseless to investigate an older version of a product if there are newer versions with bug fixes.
i'm still working several month on this work and it's not easy to measure useful metrics with strong significance :/
@Tanusree
You mean i should import the whole webdriver project into eclipse? I'll google for more information.
That's what i looking for my university thesis ;)
I compare Selenium WebDriver with Thucydides. Therefor i wanted to compare the code coverage of both frameworks ;)
And ur right, it's a little bit senseless to investigate an older version of a product if there are newer versions with bug fixes.
i'm still working several month on this work and it's not easy to measure useful metrics with strong significance :/
@Tanusree
You mean i should import the whole webdriver project into eclipse? I'll google for more information.
Tanusree Mukherjee
Aug 27, 2012, 2:01:46 PM8/27/12
to seleniu...@googlegroups.com
Josef,
--
Regards.
Tanusree
Yes, Honestly I don't know what IDE u r using but if u use Eclipse and use this eCobertura plug in u have to import the entire project in to eclipse.
I never used this plug in, but in my company other people use it and those people also work with selenium Web Driver so I can try to get / find more info.
Or you can google to get some more info about it. Or if you have some time, you can just install it in eclipse , go to the Help, and play with it to get more info about it. Or you can find some other groups to ask if anyone used any this plug in.
Regards.
Tanusree
To view this discussion on the web visit https://groups.google.com/d/msg/selenium-users/-/Vk1bpzCT9H0J.
Regards.
Tanusree
Reply all
Reply to author
Forward
0 new messages