How to measure JavaScript code coverage when running Selenium?

[Tom C]

I have recently discovered the wonderful world of Selenium. There seems however to be one obvious thing missing or have I simply not found it?

After running my suits of test cases, I would like to see a code coverage report of what part of my JavaScripts which were executed during the test run. Functions or code lines not executed during the testing will need a closer investigation by me. This will help me understand if there is a need of more test cases or if there are lines of code which are obsoleted and should be removed.

I have experimented with JSCover but unfortunately it does not seem to be ready to use. I have only managed to manually test html pages through it. My web site consists of php pages with mySQL queries. Hence testing static html pages on the in JSCover included web server will not do. I have spent several hours trying to get the -proxy option of JSCover to work, but the proxy binding does not seem to work. Unfortunately the JSCover manual (http://tntim96.github.io/JSCover/manual/manual.xml#gettingStartedProxy) is very vague on the topic of how to test a separate (local) web server. The best I managed was to manually run the tests but JSCover seemed to be unable to create the coverage report, it just hung when arriving at the Summery tab.

Furthermore JSCover does not seem to be suited for running automated Selenium tests. The JSCover manual mentions however that it shall be possible but I fail to understand how to make that happen.

Is there anyone out there who has managed to get JavaScript coverage reports after running automated Selenium test cases towards an active server ( i.e. not just static html pages fed by a tool specific web server)?

I would be really grateful for a solution to my problem. BTW I am preferably looking for a non-commercial tool. :-)

[tntim96]

I have experimented with JSCover but unfortunately it does not seem to be ready to use

 Plenty of people are using this successfully including myself.

I have spent several hours trying to get the -proxy option of JSCover to work, but the proxy binding does not seem to work

There is working regression test at https://github.com/tntim96/JSCover/blob/master/src/test-acceptance/java/jscover/server/HtmlUnitProxyTest.java. This is probably one of the lesser used features and I'm keen to try to fix any particular issues you're having.

Furthermore JSCover does not seem to be suited for running automated Selenium tests. The JSCover manual mentions however that it shall be possible but I fail to understand how to make that happen.

Sure it is. Take a look at https://github.com/tntim96/JMonitaur/blob/master/jmonitaur-parent/jmonitaur-webapp/src/test/java/com/github/tntim96/jmonitaur/web/WebDriverJasmineTest.java (NB there's no need to start the Selenium server with WebDriver).

I would be really grateful for a solution to my problem.

I'd like to help. Why not raise an issue at https://github.com/tntim96/JSCover/issues with a example of how to reproduce the issue?

[tntim96]

I have experimented with JSCover but unfortunately it does not seem to be ready to use

Plenty of people are using this successfully including myself.

I have spent several hours trying to get the -proxy option of JSCover to work, but the proxy binding does not seem to work

There is a working regression test at https://github.com/tntim96/JSCover/blob/master/src/test-acceptance/java/jscover/server/HtmlUnitProxyTest.java. This is probably one of the lesser used features, but I'm keen to try to fix any particular issues you're having.

Furthermore JSCover does not seem to be suited for running automated Selenium tests. The JSCover manual mentions however that it shall be possible but I fail to understand how to make that happen.

Sure it is. Take a look at https://github.com/tntim96/JMonitaur/blob/master/jmonitaur-parent/jmonitaur-webapp/src/test/java/com/github/tntim96/jmonitaur/web/WebDriverJasmineTest.java (NB there's no need to start the Selenium server with WebDriver).

I would be really grateful for a solution to my problem.

[tntim96]

I've added a running example at https://github.com/tntim96/JSCover-samples/blob/master/src/test/java/jscover/webdriver/jasmine/WebDriverUnderscoreProxyTest.java. Also, I've just fixed proxy issue (https://github.com/tntim96/JSCover/issues/91), so you might want to try building JSCover from trunk until 1.0.3 is released.

[Tom C]

Thanks, tntim96!

I am however primarily interested in an easy way to configure JSCover to just use my current web server without having to read a lot of java code. Even though I know java, I am just looking for a simple way of getting up an running. I am guessing that there must be more people out there looking for the same thing.

[tntim96]

a lot of java code

Removing the comments header and imports, and it's only ~100 lines of code.

Would it help to add an example that browses a site rather than executes a specific Jasmine or QUnit test?

[Tom C]

Thanks, tntim96, for you patience!

I am however afraid that I am not bright enough to get it working. What I am looking for is a step by step instruction how to be able to run my automated Selenium tests against my own web server.

What I have is:

1. Selenium test cases. They are currently stored in html. Should I export them to Java/JUnit 4/WebDriver? There is no way I can run the test cases from the Selenium IDE and still be able to measure the JS coverage?

2. The target I want to test is located at my own Apache web server installed locally. The target has some DB interaction and hence it is now suitable to move the target code to the web server supplied by JSCover.

What I want is to run the Selenium test cases and get a neat JSCover report of the JS coverage executed during the testing.

Should I run JSCover in Server Mode or Proxy Mode?

Exactly which of the supplied bat-files should I execute?

Exactly what modifications should I do to my system? Do I need to update any java files and in that case how? Where and how do I point out my test cases?

I am really eager to get this working, since I will be much more motivated designing test cases if I am able to measure that the code coverage increases.

[tntim96]

I'm working on an enhancement that may be of use to you. If you're able to follow the thread at https://github.com/tntim96/JSCover/issues/92 and checkout and build the code at https://github.com/tntim96/JSCover-localStorage, you should be able to generate the coverage reports more easily. If not you can wait until I do a complete release with full documentation and samples. The changes will enable you to generate coverage in this way in proxy mode:

1. Start your web-server
2. Start JSCover in proxy mode
3. Run your tests (no frames or windows)
   
4. Load /jscoverage.html and store the report as described in the documentation (this can be recorded in Selenium IDE in HTML, but you may need to wait for the save call to finish)
   
5. You'll have to clear the HTML localStorage['jscover'] element yourself
   

If you'd prefer to wait, I'll (hopefully) add those bits of functionality into special HTML pages so you can achieve the same without any JavaScript calls, so the process will be:

1. Start your web-server
2. Start JSCover in proxy mode
3. Run your tests (no frames or windows)
   
4. Load /jscoverage-store.html to store the report
5. Load /jscoverage-clear-local-storage.html to clear the HTML localStorage['jscover'] element

Steps 4 and 5 can probably be combined.

[tntim96]

V1.0.5 has some changes that should make what you're trying easier.

Try downloading and look at 'examples/localStorage-proxy/Readme.txt'. Let me know how you go.

[Tom C]

Thanks for the explanation. I have now had time to look more into this and I believe that I have gotten further but I am not quite there yet. I am afraid that I need an even more detailed description. Let me describe the steps I have done and please point out were I have misunderstood and go wrong. I am using JSCover-1.0.5, Firefox 24.0 and the target html page I want to run the tests on is located at http://localhost/LL/Study.php on my local PC. What I do is:

1. Make sure that my local Apache web server is running and that http://localhost/LL/Study.php is accessible.
2. I added “127.0.0.1       localhost-proxy” to my local host file.
3. I updated the Firefox network settings to use manual proxy configuration for HTTP-proxy set to localhost and Port set to 3128. I also listed no proxy to be used for “localhost, 127.0.0.1”.
4. I started examples\proxy\jscover-proxy.bat and examples\proxy\web-server.bat. Is it necessary for me to start web-server.bat or would jscover-proxy.bat be enough?
5. I believe that I have managed to set it up as expected since when I access http://localhost-proxy/LL/Study.php I get the expected content and when I inspect the loaded source code I can see that the JavaScript is instrumented. Great!
6. I run some tests and then would like to see the report. How do I do that? I try to access http://localhost-proxy:8080/jscoverage-store.html but I only get a blank page and nothing is stored in the target\jscover-proxy folder. I also tried to access http://localhost:8080/jscoverage-store.html but then I just get “Not found”. I also tried to start off by accessing http://localhost-proxy:8080/jscoverage-clear-local-storage.html before I ran the tests to make sure that everything was properly reset.

I am dying to get the information as shown in the Source tab of jscoverage.html in your server example. I am guessing that I confuse when to use “localhost”, when to use “localhost-proxy” and when to add the “:8080” afterwards. Please enlighten me.

I also tried a different approach and got further. This approach will however not allow me to run Selenium. After step 4 above I instead:

5. navigated to http://localhost-proxy:8080/jscoverage.html and in the URL field entered http://localhost/LL/Study.php . Now the test page loaded inside the JSCover frame which I believe is good. Should I have accessed http://localhost-proxy/LL/Study.php instead? That seemed to give the same result though.

 

6. I ran some tests and then choose the Summery tab. There I only found the file /ws/js/base_single_icon.js and not my own JS-file. I guess that the file base_single_icon.js is part of JSCover and should not really be instrumented. I also try to choose the Store tab and press Store Report. I get some files in target\jscover-proxy and when opening jscoverage.html I unfortunately only get information about base_single_icon.js and not about LL.js which is the name of the JS file I am including in LL/Study.php. Interestingly under target\jscover-proxy\original-src the two files \original-src\ws\js\base_single_icon.js and \original-src\LL\LL.js exist meaning that JSCover has at least at some point registered that LL.js is a JS file which should generate a report.

 Please help me out here since my dream is to easily and regularly be able to run Selenium tests and then inspect the JavaScript Coverage to be able to determine if I have any dead code or missing test cases.

[tntim96]

I think you're almost there.

Is it necessary for me to start web-server.bat or would jscover-proxy.bat be enough?

You don't need to start 'web-server.bat' if you are serving your PHP page from another server. JSCover's web-server won't know how to handle PHP files.

I try to access http://localhost-proxy:8080/jscoverage-store.html

I didn't end up implementing that URL, which is why it's not in the example. Just replace that URL, in step 6, with http://localhost-proxy:8080/jscoverage.html, click on the "Store" tab and then the "Store Report" button and it should work.

I started examples\proxy\jscover-proxy.bat

There I only found the file /ws/js/base_single_icon.js and not my own JS-file

You may need to edit jscover-proxy.bat, but just looking at it, your JS file should appear in the summary tab if it's loaded from the HTML of http://localhost/LL/Study.php

 5. navigated to http://localhost-proxy:8080/jscoverage.html and in the URL field entered http://localhost/LL/Study.php . Now the test page loaded inside the JSCover frame...

This approach can be used, but your first approach should work with the correct URL as described above.

[Tom C]

I gave it another try with your tips but unfortunately still no success.

Just to make sure that there was not problem with my source code, I copied your example files script.js and index.html to the root of my web server. I had no problem accessing them at http://localhost-proxy/. When I however called http://localhost-proxy:8080/jscoverage.html the page hung when I pressed the Summary tab. When trying again and choosing the Store tab instead and hitting Store Report, the content in target\jscover-proxy\jscoverage.html turned out to be empty. 

Since no one else seems to be having any troubles with this, I am starting to suspect that my problems could be caused by the fact that I have an old computer (5+ years). I will try to get hold of a new one and try it out again to see if that will improve the situation.

[tntim96]

The computer shouldn't matter. Can you run this example successfully?

[Tom C]

I ran the example but unfortunately it failed in the last step. When opening /target/local-storage-proxy/jscoverage.html there is no JS file listed. Between "Show missing statements column" and the headlines "File  Statements  Executed  Branches  Branches Hit  Functions Functions Hit" the following error message is displayed: "Error: NetworkError: A network error occurred."

[tntim96]

That's a problem. The '/target/local-storage-proxy' directory should have the following files:

• jscover.log
• jscoverage-highlight.css
• jscoverage-ie.css
• jscoverage-throbber.gif
• jscoverage.css
• jscoverage.html
• jscoverage.js
• jscoverage.json
• original-src
• original-src/script.js

I'm guess the bolded one is missing. Something must have gone wrong earlier. Can you step through the Selenium IDE and see which step fails? I would think 'Step 9' can't succeed if you are getting an error. Maybe you could zip the directory up and put the files somewhere I can download and examine?

[Monty]

Hi,

I am new to this Java world and me too having some kind of similar scenario,
1. I am having FitNesse UI test cases developed using Java (eClipse) & Selenium libraries
2. tomcat is running on a linux box
3. From my windows box I want to run these fitnesse UI test cases and generate code coverage report.

I am looking for is a step by step instruction to generate JS code coverage.Any help will be deeply appreciated.

Thanks,