How to measure End2End Test coverage?

[Jack Zhang]

I used angularjs e2e for my project's end2end test i can also run that in karma.

I know i can use 'karma-coverage' plugin to measure my code coverage, but this is only for unit test.

Now, i want to measure End2End Test coverage which help me know how much my test cases have coverage scenario, Anyone can tell me how to measure end2end test coverage automatically?

[Daniel Tabuenca]

I don't believe you can do this with karma for end to end tests. During normal unit testing the files are served by karma itself so it can instrument them and fully control things. During end-to-end testing, the files are being served by your web server.

If you are looking for javascript coverage, you might want to look at using Istanbul (the library karma is using for coverage) directly. 

https://github.com/yahoo/istanbul

[Jack Zhang]

Well, thank you for your replay.

But we really need a way which can help us measure the coverage of our e2e test in our project. We want a quantifiable report to guarantee the quality of e2e test. If karma can not do this, do you have other solutions ?

Very appreciate for your response.

[Jack Zhang]

I tried the tools you suggest, but seems all this kind of tools test for JavaScript file. In my understanding, End2End test is used for frond-end test which simulator user interaction. It don't related to any js files. (is my understanding right?) so this why this tools can't make e2e coverage.

But is there any tools can test html coverage? I mean how many html elements do the end2end test coverage? Is this the solution we can measure the end2end coverage?

expect for your reply.

On Thursday, October 31, 2013 2:19:07 AM UTC+8, Daniel Tabuenca wrote:

[Daniel Tabuenca]

Coverage reporting works by instrumenting code so that it can keep track of usage, then excersizing that code  somehow (usually through unit tests, but could be anything including a human manually going through test cases on a browser). For karma unit tests this is easy, because you can tell karma to instrument the script files before they are served, and automatically gather the statistics and generate a report.

With end to end testing karma can not instrument the files nor give you a report, but this does not mean that YOU can't  instrument the files that are served by the server yourself. You can even instrument non-javascript code like C#  or Java or whatever server-side language you are using. This absolutely possible, and probably not overly difficult, but you will have to understand how to use the tools available to your particular platform yourself (how to apply the instrumentation and how to gather reports). I doubt anyone will be able to give you step-by-step instructions on how to do this since it is likely highly dependent on your particular configuration, build environment, and tools. 

As far as HTML coverage, I'm not sure there are necessarily any tools for that. Coverage usually tells you that a particular line of code has been executed, but what constitutes executed in HTML? Is it that it has been displayed? That it has been served? If you really need to know this kind of information you'd likely need to write your own tool or script. Perhaps something that can compile HTTP server access logs and compare it to a set of html paths you would like to be covered. However, most people would be satisfied to concentrate on getting good coverage on unit tests, and concentrate on covering user stories or acceptance test specifications in end2end (rather than code coverage).

[Alon Nisser]

It can be done (figured it a few days ago) - Also hacky (At least the js part, not the html) I'll be posting later the exact process I use, but mainly I use istanbul intrument to create an instrumented js folder for my app/js, I create (on the fly) an alternative index.html to load this js instead of the regular js and changed scenario.js files to use the changed index.html instead of the orginial one
I automated everything in the e2e-tesh.sh.

I hope later today (or tomorrow) I'll have to to post a full solution

[Jianbo Zhang]

There is one thing confused me:

if code coverage only instrument js file, but all this js files can be instrument by karma unit test. Why we need end2edn coverage?

Like my angular project, all the js (like controller, directive, services, module etc) files coverage have already measured by unite test case using karma code coverage, Does that mean we actually don't need another coverage report for end2end test?

--
You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/leh6qVceZ8s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

[Daniel Tabuenca]

Unit tests test your code in smaller isolated units, end-to-end is a form of integration test that test how your code works together with all other units including interactions with the server.

Coverage just tells you how much of the code executed under a particular scenario. Coverage during karma unit test tells you what code was actually executed during your unit tests. Coverage under an end-to-end secenario would tell you what code was executed during end-to-end testing.

It is very common for people to concentrate on getting very high coverage for unit tests (testing all possible edge-cases), while not worrying about code coverage for end-to-end. 

What is important for end-to-end is to make sure your pages work like you expect. End-to-end tests both your client-side and server-side. You can test things like making sure all your buttons and links work like you expect, but typically you would not try to get 100% coverage through end-to-end tests since they are typically much more fragile.

[Xiaohui Chen]

Have you got chance to post your solution yet?  Cause I am working on a similar project but put files instrumented by istanbul on server did not work.

[alonn]

Since Angular moved to Protractor  I didn't post the working solution - If this is still relevant to you I can post it here

Twitter:@alonisser

LinkedIn Profile

Facebook

Tech blog:4p-tech.co.il/blog

Personal Blog:degeladom.wordpress.com

Tel:972-54-6734469

--
You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/leh6qVceZ8s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.

For more options, visit https://groups.google.com/d/optout.

[Xiaohui Chen]

I think my problem is the way instrumented files were put on the server.  You mention you have a special index.html, so I'd like to see how you did it.  Please post it if not too much trouble.  I appreciate your help. 

Tony

[alonn]

This is my script to rune2e with instrumenting:

As you can see I create an instrumented copy of my js files.

change index.html to reference the instrumented files (with sed)

the script:

#!/bin/bash

BASE_DIR=`dirname $0`

echo ""

echo "Istanbul instrumenting js files"

echo "-------------------------------------------------------------------"

istanbul instrument $BASE_DIR/../app/js -o $BASE_DIR/../app/instrumentjs

echo ""

echo "Building an instrumented index.html"

echo "------------------------------------------------------------------"

cat $BASE_DIR/../app/index.html | sed s/\"js/\"instrumentjs/ > app/index_e2etest.html

echo ""

echo "Starting node.js local webserver to run the tests"

echo "-------------------------------------------------------------------"

node $BASE_DIR/../scripts/web-server.js > /dev/null 2>&1 &

echo ""

echo "Starting Karma Server (http://karma-runner.github.io)"

echo "-------------------------------------------------------------------"

karma start $BASE_DIR/../config/karma-e2e.conf.js $*

Also: in my karma-e2e.conf I added the following lines:

preprocessors: {

            '../app/instrumentjs/**/*.js': 'coverage'

        },

Setting the coverage plugin to check the instrumented js files we created before

Twitter:@alonisser

LinkedIn Profile

Facebook

Tech blog:4p-tech.co.il/blog

Personal Blog:degeladom.wordpress.com

Tel:972-54-6734469

[Xiaohui Chen]

Thanks a lot.

Tony