Re: [testng-users] screenshot in results with Selenium and ReportNG

[Cédric Beust ♔]

Hi Kimberly,

Thanks for the detailed report and explanation, I'm sure all this will be useful to Selenium users.

Having said that, ReportNG is not part of the TestNG distribution, so I'm not sure there is much I can act on right now. 

I'd like to add some support in TestNG specifically for Selenium users, so any suggestions to that effect would be great. For example, we could have a specific listener that takes screen shots and puts them in a directory and the reports can then see if this directory exists and if it does, include the pictures found there directly in the reports.

Suggestions and contributions welcome!

-- 

Cédric

On Fri, Aug 10, 2012 at 12:26 PM, Kimberly Nicholls <kielni...@gmail.com> wrote:

I wanted to include screenshots with my failed Selenium tests in the ReportNG output, but it was harder than I thought. I started with taking a screenshot in an @AfterMethod since I need access to the WebDriver to take the screenshot.  The screenshot worked, but it was too late to add it to the report output since the @AfterMethod runs after the reporter listener.  I eventually got it working by putting the driver into the test context, extending ReportNG's HTMLReporter to implement onTestFailure and take the screenshot, and specifying a different utils class that adds the screenshot to the report output.

Some things that would have made it easier:

• Don't use the selenium-server-standalone jar.  It includes an old (6.0.1) version of testng that doesn't have the getTestContext method on ITestResult.  It conflicted with my newer testng jar that did have this method.
• I was surprised that @AfterMethod does not run immediately after the method, but after the reporter listeners.  I know it's probably too late to change that, so what about a new annotation (@ImmediatelyAfterMethod?) that runs immediately after the test method, before any listeners?
  
• I'd like to be able to override the ReportNG templates easily.  They are loaded directly from the jar file with no way to specify your own.  If I could do that, I could add a new call $utils.getTestScreenshot($testResult) that would output the img tag for the screenshot without having to turn off escaping for all of the output.  I noticed you can override the stylesheet (org.uncommons.reportng.stylesheet), so why not templates?
  
• The Velocity templates use a separate ReportNGUtils class to get their data.  That made it harder to override, since I had to substitute my class in the createContext method, and the key (in AbstractReporter) was defined private.
  
• Figuring out where to put the output was a little messy.  I got the output path from the test context in the onTestFailure method, but it included the TestNG suite directory, which ReportNG doesn't know about.  The ReportNG HTMLReporter puts its files in subdirectory, which I couldn't access in my subclass because it was declared private.
  

Here are the details; I hope this helps someone.

in test class, save driver into test context:

@BeforeSuite(alwaysRun = true)

public void setupBeforeSuite(ITestContext context) {

driver = new FirefoxDriver();

context.setAttribute(ScreenshotReportNGUtils.DRIVER_ATTR, driver);

}

extend ReportNG's HTML reporter: override createContext to use a custom utils class; implement onTestFailure from ITestListener to get the driver from the test context, take a screenshot, and save the screenshot filename and driver URL as result attributes

public class ScreenshotHTMLReporter extends HTMLReporter implements ITestListener {

    protected static final ScreenshotReportNGUtils SS_UTILS = new ScreenshotReportNGUtils();

    

    /* (non-Javadoc)

     * @see org.uncommons.reportng.AbstractReporter#createContext()

     * override to use a custom utils class

     */

    protected VelocityContext createContext() {

        VelocityContext context = super.createContext();

        // UTILS_KEY is private

        context.put("utils", SS_UTILS);

        return context;

    }

    public void onTestFailure(ITestResult tr) {

        ITestContext context = tr.getTestContext();

        WebDriver driver = (WebDriver)context.getAttribute("driver");

        try {

            File f = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

            // output dir includes suite, so go up one level 

            String outputDir = tr.getTestContext().getOutputDirectory();

            outputDir = outputDir.substring(0, outputDir.lastIndexOf("/"));

            File saved = new File(outputDir, "ss_"+tr.getMethod().getMethodName()+".png");

            FileUtils.copyFile(f, saved);

            // save screenshot path as result attribute so generateReport can access it

            tr.setAttribute("screenshot", saved.getName());

            tr.setAttribute("screenshotURL", driver.getCurrentUrl());

        } catch (Exception e) {

System.out.println("error generating screenshot: "+e);

        }

}

...

    }

custom utils class that looks for screenshot info in result attributes and adds it to the output if found

public class ScreenshotReportNGUtils extends ReportNGUtils {

    public static final String DRIVER_ATTR = "driver";

    /* (non-Javadoc)

     * @see org.uncommons.reportng.ReportNGUtils#getTestOutput(org.testng.ITestResult)

     * override to add screenshot from result attribute

     */

    public List<String> getTestOutput(ITestResult result) {

        List<String> output = super.getTestOutput(result);

        // add screenshot if there is one

        String screenshot = (String)result.getAttribute("screenshot");

        if (screenshot != null) {

            String url = (String)result.getAttribute("screenshotURL");

            if (url == null)

                url = "";

            // ReportNG output directory is private, so get screenshot from output root

            output.add("screenshot for "+result.getName()+" "+url+"<br><img src=\"../"+screenshot+"\">");

        }

        return output;

    }

}

in ant build file

• replace ReportNG HTMLReporter listener with my new ScreenshotHTMLReporter
  
• add <sysproperty key="org.uncommons.reportng.escape-output" value="false" /> to prevent escaping of img tag
  

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/t_2Ks7iq-PQJ.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Kimberly Nicholls]

Thanks for the reply.  I posted it here just because I thought it'd be the most likely place for someone to find it -- I have been reading lots of posts here recently..

The simplest thing TestNG could do is add an annotation and/or a config option for running a method after the @Test but before the listeners/reporters.  Then the screenshot method could write the screenshot details to the reporter log like any other test output.  It seems a lot cleaner to me to keep the Selenium driver/screenshot logic in the test class instead of passing it around through attributes.  

Kimberly Nicholls

[Cédric Beust ♔]

Hi Kimberly,

I recently made a change regarding Reporter.log() but I'm not sure if it would help you or not, could you try the beta and let me know if it changes anything for you? It's at http://testng.org/beta

-- 

Cédric

[edwolb]

Hi Kimberly,

We've implemented TestNG + Selenium here, and we have managed to get ReportNG to help us report on not only failure screenshots but step screenshots too.  We took some similar steps in getting where we are, and some different steps.

Rather than attempting to use an @AfterMethod, our setup actually has all tests extending a "SimpleSeleniumTemplate" class that we use to store common functions for all of our tests.  Part of this base class includes a reference to the driver, as well as a screenshot function:

  public File takeScreenShot() throws IOException {

 return DriverUtils.getScreenShot(this.driver);

  }

(in another class:)

public class DriverUtils {

public static File getScreenShot(WebDriver driver) {

   WebDriver d;

   try {

   if (driver.getClass().getName().equals("org.openqa.selenium.remote.RemoteWebDriver")) {

     d = new Augmenter().augment(driver);

   } else {

     d = driver;

   }

   return ((TakesScreenshot)d).getScreenshotAs(OutputType.FILE);

   } catch (Exception e) {

   

   }

   return null;

}

}

And finally we implement a standalone listener that sends logs to ReportNG, and has access to the test class via the TestResult:

  public void onTestFailure(ITestResult tr) {

    System.out.println("FAILED: " + tr.getInstance().getClass().getName() + "." + tr.getMethod().getMethodName());

    this.takeScreenshotAndLog(tr);

  }

  private File getScreenShot(ITestResult tr) throws IOException{

    

    if (!(tr.getInstance() instanceof SimpleSeleniumTemplate))

      return null;

    

    SimpleSeleniumTemplate testClass = (SimpleSeleniumTemplate) tr.getInstance();

    File scr = null;

    if (testClass != null) {

      scr = testClass.takeScreenShot();

      if (scr != null) {

        SimpleDateFormat sdf = new SimpleDateFormat("yyMMddHHmmssSSS");

        String date = sdf.format(new Date());

        File scrCopy = new File("test-output/SS-" + date + "-" + tr.getInstance().getClass().getName() + "." + tr.getMethod().getMethodName() + ".png");

        FileUtils.copyFile(scr, scrCopy);

        scr = scrCopy;

      }

    }

    return scr;

  }

And then we implemented the standard Report.log() to include HTML to show screenshots, stack traces and internal logs which gives you somewhat of an investigation / reproduction log all in one place, which we access via Jenkins.  For some reason, the report.log works fine with ReportNG, but has never worked with the standard TestNG results (sorry Cedric).  The additional downside is that ReportNG lumps all the results together, so they're chronological, but they're not tied to the actual tests it was executing when it failed.  We make up for this by documenting the specific test that is failing during the reporting process.

We also have a static logging feature that uses some threading magic to associate static logs to specific tests, and it allows us to capture steps, debug information and screenshots that are used during failure reporting.

We're now working towards integrating TestNG with a test management system so that our listener reports results, stacktraces, logs and screenshots to the test management system.

--

Chris

[Kimberly Nicholls]

I tried the beta and now it's calling @AfterMethod before the reporters, which is perfect.  Now all I need to get screenshots in the log is an @AfterMethod in my test class:

@AfterMethod(alwaysRun = true)

public void takeScreenshot(ITestResult result) throws IOException {

// output gets lost without this

Reporter.setCurrentTestResult(result);

if (result.isSuccess())

return;

File f = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

File outputDir = new File(result.getTestContext().getOutputDirectory());

File saved = new File(outputDir.getParent(), result.getName()+".png");

FileUtils.copyFile(f, saved);

// this works for the TestNG reporter log but not for ReportNG since the results are under the html/ subdir 

Reporter.log("screenshot for "+result.getName()+" url="+driver.getCurrentUrl()+" <img src=\""+saved.getName()+"\">", true, false);

}

I added a Reporter.log method with a boolean parameter to skip the HTML escaping.  I can upload that to github if you want.

The only tricky part was that I needed the Reporter.setCurrentTestResult(result); line.  Otherwise the output looked like it was there but got lost somewhere before it made it to the final log.  Here's what I saw (from stdout):

   [testng] afterMethod before setting test result

   [testng] afterMethod output=[[b]start testFail[/b][hr]<br>, test method start<br>, onTestFailure testFail created screenshot ss_testFail.png<br>, afterMethod before setting test result<br>]

   [testng] afterMethod after setting test result

   [testng] afterMethod output=[[b]start testFail[/b][hr]<br>, test method start<br>, onTestFailure testFail created screenshot ss_testFail.png<br>, afterMethod before setting test result<br>, afterMethod after setting test result<br>]

   [testng] afterMethod screenshot for testFail=testFail.png

   [testng] afterMethod output=[[b]start testFail[/b][hr]<br>, test method start<br>, onTestFailure testFail created screenshot ss_testFail.png<br>, afterMethod before setting test result<br>, afterMethod after setting test result<br>, afterMethod screenshot for testFail=testFail.png<br>]

   [testng] generateReport start reporter=[[b]start testFail[/b][hr]<br>, test method start<br>, onTestFailure testFail created screenshot ss_testFail.png<br>, afterMethod before setting test result<br>, afterMethod after setting test result<br>, afterMethod screenshot for testFail=testFail.png<br>]

   [testng] getTestOutput start

*** Reporter.getOutput() no longer contains the "start testFail" or "afterMethod before setting test result" lines in this utility method called from generateReport; these lines are also missing from the TestNG reporter output.

   [testng] getTestOutput reporter=[test method start<br>, onTestFailure testFail created screenshot ss_testFail.png<br>, afterMethod after setting test result<br>, afterMethod screenshot for testFail=testFail.png<br>, getTestOutput start reporter length=4]

The reporter output had

debug: test method start
debug: afterMethod after setting test result
debug: afterMethod screenshot for testFail=testFail.png

Kimberly Nicholls

[Cédric Beust ♔]

Hi Kimberly,

Thanks for the update, and glad to hear the latest version works for you.

I'm a bit puzzled by the fact that you have to call Reporter.setCurrentResult() (which should be a private method, by the way, so you shouldn't use it) because that's what the fix does: making sure we remain within the context of the current method eve in After methods...

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/GLCngFk4DEwJ.

[Kimberly Nicholls]

Here's some code without all the distracting screenshot stuff:

@AfterMethod(alwaysRun = true)

public void logResults(ITestResult result) throws IOException {

Reporter.log("debug: logResults before setCurrentTestResult", true);

Reporter.setCurrentTestResult(result);

Reporter.log("debug: logResults after setCurrentTestResult", true);

}

@Test(description="failing test")

public void testFail() {

Reporter.log("debug: test method start", true);

Assert.assertTrue(1 < 0, "fail");

}

With the 6.8 beta, I get this in stdout:

   [testng] debug: test method start

   [testng] debug: logResults before setCurrentTestResult

   [testng] debug: logResults after setCurrentTestResult

and this in the reporter log:

debug: test method start
debug: logResults after setCurrentTestResult

Kimberly

[Mike]

Try something like this (works for me):
            WebDriver augmentedDriver = getScreenshot();
            try
            {
                File file;
                String str = System.getProperty("user.dir") + "/"; // Current dir

                file = ((TakesScreenshot) augmentedDriver).getScreenshotAs(
                        OutputType.FILE);
                FileUtils.copyFile(file, new File(filename));
                Reporter.log("<img src=\"file:///" + str + filename +
                             "\" alt=\"\"/><br />");
            }
            catch (Throwable ex1)
            {
                Reporter.log("Unable to capture screentshot to " + filename +
                             "<br/>Got error: " + ex1.getMessage());
            }   // try-catch

getScreenshot() is simply this:
    private WebDriver getScreenshot()
    {
        return new Augmenter().augment(driver);
    }   // getScreenshot

You simply need to know what directory the image file will be placed in.  I control that with a passed parameter, which is why I am using a variable here, but you can use a constant if you wish.

Mike

On Tuesday, January 29, 2013 2:47:42 PM UTC-8, zeiswei...@gmail.com wrote:

Kimberly,

Can you attach the method you created? I am successfully taking the screenshots, but I cant get them to show up in my HTML report- on the location is showing up in the log.

"I added a Reporter.log method with a boolean parameter to skip the HTML escaping.  I can upload that to github if you want."

Thank you so much for the info!

[Kimberly Nicholls]

Here is my screenshot method.  I put the screenshots in the parent of the output directory, which is the root testng directory.  That way the path works for both the standard testng output and the reportng HTML files.  

protected void takeScreenshot(ITestContext context, String name) {

File f = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

File dir = new File(context.getOutputDirectory());

File saved = new File(dir.getParent(), "ss_"+name+".png");

try {

FileUtils.copyFile(f, saved);

Reporter.log("screenshot for "+name+" url="+driver.getCurrentUrl()+

"<div style=\"height:400px; width: 750px; overflow:scroll\"><img src=\""+saved.getName()+"\"></div>", 

true);

} catch (IOException e) {

Reporter.log("error generating screenshot for "+name+": "+e, true);

}

}

[Max Mazurkevych]

hi to all!
Kimberly,
I have been trying to get my reportNg results for more than 2 weeks
but my attempts were not successful!

please, can you send me some project were reportNg results are implemented!
I would be very thankful!

[shakun tyagi]

Hi Cedric,

I have been trying the same to link my screenshots to the reportng but failed to get the screenshot link.

I am new to the selenium and testng world.

Here are the methods 

@AfterMethod

  public void setScreenshot(ITestResult result) {

System.out.println("AfterMethod1");

      if (result.isSuccess()) {

          try {

               SimpleDateFormat formater = new SimpleDateFormat("dd_MM_yyyy_hh_mm_ss");

//              WebDriver returned = new Augmenter().augment(webDriver);

//               WebDriver returned = new FirefoxDriver();

              if (augmentedDriver != null) {

                  File f = ((TakesScreenshot) augmentedDriver)

                          .getScreenshotAs(OutputType.FILE);

                  try {

                      FileUtils.copyFile(f, new File(SCREENSHOT_FOLDER

                              + result.getName()+"_"+formater.format(Calendar.getInstance().getTime()) + SCREENSHOT_FORMAT));

                  } catch (IOException e) {

                      e.printStackTrace();

                  }

              }

          } catch (ScreenshotException se) {

              se.printStackTrace();

          }

      }

      wdc.driver.close();

  }

  

  

  @AfterMethod

public void Screen(ITestResult result) throws Exception

{

 

 System.out.println("AfterMethod2");

if (result.isSuccess()) 

   takeScreenShoot(augmentedDriver, result.getMethod());

  

wdc.driver.close();

}

  

  

  public void takeScreenShoot(WebDriver driver, ITestNGMethod testMethod) throws Exception {

//      WebDriver augmentedDriver = new Augmenter().augment(driver);

   

 

 

      File screenshot = ((TakesScreenshot) augmentedDriver).getScreenshotAs(OutputType.FILE);

      String nameScreenshot = testMethod.getTestClass().getRealClass().getSimpleName() + "_" + testMethod.getMethodName();

      String path = getPath(nameScreenshot);

      FileUtils.copyFile(screenshot, new File(path));

//      Reporter.log("<a href=" + path + " target='_blank' >" + this.getFileName(nameScreenshot) + "</a>");

      Reporter.log("<img src=\"file:///" + path + this.getFileName(nameScreenshot) + "\" alt=\"\"/><br />");

//      System.out.println("<a href=" + path + " target='_blank' >" + this.getFileName(nameScreenshot) + "</a>");

      System.out.println("<br><img src=\"file:///" + path + this.getFileName(nameScreenshot) + "\" alt=\"\"/></br>");

      }

      private String getFileName(String nameTest) throws IOException {

      DateFormat dateFormat = new SimpleDateFormat("MM-dd-yyyy_hh.mm.ss");

      Date date = new Date();

      return dateFormat.format(date) + "_" + nameTest + ".png";

      }

      private String getPath(String nameTest) throws IOException {

      File directory = new File(".");

      String newFileNamePath = directory.getCanonicalPath() + "\\target\\surefire-reports\\screenShots\\" + getFileName(nameTest);

      return newFileNamePath;

      }

Thanks in Advance.

Shakun

[David Goldstick]

I almost have this working. On test failure, instead of embedding the image in ReportNG, it's printing the HTML explicitly instead of rendering it. I hope this isn't a bad question, but I've been looking all over for how to do this. Thanks!

Screenshot for testForX<html><br /><img src="../ss_testForX.png"></html>

Here's the pertinent part of my ScreenshotReportNGUtils code:

public List<String> getTestOutput(ITestResult result) {
        List<String> output = super.getTestOutput(result);
        // add screenshot if there is one
        String screenshot = (String)result.getAttribute("screenshot");
        if (screenshot != null) {
            String url = (String)result.getAttribute("screenshotURL");
            if (url == null)
                url = "";

            Document.class(null);

            // ReportNG output directory is private, so get screenshot from output root

            StringBuilder str = new StringBuilder ("<html><br /><img src=\"../" + screenshot + "\"></html>");
            output.add("Screenshot for " + result.getName() + str);
        }
        return output;

            File f = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

[David Goldstick]

I'm happy to report I figured out the problem with the escaping which prevented html rendering. I hadn't yet developed the ANT script and was testing from Eclipse and was searching how to best handle it. Additionally, I added the listener to testng.xml instead of ANT and it works fine.

This link will help those with the escaping issue who are testing first in Eclipse: http://stackoverflow.com/questions/11957105/how-to-set-sysproperty-in-testng-using-eclipse

The first part of question could be addressed by accessing following menu - Run -> Run|Debug Configuration -> Your Configuration -> 'Arguments' tab -> VM arguments. Add following line to the text area:

     -Dorg.uncommons.reportng.escape-output=false

Thanks so much to Kimberly and everyone for the great example!