Why Selenium Server takes screenshots on exceptions by default?

218 views
Skip to first unread message

Alexei Barantsev

unread,
Dec 23, 2016, 9:26:46 AM12/23/16
to Selenium Developers
Hi, devs,

Why Selenium Server takes screenshots on exceptions by default? It's huge performance penalty.

An example:

      for (int i = 1; i < 100; i++) {
        try {
          driver.get("test.com");
        } catch (WebDriverException ex) {
        }
      }

This piece of code takes ~25 seconds and generates ~20MB of screenshots in ChromeDriver on my machine with 1920x1200 sceen resolution using the server running on the same machine.

With capability "webdriver.remote.quietExceptions" set to true the same code takes ~1,7 seconds.

15x time perfomance boost and -20MB of useless traffic.

And again the question is: why this feature is ON by default?

Regards,
-- 
Alexei Barantsev
Software-Tesing.Ru
Selenium2.Ru

Simon Stewart

unread,
Dec 24, 2016, 4:31:15 PM12/24/16
to selenium-developers
Same reason we include all sorts of information in our exceptions: because it's proved the best way to provide debugging information that people weren't otherwise providing. 

Normally, an exception marks the end of a test, so your example is a little contrived. The only case where I've seen people doing something that might do otherwise on a regular basis is checking to see if an element is present (by using "findElement" rather than "findElements").

In real life usage, is this causing you problems?

Simon

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/49f301e8-7f66-4a3c-b72c-a9ce410f3753%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alexei Barantsev

unread,
Dec 25, 2016, 4:08:53 AM12/25/16
to Selenium Developers
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

4) How is it supposed to be used? Do all test frameworks (JUnit/NUnit/py.test/rspec/mocha/...) allow to examine the exception occured during test execution in a kind of @After-block? No. Should a user catch an exception in every test and examine and save the screenshot if present?

5) And it is present there *only* if it is a remote session. If you run test locally you have to implement taking screenshots on failure anyway. So they do it in local/remove agnostic manner.

My suggestion:
a) disable this feature by default, because it is not very popular, but gives performance penalty to all users,
b) may be implement it for local runs too.

Alexei Barantsev,
-- 
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 0:31:15 UTC+3 пользователь Simon Stewart написал:

Paul Hammant

unread,
Dec 25, 2016, 6:25:00 AM12/25/16
to selenium-developers
I don't think it does take screenshots by default/silently:

ChromeDriver chromeDriver = new ChromeDriver();
long start = System.currentTimeMillis();

for (int i = 1; i < 100; i++) {
    try {
        chromeDriver.get("test.com");
    } catch (WebDriverException ex) {
    }
}
System.err.println("millis " + (System.currentTimeMillis() - start));

3.4 seconds on my 2011 Mac with 4GB RAM, no pngs the current directory.

Alexei Barantsev

unread,
Dec 25, 2016, 2:17:22 PM12/25/16
to Selenium Developers
Yes, if you run locally -- there are no screenshots.
Selenium Server takes them and attach them to exceptional responses.

Regards
-- 
Alexei Barantsev
Software-Testing.Ru
Selenium2.Ru

воскресенье, 25 декабря 2016 г., 14:25:00 UTC+3 пользователь Paul Hammant написал:

Paul Hammant

unread,
Dec 25, 2016, 4:18:42 PM12/25/16
to selenium-developers
Could you help me reproduce this, please :)

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/512df7c7-8dc8-488b-8899-95a003a8293d%40googlegroups.com.

Alexei Barantsev

unread,
Dec 26, 2016, 4:36:22 AM12/26/16
to Selenium Developers
1) Start Selenium Server
2) Run the sample:

    DesiredCapabilities caps = DesiredCapabilities.chrome();
    //caps.setCapability("webdriver.remote.quietExceptions", true);
    WebDriver driver = new RemoteWebDriver(caps);
    long start = System.currentTimeMillis();
    for (int i = 1; i < 100; i++) {
      try {
        driver.get("test.com");
      } catch (WebDriverException ex) {
      }
    }
    System.err.println("millis " + (System.currentTimeMillis() - start));
    driver.quit();

3) Try to uncomment the line that sets "webdriver.remote.quietExceptions" capability and run again.

Regards,
-- 
Alexei Barantsev
Software-Testing.Ru
Selenium2.Ru


понедельник, 26 декабря 2016 г., 0:18:42 UTC+3 пользователь Paul Hammant написал:

Simon Stewart

unread,
Jan 5, 2017, 10:46:07 AM1/5/17
to selenium-developers
After a heck a pause, replying. Inline.

On Sun, Dec 25, 2016 at 9:08 AM, Alexei Barantsev <bara...@gmail.com> wrote:
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

Let's fix it, then. Given the discussions around IDE, we can just leave that.
 
2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

That's why we write javadocs. And why things appear in stacktraces. We try and provide as much hand-holding as makes sense, but I think that there's a lot of value-add in making the screenshots available.
 
3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

We should document this properly, then. Not remove the feature :)
 
4) How is it supposed to be used? Do all test frameworks (JUnit/NUnit/py.test/rspec/mocha/...) allow to examine the exception occured during test execution in a kind of @After-block? No. Should a user catch an exception in every test and examine and save the screenshot if present?

It's really up to them. Personally, I use junit and a runner to capture this stuff.
 
5) And it is present there *only* if it is a remote session. If you run test locally you have to implement taking screenshots on failure anyway. So they do it in local/remove agnostic manner.

The main reason we did things this way was because people running tests locally tend to be sitting in front of the machine and can see what's happening. The remote sessions (certainly at the time, and it's still the case) are being run on machines that folks can't eyeball easily. It might be nice to add a capability to allow local implementations to take screenshots on exceptions locally.
 
My suggestion:
a) disable this feature by default, because it is not very popular, but gives performance penalty to all users,

I disagree, but perhaps we can add a flag to the server to make it easier to disable. 
 
b) may be implement it for local runs too.

I like this.

Simon
 
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscribe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/35b7ec88-c56e-4fc0-9c48-e9a96a939959%40googlegroups.com.

Jason Leyba

unread,
Jan 11, 2017, 4:44:48 PM1/11/17
to selenium-...@googlegroups.com
On Thu, Jan 5, 2017 at 7:46 AM Simon Stewart <simon.m...@gmail.com> wrote:
After a heck a pause, replying. Inline.

On Sun, Dec 25, 2016 at 9:08 AM, Alexei Barantsev <bara...@gmail.com> wrote:
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

Let's fix it, then. Given the discussions around IDE, we can just leave that.
 
2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

That's why we write javadocs. And why things appear in stacktraces. We try and provide as much hand-holding as makes sense, but I think that there's a lot of value-add in making the screenshots available.
 
3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

We should document this properly, then. Not remove the feature :)

The feature shouldn't be removed, but it is a performance killer and should be disabled by default. We disabled it here back in 2012, and a quick search doesn't turn up anyone that has turned it back on.
 
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-develo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAOrAhYHCsE4DPf_7L1REVAK5hToOJ2t06cPV%3DfMqiY3gsoBB2w%40mail.gmail.com.

Simon Stewart

unread,
Jan 12, 2017, 7:40:03 AM1/12/17
to selenium-developers
On Thu, Jan 12, 2017 at 12:44 AM, Jason Leyba <jml...@gmail.com> wrote:
On Thu, Jan 5, 2017 at 7:46 AM Simon Stewart <simon.m...@gmail.com> wrote:
After a heck a pause, replying. Inline.

On Sun, Dec 25, 2016 at 9:08 AM, Alexei Barantsev <bara...@gmail.com> wrote:
1) That's exactly what we have implemented in ExpectedConditions (Java) -- use of "findElement" rather than "findElements". We can fix it, sure.

Let's fix it, then. Given the discussions around IDE, we can just leave that.
 
2) I bet many many people do the same, because it is not clear from findElement description that it has such a performance penalty. And who reads those description anyway.

That's why we write javadocs. And why things appear in stacktraces. We try and provide as much hand-holding as makes sense, but I think that there's a lot of value-add in making the screenshots available.
 
3) How many people knows about this feature? Try to google "selenium screenshotexception" -- results are close to zero. But there are thousands of recommendations how to take screenshot on test failure, and no one of them recommends to use the screenshot already obtained in the exception. They just take another one!

We should document this properly, then. Not remove the feature :)

The feature shouldn't be removed, but it is a performance killer and should be disabled by default. We disabled it here back in 2012, and a quick search doesn't turn up anyone that has turned it back on.

I love a bit of data. Anyone want to make a command-line flag that enables/disables this feature globally, and then make the changes to capabilities processing to make sure we don't pass this anything with a DriverService if we enable per-session with a capability?

Simon
 
 
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAChzDzmjL4NaMkKLYCf-Js6F7cPR%2Bei8Fxsj3gwaT03hd2ti7A%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages