Rare and intermittent RemoteWebDriver error "Unable to determine type from: <. Last 1 characters read: <"

[jc]

I am not sure if this is a Geb issue but hoping maybe the smart people here could maybe help understand why I occasionally see this error.  We have a test suite of about 650 test cases that run 4 times overnight for different resolutions.  So that is about 2200 tests per night.  I will have occasional failures, usually 1 or 2, with this exception:

Unable to determine type from: <. Last 1 characters read: <

Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'

System info: host: 'redacted.corporate.redacted.com', ip: '192.168.1.15', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '11.0.4'

Driver info: driver.version: RemoteWebDriver

org.openqa.selenium.json.JsonException: Unable to determine type from: <. Last 1 characters read: <

Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'

System info: host: 'redacted.corporate.redacted.com', ip: '192.168.1.15', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '11.0.4'

Driver info: driver.version: RemoteWebDriver

at org.openqa.selenium.json.JsonInput.peek(JsonInput.java:122)

at org.openqa.selenium.json.JsonTypeCoercer.lambda$null$6(JsonTypeCoercer.java:140)

at org.openqa.selenium.json.JsonTypeCoercer.coerce(JsonTypeCoercer.java:126)

at org.openqa.selenium.json.Json.toType(Json.java:69)

at org.openqa.selenium.json.Json.toType(Json.java:55)

at org.openqa.selenium.json.Json.toType(Json.java:50)

at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:87)

at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)

at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)

at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)

at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:277)

at geb.Browser.go(Browser.groovy:533)

at geb.Browser.go(Browser.groovy:510)

at geb.spock.GebSpec.methodMissing(GebSpec.groovy:60)

at geb.BaseTest.cleanup(BaseTest.groovy:71)

In my last case I had this happen twice. The first when trying to click an element.  IntelliJ said this is the offending line of code:

pageHeader {$("div h3")[0]}

which is the first item inside a static content block on a page.

The second one the offending line was the end bracket "}" of an if statement.

Versions of everything being used:

Grid: 3.141.59

org.seleniumhq.selenium:selenium-remote-driver:3.141.59
org.seleniumhq.selenium:selenium-server:3.141.59

org.codehaus.groovy:groovy-all:2.5.8

org.spockframework:spock-core:1.3-groovy-2.5

org.gebish:geb-spock:3.4

The remote driver is not being invoked in any obscure way:

remoteDriver = new RemoteWebDriver(URI.create(hubUrl).toURL(), capabilities)

When this happens it's usually in the middle of a test class where several tests before it have passed with no issues using the very same code.  

Things I know:

1. It only happens when running on the grid aka with RemoteWebDriver.  I have never seen it happen locally.
2. It happens once about every 2200 tests.
3. It's never the same test, same method, or same action.
4. It happens on both Chrome and Firefox (unsure about other browsers).
5. The message is always "Unable to determine type from: <. Last 1 characters read: <"

It almost seems as if there is a sync issue somewhere Geb or WebDriver or something..  I am unsure how 1500+ test cases can pass and then all of the sudden encounter this error.  

I would gladly provide any help I can.  Unfortunately I cannot hand out this code for anyone to test and I don't even know what kind of example project I could throw together to even attempt to try to reproduce this.  I am hoping the stack trace will help someone identify something.

[jc]

Here is the stack trace of the one that failed on the locator

Unable to determine type from: <. Last 1 characters read: <

Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'

System info: host: 'redacted.corporate.redacted.com', ip: '192.168.1.15', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '11.0.4'

Driver info: driver.version: RemoteWebDriver

org.openqa.selenium.json.JsonException: Unable to determine type from: <. Last 1 characters read: <

Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'

System info: host: 'redacted.corporate.redacted.com', ip: '192.168.1.15', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '11.0.4'

Driver info: driver.version: RemoteWebDriver

at org.openqa.selenium.json.JsonInput.peek(JsonInput.java:122)

at org.openqa.selenium.json.JsonTypeCoercer.lambda$null$6(JsonTypeCoercer.java:140)

at org.openqa.selenium.json.JsonTypeCoercer.coerce(JsonTypeCoercer.java:126)

at org.openqa.selenium.json.Json.toType(Json.java:69)

at org.openqa.selenium.json.Json.toType(Json.java:55)

at org.openqa.selenium.json.Json.toType(Json.java:50)

at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:87)

at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)

at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)

at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)

at org.openqa.selenium.remote.RemoteWebDriver.findElements(RemoteWebDriver.java:353)

at org.openqa.selenium.remote.RemoteWebDriver.findElementsByCssSelector(RemoteWebDriver.java:424)

at org.openqa.selenium.By$ByCssSelector.findElements(By.java:441)

at org.openqa.selenium.remote.RemoteWebDriver.findElements(RemoteWebDriver.java:311)

at geb.navigator.SearchContextBasedBasicLocator.elementsSupplier_closure8$_closure12(SearchContextBasedBasicLocator.groovy:199)

at groovy.lang.Closure.call(Closure.java:405)

at groovy.lang.Closure.call(Closure.java:421)

at geb.navigator.SearchContextBasedBasicLocator.elementsSupplier_closure8(SearchContextBasedBasicLocator.groovy:199)

at groovy.lang.Closure.call(Closure.java:405)

at geb.navigator.SearchContextBasedBasicLocator.toNavigator(SearchContextBasedBasicLocator.groovy:223)

at geb.navigator.SearchContextBasedBasicLocator.find(SearchContextBasedBasicLocator.groovy:186)

at geb.navigator.SearchContextBasedBasicLocator.find(SearchContextBasedBasicLocator.groovy:55)

at geb.navigator.DefaultLocator.find(DefaultLocator.groovy:22)

at geb.navigator.DefaultLocator.find(DefaultLocator.groovy:31)

at geb.content.NavigableSupport.$(NavigableSupport.groovy:65)

at geb.content.PageContentTemplateFactoryDelegate.$(PageContentTemplateFactoryDelegate.groovy:27)

at geb.pages.EnrollmentStartPage._clinit__closure6$_closure8(EnrollmentStartPage.groovy:16)

at geb.pages.EnrollmentStartPage._clinit__closure6$_closure8(EnrollmentStartPage.groovy)

at geb.content.PageContentTemplate.invokeFactory(PageContentTemplate.groovy:102)

at geb.content.PageContentTemplate.create_closure1(PageContentTemplate.groovy:59)

at geb.content.PageContentTemplate.create_closure1(PageContentTemplate.groovy)

at geb.content.PageContentTemplate.create(PageContentTemplate.groovy:87)

at geb.content.PageContentTemplate.get(PageContentTemplate.groovy:54)

at geb.content.DefaultPageContentSupport.getContent(DefaultPageContentSupport.groovy:42)

at geb.content.PageContentSupport.propertyMissing(PageContentSupport.groovy:41)

at geb.Page.propertyMissing(Page.groovy:110)

at geb.pages.EnrollmentStartPage.enterCoverageEffectiveDate(EnrollmentStartPage.groovy:107)

at geb.tests.ApplicantInformationPageTest.Applicant Information: information is saved when proceeding and coming back to page(ApplicantInformationPageTest.groovy:707)

[tho...@posteo.de]

Hello jc,

the stacktrace seems to indicate that there is HTML being returned where
JSON is expected.
The "<" character is most commonly found as the first character in any
HTML document.
The "type" that is mentioned in the message is one of "JsonArray" or
"JsonObject", starting with "[" and "{" respectively.
It would be helpful to see the full response, and not only the first
character, but that might not be easy to do, as the error itself is
probably
happening on a low level in the communication between the components of
the Grid.
To further investigate this, you might want to look at logfiles written
by Grid components, but right now I don't know where or if there are
any.

I am guessing that there is some kind of resource leak inside the Grid,
which leads to some component responding with a plain HTML error page,
instead of the expected JSON document.

As a first attempt to circumvent this problem I would suggest to restart
the whole Grid between the test runs for different resolutions.

Thomas

>> System info: host: 'redacted.corporate.redacted.com [1]', ip:
>> '192.168.1.15', os.name [2]: 'Mac OS X', os.arch: 'x86_64',

>> os.version: '10.14.6', java.version: '11.0.4'
>> Driver info: driver.version: RemoteWebDriver
>> org.openqa.selenium.json.JsonException: Unable to determine type
>> from: <. Last 1 characters read: <
>> Build info: version: '3.141.59', revision: 'e82be7d358', time:
>> '2018-11-14T08:17:03'

>> System info: host: 'redacted.corporate.redacted.com [1]', ip:
>> '192.168.1.15', os.name [2]: 'Mac OS X', os.arch: 'x86_64',

>> * It only happens when running on the grid aka with

>> RemoteWebDriver. I have never seen it happen locally.

>> * It happens once about every 2200 tests.
>> * It's never the same test, same method, or same action.
>> * It happens on both Chrome and Firefox (unsure about other
>> browsers).
>> * The message is always "Unable to determine type from: <. Last 1

>> characters read: <"
>>
>> It almost seems as if there is a sync issue somewhere Geb or
>> WebDriver or something.. I am unsure how 1500+ test cases can pass
>> and then all of the sudden encounter this error.
>>
>> I would gladly provide any help I can. Unfortunately I cannot hand
>> out this code for anyone to test and I don't even know what kind of
>> example project I could throw together to even attempt to try to
>> reproduce this. I am hoping the stack trace will help someone
>> identify something.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Geb User Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to geb-user+u...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/geb-user/147d363a-18ee-4258-8c03-69f6f7799d49n%40googlegroups.com
> [3].
>
>
> Links:
> ------
> [1] http://redacted.corporate.redacted.com
> [2] http://os.name
> [3]
> https://groups.google.com/d/msgid/geb-user/147d363a-18ee-4258-8c03-69f6f7799d49n%40googlegroups.com?utm_medium=email&utm_source=footer

[jc]

Hi Thomas, thanks for the reply.  When you say "which leads to some component responding with a plain HTML error page instead of the expected JSON documen"" are you talking about the communication between the hub and node or the browser itself?  Unfortunately Geb does not take a screenshot on this kind of failure so I am never able to see what the browser is showing.  I did put together an event listener with the help of stack overflow to execute something if a test fails, so I'll see if I can print the page source or something.

The nodes get restarted every morning.  I am not sure if restarting after each test run will be feasible since this grid is shared by another team.  Is there away to tell nodes to restart after some activity or something?

[tho...@posteo.de]

I have my doubts, that it has anything to do with what is happening in
the browser or in the test, but I am really speculating.

Is there a way to set up your own Grid? I can highly recommend Zalenium
https://opensource.zalando.com/zalenium/ which is an out-of-the-box Grid
implementation, that can be started with a single (docker) command. Only
thing you need basically, is a strong enough machine to run everything,
hub and browsers. It also alleviates the problem of taking screenshots,
because Zalenium also records videos of all tests, which from my
experience are much more useful than screenshots.

> org.openqa.selenium.remote.RemoteWebDriver.findElements(RemoteWebDriver.java:353)
>>
>>> at
>>>
>>
> org.openqa.selenium.remote.RemoteWebDriver.findElementsByCssSelector(RemoteWebDriver.java:424)
>>
>>> at org.openqa.selenium.By

>> [3]$ByCssSelector.findElements(By.java:441)

>>>> System info: host: 'redacted.corporate.redacted.com [1] [1]', ip:
>>
>>>> '192.168.1.15', os.name [2] [2]: 'Mac OS X', os.arch: 'x86_64',

>>>> os.version: '10.14.6', java.version: '11.0.4'
>>>> Driver info: driver.version: RemoteWebDriver
>>>> org.openqa.selenium.json.JsonException: Unable to determine type
>>>> from: <. Last 1 characters read: <
>>>> Build info: version: '3.141.59', revision: 'e82be7d358', time:
>>>> '2018-11-14T08:17:03'

>>>> System info: host: 'redacted.corporate.redacted.com [1] [1]', ip:
>>
>>>> '192.168.1.15', os.name [2] [2]: 'Mac OS X', os.arch: 'x86_64',

>> [4]
>>> [3].
>>>
>>>
>>> Links:
>>> ------
>>> [1] http://redacted.corporate.redacted.com [1]
>>> [2] http://os.name [2]
>>> [3]
>>>
>>
> https://groups.google.com/d/msgid/geb-user/147d363a-18ee-4258-8c03-69f6f7799d49n%40googlegroups.com?utm_medium=email&utm_source=footer
>> [5]

>
> --
> You received this message because you are subscribed to the Google
> Groups "Geb User Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to geb-user+u...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/geb-user/5c761a54-9cdb-4c1e-8882-01c436793bd8n%40googlegroups.com
> [6].

>
>
> Links:
> ------
> [1] http://redacted.corporate.redacted.com
> [2] http://os.name

> [3] http://org.openqa.selenium.By
> [4]
> https://groups.google.com/d/msgid/geb-user/147d363a-18ee-4258-8c03-69f6f7799d49n%40googlegroups.com
> [5]
> https://groups.google.com/d/msgid/geb-user/147d363a-18ee-4258-8c03-69f6f7799d49n%40googlegroups.com?utm_medium=email&amp;utm_source=footer
> [6]
> https://groups.google.com/d/msgid/geb-user/5c761a54-9cdb-4c1e-8882-01c436793bd8n%40googlegroups.com?utm_medium=email&utm_source=footer

[jc]

I actually did try Zalenium and encountered this issue on their GitHub and it looks like they are no longer making any improvements or fixes, so I stopped.  I then checked out Selenoid which I believe essentially does the same thing, but I encounter this issue enough on both Chrome and Firefox to not move forward with it as well.