GoogleBot is using the wrong permutation?

[Ed]

Has anybody experiences the following situation?

GoogleBot is using a permutation that doesn't exists (anymore)? :(

Details:

Googlebot (GB) comes by to crawl the site (build with GWT 2.7.0: leuker.nl) and generates some exception on some page, namely leuker.nl/zorgtoeslag/acties), this only occurs when GB accesses the site. The exception looks like:

---

Cannot call method 'xg' of null

        at Unknown.NOb(https://leuker.nl/aangifte/acties) ~[na:na]

        at Unknown.Mrd(https://leuker.nl/aangifte/acties) ~[na:na] 

        .....

----

The deObfuscator in the backend complaints that it's receiving a permutation from the frontend, that doesn't exists, and checking it manually, it really doesn't exists... It had that already a few times when deploying new versions.

So:

1) What permutation is GB using ? I could it be a cashed one (old one) ?

2) I can't solve this issue without deObfuscating (I don't want to put a pretty-version online, it's too big).

Any idea ?

- Ed 

[Vassilis Virvilis]

Hi Ed,

Looks like the googlebot user agent is not handled by GWT permutations. I have collapsed all permutations and I am still getting 404 in my web server logs where the google bot is trying to get the undefined.cache.js

An interim solution / workaround until the GWT crew can fix this is mentioned here in the previous conversation we had at https://groups.google.com/forum/#!msg/google-web-toolkit/tmUeYATU2kA/6F8OjCH1CgAJ

The fix was to

$sudo ln -s /var/lib/tomcat7/webapps/myapp/myapp/D44347868....cache.js /var/lib/tomcat7/webapps/myapp/myappundefined.nocache.js

after every deployment.

Here is the script version
sudo ln -s `ls /var/lib/tomcat7/webapps/myapp/myapp/*.cache.js | head -n 1` /var/lib/tomcat7/webapps/myapp/myapp/undefined.nocache.js

after every deployment.

I was already doing

$sudo touch /var/lib/tomcat7/webapps/myapp/myapp/myapp.nocache.js

after every deployment, so it's one more line that shouldn't exist in the first place in the deployment script.

--
You received this message because you are subscribed to the Google Groups "GWT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-tool...@googlegroups.com.
To post to this group, send email to google-we...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-web-toolkit.
For more options, visit https://groups.google.com/d/optout.

--

Vassilis Virvilis

[Jens]

You can use <set-property-fallback> to define a fallback value in case the user.agent is not recognized. That way you can serve a default permutation for unknown user.agents. By default the fallback value is "unknown" and thats why GWT tries to load a permutation file called "unknown.cache.js". So it should be fine if you use webkit or gecko1_8 as fallback value.

-- J.

[Vassilis Virvilis]

Good to know.

thanks for the tip

On Tue, Dec 1, 2015 at 12:05 PM, Jens <jens.ne...@gmail.com> wrote:

You can use <set-property-fallback> to define a fallback value in case the user.agent is not recognized. That way you can serve a default permutation for unknown user.agents. By default the fallback value is "unknown" and thats why GWT tries to load a permutation file called "unknown.cache.js". So it should be fine if you use webkit or gecko1_8 as fallback value.

-- J.

--

You received this message because you are subscribed to the Google Groups "GWT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-tool...@googlegroups.com.
To post to this group, send email to google-we...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-web-toolkit.
For more options, visit https://groups.google.com/d/optout.

--

Vassilis Virvilis

[Ed]

@Jens: I am already using the fallback user agent. That is not the problem (I had that before). It occurs after that during crawling the website.

@ Vassilis: Thanks, I am aware of the forum discussion your refer to, I was the one that started it ;)... I am not getting any 4XX errors, all files can be found.

Any more ideas?

- Ed

[Ed Bras]

What I did that probably solved it:​ instead of reading the permutation name through the HTTP header using RpcRequestBuilder.STRONG_NAME_HEADER, I now send the permutation name along from with the exception occurred in the client.

At least I am not using any old cached http header.

Let's see if that works, I think it does as I didn't see the exception just yet ;)

[Ed]

Ok, Still googlebot comes a long and after I deployed a new version I always get google bot exceptions.

I checked the permutation, and that often seem to be ok (not always).

That is: the one I receive as http header is the same as the one that I send a long to the backend when an exception occurs.

The error I get:

---

(TypeError) : Cannot call method 'Kg' of null 

at Unknown.vSb(https://www.leuker.nl/app/aangifte/start/)

---

Any idea ?

I think it might come with the JS engine that googlebot uses. For example: I don't support IE9, as I use JS features that aren't supported by IE9 that result in similar errors. 

But how to work around this? It's hard to see the exact error that occurs.

- Ed

[Rogelio Flores]

If you compile your app with the "PRETTY" compiler flag and then deploy it and run the same test, you might know exactly what "Kg" is (what its real name is on your Java code) which hopefully will let you know exactly what is failing and why.

[Ed Bras]

Ofcourse that is possible, but that is not an option in production.

--
You received this message because you are subscribed to a topic in the Google Groups "GWT Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-web-toolkit/ixo_9kd-joU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-web-tool...@googlegroups.com.

To post to this group, send email to google-we...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-web-toolkit.

[Vassilis Virvilis]

Are you using fetch and render from google webmaster tools while to trigger it while you are in development mode?

You received this message because you are subscribed to the Google Groups "GWT Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-tool...@googlegroups.com.

[Thomas Broyer]

Have you tried "deobfuscating" those values (Kg and vSb) using the symbolMaps? That could possibly help understand what happens. 

[Michael Joyner]

You need to have a "default" permutation for Google Bot:

We use the following here:

<!-- Fallback User Agent for use by CRAWLERS -->
     <set-property-fallback name="user.agent" value="gecko1_8" />

--

You received this message because you are subscribed to the Google Groups "GWT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-tool...@googlegroups.com.
To post to this group, send email to google-we...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-web-toolkit.

[Ed Bras]

Thanks, but I already have that. If I wouldn't have that, it would result in another exception like I had in the past. Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "GWT Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-web-toolkit/ixo_9kd-joU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-web-tool...@googlegroups.com.