Captcha service is failing in mainland China due to an apparent Google implementation error

1,866 views

Skip to first unread message

Earl Zmijewski

unread,

Nov 28, 2012, 10:19:43 AM11/28/12

to reca...@googlegroups.com

Summary: Google's captcha service is failing from mainland China clients. As shown in the following analysis, this has nothing to do with the Great Firewall, but rather points to a broken Google implementation of chunked gzipped output, a format Google elects to use only for mainland China clients of the captcha API.

----

Analysis:

If you visit the following page from China using Chrome or IE ...

http://www.renesys.com/about/contact/request_trial.shtml

you will never see the Captcha at the bottom of the page. You will see the sign up form, but the complete page load will hang for a very long time before timing out. As shown by netstat on the client, the browser keeps trying various Google servers to no avail.

The exact call that is hanging is ...

http://api.recaptcha.net/challenge?k=6LeyEgIAAAAAABrYkVNRTaS-yrQe8wdtJ5hvabye

Now if you try to get the above from the command line via wget in China, it works as expected! However, from Chrome, the browser will only show a fragment of the expected output, namely, ...

var RecaptchaState = {
    site : '6LeyEgIAAAAAABrYkVNRTaS-yrQe8wdtJ5hvabye',
    rtl : false,
    challenge : '03AHJ_Vuv6zBUgUn0Lpw_JgFIT26vpRS13XKrqoMi2tVfixH0fA8I2kRmT19GRScPGyHMjthgZAqeJI4cPUMvOQ8H5TdGUf9PW8w20RVT5WUHqAFLvel2ahTfubKZagoprh3c3RVNDY3TYTUupV0bZA7EVyw8d0YSOcKZHuWivEX5ghNAvIO_6cYw',
    is_incorrect

as which point there is no further output. The following output is missing ...

                                      : false,
    programming_error : '',
    error_message : '',
    server : 'http://www.google.com/recaptcha/api/',
    lang : 'zh-CN',
    timeout : 1800
};

document.write('<scr'+'ipt type="text/javascript" s'+'rc="' + RecaptchaState.server + 'js/recaptcha.js"></scr'+'ipt>');

So why does wget work and the browser fail? The difference is in the headers that are sent. Using the developer tools in Chrome (think "Firebug"), we see the following headers used by Chrome. The bold term ("gzip") is causing the problem.

Accept:

text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:

ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:

gzip,deflate,sdch
Accept-Language:

en-US,en;q=0.8
Connection:

keep-alive
Host:

api.recaptcha.net
User-Agent:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11

Using all of the above headers via curl and leaving out only "gzip" under "Accept-Encoding", we get the expected output from China, as shown next.

$ curl -v --header 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3' --header 'Accept-Encoding:deflate,sdch' --header 'Accept-Language:en-US,en;q=0.8' --header 'Connection:keep-alive' http://api.recaptcha.net/challenge?k=6LeyEgIAAAAAABrYkVNRTaS-yrQe8wdtJ5hvabye --user-agent 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'
* About to connect() to api.recaptcha.net port 80
*   Trying 74.125.128.103... connected
* Connected to api.recaptcha.net (74.125.128.103) port 80
> GET /challenge?k=6LeyEgIAAAAAABrYkVNRTaS-yrQe8wdtJ5hvabye HTTP/1.1
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11
> Host: api.recaptcha.net
> Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
> Accept-Encoding:deflate,sdch
> Accept-Language:en-US,en;q=0.8
> Connection:keep-alive
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Tue, 27 Nov 2012 22:25:48 GMT
< Content-Type: text/javascript
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Transfer-Encoding: chunked

var RecaptchaState = {
    site : '6LeyEgIAAAAAABrYkVNRTaS-yrQe8wdtJ5hvabye',
    rtl : false,
    challenge : '03AHJ_VutW_BZ_3gGxxr8LzBN_oTSWQBJAs1wgaYy_gNaYpATLeglk-zrB4VjdFISkmIiacOvV8JGQC9apWd6T_iCiCLNA7CEtdloTfkL0kVgftrTJUkrkYre9rPv2TJ-vJw2VwxtuLPg_UsUqEqqIkTgcwwlpHDHWDOhos-1nyOgVG2eiVbKXmfwluK03xGj5Qqdfz-hUkLN5',
    is_incorrect : false,
    programming_error : '',
    error_message : '',
    server : 'http://www.google.com/recaptcha/api/',
    lang : 'zh-CN',
    timeout : 1800
};

document.write('<scr'+'ipt type="text/javascript" s'+'rc="' + RecaptchaState.server + 'js/recaptcha.js"></scr'+'ipt>');
* Connection #0 to host api.recaptcha.net left intact
* Closing connection #0

Notice that gzip is missing on the "Accept-Encoding" line and that the server reports it is transferring the data in "chunked" format. Adding "gzip" to "Accept-Encoding" in the above curl command results in the download hanging from four different mainland Chinese hosts. These hosts hang in a similar fashion to that shown above via a browser. However, the same command works from locations in the US, Europe, Pakistan, Saudi Arabia, Ukraine, Azerbaijan, Hong Kong, Vietnam and many others. This would seem to imply interference from the GFW, but that is definitely not the case here. Rather, it is a result of how Google is handling requests from mainland China. Let's consider the headers returned by Google from outside and inside China for this curl command.

Outside China

< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Tue, 27 Nov 2012 23:04:02 GMT
< Content-Type: text/javascript
< Content-Encoding: gzip
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Content-Length: 452
< Server: GSE

Notice that the data is in gzip format and its exact length is provided.

Inside China

< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Tue, 27 Nov 2012 23:44:03 GMT
< Content-Type: text/javascript
< Content-Encoding: gzip
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Transfer-Encoding: chunked

For clients inside China, the exact length is not provided, rather the server elects to "chunk" the data. Let's look at tcpdump output from China during a failed attempt. Here, we have obscured the Chinese IP as "CN-IP".

We see a normal TCP handshake next ...

07:39:06.308164 IP CN-IP.48648 > 74.125.131.105.http: S 3996998925:3996998925(0) win 5840 <mss 1460,sackOK,timestamp 557488045 0,nop,wscale 6>
07:39:06.612729 IP 74.125.131.105.http > CN-IP.48648: S 1597132228:1597132228(0) ack 3996998926 win 14180 <mss 1430,sackOK,timestamp 1171604904 557488045,nop,wscale 6>
07:39:06.612763 IP CN-IP.48648 > 74.125.131.105.http: . ack 1 win 92 <nop,nop,timestamp 557488349 1171604904>

The Chinese client makes the 426 byte request and Google responds with 780 bytes ...

07:39:06.613067 IP CN-IP.48648 > 74.125.131.105.http: P 1:427(426) ack 1 win 92 <nop,nop,timestamp 557488349 1171604904>
07:39:06.918345 IP 74.125.131.105.http > CN-IP.48648: . ack 427 win 239 <nop,nop,timestamp 1171605208 557488349>
07:39:06.979446 IP 74.125.131.105.http > CN-IP.48648: P 1:781(780) ack 427 win 239 <nop,nop,timestamp 1171605228 557488349>
07:39:06.979461 IP CN-IP.48648 > 74.125.131.105.http: . ack 781 win 116 <nop,nop,timestamp 557488716 1171605228>

The Chinese client closes down the session after a specified 2 second timeout on the curl command line ...

07:39:08.980194 IP CN-IP.48648 > 74.125.131.105.http: F 427:427(0) ack 781 win 116 <nop,nop,timestamp 557490717 1171605228>
07:39:09.284628 IP 74.125.131.105.http > CN-IP.48648: F 781:781(0) ack 428 win 239 <nop,nop,timestamp 1171607576 557490717>
07:39:09.284664 IP CN-IP.48648 > 74.125.131.105.http: . ack 782 win 116 <nop,nop,timestamp 557491021 1171607576>

Note also that we are using the Google server at 74.125.131.105, which we have forced by editing /etc/hosts on the Chinese client. This was done to rule out the Hong Kong Google servers as a potential problem. The last few hops of a traceroute from the Chinese client to this IP address are as follows ...

12 ae-1-60.edge2.SanJose3.Level3.net (4.69.152.17) 320.719 ms
13 GOOGLE-INC.edge2.SanJose3.Level3.net (4.79.40.154) 301.777 ms
14 216.239.49.170 (216.239.49.170) 383.494 ms
15 209.85.250.66 (209.85.250.66) 384.793 ms
16 64.233.174.204 (64.233.174.204) 312.583 ms
17 72.14.239.80 (72.14.239.80) 360.023 ms
18 209.85.249.238 (209.85.249.238) 380.194 ms
19 64.233.174.87 (64.233.174.87) 382.098 ms
20 api.recaptcha.net (74.125.131.105) 385.422 ms

Performing the same call from the US to the same Google IP, producing the following tcpdump snippet.

13:12:37.745584 IP 74.125.131.105.http > US-IP.54720: Flags [P.], seq 1:333, ack 427, win 239, options [nop,nop,TS val 1045722684 ecr 2562330418], length 332
13:12:37.745589 IP US-IP.com.54720 > 74.125.131.105.http: Flags [.], ack 333, win 31, options [nop,nop,TS val 2562330493 ecr 1045722684], length 0
13:12:37.745685 IP 74.125.131.105.http > US-IP.54720: Flags [P.], seq 333:783, ack 427, win 239, options [nop,nop,TS val 1045722684 ecr 2562330418], length 450
13:12:37.745688 IP UP-IP.54720 > 74.125.131.105.http: Flags [.], ack 783, win 33, options [nop,nop,TS val 2562330493 ecr 1045722684], length 0

In this successful call, the chunking option is not used by the server and Google sends the gzip output in two packets, whose total length is two bytes more than what we saw in China. Note also that chunking alone is not the blame here, but rather chunking in conjunction with gzip output. Chunking does work in China for non-gzip output.

We tried to force Google to not use chunking via the -0 option in curl, stating the client only understands HTTP 1.0. The HTTP 1.1 protocol states that the server should not set the chunking option for HTTP 1.0 clients, however, Google still insists on using chunking in this case.

We conclude with a summary of our findings.

The GFW is not implicated in this issue. Typical GFW tricks such as DNS poisoning, TCP resets and hard IP blocks are not in evidence here. The Chinese clients have no problems communicating with Google's captcha servers.
Google is electing to send captcha output in chunked gzip format for mainland Chinese clients only. Clients in all other locations receive gzip output whose length is precomputed.
This is not a function of chunked data alone. Chunked non-gzip data successfully transfers to mainland Chinese clients.
This is not a function of the Google servers that are used. Servers in Hong Kong and California display the same problem when accessed from mainland Chinese clients.
This is not a function of the browser used, IE and Chrome behave similarly.
This is not under direct control of the user. If the browsers indicate they can handle gzip format, Google will try to provide that and fail for mainland Chinese users.
2 fewer bytes are sent to Chinese users when compared to US users.

We conclude that Google's handling of chunked data for gzip encoding captcha data has an implementation error. From this link ...

http://en.wikipedia.org/wiki/Chunked_transfer_encoding

We have ...

"The size of each chunk is sent right before the chunk itself so that the receiver can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero."