Re: DECODE_BODY_ERROR For SDCH encoded page

[openvcdiff]

Hi Nikhil:

Sorry for the delay in approving your message to the group.

Are you sure that the SDCH-encoded response contains the dictionary-server-id 

in the proper format before the VCDIFF-format payload, as required by the SDCH

specification?

https://docs.google.com/document/d/1REMkwjXY5yFOkJwtJPjCMwZ4Shx3D9vfdAytV_KQCUo/preview#heading=h.8b88417df464

Cheers,

Lincoln

On Wednesday, July 22, 2015 at 10:52:00 AM UTC-7, nikhil gupta wrote:

Hi,

We in amazon are trying to implement SDCH encoding for some of our pages. But as part of implementation we are facing certain issues which are becoming difficult to debug. It would be very helpful, if someone can give their inputs on issues we are facing.

 

We are able to advertise our generated dictionaries as part of request header which we verified by checking in chrome://net-internal tools, but when we send encoded content to browser , it fails saying "DECODE_BODY_ERROR" for that request. When we tried to debug we found this instance here in code(http://osxr.org/android/source/external/chromium/net/base/sdch_filter.cc#0301) which basically comes when decodeChunk method return fails. This method return fails when readDataFileHeader(http://osxr.org/android/source/external/chromium/sdch/open-vcdiff/src/vcdecoder.cc#0875) returns false which in return false when the first four headers are not "VCD"(http://osxr.org/android/source/external/chromium/sdch/open-vcdiff/src/vcdecoder.cc#0677). 

But as part of encoded response , we are already passing these bytes. So its getting difficult to debug this issue.

Log file content:

6099: URL_REQUEST

http://nikgupta.desktop.amazon.com:11080/gp/product/B0018OOVPC?isDebug=0

Start Time: 2015-07-15 20:18:33.393

t=751751 [st=   0] +REQUEST_ALIVE  [dt=1824]
t=751752 [st=   1]   +URL_REQUEST_DELEGATE  [dt=19]
t=751752 [st=   1]      DELEGATE_INFO  [dt=19]
                        --> delegate_info = "extension AMIE Desktop"
t=751771 [st=  20]   -URL_REQUEST_DELEGATE
t=751771 [st=  20]    URL_REQUEST_START_JOB  [dt=10]
                      --> load_flags = 2704386 (BYPASS_CACHE | BYPASS_DATA_REDUCTION_PROXY | MAIN_FRAME | MAYBE_USER_GESTURE | REPORT_RAW_HEADERS | VERIFY_EV_CERT)
                      --> method = "GET"
                      --> priority = "HIGHEST"
                      --> url = "http://nikgupta.desktop.amazon.com:11080/gp/product/B0018OOVPC?isDebug=0"
t=751781 [st=  30]   +URL_REQUEST_START_JOB  [dt=1793]
                      --> load_flags = 2704386 (BYPASS_CACHE | BYPASS_DATA_REDUCTION_PROXY | MAIN_FRAME | MAYBE_USER_GESTURE | REPORT_RAW_HEADERS | VERIFY_EV_CERT)
                      --> method = "GET"
                      --> priority = "HIGHEST"
                      --> url = "http://nikgupta.desktop.amazon.com:11080/gp/product/B0018OOVPC?isDebug=0"
t=751782 [st=  31]     +URL_REQUEST_DELEGATE  [dt=4]
t=751782 [st=  31]        DELEGATE_INFO  [dt=3]
                          --> delegate_info = "extension User-Agent Switcher for Chrome"
t=751785 [st=  34]        DELEGATE_INFO  [dt=0]
                          --> delegate_info = "extension AMIE Desktop"
t=751785 [st=  34]        DELEGATE_INFO  [dt=1]
                          --> delegate_info = "extension AMIE Desktop"
t=751786 [st=  35]        CHROME_EXTENSION_MODIFIED_HEADERS
                          --> deleted_headers = []
                          --> extension_id = "idgpnmonknjnojddfkpgkljpfnnfcklj"
                          --> modified_headers = ["Accept-Encoding: gzip;q=0,deflate,sdch"]
t=751786 [st=  35]     -URL_REQUEST_DELEGATE
t=751786 [st=  35]      HTTP_CACHE_GET_BACKEND  [dt=0]
t=751786 [st=  35]      HTTP_CACHE_DOOM_ENTRY  [dt=0]
                        --> net_error = -2 (ERR_FAILED)
t=751786 [st=  35]      HTTP_CACHE_CREATE_ENTRY  [dt=0]
t=751786 [st=  35]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]
t=751786 [st=  35]      URL_REQUEST_DELEGATE  [dt=0]
t=751786 [st=  35]     +HTTP_STREAM_REQUEST  [dt=1]
t=751787 [st=  36]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                          --> source_dependency = 6178 (HTTP_STREAM_JOB)
t=751787 [st=  36]     -HTTP_STREAM_REQUEST
t=751787 [st=  36]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
t=751787 [st=  36]        HTTP_TRANSACTION_SEND_REQUEST_HEADERS
                          --> GET /gp/product/B0018OOVPC?isDebug=0 HTTP/1.1
                              Host: nikgupta.desktop.amazon.com:11080
                              Connection: keep-alive
                              Pragma: no-cache
                              Cache-Control: no-cache
                              Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
                              User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36
                              Accept-Encoding: gzip;q=0,deflate,sdch
                              Avail-Dictionary: tOamjyML
                              Accept-Language: en-US,en;q=0.8
                              Cookie: [1131 bytes were stripped]
t=751787 [st=  36]     -HTTP_TRANSACTION_SEND_REQUEST
t=751787 [st=  36]     +HTTP_TRANSACTION_READ_HEADERS  [dt=1785]
t=751787 [st=  36]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=1785]
t=753572 [st=1821]        HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                          --> HTTP/1.1 200 OK
                              Date: Wed, 15 Jul 2015 14:48:33 GMT
                              Server: Server
                              pragma: no-cache
                              x-amz-id-1: 1Z45WRSEAZDGR1NFZN66
                              p3p: policyref="http://www.amazon.com/w3c/p3p.xml",CP="CAO DSP LAW CUR ADM IVAo IVDo CONo OTPo OUR DELi PUBi OTRi BUS PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA HEA PRE LOC GOV OTC "
                              x-sap-pg: apparel_display_on_website
                              x-amazon-internal-requestcost: 981892
                              cache-control: no-cache
                              x-frame-options: SAMEORIGIN
                              expires: -1
                              x-amazon-internal-page-type: Detail
                              x-amz-id-2: 5z6bDOz1dJgppp5n1m4sZzaOeeXrEd6YOpM/ATm6y9wY2jrHXyUoVg==
                              x-ua-compatible: IE=edge
                              Set-cookie: [96 bytes were stripped]
                              Set-cookie: [94 bytes were stripped]
                              Set-cookie: [97 bytes were stripped]
                              Content-Encoding: sdch
                              Vary: Accept-Encoding,User-Agent
                              Keep-Alive: timeout=60, max=100
                              Connection: Keep-Alive
                              Transfer-Encoding: chunked
                              Content-Type: text/html; charset=UTF-8
t=753572 [st=1821]     -HTTP_TRANSACTION_READ_HEADERS
t=753572 [st=1821]      HTTP_CACHE_WRITE_INFO  [dt=1]
t=753573 [st=1822]      HTTP_CACHE_WRITE_DATA  [dt=0]
t=753573 [st=1822]      HTTP_CACHE_WRITE_INFO  [dt=0]
t=753573 [st=1822]      URL_REQUEST_DELEGATE  [dt=0]
t=753574 [st=1823]      SDCH_DECODING_ERROR
                        --> net_error = -2 (ERR_FAILED)
                        --> sdch_problem_code = 6 (OPTIONAL_GUNZIP_ENCODING_ADDED)
t=753574 [st=1823]      URL_REQUEST_FILTERS_SET
                        --> filters = "FILTER_TYPE_GZIP_HELPING_SDCH,FILTER_TYPE_SDCH"
t=753574 [st=1823]   -URL_REQUEST_START_JOB
t=753574 [st=1823]    URL_REQUEST_DELEGATE  [dt=1]
t=753575 [st=1824]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=753575 [st=1824]    HTTP_CACHE_WRITE_DATA  [dt=0]
t=753575 [st=1824]    SDCH_DECODING_ERROR
                      --> net_error = -2 (ERR_FAILED)
                      --> sdch_problem_code = 5 (DECODE_BODY_ERROR)
t=753575 [st=1824]    FAILED
                      --> net_error = -330 (ERR_CONTENT_DECODING_FAILED)
t=753575 [st=1824] -REQUEST_ALIVE
                    --> net_error = -330 (ERR_CONTENT_DECODING_FAILED)

Thanks

Nikhil

[Can Selcik]

Hi Nikhil,

Here is a hexdump of the beginning section of one of our testing dictionaries at LinkedIn. Notice the "0a 0a" (\n\n) right after the domain field. The problem might be that it is missing.

00000000  64 6f 6d 61 69 6e 3a 20  6c 69 6e 6b 65 64 69 6e  |domain: linkedin|
00000010  6c 64 33 2e 6c 69 6e 6b  65 64 69 6e 2e 62 69 7a  |ld3.linkedin.biz|
00000020  0a 0a 2f 2f 20 3d 3d 43  6c 6f 73 75 72 65 43 6f  |..// ==ClosureCo|
00000030  6d 70 69 6c 65 72 3d 3d  0a 2f 2f 20 40 63 6f 6d  |mpiler==.// @com|

Also, could you tell me where you are seeing the log output you've posted? I'm aware of the chrome://net-internals/#sdch page but a log output like the one you've shown would be immensely useful.

Let me know if you have any other questions or if you want to bounce around some ideas. At LinkedIn, we basically done with the server-side implementation (Apache Traffic Server plugin, which will be open-sourced) and currently working on refining our dictionary generation pipeline.

Thanks,

Can

[nikhil gupta]

Hi Can,

Thanks for the idea and yes we were able to resolve that issue and problem was that we were missing some of the headers required on top of the page.

You can use chrome://net-internals/#events for the kind of logs which I have posted. It summarizes all the events happening onto the page and give details about.

Even we are done with the changes and are working upon building dictionary generation thing.

Thanks

Nikhil

--

Nikhil Gupta
Website: http://web.iiit.ac.in/~nikhil.guptaug08/

[Christopher O'Connell]

Events is super useful to see SDCH requests. In addition, if you have one of the SDCH filters applied, it will show you successful requests in green and failed requests in an orange/yellow color, making it easy to see how things go at a glance.

--
You received this message because you are subscribed to the Google Groups "SDCH" group.
To unsubscribe from this group and stop receiving emails from it, send an email to SDCH+uns...@googlegroups.com.
To post to this group, send email to SD...@googlegroups.com.
Visit this group at https://groups.google.com/group/SDCH.
For more options, visit https://groups.google.com/d/optout.

[Christopher O'Connell]

Nikhil,

Are you guys working on something "better" than just femtozip for dictionary generation? We'd love to have something we could spin up a few EC2 instances and run to generate big dictionaries in a parallel manner. Also, if you're out of the Seattle office, want to meet up with us for drinks some time? It's cool to see a bit more interest in SDCH after it being very quiet for a while.

All the best,

~ Christopher

On Wed, Dec 23, 2015 at 4:04 AM, nikhil gupta <nikhilg...@gmail.com> wrote:

--

[ernest.w....@gmail.com]

>>  It's cool to see a bit more interest in SDCH after it being very quiet for a while.

When I ran into SDCH and started to dig on the internet about it, I got an impression that SDCH was POCed by Google for fun and then abandoned, I saw little to none interest of this technology, looks like Google and LinkedIn were the only two big companies to implement it. Does anybody know what is the current status of SDCH?

[Jim Roskind]

Not sure about "current" status... but I think current Chromium builds support it ... and this thread:

https://groups.google.com/a/chromium.org/forum/#!topic/net-dev/qqloNBJUqHM

may provide pointers.

YMMV,

Jim

[Christopher O'Connell]

I certainly cannot comment on Google's intentions, but we find SDCH very useful on chrome browsers, which between desktop, Android and iOS account for a significant percentage of Web users.

We've made the decision that it's worth it to support.

All the best,

~ Christopher

[ernest.w....@gmail.com]

Hi Jim,

Actually, I spent this day POCing SDCH encoding in our server, works as expected in Chrome. But... Why I dont see dictionaries being advertised by google and linkedin? these two implemented it, and looks like they (at this moment) do not use it. It is somewhat suspicious... did they find something that caused them to drop SDCH support?

[Christopher O'Connell]

Google homepage uses it. I haven't seen the time used on linkedin, although it maybe only on pages I don't visit.

All the best,

~ Christopher 

[ernest.w....@gmail.com]

dont see any SDCH related headers nor dictionaries being downloaded/advertised/used/whatever on google homepage. checked with fiddler and chrome net internals... odd stuff, maybe it is because the same page certainly served from different geographic locations for me and you

To unsubscribe from this group and stop receiving emails from it, send an email to SDCH+unsubscribe@googlegroups.com.

[ernest.w....@gmail.com]

BTW, did you encounter some problems? like proxies? what else? looks like to make it production ready takes a little bit more than just advertising the dictionary and encoding response.

[Christopher O'Connell]

I've been meaning to write up our experiences as a blog post. We did encounter some problems, proxies, however, were not a major problem, as far as we know.

[ernest.w....@gmail.com]

what blog to monitor? :)

[Christopher O'Connell]

Here's the only two domains I've ever seen a live SDCH dictionary on. If anyone else has them live, send pages, I'd love to get a screenshot with a bigger list.

[Christopher O'Connell]

Oh, as a further note, we've been serving the dictionaries using .dct as a three letter extension. There's no particular reason for that other than a general feeling that static resources *should* have extensions, for some suitable value of *should*.

[Jim Roskind]

FWIW: If you're running chromium, you can visit the URL:

chrome://histograms/Sdch

This will contain (for the given browser session) several counts associated
with SDCH encoding (decoding, advertising a dictionary, dictionary download
sizes, more error details with counts, etc.).  It won't show *what* domain
drove up the usage/counts... but it will at least show how much usage was
seen (if there was some).  For any given "histogram name," you can look in:

https://chromium.googlesource.com/chromium/src.git/+/master/tools/metrics/histograms/histograms.xml

to see (usually) what the histogram is recording (i.e., how to make sense
of the data... was it milliseconds?... or was it a count of use of
enumerated values?? etc.).

As noted earlier on the thread, you can also visit:

chrome://net-internals/#sdch

which will list domains that used SDCH, etc.

In the past, I have seen both Google and LinkedIn using SDCH.

YMMV,

Jim

Chief Optimism Officer,

Department of Optimistic Speculative Optimizations

"It is only the impossible, that takes significant time to achieve!"

[ernest.w....@gmail.com]

Checked,  no Google neither LinkedIn mentioned in dictionary list or whatever

~ Christopher 

To unsubscribe from this group and stop receiving emails from it, send an email to SDCH+unsubscribe@googlegroups.com.

To post to this group, send email to SD...@googlegroups.com.
Visit this group at https://groups.google.com/group/SDCH.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SDCH" group.
To unsubscribe from this group and stop receiving emails from it, send an email to SDCH+uns...@googlegroups.com.
To post to this group, send email to SD...@googlegroups.com.
Visit this group at https://groups.google.com/group/SDCH.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SDCH" group.
To unsubscribe from this group and stop receiving emails from it, send an email to SDCH+uns...@googlegroups.com.
To post to this group, send email to SD...@googlegroups.com.
Visit this group at https://groups.google.com/group/SDCH.
For more options, visit https://groups.google.com/d/optout.