Content-Length header causing trouble

[Hrivnak, Michael]

I am testing "docker pull" from static files, served by apache in the v2 API layout. There is a strange behavior with the Content-Length response header that leads to a parsing error in docker. I'm hoping someone can help explain what I'm seeing.

The mystery is in a behavior of the golang http client, as I've demonstrated with this tiny app:

https://gist.github.com/mhrivnak/2d35be8500db8b16f5c2

Full output is shown in comments below the gist.

If I use that to GET a blob file served by the v2 registry, the Content-Length header is present as expected on the Response.

If I do the same GET when the file is served via apache, I can see the Content-Length header in the response using wireshark, but it does not get populated on the Response. Repeating the same request with other clients, like curl, shows the Content-Length header being present.

It seems like something in golang is deliberately ignoring that header when served by apache, but I can't figure out where or why. If we can explain this behavior, then it will be more clear what docker should do when trying to parse the Content-Length value.

Thanks!

Michael

[Vincent Batts]

Cc'ing docker distribution

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen Day]

Looks Apache is trying to compress the content, which is not necessary. Turn off compression.

Sent from my iPhone

--

[Hrivnak, Michael]

Note that I'm checking for the header three different ways, only one of which is DumpResponse. All three fail when using apache, and all three succeed when using the registry.

I made an image that reproduces this: https://registry.hub.docker.com/u/mhrivnak/staticreg/

source: https://github.com/mhrivnak/dockerstaticpull/tree/master/staticreg

The URL baked into my previous gist will work with the above if you adjust the hostname.

Michael

On Tue, Jul 28, 2015 at 8:48 PM, Olivier Gambier <olivier...@docker.com> wrote:

I wouldn’t trust DumpResponse to display what’s on the wire precisely.
http://golang.org/src/net/http/httputil/dump.go?s=7108:7182#L249

Now, from apache 2.4.10 I can read a valid Content-Length header from go without any issue…

Can you make that server accessible to us?

Best,

--
#olivier

On July 28, 2015 at 3:29:37 PM, Vincent Batts (vba...@gmail.com) wrote:
> Cc'ing docker distribution

[Hrivnak, Michael]

I figured out what was going wrong. Short answer: I think everything is just fine.

Long answer...

Apache httpd on centos7 comes out-of-the-box with "mod_mime_magic" enabled, and configured in such a way that it was incorrectly adding a "Content-Encoding: x-gzip" header to responses with blob files. It did this only because it noticed that these files had been compressed with gzip [0]. UGH. Upon disabling that module, the header went away, and go's http client could read the Content-Length header.

I suspect that go received the response, honored the Content-Encoding header by unzipping the body, determined that the resulting data's size did not match the Content-Length header, and decided to handle this by censoring the Content-Length header.

There is still a small bug where a web server that legitimately does not respond with a Content-Length header will cause a parsing error in docker, but this is minor, and I'll report it separately.

Michael

[0] See the very last example in this section: http://httpd.apache.org/docs/2.4/mod/mod_mime_magic.html#format