"mojo get" fails on some HTTPS website

[jo8532]

Hi,

I get no response for this website:

$ mojo get https://www.justhappy.fr/

$

I'm able to get the content of this webpage with other user-agents (wget, telnet-ssl, Firefox, perl script using IO::Socket::SSL, ...). When I try to use Mojo::UserAgent in my own script, I get a 200 response code, but the content is empty.

Mojo works fine with other HTTPS websites:

$ mojo get https://www.google.com/ | wc -c

10803

Is this a bug in Mojolicious? Any help appreciated.

[sri]

Is this a bug in Mojolicious? Any help appreciated.

Yes, that is very likely a bug in out HTTP message framing code. We

should expect a message body in this case, but do not.

--

sebastian

[sri]

Afraid i don't have time to fix this at the moment. So we'll need a volunteer.

--

sebastian

[sri]

If you open a GitHub issue for this i'll flag it as a confirmed bug.

--

sebastian

[Tekki]

I've found a second site that doesn't work: https://gl.ch

What they have in common and what is different from other websites is that the response headers are not written in lower case.

justhappy.fr:

1. Access-Control-Allow-Origin:
   *
2. Cache-Control:
   max-age=0, private, must-revalidate
3. Content-Encoding:
   gzip
4. Content-Type:
   text/html; charset=utf-8
5. Date:
   Tue, 07 Aug 2018 06:38:04 GMT
6. ETag:
   W/"a43244322c99f68a2d2aea1b7e491de9"
7. Set-Cookie:
   _simplebo_tool_session=dGZYU3k5d2VVSS9IejQzRjBlTjk2NzZjZEhpSGRvWjFoMmlpZWRtYWJsdEF2RWpTbjYxazlRZ0ZENVlaaTVvTWNpNmtFd1R1QmFPUEFWR1A1NnJERFc3KzhUbUpDeVdzbSs3L0pXWHJwN0ZSZ1l5Y0N1b3N2QXJRTEJlRkVaVHl3dTRVYWdINjBxYVpOYjh2OFhmeUVBPT0tLTdxUHlSS0NzWTN5Z2EydVdqRmo4L1E9PQ%3D%3D--fb247d72a1e3b42cf7f940100584403db220eb73; path=/; HttpOnly
8. Vary:
   Accept-Encoding
9. Via:
   1.1 alproxy
10. X-Content-Type-Options:
    nosniff
11. X-Request-Id:
    71265db3-9b5d-46c4-807e-1629fa89fcbd
12. X-ServedBy:
    simplebo
13. X-XSS-Protection:
    1; mode=block

gl.ch:

1. Connection:
   close
2. Content-Encoding:
   gzip
3. Content-Language:
   de-CH
4. Content-Type:
   text/html;charset=ISO-8859-1
5. Date:
   Tue, 07 Aug 2018 06:40:22 GMT
6. Transfer-Encoding:
   chunked
7. Vary:
   Accept-Encoding
8. X-Powered-By:
   ASP.NET

No idea if this is related to the problem.

Tekki

[Dmitry L.]

It is problem with website, but not with mojo.

Try:
curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip'
vs
curl -v 'https://www.justhappy.fr/'

> --
> You received this message because you are subscribed to the Google Groups "Mojolicious" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
> To post to this group, send email to mojol...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mojolicious.
> For more options, visit https://groups.google.com/d/optout.



--
//wbr, Dmitry L.

[sri]

It looks like the problem is that the response does not include a Content-Length or Transfer-Encoding header. So Mojolicious defaults to expecting a HTTP/1.1 response on a keep-alive connection without a body, when it should expect a body terminated by a connection close.

--

sebastian

[sri]

I think all of this is related to Mojo::Content::expect_close. If that defaults to 1 the site can be opened. We need better code to detect which HTTP responses are supposed to be empty.

--

sebastian

[sri]

    http://paste.opensuse.org/58336341

This patch actually makes the site work. But of course it also breaks existing tests and

looks like it might not be exactly compatible with the HTTP spec, so we can't just apply

it. Unfortunately it doesn't look like i will have time to properly look into it anytime soon,

so i can only give you a few pointers.

But given the fact that nobody has even opened a GitHub issue yet, i suppose this is

not exactly a high priority bug.

--

sebastian

[Tekki]

Why a problem with the website? Try:

curl -v 'https://www.google.com/' -H 'Accept-Encoding: gzip'

curl -v 'https://www.justhappy.fr/' -H 'Accept-Encoding: gzip' | gunzip -c

[Dmitry L.]

There are no content-length header

вт, 7 авг. 2018 г., 16:53 Tekki <te...@tekki.ch>:

[Tekki]

Correct, the same for https://gl.ch

[jo8532]

Thank you to everyone who tried to help. Thank you sri for publishing a fix.