Double compression over HTTPS

[Lieven Govaerts]

Hi,



when OpenSSL is built with zlib, it will automatically compress all data sent over an SSL connection. You can see this in the initial handshake "Client Hello" and "Server Hello" where client and server agree on the compression mechanism to be used.

If the data being sent or received is already compressed, OpenSSL will compress it a second time. This can happen when already compressed binaries like .gif or .zip are sent, or when the server uses gzip encoding for a http response. This can have impact on performance and memory usage. See Paul Querna's blog post about this topic in [1]. This also has been mentioned before on svn-dev by Justin in [2].

Since OpenSSL 1.0 this automatic compression can be disabled at runtime.

Compression by OpenSSL has some advantages and disadvantages:
+ OpenSSL will compress the full data stream, so for https that includes all headers + all small requests and responses which mod_deflate skips.
+ OpenSSL compression is stateful, it will not reset its dictionary between every response like gzip/deflate-encoding does, so it will reach better compression ratio when content of multiple consecutive requests or responses are similar within a 32KB window. Side note: I have done tests with using a preset dictionary for zlib for http(s) responses and found the difference can be up to 50% extra compression.
- Where content is already compressed by the application layer (e.g. gzip encoding or transferring binary files), OpenSSL will compress these again.


I have been doing some small-scale testing to see what difference this all makes. My test case was using svn to checkout a copy of the subversion trunk branch in the asf repository.

I have tested 4 different scenario's:
1. As-is setup, OpenSSL compression enabled + gzip encoding enabled. (double compression)
2. OpenSSL compression disabled + gzip encoding enabled. (compression handled by the application)
3. OpenSSL compression disabled + gzip encoding disabled. (no compression at all)
4. OpenSSL compression enabled + gzip encoding disabled (compression handled by OpenSSL)

I found this particular scenario too small to see a measurable difference in memory or cpu usage, although this is interesting to test further.

Difference in total times are more interesting:
   | bytes read | bytes written | total time
1: | 17.50MB    | 233-284KB     | 59s
2: | 18.67MB    | 2.13-2.43MB   | 1m9s-1m18s
3: | 50.35MB    | 2.34MB        | 103s-108s
4: | 15.27MB    | 235-260KB     | 50s-56s

You can see from the above reasoning and my test results that it would be beneficial to disable gzip encoding when using https if OpenSSL was built with zlib. However, in the scenario where large compressed binary files are stored in a svn repository, I suppose disabling both OpenSSL compression and gzip encoding will provide the best results.

Given the above I propose the following:

- Add an option in serf to disable OpenSSL compression

- Add a function in serf to check if compression is enabled in OpenSSL.

- In Subversion, don't ask for gzip encoding when working over https with compression.

- In Subversion, if the config option "http-compression" is set to "no", disable both OpenSSL compression and gzip encoding.

Which makes scenario 4 the default, and the user can select for scenario 3 with the "http-compression" option.

Patch to disable OpenSSL compression in serf is attached.

Suggestions? Objections?

Lieven

[1]: http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/

[2]: http://svn.haxx.se/dev/archive-2011-05/0362.shtml

[Lieven Govaerts]

On Fri, Oct 5, 2012 at 12:15 PM, Bert Huijben <be...@qqmail.nl> wrote:
> +1 on this.
>
>
>
> Can you see if zlib is being used (after negotiation) before requesting
> compression from the Server?
>

It's possible, openssl has functions SSL_get_current_compression and
SSL_get_current_expansion for this purpose.
I'll have to check how they can fit in with the lazy initialization
used by serf though.

> Just checking if the feature is available in openssl doesn’t tell if the
> feature is compiled in at the other side.
>
>
>
>
>
> Using 1m9 (=69 seconds) and 109 seconds in the same table, hides the major
> difference between the numbers.

Indeed. This is the table with all times in seconds:

| bytes read | bytes written | total time
1: | 17.50MB | 233-284KB | 59s

2: | 18.67MB | 2.13-2.43MB | 69s-78s

3: | 50.35MB | 2.34MB | 103s-108s
4: | 15.27MB | 235-260KB | 50s-56s

>
>

> We should clearly avoid accidentally having no compression at all unless
> requested, as the double compression works much better than that scenario.
>
>
> Bert
>

Lieven
[..]

[Ivan Zhakov]

Also note that Subversion deltas (svndiff) already compressed. Deltas
compression can be disabled using "SVNCompressionLevel 0" directive
in Apache configuration file. What I'm thinking about is to add code
to mod_dav_svn to detective compressed OpenSSL connection and disable
svndiff compression regardless SVNCompressionLevel. Does it make sense?

--
Ivan Zhakov

[Ivan Zhakov]

On Fri, Oct 5, 2012 at 1:51 PM, Lieven Govaerts <l...@apache.org> wrote:
> Hi,
>

[...]

>
> I have tested 4 different scenario's:
> 1. As-is setup, OpenSSL compression enabled + gzip encoding enabled. (double
> compression)
> 2. OpenSSL compression disabled + gzip encoding enabled. (compression
> handled by the application)
> 3. OpenSSL compression disabled + gzip encoding disabled. (no compression at
> all)
> 4. OpenSSL compression enabled + gzip encoding disabled (compression handled
> by OpenSSL)
>
> I found this particular scenario too small to see a measurable difference in
> memory or cpu usage, although this is interesting to test further.
>
> Difference in total times are more interesting:
> | bytes read | bytes written | total time
> 1: | 17.50MB | 233-284KB | 59s
> 2: | 18.67MB | 2.13-2.43MB | 1m9s-1m18s
> 3: | 50.35MB | 2.34MB | 103s-108s
> 4: | 15.27MB | 235-260KB | 50s-56s
>

[...]

> Given the above I propose the following:
> - Add an option in serf to disable OpenSSL compression
> - Add a function in serf to check if compression is enabled in OpenSSL.
> - In Subversion, don't ask for gzip encoding when working over https with
> compression.
> - In Subversion, if the config option "http-compression" is set to "no",
> disable both OpenSSL compression and gzip encoding.
>
> Which makes scenario 4 the default, and the user can select for scenario 3
> with the "http-compression" option.
>
> Patch to disable OpenSSL compression in serf is attached.
>
> Suggestions? Objections?
>

It seems I found easy fix for this issue. Just add Apache directive:
SetEnvIf SSL_COMPRESS_METHOD "DEFLATE" no-gzip

This line should disable mod_deflate if OpenSSL compression is used
for current connection. I'm going to test it tomorrow.

--
Ivan Zhakov

[Lieven Govaerts]

This topic seems to be coming up in issue #3980 - serf increases
server load, so I'd like to clear that none of the proposed changes
have been implemented.
The reason is a confirmed security issue with openssl compression:
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4929
https://issues.apache.org/bugzilla/show_bug.cgi?id=53219

The proposed approach here is to disable SSL compression completely,
so in terms of the above that leaves us with scenario 2.

Serf doesn't have an option currently to disable SSL compression from
the client side. I plan to add it in the next version.

[Ivan Zhakov]

From my testing OpenSSL compression is also very slow when used for
local area connections.

--
Ivan Zhakov

[Justin Erenkrantz]

On Tue, Nov 13, 2012 at 2:41 AM, Lieven Govaerts <l...@apache.org> wrote:

The proposed approach here is to disable SSL compression completely,

so in terms of the above that leaves us with scenario 2.

Serf doesn't have an option currently to disable SSL compression from
the client side. I plan to add it in the next version.

 

+1.  -- justin 

[Lieven Govaerts]

Implemented in serf r1692, with an API for applications to enable SSL
compression. Which to be clear, I don't propose svn to use.

[Greg Stein]

On Nov 14, 2012 1:44 AM, "Lieven Govaerts" <l...@apache.org> wrote:
>...

> Implemented in serf r1692, with an API for applications to enable SSL
> compression. Which to be clear, I don't propose svn to use.

This API would go into a 1.2 and 2.0, but if nobody is asking for it, then I'd suggest we don't spin those up yet.

Cheers,
-g

[Lieven Govaerts]

Agreed on not making a release only for this change, but I do plan to
prepare a 1.2 release pretty soon, when the issue fixing work in
progress has been finalized.

Lieven

[Greg Stein]

I likely missed some stuff in your blazing action :-)

Sounds like some other APIs were introduced, for release?

Cheers,
-g