Performance issues with mod_pagespeed as forwarding proxy

[Simon Werner]

Hello,

I am setting up mod_pagespeed as a forwarding proxy.  The primary goal is to reduce the amount of data transferred (this works well), the side affect is that it is incredibly slow.  My example page (http://www.ozzy.com/us/albums  -- this is just a random link from randomwebsite.com/) takes around 9 seconds to load without pagespeed and around 70 seconds with mod_pagespeed configured as a forwarding proxy.  When mod_pagespeed is configured, the first request takes around 66 seconds to load, the subsequent page loads very quickly.

I see the following:

 - mod_pagespeed uses near 100% CPU for more than 1 minute.

 - mod_pagespeed has rewritten 117 images, although there are only 23 jpegs on the main screen, and 22 other images.

 - in the mod_pagespeed cache folder for www.ozzy.com there are 153 jpg images (using the command 'find /tmp/http,3A/,2Fwww.ozzy.com/ | grep \.jpg, | wc')

Why is mod_pagespeed rewriting so many more images than is actually downloaded?  How can I get the page to load faster?

My set-up:

 - Amazon EC2 micro instance (1 CPU, about 600MB ram, 64bit)

 - Ubuntu 13.04 (amd64)

 - mod_pagespeed version 1.4.26.3-r3101

Mod_pagespeed set-up:

 - uses tmpfs as cache (so I/O is not an issue)

 - the /mod_pagespeed_statistics file is attached.

 - Conf file is below:

    ModPagespeed on

    ModPagespeedInheritVHostConfig on

    AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER text/html

    ModPagespeedFileCachePath            "/tmp/"

    ModPagespeedRewriteLevel CoreFilters

    

    ModPagespeedEnableFilters rewrite_images

    ModPagespeedImageRecompressionQuality 10        # 10% just for testing purposes

    ModPagespeedRewriteDeadlinePerFlushMs -1        # Wait until the whole resource is optimised

    ModPagespeedDomain *

    ModPagespeedFileCacheSizeKb          102400

    ModPagespeedFileCacheCleanIntervalMs 10000

    ModPagespeedLRUCacheKbPerProcess     1024

    ModPagespeedLRUCacheByteLimit        16384

    ModPagespeedCssFlattenMaxBytes       2048

    ModPagespeedCssInlineMaxBytes        2048

    ModPagespeedCssImageInlineMaxBytes   2048

    ModPagespeedImageInlineMaxBytes      2048

    ModPagespeedJsInlineMaxBytes         2048

    ModPagespeedCssOutlineMinBytes       3000

    ModPagespeedJsOutlineMinBytes        3000

    ModPagespeedFileCacheInodeLimit        0

    ModPagespeedImageMaxRewritesAtOnce     4

    ModPagespeedNumRewriteThreads 4

    ModPagespeedNumExpensiveRewriteThreads 4

    ModPagespeedAvoidRenamingIntrospectiveJavascript on

[Joshua Marantz]

Hi,

The behavior you are seeing is not surprising given this:

   ModPagespeedRewriteDeadlinePerFlushMs -1        # Wait until the whole resource is optimised

I think this makes sense given your goals, maybe this is too extreme for you and you should set a bound of (say) 5 seconds, which would still be faster than your non-pagespeed load.  Note that when the deadline is expired, partially optimized HTML is emitted, but all the image optimizations will continue to happen in the background until the optimized images are all cached.

I think also the thread-count and concurrent rewrites might be a little aggressive for your EC2 instance.  It might be better, since you have only 1 CPU, to scale all those '4' settings back a bit, probably to 1 or 2.

You may also want to consider benchmarking non-tmpfs file system, just for comparison purposes.  Since you only have 600M physical, burning a lot on a tmpfs might not be the best way to spend it, since cache warm-up is expensive for you, and the penalty for eviction is high.

One other note: if this is to be a general purpose forward proxy, you might consider two other options:

in-place resource optimization: https://developers.google.com/speed/pagespeed/module/system#ipro

preserve URLs: https://developers.google.com/speed/pagespeed/module/config_filters#preserveurls

The "preserve URLs" feature only makes sense if you also turn on in-place optimization.  Without "preserve URLs", some small percentage of web sites you visit might break due to JavaScript introspection.

-Josh

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Simon Werner]

Hi Josh, thanks a lot for your quick reply.  Please see my response to your suggestions below.  One of my initial questions still remains open:

Why is mod_pagespeed rewriting so many more images than is actually downloaded? 

The behavior you are seeing is not surprising given this:

   ModPagespeedRewriteDeadlinePerFlushMs -1        # Wait until the whole resource is optimised

I think this makes sense given your goals, maybe this is too extreme for you and you should set a bound of (say) 5 seconds, which would still be faster than your non-pagespeed load.  Note that when the deadline is expired, partially optimized HTML is emitted, but all the image optimizations will continue to happen in the background until the optimized images are all cached.

I did this and tried a few combinations of the flush setting.  The issue is that the images come through unprocessed (still with their original compression).  I want to heavily compress images (e.g. jpegs to around 30 to 50%) to save data transfer to the client, this is my primary goal.  When I set this value to 5, or 15, most of the images for this website still came back uncompressed.   Since I am setting up a forward proxy for the internet, I don't expect many cache hits.

 

I think also the thread-count and concurrent rewrites might be a little aggressive for your EC2 instance.  It might be better, since you have only 1 CPU, to scale all those '4' settings back a bit, probably to 1 or 2.

 

I see your point and I tried this, the difference was negligible, if any.

 

You may also want to consider benchmarking non-tmpfs file system, just for comparison purposes.  Since you only have 600M physical, burning a lot on a tmpfs might not be the best way to spend it, since cache warm-up is expensive for you, and the penalty for eviction is high.

As above, I don't expect many cache hits ever, because I am running a forward proxy. 

 

One other note: if this is to be a general purpose forward proxy, you might consider two other options:

in-place resource optimization: https://developers.google.com/speed/pagespeed/module/system#ipro

preserve URLs: https://developers.google.com/speed/pagespeed/module/config_filters#preserveurls

The "preserve URLs" feature only makes sense if you also turn on in-place optimization.  Without "preserve URLs", some small percentage of web sites you visit might break due to JavaScript introspection.

Thanks, I had in-place resource optimisation configured but just not shown on my config.  But, I did not know about PreserveURLs, for my issue above it did not fix anything.

Regards,

Simon  

[Maksim Orlovich]

This is probably because we rewrite all the images inside the CSS
file; there are tons that are likely not used.
ModPagespeedInPlacePreemptiveRewriteImages off might help when
combined with ImagePreserveURLs on;
or you could turn off css optimization, though that will hurt your
compression slightly due to no minification.

[Simon Werner]

Thanks Maksim for your reply.  Your suggestion did reduce the number of images produced, thanks.

However, I have tried many permutations of filter settings and benchmarked them against ziproxy.  I test it on a few websites and have an average throughput of about 3 requests per second (@ 220 kbit/s). Ziproxy compresses images and everything and manages that with around 1% CPU load.  While apache (pagespeed configured as a forward proxy) has an average CPU load of around 27%, to deliver similar content as ziproxy. 

Even when I reduce the mod_pagespeed settings down the minimum (see below) it uses a lot of processing power.  Have I configured my server wrong?

<IfModule pagespeed_module>

    ModPagespeed on

    ModPagespeedInheritVHostConfig on

    AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER text/html

    ModPagespeedFileCachePath            "/tmp/"

    ModPagespeedRewriteLevel PassThrough

    ModPagespeedEnableFilters add_head

    ModPagespeedEnableFilters combine_css

    ModPagespeedDomain *

    ModPagespeedFileCacheSizeKb          102400

    ModPagespeedFileCacheCleanIntervalMs 10000

    ModPagespeedLRUCacheKbPerProcess     1024

    ModPagespeedLRUCacheByteLimit        16384

    ModPagespeedCssFlattenMaxBytes       2048

    ModPagespeedCssInlineMaxBytes        2048

    ModPagespeedCssImageInlineMaxBytes   2048

    ModPagespeedImageInlineMaxBytes      2048

    ModPagespeedJsInlineMaxBytes         2048

    ModPagespeedCssOutlineMinBytes       3000

    ModPagespeedJsOutlineMinBytes        3000

    ModPagespeedFileCacheInodeLimit        0

    ModPagespeedImageMaxRewritesAtOnce    1 

    ModPagespeedNumRewriteThreads 1

    ModPagespeedNumExpensiveRewriteThreads 1

    ModPagespeedAvoidRenamingIntrospectiveJavascript on

</IfModule>

[Joshua Marantz]

I haven't looked at ziproxy, but it sounds pretty useful.   I hadn't heard of it before but it's been around for a while.

The config you pasted doesn't actually do any image compression, however it does parse and re-serialize HTML which costs CPU time.  Looking at the Ziproxy home page it claims it optimizes HTML, but I'm not clear on if it's doing the same things MPS is doing.

You are paying some up-front overhead for features in mod_pagespeed that depend on having a streaming HTML lexer, including: lazyloading images, and deferring JavaScript.  There are a bunch of other features I could list as well but depend on caching to work fast, such as combining CSS (which you have enabled), prioritizing critical css, sizing images to context and transcoding image to webp.

Another difference is that mod_pagespeed runs in Apache HTTPD, which gives you a lot of capability that might be of interest in some cases though perhaps not in yours.  Ziproxy runs (I think) in its own process which makes it much simpler and streamlined, although its home page has warnings about its own limitations due to (e.g.) lack of DNS cache.

Image compression itself is quite a rich field and we are constantly adding functionality and performance.  See https://developers.google.com/speed/pagespeed/module/filter-image-optimize

So if you need those features, mod_pagespeed might be worth the extra CPU time.  But if ziproxy's image compression and text-file compression is sufficient, then it might be a better bet. 

On an absolute scale, though, 3qps seems pretty dismal.  We load-test regularly at 1500 qps on a 6-core Dell desktop.

Looking at your config again, I see this 'ModPagespeedFileCacheCleanIntervalMs 10000'.  That might impact your load, as every 10 seconds you will be garbage-collecting over 100M of disk.  That might keep your CPU & I/O systems pretty busy.  Can you increase the interval?  A better idea would be to use a small memcached instance: https://developers.google.com/speed/pagespeed/module/system#memcached.  That will be much more efficient at evicting most of your cached resources compared to the file-system.

-Josh

[Simon Werner]

Hi Josh,

Thanks for comments.  1500 qps on a 6 core desktop sounds like I am in the right ballpark with my mod_pagespeed set up.  Any idea on ngx_pagespeed's performance in comparison? (I believe it does not work a forwarding proxy though).  With that 1500 qps I presume a sizeable portion would have cache hits, I don't expect many cache hits, so I don't need a cache.  Btw, I changed my  'ModPagespeedFileCacheCleanIntervalMs 10000' to 100000 and this did not change anything.  The reason I used such a simple conf file was just to show that even the most simple configuration uses significant CPU, it gets worse with image compression.

Yes, ziproxy does not have all the features of pagespeed but some can easily be configured like a local DNS cache.  I may even look at chaining the two together.

Are there any tools to benchmark the internals of mod_pagespeed?  So I can see which filters / settings are using up the most resources?

Thanks,

Simon