Segmentation fault while enabling ModPagespeedCreateSharedMemoryMetadataCache

[Hans van Eijsden]

Dear developers,

Thanks for all your great work. To me, it's a pleasure to use MPS, it really makes a difference in user experience and performance.

I've installed mod_pagespeed 1.6.29.4-3289, the newest one. I've done a clean install and I've also done a clean install of Varnish. Then, I configured Varnish as described in your guidelines and I also configured MPS as described, together with memcached. It works great! System load 0.01 - 0.03 with 200 hits/second.

But... when I enable ModPagespeedCreateSharedMemoryMetadataCache things go wrong: Apache crashes.

Apache error.log: 

[Sun Jul 14 15:31:47 2013] [notice] child pid 17075 exit signal Segmentation fault (11)

dmesg: 

[ 3754.765774] apache2[11110]: segfault at 8 ip 00007f6665412e66 sp 00007f66556fe358 error 4 in libaprutil-1.so.0.3.9[7f6665409000+22000]

[ 6817.123737] apache2[17576] general protection ip:7f66651ee07c sp:7f6655d4f350 error:0 in libapr-1.so.0.4.2[7f66651d0000+38000]

[ 8024.609384] apache2[20921] general protection ip:7faab47f7ec1 sp:7faaa72e8280 error:0 in libaprutil-1.so.0.3.9[7faab47ee000+22000]

[ 8036.949251] apache2[21049]: segfault at 42 ip 00007faab45d3488 sp 00007faaaf531e20 error 4 in libapr-1.so.0.4.2[7faab45b5000+38000]

In mod_pagespeed_statistics everything seems fine, I see a lot of hits in the SharedMemoryMetadataCache. But Apache keeps crashing.

# uname -a

Linux ... 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux

# apache2 -v

Server version: Apache/2.2.16 (Debian)

Server built:   Mar  3 2013 12:10:30

# php -v

PHP 5.4.17-1~dotdeb.0 (cli) (built: Jul  6 2013 16:56:09) 

Copyright (c) 1997-2013 The PHP Group

Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies

    with Zend OPcache v7.0.2, Copyright (c) 1999-2013, by Zend Technologies

Vanilla Debian Squeeze (later this year I'll upgrade to Wheezy). PHP via FPM. Memcache 1.4.5 on localhost. Varnish 3.0.4. All installed via aptitude/apt-get and up-to-date.

What should I do? I'm not a developer, and I'm not used to work with strace, but I really want to help you and to contribute, if that's needed.

Or maybe if I've done something wrong. Always possible. :)

Which information do you need? And how do I obtain that (for example, if you need a dump, how do I create that)?

For now, I disabled ModPagespeedCreateSharedMemoryMetadataCache and everything runs fine.

Thank you very much and keep up the good work.

[Anupama Dutta]

Hans,

Thanks for trying out mod_pagespeed. Will it be possible for you to temporarily disable the Varnish-cache-integration (by deleting the "ModPagespeedModifyCachingHeaders off" line and removing all the ModPagespeedDownstreamCache* directives from your config), and tell us if re-enabling ModPagespeedCreateSharedMemoryMetadataCache continues to cause Apache crashes?

Thanks,

Anupama.

[Hans van Eijsden]

Thanks for the quick reply and your help.

I disabled (commented out) the directives you mentioned. 

But again:

[Mon Jul 15 18:12:55 2013] [notice] child pid 7889 exit signal Segmentation fault (11)

Re-enabling ModPagespeedCreateSharedMemoryMetadataCache continues to cause Apache crashes.

By the way I have to keep this line disabled, commented out: ModPagespeedDownstreamCacheRewritingThresholdPercentage 95

When I enable it, I receive a syntax error while (re)starting Apache:

/etc/init.d/apache2 reload

Syntax error on line 44 of /etc/apache2/mods-enabled/pagespeed.conf:

Invalid command 'ModPagespeedDownstreamCacheRewritingThresholdPercentage', perhaps misspelled or defined by a module not included in the server configuration

Action 'configtest' failed.

The Apache error log may have more information.

 failed!

// Hans

Op maandag 15 juli 2013 16:55:50 UTC+2 schreef Anupama Dutta het volgende:

[Maksim Orlovich]

FWIW: turning on 'ModPagespeedInstallCrashHandler on' might offer some
insight on what's going wrong, as it will
cause us to try to get a stack trace when crash occurs; though I am
not certain I'll be able to figure it out
from release binaries. Also, are there any messages mentioning shared
memory in the log?

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Hans van Eijsden]

The CrashHandler works. I tried it and here's the entry in the Apache error.log:

[Mon Jul 15 20:08:11 2013] [alert] [@21547] CRASH with signal:11 at Backtrace:\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0xe231a) [0x7ffa109d731a]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x1a3683) [0x7ffa10a98683]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0xc079e) [0x7ffa109b579e]\n\t/lib/libc.so.6(+0x35f0c32230) [0x7ffa15101230]\n\t/usr/lib/libapr-1.so.0(+0x35f281da70) [0x7ffa1566aa70]\n\t/usr/lib/libapr-1.so.0(apr_allocator_alloc+0x23) [0x7ffa1566ae6e]\n\t/usr/lib/libaprutil-1.so.0(apr_bucket_alloc+0x8c) [0x7ffa1588ff2c]\n\t/usr/lib/libaprutil-1.so.0(+0xbd74) [0x7ffa15891d74]\n\t/usr/lib/libaprutil-1.so.0(apr_brigade_split_line+0xc7) [0x7ffa15891947]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3b15) [0x7ffa10da8b15]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3ca1) [0x7ffa10da8ca1]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b5a80) [0x7ffa10daaa80]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b1017) [0x7ffa10da6017]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x2973b7) [0x7ffa10b8c3b7]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x298452) [0x7ffa10b8d452]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e0e1d) [0x7ffa10dd5e1d]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x2fc1f1) [0x7ffa10bf11f1]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x31c05c) [0x7ffa10c1105c]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e587f) [0x7ffa10dda87f]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e59ed) [0x7ffa10dda9ed]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x30afe6) [0x7ffa10bfffe6]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x28b7c8) [0x7ffa10b807c8]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x31caa2) [0x7ffa10c11aa2]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x33364c) [0x7ffa10c2864c]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x31caa2) [0x7ffa10c11aa2]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x33733c) [0x7ffa10c2c33c]\n\t/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b6c58) [0x7ffa10dabc58]\n\t/lib/libpthread.so.0(+0x35efe068ca) [0x7ffa154378ca]\n\t/lib/libc.so.6(clone+0x6d) [0x7ffa1519eb6d]\n

Oh, I use apache2-mpm-event. But the same crashes happen with apache2-mpm-worker too.

I don't see any messages about shared memory in the logs.

Here are my Varnish and Pagespeed config files: http://www.weblogzwolle.nl/confs/ (no confidential things in them).

// Hans

Op maandag 15 juli 2013 19:52:17 UTC+2 schreef Maksim Orlovich het volgende:

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Anupama Dutta]

Thanks, Hans, for pointing out the syntax error alert for the new configuration. I have fixed the documentation to mention "ModPagespeedDownstreamCacheRewrittenPercentageThreshold" which is the correct name for the directive. Do try out this corrected directive and let us know if you face any other issues.

You should also copy over the latest version of the generate_user_agent_based_key method from https://developers.google.com/speed/pagespeed/module/downstream-caching#varnish_conf since there was a small update made to this method a couple of days ago.

Also, could you share any numbers related to the reduction in server load with the Varnish cache integration? Since Varnish support is a new feature, any feedback you have related to it will be very helpful.

Thanks,

Anupama.

[Maksim Orlovich]

Thanks for the backtrace. Finally managed to decode it:

[Mon Jul 15 20:08:11 2013] [alert] [@21547] CRASH with signal:11 at Backtrace:

/usr/lib/apache2/modules/mod_pagespeed.so(+0xe231a)
base::debug::StackTrace::StackTrace() + 26 in section .text of
/tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x1a3683)
net_instaweb::StackTraceString() + 51 in section .text of
/tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0xc079e)
signal_handler + 46 in section .text of /tmp/libmod_pagespeed.so
/lib/libc.so.6(+0x35f0c32230)
/usr/lib/libapr-1.so.0(+0x35f281da70)
/usr/lib/libapr-1.so.0(apr_allocator_alloc+0x23)
/usr/lib/libaprutil-1.so.0(apr_bucket_alloc+0x8c)
/usr/lib/libaprutil-1.so.0(+0xbd74)
/usr/lib/libaprutil-1.so.0(apr_brigade_split_line+0xc7)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3b15)
get_server_line + 37 in section .text of /tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3ca1)
sendv_and_get_server_line_with_timeout + 129 in section .text of
/tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b5a80)
apr_memcache2_getp + 272 in section .text of /tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b1017)
net_instaweb::AprMemCache::Get(std::string const&,
net_instaweb::CacheInterface::Callback*)
src/net/instaweb/system/apr_mem_cache.cc:199
/usr/lib/apache2/modules/mod_pagespeed.so(+0x2973b7)
net_instaweb::RewriteContext::OutputCacheCallback::IsFilesystemMetadataCacheCurrent(net_instaweb::CacheInterface*,
std::string const&, net_instaweb::InputInfo const&, long)
src/net/instaweb/rewriter/rewrite_context.cc:299
/usr/lib/apache2/modules/mod_pagespeed.so(+0x298452)
net_instaweb::RewriteContext::OutputCacheCallback::IsCachedResultValid(net_instaweb::CachedResult*,
bool*, std::vector<net_instaweb::InputInfo*,
std::allocator<net_instaweb::InputInfo*> >*)
src/net/instaweb/rewriter/rewrite_context.cc:441
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e0e1d)
net_instaweb::WriteThroughCallback::ValidateCandidate(std::string
const&, net_instaweb::CacheInterface::KeyState)
src/net/instaweb/util/write_through_cache.cc:55
/usr/lib/apache2/modules/mod_pagespeed.so(+0x2fc1f1)
net_instaweb::DelegatingCacheCallback::ValidateCandidate(std::string
const&, net_instaweb::CacheInterface::KeyState)
src/net/instaweb/util/delegating_cache_callback.cc:46
/usr/lib/apache2/modules/mod_pagespeed.so(+0x31c05c)
net_instaweb::CacheInterface::ValidateAndReportResult(std::string
const&, net_instaweb::CacheInterface::KeyState,
net_instaweb::CacheInterface::Callback*)
src/pagespeed/kernel/base/cache_interface.cc:39
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e587f)
net_instaweb::SharedMemCache<64ul>::GetFromEntry(std::string const&,
net_instaweb::SharedMemCacheData::Sector<64ul>*, int,
net_instaweb::CacheInterface::Callback*) + 351 in section .text of
/tmp/libmod_pagespeed.so
(inline? ~_Vector_base)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e59ed)
net_instaweb::SharedMemCache<64ul>::Get(std::string const&,
net_instaweb::CacheInterface::Callback*) + 301 in section .text of
/tmp/libmod_pagespeed.so
(inline? std::string::_M_rep)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x30afe6)
net_instaweb::CacheStats::Get(std::string const&,
net_instaweb::CacheInterface::Callback*)
src/net/instaweb/util/cache_stats.cc:142
/usr/lib/apache2/modules/mod_pagespeed.so(+0x28b7c8)
net_instaweb::RewriteContext::Start()
src/net/instaweb/rewriter/rewrite_context.cc:1391
/usr/lib/apache2/modules/mod_pagespeed.so(+0x31caa2)
net_instaweb::Function::CallRun()
src/pagespeed/kernel/base/function.cc:52
/usr/lib/apache2/modules/mod_pagespeed.so(+0x33364c)
net_instaweb::QueuedWorkerPool::Run(net_instaweb::QueuedWorkerPool::Sequence*,
net_instaweb::QueuedWorker*)
src/pagespeed/kernel/thread/queued_worker_pool.cc:155
/usr/lib/apache2/modules/mod_pagespeed.so(+0x31caa2)
net_instaweb::Function::CallRun()
src/pagespeed/kernel/base/function.cc:52
/usr/lib/apache2/modules/mod_pagespeed.so(+0x33733c)
net_instaweb::Worker::WorkThread::Run()
src/pagespeed/kernel/thread/worker.cc:84
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b6c58)
net_instaweb::PthreadThreadImpl::InvokeRun(void*)
src/pagespeed/kernel/thread/pthread_thread_system.cc:124
/lib/libpthread.so.0(+0x35efe068ca)
/lib/libc.so.6(clone+0x6d) [0x7ffa1519eb6d]

Not sure as to what it means just yet, at this point I am merely happy
to have the info.
(Need to do one more tweak to get the line numbers for the C code, too).

[Maksim Orlovich]

And the C frames:

/usr/lib/libaprutil-1.so.0(apr_brigade_split_line+0xc7)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3b15)
get_server_line + 37 in section .text of /tmp/libmod_pagespeed.so

src/third_party/aprutil/apr_memcache2.c:679

/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3ca1)
sendv_and_get_server_line_with_timeout + 129 in section .text of
/tmp/libmod_pagespeed.so

src/third_party/aprutil/apr_memcache2.c:715

/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b5a80)
apr_memcache2_getp + 272 in section .text of /tmp/libmod_pagespeed.so

src/third_party/aprutil/apr_memcache2.c:896
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b1017)

[Hans van Eijsden]

Sorry for my late reply and thanks for the support (quite a crazy busy week here).

I tried the ModPagespeedDownstreamCacheRewrittenPercentageThreshold directive and it works, that's the right one. I also changed my Varnish config to the latest generate_user_agent_based_key method. Works!

I still have the segmentation faults though, while enabling ModPagespeedCreateSharedMemoryMetadataCache - also on the other server I manage.

I would love to share some numbers, related to the reduction in server load. These are the numbers of www.weblogzwolle.nl with about 200 requests a second:

- Bare Apache + PHP-FPM: 16.00 - 23.00

- Bare Apache + PHP-FPM + APC: 9.00 - 11.00

- Bare Apache + PHP-FPM + Zend Opcache 7.0.2 beta: 8.00 - 9.50

- Apache + mod-pagespeed via memcached + PHP-FPM + Zend Opcache 7.0.2 beta: 0.80 - 6.50

- Varnish + Apache + PHP-FPM + Zend Opcache 7.0.2 beta: 0.01 - 0.02 (Varnish caching ALL the static content)

- Varnish + Apache + mod-pagespeed via memcached + PHP-FPM + Zend Opcache 7.0.2 beta: 0.01 - 0.07 (Varnish caching ALL the static content)

- Varnish + Apache + mod-pagespeed via memcached + PHP-FPM + Zend Opcache 7.0.2 beta: 0.02 - 0.38 (New Varnish and Pagespeed config as suggested in your documentation)

So, yes, serving static requests through Apache -> mod-pagespeed -> memcached takes more time and load then serving through Varnish on top. It's still quite fast though, when the memcache was warmer. ;)

// Hans

Op dinsdag 16 juli 2013 00:30:18 UTC+2 schreef Anupama Dutta het volgende:

[Anupama Dutta]

Thanks a lot, Hans, for proving such detailed numbers - these are really useful to us!

Just to make sure I understand these correctly, the earlier "Varnish + mod_pagespeed + memcached" setup (your last-but-one row) was the best configuration in terms of server load and time, and the current "Varnish + mod_pagespeed + memcached setup with the new suggested Varnish and PS configuration" (your last row) is the second best configuration in terms of server load and time. Is that correct?

If yes, I suppose that the slightly increased load can be explained by the new configuration sending ~5 times more requests than the earlier configuration, because of the 5 cache-keys-per-URL approach? [I am assuming that you had no cache fragmentation in your earlier setup and had explicitly disabled most UserAgent-dependent optimizations at that point.]

You also mention that it is quite fast when "memcache was warmer" - does that refer to whenever shared memory metadata cache was working (has it worked for you before?) or do you simply mean that it works better when the existing memcache has been warmed up for your traffic?

Thanks,

Anupama.

[Hans van Eijsden]

Hi Anupama, you're welcome! I'm glad to contribute something to your beautiful project.

The earlier "Varnish + mod_pagespeed + memcached" setup (my last-but-one row) was the best configuration indeed, in terms of server load and time. This, because Varnish is a layer on top of all the other software, with an average response and deliver time of less than 20 ms. But resources passed through Varnish to the backend, need to be served with the slow and heavy Apache. 

Nginx gave much, much better results, but I'll wait with the migration until ngx_pagespeed is available as module or binary via apt-get, to make future upgrades easy and steady.

Yes, the current "Varnish + mod_pagespeed + memcached setup with the new suggested Varnish and mod-pagespeed configuration" (my last row) is the second best configuration in terms of server load and time, and pagespeed score of course.

No, the slightly increased load can NOT be explained by "the new configuration sending ~5 times more requests than the earlier configuration". For Varnish it's not a problem sending 5 times as much more requests, the load won't even rise with 5000 times more requests - as long as they're served completely by Varnish and not passed or piped through to the backend.

The only reason for an increased load is Varnish passing the .pagespeed. requests to the Apache backend. Apache takes much resources. Varnish doesn't - the limitation of Varnish is not IO or CPU, but the network. :-)

My mention of "memcache was warmer" did not refer to the shared memory metadata cache - I couldn't test that cache, because then Apache was and is crashing every 2 or 3 minutes with that cache enabled. The segfaulting Apache created a higher load of course, so I couldn't compare it.

But yes, the shared memory metadata cache worked for that short time: at least the stats were working. At least I could see the hits in "mod_pagespeed_statistics?memcached".

Mod-pagespeed works better when the existing memcache has been warmed up for my traffic, because then all the images are converted, optimized and stored in memcached.

I hope it's a little bit more helpful now.

Enjoy your weekend! :)

// Hans

Op vrijdag 19 juli 2013 04:49:51 UTC+2 schreef Anupama Dutta het volgende:

[Jeff Kaufman]

On Fri, Jul 19, 2013 at 3:24 PM, Hans van Eijsden <hans...@gmail.com> wrote:
> The only reason for an increased load is Varnish passing the .pagespeed.

> requests to the Apache backend. Apache takes much resources. Varnish doesn't.

It sounds like you'd rather serve the .pagespeed. resources from
Varnish instead of Apache? You could do that, and it would work fine.
The documentation says to include in your config:

# Block 5a: Bypass the cache for .pagespeed. resource. PageSpeed has its own
# cache for these, and these could bloat up the caching layer.
if (req.url ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {
# Skip the cache for .pagespeed. resource. PageSpeed has its own
# cache for these, and these could bloat up the caching layer.
return (pass);
}

but if you removed that then Varnish would cache your .pagespeed.
resources and not need to go back to Apache for most requests.

This would cause .pagespeed. resources to get cached multiple times by
Varnish under different cache keys, unfortunately. Because these
resources include the hash of the content in the url, however, the url
is a sufficient cache key on its own. While I haven't tested this, I
think you could add a section to generate_user_agent_based_key that
puts all .pagespeed. resources into one fragment. Something like:

if (req.url ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {
# Put all PageSpeed resources into the same cache fragment.
set req.http.PS-CapabilityList =
req.http.default_ps_capability_list_for_large_screens;
}

If you do try this, I would be curious what effect it has on your load numbers.

Jeff

[Hans van Eijsden]

Jeff! That's great, thanks!

I've added that code, and I will monitor the performance and load for today.

I'll post the results here, tomorrow or so, when the caches are warmed up.

// Hans

Op maandag 22 juli 2013 16:29:06 UTC+2 schreef Jeff Kaufman het volgende:

[Joshua Marantz]

Hi,

There are threads of discussion here:

   1.  Tweaking and measuring the varnish cache settings

   2.  Figuring out why shared-memory caching & memcached are not interacting well for you.

I have moved #2 to a separate Issue: https://code.google.com/p/modpagespeed/issues/detail?id=747

Let's discuss the crash & how we can repro/fix it there.  This thread can be for turning the settings.

-Josh

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

[Hans van Eijsden]

Josh, thanks for separating the discussion.

Jeff, good news: I see good results already.

The load is now steady at 0.00 - 0.01 with 220 - 280 requests/s.

I love the results! The only thing is.. it consumes twice as much RAM: the static items are stored in memcached AND Varnish. But for me it works, because I've got enough RAM to store all the contents twice.

I will continue to monitor.

// Hans

Op maandag 22 juli 2013 16:38:41 UTC+2 schreef jmarantz het volgende:

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

Yes, see also: https://code.google.com/p/modpagespeed/issues/detail?id=705 -- basically when the system already has good caches then mod_pagespeed might not need to keep its own copies of inputs &/or outputs.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

[Jeff Kaufman]

On Mon, Jul 22, 2013 at 11:30 AM, Hans van Eijsden <hans...@gmail.com> wrote:
> The only thing is.. it consumes twice as much RAM: the
> static items are stored in memcached AND Varnish. But for me it works,
> because I've got enough RAM to store all the contents twice.
>

If using twice as much RAM becomes a problem, moving PageSpeed's cache
from memcache to something backed by the filesystem might make sense.
Normally that would hurt performance, but if almost all requests are
being served out of Varnish's cache then the occasional request to
disk for a rarely requested optimized image might not be too bad.

Jeff