Pagespeed impact the apache application with Waiting for completion of URL error message

[Siva Moorthy]

Hi Team,

We did pagespeed upgrade 8 months back, recently we found that pagespeed makes apache unresponsive which impacts the application severely.

Got Waiting for completion of URL error message around that time.

Traffic to site is normal, Can you please advise ASAP why pagespeed is causing this issue, how can we rectify this one ?

[Tue Dec 15 00:39:30 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @29078] Waiting for completion of URL https://www.example.com/garden-outdoor-living for 1220.03 sec

[Tue Dec 15 00:39:30 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @26945] Waiting for completion of URL https://www.example.com/favicon.ico for 1670.04 sec

[Tue Dec 15 00:39:30 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @27024] Waiting for completion of URL https://www.example.com/bathroom-plumbing/accessories/toilet-brushes-holders for 1670.04 sec

[Tue Dec 15 00:39:30 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @30043] Waiting for completion of URL https://www.example.com/masters/inc_shop_our_range_subcategories.jsp?ajax=true&subFolderId=1&_=1450099357449 for 935.02 sec

[Tue Dec 15 00:39:30 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @30043] Waiting for completion of URL https://www.example.com/tools/power-tools/oscillating-multi-tool for 925.018 sec

Regards /Siva.

[Jeff Kaufman]

I'm sorry about that! This is fixed in the current version: 1.9.33.11.

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/45fc1e08-bef9-48e1-a2ae-c7370bca9178%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[SIVA]

Hi Jeff,

Thanks for the quick reply, can you please explain what is the issue, how it was fixed in 1.9.33.11 version ?

You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/hn6Ry7ZjiNY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUuNjoXCrbPZoXCLp4vT3ey6Ojqd6utv%2B3bB-KCK5L4VbQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

---

Siva Moorthy

 

[Siva Moorthy]

Hi Jeff,

Thanks for the quick reply, can you please explain what is the issue, how it was fixed in 1.9.33.11 version ?

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

It's a long and gory story:
https://github.com/pagespeed/mod_pagespeed/issues/1048

The main problem was that PageSpeed was writing responses out on the
wrong thread [1]. If that write blocked on a slow reader, this could
slow everything down. This was made worse because PageSpeed could get
stuck on a single slow reader forever [2] and also because the
memcache integration could go into a 100% cpu spin on some protocol
errors [3].

Version 1.9.32.10 is the most recent change that has all of these
fixed, but if you're using https-fetching then please use 1.9.32.11
which has the fix for an openssl security bug.

[1] https://github.com/pagespeed/mod_pagespeed/commit/a68006cbcf238df90760fa3be9719a64a156dc51
[2] https://github.com/pagespeed/mod_pagespeed/commit/e59644fc50c257f1360194a895286886c87481b8
[3] https://github.com/pagespeed/mod_pagespeed/commit/84a9deaf9c4df13ae707f44d06f577321de46e8c

>> > email to mod-pagespeed-di...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/45fc1e08-bef9-48e1-a2ae-c7370bca9178%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mod-pagespeed-di...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/eb7e1605-d642-443a-8d6a-8881c28e2dc3%40googlegroups.com.

[SIVA]

Hi Jeff,

I am going through the given documents. You've said to move 1.9.33.11.

Please confirm the latest version available which includes this fix and other vulnerabilities .

You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/hn6Ry7ZjiNY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUu7C_dnpVTKEDZ5Wxo4gFLgrhhOk-0y%2BRnnvHxJzSY7yA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

---

Siva Moorthy

 

[SIVA]

Hi Jeff/Team,

Can I use the same configuration and filters with latest version or do we need to make any specific changes for new version ?

Also please confirm the latest available version which includes this fix and other vulnerability fixes.

--

---

Siva Moorthy

 

[Jeff Kaufman]

You're currently running 1.9.32.3. The most recent release on the 1.9
branch is 1.9.33.11, and that has this fix and all security patches.
It doesn't require any configuration changes relative to earlier
1.9.x.x releases. All versions of 1.9 are very nearly the same code.

The first beta release of the 1.10 series came out yesterday. It is
also fully up to date on bugfixes, and it also shouldn't require
configuration changes. On the other hand, 1.10 represents about a
year of code changes, so it's more likely that something would start
working differently than it did in 1.9.

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAOjCk%3D%3DJjv6GX_3o59ChohEh9E8QazSMUtq3e%2Baha3FagiZC-w%40mail.gmail.com.

[SIVA]

Hi Jeff,

We've installed latest version 1.9.32.11 stable in our application and we are monitoring apache performance .

Could see, below new errors 

1. I hope, this is similar to file not found errors , Please confirm.

[Wed Dec 16 02:22:41 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @20066] x0037_ProductRecalls-1120x123.jpg:0: Resource based on http:/example:6666/meiaa/Media/static/Homepage/FooterBanner/0037_ProductRecalls-1120x123.jpg but cannot access the original

[Wed Dec 16 02:22:41 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @20066] [1216/022241:WARNING:resource_fetch.cc(197)] Fetch failed for resource url http://example:6666/meiaa/Media/static/Homepage/FooterBanner/x0037_ProductRecalls-1120x123.jpg.pagespeed.ic.uOCEoBzGcw.jpg

[Wed Dec 16 02:22:41 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @20066] Fetch failed for http://example:6666/meiaa/Media/static/Homepage/FooterBanner/x0037_ProductRecalls-1120x123.jpg.pagespeed.ic.uOCEoBzGcw.jpg, status=404

[Wed Dec 16 02:22:41 2015] 

[Wed Dec 16 02:22:41 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @20066] http://example:6666/meiaa/Media/static/Homepage/FooterBanner/x0037_ProductRecalls-1120x123.jpg.pagespeed.ic.uOCEoBzGcw.jpg resource_404_count: not found (404)

2 . This error seems to be new. We store files on NFS mount point from where media files will be fetched. Is there anything we need to fix with this error ?

[Wed Dec 16 03:16:30 2015] [error] [mod_pagespeed 1.9.32.11-7550 @20066] Slow read operation on file /applic/apache/httpd/logs/website/mod_pagespeed/cache/prop_page/https,3A/,2Fwww.example.com/building-timber/timber/hardwood-timber.jsp,3Bjsessionid=1DUOPnAhIQnlYTp1STnScg__,3FbmUID=l5.TjIS_r6_FdGm7Gs,40Desktop,40beacon_cohort,: 59.835ms; configure SlowFileLatencyUs to change threshold\n

[Wed Dec 16 03:16:30 2015] [error] [mod_pagespeed 1.9.32.11-7550 @20065] Slow read operation on file /applic/apache/httpd/logs/website/mod_pagespeed/cache/prop_page/https,3A/,2Fwww.example.com/navigation/include/inc_product_more_products_ajax.jsp_r6_FdGm7Gs,40Desktop,40beacon_cohort,: 543.403ms; configure SlowFileLatencyUs to change threshold\n

[Wed Dec 16 03:16:34 2015] [error] [mod_pagespeed 1.9.32.11-7550 @20065] Slow read operation on file /applic/apache/httpd/logs/website/mod_pagespeed/cache/prop_page/https,3A/,2Fwww.example.com/navigation/include/inc_product_more_products_ajax.jsp_r6_FdGm7Gs,40Tablet,40beacon_cohort,: 604.858ms; configure SlowFileLatencyUs to change threshold\n

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUt0WbsyFQiGdTvLRwcpazWEoetyWC7ZNx%2BUM%2BD_EbXbkQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

---

Siva Moorthy

 

[Joshua Marantz]

I would avoid using LoadFromFile with an NFS volume.  This is precisely the point of the SlowFileLatency warning.

When you use normal HTTP fetches in mod_pagespeed (Serf), the origin TTL is used to periodically freshen the content.  If that content is on a file server it's probably OK, because that happens in the background, and not very often -- you control how often by setting your cache-control headers.

When you use LoadFromFile, mod_pagespeed calls stat() on your file for every request -- both for the HTML and for the resource itself.  If that's on a local disk it should be fine (and cached effectively by the operating system).  But if you user using NFS, then call these calls to check the file timestamp are going to hurt.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAOjCk%3Dk9g1-r20f2sTFhupcLnewOzWvNvA0mQCWk54b%2BnXZTOg%40mail.gmail.com.

[SIVA]

Hi Joshua,

We use NFS FS to copy the files to our local system where pagespeed runs. Anyhow pagespeed fetch the files from local cache and compress it. 

Is it still an issue ?

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAGKR%2BEDDc5WQ_-cX9cjtCBNG5nOAUzcHrRoEaBmJu-wYHQD6YA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

---

Siva Moorthy

 

[SIVA]

Hi Jeff, Joshua,

Still we are receiving error after upgrading to 1.9.32.11 . Will it make an impact like before ?

[Thu Dec 17 16:14:44 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @1777] Waiting for completion of URL https://www.example.com.au/suggest-wrapper/?search=campin for 5.002 sec.

[Thu Dec 17 16:14:44 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @2614] Waiting for completion of URL https://www.example.com.au/PickedList/christmastrees.jsp;jsessionid=dkMRm2O7ebli7cHrZCv24g__.?bmUID=l60xJFR for 5.001 sec.

[Thu Dec 17 16:14:44 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @2614] Waiting for completion of URL https://www.example.com.au/suggest-wrapper?search=camping for 5.013 sec.

[Thu Dec 17 16:14:44 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @1777] Waiting for completion of URL https://www.example.com.au/navigation/find_clearance_store.jsp for 5.002 sec.

--

---

Siva Moorthy

 

[Jeff Kaufman]

Are you getting the "Slow read operation on file" messages at the same
time? Are there any other messages you're getting along side it?

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAOjCk%3Dmha7JqVw8aKe8DQDhEg-Un%3Doj2-0XuoCZMV7af86B%2BDg%40mail.gmail.com.

[SIVA]

Hi Jeff,

Here is the summary. 

1. We are getting more "Slow read operation on file" errors , we can avoid this as it will be resolved by changing the latency threshold value.

2. Some times are getting "Waiting for completion of URL " with "Slow read operation on file" errors 

[Mon Dec 21 11:25:17 2015] [warn] [mod_pagespeed 1.9.32.11-7550 @21527] Waiting for completion of URL https://www.example.com/suggest-wrapper/?search=bonda for 5.002 sec.

3. Other than that , we got "Fetch timed out for" , "File does not exist:" , "Fetch failed for" and "Rewrite failed while fetching" errors randomly. Most of them raised due to missing files.

Please advise on second point.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUvXaT4_qN%2Bdf%3Dgc07XK9ETRzi%2BPfhMAvSZcnUaGmf8Jow%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

---

Siva Moorthy

 

[Jeff Kaufman]

When you raise the threshold value you're hiding PageSpeed's warnings,
but you're not fixing the underlying problem of file reads taking too
long. PageSpeed uses blocking file reads, and was built around an
assumption that these would generally complete quickly. While a file
read is happening PageSpeed can't do anything else, and so if those
are very slow PageSpeed gets behind.

It sounds like in your case you have a file system that is sometimes
very slow, and PageSpeed is giving you "Slow read operation on file"
and "Waiting for completion of URL" errors to let you know about the
problems it's having. To fix this you could speed up the file system
or keep PageSpeed from needing the filesystem (by switching our cache
to memcached and/or turning off LoadFromFile.

(This is different than the problem you were having with 1.9.32.3,
because the wait times are much lower, even though the error messages
are similar.)

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAOjCk%3DkDfOGE%3Dj2gZFd3WVW2BRb_fQWAwkrszXx3%2BrRjWyyiCw%40mail.gmail.com.