Issue 1048 in modpagespeed: Waiting for completion of URL

[modpag...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 1048 by unsalkor...@gmail.com: Waiting for completion of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I just installed mod_pagespeed on my centos 7 and got tons of httpd errors
log in 1 minute. An example line of error:
<code>
[Tue Feb 10 11:05:14.311755 2015] [pagespeed:warn] [pid 21132:tid
139634310850304] [mod_pagespeed 1.9.32.3-4448 @21132] Waiting for
completion of URL http://exampledomain.com/example-slug/ for 45.001 sec
</code>

ALL requests got error, include image requests too.


My server hardware specs:
* Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz, 8 cores
* 32 GB DDR3 RAM
* 2 x 2 TB SATA 6 Gb/s 7200 rpm HDD (Software-RAID 1) Class Enterprise

Software specs:
Operating system: CentOS Linux 7.0.1406
Kernel: Linux 3.10.0-123.20.1.el7.x86_64 on x86_64

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9
PHP/5.6.5 mod_perl/2.0.9dev Perl/v5.16.3
Server MPM: event


What version of the product are you using (please check X-Mod-Pagespeed
header)?
mod-pagespeed-stable-1.9.32.3-4448.x86_64

URL of broken page:
I removed module after 1 minute of terror. If a google developer want to
learn more of server information, mail me an ip address that i can give
permission to look mod_info print.




--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[modpag...@googlecode.com]

Comment #1 on issue 1048 by hug...@betabrand.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I have the same problem with the same version of pagespeed.

I'm on Debian Wheezy.

[modpag...@googlecode.com]

Updates:
Status: Accepted

Comment #2 on issue 1048 by jef...@google.com: Waiting for completion of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

This looks like a bug to me in:

void ApacheFetch::Wait() {
MessageHandler* handler = server_context_->message_handler();
Timer* timer = server_context_->timer();
int64 start_ms = timer->NowMs();
{
ScopedMutex lock(mutex_.get());
while (!done_) {
condvar_->TimedWait(blocking_fetch_timeout_ms_);
if (!done_) {
int64 elapsed_ms = timer->NowMs() - start_ms;
handler->Message(
kWarning, "Waiting for completion of URL %s for %g sec",
mapped_url_.c_str(), elapsed_ms / 1000.0);
}
}
}
}

I think what we're supposed to be doing is blocking for a maximum of
blocking_fetch_timeout_ms_, but instead we block until Done() is called and
just log an error every blocking_fetch_timeout_ms_.

[modpag...@googlecode.com]

Comment #3 on issue 1048 by jef...@google.com: Waiting for completion of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

In the mean time, could you try turning off IPRO?

ModPagespeedInPlaceResourceOptimization off

[modpag...@googlecode.com]

Comment #4 on issue 1048 by hug...@betabrand.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I'll try that and report back.

Thanks.

[modpag...@googlecode.com]

Comment #5 on issue 1048 by jef...@google.com: Waiting for completion of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Turns out the logging behavior is intentional:

// Blocks indefinitely waiting for the proxy fetch to complete.
// Every 'blocking_fetch_timeout_ms', log a message so that if
// we get stuck there's noise in the logs, but we don't expect this
// to happen because underlying fetch/cache timeouts should fire.
//
// Note that enforcing a timeout in this function makes debugging
// difficult.
void Wait();

Still not clear why the fetch/cache timeouts aren't firing.

[modpag...@googlecode.com]

Comment #6 on issue 1048 by jef...@google.com: Waiting for completion of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Have you set any of the pagespeed timeouts to strange values, like -1? Are
there any other ways your PageSpeed configuration or server setup is
unusual? Or is this pretty much a stock install?

[modpag...@googlecode.com]

Comment #7 on issue 1048 by hug...@betabrand.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

The only place referring to timeouts in my config are:

ModPagespeedMemcachedTimeoutUs 50000
ModPagespeedRewriteDeadlinePerFlushMs 100

## Not a timeout, but maybe relevant
ModPagespeedImageMaxRewritesAtOnce 0

I also don't run into this problem for files that are declared with
ModPagespeedLoadFromFileMatch

[modpag...@googlecode.com]

Comment #8 on issue 1048 by hug...@betabrand.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Reporting back: disabling IPRO did not fix the issue.

[modpag...@googlecode.com]

Comment #9 on issue 1048 by hug...@betabrand.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I also noticed that:

1- contrary to what I said earlier, the bug also occurs for resources
declared with ModPagespeedLoadFromFileMatch

2- It seems to happen with .html files (which I feel like is weird; but
since I don't know the internals of MPS, could seem logical to you)

[Thu Feb 19 14:57:18 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @29090]

Waiting for completion of URL

http://www.betabrand.com/mens-gray-chambray-drawstring-pants.html for
6730.16 sec
[Thu Feb 19 14:57:18 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @29090]

Waiting for completion of URL

http://www.betabrand.com/womens/more/navy-blue-cornucopia-water-resistant-daypack.html
for 6730.14 sec
[Thu Feb 19 14:57:18 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @29090]

Waiting for completion of URL

http://www.betabrand.com/mens-removable-hood-gekko-vest.html for 6640.23 sec

[modpag...@googlecode.com]

Comment #10 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Hi there,

I'd like to mention that this bug is fairly important as it is triggered
all the time and its effect is apache2 hogging the CPU. The only fix is
killing the apache2 process.

Also, is it possible that using ModPagespeedBlockingRewriteKey could be a
cause of this bug?

I'm setting the X-PSA-Blocking-Rewrite header on all my requests.

[modpag...@googlecode.com]

Comment #11 on issue 1048 by CoroUrba...@gmail.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I have the same problem with the same version of pagespeed.

I'm on CENTOS 6.6 x86_64 standard, Apache 2.4.12.

[Sat Feb 28 15:21:10.939879 2015] [pagespeed:warn] [pid 49502:tid
140315103012608] [mod_pagespeed 1.9.32.3-4448 @49502] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2015/01/Mozart-La-Para-Ft-Anthony-Santos-Pa-Gozar-Remix.jpg
for 10 sec
[Sat Feb 28 15:21:11.526066 2015] [pagespeed:warn] [pid 49687:tid
140315144972032] [mod_pagespeed 1.9.32.3-4448 @49687] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/04/Paramba-ps-4.jpg for
40.002 sec
[Sat Feb 28 15:21:15.940041 2015] [pagespeed:warn] [pid 49502:tid
140315103012608] [mod_pagespeed 1.9.32.3-4448 @49502] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2015/01/Mozart-La-Para-Ft-Anthony-Santos-Pa-Gozar-Remix.jpg
for 15.001 sec
[Sat Feb 28 15:21:16.526242 2015] [pagespeed:warn] [pid 49687:tid
140315144972032] [mod_pagespeed 1.9.32.3-4448 @49687] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/04/Paramba-ps-4.jpg for
45.002 sec
[Sat Feb 28 15:21:21.526388 2015] [pagespeed:warn] [pid 49687:tid
140315144972032] [mod_pagespeed 1.9.32.3-4448 @49687] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/04/Paramba-ps-4.jpg for
50.002 sec
[Sat Feb 28 15:21:21.744521 2015] [pagespeed:warn] [pid 49819:tid
140315228890880] [mod_pagespeed 1.9.32.3-4448 @49819] Waiting for
completion of URL
http://www.corourbano.com/w/wp-content/uploads/2013/09/680px_dedd825780164515bdb1b16841c61be9.png
for 5 sec
[Sat Feb 28 15:21:26.526512 2015] [pagespeed:warn] [pid 49687:tid
140315144972032] [mod_pagespeed 1.9.32.3-4448 @49687] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/04/Paramba-ps-4.jpg for
55.002 sec
[Sat Feb 28 15:21:31.526660 2015] [pagespeed:warn] [pid 49687:tid
140315144972032] [mod_pagespeed 1.9.32.3-4448 @49687] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/04/Paramba-ps-4.jpg for
60.002 sec
[Sat Feb 28 15:21:36.526792 2015] [pagespeed:warn] [pid 49687:tid
140315144972032] [mod_pagespeed 1.9.32.3-4448 @49687] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/04/Paramba-ps-4.jpg for
65.002 sec
[Sat Feb 28 15:21:40.631322 2015] [core:error] [pid 49895:tid
140315355662080] [client 198.7.62.69:57760] Script timed out before
returning headers: wp-cron.php
[Sat Feb 28 15:22:26.233396 2015] [pagespeed:warn] [pid 49592:tid
140315228890880] [mod_pagespeed 1.9.32.3-4448 @49592] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/11/j-balvin-latin-grammy-slide-newsjpg.jpg
for 5 sec
[Sat Feb 28 15:22:44.002184 2015] [pagespeed:warn] [pid 49819:tid
140315176441600] [mod_pagespeed 1.9.32.3-4448 @49819] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/07/tu-quiere-chapiame-chimbala-ft-la-material.jpg
for 5 sec
[Sat Feb 28 15:22:49.002314 2015] [pagespeed:warn] [pid 49819:tid
140315176441600] [mod_pagespeed 1.9.32.3-4448 @49819] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/07/tu-quiere-chapiame-chimbala-ft-la-material.jpg
for 10 sec
[Sat Feb 28 15:22:54.002439 2015] [pagespeed:warn] [pid 49819:tid
140315176441600] [mod_pagespeed 1.9.32.3-4448 @49819] Waiting for
completion of URL
http://corourbano.com/w/wp-content/uploads/2014/07/tu-quiere-chapiame-chimbala-ft-la-material.jpg
for 15 sec

Any suggestions on this?

[modpag...@googlecode.com]

Comment #12 on issue 1048 by huynhvut...@gmail.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I also have this problem, anyone can solve ?

[modpag...@googlecode.com]

Updates:
Owner: jef...@google.com

Comment #13 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I suspect (but don't know for sure) that use of
ModPagespeedBlockingRewriteKey might not interact well with in-place
fetches. It would be better to set a longer rewrite deadline.

Nevertheless I think this is a real bug and we should figure out who should
tackle it.

[modpag...@googlecode.com]

Comment #14 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Is everybody on this thread with the bug using
ModPagespeedBlockingRewriteKey though?

[modpag...@googlecode.com]

Comment #15 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Looking at the code, the class in question (ApacheFetch) is used in three
different scenarios:

1. admin-page handling (which I think is not at issue here based on the log
messages)
2. ModPagespeedInPlaceResourceOptimization (on by default starting in 1.9)
3. ModPagespeedMapProxyDomain

It would be helpful if each person experiencing this bug let us know if
they are using any of the above features, specifying with or without a
BlockingRewriteKey.

[modpag...@googlecode.com]

Comment #16 on issue 1048 by lambert....@gmail.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Someone found the problem? I'm having this and I'm trying to find out.

[modpag...@googlecode.com]

Comment #17 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

In order to help finding the source of this bug it would help if you posted
the information stated in comment #15:
https://code.google.com/p/modpagespeed/issues/detail?id=1048#c15

[modpag...@googlecode.com]

Comment #18 on issue 1048 by foreve...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I have the same problem with an Ubuntu-server 13.10

[Tue Mar 31 14:41:20.129694 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 345.023 sec
[Tue Mar 31 14:41:25.129988 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 350.023 sec
[Tue Mar 31 14:41:30.130250 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 355.024 sec
[Tue Mar 31 14:41:35.130507 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 360.024 sec
[Tue Mar 31 14:41:40.130767 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 365.024 sec
[Tue Mar 31 14:41:45.131062 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 370.025 sec
[Tue Mar 31 14:41:50.131376 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 375.025 sec
[Tue Mar 31 14:41:55.131638 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 380.025 sec
[Tue Mar 31 14:42:00.131922 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 385.025 sec
[Tue Mar 31 14:42:05.132191 2015] [pagespeed:warn] [pid 1166]
[mod_pagespeed 1.9.32.3-4448 @1166] Waiting for completion of URL
http://cometeelmundo.net/sites/default/files/styles/originalwatermark/public/field/image/Interior%20catedral%20de%20San%20Basilio.jpg?itok=TKiOM8ek
for 390.026 sec

[modpag...@googlecode.com]

Comment #19 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

It would be helpful if each person experiencing this bug let us know if

they are using either of the these features:

- ModPagespeedInPlaceResourceOptimization (on by default starting in 1.9)
- ModPagespeedMapProxyDomain

Please also let us know if you also use a BlockingRewriteKey.

[modpag...@googlecode.com]

Comment #20 on issue 1048 by foreve...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

If I understand correctly ModPagespeedInPlaceResourceOptimization is
enabled by default on 1.9 without specifying it on the config file, so in
this case I have it enabled.

I'm not using ModPagespeedMapProxyDomain and netiher a BlockingRewriteKey.

If you need more information feel free to ask for it.

[modpag...@googlecode.com]

Comment #21 on issue 1048 by vedio...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I have this problem as well. I regularly need to restart the Apache2
service because I use 100% CPU. It happens almost twice a day. It's a
problem because the rest of the time mod_pagespeed benefits are real and
great....

[modpag...@googlecode.com]

Comment #22 on issue 1048 by jef...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

We're still trying to figure out what's causing this, but a workaround for
now is to disable in-place resource optimization:

ModPagespeedInPlaceResourceOptimization off

[modpag...@googlecode.com]

Comment #23 on issue 1048 by vedio...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

ModPagespeedInPlaceResourceOptimization off didn't stop my CPU to go
high today. I had to restart the service again.

A few examples of what I have in my apache2 error logs (no idea if it is
related) :
=> tail -F /var/log/apache2/error.log



[Wed Apr 01 13:04:40.539741 2015] [pagespeed:error] [pid 6440]
[mod_pagespeed 1.9.32.3-4448 @6440]
http://www.xxxxxx.ch/image/ico_delete_small.gif (connecting to:%22):0:
Error status=670002 (Name or service not known) serf_connection_create2
[Wed Apr 01 13:04:40.539799 2015] [pagespeed:warn] [pid 6440]
[mod_pagespeed 1.9.32.3-4448 @6440] Fetch failed to start:
http://www.xxxxxx.ch/image/ico_delete_small.gif (connecting to:%22)
[Wed Apr 01 13:27:42.192734 2015] [pagespeed:warn] [pid 8212]
[mod_pagespeed @8212] [0401/132742:WARNING:queued_worker_pool.cc(432)]
Canceling 68 functions on sequence Shutdown

[modpag...@googlecode.com]

Comment #24 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

vediovis: do you have MapProxyDomain in your config? Or
BlockingRewriteKey? Did you flush your cache after turning off
InPlaceResourceOptimization?

foreveryo: did you try flushing your cache and turning off
InPlaceResourceOptimization?

[modpag...@googlecode.com]

Comment #25 on issue 1048 by vedio...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

jmara: I do not use MapProxyDomain or BlockingRewriteKey
How can I flush the cache?

The CPU goes high all the time now. It seems the longer the installation,
the higher frequence of high CPU usage.

[modpag...@googlecode.com]

Comment #26 on issue 1048 by vedio...@gmail.com: Waiting for completion of

[modpag...@googlecode.com]

Comment #27 on issue 1048 by foreve...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I have set ModPagespeedInPlaceResourceOptimization off and cleared the
cache (I am using memcached) I will come back in a few hours to let us know
how is working.

[modpag...@googlecode.com]

Comment #28 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

To clear cache, by default you do:
sudo touch $FILE_CACHE_PATH/cache.flush

Details are
here:https://developers.google.com/speed/pagespeed/module/system#flush_cache

Note that it is not sufficient to restart memcached, because there is also
an in-memory L1 cache that will not be flushed when you do that.

[modpag...@googlecode.com]

Comment #29 on issue 1048 by vedio...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

jmara : I use OPcache from Zend as well. Do you need my configuration file ?

[modpag...@googlecode.com]

Comment #30 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I was mainly concerned about the mod_pagespeed cache. Other caches
upstream or downstream of mod_pagespeed should not matter.

[modpag...@googlecode.com]

Comment #31 on issue 1048 by m.moko...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Any updates on this issue? We are experiencing those errors on the logs
with very minor configuration differences than the default configuration.

Those are the changes we have made (comparing the original and our files):

ModPagespeedEnableFilters collapse_whitespace,elide_attributes,trim_urls
ModPagespeedEnableFilters remove_quotes,remove_comments

ModPagespeedDomain **********9.cloudfront.net
ModPagespeedDomain **********h.cloudfront.net
ModPagespeedDomain **********k.cloudfront.net

The errors we are getting:
[Thu Apr 02 08:04:51 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @13247]
Waiting for completion of URL http://www.mydomain.com/mydir/images/068.JPG
for 75.002 sec
[Thu Apr 02 08:04:51 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @6814]
Waiting for completion of URL http://www.mydomain.com/mydir/images/a1.jpg
for 75.002 sec
[Thu Apr 02 08:04:51 2015] [warn] [mod_pagespeed 1.9.32.3-4448 @4789]
Waiting for completion of URL http://www.mydomain.com/mydir/images/062.JPG
for 75.002 sec

If you need anything else, please let me know.

[modpag...@googlecode.com]

Comment #32 on issue 1048 by m.moko...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

BTW, forgot to mention that if you request the files (images in our case)
directly through a browser (chrome) they are downloaded in less than a
second. Not sure why it takes page speed so long to fetch them.

[modpag...@googlecode.com]

Comment #33 on issue 1048 by vedio...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Back from week end to see the server high loaded at 100% CPU since last
saturday. Bad.... I will have no choice but to uninstall the module. It's a
shame, because the benefits are real when it works. But I cannot have my
server going wild everytime I have to leave for a few hours so I have no
choice.

1/ images url suddenly were rewritten again, despite
ModPagespeedImagePreserveURLs on . It is wrong, I don;t want my image names
to be updated. The website exists for quite a long time, images must be
kept at the same url / name as before.

2/ I found new error logs : if it can help...
[Mon Apr 06 16:15:45.788929 2015] [pagespeed:error] [pid 24728]
[mod_pagespeed 1.9.32.3-4448 @24728] http://xxxxxx.ch/FrancaisAnim.gif
(connecting to:images):0: Error status=670002 (Name or service not known)
serf_connection_create2
[Mon Apr 06 16:15:45.789001 2015] [pagespeed:warn] [pid 24728]
[mod_pagespeed 1.9.32.3-4448 @24728] Fetch failed to start:
http://xxxxxx.ch/FrancaisAnim.gif (connecting to:images)
[Mon Apr 06 21:40:16.096440 2015] [pagespeed:warn] [pid 17239]
[mod_pagespeed 1.9.32.3-4448 @17239] Rewrite
http://xxxxxx.ch/fichiers/jpg/699.jpg failed while fetching
http://xxxxxx.ch/fichiers/jpg/x699.jpg.pagespeed.ic.EH8_oOdmdp.webp
[Mon Apr 06 21:40:16.099784 2015] [pagespeed:warn] [pid 17497]
[mod_pagespeed 1.9.32.3-4448 @17497] Rewrite
http://xxxxxx.ch/fichiers/jpg/716.jpg failed while fetching
http://xxxxxx.ch/fichiers/jpg/x716.jpg.pagespeed.ic.XhGfIr9bpq.webp
[Tue Apr 07 04:35:49.299849 2015] [pagespeed:error] [pid 11528]
[mod_pagespeed 1.9.32.3-4448 @11528]
http://xxxxxx.ch/web/images/FrancaisAnim.gif (connecting to:magmi):0: Error

status=670002 (Name or service not known) serf_connection_create2

[Tue Apr 07 04:35:49.299906 2015] [pagespeed:warn] [pid 11528]
[mod_pagespeed 1.9.32.3-4448 @11528] Fetch failed to start:
http://xxxxxx.ch/web/images/FrancaisAnim.gif (connecting to:magmi)

Thanks !

S.L.

[modpag...@googlecode.com]

Comment #34 on issue 1048 by foreve...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

After disabling InPlaceResourceOptimization I'm not having any major issues.

[modpag...@googlecode.com]

Comment #35 on issue 1048 by morlov...@google.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Other things that may shed some light:
1) Is everyone with the problem using memcached?
2) Any mentions of cache errors or the like in logs?

@other devs:
I am sort of wondering if we're deadlocking the CacheBatcher thread
again[1], as cache callback not getting invoked seems like the most obvious
reason we would get stuck for .html at least (though interestingly not all
reports mention .html --- there could be more than 1 issue, too...)...

Unfortunately attempts to reproduce this with loadtests failed...

[1] Previous episode is:
https://code.google.com/p/modpagespeed/source/detail?r=3674
We are too early to have that same scenario, though.

[modpag...@googlecode.com]

Comment #36 on issue 1048 by vedio...@gmail.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

When I have a apache process at 100% CPU for minutes, how can I find what
it is doing internally? It may help...

[modpag...@googlecode.com]

Comment #37 on issue 1048 by morlov...@google.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Maybe... If it's say, pid 12345, the command:

gdb --pid 12345 --eval-command='info proc mappings' --eval-command='thread
apply all bt' --eval-command=quit

will print out what modules are loaded at what address
(if you paste it, it's probably best to only include info on
mod_pagespeed.so or mod_pagespeed_ap24.so...) and what thread is doing
what. It may be hard to interpret, though since the modules in packages are
built release, so all it will have is addresses.

[modpag...@googlecode.com]

Comment #38 on issue 1048 by jcro...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

this issue has been migrated to github:
https://github.com/pagespeed/mod_pagespeed/issues/1048

[modpag...@googlecode.com]

Comment #39 on issue 1048 by morlov...@google.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Note that I'll still pay attention to information on here --- this bug is
too important to do otherwise, and we badly need information.

[modpag...@googlecode.com]

Comment #40 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Since last time I reported about this, I added the following config:

# Re-enable IPRO, it used to be off, but according to my
previous bug report, IPRO off did not fix the issue.
ModPagespeedInPlaceResourceOptimization on

## Disable Filters
ModPagespeedDisableFilters defer_javascript
ModPagespeedDisableFilters add_head
ModPagespeedDisableFilters
rewrite_style_attributes_with_url
ModPagespeedDisableFilters lazyload_images

ModPagespeedCriticalImagesBeaconEnabled false

I've been running MPS with these settings for a couple of hours and haven't
run into the issue.

Maybe the bug occurs because of one of the 4 filters listed above, or
because of ModPagespeedCriticalImagesBeaconEnabled.

[modpag...@googlecode.com]

Comment #41 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

The problem re-occured. My previous comment is not a good lead.

[modpag...@googlecode.com]

Comment #42 on issue 1048 by jcro...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

are you using MapProxyDomain?

This code path should really only be possible with MapProxyDomain or
InPlaceResourceOptimization.

[modpag...@googlecode.com]

Comment #43 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

I only use IPRO in my config, no MapProxyDomain.

Next time the bug occurs, I'll try running the gdb command previously
suggested.

[modpag...@googlecode.com]

Comment #44 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Bug currently occurring, here's the gdb output, anything else you'd like me
to run?

Attachments:
mod_pagespeed.log 73.8 KB

[modpag...@googlecode.com]

Comment #45 on issue 1048 by jmara...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Hughes -- can you confirm which OS & bit-size you are using, and that your
binary is from our distribution?

[modpag...@googlecode.com]

Comment #46 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64 GNU/Linux

I use the regular debian package. Currently on the latest version
available: 1.9.32.3-r4448

[modpag...@googlecode.com]

Comment #47 on issue 1048 by jef...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Thanks so much for the gdb output! We're working on symbolizing it, at
which point it should be really helpful.

[modpag...@googlecode.com]

Comment #48 on issue 1048 by morlov...@google.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

And here is decoded output + script (which has an hack to suppress
addr2line being clueless about filenames).

It still seems like ipro is on, BTW.


Attachments:
decoded.txt 42.8 KB
symbolize.rb 756 bytes

[modpag...@googlecode.com]

Comment #49 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Yes, IPRO is on. (see comment #40)

[modpag...@googlecode.com]

Comment #50 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Here's another, slightly different (see Thread 1), stack trace.



Attachments:
mod_pagespeed_stack3 74.2 KB

[modpag...@googlecode.com]

Comment #51 on issue 1048 by morlov...@google.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Thanks. One more thing that may be worth trying:

gdb --pid 12345
(with PID of tasks that's eating cpu instead of 12345)

bt <enter>
c <enter>
Ctrl+C
bt <enter>
c <enter>
Ctrl+C

(repeat a few times)
then quit <enter>

That would roughly show where it's spending the time. My suspicion at this
point is that we are screwing up in memcached client code, but it's hard to
see how that would happen in a CPU-burning way.

[modpag...@googlecode.com]

Comment #52 on issue 1048 by jef...@google.com: Waiting for completion of
URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

For debugging this, it would be useful to look at the pagespeed
configuration files for the different sites that are having this problem.
If this problem has happened to you, could you send your server
configuration file to jef...@google.com?

[modpag...@googlecode.com]

Comment #53 on issue 1048 by hug...@betabrand.com: Waiting for completion
of URL
https://code.google.com/p/modpagespeed/issues/detail?id=1048

Here's an interesting thing I noticed:

In a previous comment (#7) I mentionned that the bug did not occur for
resources declared with ModPagespeedLoadFromFileMatch. Then in comment #9,
I backed out of this statement, saying that the bug still did occur.

Fast forward to a few days ago: I added a new ModPagespeedLoadFromFileMatch
configuration.

Since that day, the bug has not occurred once, which is unfortunate because
I really wanted to send the output I would get by following comment #51.

It could be completely coincidental, but I thought I'd be worth mentioning.