Pagespeed crashing nginx 1.9.11

[MrBitbuck (DE)]

Hi there,

i noticed several bugs with the latest Release of Pagespeed with Nginx 1.9.11

First bug is really Strange.
If i try to open a mp3 file nginx is completely crashing with uknown error code.
The only i can see about in the niginx logfile is this here (sorry not more about the mp3, thats all he is producing)

2016/02/22 07:40:03 [error] 14682#17566: [ngx_pagespeed 1.10.33.5-0] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
2016/02/22 07:40:03 [error] 14682#17566: [ngx_pagespeed 1.10.33.5-0] URL https://wow.gnetwork.eu/assets/themes/gnetwork/css/ilightbox/closedhand.cur (connecting to:85.14.244.243:80) active for 12 ms
2016/02/22 07:40:03 [error] 14682#17566: [ngx_pagespeed 1.10.33.5-0] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
2016/02/22 07:40:03 [error] 14682#17566: [ngx_pagespeed 1.10.33.5-0] URL https://wow.gnetwork.eu/assets/themes/gnetwork/css/ilightbox/closedhand.cur (connecting to:85.14.244.243:80) active for 12 ms

Meanwhile the Webserver is responding with a unknown Error Code to all requests on every site.

Request

GET /assets/uploads/2015/10/DC_GeneralWalkUni02.mp3 HTTP/1.1
Host: wow.gnetwork.eu
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Accept: audio/webm,audio/ogg,audio/wav,audio/*;q=0.9,application/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5
Accept-Language: de,en-US;q=0.7,en;q=0.3
Range: bytes=0-
Referer: https://wow.gnetwork.eu/soundfiles/musik/zonen/dalaran/
Connection: keep-alive

And as Reponse the Crash with unkown error
The only way currently to fix it is to completly disable pagespeed nginx

I let the Error Live so that you can see it.. example site
https://wow.gnetwork.eu/soundfiles/musik/stadtmusik/stadtmusik-darnassus/

or directly https://wow.gnetwork.eu/assets/uploads/2015/10/Darnassus-Walking-1.mp3


The second Bug is also hard to debug.
Pagespeed is kind of destroying suddenly some of my javascripts. (i think its because of some defer). Also this bug appeard with the latest version for Nginx 1.9.11.
I let the issue now live so that you can see the Console Error Message

PSA ERROR: Exception while evaluating.jQuery(...).panelslider is not a function

on: https://wow.gnetwork.eu/team/

What i do to fix it..
I added again jquery, but worked only one time till pagespeed react on my new jquery embed


Second i tried to disable the javascript folders to stop him from file handling with

pagespeed Disallow "*/assets/plugins/dvteam/js/*";
pagespeed Disallow "*/assets/plugins/dvteam/js/mini.js*";

But that doesn't had any effect

so i tried it with

<script data-pagespeed-no-defer src="https://wow.gnetwork.eu/assets/plugins/dvteam/js/mini.js"></script>

But also with no effect

Yust with the disable rule

pagespeed Disallow "*/team/*";

all was working again.

[MrBitbuck (DE)]

Time for some Debug Logs (Needed to move them to Pastebin)

First Request to a mp3

[Mon, 22 Feb 2016 07:44:36 GMT] [Info] [3528] Trying to serve rewritten resource in-place: https://wow.gnetwork.eu/assets/uploads/2015/10/Darnassus-Walking-1.mp3

[Mon, 22 Feb 2016 07:44:36 GMT] [Info] [3528] Could not rewrite resource in-place because URL is not in cache: https://wow.gnetwork.eu/assets/uploads/2015/10/Darnassus-Walking-1.mp3

Result: Instant Crash

After some seconds ... pagespeed seems to do nothing (no more logs for some seconds)
After some seconds more.. new log entrys (like nothing happend)
http://pastebin.com/uRhb7XAs

[MrBitbuck (DE)]

And here the Nginx Debug Log

2016/02/22 09:05:48 [notice] 13386#13386: *53 "/wp-admin$" does not match "/assets/uploads/2015/10/Darnassus-Walking-1.mp3", client: 79.230.144.149, server: wow.gnetwork.eu, request: "GET /assets/uploads/2015/10/Darnassus-Walking-1.mp3 HTTP/1.1", host: "wow.gnetwork.eu"
2016/02/22 09:05:48 [notice] 13386#13386: *53 "^/assets/cache/minify.*/w3tc_rewrite_test$" does not match "/assets/uploads/2015/10/Darnassus-Walking-1.mp3", client: 79.230.144.149, server: wow.gnetwork.eu, request: "GET /assets/uploads/2015/10/Darnassus-Walking-1.mp3 HTTP/1.1", host: "wow.gnetwork.eu"
2016/02/22 09:05:48 [notice] 13386#13386: *53 "(.*)" matches "/assets/uploads/2015/10/Darnassus-Walking-1.mp3", client: 79.230.144.149, server: wow.gnetwork.eu, request: "GET /assets/uploads/2015/10/Darnassus-Walking-1.mp3 HTTP/1.1", host: "wow.gnetwork.eu"
2016/02/22 09:05:48 [notice] 13386#13386: *53 rewritten data: "/assets/uploads/2015/10/Darnassus-Walking-1.mp3", args: "", client: 79.230.144.149, server: wow.gnetwork.eu, request: "GET /assets/uploads/2015/10/Darnassus-Walking-1.mp3 HTTP/1.1", host: "wow.gnetwork.eu"
2016/02/22 09:05:48 [info] 13386#13386: [ngx_pagespeed 1.10.33.5-0] Trying to serve rewritten resource in-place: https://wow.gnetwork.eu/assets/uploads/2015/10/Darnassus-Walking-1.mp3
2016/02/22 09:05:48 [info] 13386#13386: [ngx_pagespeed 1.10.33.5-0] Could not rewrite resource in-place because URL is not in cache: https://wow.gnetwork.eu/assets/uploads/2015/10/Darnassus-Walking-1.mp3
2016/02/22 09:05:48 [info] 13386#13386: *53 client prematurely closed connection (104: Connection reset by peer) while sending response to client, client: 79.230.144.149, server: wow.gnetwork.eu, request: "GET /assets/uploads/2015/10/Darnassus-Walking-1.mp3 HTTP/1.1", host: "wow.gnetwork.eu"

[MrBitbuck (DE)]

Uh ahm the above ip adress seems sadly not to be mine (beginning with 85. - can you please remove it due privacy reason)?

[Otto van der Schaaf]

On Mon, Feb 22, 2016 at 9:19 AM, 'MrBitbuck (DE)' via ngx-pagespeed-discuss <ngx-pagesp...@googlegroups.com> wrote:

Uh ahm the above ip adress seems sadly not to be mine (beginning with 85. - can you please remove it due privacy reason)?

I'm not able to edit or remove this post, so I can't help you with that, sorry.

For the problem with the mp3 file (https://wow.gnetwork.eu/assets/uploads/2015/10/Darnassus-Walking-1.mp3) - with crash, do you mean nginx's worker process core dumps? Or does crash mean you get the cloudflare error page?

Do you have aio / threading set up in nginx.conf?

For the problem with the javascript, did that work ok with an earlier ngx_pagespeed release?

Otto

[MrBitbuck (DE)]

I'm not really sure what exactly happens with nginx because the log give out not enough information (also as debug log). I yust can say.. if someone clicking on a mp3 all nginx worker will instantly drop all connections and from what i see restarting all the worker processes.

Do you have aio / threading set up in nginx.conf?

Yes currently i use 4 worker processes.

[MrBitbuck (DE)]

Uhm sorry forget to answer,

yes the script was working before the latest update.

[Otto van der Schaaf]

Could you try turning off in-place resource optimization in nginx.conf to see if that fixes the problem with mp3 files?:

pagespeed InPlaceResourceOptimization off;

Also, could you send me the debug log  so I can have a look what happens when an mp3 is requested (with in-place resource optimization still on)?

Otto

[MrBitbuck (DE)]

Hi Otto,

after i disabled InPlaceResourceOptimization the error went away with the mp3.
Sorry for the stupid question but, is it possible to log the message log to file? I didn't found something about in the FAQ.
i mean i could copy it out of the log viewer but due the server shutdown it would be great to have it within a seperate logfile

Currently i got only

pagespeed LogDir /var/log/pagespeed;
pagespeed StatisticsLogging on;
pagespeed Statistics on;
pagespeed StatisticsLoggingIntervalMs 60000;
pagespeed StatisticsLoggingMaxFileSizeKb 1024;
pagespeed MessageBufferSize 100000;

[MrBitbuck (DE)]

This is the log i copied out from the admin console. I hit the mp3 several times

http://pastebin.com/raw/iguYZL10

[Otto van der Schaaf]

Hi,

Those messages should also be logged to nginx's error.log, see

http://nginx.org/en/docs/ngx_core_module.html#error_log

If you add --with-debug to ./configure when you build nginx there will be a lot more messages.

Otto

[MrBitbuck (DE)]

Hi Otto,

i compiled it in Debug Mode here is the nginx debug error log. I hope it might be somehow useful
https://media.gnitter.de/error.log

i clicked a few times on Darnassus-Walking-1.mp3 and let him write the logfiles.
(you can find that lines if you seek for Darnassus-Walking-1.mp3)

[Otto van der Schaaf]

update - we've been debugging this off-list.

The probable cause for the problems with mp3 responses seems to be a bad Cache-Control header [1] that is being emitted when IPRO is turned on (in the recording phase), which makes CloudFlare reject the response.

The source of this header has not been pinpointed yet, though with IPRO off it doesn't get emitted.

[1] Cache-Control: ?ax-age=300

Otto

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Otto van der Schaaf]

We have been able to find the cause for the bad cache-control value mentioned above, and the server involved is running with a change that fixes the problem.

I created https://github.com/pagespeed/ngx_pagespeed/issues/1138 to track the fix.

Otto

[Jeff Kaufman]

Just to confirm: PageSpeed wasn't crashing nginx, but instead was
corrupting the cache-control header with a memory lifetime issue, and
this was making cloudflare show an error to the user?

[Otto van der Schaaf]

On Mon, Feb 29, 2016 at 3:08 PM, 'Jeff Kaufman' via ngx-pagespeed-discuss <ngx-pagesp...@googlegroups.com> wrote:

Just to confirm: PageSpeed wasn't crashing nginx, but instead was
corrupting the cache-control header with a memory lifetime issue, and
this was making cloudflare show an error to the user?

That is an accurate recap, yes.

It looked like a crash, because CloudFlare would report errors for all further requests from the same browser for a while after receiving the bad response from the origin. I have had nginx running under gdb while reproducing this to double check.

Otto

[Jeff Kaufman]

Ok, thanks. It does look like the memory lifetime issue could cause
pagespeed to crash the nginx worker though.

[Otto van der Schaaf]

On Mon, Feb 29, 2016 at 7:50 PM, 'Jeff Kaufman' via ngx-pagespeed-discuss <ngx-pagesp...@googlegroups.com> wrote:

Ok, thanks.  It does look like the memory lifetime issue could cause
pagespeed to crash the nginx worker though.

Though that did not happen in the instance I looked at, I agree is has that potential..