4xx error code rewrite

389 views
Skip to first unread message

John Mase

unread,
Jul 19, 2016, 7:56:36 AM7/19/16
to ngx-pagespeed-discuss
my images doesnt seem to get rewritten.on webpage tools 

<img src='forum/screenshots/monthly_2016_07/news_large_J_STARS_shokai_JK.thumb.jpg.39464ddc12b37ea28a89252946d2e052.jpg' alt='' data-pagespeed-url-hash="2372681252" onload="pagespeed.CriticalImages.checkImageForCriticality(this);"><!--deadline_exceeded for filter CacheExtender--><!--4xx status code, preventing rewriting of https:///forum/screenshots/monthly_2016_07/news_large_J_STARS_shokai_JK.thumb.jpg.39464ddc12b37ea28a89252946d2e052.jpg--> </a><!--deadline_exceeded for filter CssFilter-->

access log shows nothing significant. i do restrict 30 connections per ip. doubt that has something to do with it

pagespeed on;
  pagespeed EnableCachePurge on;
pagespeed FileCachePath "/dev/shm/ngx_pagespeed/";
pagespeed HttpCacheCompressionLevel 9;
pagespeed RewriteLevel PassThrough;
pagespeed FileCacheSizeKb 3048576;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed CreateSharedMemoryMetadataCache "/dev/shm/ngx_pagespeed/" 65536;
pagespeed MessageBufferSize 100000;

pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;
    pagespeed LoadFromFile https://domain.me/cdn /home/domains/domain.me/public_html/cdn;


pagespeed Domain https://cdn.domain.me;
pagespeed Domain https:/domain.me;
pagespeed MapRewriteDomain https://cdn.domain.me https://domain.me/cdn;



#js
pagespeed JsInlineMaxBytes 2048;
pagespeed JsOutlineMinBytes 3000;
pagespeed EnableFilters rewrite_javascript;
pagespeed EnableFilters outline_javascript;
pagespeed EnableFilters inline_javascript;

#css
pagespeed CssImageInlineMaxBytes 0;
pagespeed CssInlineMaxBytes 2048;
pagespeed CssOutlineMinBytes 2050;
pagespeed EnableFilters fallback_rewrite_css_urls;
pagespeed EnableFilters rewrite_css;
pagespeed EnableFilters inline_css;
pagespeed EnableFilters outline_css;
pagespeed EnableFilters move_css_above_scripts;



#images
pagespeed ImageInlineMaxBytes 1000000;
pagespeed ImageMaxRewritesAtOnce 20;
pagespeed EnableFilters rewrite_images;
pagespeed EnableFilters resize_rendered_image_dimensions;
pagespeed EnableFilters convert_gif_to_png;
pagespeed EnableFilters convert_png_to_jpeg;
pagespeed EnableFilters convert_jpeg_to_progressive;
pagespeed EnableFilters convert_jpeg_to_webp;
pagespeed EnableFilters insert_image_dimensions;


#misc
pagespeed EnableFilters rewrite_style_attributes_with_url,convert_meta_tags;
pagespeed EnableFilters make_google_analytics_async;
pagespeed EnableFilters collapse_whitespace;
pagespeed EnableFilters extend_cache;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters remove_comments;
pagespeed EnableFilters inline_google_font_css;
'
 

John Mase

unread,
Jul 19, 2016, 8:24:13 AM7/19/16
to ngx-pagespeed-discuss
so i amended my config file but i still see the same error msg. at this time, i dont use cdn anymore to see if there are any changes.

i use this now 
    pagespeed LoadFromFileMatch "^https?://domain.me/" "/home/yuon/domains/domain.me/public_html/";
pagespeed LoadFromFileRuleMatch disallow .*;
pagespeed LoadFromFileRuleMatch allow \.css$;
pagespeed LoadFromFileRuleMatch allow \.jpe?g$;
pagespeed LoadFromFileRuleMatch allow \.png$;
pagespeed LoadFromFileRuleMatch allow \.gif$;
pagespeed LoadFromFileRuleMatch allow \.js$;

<a href='https://domain.me/forum/files/file/71083-asparagus-mont-blanc-320k/' title='More information about "asparagus- mont blanc 320k"' style='background-image:url(https://domain.me/forum/screenshots/monthly_2016_07/578bd8de4abbe_ASPARAGUS-MONTBLANC.thumb.JPG.c32f3b9dd0e1a2b87cc2226fd961dd51.JPG)' class=' ipsThumb ipsThumb_medium ipsThumb_bg'> 

<img src='https://domain.me/forum/screenshots/monthly_2016_07/578bd8de4abbe_ASPARAGUS-MONTBLANC.thumb.JPG.c32f3b9dd0e1a2b87cc2226fd961dd51.JPG' alt='' data-pagespeed-url-hash="2076280510" onload="pagespeed.CriticalImages.checkImageForCriticality(this);">

Otto van der Schaaf

unread,
Jul 19, 2016, 8:41:08 AM7/19/16
to ngx-pagesp...@googlegroups.com
You also had a (weird looking) MapOriginDomain line in the previous configuration, do you still have that?
Having LoadFromFile set up should bypass the rate limiting you have in place.

I think the 4XX you are seeing in the debug comments is remembered for five minutes before ngx_pagespeed will attempt to refetch. 
To force ngx_pagespeed to refresh right away you can purge it's cache: https://developers.google.com/speed/pagespeed/module/system#flush_cache
(I'm not sure that updating MapOriginDomain/LoadFromFile configuration invalidates the right cache entries here, but I think you need to do this)

Lastly, I think that matching will be case-sensitive. It looks like the extension you want in the example you posted is JPG (and not jpg).

Otto

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.
Message has been deleted

John Mase

unread,
Jul 19, 2016, 9:24:48 AM7/19/16
to ngx-pagespeed-discuss
hi,  i removed maporigin domain and only kept this.

pagespeed MapRewriteDomain https://cdn.domain.me(serving cdn resources) https://domain.me/cdn; (origin domain)
 i moved my static file back to cdn (cloudflare)
i get 4xx status code on css,js, jpg and png




ive cleared pagespeed cache and i see this
<hr class='ipsHr'> <div id='elDownloadStatsLatest' class='ipsClearfix'> <span style='background-image:url(https://cdn.domain.me/monthly_2016_07/cover.jpg.f77619403bb835c4c979788d4477651f.jpg)' class='ipsPos_left ipsThumb ipsThumb_small ipsThumb_bg'> <img src='https://cdn.domain.me/monthly_2016_07/cover.jpg.f77619403bb835c4c979788d4477651f.jpg' alt='' data-pagespeed-url-hash="1778898585" onload="pagespeed.CriticalImages.checkImageForCriticality(this);"><!--4xx status code, preventing rewriting of https://cdn.domain.me/monthly_2016_07/cover.jpg.f77619403bb835c4c979788d4477651f.jpg--><!--4xx status code, preventing rewriting ofhttps://cdn.domain.me/monthly_2016_07/cover.jpg.f77619403bb835c4c979788d4477651f.jpg--> </span><!--4xx status code, preventing rewriting of https://cdn.domain.me/monthly_2016_07/cover.jpg.f77619403bb835c4c979788d4477651f.jpg-->

my current pagespeed config
pagespeed on;
  pagespeed EnableCachePurge on;
pagespeed FileCachePath "/dev/shm/ngx_pagespeed/";
pagespeed HttpCacheCompressionLevel 9;
pagespeed RewriteLevel PassThrough;
pagespeed FileCacheSizeKb 3048576;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed CreateSharedMemoryMetadataCache "/dev/shm/ngx_pagespeed/" 65536;
pagespeed MessageBufferSize 100000;
pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed GlobalStatisticsPath /ngx_pagespeed_global_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;
pagespeed GlobalAdminPath /pagespeed_global_admin;
    pagespeed LoadFromFileMatch "^https?://domain.me/" "/home/user/domains/domain.me/public_html/";
pagespeed LoadFromFileRuleMatch disallow .*;
pagespeed LoadFromFileRuleMatch allow \.css$;
pagespeed LoadFromFileRuleMatch allow \.jpe?g$;
pagespeed LoadFromFileRuleMatch allow \.png$;
pagespeed LoadFromFileRuleMatch allow \.gif$;
pagespeed LoadFromFileRuleMatch allow \.js$;


John Mase

unread,
Jul 19, 2016, 10:19:54 AM7/19/16
to ngx-pagespeed-discuss
im also getting deadline exceeded not sure what i did wrong. appreciate any input thanks. ive uploaded html page of my site with debug msg enabled,, hope you can shed some light

page.rtf

Otto van der Schaaf

unread,
Jul 19, 2016, 11:01:13 AM7/19/16
to ngx-pagesp...@googlegroups.com
I'm curious, how did things look optimization-wise before you added the CDN rewriting back in?

The html you attached has a pretty large html body. Some assets are getting optimized (and have .pagespeed. in their url), and others have deadline or 4XX complaints written after them.  On the deadline complaints: Pagespeed will wait for 10 ms for optimizations to complete before it moves on and starts writing out html. 
I expect those deadline complaints to go away after a few refreshes (or if fetching/optimization fails they will be replaced by a different complaint). 

Some resources from cdn.mydomain.me are getting optimized, which makes me think the rewriting/origin mapping is working.

If rate limiting applies when pagespeed fetches its inputs over http, that may also cause deadline complaints (and perhaps 4XX responses, depending on how the rate limiting is set up precisely).

Is there a rate limiter interfering?

Otto



On Tue, Jul 19, 2016 at 4:19 PM John Mase <maid...@gmail.com> wrote:
im also getting deadline exceeded not sure what i did wrong. appreciate any input thanks. ive uploaded html page of my site with debug msg enabled,, hope you can shed some light

Message has been deleted

John Mase

unread,
Jul 19, 2016, 12:40:18 PM7/19/16
to ngx-pagespeed-discuss
i still get the same erro when i dont use cdn. ive attached my html page after clearing cache 40 minutes. this is without using cdn.
im running a forum from invision power suite between if it has anything relevent
my html page

my server dont have rate limit. nor does cloudflare but for sake of testing it i moved my static fileds back to my server- but my pagespeed config shouoldnt be the reason why it cant rewrit ersome image and css js files?
Message has been deleted

John Mase

unread,
Jul 19, 2016, 7:05:42 PM7/19/16
to ngx-pagespeed-discuss
i was wrong, after using keycdn and they insisted they werent dropping connection on their end so something seems to be wrong on my server end this error msg applies on every files it serves, js,css etc

<!--4xx status code, preventing rewriting of

maybe this config isnt enough for using cdn url to do rewrite? pagespeed message history doesnt show any error indication

Otto van der Schaaf

unread,
Jul 20, 2016, 10:14:45 AM7/20/16
to ngx-pagespeed-discuss

<!--4xx status code, preventing rewriting of

maybe this config isnt enough for using cdn url to do rewrite? pagespeed message history doesnt show any error indication
 
That configuration line looks fine. But as the site is on https you do need to make sure that ngx_pagespeed will be able to get to your images/css/js.  
It's odd that the message history does not show anything in relation to the 4xx debug comments. However, two posts earlier you posted a html response (6.5MB via "zippyshare").
That did not contain any 4xx status codes, only complaints about deadlines being missed. Optimization was happening too. I suspect that the configuration was actually working at that point?

Otto

John Mase

unread,
Jul 20, 2016, 10:38:46 AM7/20/16
to ngx-pagespeed-discuss
ah yeah i was aware later that if i dont use cdn services like keycdn and cloudflare my static files actually do get optimized by pagespeed. but yes, on maprewrite i made sure to include https so im not sure why even after 12 hours it still shows 4xx error code on my cdn services. i still get the exceeded error messages and still issue with 4xx. my server is dedicated and centos 7, firewall disabled and i have no any sort of rate limitation implemented on my server, 
i havent cleared my cache since 12 hours so im not sure why i still get deadline exceeded on some parts. ive attached html page again
page.rtf

Otto van der Schaaf

unread,
Jul 20, 2016, 11:16:05 AM7/20/16
to ngx-pagesp...@googlegroups.com
For the 4XX  debug comments:

Do you currently have LoadFromFile set up? If you have any urls that cannot be mapped one to one to a file on disk those will fail, and may explain the 4xx messages.
So urls that are rewritten or css/js/images that are dynamically generated should be excluded.
If not then I am wondering what shows up in the access logs for the fetches that ngx_pagespeed performs for these urls. You can recognize fetches from ngx_pagespeed as it mentions "pagespeed" in the user-agent. 

For the "deadlines exceeded" comments: 

You might have to check if your caches are sized appropriately. Around 2 times the size of the original css/js/images of your website would be great.
If the cache is too small ngx_pagespeed may end up continuously re-optimizing assets.

Otto



--

John Mase

unread,
Jul 20, 2016, 2:08:00 PM7/20/16
to ngx-pagespeed-discuss
i have loadfrom file and ive also have more than enough cache size for ngx pagespeed but i still get these errors despite that i cleared cache and waiting 30 minutes to see if debug msg changed but it doesnt

<link rel='stylesheet' href='https://kxcdn.com/forum/uploads/css_built_100/90eb5adf50a8c640f633d47fd7eb1778_core.css.3286c1f5025266a0b52b607173afda78.css?v=95f8d25421' media='all' rel='preload' as='stylesheet'><!--deadline_exceeded for filter CacheExtender--><!--4xx status code, preventing rewriting of

pagespeed LoadFromFile "https://mydomain.me/forum/uploads"
                       "/home/yuon/domains/domain.me/public_html/forum/uploads/";

Den tisdag 19 juli 2016 kl. 13:56:36 UTC+2 skrev John Mase:

John Mase

unread,
Jul 20, 2016, 4:31:25 PM7/20/16
to ngx-pagespeed-discuss
oh and nginx access and error log shows nothiong. no errors at all
Message has been deleted

John Mase

unread,
Jul 21, 2016, 5:37:28 AM7/21/16
to ngx-pagespeed-discuss
but ive cleared cache multiple times already and maprwrite is correctly set on my cdn url
[Thu, 21 Jul 2016 09:28:09 GMT] [Info] [4879] HTTPCache key=https://19b9.kxcdn.com/forum/uploads/css_built_39/90eb5adf50a8c640f633d47fd7eb1778_core.css.df99541cc0fcaf4cab8f634bf1ac6ef0.css?v=95f8d25421 fragment=mydomain.me: remembering recent failure for 299 seconds.
[Thu, 21 Jul 2016 09:28:09 GMT] [Info] [4879] HTTPCache key=https://19b9.kxcdn.com/forum/uploads/css_built_39/90eb5adf50a8c640f633d47fd7eb1778_core.css.df99541cc0fcaf4cab8f634bf1ac6ef0.css?v=95f8d25421 fragment=mydomain.me: remembering recent failure for 299 seconds.
[Thu, 21 Jul 2016 09:28:09 GMT] [Info] [4879] HTTPCache key=https://19b9.kxcdn.com/forum/uploads/css_built_39/5a0da001ccc2200dc5625c3f3934497d_core_responsive.css.39dd39e4826cd57942f0cc34bb8abfcf.css?v=95f8d25421 fragment=mydomain.me: remembering recent failure for 299 seconds.
[Thu, 21 Jul 2016 09:28:09 GMT] [Info] [4879] HTTPCache key=https://19b9.kxcdn.com/forum/uploads/css_built_39/5a0da001ccc2200dc5625c3f3934497d_core_responsive.css.39dd39e4826cd57942f0cc34bb8abfcf.css?v=95f8d25421 fragment=mydomain.me: remembering recent failure for 299 seconds.'

Otto van der Schaaf

unread,
Jul 21, 2016, 5:44:40 AM7/21/16
to ngx-pagespeed-discuss
These urls give me alternating "404 not found" / "503 zone not found" responses when I open them in Chrome, but that is probably because of anonymization right?
I must say I'm a bit puzzled, it looks like the problems now are mostly specific to these css files and the images referenced in there, correct?

Could you post a snapshot of the statistics page from the admin pages? I'm wondering if there is a clue in there..

Otto

John Mase

unread,
Jul 21, 2016, 7:35:18 AM7/21/16
to ngx-pagespeed-discuss
indeed, the urls are omitted so its not real url. if you want i can email you my domain. but yes the error are specific to some css/js/image file extension no idea to be honest.
atm im using cloudflare cdn again but still x4xx error code.


Den tisdag 19 juli 2016 kl. 13:56:36 UTC+2 skrev John Mase:
Message has been deleted

John Mase

unread,
Jul 21, 2016, 11:24:05 AM7/21/16
to ngx-pagespeed-discuss
im giving up on ngx pagespeed thanks for trying anyway

Otto van der Schaaf

unread,
Jul 22, 2016, 11:26:52 AM7/22/16
to ngx-pagespeed-discuss
I'm really sorry to hear that. Though I noticed the site is still running ngx_pagespeed, and that everything seems OK now as far as I can tell, so I'm hopeful the module is allowed to stay. Your admin pages didn't show me anything that would lead me to think something was wrong either.

I was trying a few things to see if I could reproduce the behavior locally, and one thing is that css files that reference a lot of images may take a while before the optimized css gets written into the html. But other then that I couldn't really find anything. 

But there is this: when you add ?PageSpeedFilters=+debug to the querystring of the html you might run into an issue [1] where doing so enables other options as well. These newly enabled options trigger re-optimization of the page and its resources, and as such will show you misleading debug comments that are no issue when the pages are requested without ?PageSpeedFilters=+debug -- 
So is is possible we have been looking into a non-issue.


Otto



On Thu, Jul 21, 2016 at 5:24 PM John Mase <maid...@gmail.com> wrote:
im giving up on ngx pagespeed thanks for trying anyway

--

John Mase

unread,
Jul 24, 2016, 1:26:22 AM7/24/16
to ngx-pagespeed-discuss
nginx plus staff checked my server and said the issue is caused because of no CORS so when they added this in server block, it started to rewrite js. and css
        add_header 'Access-Control-Allow-Origin' '*';
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";


i think your pagespeed documentation should mention this if anyone encounter similar issue when they use cdn 

Den tisdag 19 juli 2016 kl. 13:56:36 UTC+2 skrev John Mase:

John Mase

unread,
Jul 24, 2016, 1:31:12 AM7/24/16
to ngx-pagespeed-discuss
adding a cors footnote here would actually be very helpful

Otto van der Schaaf

unread,
Jul 24, 2016, 5:20:54 AM7/24/16
to ngx-pagespeed-discuss
I agree that mentioning CORS in the docs for rewriting domains may be helpful, and I also understand it may have solved a problem where the browser would refuse to process assets served via the CDN. However I can not relate this to the problem we have been looking into.

Also, note that for CORS support, a new option called AddResourceHeader was added fairly recently:


So you may also have to look into applying that option.

You mention that you (or nginx+ support) added the following to nginx's configuration:
        1. add_header 'Access-Control-Allow-Origin' '*';
        2. add_header Cache-Control "public, must-revalidate, proxy-revalidate";

As we you are using LoadFromFile, I expect these to be completely bypassed as PageSpeed does not fetch its inputs via http.
Can you share your thoughts on how these changes relate to fixing the problems with the debug comments about 4XX responses and exceeded deadlines?

In any case, I'm happy to hear it's working for you now!

Otto
Reply all
Reply to author
Forward
0 new messages