Re: [ngx-pagespeed-discuss] TTFB is more when pagespeed is enabled.

[Joshua Marantz]

30ms TTFB penalty is a bit higher than I'd expect. Depending on the hardware, it should take about 10ms to fully parse a reasonably large web page.

Can you try loading your page with ?PagespeedFilters=+debug added to the URL, and look at the resulting HTML? Comments will be added to the HTML, each time there is a flush, detailing the timing that PageSpeed observed.

How are you generating your HTML? Is it a static file? Dynamically generated? In chunks or all at once?

Also where exactly is /var/ngx_pagespeed_cache? If that's on a network file system then performance will suffer. If it's on a local hard disk it should be fine, but you could consider using an SSD or switching to memcached or redis to improve performance.

On Thu, Jan 23, 2020 at 8:43 AM Anurag Adhikary <anu...@depanacheinteriors.com> wrote:

This is my ngx pagespeed config.

pagespeed Statistics on;

pagespeed StatisticsLogging on;

pagespeed LogDir /var/log/pagespeed;

pagespeed AdminPath /pagespeed_admin;

pagespeed GlobalAdminPath /pagespeed_global_admin;

#You can also enable some pages individually.

pagespeed StatisticsPath /pagespeed_statistics;

pagespeed GlobalStatisticsPath /pagespeed_global_statistics;

pagespeed MessagesPath /pagespeed_message;

pagespeed ConsolePath /pagespeed_console;

pagespeed FileCachePath /var/ngx_pagespeed_cache;

#pagespeed FileCacheSizeKb            102400;

#pagespeed FileCacheCleanIntervalMs   3600000;

#pagespeed FileCacheInodeLimit        500000;

#pagespeed LRUCacheKbPerProcess     1024;

#pagespeed LRUCacheByteLimit        16384;

pagespeed EnableCachePurge on;

pagespeed DefaultSharedMemoryCacheKB 150000;

#pagespeed LoadFromFileCacheTtlMs 315360000000;  # ten years

pagespeed FetchWithGzip on;

pagespeed FetchHttps enable;

pagespeed MessageBufferSize 100000;

#pagespeed ImageMaxRewritesAtOnce -1;

#pagespeed UseNativeFetcher off;

#resolver 1.1.1.1;

#pagespeed ProcessScriptVariables on;

#pagespeed MaxCacheableContentLength -1;

#pagespeed MaxSegmentLength 250;

#pagespeed MaxCombinedCssBytes -1;

#let's speed up PageSpeed by storing it in the super duper fast memcached

pagespeed MemcachedThreads 1;

pagespeed MemcachedServers "127.0.0.1:11211";

pagespeed CreateSharedMemoryMetadataCache /var/ngx_pagespeed_cache 512000;

# PageSpeed Config

   pagespeed on;

   pagespeed RewriteLevel OptimizeForBandwidth;

   #pagespeed LoadFromFile "http://$host" "$document_root";

   #pagespeed LoadFromFileRuleMatch Disallow \.php$;

   #pagespeed EnableFilters debug;

   #pagespeed EnableFilters extend_cache;

   ## Text / HTML

   pagespeed EnableFilters insert_dns_prefetch;

   #pagespeed EnableFilters combine_heads;

   pagespeed EnableFilters collapse_whitespace;

   #pagespeed EnableFilters convert_meta_tags;

   pagespeed EnableFilters elide_attributes;

   pagespeed EnableFilters pedantic;

   pagespeed EnableFilters remove_comments;

   pagespeed EnableFilters remove_quotes;

   pagespeed EnableFilters trim_urls;

   ## JavaScript

   #pagespeed EnableFilters combine_javascript;

   #pagespeed EnableFilters canonicalize_javascript_libraries;

   #pagespeed EnableFilters inline_javascript;

   ##pagespeed AvoidRenamingIntrospectiveJavascript false;

   ## CSS

   #pagespeed EnableFilters outline_css;

   #pagespeed EnableFilters combine_css;

   #pagespeed EnableFilters inline_import_to_link;

   #pagespeed EnableFilters inline_css;

   pagespeed EnableFilters inline_google_font_css;

   #pagespeed EnableFilters move_css_above_scripts;

   #pagespeed EnableFilters move_css_to_head;

   pagespeed EnableFilters prioritize_critical_css;

   pagespeed EnableFilters rewrite_css;

   pagespeed EnableFilters fallback_rewrite_css_urls;

   #pagespeed EnableFilters rewrite_style_attributes_with_url;

   #pagespeed PermitIdsForCssCombining *-css;

the website URL is https://new16.depanache.in/

When pagespeed is turned on the TTFB is .042367 seconds but when it is turned off it is .011886

Is this normal?

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/aa8c4d5f-dc9e-48d0-85ea-58fe74111348%40googlegroups.com.

[Joshua Marantz]

As an experiment, can you try disabling prioritize_critical_css and  insert_dns_prefetch? I refreshed your page a few times and found there was high 'render' time annotated at the end of the HTML (with debug on) about 5/6 times. I think the most likely culprit is prioritize_critical_css. I'm also guessing the cause of prioritize_ critical_css being slow is that some of your CSS files may have low cache TTL, but it's easier just to turn the filter off for now. If that improves the Render time then we can figure out which CSS file has a low TTL or other problem.

On Thu, Jan 23, 2020 at 9:30 AM Anurag <anu...@depanacheinteriors.com> wrote:

Thanks for the quick reply Joshua.

I enabled the debug filter as you said but could not see anything that should make me worried, could you please check it once for more clarity 

view-source:https://new16.depanache.in/

How are you generating your HTML? It is a static file located in the path mentioned in the config.

Also where exactly is /var/ngx_pagespeed_cache? If that's on a network file system then performance will suffer. If it's on a local hard disk it should be fine, but you could consider using an SSD or switching to memcached or redis to improve performance. This is hosted on a $5 DigitalOcean box, the /var/ngx_pagespeed_cache is located on the disk (SSD), I was having the same thought process as you and configured memcached later but no use. Please check my config here - https://new16.depanache.in/pagespeed_admin/cache#physical_cache

To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/b48a014b-3a4b-4bb9-9d21-0e455cfaf80e%40googlegroups.com.

[Joshua Marantz]

Thanks for confirming.It looks like there's not much benefit to disabling dns-prefetch so you can leave it on if you like.

Here are some things to look at:

• are any of the CSS files not loading?
• do all the CSS files have a cache-TTL of an hour or more?
• do all the CSS files have caching enabled?

make sure all the CSS files are valid and cacheable for at least an hour, where "all" here means every file referenced by your CSS or HTML, even if they are not actually used in the layout.

I think one of the pain-points with prioritize_critical_css is that any failure loading CSS results in recomputing the critical CSS quite frequently; perhaps on every load. When I was refreshing your view-source link it seemed like about 1 in 5 or so loads had a fast 'render' time.

One more thing: I wouldn't over-emphasize TTFB. The important thing is the user experience, which, at least in the past, I've felt was best captured in webpagetest.org's "speed index" rather than TTFB. So you will definitely pay something for prioritize_critical_css (or pagespeed in general) in terms of TTFB latency, but hopefully the value will come back in a faster rendering of the page.

Please let us know what you find, and good luck.

On Thu, Jan 23, 2020 at 11:56 AM Anurag <anu...@depanacheinteriors.com> wrote:

Yes, you are right. I have disabled both prioritize_critical_css and  insert_dns_prefetch and TTFB is much better. 

The numbers are:

1. Both of them disabled - .028398 to .030666 (500 tests) 
2. ONLY DNS prefetch disabled -  .055700 to .057881 (500 tests) 

3. ONLY critical CSS disabled - .029216 to .030242 (500 tests) 

4. PageSpeed off - .019661 to .020378 (500 tests) 

As I need critical CSS, please suggest me the next steps to fix the problem. 

To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/d1d5f68c-0ec0-4586-9480-722d31c5137b%40googlegroups.com.

[Longinos]

Hi Anurag

Seem that you have disabled pagespeed and put cloudflare on, so I can´t test pagespeed.

But here my 5 cts:

Maybe you can gain some ms making rel="preconect" to some resources, like fonts.googleapis.com or www.google-analytics.com and so.

If the css files have images (gif,png,jpg) the these images are rewrited to things lile pagespeed.ic.HAHS.webp and the same image can get a pagespeed.ic.HAHS.jpg (depent on the browser capabillity) so the css changes between request from one browser and other (chrome vs. firefox for example) this makes the prioritize_critical_css restart the whole calculation.

A workaround I have fuond is making request with firefox and disallow files that have pagespeed in the url, with autoptimize turned off, copy the whole optimized css and change the original css individual files with the optimized css for firefox.

This makes you loss the webp optimization in chrome for images in the css files but get the same css file between browser and make the prioritize_critical_css filter work the same for diferent browser request.

[Longinos]

Hi Anurag

So this cloudflare server is a joke =:-))

Pagespeed still disabled, can´t test it.

For the WP... Have you considered to use a pagecache?

The most simple way to do it is using the nginx fastcgi_cache directives, these ceate a static version of the site and with a plugin like nginx-helper you can manage the delte policy when you create,modify post

For the unused css... no, really I don´t know any free service for do that. Autoptimize has a payd version of a service for that, creates a specific css file for each page that maybe reused if all the css is the same between pages.

WP core and plugins load all the css the core or plugin need even with options you don´t enable or use. There is a discussion in WP core forums to make the new gutenberg editor load only the css for blocks that are present in the page, but I think that these enhancement may be so far to come.

[Longinos]

Hi Anurag

Sorry, I have no time now, here is weekend and I go out for all of it.

The next tuesday I test this, but at a 1st glance:

Google fonts can´t be inlined cause are loaded by javascript.

The css file generated by autoptimize fail cause has inlined resource from drive.google.com

Images fail cause are lazy loaded by lazysizes, this put data-srcset and this label in sot recognized by pagespeed until a patch is applied, so only get optimization from IRPO

and in-fligth optimization

Next tuesday a will make a more detailed test.

El viernes, 24 de enero de 2020, 12:04:28 (UTC+1), Anurag escribió:

Apologies again. I made so many versions that I forgot which one it is turned on or off. Please try this https://mumbai16.depanache.in

curl -I https://mumbai16.depanache.in

HTTP/2 200

server: cloudflare-nginx

content-type: text/html

x-robots-tag: noindex, nofollow, nosnippet, noarchive

x-content-type-options: nosniff

strict-transport-security: max-age=16070400; includeSubDomains

date: Fri, 24 Jan 2020 10:59:47 GMT

x-page-speed: 1.13.35.2-0

cache-control: max-age=0, no-cache

Setup can be verified here - https://mumbai16.depanache.in/pagespeed_admin/config

[Longinos]

Hi Anurag

I have tested https://mumbai16.depanache.in/ and I see that the resources are only optimized by IPRO.

Maybe you have enabled any filter that prevents the url rewrite?

If you see the url in dev tools, you can view that the etag header is like W/"PSA-aj-DU_Ltnsr_V" , this header is set by IPRO,

but the url is not rewrited (don´t have pagespeed.xx.HASH.file-extension in it.

Same with images.

[Longinos]

HI Anurag

These are all the config directives? Where you have configured the pagespeed cache?

If no pagespeed cache is configured nginx don´t start.

The server is a single server with local disk storage?

The config you have send is not the config for https://mumbai16.depanache.in/

cause I can see images and javascript filters enabled and in these are not images filters and all the javascript are commented.

Can you paste here this page: https://mumbai16.depanache.in/pagespeed_global_admin/config?

El martes, 28 de enero de 2020, 16:37:37 (UTC+1), Anurag escribió:

This is the config for https://mumbai16.depanache.in/ 

Please let me know which filter do you want me to enable or disable to debug further.

   # PageSpeed Config
   pagespeed on;
   pagespeed RewriteLevel OptimizeForBandwidth;
   #pagespeed LoadFromFile "http://$host" "$document_root";
   #pagespeed LoadFromFileRuleMatch Disallow \.php$;
   #pagespeed EnableFilters debug;
   #pagespeed EnableFilters extend_cache;


   ## Text / HTML

   #pagespeed EnableFilters insert_dns_prefetch;
   #pagespeed EnableFilters combine_heads;
   #pagespeed EnableFilters collapse_whitespace;
   #pagespeed EnableFilters convert_meta_tags;
   #pagespeed EnableFilters elide_attributes;
   #pagespeed EnableFilters pedantic;
   #pagespeed EnableFilters remove_comments;
   #pagespeed EnableFilters remove_quotes;
   #pagespeed EnableFilters trim_urls;

   ## JavaScript
   #pagespeed EnableFilters combine_javascript;
   #pagespeed EnableFilters canonicalize_javascript_libraries;
   #pagespeed EnableFilters inline_javascript;
   ##pagespeed AvoidRenamingIntrospectiveJavascript false;

   ## CSS
   #pagespeed EnableFilters outline_css;
   #pagespeed EnableFilters combine_css;
   #pagespeed EnableFilters inline_import_to_link;
   #pagespeed EnableFilters inline_css;

   #pagespeed EnableFilters inline_google_font_css;

   #pagespeed EnableFilters move_css_above_scripts;
   #pagespeed EnableFilters move_css_to_head;

   #pagespeed EnableFilters prioritize_critical_css;
   #pagespeed EnableFilters rewrite_css;
   #pagespeed EnableFilters fallback_rewrite_css_urls;

   #pagespeed EnableFilters rewrite_style_attributes_with_url;
   #pagespeed PermitIdsForCssCombining *-css;

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/StHt5vVJMpg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/305a3e5e-cc6c-4d95-b52b-d95b867965bd%40googlegroups.com.

[Longinos]

Hi Anurag

Try to put the lines below at the same level (http block, I think) that are the cache config:

pagespeed Domain http*://*.depanache.in;

And https://www.modpagespeed.com/doc/https_support#configuring_ssl_certificates

pagespeed SslCertDirectory directory;
pagespeed SslCertFile file;

if debian/ubuntu distro only the directory, if red hat/centos both directory and file.

In the attached image you can see a permision to rewrite issue, I think s for don´t have the pagespeed Domain directive.

And you can see a message related to file cache. Can´t read the file /var/ngx_pagespeed_cache/!clean!time! , maybe a file permission in the directory. The user you use to run nginx must can read/write the directory /var/ngx_pagespeed_cache.

El viernes, 31 de enero de 2020, 5:18:49 (UTC+1), Anurag escribió:

Yes, Single server with SSD storage. This is a linode $5 box.

Please note pagespeed RewriteLevel OptimizeForBandwidth enables few filters and only these filters are enabled for this site. I am attaching the pagespeed cache directives below.

I have also enabled debug filter so that you get maximum information. Do let me know if you need more information.

[Longinos]

I don´t know why, but the image is not attached, try again

El viernes, 31 de enero de 2020, 11:08:45 (UTC+1), Longinos escribió:

Hi Anurag

Try to put the lines below at the same level (http block, I think) that are the cache config:

pagespeed Domain http*://*.depanache.in;

And https://www.modpagespeed.com/doc/https_support#configuring_ssl_certificates

pagespeed SslCertDirectory directory;
pagespeed SslCertFile file;

if debian/ubuntu distro only the directory, if red hat/centos both directory and file.

In the attached image you can see a permision to rewrite issue, I think s for don´t have the pagespeed Domain directive.

And you can see a message related to file cache. Can´t read the file /var/ngx_pagespeed_cache/!clean!time! , maybe a file permission in the directory. The user you use to run nginx must can read/write the directory /var/ngx_pagespeed_cache.

[Longinos]

Hi Anurag

Sorry, I don´t explain well.

The cert directory is NOT the Let´s encrypt.

Think in pagespeed as a browser, they need the certificates authorities to validate certificates.

For Ubuntu (or Debian) is: /etc/ssl/certs

[Longinos]

Hi Anurag

Also I think you have some problem with file permission to the pagespeed cache. Pagespeed is unable to write in it, so can´t write optimized resources.

I can´t see the images you have attached.... But I think you have

/var - root

/var/nginx_pagespeed_cache - root (this must be www-data too)

/var/nginx_pagespeed_cache/watever_else_file_or_directory - www-data

[Longinos]

Hi Anurag

Put this pagespeed FileCachePath /var/ngx_pagespeed_cache/;

Note the trailing slash

[Longinos]

Hi Anurag

The pagespeed SslCertDirectory /etc/ssl/certs/; is missing

The pagespeed Domain http*://*.depanache.in; too.

And can you add these more directives:

pagespeed FileCacheSizeKb            102400;
pagespeed FileCacheCleanIntervalMs   3600000;

Some thing is preventing to rewrite the url in the page, but it get rewrited.... I will say, for example, this url:

https://mumbai16.depanache.in/depanache-ui/uploads/2019/08/Kitchen.1-thumbnail.jpg

when was fechted in a chrome browser, get converted in this:

https://mumbai16.depanache.in/depanache-ui/uploads/2019/08/xKitchen.1-thumbnail.jpg.pagespeed.ic.b1AZp8naxF.webp

and both images can be viewed, but the html don´t get the rewrited url.

El viernes, 31 de enero de 2020, 14:11:05 (UTC+1), Anurag escribió:

pagespeed FileCachePath /var/ngx_pagespeed_cache/; is done

pagespeed SslCertDirectory /etc/ssl/certs/; is done

[Longinos]

Can you disable memcached server in the config?

Only for testing, maybe is too small in size and can´t store all the assets.

[Longinos]

Is it failing to rewrite HTML coz of this? 

[https://mumbai16.depanache.in/:6] Unrecognized script:'<script type="application/ld+json"></script> 6...36' 

No, this message is saying that pagespeed don´t recognize application/ld+json script types. It only recognizes script application/javascript or related to javascript.

Can you do this commands:

# touch /var/nginx_pagespeed_cache/!clean!time!

#chown -R www-data:www-data

Pagespeed is rewriting the url. storing the file in the cache, cause that you can request the rewrited url and get the optimized file..... but is unable to rewrite the html code.

You can see rewited resources in https://mumbai16.depanache.in/pagespeed_global_admin/message_history

[Longinos]

Can you send me all the nginx configuration files via a private email?

[Longinos]

Hi Anurag

Issue solved, now you get resources rewrited.

The module you use (or have you compiled) has enabled by default 3 filters that block the resource rewrite.

ImagePreserveURL, CssPreserveURL and JsPreserveURL.

Now you have images in data-srcset not rewrited, as spected until a pacht apply to the pagespeed module.

You can see, together to these diabled filters, a new filter pagespeed to get images added to a div as background get rewrited.

[Longinos]

Hi Anurag

Well...

1.- If you have any thing that relies in the original url then you need to no rewrite it. Think, for example, a js snipet that look for a url, if the url is rewrited then it can´t found it.

The rewrited url content a hash based in the file content, when the content changes, the hash change, so you not need to take care of versioning files related to cacheablity.

2 and 3.- Maybe rewrite the url (also optimize the resource) intruces a delay. By default pagespeed takes 10ms to do their work (this maybe configured by pagespeed RewriteDeadlinePerFlushMs XX; pagespeed InPlaceRewriteDeadlineMs XX;) If the resource is not optimized/rewrited in this time then pagespeed serves the original and do a background fecht and optiization so the next request get otimized.

Maybe using memcached, also redis or puting the fie cache in a ram disk, may lower the ttfb, but take account that pagespeed cahed need to be between 5 to 7 times the size of the original web, so you need more ram to put the cache.

Using the filesystem as cache store is a bit slower than using ram.

4.- I don´t take a look on to this in your server, but in general lowering buffer size you may have better ttfb times. As a example, in the nigx docs say that lowering ssl buffer from the default 16kb to 4kb get better ttfb.

See http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_buffer_size for that.

But now that the cause for no rewrite url is solved, you can test with or without rewrite url, and experiment what filters are good to your served or what not.

P.D. Yout ttfb is not bad, measured with webpagespeed from mumbay (https://www.webpagetest.org/result/200202_PC_ce7ebfa5e315f185af6fb45d36a9cfbd/)

[Longinos]

Hi Anurag

No. Now you have in the config

FileCacheCleanIntervalMs 33600000

This make pagespeed check the size (104857600 bytes) of the file cache.

each hour, you may increase the time to check.

El lunes, 3 de febrero de 2020, 5:38:17 (UTC+1), Anurag escribió:

Is this something to worry about?

[Mon, 03 Feb 2020 04:28:16 GMT] [Info] [22522] Need to check cache size against target 104857600

[Mon, 03 Feb 2020 04:28:16 GMT] [Info] [22522] Need to check cache size against target 104857600

[Mon, 03 Feb 2020 04:28:16 GMT] [Info] [22522] Need to check cache size against target 104857600

[Mon, 03 Feb 2020 04:28:16 GMT] [Info] [22522] Need to check cache size against target 104857600

[Longinos]

Oh....

I see you have sitemap xml files... you may exclude files that pagespeed don´t touch:

pagespeed Disallow /*sitemap*

pagespeed Disallow /*.*ico

pagespeed Disallow /*.woff

pagespeed Disallow /*.woff2

pagespeed Disallow /*.ttf

pagespeed Disallow /*.svg

[Longinos]

More.

Your site can be accessed by 2 ways: 1 by "name" and 1 by IP, but you don´t have set the canonical label, so when googlebot may accees your site it see 2 "different sites with the same content

and may index both by "name" and by IP. Setting the canonical label it only indexes the url set in canonical, so no duplicate content.

Maybe you need to redirect the IP to the dns name.