Alternatives to Memcache/Reddis (Multi Server)

[Shane Marsh]

Hiya, I hope everyone is well and staying safe.

Just wondered if I could get some advice. Currently we have a number of front end web servers all running Nginx & mod_pagespeed and currently they are connected together with two memcache servers. One servers has 7.5GB and a second at 13.07GB. 

The hit rate seems to be above 80% (percentage of get_hits vs get_misses) so I am happy with how it's performing but costs is quite an issue. AWS charge the earth for in-memory caching and the cache.r6g.large (Elasticache) instance costs $165+vat per month. Obviously Reddis would costs the same because AWS charge for the instance size, not what it's used for. 

My question: In memory caching is very expensive. Is there a cheaper alternative?

Look forward to hearing from you!

Shane

[Otto van der Schaaf]

There's two options that come to mind that could yield cost savings, and disk persistence as a bonus:

- Apache Traffic Server has a scalable disk based cache, which can be exposed over the memcached protocol via: https://github.com/apache/trafficserver/tree/master/plugins/experimental/memcache.

This would yield you a scalable disk-backed persistent cache, which can suffice with a pretty small amount of RAM.

(I've actually had that running in production for a while. It needed a tiny patch, I'm not sure if I contributed that back, but I could check)

- Not sure what the cost model is with Redis, but that's also an alternative: https://www.modpagespeed.com/doc/system#redis

Hope that helps

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/1f9afb3c-27f6-49eb-9353-e9b9ac01e3fbn%40googlegroups.com.

[Shane Marsh]

Hi Otto, 

Yes that helps thank you. I will look into it and run some tests. I have been considering whether to convert to Apache. Is mod_pagespeed in the Apache repository yet? I hope things are still going well with the incubation!

The cost model for Reddis is the same within AWS - they only care about the resources you provision what it's used for. 

[Shane Marsh]

Hi Otto, 

Are you sure tsmemcache is production ready? I ran a short test today and the whole thing just tore apart a the seams!

Several problems straight away:

1) Many errors appeared in the Message history along the lines of: [Mon, 19 Apr 2021 16:21:32 GMT] [Error] [1702] AprMemCache::Put error: Internal error (specific information not available) (20014) on key rname/cb_Iv6yX0gwkPM5jgKHHMon/XblePH60s6SPSCP_RIW1n@@_, value-size 72

2) Pagespeed refused to optimise anything - because of the recurring errors.

3) The stats page broke. Infact the whole memcache stats functionality doesn't appear to be implemented at all. The command just errors out.

What was the "tiny patch" you talk about because the first impressions I have is the whole plugin incomplete, broken and largely unmaintained. Such a shame.

Shane :)

[Otto van der Schaaf]

I found the tiny patch :-)

https://gist.github.com/oschaaf/3664892e9ee7deee8bf5b51e0787332c

Unfortunately, past me didn’t care to leave a comment with some context on why those changes where needed. It might have had something to do with subtle differences in what clients/versions put on the wire, eg pagespeed send subtly different things on the wire compared to what was used to test the plug-in during implementation. 

I remember this change did stabilize things for my use case. It is probably not a widely used ats plugin as it is still experimental (and ATS isn’t that well known). But I thought it was quite powerful in a sense that this scales to very large cache sizes without issues 

Otto

To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/788a9f2b-854d-45e7-bdf2-92f7fcaa65aan%40googlegroups.com.

[Shane Marsh]

Thanks Otto. 

I will take a another look. I think to implement your code I'll need to compile from source - the version I tested used apt packages. The only other possible way forward I could find was Fatcache: https://github.com/twitter/fatcache

Seems quite old and isn't maintained so I was going to compile it and give it a try. Beyond this I have no idea! 

Shane :)

[Shane Marsh]

Seems Twitters Fatcache is dead too. Such a shame because the code compiles cleanly and the server process runs but as soon as a load is applied the server immediatly crashes with a Segmentation fault. 

With no other way forward I guess I'd better kneel down at the alter of AWS and help Jeff buy another yacht.

Thanks for your help!

Shane

[Shane Marsh]

I don't believe it, a contributor actually replied on the Fatcache repository. Seems this Kvrocks the way forward and (as a bonus), it's in active development: https://github.com/bitleak/kvrocks

Fingers crossed!

Shane :)

[Otto van der Schaaf]

Nice, curious to hear how that works out!

Otto

To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/efd4c9cb-c053-4681-97c0-0263dc4fccfdn%40googlegroups.com.

[Shane Marsh]

Well immediately I am very impressed.  I've had it running over night and it seems to be fitting the need perfectly. Just note their current implementation of RockDB only builds on GCC v8 or below. My OS default was GCC 9.3 so it took a few moments to work what was going wrong. Now it's running, it's using about 2.4GB of RAM and 25GB of SSD storage. I've not seen any warnings in pagespeeds Message History and everything is stable. You should definitely try it :)

Shane :)

[Otto van der Schaaf]

Nice! Thanks for pointing out the alternative option. 

If it turns out stable, it might be worth pointing out the option to use this in our docs, it seems relatively easy to leverage it?

I'm sure you aren't the only one running into this, and it might help others getting easy access to that information.

Otto 

To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/a507abb4-d097-456a-b058-4a0f2fec4869n%40googlegroups.com.

[Shane Marsh]

Hi Otto, 

Yes it's very easy to use - the default settings work great so anyone who is comfortable with the command line will be able to use. It compiled cleanely and they latest release fixed the issues I was having with GCC versions. I've been running it in production for 5 days and it's not missed a beat - it's been "rock" solid in terms of stability. Additionally they seem to be very quick to act on any issues. 

A couple of things to bear in mind that I've noticed so far:

• Unlike Redis, Kvrocks has no built-in eviction policy whatsoever. The developers say Kvrocks designed to be "like a database" so the Kvrocks will consume disk space pagespeed has decided all optimisations are complete or hard limitation is reached. In the context of mod_pagespeed I would guess this disk space consumption would top out at roughly five times the size of the installation we are trying to optimise?
• You can set a hard limit to stop the server from running out of disk space but if Kvrocks meets that limit or the instance runs out of hdd/ssd space, any new entries will fail and pagespeed will stop optimising. Therefore HHD monitoring is critical.
• It's does not appear to be possible (yet) to run a script (say in python) to remove "old" entries because something (either pagespeed or Kvrocks) is setting the TTL of all cache entries to -1 so at some point there will be a need to clear the database or run a script would need to delete entries by matching the domain name contained within the Redis keys. Not sure if mod_pagespeed cache flush does this (not tested it).

Other than that, so far so good!

Shane :)

[Longinos]

Hi Shane

"... something (either pagespeed or Kvrocks) is setting the TTL of all cache entries to -..." Pagespeed can set a TTL to the redis records wirh:

ModPagespeedRedisTTLSec -1(-1 or some other value) so if Kvrocks admit this values maybe some script can clean old entries.

[Joshua Marantz]

Very cool!

But do note that depending on configuration, PageSpeed can do cache inserts on high-entropy keys and needs some kind of eviction policy to bound cache storage. For example, certain optimizations store cache entries on HTML URLs in the "property cache". Examples:

1. insert_dns_prefetch caches the DNS entries that are needed by an HTML URL.
   
2. inline_images caches the above-the-fold image URLs for the current page, to avoid having not-yet-visible image bytes competing with critical CSS or JS during initial render.
3. lazyload_images needs the same info to avoid lazy-loading images that are part of the initial visible rendered view
4. prioritize_critical_css caches the critical CSS elements for each HTML url.

Often, HTML URLs have high entropy, or can include query-params that are irrelevant to the rendering but could bust caches. You could avoid this by disabling all filters that require this.

I think you need to keep this in mind when using a cache that has no implemented eviction policy. But at a minimum I'd see if Kvrocks can be configured to have a default TTL. Ideally when PageSpeed talks to caches with a built-in TTL function it should configure that per-entry, since PageSpeed always has a TTL in mind (usually configured in the payload) and should probably pass it to the cache if possible. This would require changing the internal cache API in PageSpeed which is definitely do-able but may require updating the implementations of all the caches and adapters, so it would touch a lot of files.

To view this discussion on the web visit https://groups.google.com/d/msgid/ngx-pagespeed-discuss/598bbbe9-2794-4dde-893d-8cfbe4c5defbn%40googlegroups.com.

[Shane Marsh]

Looking at this, Kvrocks does appear have TTL support: https://github.com/bitleak/kvrocks/blob/unstable/docs/metadata-design.md

Additionally, the underlying RocksDB documentation states keys with expired TTLs are removed during compactions: https://github.com/facebook/rocksdb/wiki/Time-to-Live

So in theory, if we made changed the default ttl within mod_pagespeed something other than -1 we would have an eviction policy of sorts that would prevent database swell. Compactions are configurable within Kvrocks and can be run via cron or within specified times.  

Shane :)

[Shane Marsh]

Just wanted to update. Things continue to be stable. 

All I've added is: pagespeed RedisTTLSec 2592000;  giving a 30 day TTL.  It''s working well. 

Shane :)