Cache Performance

[Max Lord]

As our cluster grew, we moved the cache from the broker tier to the historical nodes, and saw much better latency and cache utilization. We use local cache exclusively, rather than memcached.

Recently we noticed a pretty serious performance hit once the cache filled up and the eviction rate increased. Even though the cache hit rate was still good, historical load was much higher, and GC times were about 3x normal.

Would we benefit from using the new caffeine cache extension that Gian created a while back? We're still on 0.9.0.

Thanks for any tips.

Regards,

Max

[Fangjin Yang]

Hi Max, the benchmarks for the caffeine extension showcase that it is definitely more performant than the local heap and has seen production usage at scale.

https://github.com/druid-io/druid/pull/3028 has more details. I would recommend trying it out.

[Max Lord]

Thanks, Fangjin

We got the extension installed, and it seems to be working well. We'll monitor its performance for a few days.

Regards,

Max

[charles.allen]

Just FYI, be sure to monitor heap metrics during this time. The cache is very efficient but the way the extension uses it can cause extra heap pressure under high eviction rates. On the majority of our nodes it worked out of the box, but some of our very high query load nodes had to have GC tunings.

[Max Lord]

Just to tie up this thread, the caffeine cache extension turned out to work very well for us, even under high eviction rates.

We saw about 50% less GC time while under our peak load, healthier heap usage, and lower overall query latency across the whole cluster.

Recommended, +1

Cheers,

Max

[Gian Merlino]

Thanks for posting back!

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/92f5fa8f-15bd-4977-b6e7-e5c571408d19%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.