Map with expiration

[Alex Pliutau]

In some cases your application doesn’t need Redis, and internal in-memory map with locks and expiration will suffice.

For example you already know the size of the map and you don’t need to store a lot of data. Use cases could be IP rate limiting, or any other short-lived data.

Here is how you can implement this data structure in Go, let’s call it a TTLMap:

https://gist.github.com/plutov/69d22552c696d48496563529c624e0b7 

[Nico Braun]

Hi, isnt this kind of inefficient? I think it would be better without tickers and extra goroutines. You could check if an item is expired when trying to fetch it. And then you do periodic compaction to make sure old stuff is freed up, eventually. 

[Nico Braun]

Well, you do that already. Except its having 1 goroutine per entry. Maybe 1 goroutine for all would be enough.

[Alex Pliutau]

It's one goroutine for all currently. Which is more efficient than checking each item individually.

[Nico Braun]

Ah you are right. I need to sleep. I read this code wrong, twice. Just ignore me. 

[Cleberson Pedreira Pauluci]

Hello. I found this very interesting.

I have some questions:

Have you considered the possibility of using time.Duration for the maxTTL parameter?

Maybe wouldn't it be more interesting if each item had its own maxTTL instead of just one for any item on the map?

If you'll allow me, I made those adaptations to your original code.
Here's the example: https://go.dev/play/p/rba6HCJe-4X

Thanks.

[Alex Pliutau]

Yes, time.Duration and maybe another argument for ticker interval as time.Duration as well.

[Def Ceb]

I've found this package to be useful for this in the past: https://github.com/jellydator/ttlcache

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a59272be-7277-4cb6-a7a7-8bf4bdcefc40n%40googlegroups.com.

[Nico Braun]

I started thinking about the compaction, and if there was a way to prevent stopping the world during cleanup. May be interesting to look how etcd or redis something does it.

Could you use a RWMutex for this map, or is this not a good use case?

[robert engels]

The best way (imo) to perform the compaction is to use multiple sub maps - i.e. partition the keys - and then you can compact these using RW locks to limit the stop the world.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/924e3ebf-c2a6-4288-be14-8b187a46ab25n%40googlegroups.com.

[Cleberson Pedreira Pauluci]

I'm wondering if using sync.Map wouldn't be the best option for performance reasons for this application, since it's a specialized map implementation. Does anyone know anything about this?

[robert engels]

My project github.com/robaho/go-concurrency-test may be of interest in evaluating this. The timings are old, but AFAIK the analysis still holds.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/fc43af25-566a-4e88-b1e3-ba97e226f351n%40googlegroups.com.