Millions of map keys

[Christian von Kietzell]

Hi Go nuts,

would it be a terribly bad idea to have a map with millions of keys? In my case those keys would be short to medium-length strings (anything from 1 to about 20 characters). Values would be slices of int64.

If that could provoke the GC to show less-than-optimal behaviour what are some possible alternatives?


Cheers,
Chris

[Matthew Zimmerman]

Millions should be fine in my experience, however you cannot use a slice of anything as the keys so you'll have to convert it to something else first.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Val]

@Matthew the keys are strings, which is already fine. The values are slices.

[Nick Craig-Wood]

My understanding is that big maps with no pointers are fine.

However a string has an implicit pointer in, so I suspect that might
trigger the suboptimal GC behaviour.


--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Egon]

It's hard to suggest really good optimizations based on an abstract problem. Even the characters and values in the slices may suggest a better data-structure that is not possible in the generic case.

In other words, can you show a sample of the data or a program that generates a representative dataset, (so that the character distribution, length distribution, value distributions are good). Then also include how the dataset is accessed, how often iterated, modified etc. (see https://github.com/golang/go/wiki/HowToAsk).

As a general suggestion, do the stupid thing first, measure -> if it's good enough, proceed with more important problems :)

+ Egon

[Alex Bligh]

On 3 May 2016, at 12:13, Christian von Kietzell <cub...@gmail.com> wrote:

> would it be a terribly bad idea to have a map with millions of keys? In my case those keys would be short to medium-length strings (anything from 1 to about 20 characters). Values would be slices of int64.
>
> If that could provoke the GC to show less-than-optimal behaviour what are some possible alternatives?

Do the keys change frequently? Do the values change frequently? Certainly if the map is static or nearly static there will be no issue.

--
Alex Bligh

[Daniel Skinner]

> As a general suggestion, do the stupid thing first

That can't be stressed enough. If I had a nickel for every time the naive/simple approach worked best ...

[halmstad....@gmail.com]

If you're interested in big, possibly memory efficient maps; this might be worth having a look at:
https://github.com/fncodr/godbase/tree/master/maps

They offer more control than maps at least, which is what you often crave for when working with big datasets...

Peace

[Brad Fitzpatrick]

That's fine. I have millions of things in maps.

Use Go 1.6 for best performance.

--

[Dan Kortschak]

I'm doing a genomics analysis at the moment that creates maps with
>600million map entries. It works fine.