saving map to file

[Tai Trinh]

Hello, i'm looking for advice.

I have a map, rather big one ( string keys are allways 84char long )
map[string]int

Whats the best way of saving and loading it from file.
*does not have to be human readable
*dont take to much memory doing it
*rather fast
*ease of implementation

( right now im using package json and marshaling and unmarshaling it
to bytes and then saving it to file, and the marshaling and
unmarshaling seem to be eating to much memory )

[Rob 'Commander' Pike]

I'd use package gob, which should be much more efficient.

-rob

[Tai Trinh]

Thank you.

-Tai

[Tai Trinh]

Using gob now. This is what i am doing. I ran it on my small linux
machine runing ubuntu 10.04

The map.txt file about 140mg.
Compiled with, 6g version release.r60.3 9516

What i am wondering is why does my test program seem to be using about
1gig ram when gob is decoding?

-Tai

func savePats(file string, pats map[string]int) {
f, err := os.Create(file)
if err != nil {
panic("cant open file")
}
defer f.Close()

enc := gob.NewEncoder(f)
if err := enc.Encode(pats); err != nil {
panic("cant encode")
}
}

func loadPats(file string) (pats map[string]int) {
f, err := os.Open(file)
if err != nil {
panic("cant open file")
}
defer f.Close()

enc := gob.NewDecoder(f)
if err := enc.Decode(&pats); err != nil {
panic("cant decode")
}

return pats

[Andrew Gerrand]

This blog post discusses some techniques for profiling the memory and CPU usage of Go programs:

  http://blog.golang.org/2011/06/profiling-go-programs.html

Andrew

[Andy Balholm]

What type are the values? If it's a map[string]string or something like that, it should be easy to write your own encoder/decoder.

[Tai Trinh]

Thanks Andrew

I have been profiling my code, and what i don't seem to understand is
why gob.encode seem to use so much memory...

And true i could write my own encoder/decoder.

But what i chose to do in the end was use a key value store, redis. My
hashmap was getting real big in the order of 9 000 000+ entries

-Tai

[André Moraes]

If you don't need to share that with another program directly from Redis,
you could use one of the many KV database available to Go

goleveldb (I think the author is adg),
godiskv,

And a some bindings to tokyo cabinet.

Look at:

http://go-lang.cat-v.org/go-code and http://godashboard.appspot.com/project

Don't use gocask since it store the keys on memory, with +/- 9 million
keys you will suffer a lot from memory consumption.
Also I don't work with that code all the time, so maybe it will some
fix due weekly changes.

--
André Moraes
http://andredevchannel.blogspot.com/

[Andrew Gerrand]

On Tuesday, December 6, 2011 10:31:26 PM UTC+11, André Moraes wrote:

goleveldb (I think the author is adg),

Actually it's Nigel Tao, and IIRC it's not a complete implementation yet.

Andrew

[Andrew Gerrand]

On Tuesday, December 6, 2011 7:36:27 PM UTC+11, Tai Trinh wrote:

Thanks Andrew

I have been profiling my code, and what i don't seem to understand is
why gob.encode seem to use so much memory...

It shouldn't be. Can you please file an issue with as much detail as you can provide?

http://code.google.com/p/go/issues/entry

Thanks.

Andrew 

[Tai Trinh]

Thanks.

Will do.

-Tai