mgo bson heap grows problem

[Valentin Kuznetsov]

Hi, I'm trying Go w/ mgo MongoDB driver for my project and I face constant heap grows which I want to understand how to handle. Below I post stand along web server which query MongoDB and extract some results from it. If I'll invoke multiple calls to this server I observe heap grows for every call and it doesn't shrink back. I look at heap via pprof web interface. I see grows of this type of records:

1: 576 [2: 1152] @ 0xad46 0x9ea8 0xb8d6 0xf3133 0x8957f 0x8b2a6 0x89494 0x851f3 0x6ca73 0x6d828 0x6d8fc 0x2555 0x265b 0x27bc 0xa8921 0xaa08d 0xaaa5a 0xa8477 0x2af81
#       0xf3133 reflect.Value.SetMapIndex+0x2e3                         /usr/local/go/src/reflect/value.go:1440
#       0x8957f labix.org/v2/mgo/bson.(*decoder).readDocTo+0xb7f        /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/bson/decode.go:228
#       0x8b2a6 labix.org/v2/mgo/bson.(*decoder).readElemTo+0x106       /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/bson/decode.go:404
#       0x89494 labix.org/v2/mgo/bson.(*decoder).readDocTo+0xa94        /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/bson/decode.go:227
#       0x851f3 labix.org/v2/mgo/bson.Unmarshal+0x1d3                   /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/bson/bson.go:476
#       0x6ca73 labix.org/v2/mgo.(*Iter).Next+0x583                     /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/session.go:2464
#       0x6d828 labix.org/v2/mgo.(*Iter).All+0x6c8                      /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/session.go:2534
#       0x6d8fc labix.org/v2/mgo.(*Query).All+0x5c                      /Users/vk/Work/Languages/Go/gopath/src/labix.org/v2/mgo/session.go:2546
#       0x2555  main.Get+0x435                                          /Users/vk/Work/Languages/Go/GIT/test_mongo.go:59
#       0x265b  main.processRequest+0x8b                                /Users/vk/Work/Languages/Go/GIT/test_mongo.go:70
#       0x27bc  main.RequestHandler+0x8c                                /Users/vk/Work/Languages/Go/GIT/test_mongo.go:77
#       0xa8921 net/http.HandlerFunc.ServeHTTP+0x41                     /usr/local/go/src/net/http/server.go:1265
#       0xaa08d net/http.(*ServeMux).ServeHTTP+0x17d                    /usr/local/go/src/net/http/server.go:1541
#       0xaaa5a net/http.serverHandler.ServeHTTP+0x19a                  /usr/local/go/src/net/http/server.go:1703
#       0xa8477 net/http.(*conn).serve+0xb57                            /usr/local/go/src/net/http/server.go:1204

 I"ll appreciate any feedback/suggestions. Thanks Valentin.

Here is a code:

package main

import (

"encoding/json"

"labix.org/v2/mgo"

"labix.org/v2/mgo/bson"

"log"

"net/http"

)

// profiler

import _ "net/http/pprof"

type MongoConnection struct {

Session *mgo.Session

}

func (m *MongoConnection) Connect() *mgo.Session {

var err error

if m.Session == nil {

m.Session, err = mgo.Dial("mongodb://localhost:27017")

if err != nil {

panic(err)

}

m.Session.SetMode(mgo.Monotonic, true)

}

return m.Session.Copy()

}

// single object to handle MongoDB connections

var _Mongo MongoConnection

// insert into MongoDB

func Insert(dbname, collname string, records []bson.M) {

s := _Mongo.Connect()

defer s.Close()

c := s.DB(dbname).C(collname)

for _, rec := range records {

if err := c.Insert(&rec); err != nil {

panic(err)

}

}

}

type Results bson.M

// get records from MongoDB

// func Get(dbname, collname string, spec bson.M, idx, limit int) []bson.M {

//     out := []bson.M{}

func Get(dbname, collname string, spec bson.M, idx, limit int) []Results {

var out []Results

s := _Mongo.Connect()

defer s.Close()

c := s.DB(dbname).C(collname)

var err error

if limit > 0 {

err = c.Find(spec).Skip(idx).Limit(limit).All(&out)

} else {

err = c.Find(spec).Skip(idx).All(&out)

}

if err != nil {

panic(err)

}

return out

}

// helper function to query MongoDB (test.collection database).

func processRequest(query string) map[string]interface{} {

response := make(map[string]interface{})

var spec bson.M

data := Get("test", "collection", spec, 0, -1)

response["data"] = data
        response["query"] = query

data = nil

return response

}

func RequestHandler(w http.ResponseWriter, r *http.Request) {

query := r.FormValue("query")

response := processRequest(query)

js, err := json.Marshal(&response)

if err != nil {

http.Error(w, err.Error(), http.StatusInternalServerError)

return

}

w.Header().Set("Content-Type", "application/json")

w.Write(js)

js = nil

response = nil

}

func Server() {

http.HandleFunc("/", RequestHandler)

err := http.ListenAndServe(":8212", nil)

if err != nil {

log.Fatal("ListenAndServe: ", err)

}

}

func main() {

Server()

}

[Klaus Post]

Hi!

First I would try to Clone the session instead of Copy. Otherwise I cannot see anything wrong.

js = nil

response = nil

... do nothing.

Are you sure they aren't being garbage collected? How are you testing?

/Klaus

[Valentin Kuznetsov]

Klaus, I tried Clone as you suggested and still see that heap is growing on repeating queries to the server. Regarding nil assignments, I experimented with them, but certainly you're right they do nothing. My suspicious that there is a memory leak in bson module elsewhere. When I retrieve large chunk of data, e.g. 1000 records at once, I see that it is heap size is almost double each time I place a request.

[Gustavo Niemeyer]

Valentin, as described in the mgo-users  mailing list, one issue is that you are using extremely out of date code, as current code lives in gopkg.in/mgo.v2 for years. The other issue is that you did not really describe details of what you think is a leak. It's normal for the heap to grow up to a limit since Go is garbage collected. Please provide more details on mgo-users so we can have the conversation in a single place.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.