Mgo seems to leak
HeineAndersen
Hope someone can help me with this.
I got this little function fetching files om mongodb, it works great,
but It's leaking memory.
If i hit It using apache benchmark 100000 times, the memory usage keep
growing ( +3G )
func Download(w http.ResponseWriter, r *http.Request) {
key := r.URL.Raw[10:]
session := dbSession.Copy()
defer session.Close()
c := session.DB("fs").C("keyid")
result := KeyId{}
err := c.Find(bson.M{"key": key}).One(&result)
if err != nil {
http.Error(w, "Not Found", http.StatusNotFound)
return
}
gridFile, err :=
session.DB("fs").GridFS("fs").OpenId(bson.ObjectIdHex(result.Id))
check(err)
w.Header().Set("Connection", "keep-alive")
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", "attachment;filename=" +
gridFile.Name())
w.Header().Set("Content-Length", strconv.Itoa64(gridFile.Size()))
_,err = io.Copyn(w, gridFile, gridFile.Size())
check(err)
err = gridFile.Close()
check(err)
}
To be sure It's not an Go problem I've done a test with this function,
doing the same, just streaming from the filesystem, it doesn't leak.
func DownloadDebug(w http.ResponseWriter, r *http.Request) {
filename := r.URL.Raw[15:]
log.Println("filname:", filename)
fh, err := os.Open(filename)
check(err)
defer fh.Close()
stat,err := fh.Stat()
check(err)
w.Header().Set("Connection", "keep-alive")
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", "attachment;filename=" +
filename)
w.Header().Set("Content-Length", strconv.Itoa64(stat.Size))
_,err = io.Copyn(w, fh, stat.Size)
check(err)
}
Am i doing something wrong ?
btw, I'm on 64 bit linux, and weekly go, and newest mgo.
Gustavo Niemeyer
> Hope someone can help me with this.
I'll be happy to give you a hand.
> I got this little function fetching files om mongodb, it works great,
> but It's leaking memory.
> If i hit It using apache benchmark 100000 times, the memory usage keep
> growing ( +3G )
I haven't tested your program yet, and I have a few things to do right
now, but give me a moment and I'll evaluate this locally to see what's
going on.
> btw, I'm on 64 bit linux, and weekly go, and newest mgo.
Ok.
--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter
HeineAndersen
I've cut away all the crap, and I can reproduce the problem with this
code,
It's using about 3G ram. (using top to measure)
11506 myself 20 0 3373m 3.0g 1688 S 79.1 77.6 0:47.03
Mgoleak
package main
import (
"os"
"io"
"launchpad.net/mgo"
"launchpad.net/gobson/bson"
)
var dbSession *mgo.Session
func check(err os.Error) {
if err != nil {
panic(err.String())
}
}
func ReadGridFile(objectId bson.ObjectId) {
gridSession := dbSession.Copy()
defer gridSession.Close()
gridFile, err :=
gridSession.DB("fs").GridFS("fs").OpenId(objectId)
check(err)
defer gridFile.Close()
fh, err := os.OpenFile("/dev/null", os.O_RDWR, 0666)
check(err)
defer fh.Close()
_,err = io.Copy(fh, gridFile)
check(err)
}
func main() {
var err os.Error
dbSession,err = mgo.Mongo("localhost")
check(err)
defer dbSession.Close()
objectId := bson.ObjectIdHex("4e12e771523e591cdf0000d0")
for i := 0;i < 10000; i++ {
ReadGridFile(objectId)
}
}
Gustavo Niemeyer
No problem, and sorry to see you're having issues.
> I've cut away all the crap, and I can reproduce the problem with this
> code, It's using about 3G ram. (using top to measure)
Ah, very nice. That'll make it pretty easy to sort out what's going on.
Thanks!
Gustavo Niemeyer
> code, It's using about 3G ram. (using top to measure)
Alright, I think the issue is being caused by the use of SetFinalizer
in GridFS. I can't see why it's causing a problem yet, because as far
as I can see there are no cycles involved (there's nothing pointing to
the file), but either way as long as the file is properly closed it's
really not doing anything, so should be fine to remove it until I
understand why it's misbehaving.
Can you please try it out in your real environment? Here is the trivial patch:
http://pastebin.ubuntu.com/638650/
Heine Andersen
A last question, Is it ok to use a global var to store the seesion in,
like i do in my web app ?
regards
Gustavo Niemeyer
That's great to know. Thanks for the reproducers.
> A last question, Is it ok to use a global var to store the seesion in,
> like i do in my web app ?
I'm not a fan of that kind of global variable myself, so I wouldn't
advise you to use it. That said, it's a conceptual problem (harder to
test, etc) rather than a bug, so it will work. As a note, if you
decide to go with it, mgo is concurrency-safe so you can access the
variable without locks, assuming it has been put in place without
races..