500 goroutines make ~400 mongodb connections

[jeane paul Soliva]

Greetings,

I have goroutines that do Session.Copy() but checking mongos connection after launching my app, current connections rise up to 400~

mongos> db.serverStatus().connections

{ "current" : 4, "available" : 815, "totalCreated" : NumberLong(4) }

mongos> db.serverStatus().connections

{ "current" : 433, "available" : 386, "totalCreated" : NumberLong(433) }

mongos> db.serverStatus().connections

{ "current" : 417, "available" : 402, "totalCreated" : NumberLong(435) }

mongos>

this is the loop that launch the goroutines

wg.Add(len(topStoriesIds))

for _, id := range topStoriesIds {

go func(id int) {

newsContent := NewsReader(id)

ContentOutPut(newsContent, &wg)

}(id)

}

and the insert to mongodb logic from ContentOutPut

func HackerNewsInsert(n News, nTitle string, wg *sync.WaitGroup) {

defer wg.Done()

sc := SessionCopy()

defer sc.Close()

c := sc.DB(Db).C(NewsCollection)

if !NewsFindIfExist(nTitle, sc) {

return

}

err := c.Insert(n)

if err != nil {

fmt.Println(err)

return

}

fmt.Println("saved! news!")

}

SessionCopy func 

func SessionCopy() *mongodb.Session {

return MongodbSession.Copy()

}

what could be the problem? been doing this since yesterday and running in to few workarounds... thanks!

[Klaus Post]

Hi!

The best solution would be to limit the number of goroutines you start. If you don't want that many to run at the same time, don't start them in the first place.

Otherwise look at the 'Session.SetPoolLimit()' function. Be sure to read the documentation. Not sure why 'It is [...] an unsupported use case to use the database driver to define the concurrency limit', so be careful. From what I can read from the code, it waits 100ms and retries if the pool limit is reached.

/Klaus

[Gustavo Niemeyer]

Klaus is right. Just to clarify his point, mgo will create as many connections as required to satisfy the concurrency it is submitted to. In other words, if there are 10 requests being submitted concurrently to the database, how could mgo possibly arbitrate correctly which of these requests should be submitted to possibly long delays based on completely unrelated requests?

SetPoolLimit is not meant to solve this case either. The default limit is 4096 concurrent connections, and it only exists to prevent a major breakage from taking place due to misbehavior from a particular application, not to be regularly exercised. In fact, that method was added when somebody requested the limit to be increased, rather than reduced.

Concurrency is better controlled "at the door", as Klaus mentioned. It's best to not even accept further connections if you are unable to handle them, otherwise the application will get a higher rate of incoming connections consuming resources than the application can deal with, and eventually the system will run out of resources altogether.

--
You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[jeane paul Soliva]

Greetings,

thanks for the reply, I think I misunderstood session.copy(). by saying copy, means using/copy the same connection and not making new connections regardless of what usage like launching thousand goroutine. I think copy is quite confusing for me, who is not a go wizard. for ordinary dev, copy could mean using the existing connections and not making new.