Re: Semafores Error with Mongos

[Asya Kamsky]

There is an answer to that question on this list here:

https://groups.google.com/forum/?fromgroups=#!topic/mongodb-user/vBVXNE39ow4

Asya

On Friday, March 15, 2013 12:16:42 AM UTC-7, Virendra Agarwal wrote:

Hi 

I have Mongos System in place with two shard and three config srevres

All DB server Ulimit 65536

Mongos maxConn = 20000

Mongod maxConn = 20000

Java driver options

mongo.getMongoOptions().setConnectionsPerHost(100);

mongo.getMongoOptions().setThreadsAllowedToBlockForConnectionMultiplier(150);

mongo.getMongoOptions().setConnectTimeout(5000);

Application servers10 linux servers

These Servers are writing 1000 records per second in Mongo.

I get this error very frequently 

com.mongodb.DBPortPool$SemaphoresOut: Concurrent requests for database connection have exceeded limit of 15000

        at com.mongodb.DBPortPool.get(DBPortPool.java:211)

        at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:440)

        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:177)

        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)

        at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:270)

        at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:226)

        at com.mongodb.DBCollection.insert(DBCollection.java:147)

        at com.mongodb.DBCollection.insert(DBCollection.java:90)

        at com.mongodb.DBCollection.save(DBCollection.java:809)

        at com.google.code.morphia.DatastoreImpl.save(DatastoreImpl.java:731)

        at com.google.code.morphia.DatastoreImpl.save(DatastoreImpl.java:793)

        at com.google.code.morphia.DatastoreImpl.save(DatastoreImpl.java:787)

        at com.til.ibeat.dao.mongo.PageTrendLogDaoMongo.saveLog(PageTrendLogDaoMongo.java:54)

        at com.til.ibeat.service.PageTrendLogService.writeLog(PageTrendLogService.java:23)

        at com.til.ibeat.controller.PageTrendLoggingController.handleRequest(PageTrendLoggingController.java:181)

        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:763)

        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:709)

        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:613)

        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:525)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

        at com.redhat.jboss.support.ConfigureSessionCookieValve.invoke(ConfigureSessionCookieValve.java:25)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)

        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)

        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)

        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)

        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)

        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)

        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)

[Virendra Agarwal]

HI Asya As i have mentioned in my post i am using all three parameters to optimize my connections but still error is there . :( 

[Ronald Stalder]

Hi Virendra

you say that you have 10 application servers - how many mongos are you running ? On what machines?

Regards

Ronald

 

[Virendra Agarwal]

I am running one monogs on a linux with 8 core processor machine and all application servers are talking to this mongos and then it has two shards underline which actually save data 

[Virendra Agarwal]

Hi Ronald any suggestions ??? 

On Monday, 18 March 2013 12:12:55 UTC+5:30, Ronald Stalder wrote:

[Ronald Stalder]

I'd suggest, that you start one mongos on each application server. See http://docs.mongodb.org/manual/administration/sharded-cluster-architectures/

regards

Ronald

[Virendra Agarwal]

Hi Ronald,

I have updated to Mongo 2.2.4 version and replace all my Map Reduce operations to aggregate. But I am getting com.mongodb.DBPortPool$SemaphoresOut: Concurrent requests for database connection have exceeded limit of error very frequently. When i downgrade mongo to 2.0 this error doesn't come could you please help me why this is happening