Error connecting to MongoDB Atlas
Dave DeCaprio
I’m using Mongo DB Atlas and Reactive Mongo and have suddenly started getting sporadic connection issues.
I have been running this application on Mongo for several months, but starting a week ago I started having issues where I would lose connection ot the database every couple of days. Once the connection is lost it doesn’t recover and I have to restart the app.
I have seen this with 0.20.11. I tried upgrading to 0.20.12 and 1.0.0 and am seeing the same issue. With 1.0.0 it seems to happen consistently – it never gets a solid connection.
Below is a stack trace from using the 1.0.0 version. I know this isn’t a network issue because I had a version of the app running 0.20.11 that was working just fine in a different container on the same host at the same time that I got this exception.
Everything was working great up until last week, and I didn’t deploy any new code changes. Any ideas?
Dave
2020-10-29 13:24:53,910 %PARSER_ERROR[coloredLevel] c.ApplicationController - Serving index.html from root
2020-10-29 13:35:36,778 %PARSER_ERROR[coloredLevel] application -
! @7hi8nf3pd - Internal server error, for (POST) [/api/auth/login] ->
play.api.UnexpectedException: Unexpected exception[PrimaryUnavailableException: MongoError['No primary node is available! (Supervisor-1/cizr-prod2)']]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:340)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:263)
at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:443)
at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:441)
at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
Caused by: reactivemongo.core.actors.Exceptions$PrimaryUnavailableException: MongoError['No primary node is available! (Supervisor-1/cizr-prod2)']
at play.modules.reactivemongo.DefaultReactiveMongoApi.database(DefaultReactiveMongoApi.scala:49)
at reactivemongo.api.MongoConnection.database(MongoConnection.scala:108)
Caused by: reactivemongo.core.actors.Exceptions$InternalState:
2020-10-29 14:00:21,331 %PARSER_ERROR[coloredLevel] c.ApplicationController - Serving index.html from api
2020-10-29 14:00:23,469 %PARSER_ERROR[coloredLevel] c.ApplicationController - Serving index.html from root
2020-10-29 14:00:23,582 %PARSER_ERROR[coloredLevel] c.ApplicationController - Serving index.html from root
2020-10-29 20:23:51,921 %PARSER_ERROR[coloredLevel] r.c.a.MongoDBSystem - [Supervisor-1/cizr-prod2] Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] hasn't answered in time to last ping! Please check its connectivity (<time:447505336576528>).
reactivemongo.core.actors.Exceptions$InternalState:
at reactivemongo.ConnectAll({{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1400091ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1473672ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447485344856531ns, authenticated={application-prod@admin}] }})(<time:447495354764084>)
at reactivemongo.ConnectAll$IsMaster(928, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1400091ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1473672ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447485344856531ns, authenticated={application-prod@admin}] }})(<time:447495345996694>)
at reactivemongo.ConnectAll$IsMaster(927, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1400091ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447495344232117ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447485344856531ns, authenticated={application-prod@admin}] }})(<time:447495345767780>)
at reactivemongo.IsMaster(false, 928, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1400091ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447495344232117ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447485344856531ns, authenticated={application-prod@admin}] }})(<time:447495345724110>)
at reactivemongo.IsMaster(true, 927, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=447495344187587ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447495344232117ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447485344856531ns, authenticated={application-prod@admin}] }})(<time:447495345626188>)
2020-10-29 20:23:53,460 %PARSER_ERROR[coloredLevel] r.c.a.MongoDBSystem - [Supervisor-1/cizr-prod2] Received unexpected isMaster from ds137768-shard-00-00.uzlyy.mongodb.net:27017 response #924 (expected #929)! Please check connectivity.
reactivemongo.core.actors.Exceptions$InternalState:
at reactivemongo.ConnectAll({{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1747176ns, authenticated={application-prod@admin}] }})(<time:447505354140569>)
at reactivemongo.ConnectAll$IsMaster(931, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1747176ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] }})(<time:447505339819856>)
at reactivemongo.IsMaster(true, 931, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=447505337900588ns, authenticated={application-prod@admin}] }})(<time:447505339726085>)
at reactivemongo.ConnectAll$IsMaster(930, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=447505337900588ns, authenticated={application-prod@admin}] }})(<time:447505339611903>)
at reactivemongo.IsMaster(false, 930, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=447505337798916ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=447505337900588ns, authenticated={application-prod@admin}] }})(<time:447505339335239>)
2020-10-29 20:23:53,693 %PARSER_ERROR[coloredLevel] r.c.a.MongoDBSystem - [Supervisor-1/cizr-prod2] Received unexpected isMaster from ds137768-shard-00-00.uzlyy.mongodb.net:27017 response #929 (expected #<none>)! Please check connectivity.
reactivemongo.core.actors.Exceptions$InternalState:
at reactivemongo.ConnectAll$IsMaster(924, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1747176ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=9223372036854775807ns, authenticated={application-prod@admin}] }})(<time:447506876540282>)
at reactivemongo.IsMaster(false, 924, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1747176ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (10/10/10 available connections), latency=19989986631ns, authenticated={application-prod@admin}] }})(<time:447506875976914>)
at reactivemongo.ConnectAll({{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1747176ns, authenticated={application-prod@admin}] }})(<time:447505354140569>)
at reactivemongo.ConnectAll$IsMaster(931, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1747176ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] }})(<time:447505339819856>)
at reactivemongo.IsMaster(true, 931, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19989986631ns, authenticated={}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1357870ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=447505337900588ns, authenticated={application-prod@admin}] }})(<time:447505339726085>)
2020-10-29 21:20:58,432 %PARSER_ERROR[coloredLevel] c.ApplicationController - Serving index.html from root
2020-10-30 03:34:11,930 %PARSER_ERROR[coloredLevel] r.c.a.MongoDBSystem - [Supervisor-1/cizr-prod2] Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] hasn't answered in time to last ping! Please check its connectivity (<time:473325344703894>).
reactivemongo.core.actors.Exceptions$InternalState:
at reactivemongo.ConnectAll({{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473305344782729ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1385170ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1075435ns, authenticated={application-prod@admin}] }})(<time:473315354842581>)
at reactivemongo.ConnectAll$IsMaster(672, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1075435ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1385170ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473305344782729ns, authenticated={application-prod@admin}] }})(<time:473315346293521>)
at reactivemongo.IsMaster(false, 672, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1075435ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473315344849140ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473305344782729ns, authenticated={application-prod@admin}] }})(<time:473315346252030>)
at reactivemongo.ConnectAll$IsMaster(673, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473305344782729ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473315344849140ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1075435ns, authenticated={application-prod@admin}] }})(<time:473315346042847>)
at reactivemongo.IsMaster(true, 673, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473305344782729ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473315344849140ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=473315344900881ns, authenticated={application-prod@admin}] }})(<time:473315346001206>)
2020-10-30 03:34:18,115 %PARSER_ERROR[coloredLevel] r.c.a.MongoDBSystem - [Supervisor-1/cizr-prod2] Received unexpected isMaster from ds137768-shard-00-00.uzlyy.mongodb.net:27017 response #671 (expected #676)! Please check connectivity.
reactivemongo.core.actors.Exceptions$InternalState:
at reactivemongo.ConnectAll({{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1672474ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325354256509>)
at reactivemongo.ConnectAll$IsMaster(675, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1672474ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325346127434>)
at reactivemongo.ConnectAll$IsMaster(674, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473325344186536ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325345909161>)
at reactivemongo.IsMaster(false, 675, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473325344186536ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325345879190>)
at reactivemongo.IsMaster(true, 674, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=473325344134706ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473325344186536ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325345758499>)
2020-10-30 03:34:18,116 %PARSER_ERROR[coloredLevel] r.c.a.MongoDBSystem - [Supervisor-1/cizr-prod2] Received unexpected isMaster from ds137768-shard-00-00.uzlyy.mongodb.net:27017 response #676 (expected #<none>)! Please check connectivity.
reactivemongo.core.actors.Exceptions$InternalState:
at reactivemongo.ConnectAll$IsMaster(671, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=9223372036854775807ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1672474ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] }})(<time:473331530984072>)
at reactivemongo.IsMaster(false, 671, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (10/10/10 available connections), latency=19999419037ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1672474ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] }})(<time:473331530947601>)
at reactivemongo.ConnectAll({{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1672474ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325354256509>)
at reactivemongo.ConnectAll$IsMaster(675, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=1672474ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325346127434>)
at reactivemongo.ConnectAll$IsMaster(674, {{NodeSet Some(atlas-lt96ge-shard-0) Node[ds137768-shard-00-01.uzlyy.mongodb.net:27017: Primary (10/10/10 available connections), latency=1581682ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-02.uzlyy.mongodb.net:27017: Secondary (10/10/10 available connections), latency=473325344186536ns, authenticated={application-prod@admin}] | Node[ds137768-shard-00-00.uzlyy.mongodb.net:27017: Unknown (0/10/10 available connections), latency=19999419037ns, authenticated={}] }})(<time:473325345909161>)
2020-10-30 08:48:09,020 %PARSER_ERROR[coloredLevel] c.ApplicationController - Serving index.html from root
Dave
