Upgrading to Reactivemongo 1.0.7 from 0.16 causes requests to hang

[Drew McClelland]

Hey all, we've been running Reactivemongo 0.16 for a long time, and I recently tried to upgrade to 1.0.7. When running locally and after deployment everything works as expected, but after about ~3hours we start to see requests hang and never finish.

I combed through the trace logs and it looks like the request is successfully submitted (`Request 20492 successful on channel #f943079d...`), and added to the awaitingResponses object, but the channel is never written to again. All of the hanging requests continue to be shown in the `awaitingResponses` property of the `Registering awaiting response for requestID ...` log. The `Channel received message Response...` and `Got a response from f943079d to 20492!` messages are never logged for the hanging requests.

All future requests that use that channel will hang as well. 

The requests do seem to have a node name in common (our -mongo-secondary.service.consul consul address), however I saw requests to that node succeed some of the time, when they wrote to a different channel.

I can share more of the trace logs if that would be helpful. This has consumed a large amount of time so I'm probably going to scrap the upgrade unless this is a known issue with a known fix.

Command: `coll.find(selector).cursor[T]().collect[List](-1, Cursor.FailOnError())`
DB Instantiation: 
private val driver = new AsyncDriver

driver.connect(serversOpt.get, MongoConnectionOptions(

    keepAlive = true,

    nbChannelsPerNode = 25,

    readPreference = ReadPreference.secondaryPreferred,

    readConcern = ReadConcern.Local,

    sslEnabled = false,

    writeConcern = Journaled))

ReactiveMongo Version:  1.0.7
Akka Version: akka-actor_2.11:2.3.13
Mongo Version: mongo:4.2.6
Scala Version: 2.11.12

Play Version: play_2.11:2.4.11

[Cédric Chantepie]

Try sharing log, and more useful, a minimal reproducer.

[Drew McClelland]

I've attached the failed and successful traces. 

Since I haven't been able to repro this locally, I can't provide a minimal reproducer.

Thanks,

[Carlos Saltos]

WOW !! ... that's a very crazy bug you step on Drew ... at my company we also jump from reactivemongo 0.16 to 1.0.0 and we get some difficult errors until Cédric kindly help us and it works with newer versions (TL;DR it was our own configuration mistake)

But anyway, as mentioned is key to make the error reproducible locally and have a minimal reproduce repo but that was also very difficult for us, so in the meantime, as a workaround, we loaded the connections via a cache using scala-cache2k that rotates the connections every hour and that "solved" the problem until we found a solid solution.

If you need details on how to implement the scalacache cache2k please let us know and, of course, you can use any other good cache library for this workaround.

Best regards,

Carlos

--
You received this message because you are subscribed to the Google Groups "ReactiveMongo - http://reactivemongo.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reactivemong...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reactivemongo/63cb5ccc-c6ff-4788-b1a6-7de93b42fc44n%40googlegroups.com.

[Cédric Chantepie]

The failed log doesn't indeed include error, as indicated by the TRACE/DEBUG level.