Strange errors

[Scott Smith]

Hi

We're seeing strange errors returned from our reactivemongo connections. These are the errors:

[Supervisor-1/Connection-2] Oups. 4098 not found! complete message is Response(MessageHeader(185188,170643495,4098,1),Reply(8,0,0,2),LittleEndianHeapChannelBuffer(ridx=36, widx=185188, cap=185188),ResponseInfo(-1611803325))

[Supervisor-1/Connection-2] Oups. 4202 not found! complete message is Response(MessageHeader(147,170643496,4202,1),Reply(8,0,0,1),LittleEndianHeapChannelBuffer(ridx=36, widx=147, cap=147),ResponseInfo(-1611803325))

[Supervisor-1/Connection-2] Oups. 4276 not found! complete message is Response(MessageHeader(147,170643497,4276,1),Reply(8,0,0,1),LittleEndianHeapChannelBuffer(ridx=36, widx=147, cap=147),ResponseInfo(-1611803325))

After seeing these errors, the connections no longer work and hang without returning a response. Anyone have any insight into this?

thanks

[Cédric Chantepie]

Need to be more specific about the context/versions

[Scott Smith]

We're using version 0.12 of reactivemongo.

The connection is made through a URI. The error doesn't come through with a stack trace. From the "Oups" part of it, I've traced it to the fallback value found in the reactivemongo.core.actors.MongoDBSystem trait.

On Mon, Sep 18, 2017 at 5:20 PM, Cédric Chantepie <chantep...@gmail.com> wrote:

Need to be more specific about the context/versions

--
You received this message because you are subscribed to a topic in the Google Groups "ReactiveMongo - http://reactivemongo.org" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reactivemongo/wMJorjJThrg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reactivemongo+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cédric Chantepie]

Sorry but that's not sufficiently specific about the context.

[Ori Popowski]

Hi,

We're having the exact same problems for quiet a while. It happens several of our applications.

Details:

• ReactiveMongo 0.12.7
• Play 2.6.6 application
• Scala 2.12.4
• Mongo version 3.4.9 replicated cluster

It's happening for the past month or so, and also happened when Mongo was 3.2 and the driver 0.12.0.

Mongo CPU is always below 40%.

The impact of the problem is that reads do not succeed we're forced to restart the application every few days.

Please suggest how to solve this

[Scott Smith]

Hi Ori

When we encountered these errors, we were in the middle of moving our servers to a new service provider, so I'm not sure I can remember what exactly the problem was (too many changes and too much time since then).  From my notes, though, I think it may have had something to do with trying to have too many connections against the mongodb server. 

When you see the error, try connecting to your mongodb instance and running this command:

db.serverStatus().connections

Is the number of current connections pretty close to the number available?

[Ori Popowski]

Hi,

Thanks for the suggestion.

However, looking at the timeseries graph of both connections and connections available, I can see that connection/available ratio never crosses the 2% mark.

We have 50K available connections and we usually use 800 out of them.

This also applies to all times when the errors occurred in the log, so this hypothesis is not correct.

Any other ideas?

[Ori Popowski]

I want to add that the "oups" error message does not correlate with any metric we have of MongoDB in the last 7 days:

• Number of connections
• globallock queued readers
• globallock queued writers
• globallock active client readers
• globallock active client writers
• read/write tickets
• open cursors
  
• CPU
• ...
  

Thanks