Issues in production, mongodb 1.8.2, two shards

[Oded Maimon]

Hi,

we are getting, few times a day, the following errors in our code:

com.mongodb.MongoException$Network: can't call something

        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)

        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:296)

        at com.mongodb.DBCursor._check(DBCursor.java:356)

        at com.mongodb.DBCursor._hasNext(DBCursor.java:486)

        at com.mongodb.DBCursor.hasNext(DBCursor.java:511)

        at com.google.code.morphia.query.MorphiaIterator.hasNext(MorphiaIterator.java:40)

        at com.google.code.morphia.query.QueryImpl.get(QueryImpl.java:351)

        at com.google.code.morphia.DatastoreImpl.get(DatastoreImpl.java:497)

        at com.google.code.morphia.dao.BasicDAO.get(BasicDAO.java:163)

at the same time the mongodb shards started to move a chunks of few of our main collections (the users related collection)...

as i understand mongo chunk movement, it shouldn't interrupt regular operation.

mongos is a local process on each application server.

what are the reasons for such an error?

[Eliot Horowitz]

Want to meet @ 9?
That morning is wide open.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/r-ZFTcMDk2QJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[Scott Hernandez]

Do you have any server logs from that from that period. There should
be more info there.

On Wed, Aug 17, 2011 at 6:33 PM, Oded Maimon <oded....@gmail.com> wrote:

[Oded Maimon]

yes i've it.. 

I've kept only the relevant minutes.

do you want me to also send the mongos logs for that time?

it is attached to this mail. it is password protected. i'm sending the password directly to you...

BTW, The servers are on EC2.

Thanks

[Oded Maimon]

Hi Scott, did you get my logs and find something relevant ?

thanks.

[Scott Hernandez]

Unfortunately I didn't see any issues in the logs. There were some
really slow periods of operations on the shard (mongod) but it is hard
to know the cause.

Does this happen repeatedly, or in such a maner where you can predict
it? Is so it would help to collect some mongostat numbers as well
during that time. You can run mongostat --discover against the mongos
server and it will collect stats from all shards. Also, collecting
"iostat -xm 2" on the primaries during that time would help as well.

On Fri, Aug 19, 2011 at 5:43 AM, Oded Maimon <oded....@gmail.com> wrote:
> Hi Scott, did you get my logs and find something relevant ?
> thanks.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mongodb-user/-/u-EF_UJMGj8J.