We're getting odd segmentation faults occurring on the primary node.

System specs:
64 bit (dual quad core)
32 GB RAM
1.5 TB HDD

The db has 14,420,016 documents occupying around 115GB of disk space.

When we shut down the faulting primary then the newly elected primary will
exhibit the segfault.
It doesn't seem to be related to a particular query.

Mon Oct  1 13:33:52 [conn14] command voip.$cmd command: { count:
"call_log", query: { start_time: { $lte: 1325404799.0, $gte: 1324800000.0

Mon Oct  1 13:33:52 Got signal: 11 (Segmentation fault).

Mon Oct  1 13:33:52 Backtrace:
0xade6e1 0x5582d9 0x558862 0x7f5740489cb0 0x819a7b 0x92fdf8 0x92fed5
0x931e8c 0xbda621 0xbda841 0xbda92f 0x84e7a7 0x881496 0x88181e 0x881e2b
0x881fc6 0x8824b4 0x88270c 0x850a6d 0x85120b
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262)
[0x558862]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f5740489cb0]
/usr/bin/mongod(_ZNK5mongo11BSONElement9woCompareERKS0_b+0xdb) [0x819a7b]
/usr/bin/mongod(_ZNK5mongo24FieldRangeVectorIterator20FieldIntervalMatcher8 upperCmpEv+0x48)
[0x92fdf8]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator23validateCurrentInterva lEiRKNS_11BSONElementEbbRb+0x75)
[0x92fed5]
/usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+ 0x30c)
[0x931e8c]
/usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x8 1)
[0xbda621]
/usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x41) [0xbda841]
/usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0xaf) [0xbda92f]
/usr/bin/mongod(_ZN5mongo22QueryOptimizerCursorOp4nextEv+0xb7) [0x84e7a7]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56)
[0x881496]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner5_nextEv+0x11e) [0x88181e]
/usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x7b) [0x881e2b]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner13iterateRunnerERNS_7QueryOpEb+0 x46)
[0x881fc6]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner12nextOpSimpleEv+0x14) [0x8824b4]
/usr/bin/mongod(_ZN5mongo16MultiPlanScanner6nextOpEv+0x5c) [0x88270c]
/usr/bin/mongod(_ZN5mongo24QueryOptimizerCursorImpl8_advanceEb+0xad)
[0x850a6d]
/usr/bin/mongod(_ZN5mongo24QueryOptimizerCursorImpl7advanceEv+0xb)
[0x85120b]

Hi Dylan,

Can you create a Jira ticket (https://jira.mongodb.org) in the "Core
server" project and provide information about your system so that we can
debug this?

The crash seems to be a null pointer used in BSONElement::woCompare (well
ordered compare) or in an inlined function it calls.  We may be able to
make some headway from the stack trace but we will probably need a full
log, which you could post to the Jira ticket once it is set up.

Thanks!

Tad