Should I be worried about these stats?

[John Arrowwood]

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn       time
   576      0      0      0       0       1       0   127g   254g   300m     50     90.4          0      0|12    0|13   814k     1k   438   16:44:16
   469      0      0      0       0       1       0   127g   254g   302m    107      116          0       0|1     0|2   192k     1k   438   16:44:17
   499      0      0      0       0       1       0   127g   254g   303m     77      101          0      0|10    0|10   121k     1k   438   16:44:18
   275      0      0      0       0       1       0   127g   254g   301m     56     93.1       12.5      0|49    0|50   474k     1k   438   16:44:19
   536      0      0      0       0       1       0   127g   254g   303m     39     80.3        5.5       0|3     0|3   826k     1k   438   16:44:20
     0      0      0      0       0       1       0   127g   254g   303m      0        0          0       0|3     0|3    62b     1k   438   16:44:21
   410      0      0      0       0       1       0   127g   254g   302m     58     66.2          0       0|1     0|2    88k     1k   438   16:44:22
   615      0      0      0       0       1       0   127g   254g   305m     81     99.9          0      0|21    0|22   584k     1k   438   16:44:23
   297      0      0      0       0       1       0   127g   254g   303m     27     95.2          0      0|39    0|40   421k     1k   438   16:44:24
   349      0      0      0       0       1       0   127g   254g   304m     20     25.5          0       0|2     0|2   539k     1k   438   16:44:26

It says that 127gigs is mapped, and the virtual memory size is 254gigs.  This is a server with 16gigs of ram.  It's no wonder that 'top' says my server is in around 20% I/O wait...

If you were seeing these kinds of stats, and if you knew that things are going VERY slowly because of it, what would you do?

--
John Arrowwood
--
http://www.irie-inc.com/
http://www.FindingResonance.com/

[Scott Hernandez]

I'd say you need disks with better throughput and which support more
IO operations.

Can you run mongostat and iostat -xm 2 for a few minutes and post to
gist/pastie/pastebin?

Reading this should help you understand the memory numbers:
http://www.mongodb.org/display/DOCS/Checking+Server+Memory+Usage

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[John Arrowwood]

mongostat: http://pastebin.com/anSv0N6x
iostat: http://pastebin.com/Pw1r8rw3

And yes, the disk is a SAN that was mounted onto the server, so I'm not surprised.  But this is a proof of concept anyway.  As long as I can get it to perform acceptably so I can prove that it will function for our needs, then we can look into ways of optimizing it.  I just wondered if this was something I should worry about or not, in general.  But thanks to the links you sent, I understand better what I'm looking at.  Memory Mapped Files.  I get it now.

-- John

--
John Arrowwood,
Consulting Hypnotist / Hypnotherapist
http://www.JohnArrowwood.com
503.863.4823

[Scott Hernandez]

It is hard to say what is going on. It seems like the iostat data is small subset of time in the mongostat data. Not sure where to line it up for comparison.

Are these inserts on a new database/collection or are you constantly deleting big chunks of docs and then inserting big chunks?

What is happening which is causing the load? Is this some test data or a real application?

[John Arrowwood]

Let me try to get the two lined up a little better.  Here, these two will be an almost exact alignment over a 5 minute period:

mongostat: http://pastebin.com/yJe36x2f
iostat: http://pastebin.com/tc1Quy6j

The ONLY thing happening on the server (meaning, the mongo instance) is importing.  However, I'm importing into over 200 collections simultaneously.  There is no C compiler on the machine, so I could not build the Perl driver.  So instead I open an instance of mongoimport for each target collection, and as I read the log files, I translate them into JSON, then output them through the right instance of mongoimport.

The Perl write to the file handles is buffered, which means that there may be some bursts of activity on any given import process.  Some, however, are so active that they are almost continuously receiving data.  Some of the objects being written are large, others are small.  Some border on ridiculously large, so for those, the buffering probably doesn't matter much. 

I divide into a separate database per day's worth of logs, and a separate collection per method.  Each method in our system emits a log entry for every time it is called, which contains the inputs and the outputs, so we can monitor the system activity, and understand the health of the larger system that it is servicing.  I divided it up this way to conserve space and, in theory, allow for faster searching.  It conserves space because I don't have to have the date field in every record, since the date is determined by the database.  And I don't need the layer, service, or method fields, because that is the name of the collection.  And had I put it all in one collection, like I originally tried, I would not only have had to have all of those fields in each document, I would have had to index on them, which means that my total disk utilization would have been markedly higher.  And its 50 gigs per day as it is, BEFORE importing it.

And yes, this is real production logs being pushed into Mongo, it's not test data.  I'm testing out Mongo with it, but it's representative of the real data we need Mongo to deal with.  Not only will I need to push data in, I've got to be able to do ad-hoc queries against the most recently added data so that we can identify the current state of the system AS the data is being imported.

One thought that came to my mind was to split things up a little bit.  Looking at the stats, clearly Mongo is journaling.  This is confirmed by looking in the db directory.  The thought that occurred to me is, can I put the journal directory in /tmp, so it is being written to a local disk instead of the SAN?  I could create a directory in /tmp, then replace the journal directory with a symlink.  Then the respective I/O will go to different places, and will involve less I/O to the SAN, overall.  That's my theory, at least.  Is that a valid thing to do?

Any other thoughts on how I can get this system to perform the way it clearly SHOULD perform would be appreciated.  Thanks!

-- John

[Eliot Horowitz]

Putting the journal on a difference file system would be a good test.

On /tmp its not too useful as it won't survive a reboot - but you could try disabling.

How many indexes do you have?

Can you also send the mongo log?

[John Arrowwood]

I put it on a local filesystem, and it made a significant difference.  Night and day, even.

Then I played with doing a shard configuration so that I could balance the data across the four disk mounts that we were given.  And it worked, except for one show-stopper of a problem:  I ran out of file handles after a very short while!  So much for that idea...

But it's working, so I'm happy.  I'd be more happy if that 16 TB were locally mounted.  But you take what you can get.

[Eliot Horowitz]

About running out of file descriptors, did you try just setting with ulimit to something really high?

[John Arrowwood]

Yep.  Unlimited.  But it was a lot of collections, which meant a lot of processes (since I haven't gotten permission to install the driver), so it comes as no surprise to me that it fell over on itself.  It was worth trying, it's no big deal that it didn't work.