Received: by 10.42.89.20 with SMTP id e20mr15679518icm.3.1336512125756;
        Tue, 08 May 2012 14:22:05 -0700 (PDT)
X-BeenThere: mongodb-user@googlegroups.com
Received: by 10.50.168.41 with SMTP id zt9ls3865564igb.0.gmail; Tue, 08 May
 2012 14:21:50 -0700 (PDT)
Received: by 10.50.181.198 with SMTP id dy6mr253545igc.2.1336512110786;
        Tue, 08 May 2012 14:21:50 -0700 (PDT)
Received: by 10.50.151.242 with SMTP id ut18msigb;
        Tue, 8 May 2012 14:17:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.50.95.136 with SMTP id dk8mr2016095igb.0.1336511850227; Tue,
 08 May 2012 14:17:30 -0700 (PDT)
Authentication-Results: ls.google.com; spf=pass (google.com: domain of
 zack.shoy...@gmail.com designates internal as permitted sender)
 smtp.mail=zack.shoy...@gmail.com; dkim=pass
 header...@gmail.com
Received: by o3g2000pby.googlegroups.com with HTTP; Tue, 8 May 2012 14:17:30
 -0700 (PDT)
Date: Tue, 8 May 2012 14:17:30 -0700 (PDT)
User-Agent: G2/1.0
X-HTTP-Via: 1.1 reno-gw.pc-doctor.com:3128 (squid/2.7.STABLE9)
X-HTTP-UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19
 (KHTML, like Gecko) Chrome/18.0.1025.142 Safari/535.19,gzip(gfe)
Message-ID: <de90ed98-4a51-439a-ade1-c088fde0c4b3@o3g2000pby.googlegroups.com>
Subject: Slow-down for a large inserts-only job
From: Zack Shoylev <zack.shoy...@gmail.com>
To: mongodb-user <mongodb-user@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1

The case:
32-core server running 32 mongod shards, a config server, and a
mongos. 300GB RAM and a large raid disk system (with very high
throughput).
Parallel mongoimport jobs starts with about 100k inserts/sec total,
but quickly slow down to 0 to 5k/sec

Logs show a lot of
Tue May  8 13:47:46 [conn21] warning: could have autosplit on
collection: test.test1 but: splitVector command failed: { errmsg:
"need to specify the desired max chunk size (maxChunkSize or
maxChunkSizeBytes)", ok: 0.0 }
and slow inserts:
Tue May  8 13:47:50 [conn22] insert test.test1 1320ms
Tue May  8 13:47:50 [conn31] insert test.test1 1423ms

I have chunkSize set to 20000, and 32 chunks (1 per shard) with fully
distributed splitting of data.

I need 100k min consistent inserts, but 300k+ would be preferable.

My questions are:
What's the deal with the splitVector? I am running 2.0.4 and made sure
to restart everything after setting the chunkSize. Is this what's
causing the slow-down?
If not, what could be causing the slow-down? CPU usage is low, and so
is memory, only disk activity is high. Is mongodb using a "safe mode"