mongoimport performance issue and unexpected speed changing

96 views

Skip to first unread message

Ender TUNÇ

unread,

Nov 21, 2015, 7:30:53 AM11/21/15

to mongodb-user

Important note for who has more than 50 Reputation on Stackoverflow: Fastest way to import millions of JSON documents to MongoDB this guy may help me out but I can not reach him(no e-mail etc). If you can lead him here by commenting on his answer (I do not have 50 repıtation to make comment) I would be appreciate.

I have ~2GB json file that I need to import mongoDB. I am using mongoimport to import json to mongoDB but I have some consideration about performance. This process will happen a lot of time so it needs to fast as much as possible.

My collection on db is empty and no field indexed.

I want to give information about my json file.

~48000 line, each line has ~750 words and ~47000 characters.

I am currently using this command;

mongoimport --db db_name --collection collection_name --type json --file /path/to/file --batchSize 300 -j 4 --writeConcern '{w: 0, wtimeout:500, fsync: false, j: true}'

-j => parallel worker count. Bigger than 8 make it slower.

--batchSize => can't set bigger than 350. Somehow It crashes to connection. It makes an effect on it but I could not figure out how.

I am not sure about --writeConcern part. I tried with and without it does not make big difference.

I try to change the values to make it fast but none of the work properly.When I start to import I get these graphics.More CPU usage more MB/s

Reminder: Other programs shut down during the test. Base CPU usage 5-10 %

Starting to import. When CPU get %90 average 20mb/s. It last just 2-3 seconds

After that It start to act like unpredictable. I do not know why sometimes It goes and down again. Low MB/s

And lastly, after somepoint it goes down and never goes up back. I start to die here. Very low MB/s

I think this speed is so slow. What could be the reason? What am I doing wrong?

You can check the values...

2015-11-21T00:37:40.772+0200    [........................] db.items 66.8 MB/2.0 GB (3.2%)
2015-11-21T00:37:43.757+0200    [#.......................] db.items 138.6 MB/2.0 GB (6.6%)
2015-11-21T00:37:46.757+0200    [#.......................] db.items 173.0 MB/2.0 GB (8.3%)
2015-11-21T00:37:49.762+0200    [##......................] db.items 225.4 MB/2.0 GB (10.8%)
2015-11-21T00:37:52.757+0200    [##......................] db.items 244.2 MB/2.0 GB (11.7%)
2015-11-21T00:37:55.757+0200    [###.....................] db.items 262.7 MB/2.0 GB (12.6%)
2015-11-21T00:37:58.757+0200    [###.....................] db.items 281.0 MB/2.0 GB (13.5%)
2015-11-21T00:38:01.757+0200    [###.....................] db.items 299.3 MB/2.0 GB (14.3%)
2015-11-21T00:38:04.765+0200    [###.....................] db.items 322.2 MB/2.0 GB (15.4%)
2015-11-21T00:38:07.762+0200    [####....................] db.items 381.3 MB/2.0 GB (18.3%)
2015-11-21T00:38:10.757+0200    [####....................] db.items 396.1 MB/2.0 GB (19.0%)
2015-11-21T00:38:13.757+0200    [####....................] db.items 407.0 MB/2.0 GB (19.5%)
2015-11-21T00:38:16.768+0200    [#####...................] db.items 442.8 MB/2.0 GB (21.2%)
2015-11-21T00:38:19.767+0200    [#####...................] db.items 511.3 MB/2.0 GB (24.5%)
2015-11-21T00:38:22.757+0200    [######..................] db.items 543.3 MB/2.0 GB (26.0%)
2015-11-21T00:38:25.760+0200    [######..................] db.items 585.7 MB/2.0 GB (28.0%)
2015-11-21T00:38:28.757+0200    [######..................] db.items 592.2 MB/2.0 GB (28.3%)
2015-11-21T00:38:31.757+0200    [######..................] db.items 592.2 MB/2.0 GB (28.3%)
2015-11-21T00:38:34.757+0200    [######..................] db.items 592.2 MB/2.0 GB (28.3%)
2015-11-21T00:38:37.757+0200    [#######.................] db.items 617.0 MB/2.0 GB (29.5%)
2015-11-21T00:38:40.772+0200    [#######.................] db.items 653.7 MB/2.0 GB (31.3%)
2015-11-21T00:38:43.766+0200    [########................] db.items 708.4 MB/2.0 GB (33.9%)
2015-11-21T00:38:46.758+0200    [########................] db.items 747.5 MB/2.0 GB (35.8%)
2015-11-21T00:38:49.758+0200    [########................] db.items 770.8 MB/2.0 GB (36.9%)
2015-11-21T00:38:52.757+0200    [#########...............] db.items 787.3 MB/2.0 GB (37.7%)
2015-11-21T00:38:55.757+0200    [#########...............] db.items 804.2 MB/2.0 GB (38.5%)
2015-11-21T00:38:58.759+0200    [#########...............] db.items 813.7 MB/2.0 GB (39.0%)
2015-11-21T00:39:01.757+0200    [#########...............] db.items 829.9 MB/2.0 GB (39.7%)
2015-11-21T00:39:04.757+0200    [#########...............] db.items 833.5 MB/2.0 GB (39.9%)
2015-11-21T00:39:07.757+0200    [##########..............] db.items 882.1 MB/2.0 GB (42.2%)
2015-11-21T00:39:10.757+0200    [##########..............] db.items 882.1 MB/2.0 GB (42.2%)
2015-11-21T00:39:13.757+0200    [##########..............] db.items 885.9 MB/2.0 GB (42.4%)
2015-11-21T00:39:16.757+0200    [##########..............] db.items 895.5 MB/2.0 GB (42.9%)
2015-11-21T00:39:19.757+0200    [##########..............] db.items 912.5 MB/2.0 GB (43.7%)
2015-11-21T00:39:22.757+0200    [##########..............] db.items 935.5 MB/2.0 GB (44.8%)
2015-11-21T00:39:25.757+0200    [##########..............] db.items 948.8 MB/2.0 GB (45.4%)
2015-11-21T00:39:28.757+0200    [###########.............] db.items 962.2 MB/2.0 GB (46.1%)
2015-11-21T00:39:31.757+0200    [###########.............] db.items 971.8 MB/2.0 GB (46.5%)
2015-11-21T00:39:34.757+0200    [###########.............] db.items 989.2 MB/2.0 GB (47.4%)
2015-11-21T00:39:37.757+0200    [###########.............] db.items 1006.3 MB/2.0 GB (48.2%)
2015-11-21T00:39:40.757+0200    [###########.............] db.items 1014.5 MB/2.0 GB (48.6%)
2015-11-21T00:39:43.757+0200    [###########.............] db.items 1014.5 MB/2.0 GB (48.6%)
2015-11-21T00:39:46.757+0200    [###########.............] db.items 1014.5 MB/2.0 GB (48.6%)
2015-11-21T00:39:49.757+0200    [###########.............] db.items 1023.7 MB/2.0 GB (49.0%)
2015-11-21T00:39:52.757+0200    [###########.............] db.items 1.0 GB/2.0 GB (49.8%)
2015-11-21T00:39:55.757+0200    [############............] db.items 1.0 GB/2.0 GB (50.4%)
2015-11-21T00:39:58.757+0200    [############............] db.items 1.0 GB/2.0 GB (51.1%)
2015-11-21T00:40:01.785+0200    [############............] db.items 1.1 GB/2.0 GB (51.9%)
2015-11-21T00:40:04.757+0200    [############............] db.items 1.1 GB/2.0 GB (53.9%)
2015-11-21T00:40:07.757+0200    [#############...........] db.items 1.1 GB/2.0 GB (55.2%)
2015-11-21T00:40:10.757+0200    [#############...........] db.items 1.1 GB/2.0 GB (56.1%)
2015-11-21T00:40:13.757+0200    [#############...........] db.items 1.2 GB/2.0 GB (56.7%)
2015-11-21T00:40:16.757+0200    [#############...........] db.items 1.2 GB/2.0 GB (56.7%)
2015-11-21T00:40:19.759+0200    [#############...........] db.items 1.2 GB/2.0 GB (57.6%)
2015-11-21T00:40:22.772+0200    [##############..........] db.items 1.2 GB/2.0 GB (58.6%)
2015-11-21T00:40:25.757+0200    [##############..........] db.items 1.2 GB/2.0 GB (59.3%)

Reply all

Reply to author

Forward

0 new messages