Mongo 3.6.5 WiredTiger storage engine slow startup

[valdek]

I'm testing MongoDB (community edition) upgrade from 2.6 to 3.6 (version 3.6.5 is installed) in my company. It's running on Debian Jessie on AWS EC2.

We are experiencing some strange behavior (or maybe it's normal): when we are using WiredTiger storage engine on Mongo 3.6 - between mongo service start and time when it starts to listen connections there's like 2 minutes delay (depends on disk type attached). 

With Mmapv1 storage engine it's starting to listen for connections immediately.

Database has more than 16k collections with very different numbers of documents in them (from few to tens of thousands), database size with compression enabled is around 4,5 GB, without compression ~11GB

Server on which it's experienced is AWS EC2 instance c4.xlarge (4 cores, 8GB of RAM), I've tried different volumes type attached - magnetic (here startup time was even 4-5 minutes), gp2 (300/3000 IOPS), io1 (with IOPS on 1000 level).

There was no significant difference in startup time between having database files on gp2 or io1 volume (volumes created from scratch not from snapshots so AWS disk warm up is not a case here).

Most of the startup time is spent on WT checkpoint, from mongod.log:

2018-07-18T13:25:11.119+0200 D STORAGE  [WTCheckpointThread] starting WTCheckpointThread thread

2018-07-18T13:26:11.418+0200 D STORAGE  [initandlisten] Slow WT transaction. Lifetime of SnapshotId 1 was 60259ms

No matter if filesystem type is EXT3 or XFS, filesystem is mounted with or without noatime option, transparent huge pages disabled or enabled , readahead disabled or enabled - it's always between 60s - 120s (or sometimes more).
I've also tried with separated directory for WiredTiger indexes option but it didn't speed up.
We have database journaling disabled, no replica set in place during tests (but on production replication it will be involved).

Any idea if there's anything else I could try? or maybe it's normal behavior, from startup observation I noticed that during WT checkpoint it access each collection file (around ~16k files) so I guess that's where startup is mostly delayed.

[Robert Cochran]

Hi!

You don't list the steps you took to upgrade from MongoDB version 2.6 to MongoDB version 3.6.5. Did you follow the recommended process for upgrading? You can't directly upgrade from version 2.6 to 3.6.5. Instead, you have to follow a process of upgrading from one release to the next in a sequence of steps. Consult the documentation to determine the precise upgrade sequence you need to follow.

With that said, I am new to MongoDB and I'm only commenting in a spirit of helpfulness.

Thanks

Bob

[Oleg Toropov]

I am sure this delay related to 16k files. The opening file (c or c++)  is very expensive operation.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/9008a4f8-5554-434a-82eb-d42adaad0146%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

//
// Oleg V.Toropov, MCP | oleg.t...@gmail.com
// -----------------------------------------------------------------
// I have discovered that if I read enough stuff that's over my head,
// I actually begin to understand some of it.
//

[Robert Cochran]

Also, I'm not certain if MongoDB 3.6.5 is supported on Debian 8 ("jessie"). 

Thanks

Bob

On Thursday, July 19, 2018 at 6:50:28 PM UTC-4, valdek wrote:

[valdek]

Hello All, 

thank you for you answers. 

To answer to questions:
- MySQL 3.6.5 is supported on jessie
- Upgrade procedure can be considered as installation from scratch (took dump, removed old mongo, installed 3.6.5 and imported dump)

[Ramon Fernandez]

All,

MongoDB 3.6 is supported on Debian 8 — please see the Production Notes for further information on supported platforms and the installation instructions.

Cheers,

Ramón.

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/d2443d95-f958-44e1-bf53-b9e0fc998e71%40googlegroups.com.