Time to live (TTL) slows down mongod

[David Nellessen]

Profiling my slow queries I found something interesting: About 1/4 of my top 100 slow queries are queries on the index table searching for indexes that contain expireAfterSeconds, for example:

{
 "ts" : ISODate("2013-12-05T10:20:21.549Z"),
 "op" : "query",
 "ns" : "mydb.system.indexes",
 "query" : {
 "expireAfterSeconds" : {
 "$exists" : true
 }
 },
 "ntoreturn" : 0,
 "ntoskip" : 0,
 "nscanned" : 247846,
 "keyUpdates" : 0,
 "numYield" : 0,
 "lockStats" : {
 "timeLockedMicros" : {
 "r" : NumberLong(188687),
 "w" : NumberLong(0)
 },
 "timeAcquiringMicros" : {
 "r" : NumberLong(2),
 "w" : NumberLong(6)
 }
 },
 "nreturned" : 0,
 "responseLength" : 20,
 "millis" : 188,
 "client" : "0.0.0.0",
 "user" : ""
}

These queries are made for the Time to live (TTL) feature of mongo, run every 60 seconds (http://docs.mongodb.org/manual/tutorial/expire-data/). Note that I don't use TTL at all! Also note that I am using a lot of collections (> 100,000) with a lot of indexes (247,846 in total). I assume you only run into this issue if you have a lot of indexes like me. So the question is:

How do I disable TTL? Is that possible at all? Alternatives to face this issue?

[Alan Spencer]

Hi David,

I don't believe there is a way to disable the TTL indexes completely.

The query time of 188ms for scanning the 247,846 indexes seems about right. And it would suggest these indexes are in memory.

Although these queries are "slow" (i.e. > 100ms), do they impact the running of MongoDB. Do you see any specific impact every 60 seconds?

Are you using MMS? Can you see if there is an impact to application performance when these run?

Thanks,

Alan.

[David Nellessen]

I am not using MMS. The only impact I can see is that those queries are the most slow queries over all. The database is slow in general and I am currently refactoring the structure to increase performance. I drastically reduced the amount of collections from 247,846 to 20 because it had a lot of downsides (slow backups, slow creation of collection creation/deletion, maximum number of namespaces, ...). The system is much faster now and the issue does not remain for me. Still I think this is a general problem for others dealing with a lot of collection and there should be a way to disable ttl.

[Stephen Steneker]

On Wednesday, January 8, 2014 9:49:02 AM UTC-8, David Nellessen wrote:

I am not using MMS. The only impact I can see is that those queries are the most slow queries over all. The database is slow in general and I am currently refactoring the structure to increase performance. I drastically reduced the amount of collections from 247,846 to 20 because it had a lot of downsides (slow backups, slow creation of collection creation/deletion, maximum number of namespaces, ...). The system is much faster now and the issue does not remain for me. Still I think this is a general problem for others dealing with a lot of collection and there should be a way to disable ttl.

Hi David,

If you don't have any TTL indexes you can disable the TTL monitor thread in MongoDB 2.4.6+ (see SERVER-10033).

The parameter can be either set at startup:

--setParameter ttlMonitorEnabled=false

Or at runtime:

db.adminCommand({setParameter:1, ttlMonitorEnabled:true})

This parameter seems to have been overlooked in the current release notes / documentation.

Regards,

Stephen

[Alan Spencer]

Hi David,

That's great that you were able to make your system perform much better.

I would recommend using MMS if this is possible for you as it can give you good information on the causes of performance degradation, and it's free.

https://mms.mongodb.com/user/register

Regards,

Alan.

[Alan Spencer]

Also, in 2.4.6 a parameter was added to disable the TTL index monitor thread: https://jira.mongodb.org/browse/SERVER-10033

Regards,

Alan.

[David Nellessen]

Thank you Stephen and Alan! I will consider using MMS, even though I expect a lot of privacy issues we usually face with such a service here in Germany.

[Stephen Steneker]

On Monday, January 13, 2014 8:28:28 AM UTC-5, David Nellessen wrote:

Thank you Stephen and Alan! I will consider using MMS, even though I expect a lot of privacy issues we usually face with such a service here in Germany.

Hi David,

As far as data/privacy concerns go, by default MMS Monitoring only collects counter metrics. The database stats ("DB Stats") tab for a host does show the database names, but if you are concerned you can turn off the option to "Collect Database Specific Statistics" via the Settings page in MMS.  Features which may include user data (such as log and profile data collection) are not enabled by default.

You can see a list of the commands used by the MMS agent in the documentation:

 http://mms.mongodb.com/help/monitoring/reference/#database-commands-used-by-the-monitoring-agent

If you aren't able to use the hosted MMS Monitoring, there are some alternative admin/monitoring interfaces listed in the MongoDB documentation: 

 http://docs.mongodb.org/ecosystem/tools/administration-interfaces/

FYI, there is also an On-Premise version of MMS available with MongoDB Enterprise subscriptions.

Regards,

Stephen

[s.molinari]

What privacy issue could there be? I mean, which laws are you concerned about?

Scott

[David Nellessen]

Thank you Stephen for clearing things up. I will give it a try. I am concerned with privacy issues related to user and usage data of our service i.e. the data stored in MongoDB. If this is not mandatory there won't be an issue. There are a lot of privacy laws and regulations in Germany and our company also has its own policy, s.molinari.

[s.molinari]

I also live in Germany and appreciate what you mean about the laws here, but MMS is purely a database monitoring system for monitoring database performance. No "private" data or even user specific data on their usage of the database can be seen with MMS. It is more of a top-level view of database performance and health. So, there is no need to worry about the laws in this case. You could say the performance and health data MMS collects about the database server is actually your company's data, not the user's. So, the only problem you might run into is a possible company policy about data being given to third parties, which MongoDB does use for their and your purposes too (i.e. to improve the database).

Scott

[David Nellessen]

You are referring to the monitoring feature, right? Thank's for clearing things up. Though things are different for the backup feature of MMS where all data will be transferred unencrypted if I understand correct. So that would be a big privacy issue. Nevertheless I will give the monitoring of MMS a try!

[Asya Kamsky]

Backup service is different from monitoring but to make something
perfectly clear - the data *is* transfered securely in both directions
with BRS:
https://mms.mongodb.com/learn-more
https://wiki.mongodb.com/display/mcs/Backup+FAQ

Asya

> --
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Asya Kamsky]

Sorry, you won't be able to see the second link, I just realized. But
the relevant information is that all communication to/from our
datacenter where your backup is will be over SSL *only*.

Asya

[David Nellessen]

But it is only enrcrypted for transport, right? It will be processed and
stored unencrypted. This would still be a big issue.

"Data transfer is SSL encrypted. Data is not encrypted at rest."

[s.molinari]

Yes, you are right David. The data storage/ backup service would have several critical issues for meeting data privacy laws in Germany. Especially if the servers are outside of the EU.

Since you were talking about performance the whole time, I didn't think the back-up service was part of the discussion.:-)

Scott