Stale indexes while Raven does a cleanup

Marcelo Volmaro

unread,

Apr 17, 2017, 2:35:39 PM4/17/17

to RavenDB - 2nd generation document database

Hi, I have a DB where I enabled the DocumentExpiration bundle. Also, I have the following indexes on that DB:

public sealed class DebugMessages_PerMessageTypeCounters : AbstractIndexCreationTask<DebugMessage, DebugMessages_PerMessageTypeCounters.MessageTypeCounters>
{
	public sealed class MessageTypeCounters
	{
		public int Count { get; set; }
		public MessageTypes Type { get; set; }
	}

	public DebugMessages_PerMessageTypeCounters()
	{
		Map = messages => messages.Select(message => new MessageTypeCounters
		{
			Count = message.Count,
			Type = message.Type
		});

		Reduce = counters => counters.GroupBy(counter => counter.Type).Select(g => new MessageTypeCounters
		{
			Type = g.Key,
			Count = g.Sum(x => x.Count)
		});
	}
}

and

public sealed class DebugMessages_PerMessageTypeAndAppNameCounters : AbstractIndexCreationTask<DebugMessage, DebugMessages_PerMessageTypeAndAppNameCounters.MessageAppTypeCounters>
{
	public sealed class MessageAppTypeCounters : Equatable<MessageAppTypeCounters>
	{
		public string ApplicationName { get; set; }
		public int Count { get; set; }
		public MessageTypes Type { get; set; }

		public override bool Equals(MessageAppTypeCounters other)
		{
			return ApplicationName.Equals(other.ApplicationName, StringComparison.OrdinalIgnoreCase) && Type == other.Type;
		}

		protected override int HashCode { get { return Avalanche.Create(ApplicationName, Type); } }
	}

	public DebugMessages_PerMessageTypeAndAppNameCounters()
	{
		Map = messages => messages.Select(message => new MessageAppTypeCounters
		{
			ApplicationName = message.ApplicationName,
			Count = message.Count,
			Type = message.Type
		});

		Reduce = items => items.GroupBy(appName => new { appName.ApplicationName, appName.Type }).Select(apps => new MessageAppTypeCounters
		{
			ApplicationName = apps.Key.ApplicationName,
			Type = apps.Key.Type,
			Count = apps.Sum(x => x.Count)
		});
	}
}

The problem I have is that the above indexes are quite expensive on deleting and are making my indexes stale for a long time (more than 6/8 hours).

I really don't need the "Count" to be updated on delete. I just need the count to continue increasing with every new message (or messages, as they can be grouped and that's why I have a Count property on every message).

Is there a way to re-write the above indexes so I can continue to have those values and do not have to worry about deletes?. I'm deleting messages as once they are a week or so old, they are not interesting anymore. But I still need the count as if they where there to draw my statistics...

Oren Eini (Ayende Rahien)

unread,

Apr 18, 2017, 4:02:57 AM4/18/17

to ravendb

The indexes are keeping up to date, but if the source document is deleted, we need to re-run the reduce to remove the data.

The count will go down on delete, and if you are deleting a lot, it can mean that we have to go over a lot of data to update it

Hibernating Rhinos Ltd

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marcelo Volmaro

unread,

Apr 18, 2017, 8:55:26 AM4/18/17

to RavenDB - 2nd generation document database

Yes, I understand that. And the cleanup usually removes 100k+ documents.

Is there a way I can modify the indexes so the Count never get updated on deletes? (Or written in another way: Is there a way I can create indexes that always increase based on the "Count" property but never decrease, even if I delete documents?)

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,

Apr 18, 2017, 9:39:06 AM4/18/17

to ravendb

You can do that using scripted indexes, yes.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Reply all

Reply to author

Forward