Rhino Queues - Purging improvements

[Ryan]

I've created a pull request to fix issues with purging in Rhino.Queues, namely the VersionStoreOutOfMemory exceptions that can be thrown when purging a large number of messages from the history.  It's now batching the ESENT transactions to prevent a long-running transaction that causes that exception.

I've also added a QueueManagerConfiguration class so you can configure the purge options in your application.  I'll next be submitting a pull request for RSB to expose the QueueManagerConfiguration in the container so it can be modified by the application.  It currently has settings to specify whether to enable/disable processed and outgoing message history, and specify how many/how long messages should be kept in the history. 

It uses the same defaults that were already in Rhino.Queues: history is enabled for processed and outgoing messages, a minimum of 100 messages are kept for both the processed and outgoing histories, and messages are kept in the history for a minimum of one day.  That last setting is where the exceptions were most likely to occur, especially if the application processes a lot of messages each day.  That was also the major contributor to large esent file sizes.  You'll likely want to configure that to something that makes sense for your application.

Any thoughts?
-Ryan

[miles]

Hi Ryan,

That sounds useful.  I wonder if there might be some mileage in making it possible to hook into purge / history to carry out some action?  For example, dumping the messages to some text file instead of purging completely (or if the history is disabled).

Semi-related to this, I was trying to debug an app the other day, I discovered it can be quite tricky to get to look at the actual message history when you don't want to stop the service that's running.  I stumbled across this code which I don't know whether you're aware of:

EsentInteropSamples > DbUtil
http://managedesent.codeplex.com/SourceControl/changeset/view/81201#534393

http://managedesent.codeplex.com/SourceControl/changeset/view/81201#749469
EsentInteropTests > BackupRestoreTests.cs

for backing up and dumping the esent structures.  I don't know whether easy backups and dumps open up any other avenues?

Cheers,

Miles

[Dennis Haney]

Hi Ryan

I have these exact two problems and is very interested in getting those changes. But I am unable to find out where your pull request is so I can copy to my local version and try it out.

[Ryan]

Miles,

Sorry for taking so long to reply.  Somehow I missed your post. 

I've run into friction debugging ESENT files in the past too, and I definitely agree it'd be useful to add in hooks to allow debugging without taking the service offline.  Corey and I have thrown around the idea of having the queues self-host a small web app that would allow you to view diagnostic information about the queues, similar to the FubuMVC.Diagnostics package in FubuMVC.  Corey's actually started work on a new queue project for the next incarnation of FubuTransportation. It would end up being an alternative messaging solution to Rhino.ServiceBus, and built-in diagnostics has been one of the goals.

[Ryan Hauert]

Dennis, the pull request has been merged in and is part of Rhino.Queues 2.1.0.0.  Rhino.ServiceBus 3.1.0.0 and higher has the QueueManagerConfiguration added to the container that you can resolve and configure.

--
You received this message because you are subscribed to a topic in the Google Groups "Rhino Tools Dev" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/rhino-tools-dev/oSI-DpTMfrM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rhino-tools-d...@googlegroups.com.
To post to this group, send email to rhino-t...@googlegroups.com.

Visit this group at http://groups.google.com/group/rhino-tools-dev.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dennis Haney]

Thanks. I updated it and it seems to have solved the problems with the size of the db blowing up.

Now I am getting these :S

Rhino.ServiceBus.RhinoQueues.RhinoQueuesTransport - An error occured while recieving a message, shutting down message processing thread - Rhino Service Bus Worker Thread #2
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'QueueManager'.
   at Rhino.Queues.QueueManager.AssertNotDisposedOrDisposing()
   at Rhino.Queues.QueueManager.GetMessageFromQueue(String queueName, String subqueue)
   at Rhino.Queues.QueueManager.Receive(String queueName, String subqueue, TimeSpan timeout)
   at Rhino.ServiceBus.RhinoQueues.RhinoQueuesTransport.ReceiveMessage(Object context)

And I cant figure out how to restart the processing thread to not just be dead in the water.

--
Dennis

[Ryan Hauert]

I've only seen that error occur if trying to use the bus after it was disposed.  The QueueManager isn't disposed until the bus is disposed.