Adding Performance Counters to Rhino-Queues

[Ken Baltrinic]

This is a continuation of a previous conversation (http://
groups.google.com/group/rhino-tools-dev/browse_thread/thread/
d1ec9838788be43f) the focus of which has shifted enough to merit a new
thread.

So I am now looking at adding performance counters to Rhino-Queus.
Again my intention is to keep it simple to start with and add features
as needed. I am envisioning two categories named "Rhino-Queues
Outbound" and "Rhino-Queues Inbound." Initially each will have one
NumberOfItems32 counter named "Unsent Messages" and "Arrived Messages"
respectively. Outbound instances will be per URI to which messages
are being sent and named accordingly. Inbound instances will be per
port + queue name.

Currently my design calls for adding events to IQueueManager interface
that are raised each time a message is added to or removed from an
inbound or outbound queue; with the performance counter code hooking
into these events and updating the counts appropriately. Each time a
QueueManager is initialized, the counter's values will be synced with
the counts from persistent storage.

Comments?

[Ken Baltrinic]

I have moved ahead with implementing this based on the design I
described above and have pushed my in progress work to my fork on
GitHub (https://github.com/kbaltrinic/rhino-queues/tree/perf-counters)
for review. Currently I have a few concerns.

1) I am currently initializing performance counters by a call to
QueueManager.EnablePerformanceCounters(...); Since this call is made
after the manager is constructed, send and receive operations are
already in progress and there are thus threading concerns with the
PerformanceMonitor.SyncWithCurrentQueueState() method as its trying to
hit a moving target. I would like to hear some ideas on what people
think may be the best way to solve this. I think my favorite option
is to add a override to QueueManager constructor that takes a new
"ConfigurationData" object that can be used to pass in configuration
information. This way, if preformance motinoring is desired, it can
start up before the send and receive threads do. Another option is to
add a Start method to the QueueManager and not start the
automatically, but that is a breaking change, unless we add a
autostart switch to the constructor that defaults to true. Third is
some means of thread synchronization.

2) The integration tests in the
Rhino.Queues.Tests.Monitoring.EnablingPerformanceCounters fixture
required admin rights in order to work as only admin accounts can
create and delete Perf Counter Categories. I am not sure how that
requirement should be handled in the context of this project.

3) The MessageQueuedForReceive_EventNotRaised_IfReceiveAborts test in
the Rhino.Queues.Tests.RaisingReceivedEvents fixture is currently set
to skip because, other than coding a throw statement into the Sender
class and removing it after the test, I can't figure out a set up the
needed conditions for this test.

Lastly, the final piece of this implementation that I have yet to
tackle is setting up the PerformanceMonitor class so that it
participates in transactions. Currently for example, if a message is
sent, the Unsent Messages counter gets incremented immediately and
does not decrement if the tx rolls back. Same thing for Receive,
MoveTo and EnqueueDirectlyTo actions. I have a pretty good idea of
how I intend to handle this. Just throwing it out there as an FYI of
work yet to be done.

--Ken

[Corey Kaylor]

I'll have time to take a look on Friday afternoon. Probably best if I just look at what you've done first before I comment.

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To post to this group, send email to rhino-t...@googlegroups.com.
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.

[Corey Kaylor]

Looks good so far.

1) Personally, I would opt for the breaking change by adding a Start method.

2) I think this will work fine for now. We can likely tweak the build script to elevate if needed.

3) Probably not using QueueManager for the sending and instead use a pseudo Sender that throws for this would work.

On Wed, Mar 23, 2011 at 2:33 PM, Ken Baltrinic <kbalt...@gmail.com> wrote:

[Corey Kaylor]

One other thing. It probably makes sense to install the counters outside of the call to .Enable and likely something that will only get executed from an install action inside the bus. 

[Ken Baltrinic]

Corey,

I have uploaded the modifications needed to make the performance
counters transaction-aware to https://github.com/kbaltrinic/rhino-queues/tree/perf-counters
if you wish to take a look.

I have also made the breaking change to introduce a start method. It
seemed like a good idea to do this in its own branch and let you take
a look at that before I merged it into other changes. The code is at
https://github.com/kbaltrinic/rhino-queues/tree/add-start-method.

I am having an issue with one of the tests I added.
OperationsOnUnstartedQueues.Can_get_processed_messages_without_starting()
always passes in isolation but fails intermittently when run with the
whole test fixture or with all tests. This suggests state is bleeding
between tests but as you can see, I am deleting storage between tests
so there is not much opportunity for bleed. If you or anyone have any
ideas, please chime in.

While you take a look at this, I will continue on with removing
counter category creation to an installation process.

--Ken

[Ken Baltrinic]

Corey,

At this point I have completed all the cleanup and other coding per
the recommendations in both of your posts of Mar 26. The code still
exists in two forks, as above, one for performance counters and one
for adding the start method. I am waiting on your feedback on the
start method code before I proceed.

The only thing left is to integrate the two branches, after which I
will add a line to EnablePerformanceCounters() to throw an exception
if the queue has already been started. Otherwise, barring any
comments to the contrary from you or the community, I will be ready to
submit a pull request.

Have a good weekend,
--Ken

[Ken Baltrinic]

Corey,

I see you pulled the updates to add performance counters into the RQ
master branch. Thanks.

The next step, as I see it, is to make an update to RSB to reference
the new RQ build and add a means of configuring RSB to enable
performance counters in RQ. Sound right to you? Should the build
number be increment in RQ before pulling it into RSB? I am not sure
what your processes is so please advise.

--Ken

[Corey Kaylor]

Yes, the number should get incremented to 1.3. I'll update the build myself
in the next couple days, there are other things that need to be updated as
well. What you've described seems correct. I might suggest keeping your
changes private for the next week or so and rebase after I merge over the
servicelocator branch.

Corey,

--Ken

--

[Ken Baltrinic]

Just an FYI. the project on which we need the RSB/RQ work has been
temporarily shelved due to an emergency elsewhere, so I probably will
not be active on this for at least a few weeks, could be a few months.

[Corey Kaylor]

Thanks for the update.

-----Original Message-----
From: rhino-t...@googlegroups.com
[mailto:rhino-t...@googlegroups.com] On Behalf Of Ken Baltrinic
Sent: Friday, May 13, 2011 3:48 PM
To: Rhino Tools Dev
Subject: [rhino-tools-dev] Re: Adding Performance Counters to Rhino-Queues

--

[Ken Baltrinic]

Corey,

I am getting back to this project, at least for a short time. Doesn't
look like the RQ build # has been updated yet. Any chance to get this
done soon? I am hoping to get the upgraded RQ integrated with RSB and
send you a pull request for that ASAP but need the new # and anything
else you feel needs done to RQ first.

--Regards,
Ken

On May 10, 10:50 am, "Corey Kaylor" <co...@kaylors.net> wrote:
> Yes, the number should get incremented to 1.3. I'll update the build myself
> in the next couple days, there are other things that need to be updated as
> well.  What you've described seems correct. I might suggest keeping your
> changes private for the next week or so and rebase after I merge over the
> servicelocator branch.
>
>
>
>
>
>
>
> -----Original Message-----
> From: rhino-t...@googlegroups.com
>
> [mailto:rhino-t...@googlegroups.com] On Behalf Of Ken Baltrinic
> Sent: Tuesday, May 10, 2011 5:36 AM
> To: Rhino Tools Dev

> Subject: [rhino-tools-dev] Re: AddingPerformanceCountersto Rhino-Queues
>
> Corey,
>
> I see you pulled the updates to addperformancecountersinto the RQ master

[Corey Kaylor]

This should be taken care of now.

[Ken Baltrinic]

Thanks Corey,

I have made the updates to RSB to incorporate RQ 1.3 and sent you a
pull request.

--Ken

[Ken Baltrinic]

Corey,

Have you had a chance to look at my pull request for RSB? Any issues
that I need to address?

--Ken

[Corey Kaylor]

I've been on vacation and we've been in and out camping. I'll have a chance Friday when i return to take a look. Sorry for the delay.