[Mifos-developer] Monitoring Tools/Strategies for Mifos

[Ryan Whitney]

I couldn’t find much information on mifos.org, but I think we need to add a page talking about some options people can use for monitoring Mifos.

Any ideas?  Tools?  Strategies?  

I think the areas we would cover are

1. Monitoring Mifos
2. MySQL
3. Tomcat
4. System Performance
   

We should primarily focus on Windows, but I think we can also invite feedback on other platforms as well.

Ryan
Ryan Whitney
Mifos Technical Program Manager
rwhi...@grameenfoundation.org  
Mifos - Technology that Empower Microfinance (www.mifos.org)

[Gbolahan Oshonubi]

Hi Ryan,

Can we look at the following opensoure applications:

1. JFFNMS Just For Fun Network Management                                 http://www.jffnms.org/

2. ODCNMS: Open Data Centre Network Management                    http://www.odcnms.org/

Please let me know if they meet your requirements. We can assist MFIs with implementation.

Thank you,
Gbolahan Oshonubi
Intelligent Network Services Ltd
osho...@inslimited.com
Mobile +234-8023175974
Remain blessed

Disclaimer This e-mail (and any attachment) is intended only for the attention of the addressee(s). Its unauthorised use, disclosure, storage or copying is not permitted. If you are not the intended recipient, please destroy all copies and inform the sender by return e-mail. Internet e-mail is not a secure medium. Any reply to this message could be intercepted and read by someone else. Please bear that in mind when deciding whether to send material in response to this message by e-mail.

---

From: Ryan Whitney <rwhi...@grameenfoundation.org>
Sent: Tuesday, July 08, 2008 12:28 AM
To: Developer <mifos-d...@lists.sourceforge.net>
Subject: [Mifos-developer] Monitoring Tools/Strategies for Mifos

[Graeme Ruthven]

I don't know about monitoring Mifos as such, but I've been running the Munin package for a while. It's Free! :-) With a capital "F"...

 

If there are performance hooks in Mifos it should be reasonably simple to write a plugin to gather and graph data.

 

See http://mifos.kula.co.nz/munin/tuvakawawa/tuvakawawa.html#Tomcat for an example -- and http://mifos.kula.co.nz/munin to see the whole system. It appears looks to be showing a Tomcat memory leak on a quiescent system, by the way.

 

This is pretty much out of the box and I'll probably prune some useless graphs and add some more until I'm satisfied, but I regard it as a good start. Munin also sends email notifications when thresholds are crossed.

 

Given that Mifos is an Open Source package and that we should be looking to minimise the implementation and ongoing cost for an MFI, I favour Linux over Windows as the platform of choice.

 

I'm not looking to start an OS war (let's save that effort for vi vs EMACS :), but don't believe that having a primary focus on Windows is necessarily the best way forward. In my opinion, YMMV, etc, etc, of course.

 

Regards

Graeme

---

From: mifos-devel...@lists.sourceforge.net [mailto:mifos-devel...@lists.sourceforge.net] On Behalf Of Ryan Whitney
Sent: Tuesday, 8 July 2008 10:28
To: Developer

Subject: [Mifos-developer] Monitoring Tools/Strategies for Mifos

[Ryan Whitney]

You are certainly not alone in wanting to promote linux as a solution, we have quite a following internally here within Mifos (and a lot of Mifos is developed on linux ).

Unfortunately, the reality does not line up with the ideal as a lot of MFIs are windows shops.  So when deploying to a new MFI, you have to take into consideration how difficult it would be for them to step up and get familiar with linux, or the cost of finding someone familiar with linux.  Not to mention you’d have to get them to buy off on it in the first place.  So our focus is a reflection of that.

As for the cost comparison, I actually believe that’s not much of an issue.  As Mifos is built on top of MySQL and Tomcat, both free, the real cost would be for the server software, and a license (if the MFI even chooses to buy one...) for something like MS Server 2003 is pretty cheap (rough search brings up about $250-500 per license).  In our experience, that cost is pretty negligible compared to the total cost of a deployment. So in fact linux could potentially cost more, in regards to training needs, new staffing requirements, etc.  

Of course, there are also other considerations like virus, security, third party packages, etc.

All that said, here are plans to support firefox down the road (in fact, we have quite a few bugs already in the queue) but decided it was a lower priority for this release.  I also know there are some MFIs that are running linux or planning too, and we’re putting up more and more information on that topic on mifos.org.  So I think there is quite a bit going on in regards to linux, but as I said, when it comes down to an actual deployment, you have to make the call that best reflects the situation. ;)

Ryan

Ps – Vi(m)

---

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

Ryan Whitney

Mifos Technical Program Manager
rwhi...@grameenfoundation.org  

Mifos - Technology that Empowers Microfinance (www.mifos.org)

[kdur...@aol.com]

Thanks Ryan for the insightful analysis of MIFOS deployment in a Window or Linux environment.

Thanks a great deal.

Kazeem
Sent from my BlackBerry wireless device from MTN

-----Original Message-----
From: Ryan Whitney <rwhi...@grameenfoundation.org>

Date: Tue, 08 Jul 2008 10:11:00
To: Developer<mifos-d...@lists.sourceforge.net>
Subject: Re: [Mifos-developer] Monitoring Tools/Strategies for Mifos

[Adam Monsen]

On Jul 7, 3:28 pm, Ryan Whitney wrote:
> I think we need to add a page talking about some options
> people can use for monitoring Mifos.

Great idea!

> Any ideas?  Tools?  Strategies?

Yes, lots. :)

http://mifos.org/developers/wiki/NetworkMonitoring

Free/Libre/Open Source Software tools for network monitoring are quite
mature. Most of them run on any platform.

--
Adam Monsen

[Soham Dhakal]

Hi Ryan and all,
I do understand your point. And i am not trying to lengthen this
conversation but i just thought i would point out the scenario here in
Nepal (from our limited dealings with MFIs)

The MFIs we are working with do not have any MIS system as such, if they
did (meaning that they already has the financial resources to either
custom develop a solution or buy one from vendor) they are not going to
be that interested in mifos (we have talked to a few that already have
systems) as it lacks a lot of functionality at the local level.
Additionally, because of the centralized nature of the deployment,
connectivity infrastructure costs run higher (again i am talking just
based on my limited experience in nepal).. Thus any saving that can be
realized is really needed.

In nepal:

1. Windows Server 2008 License with 5 User Cal and Media Kit : Nrs.
65,000.00 ( Inclusive of 13% VAT ) (and you need an application server,
a database server and a test server multiply that by 3)
2. Windows XP is about Nrs 26,000
3. Microsoft office ranges from Nrs 11,000 - 20,000
4. So if a MFI has 47 branches, and are going to buy computers for them,
then per branch at the minimum, there is Nrs 37,000 additional cost per
branch. This is excluding the computers needed a the regional offices or
head offices (and they could use open office ;-)
5. Then there is the additional staffing and support requirements,
because of viruses etc. (Anti viruses licenses also are not free,
kaspersky is Nrs 1000 per license).
6. Currently you can get a XP cd (pirated of course) for Nrs100 and
there is no monitoring thus most users (including organizations) use
those. But from 2010 as nepal's WTO entry comes into effect, these
things will be monitored and audited, so the cost savings issue becomes
even more prominent, even for MFIs with existing systems based on windows.

thus total savings for a mid sized MFI with 47 branches would be just
based on licensing cost Nrs 1981000 which is about $28,710.14

also e.g. with latest version of Ubuntu, the user training needed is
comparable to windows. As for admin, at least here in nepal, the number
of training centers giving linux training is growing.

Therefore, imho, if an MFI already has a MIS system and support staff
then they are more likely to continue with windows.
But if an MFI is going from manual to mifos, and starting from scratch,
the savings that they can realize is pretty high.

At the least firefox support ;-)

Disclaimer: The prices quoted are based on vendor responses few months
back, and the actual prices might differ . currrent exchange rate is
about $1 = Nrs 69, per capita income $470

Thanks
Soham

> ------------------------------------------------------------------------
> *From:* mifos-devel...@lists.sourceforge.net
> [mailto:mifos-devel...@lists.sourceforge.net] *On
> Behalf Of *Ryan Whitney
> *Sent:* Tuesday, 8 July 2008 10:28
> *To:* Developer
> *Subject:* [Mifos-developer] Monitoring Tools/Strategies for Mifos

>
>
> I couldn’t find much information on mifos.org, but I think we
> need to add a page talking about some options people can use
> for monitoring Mifos.
>
> Any ideas? Tools? Strategies?
>
> I think the areas we would cover are
>

> 1. Monitoring Mifos
> 2. MySQL
> 3. Tomcat
> 4. System Performance

>
>
> We should primarily focus on Windows, but I think we can also
> invite feedback on other platforms as well.
>
> Ryan

> *Ryan Whitney
> */Mifos Technical Program Manager
> /rwhi...@grameenfoundation.org

> Mifos - Technology that Empower Microfinance (www.mifos.org)
>
>

> ------------------------------------------------------------------------

> -------------------------------------------------------------------------
> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> Studies have shown that voting for your favorite open source project,
> along with a healthy diet, reduces your potential for chronic lameness
> and boredom. Vote Now at http://www.sourceforge.net/community/cca08
>
>

> *Ryan Whitney **
> */Mifos Technical Program Manager
> /rwhi...@grameenfoundation.org

> Mifos - Technology that Empowers Microfinance (www.mifos.org)
>

> ------------------------------------------------------------------------

[Ryan Whitney]

Hi Soham,

Thanks for the insight and I think it provides a good counterpoint to my
e-mail and gets to the heart of the matter.

For any deployment (and really, any software project), you need to use the
technology that makes the most sense for that situation.

Cheers,
Ryan

Ryan Whitney

Mifos Technical Program Manager
rwhi...@grameenfoundation.org

Mifos - Technology that Empowers Microfinance (www.mifos.org)

-------------------------------------------------------------------------

[Ryan Whitney]

Forgot to add my last point

There is a place for both linux and windows with Mifos, but we made the
assessment that the market is a bit bigger at the moment for Windows and so
that's where our initial focus went. Firefox is on the path for getting
support and Mifos already runs on linux ;)

Ryan

[Graeme Ruthven]

> -----Original Message-----
> From: mifos-devel...@lists.sourceforge.net
> [mailto:mifos-devel...@lists.sourceforge.net] On

> Behalf Of Adam Monsen
> Sent: Wednesday, 9 July 2008 06:36
> To: mifos-d...@lists.sourceforge.net
> Subject: Re: [Mifos-developer] Monitoring Tools/Strategies for Mifos
>
> On Jul 7, 3:28 pm, Ryan Whitney wrote:
> > I think we need to add a page talking about some options
> > people can use for monitoring Mifos.
>
> Great idea!

Agreed Adam and Ryan, and the Wiki page is a great start.

> > Any ideas?  Tools?  Strategies?
>
> Yes, lots. :)

Why am I not surprised? ;-)

> http://mifos.org/developers/wiki/NetworkMonitoring
>
> Free/Libre/Open Source Software tools for network monitoring are quite
> mature. Most of them run on any platform.

Picking up on the monitoring thread...

It moved to a discussion of platforms and costs, which is a good and valid
matter but deserves its own thread. I have my own opinions! :-)

I'm not sure what Mifos-specific monitoring would be useful, but would like
to suggest that an API could follow the one used by Tomcat.

Tomcat manager will answer requests with an XML file containing a variety of
information, as in the sample files below.

The Munin agent, and Nagios appears to be similar to the extent that some
agents are interchangeable, simply calls Tomcat then parses the XML that's
returned to store data and generate the graphs, using RRD.

An approach like that would appear to be a pretty good fit and would leave
the display and monitoring to one of the applications available, rather than
having to be developed within Mifos. After all, the interested parties will
probably be system admins and other technically-oriented people, rather than
the Mifos users, so having the data displayed elsewhere is probably a Good
Idea (TM).

If Mifos maintains metrics that could be exposed by an interface like this,
it should be fairly simple to code a transaction to return them. If it
doesn't, and those who know what measurements would be useful in analysing
and tuning the system can define them then, again, implementation should be
simple.

Any other opinions?

Regards
Graeme
--
-------------------------------------------------
Graeme Ruthven
Kula Services Limited
E-mail: gra...@kula.co.nz
Phone: +64 4 477 0898
Fax: +64 4 477 0810
Mobile: +64 27 450 5151
-------------------------------------------------

This example is used by the Munin plugins tomcat_access, tomcat_jvm,
tomcat_threads, and tomcat_volume to produce the graphs seen here:

http://mifos.kula.co.nz/munin/tuvakawawa/tuvakawawa.html#Tomcat

Request:
========

http://tuvakawawa:8180/manager/status?XML=true

Web page displayed (high-resolution ASCII screenshot):
======================================================

Tomcat Status
JVM: free:22197016 total:72310784 max:134152192
Connector -- http-8180
threadInfo maxThreads:150 minSpareThreads:25 maxSpareThreads:75
currentThreadCount:25 currentThreadsBusy:2
requestInfo maxTime:226 processingTime:21456 requestCount:4599
errorCount:4 bytesReceived:0 bytesSent:5617880
Stage Time B Sent B Recv Client VHost Request
R 0 0 ? ? ?????
R 0 0 ? ? ?????
S 4 0 0 10.2.0.5 tuvakawawa
GET/manager/status?XML=trueHTTP/1.1
R 0 0 ? ? ?????
Connector -- jk-8009
threadInfo maxThreads:200 minSpareThreads:4 maxSpareThreads:50
currentThreadCount:4 currentThreadsBusy:1
requestInfo maxTime:0 processingTime:0 requestCount:0
errorCount:0 bytesReceived:0 bytesSent:0
Stage Time B Sent B Recv Client VHost Request

Raw XML corresponding to above:
===============================

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="/manager/xform.xsl" ?>
<status>
<jvm>
<memory free='22197016' total='72310784' max='134152192'/>
</jvm>
<connector name='http-8180'>
<threadInfo maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
currentThreadCount="25" currentThreadsBusy="2" />
<requestInfo maxTime="226" processingTime="21456" requestCount="4599"
errorCount="4" bytesReceived="0" bytesSent="5617880" />
<workers>
<worker stage="R" requestProcessingTime="0" requestBytesSent="0"
requestBytesRecieved="0" remoteAddr="&#63;" virtualHost="&#63;"
method="&#63;"
currentUri="&#63;" currentQueryString="&#63;" protocol="&#63;"
/><worker stage="R" requestProcessingTime="0" requestBytesSent="0"
requestBytesRecieved="0" remoteAddr="&#63;" virtualHost="&#63;"
method="&#63;" currentUri="&#63;" currentQueryString="&#63;"
protocol="&#63;" />
<worker stage="S" requestProcessingTime="4" requestBytesSent="0"
requestBytesReceived="0" remoteAddr="10.2.0.5" virtualHost="tuvakawawa"
method="GET" currentUri="/manager/status"
currentQueryString="XML=true" protocol="HTTP/1.1" />
<worker stage="R" requestProcessingTime="0" requestBytesSent="0"
requestBytesRecieved="0" remoteAddr="&#63;" virtualHost="&#63;"
method="&#63;"
currentUri="&#63;" currentQueryString="&#63;" protocol="&#63;" />
</workers>
</connector>
<connector name='jk-8009'><threadInfo maxThreads="200"
minSpareThreads="4" maxSpareThreads="50" currentThreadCount="4"
currentThreadsBusy="1" />
<requestInfo maxTime="0" processingTime="0" requestCount="0"
errorCount="0" bytesReceived="0" bytesSent="0" />
<workers>
</workers>
</connector>
</status>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

[Van Mittal-Henkle]

Thanks for sharing your ideas on this! Administration and monitoring
are areas that we are very interested in and it's great to get good
input.

Cheers,
--Van

Graeme Ruthven wrote:
> I'm not sure what Mifos-specific monitoring would be useful, but
> would like to suggest that an API could follow the one used by
> Tomcat.
>
> Tomcat manager will answer requests with an XML file containing a
> variety of information, as in the sample files below.
>
> The Munin agent, and Nagios appears to be similar to the extent that
> some agents are interchangeable, simply calls Tomcat then parses the
> XML that's returned to store data and generate the graphs, using RRD.

...