jBase monitoring program(s)

[Doug Chanco]

I have gotten quite a few requests for my program(s) basically it allows
you to monitor a jBase system from another system, the ONLY requirements
are ssh installed on both system and perl with the Perl::Telnet module
installed (on the monitoring system)), while it was good enough for me
(style and documented wise its not good enough for mass distribution, so
I am in the process of cleaning it up as well as documentating it more
throughly, I will be done with it by end of week or early next week, so
please be patient, I HOPE it will ber worth your wait!

dougc

[Tony G]

Doug Chanco wrote:
> I have gotten quite a few requests for my program(s)...

I've been resisting the temptation to advertise vaporware but based on the
interest people have expressed it seems now would be a good time to share
some info. Application VARs are often responsible for a large number of
end-users. The only time a VAR hears from some of his clients is when
there is a problem. We've been working on a framework that will allow VARs
to proactively monitor their clients' environments, at the OS, DBMS, and
application levels. The framework consists of plugins which are added into
the client environments to gather data, complimentary Support-side plugins
to interpret the data, and rendering plugins to make the data available in
an attractive and meaningful user interface.

So, for example, the OSDISK01 plugin would be a standard component
distributed to all users of the framework. It simply monitors the amount
of disk available in the host OS under the DBMS/application. The VAR sets
a trigger so that he is notified if any of his clients have less than 20%
disk available on their system. Related parameters might be to monitor
only certain partitions, or to determine if disk is being consumed more
rapidly than normal. Other standard plugins will advise the VAR when
clients systems are crashing, when backups aren't being done, when CPU or
memory user are chronically at some level, etc. VARs will be able to write
their own plugins to monitor their application for runtime errors, out of
balance errors, security errors, or perhaps to monitor application usage to
help determine future enhancements. People will be able to write and sell
plugins that we can't conceive or aren't qualified to write.

The monitor will be sold to VARs but IT managers can also have their own
dashboard. Third-party providers can get a dashboard which only monitor's
hardware, for example, while the VAR only monitors the application - this
allows the right people to keep an eye on what they know best. Support
providers can sell this monitoring service as a new service to existing
clients, or they can cite environment monitoring as a unique free
feature/service in their sales cycle. Some people might see this as simply
an investment in Marketing: your clients will appreciate the fact that
you're keeping an eye out for problems for them, and may be impressed that
you know about issues on their system before they do.

Since the framework is bi-directional, the VAR can upload new plugins to
the end-user sites, add a rendering plugin to his dashboard, and get an
entirely new view on what's happening in the field with no deployment
issues. In a v1.x release we plan to allow VARs to make changes remotely
to end-user environments, and probably in a v2.x they will be able to
upload application updates, tables for taxes or product codes, etc.
Consider the value of being able to update all of your clients without
going on-site, doing a dialup, or scheduling downtime. Or what about
activating a new module in your end-user's application just by clicking a
checkbox in your browser? Again, these update and configuration features
can be offered to end-users as a free part of their package or as a special
for-fee subscription offering.

Progress on this is slow only because it is not funded but any VAR with a
few dozen sites might easily recognize the benefits and provide further
incentive for us to put this on the front burner.

Of course there are other tools in the market for doing this sort of thing
but they are usually OS-specific, very expensive, and they certainly know
nothing about the jBASE/MV DBMS and nothing about business software. The
component-orientation of the framework will allow us to keep the
entry-level costs down, to quickly enhance the software with new common
metrics, and to offer more complex plugins on an optional basis and for
varying fees. We will be happy to recommend plugins offered by other other
developers (so we aren't the only source) and to foster a sort of Open
Source plugin community where people can share their plugins entirely for
free.

If you're interested, please let me know about how many sites you have out
there and we can discuss features, funding, and development schedules.
I'll be happy to chat with people about this at Spectrum. Stop by the
jBASE booth and let's coordinate through Bob Markowitz. Sorry Bob. :)

Tony Gravagno, MV.Monitor@ removethisNebula-RnD.com
Nebula Research and Development is an authorized
Distributor for jBASE and mv.NET

[Clif Bristol]

Will this work with windows or is it a unix based solution?

[Doug Chanco]

It will work with windows PROVIDED you have perl (or more correctly
activeperl) on the machine. The email functionality may have to be
tweaked, since I use sendmail and to be honest really have no idea how
to email from windows ......

dougc

[Tony G]

Clif Bristol wrote:
> Will this work with windows or is it a unix based solution?

For the end-user side, plugins will be written to support as many functions
on as many operating systems as necessary. Each system being monitored in
a network will execute it's own OS-specific code for any given function,
like identifying free disk space, so a VAR will be able to support one
end-user who has several OS's (and DBMS's) on-site.

Initially, data will be aggregated for everyone in a central server hosted
by Nebula R&D, and the only thing support providers will need is a browser.
(No business data will be exchanged so security shouldn't be an issue.)
Later we may work out self-hosted servers so that end-users communicate
directly to the VAR/support-provider and the VAR's dashboard will be served
form that box. This will of course require the use of technologies that
are acceptable in a variety of environments and I'll put off dealing with
that for as long as possible.

Tony
MV.Monitor@ removethisNebula-RnD.com

> From: Tony G
> I've been resisting the temptation to advertise vaporware but based on
the
> interest people have expressed it seems now would be a good time to share

> some info....

[Sanjeev Mulay]

Tony,

One thing that I can say for certain is that a number of small and medium sized Banking institutions use jBase to run their core banking system.  For us, this is a mission-critical application and proactive monitoring of the database layer is an absolute must - something that is not an option at the moment. 

 

While it may not be possible to have monitoring solutions in the class of those available for other DBMS's, any monitoring solution will be a step up from what is available currrently.

 

I am looking forward to Doug's script, but whether Banks (in particular) will agree to have their performance data sent out to a remote site is a question that is a bit more difficult to answer.  My guess is probably not!

 

Just my $0.02.

 

Regards,

[Darrin Filer]

We're looking at all the available solutions for working with jBASE files
on remote computer systems.

We'll need read and delete access to records in the remote files. (Remote
files function as drop-boxes for us to pick up and remove data from.)
Traffic will travel over a 700 mile private IP connection that hiccups
quite rarely.

How do people feel about using nfs for remote jBASE data files? Is the
chance of file corruption too great to even consider it? What are the
potential pitfalls?

[Simon Verona]

I don't think record locking will work properly with nfs... I believe that
this is one of the key things that jRFS handles.

Having said that, I've not used either... so perhaps others are better to
comment!

Regards
Simon

=======================================
Simon Verona
Dealer Management Services Ltd
Stewart House
Centurion Office Park
Julian Way
Sheffield
S9 1GD

Email: si...@dmservices.co.uk
Tel: 0870 080 2300
Fax: 0870 169 6747

-----Original Message-----
From: jB...@googlegroups.com [mailto:jB...@googlegroups.com] On Behalf Of

[Kevin Bilbee]

jRFS will have better performance. Opening a file on NFS over a slow link
will be slow and the locking will not work correctly.

If the apps are not jBasic that are writing tot he files I would liik at
jRCS they have a com component and a .nt component. If .net only then you
could also look at mv.Net

Kevin Bilbee

[Darrin Filer]

Interesting, we've found nfs to perform at least 100 times quicker than
jRFS. But I suspect that this has more to do with our configuration of
jRFS. Yet, even with attribute comparisons set to execute on either client
or server, SELECTs over jRFS are too slow for use in user-interactive
programs. Hence why we were looking into nfs.

For us, record locking isn't a concern. The files operate only as
drop-boxes and locking won't be necessary. Local write and remote read/delete.

I'm not saying nfs is better. Rather, just curious... What are the pitfalls
of using nfs with jBASE.

Darrin

[Kevin Bilbee]

Using NFS the file is processed locally on the client so the entire file has
to be transported over the wire. Do you have the environment variable
"JRFS_CHK_REMOTE = 1"? With this set jRFS will execute the select on the
server not locally. This should speed up your selects.

Kevin

[brucew...@comcast.net]

I wonder if making this file type UD might help things, too. That way, if you need something from remote data, you only pull what it is you need. Just a thought...

 

[Gary Calvin]

You might also consider a message queue or a web services architecture, especially if it's important to ensure that nothing falls through the cracks. We've pretty much replaced all of our drop-box-and-polling integration links (we've had many over the years) with web services, and I find it's much better for performance and reliability.

-Gary-

[Greg Cooper]

Here is some difference between NFS and jRFS. IMHO you should always always always use jRFS, even though it was written by a balding mexican with a dodgy moustache and a panache for firing people (just joking Ted). I'll explain..

 

Lets take the example of a WRITE statement ...

 

With jRFS it will send a single packet to the server with the file ID, the record, the key and flags. If the record is very large it might take more than one packet, but let's assume it is say a 1k record. The jRFS running on the server will perform the update locally and then return a packet with a SUCCESS/FAIL code. So, two network packets to do a WRITE with jRFS.

 

With NFS, then to the eyes of jBASE it is a local file. So , we will take a binary lock (I don't mean a record lock here) on the start of the group where the data resides in the file (1 packet ) then read the group into local memory (1 packet ) modify the group in local memory (0 packets) , write the group to the file (1 packet) and then release the lock (1 packet). So, four network packets to do a WRITE with NFS.

 

In this simple example, using NFS takes double the network traffic.

 

Also, if the file on the server is badly sized, then while jRFS will still only take 2 packets, using NFS will incur additional packets as jBASE reads and writes the data from overflow space caused by the file being badly sized. Hence with a badly sized file, the performance of NFS degrades even more.

 

If the operation was a READ then jRFS still wins, 2 packets for jRFS versus 3 packets for NFS.

 

Now the fun begins. Best to get a coffee and a quiet time ....

 

Let's assume the file on the server is badly sized. As explained already, this incurs extra overhead with NFS. What jBASE will do is to created linked frames to store this extra overflow information and so it will do multiple file writes to maintain these links. As we all know, networks are particularly fickle and if the network goes down while we're doing these writes of links and chains then you have a corrupt file on the server. With jRFS it is an atomic operation, should the network go down while you are using jRFS the update may or may not work but you will not get a corrupt file on the server because of it.

 

So jRFS should be more efficient and cause no file corruptions compared to NFS.

 

To help circumvent this possible file corruption, jBASE has J3 files which are "network friendly" and are designed to work on a NFS network. So when the file overflows, jBASE is very careful about which order it writes the link information back to avoid corruption should the network fail at any point. This involves extra work over NFS and so the operation slows down even more, but at least you don't get a corrupt file. There is no such thing as a free lunch.

 

Note: Using J3 files which are NFS mounted will use NFS to provide record locking over a network. So if you really want to use NFS files, use J3 files. Alternately use J4 files which have been converted to being network friendly (jchmod +N filename) and while this then ensure problems with memory maps and record locking are okay, network friendly J4 files won't provide you with the security of corruption resistance should the network fail (which J3 provides).

 

So jRFS is usually quicker and more reliable than using NFS.

 

One possible caveat is doing a SELECT when the opposite is true (or rather *used* to be true). Imagine a file with 100 small records and a modulo of say 11. With jRFS this would take 100 or more network packets to read each of the records, while with NFS it would take just 11 packets, one for each group. So NFS wins for SELECT. To circumvent this degredation jBASE allows you to configure the SELECT to occur on the server rather than the client, as already mentioned in a previous reply, this involves setting JRFS_CHK_REMOTE = 1.

 

When Darrin says he has found NFS to be 100 times quicker than jRFS, I can only imagine it is because he isn't doing remote SELECTs as mentioned above. I hope you can see from the above why, if configured correctly, jRFS should be the way to go, even if it is a pig to configure. If your 100x speed improvement does not involve SELECT, then my guess is your files may be NFS mounted but you are using J4 files which by default aren't network friendly and use memory maps which are unreliable over NFS mounted files, let alone other problems.

 

Phew, made it ! I hope this all helps, if anyone is still reading this.

 

Greg

Darrin Filer <dfi...@cookvascular.com> wrote:

[Charlie Noah]

Greg,

Not only did I read the whole thing, but I found it extremely interesting
and informative. Since we use jRFS a lot, it is a topic near and dear to
my heart.

Thanks!
Charlie Noah
Inland Truck Parts

Greg Cooper wrote:
> Here is some difference between NFS and jRFS. IMHO you should always
> always always use jRFS, even though it was written by a balding mexican
> with a dodgy moustache and a panache for firing people (just joking
> Ted). I'll explain..
>
> Lets take the example of a WRITE statement ...
>
> With jRFS it will send a single packet to the server with the file ID,
> the record, the key and flags. If the record is very large it might take
> more than one packet, but let's assume it is say a 1k record. The jRFS
> running on the server will perform the update locally and then return a
> packet with a SUCCESS/FAIL code. So, two network packets to do a WRITE
> with jRFS.
>
> With NFS, then to the eyes of jBASE it is a local file. So , we will
> take a binary lock (I don't mean a record lock here) on the start of the
> group where the data resides in the file (1 packet ) then read the group

> into local memory (1 packe t ) modify the group in local memory (0

> operation slow s down even more, but at least you don't get a corrupt

> file. There is no such thing as a free lunch.
>
> Note: Using J3 files which are NFS mounted will use NFS to provide
> record locking over a network. So if you really want to use NFS files,
> use J3 files. Alternately use J4 files which have been converted to
> being network friendly (jchmod +N filename) and while this then ensure
> problems with memory maps and record locking are okay, network friendly
> J4 files won't provide you with the security of corruption resistance
> should the network fail (which J3 provides).
>
> So jRFS is usually quicker and more reliable than using NFS.
>
> One possible caveat is doing a SELECT when the opposite is true (or
> rather *used* to be true). Imagine a file with 100 small records and a
> modulo of say 11. With jRFS this would take 100 or more network packets

> to read each of the records, while with NFS it would ta ke just 11

> packets, one for each group. So NFS wins for SELECT. To circumvent this
> degredation jBASE allows you to configure the SELECT to occur on the
> server rather than the client, as already mentioned in a previous reply,
> this involves setting JRFS_CHK_REMOTE = 1.
>
> When Darrin says he has found NFS to be 100 times quicker than jRFS, I
> can only imagine it is because he isn't doing remote SELECTs as
> mentioned above. I hope you can see from the above why, if configured
> correctly, jRFS should be the way to go, even if it is a pig to
> configure. If your 100x speed improvement does not involve SELECT, then
> my guess is your files may be NFS mounted but you are using J4 files
> which by default aren't network friendly and use memory maps which are
> unreliable over NFS mounted files, let alone other problems.
>
> Phew, made it ! I hope this all helps, if anyone is still reading this.
>
> Greg
>

[Darrin Filer]

Thank you for the excellent info. I've even printed your reply and put it
in my good jBASE info binder. ;-)

This is where a wikipedia style reference for jBASE would be immensely
useful. Many more could benefit from these nuggets of wisdom.

Darrin