ClientDataSet speed problem
Roland
So far, the ClientDataSet's big problem is speed, from developper view, we
really don't care or might not be aware the mechanism behind the TTable,
TAdoDataSet or TClientDataSet, but we really care the speed, and the
consistent of development process. Unforunately ClientDataSet made
developpers to change their used way to display data(using TQuery or TTable
display whole data and using TClientDataSet you have to display small amount
data and still slow). Pratically displaying data and modifying data can be
two seperate process, especially in 3 tier(you don't use DBGrid to display
huge data and moidfy it, you might search and display them then get one
record to do modification). From that point view, I think if TClientDataSet
have a flag to say, this is for view only, you don't need load unneccesary
information into ClientDataSet, it might speed up the process.
This is my assumption, it may be wrong because I don't know what's happened
behind the TClientDataSet, but the speed will be the biggest obstacle for
developpers using TClientDataSet.
--
Roland Zhang
Development
Compdata Technology Services
Bjørge Sæther
news:3f4ab5ea$1...@newsgroups.borland.com...
TClientDataSet won't be as fast as a direct connection to the database. Of
course. Data transfer & network is doubled, if AppServer is on a different
machine than the db-server. So, why do you use TClientDataSet ? What is the
reason you didn't stick with TQuery/TTable (or similar concepts on other
"middleware") ?
BTW, I feel that lack of direct transaction support is worse...
--
Regards,
Bjørge Sæther
bjorge@haha_itte.no
-------------------------------------
I'll not spend any money on American Software products
until armed forces are out of Iraq.
vavan
wrote:
>So far, the ClientDataSet's big problem is speed, from developper view, we
not to mention its capacity
>This is my assumption, it may be wrong because I don't know what's happened
>behind the TClientDataSet, but the speed will be the biggest obstacle for
>developpers using TClientDataSet.
the problem isn't TClientDataSet itself (it's just kind of high-level
wrapper) but its internal mechanisms and structures buried in
midas.dll
personally I recently started to test and evaluate kbmt (the fastest
dataset) and even use it in one of the new project. Quite impressive
--
Vladimir Ulchenko aka vavan
vavan
>not to mention its capacity
and whole bag of bugs
Dave Rowntree
>So far, the ClientDataSet's big problem is speed, from developper view, we
>really don't care or might not be aware the mechanism behind the TTable,
>TAdoDataSet or TClientDataSet,
Then I suggest you probably get what you deserve. Anyone using tools
(any tools) without bothering to learn how to use them really has no
right to complain if they get poor results.
> but we really care the speed,
then learn how to use the tools.
> and the
>consistent of development process. Unforunately ClientDataSet made
>developpers to change their used way to display data
It is most certainly not unfortunate if you want to produce n-tier
apps. It is most certainly not unfortunate if you want to produce apps
that can work online/offline (briefcase). It is most certainly not
unfortunate if you want to make good use of network bandwidth and DB
resources.
>(using TQuery or TTable
>display whole data
No they don't. They display a section of table data at a time. They
also have to be continuously connected to the DB, constantly making DB
requests, continually consuming large amounts of DB and network
resources.
> and using TClientDataSet you have to display small amount
Sure, at least that's what you should be doing. CDS's (in a well
designed system where the designer has learnt how to use the tools)
also use small amounts of network and DB resources.
> Pratically displaying data and modifying data can be
>two seperate process, especially in 3 tier
It can be if you want it to be, it doesn't have to be.
>(you don't use DBGrid to display
>huge data
How many user's do you know that can effectively browse huge amounts
of data in a DBGrid. Normally making someone have to browse large
amounts of data to find what they are looking for is extremely
unproductive and in my book constitutes poor (and normally lazy)
design.
> and moidfy it, you might search and display them then get one
>record to do modification).
Again, you can if you want but you don't have to.
> From that point view, I think if TClientDataSet
>have a flag to say, this is for view only, you don't need load unneccesary
>information into ClientDataSet, it might speed up the process.
Please give an example of unnecessary information that MIDAS(datasnap)
brings into CDSs that you would like to prevent.
>This is my assumption, it may be wrong because I don't know what's happened
>behind the TClientDataSet, but the speed will be the biggest obstacle for
>developpers using TClientDataSet.
It is often more meaningful to have knowledge and experience of
something before criticizing it ;)
--
Dave Rowntree
Dave Rowntree
>not to mention its capacity
That depends on the usage you put it to. If you are trying to use a
CDS to locally store huge amounts of data then I would suggest that
you are using it for something it isn't good at anyway, and probably
wasn't designed to do.
--
Dave Rowntree
C4D - Kim Madsen
I might read your reply a bit more harsh than what you meant it to read.
But although I agree with you most of the way, I know of quite many situations where people needs loads of data in a
client dataset, specially for briefcase situations where there is no desire or need for any 'real' embedded database.
Actually CDS seems to have a problem with the number of records it can handle even of the recordstorage might not be big
memorywise, specially when you have blob fields in the record. One blob field seems to reduce the number of records
handled by CDS to only 32768, more blob fields seems to worsen the situation.
That number of records is not really huge, specially in briefcase situations. Thus I think the author starting this
thread do have a point.
--
best regards
Kim Madsen
k...@components4developers.com
www.components4developers.com
The best components for the best developers
kbmMW - kbmMemTable - kbmWABD - kbmX10
"Dave Rowntree" <daver@spam_offbrookswood.co.uk> skrev i en meddelelse
news:r7jmkvson2kvr87u7...@4ax.com...
Roland
> (any tools) without bothering to learn how to use them really has no
> right to complain if they get poor results.
Can you look into midas.dll and know exactly what's inside?
> No they don't. They display a section of table data at a time. They
> also have to be continuously connected to the DB, constantly making DB
> requests, continually consuming large amounts of DB and network
> resources.
No. Component speed itself and network resource is different issue. I would
like a component has fast capacity to fetch amount of data from network
rather then say because of network resource so we have to make a component
with limitation for data fetching.
> Sure, at least that's what you should be doing. CDS's (in a well
> designed system where the designer has learnt how to use the tools)
> also use small amounts of network and DB resources.
Is this a well designed system? The actor(designer)'s perform should be
recognized by audiance not actor self.
> It can be if you want it to be, it doesn't have to be.
That's what I sugest to minimize overhead loading(unnecessary information if
only for display purpose). I hope the designer can look into the users'
response and to improve their products rather then complain users' unaware.
If not, CDS's market or Delphi's market will prove that.
> How many user's do you know that can effectively browse huge amounts
> of data in a DBGrid. Normally making someone have to browse large
> amounts of data to find what they are looking for is extremely
> unproductive and in my book constitutes poor (and normally lazy)
> design.
You are right. Users are lazy otherwise we don't need to develop good
application for them. Remember, DBGrid is just an example. CDS can affect
all other components having feature to display a list of data.
> Please give an example of unnecessary information that MIDAS(datasnap)
> brings into CDSs that you would like to prevent.
As I said I don't know what's inside the MIDAS. It's only my suggestion or
assumption. The designer knows that and I hope designers can improve MIDAS a
lot including bug fixes and speed of course.
> It is often more meaningful to have knowledge and experience of
> something before criticizing it ;)
Criticizing it is not expert's right . Certainly I have knowledge and
experience of using CDS. You can defend MIDAS, and I hope MIDAS is every
one's first choice. But can you tell me CDS is superior then any other
third party components regarding to speed? I love Delphi, I love Borland.
That's why CDS is my only component for 3 tier development instead of using
third party components. But unfortunately, I had pain experiences with CDS's
speed and bugs. Why should I stick to CDS? This is what my collegue asked me
at the beginning of 3 tier development.
Roland Zhang
vavan
<daver@spam_offbrookswood.co.uk> wrote:
>That depends on the usage you put it to. If you are trying to use a
>CDS to locally store huge amounts of data then I would suggest that
no, I don't anymore :) there are robust alternatives which let me
handle sizeable amounts of data while being TDataSet in its nature at
the same time. I understand that one size doesn't fit all (it is
applicable to advice not to load huge amounts of data as well. Not
only human being wants to use CDS but some services too :) but CDS
SHOULD definitely operate faster and more error-free to be competitive
while there are such mature 3-rd party products (kbmt, asta to name a
few) with the same (and even richer) functionality. You know it is
very comfortable to have the same full featured TDataSet to handle a
lot of data in some real-time high performance service or use it as a
container linked to data-aware end-user <g>
>you are using it for something it isn't good at anyway, and probably
>wasn't designed to do.
though I only can speculate about goals of CDS I can see clearly how
one can spoil good idea with bad implementation. I consider lack of
support of midas.dll as another fail of borland. If only borland
changed its attitude/support or publish midas sources...
Dave Rowntree
>Can you look into midas.dll and know exactly what's inside?
No, but that does not prevent me from learning how to use MIDAS.
>> No they don't. They display a section of table data at a time. They
>> also have to be continuously connected to the DB, constantly making DB
>> requests, continually consuming large amounts of DB and network
>> resources.
>
>No. Component speed itself and network resource is different issue. I would
>like a component has fast capacity to fetch amount of data from network
>rather then say because of network resource so we have to make a component
>with limitation for data fetching.
Unless you are fetching unreasonably large amounts of data from a DB
into a CDS, (both in terms of width and length), or fetching large
amounts of complex nested structures, the time taken to pull data from
the DB into the CDS is normally insignificant from an end user
perspective.
The time taken to pull say 500 records with maybe 20 fields into a CDS
could be in the order of 0.07 seconds. A TTable doing the same job
could take about 0.03 seconds. If you are saying that the difference
of 0.04 seconds is a problem, then I rest my case.
>> Sure, at least that's what you should be doing. CDS's (in a well
>> designed system where the designer has learnt how to use the tools)
>> also use small amounts of network and DB resources.
>
>Is this a well designed system? The actor(designer)'s perform should be
>recognized by audiance not actor self.
I fail to see the relevance of that statement.
>>> Pratically displaying data and modifying data can be
>>>two seperate process, especially in 3 tier
>> It can be if you want it to be, it doesn't have to be.
>
>That's what I sugest to minimize overhead loading(unnecessary information if
>only for display purpose).
As I already said, it can be if you want, but it doesn't have to be.
> I hope the designer can look into the users'
>response and to improve their products rather then complain users' unaware.
>If not, CDS's market or Delphi's market will prove that.
I'm not sure I understand what you mean.
If you mean you think Borland should take note of what you have said
here, then I for one sincerely hope not. All I see here is you
complaining about the fact that MIDAS is designed differently to the
BDE, and requires different application design techniques.
If you mean you as a designer should improve your product in response
to your end users complaints, then of course, that is up to you.
>> How many user's do you know that can effectively browse huge amounts
>> of data in a DBGrid. Normally making someone have to browse large
>> amounts of data to find what they are looking for is extremely
>> unproductive and in my book constitutes poor (and normally lazy)
>> design.
>
>You are right. Users are lazy otherwise we don't need to develop good
>application for them.
I think you may have miss-understood me here. I meant the designer is
lazy just hooking up a TTable to a DBGrid as a means for the end user
to look through and locate one record of interest out of an entire DB
table.
> Remember, DBGrid is just an example. CDS can affect
>all other components having feature to display a list of data.
Sure. However, any UI control that displays a list as a means to
locate an item of interest should be of a usable length. Displaying a
list containing several thousand items and expecting an end user to
look through to locate one item of interest is totally unproductive.
An example of this poor, lazy design could be hooking up a TTable to a
DBGrid. A much more pro-active way for an end user to locate a record
of interest is for an application to request selection criteria from
an end user before fetching any data. The more selection criteria they
enter, the shorter the selection list becomes.
>> Please give an example of unnecessary information that MIDAS(datasnap)
>> brings into CDSs that you would like to prevent.
>
>As I said I don't know what's inside the MIDAS. It's only my suggestion or
>assumption. The designer knows that and I hope designers can improve MIDAS a
>lot including bug fixes and speed of course.
Bug fixes are a totally different matter and are not what you have
been complaining about in this thread.
What is inside Midas.dll is of little importance compared to the
importance of learning how to use and design systems for memory based,
detached dataset systems such as MIDAS.
>> It is often more meaningful to have knowledge and experience of
>> something before criticizing it ;)
>
>Criticizing it is not expert's right . Certainly I have knowledge and
>experience of using CDS. You can defend MIDAS, and I hope MIDAS is every
>one's first choice.
My reply to your post was not necessarily to jump to the defence of
MIDAS, but to decry your negativity towards learning how to use new
tools, and then blaming the tools for the results you get; in this
case a different dataset concept compared to the BDE dataset concept.
> But can you tell me CDS is superior then any other
>third party components regarding to speed?
I have never said it was.
--
Dave Rowntree
Dave Rowntree
(kbmMW/kbmMemTable/kbmWABD/kbmX10)> wrote:
>But although I agree with you most of the way, I know of quite many situations where people needs loads of data in a
>client dataset, specially for briefcase situations where there is no desire or need for any 'real' embedded database.
I would agree with you (partially anyway <g>) if that was what this
thread was about. Mainly, this thread is about the design differences
requirement between BDE apps and a memory based, detached dataset
system, in this case MIDAS; and the fact that the author thinks that
there shouldn't be any differences, and that he apparently shouldn't
have to learn anything new.
>Actually CDS seems to have a problem with the number of records it can handle even of the recordstorage might not be big
>memorywise, specially when you have blob fields in the record. One blob field seems to reduce the number of records
>handled by CDS to only 32768, more blob fields seems to worsen the situation.
Agreed. This is a known problem.
>That number of records is not really huge, specially in briefcase situations. Thus I think the author starting this
>thread do have a point.
I would agree if this thread was about the number of records you can
fit into a CDS. As it is not, I disagree.
--
Dave Rowntree
Dave Rowntree
>On Tue, 26 Aug 2003 14:18:54 +0100, Dave Rowntree
><daver@spam_offbrookswood.co.uk> wrote:
>
>>That depends on the usage you put it to. If you are trying to use a
>>CDS to locally store huge amounts of data then I would suggest that
>
>no, I don't anymore :)
I was speaking hypohetically, what you choose to do is of course up to
you.
> I understand that one size doesn't fit all (it is
>applicable to advice not to load huge amounts of data as well. Not
>only human being wants to use CDS but some services too :)
Indeed, however this thread related to displaying records in a DBGrid
for human consumption hence my comments.
> I consider lack of
>support of midas.dll as another fail of borland. If only borland
>changed its attitude/support or publish midas sources...
I agree entirely. Craig Stuntz is pressing hard for this and other
source code release. Let's hope he is successful :)
--
Dave Rowntree
Roland
Yes. The learning result is you have to limit your data fetching, you have
to change your UI to comply with MIDAS.
> Unless you are fetching unreasonably large amounts of data from a DB
> into a CDS, (both in terms of width and length), or fetching large
> amounts of complex nested structures, the time taken to pull data from
> the DB into the CDS is normally insignificant from an end user
> perspective.
What's your conclusion comes from? What's the reasonable amount of data?
Records is just one factor, the others like fields amount in record, field
size, etc.
>
> The time taken to pull say 500 records with maybe 20 fields into a CDS
> could be in the order of 0.07 seconds. A TTable doing the same job
> could take about 0.03 seconds. If you are saying that the difference
> of 0.04 seconds is a problem, then I rest my case.
Are you playing toy application? Did you test systematically?
> >That's what I sugest to minimize overhead loading(unnecessary information
if
> >only for display purpose).
>
> As I already said, it can be if you want, but it doesn't have to be.
It's not I want. It's I have to. I have to do the changes in
design(including UI design) to comply with MIDAS's technique and have a
complain of course about that changes after that experience. It's relied on
Borland's improvement. If you don't want other people to see Borland's
improvement in their products including MIDAS then I give up.
> If you mean you think Borland should take note of what you have said
> here, then I for one sincerely hope not. All I see here is you
> complaining about the fact that MIDAS is designed differently to the
> BDE, and requires different application design techniques.
Don't be harsh. Don't mis interprete what I said. SPEED IMPROVEMENT. That's
my central topic. That's what my complains.
>
> If you mean you as a designer should improve your product in response
> to your end users complaints, then of course, that is up to you.
If you think Borland treat MIDAS as not need to be improved, that will be
sad for Borland. Might not be sad for you. Whatever Borland hear from me or
not, it's up to Borland, but it's the place provided by Borland that I can
express what my thoughts.
> I think you may have miss-understood me here. I meant the designer is
> lazy just hooking up a TTable to a DBGrid as a means for the end user
> to look through and locate one record of interest out of an entire DB
> table.
>
> > Remember, DBGrid is just an example. CDS can affect
> >all other components having feature to display a list of data.
>
> Sure. However, any UI control that displays a list as a means to
> locate an item of interest should be of a usable length. Displaying a
> list containing several thousand items and expecting an end user to
> look through to locate one item of interest is totally unproductive.
> An example of this poor, lazy design could be hooking up a TTable to a
> DBGrid. A much more pro-active way for an end user to locate a record
> of interest is for an application to request selection criteria from
> an end user before fetching any data. The more selection criteria they
> enter, the shorter the selection list becomes.
What about lookup? Simple example, when you filling country, or zip code,
you need lookup from thess dataset. In real world, not every case suitable
for narrow search. I had experiences, the other developpers too.
> What is inside Midas.dll is of little importance compared to the
> importance of learning how to use and design systems for memory based,
> detached dataset systems such as MIDAS.
>
In which words did you find my complaining about learning MIDAS?
> My reply to your post was not necessarily to jump to the defence of
> MIDAS, but to decry your negativity towards learning how to use new
> tools, and then blaming the tools for the results you get; in this
> case a different dataset concept compared to the BDE dataset concept.
>
Sorry. Don't mis interprete what I means. Again I am complaining something
has to be changed for both developer and end users by using MIDAS not about
learning thing.
> > But can you tell me CDS is superior then any other
> >third party components regarding to speed?
>
> I have never said it was.
Of course you didn't say. That's my question. Can you answer to me?
Roland Zhang
Roland
> thread was about. Mainly, this thread is about the design differences
> requirement between BDE apps and a memory based, detached dataset
> system, in this case MIDAS; and the fact that the author thinks that
> there shouldn't be any differences, and that he apparently shouldn't
> have to learn anything new.
Don't rape author's original post meaning.
Roland Zhang
Roland
> for human consumption hence my comments.
Did I tell you I want to display huge amount of data into DBGrid? The fact
is you have to change you design to comply with MIDAS because of speed
problem(not network resource issue) and we all hope Borland to improve it
especially if 3rd party components are better then CDS regarding speed.
Roland Zhang
vavan
wrote:
>> Indeed, however this thread related to displaying records in a DBGrid
>> for human consumption hence my comments.
>
>Did I tell you I want to display huge amount of data into DBGrid? The fact
>is you have to change you design to comply with MIDAS because of speed
Yes and no. Yes, CDS seem to be the... ehmm, not the best :) regarding
speed comparing with competitive components. No, you HAVE to change
your design no matter what is your tool set of choice is if you want
to build solid multi-tier system
>problem(not network resource issue) and we all hope Borland to improve it
>especially if 3rd party components are better then CDS regarding speed.
as Dave already said, Craig is "pressing hard" to force borland to
release sources of midas.dll
if borland wants to keep the developers building modern distributed
systems and attract more they at borland should change their mind or
they lost the market. midas in its current incarnation isn't worthy to
choose (did I said this?? :). Let's hope we'll see its reborn :)
Dave Rowntree
That certainly wasn't my intention. I commented on what I understood
from your post.
--
Dave Rowntree
Dave Rowntree
>Yes. The learning result is you have to limit your data fetching, you have
>to change your UI to comply with MIDAS.
For the fourth time (copied from a previous post in this thread):
"Not really. As I have already said, and now I see vavan has also
re-iterated, you have to change the design because you are now using a
memory based, detached dataset on the client. You happen to be using
MIDAS, but the same equally applies to all other memory based,
detached dataset systems. These are basic building blocks for
multi-tier solutions, as is the need to keep data transfer volumes to
a sensible level. Until you understand and accept that concept, you
are unlikely to be happy producing multi-tier apps."
>> Unless you are fetching unreasonably large amounts of data from a DB
>> into a CDS, (both in terms of width and length), or fetching large
>> amounts of complex nested structures, the time taken to pull data from
>> the DB into the CDS is normally insignificant from an end user
>> perspective.
>
>What's your conclusion comes from? What's the reasonable amount of data?
>Records is just one factor, the others like fields amount in record, field
>size, etc.
That is what "width" means.
>>
>> The time taken to pull say 500 records with maybe 20 fields into a CDS
>> could be in the order of 0.07 seconds. A TTable doing the same job
>> could take about 0.03 seconds. If you are saying that the difference
>> of 0.04 seconds is a problem, then I rest my case.
>
>Are you playing toy application? Did you test systematically?
<g> No. A quick, crude test. However, the number of records quoted
should be concidered more than you would normally fetch into a DBGrid
for end user browsing.
>> >That's what I sugest to minimize overhead loading(unnecessary information
>if
>> >only for display purpose).
>>
>> As I already said, it can be if you want, but it doesn't have to be.
>
>It's not I want. It's I have to. I have to do the changes in
>design(including UI design) to comply with MIDAS's technique and have a
>complain of course about that changes after that experience.
Same complaint again, please see previous four answers to this point.
I can understand your frustration on converting an existing app if you
thought you could directly replace BDE datasets with
CDS/DSP/SQLDataSet. I think information from Borland saying that
dbexpress replaces the BDE is misleading, but there has been quite a
lot written in the newsgroups and in BDN articles about converting.
> It's relied on
>Borland's improvement. If you don't want other people to see Borland's
>improvement in their products including MIDAS then I give up.
I'm not sure I understand. Perhaps you think I am from Borland. I am
not.
>> If you mean you think Borland should take note of what you have said
>> here, then I for one sincerely hope not. All I see here is you
>> complaining about the fact that MIDAS is designed differently to the
>> BDE, and requires different application design techniques.
>
>Don't be harsh. Don't mis interprete what I said. SPEED IMPROVEMENT. That's
>my central topic. That's what my complains.
And I am saying that unless you learn how to use and design for a
memory based detached dataset multi-tier system, your problems are
unlikely to go away.
>> If you mean you as a designer should improve your product in response
>> to your end users complaints, then of course, that is up to you.
>
>If you think Borland treat MIDAS as not need to be improved, that will be
>sad for Borland. Might not be sad for you. Whatever Borland hear from me or
>not, it's up to Borland, but it's the place provided by Borland that I can
>express what my thoughts.
Of course I want to see MIDAS improved. Have a look at the Midas area
in QC. I am involved with virtually every entry in there, and a lot of
the reports have been entered by me. Why on earth would I put that
much effort into something I don't want to see improved!
>> I think you may have miss-understood me here. I meant the designer is
>> lazy just hooking up a TTable to a DBGrid as a means for the end user
>> to look through and locate one record of interest out of an entire DB
>> table.
>>
>> > Remember, DBGrid is just an example. CDS can affect
>> >all other components having feature to display a list of data.
>>
>> Sure. However, any UI control that displays a list as a means to
>> locate an item of interest should be of a usable length. Displaying a
>> list containing several thousand items and expecting an end user to
>> look through to locate one item of interest is totally unproductive.
>> An example of this poor, lazy design could be hooking up a TTable to a
>> DBGrid. A much more pro-active way for an end user to locate a record
>> of interest is for an application to request selection criteria from
>> an end user before fetching any data. The more selection criteria they
>> enter, the shorter the selection list becomes.
>
>What about lookup? Simple example, when you filling country, or zip code,
>you need lookup from thess dataset. In real world, not every case suitable
>for narrow search. I had experiences, the other developpers too.
If you mean you want use a LookupField to provide a pick list
containing thousands of entries, then you appear to have not grasped
the design concept of memory based datasets. You also appear to have
ignored my statement in the previous paragraph:
"However, any UI control that displays a list as a means to locate an
item of interest should be of a usable length. Displaying a list
containing several thousand items and expecting an end user to look
through to locate one item of interest is totally unproductive."
>> What is inside Midas.dll is of little importance compared to the
>> importance of learning how to use and design systems for memory based,
>> detached dataset systems such as MIDAS.
>>
>
>In which words did you find my complaining about learning MIDAS?
"So far, the ClientDataSet's big problem is speed, from developper
view, we really don't care or might not be aware the mechanism behind
the TTable, TAdoDataSet or TClientDataSet"
>> > But can you tell me CDS is superior then any other
>> >third party components regarding to speed?
>>
>> I have never said it was.
>
>Of course you didn't say. That's my question. Can you answer to me?
Not from personal experience, no.
--
Dave Rowntree
Dave Rowntree
>> Indeed, however this thread related to displaying records in a DBGrid
>> for human consumption hence my comments.
>
>Did I tell you I want to display huge amount of data into DBGrid? The fact
>is you have to change you design to comply with MIDAS because of speed
>problem(not network resource issue)
Not really. As I have already said in a different leg of this thread,
and now I see vavan has also re-iterated in this leg, you have to
change the design because you are now using a memory based, detached
dataset on the client. You happen to be using MIDAS, but the same
equally applies to all other memory based, detached dataset systems.
These are basic building blocks for multi-tier solutions, as is the
need to keep data transfer volumes to a sensible level. Until you
understand and accept that concept, you are unlikely to be happy
producing multi-tier apps.
> and we all hope Borland to improve it
>especially if 3rd party components are better then CDS regarding speed.
Agreed.
--
Dave Rowntree
Dave Rowntree
I disagree with you.
> Let's hope we'll see its reborn :)
I certainly hope it receives the attention it deserves ;)
--
Dave Rowntree
vavan
<daver@spam_offbrookswood.co.uk> wrote:
>> midas in its current incarnation isn't worthy to
>>choose
>
>I disagree with you.
would you recommend it to someone from the outside in its present
state? would you point him to QC at the same time? I wouldn't :)
>> Let's hope we'll see its reborn :)
>
>I certainly hope it receives the attention it deserves ;)
this could remove my doubts, but why I still feel so sceptical? :) OK,
I believe you know much more what's happening there
Dave Rowntree
>>I certainly hope it receives the attention it deserves ;)
>
>this could remove my doubts, but why I still feel so sceptical? :) OK,
>I believe you know much more what's happening there
Didn't mean to imply anything by my comment.
--
Dave Rowntree
vavan
<daver@spam_offbrookswood.co.uk> wrote:
>Didn't mean to imply anything by my comment.
OK, then I won't make up any unfounded conclusions
Bjørge Sæther
news:m8qrkvc3uiusqr5m7...@4ax.com...
> On Thu, 28 Aug 2003 12:10:40 +0100, Dave Rowntree
> <daver@spam_offbrookswood.co.uk> wrote:
>
> >> midas in its current incarnation isn't worthy to
> >>choose
> >
> >I disagree with you.
>
> would you recommend it to someone from the outside in its present
> state? would you point him to QC at the same time? I wouldn't :)
I kind of understand your feelings on MIDAS. I've been working on a MIDAS
project (D5) for about 2 years, and it *still* frustrates me.
There are a few problems of non-technical nature involved:
- The user base isn't that large. When you have a specific question and do a
web search, you notice the small number of relevant postings, articles,
etc..
- Additionally, TClientDataSet is used for a large variety of applications -
some of them "real n-tier", but maybe most of them utilize the spin-off: A
simple local, flat-file database. Even XML-capable. Number of developers who
know e.g. the combination "MIDAS / Oracle / TWebConnection" is not
particularily large.
- "n-tier" was a commonly encountered term a few years ago. Less common was
relevant examples of practical implementations where the n-tier model
actually *did* improve applications. One example is the problem with
"business rules in middle tier". In most cases, I've found business rules to
belong to the database rather than the middle tier. What the middle tier
*should* provide, though, was heavy support for formatting data, defining
lookups, etc., as well as a straight, multi-dataset transaction control.
In short: "What was MIDAS designed for ?", and "How does one actually put
functionality in AppServer and avoid creating "heavy" clients ?"
I do fully see the problems with the traditional C/S model, with datasets
being "too hands-on". But: i.e. BDE *does* actually cache data on client
side, much like TClientDataSet. I must admit, I'm still very much confused
about the usage of TClientDataSet.
C4D - Kim Madsen
> - "n-tier" was a commonly encountered term a few years ago. Less common was
> relevant examples of practical implementations where the n-tier model
> actually *did* improve applications. One example is the problem with
> "business rules in middle tier". In most cases, I've found business rules to
> belong to the database rather than the middle tier. What the middle tier
> *should* provide, though, was heavy support for formatting data, defining
> lookups, etc., as well as a straight, multi-dataset transaction control.
Unless you want to be strapped into one single database type, I strongly disagree here.
I have seen numerous times companies who are looking into getting out of one database and into another one.
Reasons: Cost, maintainance, obsolete technology etc.
But they often realize that its an impossible task without rewriting their complete app. suite which may be many
manyears of work simply because they have entered business code into the datastorage.
Maintainance wise the applications also becomes a nightmare. Delphi/Kylix/BCB or even VB really have much better
debuggers than the absolute majority of database's embedded languages. A few databases now supporting Java as the
language for stored procedures have improved debugging slightly, but its generally still much more cumbersome to setup a
good stable debugging environment.
One of the most successful, in terms of sold licenses and applications running, n-tier setups are J2EE (Java Enterprise
Edition) either utilizing RMI or EJB's. The RAD in this can be discussed, but the idea who everybody coding J2EE
follows, is to put business code in the application server and not in the database.
There can ofcourse be some situations that benefit from code in the database itself. Like to handle autoincrementing
values, referential integrety and other extremely databound operations. In a very few special cases one can benefot from
putting business code into the database too, and thats when some major computation has to be made on the data where
retrieving the data first to an external application would be way too big a job to have the app server do the
computation.
In real life, that situation covers less than 1% of all business code.
By seperating the business code from the database, changing databases, making failover and loadbalancing becomes much
easier.
In this case some type of virtual datastorage is needed, hence TClientDataset or TkbmMWxxxQuery/StoredProc etc.
> In short: "What was MIDAS designed for ?", and "How does one actually put
> functionality in AppServer and avoid creating "heavy" clients ?"
Its not only heavy clients you would avoid, you would also gain the posibility to have more control and flexibility with
the way your data is transferred from a to b. A normal C/S setup usually have serious trouble moving data cross a
firewall. Most companies do not allow opening for the ports used by the propritary protocols. (Dont know if Midas
handles this transparently though, but kbmMW does).
Eduardo Andrade Bahiense
Finally someone show me a light!!!
That's encouraging for people who are starting a project and have to lead
with so many options.
The old Keep it Simple and as much over your control as you can are still
big rules.
Best regards, Eduardo
"C4D - Kim Madsen" <k...@components4developers.com
(kbmMW/kbmMemTable/kbmWABD/kbmX10)> escreveu na mensagem
news:3f4e...@newsgroups.borland.com...
Roland
Topic: CDS speed problem.
Contents: Because of CDS speed, I have to redesign the application and the
end users have to accept the UI modification even they don't like the
changes.
Suggestion: Hope Borland can have a way to improve it (Assume having feature
only for displaying and to minimize all other features)
I can not see this is my complaining about learning MIDAS. Don't tell me I
need learn how to build distributed app or how to use MIDAS. With fast
loading feature, I am sure it will be applauded by developers.
> For the fourth time (copied from a previous post in this thread):
> "Not really. As I have already said, and now I see vavan has also
> re-iterated, you have to change the design because you are now using a
> memory based, detached dataset on the client. You happen to be using
> MIDAS, but the same equally applies to all other memory based,
> detached dataset systems. These are basic building blocks for
> multi-tier solutions, as is the need to keep data transfer volumes to
> a sensible level. Until you understand and accept that concept, you
> are unlikely to be happy producing multi-tier apps."
Yes. I am unhappy with MIDAS perfomance. I already changed my design and my
3 tier application is nearly finished. The more I do with MIDAS the more
dispoint I have. Speed is only one issue, the other is bug which I have
posted but no response and I will post it later. These kind of things made
me lost confidence for MIDAS and push me seek third party components in
future.
> <g> No. A quick, crude test. However, the number of records quoted
> should be concidered more than you would normally fetch into a DBGrid
> for end user browsing.
>
Again. I never use CDS to load data to DBGrid to end user browsing. Bon't be
harsh. If crude test can be used to convince other people, what can I say.
> Of course I want to see MIDAS improved. Have a look at the Midas area
> in QC. I am involved with virtually every entry in there, and a lot of
> the reports have been entered by me. Why on earth would I put that
> much effort into something I don't want to see improved!
So why on earth people who complain CDS speed are labeled as complaining
learning CDS?
> If you mean you want use a LookupField to provide a pick list
> containing thousands of entries, then you appear to have not grasped
> the design concept of memory based datasets. You also appear to have
> ignored my statement in the previous paragraph:
> "However, any UI control that displays a list as a means to locate an
> item of interest should be of a usable length. Displaying a list
> containing several thousand items and expecting an end user to look
> through to locate one item of interest is totally unproductive."
Please see my comments on the top of this thread. Why shouldn't have a fast
loading component for displaying purpose? Never try? or Tried can not be
made? or Don't want try?
Roland Zhang
Roland
fact
> >is you have to change you design to comply with MIDAS because of speed
> >problem(not network resource issue)
>
> Not really. As I have already said in a different leg of this thread,
> and now I see vavan has also re-iterated in this leg, you have to
> change the design because you are now using a memory based, detached
> dataset on the client. You happen to be using MIDAS, but the same
> equally applies to all other memory based, detached dataset systems.
> These are basic building blocks for multi-tier solutions, as is the
> need to keep data transfer volumes to a sensible level. Until you
> understand and accept that concept, you are unlikely to be happy
> producing multi-tier apps.
Your statement like this: Because the speed limit is 110km/h, so the car
maker made your car only can reach that max speed, and you can not drive
beyond the limit.
Same as: Because of 3 tier technique, you have to load a "reasonable" data
to client, so MIDAS implement exactly under that rule. You are not allowed
to load "unreasonable" data from server otherwise you will have pain.
Roland Zhang
Bjørge Sæther
news:3f4e...@newsgroups.borland.com...
> Hi,
>
> > - "n-tier" was a commonly encountered term a few years ago. Less common
was
> > relevant examples of practical implementations where the n-tier model
> > actually *did* improve applications. One example is the problem with
> > "business rules in middle tier". In most cases, I've found business
rules to
> > belong to the database rather than the middle tier. What the middle tier
> > *should* provide, though, was heavy support for formatting data,
defining
> > lookups, etc., as well as a straight, multi-dataset transaction control.
>
> Unless you want to be strapped into one single database type, I strongly
disagree here.
> I have seen numerous times companies who are looking into getting out of
one database and into another one.
> Reasons: Cost, maintainance, obsolete technology etc.
Well, where do I start...
First, being able to create a solid, maintainable database on 1 platform is
more important than the cross-platform stuff. I don't say I like to be
married to one database vendor, but I'm not the only one who has (at least
for now) been forced to choose a db-specific solution over e.g. BDE. Simply
because there are important issues that generalized components don't handle.
> But they often realize that its an impossible task without rewriting their
complete app. suite which may be many
> manyears of work simply because they have entered business code into the
datastorage.
> Maintainance wise the applications also becomes a nightmare.
Delphi/Kylix/BCB or even VB really have much better
> debuggers than the absolute majority of database's embedded languages. A
few databases now supporting Java as the
> language for stored procedures have improved debugging slightly, but its
generally still much more cumbersome to setup a
> good stable debugging environment.
Second, what one does is simply reducing whatever DB engine is used, to a
passive database like Access or paradox. I am supposed to write all of the
desired functionality into the AppServer, so that I may change DB's with
little effort. That's just fine, as long as multi-db support is a topic.
What I usually end up with, is a metadata database, holding all information
on tables / fields / relations, etc.. The database also has this, so I'll
have to choose what model to use. I've done both - describing the entire
structure in my metadatabase *and* extending the db's system tables with my
own "goodies". Anyhow, there is little support for handling this in
AppServer. I have e.g. not had success in transferring those metadata via
the Provider. Some of it, yes, but not all.
Also. I've been forced to write a quite sophisticated exception handling, to
deal with a large variety of error situations. Example: A column has a
unique index. If I load 1 record, modify this field and in that operation
violate the unique constraint, I get an error from the DB or from AppServer
(depending on what layer checks the uniqueness) upon ApplyUpdates. If I have
loaded the record with the matching duplicate, I get a "Key violation" upon
Post;. I can't explain this to the users, as I'd have to use one day to
explain about ther MIDAS concept. I just have to fix it.
> One of the most successful, in terms of sold licenses and applications
running, n-tier setups are J2EE (Java Enterprise
> Edition) either utilizing RMI or EJB's. The RAD in this can be discussed,
but the idea who everybody coding J2EE
> follows, is to put business code in the application server and not in the
database.
RAD is for children or non-programmers. It does *allways* show that the RAD
stuff better be moved elsewhere (from .dfm files, e.g.).
What about e.g. persistent datasets / fields ? A no-no, if you ask me.
Keeping the model persistent is hard when you have scattered designtime
objects all around. Or, maybe we're talking about totally different kinds of
applications ?
> There can ofcourse be some situations that benefit from code in the
database itself. Like to handle autoincrementing
> values, referential integrety and other extremely databound operations. In
a very few special cases one can benefot from
> putting business code into the database too, and thats when some major
computation has to be made on the data where
> retrieving the data first to an external application would be way too big
a job to have the app server do the
> computation.
> In real life, that situation covers less than 1% of all business code.
What is "business code" then ?
- Data validation, of course
- creating ID's
- "xxx should happen if I insert one row in yyy" stuff
- Handling access rights topics
- filtering of data
- ???
...or ??? This is something I actually miss, concrete examples of what one
actually puts in "business rules layer". All I've seen is silly stuff like
maximum amounts for employee's salaries, etc (things that sooner or later
create problems). My main problem may be that I don't like applications who
let me enter wrong data and say "Illegal value" when I post. I need this
information *in the control* ! In my opinion, having the AppServer layer
knowing what's a valid value is useless unless it may be propagated to the
client's field controls.
> By seperating the business code from the database, changing databases,
making failover and loadbalancing becomes much
> easier.
> In this case some type of virtual datastorage is needed, hence
TClientDataset or TkbmMWxxxQuery/StoredProc etc.
No, this is not the only solution. What's needed is an abstraction layer,
not an additional memory buffer location. Unless this is desired for
specific needs, of course. The BDE failing to be such an abstraction layer
doesn't mean MIDAS succeeds. I haven't checked out your concept, but I'd not
be surprised if it is competetive.
> > In short: "What was MIDAS designed for ?", and "How does one actually
put
> > functionality in AppServer and avoid creating "heavy" clients ?"
>
> Its not only heavy clients you would avoid, you would also gain the
posibility to have more control and flexibility with
> the way your data is transferred from a to b. A normal C/S setup usually
have serious trouble moving data cross a
> firewall. Most companies do not allow opening for the ports used by the
propritary protocols. (Dont know if Midas
> handles this transparently though, but kbmMW does).
This issue is a rather new one, and of course, if DB clients can't work
without opening a lot of ports, this may in itself stop them. OTOH, if I was
head of security, I'd not pe specially happy about a Windows .exe file
talking to the outside world through port 80. If I was to modify firewall
setup, there would be little difference between limiting ip ranges on port
80 and allowing e.g. IB traffic over port 3050.
C4D - Kim Madsen
> Well, where do I start...
> First, being able to create a solid, maintainable database on 1 platform is
> more important than the cross-platform stuff.
I completely agree! No idea to build a brilliant castle on a shaky foundation :)
But that should be the fundamental rule for all programming anyway, n-tier or not.
n-tier do not exclude that.
> I don't say I like to be married to one database vendor, but I'm not the only one who has (at least
> for now) been forced to choose a db-specific solution over e.g. BDE. Simply
> because there are important issues that generalized components don't handle.
We can agree that some access methods are superiour to others, and some offer more detailed information/control on that
specific databases inner structures.
But 99.9% of all enterprise business solutions actually couldnt care less about those structures. Those types of
projects deal with business data rather than with database administration, and the end user or the developer for that
matter could in most cases not care less what the name of the datastorage is.
There most often are some political/strategic decisions which dictate the use or preference of one datastore instead of
another, but functionality wise just about all databases can store and fetch data. Some faster or more elegant than
others. Some with higher stability than others.
I _do_ know of major projects which are built on Oracle and which utilize just about every feature of the database.
These projects usually started out with having a lowend frontend application written in Designer/2000, Forms or some
other older Oracle product which required you to code lots and lots of business code inside the database. Today
everybody agrees about that if one would do it again, it would not be the way to do it today.
Even on the mainframe front, people are trying to strive for more standardisation, seperation of business code from the
datastorage etc. All because the costs of running a mainframe is terribly high, and in lots of situations could be
reduced by moving some of the business code to a mini. This is only doable employing some n-tier technology of some
kind. Today you have Java and Corba as the main n-tier technologies on mainframes.
> Second, what one does is simply reducing whatever DB engine is used, to a
> passive database like Access or paradox. I am supposed to write all of the
> desired functionality into the AppServer, so that I may change DB's with
> little effort.
Or so you are able to much easier to scale your application. Putting all kinds of stuff inside your database strains the
CPU and storage systems its running on.
Everybody knows that to make a database perform, you need lots of spindles and a fair amount of CPU power. At some level
you have to purchase very expensive equipment to continue to scale on one machine. If you need more than 2 CPU's or more
than 4-8 harddrives you have to start looking around for much more expensive hardware. And if the need for that
equipment is only one or two hours a day (because thats when your db usage spikes), you have actually wasted your money
the rest of the time.
If you would move some of the load of the db to an app server, you in many situations can avoid having to invest loads
of money.
> That's just fine, as long as multi-db support is a topic.
> What I usually end up with, is a metadata database, holding all information
> on tables / fields / relations, etc.. The database also has this, so I'll
> have to choose what model to use. I've done both - describing the entire
> structure in my metadatabase *and* extending the db's system tables with my
> own "goodies". Anyhow, there is little support for handling this in
> AppServer. I have e.g. not had success in transferring those metadata via
> the Provider. Some of it, yes, but not all.
> Also. I've been forced to write a quite sophisticated exception handling, to
> deal with a large variety of error situations. Example: A column has a
> unique index. If I load 1 record, modify this field and in that operation
> violate the unique constraint, I get an error from the DB or from AppServer
> (depending on what layer checks the uniqueness) upon ApplyUpdates. If I have
> loaded the record with the matching duplicate, I get a "Key violation" upon
> Post;. I can't explain this to the users, as I'd have to use one day to
> explain about ther MIDAS concept. I just have to fix it.
>
Midas might not do the job this way, but that does not rule out n-tier as a whole. Midas was designed for what it was
designed for.
There are other designs which may encourage and allow you to do much more sophisticated things. I could name one :D
(check my signature)
> RAD is for children or non-programmers. It does *allways* show that the RAD
> stuff better be moved elsewhere (from .dfm files, e.g.).
Definitely not agreeing about RAD is for children or non programmers :)
RAD stands for Rapid Application Development, not for a specific way of doing things.
You may dislike to design your server using datamodules, fields etc in your final product, but its most likely that you
have started out like that
in your project to create a prototype which you could demo. In my, quite long I may humbly add :), experience its _much_
faster to get a project done,
getting some 95% specs, prototying your way along using RAD development, fixing the stuff the remaining 5 missing %
inflicts on the project, and be able to show a prototype of what the final product will be on the way. If I would write
all this from scratch, not employing RAD principes, it would take me minimum twice the time, and often would still not
satisfy the customer since the result comes as a surprise for them not as easy being part of the development process
along the way.
> What about e.g. persistent datasets / fields ? A no-no, if you ask me.
> Keeping the model persistent is hard when you have scattered designtime
> objects all around. Or, maybe we're talking about totally different kinds of
> applications ?
It depends on your n-tier setup.
If you are talking a 2-tier setup (C/S), then we can agree that it could be a disaster on a large project to put fields
in designtime code since all clients would have to be updated on each database change.
But if you would have a 4-tier setup: DB - DB frontend - Business app server - client, you could code all the designtime
persistent fields you want to since its all abstracted on the way to the database. You could even do it with a 3-tier
solution but a 4-tier solution just makes it a little bit more elegant.
Change fields and stuff on your database, modify the DB frontend app server to cope with the new changes, providing a
standardised API to the next tier.
You wouldnt need to change anything in either your business code or client code.
In complex projects there are always those 5% cases where this does not apply, but as long the n-tier product do not
block your abilities to get around it, it will still make your application much more scalable and stable. Development
time may not be significantly shorter on the initial base parts of the project, since structures has to be laid out and
strategies implemented. But after that stage, adding and testing new functionality will go much faster.
> What is "business code" then ?
> - Data validation, of course
> - creating ID's
> - "xxx should happen if I insert one row in yyy" stuff
> - Handling access rights topics
> - filtering of data
> - ???
- Datavalidation actually covers two areas... datavalidation which has to do with the structural integrity of the
database (unique fields etc.), and datavalidation which has to do with simply validating business data (name must be
filled, account numbers valid etc.). Datavalidation which have to do with the structural integrity of the database would
most often be placed within the database, unless the only access to the database would guaranteed be from the app server
in which case those validations can be placed in the app server.
- Creating ID's. This require some pragmatic thinking... if the project should be able to cover multiple databases
(might even be of same type), and the id should be guaranteed unique and valid on all databases, the creation of ID's
should be the job of the app server. If you however are only using one database, you can use sequences etc. methods on
the database if you want to. If portability is an issue, then just make sure that those ID's are transferable. Generally
its preferred to use unique ID's that are user generated, ie. social security numbers, password numbers etc. when ever
possible. This makes a potential transfer much easier and a recovery of the database much easier on a nasty partial
crash which usually have a negative effect on database created ID's.
- xxx should happen... If its of RI type of operation, that should be defined on the database if others can access the
database not going through the app server. If not then its up to your preferences. If the app server contains your RI,
it makes it easier to move to another database, even one that do not support stored procedures and triggers.
- Access rights. If the only way to access the database is through the app server, then having access authorization
there makes your life easier, but generally I would say most projects do require authentication on the database and I
would recommend at least having a basic auth. scheme setup there in _addition_ to a authrization scheme on the app
server. Database authorization in most cases only handle table, view, storedproc or schema granularity authorizations.
On the app server you can add more higher level granularities which operates on the functionality rather than on the raw
data. Generally makes security better and easier maintained if made right.
- Filtering of data. This is one of the places where code within the database can be justified imo. It depends on the
amount of data being filtered versus the network setup, database load etc. Lots of factors has to be weighed up to
figure out where its best placed. But tendency is still that most people do not want the network to move data that will
later not be used because they are filtered out later in the chain.
> ...or ??? This is something I actually miss, concrete examples of what one
> actually puts in "business rules layer". All I've seen is silly stuff like
> maximum amounts for employee's salaries, etc (things that sooner or later
> create problems). My main problem may be that I don't like applications who
> let me enter wrong data and say "Illegal value" when I post. I need this
> information *in the control* ! In my opinion, having the AppServer layer
> knowing what's a valid value is useless unless it may be propagated to the
> client's field controls.
A setup could be like this: Imaging server handling image reading, correction, scanning, storage, validation and
extraction to some media:
- One business object (kbmMW service) would handle raw image storage and retrieval from a datastore.
- One business object would handle image manipulations.
- One business object would be a high level entrance into OCRing an image and returning data from it.
- Under it you would have multiple different business objects each functioning as an abstraction layer towards each
their specific OCR/ICR engine (eg. MitekSys, Kadmos, Inerural, TextBridge, FineReader etc etc.) This abstraction makes
it possible for the service to read with the reader best suited for reading that specific part of the document.
- One business object would be handling validation of resulting inputs from either automatic OCR or manual input.
- Finally one business object would be able to extract documents and data in different ways needed.
The client then generally is only providing a display and asking the app server to most of the work. This makes it much
easier to add other types of clients later on.
> No, this is not the only solution. What's needed is an abstraction layer,
> not an additional memory buffer location. Unless this is desired for
> specific needs, of course. The BDE failing to be such an abstraction layer
> doesn't mean MIDAS succeeds. I haven't checked out your concept, but I'd not
> be surprised if it is competetive.
I agree, its not the only solution but for database type operations in the Borland world, it certainly gives a much
easier accesspoint to the backend database treating it as rows and fields, rather than some other propritary way of
getting to the data. Further, if you need scalablity and 100% uptime, you need some way of run your app server and
client in disconnected ways (not constantly connected to one database). To make this succeed, you _do_ need some level
of caching at least the latest resultset.
> This issue is a rather new one, and of course, if DB clients can't work
> without opening a lot of ports, this may in itself stop them. OTOH, if I was
> head of security, I'd not pe specially happy about a Windows .exe file
> talking to the outside world through port 80. If I was to modify firewall
> setup, there would be little difference between limiting ip ranges on port
> 80 and allowing e.g. IB traffic over port 3050.
The difference in security is quite big, because it disallows any malicious user to try to enter the database directly.
There is a filter infront of it which, designed properly, _only_ allows for specific operations towards the database.
This extra security ofcourse disappears with most current n-tier implementations which require specification of SQL on
the client and thus allow the client
to execute just about any query.
In kbmMW we encourage people to put the SQL on the app server instead, not having _any_ SQL in the client. This way a
user that has been given authorization to run the query will never be able to do things outside what has been defined in
the SQL statements on the server.
Further you can enable highly secure encryption, SSL/TLS, compression and lots of other things which makes the
communication much more difficult/impossible to monitor and spoof. With a direct hole to the DB, you would be at the
mercy of the standard settings of that database which very rarely require any encryption at all.
Bjørge Sæther
(kbmMW/kbmMemTable/kbmWABD/kbmX10)> skrev i melding
Most of all, Kim - thank you for useful input on n-tier application
development !
> > I don't say I like to be married to one database vendor, but I'm not the
only one who has (at least
> > for now) been forced to choose a db-specific solution over e.g. BDE.
Simply
> > because there are important issues that generalized components don't
handle.
>
> We can agree that some access methods are superiour to others, and some
offer more detailed information/control on that
> specific databases inner structures.
> But 99.9% of all enterprise business solutions actually couldnt care less
about those structures. Those types of
> projects deal with business data rather than with database administration,
and the end user or the developer for that
> matter could in most cases not care less what the name of the datastorage
is.
> There most often are some political/strategic decisions which dictate the
use or preference of one datastore instead of
> another, but functionality wise just about all databases can store and
fetch data. Some faster or more elegant than
> others. Some with higher stability than others.
What I meant was more like that there are differences in the internals of DB
servers that affect AppServer implementation. Example: MS SQL server has
AutoInc fields, while IB & Oracle uses generators.
But more important: As long as there is no BDE to use, one has to deal with
totally different component sets for different DB's. All these component
suites support TDataSet, but they're never the *preferred* apporoach. It may
be that dbExpress is a good model, but so far it doesn't seem like (from
what I've heard).
> I _do_ know of major projects which are built on Oracle and which utilize
just about every feature of the database.
> These projects usually started out with having a lowend frontend
application written in Designer/2000, Forms or some
> other older Oracle product which required you to code lots and lots of
business code inside the database. Today
> everybody agrees about that if one would do it again, it would not be the
way to do it today.
No, the Database Desktop Develompent era is definately over. thanks god.
> Even on the mainframe front, people are trying to strive for more
standardisation, seperation of business code from the
> datastorage etc. All because the costs of running a mainframe is terribly
high, and in lots of situations could be
> reduced by moving some of the business code to a mini. This is only doable
employing some n-tier technology of some
> kind. Today you have Java and Corba as the main n-tier technologies on
mainframes.
Well, here we're getting closer: We're talking about applications that need
access from different "channels", and we replace the direct db engine access
with a more intellingent AppServer. We agree (I guess) that an important
question is:
- What protocol do I use ("protocol" as in "interface between two layers")
?
This is important, as it amonst other things decides what's contained in
AppServer and whet belongs to the Client.
Clearly seen in WebSnap concept, where one actually puts client-side stuff
in WebServer application !
Attempting to establish a fruitful way of creating an AppServer with
"intellingence", the term "report engine" has emerged. Well, it has to be an
editor engine too, of course, but the idea of an application reachable to
both regular clients and web applications, to pass simple request and get a
result set in return, properly selected, formattered & ordered...
> > Second, what one does is simply reducing whatever DB engine is used, to
a
> > passive database like Access or paradox. I am supposed to write all of
the
> > desired functionality into the AppServer, so that I may change DB's with
> > little effort.
>
> Or so you are able to much easier to scale your application. Putting all
kinds of stuff inside your database strains the
> CPU and storage systems its running on.
This is the *strong* side of DB-servers - proper handling of large amounts
of data, together with secure file access & concurrency. One doesn't want to
write an AppServer that duplicates the core of the DB.
> Everybody knows that to make a database perform, you need lots of spindles
and a fair amount of CPU power. At some level
> you have to purchase very expensive equipment to continue to scale on one
machine. If you need more than 2 CPU's or more
> than 4-8 harddrives you have to start looking around for much more
expensive hardware. And if the need for that
> equipment is only one or two hours a day (because thats when your db usage
spikes), you have actually wasted your money
> the rest of the time.
Requesting a db isn't that different from requesting an AppServer. Being
able to divide work between multiple servers is a solution, of course, but
then we're talking about larger applications than I've ever been involved
with.
- Is this the main goal of an n-tier application ?
> If you would move some of the load of the db to an app server, you in many
situations can avoid having to invest loads
> of money.
I don't quite get this. We're more talking about replication &
synchronisation, which IMHO is an area where I would *not* trust MIDAS. Its
lack of transparency between AppServer and Client makes it very easy to
screw up and leaves the client with possibly wrong data. Also the lack of
real transaction support is serious when dealing with large, rapidly
changing databases.
<snip>
> > If I have loaded the record with the matching duplicate, I get a
> > "Key violation" upon Post;. I can't explain this to the users, as
> > I'd have to use one day to explain about the MIDAS concept.
> > I just have to fix it.
>
> Midas might not do the job this way, but that does not rule out n-tier as
a whole. Midas was designed for what it was
> designed for.
No, as I asked: "*What* was it designed for ?"
> There are other designs which may encourage and allow you to do much more
sophisticated things. I could name one :D
> (check my signature)
yes, I know and use the Memtable stuff, but I'm not sure I have the guts to
go and check out another n-tier concept
...right now, at least ;-)
> > RAD is for children or non-programmers. It does *allways* show that the
RAD
> > stuff better be moved elsewhere (from .dfm files, e.g.).
>
> Definitely not agreeing about RAD is for children or non programmers :)
RAD as in "Drop a component" is what I meant - what Borland is bragging
about on the Delphi box.
> RAD stands for Rapid Application Development, not for a specific way of
doing things.
> You may dislike to design your server using datamodules, fields etc in
your final product, but its most likely that you
> have started out like that
> in your project to create a prototype which you could demo. In my, quite
long I may humbly add :), experience its _much_
> faster to get a project done,
> getting some 95% specs, prototying your way along using RAD development,
fixing the stuff the remaining 5 missing %
> inflicts on the project, and be able to show a prototype of what the final
product will be on the way. If I would write
> all this from scratch, not employing RAD principes, it would take me
minimum twice the time, and often would still not
> satisfy the customer since the result comes as a surprise for them not as
easy being part of the development process
> along the way.
I find that this breaks about any mental model of programming. OOP it is
not, as using standardized components that you modify manually makes
applications hard to read. I did this exercise today, examining a number of
TClientDataSets to try figuring out why two forms worked differnetly. I'm
using form inheritance, and a lot of code is common to these forms. But
designtime components are manually modified in another "source file (.dfm) !
The first things to do is writing a few standard procedures to set values
that normally is set in IDE (Providername, Remote Module, etc.), and
standard routines to open whatever datasets are needed based on visible
controls. Etc., etc...
> > What about e.g. persistent datasets / fields ? A no-no, if you ask me.
> > Keeping the model persistent is hard when you have scattered designtime
> > objects all around. Or, maybe we're talking about totally different
kinds of
> > applications ?
>
> It depends on your n-tier setup.
> If you are talking a 2-tier setup (C/S), then we can agree that it could
be a disaster on a large project to put fields
> in designtime code since all clients would have to be updated on each
database change.
> But if you would have a 4-tier setup: DB - DB frontend - Business app
server - client, you could code all the designtime
> persistent fields you want to since its all abstracted on the way to the
database. You could even do it with a 3-tier
> solution but a 4-tier solution just makes it a little bit more elegant.
> Change fields and stuff on your database, modify the DB frontend app
server to cope with the new changes, providing a
> standardised API to the next tier.
Exactly. When a string field is changed form 20 to 50 chars, this should of
course not require a client side recompilation.
> You wouldnt need to change anything in either your business code or client
code.
> In complex projects there are always those 5% cases where this does not
apply, but as long the n-tier product do not
> block your abilities to get around it, it will still make your application
much more scalable and stable. Development
> time may not be significantly shorter on the initial base parts of the
project, since structures has to be laid out and
> strategies implemented. But after that stage, adding and testing new
functionality will go much faster.
My problem is getting the basics to work, as I have a hard time getting the
feel for what strategy to use. Changing things within the working model is
easy.
> > What is "business code" then ?
> > - Data validation, of course
> > - creating ID's
> > - "xxx should happen if I insert one row in yyy" stuff
> > - Handling access rights topics
> > - filtering of data
> > - ???
>
> - Datavalidation actually covers two areas... datavalidation which has to
do with the structural integrity of the
> database (unique fields etc.), and datavalidation which has to do with
simply validating business data (name must be
> filled, account numbers valid etc.). Datavalidation which have to do with
the structural integrity of the database would
> most often be placed within the database, unless the only access to the
database would guaranteed be from the app server
> in which case those validations can be placed in the app server.
The validation stuff can't be buried down in the database. When youn
ApplyUpdates on a bunch of tables in one transaction, there must be better
solutions than waiting for the error dialog. Validation should be done
already in the Client. This may have been solved by some
TClientDataSet.Validate; call, where one could get error reports before
actually applying any updates.
> - Creating ID's. This require some pragmatic thinking... if the project
should be able to cover multiple databases
> (might even be of same type), and the id should be guaranteed unique and
valid on all databases, the creation of ID's
> should be the job of the app server. If you however are only using one
database, you can use sequences etc. methods on
> the database if you want to. If portability is an issue, then just make
sure that those ID's are transferable. Generally
> its preferred to use unique ID's that are user generated, ie. social
security numbers, password numbers etc. when ever
> possible. This makes a potential transfer much easier and a recovery of
the database much easier on a nasty partial
> crash which usually have a negative effect on database created ID's.
Ideally, this should be a two-step thingy:
1. Client requests new ID from AppServer.
2. AppServer manufactures the ID, the Client is indifferent to how it
happened.
I discussed this today with the DB maintainer of the Midas project, where
IDs are retrieved from, incremented & stored in tables via a procedure. When
exchanging those tables with Oracle sequences, only the metatable for field
definitions needs updating. No recompilation necessary.
> - xxx should happen... If its of RI type of operation, that should be
defined on the database if others can access the
> database not going through the app server. If not then its up to your
preferences. If the app server contains your RI,
> it makes it easier to move to another database, even one that do not
support stored procedures and triggers.
Agreed. We have put very little RI checking in the DB.
> - Access rights. If the only way to access the database is through the app
server, then having access authorization
> there makes your life easier, but generally I would say most projects do
require authentication on the database and I
> would recommend at least having a basic auth. scheme setup there in
_addition_ to a authrization scheme on the app
> server. Database authorization in most cases only handle table, view,
storedproc or schema granularity authorizations.
> On the app server you can add more higher level granularities which
operates on the functionality rather than on the raw
> data. Generally makes security better and easier maintained if made right.
>
> - Filtering of data. This is one of the places where code within the
database can be justified imo. It depends on the
> amount of data being filtered versus the network setup, database load etc.
Lots of factors has to be weighed up to
> figure out where its best placed. But tendency is still that most people
do not want the network to move data that will
> later not be used because they are filtered out later in the chain.
I think it would be pure madness not to utilize DB filtering, unless
AppServer could easily hold all table data (=minor tables). In my Midas
project, I've defined a few static datasets, that needs not be retrieved
upon every use. Once they're loaded, they stay clientside.
> > ...or ??? This is something I actually miss, concrete examples of what
one
> > actually puts in "business rules layer". All I've seen is silly stuff
like
> > maximum amounts for employee's salaries, etc (things that sooner or
later
> > create problems). My main problem may be that I don't like applications
who
> > let me enter wrong data and say "Illegal value" when I post. I need this
> > information *in the control* ! In my opinion, having the AppServer layer
> > knowing what's a valid value is useless unless it may be propagated to
the
> > client's field controls.
>
> A setup could be like this: Imaging server handling image reading,
correction, scanning, storage, validation and
> extraction to some media:
Yes, I know this one. I once used a "download & thumbnail engine" that
communicated with the client.
> - One business object (kbmMW service) would handle raw image storage and
retrieval from a datastore.
> - One business object would handle image manipulations.
> - One business object would be a high level entrance into OCRing an image
and returning data from it.
> - Under it you would have multiple different business objects each
functioning as an abstraction layer towards each
> their specific OCR/ICR engine (eg. MitekSys, Kadmos, Inerural, TextBridge,
FineReader etc etc.) This abstraction makes
> it possible for the service to read with the reader best suited for
reading that specific part of the document.
> - One business object would be handling validation of resulting inputs
from either automatic OCR or manual input.
> - Finally one business object would be able to extract documents and data
in different ways needed.
What about rendering reports ? Could be something...not having to compile
the entire ReportBuilder into the client !
Ha ! I got an Idea...maybe move all the MS Office (Word, Excel)
communications to AppServer, and return result files. Could be handy on Web,
too.
;-)
> The client then generally is only providing a display and asking the app
server to most of the work. This makes it much
> easier to add other types of clients later on.
Yes. The problem then is to create an intelligent & elegant client with the
whole spectre of possible GUI goodies...the reason why we're not
particularily fond of HTML forms...
> > No, this is not the only solution. What's needed is an abstraction
layer,
> > not an additional memory buffer location. Unless this is desired for
> > specific needs, of course. The BDE failing to be such an abstraction
layer
> > doesn't mean MIDAS succeeds. I haven't checked out your concept, but I'd
not
> > be surprised if it is competetive.
>
> I agree, its not the only solution but for database type operations in the
Borland world, it certainly gives a much
> easier accesspoint to the backend database treating it as rows and fields,
rather than some other propritary way of
> getting to the data. Further, if you need scalablity and 100% uptime, you
need some way of run your app server and
> client in disconnected ways (not constantly connected to one database). To
make this succeed, you _do_ need some level
> of caching at least the latest resultset.
Very true.
Yes. All these are good reasons why Midas has gained some popularity. But it
surely arises new problems, too ;^)
I'm working on getting the idea on how to run TWebConnection via https...not
much info, so far.
Again - thanks !
C4D - Kim Madsen
> Most of all, Kim - thank you for useful input on n-tier application
> development !
You're welcome. Glad it can be of use :)
> What I meant was more like that there are differences in the internals of DB
> servers that affect AppServer implementation. Example: MS SQL server has
> AutoInc fields, while IB & Oracle uses generators.
> But more important: As long as there is no BDE to use, one has to deal with
> totally different component sets for different DB's. All these component
> suites support TDataSet, but they're never the *preferred* apporoach. It may
> be that dbExpress is a good model, but so far it doesn't seem like (from
> what I've heard).
True, each native api have its quirks. One seeminly solid multidatabase API is ADO.
ADOExpress works quite well. DBExpress seems to work fine after the patch for it.
> Well, here we're getting closer: We're talking about applications that need
> access from different "channels", and we replace the direct db engine access
> with a more intellingent AppServer. We agree (I guess) that an important
> question is:
> - What protocol do I use ("protocol" as in "interface between two layers")
> ?
Its an important question! People often scream... lets use SOAP then everything can talk with everything.
But outside the drawing board, in the real world, its a much better solution to determine if both ends of a client/app
server relation
is known and under full control of the developer. If it is, then choose a binary, propritary protocol as it will perform
much much better
than using SOAP or any other standardized bloated protocol. If in some cases there would be something from the outside
not under
the developers control, then add another transport/channel opening up for a secondary 'standard' way of access.
> This is important, as it amonst other things decides what's contained in
> AppServer and whet belongs to the Client.
> Clearly seen in WebSnap concept, where one actually puts client-side stuff
> in WebServer application !
True... which is why I would not consider a websnap application to be an application server, but rather a HTML GUI
client application residing on the server side.
Web applications should always be created as a 4-tier setup: Browser - WebServer (optionally with WebSnap etc) -
Application server - Datastore.
The webserver would act as nothing more than a client towards the application server.
> Attempting to establish a fruitful way of creating an AppServer with
> "intellingence", the term "report engine" has emerged. Well, it has to be an
> editor engine too, of course, but the idea of an application reachable to
> both regular clients and web applications, to pass simple request and get a
> result set in return, properly selected, formattered & ordered...
Its one side of application servers, the potential for extracting data already somewhat preformatted (ReportBuilder via
kbmMW forexample).
> > Or so you are able to much easier to scale your application. Putting all
> kinds of stuff inside your database strains the
> > CPU and storage systems its running on.
>
> This is the *strong* side of DB-servers - proper handling of large amounts
> of data, together with secure file access & concurrency. One doesn't want to
> write an AppServer that duplicates the core of the DB.
No, that you dont want to, but you want to create an app server that allow you to scale your existing DB, either by
having it moved to another (more expensive) DB type, or by utilizing load balancing between databases on more physical
machines.
Also you may want to hide 'stupid' or redundant requests, coming from clients, from the DB. You can easily do that if
the app server have some form of build in caching. I can tell you from personal experience that it _will_ speed up most
application severely and reduce load on the DB allowing it to handle even more users on the same hardware.
> Requesting a db isn't that different from requesting an AppServer. Being
> able to divide work between multiple servers is a solution, of course, but
> then we're talking about larger applications than I've ever been involved
> with.
> - Is this the main goal of an n-tier application ?
Scalability is one of the goals of n-tier.
As are maintainability, traceability and flexibility. Many large companies move to n-tier to have the freedom to be able
to throw out their old expensive database and replace it with something newer (or at least threathen with it towards the
vendor to get license and service fees reduced).
Maintainability - Its usually much easier to debug an application server than code in a database. Further if the
database do not contain any code, the data in it is 'more protected' in terms of its easier to deduce what has happened
with the data. The effect of DB triggers combined with stored procedures can be a terrible nightmare to document and to
proof how that will work in any given case.
>
> > If you would move some of the load of the db to an app server, you in many
> situations can avoid having to invest loads
> > of money.
>
> I don't quite get this. We're more talking about replication &
> synchronisation, which IMHO is an area where I would *not* trust MIDAS. Its
> lack of transparency between AppServer and Client makes it very easy to
> screw up and leaves the client with possibly wrong data. Also the lack of
> real transaction support is serious when dealing with large, rapidly
> changing databases.
True... but although that may rule out Midas/Datasnap it doesnt rule out the n-tier idea as such. kbmMW for example
support multidatabase transaction control.
> No, as I asked: "*What* was it designed for ?"
Remote procedures and remote datasets I guess to easier get remote access via the internet.
> yes, I know and use the Memtable stuff, but I'm not sure I have the guts to
> go and check out another n-tier concept
> ...right now, at least ;-)
:) Might come as a pleasent surprise to you!
> RAD as in "Drop a component" is what I meant - what Borland is bragging
> about on the Delphi box.
I would rather say that if one thinks one can create a complete, complex, full blown, performing application by the LEGO
way not needing to carve some building blocks one self, one will get disappointed with RAD. But the idea is imo to
combine the best of the two worlds... RAD with building blocks, and more traditional coding to make the overall
experience better and the result produced faster.
> I find that this breaks about any mental model of programming. OOP it is
> not, as using standardized components that you modify manually makes
> applications hard to read. I did this exercise today, examining a number of
> TClientDataSets to try figuring out why two forms worked differnetly. I'm
> using form inheritance, and a lot of code is common to these forms. But
> designtime components are manually modified in another "source file (.dfm) !
> The first things to do is writing a few standard procedures to set values
> that normally is set in IDE (Providername, Remote Module, etc.), and
> standard routines to open whatever datasets are needed based on visible
> controls. Etc., etc...
true... RAD is just a tool which must be used in the right circumstances. Sometimes pure old coding is the better
choise.
> Exactly. When a string field is changed form 20 to 50 chars, this should of
> course not require a client side recompilation.
Agreed.
> My problem is getting the basics to work, as I have a hard time getting the
> feel for what strategy to use. Changing things within the working model is
> easy.
I remember my first time doing n-tier from scratch long time ago. It took quite some time to get my brain tuned into
what it really means and how to implement it.
Now when I have seen the light, I would not dream of going back. It does take some getting used to, and design mistakes
will be made in the first project about where code should go.. client or app server and how. Like with all programming,
one grows experience. In all cases, ask! You can go visit the newsgroups at news://news.components4developer.com . Many
questions there are about design considerations although many ofcourse are library specific.
> The validation stuff can't be buried down in the database. When youn
> ApplyUpdates on a bunch of tables in one transaction, there must be better
> solutions than waiting for the error dialog. Validation should be done
> already in the Client. This may have been solved by some
> TClientDataSet.Validate; call, where one could get error reports before
> actually applying any updates.
Its in some situations the same as HTML forms... often they are not validated until you click send in which case you get
to know about what the problem is before you can proceed.
This is an acceptable way to do it today since the communication is so quick that the result is instantanious.
You can put more validation into the client if you want to, and in some cases its even prudent to do so, but that means
that you cannot as easily change business rules without having to change the client (unless your client is as flexible
designed that business rules can be changed there on the fly provoked by the server).
The validate call would anyhow have to get in touch with the app server to validate the data.
As long it happens inside a transaction on the datastore, there are really no significant problem with it that way.
> Ideally, this should be a two-step thingy:
> 1. Client requests new ID from AppServer.
> 2. AppServer manufactures the ID, the Client is indifferent to how it
> happened.
Yep. Agreed!
> I discussed this today with the DB maintainer of the Midas project, where
> IDs are retrieved from, incremented & stored in tables via a procedure. When
> exchanging those tables with Oracle sequences, only the metatable for field
> definitions needs updating. No recompilation necessary.
I agree thats the best way seen from a portability point of view as long as the applications reaching the database are
wellknown and programmed to handle this.
> I think it would be pure madness not to utilize DB filtering, unless
> AppServer could easily hold all table data (=minor tables). In my Midas
> project, I've defined a few static datasets, that needs not be retrieved
> upon every use. Once they're loaded, they stay clientside.
Filtering in terms of 'select' statements should certainly happen on the db rather than on the app server.
But the app server may choose to put another layer of filters on top of the data of some reasons, perhaps even
transforming the data into something else.
Caching is also an important issue where a good app server architecture can assist you. I have seen tremendous
performance improvement employing intelligent caching via my products to applications running on f.ex. Oracle.
> What about rendering reports ? Could be something...not having to compile
> the entire ReportBuilder into the client !
Yep... could create a PDF file on the server and send that one to the client as a simple file transfer.
> Ha ! I got an Idea...maybe move all the MS Office (Word, Excel)
> communications to AppServer, and return result files. Could be handy on Web,
> too.
> ;-)
:) Isnt this _almost_ what Citrix is about?
> Yes. The problem then is to create an intelligent & elegant client with the
> whole spectre of possible GUI goodies...the reason why we're not
> particularily fond of HTML forms...
Its all about weighin features vs simplicity. Its getting more and more accepted today to have a really simple client. I
dont say that I actually personally like that, but thats the trend. And users are getting used to it.
> Yes. All these are good reasons why Midas has gained some popularity. But it
> surely arises new problems, too ;^)
> I'm working on getting the idea on how to run TWebConnection via https...not
> much info, so far.
You most likely need to look at OpenSSL or something like that to put SSL certificate handling on your communication
channel.
kbmMW bundles support for this by combining with StrSecII.
> Again - thanks !
You're very welcome :)