Slow query in 100000 record
Luqman Hakim
Kevin Frevert
Depends on what you consider 'slow'. I can select data against a 4
million+ record table (and joining 5+ tables) with nearly instantaneous
response (< 1 second). Is that too slow?
krf
"Luqman Hakim" <lha...@sby.ialf.edu> wrote in message
news:4339...@newsgroups.borland.com...
Jason Smith
> million+ record table (and joining 5+ tables) with nearly instantaneous
> response (< 1 second). Is that too slow?
Wow, that's impressive. What kind of machine are you using?
And how do u define 'select', does it include fetching of data from server
to client as well? or just executing the sql without fetching?
Luqman Hakim
this is my query : "Select * from Tanah"
it's simple...
this is my code:
with ADOQuery1 do
begin
Close;
SQL.Clear;
SQL.Add('Select * from Tanah');
Open;
end;
but why it's running slowly?
what is fetching?
"Jason Smith" <blue...@yahoo.com> wrote in message
news:433a...@newsgroups.borland.com...
Vitali Kalinin
"Luqman Hakim" <lha...@sby.ialf.edu> сообщил/сообщила в новостях следующее:
news:433a...@newsgroups.borland.com...
>I am using ADOQuery in Delphi 7.
> this is my query : "Select * from Tanah"
> it's simple...
> this is my code:
>
> with ADOQuery1 do
> begin
> Close;
> SQL.Clear;
> SQL.Add('Select * from Tanah');
> Open;
> end;
>
> but why it's running slowly?
> what is fetching?
RTFM. Meet yours customers and tell them that you are planning to force them
browsing through 100000 records. In this case I think yours problems will be
resolved immediately, since no one could expect that human being could work
with that much amount of data.
Kevin Frevert
It's a pretty beefy machine. 4 processors, 3 gig of ram, fiber network
(between servers, everybody else is ethernet), enough disk space for a small
country, etc etc.
Our selects only bring back the data the application/client requires. It
would be foolish to select the entire 4 million record table, then perform
some kind of local filtering. Being as specific as possible, normalized
table design, indexes, and let the server, not the client do the work (the
whole point of client/server development) allows for that kind of
performance.
krf
"Jason Smith" <blue...@yahoo.com> wrote in message
news:433a...@newsgroups.borland.com...
Kevin Frevert
The query is running slowly because you asked it to. Selecting the entire
table with no where clause will do exactly that, return 100,000 records
across that little wire (probably the biggest bottleneck), the client has to
allocate memory for all those records, and if the query is bound to a GUI
control (DBGrid), all that has to be allocated.
To get the performance you would expect, select only the fields you need and
only the records you need. For example..
(At design time, set the SQL)
Select
Field1,
Field2
From
Tanah
Where
Field1 = :Field1
(make sure the query component's parameters are set accordingly)
In code (ex, the query component sits on a Datamodule. Code is off the top
of my head, so there are probably syntax errors)
private
FErrorMsg :String;
public
property ErrorMsg :String read FErrorMsg;
function OpenTanah(const Value :String): Boolean;
end;
function TdmMyDatamodule.OpenTanah(const Value :String) :Boolean;
begin
with qTanah do
begin
try
FErrorMsg := '';
Close;
Parameters.ParamByName('Field1').Value := Value;
Open;
Result := NOT(IsEmpty);
if (IsEmpty) then
begin
FErrorMsg := 'No records found for ' + Value;
end;
except
on E:Exception do
begin
Result := False;
FErrorMsg := 'Could not open qTanah because of exception ' +
E.Message;
end;
end;
end
end;
On an OnClick (or whatever) event on a form...
begin
if dmMyDataModule.OpenTanah(Edit1.Text) then
begin
ShowMessage('Hey, this works and it is fast!');
end
else
begin
ShowMessage(dmMyDataModule.ErrorMsg);
end;
end;
Good luck,
krf
"Luqman Hakim" <lha...@sby.ialf.edu> wrote in message
news:433a...@newsgroups.borland.com...
Jason Smith
Some people argue it makes no sense to select all the 100,000 records to
client. For whatever reason, I assume you want to do it and NEED it.
Here I share some of my experience: In an ordinary desktop PC with 7200 rpm
harddisk, I can select 200,000+ records in 10 sec.
1) Don't use ADO component from Borland. It is very slow and might take
forever to fetch all 100,000 records.
2) If you are looking for solution without buying commercial components,
then I suggest you to use dbExpress. It is very fast.
3) For dbExpress, if you are using Delphi Enterprise, you can use the MSSQL
driver shipped with Delphi.
4) If you are using Delphi professional, then u can download dbExpress Odbc
driver from sourceforge. And then connect to MSSQL server using Odbc from
Microsoft. Some people argue ODBC is slow than OLEDB. But it isn't the case
with MSSQL server odbc driver provided by Microsoft.
5) You need a in-memory dataset to fetch and contain the data from dbExpress
query. Don't use ClientDataSet from Borland, it is a slow pig. Instead, I
suggest you to use KbmMemtable which is good and free.
By using the above-mentioned methods, I can select 200,000+ records in 10
secs. For you case, 100,000 records, I would assume you only need 5 secs.
Cheers,
Jason
Jason Smith
Ic. You are selecting a subset from big tables with join. With proper
indexing and 'where' condition, it is very possible to make it less than 1
second.
I thought u were to select all the 4 million records in the returned result
set.
Jason Smith
> project to DBExpress and 3tier architecture than switch CursorLocation to
> clUseServer, CursorType to ctOpenForwardOnly and LockType to ltReadOnly
> for a single TADOQuery. ;-)
Even you have the similar setting in TADOQuery or TIBQuery or
TWhatEverQuery, the speed still can't beat dbExpress TSQLQuery. Because
Unidirectional TWhateverQuery still cache something which decreases the
performance.
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433b...@newsgroups.borland.com...
> Luqman,
>
> Some people argue it makes no sense to select all the 100,000 records to
> client. For whatever reason, I assume you want to do it and NEED it.
>
> Here I share some of my experience: In an ordinary desktop PC with 7200
> rpm harddisk, I can select 200,000+ records in 10 sec.
>
> 1) Don't use ADO component from Borland. It is very slow and might take
> forever to fetch all 100,000 records.
>
> 2) If you are looking for solution without buying commercial components,
> then I suggest you to use dbExpress. It is very fast.
>
> 3) For dbExpress, if you are using Delphi Enterprise, you can use the
> MSSQL driver shipped with Delphi.
Yes this is really good tip! Sure it is much easier to port the whole
project to DBExpress and 3tier architecture than switch CursorLocation to
clUseServer, CursorType to ctOpenForwardOnly and LockType to ltReadOnly for
a single TADOQuery. ;-)
Reagrds,
Vitali
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433b...@newsgroups.borland.com...
The only performance loss would be in ADO and Delphi wrapper layer, since
both ADO and DBExpress driver are exploiting the same OLEDB driver. So with
proper settings you could get comparable performance but with much more
stability and much less coding. BTW intensive usage of unidirectional
cursors could dramatically decrease performance of the server (and hence
client applications) and lead to errors with transaction handling, since you
could have only single "fire horse" cursor when you are in transaction or
going to start transaction. So one should consider all this factors and
choose. Comparison of ADO and DBExpress groups posts and my experience make
me think that ADO is better choise for developing with MSSQL Server.
Regards,
Vitali
Jason Smith
> both ADO and DBExpress driver are exploiting the same OLEDB driver. So
> with proper settings you could get comparable performance but with much
> more stability and much less coding.
I don't agree. ADO read-only, unidirectional, server-side-cursor query is
still about 2 times slower than dbExpress OLEDB/ODBC driver from my
experience.
Same thing with Interbase, TIBQuery is never going to be faster than TIBSQL.
From my understanding, it's bcoz TIBQuery or TADOQuery or TSDACQuery takes
time to cache something even they are not supposed to cache something in
uni-directional setting.
But dbExpress Query and TIbSQL doesn't catch, thus much faster.
> BTW intensive usage of unidirectional cursors could dramatically decrease
> performance of the server (and hence client applications)
How does it decrease performance of the server? Since Client-Server RDBMS is
initially designed in uni-directional way, I think dbExpress makes more
sense than bi-directional model.
> and lead to errors with transaction handling, since you could have only
> single "fire horse" cursor when you are in transaction or going to start
> transaction.
MSSQL 2000 or older version only allows single active cursor per connection,
thus I think uni-direction cursor makes not much impact on transaction
handling part.
> So one should consider all this factors and choose. Comparison of ADO and
> DBExpress groups posts and my experience make me think that ADO is better
> choise for developing with MSSQL Server.
Would like you to enlighten me how ADO is a better choice over dbExpress in
MSSQL context
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433b...@newsgroups.borland.com...
>> both ADO and DBExpress driver are exploiting the same OLEDB driver. So
>> with proper settings you could get comparable performance but with much
>> more stability and much less coding.
>
> I don't agree. ADO read-only, unidirectional, server-side-cursor query is
> still about 2 times slower than dbExpress OLEDB/ODBC driver from my
> experience.
It could be that you are right. But it is just because there are 2
additional layers (ADO and Delphi ADO wrapper) between Application and OLEDB
provider. BTW simple call to DisableControls before loop through records
drastically increase speed. Also you still could eliminate Delphi layer by
using ADO directly. Anyway I think that stability and usability of ADO beets
DBexpress speed improvment.
> Same thing with Interbase, TIBQuery is never going to be faster than
> TIBSQL.
>
> From my understanding, it's bcoz TIBQuery or TADOQuery or TSDACQuery takes
> time to cache something even they are not supposed to cache something in
> uni-directional setting.
It looks like you are still searching for the Holly Graal of RDBMS
compatibility, but I am among those who don't think that it exists. And all
components that you were referenced for are better suited for specific RDBMS
then DBExpress does.
> But dbExpress Query and TIbSQL doesn't catch, thus much faster.
>
>> BTW intensive usage of unidirectional cursors could dramatically decrease
>> performance of the server (and hence client applications)
>
> How does it decrease performance of the server? Since Client-Server RDBMS
> is initially designed in uni-directional way, I think dbExpress makes more
> sense than bi-directional model.
You could found a lot on this in BOL, MSDN and on the WEB.
>> and lead to errors with transaction handling, since you could have only
>> single "fire horse" cursor when you are in transaction or going to start
>> transaction.
>
> MSSQL 2000 or older version only allows single active cursor per
> connection, thus I think uni-direction cursor makes not much impact on
> transaction handling part.
You could think that but reality is far away from yours understanding.
>> So one should consider all this factors and choose. Comparison of ADO and
>> DBExpress groups posts and my experience make me think that ADO is
>> better choise for developing with MSSQL Server.
>
> Would like you to enlighten me how ADO is a better choice over dbExpress
> in MSSQL context
I don't think that you are really want to be enlightened so why I should?
Jason Smith
> improvment.
From my experience, I find stability and usability of dbexpress beats ADO.
Unless you are still using the bi-directional way to code uni-directional
way of dbexpress. Even for Visual Studio .Net, they abondan the traditional
bi-directional way to go uni-directional + in-memory dataset in ADO .Net.
Have you really tried out the performance difference between ADO and
DBExpress by urself? I have.
> It looks like you are still searching for the Holly Graal of RDBMS
> compatibility, but I am among those who don't think that it exists. And
> all components that you were referenced for are better suited for specific
> RDBMS then DBExpress does.
In my previous work, we abandon the bi-directional way to go for dbExpress.
dbExpress isn't easy to use at the first place. But once you understand the
essense, u would appreciate it.
>> How does it decrease performance of the server? Since Client-Server RDBMS
>> is initially designed in uni-directional way, I think dbExpress makes
>> more sense than bi-directional model.
> You could found a lot on this in BOL, MSDN and on the WEB.
I just have heard bi-directional will decrease peformance of server and
cause transaction hanlding problem. Bi-directional model keeps the cursor
opening for long time, it's easier to cause transaction problem and more
resource-intensive for server to maintain all those cursor state. Thus
dbExpress and ADO .Net is introduced.
> You could think that but reality is far away from yours understanding.
Unidirectional model in application server + scrollable in-memory dataset is
the default and recommended behavior in 3-tier world. And even it is adopted
in ADO.Net world. Are you suggesting they are far from reality?
> I don't think that you are really want to be enlightened so why I should?
I was just hoping if you have more concrete explanation and rationale to
point out something that I could be wrong.
Cheers,
Jason
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433b...@newsgroups.borland.com...
>> improvment.
>
> From my experience, I find stability and usability of dbexpress beats ADO.
> Unless you are still using the bi-directional way to code uni-directional
> way of dbexpress. Even for Visual Studio .Net, they abondan the
> traditional bi-directional way to go uni-directional + in-memory dataset
> in ADO .Net.
>
> Have you really tried out the performance difference between ADO and
> DBExpress by urself? I have.
.NET.
>> It looks like you are still searching for the Holly Graal of RDBMS
>> compatibility, but I am among those who don't think that it exists. And
>> all components that you were referenced for are better suited for
>> specific RDBMS then DBExpress does.
>
> In my previous work, we abandon the bi-directional way to go for
> dbExpress. dbExpress isn't easy to use at the first place. But once you
> understand the essense, u would appreciate it.
>>> How does it decrease performance of the server? Since Client-Server
>>> RDBMS is initially designed in uni-directional way, I think dbExpress
>>> makes more sense than bi-directional model.
>> You could found a lot on this in BOL, MSDN and on the WEB.
> I just have heard bi-directional will decrease peformance of server and
> cause transaction hanlding problem. Bi-directional model keeps the cursor
> opening for long time, it's easier to cause transaction problem and more
> resource-intensive for server to maintain all those cursor state. Thus
> dbExpress and ADO .Net is introduced.
While in general it could be true for ADO client side cursors it is not.
Just make some tests with SQL Profiler and you will see the difference.
Since by default TADODataSet is in memory shapshot of underlying select and
once it opens it don't require server support in any way until post or batch
update. In fact it give you same functionality as combination of TSQLDataSet
+ TDataSetProvider + TClientDataSet (or any inmemory dataset which you
prefer) out of box. So IMHO this is worth of some performance penalties.
>> You could think that but reality is far away from yours understanding.
>
> Unidirectional model in application server + scrollable in-memory dataset
> is the default and recommended behavior in 3-tier world. And even it is
> adopted in ADO.Net world. Are you suggesting they are far from reality?
Nope I am saying that yours explanation on difference of cursor types if MS
SQL Server is far from reality. And also I don't think that 3-tier world is
really such wide as someone might thinking it is just popular nowdays.
ADO.Net implies that you are pumping all neccessary data in memory and
disconnect from Server for such task unidirectional
>> I don't think that you are really want to be enlightened so why I should?
>
> I was just hoping if you have more concrete explanation and rationale to
> point out something that I could be wrong.
It doesn't look for me like this is the case.
Jason Smith
> Just make some tests with SQL Profiler and you will see the difference.
> Since by default TADODataSet is in memory shapshot of underlying select
> and once it opens it don't require server support in any way until post or
> batch update. In fact it give you same functionality as combination of
> TSQLDataSet + TDataSetProvider + TClientDataSet (or any inmemory dataset
> which you prefer) out of box. So IMHO this is worth of some performance
> penalties.
I think u are comparing ADO Client side cursor Vs (TSQLDataSet +
TClientDataSet), then in this case, TSQLDataSet + TClientDataSet is not
better than ADO Client side cursor.
But if u are comparing
(ADO server side + readonly + forward only) vs TSQLDataSet alone, u will
find that TSQLDataSet performance at least twice as faster than former.
And if u are comparing
ADO Client side cursor vs (TSQLDataSet + any in-memory data set except
TClientDataSet), u will find the later performs much better.
The bottleneck is that TADOQuery will cache certain data in memory even in
uni-directional + readonly mode. So does TIBQuery.
But TSQLQuery is like TIBSQL, they cache nothing.
And for in-memory dataset, ClientDataSet is definitely much slower than
KbmMemtable or TdxMemData or something like that.
Compare TADOQuery client cursor vs (TSQLDataSet + kbmmemTable), the later
will boost your performance if you are interested to try out.
> Nope I am saying that yours explanation on difference of cursor types if
> MS SQL Server is far from reality.
Uni-directional model is the default CS RDBMS behavior. Bi-directional mode
in MSSQL is the extra feature that MSSQL implements and requires more
resource and impose performance overhead. Oracle support Bi-directional mode
only since version 9. And interbase/FB doesn't support bi-directional mode.
Which part of my explanation is far from reality? I was trying to say that
uni-directional model is the most natural way of CS RDMBS. Borland prefers
it since Delphi 6, and Microsoft prefers it since VS .Net.
> And also I don't think that 3-tier world is really such wide as someone
> might thinking it is just popular nowdays.
maybe not in small-scaled project. But in enterprise application it is.
> ADO.Net implies that you are pumping all neccessary data in memory and
> disconnect from Server for such task unidirectional
Is TADOQuery client cursor mode not doing the same?
I'm not a TClientDataSet fan. But TClientDataSet can implement incremental
stateless fetching with extra coding. So does ADO .Net.
> It doesn't look for me like this is the case.
Well, I was really curious that u are sceptical of uni-directional +
in-memory dataset, which is the current trend.
Kevin Frevert
> Luqman,
>
> Some people argue it makes no sense to select all the 100,000 records to
> client. For whatever reason, I assume you want to do it and NEED it.
>
> Here I share some of my experience: In an ordinary desktop PC with 7200
rpm
> harddisk, I can select 200,000+ records in 10 sec.
Just curious, what kind of business are you in where users need 200,000
records in 10 seconds?
krf
Jason Smith
> records in 10 seconds?
Well, I prefer load all mode because it is easier to design and can take
advantage of devExpress grid to do all those sorting, filtering, grouping,
incremental searching, data summary.
For low-end user, load all mode is very convenient. Many applications are
implementing in load all mode. E.g. Excel, Outlook, Outlook express
So I was trying to find the quickest way to load data. And if the
performance is good, why not?
Unless the data is more than 200,000 rows, which is rare in small-to-medium
size application, I would be skeptical about the SQL "where" approach, in
which I would need to sacrifice nice features in load all mode.
Jason
Vitali Kalinin
> I think u are comparing ADO Client side cursor Vs (TSQLDataSet +
> TClientDataSet), then in this case, TSQLDataSet + TClientDataSet is not
> better than ADO Client side cursor.
So you got the point. Also you are ommiting stability and usability where
ADO much better than DBExpress.
> But if u are comparing
> (ADO server side + readonly + forward only) vs TSQLDataSet alone, u will
> find that TSQLDataSet performance at least twice as faster than former.
>
> And if u are comparing
> ADO Client side cursor vs (TSQLDataSet + any in-memory data set except
> TClientDataSet), u will find the later performs much better.
>
> The bottleneck is that TADOQuery will cache certain data in memory even in
> uni-directional + readonly mode. So does TIBQuery.
In well designed 2 tier application such combination not needed at all or
needed only in certain rare situations. And if it really needed one could
use ADO directly which speed up things aprox 2 times.
> But TSQLQuery is like TIBSQL, they cache nothing.
>
> And for in-memory dataset, ClientDataSet is definitely much slower than
> KbmMemtable or TdxMemData or something like that.
>
> Compare TADOQuery client cursor vs (TSQLDataSet + kbmmemTable), the later
> will boost your performance if you are interested to try out.
I don't need that. Since I have never experienced problems with slow opening
of TADODataSet due to right design.
>> Nope I am saying that yours explanation on difference of cursor types if
>> MS SQL Server is far from reality.
>
> Uni-directional model is the default CS RDBMS behavior. Bi-directional
> mode in MSSQL is the extra feature that MSSQL implements and requires more
> resource and impose performance overhead. Oracle support Bi-directional
> mode only since version 9. And interbase/FB doesn't support bi-directional
> mode. Which part of my explanation is far from reality? I was trying to
> say that uni-directional model is the most natural way of CS RDMBS.
> Borland prefers it since Delphi 6, and Microsoft prefers it since VS .Net.
Ok lets agree on this: server side cursors are bad for performance and
additionay extensive usage of fire horse cursors MS SQL cursors could result
in huge problems with transactions.
>> And also I don't think that 3-tier world is really such wide as someone
>> might thinking it is just popular nowdays.
>
> maybe not in small-scaled project. But in enterprise application it is.
>
>> ADO.Net implies that you are pumping all neccessary data in memory and
>> disconnect from Server for such task unidirectional
>
> Is TADOQuery client cursor mode not doing the same?
Nope by default. Since it still needs live connection during posting. But in
BatchOptimistic mode it almost the same as ADO.Net. Also prefix ADO implies
that it kinda of descendent for ADO technology. Don't you think so.
> I'm not a TClientDataSet fan. But TClientDataSet can implement incremental
> stateless fetching with extra coding. So does ADO .Net.
>
>> It doesn't look for me like this is the case.
>
> Well, I was really curious that u are sceptical of uni-directional +
> in-memory dataset, which is the current trend.
I was really skeptical about DBExpress + MS SQL (and any other RDMBS)
combination at least with native driver from Borland. The idea was good
enough but implementation sucks. And I even more skeptical about advice to
port whole damn application from ADO to DBExpress since one THINK that he
need to fetch 100000 records on client and show them to end user.
Kevin Frevert
hundred records in a dbgrid (DevExpress or other wise). At least for our
business, time is money and we (the MIS department) are responsible for
maximizing that ratio. If a user is wasting time scrolling a 200,000+
record grid , we have failed (doesn't matter if it loads fast.) We force
users to enter some kind of search criteria (implement 'query-by-form'
search forms) before showing them data. I would assume most businesses work
that way. Your mileage may vary.
Again, just curious, what kind of business would consider the 'where'
approach a sacrifice?
krf
"Jason Smith" <blue...@yahoo.com> wrote in message
news:433c...@newsgroups.borland.com...
John Herbster
"Kevin Frevert" <ke...@workdrinkingdietcoke.com> wrote
> Just curious, what kind of business are you in where
> users need 200,000 records in 10 seconds?
I can easily imagine algorithms which could use 200,000 records
in 10 seconds for statistical studies, data modeling, or optimization
studies. --JohnH
Bill Todd
Jason Smith wrote:
> For low-end user, load all mode is very convenient. Many applications
> are implementing in load all mode. E.g. Excel, Outlook, Outlook
> express
Have you actually seen someone with 200,000 records in Excel, Outlook
or Outlook Express? Just curious.
My experience is that load all mode is fine for small volumes of data
but not for the volume that you are talking about.
--
Bill Todd (TeamB)
Jason Smith
>> I think u are comparing ADO Client side cursor Vs (TSQLDataSet +
>> TClientDataSet), then in this case, TSQLDataSet + TClientDataSet is not
>> better than ADO Client side cursor.
>
> So you got the point. Also you are ommiting stability and usability where
> ADO much better than DBExpress.
That's because TClientDataSet is a slower pig. But TADOQuery from Borland
isn't much faster, for the task to fetch 100,000 records. Borland's
in-memory technology like TClientDataSet or TADOQuery only work well within
few ten thousand records.
>In well designed 2 tier application such combination not needed at all or
>needed only in certain rare situations. And if it really needed one could
>use ADO directly which speed up things aprox 2 times.
I guess u might need to use OLEDB directly. But even so dbExpress + OLEDB is
still much faster than OLEDB + else access method.
> I don't need that. Since I have never experienced problems with slow
> opening > of TADODataSet due to right design.
Luqman was asking how to boost up loading 100,000 records. Your design
wouldn't be suitable for him. TADOQuery client-side cursor will take a
substantial long time to open it.
>> Is TADOQuery client cursor mode not doing the same?
> Nope by default. Since it still needs live connection during posting. But
> in BatchOptimistic mode it almost the same as ADO.Net. Also prefix ADO
> implies that it kinda of descendent for ADO technology. Don't you think
> so.
If you have tried out ADO .Net, u will realize that ADO .Net and ADO are
totally different animals, as in dbExpress is a totally animal to BDE. Same
prefix suggests a marketing tone from Microsoft.
> I was really skeptical about DBExpress + MS SQL (and any other RDMBS)
> combination at least with native driver from Borland.
Third party component will work well.
I mean third party driver + THIRD party in-memory dataset.
> The idea was good enough but implementation sucks.
So you agree the idea is good? And if the implementation by using 3rd party
components is also good? How would you judge this technology?
Anyway, u might not need it since you don't face the pressure to fetch
100,000 records yet.
> And I even more skeptical about advice to port whole damn application from
> ADO to DBExpress
depends on how you port.
My previous work ported IBX to DBExpress and we appreciate the flexibility
and performance of DBX.
And my previous research project was using ADO .Net similar model to connect
to MSSQL.
Both work very well. I wouldn't go back to using ADO components.
> since one THINK that he need to fetch 100000 records on client and show
> them to end user.
Like in ADO does, dbExpress or ADO.Net do not have to fetch 100000 records
on client and show them to end user. You still can use the SQL where
approach.
Fetching 100000 records is purely Luqman's choice. And I assume he needs it.
And I gave him advice how to boost up the speed.
Jason Smith
> business, time is money and we (the MIS department) are responsible for
> maximizing that ratio.
Well, it's your choice. But it doesn't exclude the possibility that loading
200,000+ records is handy for some small application.
> If a user is wasting time scrolling a 200,000+
> record grid , we have failed (doesn't matter if it loads fast.)
loading 200,000 records doesn't mean the user has to waste time scrolling a
200,000+ record grid. They still can use the incremental search, filtering
function from the grid without extra coding and going back to server.
And 10 seconds isn't that long. When you are using Windows Explorer, it
sometimes might take 10 seconds to wait it to show up the files.
> We force
> users to enter some kind of search criteria (implement 'query-by-form'
> search forms) before showing them data. I would assume most businesses
> work
> that way.
It depends how you define "most businesses". Excel is the application that
load all records. And Excel is one of the most application in the world. And
I find the load all idea is good when the table won't exceed 200,000
records.
> Your mileage may vary.
That's what I'm trying to say. I think both approaches have real-life value
depending on how you apply it.
Can you imagine, if Excel is using SQL where, order by approach, how would
it be? Even it can only handle up to 60,000+ records, not as much as
200,000+ records, it is still too much for those who don't like load all
approach.
> Again, just curious, what kind of business would consider the 'where'
> approach a sacrifice?
For example, arbitary sorting in the grid would be sacrifice. You can sort
table by using SQL order by. It works fast only when u pre-build an index.
But u are not likely to build the index for every field combination. Unless
you restrict your user to certain sorting combination only, otherwise the
'go-back-to-server' approach might still take some time to sort (order by)
it.
Another example, grouping in cxGrid works in load all mode only. And if you
don't load all, the grouping will force it to load all (At least up to
version 4. I haven't tried version 5 for grouping behavior yet)
Cheers,
Jason
Jason Smith
> in 10 seconds for statistical studies, data modeling, or optimization
> studies. --JohnH
Totally agree.
10 seconds for 200,000 records is when I select 10+ fields. For statistical
studies, if I need less fields, the performance might take less than 10
seconds.
And for statistical studies, if I don't need to cache the records in memory
at all, then I would prefer to use dbExpress TSQLDataSet. It's much faster
than using TADOQuery + readonly + forward only. The peformance gain is about
2x faster.
Jason Smith
> or Outlook Express? Just curious.
Not 200,000 records in excel. But I have seen 60,000+ records in excel from
a statistical research project.
Excel has the limit of 65535 (or 65536 what so ever) records. So 200,000 is
not possible with excel.
> My experience is that load all mode is fine for small volumes of data
> but not for the volume that you are talking about.
Previously when I used TSQLDataSet + TClientDataSet, since 50,000 records
already took long time. Then I would say load all is good for 50,000
records.
But then I used TSQLDataSet + KbmMemtable, or TSQLDataSet + cxGrid
DataController (unbound mode), it takes 200,000 records for only 10 seconds.
I'm trying to say that, if ur tolerance threshold is 10 seconds. Then the
load all will work just well.
And yes, 200,000 records are quite a lot, and seldom small application will
reach 200,000. I was suggesting that for small application, we just load all
and don't have to worry about extra coding to handle SQL 'where' approach,
if the data access method is fast enough and within your tolerance
threshold.
Jason Smith
practice. And suggested a way to boost up its performance.
Some people even think a grid with more than 1000 or 10,000 records is a bad
design and should be redesigned. While in my opinion, it's not necessary a
bad design, depends on what you want.
Cheers,
Jason
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433c...@newsgroups.borland.com...
And if you will do yours studies on the server side and return to the client
only necessary data performance boost will be even more evident. Also when
you are talking that load all mode is good for you it implies very low level
of concurrency and from that end a question arises why to use Client\Server
approach? For that kind of applications desktop databases fit much better.
Also could you give real life example where users are happy when presented
with 200000 records and nothing could be done to organize results in smaller
batches?
Mike Shkolnik
> with 200000 records and nothing could be done to organize results in
smaller
> batches?
in cube (a-la pivot table/DecisionCube etc)
2. any accounting with large number of customers (for example, to print the
end-of-month report for every customer of cell/mobil company)
--
With best regards, Mike Shkolnik
EMail: mshk...@scalabium.com
http://www.scalabium.com
John Herbster
"Vitali Kalinin" <vitka...@yahoo.com> wrote
> Also could you give real life example where users are
> happy when presented with 200000 records and nothing
> could be done to organize results in smaller batches?
Vatali, By your use of the term "users", are you imagining
human users viewing data via a TDBGrid? That is not always
the case with data selected and downloaded from a database.
Often the data goes into a processing program that does all
kinds of statistics, interpolations, further selections, etc.
Regards, JohnH
Kevin Frevert
asked twice and haven't gotten an answer (which is fine)
Too often the 'Select *' approach is done for the convenience of the
developer, not the user. I can provide a number of real life examples.
krf
"Mike Shkolnik" <mshkol...@ukr.net> wrote in message
news:433d09bb$1...@newsgroups.borland.com...
Vitali Kalinin
"Mike Shkolnik" <mshkol...@ukr.net> сообщил/сообщила в новостях
следующее: news:433d09bb$1...@newsgroups.borland.com...
>> Also could you give real life example where users are happy when
>> presented
>> with 200000 records and nothing could be done to organize results in
> smaller
>> batches?
> 1. any analitical software where end-user want to rollup/drilldown the
> data
> in cube (a-la pivot table/DecisionCube etc)
200000 records. People usually making decision looking at sums, trends
e.t.c. As for the pivoting it doesn't require to fetch 200000 records to the
client since most of enterprise level RDBMS like MS SQL , Oracle have
special tools for pivoting (in fact even Access does) and even if you wont
to implement it manually it could be implemented much faster and smarter
then fetching 200000 records to the client.
> 2. any accounting with large number of customers (for example, to print
> the
> end-of-month report for every customer of cell/mobil company)
This example is about reporting but not GUI app and bottleneck there not at
RDBMS client interaction but at Client output device interaction.
Vitali Kalinin
"John Herbster" <herb-sci1_AT_sbcglobal.net> ???????/???????? ? ????????
?????????: news:433d182f$1...@newsgroups.borland.com...
> Vatali, By your use of the term "users", are you imagining
> human users viewing data via a TDBGrid? That is not always
> the case with data selected and downloaded from a database.
> Often the data goes into a processing program that does all
> kinds of statistics, interpolations, further selections, etc.
> Regards, JohnH
And we are discussing efficiency correct? Would you agree that if
calculation over 200000 records could be handled on the server side without
fetching them all to client, it should be done on the server side? And the
only reason for not doing this is if calculations are really complicated for
server side programming language (even so that there is extended stored
procs for MS SQL). If last is the case than fetching data shouldn’t affect
application performance greatly and such kind of tasks doesn’t require being
interactive.
John Herbster
"Vitali Kalinin" <vitka...@yahoo.com> wrote
> And we are discussing efficiency correct?
Yes.
And I have taken the opportunity to point out that not all
database selects have a human operator viewing the
results.
> Would you agree that if calculation over 200000
> records could be handled on the server side without
> fetching them all to client, it should be done on the
> server side?
Generally speaking, yes.
> And the only reason for not doing this is if calculations
> are really complicated for server side programming ...
> and such kind of tasks doesn’t require being interactive.
Yes. You summed it up well.
Regards, JohnH
Vitali Kalinin
Mike Shkolnik
> 200000 records.
must be fetched to client and loaded to decision cube.
> People usually making decision looking at sums, trends e.t.c.
Did you hear about "what if" analysis? The sum and trends is the
presentation only. But to recalculate these sums/threads with any custom
consts, to display the data with different views - we need to fetch all
records.
> As for the pivoting it doesn't require to fetch 200000 records to the
> client since most of enterprise level RDBMS like MS SQL , Oracle have
> special tools for pivoting (in fact even Access does)
The Oracle and MS SQL have the cubes and they are for LARGE recordsets.
Nobody will use these tools for 200 000 records!
And some RDBMS (IB, for example) haven't such tools.
> This example is about reporting but not GUI app and bottleneck there not
at
> RDBMS client interaction but at Client output device interaction.
This is example when 200 000 records must be fetched from server to client.
Mike Shkolnik
I've
> asked twice and haven't gotten an answer (which is fine)
from my projects.
Vitali Kalinin
>> Yeah it should be really head egged analytic to make decision looking at
>> 200000 records.
> To change the dimensions, to drilldown or rollup, all these 200000 records
> must be fetched to client and loaded to decision cube.
>
>> People usually making decision looking at sums, trends e.t.c.
> Did you hear about "what if" analysis? The sum and trends is the
> presentation only. But to recalculate these sums/threads with any custom
> consts, to display the data with different views - we need to fetch all
> records.
Still desagree that this is rigth solution for this task. I can suggest a
lot of better ways.
>> As for the pivoting it doesn't require to fetch 200000 records to the
>> client since most of enterprise level RDBMS like MS SQL , Oracle have
>> special tools for pivoting (in fact even Access does)
> The Oracle and MS SQL have the cubes and they are for LARGE recordsets.
> Nobody will use these tools for 200 000 records!
If something works for 1000000 records it will handle 200000 records also.
It is one of the areas where size doesn't matter :-)
> And some RDBMS (IB, for example) haven't such tools.
I have stated already that pivoting could be implemented without fetching
all data using any rdbms that could group and sort results.
>> This example is about reporting but not GUI app and bottleneck there not
> at
>> RDBMS client interaction but at Client output device interaction.
> This is example when 200 000 records must be fetched from server to
> client.
Seems like you forgot my question - "Also could you give real life example
where USERS are happy when PRESENTED with 200000 records and nothing could
be done to organize results in smaller batches".
I am not trying to argue that there is no situations when fetching 200000
recs is really needed in fact there is a lot of such situations. But doing
so in order to display for end user is useless IMHO.
Mike Shkolnik
> lot of better ways.
> > The Oracle and MS SQL have the cubes and they are for LARGE recordsets.
> > Nobody will use these tools for 200 000 records!
> If something works for 1000000 records it will handle 200000 records also.
Vitali, do you know how works the Oracle Discoverer? This tool will create
the multi-cube (a lot of tables with all intersections for every dimensions
and will calculate all predefined sub-totals (sum/avg/min/max etc)
This is one way for large resultsets because there is no any other way to
speed the analysis (only use the pre-calculated sub-totals).
Also don't forget about own administration for Oracle Discoverer.
But for small resultsets (200 000 is small resultset for Oracle/MS SQL etc)
such tools are overmuch
> It is one of the areas where size doesn't matter :-)
Sorry, but from my experience you're not right here.
> > And some RDBMS (IB, for example) haven't such tools.
>
> I have stated already that pivoting could be implemented without fetching
> all data using any rdbms that could group and sort results.
Re-read my original message - end-user must rollup/drilldown the data. There
are two ways to implement this feature:
- to use the pre-calculated all subtotals for all dimensions
- to fetch all records to client and caclualte the subtotals there
The first way is for large resultsets. The second is for small.
> I am not trying to argue that there is no situations when fetching 200000
> recs is really needed in fact there is a lot of such situations. But doing
> so in order to display for end user is useless IMHO.
IMHO you're not right. There are a lot of different controls to display the
data. From your message I see that you say about standard DBGrid only (here
I'll agree with your).
But also exists the another visual controls where user need see the all
records.
Jason Smith
> client only necessary data performance boost will be even more evident.
But then I lost the ability for arbitary sorting (since you need to index it
first, and u are not likely to index for every possible combination)
But then I lost the ability to perform grouping, data summary.
If u need to include all the power of cxGrid by using SQL way, then u will
end up writing too much code, and the performance doesn't necessary improve.
My point is, why bother optimization when u can load all in high speed? I'm
not going to waste time for optimize on this when the record count is less
than 200,000 rows.
And I know u are not going to cope with fetching all 200,000 rows since you
are using a slow TADOcomponent.
> Also when you are talking that load all mode is good for you it implies
> very low level of concurrency and from that end a question arises why to
> use Client\Server approach? For that kind of applications desktop
> databases fit much better.
The advantage of CS is not only for the use of SQL where. Besides, we expect
the stability and consitency in network environment.
Don't get me wrong, I use 'SQL where' in some cases that I don't need user
interaction. And in this case I need to leverage the power of CS RDBMS. But
to present data in grid, I won't waste my energy to optimize it by using
'SQL where' since our users and I love the rich ad hoc features (arbitary
grouping, flitering, sorting, summary, incremental search) that the cxGrid
provides.
In this case I will optimize it by using better components to boost up the
fetching speed.
> Also could you give real life example where users are happy when presented
> with 200000 records and nothing could be done to organize results in
> smaller batches?
What do u mean by nothing could be done? You can sort it, group it the way u
like. You also can search it by the handy 'incremental search'. You can do a
universal powerful local filter to organize the results in a smaller batch
that 'SQL where' approach is unlikely to provide, since in 'SQL where', u
need to index again and u are not likely to index for every combination of
where condition.
Cheers,
Jason
Vitali Kalinin
Vitali Kalinin
>> Still desagree that this is rigth solution for this task. I can suggest a
>> lot of better ways.
> For example?
>> > The Oracle and MS SQL have the cubes and they are for LARGE recordsets.
>> > Nobody will use these tools for 200 000 records!
>> If something works for 1000000 records it will handle 200000 records
>> also.
> Vitali, do you know how works the Oracle Discoverer? This tool will create
> the multi-cube (a lot of tables with all intersections for every
> dimensions
> and will calculate all predefined sub-totals (sum/avg/min/max etc)
> This is one way for large resultsets because there is no any other way to
> speed the analysis (only use the pre-calculated sub-totals).
> Also don't forget about own administration for Oracle Discoverer.
> But for small resultsets (200 000 is small resultset for Oracle/MS SQL
> etc)
> such tools are overmuch
I don't see any reason to invent bicycle when I have racing car.
>> > And some RDBMS (IB, for example) haven't such tools.
>>
>> I have stated already that pivoting could be implemented without fetching
>> all data using any rdbms that could group and sort results.
> Re-read my original message - end-user must rollup/drilldown the data.
> There
> are two ways to implement this feature:
> - to use the pre-calculated all subtotals for all dimensions
> - to fetch all records to client and caclualte the subtotals there
> The first way is for large resultsets. The second is for small.
I prefer to do something when I am asked for it, i.e. when user request for
rollup or drill down I could make it on as needed basis and ir still will be
much faster.
>> I am not trying to argue that there is no situations when fetching 200000
>> recs is really needed in fact there is a lot of such situations. But
>> doing
>> so in order to display for end user is useless IMHO.
> IMHO you're not right. There are a lot of different controls to display
> the
> data. From your message I see that you say about standard DBGrid only
> (here
> I'll agree with your).
> But also exists the another visual controls where user need see the all
> records.
You are again trying to replace my question with yours understanding of it.
Where did you see in comments anything about TDBGrid or where I have stated
that it is not possible to present user with 200000 records. My point is
quite straight - it is possible (even with TDBGrid) but it is useless.
Jason Smith
>I've
> asked twice and haven't gotten an answer (which is fine)
I already gave the real life example of statical research project using
excel to load all 60,000+ records. And in Outlook express, you also load few
ten thousands of threads for newsgroup.
And for people who are skeptical of load all approach, 60,000+ is already
too much.
I was trying to say that if u can load 200,000+ records in good speed. Why
bother to use "SQL where" to optimize for records under 200,000+ records?
e.g. 60,000+
> Too often the 'Select *' approach is done for the convenience of the
> developer, not the user. I can provide a number of real life examples.
Depends on if you are using TdbGrid or TcxGrid. devExpress TcxGrid is too
powerful even for navigation on 200,000 records.
Jason Smith
> lot of better ways.
How do u mean by better? If fetching 200,000 records can provide rich UI
feature in high speed? What better ways do I need?
> Seems like you forgot my question - "Also could you give real life example
> where USERS are happy when PRESENTED with 200000 records and nothing could
> be done to organize results in smaller batches".
Already answered in previous thread
> I am not trying to argue that there is no situations when fetching 200000
> recs is really needed in fact there is a lot of such situations. But doing
> so in order to display for end user is useless IMHO.
Not really useless, because I gain the rich UI features from cxGrid by doing
high speed fetching.
Jason Smith
> 200000 records.
In this case, the aim is not to make decision by looking at 200,000 records.
But to fetch all the records locally (with high speed). And then do whatever
analysis you want without the need to fetch data from server again.
> People usually making decision looking at sums, trends e.t.c. As for the
> pivoting it doesn't require to fetch 200000 records to the client since
> most of enterprise level RDBMS like MS SQL , Oracle have special tools for
> pivoting (in fact even Access does) and even if you wont to implement it
> manually it could be implemented much faster and smarter then fetching
> 200000 records to the client.
What about those calculation like trend, forecasting, data mining in which
SQL is not adequate with?
>> 2. any accounting with large number of customers (for example, to print
>> the
>> end-of-month report for every customer of cell/mobil company)
> This example is about reporting but not GUI app and bottleneck there not
> at RDBMS client interaction but at Client output device interaction.
Sure. But then in this case. We can use TSQLDataSet without the need for
in-memory dataset, the peformance would be decreased from 10 seconds to 5
seconds to fetch 200,000 records. And the TADOQuery in any setting can only
take more than 10 seconds, say 20 seconds to fetch 200,000 records. Which
one would u prefer?
Jason Smith
What is ur point on GUI part. By using cxGrid, 'SQL where' approach is not
going to provide total rich UI features.
>> What about those calculation like trend, forecasting, data mining in
>> which SQL is not adequate with?
>
> This one is elaborated already in JhonH tree.
And I did elaborate against that point.
> So you will get yours records printed out in say 2 hrs 30 mins 5 secs and
> I will get them only in 2 hrs 30 mins 10 secs.
Not really. I will get my result printed out in 2 hrs 30 mins 5 secs by
using direct SQLDataSet. And u will get them in 4 hrs 59mins 20 secs or more
by using TADOQuery.
Vitali Kalinin
> stored procs, because lack of stored procedure standard, thus difficult to
> port.
You know man, I think that cross-database ability is bullshit.
Jason Smith
"Vitali Kalinin" <vitka...@yahoo.com> wrote in message
news:433d...@newsgroups.borland.com...
Jason Smith
> calculation over 200000 records could be handled on the server side
> without fetching them all to client, it should be done on the server side?
If you are very sure what taks user wants to do, then summing in server side
is better by doing indexing. But we provide the convenience not to assume
what user will do in the first place, e.g. we don't assume they want to sum
or doing trend analysis, we don't assume what order they need, we don't
assume what data they want to filter. And we are not likely to index for
every situation. so "sql where" might not help in this situation.
> And the only reason for not doing this is if calculations are really
> complicated for server side programming language (even so that there is
> extended stored procs for MS SQL). If last is the case than fetching data
> shouldn't affect application performance greatly and such kind of tasks
> doesn't require being interactive.
If you want cross-database ability, it's not encourgable to use extended
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433d...@newsgroups.borland.com...
> In this case, the aim is not to make decision by looking at 200,000
> records. But to fetch all the records locally (with high speed). And then
> do whatever analysis you want without the need to fetch data from server
> again.
So you are agreed with my point on GUI part.
> What about those calculation like trend, forecasting, data mining in which
> SQL is not adequate with?
This one is elaborated already in JhonH tree.
>>> 2. any accounting with large number of customers (for example, to print
>>> the
>>> end-of-month report for every customer of cell/mobil company)
>> This example is about reporting but not GUI app and bottleneck there not
>> at RDBMS client interaction but at Client output device interaction.
>
> Sure. But then in this case. We can use TSQLDataSet without the need for
> in-memory dataset, the peformance would be decreased from 10 seconds to 5
> seconds to fetch 200,000 records. And the TADOQuery in any setting can
> only take more than 10 seconds, say 20 seconds to fetch 200,000 records.
> Which one would u prefer?
Vitali Kalinin
>I already gave u guys example. Excel, Outlook, and my customer's database.
working with some kind of C/S RDBMS?
Jason Smith
The world's largest ERP system SAP is cross-database by using ODBC. Is SAP
bullshit?
Cheers,
Jason
U don't need it, doesn't mean no one else needs it.
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433d...@newsgroups.borland.com...
Extraction of data 10 - 5 secs
Print out - 2 hrs 30 mins
And what is yours?
Jason Smith
> are working with some kind of C/S RDBMS?
Nah, I'm not developer of Excel and Outlook.
But I love their UI idea. They are so successful.
devExpress is mimicing it into their components to provide greater UI
features and values for end-users. And I'm very happy with it.
And I'm not going to tell Microsoft and devExpress that their loadAll
feature is of wrong design. But I love them.
And I'm not going to tell you that u should stop ur 'sql where' approach.
We all have choices. Just do whatever you and your customers are happy.
My arguement is that "loadAll" feature is not without pratical value. In
fact, it's one of the most popular feature in the most popular softwares
(Excel, Outlook) in the world.
The problem some people don't like "loadAll" is bcoz they can't do it fast.
And also they have difficult of navigation using traditional grid.
When I can do it fast, and provide very easy navigation. Why not load all?
Vitali Kalinin
>> You know man, I think that cross-database ability is bullshit.
>
> The world's largest ERP system SAP is cross-database by using ODBC. Is SAP
> bullshit?
Jason Smith
> Print out - 2 hrs 30 mins
> And what is yours?
More exactly,
Your fomular is
(10 - 5 secs) * as many time as users do requery - many rich UI features
Print out - 4 hrs 30mins
my formula is
(10 secs ) * only 1 time (without requery since all the data are in local) +
many rich UI features.
Printer out - 2 hrs 15 mins (since TSQLQuery alone is 2 times faster than
TADOQuery in any setting)
Which one do u prefer?
Jason Smith
>> SAP bullshit?
> With ODBC back end apparently it is.
ODBC is not necessary bullshit. Depends on who writes the driver. MSSQL odbc
from MS is as fast as MSSQL OLE-DB. MaxDB (SAPDB)'s odbc is also one of the
fasters bcoz their ODBC is native.
Vitali Kalinin
>> Extraction of data 10 - 5 secs
>> Print out - 2 hrs 30 mins
>> And what is yours?
>
> More exactly,
>
> Your fomular is
> (10 - 5 secs) * as many time as users do requery - many rich UI features
> Print out - 4 hrs 30mins
Which UI features in reporting 8-)?
> my formula is
> (10 secs ) * only 1 time (without requery since all the data are in local)
> + many rich UI features.
> Printer out - 2 hrs 15 mins (since TSQLQuery alone is 2 times faster than
> TADOQuery in any setting)
I prefer correct estimations but not speculations.
Mike Shkolnik
odbc
> from MS is as fast as MSSQL OLE-DB. MaxDB (SAPDB)'s odbc is also one of
the
> fasters bcoz their ODBC is native.
speed is not one reason only. The ODBC is very limited in features (to
compare with ADO)
Of course, ODBC supported by vendors for back-compartibility (old projects)
but personally I'll not use the ODBC in new projects:-)
Jason Smith
And I will buy a bicycle instead of racing car if I can't afford a racing
car and bicycle suits my need.
Vitali, Mike's approach is called desktop OLAP. It is a very cost-effective
approach for small business and definitely has value in real life practise.
You can search 'desktop OLAP' in google for that.
In desktop OLAP, downloading 200,000 records is not unusual. Because
multi-dimensional structure is huge and easily has more than millions of
records for a medium-sized data warehouse. Downloading 200,000 records in
this way is not to display all 200,000 records. But they can do a lot of
flexible data analysis (for example, forecasting, data mining by using
neural network, decision tree, etc which SQL is inadequate with) without
going back to server.
devExpress' xtrapivotgrid is a good example of desktop OLAP. I wouldn't
regard desktop OLAP is a bad design. It's just depends on what ur need is.
A bicycle or a racing car both have their own value. I'm not going say the
existance of bicycle is useless.
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433d...@newsgroups.borland.com...
>> are working with some kind of C/S RDBMS?
>
> Nah, I'm not developer of Excel and Outlook.
>
> But I love their UI idea. They are so successful.
> devExpress is mimicing it into their components to provide greater UI
> features and values for end-users. And I'm very happy with it.
Then you should notice: default grouping by date in Outlook, worksheets in
Excel e.t.c.
> And I'm not going to tell Microsoft and devExpress that their loadAll
> feature is of wrong design. But I love them.
>
> And I'm not going to tell you that u should stop ur 'sql where' approach.
>
> We all have choices. Just do whatever you and your customers are happy.
>
> My arguement is that "loadAll" feature is not without pratical value. In
> fact, it's one of the most popular feature in the most popular softwares
> (Excel, Outlook) in the world.
>
> The problem some people don't like "loadAll" is bcoz they can't do it
> fast. And also they have difficult of navigation using traditional grid.
>
>
> When I can do it fast, and provide very easy navigation. Why not load all?
So many words but where an example from yours personal experience?
Jason Smith
> in Excel e.t.c.
That's the power of navigation as opposed to ur opinion that they can't
navigate well. But in memory, they still load all.
>> And I'm not going to tell Microsoft and devExpress that their loadAll
>> feature is of wrong design. But I love them.
>>
>> And I'm not going to tell you that u should stop ur 'sql where' approach.
>>
>> We all have choices. Just do whatever you and your customers are happy.
>>
>> My arguement is that "loadAll" feature is not without pratical value. In
>> fact, it's one of the most popular feature in the most popular softwares
>> (Excel, Outlook) in the world.
>>
>> The problem some people don't like "loadAll" is bcoz they can't do it
>> fast. And also they have difficult of navigation using traditional grid.
>>
>>
>> When I can do it fast, and provide very easy navigation. Why not load
>> all?
> So many words but where an example from yours personal experience?
The words are all my personal experience. Can't you read between the lines?
Jason Smith
> compare with ADO)
Hi Mike, I agree ODBC feature is not rich. And ADO is very complicated. Now
the trend is go back to simplicity by introducing dbExpress or ADO .Net off
line model: uni-directional + scrollable in-memory dataset + resolver, this
architecture is more complete and more efficient and more flexible as a
whole.
But I don't exclude that you might need the specific feature of ADO.
John Herbster
"Vitali Kalinin" <vitka...@yahoo.com> wrote
> ... it should be really head egged analytic to make decision looking at
> 200000 records. People usually making decision
> looking at sums, trends, etc. ...
Making visual displays of masses of data as points in 2D or
3D can show things about data that are not readily noticable
just from statistics. --JohnH
Jason Smith
And my personal experience might not be sufficient to convince you. The
design of excel and outlook from Microsoft represents personal experience of
many people in the world by Microsoft UI research
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433d...@newsgroups.borland.com...
>> I don't see any reason to invent bicycle when I have racing car.
>
> And I will buy a bicycle instead of racing car if I can't afford a racing
> car and bicycle suits my need.
>
> Vitali, Mike's approach is called desktop OLAP. It is a very
> cost-effective approach for small business and definitely has value in
> real life practise. You can search 'desktop OLAP' in google for that.
>
> In desktop OLAP, downloading 200,000 records is not unusual. Because
> multi-dimensional structure is huge and easily has more than millions of
> records for a medium-sized data warehouse. Downloading 200,000 records in
> this way is not to display all 200,000 records. But they can do a lot of
> flexible data analysis (for example, forecasting, data mining by using
> neural network, decision tree, etc which SQL is inadequate with) without
> going back to server.
>
> devExpress' xtrapivotgrid is a good example of desktop OLAP. I wouldn't
> regard desktop OLAP is a bad design. It's just depends on what ur need is.
LOL
> A bicycle or a racing car both have their own value. I'm not going say the
> existance of bicycle is useless.
And again yours answer is full of speculations - you already have racing car
MS SQL server, Oracle e.t.c. so why to buy bicycle. And again you and Mike
not suggesting to buy it, you are suggesting to invent and than manufacter
it from a scratch.
Jason Smith
>>> Extraction of data 10 - 5 secs
>>> Print out - 2 hrs 30 mins
>>> And what is yours?
What does all that mean? I thought you were talking about both grid and
reporting.
In reporting of course I will use 'SQL where', if the data is not already
cached by my previous 'loadall'. And if my data already are cached by the
previous "load all", I will take the data from previous cache and print out
Vitali Kalinin
1)you can't give us example
2)I don't see any point to user to buy yours product since they have Excel
and Outlook.
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433d...@newsgroups.borland.com...
>> in Excel e.t.c.
>
> That's the power of navigation as opposed to ur opinion that they can't
> navigate well. But in memory, they still load all.
I know for sure that you are wrong for Outlook it uses some kind of adhoc
desktop database and no one use Excel to store large amout of data.
>>> And I'm not going to tell Microsoft and devExpress that their loadAll
>>> feature is of wrong design. But I love them.
>>>
>>> And I'm not going to tell you that u should stop ur 'sql where'
>>> approach.
>>>
>>> We all have choices. Just do whatever you and your customers are happy.
>>>
>>> My arguement is that "loadAll" feature is not without pratical value. In
>>> fact, it's one of the most popular feature in the most popular softwares
>>> (Excel, Outlook) in the world.
>>>
>>> The problem some people don't like "loadAll" is bcoz they can't do it
>>> fast. And also they have difficult of navigation using traditional grid.
>>> When I can do it fast, and provide very easy navigation. Why not load
>>> all?
I have nothing against DevExpress grids but their power should be properly
combined with right design. Which mean that you could for exmple apply some
grouping by default, upload group headers in a single shoot and then upload
group content while requested from user. And if someone doesn't know how to
implement this it is up to him.
>> So many words but where an example from yours personal experience?
>
> The words are all my personal experience. Can't you read between the
> lines?
Nope I can't
Jason Smith
> car MS SQL server, Oracle e.t.c. so why to buy bicycle. And again you and
> Mike not suggesting to buy it, you are suggesting to invent and than
> manufacter it from a scratch.
Invent from a scratch? I don't think so. Instead, I would buy devExpress
xtraPivotGrid (bicycle) for small business. Because customers in small
business wouldn't want or can't afford to pay for MSSQL, Oracle.
Jason Smith
To end up:
1) I already gave u examples. But u can't get it. That's up to you.
2) Our product is selling very well. And devExpress cxGrid with LoadAll mode
is also doing very well.
Jason Smith
> desktop database
and still load all. And synchrinize with outlook exchange server for
multi-user mode as well.
> and no one use Excel to store large amout of data.
I saw a statical research project using Excel to store 60,000+ records of
data. According to you guys, it's already a bad design. But people who did
that are happy with that.
> I have nothing against DevExpress grids but their power should be properly
> combined with right design. Which mean that you could for exmple apply
> some grouping by default, upload group headers in a single shoot and then
> upload group content while requested from user.
Believe me, it's going to be very complicated and not worth while if the
record count is less than 200,000 records. And u need to do it for many sql
request in order to get the same thing done. Which is not economical to do.
And for abitary sorting, filtering, grouping, summary, you are not likely to
index ur table for every combination. Thus 'SQL where' wouldn't be faster.
'SQL where' work well only when you allow end user certain combination of
sorting, filtering capabilities.
And 'SQL where' doesn't solve all problem. for example, if you have a table
with 200,000 records. if the distribution is not even, say you have 190,000
products in category A, and 10,000 product in category B. 'SQL where product
category = A' will still return you 190,000 records. And are you going to
count the record before hand and tell the end-user that their criteria is
too broad. Or simply return maximum N records?
> And if someone doesn't know how to implement this it is up to him.
>
>>> So many words but where an example from yours personal experience?
>>
>> The words are all my personal experience. Can't you read between the
>> lines?
> Nope I can't
If someone can't read between the lines, it's also up to him :-)
Mike Shkolnik
car
> MS SQL server, Oracle e.t.c. so why to buy bicycle.
Oracle Developer license:-)
> And again you and Mike not suggesting to buy it, you are suggesting
> to invent and than manufacter it from a scratch.
I suggest to use not only one tool/engine but use the best for task
Brian Hollister
PTI, i just wanted to point out in my Call Log application we use in-house i
implement the SQL-Where approach along with the cxGrid. I do not use the
Incremental Search in every grid displayed to the user though. My users and
I still feel that we get enough mileage out of it. I use the Where to limit
the resultset by Que Levels (there are 3). Everyone needs to see call
tickets in Levels 1 and 2 but only developers need to see those in Level 3.
I must say the application requires a second trip to collect data if a
developer wants to see calls in the Level 3 que. But since we have a small
team that works just fine for us. I tried the load every record thing and it
just took too long. So by limiting the records returned and still using the
features of the cxGrid we are very happy. I guess i just wanted to point out
another situation where it seems as if the shoe size is slightly different.
Have a wonderful day,
Brian
PS. - Big shout out to DevExpress for such an amazing component
--
Got a big event coming up? Let us
help coordinate your event. For more
visit www.kissemgoodbye.com
Jason Smith
> just took too long. So by limiting the records returned and still using
> the
> features of the cxGrid we are very happy. I guess i just wanted to point
> out
> another situation where it seems as if the shoe size is slightly
> different.
So you are in context of master-detail-detail? It makes to use SQL-where
here.
But if you want to try to boost up the performance, it's possible.
If you are using MSSQL, the fastest way is to use TSQLDataset with MSSQL
driver (either OLEDB or ODBC). And pump the data directly into
cxGrid.DataController in unbound mode. In this case cxGrid.DataController
acts as in-memory dataset.
Jason Smith
> so if someone bought it, he/she should exploit it power as much as
> possible.
And for records < 200,000 records, desktop OLAP load all approach is very
economical.
Jason Smith
Yes, I know that and I did mention that in previous post. I was trying to
say that if speed of 200,000 records is not a problem. Then the speed of
60,000+ is even not a problem. That's why excel has no problem for 60,000+
record.
> 2)So you telling me that people were seraching something in 60,000+ using
> GUI
They can use a lot of navigation or searching method provided by Excel
instead of scrolling around even they are in load all mode.
> or they were using them as raw data for calculations and only results of
> calculations were significant for them?
It's just simple, to be able to calculate the result, u need to set up the
formula and often beside the cell of raw data and then copy it into
everywhere where u need to result to be displayed.
Vitali Kalinin
"Mike Shkolnik" <mshkol...@ukr.net> сообщил/сообщила в новостях
следующее: news:433d4f5d$1...@newsgroups.borland.com...
>> 1)Excel have limit on number of record
64K to be precise :-)
Vitali Kalinin
"Jason Smith" <blue...@yahoo.com> сообщил/сообщила в новостях следующее:
news:433d...@newsgroups.borland.com...
>> desktop database
>
> and still load all. And synchrinize with outlook exchange server for
> multi-user mode as well.
>
>> and no one use Excel to store large amout of data.
>
> I saw a statical research project using Excel to store 60,000+ records of
> data. According to you guys, it's already a bad design. But people who did
> that are happy with that.
Vitali Kalinin
"Mike Shkolnik" <mshkol...@ukr.net> сообщил/сообщила в новостях
следующее: news:433d...@newsgroups.borland.com...
>> And again yours answer is full of speculations - you already have racing
> car
>> MS SQL server, Oracle e.t.c. so why to buy bicycle.
> OLAP toolas are not part of database server. Just look how much costs the
> Oracle Developer license:-)
I know nothing about Oracle licensing. But AFAIK Oracle is very expensive so
Jason Smith
context"
By the way, what data access component and RDBMS are u using?
Mike Shkolnik
--
Wayne Niddery [TeamB]
> 2. any accounting with large number of customers (for example, to
> print the end-of-month report for every customer of cell/mobil
> company)
Copying all records to a client machine for such a report is going to be
much slower than running a report process directly on the server, there is
no need to display these records in a GUI and thus no reason for anything
but a forward-only, non-cached query.
--
Wayne Niddery - Logic Fundamentals, Inc. (www.logicfundamentals.com)
RADBooks: http://www.logicfundamentals.com/RADBooks.html
"The moment the idea is admitted into society that property is not as
sacred as the laws of God and there is not a force of law and public
justice to protect it, anarchy and tyranny commence." - John Adams
Brian Hollister
I use TADODataset and MS SQL 2000 currently.
Just for a little more info, i did test prior to selecting ADO with both the
ADO and dbExpress componentry. I did find that dbExpress was faster.
However, the only difference in it seemed to be the caching of the data on
the client side that made TADODataset so much slower. Even though the
ADODataset was slower i ad been watching newsgroups and researching others
experiences with dbExpress and i was really concered with the instability
and difficulty people were having with it at the time. Being a bit hesistant
i decided to go with the TADODataset as it appeared to be more stable and
trust-worthy. Mind you this is what i saw from the outside looking in, i
have only been a professional developer for a little over four years.
<cut>
If you are using MSSQL, the fastest way is to use TSQLDataset with MSSQL
driver (either OLEDB or ODBC). And pump the data directly into
cxGrid.DataController in unbound mode. In this case cxGrid.DataController
acts as in-memory dataset.
</cut>
Our software package here that is sold is built on an old proprietary db
system and i currently use the above mentioned technique in a data viewer
utility that i built recently. Once you get the data in the grid it is nice
not to have to go back to see another record.
I would like to ask you how in this situation do you allow for updates to
data? If you have 200,000 records how do you know which one has been altered
out of all of them or am i off-base here and the technique your describing
is only used for viewing?
Vitali Kalinin
> It's just simple, to be able to calculate the result, u need to set up the
> formula and often beside the cell of raw data and then copy it into
> everywhere where u need to result to be displayed.
but they just need results of calculations and they were happy about them.
Mike Shkolnik
> much slower than running a report process directly on the server
PL/SQL? :-)
Anyway will be created the client project for report generation. Of course,
this exe will be ran on computer that is "near" to server. And maybe there
will be not any visual presentation. But anyway all records must be fetched
from server (db-server) to client app, processed in some way and printed.
> there is no need to display these records in a GUI and thus no reason for
anything
> but a forward-only, non-cached query.
Depends from report type (exists a lot of two-pass reports)
Mike Shkolnik
so
Oracle have a lot of different licence types for database and some from them
are cheap.
But _any_ OLAP tool (no matter who is vendor) are very expensive.
> if someone bought it, he/she should exploit it power as much as possible.
:-) How this is related to original topic?
Every task have a few solutions and what way to select will decide the
project manager.
PS: Vitali, re-read this topic again - you asked the samples and you got a
lot. The 200 000 is not so large dataset and exists a lot of tasks when all
records must be fetched/processed in client app.
Jason Smith
Not limited to. And you can't predict what they want next. Thus load all +
rich set of features are handy to provide a full-set browsing solution
(arbitary sorting, filtering, searching etc)
Vitali Kalinin
> Not limited to. And you can't predict what they want next. Thus load all +
> rich set of features are handy to provide a full-set browsing solution
> (arbitary sorting, filtering, searching etc)
app for them. They could happily use general tool like Excel or even do
whatever they want running queries in QA or similar tool.
Jason Smith
> the
> ADO and dbExpress componentry. I did find that dbExpress was faster.
> However, the only difference in it seemed to be the caching of the data on
> the client side that made TADODataset so much slower.
TADODataSet is always slower than dbExpress, Even you set the TADODataSet to
be uni-directional (forward only) + readonly + server cursor, which is of
equal condition of dbExpress. You can do the experiment urself.
I think you are right, the reason is TADODataSet cache certain data even
they are not supposed to cache in uni-directional mode.
> Even though the
> ADODataset was slower i ad been watching newsgroups and researching others
> experiences with dbExpress and i was really concered with the instability
> and difficulty people were having with it at the time. Being a bit
> hesistant
> i decided to go with the TADODataset as it appeared to be more stable and
> trust-worthy. Mind you this is what i saw from the outside looking in, i
> have only been a professional developer for a little over four years.
I think dbExpress itself is quite stable. But if you want a good driver, u
can use cr-lab one. If you don't mind ODBC, you can use the openodbc for
free from sourceforge. ODBC in MSSQL makes no difference in speed as
compared to OLEDB. Except that, in dbxODBC, One open connection can allow
only one active statement (as original design of MSSQL 2000 and MySQL, u
need to manually clone connection for more than 1 active statement). and
OLEDB will automatically clone connections for you if u have more than one
active statement.
> <cut>
> If you are using MSSQL, the fastest way is to use TSQLDataset with MSSQL
> driver (either OLEDB or ODBC). And pump the data directly into
> cxGrid.DataController in unbound mode. In this case cxGrid.DataController
> acts as in-memory dataset.
> </cut>
>
>
> Our software package here that is sold is built on an old proprietary db
> system and i currently use the above mentioned technique in a data viewer
> utility that i built recently. Once you get the data in the grid it is
> nice
> not to have to go back to see another record.
Do you mean you prefer not to go back to server to fetch other records?
> I would like to ask you how in this situation do you allow for updates to
> data?
Usually, we don't allow data updation in grid. But if you want, then you
will have to handle it urself,
if you are using TSQLDataSet -> DataContoller method (unbound mode), which
is the most efficient and most flexible way but most complicated to handle.
If you are using TSQLDataSet -> KbmMemtable -> DataController, then u have
to handle the kbmmemtable delta.
If you are using TADODataSet/TClientDataSet -> DataController, updation are
handled automatically. But TADODataSet and TClientDataSet have severe
performance issue as compared to many 3rd party components, especially if
you have 200,000 records, it might take forever to perform certain task.
> If you have 200,000 records how do you know which one has been altered
> out of all of them or am i off-base here and the technique your describing
> is only used for viewing?
Do you mean the refreshing part? If you want to handle it efficiently, it's
going complicated again. There are few techniques.
One example is to use counter approach. You can you can implement the
counter by urself.
Record A: counter = 1
Record B: counter = 2
...
Record Z: counter = 26
After u fetch all records into Grid, u record the max counter is 26.
When user modified Record B, update the counter, so that
Record A: counter = 1
Record B: counter = 27
...
Record Z: counter = 26
And then when you want to refresh, you don't have to refetch every 200,000
records again, instead
select * from table where counter > 26
And update your grid data accordingly.
=============================
Well, if you really want your application to have top peformance, a lot of
optimization techniques are required and it can go very complicated and
require a lot of understanding or study. It depends on whether you are going
to take the challenge.
Jason Smith
Not really. We can't predict, thus we need to provide powerful,
full-featured solution, which cxGrid does.
'SQL where' approach is to RESTRICT them, since you only provide 'where',
'sort' condition for certain combination by the fact that you are not going
to have index for every condition.
Jason Smith
most problem raises from TClientDataSet part.
Brian Hollister
In the situation that getting the data means hitting our proprietary system,
I do NOT want to hit the server again. I have no way to fetch one record
easily. Some datasets offer a restriction file but many do not, so generally
speaking its a 1 to n loop until you find the record in question. Because of
this we are porting the application to MS SQL. This utility's purpose is to
bring back all records. So no I do not want to go back again. Its basically
just used for analytical purposes for developers. However, in the Call Log
application I mentioned earlier I do not mind because I am limiting the
recordset.
I have heard that other drivers for use with dbExpress have much better
performance and reliability. Our small shop prefers to attempt to work with
what we have first then move forward from there. I got them to buy the DevEx
stuff and that was hard enough.
--
Got a big event coming up? Let us
help coordinate your event. For more
visit www.kissemgoodbye.com
"Jason Smith" <blue...@yahoo.com> wrote in message
news:433d...@newsgroups.borland.com...
Jason Smith
>
> In the situation that getting the data means hitting our proprietary
> system,
> I do NOT want to hit the server again. I have no way to fetch one record
> easily. Some datasets offer a restriction file but many do not, so
> generally
> speaking its a 1 to n loop until you find the record in question.
Do you mind to disclose, from your end-user experience, how big (record
count) does the level 1, 2, 3 reach?
> Because of
> this we are porting the application to MS SQL. This utility's purpose is
> to
> bring back all records. So no I do not want to go back again. Its
> basically
> just used for analytical purposes for developers. However, in the Call Log
> application I mentioned earlier I do not mind because I am limiting the
> recordset.
If you are viewing a call log, do u need updation in the grid? If it's
read-only, then life is much easier.
> I have heard that other drivers for use with dbExpress have much better
> performance and reliability.
Generally yes. Not many components originally from Borland are very good.
That leaves the room for 3rd party market.
Even for ADO, people are saying TBetterADO or OLEDB Direct etc are better.
> Our small shop prefers to attempt to work with
> what we have first then move forward from there. I got them to buy the
> DevEx
> stuff and that was hard enough.
:-) Can understanding. For whatever reason, usually people are relunctant
for a change. While Borland is promoting dbExpress and MS is promoting ADO
.Net (both are unidirectional dataset + scrollable in-memory dataset,
offline model), many people are still using the older model (TIBQuery,
TADOQuery). I'm not going to be judgemental what people are choosing to use.
But for speed gain, dbExpress is currently one of the fastest ways to go.
And TClientDataSet is one of the slowest in-memory datasets available.
Brian Hollister
> count) does the level 1, 2, 3 reach?
>
Total row count in the table is roughly 70K. I know pretty small in general.
I will also admit that time needs to be devoted into a better indexing of
the database and such. Another thing that bothers me is that this table is
full of lookup fields, you know when you display them you have to open other
tables. I truly have learned so much more than what i knew as i was creating
the application over a year ago. Being the small shop we are there are many
projects more important.
This raises another questo
> If you are viewing a call log, do u need updation in the grid? If it's
> read-only, then life is much easier.
My grids are only used to locate records of interest. The user then
dbl-clicks to open another form that is used to edit the record and its
detail tables. One thing i like about the grid is that even though my query
brings almost all fields from the table the user can customize what is
displayed and that can also be stored in the database. I really like that
one.
> Even for ADO, people are saying TBetterADO or OLEDB Direct etc are better.
Is TBetterAdoDataset really that much better? If so why?
Thanks for the chat back and forth,
brian
Jason Smith
> general.
True. 70K isn't that much. In an average PC with 7200 Rpm harddisk, I think
u can boost it up and fetch it in 4-5 seconds.
> I will also admit that time needs to be devoted into a better indexing of
> the database and such. Another thing that bothers me is that this table is
> full of lookup fields, you know when you display them you have to open
> other
> tables.
Lookup is quite inefficient. And it could be the critical bottleneck when
you load all. for your level 1, 2.
for lookup field, it will perform TLookupTable.Locate for each field of each
record.
Say you have 70K rows, and you will 5 lookup fields in total. Internally,
they will perform .Locate for 70K * 5 = 350,000 times of lookup.
And in TADODataSet, the lookup doesn't make use of index, that's the
problem.
I suggest you to use SQL join with proper indexing to substitue lookup
method. SQL join again requires more work but it's worthwhile.
> My grids are only used to locate records of interest. The user then
> dbl-clicks to open another form that is used to edit the record and its
> detail tables.
That's fine and makes sense. And easier to handle too, as compared to direct
updation on grid.
> One thing i like about the grid is that even though my query
> brings almost all fields from the table the user can customize what is
> displayed and that can also be stored in the database. I really like that
> one.
I like that one too. Some people argue that in CS RDBMS model, we shouldn't
select all fields but only fields of interest. But I think in 70K rows, the
field count doesn't make a very significant difference.
>> Even for ADO, people are saying TBetterADO or OLEDB Direct etc are
>> better.
>
> Is TBetterAdoDataset really that much better? If so why?
Sorry about this part. I didn't have much experience on TBetterADO etc. Just
"people are saying" :-)
===============================================================
Another issues when u are using TADODataSet with cxGrid.
I don't know what kind of cursor setting (client side or server side) you
are using. If you are using client side, then your memory usage is doubled.
In this case, TADODataSet will cache one copy of data from server. And then
cxGrid.DataController will cache another copy of data from TADODataSet.
Unless your user has big RAM, otherwise the data will reside on Virtual
Memory instead of RAM. And then it decreases the speed since it needs to
spend more time to write the extra copy in to virtual memory (harddisk).
The solution is to use unidirectional server-side cursor. And pump data
directly into cxGrid.DataController in unbound mode. After pumping data,
close the dataset (to make it stateless by detaching the table connection
from db server). In this case, the memory usage is kept minimum (1 copy
only) and thus improved the speed.
Jason Smith
using Query Analyzer provided by MSSQL to measure. It tells you the time it
uses to fetch record rows -- And that's the fastest you can get.
Due to the caching machanism in cxGrid.DataController, we can't make it as
fast as u can get from Query Analyzer. But when we use dbExpress ->
dataController. The speed is not too far from what query analyzer can
provide. If you are using TADODataSet, you will know how slow it is! as
compared to the result from Query Analyzer.
Kevin Frevert
> >I've
> > asked twice and haven't gotten an answer (which is fine)
>
> I already gave the real life example of statical research project using
> excel to load all 60,000+ records. And in Outlook express, you also load
few
> ten thousands of threads for newsgroup.
>
No offense, but you didn't write Outlook and Excel and the continuous
mentioning of those products gives you zero credibility. No one is arguing
the necessity of statical analysis apps, reporting processes that handle
large quantity of data, etc, so stop repeating it. All we are asking is
*your* experience. Not how users use Excel, repeating what Mike S. posted,
*your* experience (sorry, throwing 200K in a grid isn't statical analysis).
You mentioned in another thread your product is for sale. Do you have a
link? Screenshots? Describe the application you've written, not how users
use Outlook or Excel for data analysis.
If you don't want to or can't (your software is used by the CIA), that's
fine (I was just curious), but simply be honest.
krf
Kevin Frevert
>
> 1) I already gave u examples. But u can't get it. That's up to you.
>
You keep repeating Outlook and Excel. Those are examples of other programs
users use, not *your* application and how *your* users use it.
> 2) Our product is selling very well. And devExpress cxGrid with LoadAll
mode
> is also doing very well.
Link? Screenshots?