Loading items in TTreeView becomes 10 times slower in Delphi 5
Denis Dresse
Passing from Delphi 4 to Delphi 5, my TtreeView becomes 10 times
slower in loading items. It seems to be the
TTreeView.items.addChildObject method.
(This TTreeview has a lot of items, more than 5.000)
What can be done ?
Thanks for help.
Denis
Craig Leidy
Thanks, Craig.
Robert Kuhlmann
Treeviews are my favorite, and my hobby is to make them as fast as
possible.
So here some useful hints:
TIP 1: Create the tree in order
I found out that there's a big difference, if a tree is filled with the
items in order or not. With 'in order' I mean, that if you would fully
expand the tree, the items are a list of entries, attached to different
levels of the tree.
If you fill in these items in exactly that order, it is done within a
finger snip (tested with up to 120.000 entries).
Therefore it is necessary to build an internal structure (I use a
TCollection-descendant for that) in memory, before passing the entries
to the tree.
To fill the tree the internal structure has to be operated via a
recursive-operation (see sample code, later in this posting).
TIP2: Create Items without captions
Because I have an internal structure already (maybe with captions
already loaded into it), I create the tree-items without the caption:
NewNode := AddChild(Parent,'');
NewNode.Data := AktItem;
To see captions when a node shows up, I implement the
OnCustomDrawItem-event, loading the caption to show-up from the internal
structure (runs in realtime, even if a 65.000-items tree is fully
expanded - TTreeview is not able to show ore than 65.000 items expanded
at the same time, but can hold a lot more than that).
I've realized that and it's even functional when load the captions via
lan at runtime (TIBSQL of course -> no overhead).
TIP 3: Hide Treeview while filling or destroying it
For mass-operations allways set the Visible-property of the treeview to
false and call TreeView.Items.BeginUpdate.
You may wish to place a dummy-panel behind the TreeView so your customer
will even not notice the flicker (BevelOuter=bvLowered, BevelWidth=2,
Color=clWhite).
Don't forget to make it visible again and call TreeView.Items.EndUpdate
after you finished the build.
TIP 4:No recursions, except in memory-access
Don't load items recursively from wherever. Load them as they come in
and rebuild the treestructure internally.
TIP 5:Node-captions should be thrown from within a thread
If you follow TIP 2 and create the node-items without text, the
implementation of OnCustomDrawItem should send commissions to a thread,
that handles the work to set the node's Text-properties.
TIP 6:Use a timer if you locate Datasets via a treeview
I use a little trick, that avoids dataset-access while scrolling through
the treeview:
<TIP 6:Sample Code Start>
...
FTimer := TTimer.Create(Application);
FTimer.Interval := 250;
FTimer.Enabled := false;
FTimer.OnTimer := OnTimerProc;
...
procedure TStructureFactory.OnTimerProc(Sender: TObject);
begin
FTimer.Enabled := false;
if assigned(FTreeView) then
if assigned(FTreeView.Selected) then
if assigned(FDataDataset) then
FDataDataset.Locate(FKeyFieldname,TStructureElement(FTreeView.Selected.Data).ID,[]);
end;
procedure TStructureFactory.OnTreeViewChange(Sender: TObject;
Node: TTreeNode);
begin
if assigned(FTimer) then
begin
FTimer.Enabled := false;
FTimer.Enabled := true;
end;
end;
<TIP 6:Sample Code End>
Okay, so far so good. The hole source wil be too much to post in here.
Just a little sample of the procedure that builds the tree:
<Sample Code Start>
//needed to make the parameter TCustomTreeView possible
//and still have access the protected methods.
TTreeViewHacker = class(TTreeView)
end;
procedure TStructureFactory.ShowItemsTo(aTreeView: TCustomTreeView);
var
aktMain : TStructureElement;
i : integer;
procedure ShowSubStructure(const MainNode : TStructureElement);
var
aktSub : TStructureElement;
begin
aktSub := MainNode.FirstChild;
repeat
inc(i);
aktSub.Node :=
TTreeViewHacker(aTreeView).Items.AddChild(MainNode.Node,'');
aktSub.Node.Data := aktSub;
if aktSub.HasChildren then
ShowSubStructure(aktSub);
aktSub := aktSub.Next;
until not assigned(aktSub);
end;
begin
FTreeView := aTreeView;
TTreeViewHacker(aTreeView).OnChange := OnTreeViewChange; //needed to
set a Dataset into position
TTreeViewHacker(aTreeView).OnCustomDrawItem := TreeViewCustomDrawItem;
//show captions on demand
aTreeView.visible := false;
TTreeViewHacker(aTreeView).Items.BeginUpdate;
aktMain := FRootItem;
i := 0;
repeat
inc(i);
aktMain.Node := TTreeViewHacker(aTreeView).Items.Add(nil,'');
aktMain.Node.Data := aktMain;
if aktMain.HasChildren then
ShowSubStructure(aktMain);
aktMain := aktMain.Next;
until not assigned(aktMain);
CapThread.Resume;
TTreeViewHacker(aTreeView).Items.EndUpdate;
aTreeView.visible := true;
end;
<Sample Code End>
The actual speeds I can reach for a 15.800 items structure (app. 8-10
Levels) are
Pentium 333, Interbase, 100MBit LAN: 4 seconds
Pentium 333, Paradox, local: 2 seconds
Athlon 650, Paradox,local: < 1 second
I hope I could give a few good tips on that.
BTW: To load the same structure (with captions) via
TreeView.LoadFromStream is (much) slower.
Bye
Robert
Mike Orriss (TeamB)
> Passing from Delphi 4 to Delphi 5, my TtreeView becomes 10 times
> slower in loading items.
>
You need to wrap your code in BeginUpdate/EndUpdate - its *much* more
important for D5.
Mike Orriss (TeamB)
(Unless stated otherwise, my replies relate to Delphi 4.03/5.00)
(Unsolicited e-mail replies will most likely be ignored)
Mike Orriss (TeamB)
> The actual speeds I can reach for a 15.800 items structure (app. 8-10
> Levels) are
> Pentium 333, Interbase, 100MBit LAN: 4 seconds
> Pentium 333, Paradox, local: 2 seconds
> Athlon 650, Paradox,local: < 1 second
>
Why bother?
If you are holding all the info to populate the tree, there is no need to
add nodes until they are displayed. By creating/destroying nodes during
OnExpand/OnCollapse you get a very speedy tree indeed.
As it happens, I have just built a tree today controlled by a DBISAM table
with 8238 records at seven levels. Using SetRange as appropriate, loading
appears instantaneous.
Paul Ferrara
Begin/EndUpdate it was taking roughly 2 minutes. After adding those, it now
takes about 5 secs. And that includes opening about 100 tables and reading
the fielddefs.
Paul / ColumbuSoft
www.columbusoft.com
Mike Orriss (TeamB) <m...@3kcc.co.uk> wrote in message
news:VA.000017b5.103e3ddd@mikemain...
> In article <38CD3978...@decis.be>, Denis Dresse wrote:
> > Passing from Delphi 4 to Delphi 5, my TtreeView becomes 10 times
> > slower in loading items.
> >
>
> You need to wrap your code in BeginUpdate/EndUpdate - its *much* more
> important for D5.
>
>
Denis Dresse
seconds...
I still have to optimize some operations...as avoiding recursive
searching when building the tree in order to get under 3 seconds.
Thanks,
Denis
Pavel A
........
> If you follow TIP 2 and create the node-items without text, the
> implementation of OnCustomDrawItem should send commissions to a thread,
> that handles the work to set the node's Text-properties.
Great info, Robert!
but only the main thread can call VCL? How do you delegate this to
another thread?
--PA
Robert Kuhlmann
your not right. Any TThread decendant can "call" the VCL, but you have
to comply with some rules to do that (see Delphi-help).
I'll give you the code of the thread. There are some examples how to
access visual and non-visiual VCL-components from within a thread. Take
a special look at the threaded queries. They don't block the application
while looking for the caption-entries.
It's not fully documented because it is only a study to check the
techniques:
(I've added some comments though)
<Sample code start>
unit UnitCapThread;
interface
uses
Classes, DB, DBTables, IBDatabase, IBSQL, TreeStructure;
type
TCapThread = class(TThread)
private
FJobs : TList;
FCapDataset : TTable;
FCapQuery : TIBSQL;
FCapStrLen : integer;
FOrigDatabase : TIBDatabase;
FDatabase : TIBDatabase;
FTransaction : TIBTransaction;
FDatabasename : string;
FTablename : string;
FKeyFieldname : string;
FCapFieldname : string;
FCapFieldIndex : integer;
FCapQueryInit : procedure of object;
FGetNodeCap : procedure of object;
FGetStrucStr : TStrucProc;
CapStr : string;
tmpItem : TStructureElement;
tmpLines : TStrings;
//Initialization of Caption-Query
//1. plain BDE
procedure DSInit1;
procedure Locate1;
//2. plain IBX
procedure DSInit2;
procedure Locate2;
protected
procedure Execute; override;
procedure SetCaptionToNode;
procedure SetNodeCap;
procedure Deltmpitem;
public
//Parameters for BDE - Stand-alone, TTable, no Database-object
constructor Create(const ATable : TTable; const KeyFieldname,
CapFieldname : string); overload;
//Parameters for IBX, Stand-alone
constructor Create(const AIBDatabase : TIBDatabase; const
CaptionTablename, CaptionKeyFieldname, CapFieldname: string); overload;
destructor destroy; override;
procedure SetGetStr(AGetStrucStrProc : TStrucProc);
procedure AddJob(const AnElement : TStructureElement);
procedure AddUrgentJob(const AnElement : TStructureElement);
property Jobs : TList read FJobs;
end;
var
CapThread : TCapThread;
implementation
uses Windows, SysUtils, Forms;
{ TCapThread }
procedure TCapThread.AddJob(const AnElement: TStructureElement);
begin
FJobs.Add(AnElement);
end;
procedure TCapThread.AddUrgentJob(const AnElement: TStructureElement);
begin
if AnElement.Node.IsVisible then
begin
AnElement.Urgent := AnElement.Node.IsVisible;
FJobs.Insert(0,AnElement);
end;
end;
constructor TCapThread.Create(const ATable : TTable; const KeyFieldname,
CapFieldname : string);
begin
inherited Create(true);
FGetStrucStr := nil;
FCapQueryInit := DSInit1;
FGetNodeCap := Locate1;
FJobs := TList.Create;
FDatabaseName := ATable.Databasename;
FDatabase := nil;
FTablename := ATable.Tablename;
FKeyFieldname := KeyFieldname;
FCapFieldname := CapFieldname;
FreeOnTerminate := true;
end;
constructor TCapThread.Create(const AIBDatabase: TIBDatabase;
const CaptionTablename, CaptionKeyFieldname, CapFieldname: string);
begin
inherited Create(true);
FGetStrucStr := nil;
FCapQueryInit := DSInit2;
FGetNodeCap := Locate2;
FJobs := TList.Create;
FOrigDatabase := AIBDatabase;
FTablename := CaptionTableName;
FKeyFieldname := CaptionKeyFieldname;
FCapFieldname := CapFieldname;
FCapDataset := nil;
FreeOnTerminate := true;
end;
procedure TCapThread.Deltmpitem;
begin
FJobs.Delete(FJobs.IndexOf(tmpitem));
end;
destructor TCapThread.destroy;
begin
try
if not assigned(FCapDataset) then
FCapQuery.Free;
except
end;
FJobs.Clear;
FJobs.Free;
inherited;
end;
//Use a local Paradox-table
procedure TCapThread.DSInit1;
begin
FCapDataset := TTable.Create(nil);
TTable(FCapDataset).DatabaseName := FDatabaseName;
TTable(FCapDataset).Tablename := FTablename;
FCapDataset.Open;
FCapFieldIndex := FCapDataset.FieldByName(FCapFieldname).Index;
end;
//use an InterBase-table via IBX (own connection -> thread-safe)
procedure TCapThread.DSInit2;
begin
FDatabase := TIBDatabase.Create(Application);
FDatabase.Params.Assign(FOrigDatabase.Params);
FDatabase.DatabaseName := FOrigDatabase.DatabaseName;
FDatabase.LoginPrompt := false;
FTransaction := TIBTransaction.Create(Application);
FTransaction.AddDatabase(FDatabase);
FTransaction.DefaultAction := TACommit;
FTransaction.Params.Text :=
FOrigDatabase.DefaultTransaction.params.Text;
FDatabase.DefaultTransaction := FTransaction;
FDatabase.Open;
FTransaction.Active := true;
FCapQuery := TIBSQL.Create(Application);
FCapQuery.Database := FDatabase;
FCapQuery.Transaction := FTransaction;
FCapQuery.SQL.Text := 'select ' + FCapFieldName + ' from ' +
FTablename + ' where ' + FKeyFieldname + '=';
FCapStrLen := Length(FCapQuery.SQL.Text)-2;
FCapFieldIndex := 1;
Application.ProcessMessages;
end;
procedure TCapThread.Execute;
var
intensecount : integer;
begin
Synchronize(FCapQueryInit);
while not Terminated do
try
if FJobs.count > 0 then
begin
intensecount := 50; //work hard for 50 items (no sleep)
while (intensecount > 0) and (FJobs.count > 0) do
begin
if FJobs.count > 0 then
begin
SetCaptionToNode;
Dec(intensecount);
end
else
intensecount := 0;
end;
end
else
//give some time to the application
Sleep(20);
except
end;
end;
procedure TCapThread.Locate1;
begin
FCapDataset.FindKey([tmpItem.ID]);
tmpLines.Text := FCapDataset.Fields[FCapFieldIndex].AsString;
end;
procedure TCapThread.Locate2;
begin
FCapQuery.SQL.Text :=
copy(FCapQuery.SQL.Text,1,FCapStrLen)+IntToStr(tmpitem.ID);
FCapQuery.Prepare;
FCapQuery.ExecQuery;
tmpLines.Text := FCapQuery.Fields[0].AsString;
FCapQuery.Close;
end;
procedure TCapThread.SetCaptionToNode;
var
PreStr : string;
begin
try
tmpItem := TStructureElement(FJobs[0]);
if assigned(tmpitem.Node) then
begin
if tmpitem.Urgent then
begin
try
if length(tmpItem.NodeCaption) = 0 then
begin
if assigned(FGetStrucStr) then
FGetStrucStr(tmpitem, PreStr)
else
PreStr := '';
tmpLines := TStringlist.Create;
try
FGetNodeCap;
if length(tmpLines.Text) > 0 then
CapStr := PreStr + tmpLines.Strings[0]
else
CapStr := PreStr;
//The node-caption should only be modiffied within a
Synchronize!
Synchronize(SetNodeCap);
finally
tmpLines.Free;
end;
end;
finally
//access to treeview-items should be synchronized!
Synchronize(DeltmpItem);
end;
end
else
begin
//move to the end of the queue - not urgent enough
FJobs.Move(FJobs.IndexOf(tmpitem),FJobs.count-1);
tmpitem.Urgent := true;
end;
end
else
Synchronize(Deltmpitem);
except
end;
end;
//Get a procedure that modifies the caption-look (if needed)
procedure TCapThread.SetGetStr(AGetStrucStrProc: TStrucProc);
begin
FGetStrucStr := AGetStrucStrProc;
end;
procedure TCapThread.SetNodeCap;
begin
try
if assigned(tmpitem.Node) then
tmpItem.NodeCaption := Capstr;
except
end;
end;
end.
<Sample code end>
I hope this is clear enough. It works (copied without modifications) and
is very permormant.
Bye
Robert
Robert Kuhlmann
I'm answering a bit late because I was on a conference for three days
(no internet connection - incredible!).
If there's a fast way to load the nodes, why not use it?
I'm creating the treenodes is about 0.3 sec., the rest is DB-access. And
the treeview is able to handle a large amount of nodes. What is limited
is the amount of nodes can be expanded at the same time. So it is
neccessary to prevent more than 65000 open nodes (e.g. by closing nodes
that were not accessed for a longer time).
But even if more than 65000 nodes are "in" the tree, they are still
there, there's only trouble to display them. The treeview refuses to
scroll down further. After closing some nodes the missing ones show up
correctly.
At last, my internal structure has to be in memory as a whole, because
the entries are interconnected in multiple ways (and plymorphic too).
But this depends on the requirements the application has to fulfill.
Have a nice day
Robert
Mike Orriss (TeamB)
> If there's a fast way to load the nodes, why not use it?
>
Just because you can you mean?
I don't see the point of wasting machine cycles adding items which never
get displayed.
Why wait even 0.3 secs?
Robert Kuhlmann
maybe I wasn't detailed enough. In my application the treeview is just
one sight on items that are not only connected via the treestructure,
but with cross references. So in this case it makes sense to load them
at once, because the internal structure can be handled more easy and
faster. It's not only the treenodes that have to be displayed, but e.g.
connection lists of an item (displayed in the treeview) to other items
(not displayed in the tv yet). The goal was a fast transfer of the
internal structure to the treeview. Of course, if I can transfer fast
from within memory that's from a local DB too. The Tips in my original
posting work, no matter what kind of source the items come from.
I'm not "against" loading on demand. I just want to show that it is not
unalterable and that even big structures can be handled in memory. That
is useful if a fairly slow connection to the DB would slow down the
handling of the treeview. It may be worth to wait though to avoid
display delays while working within the tree (because of loading
operations).
Bye
Robert
Mike Orriss (TeamB)
> Of course, if I can transfer fast
> from within memory that's from a local DB too. The Tips in my original
> posting work, no matter what kind of source the items come from.
>
I was merely making the point that your tips were incomplete.
If the treeview only ever displays 500 nodes, then however you load them
it *must* be quicker to load 500 rather than 10,000!
Robert Kuhlmann
bob mackey
> > Levels) are
> > Pentium 333, Interbase, 100MBit LAN: 4 seconds
> > Pentium 333, Paradox, local: 2 seconds
> > Athlon 650, Paradox,local: < 1 second
> >
>
>
> If you are holding all the info to populate the tree, there is no need to
> add nodes until they are displayed. By creating/destroying nodes during
> OnExpand/OnCollapse you get a very speedy tree indeed.
>
> As it happens, I have just built a tree today controlled by a DBISAM table
> with 8238 records at seven levels. Using SetRange as appropriate, loading
> appears instantaneous.
Does you database contain the tree structure explicitly, or have you added
additional fields to describe the tree structure?
I have a flat db. The user can create a tree on all or a subset of the db. I
have to store this info for later recall. There are no constraints on the
number of children a node can have.
I have a scheme for doing this, but I am interested in hearing about how others
do it.
Do you mind sharing your method?
--
bob
myName = 'bobmackey';
myISP = 'annapolis.net';
myaddress := myName + '@' + myISP;
Robert Kuhlmann
I've done both:
1. Seperate Table
In this case, the structure-table has to contain the info on the
structure and infos on where the data can be found. This typeinfos are
just simple integer fields that indicate a tablename out of a list of
tablenames (stored in a table too, of course).
I've tested severeal mechanisms to represent the structure in a DB and
the following wins all tests:
"ID, FatherID, SeqNo, DatakindID, DataID", where SeqNo represents the
position within a level.
A caption may be added, but I use to load it (from the different
datatables) in a thread.
Advantage:
Very flexible, can hold any kind of information and can be extended with
additional datatypes anytime.
2. One-Datakind-Structure
The more simple form, if data of only one kind has to be stored, is to
add
"ID, FatherID, SeqNo"
to the "normal" datafields. Which is appropriate if there's only one
kind of data to be stored in the structure (or the data is hidden in
Blobs).
Advantage:
Easy to implement
To allow easy and fast handling of the structure I read the records
(only ID, FatherID, SeqNo) as they come into memory and then add some
structure infos that can be retrieved easily out of these three
structure-info-fields.
Result of this is a structure in memory, where any item knows about it's
father and holds a reference-list to all of it's childs. This means
extra memory-usage but speeds up handling a lot.
Interested in more of this?
Let me know.
Bye
Robert
Mike Orriss (TeamB)
> Does you database contain the tree structure explicitly, or have you added
> additional fields to describe the tree structure?
>
It is difficult to be precise because I use treeviews (in particular the
TdxTreeList from Developer Express) all over the place. Thinking about it,
the one common thing is that the treeview never owns the data, merely
displays the whole or a subset of data stored elsewhere.
Bruce
control when the display refreshes?
Bruce
"bob mackey" <nos...@annapolis.net> wrote in message
news:38DF7123...@annapolis.net...
> "Mike Orriss (TeamB)" wrote:
> >
> > In article <38CD7675...@planet-interkom.de>, Robert Kuhlmann
wrote:
> > > The actual speeds I can reach for a 15.800 items structure (app. 8-10
> > > Levels) are
> > > Pentium 333, Interbase, 100MBit LAN: 4 seconds
> > > Pentium 333, Paradox, local: 2 seconds
> > > Athlon 650, Paradox,local: < 1 second
> > >
> >
> > Why bother?
> >
> > If you are holding all the info to populate the tree, there is no need
to
> > add nodes until they are displayed. By creating/destroying nodes during
> > OnExpand/OnCollapse you get a very speedy tree indeed.
> >
> > As it happens, I have just built a tree today controlled by a DBISAM
table
> > with 8238 records at seven levels. Using SetRange as appropriate,
loading
> > appears instantaneous.
>
> Does you database contain the tree structure explicitly, or have you added
> additional fields to describe the tree structure?
>
Robert Kuhlmann
>
> Sorry if I am coming into this late, but are you using .BeginUpdate to
> control when the display refreshes?
>
> Bruce
If you're asking me, yes.
Bye
Robert