Virtual TreeView for Firemonkey (FMX)

[Dmitri]

Hi all,

 

Long time ago I needed to port my project written in Delphi6 to Kylix (Linux). Because VirtualTreeView (VT) was heavily used in almost all the project forms and I didn't find any suitable replacement in Kylix, I ported VT to this platform by myself. Unfortunately, I was in hurry and didn't hesitate to add many many IFDEF/ENDIF conditionals through all the VT sources which resulted in code that could hardly be maintained. So it was "forked" and lived in this state up to date. I mean it didn't get any update or features since that, although I have to admit it worked good.

 

Now I need FMX version of VT and seems it's time to discuss how to port VT-for-VCL to FMX in a right way.

I know, I'm not alone who need VT under Firemonkey. It was asked for here:

http://stackoverflow.com/questions/7433249/firemonkey-version-of-virtualtreeview

and there:

https://forums.embarcadero.com/message.jspa?messageID=411340

and here:

http://stackoverflow.com/questions/8467084/virtualtreeview-component-for-xe2

 

My idea on how to do porting is below.

VT is a Windows component and uses Windows direct calls in many places. It is done partially for efficiency and partially because VCL didn't (and still doesn't) support things needed by VT.

I'd split all calls to external functions invoked by VT into 3 categories:

-pure Windows calls

-calls to functions/methods in VCL that do not have  corresponding functions/methods in FMX

-calls to functions/methods in VCL that can be ported to FMX directly

1) and 2) are the things we need to replace with something or shut-up corresponding feature in VT

This can be done with conditional statements IFDEF/ENDF like in case with VT/Kylix or VCL itself (see lot of PUREPASCAL IFDEFs in VCL code). But I'd like to avoid it. 

First idea that may come in response on how to avoid lot of IFDEFs, is to use OOP power: put all platform-independent code into a base class, then write successor class implementing all platform-specific calls in a dedicated unit.

This approach works in some extends but it has the following weak sides. Suppose VT is ported to platform A and B:

-if development of VT for platform A moved forward and component got some new features, while code for platform B didn't move on, code for B will most probably compile successfully leading to errors in run time due to new methods for A in base class that are not overriden/implemented in code for platform B.

-if you develop a project for both A and B and platform-specific code parts of VT are in different units, you'll have to wrap "use" statements with IFDEF for plafrom A and B (this wouldn't be a big issue if Delphi designer worked transparently with conditionals in the "use" statements)

 

Another approach and I recently tried it with another component, is to put all platform-specific code into a base class, create successor class implemented all features in a plaform-independent way (relying wherever necessary onto base class), create as many base classes as many platforms are needed to support. For example if we need to support VCL and FMX, we'll have: TCustomVirtualStringTreeVCL base VCL-platform class in VirtualTrees.VCL.pas file, TCustomVirtualStringTreeFMX base class in VirtualTrees.FMX.pas unit and their single successor TCustomVirtualStringTree in VirtualTrees.pas.

This is similar to what we have in VCL & Windows -- VCL wraps Windows (platform-specific) calls into VCL ("platform-independent") classes and functions.

Benefits:

-if development with VT for platform A moved forward, got some new features, while B didn't, code for B will most probably NOT compile successfully until you implement all features in platf-B base class the successor referred to. It's always better to encounter comple-time errors than run-time ones. Also this would allow to make development for two platforms almost indepenendently.

-the "platform-independent" class (the successor) lives in its own unit. if you develop a project for both A and B, you can use this single unit. You don't need to unclude platform-depended units everywhere in your project.

 

any comments?

 

Best regards,

Dmitri

[Mike Lischke]

Hey Dmitri,

VT is a Windows component and uses Windows direct calls in many places. It is done partially for efficiency and partially because VCL didn't (and still doesn't) support things needed by VT.

I'd split all calls to external functions invoked by VT into 3 categories:

-pure Windows calls

-calls to functions/methods in VCL that do not have  corresponding functions/methods in FMX

-calls to functions/methods in VCL that can be ported to FMX directly

Maybe it is time to modularize VT in general. I'm not sure how that should be in the end, but the unit is a big beast and Google Code doesn't even show diffs for changes. I loved the idea to have just one file, store it somewhere in your lib folder and add it to your project and you are done. But this has drawbacks.

1) and 2) are the things we need to replace with something or shut-up corresponding feature in VT

This can be done with conditional statements IFDEF/ENDF like in case with VT/Kylix or VCL itself (see lot of PUREPASCAL IFDEFs in VCL code). But I'd like to avoid it. 

First idea that may come in response on how to avoid lot of IFDEFs, is to use OOP power: put all platform-independent code into a base class, then write successor class implementing all platform-specific calls in a dedicated unit.

Basically the string tree does this partially already. So this needs to be extended.

-if development with VT for platform A moved forward, got some new features, while B didn't, code for B will most probably NOT compile successfully until you implement all features in platf-B base class the successor referred to. It's always better to encounter comple-time errors than run-time ones. Also this would allow to make development for two platforms almost indepenendently.

-the "platform-independent" class (the successor) lives in its own unit. if you develop a project for both A and B, you can use this single unit. You don't need to unclude platform-depended units everywhere in your project.

I guess nobody here has experiences with that so the best approach is probably to start and see how this works out. Please use a separate dev branch for this ongoing development.

Mike

-- 

www.soft-gems.net

[Dmitri]

Hi Mike,

 

Maybe it is time to modularize VT in general. I'm not sure how that should be in the end, but the unit is a big beast and Google Code doesn't even show diffs for changes. I loved the idea to have just one file, store it somewhere in your lib folder and add it to your project and you are done. But this has drawbacks.

I managed to live with these  drawbacks :), not sure of SVN :)

In general, what I'm going to get  won't break single-unit-for-the-end-user approach.

All new units to be added while porting, will work as, say, platform-specific service-providers and type-definitions. End user won't need to use these units directly.

Example with type-definitions:

all coordinates in FMX are floating point (good stuff for scalable GUI btw) - so we'll have to define our TPoint, TSize, and all values returned/received by Width/Height-related procedures are not integers anymore. More precisely, the remain integers for Windows, but with wrappers, like

type TVtCoordinate = integer; // in Win32 type-def unit and type = TVtCoordinate = single; // in FMX

Example with service-providers:

ScrollDC and BitBlt aren't available/shouldn't be invoked directly under FMX

So, we'll have to move whole TVTDragImage.DragTo() method or its parts to platform-specific service-provider class and call its method from TVTDragImage.DragTo().

same goes to TVirtualTreeColumns.AnimatedResize(), TBaseVirtualTree.ToggleCallback() and so forth.

I think this can be done similar to how painter classes are implemented in themed software. Often they are just set of class-methods, so no instances are required. Inside these methods they implement all theme-related specifics.

Looks like implementing stand-alone service providers for drawing is better than moving platform specific methods to parent class because the service-provider class (or classes) can be re-used from many different (unrelated in the inheritance tree) classes - like TVTDragImage, TVirtualTreeColumns and  TBaseVirtualTree which can hardly be done with "move-to-parent" approach I mentioned earlier.

Still, move-to-parent approach may work good with features that aren't avilable in other platforms. For example, OLE is not avilable under Mac. So we may want to move OLE-related things to platform-specific parent class.

 

From the end-user point of view, after porting is done he/she will still work with VirtualTrees.pas unit and TVirtualStringTree class in it like it was before.

If under modularization you mean that you want to change something in this area, please explain the details.

Porting is a big work that can't be finished over night :). It's desirable to have as low changes in the main code branch while porting as possible until it is finished. This would simplify further merging the branches if the porting code appears acceptable.

Alternatively, we can do changes together while porting.

I understand that VT is a quite large unit, probably the largest in my project, but look - it will be smaller after all platform-specific stuff is moved away to platform-specific modules. Maybe some of the problems related to its size will go away with porting?

  

I guess nobody here has experiences with that so the best approach is probably to start and see how this works out. Please use a separate dev branch for this ongoing development.

I saw 2 versions of VT ported to LCL

Old LCL Port: Joerg Thaler,Christian Ulrich
New LCL Port: Luiz Américo

(from here: http://wiki.freepascal.org/VirtualTreeview )

 

The latest VT for LCL is 4.8.6. Looks like the authors are porting each time from scratches.

I think they may be interested in porting VirtualTrees.pas in a right way that would simplify futher maintenance under their platform too.

  

Regards,

Dmitri

[Joachim Marder]

Hi there.

Making VTV compatible with FMX having just one source code means really a huge amount of work!

so far we have nearly 100 open issues and nobody has the time to take care of them. In think should fist come to a stable base and fix the open issues before doing such large refactorings, which will for sure introduce some new bugs. Having some unit test before would also be a good thing.

Best regards,

Joachim

2012/11/30 Dmitri <ddmi...@gmail.com>

--
 
 

[Dmitri]

Hi Joachim,

Making VTV compatible with FMX having just one source code means really a huge amount of work!

I'd estimate ~ 8weeks is needed for this work in my spare time. It's not that huge. Not for me at least.

 

so far we have nearly 100 open issues and nobody has the time to take care of them.

In think should fist come to a stable base and fix the open issues before doing such large refactorings, which will for sure introduce some new bugs.

Having some unit test before would also be a good thing.

Good idea, indeed. If you write unit tests illustrating the issues - I promise to take a look at them. Meanwhile I'm not even sure what to check :)

I'm using VT for 11 years now and don't have much troubles with it. According to reports from my customers, they don't have problems related to VT at all. A few problems (~10-15 in total) I encountered during all the time, I already fixed many years ago and I do merge the fixes to each new version I switch to. The latest I'm using is 5.0. If the issues you referred to have any relation to what I have fixed, I'll take care to provide the updates.

But I don't think that having issues in VT is a showstopper for our port-to-FMX process. It's because VT is very stable and probably most of those 100 issues are either caused by wrong/unrecommended use-case or they relate to features I personally didn't touch/didn't need/didn't use. In latter case, I have nothing to help.

Best regards,

Dmitri.

[Joachim Marder]

Hi Dimitri.

> A few problems (~10-15 in total) I encountered during all the time, I already fixed many years ago and I do merge the fixes to each new version I switch to.

So I guess you intend to merge them to the trunk now? It might be useful to open issue in advance, which makes code changes easier to track if you connect both.

> I'm using VT for 11 years now and don't have much troubles with it.

Except the 10-15 issues you mention above?!? I also do not have much trouble, because I fixed all annoying issues that affected my apps. But there are many aspects of VTV that I do not use in my apps.

> probably most of those 100 issues are either caused by wrong/unrecommended use-case or they relate to features I personally didn't touch/didn't need/didn't use.

That might be. But I think it is the wrong attitude for such a community project to focus on the personal needs only. If a feature of VTV is broken, then it should be fixed before adding new features, that my personal opinion on this matter as a professional software developer.

I did not yet manage to go through all open issues, I just managed to keep the total amount stable by fixing about as many issues as new have been opened.

Best regards,

Joachim

2012/12/1 Dmitri <ddmi...@gmail.com>

--
 
 

[Dmitri]

Hi Joachim,

 

> A few problems (~10-15 in total) I encountered during all the time, I already fixed many years ago and I do merge the fixes to each new version I switch to.

So I guess you intend to merge them to the trunk now? It might be useful to open issue in advance, which makes code changes easier to track if you connect both.

I'd like to, but I can hardly recall their details. Sorry, I didn't write formal reports. In the beginning some of the bugs (together with possible fixes) I sent to Mike directly and he re-implemented them in a different way.

I'd suggest to try another way for now - 

1) mark all feature/enhancement requests as such.

2) if possible, add a mandatory field for the bug reports - version of Delphi the problem is reproduced. Also, it would be good to know if each particular problem can be reproduced each time or not. Sporadic or rarely happening issues can be fixed later.

3) create testing suite for VT in google/code, add a few tests demonstrating the rules on how to write tests (I can do this but not sure you'll like it), then ask _all_ bug-reporters to submit use-cases or complete tests demonstrating their issues.

4) I'll walk through all submitted tests and check whether they are reproducible in my case. Note that I have D7, D2009 (XE) and D2011 (XE2), so I can check only issues related to these versions. Whatever issues are resolved by my patch will get their fixes.

> I'm using VT for 11 years now and don't have much troubles with it.

Except the 10-15 issues you mention above?!? I also do not have much trouble, because I fixed all annoying issues that affected my apps. But there are many aspects of VTV that I do not use in my apps.

I mean that those 10-15 issues didn't bring much trouble except perhaps one issue with multi-selection. I noticed that in some cases list of selected nodes may contain deleted nodes and it wasn't easy to find a good solution on how and when update the list.

> probably most of those 100 issues are either caused by wrong/unrecommended use-case or they relate to features I personally didn't touch/didn't need/didn't use.

That might be. But I think it is the wrong attitude for such a community project to focus on the personal needs only. If a feature of VTV is broken, then it should be fixed before adding new features, that my personal opinion on this matter as a professional software developer.

We have to admit - we can't fix all problems for all users without their response and help. 

For example - if we can't reproduce an issue and reporter didn't come back in a reasonable time (say, 6months), we have to close issue with appropriate status, such as "didn't reproduce". 

Having unattended issues for long time does not help to anybody. 

Additionally if you allow  people to send fixes, you need to find a way to accept/reject them timely.

 

I did not yet manage to go through all open issues, I just managed to keep the total amount stable by fixing about as many issues as new have been opened.

I'm not sure that this is the best approach. Getting number of open issues within some margins is not bad, but not a good target either. It looks more like a work for hire which is not the case, right? What we need is a process that would allow to keep VT in stable state and check all coming fixes against test-suite with good code coverage. So I think a good target is test-suite with good (for example 90%) code coverage. 

If a feature of VTV is broken, then it should be fixed before adding new features, that my personal opinion on this matter as a professional software developer.

Still, I don't see how this relates to porting. Under porting I do not mean to add/remove features, there will be only re-arrangement of the code to allow other platforms to implement their specifics in a more transparent way -- notable without much IFDEF/ENDIF hassle. Current VT code/algorithms for VCL will not (should not) be affected and if you get testsuite when I have the porting finished, we'll check both - original and ported and make conclusion on the quality. 

Best regards,

Dmitri. 

[Joachim Marder]

> 1) mark all feature/enhancement requests as such.

Agreed.

> 2) if possible, add a mandatory field for the bug reports - version of Delphi the problem is reproduced.

Agreed, but I don't think Google Code offers this feature.

> 3) create testing suite for VT in google/code, add a few tests demonstrating the rules on how to write tests (I can do this but not sure you'll like it), then ask _all_ bug-reporters to submit use-cases or complete tests demonstrating their issues.

I like the idea, but writing good unit tests isn't easy if you do not have experience with it.

> We have to admit - we can't fix all problems for all users without their response and help.
> For example - if we can't reproduce an issue and reporter didn't come back in a reasonable time (say, 6months), we have to close issue with appropriate status, such as "didn't reproduce".

Agreed.

>Having unattended issues for long time does not help to anybody.

Yep, that's why I wanted to address them. And close as many of them as possible. I got stuck however due top a lack of time.

> Still, I don't see how this relates to porting.

Porting will require refactorings, and refactorings usually introduce new bugs as every experienced developer know. So you usually start major refactorings from a stable base and ideally have a set of unit tests. While the latter one may be difficult to achieve, we should at least aim at the first one.

Best regards,

Joachim

2012/12/2 Dmitri <ddmi...@gmail.com>

--
 
 

[Omorote Hideoshi]

Dear Dmitri,

did you manage to progress with the migration of the VirtualTreeView?

For me it's also the major component and I'm planning to do the migration.

If you already started I can help you with the work.

best,

hide

[Dominik Krickl-Vorreiter]

I'm also interested in a port to FMX.

Regards

Dominik