IntellisensX: Performance optimizations

[Walter Meester]

Hi Jim,

I tried to use IntellisenseX but the list by dot implementation was so slow that is was unusable to me.

Thankfully, you've included all the sourcecode so I went out to investigate where all the time is spend on.

Turned out to be that 85% of all time is spend on compiling scripts on EXECSCRIPT(). But tweaking that through compiling it only once, performance got acceptable.
Further when executing procedures the procedure - file is searched through FILE() function operations, also slowing down the whole operation. By eliminating that, performance got even better.
I did some other little performance tweaks to save some time.

I realize that my changes might infringe with some for me unknown functionality, but I'd like to exchange some thoughts in improving in IntellisensX to make it perform better.


Also the inability to click away the intellisense list is easy to solve by using the following trick

In the show of the form put a line:
ON KEY LABEL MOUSE _Screen.ActiveForm.Clicked

In the destroy of the form, put
ON KEY LABEL MOUSE

Create a clicked event and set it to
IF MCOL() = -1 OR MROW() = -1
      ON KEY LABEL MOUSE
      THISFORM.Release()
ENDIF


Please let me know what you think.

[Jim Nelson]

Walter --

Sounds like you've done some really good work on a project has been dormant for a long time.  You mentioned two different topics:

[1]  Optimizing by removing calls go EXECSCRIPT -- do you have changes that can you send to do this?  I am most interested in seeing your work (without making any promises what I will be able to do with them)

[2]  As for removing the IntellisenseX list -- can you supply the code for this as well?  I don't see how it can work, since that dropdown list is in a modal form, so there must be something I am missing.

--
You received this message because you are subscribed to the Google Groups "Thor, the Tool Manager for FoxPro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to FoxProThor+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jim Nelson

(805) 498-9195 (preferred)
(720) 837-3536 (cell)

www.kongcompany.com

[Walter Meester]

Hi Jim,

I'm still in the process going over the project to make some changes. I'll ping you when I'm done.

So far I've been able to reduce the time from dot to list from over 2 seconds to about 0.1. The callbacks to the thordispatcher through EXECSCRIPTS in tights loops is the main culprit, but I made some other optimizations too. I really like the prgs to be called directly rather than using the dispatcher, or to put all the procedures in classes so that functions are called as methods rather than procedural code where a file is the function. I'm not creating any new functionality, but rather to refactor it for the purpose of improving performance.

As for the modal form, just put the code in my first email in the form events and see for yourself. It works. If you click outside of the form, whether modal or not, it will fire the clicked event. The click event will determine that the click was outside of the form (MCOL() and/or MROW() will return -1) and if so, you can call thisform.release().

Walter,

Op dinsdag 14 juni 2016 15:16:51 UTC+2 schreef Jim:

[Jim Nelson]

Walter --

I am not in the position to do any testing or evaluation on this for some time.  Any work that you do in the meantime will be greatly appreciated.

All of IntellisenseX was written in a highly modularized form, which includes the opportunity for user customization (which I and a few others take full advantage of).  However, since it now as stable as it's every going to be, that modularization is no longer necessary.

I have no doubts that you are achieving great results, and really appreciate your work in doing so.  My timings are not as slow as yours, probably because of differences in paths, etc.

I really look forward to seeing what you have, although I will rely on others to do a lot of the testing.  No doubt this will require a Beta version ....

[Matt Slay]

Walter - I'd be happy to test any updates that you make.  I have some large PRG-based classes that I code against frequently, and I too have noticed a small lag, but it has been bearable to me so far.

Matt Slay

[Mike Potjer]

Walter,

I would like to test this, too, if the code can be made available somewhere.

[Jim Nelson]

Mike and Matt --

I was sure you guys would show up!

[Walter Meester]

All,

Just want ot make clear that my performance improvements are on the intellisenseX by dot.
Since I do not want anything to get broken, I'll post only two files of my changes.

The difference in performance should be well noticable, but is not as fast as what I got here locally because I've taken the root to compile some PRGs on the fly and changed lots of calls with EXECSCRIPT() in other places as well. I'm not confident about those changes to post them here.

So the thor_proc_getprgbasedclasspems.PRG contains the performance optimizations
and the Thor_Proc_ISXForm.PRG contains the enhancement to be able to click away the popuplist.










Op dinsdag 14 juni 2016 22:29:11 UTC+2 schreef Jim:

[Jim Nelson]

Walter --

Thanks for this.  The PRG you sent with optimizations is one that is called for PRG-based classes (hence its name) and would of course be quite slow for large PRGs.  (I use PRG-based classes rarely, so have not observed this.)

Perhaps, before working on any more such changes, we should talk about what you are doing, just to make sure that the path you are following makes the most sense.  There may be a more general approach that could be applied that would benefit not only IntellisenseX but also other Thor tools as well -- this could have quite some significant impact.

Your suggestion is, it seems, to essentially bypass the call to ExecScript, which re-compiles _Screen.cThorDispatcher and waddles thru some code before finding the file it wants to execute. As you have identified this as a bottleneck, it pretty much screams for some optimization.

[Walter Meester]

Jim,

In fact analyzing larger PRGs is not that slow. Analyzing a 2000 line class in a PRG takes approx 0.04 seconds on my machine. Its the stuff arround it that makes it slow.

I think replacing the dispatcher with a PRG Would be the first step. Then finding a better (faster)  way to call functions and procedures from any program.

[Jim Nelson]

Walter --

Replacing the dispatcher with a PRG will never happen. The decision to use a property of _Screen was made very consciously -- that property survives Clear All and is always accessible without touching anybody's PATH statement (one of the important design criteria in THOR was not to muck with your environment). 

That being said, your follow on point about finding a better (faster) way to call functions and procedures can definitely be addressed.

--
You received this message because you are subscribed to the Google Groups "Thor, the Tool Manager for FoxPro" group.
To unsubscribe from this group and stop receiving emails from it, send an email to FoxProThor+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Potjer]

Jim,

Maybe you're already thinking along these lines, but I wonder if it would help to strip the dispatcher down to a stub that receives the parameters and knows where Thor is installed, then calls a .PRG containing the rest of the code currently in the dispatcher.

[Mike Potjer]

Thanks Walter!

The enhancement to close the form by clicking outside of it is working fine.  I still need to find a specific example I can use to check the performance enhancements, though.

[Jim Nelson]

Mike --

I wonder if it would help to strip the dispatcher down to a stub that receives the parameters and knows where Thor is installed, then calls a .PRG containing the rest of the code currently in the dispatcher.

Actually, no, that is not what I was thinking.  I am loathe to make any other than the most trivial changes to Thor, for a number of reasons (fear of unintended consequences, lack of time, memory of difficulties in getting it to work in the first place, lack of memory of how that was done, and so on)

My thought was to provide a new direct way for calling Thor procedures and functions, identical in effect, which could be used where desired instead of the current use of ExecScript.   This would mean more grunt work changing existing references, but safer because only those changed places would work differently.

[Walter Meester]

Well, the thing we can do is eliminate the usage of the dispatcher altogether.

1. Combine several prgs so that functions do not have to be called through the dispatcher. For example, include the thor_proc_getparameters.prg into thor_proc_getprgbasedclasspems.PRG

2. Since the relative locations of the PRG are assumed not to change anymore, you can use

This.cImagesFolder = _Screen.cThorfolder + 'Tools\Images\'
In stead of:
This.cImagesFolder = Addbs(ThorDispatcher( 'Tool Folder=')) + 'Images\'

AND

loISX = NEWOBJECT('ISX_Utilities', FORCEPATH('Thor_Proc_ISX_utilities.PRG', _Screen.cthorFolder+'tools\procs'))
In stead of
loISX = ThorDispatcher('class= ISX_Utilities from Thor_Proc_ISX_Utilities')

I've not cut out all EXECSCRIPTS() for the mdot  intellisense for PRGs, and getting the list takes approx 0.04 seconds for a 2000 line PRG where previously it took almost 2 seconds.
Now however displaying the list on the screen seems to take the most time: Approx 0.2 seconds.

3. Reuse objects

    IF VARTYPE( _oPEMEditor) <> 'O'
        loEditorWin = ThorDispatcher('Class= editorwin from pemeditor')
    ELSE
        loEditorWin = _oPEMEditor.oEditorWin
        loEditorWin.FindWindow()
    ENDIF




Op vrijdag 17 juni 2016 01:19:46 UTC+2 schreef Jim:

[Jim Nelson]

Walter --

Now that I am back from (and have recovered from) vacation, I have had a chance to read through this thread about optimizing performance of IntellisenseX.

It's kind of amazing, after all this time, that you found that such significant improvement was possible and that it had to do with Thor, not IntellisenseX.  Perhaps there's something about your environment that makes it more visible to you ... or, maybe, the rest of us have simply adjusted to and accepted the performance as is.

Happily, you also identified the major bottleneck:

"Turned out to be that 85% of all time is spend on compiling scripts on EXECSCRIPT(). But tweaking that through compiling it only once, performance got acceptable."

To correct this, I have created a new object as part of today's update to Thor, which provides the same capabilities as the familiar ExecScript(_Screen.cThorDispatcher) but does not suffer from the problem of re-compilation.  

Thus, if your statement above is reasonably accurate, this alone should achieve acceptable performance for you.

To use this new object, simply replace  ExecScript(_Screen.cThorDispatcher  with _Screen.oThor.Do(

Note that this change does not affect existing code, unless it is altered as suggested, and it can be taken advantage of in all Thor tools (but probably will only be used on those where it is found to be needed.)

In addition, this is a change that can be made rather painlessly (GoFish!). This is a critical concept for me, since I am ultimately responsible for all changes to IntellisenseX and am extremely leery of even the simplest changes to a tool as complex as IntellisenseX.  Restricting the changes in the IntellisenseX procedure to something as simple as this should not be too difficult to validate and would fit within the amount of time I have to devote to this.

P.S.: I have not yet included your suggestion for On Mouse Down, which allows a mouse-click to close the IntellisenseX dropdown (the same as pressing Esc.), since that is actually part of the IntellisenseX project.