Leo's new load process
Edward K. Ream
Leo's implementation. Feel free to ignore.
Otoh, this post explains why progress on the grand refactoring now in
progress may be a bit "stately" :-)
Last night I wrote some notes to myself. They contain questions that
will require experimentation to resolve. The code is literally too
difficult to analyze directly. The essence of the situation is that
initing Leo's various classes is extremely complex, especially the
Commander, leoFrame and leoKeyHandler classes.
I may create a new guideline for all "complex" ctors: they should be
able to execute successfully regardless of whether other objects have
been completely inited. This is hardly a complete solution, but it may
be a step forward.
Anyway, here are my notes from last night. They should give you a
(partial) feel for the complexities that the LoadManager must handle:
QQQQQ
Reading any file (including settings files) will be done as follows:
1. Create a commander for the file.
**Important**: The gui is known, so we can create a frame.
However, we don't want to *draw* the frame without knowing all
settings.
To do: Can we create a frame without drawing it??
2. Physically read the .leo file using the sax parser.
To do: Can be done at all?? That is, can it be done using only
partially inited commander?
Experiment!
To do: Can this be done *without* knowing any settings??
It would seem essential, but I'm not entirely clear about this.
3. Handle the settings in the file.
A. Parse the @settings tree in the file.
B. Merge the settings with previous settings.
Note: this may set LoadManager ivars.
C. (New) create a *single* config dict, so that c.config.get is
very fast.
4. Enable plugins after *all* settings have been read?
To do: do we have to enable plugins all at once, or can we do that
for each file??
5. Draw the frame, thereby generating events to be handled by plugins.
QQQQQ
These notes are merely a starting point. Experimentation will likely
reveal further complexities and different strategies for getting .leo
files loaded in a single pass, without loading any file twice.
Edward
Edward K. Ream
> Experimentation will likely reveal further complexities and different strategies for getting .leo
> files loaded in a single pass, without loading any file twice.
"infrastructure" is required to load a .leo file? If I can make this
process self-contained, I may be able to avoid the immense
difficulties that would be entailed by changing Leo's ctors.
The idea is simple: have the load code work *independently* of the
Commands class. The process becomes:
1. Load the .leo file *without* any Commander.
2. Parse settings in the loaded file, creating a config object, again
without any Commander.
3. Create the Commander (and all the subsidiary objects, including
frames) using the data gathered in steps 1 and 2.
This looks like the simplest thing that could possibly work. However,
it will require that steps 1 and 2 work without a commander. I
believe that is feasible, but only time will tell...
Edward
Edward K. Ream
> A new strategy has just come to mind. The question is, what kind of "infrastructure" is required to load a .leo file? If I can make this process self-contained, I may be able to avoid the immense difficulties that would be entailed by changing Leo's ctors.
I studied all the calls to config.getX in the classes that become part
of the Commands class. Most of these calls are irrelevant to the
question at hand. In particular, the ctors for the atFile,
shadowController, and tangleCommands classes do contain calls to
config.getX, but such calls don't matter, because we don't have to
read external files to read a .leo file!
The fileCommands class *does* use two config settings, but happily
neither is used when *reading* a .leo file.
So it should indeed be possible to refactor leoFileCommands.getLeoFile
in so that it reads (only) the .leo file *without* relying on any
settings. Such a refactoring isn't trivial, but it is a local project
that will not have wide repercussions.
This is the way forward that I have been seeking.
The next step will be to discover what kind of data
leoFileCommands.getLeoFile presently sets in the commander. We can
probably create a "fake" commander that contains only those data. We
pass that fake to getLeoFile which sets the ivars. We can then set
the ivars in the "real" commander when we are ready to create it.
This is far from a straightforward plan, but let's keep the prize in
view: it allows us to load a .leo file in a single pass, as follows,
*without* greatly changing how Leo's many objects get inited. Here is
the overview of the process...
1. Load the .leo file without using any settings, and without loading
any external files.
2. Discover the settings by scanning the loaded .leo files.
3. Complete the loading using the settings::
A. Init all sub-objects of the commander, by calling the ctors and
the various finishCreate methods.
B. Read the external files.
C. Create the gui's frames, thereby generating events to be handled
by plugins.
Edward
Edward K. Ream
> The next step will be to discover what kind of data leoFileCommands.getLeoFile presently sets in the commander.
kind of dummy commander is required. The following works::
# fc.ctor references c.frame.
class dummyFrame:
def __init__(self):
self.ratio = 0.0
class dummyCommander:
def __init__ (self,frame,fn):
c = self
self.changed = False
self.fn = self.mFileName = fn
self.frame = frame
self.hiddenRootNode = leoNodes.vnode(context=c)
self.p = leoNodes.position(v=None)
# Call these after setting the ivars.
self.config = leoCommands.configSettings(c)
self.cacher = leoCache.cacher(c)
def all_unique_positions(self): return []
def hash(self): return 0
def rootPosition(self): return self.p
def selectVnode(self,p): self.p = p
def setChanged(self,flag): self.changed = flag
def setCurrentPosition(self,p): self.p = p
def shortFileName(self): return self.fn
With these classes defined in the @others section, the following unit
test passes:
fn = r'C:\leo.repo\trunk\leo\doc\LeoDocs.leo'
@others
frame = dummyFrame()
c = dummyCommander(frame,fn)
theFile = open(fn,'rb')
fc = leoFileCommands.fileCommands(c)
ok,ratio =
fc.getLeoFile(theFile,fn,readAtFileNodesFlag=False,silent=False)
print(ok,ratio)
This shows that I could pass a *real* (but only partly inited)
Commander to the ctor for the fileCommands class. I could then use
the "half-inited" fileCommands class to read the .leo file.
This is simply a proof of concept. Instead of relying on a commander
at all, I might rather change fc.getLeoFile so that it simply
remembers the desired results. But surely, some way can be found to
get the job done relatively cleanly.
Edward
Edward K. Ream
>> The next step will be to discover what kind of data leoFileCommands.getLeoFile presently sets in the commander.
>
> A little experimentation revealed that the proper question is what
> kind of dummy commander is required.
Using a dummy commander probably creates more work than is necessary.
The following puts up an empty window.
c,frame = g.app.newLeoCommanderAndFrame(
fileName=fn,gui=g.app.gui)
fc = c.fileCommands
ok,ratio = fc.getLeoFile(theFile,fn,readAtFileNodesFlag=False,silent=False)
This topic is enough to make my head explode, but I think Leo can load
a .leo file in a single pass, using a single commander, provided that
we adopt the following rule:
** The startup code should use settings only in the finishCreate methods. **
There are several violations of this rule at present, including the following::
1. fc.getLeoFile contains::
if c.config.getBool('check_outline_after_read'):
c.checkOutline(event=None,verbose=True,unittest=False,full=True)
It will not be a big problem to honor this setting (say in
g.openWithFileName) after config settings have been handled.
2. g.openWrapperLeoFile contains:
if c.config.getBool('use_chapters') and c.chapterController:
c.chapterController.finishCreate()
This should be even easier to deal with, as follows...
At present, several finishCreate methods call other finishCreate
methods. It be better to have a single method,
LoadManager.finishCreate, be responsible for calling all other
finishCreate methods. This removes tricky order considerations from
the subsidiary finishCreate methods, and puts them in a single place
where the order is made completely explicit.
lm.finishCreate will be called after all settings have been handled,
so all the other finishCreate methods can use settings.
To make this as painless as possible, I plan to retain the overall
loading structure. This means retaining the names and calling
sequences of load-related methods . In particular, I want to retain
the signatures of the following methods:
g.openWithFileName
g.app.newLeoCommanderAndFrame
fc.getLeoFile
fc.open
The signature of g.openWithFileName will remain unchanged, but its
entire code will probably move to the LoadManager.
The status of the following settings methods is unclear. Their
signatures may change, or perhaps they may disappear completely.
g.app.config.openSettingsFile
g.app.config.updateSettings
===== Conclusions
We are getting close to a strategy. The "use no settings until
finishCreate time" rule is easy to understand and should be relatively
easy to enforce. Indeed a single assert in the config code could
ensure that no calls to it are made until a LoadManager flag is set.
This strategy is still provisional. More experimentation is required,
but I am beginning to have hopes that the new loading code will work.
Edward
Edward K. Ream
> This post will be of interest only to those with a deep interest in
> Leo's implementation. Feel free to ignore.
>
> Otoh, this post explains why progress on the grand refactoring now in
> progress may be a bit "stately" :-)
I am working on two tracks: development of the LoadManager class and
work (in Leo's existing code base) that will eventually result in
reading Leo files only once. This will be a bit tricky if the
standard settings files (leoSettings.leo or myLeoSettings.leo) appear
in the list of files to be loaded.
Here is the checkin log for rev 4983. This should be safe enough to
use, but one never knows...
leoApp.py:
- Removed unused code from LoadManager class.
- Completed LM.computeLeoSettingsPath & LM.computeMyLeoSettingsPath.
- Replaced LM.computeSettingsPath by a single statement in
LM.scanOptions.
- Completed first draft of LM.readGlobalSettingsFiles: more work is
needed.
leoAtFile.py:
- Removed ancient, unused, new_write constant.
- Added several tests of g.new_load, whose effect is that
atFile class will call c.config only int new atFinishCreate method.
leoConfig.py:
- Fixed an old bug in readSettingsFiles: config.updateSettings
must be called *before* g.app.destroyWindow
leoFile.py:
- Refactored fc.getLeoFile using fc.readExternalFiles.
- Renamed fc.open to be fc.openLeoFile and changed all corresponding
calls.
runLeo.py:
- Connected the LoadManager class to the leoApp class.
numerous files:
- added class comments in headlines of ctors so it is obvious
to which class a cloned ctor belongs.
Edward
Edward K. Ream
> I am working on two tracks: development of the LoadManager class and work (in Leo's existing code base) that will eventually result in reading Leo files only once.
The simplest thing that could possibly work is to *reread*
leoSettings.leo or myLeoSettings.leo if they appear on the command
line, that is, if Leo is going to create a window for these files.
Leo will load files in two phases:
Phase 1: Load the standard settings files (using a nullGui).
The LoadManager will remember the created commander and the resulting
settings dictionaries.
Phase 2: Load all files on the command line (using the Qt Gui).
For each file on the command line (*including* standard settings files
if they appear there) the LoadManager will merge the global settings
computed in Phase 1 with the local settings.
To make this work, we need only ensure that calls to config.getX
happen after local settings have been computed. To do this, we'll
enforce the rule that all calls to config.getX happen in the
finishCreate method or at some later time, not in any ctor.
Edward
Edward K. Ream
>
> The LoadManager will remember the created commander and the resulting
> settings dictionaries.
>
> Phase 2: Load all files on the command line (using the Qt Gui).
>
> For each file on the command line (*including* standard settings files
> if they appear there) the LoadManager will merge the global settings
> computed in Phase 1 with the local settings.
c.config.getX. Indeed, rather than the baroque code in
g.app.config.getX, each c.config class will contain two "finalized"
settings dicts, one for shortcuts and one for all other settings. To
get a setting, the c.config.getX methods will simply look up the
setting in the appropriate dictionary.
Note that this scheme will produce the proper results for standard
settings files themselves if they appear on the command line.
Edward
Edward K. Ream
> Work continues. As my brother would say, slow but slow.
I've just seen how to take smaller bites of the elephant.
The Aha is this: there are two parts of the LoadManager class:
1. The part that controls the overall load process. This is the
analog to the run() function in runLeo.py.
2. Various components that init settings and do other "smallish" tasks.
I can develop the pieces of part 2 without using part 1. Indeed,
specific parts of Leo's existing code can call parts of part 2 now.
For example, I can change the *existing* doPrePluginsInit in runLeo.py
as follows:
if g.new_load:
g.app.lm.readGlobalSettingsFiles(file)
# reads only standard settings files, using a null gui.
# uses files[0] to compute the local directory
# that might contain myLeoSettings.leo.
else:
g.app.config.readSettingsFiles(None,verbose)
for fn in files:
g.app.config.readSettingsFiles(fn,verbose)
Imo, this kind of strategy is essential in order to manage the
complexity of the required changes. There is simply no reasonable way
to eat the elephant in a single bite. An incremental approach allows
me to do full tests after each of these smallish bites.
This will continue to be a slow, picky process. For example, there
will be ongoing issues of initing the LoadManager class while we
incorporate more and more of the LoadManager code into Leo.
Edward