Inspired by Vitalije's prototype

[Edward K. Ream]

Here are my thoughts after several days of intense study of Vitalije's prototype.

1. The top priority will be to accelerate Leo's read code, using xml.etree.ElementTree and much faster scanning of external files.  The goal will be to be able to do without caching. Imo, it's doable. I'll create a fast-read branch for this.

2. The second, lower, priority will be to heavily revise Leo's tree-drawing code, following several ideas in the prototype.  I'll create a fast-draw branch for this.  To summarize the essentials of Vitalije's tree-drawing code:

- No Tk items are ever recycled, they are reused instead.

- The Tk draw code draws only number of lines that are actually visible.

It should be possible to apply exactly the same principles to the Qt redraw code. In particular, Leo should only do partial redraws, that is, redraws from a given top node to calculated bottom node.  Not sure why I didn't do that originally. And some new ideas have appeared:

- The tree code might not ever have to save positions.  Instead, an item_to_position method might reconstruct a position from visible tree, using just the position of the first visible line of the outline.

- Leo should have a draw_from_position method, that will only ever do partial redraws. To make this work, scrolling must trigger redraws.  That should be straightforward: we'll scroll only by whole lines.

3. Imo, all this can be done without changing Leo's existing vnode and position classes.  I have spent all morning studying the prototypes redraw code.  It's tricky to separate the effects of drawing from tree-traversal issues, but my conclusion is that Leo's existing tree-traversal code is roughly as fast as Vitalije's proposed revisions.

Summary

Imo, the present position and vnode classes are good enough.

The real wins will be in the fast-read and fast-draw branches.

Edward

[vitalije]

This is exactly why I was reluctant to share unfinished prototype. I did beg you though, not to make any decisions before the prototype is finished, but it seems too hard to resist. I couldn't resist the temptation to share the unfinished code, you couldn't resits to make decisions.

I'll write more when I finish my prototype which can't happen in the next ten days. I have to focus on some other things in the following days.

Vitalije

[Terry Brown]

On Fri, Jun 1, 2018 at 2:57 PM, vitalije <vita...@gmail.com> wrote:

This is exactly why I was reluctant to share unfinished prototype. I did beg you though, not to make any decisions before the prototype is finished, but it seems too hard to resist. I couldn't resist the temptation to share the unfinished code, you couldn't resits to make decisions.

I feel a lot of sympathy for Vitalije here, have been wanting to say something for a couple of days, but phrasing is hard.

Part of the challenge of contributing to Leo is the inevitable refactoring of code you contribute, sometimes almost immediately, sometimes after many months.  This refactoring often alters the functionality of contributed code, and certainly makes it more of a challenge for the original submitter to extend the work.  This has happened to me many times - the vs-eval commands are a good recent example.

I think, Edward, that sometimes you re-write code as means to fully understanding it, I'm sure that's effective, but it's also hard on the contributor who understood the code the way they wrote it.  There are big chunks of Leo I only have a vague knowledge of, but I've never considered re-writing them just so I know how they work.

With contributions as significant as the ones Vitalije's is making, I think he should be allowed to complete the work in the form that works best for him, and to guide how the product's evaluated. If he disappears and stops maintaining his contributions, then refactoring's fine, if it's really needed, but while he's in mid-flow he should be allowed to focus on the work without edicts and pronouncements about what can and can't be done.

Cheers -Terry

[rengel]

+++

Well said!

A Not-Invented-Here-so-let's-redo-it attitude is prone to block creativity and the motivation to contribute. The same holds for the 'This will never happen' verdicts that pop up once in a while.

Vitalije is trying to make a major contribution. Why not wait a couple of days to see how it works out? What are 10 days compared to the decades Leo has been out in the open?

Reinhard

[Edward K. Ream]

On Friday, June 1, 2018 at 2:57:52 PM UTC-5, vitalije wrote:

This is exactly why I was reluctant to share unfinished prototype. I did beg you though, not to make any decisions before the prototype is finished, but it seems too hard to resist. I couldn't resist the temptation to share the unfinished code, you couldn't resits to make decisions.

On my walk just now, before reading Terry's comment, I realized that I should have responded to you, here, directly.

I apologize for unjust criticism

Very likely I have caused you severe pain in several recent comments, by criticizing your work unfairly. I apologize profusely. For me, unjustified criticism is almost unbearably painful.

I particularly regret the choice of the word "edict".  It's a principle, a suggestion.  Moreover, it was totally unfair for me to criticize any style choices you made in the prototype.  The last thing I want is for people to worry about criticism of unfinished code.  Again, I sincerely apologize.

I also regret other comments about your code. I was attempting to understand all the ramifications of helper functions, as contrasted with helper methods.  But again, my comments must have seemed like clueless criticisms. Again, I apologize.

BTW, face to face communication has its own failure modes :-)  Thoughtless spoken comments might even be more painful.  Which is why I like slower, more exact communication.

Let's move on...

The prototype

Vitalije, you may feel that the prototype has been misunderstood.  I'm not sure that's so, so let me explain.

Imo, the prototype has/had at least 5 purposes. For me, the first three have already succeeded spectacularly.  The other two, not so much, at present.

1. The prototype demonstrates that .leo files and external files can be read must faster than at present.  In this regard, the prototype needs no further work.  It is a complete success.

2. The prototype demonstrates that redrawing Leo's outline pane might also be orders of magnitude faster than at present. I am completely convinced, so prototype is a complete success, as it is.

3. The prototype demonstrates several new (to me) aspects of programming style.  I'm pretty sure I understand the benefits of helper functions instead of helper methods.  Dereferencing local vars is quicker, and there is no "self" clutter.

The fast-read branch uses this style where appropriate.  It's not needed everywhere, but it is spectacularly good in crucial places.  Once again, the prototype is a complete success, as it is.

Now to the less successful (so far) aspects of the prototype.

4. The prototype contains several examples of what I'll call "iterator style".  The code builds up lists (or bunches), and delivers them to code that then uses the lists/bunches.  This style is still confusing to me, even after detailed study.  A different prototype, might reveal why this style appeals to you.

5. The prototype may be attempting to demonstrate that new positions/vnodes would be better.  At present, I am completely unconvinced.  I see nothing in the prototype code that is persuasive.  Perhaps the reverse.

Perhaps you could explain, with words, not code, why new positions would be better.  In particular, I do not understand what it means to have a position "persist" when the outline changes.  I need a picture as good, say, as the git branch picture.  That made all the difference.

Future prototypes

I do hope I haven't made you shy about making further contributions.  That would be a big loss to the whole Leo community. 

It would be great to see a Qt prototype of fast screen redrawing.  The Tk prototype is completely finished, imo. I've written Tk drawing code, and the prototype Tk code was furlongs ahead of mine.  But Tk is way different from Qt, so the Tk prototype demonstrates only that Leo's Qt drawing code might be way faster than it is.

Aside: I now remember why I gave up incremental drawing in Qt.  It was just too difficult for me to handle scrolling.  The skip argument to ltm.display_items is really clever, though I'm not sure exactly how it works ;-)  It would be great if you could adapt that idea to Qt.  That would be a huge contribution.

Summary

I regret some of my recent comments. I apologize for them I promise not to criticize future prototypes!

I particularly regret the edict post.  Style considerations need not apply to prototypes!

The miniTkLeo prototype has been a great success. Imo, no further work on it is needed. It might have been just as useful if it had come even earlier!  In future, I hope you will publish early and often.

I would very much like to see a prototype of Qt incremental drawing.

I still do not understand iterator style. I'm not sure what it would take to open my eyes.

Code alone will not convince me that new positions are worth having. I'll probably need a simple picture.

Edward

[Offray Vladimir Luna Cárdenas]

On 02/06/18 16:11, Edward K. Ream wrote:

Future prototypes

I do hope I haven't made you shy about making further contributions.  That would be a big loss to the whole Leo community. 

+1000.

Please find the way that works for having prototypes going in this community with proper times for finishing them and dialogue in a timely manner.

Even not being so active as before here, I find this community and its dynamics inspiring and that's in part, because of the intensive prototyping and detailed talk about them (despite of not using Python as my favorite language anymore and loosing a lot of details).

Cheers,

Offray

[Edward K. Ream]

On Friday, June 1, 2018 at 3:37:55 PM UTC-5, Terry Brown wrote:

I think, Edward, that sometimes you re-write code as means to fully understanding it, I'm sure that's effective, but it's also hard on the contributor who understood the code the way they wrote it.  There are big chunks of Leo I only have a vague knowledge of, but I've never considered re-writing them just so I know how they work.

With contributions as significant as the ones Vitalije's is making, I think he should be allowed to complete the work in the form that works best for him, and to guide how the product's evaluated. If he disappears and stops maintaining his contributions, then refactoring's fine, if it's really needed, but while he's in mid-flow he should be allowed to focus on the work without edicts and pronouncements about what can and can't be done.

Vitalije may have been upset recently, but I have been upset as well.

My upset arises from shock that my excitement and admiration for Vitalije's work could have been construed as criticism.

The word 'edict' may have been ill chosen, but it was clearly stated to apply only to finished work, not to prototypes.

Furthermore, I have found a synthesis between lisp/Vitalije style and class-based style.  I'm not best pleased that nobody seems to understand this.

The FastRead class in leoFileCommands.py (the fast-read branch) gets the best of both worlds:

- fast.readWithElementTree calls methods, with all variables "bound", because doing so only adds 3 method calls.

- The crucial fast.scanVnodes uses Vitalije's insight.  It avoids all inner calls, yet it is clear because of section references.

The result is, imo, much much better than either I or Vitalije would have created on our own.

Last, I have repeatedly said that Vitalije's prototypes have already been a complete success.  I am not best pleased that nobody seems to have acknowledged this. There seems little point in continuing a prototype after it has served its purpose.

All comments welcome.

Edward

[jkn]

 Speaking for myself - but I suspect I am not along...

I think Edward's response has seemed very positive, in general, and I see both him and Vitalije as being remarkebly ego-free in all of this. This is another reason this part of the internet is so refreshing.

I suspect one reason there has not been much more general response:

> Furthermore, I have found a synthesis between lisp/Vitalije style and class-based style.  I'm not best pleased that nobody seems to understand this.

is that it just feels like quite early days with this.

Edward is very generous with reporting his 'aha' moments, as just that. Sometimes they pan out, sometimes they don't. He (Edward) is probably clearer  than me at the moment that this current work will lead to major improvements.  That is terrific, but I am only in the position of watching with interest from the sidelines.

In any case I think it is *terrific* that Vitalije has published his proof-of-concept work and that Edward has taken it and run with it. I don't see the latter as an example of  Inot Invented Here syndrome. adn a surely hope that Vitalije continues to make such contributions. I am pretty sure he is mature enough to realise that not everything will be accepted, but that it will be considered.

    HTH
    Jon N

[Offray Vladimir Luna Cárdenas]

I agreed with Jon. There are some parts of the discussion that are quite technical (that's why my other proposal of being able to test features and docs in Leo prototypes) and I think that only when Vitalije comes back from his 10 days immersion we will be able to follow what has been mostly a polite, constructive and ego free conversation.

Cheers,

Offray

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

Here is an exchange that happened privately.  Imo, it should be public:

EKR:

> My new work is itself an important demo.  It conclusively proves that
> the read-logic speedups are completely independent not only of new
> positions, but *any* positions.  Everything happens in "vnode space":

Terry:

It's not really about the code / performance outcome, but about not
respecting Vitalije's clearly stated desire to have his ideas considered
in a relative complete state.

Vitalije's ideas, so far as they relate to speed issues, have already been:

a) considered and fully accepted.

b) acknowledged as brilliant work.

c) put into Leo's code base (the fast-read) branch.

d) separated from other, more speculative ideas, such as new positions.

Point d is important.  No amount of speed-related work can address the difficulties of new positions.  That's just the way it is.  New positions may work. Or not. I don't have a strong opinion about them. But in order to work new positions must be:

1. Compatible with Leo's existing code base and

2. Worth doing for reasons not yet made clear to me.

I am willing a long time for a convincing prototype of new positions.

Summary

I see no reason to delay considering brilliant ideas and code ;-)

Otoh, I intend to make no judgement at all about more speculative ideas.  I don't believe I have disparaged any aspect of Vitalije's work.

Edward

[vitalije]

Terry, Reinhard, Offray, Jon many thanks for your support and understanding.

Edward thank you. You didn't have to apologize, but since you did, thanks. Apologize accepted.

I had to focus on something else and I couldn't afford to enter into lengthy discussions, so, I had to ban myself out of this forum for some time.

I am going to respond in separate thread.

Vitalije

[vitalije]

An ancient Greek philosopher Socrates said something like this: "Trying to teach someone faster than he can learn is dangerous and unwise". To be a good teacher one needs to be tremendously patient.

As a teacher I am very happy with the progress we made. About a year ago I have studied Leo’s code base very thoroughly. I guess there are lots of parts of Leo’s code that no one but Edward has seen or read. Theoretically speaking Leo is open source project and everyone is allowed to look at the code and read it. However, it rarely happens that someone takes the time to read such a large code base. By doing this I have learned both about Leo’s good parts and Leo’s weaknesses. About six or seven months ago I have tried to share what I had learned about Leo’s weaknesses. I tried to teach Edward to improve his coding style by avoiding some things that he usually does, things that lead to unreadable and hard to maintain code. But he didn’t accept any of my suggestions at first.

I decided to start a blog and for the first few posts I picked Leo import code to write about. A year ago Edward has put lots of effort in improving this part of code. He wrote about it in this forum; some great collapse of complexity has happened,... But in my opinion code still could be much simpler. When I picked the theme I didn’t know what will I find. What I did find was that Leo’s importer for JavaScript was useless. It produced unusable outline, taking way too much time and causing Leo to hang. However, Edward didn’t accept new coding style at that time and even suggested that it is a bad style. He has fixed JavaScript importer by avoiding section references. That has solved the problem of Leo becoming unresponsive, but the shape of the imported outline was still useless. No JavaScript programmer would want to see his code displayed in such outline. However, Edward, declared that importer is well designed and the result outline is as it should be – end of story. This left me no option but to let it go and wait. After a while, Edward himself, has opened an issue: we need a better importer for JavaScript. When he tried himself to import JavaScript of leo-vue, he realized himself that the present JavaScript importer was less than satisfactory. Little bit later he started to adopt “lispy-style” himself. He still feels a need to put a class somewhere even if it is not required, but he has adopted a lot of the style that I was suggesting.

Function that reads external files using so called fast-read algorithm has been present in Leo since July 2017. I wrote both the read and write function and had put them in leo-snippets repository. What I didn’t do at that time was to test how long would it take to read all the external files. I have made some comparisons on a single file and it was faster than Leo’s original read code, but not so much faster. Perhaps the file I made comparison on was not the best example. It turned out that the overall new read algorithm is more than 27 times faster (*Edward has improved this even more). However, it was not my initial goal. My goal was to make a new data model that would allow other parts of Leo to be much simpler. But, I was so surprised by its speed that I couldn’t resist the temptation to talk about it and somehow the discussion went in the direction that I didn’t want it to go. It became discussion about speed and profiling code and some shortcuts and not about the code structure. Well, the speed is important, but I feel like all these talks about speed caused main point to pass unnoticed.

The one good thing, though, about Edward’s rushing into fast-read brunch is that he himself realized that there are quite a few ancient options in external file format that can and should be deprecated. If I made this suggestion, I guess, Edward’s response would be that Leo must support all those options indefinitely. But when he alone comes to this conclusion, then it is a good idea.

All in all, as a teacher I have tons of reasons to be very satisfied with the progress we made. In time, Edward, will realize himself, other parts that he still misses at the moment and then he will do almost exactly as I wish him to do now.

However, as a developer I am less happy. I feel like my code contribution is not welcomed in this project. I am quite sure that it isn’t true, but still I get this feeling. I tell to myself: Vitalije, if you wanted to have some reward for your contributions then you shouldn’t contribute to the open source project. If you wanted your name to be credited then you should put the code into your own repository under your own license.

I really don’t care about credits, GitHub contribution counter and such. But, OTOH, perhaps some other developers do care about these. Perhaps someone’s career depends on GitHub contribution counter. Since March 2018, when it was declared that only Edward should commit to the master brunch, no contribution is counted for any other developer in leo-editor team. The prototype that I wrote, Edward has pushed to GitHub. Now, as far as GitHub is concerned, the author of the prototype is 100% Edward. Again, I really don’t mind it. I didn’t write it to earn some points or to impress anybody. For me, it is perfectly OK for my contributions to remain counted as Edward’s or someone’s else. But, it can be issue for people who actually need to earn some points to advance in their career. For such people it would be very unappealing to contribute to Leo.

I can see that Edward is trying to attract and encourage more contributors to Leo project. I believe he is doing a very good job in this area. He is eager to accept every contribution but only to the some extent. If the contribution is rather small it is going to be welcomed and accepted without too much noise. However, I’ve got the impression that if the contribution is above some limit, if it makes Leo much better in some area, then it won’t be accepted as such. First, Edward must rewrite it from scratch and only then he will accept it. Well, I still wouldn’t mind. I would be happy if he rewrite and use all of the prototype I made. What bothers me is that if he can’t fully understand some part of the contribution, he won’t ask for the explanation. Instead he will usually put his boss hat on and then just declare that the contribution is unacceptable and that the present code must remain as it is – end of story. I don’t think Edward is doing this on purpose. Most likely he is doing it subconsciously. One way or the other, that is something that really bothers me, and I believe it makes awful and non-inspiring atmosphere. I guess not many developers would be ready to accept that kind of treatment. I am writing all these with the hope that Edward will change this bad habits. I am also writing this to set an example for other developers. If you ever find yourself in this situation, be patient and keep contributing and explaining your contribution. Eventually all your contributions will made to the Leo’s core.

It would be much nicer though if Edward were willing to accept all contributions even when he can’t fully grasp them at once, or if at least he were willing to ask for the explanation instead of just dismissing the propositions he can’t understand at the moment.

I really hope that I didn’t offend Edward (or anyone else) by writing this. That certainly was not my intention. I just wish a better atmosphere for all contributors. Many would agree that this forum has been a source of inspiration for many people for a long time. Let it be even better for the generations to come.

Vitalije

[rengel]

What an eloquent heart-breaking account.

A well-meaning soul, hurt but stiil not (yet) giving up.

Reading it, just makes me sad.

It will never be known how many just walked away silently...

Reinhard

[jkn]

Vitalije has replied with great thoughtfullness and dignity. It's a pity that your contributions seem consistently at the trollish end to me.

[rengel]

On Tuesday, June 12, 2018 at 9:04:18 AM UTC+2, jkn wrote:

Vitalije has replied with great thoughtfullness and dignity. It's a pity that your contributions seem consistently at the trollish end to me.

Sorry, you see it that way. I don't want to argue with you, but these have been my spontaneous feelings. I just wonder if you are not over-generalzing my two posts in this thread a little bit? I've been for years on this forum and tried to contribute in various ways, some tips, some hints, some suggestions, some plain remarks, and, yes, some criticisms.

And yes, I totally agree with you, that Vitalije has replied with great thoughtfullness and dignity.

Just what he said made me sad.

Reinhard

[Marcel Franke]

Am Montag, 11. Juni 2018 15:48:00 UTC+2 schrieb vitalije:

 Since March 2018, when it was declared that only Edward should commit to the master brunch, no contribution is counted for any other developer in leo-editor team.

This is not true. The individual commits of every developer are still existing in the history and count for git-points. A commit to a branch does not rewrite previous commits, does not change names and contribution-history. Though, this is of course only valid if the developer actually commited himself to git(hub), and did not just send edward a patch of code to let him commit it. But every time you commit something to github and merge it to some branch, your name will appear in that branch. If someone then merge this branch into some other branch, your name still remains as it was before.

In git, a branch is basically a collection of commits, similar to an outline. A new branch is just a clone of a previous branch/outline with a new commit/root-element. Thus, everytime Edward merges the develop-branch to master, he only clones references of your new commits from develop to master, and adds a new commit of his own about those merge. At least that's the normal way this things go. There are ways to manipulate commit-history, but I don't think Edward is willing using such black magic.

 He is eager to accept every contribution but only to the some extent. If the contribution is rather small it is going to be welcomed and accepted without too much noise. However, I’ve got the impression that if the contribution is above some limit, if it makes Leo much better in some area, then it won’t be accepted as such.

To be fair, this is exactly the job of the project-manager. He must ensure that the code works and everthing follows whatever goal for the project exists. It actually is his job to decline everything that is to complex to be understandable on a first peek, because it can break the software, or introduce nasty bugs or even misusage. I know for the eager developer this can be hard, and I too think Edward has much to learn. But I think the biggest problem is not so much the project itself, but the codebase and general culture. From my impression, the Leo Editor is a case of a Cathedral instead of an open Bazaar (see https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar). On surface it's open source, but the source if unneccessary complicated and unhealthy thighly integrated, while still controlled by one single instance of authority. Access is hard, relevant contribution even harder. This has advantages, but also disadvantages. It's everyones choice which one you prefer.

 I guess not many developers would be ready to accept that kind of treatment.

Not really. A good developer should accept that the project has priority and that his own work has shortcomings too. If your code is rejected, then you should figure out why. Maybe it's the explenation, or the lack of test, or your code has some hidden sideeffects, or really is just bad. Team-Projects are less about code, and more about people and their characters and experiences. You need to learn to understand each other to create good results. Working against each other is not helpful. Though, I admit Edward isn't the easiest pal in that regard. And the horribled close design of Leos architecture isn't very helpful to loosen it.

 It would be much nicer though if Edward were willing to accept all contributions even when he can’t fully grasp them at once

There is a simply solution: if something is complex, make so many unit- and system-tests that they can cover the parts which are hard to understand. For a reviewer to do good work, he MUST understand the code completly, or have someone or something he can trust enough to accept their judgement.

[Edward K. Ream]

On Mon, Jun 11, 2018 at 8:48 AM, vitalije <vita...@gmail.com> wrote:

An ancient Greek philosopher Socrates said something like this: "Trying to teach someone faster than he can learn is dangerous and unwise". To be a good teacher one needs to be tremendously patient.

​Patience is always a good idea ;-)​

As a teacher I am very happy with the progress we made. About a year ago I have studied Leo’s code base very thoroughly. I guess there are lots of parts of Leo’s code that no one but Edward has seen or read. Theoretically speaking Leo is open source project and everyone is allowed to look at the code and read it. However, it rarely happens that someone takes the time to read such a large code base. By doing this I have learned both about Leo’s good parts and Leo’s weaknesses. About six or seven months ago I have tried to share what I had learned about Leo’s weaknesses. I tried to teach Edward to improve his coding style by avoiding some things that he usually does, things that lead to unreadable and hard to maintain code. But he didn’t accept any of my suggestions at first​.

​

Since we are quoting the ancients, ​Aristotle said, "That which we learn we learn by doing".  That's especially true for me.  Pictures also help...

​

I decided to start a blog and for the first few posts I picked Leo import code to write about. A year ago Edward has put lots of effort in improving this part of code. He wrote about it in this forum; some great collapse of complexity has happened,... But in my opinion code still could be much simpler.

​​

 

​Yeah, code can always be improved.  Simplicity is important, and so is code style.  But design is much more important, and "vision" is most important of all.​

When I picked the theme I didn’t know what will I find. What I did find was that Leo’s importer for JavaScript was useless. It produced unusable outline, taking way too much time and causing Leo to hang. However, Edward didn’t accept new coding style at that time and even suggested that it is a bad style. He has fixed JavaScript importer by avoiding section references. That has solved the problem of Leo becoming unresponsive, but the shape of the imported outline was still useless. No JavaScript programmer would want to see his code displayed in such outline. However, Edward, declared that importer is well designed and the result outline is as it should be – end of story. This left me no option but to let it go and wait. After a while, Edward himself, has opened an issue: we need a better importer for JavaScript. When he tried himself to import JavaScript of leo-vue, he realized himself that the present JavaScript importer was less than satisfactory. Little bit later he started to adopt “lispy-style” himself. He still feels a need to put a class somewhere even if it is not required, but he has adopted a lot of the style that I was suggesting​.

​The new code base for importers is miles ahead of the old.  But the javascript importer needs something besides better code.  It needs real knowledge of javascript.  I would be happy to have an improved javascript importer, even if it used none of the common importer code base.​

​

Function that reads external files using so called fast-read algorithm has been present in Leo since July 2017. I wrote both the read and write function and had put them in leo-snippets repository. What I didn’t do at that time was to test how long would it take to read all the external files. I have made some comparisons on a single file and it was faster than Leo’s original read code, but not so much faster. Perhaps the file I made comparison on was not the best example. It turned out that the overall new read algorithm is more than 27 times faster (*Edward has improved this even more). However, it was not my initial goal.

​I didn't know (or remember) any of this until last week.  And until your email of yesterday, I didn't know that speed was not your initial/primary goal.​

My goal was to make a new data model that would allow other parts of Leo to be much simpler. But, I was so surprised by its speed that I couldn’t resist the temptation to talk about it and somehow the discussion went in the direction that I didn’t want it to go. It became discussion about speed and profiling code and some shortcuts and not about the code structure. Well, the speed is important, but I feel like all these talks about speed caused main point to pass unnoticed​.

​Yesterday's email provided a picture that will stick, namely that Leo's DAG could be called a database without an index.

​

The one good thing, though, about Edward’s rushing into fast-read brunch is that he himself realized that there are quite a few ancient options in external file format that can and should be deprecated. If I made this suggestion, I guess, Edward’s response would be that Leo must support all those options indefinitely. But when he alone comes to this conclusion, then it is a good idea.

​Your code tipped the balance.  My own code is impossible, now, for me to understand​.  In the process of modifying (editing) your code I saw what is essential.  I also ran across comments saying the old file formats were already not being supported.

All in all, as a teacher I have tons of reasons to be very satisfied with the progress we made. In time, Edward, will realize himself, other parts that he still misses at the moment and then he will do almost exactly as I wish him to do now.

​As a student and project manager, I also am thrilled by our collaboration.  ​

However, as a developer I am less happy. I feel like my code contribution is not welcomed in this project. I am quite sure that it isn’t true, but still I get this feeling. I tell to myself: Vitalije, if you wanted to have some reward for your contributions then you shouldn’t contribute to the open source project. If you wanted your name to be credited then you should put the code into your own repository under your own license.

​I think there is a simple procedural remedy for this ​dissatisfaction, namely doing your prototypes in git branches, or even in new leo repos. You can create a rep on this page. I see there is a leo-model repo, but apparently I created it.  It was created 11 days ago, but I had already forgotten all about it.

I really don’t care about credits, GitHub contribution counter and such. But, OTOH, perhaps some other developers do care about these. Perhaps someone’s career depends on GitHub contribution counter. Since March 2018, when it was declared that only Edward should commit to the master brunch, no contribution is counted for any other developer in leo-editor team.

​This page shows the commits from all contributors.​ You're on the list.

In your case, though, your contributions go far beyond commits.

The prototype that I wrote, Edward has pushed to GitHub. Now, as far as GitHub is concerned, the author of the prototype is 100% Edward. Again, I really don’t mind it. I didn’t write it to earn some points or to impress anybody. For me, it is perfectly OK for my contributions to remain counted as Edward’s or someone’s else. But, it can be issue for people who actually need to earn some points to advance in their career. For such people it would be very unappealing to contribute to Leo.

​I would be happy to delete the leo-model repo.  It would be best if you created your own repo.  If you want to get credit for your work, then you must publish it.

I can see that Edward is trying to attract and encourage more contributors to Leo project. I believe he is doing a very good job in this area. He is eager to accept every contribution but only to the some extent. If the contribution is rather small it is going to be welcomed and accepted without too much noise. However, I’ve got the impression that if the contribution is above some limit, if it makes Leo much better in some area, then it won’t be accepted as such. First, Edward must rewrite it from scratch and only then he will accept it.

​As project manager, I have to ensure that the new code is sound.  I have enough experience with Terry's work that I am happy to trust what he has done.

For grand changes, I first understand what the change is supposed to do.  The "picture" of a db index is a start, but it's not nearly sufficient to allow me to evaluate the merits of a new data model.

Well, I still wouldn’t mind. I would be happy if he rewrite and use all of the prototype I made. What bothers me is that if he can’t fully understand some part of the contribution, he won’t ask for the explanation. Instead he will usually put his boss hat on and then just declare that the contribution is unacceptable and that the present code must remain as it is – end of story.

I don’t think Edward is doing this on purpose. Most likely he is doing it subconsciously. One way or the other, that is something that really bothers me, and I believe it makes awful and non-inspiring atmosphere. I guess not many developers would be ready to accept that kind of treatment. I am writing all these with the hope that Edward will change this bad habits. I am also writing this to set an example for other developers. If you ever find yourself in this situation, be patient and keep contributing and explaining your contribution. Eventually all your contributions will made to the Leo’s core.

​​It's up to me, as the "boss" to preserve Leo's integrity.

It's up to you, as a contributor, to convince me that a contribution is worthwhile.  This means, at a minimum:

a) I understand the general idea of the contribution and

b) That I have a git branch to evaluate. 

At present, the new data model does neither.  I am not rejecting the idea, but more work is needed.

It would be much nicer though if Edward were willing to accept all contributions even when he can’t fully grasp them at once, or if at least he were willing to ask for the explanation instead of just dismissing the propositions he can’t understand at the moment​.

​I will, quite naturally, reject ideas that I don't understand. It's your job to convince me.

Please do not take "I'm not convinced" as a rejection.  It is not intended that way.​ It's supposed to be a request for clarification.

​

I really hope that I didn’t offend Edward (or anyone else) by writing this. That certainly was not my intention. I just wish a better atmosphere for all contributors. Many would agree that this forum has been a source of inspiration for many people for a long time. Let it be even better for the generations to come.

​It's good that you expressed your frustrations.  This letter will lead to better collaboration.

Summary

1. If you want full credit for your work, do your work in a git repo or git branch.

2. The more important a proposed contribution, the more important it is for you to explain it and to provide a git branch that can be evaluated.

3. When I say "I don't understand", I am talking about my own understanding.  Do not interpret those words as a judgement about the technical merits of a proposal.

Edward

[Edward K. Ream]

On Tue, Jun 12, 2018 at 4:18 AM, 'Marcel Franke' wrote:

​

On surface

​[Leo is] open source, but the source if unnecessarily complicated and unhealthy highly integrated, while still controlled by one single instance of authority. Access is hard, relevant contribution even harder.

Terry, Vitalije, Ville Vainio, Bernhard Mulder, Kent Tenney, Marc-Antoine Parent, Paul Paterson and others have all made significant contributions to Leo. By any reasonable measure, Leo is open software.

Leo's source code is difficult in places because it's always dealing with the DAG. That can't be helped. Leo devs are special people.

The alternative isn't utopian simplicity, it's org mode.  There, everything is a string.  This can work: org mode simulates Leo's structures using filters.  It's a clever idea, but Leo's world is much richer.

Edward

[vitalije]

a) I understand the general idea of the contribution and

b) That I have a git branch to evaluate.  

Both these items were on my todo list. I have planed to share code perhaps in some branch or in its own repository, and surely I planed to explain the model in full detail once it has been settled down. I just haven't finished doing my todo list. As I said before, my primary mistake was not resisting the temptation to write about code that is not ready for showing, not documented, not even stabilized. Once I wrote about it, it somehow get out of my control and I couldn't do anything about it. 

As I said before, I am not really after credit for code. I have already too much publicity for my taste, so please do not delete your repo. I will push there once I am ready to show and discuss code.

TO: Marcel Franke, you made very good points and I totally agree with you. If git counts do not bother anyone else, they certainly won't bother me. But just FYI here is the explanation from github how they count contributions.

Vitalije

[Marcel Franke]

Am Dienstag, 12. Juni 2018 12:50:00 UTC+2 schrieb vitalije:

> But just FYI here is the explanation from github how they count contributions.

Ok, I see where you are coming from. Did you confirm that commits to develop are not appearing in your contribution-list after devel was merged to master? I see some contributions from the last weeks for some people at leo, including you, but I can only assume that they were not originally made to master and where just merged from devel to master. That explanation you linked is not really clear whether contributions are measured by commits available in default-branch, or actually commited directly to default-branch.

[Marcel Franke]

Am Dienstag, 12. Juni 2018 12:43:11 UTC+2 schrieb Edward K. Ream:

 Terry, Vitalije, Ville Vainio, Bernhard Mulder, Kent Tenney, Marc-Antoine Parent, Paul Paterson and others have all made significant contributions to Leo. By any reasonable measure, Leo is open software.

It's not whether people contribute, but how it happens. I think my usage of the term open source is a bit misleading for you. I do not mean open source in the sense the code is open available or under a open source license, but in the sense how accessable the code is to other people and how easy it is for other people to contribute to the project or even just for themself. Open source is also a mindset of how easily you can customize code for your own means.

 Leo's source code is difficult in places because it's always dealing with the DAG. That can't be helped. Leo devs are special people.

And I say Leo's architecture could be way more simpler and more modular, allowing people to more easily contribute with less control from anyone. There is only a minimal neccessary core for what leo is doing, the rest is just bloat for comfort, though leo is not designed that way. Or more exactly, it did not grow that way, after all this is the original problem here. Historical grown design. Today everyone here want's to do way more with leo than you originally planned for it of what the architecture can easily deliver.

 The alternative isn't utopian simplicity, it's org mode.

No org-mode is a complete different project that just happens to overlap in functionality. Funny enough, in it's own way it's also a case of historical grown design. Microsoft Access, Jupiter Notebook or VS Code would be more approbiated as a comparation for what leo is and what people like to use it for.

 There, everything is a string.

A leo-file is also just a string of xml. And Leo Editor is a tool working on top of it and adding a higher level of complexity. Org-mode is also primary a collection of tools to work with org-files, but also on top of org-files. And There is nothing preventing people to do the same with org-files as people here are doing with leo-files. Actually there are projects doing similar things, but it's not a main focus, after all in generally there are different goals in that culture.

[Edward K. Ream]

​On Tue, Jun 12, 2018 at 7:20 AM, 'Marcel Franke' via leo-editor <leo-e...@googlegroups.com> wrote:

 Leo's source code is difficult in places because it's always dealing with the DAG. That can't be helped. Leo devs are special people.

And I say Leo's architecture could be way more simpler and more modular, allowing people to more easily contribute with less control from anyone. There is only a minimal neccessary core for what leo is doing, the rest is just bloat for comfort, though leo is not designed that way. Or more exactly, it did not grow that way, after all this is the original problem here. Historical grown design. Today everyone here want's to do way more with leo than you originally planned for it of what the architecture can easily deliver.

​No way is this even remotely true.  Leo's classes are mostly what they were 20 years ago. To my knowledge, there is no significant interactions between classes or modules. That's what modularity means. This is what allows me, say, to contemplate rewriting Leo's undo code.

This kind of general criticism, without any grounding in fact, is disrespectful and unhelpful.  If you have specific suggestions to make, then make them.

Edward

[Terry Brown]

On Tue, 12 Jun 2018 09:30:51 -0500
"Edward K. Ream" <edre...@gmail.com> wrote:

> > And I say Leo's architecture could be way more simpler and more
> > modular, allowing people to more easily contribute with less
> > control from anyone. There is only a minimal neccessary core for
> > what leo is doing, the rest is just bloat for comfort, though leo
> > is not designed that way. Or more exactly, it did not grow that
> > way, after all this is the original problem here. Historical grown
> > design. Today everyone here want's to do way more with leo than you
> > originally planned for it of what the architecture can easily
> > deliver.
> >
> ​No way is this even remotely true.  Leo's classes are mostly what
> they were 20 years ago. To my knowledge, there is no significant
> interactions between classes or modules. That's what modularity
> means. This is what allows me, say, to contemplate rewriting Leo's
> undo code.
>
> This kind of general criticism, without any grounding in fact, is
> disrespectful and unhelpful.  If you have specific suggestions to
> make, then make them.

I think the existence of this problem is perhaps a matter of
perspective or perception. It may not be specifically an interaction
between classes or modules, but to me there does seem to be a high
level of interconnectedness in Leo which can make development
challenging.

For example Leo has a text editor component that has some really cool
features, like abbreviation expansion and general integration with Leo
such that keystrokes can invoke Python scripts etc. But, and this has
been the challenge I've run into in the Leo edit pane project,
Leo's text editor component (I believe, this is were the issue of
perception comes in) edits not some abstract piece of text, but the
text at c.p.b. Which makes it hard to use it in pop up reference
sticky notes or extra editor windows that don't change the current tree
selection.

There are other examples - the way the logging subcomponent is tied in
with start up code (because of the need to accept log messages before
there's a place to put them) and managing other widgets like Find and
Spell tabs - this is an issue I still need to work on in the dock
layout project. The find functionality is another one, several
different actions on found nodes are possible by passing in
instructions to the find machinery, but it would be more flexible the
machinery to just report what was found and have the actions on that
collection handled separately. And of course it's easy to complain
about these things :-} when you're not actually dealing with the
complexities involved, like implementing find-next given the
possibility that the outline changed since the last find.

I don't think these scleroses, as Edward has perhaps termed them in the
past, are fatal flaws, but I do think the impede development and the
future of Leo, and require major work to remove. Which is one of the
reasons I'd like to have seen Vitalije's prototype demonstration as a
whole rather than piecemeal - the interconnection between vnodes and
commanders (needing c to instantiate v) may be one of the less
intrusive, but is still jarring design wise and perhaps an impediment
to alternate back ends / interfaces.

Cheers -Terry

[Edward K. Ream]

​Yes.  You and I agree about this. ​But many people, including you, have overcome these difficulties and contributed code.  Leo's code base is huge, and parts of it are complex. That does not prove that the code could is defective in any significant way.

For example Leo has a text editor component that has some really cool
features, like abbreviation expansion and general integration with Leo
such that keystrokes can invoke Python scripts etc.  But, and this has
been the challenge I've run into in the

​​

Leo edit pane project,
Leo's text editor component (I believe, this is were the issue of
perception comes in) edits not some abstract piece of text, but the
text at c.p.b.  Which makes it hard to use it in pop up reference
sticky notes or extra editor windows that don't change the current tree
selection.

​Yes.  Furthermore, there are built-in "connections" between the outline an body panes.​

There are other examples - the way the logging subcomponent is tied in
with start up code (because of the need to accept log messages before
there's a place to put them) and managing other widgets like Find and
Spell tabs - this is an issue I still need to work on in the dock
layout project.  The find functionality is another one, several
different actions on found nodes are possible by passing in
instructions to the find machinery, but it would be more flexible the
machinery to just report what was found and have the actions on that
collection handled separately.  And of course it's easy to complain
about these things :-} when you're not actually dealing with the
complexities involved, like implementing find-next given the
possibility that the outline changed since the last find.

​Exactly.  We want the find-next/find-prev commands to be "stateless", but that's really tricky when the outline itself changes.  Maybe there is a brilliant way around this problem, but nobody has suggested one yet.

This is an example of huge complexities arising from creating the illusion of simplicity for the user. That was the big Aha of MacOS.  I have little patience with would-be devs who don't understand this basic fact of life.

This does not give me a free pass to do stupid things.  Conversely, not every complication is evidence of stupidity.

I don't think these scleroses, as Edward has perhaps termed them in the
past, are fatal flaws, but I do think the impede development and the
future of Leo, and require major work to remove.  Which is one of the
reasons I'd like to have seen Vitalije's prototype demonstration as a
whole rather than piecemeal - the interconnection between vnodes and
commanders (needing c to instantiate v) may be one of the less
intrusive, but is still jarring design wise and perhaps an impediment
to alternate back ends / interfaces.

​I agree with all of this.

Leo's code base isn't ever going to be perfect. One reason is that there are much more interesting things to do.  Like adding indices to Leo's DAG, and like supporting client-server architectures in Leo.

BTW, I have been meaning to ask.  Somewhere, don't remember now where, you seemed to imply that you needed my approval for the

Leo edit pane project.  Imo, you don't.  Of course, we'll all evaluate it when you think it's ready...

Edward