How to collaborate using Leo

[Edward K. Ream]

Our recent collaborative discussions have created a stunning new way to collaborate using Leo.

Suppose a group contains L Leo users (Leonard, Lester, ...) and N non-Leo users (Nancy, Nadine,...).

We may as well assume L > 1 and N > 1.

Kent and Terry have (finally!) convinced me that that Nancy a) must never see sentinels and b) must never pay any substantial price for not seeing them.  Furthermore, only one .leo folder can exist anywhere on Nancy's machine (and that should be in an innocuous place).  Let's call this folder the **cache**.  For now, we care nothing about how it works.

Zoom.Quiet provided an essential hint: use bzr/git hooks.  Earlier schemes for using such hooks failed because there were **pairs** of files floating around: with sentinels and without.  This creates an impossible-to-solve problem: namely ensuring that pairs were always in synch.

Standing in the shower after my swim this afternoon I saw the solution.

Suppose the "transmission line" (pushes and pulls) only carries files *with* sentinels.  This ensures lossless transmission of data and sidesteps the intractable synchronization problem.

The Leo project must provide *two kinds* of bzr hooks, one for Nancy and one for Leonard.

When Nancy does a bzr pull, **her pull hook will strip Leo sentinels**. My first thought was OMG: it can't be that easy!

Well, not quite.  Nancy can change a file and then push it.  How does Nancy's *push* hook recreate the file with sentinels??  Think about it...

Answer: Nancy's push hook uses the @shadow algorithm!!!  Nancy's *pull* hook will save the file *with* sentinels in the cache.  When Nancy pushes a file (without sentinels), the push hook will use the @shadow algorithm to create the file (with sentinels) that actually gets transmitted over the wire.

This will work!!!

Leonard needs neither a push hook nor a pull hook, but a pull hook that simulated @auto for files that *never* had sentinels might be useful.

Let's verify that Nancy will be (fairly) happy:

- Installation is low cost: she just installs Leo's push/pull hooks.

- Nancy will never, ever, will see a Leo sentinel.

- Nancy will see *slightly* slower pushes and pulls than will Leonard, but probably not slow enough to be noticeable.  In particular, paired files never get transmitted--only the (recreated) file *with* sentinels goes over the wire.

This scheme completely appears to solve the technical issues!

Let's consider management (team) issues.  They will typically depend on the ratio L/N.

- If L or N is zero, there is no problem ;-)

If L is one, this scheme has no advantages over @shadow: Leonard is not sharing Leo data with anyone!

If N is one, Nancy can probably be persuaded to accept *slightly* lower performance in order to please the group.

Otherwise, negotiations (or an edict from management) will decide whether this scheme will be acceptable to Nancy, Nadine, et. al.  If it isn't, Leonard, Lester et. al. will have to use @auto or @shadow.

This may be a perfect technical solution; team-related considerations might kill it.  Nevertheless, it is *far* better than anything I ever imagined before. 

This Aha would not have happened without Zoom.Quiet's questions and comments about collaboration. The Aha is a direct response to Terry's post that *finally* convinced me that is, um, unreasonable to blame people for not accepting Leo sentinels ;-)  It's a good thing Kent and Terry were around to stop me from making an ass of myself. This has been a stellar collaboration.  Thank you all!

Your comments please.

Edward

P.S.  No, this won't delay b1 further :-)

EKR

[gatesphere]

Ooh! This I like.

If you want to draw more help, though, you'll have to generalize from
'bzr hook' to 'version-control-system-of-choice hook'. Whether this
entails a single universal python script for all vcs', or tailored
approaches for each system remains to be seen.

I postulate that for this scheme to become successful, the following
version control systems need to be supported (at least!):
- bzr
- git
- hg (mercurial)
- cvs
- svn

-->Jake

[Edward K. Ream]

On Thu, Oct 24, 2013 at 5:44 PM, gatesphere <gates...@gmail.com> wrote:

 

Ooh!  This I like.

If you want to draw more help, though, you'll have to generalize from 'bzr hook' to 'version-control-system-of-choice hook'.

​Right.  I used bzr just as an example to simplify the presentation.​

 
 

 Whether this entails a single universal python script for all vcs', or tailored approaches for each system remains to be seen.

The latter, unless all hooks can be written in Python.
​

 

I postulate that for this scheme to become successful, the following version control systems need to be supported (at least!):
  - bzr
  - git
  - hg (mercurial)
  - cvs
  - svn

​I might drop the last two, but that's a quibble.  What matters is getting *one* to work.  Bzr is the obvious first choice. It will serve as a proof of concept applicable to all others.

EKR

[Kent Tenney]

Nice!

Can this machinery go in core? an opportunity to
demonstrate the power of the Leo framework.?

If this moves forward you might want to write it up as a story:
'How I stopped worrying and made peace with the sentinelphobes'
then submit the story to various tech magazines and online venues.

An entertaining history of Leo, your frustrations with mindshare, how
you solved it via the brilliance of your creation ...

>Bzr is the obvious first choice.

Not obvious to me, of the dozens of repos I've cloned, I think
Leo is the only bzr one: 90% git 9% mercurial

> --
> You received this message because you are subscribed to the Google Groups
> "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leo-editor+...@googlegroups.com.
> To post to this group, send email to leo-e...@googlegroups.com.
> Visit this group at http://groups.google.com/group/leo-editor.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jacob Peck]

On 10/25/2013 9:33 AM, Kent Tenney wrote:
>> >Bzr is the obvious first choice.
> Not obvious to me, of the dozens of repos I've cloned, I think
> Leo is the only bzr one: 90% git 9% mercurial

Ditto, but it's probably obvious as Leo is on bzr.

-->Jake

[Edward K. Ream]

On Fri, Oct 25, 2013 at 8:33 AM, Kent Tenney <kte...@gmail.com> wrote:

Nice!

​Thanks.  This is the best Aha I've had in a long time.​

 

Can this machinery go in core? an opportunity to
demonstrate the power of the Leo framework.?

​That's a packaging issue.  leoShadow.py is the obvious place, but it might be more convenient to create stand-alone hooks.​

 

If this moves forward you might want to write it up as a story:
'How I stopped worrying and made peace with the sentinelphobes'
then submit the story to various tech magazines and online venues.

​Haha.  It might make a good vid too.​

 

>Bzr is the obvious first choice.
Not obvious to me, of the dozens of repos I've cloned, I think
Leo is the only bzr one: 90% git 9% mercurial

​I meant, obvious for a prototype, because bzr is Python.

One moment pls... From:
http://git-scm.com/book/en/Customizing-Git-Git-Hooks
"any properly named executable scripts will work fine—you can write them in Ruby or Python or what have you."

So yes, git hooks in Python are possible.  Presumably, git hooks will be a bit different from bzr hooks, even if the underlying logic is the same.

EKR

[Zoom.Quiet]

Huuuu?! in my eyes, this flow just clone from git work flow?

- cleanness files ~= work space
- cache ~=stage index
- shadow ~= local repo.
- .leo ~= remote repo.

with local DVCS's hook:
- Nancy always face cleanness files, but when commit
- Hooks generate sentinels file base @shadow algorithm
- Leonard always face Leo, but when commit
- Hooks generate sentinels file base @shadow algorithm

Huuu?
- but not neet SCCS the realy @shadow files ?


2013/10/25 Edward K. Ream <edre...@gmail.com>:

btw:
人生苦短, Pythonic! 冗余不做,日子甭过!备份不做,十恶不赦!

means:

life is short,
Pythonic!
No Redundancy,
No Life!
No Backup,
Is Heinous!


--
人生苦短, Pythonic! 冗余不做,日子甭过!备份不做,十恶不赦!
KM keep growing environment culture which promoting organization be learnning!
俺: http://about.me/zoom.quiet
许: http://creativecommons.org/licenses/by-sa/2.5/cn/

[Edward K. Ream]

On Fri, Oct 25, 2013 at 12:09 PM, Zoom.Quiet <zoom....@gmail.com> wrote:

Huuuu?! in my eyes, this flow just clone from git work flow?

- cleanness files ~= work space
- cache ~=stage index
- shadow ~= local repo.
- .leo ~= remote repo.

​I don't know enough about git to comment.  Anyone else want to try?​

 

with local DVCS's hook:
- Nancy always face cleanness files, but when commit
    - Hooks generate sentinels file base @shadow algorithm

​That was the idea, yes.
​

 

- Leonard always face Leo, but when commit
    - Hooks generate sentinels file base @shadow algorithm

​No.  Leonard uses Leo, so his files already have sentinels.  In fact, the sentinels in *his* files are the sentinels that *Nancy's* push hook must have in order to recreate sentinels.

​Edward​

P.S.  I may not be expecting a call from Sweden, but this is the best idea I've had in a long time.

EKR

[Zoom.Quiet]

2013/10/26 Edward K. Ream <edre...@gmail.com>:

> On Fri, Oct 25, 2013 at 12:09 PM, Zoom.Quiet <zoom....@gmail.com> wrote:
>>
>> Huuuu?! in my eyes, this flow just clone from git work flow?
>>
>> - cleanness files ~= work space
>> - cache ~=stage index
>> - shadow ~= local repo.
>> - .leo ~= remote repo.
>
> I don't know enough about git to comment. Anyone else want to try?
>

- Huuu, pls. ignore it...
- though one night, i get new understand:

Leonard always face Leo:
- whatever usage @file/@shadow
- shring SCCS forever is sentinels file
- and begin working ,must:
- bzr/hg/git pull and merge local sentinels file
- and usage read @file, merge into Leo node tree
- HOOKs for Leonard just fixed:
- base clean file or shadow, gen. sentinels file
- and the reverse

Nancy alwasy face clean files:
- but in fact shring SCCS forever is sentinels file
- and begin working ,must:
- bzr/hg/git pull and merge local sentinels file
- and the HOOKs for Nancy, base sentinels file through @shadow
algorithm gen. clean files
- HOOKs for Nancy just fixed:
- usage @shadow algorithm base the cached sentinels file
- merge local clean files gen. new sentinels file

so the local cache is the key problem:
- Invisible for Nancy
- Available for SCCS
- Natural fro Leo

Mnnn... just usage .leo dir?

[Edward K. Ream]

​​

On Sun, Oct 27, 2013 at 8:57 AM, Zoom.Quiet <zoom....@gmail.com> wrote:

 

- though one night, i get new understand:

Leonard always face Leo:
    - whatever usage @file/@shadow

​Leonard should *always* use @file, to provide sentinels for Nancy.
​

 

    - sh

​a​

ring SCCS forever is sentinels file

​Correct.  This is really the key idea.
​

 

    - and begin working ,must:
        - bzr/hg/git pull and merge local sentinels file
        - and usage read @file, merge into Leo node tree
    - HOOKs for Leonard just fixed:
        - base clean file or shadow, gen. sentinels file
        - and the reverse

​It's better to think that Leonard needs no hooks.​

 

Nancy alwa

​ys​

face clean files:

​Correct!
​

 

    - but in fact sh

​a​

ring SCCS forever is sentinels file

​Correct.
​

 

    - and begin working ,must:
        - bzr/hg/git pull and merge local sentinels file
        - and the HOOKs for Nancy, base sentinels file through @shadow
algorithm gen. clean files

​The pull hook puts sentinels to the cache and strips sentinels for Nancy.
​

 

    - HOOKs for Nancy just fixed:
        - usage @shadow algorithm base the cached sentinels file
        - merge local clean files gen. new sentinels file

so the local cache is the key problem:

​No.  The local cache is just a detail :-)  It merely needs to remember the files *with* sentinels **on Nancy's local machine.**  SCCS repository *never* sees the cache.  Only Nancy's push/pull hooks see the cache.

The push hook uses the @shadow algorithm to push files *with* sentinels.  That's *all* that the cache is used for.
​

 

- Invisible for Nancy
- Available for SCCS
- Natural fro Leo

​Yes, the cache is invisible to Nancy.

Only the pull hook writes to the cache.
Only the push hook reads from the cache.​

 

Mnnn... just usage .leo dir?

​​

​Correct.  The cache will be in Nancy's .leo directory.

Edward

[Edward K. Ream]

On Sunday, October 27, 2013 1:23:21 PM UTC-5, Edward K. Ream wrote:

> Leonard should *always* use @file, to provide sentinels for Nancy.

More importantly, Leonard should use @file to provide sentinels for Lucinda, Larry, etc.

EKR

[Zoom.Quiet]

2013/10/28 Edward K. Ream <edre...@gmail.com>:

> On Sun, Oct 27, 2013 at 8:57 AM, Zoom.Quiet <zoom....@gmail.com> wrote:
>
>>
>> - though one night, i get new understand:
>>
>> Leonard always face Leo:
>> - whatever usage @file/@shadow
>
>

> Leonard should *always* use @file, to provide sentinels for Nancy.
>
>>

>> - sh
>> a
>> ring SCCS forever is sentinels file
>
>
> Correct. This is really the key idea.
>
>>
>> - and begin working ,must:
>> - bzr/hg/git pull and merge local sentinels file
>> - and usage read @file, merge into Leo node tree
>> - HOOKs for Leonard just fixed:
>> - base clean file or shadow, gen. sentinels file
>> - and the reverse
>
>
> It's better to think that Leonard needs no hooks.
>

- Big Love this rule ;-)

>>
>>
>> Nancy alwa
>> ys
>> face clean files:
>
>
> Correct!
>
>>
>> - but in fact sh
>> a
>> ring SCCS forever is sentinels file
>
>
> Correct.
>
>>
>> - and begin working ,must:
>> - bzr/hg/git pull and merge local sentinels file
>> - and the HOOKs for Nancy, base sentinels file through @shadow
>> algorithm gen. clean files
>
>
> The pull hook puts sentinels to the cache and strips sentinels for Nancy.
>
>>
>> - HOOKs for Nancy just fixed:
>> - usage @shadow algorithm base the cached sentinels file
>> - merge local clean files gen. new sentinels file
>>
>> so the local cache is the key problem:
>
>
> No. The local cache is just a detail :-) It merely needs to remember the
> files *with* sentinels **on Nancy's local machine.** SCCS repository
> *never* sees the cache. Only Nancy's push/pull hooks see the cache.
>

- the ~/.leo dir? sure, is the right space!
- but, the @shadow algorithm is usage between who?
- i guess just only can base the local user file and the lastest
verion in SCCS ?
- so this ways is perfect for DVCS but svn/csv/vss etc. Center VCS is not good.

ok all clean!
- wait the brz first show us the prototype hooks
- just need know how call the @shadow algorithm from Leo
- in fact, this @shadow algorithm is very useful for other action,
etc. co-writing system

> The push hook uses the @shadow algorithm to push files *with* sentinels.
> That's *all* that the cache is used for.
>
>>
>> - Invisible for Nancy
>> - Available for SCCS
>> - Natural fro Leo
>
>
> Yes, the cache is invisible to Nancy.
> Only the pull hook writes to the cache.
> Only the push hook reads from the cache.
>
>>
>>
>> Mnnn... just usage .leo dir?
>
> Correct. The cache will be in Nancy's .leo directory.
>
> Edward
>

> --
> You received this message because you are subscribed to the Google Groups
> "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leo-editor+...@googlegroups.com.
> To post to this group, send email to leo-e...@googlegroups.com.
> Visit this group at http://groups.google.com/group/leo-editor.
> For more options, visit https://groups.google.com/groups/opt_out.

[Edward K. Ream]

On Sun, Oct 27, 2013 at 9:01 PM, Zoom.Quiet <zoom....@gmail.com> wrote:

2013/10/28 Edward K. Ream <edre...@gmail.com>:

 

> No.  The local cache is just a detail :-)  It merely needs to remember the
> files *with* sentinels **on Nancy's local machine.**  SCCS repository
> *never* sees the cache.  Only Nancy's push/pull hooks see the cache.
>

- the ~/.leo dir? sure, is the right space!
- but, the @shadow algorithm is usage between who?
- i guess just only can base the local user file and the lastest
ver

​s​

ion in SCCS ?

​The ​@shadow algorithm is applied just before pushing a new version of one of Nancy's files, say x.py.  Therefore the input to the @shadow algorithm is:

1.  The cached version of x.py  This is the latest version of x.py that was pulled from SCCS.  It contains sentinels.  In @shadow terminology, this is the **private file**.

2. The version of x.py that Nancy is pushing.  This file is in Nancy's local SCCS repository.  This file contains no sentinels. In @shadow terminology, this is the **public file**.

- so this ways is perfect for DVCS but svn/csv/vss etc. Center VCS is not good.

​I believe this scheme will work for any kind of SCCS because the hooks only work on a) the local repository and b) the local cache.​

ok all clean!
- wait the brz first show us the prototype hooks
- just need know how call the @shadow algorithm from Leo
- in fact, this @shadow algorithm is very useful for other action,
etc. co-writing system

​I don't have time to do a prototype now.  Here are the places containing relevant code:

- leoShadow.py contains the fundamental @shadow algorithm plus quite a bit of support code.

- leoAtFile.py contains​ the code actually reads and writes @shadow trees: See at.readOneAtShadowNode and at.openFileForReading.

The at.atShadow ivar is set True in the init code for @shadow trees, so searching for atShadow will show you the special-case code involved.  This code is tricky, as you will see.

- I have been saying that we can ignore the details of the cache as far as the *design* of this scheme goes, but the actual code is non-trivial.

The leoCache.py file contains the fundamental caching code that Leo uses to cache files. Using this code is *very* tricky.  Here is the relevant code from at.read::

    if fromString:
        s,loaded,fileKey = fromString,False,None
    else:
        s,loaded,fileKey = c.cacher.readFile(fileName,root)
    # Never read an external file with file-like sentinels from the cache.
    isFileLike = loaded and at.isFileLike(s)
    if not loaded or isFileLike:
        # if trace: g.trace('file-like file',fileName)
        force = True # Disable caching.
    if loaded and not force:
        if trace: g.trace('in cache',fileName)
        at.inputFile.close()
        root.clearDirty()
        return True

For the push hook, fromString will always be False.

It appears that at.isFileLike returns True is the file uses *old-style* @file sentinels, so we can ignore this test.

So for the bzr push hook, the simplified code to get the cached file will be::

    s,loaded,fileKey = c.cacher.readFile(fileName,root)
    if loaded:

        << use s as the contents of the private file >>

The fileKey is used later in at.read to write the newly-read contents of the file back to the cache.  This write may not be strictly necessary, but it would simplify the @shadow algorithm if Nancy pushes the file again before pulling it.

To repeat, I don't have time to do this now.  I've given you the places from which I would start.

Edward