Proposal: remove commit_timestamp.json

[Edward K. Ream]

At present, leoVersion.py gets version info by parsing leo/core/commit_timestamp.json. There are at least two unacceptable problems with this approach:

1. The dates don't get updated if a developer (like me!) forgets to install the git hooks.

Just today I noticed that the last update was supposedly in July!

2. Continually changing commit_timestamp.json creates merge conflicts.

There is a much easier way. Rather than parsing commit_timestamp.json, we can get info about the last commit from  "git log -1". Like this:

    p = subprocess.Popen(
        ["git", "log" , '-1', '--date=default-local'],
        stdout=subprocess.PIPE,
        shell=True,
    )
    out, err = p.communicate()
    out = g.toUnicode(out)
    m = re.search('commit (.*)\n', out)
    commit = m.group(1)[0:8].strip() if m else ''
    m = re.search('Date: (.*)\n', out)
    date = m.group(1).strip() if m else '

This works well in my tests.

For official releases, we expect that git won't be installed, or that "git log" will fail because the official release contains no .git directory.  Either way, the code will catch exceptions thrown by subprocess and return the official (static) release date defined in leoVersion.py.

My release testing will ensure that this approach works.  Do you see any problems with it?

Edward

[Edward K. Ream]

On Sunday, October 16, 2016 at 11:18:02 AM UTC-5, Edward K. Ream wrote:

p = subprocess.Popen(
    ["git", "log" , '-1', '--date=default-local'],
    stdout=subprocess.PIPE,

    shell=True)

Testing on Linux shows that the shell argument should be set as follows:

    shell=sys.platform.startswith('win'),

EKR

[Edward K. Ream]

On Sunday, October 16, 2016 at 11:50:00 AM UTC-5, Edward K. Ream wrote:

On Sunday, October 16, 2016 at 11:18:02 AM UTC-5, Edward K. Ream wrote:

p = subprocess.Popen(
    ["git", "log" , '-1', '--date=default-local'],
    stdout=subprocess.PIPE,
    shell=True)

When this code is run outside a git repo, git log will issue a "fatal" error. This will typically be true when the user is running an "official" release. We can suppress such irrelevant messages by adding this arg to the call to subprocess.Popen:

    stderr=None if trace else subprocess.DEVNULL,

Edward

[Terry Brown]

On Sun, 16 Oct 2016 09:18:02 -0700 (PDT)
"Edward K. Ream" <edre...@gmail.com> wrote:

> At present, leoVersion.py gets version info by parsing
> leo/core/commit_timestamp.json. There are at least two unacceptable
> problems with this approach:
>
> 1. The dates don't get updated if a developer (like me!) forgets to
> install the git hooks.
>
> Just today I noticed that the last update was supposedly in July!

Heh, I don't think it's the end of the world if this happens, I guess
we / I could set up some sort of monitoring for that case, but just
noticing it (eventually) and installing the hooks works too.

> 2. Continually changing commit_timestamp.json creates merge conflicts.

Which can be resolved with

git checkout --ours leo/core/commit_timestamp.json

(seeing it's not that critical which one you use).

Certainly in a git repo. commit_timestamp.json isn't needed.
The point of it was to give useful, *fine grained* version ID in non-git
contexts. I think "you have to use git to use Leo" in an unreasonable
barrier, seeing not all Leo users are coders and therefore may never
use a VCS, and I usually point people to
https://github.com/leo-editor/leo-editor/archive/master.zip
to get the current Leo. There's no git repo. in that .zip.
O
fficial releases don't offer fine grained version info., the first
thing we'd tell someone with a problem to try is running the latest
code, and if they're getting it from master.zip, commit_timestamp.json
is useful.

So I'd argue for leaving commit_timestamp.json as is and living with
the two shortcomings you identified. But if it's not bearable, we can
do something like you suggest.

[Edward K. Ream]

On Sun, Oct 16, 2016 at 4:08 PM, 'Terry Brown' via leo-editor

So I'd argue for leaving commit_timestamp.json as is and living with
the two shortcomings you identified.  But if it's not bearable, we can
do something like you suggest.

​I've already done what I suggested, and it works fine, especially for official releases. But I see your point: the daily builds won't have detailed version info.

Most importantly, I want to make things solid for us core developers, and for those that get Leo using git. Having to install hooks is intolerable in the long run. The new way is far better.

What to do about daily snapshots? Maybe the script that creates the snapshot could create commit_timestamp.json. In this case, leoVersion.py would fall back on the legacy code if git is not around.  It would be straightforward to do this.

I like this approach--it doesn't let the tail wag the dog. What do you think? For sure I am not going back to the old way.

Edward

[Edward K. Ream]

On Sunday, October 16, 2016 at 1:15:56 PM UTC-5, Edward K. Ream wrote:

When this code is run outside a git repo, git log will issue a "fatal" [sic] error. This will typically be true when the user is running an "official" release. We can suppress such irrelevant messages by adding this arg to the call to subprocess.Popen:

    stderr=None if trace else subprocess.DEVNULL,

Arrgh.  pylint informs me that DEVNULL does not exist with Python2. Using subprocess.PIPE works with both 2 and 3.

EKR

[Edward K. Ream]

On Sun, Oct 16, 2016 at 5:13 PM, Edward K. Ream <edre...@gmail.com> wrote:

But I see your point: the daily builds won't have detailed version info.

​[snip]​

 

What to do about daily snapshots? Maybe the script that creates the snapshot could create commit_timestamp.json.

​Heh.  The zip file's name contains the commit hash, so the user can find out the exact version. Isn't that enough? I don't want write any more code to deal with this issue.

EKR

[Mike Hodson]

On Sun, Oct 16, 2016 at 4:24 PM, Edward K. Ream <edre...@gmail.com> wrote:
> The zip file's name contains the commit hash, so the user can find out the
> exact version. Isn't that enough? I don't want write any more code to deal
> with this issue.
>

I just tried downloading the master.zip from github; the only version
info I see is that this is supposed to be "5.4-b1" inside of
PKG-INFO.TXT.

Checking the actual 'master' on github shows that leoApp.py was the
latest modified file, and yes indeed, this is in the zip(after
extracting):
-rw-r--r-- 1 mike users 148K Oct 16 16:34 ./leo/core/leoApp.py

The current git hash is e09c486741b587508314dddff90bfc368c3af8f0
Searching for both the beginning, and end, (as I'm really not sure
what the 'short' hashes are, either the beginning or end..)


mike@odin ~/leo-editor-master $ grep -r 3af8f0 *
mike@odin ~/leo-editor-master $ grep -r e09c48 *
mike@odin ~/leo-editor-master $

Where exactly am I supposed to be looking that I'm unable to find this
version/commit of the master?

Or do I misunderstand?

Thanks,

Mike

[Mike Hodson]

And, there is no hash in the filename. The filename it downloads is
either "master" "master.zip" or "leo-editor-master.zip" depending on
Chrome, Curl, or Wget naming the file (and/or after following the
redirect manually with curl)

Thanks,

Mike

[Edward K. Ream]

On Sun, Oct 16, 2016 at 7:49 PM, Mike Hodson <mys...@gmail.com> wrote:

On Sun, Oct 16, 2016 at 4:24 PM, Edward K. Ream <edre...@gmail.com> wrote:
> The zip file's name contains the commit hash, so the user can find out the

​ ​

exact version.

Where exactly am I supposed to be looking that I'm unable to find this
version/commit of the master?

​Here's an example.  I downloaded
leo-editor-281522323a89c27b7c58338b1eaa624cbd383078.zip

from two days ago.

Using gitk, ​

 

​I found the Sha1 hash​ for this commit:

Author: Edward K. Ream <edre...@gmail.com>  2016-10-14 16:13:27

[snip]

    Updated the 5.4 docs from my "to be documented" mailbox.

Yes, the user must have git installed to get this information, so the hash isn't all that useful.  Otoh, if there were a problem with the .zip file, one of Leo's devs could recover any necessary info.

Put another way, the best, easiest way to get any version of Leo is to use git.  Any other way is less convenient.

EKR

[Mike Hodson]

On Sun, Oct 16, 2016 at 7:12 PM, Edward K. Ream <edre...@gmail.com> wrote:
> Here's an example. I downloaded
> leo-editor-281522323a89c27b7c58338b1eaa624cbd383078.zip
> from two days ago.

How?
I mean I must be missing something horribly wrong if I'm going to the
same URL posted by Terry and getting something without a hash.

That said, I did more digging. Seems that the full hash is contained
-as a comment- in the zip file...
With >2000 files in the zip file, infozip at the commandline shows the
comment -at the top- and in 1 second flat my screen goes from me
hitting enter on 'unzip master.zip' and seeing the end of the files
spit out verbosely and my prompt again.

So for future reference: If you link the redirectable "master.zip"
link, the hash _does not appear_ to be contained in the filename, but
_is_ contained in the Comment.
Maybe I should send a bug/feature request up the Infozip pipeline to
put comments "after" the verbose extracted files...

Thanks,

Mike

[Terry Brown]

On Sun, 16 Oct 2016 20:12:26 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> On Sun, Oct 16, 2016 at 7:49 PM, Mike Hodson <mys...@gmail.com>
> wrote:
>
> > On Sun, Oct 16, 2016 at 4:24 PM, Edward K. Ream
> > <edre...@gmail.com> wrote:
> > > The zip file's name contains the commit hash, so the user can
> > > find out
> > the
> > ​ ​
> > exact version.
> >
> > Where exactly am I supposed to be looking that I'm unable to find
> > this version/commit of the master?
> >
>
> ​Here's an example. I downloaded
> leo-editor-281522323a89c27b7c58338b1eaa624cbd383078.zip
> from two days ago.

The obvious non-git link is
https://github.com/leo-editor/leo-editor/archive/master.zip ('Latest' at
http://leoeditor.com/download.html), which has no hash, as Mike said.

> Using gitk, ​
>
> ​I found the Sha1 hash​ for this commit:
>
> Author: Edward K. Ream <edre...@gmail.com> 2016-10-14 16:13:27
> [snip]
>
> Updated the 5.4 docs from my "to be documented" mailbox.
>
> Yes, the user must have git installed to get this information, so the
> hash isn't all that useful. Otoh, if there were a problem with
> the .zip file, one of Leo's devs could recover any necessary info.
>
> Put another way, the best, easiest way to get any version of Leo is
> to use git. Any other way is less convenient.

This is still the "should using git be a requirement for using Leo"?
question. Without some non-git source of version info., non-git users
have no easy way of downloading a known version, other than official
releases, which is almost certainly not helpful if they're reporting a
problem and we're trying to fix it.

There is no script that makes these files any more, they're just
supplied automagically by GitHub.

But hah - a possible, partial, solution. Instead of this link:
https://github.com/leo-editor/leo-editor/archive/master.zip
we could use this:
https://github.com/leo-editor/leo-editor/archive/master@%7B0%20day%20ago%7D.zip
i.e.
https://github.com/leo-editor/leo-editor/archive/master@{0 day ago}.zip
which downloads the latest commit, *with the hash included*, in the file name.

It's still somewhat less convenient for the non-git user, seeing
the .zip unpacks to a different folder every time you download it (and
there's a new revision, of course). People just wanting a link they can
grab and unzip to stay current can still use
https://github.com/leo-editor/leo-editor/archive/master.zip of course,
sans useful versioning information.

Also, reporting version info. in the log will be less robust. You have
to rely on people not renaming the directory, so you can parse the
version out of sys.argv[0] or whatever. But surely no one would rename
a directly called
leo-editor-4373f88f0dd9b25daa22e0b31f2f23acf88fe3fe ;-)

I think we're getting rid of a couple of minor (well, ok, one-time
medium in the case of installing the hooks) inconveniences for Leo
devs., at the cost of the most robust and frictionless way to get
accurate versioning info. from users reporting problems.

Cheers -Terry

[Terry Brown]

On Sun, 16 Oct 2016 19:20:23 -0600
Mike Hodson <mys...@gmail.com> wrote:

> That said, I did more digging. Seems that the full hash is contained
> -as a comment- in the zip file...
> With >2000 files in the zip file, infozip at the commandline shows the
> comment -at the top- and in 1 second flat my screen goes from me
> hitting enter on 'unzip master.zip' and seeing the end of the files
> spit out verbosely and my prompt again.

Heh, well spotted, seems GitHub puts the hash in the .zip as a comment.

But I don't think that helps us, seeing there's no obvious way to know
where the user stored the .zip, which you'd need to know to access this
info.

Cheers -Terry

[Edward K. Ream]

​​

On Sun, Oct 16, 2016 at 8:20 PM, Mike Hodson <mys...@gmail.com> wrote:

On Sun, Oct 16, 2016 at 7:12 PM, Edward K. Ream <edre...@gmail.com> wrote:
> Here's an example.  I downloaded
> leo-editor-281522323a89c27b7c58338b1eaa624cbd383078.zip
> from two days ago.

How?

​From the two days ago link on the download leo page.​

 

​When I awoke early this morning, I saw the way forward.  It should satisfy everybody:

1. Add​ commit_timestamp.json to .gitignore.  No more merge conflicts and no more pollution of the git record.

2. Create leo/core/commit_timestamp.json in @button make-leo in leoDist.leo. This will ensure that a valid commit_timestamp.json appears in all official distributions, without the need for git hooks.

3. Add commit_timestamp.json back to the manifest in  launchLeo-unified.spec, so that pyinstaller will include it.

4. Change leoVersion.py so that it gets the commit hash and date from commit_timestamp.json if it can not get it from git.

Summary

The correct commit hash and date will appear at signon in all official distros, without devs having to install git hooks or worry about merge commits on commit_timestamp.json.

I'll do this today, and test it, as part of releasing 5.4b1.

Edward

[Edward K. Ream]

On Mon, Oct 17, 2016 at 3:51 AM, Edward K. Ream <edre...@gmail.com> wrote:

​​

 

When I awoke early this morning, I saw the way forward.  It should satisfy everybody:

​[snip]​

 

2. Create leo/core/

​​

commit_timestamp.json in @button make-leo in leoDist.leo.

​The create_json function in leoVersion.py will create ​commit_timestamp.json. This file will contain the commit hash as well as the timestamp.  This ensures that get_version_from_json does not need git at all.

Edward

[Edward K. Ream]

​​

On Mon, Oct 17, 2016 at 3:51 AM, Edward K. Ream <edre...@gmail.com> wrote:

​When I awoke early this morning, I saw the way forward.  It should satisfy everybody:

​Rev 4c953099 puts the new scheme in place.  It seems to work well.

There is now no need for any git hooks.  You can disable them if you like, but they won't interfere with anything if they exist.

leoVersion.py contains the new code. It is much simpler than the old.​ 

I'll be testing this further while creating Leo 5.4b1.  Expect it later today.

Edward

[Edward K. Ream]

On Mon, Oct 17, 2016 at 5:17 AM, Edward K. Ream <edre...@gmail.com> wrote:

​Rev 4c953099 puts the new scheme in place.  It seems to work well.

There is now no need for any git hooks.  You can disable them if you like, but they won't interfere with anything if they exist.

​The format of ​commit_timestamp.json has changed.  It now looks like:

{"date": "Mon Oct 17 05:07:33 2016",
  "hash": "4c953099afa22f39def13f04a0df5cae62c86a1b"}

@button make-leo creates this file, so it doesn't matter if your git hook messes with it, especially since your version won't be pushed ;-)

EKR

[Edward K. Ream]

On Monday, October 17, 2016 at 5:24:23 AM UTC-5, Edward K. Ream wrote:

​Rev 4c953099 puts the new scheme in place.  It seems to work well.

There is now no need for any git hooks.  You can disable them if you like, but they won't interfere with anything if they exist.

​The format of ​commit_timestamp.json has changed.  It now looks like:

{"date": "Mon Oct 17 05:07:33 2016",
  "hash": "4c953099afa22f39def13f04a0df5cae62c86a1b"}

@button make-leo creates this file, so it doesn't matter if your git hook messes with it, especially since your version won't be pushed ;-)

Well, I had to remove commit_timestamp.json from .gitignore and add it back to the list of tracked files so that git archive would write it.

This is a bit disappointing, because it means that commit_timestamp.json will still be around.  Otoh, it won't cause any conflicts if we all remove our git hooks...

Edward

[Terry Brown]

I'm not really sure this has any utility :-/

We know the commit hash of official releases.  That was never the problem. The problem in some ways is that a git repo. can't contain its own hash. But more usefully, the issue is with interacting with users when addressing bugs.  You need to know the exact commit, it matters whether they're testing the version with an attempted fix, or the version before.  So, will we entertain this kind of interaction with non-git users? If Leo was a code editor, maybe we could justify excluding non-git users from that kind of interaction.  But unfortunately ;-) Leo is a multi-purpose tool used by authors, designers, hobbiests in various realms, etc. etc.

So, UserA reports an issue.  A dev. pushes an attempted fix and asks that they grab the latest version, https://github.com/leo-editor/leo-editor/archive/master.zip. Iterate a couple of times.  Are we on the same page version wise?  The most robust and *user* friendly way to be sure is useful info. in the log window.  I think only the original use of commit_timestamp.json achieved that.  The best compromise I've seen so far is to have them download https://github.com/leo-editor/leo-editor/archive/master@%7B0%20day%20ago%7D.zip and tell us that they're running Leo from a folder called leo-editor-f94ab524d406a5431d94523d4bc49b47c380e5c7.

Cheers -Terry

Edward

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

On Monday, October 17, 2016 at 9:04:52 AM UTC-5, Terry Brown wrote:

I'm not really sure this has any utility :-/

I am about to release Leo 5.4b1. Imo, the new scheme works very well.

Leo always reports first 8 characters of the git hash and git date, even when there is no git around, which is almost all the time for most users.

This works because official releases now do include commit_timestamp.json.

The script that creates daily builds should be modified to create commit_timestamp.json in the new format. Once that is done, everything will "just work".

To recap, the new scheme is much better than the old because:

- No git hooks are ever needed.
- There will be no git merge conflicts on commit_timestamp.json once people remove their git hooks.
- Leo always reports the hash and the date of the build that is being run, or will, once the daily build script is updated.)
- The code in leoVersion.py is really simple.

Edward

[Edward K. Ream]

On Monday, October 17, 2016 at 11:23:08 AM UTC-5, Edward K. Ream wrote:

> To recap, the new scheme is much better than the old because:

That summary, while true, doesn't get to the heart of the matter.  And it's repetitious.

In fact, we need commit_timestamp.json only at the exact instant an official release or daily build is being created. There is no need to create the .json file on every single commit.

The new format of the .json file describes exactly the commit, avoiding the problem that a build can't mention its own hash.

The @button make-leo script makes the .json file for official releases. The script that makes daily snapshots should also make this .json file, in the new format.

Edward

[Terry Brown]

[meant to post to the list, not just Edward]

From: Edward K. Ream <edre...@gmail.com>
To: leo-editor <leo-e...@googlegroups.com>
Cc: terry_...@yahoo.com
Sent: Monday, October 17, 2016 11:23 AM
Subject: Re: Proposal: remove commit_timestamp.json

On Monday, October 17, 2016 at 9:04:52 AM UTC-5, Terry Brown wrote:

I'm not really sure this has any utility :-/

I am about to release Leo 5.4b1. Imo, the new scheme works very well.

Leo always reports first 8 characters of the git hash and git date, even when there is no git around, which is almost all the time for most users.

This works because official releases now do include commit_timestamp.json.

The script that creates daily builds should be modified to create commit_timestamp.json in the new format. Once that is done, everything will "just work".

As noted earlier in this thread, there is no script that creates the daily builds, there hasn't been since 

April 6, 2014: https://groups.google.com/d/msg/leo-editor/Hr-ygxeMPm4/tfCZxOCcv4wJ

When you use a link like:

https://github.com/leo-editor/leo-editor/archive/master@{5 days ago}.zip

GitHub sends you leo-editor-438e...92e4.zip.  That's what we have now on the Leo downloads page.

So with the new scheme all we have for user interaction / bug resolution is asking the user the funky name of the folder they're running from, if we re-link "Latest" to ...{0 days ago}.zip

*Even if there was a script*, it wouldn't help that much, even running at 15 minute intervals could create significant confusion re latest versions.

Cheers -Terry

[Jacob Peck]

I'm with Terry on this -- the commit_timestamp.json file has been immensely helpful in debugging user issues in the past.  Relegating the commit hash to some funky filename is not an acceptable option.

Git hooks are painless.  Why is this suddenly an issue?

-->Jake

[Edward K. Ream]

On Mon, Oct 17, 2016 at 11:57 AM, Jacob Peck <gates...@gmail.com> wrote:

I'm with Terry on this -- the commit_timestamp.json file has been immensely helpful in debugging user issues in the past.  Relegating the commit hash to some funky filename is not an acceptable option.

​Ok.  I can live with this.

I would like the hook that creates commit_timestamp.json to use the new format. I'll see what I can do today.

Git hooks are painless.  Why is this suddenly an issue?

​Because they aren't painless :-).  And useless if a dev forgets to copy them into leo-editor/.git when cloning a new repo.

In any event, the old code in leoVersion.py is much simpler than the old, which knew way too much the structure of .git folders.  The new code just uses git log -1.

EKR

[Terry Brown]

From: Edward K. Ream <edre...@gmail.com>
To: leo-editor <leo-e...@googlegroups.com>

Sent: Monday, October 17, 2016 1:40 PM
Subject: Re: Fw: Proposal: remove commit_timestamp.json

On Mon, Oct 17, 2016 at 11:57 AM, Jacob Peck <gates...@gmail.com> wrote:

I'm with Terry on this -- the commit_timestamp.json file has been immensely helpful in debugging user issues in the past.  Relegating the commit hash to some funky filename is not an acceptable option.

​Ok.  I can live with this.

Ok :-) next time I want to make a point I'll just ask Jacob to post it :-)

I would like the hook that creates commit_timestamp.json to use the new format. I'll see what I can do today.

It might be a source of confusion if the commit hash and timestamp sometimes agree (i.e. when .git is present), and sometimes don't (i.e. when the commit hash comes from git log -1).

If I'm understanding the new format, the githash Leo displays in the log would be for the preceding commit, when .git is not present, and the current commit, when it is?  Or perhaps it doesn't display the hash when .git is not present, but then there's not much point having it in the .json file.

This is how the use of "20160722143100" as a replacement for a commit hash evolved.  I think it's a better atomic identifier than 

"Mon Oct 17 05:07:33 2016".  For one thing it has no spaces in it.

Cheers -Terry

Git hooks are painless.  Why is this suddenly an issue?

​Because they aren't painless :-).  And useless if a dev forgets to copy them into leo-editor/.git when cloning a new repo.

In any event, the old code in leoVersion.py is much simpler than the old, which knew way too much the structure of .git folders.  The new code just uses git log -1.

EKR

[Edward K. Ream]

On Mon, Oct 17, 2016 at 11:57 AM, Jacob Peck <gates...@gmail.com> wrote:

​> ​

Why is this suddenly an issue?

​A better answer is that I have been working on distribution for the last three days.  Problems come up in that context that we completely ignore normally.

Let's be clear.  The new signon message is much better than before when using git.  That's not going away.

EKR