Automatic git commit on save using entr

[Brian Theado]

I'm currently using leo as my main note-taking application and I find it very useful to have auto-saving, so I enabled the auto-save module.

I felt a little nervous about using auto-save and decided I wanted a git commit on every save so I would have a history in case anything goes wrong.

I read Vitalije's post from not too long ago about overriding the ctrl-s key to perform extra actions, but with auto-save I'm not using any keystrokes. I decided to use the 'entr' utility (http://entrproject.org - which I learned about on this list a long time ago) to watch for changes to my .leo file and automatically execute a git commit.

I use an @script script in my leo outline to launch the entr process in the background and it is working great. I did encounter one problem with the entr process out-living the leo process. IOW, it doesn't exit when leo exits. So then when I launch leo again I have two entr processes running, both of which are trying to run auto git commits.

To work around that problem, I found a way to ensure a 2nd instance of that process doesn't get launched if it is already running and I'm satisfied with it. It doesn't solve the issue of the entr process out-living leo, but it solves the more serious concern of having two instances competing with each other. It would have been better if instead (or in addtion), I would have found a way to get the pid of the backround entr process and kill it when leo exits. But as I said, I'm satisfied enough for now with what I have.

Here is the (linux only) script code (assumes the 'git init' has already been run in the directory with the leo file and the leo file has been commited once):

# The ampersand is special syntax for leo to execute the command

# in the background. The "flock ... 9>" part is because when leo

# exits, the background process doesn't get cleaned up and this

# flock "incantation" will prevent a second instance from launching

# see http://man7.org/linux/man-pages/man1/flock.1.html

# The flock utility is Linux only.

# The 'entr' utility (http://entrproject.org) will watch the given list

# of filenames and run the commands whenever changed

g.execute_shell_commands(

    f"""&(

        flock -n 9 || exit 1

        ls {c.fileName()} | entr -s 'git diff . | cat

        git commit -m "$(date +%Y-%m-%d) autocommit" .'

        ) 9> {c.fileName()}.entr.lock

    """

)

Brian

[Chris George]

On Saturday, September 14, 2019 at 1:58:57 PM UTC-7, btheado wrote:

I'm currently using leo as my main note-taking application and I find it very useful to have auto-saving, so I enabled the auto-save module.

I felt a little nervous about using auto-save and decided I wanted a git commit on every save so I would have a history in case anything goes wrong.

Hi Brian,

I have been using bash and fossil to do something similar.

I start Leo using a bash script.

#!/bin/bash

cd ~/leo-editor
git pull
cd /working/MEGA/leo-files
./rerun2 ./push &
python3 ~/leo-editor/launchLeo.py $1 $2 $3

I treat Leo as an abstraction layer that sits on top of my filesystem and all of my work in external files. I keep all of my .leo files under version control in the same directory. Every time I start a new session, I pull from git (pretty much always from git checkout devel).

I then change to the leo-files directory which becomes the working directory. I run rerun2 which is a bash script I found online that monitors all the files in the current directory and will run a command of my choice with every change. I run this detached and then run Leo.

#!/usr/bin/env bash

# Events that occur within this time from an initial one are ignored
ignore_secs=0.25
clear='false'
verbose='false'

function usage {
    echo "Rerun a given command every time filesystem changes are detected."
    echo ""
    echo "Usage: $(basename $0) [OPTIONS] COMMAND"
    echo ""
    echo "  -c, --clear     Clear the screen before each execution of COMMAND."
    echo "  -v, --verbose   Print the name of the files that changed to cause"
    echo "                  each execution of COMMAND."
    echo "  -h, --help      Display this help and exit."
    echo ""
    echo "Run the given COMMAND, and then every time filesystem changes are"
    echo "detected in or below the current directory, run COMMAND again."
    echo "Changes within $ignore_secs seconds are grouped into one."
    echo ""
    echo "This is useful for running commands to regenerate visual output every"
    echo "time you hit [save] in your editor. For example, re-run tests, or"
    echo "refresh markdown or graphviz rendering."
    echo ""
    echo "COMMAND can only be a simple command, ie. \"executable arg arg...\"."
    echo "For compound commands, use:"
    echo ""
    echo "    rerun bash -c \"ls -l | grep ^d\""
    echo ""
    echo "Using this it's pretty easy to rig up ad-hoc GUI apps on the fly."
    echo "For example, every time you save a .dot file from the comfort of"
    echo "your favourite editor, rerun can execute GraphViz to render it to"
    echo "SVG, and refresh whatever GUI program you use to view that SVG."
    echo ""
    echo "COMMAND can't be a shell alias, and I don't understand why not."
}

while [ $# -gt 0 ]; do
    case "$1" in
      -c|--clear) clear='true';;
      -v|--verbose) verbose='true' ;;
      -h|--help) usage; exit;;
      *) break;;
    esac
    shift
done

function execute() {
    if [ $clear = "true" ]; then
        clear
    fi
    if [ $verbose = "true" ]; then
        if [ -n "$changes" ]; then
            echo -e "Changed: $(echo -e $changes | cut -d' ' -f2 | sort -u | tr '\n' ' ')"
            changes=""
        fi
        echo "$@"
    fi
    "$@"
}

execute "$@"
ignore_until=$(date +%s.%N)

inotifywait --quiet --recursive --monitor --format "%e %w%f" \
    --event modify --event move --event create --event delete \
    --exclude '__pycache__' --exclude '.cache' \
    . | while read changed
do

    changes="$changes\n$changed"

    if [ $(echo "$(date +%s.%N) > $ignore_until" | bc) -eq 1 ] ; then
        ignore_until=$(echo "$(date +%s.%N) + $ignore_secs" | bc)
        ( sleep $ignore_secs ; execute "$@" ) &
    fi

done

The push script is pretty straight forward and could be easily modified to use git instead of fossil.

#!/bin/bash

fossil add .
fossil commit . -m "Autocommit"

Since rerun2 is still part of the console session it closes when I close Leo which looks after having it hang around clutttering up the joint.

This might be a little tighter than the method you use.

Chris

[vitalije]

If you want something to happen on each save regardless whether it was initiated by hitting Ctrl-s or by auto-save, the best approach would be to register a callback on 'save2' event which fires after any save. For example:

from collections import Counter
counter = Counter()
def activate():
    c.user_dict['my_callback'] = my_callback
    g.registerHandler('save2', my_callback)


def deactivate():
    h = c.user_dict.pop('my_callback', None)
    if h:
        g.unregisterHandler('save2', h)


def my_callback(tag, kw):
    c = kw.get('c')
    counter[c.mFileName] += 1
    g.es('git commit -a -m "version %d"'%counter[c.mFileName]) # do whatever

activate() # or deactivate()

Of course, you would use some method to actually run git command in the right folder instead of writing to Log pane.

HTH Vitalije

[vitalije]

The essential part of the above script is just g.registerHandler('save2', my_callback). The rest is just for making possible to deactivate handler and to make every commit with the next natural number. While developing your handler, it is useful to be able to deactivate it in case of an error. Also if you change script and run it again it will register another version of your handler. Quite often, in development, I use the same method to un-register old version and then register the new one.

def reactivate(tag, handler):
    k = '_activation_%s_handler'%tag
        # this is just to prevent collisions
        # you can use whatever string as a key
    old_handler = c.user_dict.pop(k, None)
    if old_handler:
        g.unregisterHandler(tag, old_handler)
    g.registerHandler(tag, handler)
    c.user_dict[k] = handler
reactivate('save2', my_callback)

HTH Vitalije.

[Brian Theado]

Thanks, Vitalije. I had rejected the 'save2' approach because I wanted to run the git commit in the background so it wouldn't add any extra time to the save. Using 'save2' doesn't preclude running the process in the background, but if I take the "naive" approach of using g.execute_shell_commands, then I'm left with the issue of defunct processes (since that function never calls the communicate method for background processes). So I was left with the choice between learning more about Popen and just using the entr utility which I was already familiar with. I chose the latter but that led to the issue of needing to use flock. I may reconsider my choice.

If I do reconsider, then your response will give me a head start on properly using the 'save' hook.

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/630dd216-994f-4a7a-805a-e1b20ed2ae34%40googlegroups.com.

[vitalije]

If you precede git command with the '&' g.execute_shell_commands will not wait for command to finish. IOW it will run in the background as you wish. You may wish to use subprocess module on your own and starting process with subprocess.Popen is fast enough, it won't block your main thread for more than a few milliseconds. Git command will run in its own separate process and exit after finishing.  Notice that there is no files watching involved, there is no need for long running processes either. 

Vitalije

To unsubscribe from this group and stop receiving emails from it, send an email to leo-e...@googlegroups.com.

[vitalije]

Also Popen objects have pid attribute if you need it. And also they have methods like terminate or kill. If you want to terminate background process on closing Leo, then you need to register handler on 'end1' event which should call proc.terminate() on the process you have created with subprocess.Popen.

Vitalije

[vitalije]

YMMV, but I had a bad experience with watching files when using Leo. Leo often writes files in two phases and it happened to me more than once that process watching on files take an empty file or not completely written because of this. So, I had to add some latency to watcher. That is why I came up with my solution using 'save2' handlers, or changing Ctrl-s binding. This way you don't need a running process that watches file system (which by the way, on some platforms is implemented by constant polling file system) and this process wastes some resources too. There is nothing to clean up when closing Leo.

Vitalije

[Brian Theado]

Chris,

On Sun, Sep 15, 2019 at 7:11 AM Chris George <techn...@gmail.com> wrote:

[...] 

#!/bin/bash

cd ~/leo-editor
git pull
cd /working/MEGA/leo-files
./rerun2 ./push &
python3 ~/leo-editor/launchLeo.py $1 $2 $3

Thanks for sharing.

[...rerun2 script...]

 

At a glance, that script seems to be performing the function of the entr utility, so 6 one way, half-dozen the other

Since rerun2 is still part of the console session it closes when I close Leo which looks after having it hang around clutttering up the joint.

 

This might be a little tighter than the method you use.

Here is where I'm confused. You are launching rerun2 in the background, so it will outlast your bash script for sure. Are you saying you close your console when leo exits and that causes rerun to exit? I run a console with many tabs open. In one tab I launch leo. If I leo exits, I just launch again within the same console instance. With that use case I expect to end up with multiple instances of rerun2.

Brian

[Chris George]

I use a .desktop panel shortcut to run Leo. I always run it in a terminal window and close the terminal on exit of Leo.

 

Here is where I'm confused. You are launching rerun2 in the background, so it will outlast your bash script for sure. Are you saying you close your console when leo exits and that causes rerun to exit? I run a console with many tabs open. In one tab I launch leo. If I leo exits, I just launch again within the same console instance. With that use case I expect to end up with multiple instances of rerun2.

Brian

By using the & after the command in the bash script, the command has been detached and runs as a subprocess of the terminal window that is running Leo. When I close that terminal the subprocess goes with it. In fact I would have to take steps, like disown the subprocess or use nohup to have the process continue beyond the closing of the terminal.





HTH,





Chris


[Brian Theado]

Vitalije,

YMMV, but I had a bad experience with watching files when using Leo. Leo often writes files in two phases and it happened to me more than once that process watching on files take an empty file or not completely written because of this. So, I had to add some latency to watcher.

Thanks for the warning, I hadn't thought about that. I just checked my git history of 321 autocommits over the last 3 weeks of using this setup and in no cases do I see an empty or partial file being committed. It might be I've just gotten lucky so far. Or it could also be because the entr utility has quite a few checks built in. See the Theory of Operation at http://entrproject.org. 

If you precede git command with the '&' g.execute_shell_commands will not wait for command to finish

Right. I'm already using the ampersand for launching the long-lived process. Using it for a short-lived process is worse as with every execution you are left with a defunct process.

Put this in a node and hit ctrl-b several times:

g.execute_shell_commands('&echo My pid is $$')

In ps output, you will find a defunct process like this for each one:

[sh] <defunct>

So every time I save, I will end up with one of these. The good news with these is they go away when leo exits.

If I get motivated to improve what I have, I will look into some of your suggestions regarding Popen and proc.terminate.

[Brian Theado]

Ok, thanks for the confirmation. I never close terminal windows and always launch leo with a command on the command-line.

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/d825e886-1465-4dbd-a407-d71af4e93763%40googlegroups.com.