[PATCH] Asynchronous functions (settimeout, setinterval, and cancelinterval)

29,724 views
Skip to first unread message

Geoff Greer

unread,
Sep 1, 2013, 9:42:25 PM9/1/13
to vim...@googlegroups.com
This patch adds asynchronous functions to vimscript. If you want to perform an action in 700ms, simply:

let timeout_id = settimeout(700, 'echo("hello")')

To cancel the timeout before it's fired:

canceltimeout(timeout_id)

setinterval() also returns an id that can be used with canceltimeout.

The reason for this patch is simple: asynchronous functionality is needed to implement real-time collaborative editing in Vim. This is one of the most voted-for features (see http://www.vim.org/sponsor/vote_results.php).

Along with Matt Kaniaris, I founded Floobits to build real-time collaboration into every editor. We wrote a plugin for Vim, but we had to use hacks to get async behavior (abusing feedkeys or client-server). These methods had side-effects such as breaking leaderkeys or other shortcuts. After a lot of experimenting, we decided to try patching Vim.

Since Vim is character-driven, we had to munge some low-level input functions to get the desired behavior. We changed gui_wait_for_chars() and mch_inchar() so that call_timeouts() is run every ticktime milliseconds. The default ticktime is 100ms.

This patch isn't finished yet, but it works on unix-based OSes. If the reaction is positive, our intention is to change mch_inchar() (or something similar) in other OS-specific files. That will get async functions working for everyone.

Even if our patch isn't the best approach, we'd love to help get async functions in Vim. Doing so will open the door to a lot of cool plugins.

Oh, and this is the first time either myself or Matt have submitted a patch to Vim, so please be gentle.

Sincerely,

Geoff Greer

async.patch

Tony Mechelynck

unread,
Sep 2, 2013, 8:00:00 PM9/2/13
to vim...@googlegroups.com
Your patch is not in the approved coding style (see :help style-examples)

Wrong:

void
insert_timeouts(timeout_T *to) {
timeout_T *cur = timeouts
timeout_T *prev = NULL


OK:
/*
* Explanation of what the function is used for
* and of how to call it
*/
void
insert_timeouts(to)
timeout_T *to; /* short comment about to */
{
timeout_T *cur = timeouts; /* short comment about cur */
timeout_T *prev = NULL; /* short comment about prev */

NOTE: Don't use ANSI style function declarations. A few people still
have to
use a compiler that doesn't support it.




Best regards,
Tony.
--
Consensus Terrorism:
The process that decides in-office attitudes and behavior.
-- Douglas Coupland, "Generation X: Tales for an Accelerated
Culture"

Geoff Greer

unread,
Sep 2, 2013, 9:31:59 PM9/2/13
to vim...@googlegroups.com
Tony,

Thanks for the feedback. I've attached a new patch with some basic comments and non-ANSI C function definitions.

I'd really like to get feedback on the changes to gui_wait_for_chars() and mch_inchar(). That's where the meat of this patch is.

Sincerely,

Geoff

async.patch

mattn

unread,
Sep 3, 2013, 3:25:03 AM9/3/13
to vim...@googlegroups.com

I live this patch. Bram, we discused about this similar topic at several times. And you didn't answer whether vim will include this feature clearly.
However, I think this feature is extendable way which vim get modern software designs. Let's include this feature!

Thanks.

- Yasuhiro Matsumoto

Ben Fritz

unread,
Sep 3, 2013, 12:04:40 PM9/3/13
to vim...@googlegroups.com

Forgive me if I'm wrong, I didn't spend more than a few minutes looking over the patch.

It looks like this is not really "asynchronous" behavior, but rather adding a timer-like hook into the main loop. Pro: no worries about thread safety, race conditions, etc. Con: if the function called by the timer event takes a long time, Vim hangs until it's done.

If I'm correct, and this gets included, the help will need to be very explicit about making sure to limit the time taken by the function called! And there should definitely be a way to interrupt the action.

Bjorn Tipling

unread,
Sep 3, 2013, 2:17:59 PM9/3/13
to vim...@googlegroups.com
I am pretty excited about this patch and hope it makes it in.


Ben, why should asynchronous receive any type of restriction not already placed on code that runs some milliseconds after a key press (an existing hook). Cancelling asynchronous requests can lead to unpredictable behavior is probably never desirable. I've already experienced plugins that block for too long and what I do is, I just remove the plugin.

Ben Fritz

unread,
Sep 3, 2013, 2:52:58 PM9/3/13
to vim...@googlegroups.com
On Tuesday, September 3, 2013 1:17:59 PM UTC-5, Bjorn Tipling wrote:
> I am pretty excited about this patch and hope it makes it in.
>
>
> Ben, why should asynchronous receive any type of restriction not already placed on code that runs some milliseconds after a key press (an existing hook).

And most of these can be interrupted with CTRL-C. I am not certain whether the async functions will be since you added their processing to the functions which get keypresses (I think...I'm not actually familiar with most of Vim's code). I'm not saying they're not...only that the need to be. We've already had a few "Vim hangs on this regex" bugs in 7.4, and I could easily see a plugin that uses a search() call inside one of these timer functions. Or maybe it calls an external tool that is deadlocked for some reason. Or any number of other things could go wrong that the user doesn't want to wait for.

And I'm more worried about function that takes seconds, or minutes, to complete, than I am about milliseconds. My point was that users of these functions need to be aware that the action is not actually asynchronous in the sense of running in the background and allowing user input to continue in the foreground; you just mean it can be synced to a timer rather than a specific user input, correct?

> Cancelling asynchronous requests can lead to unpredictable behavior is probably never desirable. I've already experienced plugins that block for too long and what I do is, I just remove the plugin.
>

If you're typing something, suddenly Vim stops responding to any keyboard input, you've waited most of a minute for Vim to start responding, and you have unsaved work you wish to keep...it can be more desirable to interrupt the action and then continue using Vim long enough to save your work and restart.

Maybe if the user interrupts the action with CTRL-C you need to set a flag to remove the timer from the list, to prevent it firing off again in a few milliseconds?

kans

unread,
Sep 3, 2013, 2:53:31 PM9/3/13
to vim...@googlegroups.com
On Tuesday, September 3, 2013 9:04:40 AM UTC-7, Ben Fritz wrote:

Ben,

You are correct in that we added a timer to the main loop. Looking over the code once again, I think we should have altered the calls to select/poll instead, but lets discuss the practical effect of this patch since we can work out the details some time later.

Async does not imply parallelism nor safety- it only provides the illusion of concurrency. Idle cpu time spent waiting is instead used to perform some other task. Async frameworks make no such guarantee that every call not block the event loop; it is incumbent upon the programmer to ensure that the event loop is not blocked indefinitely. This is the definition and standard interface of async programming for every popular framework I can think of that lives in a scripting language- JavaScript, nodejs, twisted, event machine, and luvit.

The current alternative in Vim is to block forever, or worse yet, not write useful software because it is impossible to do so as a Vim plugin. With a proper settimeout, plugins have the option of returning control to the main loop and checking in latter. I'd suggest a settimeout on the settimeouts is not needed; plugins which cause too much trouble will either be fixed or uninstalled.

Ben Fritz

unread,
Sep 3, 2013, 4:07:00 PM9/3/13
to vim...@googlegroups.com
On Tuesday, September 3, 2013 1:53:31 PM UTC-5, kans wrote:
> On Tuesday, September 3, 2013 9:04:40 AM UTC-7, Ben Fritz wrote:
> > On Sunday, September 1, 2013 8:42:25 PM UTC-5, Geoff Greer wrote:
> > > This patch adds asynchronous functions to vimscript. If you want to perform an action in 700ms, simply:
> > >
> > > let timeout_id = settimeout(700, 'echo("hello")')
> > >
> > > To cancel the timeout before it's fired:
> > >
> > > canceltimeout(timeout_id)
> > >
> > > setinterval() also returns an id that can be used with canceltimeout.
> > >
> > > The reason for this patch is simple: asynchronous functionality is needed to implement real-time collaborative editing in Vim. This is one of the most voted-for features (see http://www.vim.org/sponsor/vote_results.php).
> > >
> > > Along with Matt Kaniaris, I founded Floobits to build real-time collaboration into every editor. We wrote a plugin for Vim, but we had to use hacks to get async behavior (abusing feedkeys or client-server). These methods had side-effects such as breaking leaderkeys or other shortcuts. After a lot of experimenting, we decided to try patching Vim.
> > >
> > > Since Vim is character-driven, we had to munge some low-level input functions to get the desired behavior. We changed gui_wait_for_chars() and mch_inchar() so that call_timeouts() is run every ticktime milliseconds. The default ticktime is 100ms.
> > >
> > > This patch isn't finished yet, but it works on unix-based OSes. If the reaction is positive, our intention is to change mch_inchar() (or something similar) in other OS-specific files. That will get async functions working for everyone.
> > >
> > > Even if our patch isn't the best approach, we'd love to help get async functions in Vim. Doing so will open the door to a lot of cool plugins.
> > >
> > > Oh, and this is the first time either myself or Matt have submitted a patch to Vim, so please be gentle.
> > >
> > > Sincerely,
> > >
> > > Geoff Greer
> >
> > Forgive me if I'm wrong, I didn't spend more than a few minutes looking over the patch.
> >
> > It looks like this is not really "asynchronous" behavior, but rather adding a timer-like hook into the main loop. Pro: no worries about thread safety, race conditions, etc. Con: if the function called by the timer event takes a long time, Vim hangs until it's done.
> >
> > If I'm correct, and this gets included, the help will need to be very explicit about making sure to limit the time taken by the function called! And there should definitely be a way to interrupt the action.
>
> Ben,
>
> You are correct in that we added a timer to the main loop. Looking over the code once again, I think we should have altered the calls to select/poll instead, but lets discuss the practical effect of this patch since we can work out the details some time later.
>

I like the practical implications of being able to respond to a timer event in a plugin. Polling is useful for many, many purposes, and doing it by shelling out and calling back with --remote calls is awkward at best.

> Async does not imply parallelism nor safety- it only provides the illusion of concurrency. Idle cpu time spent waiting is instead used to perform some other task. Async frameworks make no such guarantee that every call not block the event loop; it is incumbent upon the programmer to ensure that the event loop is not blocked indefinitely. This is the definition and standard interface of async programming for every popular framework I can think of that lives in a scripting language- JavaScript, nodejs, twisted, event machine, and luvit.
>

I'm more familiar with C/C++/Java, and even shell scripting, where "async" means it doesn't block the main program flow, as wikipedia says on http://en.wikipedia.org/wiki/Asynchrony: "Asynchronous actions are actions executed in a non-blocking scheme, allowing the main program flow to continue processing."

So, in my mind..."async" normally implies parallelism.

You've created an interface that is probably close enough, if developers are careful not to allow their timer callbacks to take too long, or use them to spawn and monitor a truly async process via the shell or python or whatever.

The nice thing is that you thus avoid the problem of concurrent access/thread safety, because you've not used multiple threads or processes at all. The not-so-nice thing is the inability to call long-running worker functions with your interface.

> The current alternative in Vim is to block forever, or worse yet, not write useful software because it is impossible to do so as a Vim plugin. With a proper settimeout, plugins have the option of returning control to the main loop and checking in latter.

I agree, we need something. That concurrent editing thing looked really neat! And doubtless it would be much more robust/easier to maintain with timer events in Vim.

> I'd suggest a settimeout on the settimeouts is not needed; plugins which cause too much trouble will either be fixed or uninstalled.

Possibly; but I'd rather not find the bug in the plugin by losing a lot of work when my Vim goes into an infinite loop that can't be interrupted.

kans

unread,
Sep 3, 2013, 7:31:53 PM9/3/13
to vim...@googlegroups.com

Ben,

Thanks for the explanation. End users will probably never directly call set_timeout. Hopefully, plugin authors know better than to make a call that can block forever. Good documentation will help.

Maybe control-c should cancel intervals. As you mentioned, this would let users save stuff and restart vim. At worse, plugins would be dumped into an inconsistent state that could cause more harm. Also, canceling internal callbacks is quite opaque to users. I don't see any great options for dealing with runaway plugins.

-kans


Thanks for explanation.

David Fishburn

unread,
Sep 3, 2013, 7:49:21 PM9/3/13
to vim_dev
...
Maybe control-c should cancel intervals.  As you mentioned, this would let users save stuff and restart vim.  At worse, plugins would be dumped into an inconsistent state that could cause more harm.  Also, canceling internal callbacks is quite opaque to users.  I don't see any great options for dealing with runaway plugins.


What I typically see if lots of users have many plugins installed.  It is very easy to forget what was installed last that could be leading to "strange" problems in Vim.  Anything running asynchronously I assume could easily fall under this category.

So yes, I believe an option or something to turn off asynchronous execution would be a great idea, as it would be one more step people on this list could suggest to see if the "symptom" goes away and allows the user to narrow their search to find the offending code.

My 2 cents.
David 

Bram Moolenaar

unread,
Sep 5, 2013, 7:26:41 AM9/5/13
to Geoff Greer, vim...@googlegroups.com

A few thoughts about this patch.

This is not asynchronous but these are timed commands. It should be
clear that the commands are executed in the "idle" loop, when Vim waits
for a character to be typed.

There might be two types: One that has high priority, and gets handled
even when the user has typed something. Another that has lower
priority, only gets handled when waiting for the user to type.

As someone already mentioned: If the command takes too much time, Vim
will get stuck. E.g. when it reads from a socket. It might be possible
to have this kind of work done in another thread, e.g. using Python.

The documentation is still missing. A lot of details need to be
explained there.

It must be possible to get a list of these commands, so that they can be
cancelled without having the id. This will then possibly break the
plugin that installed the command, so this gets messy quickly.

The code uses "long" to store msec, but you need 64 bits for that and
long can be 32 bits.

I don't see the need for 'ticktime'. The remaining time until the next
timeout can be computed. If it's smaller than 'updatetime' then wait
for that long.


--
hundred-and-one symptoms of being an internet addict:
170. You introduce your wife as "my_...@home.wife" and refer to your
children as "forked processes."

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Bjorn Tipling

unread,
Sep 5, 2013, 9:44:15 AM9/5/13
to vim...@googlegroups.com, Geoff Greer
> As someone already mentioned: If the command takes too much time, Vim
> will get stuck. E.g. when it reads from a socket. It might be possible
> to have this kind of work done in another thread, e.g. using Python.


Hey check this out:

au CursorHold * call BlockForever()


" Blocks forever
function! BlockForever()
let c = 1
while c >= 0
let c += 1
endwhile
endfunction

You can already do this blocking, I've already experienced it with plugins I've removed. I ask again why setTimeout should be treated differently than CursorHold? You can cntrl-c out of CursorHold, but just then make that a requirement for settimeout.


> This is not asynchronous but these are timed commands. It should be
> clear that the commands are executed in the "idle" loop, when Vim waits
> for a character to be typed.

As was already pointed out, this is what asynchronous execution looks like in JavaScript. JavaScript in the browser and elsewhere has only one thread of execution, and the behavior of setTimeout in these contexts is similar to this patch. Whatever you want to call this patch, it is very powerful, and vim is missing it. Try implementing a clock for the status line in current vim. It updates only after a keystroke. With this patch, you can have a real clock, that shows you the actual time! Not the time you last pressed a key.

Try this in a Chrome console:

function blockForever() {
var c = 1;
while (c > 0) {
c++;
}
}
window.setTimeout(blockForever, 1000);

You get the same behavior that you would from this patch, the same behavior that CursorHold gives you.

I agree there needs to be documentation on how to use this. I've built vim with this patch and was able to get one plugin that uses setinterval to work, but was not able to get my own attempt to work after a quick attempt.

On Thursday, September 5, 2013 4:26:41 AM UTC-7, Bram Moolenaar wrote:
> A few thoughts about this patch.
>
>
>
> This is not asynchronous but these are timed commands. It should be
>
> clear that the commands are executed in the "idle" loop, when Vim waits
>
> for a character to be typed.
>
>
>
> There might be two types: One that has high priority, and gets handled
>
> even when the user has typed something. Another that has lower
>
> priority, only gets handled when waiting for the user to type.
>
>
>
> As someone already mentioned: If the command takes too much time, Vim
>
> will get stuck. E.g. when it reads from a socket. It might be possible
>
> to have this kind of work done in another thread, e.g. using Python.
>
>
>
> The documentation is still missing. A lot of details need to be
>
> explained there.
>
>
>
> It must be possible to get a list of these commands, so that they can be
>
> cancelled without having the id. This will then possibly break the
>
> plugin that installed the command, so this gets messy quickly.
>
>
>
> The code uses "long" to store msec, but you need 64 bits for that and
>
> long can be 32 bits.
>
>
>
> I don't see the need for 'ticktime'. The remaining time until the next
>
> timeout can be computed. If it's smaller than 'updatetime' then wait
>
> for that long.
>
>
>
>
>
> --
>
> hundred-and-one symptoms of being an internet addict:
>

> 170. You introduce your wife as "my_la.wife" and refer to your
>
> children as "forked processes."
>
>
>
> /// Bram Moolenaar -- Bra....net -- http://www.Moolenaar.net \\\

Yasuhiro MATSUMOTO

unread,
Sep 5, 2013, 10:26:32 AM9/5/13
to vim...@googlegroups.com, Geoff Greer
We can use python's thread to get async, but vim don't have GIL. So it
can't get events safety from the thread.
> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups
> "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vim_dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>


--
- Yasuhiro Matsumoto

Ben Fritz

unread,
Sep 5, 2013, 12:08:16 PM9/5/13
to vim...@googlegroups.com, Geoff Greer
On Thursday, September 5, 2013 8:44:15 AM UTC-5, Bjorn Tipling wrote:
> > As someone already mentioned: If the command takes too much time, Vim
> > will get stuck. E.g. when it reads from a socket. It might be possible
> > to have this kind of work done in another thread, e.g. using Python.
>
>
> Hey check this out:
>
> au CursorHold * call BlockForever()
>
>
> " Blocks forever
> function! BlockForever()
> let c = 1
> while c >= 0
> let c += 1
> endwhile
> endfunction
>
> You can already do this blocking, I've already experienced it with plugins I've removed. I ask again why setTimeout should be treated differently than CursorHold? You can cntrl-c out of CursorHold, but just then make that a requirement for settimeout.
>

Exactly. I think as long as you can interrupt the timer with a keystroke it should be fine. But since you call your timer in between waiting for a keystroke, won't that prevent CTRL-C from interrupting it? Maybe it won't...I haven't tried it. I don't think we have any unique constraints, I just think that (a) it should be mentioned in the help that the timer blocks user input while processing, it's not a true asynchronous process and (b) it is required that CTRL-C or similar can interrupt the timer like it can already for autocmds, etc.

kans

unread,
Sep 5, 2013, 1:53:39 PM9/5/13
to vim...@googlegroups.com, Geoff Greer
On Thursday, September 5, 2013 9:08:16 AM UTC-7, Ben Fritz wrote:
> On Thursday, September 5, 2013 8:44:15 AM UTC-5, Bjorn Tipling wrote:
> > > As someone already mentioned: If the command takes too much time, Vim
> > > will get stuck. E.g. when it reads from a socket. It might be possible
> > > to have this kind of work done in another thread, e.g. using Python.
> >
> >
> > Hey check this out:
> >
> > au CursorHold * call BlockForever()
> >
> >
> > " Blocks forever
> > function! BlockForever()
> > let c = 1
> > while c >= 0
> > let c += 1
> > endwhile
> > endfunction
> >
> > You can already do this blocking, I've already experienced it with plugins I've removed. I ask again why setTimeout should be treated differently than CursorHold? You can cntrl-c out of CursorHold, but just then make that a requirement for settimeout.
> >
>
> Exactly. I think as long as you can interrupt the timer with a keystroke it should be fine. But since you call your timer in between waiting for a keystroke, won't that prevent CTRL-C from interrupting it? Maybe it won't...I haven't tried it.

ctrl-c works (in this patch) because the OS converts it into a SIGINT which vim knows how to handle.

>There might be two types: One that has high priority, and gets handled
even when the user has typed something. Another that has lower
priority, only gets handled when waiting for the user to type.

As mattn pointed out, this won't work unless the internals of vim are threadsafe.

> It must be possible to get a list of these commands, so that they can be
>
> cancelled without having the id.

This wouldn't be much work, but to my knowledge, no other implementation has this ability.

> I don't see the need for 'ticktime'. The remaining time until the next
>
> timeout can be computed. If it's smaller than 'updatetime' then wait
>
> for that long.

This is true. ticktime is only useful for end users if they want to trade off efficiency for a better timer function. 50ms is probably a sane default that is good enough (and once again, no other implementation lets this bubble up to user land).


Does anyone have any feedback on how this implemented; ie, putting this logic in mch_inchar vs. RealWaitForChar? Another issue is that vim makes other blocking calls that will block the timers.

kans

unread,
Sep 10, 2013, 5:54:38 PM9/10/13
to vim...@googlegroups.com, Geoff Greer

I've attached our latest patch. We moved some of the logic down to the level of poll/select (RealWaitForChar) using MAY_LOOP which is similar to what the scheme bindings do; it is cleaner and works better. We also changed the eval call to use do_cmdline_cmd which is far more convenient. Now, the timeouts don't have to be user defined functions. Also, all timeouts can now be disabled by setting 'ticktime' to -1. We discussed disabling intervals that fail, but we ultimately decided that the onus should be on programmers to make sane plugins and canceling intervals is unexpected behavior. Finally, the new features are now documented. Please let us know what you think.

async.patch

Nikolay Pavlov

unread,
Sep 10, 2013, 6:18:42 PM9/10/13
to vim...@googlegroups.com, Geoff Greer

Why do you keep calling it async?

Also other issues:
1. async.c does not follow coding style:
1.1. each 8 indentation spaces should be replaced with a tab
1.2. each opening figure brace should be present on its own line
1.3. no (void) in function declaration
2. async.h should be split into globals.h and proto/async.pro. async.h itself is to be removed.
3. Never use gettimeofday for intervals. There are special monotonic clocks in all operating system, for linux it is clock_gettime(CLOCK_MONOTONIC_RAW). What you use is subject for ntp daemon/ntpclient updates.
4. There is a proper way to disable a setting. Putting the whole setting into an #ifdef is not the one.
5. Documentation of 'ticktime' lacks references to +smth. I repeat, do not call smth "async". Not until you will make it execute asynchronously.
6. I do not see patch for the file with a list of features.

Thomas

unread,
Sep 11, 2013, 2:59:34 AM9/11/13
to vim...@googlegroups.com
On 11 September 2013 00:18, Nikolay Pavlov <zyx...@gmail.com> wrote:

Why do you keep calling it async?

Referring to the introduction to asynchronous programming linked below, the patch is an almost asynchronous feature, or at least a completition of the already existing asynchronous features of vim. With all the "autocmd-events" we already have the posibillity to run code asynchronous.
Anyway asynchronous doesn't mean threaded, as far as I know and referring again to the link. That a two different concepts (which you can combine if you want).

http://cs.brown.edu/courses/cs168/f12/handouts/async.pdf

Bjorn Tipling

unread,
Sep 11, 2013, 7:41:49 AM9/11/13
to vim...@googlegroups.com, wiene...@googlemail.com
The async issue seems like a bike shed. Whatever you want to call it, this feature is very powerful and I'm very grateful many are taking it serious. Let's focus on encouraging the devs and reviewers to getting a patch the community can accept. If the name really is an issue, I'm sure another name can work as well. Perhaps 'setinterval' or something like that.

Nikolay Pavlov

unread,
Sep 11, 2013, 10:46:06 AM9/11/13
to vim...@googlegroups.com

Now please show how you can run one sequence of commands interleaved with the other with this patch. Function that runs with given interval blocks everything until it completes, same for autocommands. You cannot say you implement preemptive multitasking if you have to wait for one task to complete before starting the other. It is not async feature.

You can, of course, do some hacks with saving a state, exiting and resuming, but you will then be forced to *emulate* preemption without any support from vim. I used to have some simple emulation even without such patch.

Note though that I would really like it merged. But not with the current name.

kans

unread,
Sep 11, 2013, 7:28:24 PM9/11/13
to vim...@googlegroups.com
ZyX,

The new patch addresses your criticism apart from changing the name. I believe we obey Vim code conventions everywhere. We have also implemented a monotonically increasing timer for unix and osx which are the only two OSes we have on hand. Otherwise, we fall back to gettimeofday. Async is only #defined if gettimeofday exists.

Bram,

We now use long longs for keeping track of the timers. We have also documented the new functions and some of their limitations.


Please let us know if you have further feedback.

-Matt

async.patch

Nikolay Pavlov

unread,
Sep 12, 2013, 4:59:51 AM9/12/13
to vim...@googlegroups.com

You can take the code from python: since 3.3 it has monotonic() function defined in C code in time module: pymonotonic function in Modules/timemodule.c. It has implementation for windows as well. Linux implementation used CLOCK_MONOTONIC (or, with higher priority, CLOCK_HIRES which is not mentioned in my clock_gettime man page; I guess there is a reason for it) without _RAW, though I would write

#ifdef CLOCK_MONOTONIC_RAW
                             CLOCK_MONOTONIC_RAW
#else
                             CLOCK_MONOTONIC
#endif

in function arguments. But it is not as severe as issue with gettimeofday: you cannot see 2 centuries hop with CLOCK_MONOTONIC, maximum slightly increased/decreased second duration when NTP daemon does his adjustments.

By the way, two pairs of figure braces may be removed from insert_timeout. Not that it is required by coding style, but this makes code look better.

> Bram,
>
> We now use long longs for keeping track of the timers.  We have also documented the new functions and some of their limitations.
>
>
> Please let us know if you have further feedback.
>
> -Matt
>

kans

unread,
Sep 13, 2013, 5:43:13 AM9/13/13
to vim...@googlegroups.com
ZyX,

I added a monotonic timer for windows; I basically took the one from python. I also changed all the make files to include the new stuff as well. The patch is 1200 lines long now (37 files changed, 476 insertions, 24 deletions). The current state includes monotonically increasing timers for OsX, Win32, and *nix. It really doesn't make sense to add any more features, but I'm happy to do cleanup as needed.

-Matt
async.monotonic.patch

kans

unread,
Sep 19, 2013, 6:01:17 AM9/19/13
to vim...@googlegroups.com
I was doing some work cleaning up the patch. It really does make sense to kill all intervals if one of them fails; its too annoying as an end user if a timeout fails every 50ms. The docstring for do_cmdline_cmd implies it should return FAIL if it can't evaluate "cmdline", but in practice it returns OK most of the time. For instance, evaluating a function that doesn't exist throws an error and returns OK. Some invalid expressions return OK, some, like "if (" don't. Does anyone know of an easy way to figure out of the expression could be evaluated and didn't throw?

-Matt

Nikolay Pavlov

unread,
Sep 19, 2013, 6:29:35 AM9/19/13
to vim...@googlegroups.com

I can suggest looking at VimTry* functions in src/if_py_both.h: they are used to transform all vim errors into python exceptions. Purge the python stuff and you will get C functions for wrapping arbitrary code into try/catch.

kans

unread,
Sep 21, 2013, 7:29:11 PM9/21/13
to vim...@googlegroups.com
Zyx,

Thanks for the help. Intervals are no longer run if they error of any sort (apart from a user interrupt which will likely happen accidentally). I was also forced to change configure.in as the monotonic timer on Linux needs to sometimes link against rt. I ran autoconf and also dumped those changes into auto/configure file. They run in both Linux and OsX.


Originally, I had hopped that this patch will be 100 lines tops, but it is closer to 600. A good chunk of that is documentation and touching the various make files. I think we have addressed all the concerns (apart from changing the name). What would it take to get this merged?

-Matt
async.stop_interval_errors.patch

Ben Fritz

unread,
Sep 21, 2013, 11:15:37 PM9/21/13
to vim...@googlegroups.com
On Saturday, September 21, 2013 6:29:11 PM UTC-5, kans wrote:
>
> Thanks for the help. Intervals are no longer run if they error of any sort (apart from a user interrupt which will likely happen accidentally).

So, if I install a plugin using this feature, and some weird sequence of events puts it into an infinite loop, I just need to kill Vim and lose my work?

I really, really want a way to force all timers to stop firing. Maybe not CTRL-C but SOMETHING should be possible for the user to kill misbehaving periodic tasks.

kans

unread,
Sep 22, 2013, 12:21:28 AM9/22/13
to vim...@googlegroups.com


Ben,

Ctrl-C will kill a given task, but it won't cancel future timeouts (if it was an interval). Ctrl-C can't do that or else it would happen accidentally since the end user should never notice timers firing to begin with. I'm open to suggestions, but you are asking for a solution to a problem that doesn't exist in any other text editor or IDE.

-Matt

Bram Moolenaar

unread,
Sep 22, 2013, 9:38:32 AM9/22/13
to kans, vim...@googlegroups.com

kans wrote:

> Zyx,
>
> Thanks for the help. Intervals are no longer run if they error of any
> sort (apart from a user interrupt which will likely happen
> accidentally). I was also forced to change configure.in as the
> monotonic timer on Linux needs to sometimes link against rt. I ran
> autoconf and also dumped those changes into auto/configure file. They
> run in both Linux and OsX.
>
>
> Originally, I had hopped that this patch will be 100 lines tops, but
> it is closer to 600. A good chunk of that is documentation and
> touching the various make files. I think we have addressed all the
> concerns (apart from changing the name). What would it take to get
> this merged?

I still have several problems with including this functionality.

I would prefer this to be called timed events instead of async, since
it's not really asynchronous. A timeout is for when you do something
and it takes too long. What this is doing is setting a timer or a
repeated timer.

There is no need for 'ticktime', one can compute the time until the next
event and use that. It's also more accurate and has less overhead.

Like others I'm worried that some plugin creates a timer with a problem,
and it keeps triggering over and over. The user must have a way to stop
it. Also, there will be plugins that have interference with what some
timer is doing, there needs to be a way to temporary disable them.
It might be required to give the timers a name instead of an ID, so that
specific timers can be disabled by name, instead of having to lookup
their ID (how would one do that anyway?).

What happens with timers when at a prompt, e.g. for ":s/pat/repl/gc"?


This is the kind of functionality where we may need to fix many bugs
that are uncovered over time, like with the conceal feature. I am not
going to include it before I'm convinced it works properly.

--
hundred-and-one symptoms of being an internet addict:
250. You've given up the search for the "perfect woman" and instead,
sit in front of the PC until you're just too tired to care.

Ben Fritz

unread,
Sep 23, 2013, 10:48:35 AM9/23/13
to vim...@googlegroups.com

That's because other editors and IDEs that have periodic timers don't freeze all user input while waiting for the periodic task to finish.

You're right about CTRL-C canceling all future timeouts would be a bad thing, since that would happen accidentally more than on purpose.

So probably a new keyboard command is needed. It can't be just a function or ex command, because if user input is frozen due to a bad timer, the user will never see the command-line.

Or, if a timer is interrupted with CTRL-C, you could prompt to continue running timers. Kind of like some browsers prompt you "an error occurred running javascript, continue running scripts on this page?" when they load a bad website.

Geoff Greer

unread,
Sep 23, 2013, 5:48:48 PM9/23/13
to vim...@googlegroups.com, kans
On Sunday, September 22, 2013 6:38:32 AM UTC-7, Bram Moolenaar wrote:
> kans wrote:
>
>
>
> > Zyx,
>
> >
>
> > Thanks for the help. Intervals are no longer run if they error of any
>
> > sort (apart from a user interrupt which will likely happen
>
> > accidentally). I was also forced to change configure.in as the
>
> > monotonic timer on Linux needs to sometimes link against rt. I ran
>
> > autoconf and also dumped those changes into auto/configure file. They
>
> > run in both Linux and OsX.
>
> >
>
> >
>
> > Originally, I had hopped that this patch will be 100 lines tops, but
>
> > it is closer to 600. A good chunk of that is documentation and
>
> > touching the various make files. I think we have addressed all the
>
> > concerns (apart from changing the name). What would it take to get
>
> > this merged?
>
>
>
> I still have several problems with including this functionality.
>
>
>
> I would prefer this to be called timed events instead of async, since
>
> it's not really asynchronous. A timeout is for when you do something
>
> and it takes too long. What this is doing is setting a timer or a
>
> repeated timer.
>

We can change the vocabulary if you like, but we wanted to follow a pattern that developers were already familiar with. We used the JavaScript names of setTimeout/setInterval/cancelTimeout. Sublime Text uses the same names. Emacs has a similar API.

>
>
> There is no need for 'ticktime', one can compute the time until the next
>
> event and use that. It's also more accurate and has less overhead.
>

Matt is working on a fix for this.

>
> Like others I'm worried that some plugin creates a timer with a problem,
>
> and it keeps triggering over and over. The user must have a way to stop
>
> it.

We'll change ctrl+c so that it stops future intervals as well as canceling the current timer. That should prevent Vim from hanging and let users save their work.

We thought about something more complicated, such as showing a list of pending timers and letting the user cancel them one-by-one. Unfortunately, users can only guess at the problem timer(s). They'd need the equivalent of `top` to find the culprit(s). Worse, canceling the wrong one could easily break a well-behaved plugin. The only fix for the well-behaved plugin would be to restart Vim.

Most other editors and frameworks have taken the stance that with great power comes great responsibility. If a timer causes problems, then it causes problems. Fault lies with the developer of the code that set the timer. Sublime Text, Emacs, JavaScript, Twisted Python (and more) work this way.

> Also, there will be plugins that have interference with what some
>
> timer is doing, there needs to be a way to temporary disable them.
>
> It might be required to give the timers a name instead of an ID, so that
>
> specific timers can be disabled by name, instead of having to lookup
>
> their ID (how would one do that anyway?).
>

I think names for timers would cause more problems than they solve. Plugins could conflict with each other if they chose the same names for their timers. IDs assigned at runtime make conflicts impossible. Also, the pattern of returning an ID is not foreign to those who have used other setTimeout implementations.

>
>
> What happens with timers when at a prompt, e.g. for ":s/pat/repl/gc"?
>
>

Timers are run while the prompt is prompting. It's the responsibility of the plugin developer to not cause problems. Great power, great responsibility and all that.

>
>
>
> This is the kind of functionality where we may need to fix many bugs
>
> that are uncovered over time, like with the conceal feature. I am not
>
> going to include it before I'm convinced it works properly.
>
>

I don't think there will be many issues, but I do see where you're coming from. You're responsible for maintaining Vim indefinitely, while people like Matt and myself can submit a patch and disappear. A conservative approach to new features is entirely understandable. If I were in your position, I'd probably behave similarly.

But please be aware of the costs of *not* adding timers to Vim. Without them, whole categories of plugins are simply impossible: Indexing files in the background. Collaborative editing. Deep integration with debuggers and browsers. That's just what I can imagine in a few minutes. The entire Vim community could imagine (and build) so much more.

There's a new generation of plugins waiting to be made. They just need timers. I hope you'll soften your stance on this patch.

Sincerely,

Geoff Greer
http://geoff.greer.fm/
https://github.com/ggreer

kans

unread,
Sep 23, 2013, 8:53:26 PM9/23/13
to vim...@googlegroups.com, kans
On Sunday, September 22, 2013 6:38:32 AM UTC-7, Bram Moolenaar wrote:
> kans wrote:
>
>
>
> > Zyx,
>
> >
>
> > Thanks for the help. Intervals are no longer run if they error of any
>
> > sort (apart from a user interrupt which will likely happen
>
> > accidentally). I was also forced to change configure.in as the
>
> > monotonic timer on Linux needs to sometimes link against rt. I ran
>
> > autoconf and also dumped those changes into auto/configure file. They
>
> > run in both Linux and OsX.
>
> >
>
> >
>
> > Originally, I had hopped that this patch will be 100 lines tops, but
>
> > it is closer to 600. A good chunk of that is documentation and
>
> > touching the various make files. I think we have addressed all the
>
> > concerns (apart from changing the name). What would it take to get
>
> > this merged?
>
>
>
> I still have several problems with including this functionality.
>
>
>
> I would prefer this to be called timed events instead of async, since
>
> it's not really asynchronous. A timeout is for when you do something
>
> and it takes too long. What this is doing is setting a timer or a
>
> repeated timer.

The named has been changed from async to timers everywhere.

>
>
> There is no need for 'ticktime', one can compute the time until the next
>
> event and use that. It's also more accurate and has less overhead.

Agreed. 'ticktime' is now the minimum time between calls to call_timeouts() and is set to 20ms. Vim waits the appropriate amount of time (which could be ticktime, -1, 0, or the time until the next timeout). Say we have intervals every .5ms; the overhead of calling select that often is way too high given we don't care about that level of precision.


>
>
> Like others I'm worried that some plugin creates a timer with a problem,
>
> and it keeps triggering over and over. The user must have a way to stop
>
> it.

The new behavior cancels intervals if they error in any way. Additionally, upon futher thought, there is no way around ctrl-C. That is, we have to use signals and ctrl-c (SIGINT) is the only convenient one. In the newest version, ctrl-c will cancel a given timeout/interval and all future calls.

> Also, there will be plugins that have interference with what some
>
> timer is doing, there needs to be a way to temporary disable them.

The new solution is good enough from the user's perspective. If a rogue timer exists, the only possible action is to cancel it, save buffers, and restart vim after removing the offending plugin. No plugin is going to continue to work after its timers are canceled.

Disabling a specific timer would violate internal assumptions plugins make. We could disable all timers for X seconds, but this functionality isn't necessary and would result in plugins battling each other over when, if ever, timeouts should be called.

>
> It might be required to give the timers a name instead of an ID, so that
>
> specific timers can be disabled by name, instead of having to lookup
>
> their ID (how would one do that anyway?).

I don't follow. Letting plugins cancel non-local timeouts will result in plugin wars, unintentional or otherwise. I don't ever see a user canceling a given timeout based on some list; at the very least, its a poor user experience. Why should users ever need to know timers exist in the first place? Allowing mutable global state with side effects *across plugins* causes more problems than it solves.

async.patch

David Fishburn

unread,
Sep 24, 2013, 8:51:44 AM9/24/13
to vim_dev

On Mon, Sep 23, 2013 at 8:53 PM, kans <mkan...@gmail.com> wrote:
...
 
> Like others I'm worried that some plugin creates a timer with a problem,
>
> and it keeps triggering over and over.  The user must have a way to stop
>
> it.

The new behavior cancels intervals if they error in any way.  Additionally, upon further thought, there is no way around ctrl-C.  That is, we have to use signals and ctrl-c (SIGINT) is the only convenient one.  In the newest version, ctrl-c will cancel a given timeout/interval and all future calls.


FYI, I often use CTRL-C when Vim is taking too long (for example, my home drive is N: and I am not connected to the network).  I understand why we want this, but I am just wondering how often people use CTRL-C within Vim.

What I would _very_ much like to see is when CTRL-C is pressed, and there are timed events, Vim would display the name of each timed event canceled.  I would also like to see future events differentiated.

CTRL-C
Cancelled 21_DB_timedEvent
Future events cancelled
21_DB_timedEvent
34_SS_MyTimedEvent
87_GG_hisTimedEvent

That way, if I see a pattern that I am a canceling the same event over and over, I can start vimgrepping my plugin / autoload directories to figure out what plugin is installing that timed event.

 
> Also, there will be plugins that have interference with what some
>
> timer is doing, there needs to be a way to temporary disable them.

The new solution is good enough from the user's perspective.  If a rogue timer exists, the only possible action is to cancel it, save buffers, and restart vim after removing the offending plugin.  No plugin is going to continue to work after its timers are canceled.

If the CTRL-C shows the above information, we can at least identify potential issue plugins.
 
I think when we document this feature in the Vim Help files, it is important to indicated every timed event should reference a GLOBAL variable which will allow the plugin to disable the timed event.

So, similar to how plugin developers follow the standard:

if exists('g:loaded_myplugin') 
    finish
endif
let g:loaded_myplugin = 1

In our timed event sample (prominently displayed) in the help we could have:

let g:myplugin_allow_timedevent = 1

function! MyPlugin_timedEvent() 
    if g:myplugin_allow_timedevent == 0 
        return
    endif

     -- Do stuff here
endfunction 

This might help (especially) first time plugin writers of the timed event feature to put in some safe guards ahead of time.


David

Matthew Kaniaris

unread,
Sep 24, 2013, 3:35:26 PM9/24/13
to vim...@googlegroups.com
David,

Is it currently possible to introspect the file and line number of call site of sitetimeout?  I think it would be enough to just error message that info when a timer is canceled.

-Matt


--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
 
---
You received this message because you are subscribed to a topic in the Google Groups "vim_dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_dev/-4pqDJfHCsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vim_dev+u...@googlegroups.com.