[PATCH] Asynchronous functions (settimeout, setinterval, and cancelinterval)

27585 views
Skip to first unread message

Geoff Greer

unread,
Sep 1, 2013, 9:42:25 PM9/1/13
to vim...@googlegroups.com
This patch adds asynchronous functions to vimscript. If you want to perform an action in 700ms, simply:

let timeout_id = settimeout(700, 'echo("hello")')

To cancel the timeout before it's fired:

canceltimeout(timeout_id)

setinterval() also returns an id that can be used with canceltimeout.

The reason for this patch is simple: asynchronous functionality is needed to implement real-time collaborative editing in Vim. This is one of the most voted-for features (see http://www.vim.org/sponsor/vote_results.php).

Along with Matt Kaniaris, I founded Floobits to build real-time collaboration into every editor. We wrote a plugin for Vim, but we had to use hacks to get async behavior (abusing feedkeys or client-server). These methods had side-effects such as breaking leaderkeys or other shortcuts. After a lot of experimenting, we decided to try patching Vim.

Since Vim is character-driven, we had to munge some low-level input functions to get the desired behavior. We changed gui_wait_for_chars() and mch_inchar() so that call_timeouts() is run every ticktime milliseconds. The default ticktime is 100ms.

This patch isn't finished yet, but it works on unix-based OSes. If the reaction is positive, our intention is to change mch_inchar() (or something similar) in other OS-specific files. That will get async functions working for everyone.

Even if our patch isn't the best approach, we'd love to help get async functions in Vim. Doing so will open the door to a lot of cool plugins.

Oh, and this is the first time either myself or Matt have submitted a patch to Vim, so please be gentle.

Sincerely,

Geoff Greer

async.patch

Tony Mechelynck

unread,
Sep 2, 2013, 8:00:00 PM9/2/13
to vim...@googlegroups.com
Your patch is not in the approved coding style (see :help style-examples)

Wrong:

void
insert_timeouts(timeout_T *to) {
timeout_T *cur = timeouts
timeout_T *prev = NULL


OK:
/*
* Explanation of what the function is used for
* and of how to call it
*/
void
insert_timeouts(to)
timeout_T *to; /* short comment about to */
{
timeout_T *cur = timeouts; /* short comment about cur */
timeout_T *prev = NULL; /* short comment about prev */

NOTE: Don't use ANSI style function declarations. A few people still
have to
use a compiler that doesn't support it.




Best regards,
Tony.
--
Consensus Terrorism:
The process that decides in-office attitudes and behavior.
-- Douglas Coupland, "Generation X: Tales for an Accelerated
Culture"

Geoff Greer

unread,
Sep 2, 2013, 9:31:59 PM9/2/13
to vim...@googlegroups.com
Tony,

Thanks for the feedback. I've attached a new patch with some basic comments and non-ANSI C function definitions.

I'd really like to get feedback on the changes to gui_wait_for_chars() and mch_inchar(). That's where the meat of this patch is.

Sincerely,

Geoff

async.patch

mattn

unread,
Sep 3, 2013, 3:25:03 AM9/3/13
to vim...@googlegroups.com

I live this patch. Bram, we discused about this similar topic at several times. And you didn't answer whether vim will include this feature clearly.
However, I think this feature is extendable way which vim get modern software designs. Let's include this feature!

Thanks.

- Yasuhiro Matsumoto

Ben Fritz

unread,
Sep 3, 2013, 12:04:40 PM9/3/13
to vim...@googlegroups.com

Forgive me if I'm wrong, I didn't spend more than a few minutes looking over the patch.

It looks like this is not really "asynchronous" behavior, but rather adding a timer-like hook into the main loop. Pro: no worries about thread safety, race conditions, etc. Con: if the function called by the timer event takes a long time, Vim hangs until it's done.

If I'm correct, and this gets included, the help will need to be very explicit about making sure to limit the time taken by the function called! And there should definitely be a way to interrupt the action.

Bjorn Tipling

unread,
Sep 3, 2013, 2:17:59 PM9/3/13
to vim...@googlegroups.com
I am pretty excited about this patch and hope it makes it in.


Ben, why should asynchronous receive any type of restriction not already placed on code that runs some milliseconds after a key press (an existing hook). Cancelling asynchronous requests can lead to unpredictable behavior is probably never desirable. I've already experienced plugins that block for too long and what I do is, I just remove the plugin.

Ben Fritz

unread,
Sep 3, 2013, 2:52:58 PM9/3/13
to vim...@googlegroups.com
On Tuesday, September 3, 2013 1:17:59 PM UTC-5, Bjorn Tipling wrote:
> I am pretty excited about this patch and hope it makes it in.
>
>
> Ben, why should asynchronous receive any type of restriction not already placed on code that runs some milliseconds after a key press (an existing hook).

And most of these can be interrupted with CTRL-C. I am not certain whether the async functions will be since you added their processing to the functions which get keypresses (I think...I'm not actually familiar with most of Vim's code). I'm not saying they're not...only that the need to be. We've already had a few "Vim hangs on this regex" bugs in 7.4, and I could easily see a plugin that uses a search() call inside one of these timer functions. Or maybe it calls an external tool that is deadlocked for some reason. Or any number of other things could go wrong that the user doesn't want to wait for.

And I'm more worried about function that takes seconds, or minutes, to complete, than I am about milliseconds. My point was that users of these functions need to be aware that the action is not actually asynchronous in the sense of running in the background and allowing user input to continue in the foreground; you just mean it can be synced to a timer rather than a specific user input, correct?

> Cancelling asynchronous requests can lead to unpredictable behavior is probably never desirable. I've already experienced plugins that block for too long and what I do is, I just remove the plugin.
>

If you're typing something, suddenly Vim stops responding to any keyboard input, you've waited most of a minute for Vim to start responding, and you have unsaved work you wish to keep...it can be more desirable to interrupt the action and then continue using Vim long enough to save your work and restart.

Maybe if the user interrupts the action with CTRL-C you need to set a flag to remove the timer from the list, to prevent it firing off again in a few milliseconds?

kans

unread,
Sep 3, 2013, 2:53:31 PM9/3/13
to vim...@googlegroups.com
On Tuesday, September 3, 2013 9:04:40 AM UTC-7, Ben Fritz wrote:

Ben,

You are correct in that we added a timer to the main loop. Looking over the code once again, I think we should have altered the calls to select/poll instead, but lets discuss the practical effect of this patch since we can work out the details some time later.

Async does not imply parallelism nor safety- it only provides the illusion of concurrency. Idle cpu time spent waiting is instead used to perform some other task. Async frameworks make no such guarantee that every call not block the event loop; it is incumbent upon the programmer to ensure that the event loop is not blocked indefinitely. This is the definition and standard interface of async programming for every popular framework I can think of that lives in a scripting language- JavaScript, nodejs, twisted, event machine, and luvit.

The current alternative in Vim is to block forever, or worse yet, not write useful software because it is impossible to do so as a Vim plugin. With a proper settimeout, plugins have the option of returning control to the main loop and checking in latter. I'd suggest a settimeout on the settimeouts is not needed; plugins which cause too much trouble will either be fixed or uninstalled.

Ben Fritz

unread,
Sep 3, 2013, 4:07:00 PM9/3/13
to vim...@googlegroups.com
On Tuesday, September 3, 2013 1:53:31 PM UTC-5, kans wrote:
> On Tuesday, September 3, 2013 9:04:40 AM UTC-7, Ben Fritz wrote:
> > On Sunday, September 1, 2013 8:42:25 PM UTC-5, Geoff Greer wrote:
> > > This patch adds asynchronous functions to vimscript. If you want to perform an action in 700ms, simply:
> > >
> > > let timeout_id = settimeout(700, 'echo("hello")')
> > >
> > > To cancel the timeout before it's fired:
> > >
> > > canceltimeout(timeout_id)
> > >
> > > setinterval() also returns an id that can be used with canceltimeout.
> > >
> > > The reason for this patch is simple: asynchronous functionality is needed to implement real-time collaborative editing in Vim. This is one of the most voted-for features (see http://www.vim.org/sponsor/vote_results.php).
> > >
> > > Along with Matt Kaniaris, I founded Floobits to build real-time collaboration into every editor. We wrote a plugin for Vim, but we had to use hacks to get async behavior (abusing feedkeys or client-server). These methods had side-effects such as breaking leaderkeys or other shortcuts. After a lot of experimenting, we decided to try patching Vim.
> > >
> > > Since Vim is character-driven, we had to munge some low-level input functions to get the desired behavior. We changed gui_wait_for_chars() and mch_inchar() so that call_timeouts() is run every ticktime milliseconds. The default ticktime is 100ms.
> > >
> > > This patch isn't finished yet, but it works on unix-based OSes. If the reaction is positive, our intention is to change mch_inchar() (or something similar) in other OS-specific files. That will get async functions working for everyone.
> > >
> > > Even if our patch isn't the best approach, we'd love to help get async functions in Vim. Doing so will open the door to a lot of cool plugins.
> > >
> > > Oh, and this is the first time either myself or Matt have submitted a patch to Vim, so please be gentle.
> > >
> > > Sincerely,
> > >
> > > Geoff Greer
> >
> > Forgive me if I'm wrong, I didn't spend more than a few minutes looking over the patch.
> >
> > It looks like this is not really "asynchronous" behavior, but rather adding a timer-like hook into the main loop. Pro: no worries about thread safety, race conditions, etc. Con: if the function called by the timer event takes a long time, Vim hangs until it's done.
> >
> > If I'm correct, and this gets included, the help will need to be very explicit about making sure to limit the time taken by the function called! And there should definitely be a way to interrupt the action.
>
> Ben,
>
> You are correct in that we added a timer to the main loop. Looking over the code once again, I think we should have altered the calls to select/poll instead, but lets discuss the practical effect of this patch since we can work out the details some time later.
>

I like the practical implications of being able to respond to a timer event in a plugin. Polling is useful for many, many purposes, and doing it by shelling out and calling back with --remote calls is awkward at best.

> Async does not imply parallelism nor safety- it only provides the illusion of concurrency. Idle cpu time spent waiting is instead used to perform some other task. Async frameworks make no such guarantee that every call not block the event loop; it is incumbent upon the programmer to ensure that the event loop is not blocked indefinitely. This is the definition and standard interface of async programming for every popular framework I can think of that lives in a scripting language- JavaScript, nodejs, twisted, event machine, and luvit.
>

I'm more familiar with C/C++/Java, and even shell scripting, where "async" means it doesn't block the main program flow, as wikipedia says on http://en.wikipedia.org/wiki/Asynchrony: "Asynchronous actions are actions executed in a non-blocking scheme, allowing the main program flow to continue processing."

So, in my mind..."async" normally implies parallelism.

You've created an interface that is probably close enough, if developers are careful not to allow their timer callbacks to take too long, or use them to spawn and monitor a truly async process via the shell or python or whatever.

The nice thing is that you thus avoid the problem of concurrent access/thread safety, because you've not used multiple threads or processes at all. The not-so-nice thing is the inability to call long-running worker functions with your interface.

> The current alternative in Vim is to block forever, or worse yet, not write useful software because it is impossible to do so as a Vim plugin. With a proper settimeout, plugins have the option of returning control to the main loop and checking in latter.

I agree, we need something. That concurrent editing thing looked really neat! And doubtless it would be much more robust/easier to maintain with timer events in Vim.

> I'd suggest a settimeout on the settimeouts is not needed; plugins which cause too much trouble will either be fixed or uninstalled.

Possibly; but I'd rather not find the bug in the plugin by losing a lot of work when my Vim goes into an infinite loop that can't be interrupted.

kans

unread,
Sep 3, 2013, 7:31:53 PM9/3/13
to vim...@googlegroups.com

Ben,

Thanks for the explanation. End users will probably never directly call set_timeout. Hopefully, plugin authors know better than to make a call that can block forever. Good documentation will help.

Maybe control-c should cancel intervals. As you mentioned, this would let users save stuff and restart vim. At worse, plugins would be dumped into an inconsistent state that could cause more harm. Also, canceling internal callbacks is quite opaque to users. I don't see any great options for dealing with runaway plugins.

-kans


Thanks for explanation.

David Fishburn

unread,
Sep 3, 2013, 7:49:21 PM9/3/13
to vim_dev
...
Maybe control-c should cancel intervals.  As you mentioned, this would let users save stuff and restart vim.  At worse, plugins would be dumped into an inconsistent state that could cause more harm.  Also, canceling internal callbacks is quite opaque to users.  I don't see any great options for dealing with runaway plugins.


What I typically see if lots of users have many plugins installed.  It is very easy to forget what was installed last that could be leading to "strange" problems in Vim.  Anything running asynchronously I assume could easily fall under this category.

So yes, I believe an option or something to turn off asynchronous execution would be a great idea, as it would be one more step people on this list could suggest to see if the "symptom" goes away and allows the user to narrow their search to find the offending code.

My 2 cents.
David 

Bram Moolenaar

unread,
Sep 5, 2013, 7:26:41 AM9/5/13
to Geoff Greer, vim...@googlegroups.com

A few thoughts about this patch.

This is not asynchronous but these are timed commands. It should be
clear that the commands are executed in the "idle" loop, when Vim waits
for a character to be typed.

There might be two types: One that has high priority, and gets handled
even when the user has typed something. Another that has lower
priority, only gets handled when waiting for the user to type.

As someone already mentioned: If the command takes too much time, Vim
will get stuck. E.g. when it reads from a socket. It might be possible
to have this kind of work done in another thread, e.g. using Python.

The documentation is still missing. A lot of details need to be
explained there.

It must be possible to get a list of these commands, so that they can be
cancelled without having the id. This will then possibly break the
plugin that installed the command, so this gets messy quickly.

The code uses "long" to store msec, but you need 64 bits for that and
long can be 32 bits.

I don't see the need for 'ticktime'. The remaining time until the next
timeout can be computed. If it's smaller than 'updatetime' then wait
for that long.


--
hundred-and-one symptoms of being an internet addict:
170. You introduce your wife as "my_...@home.wife" and refer to your
children as "forked processes."

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Bjorn Tipling

unread,
Sep 5, 2013, 9:44:15 AM9/5/13
to vim...@googlegroups.com, Geoff Greer
> As someone already mentioned: If the command takes too much time, Vim
> will get stuck. E.g. when it reads from a socket. It might be possible
> to have this kind of work done in another thread, e.g. using Python.


Hey check this out:

au CursorHold * call BlockForever()


" Blocks forever
function! BlockForever()
let c = 1
while c >= 0
let c += 1
endwhile
endfunction

You can already do this blocking, I've already experienced it with plugins I've removed. I ask again why setTimeout should be treated differently than CursorHold? You can cntrl-c out of CursorHold, but just then make that a requirement for settimeout.


> This is not asynchronous but these are timed commands. It should be
> clear that the commands are executed in the "idle" loop, when Vim waits
> for a character to be typed.

As was already pointed out, this is what asynchronous execution looks like in JavaScript. JavaScript in the browser and elsewhere has only one thread of execution, and the behavior of setTimeout in these contexts is similar to this patch. Whatever you want to call this patch, it is very powerful, and vim is missing it. Try implementing a clock for the status line in current vim. It updates only after a keystroke. With this patch, you can have a real clock, that shows you the actual time! Not the time you last pressed a key.

Try this in a Chrome console:

function blockForever() {
var c = 1;
while (c > 0) {
c++;
}
}
window.setTimeout(blockForever, 1000);

You get the same behavior that you would from this patch, the same behavior that CursorHold gives you.

I agree there needs to be documentation on how to use this. I've built vim with this patch and was able to get one plugin that uses setinterval to work, but was not able to get my own attempt to work after a quick attempt.

On Thursday, September 5, 2013 4:26:41 AM UTC-7, Bram Moolenaar wrote:
> A few thoughts about this patch.
>
>
>
> This is not asynchronous but these are timed commands. It should be
>
> clear that the commands are executed in the "idle" loop, when Vim waits
>
> for a character to be typed.
>
>
>
> There might be two types: One that has high priority, and gets handled
>
> even when the user has typed something. Another that has lower
>
> priority, only gets handled when waiting for the user to type.
>
>
>
> As someone already mentioned: If the command takes too much time, Vim
>
> will get stuck. E.g. when it reads from a socket. It might be possible
>
> to have this kind of work done in another thread, e.g. using Python.
>
>
>
> The documentation is still missing. A lot of details need to be
>
> explained there.
>
>
>
> It must be possible to get a list of these commands, so that they can be
>
> cancelled without having the id. This will then possibly break the
>
> plugin that installed the command, so this gets messy quickly.
>
>
>
> The code uses "long" to store msec, but you need 64 bits for that and
>
> long can be 32 bits.
>
>
>
> I don't see the need for 'ticktime'. The remaining time until the next
>
> timeout can be computed. If it's smaller than 'updatetime' then wait
>
> for that long.
>
>
>
>
>
> --
>
> hundred-and-one symptoms of being an internet addict:
>

> 170. You introduce your wife as "my_la.wife" and refer to your
>
> children as "forked processes."
>
>
>
> /// Bram Moolenaar -- Bra....net -- http://www.Moolenaar.net \\\

Yasuhiro MATSUMOTO

unread,
Sep 5, 2013, 10:26:32 AM9/5/13
to vim...@googlegroups.com, Geoff Greer
We can use python's thread to get async, but vim don't have GIL. So it
can't get events safety from the thread.
> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups
> "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vim_dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>


--
- Yasuhiro Matsumoto

Ben Fritz

unread,
Sep 5, 2013, 12:08:16 PM9/5/13
to vim...@googlegroups.com, Geoff Greer
On Thursday, September 5, 2013 8:44:15 AM UTC-5, Bjorn Tipling wrote:
> > As someone already mentioned: If the command takes too much time, Vim
> > will get stuck. E.g. when it reads from a socket. It might be possible
> > to have this kind of work done in another thread, e.g. using Python.
>
>
> Hey check this out:
>
> au CursorHold * call BlockForever()
>
>
> " Blocks forever
> function! BlockForever()
> let c = 1
> while c >= 0
> let c += 1
> endwhile
> endfunction
>
> You can already do this blocking, I've already experienced it with plugins I've removed. I ask again why setTimeout should be treated differently than CursorHold? You can cntrl-c out of CursorHold, but just then make that a requirement for settimeout.
>

Exactly. I think as long as you can interrupt the timer with a keystroke it should be fine. But since you call your timer in between waiting for a keystroke, won't that prevent CTRL-C from interrupting it? Maybe it won't...I haven't tried it. I don't think we have any unique constraints, I just think that (a) it should be mentioned in the help that the timer blocks user input while processing, it's not a true asynchronous process and (b) it is required that CTRL-C or similar can interrupt the timer like it can already for autocmds, etc.

kans

unread,
Sep 5, 2013, 1:53:39 PM9/5/13
to vim...@googlegroups.com, Geoff Greer
On Thursday, September 5, 2013 9:08:16 AM UTC-7, Ben Fritz wrote:
> On Thursday, September 5, 2013 8:44:15 AM UTC-5, Bjorn Tipling wrote:
> > > As someone already mentioned: If the command takes too much time, Vim
> > > will get stuck. E.g. when it reads from a socket. It might be possible
> > > to have this kind of work done in another thread, e.g. using Python.
> >
> >
> > Hey check this out:
> >
> > au CursorHold * call BlockForever()
> >
> >
> > " Blocks forever
> > function! BlockForever()
> > let c = 1
> > while c >= 0
> > let c += 1
> > endwhile
> > endfunction
> >
> > You can already do this blocking, I've already experienced it with plugins I've removed. I ask again why setTimeout should be treated differently than CursorHold? You can cntrl-c out of CursorHold, but just then make that a requirement for settimeout.
> >
>
> Exactly. I think as long as you can interrupt the timer with a keystroke it should be fine. But since you call your timer in between waiting for a keystroke, won't that prevent CTRL-C from interrupting it? Maybe it won't...I haven't tried it.

ctrl-c works (in this patch) because the OS converts it into a SIGINT which vim knows how to handle.

>There might be two types: One that has high priority, and gets handled
even when the user has typed something. Another that has lower
priority, only gets handled when waiting for the user to type.

As mattn pointed out, this won't work unless the internals of vim are threadsafe.

> It must be possible to get a list of these commands, so that they can be
>
> cancelled without having the id.

This wouldn't be much work, but to my knowledge, no other implementation has this ability.

> I don't see the need for 'ticktime'. The remaining time until the next
>
> timeout can be computed. If it's smaller than 'updatetime' then wait
>
> for that long.

This is true. ticktime is only useful for end users if they want to trade off efficiency for a better timer function. 50ms is probably a sane default that is good enough (and once again, no other implementation lets this bubble up to user land).


Does anyone have any feedback on how this implemented; ie, putting this logic in mch_inchar vs. RealWaitForChar? Another issue is that vim makes other blocking calls that will block the timers.

kans

unread,
Sep 10, 2013, 5:54:38 PM9/10/13
to vim...@googlegroups.com, Geoff Greer

I've attached our latest patch. We moved some of the logic down to the level of poll/select (RealWaitForChar) using MAY_LOOP which is similar to what the scheme bindings do; it is cleaner and works better. We also changed the eval call to use do_cmdline_cmd which is far more convenient. Now, the timeouts don't have to be user defined functions. Also, all timeouts can now be disabled by setting 'ticktime' to -1. We discussed disabling intervals that fail, but we ultimately decided that the onus should be on programmers to make sane plugins and canceling intervals is unexpected behavior. Finally, the new features are now documented. Please let us know what you think.

async.patch

Nikolay Pavlov

unread,
Sep 10, 2013, 6:18:42 PM9/10/13
to vim...@googlegroups.com, Geoff Greer

Why do you keep calling it async?

Also other issues:
1. async.c does not follow coding style:
1.1. each 8 indentation spaces should be replaced with a tab
1.2. each opening figure brace should be present on its own line
1.3. no (void) in function declaration
2. async.h should be split into globals.h and proto/async.pro. async.h itself is to be removed.
3. Never use gettimeofday for intervals. There are special monotonic clocks in all operating system, for linux it is clock_gettime(CLOCK_MONOTONIC_RAW). What you use is subject for ntp daemon/ntpclient updates.
4. There is a proper way to disable a setting. Putting the whole setting into an #ifdef is not the one.
5. Documentation of 'ticktime' lacks references to +smth. I repeat, do not call smth "async". Not until you will make it execute asynchronously.
6. I do not see patch for the file with a list of features.

Thomas

unread,
Sep 11, 2013, 2:59:34 AM9/11/13
to vim...@googlegroups.com
On 11 September 2013 00:18, Nikolay Pavlov <zyx...@gmail.com> wrote:

Why do you keep calling it async?

Referring to the introduction to asynchronous programming linked below, the patch is an almost asynchronous feature, or at least a completition of the already existing asynchronous features of vim. With all the "autocmd-events" we already have the posibillity to run code asynchronous.
Anyway asynchronous doesn't mean threaded, as far as I know and referring again to the link. That a two different concepts (which you can combine if you want).

http://cs.brown.edu/courses/cs168/f12/handouts/async.pdf

Bjorn Tipling

unread,
Sep 11, 2013, 7:41:49 AM9/11/13
to vim...@googlegroups.com, wiene...@googlemail.com
The async issue seems like a bike shed. Whatever you want to call it, this feature is very powerful and I'm very grateful many are taking it serious. Let's focus on encouraging the devs and reviewers to getting a patch the community can accept. If the name really is an issue, I'm sure another name can work as well. Perhaps 'setinterval' or something like that.

Nikolay Pavlov

unread,
Sep 11, 2013, 10:46:06 AM9/11/13
to vim...@googlegroups.com

Now please show how you can run one sequence of commands interleaved with the other with this patch. Function that runs with given interval blocks everything until it completes, same for autocommands. You cannot say you implement preemptive multitasking if you have to wait for one task to complete before starting the other. It is not async feature.

You can, of course, do some hacks with saving a state, exiting and resuming, but you will then be forced to *emulate* preemption without any support from vim. I used to have some simple emulation even without such patch.

Note though that I would really like it merged. But not with the current name.

kans

unread,
Sep 11, 2013, 7:28:24 PM9/11/13
to vim...@googlegroups.com
ZyX,

The new patch addresses your criticism apart from changing the name. I believe we obey Vim code conventions everywhere. We have also implemented a monotonically increasing timer for unix and osx which are the only two OSes we have on hand. Otherwise, we fall back to gettimeofday. Async is only #defined if gettimeofday exists.

Bram,

We now use long longs for keeping track of the timers. We have also documented the new functions and some of their limitations.


Please let us know if you have further feedback.

-Matt

async.patch

Nikolay Pavlov

unread,
Sep 12, 2013, 4:59:51 AM9/12/13
to vim...@googlegroups.com

You can take the code from python: since 3.3 it has monotonic() function defined in C code in time module: pymonotonic function in Modules/timemodule.c. It has implementation for windows as well. Linux implementation used CLOCK_MONOTONIC (or, with higher priority, CLOCK_HIRES which is not mentioned in my clock_gettime man page; I guess there is a reason for it) without _RAW, though I would write

#ifdef CLOCK_MONOTONIC_RAW
                             CLOCK_MONOTONIC_RAW
#else
                             CLOCK_MONOTONIC
#endif

in function arguments. But it is not as severe as issue with gettimeofday: you cannot see 2 centuries hop with CLOCK_MONOTONIC, maximum slightly increased/decreased second duration when NTP daemon does his adjustments.

By the way, two pairs of figure braces may be removed from insert_timeout. Not that it is required by coding style, but this makes code look better.

> Bram,
>
> We now use long longs for keeping track of the timers.  We have also documented the new functions and some of their limitations.
>
>
> Please let us know if you have further feedback.
>
> -Matt
>

kans

unread,
Sep 13, 2013, 5:43:13 AM9/13/13
to vim...@googlegroups.com
ZyX,

I added a monotonic timer for windows; I basically took the one from python. I also changed all the make files to include the new stuff as well. The patch is 1200 lines long now (37 files changed, 476 insertions, 24 deletions). The current state includes monotonically increasing timers for OsX, Win32, and *nix. It really doesn't make sense to add any more features, but I'm happy to do cleanup as needed.

-Matt
async.monotonic.patch

kans

unread,
Sep 19, 2013, 6:01:17 AM9/19/13
to vim...@googlegroups.com
I was doing some work cleaning up the patch. It really does make sense to kill all intervals if one of them fails; its too annoying as an end user if a timeout fails every 50ms. The docstring for do_cmdline_cmd implies it should return FAIL if it can't evaluate "cmdline", but in practice it returns OK most of the time. For instance, evaluating a function that doesn't exist throws an error and returns OK. Some invalid expressions return OK, some, like "if (" don't. Does anyone know of an easy way to figure out of the expression could be evaluated and didn't throw?

-Matt

Nikolay Pavlov

unread,
Sep 19, 2013, 6:29:35 AM9/19/13
to vim...@googlegroups.com

I can suggest looking at VimTry* functions in src/if_py_both.h: they are used to transform all vim errors into python exceptions. Purge the python stuff and you will get C functions for wrapping arbitrary code into try/catch.

kans

unread,
Sep 21, 2013, 7:29:11 PM9/21/13
to vim...@googlegroups.com
Zyx,

Thanks for the help. Intervals are no longer run if they error of any sort (apart from a user interrupt which will likely happen accidentally). I was also forced to change configure.in as the monotonic timer on Linux needs to sometimes link against rt. I ran autoconf and also dumped those changes into auto/configure file. They run in both Linux and OsX.


Originally, I had hopped that this patch will be 100 lines tops, but it is closer to 600. A good chunk of that is documentation and touching the various make files. I think we have addressed all the concerns (apart from changing the name). What would it take to get this merged?

-Matt
async.stop_interval_errors.patch

Ben Fritz

unread,
Sep 21, 2013, 11:15:37 PM9/21/13
to vim...@googlegroups.com
On Saturday, September 21, 2013 6:29:11 PM UTC-5, kans wrote:
>
> Thanks for the help. Intervals are no longer run if they error of any sort (apart from a user interrupt which will likely happen accidentally).

So, if I install a plugin using this feature, and some weird sequence of events puts it into an infinite loop, I just need to kill Vim and lose my work?

I really, really want a way to force all timers to stop firing. Maybe not CTRL-C but SOMETHING should be possible for the user to kill misbehaving periodic tasks.

kans

unread,
Sep 22, 2013, 12:21:28 AM9/22/13
to vim...@googlegroups.com


Ben,

Ctrl-C will kill a given task, but it won't cancel future timeouts (if it was an interval). Ctrl-C can't do that or else it would happen accidentally since the end user should never notice timers firing to begin with. I'm open to suggestions, but you are asking for a solution to a problem that doesn't exist in any other text editor or IDE.

-Matt

Bram Moolenaar

unread,
Sep 22, 2013, 9:38:32 AM9/22/13
to kans, vim...@googlegroups.com

kans wrote:

> Zyx,
>
> Thanks for the help. Intervals are no longer run if they error of any
> sort (apart from a user interrupt which will likely happen
> accidentally). I was also forced to change configure.in as the
> monotonic timer on Linux needs to sometimes link against rt. I ran
> autoconf and also dumped those changes into auto/configure file. They
> run in both Linux and OsX.
>
>
> Originally, I had hopped that this patch will be 100 lines tops, but
> it is closer to 600. A good chunk of that is documentation and
> touching the various make files. I think we have addressed all the
> concerns (apart from changing the name). What would it take to get
> this merged?

I still have several problems with including this functionality.

I would prefer this to be called timed events instead of async, since
it's not really asynchronous. A timeout is for when you do something
and it takes too long. What this is doing is setting a timer or a
repeated timer.

There is no need for 'ticktime', one can compute the time until the next
event and use that. It's also more accurate and has less overhead.

Like others I'm worried that some plugin creates a timer with a problem,
and it keeps triggering over and over. The user must have a way to stop
it. Also, there will be plugins that have interference with what some
timer is doing, there needs to be a way to temporary disable them.
It might be required to give the timers a name instead of an ID, so that
specific timers can be disabled by name, instead of having to lookup
their ID (how would one do that anyway?).

What happens with timers when at a prompt, e.g. for ":s/pat/repl/gc"?


This is the kind of functionality where we may need to fix many bugs
that are uncovered over time, like with the conceal feature. I am not
going to include it before I'm convinced it works properly.

--
hundred-and-one symptoms of being an internet addict:
250. You've given up the search for the "perfect woman" and instead,
sit in front of the PC until you're just too tired to care.

Ben Fritz

unread,
Sep 23, 2013, 10:48:35 AM9/23/13
to vim...@googlegroups.com

That's because other editors and IDEs that have periodic timers don't freeze all user input while waiting for the periodic task to finish.

You're right about CTRL-C canceling all future timeouts would be a bad thing, since that would happen accidentally more than on purpose.

So probably a new keyboard command is needed. It can't be just a function or ex command, because if user input is frozen due to a bad timer, the user will never see the command-line.

Or, if a timer is interrupted with CTRL-C, you could prompt to continue running timers. Kind of like some browsers prompt you "an error occurred running javascript, continue running scripts on this page?" when they load a bad website.

Geoff Greer

unread,
Sep 23, 2013, 5:48:48 PM9/23/13
to vim...@googlegroups.com, kans
On Sunday, September 22, 2013 6:38:32 AM UTC-7, Bram Moolenaar wrote:
> kans wrote:
>
>
>
> > Zyx,
>
> >
>
> > Thanks for the help. Intervals are no longer run if they error of any
>
> > sort (apart from a user interrupt which will likely happen
>
> > accidentally). I was also forced to change configure.in as the
>
> > monotonic timer on Linux needs to sometimes link against rt. I ran
>
> > autoconf and also dumped those changes into auto/configure file. They
>
> > run in both Linux and OsX.
>
> >
>
> >
>
> > Originally, I had hopped that this patch will be 100 lines tops, but
>
> > it is closer to 600. A good chunk of that is documentation and
>
> > touching the various make files. I think we have addressed all the
>
> > concerns (apart from changing the name). What would it take to get
>
> > this merged?
>
>
>
> I still have several problems with including this functionality.
>
>
>
> I would prefer this to be called timed events instead of async, since
>
> it's not really asynchronous. A timeout is for when you do something
>
> and it takes too long. What this is doing is setting a timer or a
>
> repeated timer.
>

We can change the vocabulary if you like, but we wanted to follow a pattern that developers were already familiar with. We used the JavaScript names of setTimeout/setInterval/cancelTimeout. Sublime Text uses the same names. Emacs has a similar API.

>
>
> There is no need for 'ticktime', one can compute the time until the next
>
> event and use that. It's also more accurate and has less overhead.
>

Matt is working on a fix for this.

>
> Like others I'm worried that some plugin creates a timer with a problem,
>
> and it keeps triggering over and over. The user must have a way to stop
>
> it.

We'll change ctrl+c so that it stops future intervals as well as canceling the current timer. That should prevent Vim from hanging and let users save their work.

We thought about something more complicated, such as showing a list of pending timers and letting the user cancel them one-by-one. Unfortunately, users can only guess at the problem timer(s). They'd need the equivalent of `top` to find the culprit(s). Worse, canceling the wrong one could easily break a well-behaved plugin. The only fix for the well-behaved plugin would be to restart Vim.

Most other editors and frameworks have taken the stance that with great power comes great responsibility. If a timer causes problems, then it causes problems. Fault lies with the developer of the code that set the timer. Sublime Text, Emacs, JavaScript, Twisted Python (and more) work this way.

> Also, there will be plugins that have interference with what some
>
> timer is doing, there needs to be a way to temporary disable them.
>
> It might be required to give the timers a name instead of an ID, so that
>
> specific timers can be disabled by name, instead of having to lookup
>
> their ID (how would one do that anyway?).
>

I think names for timers would cause more problems than they solve. Plugins could conflict with each other if they chose the same names for their timers. IDs assigned at runtime make conflicts impossible. Also, the pattern of returning an ID is not foreign to those who have used other setTimeout implementations.

>
>
> What happens with timers when at a prompt, e.g. for ":s/pat/repl/gc"?
>
>

Timers are run while the prompt is prompting. It's the responsibility of the plugin developer to not cause problems. Great power, great responsibility and all that.

>
>
>
> This is the kind of functionality where we may need to fix many bugs
>
> that are uncovered over time, like with the conceal feature. I am not
>
> going to include it before I'm convinced it works properly.
>
>

I don't think there will be many issues, but I do see where you're coming from. You're responsible for maintaining Vim indefinitely, while people like Matt and myself can submit a patch and disappear. A conservative approach to new features is entirely understandable. If I were in your position, I'd probably behave similarly.

But please be aware of the costs of *not* adding timers to Vim. Without them, whole categories of plugins are simply impossible: Indexing files in the background. Collaborative editing. Deep integration with debuggers and browsers. That's just what I can imagine in a few minutes. The entire Vim community could imagine (and build) so much more.

There's a new generation of plugins waiting to be made. They just need timers. I hope you'll soften your stance on this patch.

Sincerely,

Geoff Greer
http://geoff.greer.fm/
https://github.com/ggreer

kans

unread,
Sep 23, 2013, 8:53:26 PM9/23/13
to vim...@googlegroups.com, kans
On Sunday, September 22, 2013 6:38:32 AM UTC-7, Bram Moolenaar wrote:
> kans wrote:
>
>
>
> > Zyx,
>
> >
>
> > Thanks for the help. Intervals are no longer run if they error of any
>
> > sort (apart from a user interrupt which will likely happen
>
> > accidentally). I was also forced to change configure.in as the
>
> > monotonic timer on Linux needs to sometimes link against rt. I ran
>
> > autoconf and also dumped those changes into auto/configure file. They
>
> > run in both Linux and OsX.
>
> >
>
> >
>
> > Originally, I had hopped that this patch will be 100 lines tops, but
>
> > it is closer to 600. A good chunk of that is documentation and
>
> > touching the various make files. I think we have addressed all the
>
> > concerns (apart from changing the name). What would it take to get
>
> > this merged?
>
>
>
> I still have several problems with including this functionality.
>
>
>
> I would prefer this to be called timed events instead of async, since
>
> it's not really asynchronous. A timeout is for when you do something
>
> and it takes too long. What this is doing is setting a timer or a
>
> repeated timer.

The named has been changed from async to timers everywhere.

>
>
> There is no need for 'ticktime', one can compute the time until the next
>
> event and use that. It's also more accurate and has less overhead.

Agreed. 'ticktime' is now the minimum time between calls to call_timeouts() and is set to 20ms. Vim waits the appropriate amount of time (which could be ticktime, -1, 0, or the time until the next timeout). Say we have intervals every .5ms; the overhead of calling select that often is way too high given we don't care about that level of precision.


>
>
> Like others I'm worried that some plugin creates a timer with a problem,
>
> and it keeps triggering over and over. The user must have a way to stop
>
> it.

The new behavior cancels intervals if they error in any way. Additionally, upon futher thought, there is no way around ctrl-C. That is, we have to use signals and ctrl-c (SIGINT) is the only convenient one. In the newest version, ctrl-c will cancel a given timeout/interval and all future calls.

> Also, there will be plugins that have interference with what some
>
> timer is doing, there needs to be a way to temporary disable them.

The new solution is good enough from the user's perspective. If a rogue timer exists, the only possible action is to cancel it, save buffers, and restart vim after removing the offending plugin. No plugin is going to continue to work after its timers are canceled.

Disabling a specific timer would violate internal assumptions plugins make. We could disable all timers for X seconds, but this functionality isn't necessary and would result in plugins battling each other over when, if ever, timeouts should be called.

>
> It might be required to give the timers a name instead of an ID, so that
>
> specific timers can be disabled by name, instead of having to lookup
>
> their ID (how would one do that anyway?).

I don't follow. Letting plugins cancel non-local timeouts will result in plugin wars, unintentional or otherwise. I don't ever see a user canceling a given timeout based on some list; at the very least, its a poor user experience. Why should users ever need to know timers exist in the first place? Allowing mutable global state with side effects *across plugins* causes more problems than it solves.

async.patch

David Fishburn

unread,
Sep 24, 2013, 8:51:44 AM9/24/13
to vim_dev

On Mon, Sep 23, 2013 at 8:53 PM, kans <mkan...@gmail.com> wrote:
...
 
> Like others I'm worried that some plugin creates a timer with a problem,
>
> and it keeps triggering over and over.  The user must have a way to stop
>
> it.

The new behavior cancels intervals if they error in any way.  Additionally, upon further thought, there is no way around ctrl-C.  That is, we have to use signals and ctrl-c (SIGINT) is the only convenient one.  In the newest version, ctrl-c will cancel a given timeout/interval and all future calls.


FYI, I often use CTRL-C when Vim is taking too long (for example, my home drive is N: and I am not connected to the network).  I understand why we want this, but I am just wondering how often people use CTRL-C within Vim.

What I would _very_ much like to see is when CTRL-C is pressed, and there are timed events, Vim would display the name of each timed event canceled.  I would also like to see future events differentiated.

CTRL-C
Cancelled 21_DB_timedEvent
Future events cancelled
21_DB_timedEvent
34_SS_MyTimedEvent
87_GG_hisTimedEvent

That way, if I see a pattern that I am a canceling the same event over and over, I can start vimgrepping my plugin / autoload directories to figure out what plugin is installing that timed event.

 
> Also, there will be plugins that have interference with what some
>
> timer is doing, there needs to be a way to temporary disable them.

The new solution is good enough from the user's perspective.  If a rogue timer exists, the only possible action is to cancel it, save buffers, and restart vim after removing the offending plugin.  No plugin is going to continue to work after its timers are canceled.

If the CTRL-C shows the above information, we can at least identify potential issue plugins.
 
I think when we document this feature in the Vim Help files, it is important to indicated every timed event should reference a GLOBAL variable which will allow the plugin to disable the timed event.

So, similar to how plugin developers follow the standard:

if exists('g:loaded_myplugin') 
    finish
endif
let g:loaded_myplugin = 1

In our timed event sample (prominently displayed) in the help we could have:

let g:myplugin_allow_timedevent = 1

function! MyPlugin_timedEvent() 
    if g:myplugin_allow_timedevent == 0 
        return
    endif

     -- Do stuff here
endfunction 

This might help (especially) first time plugin writers of the timed event feature to put in some safe guards ahead of time.


David

Matthew Kaniaris

unread,
Sep 24, 2013, 3:35:26 PM9/24/13
to vim...@googlegroups.com
David,

Is it currently possible to introspect the file and line number of call site of sitetimeout?  I think it would be enough to just error message that info when a timer is canceled.

-Matt


--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
 
---
You received this message because you are subscribed to a topic in the Google Groups "vim_dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_dev/-4pqDJfHCsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vim_dev+u...@googlegroups.com.

Bram Moolenaar

unread,
Sep 25, 2013, 7:53:25 AM9/25/13
to Geoff Greer, vim...@googlegroups.com, kans

Geoff Greer wrote:


> On Sunday, September 22, 2013 6:38:32 AM UTC-7, Bram Moolenaar wrote:
> > I still have several problems with including this functionality.
> >
> > I would prefer this to be called timed events instead of async, since
> > it's not really asynchronous. A timeout is for when you do something
> > and it takes too long. What this is doing is setting a timer or a
> > repeated timer.
>
> We can change the vocabulary if you like, but we wanted to follow a
> pattern that developers were already familiar with. We used the
> JavaScript names of setTimeout/setInterval/cancelTimeout. Sublime Text
> uses the same names. Emacs has a similar API.

Java is the most used language these days and uses the Timer class.
JavaScript wasn't designed properly, it has lots of confusing stuff, I
would never use it as an example for proper design. Only for ideas.
What is Emacs? :-)
Your attitude appears to be that you are making things work. Well,
unfortunately things break once in a while. At some point in this
design you have to think about what might go wrong and how to handle
that properly. This is not something to be done as an afterthought.

Plugins WILL interfere in ways you cannot imagine. It's naive to assume
that they will all work properly. And the plugin writer cannot possibly
overview all corner cases, no matter how clever he is. We must prepare
for some plugin, or combination of plugins, to malfunction and provide
the user with a way out.

At least it must be possible for the user to find out what is causing
problems, find the source of it and fix it or disable it. That is why
we have the :verbose command, for example to find out who last set an
option. For timers there must be at least a command that lists all
active timers, when they trigger and where they were defined.

Interrupting a timer with CTRL-C is hit-and-miss, by the time the key
press gets through another timer may be active and you kill the wrong
one. Having a prompt to ask the user what he wants to kill might help.
But also interrupt the normal working sequence. And on Windows CTRL-C
means copy...

> > This is the kind of functionality where we may need to fix many bugs
> > that are uncovered over time, like with the conceal feature. I am not
> > going to include it before I'm convinced it works properly.
>
> I don't think there will be many issues, but I do see where you're
> coming from. You're responsible for maintaining Vim indefinitely,
> while people like Matt and myself can submit a patch and disappear. A
> conservative approach to new features is entirely understandable. If I
> were in your position, I'd probably behave similarly.
>
> But please be aware of the costs of *not* adding timers to Vim.
> Without them, whole categories of plugins are simply impossible:
> Indexing files in the background. Collaborative editing. Deep
> integration with debuggers and browsers. That's just what I can
> imagine in a few minutes. The entire Vim community could imagine (and
> build) so much more.

You are exactly mentioning things that can go haywire:
- Indexing in the background may hang, run at 100% CPU, change files
that are being used, etc.
- Collaborative editing will use network functions, which may hang.
- Debugger integration is very complex, anything can go wrong.

> There's a new generation of plugins waiting to be made. They just need
> timers. I hope you'll soften your stance on this patch.

I won't include it before it has been done properly.

--
Women are probably the main cause of free software starvation.

Nikolay Pavlov

unread,
Sep 25, 2013, 8:20:36 AM9/25/13
to vim...@googlegroups.com, kans, Geoff Greer

I would also add information about times to such command: it or another command should list average, maximum and time consumed for last call for debugging purposes. Possibly with the number of times they were triggered and maybe when they were defined (in format "N time ago" (e.g. "1 minute 5 seconds ago" and not "25.09.2013 16:16")) (assuming timers are constantly added and deleted it is good to know that timer A was defined nearly when you started to experience input lags).

> Interrupting a timer with CTRL-C is hit-and-miss, by the time the key
> press gets through another timer may be active and you kill the wrong
> one.  Having a prompt to ask the user what he wants to kill might help.
> But also interrupt the normal working sequence.  And on Windows CTRL-C
> means copy...

Is not interrupt happening because the job done by timer did hang? This way interrupting wrong timer is much unlikely.

>
> > > This is the kind of functionality where we may need to fix many bugs
> > > that are uncovered over time, like with the conceal feature.  I am not
> > > going to include it before I'm convinced it works properly.
> >
> > I don't think there will be many issues, but I do see where you're
> > coming from. You're responsible for maintaining Vim indefinitely,
> > while people like Matt and myself can submit a patch and disappear. A
> > conservative approach to new features is entirely understandable. If I
> > were in your position, I'd probably behave similarly.
> >
> > But please be aware of the costs of *not* adding timers to Vim.
> > Without them, whole categories of plugins are simply impossible:
> > Indexing files in the background. Collaborative editing. Deep
> > integration with debuggers and browsers. That's just what I can
> > imagine in a few minutes. The entire Vim community could imagine (and
> > build) so much more.
>
> You are exactly mentioning things that can go haywire:
> - Indexing in the background may hang, run at 100% CPU, change files
>   that are being used, etc.

Since the patch does not add preemptive multitasking I doubt anybody sane will write indexing in the background using these timers. But they are handy at updating in-app copy of index.

In fact if I was writing plugin doing any job mentioned here I would use timers solely for updating state after all necessary computations were done in some other thread/process. Though you cannot and should not expect other plugin writers share the same thoughts.

> - Collaborative editing will use network functions, which may hang.
> - Debugger integration is very complex, anything can go wrong.
>
> > There's a new generation of plugins waiting to be made. They just need
> > timers. I hope you'll soften your stance on this patch.
>
> I won't include it before it has been done properly.
>
> --
> Women are probably the main cause of free software starvation.
>
>  /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net   \\\
> ///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\  an exciting new programming language -- http://www.Zimbu.org        ///
>  \\\            help me help AIDS victims -- http://ICCF-Holland.org    ///
>

> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---

> You received this message because you are subscribed to the Google Groups "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.

Tux.

unread,
Sep 25, 2013, 10:11:58 AM9/25/13
to vim...@googlegroups.com
Bram Moolenaar <Br...@moolenaar.net> schrieb:


>Java is the most used language these days and uses the Timer class.

"Most people use Java" (which I doubt as better programmers prefer sane languages like C) is not a reason to state that Java would be any better than other languages. They just have a working Marketing department.
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Tux.

unread,
Sep 25, 2013, 10:10:15 AM9/25/13
to vim...@googlegroups.com




Bram Moolenaar <Br...@moolenaar.net> schrieb:

>Java is the most used language these days and uses the Timer class.

Bjorn Tipling

unread,
Sep 25, 2013, 10:21:20 AM9/25/13
to vim...@googlegroups.com, Geoff Greer, kans
Bram,

Given that your concern is to not present a situation to the user for which they have no remedy, it seems like providing information might be a possible remedy The more information the user has, the easier it might be to figure out how to get out of a possible sticky situation. You mentioned being able to list a list of names of timers might be a workable solution for you to get this patch in? Maybe being able to remove a timer by name? Would these additions suffice? Just curious. I'm not involved in the development of this plugin, but I would like to see it make it into vim. Maybe the timer can print its name as it starts to :messages so the user might have an idea as to what is currently happening if things hang.

kans

unread,
Sep 25, 2013, 7:30:15 PM9/25/13
to vim...@googlegroups.com, Geoff Greer, kans
> Java is the most used language these days and uses the Timer class.
>
> JavaScript wasn't designed properly, it has lots of confusing stuff, I
>
> would never use it as an example for proper design. Only for ideas.

Its true that JS was designed in a weekend, but it is easily the most popular event driven language (where asynchronous programming is the norm). We chose setTimeout and setInterval as our model because they are the closest thing to what we wanted, are familiar to most devs, and can be reasonably implemented inside Vim. Also, they aren't one of the bad parts (http://oreilly.com/javascript/excerpts/javascript-good-parts/bad-parts.html) :P

>
> > Timers are run while the prompt is prompting. It's the responsibility
>
> > of the plugin developer to not cause problems. Great power, great
>
> > responsibility and all that.
>
>
>
> Your attitude appears to be that you are making things work. Well,
>
> unfortunately things break once in a while. At some point in this
>
> design you have to think about what might go wrong and how to handle
>
> that properly. This is not something to be done as an afterthought.

I recognize that things do go wrong, but I don't expect end users to do anything about it other than exit and change their ~/.vimrc. Fixing broken plugins and writing sane ones is squarely the responsibility of the plugin authors. Installing plugins (working or otherwise) is up to end users, and this has always been the case. I don't see why this patch should be treated any differently.

If there is no way to diagnose or get out of infinite loops in general in Vim, what necessitates such a system exist here? Currently, any plugin can trivially create a busy loop that blocks forever. It is the job of the plugin authors to write code that doesn't do that.

> At least it must be possible for the user to find out what is causing
>
> problems, find the source of it and fix it or disable it. That is why
>
> we have the :verbose command, for example to find out who last set an
>
> option. For timers there must be at least a command that lists all
>
> active timers, when they trigger and where they were defined.

I disagree that a list of active timers is particularly helpful to end users for the following reasons:

1. Timers aren't like options or autocommands which are both static. They are mostly set at startup and never changed. autocommands exist on the human time scale. Timers come and go on the computer time scale. A well behaving plugin could realistically run a timer ever 30ms on the low end without ever bothering the user.
2. How will the user see the list of timers if Vim is blocked by a timer? Diagnosing troublesome timers is the only time a user would want this feature.
3. How will the user identify which timer is causing the slowdown? A timeout is only fired once; previous (troublesome) timeouts won't be in the list. Moreover, the list can be huge.
4. How does the end user even know that a timer is the problem in the first place? From his perspective, all he sees is that vim is unresponsive.
5. Canceling the wrong timer will lead to even more trouble (unlike autocommands which have no expectation of ever firing) and will make diagnosing the problem even harder.
6. The ability to cancel third party timers is dangerous. Imagine if different pages in a web browser could cancel timers they didn't set.

Perhaps this feature would be OK at helping the end user diagnose slightly problematic intervals- ie, ones that block for a second or two, but it wouldn't do anything for infinite loops. I tend to think its easy enough to diagnose that problem without this feature.

If a timers list is a hard requirement, I'm willing to write the code, but I don't think it adds much.

> Interrupting a timer with CTRL-C is hit-and-miss, by the time the key
>
> press gets through another timer may be active and you kill the wrong
>
> one. Having a prompt to ask the user what he wants to kill might help.
>
> But also interrupt the normal working sequence. And on Windows CTRL-C
>
> means copy...

Yeah, this is trouble and I'm open to suggestions. What does Vim currently offer for letting users get out of infinite loops? I would like to implement a solution, but I can't think of a satisfactory one that would take less than a month to write. I would guess the difficulty in implementing something like this is why it doesn't already exist (for unresponsive vim scripts).

-Matt

Nikolay Pavlov

unread,
Sep 26, 2013, 12:09:48 AM9/26/13
to vim...@googlegroups.com, kans, Geoff Greer


On Sep 26, 2013 3:30 AM, "kans" <mkan...@gmail.com> wrote:
>
> > Java is the most used language these days and uses the Timer class.
> >
> > JavaScript wasn't designed properly, it has lots of confusing stuff, I
> >
> > would never use it as an example for proper design.  Only for ideas.
>
> Its true that JS was designed in a weekend, but it is easily the most popular event driven language (where asynchronous programming is the norm).  We chose setTimeout and setInterval as our model because they are the closest thing to what we wanted, are familiar to most devs, and can be reasonably implemented inside Vim.  Also, they aren't one of the bad parts (http://oreilly.com/javascript/excerpts/javascript-good-parts/bad-parts.html) :P
>
> >
> > > Timers are run while the prompt is prompting. It's the responsibility
> >
> > > of the plugin developer to not cause problems. Great power, great
> >
> > > responsibility and all that.
> >
> >
> >
> > Your attitude appears to be that you are making things work.  Well,
> >
> > unfortunately things break once in a while.  At some point in this
> >
> > design you have to think about what might go wrong and how to handle
> >
> > that properly.  This is not something to be done as an afterthought.
>
> I recognize that things do go wrong, but I don't expect end users to do anything about it other than exit and change their ~/.vimrc. Fixing broken plugins and writing sane ones is squarely the responsibility of the plugin authors. Installing plugins (working or otherwise) is up to end users, and this has always been the case. I don't see why this patch should be treated any differently.
>
> If there is no way to diagnose or get out of infinite loops in general in Vim, what necessitates such a system exist here? Currently, any plugin can trivially create a busy loop that blocks forever. It is the job of the plugin authors to write code that doesn't do that.

This is  only true for python. Infinite loop can be canceled by CTRL-C if it is written in VimL and author is not intercepting interrupts just to make the loop unkillable.

> > At least it must be possible for the user to find out what is causing
> >
> > problems, find the source of it and fix it or disable it. That is why
> >
> > we have the :verbose command, for example to find out who last set an
> >
> > option.  For timers there must be at least a command that lists all
> >
> > active timers, when they trigger and where they were defined.
>
> I disagree that a list of active timers is particularly helpful to end users for the following reasons:
>
> 1. Timers aren't like options or autocommands which are both static. They are mostly set at startup and never changed. autocommands exist on the human time scale.  Timers come and go on the computer time scale.  A well behaving plugin could realistically run a timer ever 30ms on the low end without ever bothering the user.

Plugin should not be expected to be well behaving.

> 2. How will the user see the list of timers if Vim is blocked by a timer? Diagnosing troublesome timers is the only time a user would want this feature.

He will use CTRL-C. Then see the list.

> 3. How will the user identify which timer is causing the slowdown? A timeout is only fired once; previous (troublesome) timeouts won't be in the list. Moreover, the list can be huge.

Use string identifiers as suggested. This is how you can collect stats I said to be fine to see in previous message.

> 4. How does the end user even know that a timer is the problem in the first place? From his perspective, all he sees is that vim is unresponsive.

If he is not already experienced he will be asked to launch command that shows stats when he comes with his problem to stackoverflow.

> 5. Canceling the wrong timer will lead to even more trouble (unlike autocommands which have no expectation of ever firing) and will make diagnosing the problem even harder.

You can undefine plugin functions. If one wants to make problems for the plugin he already has enough capabilities to do this.

> 6. The ability to cancel third party timers is dangerous. Imagine if different pages in a web browser could cancel timers they didn't set.

Commands, functions, autocommands, mappings - they all can be undefined/altered by a third party timers. This is less dangerous then expecting deferred function to take 30 ms when launched.

>
> Perhaps this feature would be OK at helping the end user diagnose slightly problematic intervals- ie, ones that block for a second or two, but it wouldn't do anything for infinite loops.  I tend to think its easy enough to diagnose that problem without this feature.
>
> If a timers list is a hard requirement, I'm willing to write the code, but I don't think it adds much.
>
> > Interrupting a timer with CTRL-C is hit-and-miss, by the time the key
> >
> > press gets through another timer may be active and you kill the wrong
> >
> > one.  Having a prompt to ask the user what he wants to kill might help.
> >
> > But also interrupt the normal working sequence.  And on Windows CTRL-C
> >
> > means copy...
>
> Yeah, this is trouble and I'm open to suggestions. What does Vim currently offer for letting users get out of infinite loops? I would like to implement a solution, but I can't think of a satisfactory one that would take less than a month to write. I would guess the difficulty in implementing something like this is why it doesn't already exist (for unresponsive vim scripts).

I am wondering why you are acting  here like if CTRL-C does not work. From my experience this always helps against problematic VimL scripts (but not python plugins; likely ruby/lua/perl/... have the same property).

It makes sense though to collect such stats only on request. Currently there is only one thing that can add problems without easy method (I mean profiling) to determine that it is it: statusline that does not use VimL functions. AFAIR you execute Ex commands in timer. If I am right how one is going to debug if problematic timer is running only :ruby $command_t.update_index (an example, I do not claim Command-T has such method) (except for the usual disabling of the plugins)? Also are timers run shown when you :set verbose to some high enough value?

It would be fine if such stats collection functions are written in a way that can be later attached to autocommands: if e.g. InsertCharPre event fires only some python/... script this will also come unprofilable and only seen in verbose mode.

Note that stats provide only information for the user. Developer in such cases will use in-lang profiling capabilities.

> -Matt

kans

unread,
Sep 26, 2013, 5:53:55 PM9/26/13
to vim...@googlegroups.com, kans, Geoff Greer
ZyX,

As Bram pointed out ctrl-c doesn't work on windows. I'm not going to try to fix that.

Also, as Bram pointed out, ctrl-c breaking the normal flow (to ask the user if he really wanted to cancel a timeout) is annoying. Instead, we print a warning when a user cancels a timer via ctrl-c.

The newest patch includes an ex command that lists all pending timeouts and a bunch of meta info around them. We could add extra info about running times, but the current patch is already 1400 lines is good enough.

-Matt
show_timers.patch

Ben Fritz

unread,
Sep 27, 2013, 11:04:25 AM9/27/13
to vim...@googlegroups.com, kans, Geoff Greer
On Thursday, September 26, 2013 4:53:55 PM UTC-5, kans wrote:
>
> As Bram pointed out ctrl-c doesn't work on windows. I'm not going to try to fix that.
>

CTRL-C works on Windows as long as a user doesn't source the infamous mswin.vim somewhere in their config. Or if they do, they need to edit out the mapping for CTRL-C.

kans

unread,
Oct 2, 2013, 6:40:05 PM10/2/13
to vim...@googlegroups.com, kans, Geoff Greer


I believe we have addressed all major concerns- the last one being the ex_timers command which shows all pending timers. If we have missed something, or you have other concerns, please let us know. As I wrote before, we'd really like to see this feature in Vim.

-Matt

kans

unread,
Oct 8, 2013, 6:16:55 PM10/8/13
to vim...@googlegroups.com, kans, Geoff Greer


We recently stumbled into one strange bug with timers. Everything we've thrown at them works as expected apart from :Explore. When run in a timer, netrw stuff slows down by about two orders of magnitude. The commands for timers are executed like so with no other setup:

do_cmdline_cmd(timeouts->cmd)

None of the extra time is spent in self and the cpu usage doesn't spike. There isn't anything too crazy in Instruments.app (strace). Does anyone have any idea about the cause? I've included some basis profiling info below:


:call settimeout(0, "Explore ~")
count total (s) self (s) function
1 2.620059 0.027059 netrw#Explore()
1 2.593000 0.000131 netrw#LocalBrowseCheck()
1 2.446097 0.000347 <SNR>26_NetrwBrowse()
1 1.443353 0.062179 <SNR>26_PerformListing()
1 0.978938 0.000448 <SNR>26_NetrwGetBuffer()
1 0.978304 0.000426 <SNR>26_NetrwEnew()
3 0.957588 0.001165 <SNR>26_NetrwOptionRestore()
1 0.748296 <SNR>26_NetrwSetSort()
1 0.271453 <SNR>26_NetrwListHide()
2 0.168370 0.042646 <SNR>13_SynSet()
1 0.128560 0.128315 <SNR>26_LocalListing()
1 0.126958 0.000222 <SNR>26_NetrwSafeOptions()
2 0.125646 netrw#NetrwRestorePosn()
1 0.083888 0.021126 netrw#NetrwSavePosn()
2 0.041997 <SNR>10_LoadFTPlugin()
1 0.021434 0.021221 <SNR>26_NetrwBookHistRead()
68 0.001621 <SNR>26_NetrwInit()
1 0.001439 0.001433 <SNR>26_NetrwMaps()
2 0.000417 <SNR>26_NetrwOptionSave()
1 0.000228 <SNR>26_LocalFastBrowser()


:Explore ~
count total (s) self (s) function
1 0.021241 0.006975 netrw#Explore()
1 0.014266 0.000074 netrw#LocalBrowseCheck()
1 0.013968 0.000296 <SNR>26_NetrwBrowse()
1 0.008547 0.000448 <SNR>26_PerformListing()
1 0.003741 0.003255 <SNR>26_NetrwBookHistSave()
1 0.003012 0.000424 <SNR>26_NetrwGetBuffer()
1 0.002976 0.002806 <SNR>26_LocalListing()
2 0.002871 0.000588 <SNR>26_NetrwEnew()
4 0.002356 0.001092 <SNR>26_NetrwOptionRestore()
1 0.001364 0.001358 <SNR>26_NetrwMaps()
1 0.001356 0.000218 <SNR>26_NetrwSafeOptions()
2 0.001330 0.000472 <SNR>13_SynSet()
1 0.001310 <SNR>26_NetrwSetSort()
68 0.001304 <SNR>26_NetrwInit()
2 0.000763 <SNR>10_LoadFTPlugin()
1 0.000630 0.000156 <SNR>4_mergelists()
3 0.000572 <SNR>26_NetrwOptionSave()
1 0.000393 0.000133 ctrlp#mrufiles#cachefile()
1 0.000334 0.000008 <SNR>4_savetofile()
1 0.000326 0.000313 ctrlp#utils#writecache()

mattn

unread,
Oct 9, 2013, 6:36:24 AM10/9/13
to vim...@googlegroups.com, kans, Geoff Greer

Bjorn T

unread,
Oct 9, 2013, 2:36:18 PM10/9/13
to vim...@googlegroups.com, kans, Geoff Greer
They've gone ahead and implemented pretty much everything that was asked for. Geoff and kans did a lot of work. That effort is at least worth a response in this thread why the patch isn't going to be accepted, even though all that was asked for was added to the patch. I don't understand the rational.

B

Bram Moolenaar

unread,
Oct 9, 2013, 5:26:23 PM10/9/13
to mattn, vim...@googlegroups.com, kans, Geoff Greer
There are several changes. Can you quote the one you refer to?


--
A computer programmer is a device for turning requirements into
undocumented features. It runs on cola, pizza and Dilbert cartoons.
Bram Moolenaar

Christian Brabandt

unread,
Oct 9, 2013, 5:30:19 PM10/9/13
to vim...@googlegroups.com
Hi Bram!

On Mi, 09 Okt 2013, Bram Moolenaar wrote:

>
> Yasuhiro Matsumoto wrote:
>
> > Bram, why you remove this patch from todo list?
> >
> > https://code.google.com/p/vim/source/diff?spec=svn173c9c860e42e2af31d598aa6924b99d7e73bd1b&r=173c9c860e42e2af31d598aa6924b99d7e73bd1b&format=side&path=/runtime/doc/todo.txt
> >
> > I can't assent.
>
> There are several changes. Can you quote the one you refer to?

I guess, he means why you removed the async patch from the todo list.

regards,
Christian
--
Weiber gew�hnen sich Gleichg�ltigkeit und Unaufmerksamkeit gegen
Wissenschaft und Taubheit an, weil die M�nner zu oft vor ihnen von
wissenschaftlichen Dingen reden, die ihnen unbekannt.
-- Jean Paul

Bram Moolenaar

unread,
Oct 10, 2013, 5:38:04 PM10/10/13
to Christian Brabandt, vim...@googlegroups.com

Christian Brabandt wrote:

> On Mi, 09 Okt 2013, Bram Moolenaar wrote:
> >
> > Yasuhiro Matsumoto wrote:
> >
> > > Bram, why you remove this patch from todo list?
> > >
> > > https://code.google.com/p/vim/source/diff?spec=svn173c9c860e42e2af31d598aa6924b99d7e73bd1b&r=173c9c860e42e2af31d598aa6924b99d7e73bd1b&format=side&path=/runtime/doc/todo.txt
> > >
> > > I can't assent.
> >
> > There are several changes. Can you quote the one you refer to?
>
> I guess, he means why you removed the async patch from the todo list.

It thought it was there in some form... I don't like calling it async,
because it isn't. I'll add something. It's actually similar to the
request for the :timer command.

--
WOMAN: Dennis, there's some lovely filth down here. Oh -- how d'you do?
ARTHUR: How do you do, good lady. I am Arthur, King of the Britons.
Who's castle is that?
WOMAN: King of the who?
The Quest for the Holy Grail (Monty Python)

kans

unread,
Oct 10, 2013, 7:03:50 PM10/10/13
to vim...@googlegroups.com, Christian Brabandt
> It's actually similar to the
>
> request for the :timer command.

We renamed the feature to +timers. Are you referring to a different patch or to this one?

At any rate, I tracked down the final bug. We were actually stealing cpu time by calling select far too often (20ms) when select is call reentrantly. Apparently, some timeouts cause select to be called again. We were correctly dealing with not calling timeouts in this case, but not in scheduling the next call to call_timeouts. This was really hard to track because the cpu time wasn't spent inside Vim and the only thing that exhibited this problem was :Explore which is very hard to grok. To the best of my knowledge, this patch is now bug free!

-Matt

timers.patch

Ken Takata

unread,
Oct 10, 2013, 9:13:34 PM10/10/13
to vim...@googlegroups.com
Hi Matt,

Your patch still have some coding style errors and I fixed them.
Please check the attached patch.

Additionally, there are still some issues I think:

1. Copyright information is missing in the timers.c.
2. MCH_MONOTONIC_TIME is defined in os_macosx.m and os_unix.c, but it should be
defined in header files because the definition is used in timers.c.
3. Do we still need to support MSVC6? (Note that 7.4.044 is a fix for VC6.)
If it is needed, we can't use "long long". We should define a 64-bit type.
Similar (but not the same) thing is written in the todo.txt:

> On 64 bit MS-Windows "long" is only 32 bits, but we sometimes need to store a
> 64 bits value. Change all number options to use nropt_T and define it to the
> right type.

Regards,
Ken Takata

timers.patch

cptstubing

unread,
Oct 11, 2013, 7:58:48 AM10/11/13
to vim...@googlegroups.com
Calling these timers is the right way to go. Synchronicity is more typically referring to starting some task without blocking while waiting for it to complete its work, not waiting some amount of time. Yes, timers give you a kludgy way to do this in a single-threaded process, but they more specifically give you the ability to schedule an event based on a time span. Nice job on getting this work going -- timers are a pretty fundamental part of any ui framework these days and will quickly become a much-loved addition. So many cursorhold hacks can be fixed up now.

Ken Takata

unread,
Oct 12, 2013, 12:25:47 PM