Keeping GUI alive with [after idle [after 0 ...]]
John Seal
I'm developing a tool that displays a file in a scrolling text widget, then
color-codes each line according to results returned by an external program.
My first implementation was quick & dirty and didn't keep the GUI alive
while processing long files. I refactored to process one line at a time,
scheduling the next line with the [after idle [after 0 ...]] idiom as
described on http://wiki.tcl.tk/9926.
That kept the GUI alive... mostly. Everything worked except the scrollbars,
both the one on the text widget being color-coded, and another one attached
to a log text widget. If I change the code to [after idle [after 1 ...]]
the scrollbars stay alive, too.
At this point my brain is too fried to puzzle out exactly why that might
be... can anyone explain it to me?
MSE...@gmail.com
> All:
>
> I'm developing a tool that displays a file in a scrolling text widget, then
> color-codes each line according to results returned by an external program.
> My first implementation was quick & dirty and didn't keep the GUI alive
> while processing long files. I refactored to process one line at a time,
> scheduling the next line with the [after idle [after 0 ...]] idiom as
>
> That kept the GUI alive... mostly. Everything worked except the scrollbars,
> both the one on the text widget being color-coded, and another one attached
> to a log text widget. If I change the code to [after idle [after 1 ...]]
> the scrollbars stay alive, too.
>
> At this point my brain is too fried to puzzle out exactly why that might
> be... can anyone explain it to me?
I think you should be using 'filevent readable' then read all
available rather than polling as this automatically keeps your GUI
alive.
On other usefull trick for log files is to look at the scrollbar
position before inserting the text at the end and only do the 'see
end' if the scrollbar was at the end. I also change the color of the
scrollbar to show that new information has been added.
There is nothing more annoying than scrolling up in a log file to hav
it automaticlly scrolled away from that interesting message every time
a new log arrives.
PS if you change the color dont forget to change it bac to the default
when the user scrolls back to the end.
Martyn SMITH
George Peter Staplin
Martyn's advice is generally right. However, if you have a lot of data to
read, it may block the GUI anyway, because the Tcl event loop won't become
idle. I ran into this problem with a socket interface, and something that
used pipes. In other words the [after idle]/Tcl_DoWhenIdle callbacks are
never called during times of heavy activity. In some situations it's
better to spawn another process that does a lot of heavy lifting, so that
your UI is still interactive, and draws correctly.
Another problem that can occur is that the idle queue builds up, and
eventually Tcl is no longer able to allocate memory. The result is that
the process abort()s. Unfortunately this has happened to me several times
in actual usage of some programs that transferred or manipulated a lot of
data. The pattern is either unbounded memory growth at times, or a failed
allocation leading to the abort(). If we had a bounded idle queue, it
might lead to deadlocks or races in some cases, so we probably can't make
the idle queue have a boundary or limit.
Someone last night in the Tcler's Chat was asking for an example of how to
make a parallel program in Tcl. I wrote the following:
http://paste.tclers.tk/1553
There are of course performance optimizations that can be made, and in many
cases you might want a binary protocol instead. It's basically a useful
example and pattern you can use to create more workers for your main
program path.
Here is the code in full:
set ::script [info script]
proc add {} {
while 1 {
gets stdin e
set code ""
foreach i $e {
append code "$i + "
}
set code [string trimright $code "+ "]
puts stdout [expr $code]
}
}
proc start-add {} {
global script
set pipe [open |[list [info nameofexecutable] $script listen] r+]
return $pipe
}
proc eval-stdin {} {
uplevel #0 [gets stdin]
}
proc main {argc argv} {
if {1 == $argc} {
if {[lindex $argv 0] eq "listen"} {
add
exit 0
}
}
set adders [list [start-add] [start-add] [start-add]]
set n 1
foreach a $adders {
puts $a [list $n $n]
flush $a
puts "SENT TO:$a"
incr n
}
foreach a $adders {
puts RESULTS:[gets $a]
}
set n -1
foreach a $adders {
puts $a [list $n $n]
flush $a
incr n -1
}
foreach a $adders {
puts RESULTS:[gets $a]
}
fileevent stdin readable [list eval-stdin]
vwait until_exit
}
main $::argc $::argv
John Seal
<MSE...@gmail.com> wrote in message
news:753e1bbe-1201-4abf...@v19g2000yqn.googlegroups.com...
> I think you should be using 'filevent readable' then read all
> available rather than polling
Martyn:
I think you misunderstood the problem. There is no file reading, there is
no polling. I'm simply processing each line of a text widget, updating the
color of the line according to results returned by an external program.
> On other usefull trick for log files
I'm not doing anything with the log pane, I just noted that it's scrollbar
didn't work either. My thinking was that maybe the scrollbar of the text
widget being processed and colorized was unresponsive *because* of the
processing. But the log pane scrollbar is also unresponsive, even though
the log text is not being touched.
I think George Peter Staplin is on the right track with his suggestion that
the idle handler is getting swamped (I'm paraphrasing). The source of my
puzzlement is that some GUI processing works (menubar rollover highlighting,
tooltips, clicking and typing into entry boxes) and some doesn't (moving
scrollbars). I would expect all those to work via idle events.
My assumption is that the [after idle [after 1 ...]] workaround works by
allowing extra time in the idle loop. I just wonder what's different about
the scrollbar events that causes them NOT to be processed if I say [after
idle [after 0 ...]], which is the canonical idiom documented on the wiki and
elsewhere.
There's an excellent discussion of idle handling at
http://groups.google.com/group/comp.lang.tcl/browse_thread/thread/62a98bf8c0637cb4/5144376d2ce65c30.
George Peter Staplin
Tk widgets typically redraw using a pattern like this:
1. a readable socket event occurs to indicate the window system has sent
something ready for Tk.
2. an event is read such as ButtonPress, or an Expose (indicating window was
exposed and has a dirty area that needs to be redrawn), window resize, etc.
3. the widget gets that event, which is dispatched by a single thread to the
widget code. At this point if the widget tried to redraw it could hurt
performance in some cases, because there may be several Expose or other
events that occur one after the other. So Tk does a lazy/delayed redraw.
4. the widget uses Tcl_DoWhenIdle() to setup an idle callback.
Tcl_DoWhenIdle appends a callback to the Tcl event loop's idle queue.
(Note: the event loop used to be in Tk, so you may see remains of
Tk_DoWhenIdle IIRC.).
5. Tk may dispatch events to N more widgets. Where N is >= 0, and those
events may trigger more idle queue appends.
6. Finally the specific event source for the window-system-related events
may be finished.
At this point see: generic/tclNotify.c for Tcl_DoOneEvent().
7. The Tcl_DoOneEvent code is the core event loop started by wish, it's like
the [vwait forever] pattern used in tclsh.
This is where the event sources are checked, such as the window system
events, which result in things like Tcl_QueueEvent being called:
This is all part of a large while loop:
for (sourcePtr = tsdPtr->firstEventSourcePtr; sourcePtr != NULL;
sourcePtr = sourcePtr->nextPtr) {
if (sourcePtr->checkProc) {
sourcePtr->checkProc(sourcePtr->clientData, flags);
}
}
This actually is what removes and dispatches the event.
/*
* Check for events queued by the notifier or event sources.
*/
if (Tcl_ServiceEvent(flags)) {
result = 1;
break;
}
The path below is what isn't run when the program is very busy, and there is
a constant stream of serviceable events above, so you have none of the
Tcl_DoWhenIdle callbacks being called. Normally this isn't a problem, but
program usage can change that.
/*
* We've tried everything at this point, but nobody we know about
had
* anything to do. Check for idle events. If none, either quit or go
* back to the top and try again.
*/
idleEvents:
if (flags & TCL_IDLE_EVENTS) {
if (TclServiceIdle()) {
result = 1;
break;
}
}
So, this is why Tk feels fast sometimes, and it's the Tcl and Tk way of
handling multiplexing, but it can lead to problems.
The idle callbacks are used extensively throughout Tk to deal with things
from window creation (this is why we have Tk_MakeWindowExist()) to the grid
manager. Tcl even uses idle callbacks for generating some errors.
So, let's say I do:
bind .widget <ButtonPress-1> {
long_calculation
}
That will often leave Tcl in the Tcl_ServiceEvent (see above), and prevents
the idle events from being serviced that would otherwise redraw the GUI.
So when you break up your algorithm into incremental parts like this, it's
much better:
proc do_step {step} {
#do part of the long calculation using $step
incr step
after N [list do_step $step]
}
bind .widget <ButtonPress-1> { do_step 1 }
That allows the idle path to run. There are other ways of getting the idle
path to run, such as [update idletasks], but [update] is harmful in some
cases (see the wiki).
You should also guard against having 2 do_step [after]s being called due to
repeating ButtonPress events, because that can result in unexpected
problems, or side effects. So in reality you might have one proc that
starts it all, and another that does each step, and the proc that starts
the calculation might return if the task is already active.
This is a graphical demonstration I wrote that shows how to use Tcl's event
loop effectively for many graphical events:
http://wiki.tcl.tk/7456
-George
George Peter Staplin
> I think George Peter Staplin is on the right track with his suggestion
> that
> the idle handler is getting swamped (I'm paraphrasing). The source of my
> puzzlement is that some GUI processing works (menubar rollover
> highlighting, tooltips, clicking and typing into entry boxes) and some
> doesn't (moving
> scrollbars). I would expect all those to work via idle events.
>
One more thing: if this is Windows, and if I recall correctly, then some of
the widgets are actually using different threads. So you may see some
native Win32 widgets react differently with respect to redrawing, and
interaction, depending on what state the event loop is in.
-George
John Seal
news:grolrp$n7p$1...@news.xmission.com...
> So when you break up your algorithm into incremental parts like this, it's
> much better:
>
> proc do_step {step} {
> #do part of the long calculation using $step
> incr step
> after N [list do_step $step]
> }
> bind .widget <ButtonPress-1> { do_step 1 }
George:
This is similar to how I broke up my long-running process, except that I
used [after idle [after 0 ...]]. Are you saying specifically that [after N
...] (presumably where N>0 ) is "much better" because it creates time for
the idle path to run?
I found that [after idle [after 1 ...]] also allowed the scrollbars to work.
Is that better or worse than just [after N ...]? Dean Brettle's discussion,
which I linked to earlier, seems to imply that [after N ...] can still get
swamped.
> That allows the idle path to run. There are other ways of getting the
> idle
> path to run, such as [update idletasks], but [update] is harmful in some
> cases (see the wiki).
Oh yeah, I learned my lesson about [update] long ago!
George Peter Staplin
> "George Peter Staplin" <geor...@xmission.com> wrote in message
> news:grolrp$n7p$1...@news.xmission.com...
>> So when you break up your algorithm into incremental parts like this,
>> it's much better:
>>
>> proc do_step {step} {
>> #do part of the long calculation using $step
>> incr step
>> after N [list do_step $step]
>> }
>> bind .widget <ButtonPress-1> { do_step 1 }
>
> George:
>
> This is similar to how I broke up my long-running process, except that I
> used [after idle [after 0 ...]]. Are you saying specifically that [after
> N ...] (presumably where N>0 ) is "much better" because it creates time
> for the idle path to run?
>
> I found that [after idle [after 1 ...]] also allowed the scrollbars to
> work.
> Is that better or worse than just [after N ...]? Dean Brettle's
> discussion, which I linked to earlier, seems to imply that [after N ...]
> can still get swamped.
John,
The problem with after 0 is that if you use it repeatedly the idle events
are never processed. That can also happen with after N when N is small, so
the [after idle [after N]] pattern is designed to work around the problem
sometimes occurring.
Unfortunately the [after N] behavior as I mentioned can lead to idle
starvation problems depending on the amount of work done in each callback,
so for your solution you [after idle [after N]] may be ideal. The load
necessary to lead to starvation will also vary depending on the CPU speed,
and other factors.
I have seen a script essentially stop redrawing on one slower machine, while
a faster machine continues to redraw and work fine. The problem was that
the time consumed by the processing varied between the systems. When the
slower machine stops redrawing the idle queue fills, and eventually the
program crashes from a Tcl_Panic()/abort() due to an out of memory
condition.
If you are doing a lot of work, then you might very well want the [after
idle [after N]] pattern to work around this.
I think a good and reasonable alternative is often to spawn another process.
You can also reuse the same script for different process paths that are
activated by different argv options. Then it becomes a matter of
interacting with those process paths via pipes created by [open |].
>> That allows the idle path to run. There are other ways of getting the
>> idle
>> path to run, such as [update idletasks], but [update] is harmful in some
>> cases (see the wiki).
>
> Oh yeah, I learned my lesson about [update] long ago!
Me too :) The Tcl event loop isn't ideal. It's a compromise, and sometimes
it's a bad compromise for some situations.
-George
dkf
> I have seen a script essentially stop redrawing on one slower machine, while
> a faster machine continues to redraw and work fine. The problem was that
> the time consumed by the processing varied between the systems. When the
> slower machine stops redrawing the idle queue fills, and eventually the
> program crashes from a Tcl_Panic()/abort() due to an out of memory
> condition.
One fix for this is to do (using the 'every' code of the wiki):
every 1000 {update idletasks}
Explanation below.
> If you are doing a lot of work, then you might very well want the [after
> idle [after N]] pattern to work around this.
While timer events are processed even when the event queue is flooded,
it's idle events that are not (because the system never becomes idle,
of course). The fix in these extreme situations is to make sure that
idle events get their day in the sun using a technique like above.
(Most of the time it isn't necessary and is actually advised
against...)
Donal.
Ralf Fassel
| I refactored to process one line at a time, scheduling the next line
| with the [after idle [after 0 ...]] idiom as described on
| http://wiki.tcl.tk/9926.
Just in case... note that the wiki reads
after idle [list after 0 $myCommand]
which is quite different from the
[after idle [after 0 ...]]
you cited above.
I guess that your example was just a 'placeholder' for the 'real' code,
since "[after idle [after 0 ...]]" will most probably produce a "command
not found" background error.
R'
Donald Arseneau
> Just in case... note that the wiki reads
> after idle [list after 0 $myCommand]
> which is quite different from the
> [after idle [after 0 ...]]
> you cited above.
The version with [after 0 ...] will execute its payload (...)
immediately, then try to execute the after-identifier as a
command some time later.
There doesn't even need to be any [list ] or other
grouping there, since
after idle after 0 ...
works fine.
Donald Arseneau
John Seal
news:ygaws9n...@akutech.de...
> Just in case... note that the wiki reads
> after idle [list after 0 $myCommand]
> which is quite different from the
> [after idle [after 0 ...]]
> you cited above.
>
> I guess that your example was just a 'placeholder' for the 'real' code,
> since "[after idle [after 0 ...]]" will most probably produce a "command
> not found" background error.
Ralf:
Indeed the ellipsis was just a placeholder for my actual code, which was
carefully "listified" to perform correctly as an [after] script.
Ralf Fassel
| > Just in case... note that the wiki reads
| > after idle [list after 0 $myCommand]
| > which is quite different from the
| > [after idle [after 0 ...]]
| > you cited above.
| >
| > I guess that your example was just a 'placeholder' for the 'real' code,
| > since "[after idle [after 0 ...]]" will most probably produce a "command
| > not found" background error.
|
| Ralf:
|
| Indeed the ellipsis was just a placeholder for my actual code, which
| was carefully "listified" to perform correctly as an [after] script.
The point I probably failed to make was that one should not use
after idle [after 0 ...]
regardless of the ellipsis listified or not.
As Donald already pointed out in Message-ID: <f458d299-ec19-4fdf...@z16g2000prd.googlegroups.com>,
after idle [after 0 ...]
will execute right now the
after 0 ...
part of the command and return the 'after'-handler for it (say 'after0'),
and then execute the
after idle after0
command, which will likely produce an error when the idle handlers
finally run, if 'after0' does not exist as a command.
So you either need
after idle [list after 0 ...]
or
after idle after 0 ...
For more 'complicated' commands than simply 'myCommand', the latter form
should take care of the extra round of substitution when the idle
handler finally runs. Compare:
proc foo arg { puts "foo: {$arg}" }
after idle after 0 [list foo with\ space]
=> bgerror: wrong num #args...
after idle [list after 0 [list foo with\ space]]
=> foo: {with space}
HTH
R'