Fast text display?
Christopher Subich
and is also a good excuse to become familiar with the Python language.
I have a conceptual handle on most of the implementation, but the
biggest unknown for me is the seemingly trivial matter of text display.
My first requirement is raw speed; none of what I'm doing is
processing-intensive, so Python itself shouldn't be a problem here. But
still, it's highly desirable to have very fast text updates (text
inserted only at the end)-- any slower than 20ms/line stretches
usability for fast-scrolling. EVERY other action on the text display,
though, like scrolling backwards or text selection, can be orders of
magnitude slower.
The second requirement is that it support text coloration. The exact
markup method isn't important, just so long as individual characters can
be independently colored.
The third requirement is cross-platform-osity; if you won't hold it
against me I'll tell you that I'm developing under Cygwin in Win2k, but
I'd really like it if the app could run under 'nix and mac-osx also.
I'm pretty open to any graphical toolkit -- I have experience with none
of them, so I have little in the way of prejudice.
I'm not even sure where to start looking to see if a widget that does
this has been premade -- the text widgets that I've seen so far have all
had documentation geared more towards text editing than pure display.
My closest look has been at PyGTK -- the GTK text widget is certainly
complex enough to handle the markup, but it also seems a little bit
feature-rich for my needs so I'm concerned about the overhead.
In a worst-case scenario, I could try writing a widget-lite with SDL or
possibly an OpenGL texture, but if a widget somewhere will already do
this then it'd be needless work. To boot, I have no idea how to handle
selection on such a homebrew-display if I'm using a proportional font,
but that's a minor matter.
So, the one-line summation of the novel above, does any one know of a
gui toolkit/widget that will do (preferably extremely) fast text display
in a cross-platform manner?
PS: I couldn't get a prototype curses display working either, the
scroll() function didn't work -- interactive log below:
>>> import curses
>>> x = curses.initscr()
>>> print curses.has_il()
True
>>> x.idlok(1)
>>> x.scroll(1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
_curses.error: scroll() returned ERR
Andrew Dalke
> My first requirement is raw speed; none of what I'm doing is
> processing-intensive, so Python itself shouldn't be a problem here.
There's raw speed and then there's raw speed. Do you want to
display, say, a megacharacter/second?
> it's highly desirable to have very fast text updates (text
> inserted only at the end)-- any slower than 20ms/line stretches
> usability for fast-scrolling.
Ahh, that's 400 bytes per second. That's pretty slow.
> The second requirement is that it support text coloration.
> The third requirement is cross-platform-osity
qtextedit has all of those. See
http://doc.trolltech.com/3.3/qtextedit.html
Looks like LogText mode is exactly what you want
http://doc.trolltech.com/3.3/qtextedit.html#logtextmode
] Setting the text format to LogText puts the widget in a special mode
] which is optimized for very large texts. Editing, word wrap, and rich
] text support are disabled in this mode (the widget is explicitly made
] read-only). This allows the text to be stored in a different, more
] memory efficient manner.
and
] By using tags it is possible to change the color, bold, italic and
] underline settings for a piece of text.
Depending on what you want, curses talking to a terminal might be
a great fit. That's how we did MUDs back in the old days. :)
Andrew
da...@dalkescientific.com
Paul Rubin
> The third requirement is cross-platform-osity; if you won't hold it
> against me I'll tell you that I'm developing under Cygwin in Win2k,
> but I'd really like it if the app could run under 'nix and mac-osx
> also.
>
> I'm pretty open to any graphical toolkit -- I have experience with
> none of them, so I have little in the way of prejudice.
Use tkinter if you care about cross-platform operation. Everything
else requires downloading and installing separate toolkits.
Christopher Subich
> Christopher Subich wrote:
>
>> My first requirement is raw speed; none of what I'm doing is
processing-intensive, so Python itself shouldn't be a problem here.
>
>
>
> There's raw speed and then there's raw speed. Do you want to
> display, say, a megacharacter/second?
[snip]
> Ahh, that's 400 bytes per second. That's pretty slow.
Scrolling the text at any faster than a blur is counterproductive for
the user, after all. You're off by a decimal, though, an 80-column line
at 20ms is 4kbytes/sec. My guess is that any faster throughput than
10kbytes/sec is getting amusing for a mud, which in theory intends for
most of this text to be read anyway.
>
> qtextedit has all of those. See
> http://doc.trolltech.com/3.3/qtextedit.html
That looks quite good, except that Trolltech doesn't yet have a GPL-qt
for Win32. I might take another look at it whenever qt4 comes out, but
in the meantime (since I'm unfortunately developing on a win2k system)
it's not useful.
> Depending on what you want, curses talking to a terminal might be
> a great fit. That's how we did MUDs back in the old days. :)
See the scrolling problem in the original post, as to why I can't use it
as a temporary user interface. :)
Christopher Subich
> Use tkinter if you care about cross-platform operation. Everything
> else requires downloading and installing separate toolkits.
Oh, the downloading and installing isn't a big deal. If in the
far-flung future anyone else uses this program, they'll be big boys who
can install things themselves. :)
I'm just concerned about availability; the cross-platform operation for
me would exclude things like direct Win32API calls, or direct
linux-terminal calls (which apparently mcl uses to great effect).
Paul Rubin
> > Use tkinter if you care about cross-platform operation. Everything
> > else requires downloading and installing separate toolkits.
>
> Oh, the downloading and installing isn't a big deal. If in the
> far-flung future anyone else uses this program, they'll be big boys
> who can install things themselves. :)
No, it's a big pain. I'm a big boy and gave up on trying to install
wxpython for bittorrent on FC3 (the problem was with wxwidgets needing
an obsolete version of gtk, not with wxpython itself). There's just
no compelling reason to want to deal with this stuff. Tkinter has its
warts but it allows making functional gui's without all that much
fuss. If it were replaced in the default distro then I'd say use
whatever the new default is, but as it is, it's best to not impose
unnecessary extra headache on users.
Christopher Subich
> No, it's a big pain. I'm a big boy and gave up on trying to install
> wxpython for bittorrent on FC3 (the problem was with wxwidgets needing
> an obsolete version of gtk, not with wxpython itself). There's just
> no compelling reason to want to deal with this stuff. Tkinter has its
> warts but it allows making functional gui's without all that much
> fuss. If it were replaced in the default distro then I'd say use
> whatever the new default is, but as it is, it's best to not impose
> unnecessary extra headache on users.
Fair enough. At the moment, the expected user base for the program is
exactly one, but making it easy has its advantages. Since you've
obviously used it yourself, if I end up going with tkinter, are there
any performance gotchas on text rendering that I should be aware of?
Andrew Dalke
> You're off by a decimal, though, an 80-column line
> at 20ms is 4kbytes/sec.
D'oh! Yeah, I did hundredths of a second instead of thousands.
> My guess is that any faster throughput than
> 10kbytes/sec is getting amusing for a mud, which in theory intends for
> most of this text to be read anyway.
Which is why I don't think you'll have a problem with any of
the standard GUI libraries.
> That looks quite good, except that Trolltech doesn't yet have a GPL-qt
> for Win32.
Cost and license weren't listed as requirements. :)
You *did* say "hobby" though in post-hoc justification, I've known
people with some pretty expensive hobbies.
> See the scrolling problem in the original post, as to why I can't use it
> as a temporary user interface. :)
Indeed, but MUDs 15 years ago could run in a terminal and display
colored text via ANSI terminal controls, letting the terminal
itself manage history and scrolling. I had some sort of TSR for
the latter, under DOS.
Andrew
da...@dalkescientific.com
Jp Calderone
>As a hobby project, I'm writing a MUD client -- this scratches an itch,
>and is also a good excuse to become familiar with the Python language.
>I have a conceptual handle on most of the implementation, but the
>biggest unknown for me is the seemingly trivial matter of text display.
>
>My first requirement is raw speed; none of what I'm doing is
>processing-intensive, so Python itself shouldn't be a problem here. But
>still, it's highly desirable to have very fast text updates (text
>inserted only at the end)-- any slower than 20ms/line stretches
>usability for fast-scrolling. EVERY other action on the text display,
>though, like scrolling backwards or text selection, can be orders of
>magnitude slower.
>
>The second requirement is that it support text coloration. The exact
>markup method isn't important, just so long as individual characters can
>be independently colored.
>
>The third requirement is cross-platform-osity; if you won't hold it
>against me I'll tell you that I'm developing under Cygwin in Win2k, but
>I'd really like it if the app could run under 'nix and mac-osx also.
I've done this with Tkinter before. At the time, I surveyed the various toolkits for the quality of their text widgets, and of Tkinter, PyGTK, PyQT, and wxPython, only Tkinter could satisfy the performance requirements. This was about three years ago, so the field may have changed.
If you like, you can check out the code:
http://sourceforge.net/projects/originalgamer
As MUD clients go, it's pretty weak, but it solves the text display problem pretty decently.
Hope this helps,
Jp
Christopher Subich
> If you like, you can check out the code:
>
> http://sourceforge.net/projects/originalgamer
>
> As MUD clients go, it's pretty weak, but it solves the text display
> problem pretty decently.
Oooh! Code! Thanks! After taking an extremely quick look, I think I
might be kicking myself soon for not knowing about Twister and writing
socket code thus far instead. Interesting to see how someone else
solves a the same problem. :)
Paul Rubin
> Fair enough. At the moment, the expected user base for the program is
> exactly one, but making it easy has its advantages. Since you've
> obviously used it yourself, if I end up going with tkinter, are there
> any performance gotchas on text rendering that I should be aware of?
We're just talking about a scrolling text window that has to run at
human reading and typing speeds, right? It shouldn't be a problem.
I've used the text widget and found it to be fast enough for what I
was doing, though I didn't exactly stress it.
Diez B. Roggisch
> That looks quite good, except that Trolltech doesn't yet have a GPL-qt
> for Win32. I might take another look at it whenever qt4 comes out, but
> in the meantime (since I'm unfortunately developing on a win2k system)
> it's not useful.
Look at qt feee win edition, available here:
http://pythonqt.vanrietpaap.nl/
Diez
Riccardo Galli
> Christopher Subich <spam.csub...@block.subich.spam.com> writes:
>> > Use tkinter if you care about cross-platform operation. Everything
>> > else requires downloading and installing separate toolkits.
>>
>> Oh, the downloading and installing isn't a big deal. If in the
>> far-flung future anyone else uses this program, they'll be big boys who
>> can install things themselves. :)
>
> No, it's a big pain. I'm a big boy and gave up on trying to install
> wxpython for bittorrent on FC3 (the problem was with wxwidgets needing an
> obsolete version of gtk, not with wxpython itself). There's just no
> compelling reason to want to deal with this stuff. Tkinter has its warts
> but it allows making functional gui's without all that much fuss.
Using tkinter doesn't need downloading and installing only in Windows.
In *nix is not so common to have tcl/tk installed (and probably in Mac too)
GUI cross platform need external support, in a OS or in another.
Bye,
Riccardo
--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Paul Rubin
> Using tkinter doesn't need downloading and installing only in Windows.
> In *nix is not so common to have tcl/tk installed (and probably in Mac too)
Hmm, in the Linux distros that I'm used to, tcl/tk is preinstalled. I
had the impression that it was included with Python but obviously I
haven't looked that closely.
Jp Calderone
What does "included with Python" mean anyway? Different packagers make different decisions. Some may include Tcl/Tk, others may exclude it. Some may provide a separate but trivially-installable package for it. On systems with reasonable package managers, it barely makes a difference, as any packaged software is at most one or two simple commands away.
This applies to other libraries as well, of course. Installing wxPython on Debian is a 5 second ordeal. This is not to say debian is awesome and you should go install it right now or *else*, just to say that the installation of a single piece of software can vary greatly in difficulty between different platforms.
Jp
Paul Rubin
> What does "included with Python" mean anyway? Different packagers
> make different decisions.
I mean included with the source distro from python.org.
> This applies to other libraries as well, of course. Installing
> wxPython on Debian is a 5 second ordeal.
I don't believe that, unless you're installing a binary. I only want
to install from source.
Christopher Subich
> We're just talking about a scrolling text window that has to run at
> human reading and typing speeds, right? It shouldn't be a problem.
> I've used the text widget and found it to be fast enough for what I
> was doing, though I didn't exactly stress it.
It should have burst speeds of 2-10x reading speed; given the
stop-and-start nature of most muds, the data comes in bursts that the
player processes and reacts to before the next burst.
It's good to know that my concerns seem mostly unfounded. When I get to
a reasonable state of implementation, I'll post here if I've made any
unexpected discoveries.
--
PS: Really sorry to everyone about all the e-mails I've generated. I'm
not quite used to Thunderbird, and it puts "reply to sender only" where
I'd expect "reply to newsgroup" to be.
Eric Brunel
> Using tkinter doesn't need downloading and installing only in Windows.
> In *nix is not so common to have tcl/tk installed (and probably in Mac too)
>
> GUI cross platform need external support, in a OS or in another.
Even if tcl/tk is not installed by default on your system, doing it is usually not a pain, since the guys making it insist on keeping its required dependencies minimal: IIRC, to install tcl/tk on any Unix, you just need to have X and a C compiler.
This may be a slight advantage over most other toolkits, that usually require other libraries that you may not have installed on your system, or installed at the wrong version. I was sometimes caught in such a dependency hell when trying to install a toolkit that I eventually gave up...
--
python -c "print ''.join([chr(154 - ord(c)) for c in 'U(17zX(%,5.zmz5(17;8(%,5.Z65\'*9--56l7+-'])"
Mike Meyer
> Jp Calderone <exa...@divmod.com> writes:
>> What does "included with Python" mean anyway? Different packagers
>> make different decisions.
> I mean included with the source distro from python.org.
Well, it includes Tkinter, but it doesn't include tcl/tk. So you get a
non-functional version of Tkinter if you install from those
sources. Not very useful.
>> This applies to other libraries as well, of course. Installing
>> wxPython on Debian is a 5 second ordeal.
> I don't believe that, unless you're installing a binary. I only want
> to install from source.
On FreeBSD, it takes about 5 seconds of my time to install
wxPython from source:
# cd /usr/ports/x11-toolkits/py-wxPython
# make install
Then I can go do something useful while the package system downloads,
compiles and installs all the required software. I don't know if
Debian has a similar system for installing from source. Gentoo does if
you want to stay with Linux.
FWIW, FreeBSD's Python package doesn't include tcl or tk, but will
install them as dependencies when you install Python so you get a
working Tkinter out of the box.
And in spite of all that, I use PyQt by preference.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Renato Ramonda
> GUI cross platform need external support, in a OS or in another.
Sure, but using win32 versions of gtk and glade plus py2exe and
InnoSetup (if you want to build a fancy installer) will give you a self
contained gui program in windows.
Big, sure, but self contained :-D
--
Renato
--------------------------------
Usi Fedora? Fai un salto da noi:
http://www.fedoraitalia.org
Renato Ramonda
> Then I can go do something useful while the package system downloads,
> compiles and installs all the required software. I don't know if
> Debian has a similar system for installing from source. Gentoo does if
> you want to stay with Linux.
Debian actually has such a mechanism, if you have deb-src repositories
in your apt sources list. The real problem will come from conflicting
versions of wx and/or gtk required by different (other) programs.
Paul Rubin
> > I mean included with the source distro from python.org.
>
> Well, it includes Tkinter, but it doesn't include tcl/tk. So you get a
> non-functional version of Tkinter if you install from those
> sources. Not very useful.
OK. But I found on RH9 and FC3, as well as on Windows, that tcl/tk
was preinstalled (or included with Python). I didn't find wxwidgets
preinstalled on any of those systems.
> On FreeBSD, it takes about 5 seconds of my time to install
> wxPython from source:
>
> # cd /usr/ports/x11-toolkits/py-wxPython
> # make install
What about wxwidgets, which wxpython requires?
> Then I can go do something useful while the package system downloads,
> compiles and installs all the required software.
The installer is going to download more stuff? Yuccch, I'd rather not
need a network connection to do an install. Anyway, wxpython built ok
on FC3. The problem was wxwidgets, which needed an obsolete version
of GTK. I spent at least an hour or two messing around with it before
deciding it wasn't worth the hassle.
Renato Ramonda
> The installer is going to download more stuff? Yuccch, I'd rather not
> need a network connection to do an install. Anyway, wxpython built ok
> on FC3. The problem was wxwidgets, which needed an obsolete version
> of GTK. I spent at least an hour or two messing around with it before
> deciding it wasn't worth the hassle.
AFAIR the linux package for wxpython does not require wxWidgets at all
(it's embedded).
As for the obsolete gtk look for a "gtk+" package, not for the "gtk"
which is the up-to-date gtk2 package. And good luck :-)
Mike Meyer
> Mike Meyer <m...@mired.org> writes:
>> > I mean included with the source distro from python.org.
>> Well, it includes Tkinter, but it doesn't include tcl/tk. So you get a
>> non-functional version of Tkinter if you install from those
>> sources. Not very useful.
> OK. But I found on RH9 and FC3, as well as on Windows, that tcl/tk
> was preinstalled (or included with Python). I didn't find wxwidgets
> preinstalled on any of those systems.
I think posts are getting crossed here. The python sources you get
from www.python.org don't include tcl or tk, which is how the poster I
was replying to specified things.
>> On FreeBSD, it takes about 5 seconds of my time to install
>> wxPython from source:
>>
>> # cd /usr/ports/x11-toolkits/py-wxPython
>> # make install
>
> What about wxwidgets, which wxpython requires?
If the package needs something and it's not installed, the package
system will get it and install it. That's the *point* of a package
system.
>> Then I can go do something useful while the package system downloads,
>> compiles and installs all the required software.
>
> The installer is going to download more stuff? Yuccch, I'd rather not
> need a network connection to do an install.
The package system can be configured in a number of different ways. My
configuration - with a broadband network connection - is to check the
local disk cache, then download anything that's missing. You can
preload the disk cache by doing "make fetch-recursive" if you
want. You can preload the cache by hand if you want, but the package
system is very picky about the version numbers of things it's trying
to build, so that'd be a pain.
Paul Rubin
> > OK. But I found on RH9 and FC3, as well as on Windows, that tcl/tk
> > was preinstalled (or included with Python). I didn't find wxwidgets
> > preinstalled on any of those systems.
>
> I think posts are getting crossed here. The python sources you get
> from www.python.org don't include tcl or tk, which is how the poster I
> was replying to specified things.
Here's what I believe the situation is:
RH9: tcl/tk preinstalled, tkinter included with python. Once I
install RH9, I can use Python, tkinter, and tcl/tk without
downloading anything additional.
FC3: same as RH9
Windows: tcl and tk and tkinter all included in python.msi. Once I
install Python, I can use tkinter/tcl/tk without installing anything
additional.
> > What about wxwidgets, which wxpython requires?
>
> If the package needs something and it's not installed, the package
> system will get it and install it. That's the *point* of a package
> system.
Not really. A package system turns a software distro into a single
file you can install. In some instances a package depends other
packages and some package managers can then go chase down the
dependencies, but that's not characteristic of packages per se, and
while it's better than nothing, it's best to avoid it since the mechanism
breaks a lot. Tcl/tk is a boneheaded system in many ways, but it's
been very successful partly through the strategy of avoiding installation
hassle by minimizing dependencies.
> The package system can be configured in a number of different ways. My
> configuration - with a broadband network connection - is to check the
> local disk cache, then download anything that's missing. You can
> preload the disk cache by doing "make fetch-recursive" if you
> want. You can preload the cache by hand if you want, but the package
> system is very picky about the version numbers of things it's trying
> to build, so that'd be a pain.
Best is to have just one package that doesn't depend on any other
packages and doesn't need any further downloads, so once you've
installed it, you're done.
Mike Meyer
>>
>> If the package needs something and it's not installed, the package
>> system will get it and install it. That's the *point* of a package
>> system.
>
> Not really. A package system turns a software distro into a single
> file you can install. In some instances a package depends other
> packages and some package managers can then go chase down the
> dependencies, but that's not characteristic of packages per se, and
> while it's better than nothing, it's best to avoid it since the mechanism
> breaks a lot. Tcl/tk is a boneheaded system in many ways, but it's
> been very successful partly through the strategy of avoiding installation
> hassle by minimizing dependencies.
The crucial word in what I said was "system". You're right that
packages per se don't handle dependencies. That's up to the package
system. In my experience, the "breaks a lot" depends on the system
you've got, and how you use it. I've spent a lot of time cursing RPMs
because the dependency system does seem to break a lot. On the other
hand, the only times I've had the FreeBSD package system break is when
I was trying to force it to build things that weren't in the package
system as supplied.
>> The package system can be configured in a number of different ways. My
>> configuration - with a broadband network connection - is to check the
>> local disk cache, then download anything that's missing. You can
>> preload the disk cache by doing "make fetch-recursive" if you
>> want. You can preload the cache by hand if you want, but the package
>> system is very picky about the version numbers of things it's trying
>> to build, so that'd be a pain.
>
> Best is to have just one package that doesn't depend on any other
> packages and doesn't need any further downloads, so once you've
> installed it, you're done.
This is the best solution for a naive user with a primitive package
system. If either condition doesn't hold, it's no longer clear that
that's the case. With a good package system, the dependencies will be
handled invisibly and transparently, and the only way to find out that
multiple packages were installed after you're done is by checking the
installed packages database. If the user is advanced, they may prefer
not to have multiple versions of the dependency installed, or that it
be installed where it's available to other packages instead of just
the installing package - or vice versa.