[erlang-questions] erlang for programming a text editor
Ted Henry
I wonder about writing a multiprocess vi-like or emacs-like text editor
(probably more emacs-like in implementation.) Sometimes single threaded
editors really bog down when searching multiple files, syntax highlighting
multiple files. I'm not that knowledgeable in the Erlang world. It seems
like Erlang's fault tolerance and ability to upgrade code without restarting
would not be big advantages for a text editor. Also a text editor isn't
usually distributed across multiple physical machines. I think the main
advantage would be Erlang's processes for concurrency. Would Erlang offer a
good advantage over a programming language with software transactional
memory (e.g. Haskell, Clojure)? Would *you* use Erlang to program a text
editor or desktop application?
Very interested to read your thoughts.
Ted
Jayson Vantuyl
Well, if I were doing this, I probably wouldn't do it in Erlang. Text
handling in Erlang is byzantine at best, so an editor very well could
be an exercise in pain.
I wouldn't bet that Erlang's good distribution / parallelization would
offset the rest of the trouble. I'm unsure how production-ready it
is, but Reia might make certain bits easier, just due to its string
handling alone.
JRuby or Clojure might be a better fit just because Java has MUCH more
mature Unicode handling and syntax highlighting. I don't know of
anything for Erlang that even tries to do syntax highlighting,
although you could probably get a good start on the parsers with
Neotoma.
In any case, I wish you the best of luck. It's good to see people
trying to tackle useful problems with the right tools.
--
Jayson Vantuyl
kag...@souja.net
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Mats
want my editor to be distributed so I don't lose state when I restart my
computer. A wonderful thought. Updating the source could be done 2-3 times a
day. Update cycles never seen before with text editors. Bye bye Emacs. ;)
Luke Gorrie made an emacs clone[1]. It could be a good start.
The only part missing in Ermacs is LFE (Lisp Flavored Erlang).
Think about the accumulated runtime of all users, if Ermacs was running 24/7
of the latest Erlang release. It could be a nice way of introducing "Test
what you fly, fly what you test" to the QA routine (both for Ermacs and
Erlang).
[1] http://fresh.homeunix.net/~luke/ermacs/
Joe Armstrong
> Ted,
>
> Well, if I were doing this, I probably wouldn't do it in Erlang. Text
> handling in Erlang is byzantine at best, so an editor very well could be an
> exercise in pain.
I beg to differ.
In my opinion Erlang is brilliant at handling text - text is stored in
lists and list processing is blindingly fast and there are large
numbers of library functions
that work on lists. A very unpainful experience.
I have written a full text edit in both Erlang and C - the C *was* painful
Imagine how to implement "undo" in Erlang. Since data is immutable,
you just revert to some old state by accessing the data at some
historical point in the edit - trail all old states in a history stack
and a difficult operation become trivial to program (and efficient
because of the sharing) by popping the history stack. Easy in Erlang - difficult
in a language with mutable state.
I have written *many* text processing applications in Erlang and only
once had problems. My usual experience is that I can write
mind-boggelingly inefficient code which never the less executes blindingly fast.
I always use the get-it-right then optimize philosophy. But the number
of times I have
to optimize is very very few. For text applications virtually *never*.
Text processing
is trivial (unless you're talking of Gigabytes, but then the problem is one of
algorithms)
Only one text application ever caused me a problem - writing a full
text indexer for Gigabytes of data - the problem was not with Erlang
but with my knowledge of algorithms - after I read "managing gigabyes"
and implemented gamma encoding
in Erlang the problem went away. (the ease of which I could change
representations
for lists, to (in this case) gamma encoded integers, was a big big win)
Text process applications involve mainly list processing.
If lists don't cut it then custom abstractions can be easily made.
Erlang is *brilliant* at text applications. Lists are blindingly fast.
The bit syntax
is fantastic if you need to do things like huffman encoding or the
like for tricky text
representations. Complex data structures are trivial to create.
A text editor hardy needs concurrency it should be way fast enough on one core -
there is no "natural concurrency in the problem" - emacs used to run
blindingly fast
on a 40Mz PC with 128KB of memory - it's *not* a difficult problem making
something like emacs run fast enough (unless you want to use emacs to search
through GBytes of data, in which case you're probably should not be
using emacs anyway)
I wrote a simple emacs editor years ago - it's in the widgets subdirectory
of http://www.sics.se/~joe/ex11/download/release-2.5.tgz
The only tricky part was not the emacs logic, but the screen display
and catching the
keystrokes and mouse events.
/Joe
Toby Thain
On 1-Nov-09, at 1:31 AM, Ted Henry wrote:
> With 100 core chips probably coming to desktop computers in the
> near future,
> I wonder about writing a multiprocess vi-like or emacs-like text
> editor
> (probably more emacs-like in implementation.) Sometimes single
> threaded
> editors really bog down when searching multiple files,
This would seem to be I/O bound.
> syntax highlighting
> multiple files.
If you can't get syntax highlighting working fast enough on today's
single core, your problems aren't going to be solved by language choice.
--Toby
> I'm not that knowledgeable in the Erlang world. It seems
> like Erlang's fault tolerance and ability to upgrade code without
> restarting
> would not be big advantages for a text editor. Also a text editor
> isn't
> usually distributed across multiple physical machines. I think the
> main
> advantage would be Erlang's processes for concurrency. Would Erlang
> offer a
> good advantage over a programming language with software transactional
> memory (e.g. Haskell, Clojure)? Would *you* use Erlang to program a
> text
> editor or desktop application?
>
> Very interested to read your thoughts.
>
> Ted
Ted Henry
> Text process applications involve mainly list processing.
> If lists don't cut it then custom abstractions can be easily made.
In the context of a text editor, by "custom abstractions can be easily
made", do you mean something written in C (e.g. gap buffer) and linked
into the Erlang program with an FFI?
Ted
Richard O'Keefe
On Nov 1, 2009, at 6:31 PM, Ted Henry wrote:
> With 100 core chips probably coming to desktop computers in the near
> future,
> I wonder about writing a multiprocess vi-like or emacs-like text
> editor
> (probably more emacs-like in implementation.)
David Warren used to use a home-brew text editor he'd written in Prolog,
so why not?
>
> Would Erlang offer a
> good advantage over a programming language with software transactional
> memory (e.g. Haskell, Clojure)? Would *you* use Erlang to program a
> text
> editor or desktop application?
What exactly do you want transactional memory _for_?
Have you read Joe Armstrong's blog entry about STM in Erlang?
One issue in a text editor is undo. I have my own small but powerful
text editor written in C that I still use. The only thing I've ever
really missed in it is a good undo. Erlang should at least remove
the temptation to use data structures that make it hard to provide
undo. (AVL DAGs in Erlang, anyone?)
My editor could keep up with me on a PDP-11/60. So it's not things
that require human involvement that would benefit from concurrency.
Rendering with fancy fonts and bizarre hard-to-read colours is
more demanding, but then, who needs to do _that_?
So the most interesting questions are not "what language" or even "what
features", but
- what is it that we want extra computing power to DO when we are
"editing", and
- what concurrency pattern(s) is(are) useful to express that?
Richard O'Keefe
On Nov 2, 2009, at 3:54 AM, Joe Armstrong wrote:
> On Sun, Nov 1, 2009 at 6:54 AM, Jayson Vantuyl <kag...@souja.net>
> wrote:
>> Ted,
>>
>> Well, if I were doing this, I probably wouldn't do it in Erlang.
>> Text
>> handling in Erlang is byzantine at best, so an editor very well
>> could be an
>> exercise in pain.
>
> I beg to differ.
>
> In my opinion Erlang is brilliant at handling text - text is stored in
> lists and list processing is blindingly fast and there are large
> numbers of library functions
> that work on lists. A very unpainful experience.
For what it's worth, ermacs has a 'cords.erl' module written by
Luke Gorrie that stores a document as a tree of binaries.
The thing very few people realise is just how very LITTLE help
strings give you for text processing. For example, back in 1979
a friend of mine wrote a batch text editor for IBM mainframes.
(Think of it as an MVS version of sed(1), but a *lot* dumber.)
It took him 150 pages of PL/I, and he had to fight PL/I strings
every step of the way. When I started at Edinburgh and had to
learn C, I wrote a version of his program in C, *with* undo,
and it took me something between 10 and 15 pages of C (I no
longer recall precisely). Why was C so much better? Because
it DIDN'T have strings. It had completely general purpose
data types (like arrays), and while it didn't actually help me
much, it didn't get in my way either. It gave me the _building
blocks_ I needed.
Edinburgh had a succession of text editors written in Pop-2.
There was the "77 editor" written by Boyer and Moore (of Boyer/Moore
theorem prover fame). The architecture of that was later written up
as a Xerox report that I have somewhere. Then there was the Pop
Editor, which became the Display-Oriented Pop Editor (DOPE), which
was rather Emacs-like. (If by any chance Dave Bowen should read
this, hi!) What data structure did that use? A list of records,
each with a block of characters and some counters, plus an overflow
area for type-in. Not strings, as such.
>
> A text editor hardy needs concurrency it should be way fast enough
> on one core -
> there is no "natural concurrency in the problem" - emacs used to run
> blindingly fast
> on a 40Mz PC with 128KB of memory - it's *not* a difficult problem
> making
> something like emacs run fast enough (unless you want to use emacs
> to search
> through GBytes of data, in which case you're probably should not be
> using emacs anyway)
Grepping the R12B-5 sources on a 500 MHz CPU, 84 MHz memory machine
took 20 seconds. Grepping a 500MHz file took 24 seconds. On a 2.2 GHz
CPU, 667 Mhz memory machine, the same grep took 6 seconds. The same
search would take maybe a second using Informatio Retrieval techniques.
The limiting factor on the fast machine seems to be how fast the data
can be pulled off the disc, and more cores would not help.
One of the programs hanging around on my disc is SubEthaEdit,
which is a collaborative editing program. Every so often I accidentally
invoke it and am reminded of its existence.
What about collaborative editing?
What about fusing version control and collaborative editing?
For code, isn't testing going to get a bigger payoff from multicore?
Robert Virding
> On Sun, Nov 1, 2009 at 6:54 AM, Jayson Vantuyl <kag...@souja.net> wrote:
> > Ted,
> >
> > Well, if I were doing this, I probably wouldn't do it in Erlang. Text
> > handling in Erlang is byzantine at best, so an editor very well could be
> an
> > exercise in pain.
>
> I beg to differ.
>
> In my opinion Erlang is brilliant at handling text - text is stored in
> lists and list processing is blindingly fast and there are large
> numbers of library functions
> that work on lists. A very unpainful experience.
>
Joe is absolutely right here. The problem is not the lack of traditional
string handling, the problem is not realising how powerful and easy it is to
use lists instead. It is necessary to "think outside the box"* of strings.
Robert
* That is a terrible expression but I couldn't think of a better one just
now.
Michael Turner
But aren't text editors supposed to be written in TECO?
So, obviously, the first order of business is a TECO interpreter
implemented in Erlang, taking all possible advantage of its ease of
hotswapping modules, and of seamlessly scaling to many processes and
many processors.
It could be a challenge. I believe that nobody has looked at this
particular problem before. I feel as if I am standing silent, with a
wild surmise, upon a peak in Darien.
-michael turenr
Alpar Juttner
> A text editor hardy needs concurrency it should be way fast enough on one core -
> there is no "natural concurrency in the problem" - emacs used to run
> blindingly fast
> on a 40Mz PC with 128KB of memory - it's *not* a difficult problem making
> something like emacs run fast enough (unless you want to use emacs to search
> through GBytes of data, in which case you're probably should not be
> using emacs anyway)
An editor like emacs does not need multicore, but it _does_ need
concurrency, as it could provide much better responsiveness. When emacs
blocks for a couple of seconds, it is already annoying. When it blocks
for tens of seconds, it can be a considerable pain.
* Several years ago, I had gave up using a Emacs-RMAIL, because
fetching new mail took a long time on our environment and this
process blocks the whole emacs.
* Emacs can open remote files through ssh/ftp. A wonderful feature
but opening and (auto)saving may take quite a while through the
net and these processes also block whole emacs.
* Emacs-w3 is also practically unusable due to its blocking
behavior.
In some important cases the non-blocking processing are solved in emacs
(such as in case of the compile command or the gdb interface) but with
complex design and sometimes with dirty hacking.
An inherently concurrent design would immediately eliminate all of these
issues.
Regards,
Alpar
Robert Virding
>
> But aren't text editors supposed to be written in TECO?
>
> So, obviously, the first order of business is a TECO interpreter
> implemented in Erlang, taking all possible advantage of its ease of
> hotswapping modules, and of seamlessly scaling to many processes and
> many processors.
>
> It could be a challenge. I believe that nobody has looked at this
> particular problem before. I feel as if I am standing silent, with a
> wild surmise, upon a peak in Darien.
>
Ah, a true believer in the old gods! Yes, of course this is the true path to
greatness, but unfortunately many young apprentices have no knowledge of the
old ways.
Robert
Steve Vinoski
<ajuttn...@googlemail.com>wrote:
> An editor like emacs does not need multicore, but it _does_ need
> concurrency, as it could provide much better responsiveness. When emacs
> blocks for a couple of seconds, it is already annoying. When it blocks
> for tens of seconds, it can be a considerable pain.
>
I use a package called emacs-jabber for doing IM from within emacs.
<http://emacs-jabber.sourceforge.net/>
I've never had it block emacs on me. When I started looking into how it was
doing event handling, I found that it uses something called fsm.el:
<http://www.mail-archive.com/gnu-emac...@gnu.org/msg00308/fsm.el>
which has the following interesting comment near the top of the source code:
;; fsm.el is an exercise in metaprogramming inspired by gen_fsm of
;; Erlang/OTP. It aims to make asynchronous programming in Emacs Lisp
;; easy and fun. By "asynchronous" I mean that long-lasting tasks
;; don't interfer with normal editing.
It's based on gen_fsm -- we've come full circle. :-)
More emacs packages should make use of this to avoid blocking.
--steve
Luke Gorrie
> Very interested to read your thoughts [on text editors in Erlang]
The desire to rewrite Emacs in my-pet-language is a form of insanity!
The only cure I know of is to actually do it, and then you'll realise
that Emacs Lisp is a great language to write Emacs in and wonder what
all the fuss is about :-)
I have a strong suspicion that loving Emacs in the 21st century is
also a form of insanity, but I'm not sure of the cure yet.. I think it
involves making friends with Smalltalk hackers :-)
-Luke
P.S. Apologies for probable poor threading of this message.
Ted Henry
<ajuttn...@googlemail.com> wrote:
> An editor like emacs does not need multicore, but it _does_ need
> concurrency, as it could provide much better responsiveness. When emacs
> blocks for a couple of seconds, it is already annoying. When it blocks
> for tens of seconds, it can be a considerable pain.
>
> * Several years ago, I had gave up using a Emacs-RMAIL, because
> fetching new mail took a long time on our environment and this
> process blocks the whole emacs.
> * Emacs can open remote files through ssh/ftp. A wonderful feature
> but opening and (auto)saving may take quite a while through the
> net and these processes also block whole emacs.
> * Emacs-w3 is also practically unusable due to its blocking
> behavior.
Thanks for your response and examples. These issues seem more related
to Emacs as an OS (as some non-Emacs users like to accuse) rather than
Emacs as a simple text editor...not that I'd be shooting for the
simple text editor. ;-)
Ted
Ted Henry
> The desire to rewrite Emacs in my-pet-language is a form of insanity!
And what if you want to invent your own language and then implement an
Emacs in that language? :-S
Ted
Richard O'Keefe
On Nov 2, 2009, at 8:20 PM, Alpar Juttner wrote:
> * Several years ago, I had gave up using a Emacs-RMAIL, because
> fetching new mail took a long time on our environment and this
> process blocks the whole emacs.
I use "Mail" version 3.6 on a 2.2GHz intel Mac running MacOS 10.5.7.
I can assure you that l o n g pauses to fetch new mail are
by no means confined to Emacs. Activity Monitor reports that Mail
is using 12 ... no wait ... 11 threads, no wait, it's 12 again.
So simply having and using concurrency is no guarantee that you won't
get long waits.
Our sysadmins made a unilateral decision to move everyone's files off
their local machines onto a file server. There are now frequent
jarring waits while Mail waits for a file server in order to touch
some file I don't particularly want it to touch at that time.
The fundamental issue is *system design*. It's thinking about what
kinds
of delays there might be and arranging for the program to let you do
something else (such as compose a message) while that's happening. For
this, threads are a great help, but not a necessity. (Well, you _could_
write the whole thing in assembly code (:-) (:-).)
>
> An inherently concurrent design would immediately eliminate all of
> these
> issues.
Mail is evidence that an inherently concurrent design *as such* need not
eliminate *any* of these issues. Concurrency makes it *easier* to
create
good designs, but good design never just happens.
Michael Turner
Great bumpersticker line! Copyright it immediately.
The main useful purpose I can see for using Erlang to implement a
text-editor-cum-general-purpose-shell-programming-environment (Emacs
being exemplary) is to make a promotional statement about Erlang: your
success says, "Look, Erlang is not just great for what it was designed
for, it's also quite adequate for other purposes."
Wings3D already does a pretty good job in that department, though.
Offhand, I can't think of a good reason to write a 3D modeler in Erlang
specifically. What Wings3D says, in effect, is that if you just want to
write something complex for fun and learning value, you shouldn't worry
that Erlang would hamper you significantly -- it might make some things
harder, but not prohibitively so.
Wings3D is better than some hypothetical Emacs clone in another
(promotional) respect: your average *user* can relate to it better.
Emacs grew up in the glass tty days, it's got a face (and a raison
d'etre) that only a programmer could love. Most people these days
don't even know what you mean when you say "text editor", any more
than they know what you mean when you say "Erlang is Turing-complete."
-michael turner
Torben Hoffmann
Since
"Look, Erlang is not just great for what it was designedfor, it's also quite
adequate for other purposes."
is part of the reason for wanting to inflict almost infinite pain on
one-self - I mean, writing a text editor is hard work - I would like to
point the attention to the original domain Erlang was supposed to address
(nice form @ http://en.citizendium.org/wiki/Erlang%27s_original_domain):
1. Handling of a very large number of concurrent activities
2. Actions to be performed at a certain point in time or within a certain
time
3. Systems distributed over several computers
4. Interaction with hardware
5. Very large software systems
6. Complex functionality such as feature interaction
7. Continuous operation for many years
8. Software maintenance (reconfiguration et cetera) without stopping the
system
9. Stringent quality and reliability requirements
10. Fault tolerance both to hardware failures and software errors
My argument, as that of many others smarter than me, is that whenever you
have a problem domain that has needs that matches with the above list then
Erlang might be the tool to use.
Web servers, instant messaging and enterprise messaging are other areas that
have similar requirements, and - surprise, surprise - quite a few Erlang
solutions has turned up in these areas.
Final remark: if you are going to invest your time and energy into something
why not solve a new problem and try to make some money in the process?
I love Erlang,
Torben - High Priest of Erlang (self-proclaimed)