Can Luddite be used to write a lexer for any Scintilla based editor?

[Randy Kramer]

In a comment to scintilla-Feature Requests-3531865, Neil (Hodgson) said this:

<quote>
Message:
Lexers are written in code for control and performance. There are other
ways to write lexers. The most popular appear to be
* Scintillua which uses Lua LPEG parsers http://foicica.com/scintillua/
* Komodo's Luddite http://docs.activestate.com/komodo/4.4/luddite.html
</quote>

My question is (for Neil or anybody) is: can I write a Lexer in Luddite and
use it in, for example, Geany, Scite, or basically any Scintilla based
editor?

Randy Kramer

[Lex Trotman]

If, as the active state documentation states, it compiles into a
standard scintilla lexer then it "should" be able to be used in any
Scintilla editor. But only if the compiled lexer is fully self
contained and doesn't need any of Komodo code to run.

Cheers
Lex

>
> Randy Kramer
>
> --
> You received this message because you are subscribed to the Google Groups "scintilla-interest" group.
> To post to this group, send email to scintilla...@googlegroups.com.
> To unsubscribe from this group, send email to scintilla-inter...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scintilla-interest?hl=en.
>

[Eric Promislow]

I've never tried it that way, but I don't see why not. The key is that
Luddite doesn't
generate a C++ lexer. It generates a lexres file, a pseudo-binary
file. The full path
to this file is placed in the first entry in the first keyword list
for the language you
want to lex (which is an instance of SCLEX_UDL), and ColouriseTemplate1Doc
will load it the first time it sees it and build essentially a state
machine that
drives the lexing process.

All of Luddite and sample ko*Language.py files for about a dozen languages
are all available under MPL/GPL/LGPL at
http://svn.openkomodo.com/openkomodo/browse/openkomodo/trunk/src/udl
and
the actual lexer is at
http://svn.openkomodo.com/openkomodo/view/openkomodo/trunk/contrib/patches/scintilla/lexers/LexUDL.cxx
with patches in the containing directory, in particular
http://svn.openkomodo.com/openkomodo/view/openkomodo/trunk/contrib/patches/scintilla/includes.patch

Unlike everything I said about keywords in lexers in the recent Visual Prolog
thread, keywords are defined in a .udl file, are compiled into the .lexres file,
and are managed by LexUDL.cxx, not a containing application.

UDL files are limited to 54 styles, as it was designed before indicators were
added. The styles are defined in
http://svn.openkomodo.com/openkomodo/view/openkomodo/trunk/contrib/patches/scintilla/includes.patch

Hope this helps you get started. Let me know if you run into any problems.

- Eric

[Randy Kramer]

Eric and Lex,

Thanks for your responses!

Now I understand, LexUDL.cxx is the key:

> keywords are defined in a .udl file, are compiled into the
> .lexres file, and are managed by LexUDL.cxx, not a containing application.

So, as long as I "install" LexUDL.cxx and arrange to compile my (custom) .udf
file into a .lexres file, I should be good to go.

BTW, I started to write a lexer (for TWiki markup language, plus some
modifications) in C++ but that effort is currently on hold (lack of time,
frustration). When I get time and the ambition to restart that effort, I'll
look more closely at Luddite. (C++ became very cumbersome with the things I
wanted to do.)

Randy Kramer

[Eric Promislow]

Luddite isn't ideal for some of the reduced markup languages because
it works on one line at a time, and it provides limited state manipulation
facilities, mostly for programming-language constructs like
arbitrary string delimiters and here-documents.

But if it works for you, it's certainly faster than writing a
from-scratch C++ lexer.

- Eric

[Randy Kramer]

Eric,

Thanks for the followup. The one line at a time might be the problem--one of
my modifications to the TWiki markup is that I use the mbox From header to
serve as a record separator.

If you're not familiar with the From header, it consists of an arbitrary
number of lines, it can look something like this:

From "file index: askRhk04" Mon May 28 07:31:07 2007
Date: 05/28/07 07:31 am
Subject: file index: askRhk04

(Normally the field after the From is the email address of the sender, but
I've found in the software that I plan to use (including kmail, nail/mail)
that I can put an arbitrary phrase in quotes and that works just fine.)

It can have any number of email header lines of the form:

<keyword>: <whatever>

and is finally ended by a blank line. For my purposes, I require that it have
at least one Date: line and at least one Subject: line, but there may be an
arbitrary number of such lines.

Does luddite have any ability to maintain state between lines? And then, for
this case, after having recognized a valid mbox From header, go back to the
From line and colorize and fold starting from there?

There are some other instances where I need to maintain state between lines,
but this is the easiest example to give.

Thanks,
Randy Kramer

[Randy Kramer]

Oh, forgot to mention the most important / critical application I want to
use--recoll, and the reason to use the mbox header is because recoll
recognizes that as a record separator. (At one time that was the only record
separator that recoll recognized, that may have changed by now. If it has, I
probably still prefer to use that for the flexibility it gives me to view
records in an email client.)

[Eric Promislow]

Sure, this is similar to the way Luddite recognizes that it's transitioned from
HTML markup to the JS inside a script tag, or CSS inside a style tag.

Best to follow the links I gave. Start with html-mainlex.udl and follow the
include's to get an idea of how it works.

- Eric

[Randy Kramer]

Eric,

Sounds good--thanks again!

Randy Kramer