a lexilla lexer crashes

[ISTICG BERRECHID]

hello, ive been trying to get a lexer for 'typst' to work on scintilla but no luck,
basically i found the lexer in here:
https://github.com/orbitalquark/scintillua/pull/149

it wasnt hard to port it to lexilla, but if i try it, it just crashes scintilla, however, disabling the embedding lines fixes it, but now the markup grammar doesnt get embedded in code grammar and vice versa

is the problem relative to the library or the implementation of the lexer itself???

(ps: i tried using the same lexer as is using some automated tests with something i found called scintillua, but it just halts!) 

[Mitchell]

Hi,

> On Aug 3, 2025, at 12:18 PM, ISTICG BERRECHID <ist...@gmail.com> wrote:
>
> hello, ive been trying to get a lexer for 'typst' to work on scintilla but no luck,
> basically i found the lexer in here:
> https://github.com/orbitalquark/scintillua/pull/149
>
> it wasnt hard to port it to lexilla, but if i try it, it just crashes scintilla, however, disabling the embedding lines fixes it, but now the markup grammar doesnt get embedded in code grammar and vice versa
>
> is the problem relative to the library or the implementation of the lexer itself???

It’s a problem with your lexer implementation.

Lexilla and Scintillua (where you found the lexer from) approach lexing in very different ways. I’m surprised it “wasn’t hard” to port a Scintillua Lua lexer to a Lexilla C++ lexer. Lexilla does not support embedded lexers like Scintillua does. You’d have to lex Markdown from within your Typst lexer, much like the HTML lexer lexes embedded languages like PHP.

Since you’ve written the lexer and you’re getting a crash, I’d use a debugger to inspect why the lexer is crashing. The Scintilla documentation (https://scintilla.org/ScintillaDoc.html) has some links on how to write lexers that could be helpful. I would start with a basic lexer that works/doesn’t crash and then slowly add Typst features until you have the desired functionality.

> (ps: i tried using the same lexer as is using some automated tests with something i found called scintillua, but it just halts!)

Scintillua can drop into an existing Scintilla-based editor (like SciTE) with some configuration, so you should be able to get it to work with the Typst lexer you found. Feel free to open up a new discussion if you’d like help in that regard (https://github.com/orbitalquark/scintillua/discussions).

Cheers,
Mitchell

[ISTICG BERRECHID]

Hi, thank you for answering,

I choose to go with the original lua lexer and indeed got it to work with SciTE!

I was also able to identify the source of the problem: recursive embedding

See, typst allows you to embed 'code' blocks within markup, and within that code block, you can also embed markup and so on

For example:

`

This is a *markup* text. #{

// This is code

let x = [
  This is text
  #let z = "This is code"
]
};

`

Keywords inside code blocks don't need the pre '#', but do when in markup

I believe Scintillua doesn't allow this? If not is there any intention to implement it?

[Mitchell]

Hi,

> On Aug 3, 2025, at 3:02 PM, ISTICG BERRECHID <ist...@gmail.com> wrote:
>
> Hi, thank you for answering,
> I choose to go with the original lua lexer and indeed got it to work with SciTE!
> I was also able to identify the source of the problem: recursive embedding
>
> See, typst allows you to embed 'code' blocks within markup, and within that code block, you can also embed markup and so on
> For example:
> `
> This is a *markup* text. #{
> // This is code
> let x = [
> This is text
> #let z = "This is code"
> ]
> };
> `
> Keywords inside code blocks don't need the pre '#', but do when in markup
>
> I believe Scintillua doesn't allow this? If not is there any intention to implement it?

I have not found a way to get recursive embedding to work. It’s a hard problem to solve.

Cheers,
Mitchell

> On Sunday, August 3, 2025 at 5:46:59 PM UTC+1 Mitchell wrote:
> Hi,
>
> > On Aug 3, 2025, at 12:18 PM, ISTICG BERRECHID <ist...@gmail.com> wrote:
> >
> > hello, ive been trying to get a lexer for 'typst' to work on scintilla but no luck,
> > basically i found the lexer in here:
> > https://github.com/orbitalquark/scintillua/pull/149
> >
> > it wasnt hard to port it to lexilla, but if i try it, it just crashes scintilla, however, disabling the embedding lines fixes it, but now the markup grammar doesnt get embedded in code grammar and vice versa
> >
> > is the problem relative to the library or the implementation of the lexer itself???
>
> It’s a problem with your lexer implementation.
>
> Lexilla and Scintillua (where you found the lexer from) approach lexing in very different ways. I’m surprised it “wasn’t hard” to port a Scintillua Lua lexer to a Lexilla C++ lexer. Lexilla does not support embedded lexers like Scintillua does. You’d have to lex Markdown from within your Typst lexer, much like the HTML lexer lexes embedded languages like PHP.
>
> Since you’ve written the lexer and you’re getting a crash, I’d use a debugger to inspect why the lexer is crashing. The Scintilla documentation (https://scintilla.org/ScintillaDoc.html) has some links on how to write lexers that could be helpful. I would start with a basic lexer that works/doesn’t crash and then slowly add Typst features until you have the desired functionality.
>
> > (ps: i tried using the same lexer as is using some automated tests with something i found called scintillua, but it just halts!)
>
> Scintillua can drop into an existing Scintilla-based editor (like SciTE) with some configuration, so you should be able to get it to work with the Typst lexer you found. Feel free to open up a new discussion if you’d like help in that regard (https://github.com/orbitalquark/scintillua/discussions).
>
> Cheers,
> Mitchell
>

> --
> You received this message because you are subscribed to the Google Groups "scintilla-interest" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scintilla-inter...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/scintilla-interest/1d397e8b-1dea-4332-80bd-fb2325021dd7n%40googlegroups.com.