Custom Pygments' lexer not found

[Denis Bitouzé]

Hi,

(Question already asked but still not answered on SO:
https://stackoverflow.com/q/77329701/1682470)

I created Pygments customized lexer and style:

- `acetexlexer.py` (lexer file),
- `acedracula.py` (style file),

that work pretty well since the following command returns the expected
result:

┌────
│ pygmentize -O style=acedracula -x -l acetexlexer.py:AceTexLexer test.tex
└────

But I can't get them to work with Sphinx, despite extensive research on
the Internet.

I tried for instance the following setup, partially based on:

┌────
│ https://github.com/AdaCore/aunit/blob/v21.0.0/doc/share/ada_pygments.py
│ https://github.com/AdaCore/aunit/blob/v21.0.0/doc/share/conf.py
└────

that I found from:

┌────
│ https://github.com/sphinx-doc/sphinx/issues/9544
└────

1. At the root of the `source` of the project, I created a `_pygments`
subdirectory containing the lexer and style files (`acetexlexer.py` and
`qacedracula.py`).
2. The relevant lines of `acetexlexer.py` are:
┌────
│ from pygments.lexer import inherit, bygroups
│ from pygments.token import Text, Comment, Operator, Keyword, Name, String, \
│ Number, Punctuation, Generic, Other, Whitespace
│ from pygments.lexers.markup import TexLexer
│
│ __all__ = ['AceTexLexer']
│
│ class AceTexLexer(TexLexer):
│ aliases = ['xtex', 'xlatex']
└────
3. The relevant lines of my `conf.py` file are:
┌────
│ import os
│ import sys
│ sys.path.insert(0, os.path.abspath('.'))
│ sys.path.append(os.path.abspath("./_ext"))
│ sys.path.append(os.path.abspath("./_pygments"))
│
│ def setup(app):
│ from acetexlexer import AceTexLexer
│ app.add_lexer('xlatex', acetexlexer.AceTexLexer)
└────

But, when I run:

┌────
│ sphinx-build -v -j auto source build/html source/test.md
└────

I get:

┌────
│ [...]/source/test.md:16: WARNING: Pygments lexer name 'xlatex' is not known.
└────

What am I doing wrong?

Thanks!
--
Denis

[Slavko]

Dňa 22. októbra 2023 13:51:21 UTC používateľ "Denis Bitouzé" <denis....@gmail.com> napísal:

> │ def setup(app):
> │ from acetexlexer import AceTexLexer
> │ app.add_lexer('xlatex', acetexlexer.AceTexLexer)
> └────

I cannot comment lexer implementation itself, i never did
own, but i use:

def setup(app):
from pylogrotate.lexer import LogrotateLexer
app.add_lexer('logrotate', LogrotateLexer())

Note the "()" at end, to pass instance (object), you are
passing class. And as you directly imported "AceTexLexer",
you don't need to use "acetexlexer.AceTexLexer", just
"AceTexLexer()".

regards


--
Slavko
https://www.slavino.sk/

[Denis Bitouzé]

Le 22/10/23 à 14h20, Slavko a écrit :

> Dňa 22. októbra 2023 13:51:21 UTC používateľ "Denis Bitouzé" <denis....@gmail.com> napísal:
>
>> │ def setup(app):
>> │ from acetexlexer import AceTexLexer
>> │ app.add_lexer('xlatex', acetexlexer.AceTexLexer)
>> └────
>
> I cannot comment lexer implementation itself, i never did
> own, but i use:
>
> def setup(app):
> from pylogrotate.lexer import LogrotateLexer
> app.add_lexer('logrotate', LogrotateLexer())
>
> Note the "()" at end, to pass instance (object), you are
> passing class.

Unfortunaltely, that doesn't work either. And I precisely didn't use
these parentheses because of :

┌────
│ https://github.com/sphinx-doc/sphinx/issues/9544#issuecomment-898865061
└────

where tk0miya said:

┌────
│ add_lexer() takes a Lexer class instead of an instance. Could you pass
│ the AdaLexer as a class?
└────

> And as you directly imported "AceTexLexer", you don't need to use
> "acetexlexer.AceTexLexer", just "AceTexLexer()".

Ah, thanks.

Regards.
--
Denis

[Slavko]

Dňa 22. októbra 2023 14:50:58 UTC používateľ "Denis Bitouzé" <denis....@gmail.com> napísal:

>Unfortunaltely, that doesn't work either. And I precisely didn't use
>these parentheses because of :

Yes, you are right, my notes are outdated, i checked conf and it is
without (), I am sorry.

[Denis Bitouzé]

Hi,

Le 22/10/23 à 15h51, Denis Bitouzé a écrit :

> (Question already asked but still not answered on SO:
> https://stackoverflow.com/q/77329701/1682470)
>
> I created Pygments customized lexer and style:
>
> - `acetexlexer.py` (lexer file),
> - `acedracula.py` (style file),
>
> that work pretty well since the following command returns the expected
> result:
>
> ┌────
> │ pygmentize -O style=acedracula -x -l acetexlexer.py:AceTexLexer test.tex
> └────
>
> But I can't get them to work with Sphinx, despite extensive research on
> the Internet.

Okay, I succeeded, thanks to:

┌────
│ https://stackoverflow.com/a/62084359/1682470
└────

For the record, here is how I could make it work.

1. In fact, no need to create a `_pygments/` directory containing:
- `acetexlexer.py` (lexer file),
- `acedracula.py` (style file).
2. (As I did from the beginning, )
- the `acetexlexer.py` file is in the directory (in my case)
`~/.local/lib/python3.10/site-packages/pygments/lexers`,
- the `acedracula.py` file is in the directory (in my case)
`~/.local/lib/python3.10/site-packages/pygments/styles` with:
┌────
│ 'acedracula': 'acedracula::AceDraculaStyle',
└────
in the `STYLE_MAP` dictionary in the `__init__.py` file in this directory.
3. In the `source/_ext` directory, I created a file named (for the
moment) `test.py` containing:
┌────
│ from pygments.lexers.acetexlexer import AceTexLexer
│
│ def setup(app):
│ app.add_lexer('xlatex', AceTexLexer)
│ return {
│ 'parallel_read_safe': True,
│ 'parallel_write_safe': True,
│ }
└────
(the 4 last line in order to be able to distribute the build over
N processes in parallel: `-j` option of `sphinx-build`).
4. In `conf.py`, I removed the `def setup(app):` function and
everything related to lexers but added to the `extensions` list, the
'test' extension (see `test.py` above).

So far, so good.

Now, I've no idea what could be a proper name instead of `test.py` for
my `test` “extension”. I guess reusing the same name `acetexlexer.py` as
used for my own lexer would be a bad idea. Any suggestion is welcome.

Thanks!
--
Denis

[Slavko]

Dňa 23. októbra 2023 10:10:41 UTC používateľ "Denis Bitouzé" <denis....@gmail.com> napísal:

>Okay, I succeeded, thanks to:

Nice ;-)

>Now, I've no idea what could be a proper name instead of `test.py` for
>my `test` “extension”. I guess reusing the same name `acetexlexer.py` as
>used for my own lexer would be a bad idea. Any suggestion is welcome.

my_lexers.py :-D

[Denis Bitouzé]

Le 23/10/23 à 10h42, Slavko a écrit :

> Dňa 23. októbra 2023 10:10:41 UTC používateľ "Denis Bitouzé"
> <denis....@gmail.com> napísal:
>

>>Now, I've no idea what could be a proper name instead of `test.py` for
>>my `test` “extension”. I guess reusing the same name `acetexlexer.py` as
>>used for my own lexer would be a bad idea. Any suggestion is welcome.
>
> my_lexers.py :-D

;)

The problem is that I seem to already have a file dedicated to my lexer:
`acetexlexer.py`. So my question is, since my lexer file is
`acetexlexer.py`, what is the purpose/role/status/whatever of the file
I've (temporarily) named `test.py` and containing:

┌────
│ from pygments.lexers.acetexlexer import AceTexLexer
│
│ def setup(app):
│ app.add_lexer('xlatex', AceTexLexer)
│ return {
│ 'parallel_read_safe': True,
│ 'parallel_write_safe': True,
│ }
└────

Regards.
--
Denis