sphinx tcl highlighting problem

[bulletmark]

The small tcl code fragment:

.. code-block:: tcl

   :linenos:

   set a 1

   puts ${a}

fails to render in sphinx but does render when I run it through pygmentize manually. Also runs fine in tclsh of course. Have tried sphinx 1.0.7, 1.0.8, and 1.1. If I take out the {} then it renders ok. This problem also occurs with other valid tcl contructs like "\" end of line continuation etc. Fails as a literal include also. Beats me.

[Georg Brandl]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I can't reproduce that this works with standalone pygmentize: Pygments also
generates an error token for the snippet you posted.

Insofar, if it is valid TCL, this is a bug in Pygments.

cheers,
Georg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iEYEARECAAYFAk6MXycACgkQN9GcIYhpnLAixACdF4O4Apk58s7NlJpsR+PT/rsK
GxEAn3Jr2TPYIjSI60T9VNqmmQu3CTGG
=17Ts
-----END PGP SIGNATURE-----

[bulletmark]

I was running pygmentize to my terminal and after your comment I have realized that some of the output is underlined so I guess that means pygmentize believes it is invalid syntax?

Either way, why does sphinx just fail to render the output at all if the highlighter trips over some syntax? Is there any message where a user can see the reason why his code is not highlighting? I spent ages on this problem thinking I had an environment problem. The code section is not even being line numbered at all. More particularly, can a user tell sphinx to ignore syntax errors and render (as best it can) anyway? I imagine there will always be different language dialects which a general purpose highlighter will not completely accommodate.

[Georg Brandl]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/05/11 16:02, bulletmark wrote:
> I was running pygmentize to my terminal and after your comment I have
> realized that some of the output is underlined so I guess that means
> pygmentize believes it is invalid syntax?

Yep, this is how the default style shows Pygments "Error" tokens.

> Either way, why does sphinx just fail to render the output at all if the
> highlighter trips over some syntax? Is there any message where a user can
> see the reason why his code is not highlighting?

It's because then you can keep the default highlight language even if you have
a shell/text/whatever sample inbetween.

> I spent ages on this problem thinking I had an environment problem. The
> code section is not even being line numbered at all. More particularly, can
> a user tell sphinx to ignore syntax errors and render (as best it can)
> anyway? I imagine there will always be different language dialects which a
> general purpose highlighter will not completely accommodate.

Yep, we're planning an option to customize this in 1.1.

Georg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iEYEARECAAYFAk6MbXgACgkQN9GcIYhpnLCwswCeNecHcgiZiBPKj5lcp35AKhUJ
FcAAoJwLXB2+t7L6ghsVtB5Wx6+g93lK
=Z0cv
-----END PGP SIGNATURE-----

[bulletmark]

Thanks Georg.

For the benefit of others also looking for an short-term solution, here is a crude workaround I have devised which seems to work. Just add the following file as a custom extension in your sphinx source dir:

from pygments.lexers.agile import TclLexer

from pygments import token

class myTclLexer(TclLexer):

    name = 'My custom Tcl syntax lexer'

    aliases = ['mytcl']

def setup(app):

    # HACK: suppress tcl syntax parsing errors

    token.Error = token.Other

    # Add our custom lexer

    app.add_lexer(myTclLexer.aliases[0], myTclLexer())

It seems that just deriving a custom lexer allows sphinx syntax highlighting to work. The "HACK" monkey patch line then suppresses error highlighting (i.e. the red box in html). You can choose to install this file as a system wide pygments lexer using setuptools if you prefer.

[bulletmark]

On Thursday, 6 October 2011 00:45:12 UTC+10, Georg Brandl wrote:

Yep, we're planning an option to customize this in 1.1.

I can't see this option in the release notes or documentation for recently released 1.1. Is it still being considered?