Regression highlighting java code snippet

22 views
Skip to first unread message

Andrea Aime

unread,
Jan 1, 2022, 5:10:47 AMJan 1
to sphinx-users
Hi all,
I work on the GeoTools project, which has been successfully building its documentation using Sphinx for years. Thanks for sharing Sphinx with the rest of the world :-D

In the recent days one of our Github actions picked up the latest versions of Sphinx and Pygments, which results in a build warning (which we treat as a failure, trying to keep documentation in good shape). The versions involved are:
  • Sphinx is version 4.3.2
  • Pygments version is 2.11.1

The directive failing is a simple:

.. literalinclude:: /../src/main/java/org/geotools/tutorial/csv3/parse/CSVStrategy.java
:language: java
:start-after: // docs start createBuilder
:end-before: // docs end findMostSpecificTypesFromData

and the file included is this one:

The error message just states lexing did not work:

     [exec] Warning, treated as error:
     [exec] /home/aaime/devel/git-gt/docs/user/tutorial/datastore/strategy.rst:245:Could not lex literal_block as "java". Highlighting skipped.

However I've tried to use "pygmentize" on the command line, it highlights the file without errors. Also tried leaving just the section of code that is meant to be highlighted, same results, no errors, e.g:

> pygmentize   -V
Pygments version 2.11.1, (c) 2006-2021 by Georg Brandl, Matthäus Chajdas and contributors.
> pygmentize   -l java /home/aaime/devel/git-gt/docs/src/main/java/org/geotools/tutorial/csv3/parse/CSVStrategy.java
(works fine, output omitted)

I'm stuck here... any idea of what I could do to address this issue?

Cheers
Andrea



Komiya Takeshi

unread,
Jan 1, 2022, 5:48:39 AMJan 1
to sphinx...@googlegroups.com
Hi Andrea,

Happy new year!

I guess new pygments would get errors internally on processing your code.
Could you try passing "-F raiseonerror" option to pygmentize?

Thanks,
Takeshi KOMIYA

2022年1月1日(土) 19:10 Andrea Aime <andre...@gmail.com>:
> --
> You received this message because you are subscribed to the Google Groups "sphinx-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sphinx-users/84e651ea-ae48-4153-b51d-76a1f5b90944n%40googlegroups.com.

Andrea Aime

unread,
Jan 3, 2022, 7:51:11 AMJan 3
to sphinx-users
Oh, I overlooked that option. Indeed it provides information:

*** Error while highlighting:
pygments.filters.ErrorToken: =
   (file "/usr/local/lib/python3.8/dist-packages/pygments/filters/__init__.py", line 782, in filter)
*** If this is a bug you want to report, please rerun with -v.
<String, Class<?>> findMostSpecificTypesFromData(
            CsvReader csvReader, String[] headers) throws IOException {
        Map<String, Class<?>> result = new HashMap<>();
        // start off assuming Integers for everything
        for (String header : headers) {
            result.put(header, Integer.class);
        }
        // Read through the whole file in case the type changes in later rows
        while (csvReader.readRecord()) {
            String[] record

That's all of the output. Oh, maybe I see what's going on. The last item is "record",
maybe some Java 17 compatibility crept in, and considers "record" a keyword?
However the java compiler does not, tried with Java 17 with no issues, so it looks
like a bug.

I'll raise a bug report against pygments, thanks a lot.

Cheers
Andrea
Reply all
Reply to author
Forward
0 new messages