Fatal (but not really?) charmap error

[Sean Fitzpatrick]

I am getting weird behaviour running pretext 2.45.0 on Windows.

(Maybe I would see this on Linux too but I'm stuck with Windows for the next two weeks.)

I am trying to build PDF, with some custom XSL.

The build appears to succeed. I get a .tex file, and then the CLI continues, running XeLaTeX without errors.

But between these two steps, a "report-publisher-variables" step runs, and it throws a fatal error. So even though the PDF was built successfully, it does not get copied over by the CLI, and it goes to the graveyard along with the rest of the temp directory.

Here is the relevant bit of log output:

INFO    : successful application of C:\Users\SEANFI~1\AppData\Local\Temp\ptxcli_kxjqgp6p\cli_xsl\apex-latex-style.xsl
INFO    : XSL conversion of C:\Users\Sean Fitzpatrick\Documents\github\APEXCalculusPTX\ptx\apex.ptx by C:\Users\Sean Fitzpatrick\.ptx\2.45.0\core\xsl\utilities\report-publisher-variables.xsl
INFO    : successful application of C:\Users\Sean Fitzpatrick\.ptx\2.45.0\core\xsl\utilities\report-publisher-variables.xsl
FATAL   : 'charmap' codec can't decode byte 0x9d in position 63479: character maps to <undefined>
INFO    : ------------------------
FATAL   : Failed to build without errors.  Exiting...

(Except it doesn't exit at this point! It builds the PDF, and then insists that it didn't.)

How do I go about tracking down the place in my source where the offending byte is?

[Sean Fitzpatrick]

I see someone else ran into this a few years ago, also on Windows. But that wasn't their main issue.

[Rob Beezer]

> I see someone else ran into this a few years ago, also on Windows.

Can you add a pointer? Anything valuable there?

I think we are going to prohibit you from being 100% reliant on a single Windows
machine for weeks at a time. ;-)

Rob

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-suppo...@googlegroups.com <mailto:pretext-
> support+u...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-support/
> f5fdcdd8-daa3-4d09-a7f9-0908d829d101n%40googlegroups.com <https://
> groups.google.com/d/msgid/pretext-support/f5fdcdd8-daa3-4d09-
> a7f9-0908d829d101n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Valerio Monti]

I reported an issue with charmap on Windows when building an EPUB from a source containing a <q> tag:

https://groups.google.com/g/pretext-dev/c/vyA57GFN8IY/m/codAj23uBgAJ

Every now and then I would retry the EPUB build, and the issue was still there. However, with PreTeXt-CLI 2.45 the build succeeded, so this should be unrelated to the issue reported by the OP.

Since I hadn't tested it for a long time, I can't tell which version actually fixed the issue. It may have been resolved when support for internationalized quotation marks was introduced. That also reminds me that I still need to work on localization.

All the best,

Valerio

[Sean Fitzpatrick]

The earlier thread that referenced the same 0x9d character is here: 

https://groups.google.com/g/pretext-support/c/xstrmXP_Tns/m/2t6UHUHQAAAJ

I think it might be the TM character. Maybe the one supplied by PreTeXt is causing trouble? I'll try removing any instances of it. 

But it seems like this error should not be fatal: PDF builds successfully, but the CLI refuses to copy it over.

To unsubscribe from this group and stop receiving emails from it, send an email to pretext-suppo...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-support/e4c973d2-274c-405b-8355-f5239c264760n%40googlegroups.com.

[Sean Fitzpatrick]

Removing <trademark/> from my source doesn't help. If I build with debugging on and keep the temp directories, I can get a completed PDF that doesn't seem to have anything wrong with it.

An error, maybe, but definitely not fatal!

[Sean Fitzpatrick]

Here is the larger traceback that I get with debugging on. It seems to happen after everything is done? I'm starting from the line where the .tex is built successfully:

INFO    : successful application of C:\Users\SEANFI~1\AppData\Local\Temp\ptxcli_jan7lj76\cli_xsl\apex-latex-style.xsl
DEBUG   : parsing the publisher file variables
DEBUG   : temporary directory for publisher variables: C:\Users\SEANFI~1\AppData\Local\Temp\ptx-3s21asjc
DEBUG   : file of publisher variables: C:\Users\SEANFI~1\AppData\Local\Temp\ptx-3s21asjc\pub_var.txt

INFO    : XSL conversion of C:\Users\Sean Fitzpatrick\Documents\github\APEXCalculusPTX\ptx\apex.ptx by C:\Users\Sean Fitzpatrick\.ptx\2.45.0\core\xsl\utilities\report-publisher-variables.xsl

DEBUG   : XSL conversion via C:\Users\Sean Fitzpatrick\.ptx\2.45.0\core\xsl\utilities\report-publisher-variables.xsl of C:\Users\Sean Fitzpatrick\Documents\github\APEXCalculusPTX\ptx\apex.ptx to C:\Users\SEANFI~1\AppData\Local\Temp\ptx-3s21asjc\pub_var.txt and/or into directory None with parameters {'cli.version': '2.45', 'publisher': 'C:/Users/Sean Fitzpatrick/Documents/github/APEXCalculusPTX/publication/publication-pdf.ptx', 'assembly.version-only': 'yes'}

INFO    : successful application of C:\Users\Sean Fitzpatrick\.ptx\2.45.0\core\xsl\utilities\report-publisher-variables.xsl

DEBUG   : determining value of publisher variable 'journal-name'
DEBUG   : Now working in directory C:\Users\SEANFI~1\AppData\Local\Temp\ptx-bvioxmfq
DEBUG   : locating "PdfMethod.XELATEX" in [executables] section of configuration file
DEBUG   : PdfMethod.XELATEX executable: xelatex, options:
DEBUG   : Successfully changed directory back to C:\Users\Sean Fitzpatrick\Documents\github\APEXCalculusPTX

FATAL   : 'charmap' codec can't decode byte 0x9d in position 63479: character maps to <undefined>

DEBUG   : Exception info:
------------------------
Traceback (most recent call last):
  File "C:\Python314\Lib\site-packages\pretext\cli.py", line 693, in build
    t.build(
    ~~~~~~~^
        clean=clean,
        ^^^^^^^^^^^^
    ...<4 lines>...
        clean_tmp_dirs=not save_tmp_dirs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Python314\Lib\site-packages\pretext\project\__init__.py", line 870, in build
    core.pdf(
    ~~~~~~~~^
        xml=self.source_abspath(),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
        latex_format="latex",
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Python314\Lib\site-packages\pretext\core\pretext.py", line 4548, in pdf
    result = _latex_compile(latex_cmd, logname, sourcename)
  File "C:\Python314\Lib\site-packages\pretext\core\pretext.py", line 4453, in _latex_compile
    and "Rerun to get" in open(log_file).read()
                          ~~~~~~~~~~~~~~~~~~~^^
  File "C:\Python314\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 63479: character maps to <undefined>

INFO    : ------------------------
FATAL   : Failed to build without errors.  Exiting...

[Sean Fitzpatrick]

I was hoping that the new validation tools would help me out, but:

Sean Fitzpatrick@cr8ndy3-mat23 MINGW64 ~
$ python ~/Documents/github/pretext/pretext/pretext -V -p ~/Documents/github/APEXCalculusPTX/publication/publication-color-print.ptx ~/Documents/github/APEXCalculusPTX/ptx/apex.ptx

Traceback (most recent call last):

  File "C:\Users\Sean Fitzpatrick\Documents\github\pretext\pretext\pretext", line 1033, in <module>
    main()
    ~~~~^^
  File "C:\Users\Sean Fitzpatrick\Documents\github\pretext\pretext\pretext", line 686, in main
    ptx.validate(xml_source, publication_file, stringparams, out_file, dest_dir, method)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Sean Fitzpatrick\Documents\github\pretext\pretext\lib\pretext.py", line 5313, in validate
    source_lines = f.readlines()

  File "C:\Python314\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 7315: character maps to <undefined>

I can't tell if the offending character is in my source, or somewhere in the script.

[Rob Beezer]

0x9d is a non-printing character.

https://www.fileformat.info/info/unicode/char/9d/index.htm

copy it (yes, it is invisible) and search for it in your source.

"cp1252" is a (Windows) "code page", which I have never totally grokked, and
goes back to DOS (I think). We might be missing a UTF-8 encoding.

My assistant just confirmed, and is on it.

Rob

> m/2t6UHUHQAAAJ <https://groups.google.com/g/pretext-support/c/
> xstrmXP_Tns/m/2t6UHUHQAAAJ>

>
> I think it might be the TM character. Maybe the one supplied by
> PreTeXt is causing trouble? I'll try removing any instances of it.
>
> But it seems like this error should not be fatal: PDF builds
> successfully, but the CLI refuses to copy it over.
>
> On Tue, Jul 28, 2026, 2:34 a.m. Valerio Monti
> <valeriom...@gmail.com> wrote:
>
>
> I reported an issue with charmap on Windows when building an
> EPUB from a source containing a <q> tag:
>
> https://groups.google.com/g/pretext-dev/c/vyA57GFN8IY/m/

> codAj23uBgAJ <https://groups.google.com/g/pretext-dev/c/
> vyA57GFN8IY/m/codAj23uBgAJ>

> > INFO    : successful application of C:\Users\Sean
> > Fitzpatrick\.ptx\2.45.0\core\xsl\utilities\report-
> publisher-variables.xsl

> > FATAL   : 'charmap' codec can't decode byte 0x9d in
> position 63479:
> > character maps to <undefined>
> > INFO    : ------------------------
> > FATAL   : Failed to build without errors.  Exiting...
> >
> > (Except it doesn't exit at this point! It builds the PDF,
> and then insists
> > that it didn't.)
> >
> > How do I go about tracking down the place in my source
> where the offending
> > byte is?
> >
> > --
> > You received this message because you are subscribed to
> the Google Groups
> > "PreTeXt support" group.
> > To unsubscribe from this group and stop receiving emails
> from it, send an email
> > to pretext-suppo...@googlegroups.com <mailto:pretext-
> > support+u...@googlegroups.com>.
> > To view this discussion visit https://groups.google.com/

> d/msgid/pretext-support/ <https://groups.google.com/d/msgid/
> pretext-support/>
> > f5fdcdd8-daa3-4d09-a7f9-0908d829d101n%40googlegroups.com
> <http://40googlegroups.com> <https://
> > groups.google.com/d/msgid/pretext-support/f5fdcdd8-
> daa3-4d09- <http://groups.google.com/d/msgid/pretext-
> support/f5fdcdd8-daa3-4d09->
> > a7f9-0908d829d101n%40googlegroups.com?
> utm_medium=email&utm_source=footer
> <http://40googlegroups.com?

> utm_medium=email&utm_source=footer>>.
>
> --
> You received this message because you are subscribed to the
> Google Groups "PreTeXt support" group.
>
> To unsubscribe from this group and stop receiving emails from
> it, send an email to pretext-suppo...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/
> pretext-support/e4c973d2-274c-405b-8355-

> f5239c264760n%40googlegroups.com <https://groups.google.com/d/
> msgid/pretext-support/e4c973d2-274c-405b-8355-
> f5239c264760n%40googlegroups.com?

> utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email

> to pretext-suppo...@googlegroups.com <mailto:pretext-
> support+u...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-

> support/6d9bc6ce-d6e9-4045-a757-308a3ba8f316n%40googlegroups.com <https://
> groups.google.com/d/msgid/pretext-support/6d9bc6ce-d6e9-4045-
> a757-308a3ba8f316n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Rob Beezer]

Not clear why we have not hit this one many times before? Anyway, there will be
a fix up in a minute. In the meantime I asked my assistant for the full story
in narrative form.

Rob

----

Sean, Valerio - thanks, this one is now diagnosed and fixed, and it is
entirely our bug, not anything about your source.

The short version: PreTeXt's Python was opening text files without saying
what encoding to use. When you do that, Python falls back on a platform
default, and that default is UTF-8 on Linux and macOS but cp1252 on
Windows. Every file PreTeXt writes is UTF-8, so the moment PreTeXt reads
one of its own files back on Windows, any byte outside the cp1252 repertoire
stops the run with "'charmap' codec can't decode byte 0x9d". Nothing is
wrong with the byte; the reader was simply told to expect the wrong
alphabet. So Sean's instinct to go hunting for a trademark character, and
Valerio's suspicion of the internationalized quotation marks, were both
chasing a symptom - any non-ASCII content anywhere in the document can
trigger it, and which character happens to be first is an accident of your
source.

That also explains the "fatal but not really" part, which is the most
useful clue in the whole thread. The PDF really is finished and correct by
the time this happens. The failure comes afterward, when PreTeXt reads the
LaTeX log file to decide whether LaTeX needs another pass to settle
cross-references. The read throws, the run is marked as failed, and so the
finished PDF never gets copied out of the temporary directory. Sean, when
you turned on debugging, kept the temp directory, and found a perfectly good
PDF sitting in it, you had already found the answer.

Your second report, from validation, is the same bug in a different place:
there PreTeXt reads back the assembled version of your source, which is full
of your own prose and therefore full of non-ASCII. I was able to reproduce
that one here almost exactly by forcing Python to use a non-UTF-8 default,
and got a decode error at position 7290 of the sample article against your
position 7315. Same site, different document.

Looking for other instances turned up thirty-eight in all, so this would
have kept resurfacing in new places as people did new things on Windows. A
couple of them were worse than a crash: the Asymptote path would have
decoded a diagram as cp1252 and then re-encoded it as UTF-8, quietly
mangling any non-ASCII in it rather than complaining. All of them now name
UTF-8 explicitly. Two reads additionally tolerate undecodable bytes instead
of failing, because they consume output from other programs whose encoding
we do not control: the LaTeX log, which is whatever the engine and its
packages chose to emit, and the output of jing, which is a Java program
whose stdout encoding varies by platform and Java version.

Verifying a Windows bug without Windows takes a little care, so for the
record: I reproduced the failure by forcing Python's default encoding to
ASCII, which is a stricter version of the same condition, confirmed the
build fails there exactly as you describe, and then confirmed that with the
fix HTML, LaTeX, EPUB, PDF, braille, and validation all complete under that
same setting and produce byte-identical output to a normal run.

Two practical notes. This is a fix in the core repository, which the CLI
carries along, so it will reach you with the next CLI release rather than
today. In the meantime you can get the correct behavior on Windows right
now by setting the environment variable PYTHONUTF8 to 1, which tells Python
to use UTF-8 regardless of the system locale. That is worth doing anyway if
you work with PreTeXt on Windows.

[Rob Beezer]

https://github.com/PreTeXtBook/pretext/pull/3083

[Sean Fitzpatrick]

Well, at least when I travel with the wrong operating system I manage to uncover some interesting bugs. 

I think I know why I never ran into this before: I've only just started to build directly to PDF. In the past I always built the .tex file and then compiled that separately.

On Tue, Jul 28, 2026, 11:55 a.m. 'Rob Beezer' via PreTeXt support <pretext...@googlegroups.com> wrote:

https://github.com/PreTeXtBook/pretext/pull/3083

--
You received this message because you are subscribed to the Google Groups "PreTeXt support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-suppo...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/pretext-support/MTAwMDAwT1FZT0p6djE.1785250503%40pnsh.