Tikal defaulting to windows-1252 on Windows...

[jimbo]

If no encoding is specified Tikal uses "Charset.defaultCharset()" -
which on Windows returns windows-1252. This is causing an XML parsing
error for some of our docx documents:

An error occurred during extraction
Unexpected first character (char code 0xEF), not valid in xml document:
could be mangled UTF-8 BOM marker. Make sure that the Reader uses
correct encoding or pass an InputStream instead

Should we change the "Charset.defaultCharset()" code and simply force
utf-8 as the default encoding? I think that's what we do for Rainbow now.

Jim

[jimbo]

There must be an issue with how the Tikal extract pipeline is setup.
Even when we force utf-8 with "-ie utf-8 -oe utf-8" we still get the
error below.

[jimbo]

Forgot to mention that the same file runs fine on Linux.

[jimbo]

I found the problem - the OpenXml filter depends on the default charset
- which on Windows is still windows-1252. This fails on some documents
that have BOMS.

See ticket:
https://bitbucket.org/okapiframework/okapi/issues/1162/openxml-filter-fails-on-windows

Jim