Back-of-book indexes

[Ian Fantom]

Working on an end-of-book index I realised that the sort order in the index was wrong – it followed English sorting rules rather than those of the declared language:

\usepackage{polyglossia}

\setmainlanguage{esperanto}

I traced the problem to makeindex, which accepts only Latin-1 8-bit encoding. So I found out about xindy and its Latex wrapper texindy, and tried:

- ---------------------

$ texindy -L esperanto filename.idx

Opening logfile "/dev/null" (done)

Reading indexstyle...

Loading module "/tmp/7rb6Yof7sM"...

Loading module "lang/esperanto/latin3-lang.xdy"...

Loading module "lang/esperanto/latin3.xdy"...

Finished loading module "lang/esperanto/latin3.xdy".

Finished loading module "lang/esperanto/latin3-lang.xdy".

Loading module "tex/inputenc/latin.xdy"...

Error in line 4:

;; Generated from latin?.xdy

Binary

ERROR: EVAL: variable BINARY has no value

- --------------------

I don't understand the error messages.

It looks as if texindy is expecting Latin-3 rather than utf-8. (Latin-3 is the 8-bit encoding for Esperanto and Turkish).

I found this example (https://tex.stackexchange.com/questions/25623/how-to-use-xindy-with-xetex-luatex):

xindy -M texindy -C utf8 -L german-duden filename.idx
So I tried it for Esperanto and got:

- ----------------------

$ xindy -M texindy -C utf8 -L esperanto *.idx

Opening logfile "/dev/null" (done)

Reading indexstyle...

Loading module "/tmp/FxGCZvtV39"...

Loading module "lang/esperanto/utf8-lang.xdy"...

Loading module "lang/esperanto/utf8.xdy"...

Finished loading module "lang/esperanto/utf8.xdy".

Finished loading module "lang/esperanto/utf8-lang.xdy".

Loading module "texindy.xdy"...

Loading module "numeric-sort.xdy"...

Finished loading module "numeric-sort.xdy".

Loading module "latex.xdy"...

Loading module "tex.xdy"...

Finished loading module "tex.xdy".

Finished loading module "latex.xdy".

Loading module "latex-loc-fmts.xdy"...

Finished loading module "latex-loc-fmts.xdy".

Loading module "makeindex.xdy"...

Finished loading module "makeindex.xdy".

Loading module "latin-lettergroups.xdy"...

Finished loading module "latin-lettergroups.xdy".

Finished loading module "texindy.xdy".

Finished loading module "/tmp/FxGCZvtV39".

Finished reading indexstyle.

Finalizing indexstyle... (done)

Reading raw-index "/tmp/x51eOsF6o4"...

WARNING: unknown cross-reference-class `hyperindexformat'! (ignored)

WARNING: unknown cross-reference-class `hyperindexformat'! (ignored)

ERROR: CHAR: index 0 should be less than the length of the string

- ---------------------

I don't understand the warnings and the error message. I checked with my compilation in TexStudio and nothing had changed in the index.

I checked my declarations and compared with those in stackexchange:

- --------------------

\ifPDFTeX

\usepackage[T1]{fontenc}

\usepackage[utf8]{inputenc}

% 2024-10-25 Added above line

\usepackage{textcomp} % provide euro and other symbols

\else % if luatex or xetex

\usepackage{unicode-math} % this also loads fontspec

\defaultfontfeatures{Scale=MatchLowercase}

\defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}

\fi

\usepackage{makeidx}\makeindex

- ----------------------

I'm not sure about '\ifPDFTeX' – I couldn't see any reference to pdftex in the headers.

I'm using Xelex in TexStudio under Ubuntu.

I've tried following http://www.xindy.org/documentation.html , where it states "Documentation is one of our big problems in this project. If you are willing to give us a hand here, you're more than welcome". Perhaps that is a clue to my problem! There are too many subtleties I don't understand at this stage.

Any help would be appreciated!

Many thanks.

Ian

[Peter Flynn]

On 07/07/2025 08:49, Ian Fantom wrote:
> It looks as if texindy is expecting Latin-3 rather than utf-8. (Latin-3
> is the 8-bit encoding for Esperanto and Turkish).

But what encoding have you actually used for your document?
I thought UTF-8 included everything that Esperanto used.

> |xindy -M texindy -C utf8 -L german-duden filename.idx|
> |So I tried it for Esperanto and got:|

I'm not sure what the german-duden would do, maybe hyphenation?

> Reading raw-index "/tmp/x51eOsF6o4"...
>
> WARNING: unknown cross-reference-class `hyperindexformat'! (ignored)
> WARNING: unknown cross-reference-class `hyperindexformat'! (ignored)

Ignorable.

> ERROR: CHAR: index 0 should be less than the length of the string

Bug. It should specify WHERE in the data stream the error occurred.
Is there an xindy.log file that gives more information?

> I checked my declarations and compared with those in stackexchange:
>
> - --------------------
>
> \ifPDFTeX
>
> \usepackage[T1]{fontenc}
>
> \usepackage[utf8]{inputenc}

Don't do this if you use UTF-8. pdflatex is not fully UTF-8 compliant.
With XeLaTeX you should be using \usepackage{fontspec}, but you should
switch to LuaLaTeX as XeLaTeX is no longer supported.

The key is making sure ALL files are UTF-8. It's very easy for an
unguarded command, event, or edit to accidentally switch a file to (eg)
ISO 8859-* or even Windows-1252 without realising it.

> I'm not sure about '\ifPDFTeX' – I couldn't see any reference to pdftex
> in the headers.

It's only there to fix things for users of pdflatex, AFAIK.

> I'm using Xelex in TexStudio under Ubuntu.
>

> Perhaps that is a clue to my problem! There are too many subtleties
> I don't understand at this stage.

I have been using makeindex still, for this very reason. I do still have
some issues (see
https://tex.stackexchange.com/questions/731952/makeindex-mangling-multibyte-characters)
but was recommended to use xindex rather than xindy.

> I've tried following http://www.xindy.org/documentation.html , where
> it states "Documentation is one of our big problems in this project.
> If you are willing to give us a hand here, you're more than welcome".

IMNSHO if their software needs that much documentation to use, maybe it
needs rewriting to be more obvious (says the man writing a 500pp manual :-)

Peter

[Ian Fantom]

Thanks. I hadn't heard of xindex - or if I had I was confusing it with
xindy.

In that case I'll set up for luatex. I didn't convert earlier because I
didn't want to get into another rabbit hole! The other comments were
useful, too, but the other issues may be solved with xindex.

Thanks,

Ian

[Peter Flynn]

On 07/07/2025 11:20, Ian Fantom wrote:
> Thanks. I hadn't heard of xindex - or if I had I was confusing it with
> xindy.

Let us know how it goes.

> In that case I'll set up for luatex. I didn't convert earlier
> because I didn't want to get into another rabbit hole! The other
> comments were useful, too, but the other issues may be solved with
> xindex.

I had reservations (having been a XeLaTeX user for a long time) but
apart from one initial hiccup with fonts, LuaLaTeX has been working
fine. It's fractionally slower than XeLaTeX, but compensates for that by
creating PDF direct, whereas XeLaTeX creates an intermediary file and
then converts that to PDF (similar how the original LaTeX created a DVI
file which needed converting to PS afterwards).

Peter

[Ian Fantom]

I've found a quick work-around, keeping things as they are. I know it's
a sin, but I'm trying to get a book out quickly now, and I'll look at
LuaLatex after the hols.

Instead of: \index{Ĝenevo}Ĝenevo

I can write: \index{Gxenevo@Ĝenevo}Ĝenevo

The first term defines where it is in the index listing, and the second
defines how it appears. I'm not sure whether that's by design or not!
The transliteration Ĝ -> Gx is standard in Esperanto, because there is
no 'x' in the Esperanto alphabet, and so 'Gx' comes after 'G', where it
should come. The only difference is that there's no spacing between the
G's and the Ĝ's, but that it hardly noticable.

:-)

Thanks,

Ian