utf8 support in fltk-1.4.x

[Flap Circona]

Hi all,

I'm not sure if this is the correct group to ask such a question, please drop me a note if it isn't.

I'm using different FLTK versions for my small program on three different machines. My desktop is an older ubuntu "focal", my laptop ubuntu "jammy" and a RaspberryPi with a current distribution (downloaded a few days ago from their website).

All FLTK installations are from the git repo and manually built into /usr/local. The desktop uses FLTK 1.4.x, the laptop and the RaspberryPi FLTK 1.3.x. With this constellation my small program behaves the same on all three platforms.

When I change the laptop from FLTK 1.3.x to 1.4.x, all labels, titles and so on of my small program stop displaying German "umlauts". Only a single question-mark ('?') is shown instead. If I change the laptop back to 1.3.x everything is as expected again.

There seems an external dependency which I don't know yet, which seems missing on my laptop's ubuntu "jammy". Any ideas?

Cheers, Juergen

[Flap Circona]

One more observation:

With FLTK 1.4.x the German "umlauts" disappear if I run my prgram with "FLTK_BACKEND=wayland" and they are back again with "FLTK_BACKEND=x11".

[imm]

My _guess_ is that it may be a font selection issue - do you know what font face is being selected on each host machine?

Further, note that the actual text rendering scheme may differ: fltk-1.4 will use either pango or XFT (or as a fallback X11) to render text (where pango is generally viewed as "better" in that it's capable of substituting for missing glyphs in a face and generally superior at handling mixed languages and so forth.)

But for any of these to work, the relevant libs need to be available at runtime, and the relevant dev packages available at compile time...

Fltk-1.3 doesn't have pango support, only XFT and X11, but again the relevant libs / dev packages need to be available.

And ultimately any of these can only render character combinations that the selected font face has the glyphs for...

--
Ian
From my Fairphone FP3
   

--
You received this message because you are subscribed to the Google Groups "fltk.general" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fltkgeneral...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fltkgeneral/19a73b57-566c-488e-92d7-f5da77c19f88n%40googlegroups.com.

[Flap Circona]

I'm using the KDE-Neon distribution on the laptop. In its KDE settings "Noto sans" is the used font. I don't know, how FLTK selects a font. Does it use system wide settings?

I took a look into both run-time maps of the running program (with wayland or x11 setting ), and it seems more or less the same. libpango is mapped, but no libxft in both case.

I followed the README for the required development libraries when building the 1.4.x FLTK libs.

[....]

Configuration Summary
-------------------------------------------------------------------------
    Directories: prefix=/usr/local
                 bindir=${exec_prefix}/bin
                 datadir=${datarootdir}
                 datarootdir=${prefix}/share
                 exec_prefix=${prefix}
                 includedir=${prefix}/include
                 libdir=${exec_prefix}/lib
                 mandir=${datarootdir}/man
    Build fluid: YES
    Build tests: YES
Cross-compiling: NO
       Graphics: Wayland or X11 with cairo + Xfixes + Xinerama + Xcursor + Xrender + Pango
Image Libraries: JPEG=System
                 PNG=System
                 ZLIB=System
Forms library:   YES
    Large Files: YES
         OpenGL: YES
        Threads: YES
   Allow std:: : NO
configure: creating ./config.status
config.status: creating makeinclude
config.status: creating fltk.list
config.status: creating fltk-config
config.status: creating fltk.spec
config.status: creating FL/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: creating FL/fl_config.h
config.status: FL/fl_config.h is unchanged

[...]

When I build the 1.3.x version on the same system, it states at the end:

[...]

       Graphics: X11 + Xft + Xdbe + Xfixes + Xinerama + Xcursor + Xrender

[...]

[Bill Spitzak]

Older versions of flak converted UTF-8 to printable characters itself, and invalid UTF-8 bytes were assumed to be in CP1252 instead, which in effect made FLTK draw both UTF-8 and CP1252, even when mixed together in the same string.

I think Pango/etc being used for rendering are much more strict about the strings being valid UTF-8 and print all the errors as question marks.

It is possible to convert the existing strings to UTF-8. FLTK’s own converters are unchanged so you may be able to use them to convert the UTF-8 to UTF-16 and then convert back, which will change all the CP1252 to the correct UTF-8.

To view this discussion on the web visit https://groups.google.com/d/msgid/fltkgeneral/652a2a12-0e63-487b-8328-edd15907c54fn%40googlegroups.com.

[Flap Circona]

Hi all,

more time, more tests. Thanks for the codepage hint: It was a gettext issue. I'm using gettext() for all strings in my program. And gettext() still returns UTF-8 (as used in the po sources) when my program is linked againt FLTK-1.3.x.

And it returns the question mark for the umlauts if I link my program against FLTK-1.4.x. My LANG variable is 'de_DE.UTF-8'. Querying the settings of gettext do not show differences between 1.3.x and 1.4.x. So I'm out of ideas, what

is going different between both FLTK versions.

Now I'm using an additional call to bind_textdomain_codeset() in my early initialisation code to define the UTF-8 codepage and all umlauts are printed again with FLTK-1.4. I'm not sure if this is only a workaround or a requirement, since the gettext documentation suggests it uses the part "[.codeset]" from the LANG environment variable by default.

Thanks again for the codepage hint.

Jürgen

[Manolo]

The problem is that the Wayland and the X11/Cairo FLTK backends used in FLTK 1.4

require proper UTF-8 strings to draw text. FLTK 1.3 uses a procedure that accepts UTF-8

strings but that also converts text encoded in Windows Latin-1 to correct UTF-16

before drawing it.
Your source code most probably uses Windows Latin-1 strings rather than UTF-8

strings. That's what explains the diverse effects you report.

UTF encoding is THE way to get rid of the nightmare of distinct code pages

once and for all. The way out for you, in my view, is to convert all strings

that your source code uses, either directly or via po/gettext, to proper UTF-8.

We should discuss between developers if supporting Windows Latin-1 strings

in the Wayland and X11/Cairo FLTK backends of FLTK 1.4 is desirable.
An incentive to abandon codepages and adopt UTF-8 makes sense to me.

[Flap Circona]

Hi,

my source code uses UTF-8 (fixed setting in my editor). I double checked this by printing the hex numbers of the strings I use. The "ä" is encoded in the string with 0xc3 0xa4 which is correct according to https://www.utf8-zeichentabelle.de/. If I hexdump the MO file used by gettext in /usr/share/locale/de/LC_MESSAGES/my-app.mo it contains the german translations as I wrote them in the corresponding po files in my project in UTF-8. So I guess, my setup is correct.

This works in both 1.3.x and 1.4.x:

   widget = new Fl_Button(some_x, some_y, some_w, some_h, "@fileopen Dokument auswählen…"));

(so I'm sure the string reaches the contructor of Fl-Button in UTF-8)

This works in 1.3.x, but not in 1.4.x without setting the codeset:

   widget = new Fl_Button(some_x, some_y, some_w, some_h, gettext("@fileopen Select Document…"));

Something in FLTK 1.4.x (just a guess) instructs gettext() to adapt the codepage to something different than UTF-8.

Does somthing run in FLTK-1.4.x prior calling my apllication's main()? Setting up the locale is the first thing I do in my main(). And after that I create the main window.

[Bill Spitzak]

I think what was happening is gettext() was returning Latin-1 in both versions, it’s just that fltk-1.3 was able to recognize and print this (as well as correct UTF-8 which you were using elsewhere). Fltk 1.4 can only print correct UTF-8.

That call you found to make gettext return UTF-8 is the correct fix. Though they really should completly scrap locales and make the entire get text system UTF-8 only (maybe with work to read databases in other code sets).

-- 
You received this message because you are subscribed to the Google Groups "fltk.general" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fltkgeneral...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fltkgeneral/a59243c0-0c5a-4290-b9b7-838e08061060n%40googlegroups.com.

[Flap Circona]

I dug deeper.... And yes, you are right. gettext() defaults to plain ASCII. The missing piece was "LC_CTYPE":

"11.2.4 How to specify the output character set gettext uses

[...]

The output character set is, by default, the value of nl_langinfo (CODESET), which depends on the LC_CTYPE part of the current locale. But programs which store strings in a locale independent way (e.g. UTF-8) can request that gettext and related functions return the translations in that encoding, by use of the bind_textdomain_codeset function.

[...]"

I only set LC_MESSAGES in my program and in FLTK-1.3.x LC_TYPE is set as well (it is enough to call "Fl::args(argc, argv);" and nothing else), in FLTK-1.4.x it seems not set.

So in FLTK-1.3. it only works by accident, because my LANG contains the "[.codeset]" part with "UTF-8".

The correct solution is calling bind_textdomain_codeset() and thus, forcing the UTF-8 encoding, FLTK wants.

Many thanks to all of you for your comments which points me in the correct direction.

Jürgen