RFC: Small mod to fl_utf8len() to detect invalid single char UTF-8 bytes

[Greg Ercolano]

Suggesting this one line change to fl_utf8len() to detect the two invalid
single character UTF-8 chars 0xFE and 0xFF:

@@ -107,6 +107,7 @@ int fl_utf8len(char c)
       if (c & 0x10) {
         if (c & 0x08) {
           if (c & 0x04) {
+            if (c & 0x02) return -1;
             return 6;
           }
           return 5;

A similar change can be made to fl_utf8len1() for symmetry.

The change seems a good way to implement, as it /only/ does the extra test if the utf-8 string is the rare length of 6, so it won't even affect the speed of shorter lengths.

I carefully made a test app comparing old code to the new for all 256 possible input values to the function, and the only difference the change makes is it now returns -1 for the 0xFE and 0xFF cases.

Since this isn't my code, I thought it best to run it up the flagpole to see if there's any negative effects you all who know utf8 might foresee.

Why this came up:

I'm working on Fl_Terminal to make sure it properly detects invalid UTF-8 strings, and since the single characters 0xFF and 0xFE are invalid anywhere in UTF-8 strings, they make a convenient test case. However, I noticed our fl_utf8len() function wasn't catching them.

[imacarthur]

On Wednesday, 12 August 2026 at 04:35:58 UTC+1 Greg wrote:

Suggesting this one line change to fl_utf8len() to detect the two invalid
single character UTF-8 chars 0xFE and 0xFF:

@@ -107,6 +107,7 @@ int fl_utf8len(char c)
       if (c & 0x10) {
         if (c & 0x08) {
           if (c & 0x04) {
+            if (c & 0x02) return -1;
             return 6;
           }
           return 5;

Looks plausible to me, FWIW.

Though my OCD would probably code it as  

                           if (c & 0x02) { return -1; }

Because "goto FAIL;" etc...

[Albrecht Schlosser]

On 8/12/26 05:35 Greg Ercolano wrote:

Suggesting this one line change to fl_utf8len() to detect the two invalid
single character UTF-8 chars 0xFE and 0xFF:

@@ -107,6 +107,7 @@ int fl_utf8len(char c)
       if (c & 0x10) {
         if (c & 0x08) {
           if (c & 0x04) {
+            if (c & 0x02) return -1;
             return 6;
           }
           return 5;

A similar change can be made to fl_utf8len1() for symmetry.

The change seems a good way to implement, as it /only/ does the extra test if the utf-8 string is the rare length of 6, so it won't even affect the speed of shorter lengths.

The problem with the current implementation is (and has "always" been) that proper UTF-8 encoding covers only 4 bytes (21 bits of Unicode code points, which is, AFAICT, the limit of current and future Unicode), and therefore returning 5 or 6 is faulty anyway.

Instead of adding this line we should return -1 instead of 5 or 6 anyway, to be Unicode compliant.

Wikipedia says (I copied the full reference, which may only be readable if you read the HTML encoding of this mail):
https://en.wikipedia.org/wiki/UTF-8#Description :

UTF-8 encodes code points in one to four bytes, depending on the value of the code point. In the following table, the letters u through z represent the hexadecimal digits of a character’s Unicode number (U+uvwxyz, or U+wxyz for a four-digit number). In the binary column, each hex digit is expanded into its 4-bit binary equivalent (uuuu to zzzz) to show how its bits are distributed across the UTF-8 bytes.

Code point ↔ UTF-8 conversion

First code point	Last code point	Byte 1	Byte 2	Byte 3	Byte 4
U+0000	U+007F	0yyyzzzz
U+0080	U+07FF	110xxxyy	10yyzzzz
U+0800	U+FFFF	1110wwww	10xxxxyy	10yyzzzz
U+010000	U+10FFFF	11110uvv	10vvwwww	10xxxxyy	10yyzzzz

(end of citation)

The tables (and the code) in our lib obviously predate the reduction of the Unicode range from full 32 bits (or 31 bits?) to only 21 bits which is the standard.

I carefully made a test app comparing old code to the new for all 256 possible input values to the function, and the only difference the change makes is it now returns -1 for the 0xFE and 0xFF cases.

Since this isn't my code, I thought it best to run it up the flagpole to see if there's any negative effects you all who know utf8 might foresee.

Why this came up:

I'm working on Fl_Terminal to make sure it properly detects invalid UTF-8 strings, and since the single characters 0xFF and 0xFE are invalid anywhere in UTF-8 strings, they make a convenient test case. However, I noticed our fl_utf8len() function wasn't catching them.

I'm not sure if it is now a good time (before the release of 1.5) to change such a function in the way I would suggest for correctness (handling length 5 and 6 as error and returning -1 instead), and I didn't check FLTK's code for any side effects, but I think it would be OK. It would definitely not violate the documentation because the documentation doesn't specify what valid and invalid encodings are.

I'm pretty sure, however, that our UTF-8 to Unicode Code Points (and vice versa) also treat such invalid Unicode values in the wrong way.

Regarding the tests in Fl_Terminal: you could test for `x < 0 || x > 4` (or equivalent) to catch all really invalid UTF-8 encodings - which would catch the mentioned byte values but doesn't mean that this would catch all invalid Unicode code points, BTW.

[Greg Ercolano]

On 8/12/26 03:40, 'Albrecht Schlosser' via fltk.coredev wrote:

The problem with the current implementation is (and has "always" been) that proper UTF-8 encoding covers only 4 bytes (21 bits of Unicode code points, which is, AFAICT, the limit of current and future Unicode), and therefore returning 5 or 6 is faulty anyway.

    Ya, at least for UTF-8 encoding of Unicode.
    I caught that 4 byte Unicode limit too, which is why I use a 4 byte buffer in Fl_Terminal for saving "chars".

Instead of adding this line we should return -1 instead of 5 or 6 anyway, to be Unicode compliant.

    I guess it depends on whether our fl_utf8 functions should allow for full utf-8 encoding, perhaps to support utf-8 encoding of other types of data that might allow 6 bytes, or if we want to limit it to Unicode purposes only.

I'm not sure if it is now a good time (before the release of 1.5) to change such a function in the way I would suggest for correctness (handling length 5 and 6 as error and returning -1 instead), and I didn't check FLTK's code for any side effects, but I think it would be OK. It would definitely not violate the documentation because the documentation doesn't specify what valid and invalid encodings are.

    I think improving invalidation detection is always good, and it would help my terminal tests for sure, as I need a reliable way to test for invalid utf-8 chars.

    This has to do with me working on an issue you (Albrecht) opened against Fl_Terminal, issue #950. I've only now been able to get around to it, because I needed to find related changes I'd made years ago but not committed, and lost track of them because it was floating around on the local filesystem of a workstation I had to decomission, and I only recently tracked down and restored.

    I have the mods reapplied, and expect a commit today after some final testing which should close that old issue.

Regarding the tests in Fl_Terminal: you could test for `x < 0 || x > 4` (or equivalent) to catch all really invalid UTF-8 encodings - which would catch the mentioned byte values but doesn't mean that this would catch all invalid Unicode code points, BTW.

    Yes, in fact there's a few other bytes that are not allowed; bytes above 0xF4 are apparently invalid,
    and there are two others, I think 0xC0 or some such.

[Greg Ercolano]

On 8/12/26 03:40, 'Albrecht Schlosser' via fltk.coredev wrote:

I'm not sure if it is now a good time (before the release of 1.5) to change such a function in the way I would suggest for correctness (handling length 5 and 6 as error and returning -1 instead), and I didn't check FLTK's code for any side effects, but I think it would be OK.

    OK, yeah, I didn't want to change much, so I think the one line change is the right one "for now".
    Later we can figure out how to make it more correct.

    I should mention though; FWIW, codex initially suggested rewriting the code to be as follows, as per RFC 3629:

  int fl_utf8len(char c) {
    const unsigned char u = (unsigned char)c;
    if (u <= 0x7f) return 1;
    if (u >= 0xc2 && u <= 0xdf) return 2;
    if (u >= 0xe0 && u <= 0xef) return 3;
    if (u >= 0xf0 && u <= 0xf4) return 4;
    return -1;
  }

  int fl_utf8len1(char c) {
    const int len = fl_utf8len(c);
    return len < 0 ? 1 : len;
  }

    ..but I didn't want to make such a large change, and asked for a simpler modification retaining the spirit and formatting of the old code, so it suggested that one line mod for the return 6 case instead.

    However, using the above code, it seems to properly detect all possible "invalid bytes", not only for the first byte in the string (to determine length), but actually anywhere in the utf8 string.

    I tested the above code against our old code, and its results are exactly the same, except for detecting these as invalid bytes:

        0xc0, 0xc1
        0xf5 thru 0xff

    And RFC 3629 confirms explicitly that those particular values are in fact invalid:

    ..that last bit being relevant, and seems to be exactly what the new code (above) does.

    So I'm kinda liking that as the way to go.

    So I can either add that in a future commit (+1), or my one line change (also +1), vs not making any change (-1).

[Greg Ercolano]

On 8/12/26 09:30, Greg Ercolano wrote:

    I should mention though; FWIW, codex initially suggested rewriting the code to be as follows, as per RFC 3629:

  int fl_utf8len(char c) {
    const unsigned char u = (unsigned char)c;
    if (u <= 0x7f) return 1;
    if (u >= 0xc2 && u <= 0xdf) return 2;
    if (u >= 0xe0 && u <= 0xef) return 3;
    if (u >= 0xf0 && u <= 0xf4) return 4;
    return -1;
  }

  int fl_utf8len1(char c) {
    const int len = fl_utf8len(c);
    return len < 0 ? 1 : len;
  }

[..]

    Here's how it helps the test/terminal program which wants to demonstrate handling of invalid ANSI/XTERM/UTF-8 sequences, showing the difference between the old fl_utf8len(), which doesn't detect any of the 0xC0/1 and >0xF4 cases, vs new code suggested above by codex that detects all the single-byte issues. We probably should consider this change, as it probably prevents some accidental overruns from malformed utf-8:

[Albrecht Schlosser]

On 8/12/26 19:02 Greg Ercolano wrote:

On 8/12/26 09:30, Greg Ercolano wrote:

    I should mention though; FWIW, codex initially suggested rewriting the code to be as follows, as per RFC 3629:

  int fl_utf8len(char c) {
    const unsigned char u = (unsigned char)c;
    if (u <= 0x7f) return 1;
    if (u >= 0xc2 && u <= 0xdf) return 2;
    if (u >= 0xe0 && u <= 0xef) return 3;
    if (u >= 0xf0 && u <= 0xf4) return 4;
    return -1;
  }

  int fl_utf8len1(char c) {
    const int len = fl_utf8len(c);
    return len < 0 ? 1 : len;
  }

[..]

+1 on this version of the code, for these reasons:

- clarity and maintainability (fl_utf8len)
- correctness for UTF-8, according to RFC 3629 (fl_utf8len)
- no duplication of code in fl_utf8len1

BTW, you wrote in an earlier message:

> I guess it depends on whether our fl_utf8 functions should allow for full utf-8 encoding, perhaps to support utf-8 encoding of other types of data that might allow 6 bytes, or if we want to limit it to Unicode purposes only.

FLTK's purpose is to display Unicode text, encoded as UTF-8 strings (besides any other GUI specific stuff, of course). The functions in question deal with Unicode text, not any binary data that is - intentionally or not - encoded in UTF-8. If this is not clearly documented, then we should do this.

In old documentation we had these tables with up to 6 bytes for encoding, but this was not RFC 3629 compliant. Historically Unicode was designed for 31-bit Code Points, but that was revised long ago, and now Unicode supports only 21 bits (U+0000 .. U+10FFFF), and that's what we should document and use. I also read in RFC 3629 that doing the conversion wrong can result in security issues.

Note: meanwhile I'm convinced that we should make this kind of changes NOW, i.e. before we release FLTK 1.5.0 !

[Greg Ercolano]

On 8/12/26 14:14, 'Albrecht Schlosser' via fltk.coredev wrote:

On 8/12/26 09:30, Greg Ercolano wrote:

    I should mention though; FWIW, codex initially suggested rewriting the code to be as follows, as per RFC 3629:

  int fl_utf8len(char c) {
    const unsigned char u = (unsigned char)c;
    if (u <= 0x7f) return 1;
    if (u >= 0xc2 && u <= 0xdf) return 2;
    if (u >= 0xe0 && u <= 0xef) return 3;
    if (u >= 0xf0 && u <= 0xf4) return 4;
    return -1;
  }

  int fl_utf8len1(char c) {
    const int len = fl_utf8len(c);
    return len < 0 ? 1 : len;
  }
[..]

+1 on this version of the code, for these reasons:

- clarity and maintainability (fl_utf8len)
- correctness for UTF-8, according to RFC 3629 (fl_utf8len)
- no duplication of code in fl_utf8len1

[..]

Note: meanwhile I'm convinced that we should make this kind of changes NOW, i.e. before we release FLTK 1.5.0 !

    OK, great, I'll commit it today, as it would seriously help the terminal tests.
    We can always revert if other devs makes a compelling argument to the contrary.

[Greg Ercolano]

    I should mention though; FWIW, codex initially suggested rewriting the code to be as follows, as per RFC 3629:

  int fl_utf8len(char c) {
    const unsigned char u = (unsigned char)c;
    if (u <= 0x7f) return 1;
    if (u >= 0xc2 && u <= 0xdf) return 2;
    if (u >= 0xe0 && u <= 0xef) return 3;
    if (u >= 0xf0 && u <= 0xf4) return 4;
    return -1;
  }

  int fl_utf8len1(char c) {
    const int len = fl_utf8len(c);
    return len < 0 ? 1 : len;
  }
[..]

[..]

    OK, great, I'll commit it today, as it would seriously help the terminal tests.
    We can always revert if other devs makes a compelling argument to the contrary.

    I've opened up issue # 1576 for this, and will associate the commit with that issue, so there can be a chance for comments and digressions.

    Due to the increased chances of invalid return values of -1, I asked codex to check all of FLTK code to look for possible mishandling of the -1 return from fl_utf8len(), and it did find at least one case in Fl_Input_::expandpos(), so solving that and whatever else can be connected to that issue.

[melcher....@googlemail.com]

+1 from me FWIW

[Greg Ercolano]

Albrecht-S schrieb am Mittwoch, 12. August 2026 um 23:14:21 UTC+2:

+1 on this version of the code [..]

On 8/13/26 09:16, 'melcher....@googlemail.com' via fltk.coredev wrote:

+1 from me FWIW

    Great, thanks Matt, that's three +1's.
    Committed as of yesterday in 8168811.