RFC/vote: A fl_utf8 function for detecting continuation bytes

[Greg Ercolano]

Would like to suggest a new fl_utf8_xxx() function that detects continuation bytes, e.g.

inline bool fl_utf8_is_aligned(char c) {
  //           Byte 1    Byte 2    Byte 3   ..etc..
  //   ASCII:  0xxxxxxx
  // UTF8(2):  110xxxxx  10xxxxxx
  // UTF8(3):  1110xxxx  10xxxxxx  10xxxxxx
  // UTF8(4):  11110xxx  10xxxxxx  10xxxxxx  10xxxxxx
  // UTF8(5):  111110xx  10xxxxxx  10xxxxxx  10xxxxxx  10xxxxxx
  // UTF8(6):  1111110x  10xxxxxx  10xxxxxx  10xxxxxx  10xxxxxx  10xxxxxx
  //           \______/  \______________________________________________/
  //          Start byte           Continuation bytes
  //                               (c & 0xc0) == 0x80
  return ((c & 0xc0) == 0x80);
}

The above code pattern of testing (byte & 0xc0) against 0x80 is repeated throughout fltk. It seems like this should be an actual utf8 function, to avoid having utf8 magic numbers scattered everywhere:

$ git grep '0xc0.*=.*0x80'
FL/Fl_Terminal.H:      //                               (c & 0xc0) == 0x80
FL/Fl_Terminal.H:      return ((c & 0xc0) == 0x80);
src/Fl_File_Chooser2.cxx:        if (ptr[1] && (ptr[1]&0xc0)!=0x80) break;
src/Fl_File_Chooser2.cxx:        if (ptr[1] && (ptr[1]&0xc0)!=0x80) break;
src/Fl_File_Chooser2.cxx:        if (ptr[1] && (ptr[1]&0xc0)!=0x80) break;
src/Fl_File_Chooser2.cxx:        if (ptr[1] && (ptr[1]&0xc0)!=0x80) break;
src/Fl_File_Chooser2.cxx:        if (ptr[1] && (ptr[1]&0xc0)!=0x80) break;
src/Fl_File_Chooser2.cxx:        if (ptr[1] && (ptr[1]&0xc0)!=0x80) break;
src/Fl_Text_Buffer.cxx:  while ( (c&0xc0) == 0x80) {
src/fl_utf8.cxx:  if ( (end && p+1 >= end) || (p[1]&0xc0) != 0x80) goto FAIL;
src/fl_utf8.cxx:    if ( (end && p+2 >= end) || (p[2]&0xc0) != 0x80) goto FAIL;
src/fl_utf8.cxx:    if ( (end && p+3 >= end) || (p[2]&0xc0) != 0x80 || (p[3]&0xc0) != 0x80) goto FAIL;
src/fl_utf8.cxx:  if ((*p&0xc0) != 0x80) return p;
src/fl_utf8.cxx:  if ((*p&0xc0) != 0x80) return p;

Seeking comments/votes:

    > Does it make sense to add this as an fl_utf8() inline function, and refit existing code to use it (+1/-1/0)
    > Suggesting 'is_aligned' or 'is_align' or 'is_continuation' as the fl_utf8_xxx suffix, but open to other suggestions

[Manolo]

Le jeudi 16 juillet 2026 à 19:21:06 UTC+2, er...@seriss.com a écrit :

Seeking comments/votes:

    > Does it make sense to add this as an fl_utf8() inline function, and refit existing code to use it (+1/-1/0)

Yes it makes sense. +1

 

    > Suggesting 'is_aligned' or 'is_align' or 'is_continuation' as the fl_utf8_xxx suffix, but open to other suggestions

I would support  fl_utf8_is_continuation_byte().

[Bill Spitzak]

I have no idea why you think "align" is a descriptive word for this.

I would use either is_continuation() or !is_leading()

Fl_File_Chooser2 should be checked, it seems to be acting like NUL is a continuation byte, and it certainly is not!

--
You received this message because you are subscribed to the Google Groups "fltk.coredev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fltkcoredev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/fltkcoredev/40a63ee1-4842-41be-b7ec-c2bba74784e4n%40googlegroups.com.

[Greg Ercolano]

On 7/17/26 09:49, Bill Spitzak wrote:

I have no idea why you think "align" is a descriptive word for this.

    It's a reference to a precedent set in Fl_Text_Buffer.cxx called utf8_align() which uses the same code to walk the string to align to the next utf8 character boundary:

/*
 Align an index to the current UTF-8 boundary.
 */
int Fl_Text_Buffer::utf8_align(int pos) const
{
  char c = byte_at(pos);

  while ( (c&0xc0) == 0x80) {

    pos--;
    c = byte_at(pos);
  }
  return pos;
}

    But yeah, "align" is not as good as "continuation" (the actual utf8 name for detecting the bit pattern).
    The only nice thing about "align" is it's short, as short names are something I've always liked about forms/xforms/fltk function/method names.

I would use either is_continuation() or !is_leading()

    Ya, is_continuation() is what I went with in Fl_Terminal, so a move to the fl_utf8 suite would mean fl_utf8_is_continuation(), whose name is longer than the actual in-line code `(c&0xc0)==0x80`, so I'm always trying to find something shorter.

[Albrecht Schlosser]

On 7/17/26 10:46 Manolo wrote:

Le jeudi 16 juillet 2026 à 19:21:06 UTC+2, er...@seriss.com a écrit :

Seeking comments/votes:

    > Does it make sense to add this as an fl_utf8() inline function, and refit existing code to use it (+1/-1/0)

Yes it makes sense. +1

+1 from me too.

    > Suggesting 'is_aligned' or 'is_align' or 'is_continuation' as the fl_utf8_xxx suffix, but open to other suggestions

I would support  fl_utf8_is_continuation_byte().

I vote for fl_utf8_is_continuation() w/o the redundant "_byte" suffix.

I'd also agree with fl_utf8_is_cont() for brevity, or even shorter w/o "_is": fl_utf8_cont(). If anybody doesn't know what it means, ... RTFM ;-)

[Albrecht Schlosser]

On 7/17/26 20:19 Greg Ercolano wrote:
>     Ya,is_continuation() is what I went with in Fl_Terminal, so a move

> to the fl_utf8 suite would mean fl_utf8_is_continuation(), whose name
> is longer than the actual in-line code `(c&0xc0)==0x80`, so I'm always
> trying to find something shorter.

See my suggested abbreviations fl_utf8_is_cont() and/or fl_utf8_cont(),
or add fl_utf8_is_cont() to the list.

It doesn't matter if the function call is longer than the actual code:
it makes clear what it's about, rather than the cryptic
`(c&0xc0)==0x80`. I definitely prefer a longer (inline) function call
than such cryptic code. Calling a function is also safer, because
there's no chance to make an undiscovered typo.

[Melcher Matthias]

I wrote utf8_align 20 years ago, knowing very little about Unicode. The method is used to ensure that the text buffer cursor is at the start of a utf8 sequence and was introduced when text buffer was extended to support utf-8.

For me personally, I don't really care what this kind of function should be called. This was pure utility back then.

The standard uses "self-synchronizing", "lead byte", and "continuation byte", so our functions should reflect that terminology.

As for Fl_Text_Buffer::utf8_align(), code should instead use the new correctly named functions, and this method could be marked deprecated.

+1 for completing utf-8 support and using the right terminology.

> --
> You received this message because you are subscribed to the Google Groups "fltk.coredev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fltkcoredev...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/fltkcoredev/bfdf5f47-ad97-4641-9efb-5e4e5bb307f7%40seriss.com.

[Albrecht Schlosser]

On 7/20/26 12:01 'Melcher Matthias' via fltk.coredev wrote:

I wrote utf8_align 20 years ago, knowing very little about Unicode. The method is used to ensure that the text buffer cursor is at the start of a utf8 sequence and was introduced when text buffer was extended to support utf-8.

OK, this explains the naming: the text position (index) should be aligned to a character boundary. This method is not documented, but according to the code it does something similar to fl_utf8back() inside the text buffer.

Note that we have fl_utf8back() and fl_utf8fwd() but these functions operate on plain character arrays (strings). 

The standard uses "self-synchronizing", "lead byte", and "continuation byte", so our functions should reflect that terminology.

As for Fl_Text_Buffer::utf8_align(), code should instead use the new correctly named functions, and this method could be marked deprecated.

I agree that it should use the correct function (internally). But it's a very special case because it operates on Fl_Text_Buffer. No matter how we call it, it should be documented with doxygen comments, since it's a public method. I'm not sure whether we want (need) to create another method and deprecate the old one. Proper documentation should IMHO suffice.

+1 for completing utf-8 support and using the right terminology.

Summary: we have three types of bytes in a valid UTF-8 sequence: (1) ASCII, (2) UTF-8 lead bytes, and (3) UTF-8 continuation bytes. If a byte is neither a lead nor a continuation byte, then it's pure (7-bit) ASCII.

IMHO it would be enough to detect continuation bytes (3). If a byte is not a continuation byte, it is a valid start of a UTF-8 character, either ASCII (1) or a lead byte (2) -- or are there invalid bytes that can't appear in a UTF-8 sequence?

My personal conclusion is that we should add fl_utf8_[is_]cont[inuation]() to detect continuation bytes as Greg suggested, and that's enough. And of course document Fl_Text_Buffer::utf8_align() properly.

The question for a good name of this new function remains. I propose:

(1) fl_utf8_is_continuation()
(2) fl_utf8_is_cont()
(3) fl_utf8_continuation()
(4) fl_utf8_cont()
(5) none of these, I propose: ...

If you agree, please vote for one of these (1-5).

My vote is (2).

Please feel free to suggest other useful functions like fl_utf8_lead() if you think these are useful (I don't).

[Greg Ercolano]

The question for a good name of this new function remains. I propose:

(1) fl_utf8_is_continuation()
(2) fl_utf8_is_cont()
(3) fl_utf8_continuation()
(4) fl_utf8_cont()
(5) none of these, I propose: ...

    +1 for (2) only because:

• it's short
• I like "is" as it implies a test and a boolean return
• "continuation" is correct terminology based on the this wikipedia utf8 page

    I did some further research in the "continuation" terminology, and was surprised to find: while the wikipedia page uses it, the RFCs for the UTF-8 spec and Unicode specs does not.

    I went down that rabbit hole, and I'll follow up separately in this thread with all that noise, but in the end I think "continuation" is fine.

[Greg Ercolano]

On 7/20/26 09:13, Greg Ercolano wrote:

[..]

I did some further research in the "continuation" terminology, and was surprised to find: while the wikipedia page uses it, the RFCs for the UTF-8 spec and Unicode specs does not. I went down that rabbit hole, and I'll follow up separately in this thread with all that noise, but in the end I think "continuation" is fine.

    So here's my adventure down the rabbit hole regarding "continuation" terminology.
    As I said, after my return from this excursion, I'm left thinking "continuation" is still OK; see green text below.

* * *

   Regarding "continuation" terminology, I put on my pedantic hat to dig down into the actual RFC for UTF-8 to verify the terminology the spec uses, since wikipedia is not really authoritative.

    So I looked up the RFC that defines UTF-8, specifically RFC 3629, there is no mention of this "continuation" terminology.
    They seem instead to refer to the sequence this way, e.g. Section 3:

• First byte is the "initial octet"
• The continuation bytes are called "subsequent octets" or "following octets"

    Apparently the UTF-8 spec was written with the Unicode standard in mind, and that refers to these bytes as:

• "leading byte"
• "trailing bytes"

    So I asked AI (google AI/Gemini) a few questions:

I'm trying to determine if there's specific terminology for the bytes in the middle of a UTF-8 sequence. The wikipedia page calls them "continuation" bytes, but the RFCs do not mention this.

    ..and it replied:

---

You are completely right to notice the vocabulary difference. [Gemini, lol, you are such a suck-up! -erco] While Wikipedia relies on informal industry jargon like "continuation bytes", formal specifications use much stricter structural terminology. [1]

The official terminology for those middle or subsequent bytes depends on which standard or section you look at:

1. The Official ABNF Term: UTF8-tail

In Section 4 of RFC 3629, the Internet Engineering Task Force (IETF) defines the formal syntax of UTF-8 using Augmented Backus-Naur Form (ABNF). [2]

• The specification explicitly brands these bytes as UTF8-tail.
• The ABNF rule is normatively defined as: UTF8-tail = %x80-BF. [2, 3]

2. The Textual Description Term: "Subsequent Octets"

In the narrative text of RFC 3629 (Section 3), the author does not use the word "byte" (preferring the networking standard term "octet"). [4, 5]

• The first byte is explicitly called the "initial octet".
• Any bytes following it are formally referred to as "subsequent octet(s)" or "following octet(s)". [4]

3. The Unicode Standard Term: "Trailing Bytes"

Because RFC 3629 is explicitly written to align with the Unicode Standard, it is also helpful to check their terminology. The Unicode Consortium formally refers to the initial byte as the leading byte and all subsequent bytes in the sequence as trailing bytes. [1, 4, 6, 7]

Summary Comparison

Source	First Byte Terminology	Following/Middle Bytes Terminology
RFC 3629 (ABNF Syntax)	UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4	UTF8-tail
RFC 3629 (Text Narrative)	Initial octet	Subsequent octet(s) / Following octet(s)
The Unicode Standard	Leading byte	Trailing byte
Wikipedia / Industry Jargon	Leading byte	Continuation byte

Are you writing a parser or an ABNF schema where you need to implement these exact UTF8-tail constraints? [..]

---

    So I followed up to google/gemini with this reply to its question at the end of the above:

I'm writing a utf-8 parser, in particular trying to name a function that tests if it's in the middle of a utf-8 sequence, or has reached the beginning or end. Can you suggest a proper name for a function that tests bytes (octets) in a utf-8 string?

    ..and it replied:

---

Since you are writing a parser, naming your functions after the official Unicode Standard or RFC 3629 ABNF terms will make your code immediately understandable to other systems engineers.

Here are the best function names categorized by the specific design pattern you are using, alongside names for your beginning/end checks.

Option 1: The "Trailing" Pattern (Recommended)

This maps directly to the Unicode Standard. It is the most common industry convention for systems-level parsers (like those in Rust, Go, or C++).

• is_trailing_byte(b) (or is_trail_byte): Returns true if the byte is in the middle of a sequence (0x80 to 0xBF).
• is_leading_byte(b) (or is_lead_byte): Returns true if the byte can start a sequence (0x00 to 0x7F, or 0xC2 to 0xF4).

Option 2: The "Tail" Pattern (RFC 3629 ABNF Alignment)

If your parser strictly follows the formal IETF grammar nomenclature, name it after the exact ABNF production rule.

• is_utf8_tail(b): Returns true if the byte matches the UTF8-tail definition (%x80-BF).

Option 3: The Boundary Check Pattern

If your function is not just inspecting a single byte, but checking an iterator or index pointer to see if it is currently sitting safely on a character boundary (the beginning or end of a full sequence):

• is_char_boundary(index): Returns true if the index points to a valid starting byte or the end of the string.
• is_inside_sequence(index): Returns true if the index is currently sitting on a middle/subsequent byte.

---

Implementation Reference (Hex Ranges)

[..hex table snipped..]

In code, a byte is a middle/trailing byte if and only if its top two bits are 10. You can test this instantly with a bitwise mask:

// Evaluates to true only for middle/subsequent bytes (0x80 to 0xBF)
bool is_trailing_byte(uint8_t byte) {
    return (byte & 0xC0) == 0x80;                // <-- SAME CODE WE USE IN FLTK -ERCO
}

// Evaluates to true if the pointer is at a safe splitting/starting point
bool is_char_boundary(uint8_t byte) {
    return !is_trailing_byte(byte);
}

[So Gemini is suggesting "is_trailing_byte()", or I guess we could abbreviate "is_trail()", but I think that's not as clear as "continuation" -erco] 

---

    For competeness, I switched to using Codex/ChatGPT, and asked it the same two questions..
    Here's that conversation, which seemed to prefer the "continuation" terminology:

---

> I'm trying to determine if there's specific terminology for the bytes in the middle of a UTF-8 sequence. The wikipedia page calls
  them "continuation" bytes, but the RFCs do not mention this.

• I’ll check the primary specs rather than relying on memory, since UTF-8 terminology varies a bit between Unicode, RFC 3629, and secondary summaries.

• Searched the web for Unicode Standard UTF-8 continuation byte terminology
• Searched the web for https://www.rfc-editor.org/info/rfc3629/
• Searched the web for 'continuation' in https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/
• Searched the web for WHATWG Encoding UTF-8 continuation byte definition

• Yes. The common term is continuation byte, and it is widely understood for UTF-8 bytes of the form 10xxxxxx, generally 0x80..0xBF.

  If you want terminology closer to primary standards:

  - The Unicode Standard uses leading byte and trailing byte terminology. Chapter 2 says UTF-8 has a first/leading element and
    subsequent/trailing elements, and Chapter 3 refers to “trailing byte range” in the well-formed UTF-8 table. Sources: Unicode Chapter
    2, §2.5.3 and Unicode Chapter 3, §3.9.3.

  - RFC 3629 does not appear to use “continuation byte”. Its ABNF calls them UTF8-tail, defined as %x80-BF, and the prose calls them the
    “following octet(s)”. Source: RFC 3629 §3-4.

  So for spec-aligned wording, use UTF-8 trailing byte or UTF-8 tail byte. For general programming/documentation wording, UTF-8
  continuation byte is normal and clear. [Because of this, I think we're OK with "continuation" - erco]

  One nuance: every UTF-8 continuation/trailing byte has the bit shape 10xxxxxx, but not every syntactically valid second byte can be
  described only as 0x80..0xBF in context. Well-formed UTF-8 has extra restrictions for some first bytes, e.g. E0 A0..BF, F0 90..BF, F4
  80..8F.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

› I'm writing a utf-8 parser, in particular trying to name a function that tests if it's in the middle of a utf-8 sequence. Can you
  suggest a proper name for such a function that tests bytes (octets) in a utf-8 string?

• Use a name based on continuation or trail bytes.

  Good options:

      is_utf8_continuation_byte(unsigned char c)

  or, if you want Unicode/ICU-style terminology:

     is_utf8_trail_byte(unsigned char c)

  I’d pick is_utf8_continuation_byte() for readability unless the surrounding code already uses “lead/trail” terminology. [ <-- So here again, chatgpt seems OK with "continuation" and even prefers it. And I'm fine leaving out the "byte" suffix, which seems unnecessary, due to the function prototype. -erco]

  The test itself is usually:

  return (c & 0xC0) == 0x80;  /* 10xxxxxx */

  If the function means “this byte cannot begin a UTF-8 character and must continue one,” then is_utf8_continuation_byte is exact. If it
  means “we are currently after a lead byte and still expecting more bytes,” name the parser-state function differently, e.g.:

      utf8_in_sequence()
      utf8_expect_continuation()
      utf8_need_trail_byte()

 

[melcher....@googlemail.com]

er...@seriss.com schrieb am Montag, 20. Juli 2026 um 18:43:07 UTC+2:

That was ssurprsingly entertaining for me ;-D

So how about (which could be confused with the start and end of a string):

`fl_utf8_is_head()` and `fl_utf8_is_tail()`

or the long version which makes the job very clear

`fl_utf8_is_leading_byte()` and `fl_utf8_is_trailing_byte()` or `fl_utf8_is_continuation_byte()`

[Manolo Gouy]

> Le 20 juil. 2026 à 18:13, Greg Ercolano <er...@seriss.com> a écrit :
>
>
>> The question for a good name of this new function remains. I propose:
>>
>> (1) fl_utf8_is_continuation()
>> (2) fl_utf8_is_cont()
>> (3) fl_utf8_continuation()
>> (4) fl_utf8_cont()

>> (5) none of these, I propose: …
My vote is for (1) although I would have preferred fl_utf8_is_continuation_byte()

[Greg Ercolano]

On 7/20/26 10:33, 'melcher....@googlemail.com' via fltk.coredev wrote:

That was ssurprsingly entertaining for me ;-D

    Ha, glad for that..!

So how about (which could be confused with the start and end of a string):

    `fl_utf8_is_head()` and `fl_utf8_is_tail()`

or the long version which makes the job very clear

    `fl_utf8_is_leading_byte()` and `fl_utf8_is_trailing_byte()` or `fl_utf8_is_continuation_byte()`

    Interesting.. but I'mma stick with (2) for fltk_utf8_is_cont() or even fltk_utf8_is_continuation() because that prevents needing two functions (e.g. head vs. tail), since is_continuation() covers both by simply telling you if you're 'in the middle or not'.

    Also, having one function instead of two means a parent walking the string either 'to the beginning' or 'to the end' doesn't have to worry about using the wrong function accidentally.

[Bill Spitzak]

"tail" seems good to me.

You don't need a "head" function. Characters start on !is_tail() bytes. "head" is rather ambiguous as to whether it includes ASCII or invalid bytes, but those all start a character as far as fltk is concerned.

--

You received this message because you are subscribed to the Google Groups "fltk.coredev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fltkcoredev...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/fltkcoredev/4331f9e1-6329-41ba-9d83-4ef1b789f136%40seriss.com.

[Albrecht Schlosser]

On 7/21/26 16:36 Bill Spitzak wrote:
> "tail" seems good to me.

+1 "tail" is a full but definitely short word, better than "cont" which
is an arbitrary abbreviation.

> You don't need a "head" function. Characters start on !is_tail()
> bytes. "head" is rather ambiguous as to whether it includes ASCII or
> invalid bytes, but those all start a character as far as fltk is
> concerned.

Good point.

[Greg Ercolano]

So naming things really is the hardest thing.. I always thought so, lol.

It's looking like everyone agrees on fl_utf8_is_continuation(); adding the results up:

• fl_utf8_is_continuation()
      manolo: My vote is for (1) although I would have preferred fl_utf8_is_continuation_byte()
      bill: I would use either is_continuation() or !is_leading()
      albrecht: I vote for fl_utf8_is_continuation() w/o the redundant "_byte" suffix.
      greg: Suggesting 'is_aligned' or 'is_align' or 'is_continuation'
  
  
• fl_utf8_is_cont()
      albrecht: I'd also agree with fl_utf8_is_cont() for brevity
      greg: +1 for (2) [fl_utf8_is_cont()]
  
  
• fl_utf8_is_tail()
      bill: "tail" seems good to me
      albrecht: +1 "tail" is a full but definitely short word, better than "cont" which is an arbitrary abbreviation.
  
  

There's some other suggs but not more than one or two votes.

Matt doesn't show up because he seemed to not care what it's named
as long as it's "using the right terminology".

I think the "_byte" suffix has been voted out: manolo prefers it, albrecht and greg
think it's redundant, all bill's suggestions leave it out.

I'll leave the voting open a bit longer, just in case there's a change, e.g. if "_tail" gets traction.

I'm -1 for _tail() because it's unclear to me: Even though it's actually used in the RFC spec as ABNF (augmented bakus-naur format), I think bakus-naur is its own world, and outside of that limited and specific context it's unclear; to the casual reader it implies "tail" is the very end of something (last byte), not possibly somewhere in the middle. But I could be swayed with conversation; "tail" is very nicely short!