Enforcing UTF-8 validity

[Neil Hodgson]

Scintilla has defined a document as a sequence of octets. An encoding is used to specify how those octets are displayed and selected as characters but is not used to filter input to ensure that the document is valid in that encoding. This allows Scintilla to load any file, even those that are invalid which can be very useful when investigating and fixing encoding problems.

However, some users would prefer to define a document as valid Unicode text and to only allow modifications that retain validity. Applications using this definition might refuse to load files that are not invalid Unicode text or may offer to fix any broken sections. Once loaded into Scintilla, it would do its part by refusing to perform any insertions or deletions that are not performed on character boundaries or are not over complete valid characters. Thus the application would be sure that the document is still valid when it is time to save.

Since applications may not wish this validating behaviour, there would be an option to turn the validation mode on or off. The important encoding for most is UTF-8 although this could be extended to other encodings if there is sufficient interest. Whenever Scintilla detects an operation that breaks validity, it will instead throw an exception which will be trapped at the Scintilla boundary and converted into status code.

Applications may watch for these failures and so fix the problem and retry, possibly by showing UI. Another possibility that may work better for some applications would be to provide an event when a validity-breaking operation is attempted.

Neil