NUL-terminating APIs, was [scite] Serious problem with lua GetSelText function in Scite 5.1.5
19 views
Skip to first unread message
Neil Hodgson
Nov 10, 2021, 1:07:01 AM11/10/21
to scite-interest, Scintilla mailing list
Paolo Gotti:
> I'm sorry to bother you, but the behaviour of lua GetSelText function has changed.
> The returned string now always include the terminating 0.
In previous versions of SciTE, string results from Scintilla API calls that included NUL bytes were truncated early at the NUL. A change was made to pass the whole string including any NUL bytes.
There have been some inconsistencies in the way that string returning APIs were handled by Scintilla, particularly SCI_GETTEXT, SCI_GETSELTEXT, and SCI_GETCURLINE. Most APIs return the length of the string excluding any NUL byte terminator but the mentioned APIs returned one more, including the NUL.
It is likely best to change these 3 APIs to be consistent with others. However, this could break applications that will no longer allocate enough memory to include the NUL terminator. Downstream projects should check their use of these APIs.
The normal technique is to call the Scintilla API with NULL as the string argument which returns the length. Then allocate a string of length+1 bytes (the +1 may be set to NUL). Then call the API with this allocated string. Use the resulting string in a way that ignores the trailing NUL. If using the C++ std::string type, an extra NUL byte is automatically allocated at the end which is not counted in std::string::length() so works without problems.
The attached LengthFix.patch changes Scintilla to return the string length for the 3 above mentioned APIs.
The attached LengthFixSciTE.patch changes the Lua Scintilla interface to return an empty string instead of nil when the API returns an empty string. This was previously hidden by the +1 on the length.
> As a consequence, even no selection at all gets length=1.
> Many scripts of mine are broken, but I have already reverted to the previous version.
> Nonetheless, somebody could get strange results.
The above changes modify behaviour too much to be pushed in a quick update, so users should either strip terminating NULs off API results or return to the previous SciTE release for now.
Neil
> I'm sorry to bother you, but the behaviour of lua GetSelText function has changed.
> The returned string now always include the terminating 0.
In previous versions of SciTE, string results from Scintilla API calls that included NUL bytes were truncated early at the NUL. A change was made to pass the whole string including any NUL bytes.
There have been some inconsistencies in the way that string returning APIs were handled by Scintilla, particularly SCI_GETTEXT, SCI_GETSELTEXT, and SCI_GETCURLINE. Most APIs return the length of the string excluding any NUL byte terminator but the mentioned APIs returned one more, including the NUL.
It is likely best to change these 3 APIs to be consistent with others. However, this could break applications that will no longer allocate enough memory to include the NUL terminator. Downstream projects should check their use of these APIs.
The normal technique is to call the Scintilla API with NULL as the string argument which returns the length. Then allocate a string of length+1 bytes (the +1 may be set to NUL). Then call the API with this allocated string. Use the resulting string in a way that ignores the trailing NUL. If using the C++ std::string type, an extra NUL byte is automatically allocated at the end which is not counted in std::string::length() so works without problems.
The attached LengthFix.patch changes Scintilla to return the string length for the 3 above mentioned APIs.
The attached LengthFixSciTE.patch changes the Lua Scintilla interface to return an empty string instead of nil when the API returns an empty string. This was previously hidden by the +1 on the length.
> As a consequence, even no selection at all gets length=1.
> Many scripts of mine are broken, but I have already reverted to the previous version.
> Nonetheless, somebody could get strange results.
The above changes modify behaviour too much to be pushed in a quick update, so users should either strip terminating NULs off API results or return to the previous SciTE release for now.
Neil
Mitchell
Nov 10, 2021, 10:37:28 PM11/10/21
to scintilla...@googlegroups.com
Hi,
On Wed, 10 Nov 2021 17:06:57 +1100
"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:
> Paolo Gotti:
>
> > I'm sorry to bother you, but the behaviour of lua GetSelText function has changed.
> > The returned string now always include the terminating 0.
>
> In previous versions of SciTE, string results from Scintilla API calls that included NUL bytes were truncated early at the NUL. A change was made to pass the whole string including any NUL bytes.
>
> There have been some inconsistencies in the way that string returning APIs were handled by Scintilla, particularly SCI_GETTEXT, SCI_GETSELTEXT, and SCI_GETCURLINE. Most APIs return the length of the string excluding any NUL byte terminator but the mentioned APIs returned one more, including the NUL.
>
> It is likely best to change these 3 APIs to be consistent with others. However, this could break applications that will no longer allocate enough memory to include the NUL terminator. Downstream projects should check their use of these APIs.
>
> The normal technique is to call the Scintilla API with NULL as the string argument which returns the length. Then allocate a string of length+1 bytes (the +1 may be set to NUL). Then call the API with this allocated string. Use the resulting string in a way that ignores the trailing NUL. If using the C++ std::string type, an extra NUL byte is automatically allocated at the end which is not counted in std::string::length() so works without problems.
My project uses this normal technique, but has special cases for SCI_GETTEXT, SCI_GETSELTEXT, and SCI_GETCURLINE to throw out the extra NUL byte returned. If Scintilla does not include it by default (i.e. Neil's proposed patch), I can remove this special handling and that would be nice.
Cheers,
Mitchell
On Wed, 10 Nov 2021 17:06:57 +1100
"'Neil Hodgson' via scintilla-interest" <scintilla...@googlegroups.com> wrote:
> Paolo Gotti:
>
> > I'm sorry to bother you, but the behaviour of lua GetSelText function has changed.
> > The returned string now always include the terminating 0.
>
> In previous versions of SciTE, string results from Scintilla API calls that included NUL bytes were truncated early at the NUL. A change was made to pass the whole string including any NUL bytes.
>
> There have been some inconsistencies in the way that string returning APIs were handled by Scintilla, particularly SCI_GETTEXT, SCI_GETSELTEXT, and SCI_GETCURLINE. Most APIs return the length of the string excluding any NUL byte terminator but the mentioned APIs returned one more, including the NUL.
>
> It is likely best to change these 3 APIs to be consistent with others. However, this could break applications that will no longer allocate enough memory to include the NUL terminator. Downstream projects should check their use of these APIs.
>
> The normal technique is to call the Scintilla API with NULL as the string argument which returns the length. Then allocate a string of length+1 bytes (the +1 may be set to NUL). Then call the API with this allocated string. Use the resulting string in a way that ignores the trailing NUL. If using the C++ std::string type, an extra NUL byte is automatically allocated at the end which is not counted in std::string::length() so works without problems.
Cheers,
Mitchell
Neil Hodgson
Nov 11, 2021, 6:20:46 PM11/11/21
to Scintilla mailing list, scite-i...@googlegroups.com
The proposed change has now been committed. An additional change was made to SCI_GETTEXT so that the length (wParam) argument is the length of the buffer excluding the NUL as this is more consistent.
Neil
Reply all
Reply to author
Forward
0 new messages