[TCLCORE] Full Unicode for Tcl 8.7 (TIP #622)
17 views
Skip to first unread message
Jan Nijtmans
Mar 29, 2022, 12:30:59 PM3/29/22
to Tcl Core List
Hi all,
Recently, there was a lot of discussion about a side-effect
for TIP #389: Full support for Unicode 10.0 and later (part 1),
more specifically:
$ tclsh8.7
% string length 🤝
2
The only option I see which can solve that is defining
TCL_UTF_MAX=4 when building Tcl. But .... that would
mean that sizeof(Tcl_UniChar) = 4 (was: 2), which means
that all extensions need to be recompiled, something which
is not acceptable for a minor release.
However, I have been investigating a way out of this: We
can build Tcl with TCL_UTF_MAX=4, but still keep
TCL_UTF_MAX=3 in tcl.h. A number of functions affected
by this change, can be split in UTF-16 and UTF-32 functions,
where the UTF-16 variant is used when TCL_UTF_MAX=3 and
the UTF-32 variant in TCL_UTF_MAX=4. It's a kind of
UTF-16 compatibility layer restoring binary compatibility
for extensions.
This approach appears to be working now, and a TIP
is written describing this:
<https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
Implementation is in the "full-utf-for-87" branch.
<https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
So, I would like to start discussing this idea.
Regards,
Jan Nijtmans
_______________________________________________
Tcl-Core mailing list
Tcl-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tcl-core
Porter, Don (Fed) via Tcl-Core
Apr 1, 2022, 10:13:21 AM4/1/22
to Jan Nijtmans, Tcl Core List
I don't want to give any false impressions. I have not yet read the TIP. I have not yet reviewed the code changes on the full-utf-for-87 branch. My sense of things is limited to seeing checkin comments go by on the tcl-bugs mailing list.
That said, on that limited foundation, I'm excited and pleased about at least what I think is happening there. I like the idea of offering a greater amount of Unicode support in Tcl 8.7, and doing it in a way that can move with little to no change into Tcl 9, all while overcoming some of the most jarring incompatibilities with Tcl 8.5 that have troubled a lot of us.
I can't be more concrete or detailed until I do the work, but I hope this positive comment from me will encourage others to examine the branch, try it out, and comment on how it interacts with deployed uses of Tcl.
That said, on that limited foundation, I'm excited and pleased about at least what I think is happening there. I like the idea of offering a greater amount of Unicode support in Tcl 8.7, and doing it in a way that can move with little to no change into Tcl 9, all while overcoming some of the most jarring incompatibilities with Tcl 8.5 that have troubled a lot of us.
I can't be more concrete or detailed until I do the work, but I hope this positive comment from me will encourage others to examine the branch, try it out, and comment on how it interacts with deployed uses of Tcl.
DGP
From: Jan Nijtmans <jan.ni...@gmail.com>
Sent: Tuesday, March 29, 2022 12:30 PM
To: Tcl Core List <tcl-...@lists.sourceforge.net>
Subject: [TCLCORE] Full Unicode for Tcl 8.7 (TIP #622)
Sent: Tuesday, March 29, 2022 12:30 PM
To: Tcl Core List <tcl-...@lists.sourceforge.net>
Subject: [TCLCORE] Full Unicode for Tcl 8.7 (TIP #622)
Hi all,
Recently, there was a lot of discussion about a side-effect
for TIP #389: Full support for Unicode 10.0 and later (part 1),
more specifically:
$ tclsh8.7
% string length 🤝
2
The only option I see which can solve that is defining
TCL_UTF_MAX=4 when building Tcl. But .... that would
mean that sizeof(Tcl_UniChar) = 4 (was: 2), which means
that all extensions need to be recompiled, something which
is not acceptable for a minor release.
However, I have been investigating a way out of this: We
can build Tcl with TCL_UTF_MAX=4, but still keep
TCL_UTF_MAX=3 in tcl.h. A number of functions affected
by this change, can be split in UTF-16 and UTF-32 functions,
where the UTF-16 variant is used when TCL_UTF_MAX=3 and
the UTF-32 variant in TCL_UTF_MAX=4. It's a kind of
UTF-16 compatibility layer restoring binary compatibility
for extensions.
This approach appears to be working now, and a TIP
is written describing this:
Recently, there was a lot of discussion about a side-effect
for TIP #389: Full support for Unicode 10.0 and later (part 1),
more specifically:
$ tclsh8.7
% string length 🤝
2
The only option I see which can solve that is defining
TCL_UTF_MAX=4 when building Tcl. But .... that would
mean that sizeof(Tcl_UniChar) = 4 (was: 2), which means
that all extensions need to be recompiled, something which
is not acceptable for a minor release.
However, I have been investigating a way out of this: We
can build Tcl with TCL_UTF_MAX=4, but still keep
TCL_UTF_MAX=3 in tcl.h. A number of functions affected
by this change, can be split in UTF-16 and UTF-32 functions,
where the UTF-16 variant is used when TCL_UTF_MAX=3 and
the UTF-32 variant in TCL_UTF_MAX=4. It's a kind of
UTF-16 compatibility layer restoring binary compatibility
for extensions.
This approach appears to be working now, and a TIP
is written describing this:
Implementation is in the "full-utf-for-87" branch.
So, I would like to start discussing this idea.
Regards,
Jan Nijtmans
_______________________________________________
Tcl-Core mailing list
Tcl-...@lists.sourceforge.net
Jan Nijtmans
Apr 14, 2022, 8:17:12 AM4/14/22
to Porter, Don (Fed), Tcl Core List
Op vr 1 apr. 2022 om 16:12 schreef Porter, Don:
How much time do you need to check out this branch and review it? I think the
branch is ready for vote, but - this time - I don't want people to start looking
at it when the vote already started....
The branch is here:
<https://core.tcl-lang.org/tcl/info/f8cdc915ce52bcc9>
Thanks!
>
> I don't want to give any false impressions. I have not yet read the TIP. I have not yet reviewed the code changes on the full-utf-for-87 branch. My sense of things is limited to seeing checkin comments go by on the tcl-bugs mailing list.
>
> That said, on that limited foundation, I'm excited and pleased about at least what I think is happening there. I like the idea of offering a greater amount of Unicode support in Tcl 8.7, and doing it in a way that can move with little to no change into Tcl 9, all while overcoming some of the most jarring incompatibilities with Tcl 8.5 that have troubled a lot of us.
>
> I can't be more concrete or detailed until I do the work, but I hope this positive comment from me will encourage others to examine the branch, try it out, and comment on how it interacts with deployed uses of Tcl.
Don (and others)!
> I don't want to give any false impressions. I have not yet read the TIP. I have not yet reviewed the code changes on the full-utf-for-87 branch. My sense of things is limited to seeing checkin comments go by on the tcl-bugs mailing list.
>
> That said, on that limited foundation, I'm excited and pleased about at least what I think is happening there. I like the idea of offering a greater amount of Unicode support in Tcl 8.7, and doing it in a way that can move with little to no change into Tcl 9, all while overcoming some of the most jarring incompatibilities with Tcl 8.5 that have troubled a lot of us.
>
> I can't be more concrete or detailed until I do the work, but I hope this positive comment from me will encourage others to examine the branch, try it out, and comment on how it interacts with deployed uses of Tcl.
How much time do you need to check out this branch and review it? I think the
branch is ready for vote, but - this time - I don't want people to start looking
at it when the vote already started....
The branch is here:
<https://core.tcl-lang.org/tcl/info/f8cdc915ce52bcc9>
Thanks!
Jan Nijtmans
_______________________________________________
Tcl-Core mailing list
Tcl-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tcl-core
_______________________________________________
Tcl-Core mailing list
Tcl-...@lists.sourceforge.net
Jan Nijtmans
Apr 27, 2022, 1:02:32 PM4/27/22
to Tcl Core List
This is the Call For Votes for TIP #622.
<https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
Almost one month has passed since I asked people to
look at the branch, I don't think anyone did (at least
no-one reported any feedback). Since the implementation
and TIP are finished, no-one appears to have issues,
let's start the vote.
Implementation is in the "full-utf-for-87" branch.
<https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
Let's try to have votes in about three weeks, to conclude
Monday May 16 noon, UTC, or [clock format 1652702400].
If more time is needed, let me know, then the period
can still be extended.
My vote:
TIP 622: YES
Regards,
Jan Nijtmans
Op di 29 mrt. 2022 om 18:30 schreef Jan Nijtmans:
Harald Oehlmann
Apr 27, 2022, 1:51:28 PM4/27/22
to tcl-...@lists.sourceforge.net
Am 27.04.2022 um 19:01 schrieb Jan Nijtmans:
> This is the Call For Votes for TIP #622.
>
> <https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
>
> Almost one month has passed since I asked people to
> look at the branch, I don't think anyone did (at least
> no-one reported any feedback). Since the implementation
> and TIP are finished, no-one appears to have issues,
> let's start the vote.
Thank you, Jan, for the work. As a side note, I saw an appreciation
> This is the Call For Votes for TIP #622.
>
> <https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
>
> Almost one month has passed since I asked people to
> look at the branch, I don't think anyone did (at least
> no-one reported any feedback). Since the implementation
> and TIP are finished, no-one appears to have issues,
> let's start the vote.
E-Mail on the core list for that by PoorYorik what is IMHO a very
important voice.
Thank you,
Harald
Jan Nijtmans
May 10, 2022, 2:41:09 AM5/10/22
to Tcl Core List
Op wo 27 apr. 2022 om 19:01 schreef Jan Nijtmans:
>
> This is the Call For Votes for TIP #622.
>
> <https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
...
> This is the Call For Votes for TIP #622.
>
> <https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
>
> Implementation is in the "full-utf-for-87" branch.
> <https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
>
> Let's try to have votes in about three weeks, to conclude
> Monday May 16 noon, UTC, or [clock format 1652702400].
> If more time is needed, let me know, then the period
> can still be extended.
Reminder: 6 days left to go.
> Implementation is in the "full-utf-for-87" branch.
> <https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
>
> Let's try to have votes in about three weeks, to conclude
> Monday May 16 noon, UTC, or [clock format 1652702400].
> If more time is needed, let me know, then the period
> can still be extended.
Kevin Walzer
May 10, 2022, 9:43:25 AM5/10/22
to tcl-...@lists.sourceforge.net
On 5/10/22 2:40 AM, Jan Nijtmans wrote:
>
>> Implementation is in the "full-utf-for-87" branch.
>> <https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
>>
>> Let's try to have votes in about three weeks, to conclude
>> Monday May 16 noon, UTC, or [clock format 1652702400].
>> If more time is needed, let me know, then the period
>> can still be extended.
> Reminder: 6 days left to go.
>
> Regards,
> Jan Nijtmans
>
TIP 622: YES
>
>> Implementation is in the "full-utf-for-87" branch.
>> <https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
>>
>> Let's try to have votes in about three weeks, to conclude
>> Monday May 16 noon, UTC, or [clock format 1652702400].
>> If more time is needed, let me know, then the period
>> can still be extended.
> Reminder: 6 days left to go.
>
> Regards,
> Jan Nijtmans
>
--
Kevin Walzer
Code by Kevin
http://www.codebykevin.com
Jos Decoster
May 10, 2022, 2:41:00 PM5/10/22
to Jan Nijtmans, Tcl Core List
Hi,
I'm not into unicode. Vote: PRESENT.
Kind regards,
Jos.
Andreas Kupries
May 11, 2022, 12:18:25 PM5/11/22
to Jan Nijtmans, Tcl Core List
> This is the Call For Votes for TIP #622.
>
> <https://core.tcl-lang.org/tips/doc/trunk/tip/622.md>
>
> Almost one month has passed since I asked people to
> look at the branch, I don't think anyone did (at least
> no-one reported any feedback). Since the implementation
> and TIP are finished, no-one appears to have issues,
> let's start the vote.
--
Happy Tcling,
Andreas Kupries <andreas...@gmail.com>
<https://core.tcl-lang.org/akupries/>
<https://akupries.tclers.tk/>
Developer @ SUSE Software Solutions Germany GmbH
-------------------------------------------------------------------------------
st...@digitalsmarties.com
May 11, 2022, 10:44:37 PM5/11/22
to Tcl Core List, Jan Nijtmans
On 28 Apr 2022, 1:03 AM +0800, Jan Nijtmans <jan.ni...@gmail.com>, wrote:
This is the Call For Votes for TIP #622.
TIP #622: YES
Jan Nijtmans
May 17, 2022, 5:02:44 PM5/17/22
to Tcl Core List
Op wo 27 apr. 2022 om 19:01 schreef Jan Nijtmans:
>
....
>
> Implementation is in the "full-utf-for-87" branch.
> <https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
>
> Let's try to have votes in about three weeks, to conclude
> Monday May 16 noon, UTC, or [clock format 1652702400].
The deadline passed, here are the results:
> <https://core.tcl-lang.org/tcl/timeline?r=full-utf-for-87>
>
> Let's try to have votes in about three weeks, to conclude
> Monday May 16 noon, UTC, or [clock format 1652702400].
Vote-Summary: Accepted 5/0/1
Votes-For: AK, JN, KBK, KW, SL
Votes-Against: none
Votes-Present: JD
This means, the TIP is accepted and now merged into
core-8-branch (for 8.7 only, _not_ for 9.0!)
Kevin Kenny wrote:
> There are still problems, but they're fixable in further TIPs. There are certain to
> be integration issues, which will be revealed only when this stuff gets deployed more widely.
So, let the bug-reports come in then ;-) I'm not aware of any
problems left but - please - check this with your own favorite
extension or application.
Thanks, all, for voting!
Jan Nijtmans
May 24, 2022, 4:27:55 AM5/24/22
to Tcl Core List
Op di 17 mei 2022 om 23:02 schreef Jan Nijtmans:
https://core.tcl-lang.org/tcl/tktview/f160f9f982e35beb8789431b59e1108edcaefc7e
There's a fix available, being tested now.
So, please, report any problems you find in core-8-branch, which might
be caused by TIP #622, using your favorite extension/application!
Thanks!
> Kevin Kenny wrote:
> > There are certain to
> > be integration issues, which will be revealed only when this stuff gets deployed more widely.
>
> So, let the bug-reports come in then ;-) I'm not aware of any
> problems left but - please - check this with your own favorite
> extension or application.
The first bug-report related to TIP #622 came in (Thanks, Nicolas!)
> > There are certain to
> > be integration issues, which will be revealed only when this stuff gets deployed more widely.
>
> So, let the bug-reports come in then ;-) I'm not aware of any
> problems left but - please - check this with your own favorite
> extension or application.
https://core.tcl-lang.org/tcl/tktview/f160f9f982e35beb8789431b59e1108edcaefc7e
There's a fix available, being tested now.
So, please, report any problems you find in core-8-branch, which might
be caused by TIP #622, using your favorite extension/application!
Thanks!
Reply all
Reply to author
Forward
0 new messages