fixing locale problems with floating numbers in printf and scanf on OSX

[Stefan Csomor]

Hi

there are some problems with the c-runtime locale not being respected correctly for all aspects, e.g. decimal points in my home locale de_CH '.' not being correctly, the fallback de_DE is having a ','. Now Core Foundation does have a printf like method, and everything were fine – if only wchar_t was a supported type, but alas it is not, only UniChar MacOS utf16 character is supported. 

I have undertaken several attempts to integrate this Core Foundation Method but failed 

• is there a printf code we have which I could use ? where I could redirect all %d %f etc to the Core Foundation implementation and only do %c and %s by hand …
• how about adding wchar -> UniChar conversions in the arg array, would this be feasible with a reasonable amount of effort
• how about just fixing the ToDouble / FromDouble calls in String and NumFormatter ?

Thanks,

Stefan

[Vadim Zeitlin]

On Thu, 13 Jun 2013 20:34:57 +0000 Stefan Csomor wrote:

SC> there are some problems with the c-runtime locale not being respected
SC> correctly for all aspects, e.g. decimal points in my home locale de_CH
SC> '.' not being correctly, the fallback de_DE is having a ','.

Sorry, do you mean that "de_CH" locale is not supported by Unix part of OS
X while "de_DE" is supported?

SC> Now Core Foundation does have a printf like method, and everything were
SC> fine – if only wchar_t was a supported type, but alas it is not, only
SC> UniChar MacOS utf16 character is supported.

In principle, we could convert to/from UTF-16 and still use them...

SC> I have undertaken several attempts to integrate this Core Foundation
SC> Method but failed
SC>
SC> * is there a printf code we have which I could use ? where I could
SC> redirect all %d %f etc to the Core Foundation implementation and only
SC> do %c and %s by hand …

Yes, we have our own implementation in src/common/wxprintf.cpp. It uses
system_sprintf() (see include/wx/private/wxprintf.h) to actually convert
each field to a string, so you could, in principle, change the calls to it
in wxPrintfConvSpec<CharType>::Process() for wxPAT_[LONG]DOUBLE cases to
some other function which would have a separate implementation for OS X.
Whether or not this is a good idea, I'm less sure.

SC> * how about adding wchar -> UniChar conversions in the arg array,
SC> would this be feasible with a reasonable amount of effort

The conversion is definitely feasible, I'm just not sure where would we
have to do it.

SC> * how about just fixing the ToDouble / FromDouble calls in String
SC> and NumFormatter ?

wxString is non-GUI code, it would be really unexpected for it to do
something different from printf(). For that matter, it would be probably
unexpected for wxPrintf() to behave differently from printf() too... OTOH
it could be argued that they should behave consistently with NSLog() or
whatever is used in CF for printing strings, of course.

Frankly, I don't see any good solution here. It would be so much simpler
if Apple just didn't randomly break things at Darwin level :-(

Regards,
VZ

[Stefan Csomor]

Hi

>SC> there are some problems with the c-runtime locale not being respected
>SC> correctly for all aspects, e.g. decimal points in my home locale de_CH
>SC> '.' not being correctly, the fallback de_DE is having a ','.
>
> Sorry, do you mean that "de_CH" locale is not supported by Unix part of
>OS
>X while "de_DE" is supported?

yes, basically it's just that it has a de_XX impl, but no de_CH variant

>SC> Now Core Foundation does have a printf like method, and everything
>were
>SC> fine ­ if only wchar_t was a supported type, but alas it is not, only
>SC> UniChar MacOS utf16 character is supported.
>
> In principle, we could convert to/from UTF-16 and still use them...
>
>SC> I have undertaken several attempts to integrate this Core Foundation
>SC> Method but failed
>SC>
>SC> * is there a printf code we have which I could use ? where I could
>SC> redirect all %d %f etc to the Core Foundation implementation and
>only

>SC> do %c and %s by hand Š

>
> Yes, we have our own implementation in src/common/wxprintf.cpp. It uses
>system_sprintf() (see include/wx/private/wxprintf.h) to actually convert
>each field to a string, so you could, in principle, change the calls to it
>in wxPrintfConvSpec<CharType>::Process() for wxPAT_[LONG]DOUBLE cases to
>some other function which would have a separate implementation for OS X.
>Whether or not this is a good idea, I'm less sure.

this would allow to fix things at a central level, now do we have the same
for scanf? because whatever I change, at least at the level I change it,
it has to be round-trip-safe ..to.. and ..from.. have to be at the same
workaround level

>
>SC> * how about adding wchar -> UniChar conversions in the arg array,
>SC> would this be feasible with a reasonable amount of effort
>
> The conversion is definitely feasible, I'm just not sure where would we
>have to do it.

me neither, unfortunately

>SC> * how about just fixing the ToDouble / FromDouble calls in String
>SC> and NumFormatter ?
>
> wxString is non-GUI code, it would be really unexpected for it to do
>something different from printf(). For that matter, it would be probably
>unexpected for wxPrintf() to behave differently from printf() too... OTOH
>it could be argued that they should behave consistently with NSLog() or
>whatever is used in CF for printing strings, of course.

exactly, actually we have one printf implementation a c-runtime level
which is rather incomplete, one a core foundation level which works
correctly, but needs utf16 strings

>Frankly, I don't see any good solution here. It would be so much simpler
>if Apple just didn't randomly break things at Darwin level :-(

yes, definitely Š

Thanks,

Stefan

[Vadim Zeitlin]

On Fri, 14 Jun 2013 13:56:57 +0000 Stefan Csomor wrote:

SC> > Sorry, do you mean that "de_CH" locale is not supported by Unix part of
SC> >OS X while "de_DE" is supported?
SC>
SC> yes, basically it's just that it has a de_XX impl, but no de_CH variant

Why hasn't Switzerland officially complained about it yet? In France there
would be demonstrations in the street if something like this happened :-/

SC> >SC> * is there a printf code we have which I could use ? where I could

SC> >SC> redirect all %d %f etc to the Core Foundation implementation and only
SC> >SC> do %c and %s by hand Š
SC> >
SC> > Yes, we have our own implementation in src/common/wxprintf.cpp. It uses
SC> >system_sprintf() (see include/wx/private/wxprintf.h) to actually convert
SC> >each field to a string, so you could, in principle, change the calls to it
SC> >in wxPrintfConvSpec<CharType>::Process() for wxPAT_[LONG]DOUBLE cases to
SC> >some other function which would have a separate implementation for OS X.
SC> >Whether or not this is a good idea, I'm less sure.
SC>
SC> this would allow to fix things at a central level, now do we have the same
SC> for scanf?

No, we rely on system vswscanf(). And it's really not obvious to solve
this from outside, i.e. the trick with replacing commas with dots (or vice
versa) wouldn't work for arbitrary strings, of course.

Does CF provide a vswscanf() replacement?

SC> because whatever I change, at least at the level I change it,
SC> it has to be round-trip-safe ..to.. and ..from.. have to be at the same
SC> workaround level

notice that wxString::ToDouble() doesn't use sscanf() but rather strtod().


SC> exactly, actually we have one printf implementation a c-runtime level
SC> which is rather incomplete, one a core foundation level which works
SC> correctly, but needs utf16 strings

I guess we really need to use wxNumberFormatter for floating point number
formatting. We can do whatever we want/need there and it's not a problem if
its behaviour is not exactly the same as that of printf(), that's the whole
point of having this class, in fact. So my suggestion is to do this there
and then also check that wxNumberFormatter is used in the components that
you care about, e.g. wxGrid.

SC> >Frankly, I don't see any good solution here. It would be so much simpler
SC> >if Apple just didn't randomly break things at Darwin level :-(
SC>
SC> yes, definitely Š

Would you have asked them about the probability of this being fixed at
WWDC by chance?
VZ

[John Ralls]

Apple doesn't use libc for string output or localization, so it's probably not their fault. The CoreFoundation
Unicode-code is a wrapper around ICU, for which Apple provides at least /usr/lib/libicucore.A.dylib. If
you don't like the CFString interface you can use ICU directly after installing the headers.

CFString supports transcoding with CFStringCreateFromExternalRepresentation and formatting with
CFStringCreateWithFormat(). What's wrong with using the former to convert wchar_t strings to CFStrings
and the latter to create a CFString for insertion into a control or for terminal output?

As for the de_CH locale, /usr/share/locale/de_CH.UTF-8/LC_NUMERIC is symlinked to de_DE.UTF-8.
Again, Apple doesn't use those locales themselves, they're there for the convenience of unix programs that
don't use ICU.

Regards,
John Ralls

[Vadim Zeitlin]

On Fri, 14 Jun 2013 10:16:54 -0400 John Ralls wrote:

JR> Apple doesn't use libc for string output or localization, so it's
JR> probably not their fault.

While I don't think it really matters whether it's their fault or not
because it doesn't help us anyhow and doesn't indicate in any whether Apple
is going to change or not, I have trouble understanding this point of view.
Whose fault is it that they took random bits and pieces of FreeBSD but not
all of them? I would have understood if they didn't provide any POSIX API
at all, this would have been stupid but at least consistent. But providing
an API which is half broken is just ridiculous. Not as bad as their locale
support in C++ (which crashes if you try to use it), but still pretty much
their fault, I believe.


JR> The CoreFoundation Unicode-code is a wrapper around ICU, for which
JR> Apple provides at least /usr/lib/libicucore.A.dylib. If you don't like
JR> the CFString interface you can use ICU directly after installing the
JR> headers.

This is not really an option, we don't want to make wxOSX installation
even more difficult. And nothing guarantees us that these libraries will
still be available in the future. Without even saying that I am not sure if
they compiled them from the unmodified ICU sources, i.e. that we can't be
sure that the headers from the upstream ICU are compatible with them.


JR> CFString supports transcoding with
JR> CFStringCreateFromExternalRepresentation and formatting with
JR> CFStringCreateWithFormat(). What's wrong with using the former to
JR> convert wchar_t strings to CFStrings and the latter to create a
JR> CFString for insertion into a control or for terminal output?

But where would we do this in wx API? This is the real problem: we use
wxPrintf() and/or wxString::{To,From}Double() in a lot of places inside
wxWidgets and the user application code uses these functions too. So should
we use CF functions there? If so, we also need them for wxScanf(). But I'm
not sure if CF provides its replacement, so it would need to be written.
And this would still leave the problem of wxPrintf() being incompatible
with printf() which I feel uncomfortable about.

Only using the CF functions in wxNumberFormatter is not a problem, of
course, as I wrote in another reply. But it's not really the full solution
neither.


JR> As for the de_CH locale, /usr/share/locale/de_CH.UTF-8/LC_NUMERIC is
JR> symlinked to de_DE.UTF-8. Again, Apple doesn't use those locales
JR> themselves, they're there for the convenience of unix programs that
JR> don't use ICU.

They could have just copied de_CH locale from the same place they copied
de_DE files from and avoided the problem completely. And for once I'm not
even going to accuse them of behaving maliciously (perhaps because I'm not
Swiss), it just seems very sloppy.

Regards,
VZ

[Stefan Csomor]

Hi

>CFString supports transcoding with
>CFStringCreateFromExternalRepresentation and formatting with
>CFStringCreateWithFormat(). What's wrong with using the former to convert
>wchar_t strings to CFStrings
>and the latter to create a CFString for insertion into a control or for
>terminal output?

I'm using CFStringCreateWithFormat in my current sandbox, but that one is
expecting utf16 strings, that's why I was asking on where to put that
conversion Š

Best,

Stefan

[Stefan Csomor]

Hi

>Why hasn't Switzerland officially complained about it yet? In France there
>would be demonstrations in the street if something like this happened :-/

I think Switzerland is at a bad position right now towards the USA for
these things ;-)

no, only for individual conversions like strod there's a
CFStringGetDoubleValue

>
>SC> because whatever I change, at least at the level I change it,
>SC> it has to be round-trip-safe ..to.. and ..from.. have to be at the
>same
>SC> workaround level
>
> notice that wxString::ToDouble() doesn't use sscanf() but rather
>strtod().
>
>
>SC> exactly, actually we have one printf implementation a c-runtime level
>SC> which is rather incomplete, one a core foundation level which works
>SC> correctly, but needs utf16 strings
>
> I guess we really need to use wxNumberFormatter for floating point number
>formatting. We can do whatever we want/need there and it's not a problem
>if
>its behaviour is not exactly the same as that of printf(), that's the
>whole
>point of having this class, in fact. So my suggestion is to do this there
>and then also check that wxNumberFormatter is used in the components that
>you care about, e.g. wxGrid.

Ok, I'll do that, or how about doing the replacements at the

level

wxString::ToDouble
wxString::FromDouble


and use that (btw. shouldn't num wxNumberFormatter::ToString use
wxString::FromDouble as well instead of yet another wxString::Format in
the first lines) ?

>
>SC> >Frankly, I don't see any good solution here. It would be so much
>simpler
>SC> >if Apple just didn't randomly break things at Darwin level :-(
>SC>
>SC> yes, definitely Š
>
> Would you have asked them about the probability of this being fixed at
>WWDC by chance?

any non critical bug with implications like this would most probably only
be fixed for the next major release - so this doesn't help - but I was
glad I could have the cocoa engineers review my workarounds - and look at
their NSComoBoxCell source code to verify that there's no other way ;-)

Best,

Stefan

[Vadim Zeitlin]

On Fri, 14 Jun 2013 16:25:25 +0000 Stefan Csomor wrote:

SC> >CFString supports transcoding with
SC> >CFStringCreateFromExternalRepresentation and formatting with
SC> >CFStringCreateWithFormat(). What's wrong with using the former to convert
SC> >wchar_t strings to CFStrings
SC> >and the latter to create a CFString for insertion into a control or for
SC> >terminal output?
SC>
SC> I'm using CFStringCreateWithFormat in my current sandbox, but that one is
SC> expecting utf16 strings, that's why I was asking on where to put that
SC> conversion Š

Sorry, but I just don't see how is the conversion the problem. We can
convert just before calling this function, it's trivial (well, ignoring the
performance implications for larger strings). The question is where to call
it from? And I still don't see any other place than wxNumberFormatter to do
it.

Regards,
VZ

[Vadim Zeitlin]

On Fri, 14 Jun 2013 16:52:43 +0000 Stefan Csomor wrote:

SC> > Does CF provide a vswscanf() replacement?
SC>
SC> no, only for individual conversions like strod there's a
SC> CFStringGetDoubleValue

Well, this means we shouldn't touch wxPrintf() as it should continue being
symmetric/consistent with wxScanf().

SC> > I guess we really need to use wxNumberFormatter for floating point number
SC> >formatting. We can do whatever we want/need there and it's not a problem
SC> >if its behaviour is not exactly the same as that of printf(), that's
SC> >the whole point of having this class, in fact. So my suggestion is to
SC> >do this there and then also check that wxNumberFormatter is used in
SC> >the components that you care about, e.g. wxGrid.
SC>
SC> Ok, I'll do that,

TIA!

SC> or how about doing the replacements at the level
SC>
SC> wxString::ToDouble
SC> wxString::FromDouble
SC>
SC> and use that

I'm still somewhat uncomfortable with using CF methods at this level. But
if you really think we must (why?), let's do it. Personally I'd prefer to
keep this in wxNumberFormatter and tell people to use wxString::ToCDouble()
for portable data which should be locale-independent and wxNumberFormatter
for the locale-dependent stuff in the UI (in the large sense, i.e. anything
shown to the user, not just the GUI).

SC> (btw. shouldn't num wxNumberFormatter::ToString use
SC> wxString::FromDouble as well instead of yet another wxString::Format in
SC> the first lines) ?

Well, right now it doesn't matter, but if we do the above, then it should,
yes.

Regards,
VZ

[Stefan Csomor]

Hi

>SC> or how about doing the replacements at the level
>SC>
>SC> wxString::ToDouble
>SC> wxString::FromDouble
>SC>
>SC> and use that
>
> I'm still somewhat uncomfortable with using CF methods at this level. But
>if you really think we must (why?), let's do it. Personally I'd prefer to
>keep this in wxNumberFormatter and tell people to use
>wxString::ToCDouble()
>for portable data which should be locale-independent and wxNumberFormatter
>for the locale-dependent stuff in the UI (in the large sense, i.e.
>anything
>shown to the user, not just the GUI).

right now we have quite a lot of code using wxString::Format with for
doubles for things that end up in the UI. Always replacing this with
wxNumberFormatter would be a kind of overkill for me, so I'd suggest the
following:

wxNumberFormatter


doubles represented with locale-correct decimal point AND if desired
thousands separator

wxString::ToDouble
wxString::FromDouble

doubles represented with the locale-correct decimal point but NO thousands
separator

wxString::ToCDouble
wxString::FromCDouble

always using '.' as decimal separator



Best,

STefan

[Stefan Csomor]

Hi

>wxString::ToDouble
>wxString::FromDouble
>
>doubles represented with the locale-correct decimal point but NO thousands
>separator

BTW: I wouldn't necessarily use CFString methods here, but just a
replacement like

#ifdef __WXOSX__

// the c locale system is not as correct as the native one on OSX
// eg. de_CH gets rerouted to de_DE which is wrong for LC_NUMERIC
lconv * const lc = localeconv();
if ( lc )
{
wxString dp = GetDecimalSeparator();
wxString ldp = lc->decimal_point;
if ( ldp != dp )
s.Replace(ldp,dp);
}

#endif

Best,

Stefan

[John Ralls]

Not interested in fighting over this, I have better things to do. I was just making some
suggestions.

Core Foundation doesn't have a sscanf replacement: That's provided by NSScanner[1], which
may have too much of an impedance mismatch for you to wrap. CFCreateFromFormat(),
though, is called more or less just like printf, so it shouldn't be too hard to wrap it however you like.

There's a FreeBSD patch [2] for the Swiss locales, apparently never committed to Darwin. I'll file a radr
on it when I can get in (I'm got an error when I tried to log in right now). Doesn't do any good for wx in
general, though Stefan could apply it to his system as a work-around, as could other Swiss users.

Regards,
John Ralls

[1] https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Strings/Articles/Scanners.html#//apple_ref/doc/uid/20000147-BCIEFGHC
[2]www.freebsd.org/cgi/query-pr.cgi?pr=75502

[Vadim Zeitlin]

On Fri, 14 Jun 2013 21:00:44 +0000 Stefan Csomor wrote:

SC> right now we have quite a lot of code using wxString::Format with for
SC> doubles for things that end up in the UI. Always replacing this with
SC> wxNumberFormatter would be a kind of overkill for me, so I'd suggest the
SC> following:
SC>
SC> wxNumberFormatter
SC>
SC> doubles represented with locale-correct decimal point AND if desired
SC> thousands separator

SC>
SC> wxString::ToDouble
SC> wxString::FromDouble
SC>

SC> doubles represented with the locale-correct decimal point but NO thousands
SC> separator
SC>
SC> wxString::ToCDouble
SC> wxString::FromCDouble
SC>
SC> always using '.' as decimal separator

Yes, I understand this proposal, but what I'm trying to say is that the
difference between wxNumberFormatter and {To,From}Double seems rather
arbitrary to me. Presumably the numbers meant for people consumption should
be formatted according to their preferences and this includes both the
correct decimal character and correct thousands grouping. Of course the
former is more important but, still, I don't see any valid reason for not
using wxNumberFormatter which gives you both.

I understand that there are practical constraints on the time you can
spend on fixing this and I don't expect you to replace all calls to
{To,From}Double() with the use of wxNumberFormatter. But maybe you could do
it at least for the few of them that are important to you?

To summarize, IMHO it would be preferable to use wxNumberFormatter as much
as possible, even if we do what you propose above.

Regards,
VZ

[Stefan Csomor]

Hi

-----Ursprüngliche Nachricht-----
Von: Vadim Zeitlin <va...@wxwidgets.org>
Antworten an: <wx-...@googlegroups.com>
Datum: Samstag, 15. Juni 2013 03:45
An: <wx-...@googlegroups.com>
Betreff: Re[6]: [wx-dev] fixing locale problems with floating numbers in
printf and scanf on OSX

ok, agreed, but in order for wxNumberFormatter to have any advantage over
wxString::FromDouble we should be able to express whether people really
want the thousands separator, if not, then it doesn't offer more, and I
wouldn't dare to default to showing it, as this definitely would suddenly
behave differently than before.

Of course FromDouble and ToDouble could be implemented using
wxNumberFormatter completely - then it would always use the formatter and
still have the calling convenience the wxString methods have ?

Thanks,

Stefan

[Vadim Zeitlin]

On Sat, 15 Jun 2013 02:02:25 +0000 Stefan Csomor wrote:

SC> > To summarize, IMHO it would be preferable to use wxNumberFormatter as
SC> >much as possible, even if we do what you propose above.
SC>
SC> ok, agreed, but in order for wxNumberFormatter to have any advantage over
SC> wxString::FromDouble we should be able to express whether people really
SC> want the thousands separator,

You can express it on the call site (by specifying Style_None or
Style_WithThousandsSep) and I don't think we need anything else, e.g. I'd
be against having a global default style. Of course, each component using
wxNumberFormatter, e.g. wxGrid, would provide its own methods for
configuring the default floating point numbers display but it shouldn't be
done at this low level.

SC> if not, then it doesn't offer more, and I wouldn't dare to default to
SC> showing it, as this definitely would suddenly behave differently than
SC> before.

Actually, wxString::FromDouble() -- which would be affected by this -- is
not used anywhere at all in wxWidgets sources currently. And I don't think
we can start having problems if we accept (optional) thousand separators in
the places where wxString::ToDouble() is used, e.g. for parsing of the
command line double arguments or values entered in wxGrid cells.

SC> Of course FromDouble and ToDouble could be implemented using
SC> wxNumberFormatter completely - then it would always use the formatter
SC> and still have the calling convenience the wxString methods have ?

Well, then why would we have wxNumberFormatter at all? My initial reason
for introducing it was to keep wxString simple or, at least, to avoid
making it even more complicated and I'd still prefer to just leave its
methods be simple wrappers around the corresponding standard functions.
I.e. for me wxString::ToDouble() should be symmetric with ToLong() and
ToULong() and wrap strtod() in the same way the other functions wrap
strtol() and strtoul(). And wxNumberFormatter should be used for anything
more involved.

Regards,
VZ

[Stefan Csomor]

Hi

>Actually, wxString::FromDouble() -- which would be affected by this -- is
>not used anywhere at all in wxWidgets sources currently. And I don't think
>we can start having problems if we accept (optional) thousand separators
>in
>the places where wxString::ToDouble() is used, e.g. for parsing of the
>command line double arguments or values entered in wxGrid cells.

Ok, I'll replace them with a Style_None for output and be as lenient as
possible for input when I come across them.

>
>SC> Of course FromDouble and ToDouble could be implemented using
>SC> wxNumberFormatter completely - then it would always use the formatter
>SC> and still have the calling convenience the wxString methods have ?
>
> Well, then why would we have wxNumberFormatter at all? My initial reason
>for introducing it was to keep wxString simple or, at least, to avoid
>making it even more complicated and I'd still prefer to just leave its
>methods be simple wrappers around the corresponding standard functions.

ok, I get your point, I was just trying to isolate my fixes at exactly one
point. I'll do the following then: fix the c-runtime bug of OSX at the
level wxString, by simple replacing the decimal point. And reroute the num
formatter also in the FromDouble to use the wxString case as it already
does for the ToDouble case

Thanks,

Stefan