[wx/msw/winundef.h]: restore the UTF8 support that was mistakenly stripped in 3.3.0 (Issue #25803)
jamarr
jamarr81 created an issue (wxWidgets/wxWidgets#25803)Build System Used
I build wxWidgets and/or my application using:
- vcpkg -> cmake -> msbuild
vcpkg install wxWidgets
Description
The wx/msw/winundef.h header was broken for Windows UTF8 Builds in 53b3b979 (v3.3.0-rc1) by stripping the Ansi/UTF8 (Narrow) declarations; see 042d922e (v3.1.1-rc) for contrast. This header should support both the Wide (UTF16) and Narrow (UTF8) variations of the WinSDK.
Background
Given an application context utilizing pugixml (utf8), sqlite (utf8), luajit (utf8), curl (utf8), ws2 (utf8), winsdk (utf8), I have run into a WinSDK mistranslation issue because the winundef.h header is redefining WinSDK calls like CreateWindow, MessageBox, etc., to their W (Wide) variations, when I need the A (Narrow) version, preventing the application from compiling.
I am stuck using v3.2.8.1 until the UTF8 support can be restored.
Concerns
I have not been able to fully build/test v3.3.1 and I am worried that some other changes in 53b3b979 (v3.3.0-rc1) may have broken other parts of wxWidgets for Windows UTF8 Builds as well:
install.md- seems fine.setup.h- seems fine.htmlhelp.cpp- stripped: HH_POPUPA, HH_ALINKA, HH_FTS_QUERYA, HH_WINTYPEA, HH_NTRACKA?activex.cpp- stripped: GetObjectA?uuid.cpp- stripped: UuidToStringA?version.rc- seems fine.
These changes make me worried that there could be other changesets that also mistakenly stripped support for Windows UTF8 Builds besides this one. Ideally, these translations should be confined/moved to winundef.h for simpler isolation/maintenance.
Solutions
We should start by restoring this file from 042d922e (v3.1.1-rc). In addition, it may be prudent to replace the definition/usage of wxUSE_UNICODE_WINDOWS_H in this file with wxUSE_UTF8_LOCALE_ONLY. And then we should correct any other relevant concerns, if warranted.
Platform and version information
- wxWidgets: 3.3.1
- wxWidgets port: wxMSW
- OS: Windows 11
- Compiler: MSVS 2022
- Non-default compiler options: /utf-8
As an aside, I am using static linking for this project for all dependencies, including winsdk/ucrt and wxwidgets. I'm unsure how dynamic-linking may factor in, but I assume it should not matter in this use case.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Maarten
MaartenBent left a comment (wxWidgets/wxWidgets#25803)Apparently Windows is reusing the ANSI API for UTF-8: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page#-a-vs--w-apis
I suppose the file can be restored with wxUSE_UTF8_LOCALE_ONLY (or just wxUSE_UNICODE_UTF8?). Or a new option? I suppose there are also people that use UTF-8 build with the Wide API.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
vadz left a comment (wxWidgets/wxWidgets#25803)Please explain what exactly is broken. Using UTF-8 with wxMSW still works, we have CI builds using it that continue passing and there is nothing wrong with using UTF-8 with wxWidgets but wide versions of Windows API. If the problem is just that Foo gets redefined as FooA by windows.h, you can fix this by including include/wx/msw/winundef.h.
Irrespectively of this, it might make sense to use Windows UTF-8 CP support with wxUSE_UTF8_LOCALE_ONLY under MSW, but this would be a new feature, not fixing something that "was broken".
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Maarten
MaartenBent left a comment (wxWidgets/wxWidgets#25803)As I understand it, this is not really an UTF-8 problem, but a problem of changing Windows #defines.
I assume they don't define UNICODE. So when using the Foo function, it is defined to FooA. And this is what all the other libraries expect. But then wxWidgets is redefining/inlining Foo as FooW, breaking stuff.
From my link, the FooA ANSI functions apparently support UTF-8 when specifying in the manifest:
<activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
vadz left a comment (wxWidgets/wxWidgets#25803)I think so too, but I'm not sure, so I'd like to establish what precisely "got broken".
Anyhow, my advice about including wx/msw/winundef.h or just including the headers relying on A version of the Windows functions before any wx headers, would be enough to fix the problem if it's really just this, we don't need to change anything in wx itself.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Maarten
MaartenBent left a comment (wxWidgets/wxWidgets#25803)my advice about including wx/msw/winundef.h
This file is exactly what breaks it. Because it changes FooA to FooW. Including wx headers last can fix it, but maybe not if they use win32 API in their own code.
A possible fix that doesn't need wxUSE_UTF8_LOCALE_ONLY/wxUSE_UNICODE_UTF8 is the following. When we define UNICODE in platform.h, add:
#ifndef UNICODE # define UNICODE +# define UNICODE_IS_SET_BY_WXWIDGETS #endif
And then change winundef.h to (with LPCTSTR also correctly redefined):
inline HWND CreateDialog(HINSTANCE hInstance, LPCTSTR pTemplate, HWND hwndParent, DLGPROC pDlgProc) { #if defined(UNICODE) && !defined(UNICODE_IS_SET_BY_WXWIDGETS) return CreateDialogW(hInstance, pTemplate, hwndParent, pDlgProc); #else return CreateDialogA(hInstance, pTemplate, hwndParent, pDlgProc); #endif }
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
vadz left a comment (wxWidgets/wxWidgets#25803)Oops, yes, you're right, sorry, I forgot that, contrary to its name, it didn't just #undef these identifiers but created inline functions replacing them.
Another possibility would be to just stop doing this and replace all Foo calls in wx code with explicit FooW().
I'd still like to know what exactly got broken for the OP, as I/we could still be misunderstanding it. So a minimal example would be really helpful to make sure we're fixing the right problem.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Maarten
MaartenBent left a comment (wxWidgets/wxWidgets#25803)Another possibility would be to just stop doing this and replace all Foo calls in wx code with explicit FooW().
This is not about the win32 functions that wx uses internally (most of these aren't even used). It is used to fix name clashes between the win32 define, and function definitions.
For example, if I remove CreateDialog from winundef.h, building results in the following error in the wxTopLevelWindowMSW::CreateDialog declaration:
include\wx\msw\toplevel.h(139,10): error C2059: syntax error: 'constant'
C2059 is raised when a preprocessor symbol name is re-used as an identifier.
But lets wait for a minimal example and/or confirmation that OP indeed doesn't define UNICODE.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
vadz left a comment (wxWidgets/wxWidgets#25803)This is not about the win32 functions that wx uses internally (most of these aren't even used). It is used to fix name clashes between the win32 define, and function definitions.
Yes, I know, but (just) #undef would take care of this, wouldn't it. I meant that we don't need to replace Foo with something else.
But lets wait for a minimal example and/or confirmation that OP indeed doesn't define
UNICODE.
Indeed, because it's still possible that this is about something different (e.g. I have no idea what does stuff like HH_POPUPA have to do with anything).
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
jamarr
jamarr81 left a comment (wxWidgets/wxWidgets#25803)@MaartenBent, yes, thank you so much for contributing/elaborating. We are not defining UNICODE/_UNICODE in our downstream project, because in a Windows/WinSDK context these have a special meaning that forces the use of its Wide/UTF16 API, and we are specifically only using its Narrow/UTF8 API. In this context, it's a bit of a misnomer, because Unicode can (and is) used even when these are NOT defined; developers should treat these as if they're actually called UNICODE_WIDE/_UNICODE_WIDE. So, you have precisely described the problem:
Windows is reusing the ANSI API for UTF-8... I assume they don't define UNICODE. So when using the Foo function, it is defined to FooA. And this is what all the other libraries expect. But then wxWidgets is redefining/inlining Foo as FooW, breaking stuff... it changes FooA to FooW... [when] they use win32 API in their own code... This is not about the Win3 functions that wx uses internally...
@vadz, in regards to this comment:
... Indeed, because it's still possible that this is about something different (e.g. I have no idea what does stuff like HH_POPUPA have to do with anything)...
I only mentioned those changes because I personally am unsure if stripping the A/Narrow version of these declarations is problematic or not? I'm not using these, but they were changed in the same changeset as winundef.h. I am merely suggesting that someone should perhaps double-check that.
Personally, I would prefer that wxWidgets stop trying to dictate which version of the WinSDK (Wide/Narrow) that it is built with, and instead let the dependent project dictate that; for example, when wxUSE_UTF8_LOCALE_ONLY is defined on Windows, wxWidgets should interpret this as the project explicitly stating that it wants to use the A/Narrow/UTF8 version of the WinSDK.
But lets wait for a minimal example and/or confirmation that OP indeed doesn't define UNICODE.
@MaartenBent, to confirm, we do not define UNICODE/_UNICODE, and even undefine it when building wxWidgets. This has worked from v2.x up to v3.2.8.1, allowing UTF8 builds for Windows. It stopped working in v3.3.x, at least, because of the winundef.h changes, which now redefine the A/Narrow/UTF8 declarations as W/Wide/UTF16 versions, breaking the downstream project code that needs to call some of those functions.
I do not have a minimal example atm, but maybe in a couple of days I can make time to rebuild v3.3.1 with the sample projects and manually remove the UNICODE/_UNICODE declarations in the wx/platform.h header and see what happens; the samples would need to call some WinSDK functions, so depending on the sample, that also might need to be added.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
vadz left a comment (wxWidgets/wxWidgets#25803)Perhaps we could avoid including <windows.h> from public headers completely, but this risks breaking a lot of existing code because I'm sure there are tons of it that actually implicitly rely on <windows.h> getting included by wx headers, so it would have to be opt-in. And we'd also still have to #undef the symbols used in the headers to prevent our headers from being broken by something the application includes before including wx headers.
I think the simplest fix for you is to just use FooA() explicitly. And ideally wxWidgets should use FooW() explicitly too, if only for clarity.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
jamarr
jamarr81 left a comment (wxWidgets/wxWidgets#25803)@vadz Perhaps we could avoid including <windows.h> from public headers completely, but this risks breaking a lot of existing code because I'm sure there are tons of it that actually implicitly rely on <windows.h> getting included by wx headers, so it would have to be opt-in. And we'd also still have to #undef the symbols used in the headers to prevent our headers from being broken by something the application includes before including wx headers.
It would be nice if, down the road, we could all avoid the headaches from the windows headers and their annoying API Macros that everyone hates. But for now, in the meantime, restoring the winundef.h functionality so Windows UTF8 Builds like mine can continue to compile (without custom work-arounds per-project) seems like the best option for myself, and also for the wxWidgets community?
@vadz ... I think the simplest fix for you is to just use FooA() explicitly...
Probably, but that is not how the WinSDK is intended to be used; this would be abnormal in the broader market of all WinSDK projects. It makes more sense for wxWidgets to support the normal/expected WinSDK declarations; both the W/UTF16 and A/UTF8 variations. Lastly, I will quote you from this comment:
@vadz ... Using wxUSE_UNICODE_UTF8 is almost orthogonal to using _UNICODE: the former only defines how wxString stores its contents, while the latter defines whether we use Windows A or W functions. Of course, if you use wxUSE_UNICODE_UTF8 you probably want to use A functions to avoid conversions and vice versa,..
I hope you would still agree with this?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
vadz left a comment (wxWidgets/wxWidgets#25803)It would be nice if, down the road, we could all avoid the headaches from the windows headers and their annoying API Macros that everyone hates. But for now, in the meantime, restoring the
winundef.hfunctionality
We can't restore it because we don't support building using ANSI APIs any more. We need to find some way of solving, or at least avoiding, the problem now.
@vadz ... I think the simplest fix for you is to just use FooA() explicitly...
Probably, but that is not how the WinSDK is intended to be used
Citation needed.
@vadz ... Using wxUSE_UNICODE_UTF8 is almost orthogonal to using _UNICODE: the former only defines how wxString stores its contents, while the latter defines whether we use Windows A or W functions. Of course, if you use wxUSE_UNICODE_UTF8 you probably want to use A functions to avoid conversions and vice versa,..
I hope you would still agree with this?
I still think that these options are orthogonal. We could repurpose wxUSE_UTF8_LOCALE_ONLY for the A/W choice, but we shouldn't use wxUSE_UNICODE_UTF8 itself for it.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
VZ
Closed #25803 as completed via b7705f8.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.