C++0x standard code conversion facets and text in UTF-8 encoding
Ángel José Riesgo
text files. I would like to support UTF-8 encoding, with or without a
BOM, and also UCS-2, since text editors typically allow these options.
So I have used Notepad++ to save a few files with different encodings
and I have been trying to learn how to use the C++0x code-conversion
facets codecvt_utf8, codecvt_utf16, and codecvt_utf8_utf16 in order to
read the files. Now this has proved to be a bit difficult and, in the
end, I have been able to read the contents of the files into
std::wstring objects, but not into UTF-8-encoded std::strings, which
is what I would like.
So, my question is whether there is a way to use these standard code-
conversion facets to read the contents of a text file into UTF-8-
encoded std::strings.
Just to clarify what I've done so far, I have the following test,
which builds and runs correctly in Microsoft Visual C++ 10:
#include <string>
#include <locale>
#include <codecvt>
#include <iostream>
#include <fstream>
const std::wstring kUtf8NoBOMFile = L"../../resources/utf8-no-bom.txt";
const std::wstring kUtf8WithBOMFile = L"../../resources/utf8-with-bom.txt";
const std::wstring kUcs2BigEndianFile = L"../../resources/ucs2-big-endian.txt";
const std::wstring kUcs2LittleEndianFile = L"../../resources/ucs2-little-endian.txt";
void Log(const std::wstring& line)
{
//TO DO: Dump lines to a UTF-8-encoded file.
}
std::locale GetAppropriateLocale(const std::wstring& fileName)
{
std::locale loc;
const std::codecvt_mode mode = std::consume_header;
if(fileName == kUtf8NoBOMFile || fileName == kUtf8WithBOMFile)
{
loc = std::locale(loc, new std::codecvt_utf8<wchar_t, 0x10ffff, mode>);
}
else if(fileName == kUcs2BigEndianFile)
{
loc = std::locale(loc, new std::codecvt_utf16<wchar_t, 0x10ffff, mode>);
}
else if(fileName == kUcs2LittleEndianFile)
{
const std::codecvt_mode little_endian_mode = static_cast<const std::codecvt_mode>(mode|std::little_endian);
loc = std::locale(loc, new std::codecvt_utf16<wchar_t, 0x10ffff, little_endian_mode>);
}
return loc;
}
void CheckFile(const std::wstring& fileName)
{
std::wifstream wifstr(fileName);
if(wifstr)
{
std::locale loc = GetAppropriateLocale(fileName);
wifstr.imbue(loc);
std::wstring nextLine;
while(std::getline(wifstr, nextLine))
{
Log(nextLine);
}
}
}
int main()
{
CheckFile(kUtf8NoBOMFile);
CheckFile(kUtf8WithBOMFile);
CheckFile(kUcs2BigEndianFile);
CheckFile(kUcs2LittleEndianFile);
}
Now what I would like to do is to replace the line "std::wstring
nextLine;" with "std::string nextLine;" and change the signature of
the Log function so that it takes an std::string (which I will assume
is well-formed UTF-8) as its parameter. After doing that, I have tried
to modify the GetAppropriateLocale function above so that the codecvt
templates use char rather than wchar_t, and I have also tried to use
codecvt_utf8_utf16, but to no avail. The little information I have
found online seems to imply that these standard codecvt templates
require a wide-character type as its first Elem parameter, which I
find a bit odd, since I would expect that using an internal UTF-8
std::string representation for reading UCS-2 text files should be as
valid as using an internal UCS-2 representation for reading UTF-8
files.
Actually, I was expecting that the new C++0x standard would enhance
the support for UTF-8 using plain chars and std::strings, but instead
I've found that now there are even more wide character types now! I
suppose I could change my code to use an internal std::u32string
representation for text, but I would rather not go that way and stick
to plain old std::strings.
Any help will be appreciated. Thanks in advance.
Ángel José Riesgo
--
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
Ulrich Eckhardt
> I have been trying to write some code that handles reading text from
> text files. I would like to support UTF-8 encoding, with or without a
> BOM, and also UCS-2, since text editors typically allow these options.
> So I have used Notepad++ to save a few files with different encodings
> and I have been trying to learn how to use the C++0x code-conversion
> facets codecvt_utf8, codecvt_utf16, and codecvt_utf8_utf16 in order to
> read the files. Now this has proved to be a bit difficult and, in the
> end, I have been able to read the contents of the files into
> std::wstring objects, but not into UTF-8-encoded std::strings, which
> is what I would like.
You have internally a character and externally bytes that represent the
same. You can also turn off any transcoding and have the same sequence of
bytes internally as externally. Having internally a sequence of bytes and a
different sequence of bytes externally is simply not supported, so having
UTF-8 internally and UCS2 externally won't work.
> Actually, I was expecting that the new C++0x standard would enhance
> the support for UTF-8 using plain chars and std::strings, but instead
> I've found that now there are even more wide character types now! I
> suppose I could change my code to use an internal std::u32string
> representation for text, but I would rather not go that way and stick
> to plain old std::strings.
I'd try to use std::wstring internally, maybe some of the C++0x types if I
was using C++0x.
Otherwise, you can store anything in a std::string, even null bytes, so you
could even store UTF2 there. Things like operator[] on that string would
become rather meaningless or complicated to use, but you have the same
problem with UTF-8, too.
Uli
--
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
Ángel José Riesgo
> =C1ngel Jos=E9 Riesgo wrote:
> > I have been trying to write some code that handles reading text from
> > text files. I would like to support UTF-8 encoding, with or without a
> > BOM, and also UCS-2, since text editors typically allow these options.
> > So I have used Notepad++ to save a few files with different encodings
> > and I have been trying to learn how to use the C++0x code-conversion
> > facets codecvt_utf8, codecvt_utf16, and codecvt_utf8_utf16 in order to
> > read the files. Now this has proved to be a bit difficult and, in the
> > end, I have been able to read the contents of the files into
> > std::wstring objects, but not into UTF-8-encoded std::strings, which
> > is what I would like.
>
> You have internally a character and externally bytes that represent the
> same. You can also turn off any transcoding and have the same sequence of
a
> different sequence of bytes externally is simply not supported, so having
> UTF-8 internally and UCS2 externally won't work.
Thanks for your reply. So, if I understand you correctly,
codecvt_utf8, codecvt_utf16, and codecvt_utf8_utf16 are only meant to
be used to transcode sequences of bytes into the "character" types
wchar_t, char16_t and char32_t? What I don't really understand is why
using std::wstrings when sizeof(wchar_t) == 2, as in Microsoft Visual C
++ (MSVC), or std::u16strings should be essentially different from
using std::strings. As far as I know, these wide strings don't take
into account UTF-16 surrogate pairs, so just like a std::string can be
regarded as "a sequence of bytes", a std::wstring in MSVC is simply a
sequence of 16-bit units, not a real sequence of characters (in the
sense of Unicode code points).
It looks as if transcoding between external sequences of bytes and
internal sequences of 16-bit units is supported, whereas transcoding
between external sequences of bytes and internal sequences of bytes is
not, which I find a bit inconsistent. But I suppose there must be some
rationale for this that I don't understand.
> > Actually, I was expecting that the new C++0x standard would enhance
> > the support for UTF-8 using plain chars and std::strings, but instead
> > I've found that now there are even more wide character types now! I
> > suppose I could change my code to use an internal std::u32string
> > representation for text, but I would rather not go that way and stick
> > to plain old std::strings.
>
> I'd try to use std::wstring internally, maybe some of the C++0x types if =
I
> was using C++0x.
I have used std::wstrings in the past. However, I now feel that if I
want my code to be as cross-platform as possible, things get simpler
if I use std::strings and I expect (or enforce) that such strings are
UTF-8-encoded. The advantage is that I can serialise any text without
worrying about the encoding. Because the size of wchar_t varies across
platforms, if I use wchar_t's internally I have to carry out far more
conversions (every time I serialise data) than if I stick to
std::strings. I still have to transcode into UTF-16 std::wstrings when
I have to interact with the Windows file system (to open a file
stream, for example) or with the Win32 API (to display a MessageBox,
for example), but I feel that having std::strings in the platform-
independent and serialisation code is a better compromise for me than
having std::wstrings all over the place and worrying about transcoding
every time I serialise or deserialise. Regarding deserialisation, I
also expect the text I read from files to be UTF-8-encoded in general.
The reason I've posted this question is that I have some configuration
files that are plain text files. Now a user can potentially edit such
text files by hand using, say, Notepad on Windows. And such manually
edited files can end up having different encodings (especially if the
text has characters outside the local code page, where Notepad
recommends you save them as either UTF-8 or UTF-16). That's why I am
trying to have a solution that allows me to open and read text files
in a way that can sort out the various BOMs and encoding options
(although I may end up using XML for all my configuration files and
read those files through an XML library, but that's irrelevant to my
question here).
> Otherwise, you can store anything in a std::string, even null bytes, so y=
ou
> could even store UTF2 there. Things like operator[] on that string would
> become rather meaningless or complicated to use, but you have the same
> problem with UTF-8, too.
>
You're right that if I use std::strings to store UTF-8 text I am
effectively using the standard string class as little more than a
glorified vector<char>. But I feel that such an approach is perfectly
valid for me. I can't think of any realistic scenario where I will
need to identify code point boundaries (I don't intend to write any
word wrap algorithm, for example), and if I want to know the size of a
string, I only need to know its size in bytes for serialisation/
allocation purposes. I can't think of any real-life situation where I
need to know how many Unicode code points make up a string (other than
implementing a simplistic, probably useless, "letter count" option in
a word processor). As for the issue of comparison and Unicode
normalisation, if I read my data from disk in a consistent way, I
can't think of any situation where I will get an NFC-normalised string
as the left-hand-side of a comparison against an NFD-normalised string
on the right-hand-side. I don't know if there may be any subtler point
I'm missing, but I think all the situations where code point
boundaries and Unicode normalisation are relevant are either contrived
artificial examples or irrelevant to my needs. Based on these
considerations, I tend to think that sticking to UTF-8 and
std::strings internally makes sense.
=C1ngel Jos=E9 Riesgo
--