std::string -> javascript string seems not work correctly using non ascii codepoints

[Sören König]

when using embind to call the following method test, the resulting string is broken (only non ascii codepoints)

std::string test()
{
    return u8"test ö ä ü ende";
}

std::wstring wtest()
{
    return L"test ö ä ü ende";
}

EMSCRIPTEN_BINDINGS(my_module) {
    function("test", &test);
    function("wtest", &wtest);
}

The version using wstring works with wchar_t size 4 byte but not with size 2 byte.(via Compiler-flag -fshort-wchar)


i did a similar experiment using cwrap/  EXPORT_FUNCTION to call

const char* test()
{
    return u8"test ö ä ü ende";
}


which is working fine.

is this a bug?

[Brion Vibber]

If I'm reading the code correctly, embind's marshaling code currently just assumes Latin-1 encoding for std::string, which is incorrect in your case since you're using UTF-8 strings.

(You can find this in embind.js in the emscripten source tree; search for '_embind_register_std_string' and see the 'fromWireType' method that gets defined there.)

In a pinch, you can probably swap that code out, though I'm not sure there's an easy way to do it at runtime. Ideally this would be locale-sensitive or something, but I don't know offhand the encoding-safety properties of std::string...

-- brion

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charles Vaughn]

Might be worth looking into leveraging TextEncoder/TextDecoder for browsers that support it

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Brion Vibber]

On Fri, Apr 1, 2016 at 7:55 PM, Charles Vaughn <cva...@gmail.com> wrote:

Might be worth looking into leveraging TextEncoder/TextDecoder for browsers that support it

I've filed issues in the tracker so we don't lose track:

https://github.com/kripken/emscripten/issues/4221 - lack of support for non-latin1
https://github.com/kripken/emscripten/issues/4222 - support TextEncoder/TextDecoder

-- brion