Mojo::Log and UTF-8

[Alex Povolotsky]

Hello

How do I send UTF-8 constants to Mojo::Log? It forcefully encodes everything so UTF-8 goes with double encoding and unreadable. Trying to decode UTF-8 cat 

$log->info(decode('UTF-8', 'тест лога'));

results in

Use of uninitialized value $_[0] in join or string at /usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.

Looks like something is wrong...

[Alex Povolotsky]

Quite interesting. The simple script

=== cut mojolog ===

#!/usr/bin/env perl
use strict;
use utf8;
use Mojo::Log;
my $log = new Mojo::Log;
print "Просто принт\n";
$log->info('тест лога');
=== cut mojolog === 

yields fine results on all my servers but one.

On it, I'm getting

% perl mojolog
Просто принт
[Sat Oct  6 15:22:43 2018] [info] Ñ�еÑ�Ñ� лога 

The second line is clearly wrong and seems to be badly encoded. Same version of OS, Perl, Mojolicious, same set of environment variables. What else could influence unicode processing?

суббота, 6 октября 2018 г., 13:35:48 UTC+3 пользователь Alex Povolotsky написал:

[Dan Book]

The script is not quite correct. I suspect if you add "use warnings" you will see a "Wide character in print" warning, because you are not encoding your unicode characters to UTF-8 before printing them. However, on the server where you are seeing the double encoding, perhaps you have a PERL5OPT environment variable which is setting something like "-CSAD" and thus encoding all output handles by default (this should not be globally set, because it can mess with modules that don't expect it like this).

-Dan

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Alex Povolotsky]

So I should not binmode STDERR, ':utf8' if I'm using Mojo::Log and always explicitly encode() data?

суббота, 6 октября 2018 г., 20:30:12 UTC+3 пользователь Dan Book написал:

[Dan Book]

Mojo::Log already encodes data by default. Anything you send to stderr outside of that should be encoded if necessary.

-Dan

[kon...@gmail.com]

Hi.

Ok, data is encoded from UTF-8 into bytes when I log a message,

but when this data is printed to STDERR

How make it decoded from bytes to UTF-8 back?

my terminal supports UTF-8 and I want see nice messages instead of ХабÐ

[Sebastian Riedel]

Ok, data is encoded from UTF-8 into bytes when I log a message,

but when this data is printed to STDERR

How make it decoded from bytes to UTF-8 back?

my terminal supports UTF-8 and I want see nice messages instead of ХабÐ

UTF-8 *is* bytes. If you are encoding already encoded UTF-8 then you are double encoding.

--

sebastian 

[kon...@gmail.com]

I found a problem. Some module did:

binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';

and, probably, when STDERR was redirected to STDOUT by Mojo::Log then double encoding occur.

Is it worth to check that STDERR and STDOUT have not same layers before redirecting?

[Dan Book]

Any layer on standard handles will cause double encoding when a module like Mojo::Log encodes to bytes before printing. These layers are global unfortunately.

-Dan

--

You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/f276d64e-f215-47d8-af7b-893d9c200ef4n%40googlegroups.com.