Mojo::Log and UTF-8

72 views
Skip to first unread message

Alex Povolotsky

unread,
Oct 6, 2018, 6:35:48 AM10/6/18
to Mojolicious
Hello

How do I send UTF-8 constants to Mojo::Log? It forcefully encodes everything so UTF-8 goes with double encoding and unreadable. Trying to decode UTF-8 cat 

$log->info(decode('UTF-8', 'тест лога'));

results in

Use of uninitialized value $_[0] in join or string at /usr/local/lib/perl5/site_perl/Mojo/Log.pm line 55.

Looks like something is wrong...

Alex Povolotsky

unread,
Oct 6, 2018, 10:15:06 AM10/6/18
to Mojolicious
Quite interesting. The simple script

=== cut mojolog ===

#!/usr/bin/env perl
use strict;
use utf8;
use Mojo::Log;
my $log = new Mojo::Log;
print "Просто принт\n";
$log->info('тест лога');
=== cut mojolog === 

yields fine results on all my servers but one.

On it, I'm getting

% perl mojolog
Просто принт
[Sat Oct  6 15:22:43 2018] [info] Ñ�еÑ�Ñ� лога 

The second line is clearly wrong and seems to be badly encoded. Same version of OS, Perl, Mojolicious, same set of environment variables. What else could influence unicode processing?

суббота, 6 октября 2018 г., 13:35:48 UTC+3 пользователь Alex Povolotsky написал:

Dan Book

unread,
Oct 6, 2018, 1:30:12 PM10/6/18
to mojol...@googlegroups.com
The script is not quite correct. I suspect if you add "use warnings" you will see a "Wide character in print" warning, because you are not encoding your unicode characters to UTF-8 before printing them. However, on the server where you are seeing the double encoding, perhaps you have a PERL5OPT environment variable which is setting something like "-CSAD" and thus encoding all output handles by default (this should not be globally set, because it can mess with modules that don't expect it like this).

-Dan

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

Alex Povolotsky

unread,
Oct 10, 2018, 11:53:27 AM10/10/18
to Mojolicious
So I should not binmode STDERR, ':utf8' if I'm using Mojo::Log and always explicitly encode() data?

суббота, 6 октября 2018 г., 20:30:12 UTC+3 пользователь Dan Book написал:

Dan Book

unread,
Oct 10, 2018, 12:08:03 PM10/10/18
to mojol...@googlegroups.com
Mojo::Log already encodes data by default. Anything you send to stderr outside of that should be encoded if necessary.

-Dan

kon...@gmail.com

unread,
Oct 31, 2020, 12:04:27 PM10/31/20
to Mojolicious
Hi.
Ok, data is encoded from UTF-8 into bytes when I log a message,
but when this data is printed to STDERR
How make it decoded from bytes to UTF-8 back?
my terminal supports UTF-8 and I want see nice messages instead of ХабÐ

Sebastian Riedel

unread,
Oct 31, 2020, 12:16:55 PM10/31/20
to Mojolicious
Ok, data is encoded from UTF-8 into bytes when I log a message,
but when this data is printed to STDERR
How make it decoded from bytes to UTF-8 back?
my terminal supports UTF-8 and I want see nice messages instead of ХабÐ

UTF-8 *is* bytes. If you are encoding already encoded UTF-8 then you are double encoding.

--
sebastian 
Message has been deleted

kon...@gmail.com

unread,
Oct 31, 2020, 1:54:45 PM10/31/20
to Mojolicious
I found a problem. Some module did:

binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';

and, probably, when STDERR was redirected to STDOUT by Mojo::Log then double encoding occur.

Is it worth to check that STDERR and STDOUT have not same layers before redirecting?

Dan Book

unread,
Oct 31, 2020, 1:57:22 PM10/31/20
to mojol...@googlegroups.com
Any layer on standard handles will cause double encoding when a module like Mojo::Log encodes to bytes before printing. These layers are global unfortunately.

-Dan

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages