Re: [nokogiri-talk] utf-8 string mojibake was happened in using "node << string". nokogiri 1.5.5

Mike Dalessio

unread,

Mar 21, 2013, 8:36:13 AM3/21/13

to nokogiri-talk

Hello!

Thank you for asking this question. Please help me reproduce the issue?

Here is my code:

```

#!/usr/bin/env ruby

# -*- coding: utf-8 -*-

require 'nokogiri'

puts Nokogiri::VERSION_INFO

# => # {"warnings"=>[], "nokogiri"=>"1.5.7", "ruby"=>{"version"=>"1.9.3", "platform"=>"x86_64-linux", "description"=>"ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]", "engine"=>"ruby"}, "libxml"=>{"binding"=>"extension", "compiled"=>"2.8.0", "loaded"=>"2.8.0"}}

docstring = "<foo><bar>鋸</bar><bazz /></foo>"

doc = Nokogiri::XML(docstring, nil, "UTF-8")

puts doc.encoding

# => "UTF-8"

string = "重田一聖"

puts string.encoding

# => "UTF-8"

doc.at("bazz") << string

puts doc.to_xml

# =>

# <?xml version="1.0" encoding="UTF-8"?>

# <foo>

# <bar>鋸</bar>

# <bazz>重田一聖</bazz>

# </foo>

```

---

mike dalessio / @flavorjones

On Thu, Mar 21, 2013 at 1:28 AM, 重田一聖 <shi...@ga.kanamei.com> wrote:

when i use "node << string"(document.encoding = utf-8 and string.encoding = utf-8) on LANG=c machine. so string was converted to us-ascii forcibly and cause mojibake.

it is not happen when i use "node << doc.create_text_node(string)".

is nokogiri influenced by LANG(env) (or other something)?

this is not happen on my pc.(mac osx)

sorry my bad english.
Kazumasa Shigeta.

--
You received this message because you are subscribed to the Google Groups "nokogiri-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nokogiri-tal...@googlegroups.com.
To post to this group, send email to nokogi...@googlegroups.com.
Visit this group at http://groups.google.com/group/nokogiri-talk?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

重田一聖

unread,

Mar 26, 2013, 1:33:27 AM3/26/13

to nokogi...@googlegroups.com

thank you for your reply.

this issue happen only on my client's server machine.

so i can't reproduce this issue instantly.

my client's server machine is may be "Ubuntu 10.04.4 LTS"(i have only this information now) .

i' ll try negotiate with my client about trying run your cord on target machine .

ps. i use to_html method when this issue was happened.

2013年3月21日木曜日 21時36分13秒 UTC+9 Mike Dalessio:

重田一聖

unread,

May 17, 2013, 12:33:00 AM5/17/13

to nokogi...@googlegroups.com

Why mojibake is happen on centos ?

it seems to happen only when i use Nokogiri::HTML .

I Don't use it is better?

Kazumasa Shigeta.

# -*- coding: utf-8 -*-

require 'nokogiri'

puts Nokogiri::VERSION_INFO

html = "<!DOCTYPE html><html><head></head><body></body></html>"

tag = "<span>あいうえお</span>"

doc = Nokogiri::XML(html, nil, "UTF-8")

body = doc.at("body")

fragment = doc.fragment(tag)

puts fragment.to_html

body << fragment

fragment = doc.fragment(nil)

fragment << tag

puts fragment.to_html

body << fragment

fragment = doc.fragment(nil)

fragment.add_child tag

puts fragment.to_html

body << fragment

body << tag

body.add_child tag

puts

puts doc.to_html

puts "-" * 80

#----------------------------------------------------------

doc = Nokogiri::HTML(html, nil, "UTF-8")

body = doc.at("body")

fragment = doc.fragment(tag)

puts fragment.to_html

body << fragment

fragment = doc.fragment(nil)

fragment << tag

puts fragment.to_html

body << fragment

fragment = doc.fragment(nil)

fragment.add_child tag

puts fragment.to_html

body << fragment

body << tag

body.add_child tag

puts

puts doc.to_html

=========================================

result on centos

{"warnings"=>[], "nokogiri"=>"1.5.6", "ruby"=>{"version"=>"1.9.3", "platform"=>"i686-linux", "description"=>"ruby 1.9.3p194 (2012-04-20 revision 35410) [i686-linux]", "engine"=>"ruby"}, "libxml"=>{"binding"=>"extension", "compiled"=>"2.7.6", "loaded"=>"2.7.6"}}
<span>あいうえお</span>
<span>あいうえお</span>
<span>あいうえお</span>

<!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head>
<body>
<span>あいうえお</span><span>あいうえお</span><span>あいうえお</span><span>あいうえお</span><span>あいうえお</span>
</body>
</html>
--------------------------------------------------------------------------------
<span>ã ‚ã „ã †ã ˆã Š</span>
<span>ã ‚ã „ã †ã ˆã Š</span>
<span>ã ‚ã „ã †ã ˆã Š</span>

<!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head>
<body>
<span>ã ‚ã „ã †ã ˆã Š</span><span>ã ‚ã „ã †ã ˆã Š</span><span>ã ‚ã „ã †ã ˆã Š</span><span>ã ‚ã „ã †ã ˆã Š</span><span>ã ‚ã „ã †ã ˆã Š</span>
</body>
</html>

=======================

result on osx (no mojibake)

{"warnings"=>[], "nokogiri"=>"1.5.5", "ruby"=>{"version"=>"1.9.3", "platform"=>"x86_64-darwin12.2.0", "description"=>"ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12.2.0]", "engine"=>"ruby"}, "libxml"=>{"binding"=>"extension", "compiled"=>"2.7.8", "loaded"=>"2.7.8"}}
<span>あいうえお</span>
<span>あいうえお</span>
<span>あいうえお</span>

<!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head>
<body>
<span>あいうえお</span><span>あいうえお</span><span>あいうえお</span><span>あいうえお</span><span>あいうえお</span>
</body>
</html>
--------------------------------------------------------------------------------
<span>あいうえお</span>
<span>あいうえお</span>
<span>あいうえお</span>

<!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head>
<body>
<span>あいうえお</span><span>あいうえお</span><span>あいうえお</span><span>あいうえお</span><span>あいうえお</span>
</body>
</html>

Eito Katagiri

unread,

May 17, 2013, 12:46:33 AM5/17/13

to nokogi...@googlegroups.com

Hi,

On Fri, May 17, 2013 at 1:33 PM, 重田一聖 <shi...@ga.kanamei.com> wrote:
> Why mojibake is happen on centos ?
> it seems to happen only when i use Nokogiri::HTML .

You should upgrade libxml2 to 2.7.7 or above.

<snip>

> {"warnings"=>[], "nokogiri"=>"1.5.6", "ruby"=>{"version"=>"1.9.3",
> "platform"=>"i686-linux", "description"=>"ruby 1.9.3p194 (2012-04-20
> revision 35410) [i686-linux]", "engine"=>"ruby"},
> "libxml"=>{"binding"=>"extension", "compiled"=>"2.7.6", "loaded"=>"2.7.6"}}

<snip>

HTH,

:eito

重田一聖

unread,

May 17, 2013, 4:39:51 AM5/17/13

to nokogi...@googlegroups.com

thank you very much!

hmmm...if this centos is my machine i can do so . but it is not mine ..... so i may not do so.

i repeat to say thank you very much .

Kazumasa Shigeta.

Mike Dalessio

unread,

May 21, 2013, 8:48:56 PM5/21/13

to nokogiri-talk

You should try nokogiri 1.6.0.rc1 which packages libxml 2.8.0.

--

重田一聖

unread,

May 23, 2013, 11:51:39 PM5/23/13

to nokogi...@googlegroups.com

SUBARASHII!!

it is surely fixed "mojibake"!!

thank you very match!

Mike Dalessio

unread,

May 24, 2013, 1:38:57 PM5/24/13

to nokogiri-talk

You are welcome! Thank you for using Nokogiri!

Reply all

Reply to author

Forward