Serious YAML bug
Frantisek Fuka
I am using YAML to store around 100KB of strings (just simple arrays
and hashes) and I found out the "unpacking" of YAML data stops after
4096 (or so) characters of yaml file (without raising any errors) - the
parsed data just ends in the middle of the string. And it has nothing
to do with disk operations, it happens even when I read the whole yaml
file into string and then use "YAML.parse(longstring)" call. Maybe it
has something to do with the fact that my strings contains many
non-Ascii characters?
Here is the file (zipped) that gives me this problem:
http://fuxoft.cz/test.yaml.zip
I am using the latest Ruby included in Dapper Drake Linux distro and
YAML::Slyck version is "0.60".
James H.
other day and found that I needed to upgrade -- I believe Dapper comes
with 1.8.2 which appears to be somewhat problematic. The process for
apt-getting Ruby 1.8.4 is a little tricky, and long enough that I can't
remember it off of the top of my head.
Can you post your code?
James H
Frantisek Fuka
The code is rather simple:
str=File.open(fname).read
puts str #prints 100KB of data
data=YAML.parse(str)
puts data.emit #prints 4KB of data
However, using "data = YAML.load(File.open(fname))" (which I used
originally) produces the same error.
Robert Klemme
> Version is: ruby 1.8.4 (2005-12-24) [i486-linux]
>
> The code is rather simple:
>
> str=File.open(fname).read
Note that you do not close the file descriptor properly which may cause
problems if you access that file later on in the same process. Better
do
str = File.read fname
> puts str #prints 100KB of data
> data=YAML.parse(str)
You probably used the wrong method:
irb(main):030:0> YAML.parse( "foo".to_yaml )
=> #<YAML::Syck::Scalar:0x488b388>
irb(main):031:0> YAML.load( "foo".to_yaml )
=> "foo"
> puts data.emit #prints 4KB of data
>
> However, using "data = YAML.load(File.open(fname))" (which I used
> originally) produces the same error.
Again, here you do not close the file handle properly. When reading
from a file you can also use
data = YAML.load_file fname
Cheers
robert
ts
F> Version is: ruby 1.8.4 (2005-12-24) [i486-linux]
It's fixed in cvs
svg% ./ruby -v ~/b.rb
ruby 1.8.4 (2005-12-24) [i686-linux]
:subtitles
"WARNER BROS. uv\303\241d\303\255"
"Ud\304\233lala jsem chybu. V\303\255m, \305\276"
svg%
svg% ./ruby -v ~/b.rb
ruby 1.8.4 (2006-06-02) [i686-linux]
:subtitles
"WARNER BROS. uv\303\241d\303\255"
""
:saved_at
Fri May 26 19:25:06 CEST 2006
:custom_splits
{"Po \304\215lov\304\233ku, kter\303\275 mi p\305\231ipomn\304\233l p\303\241t\303\275 listopad."=>"Po \304\215lov\304\233ku, kter\303\275 mi\np\305\231ipomn\304\233l p\303\241t\303\275 listopad.", "A mn\304\233 se nest\303\275sk\303\241 po ideji, ale po \304\215lov\304\233ku."=>"A mn\304\233 se nest\303\275sk\303\241\npo ideji, ale po \304\215lov\304\233ku."}
svg%
--
Guy Decoux
Ross Bamford
> Frantisek Fuka wrote:
>> Version is: ruby 1.8.4 (2005-12-24) [i486-linux]
>> The code is rather simple:
>> str=File.open(fname).read
>
> Again, here you do not close the file handle properly. When reading
> from a file you can also use
>
> data = YAML.load_file fname
>
Hmm, the behaviour I'm seeing here would suggest this to be a bug. With
load_file, sometimes it works, sometimes it doesn't. Loading the file in
Ruby and passing it in always fails.
$ ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
$ irb -ryaml
YAML::Syck::VERSION
# => "0.60"
YAML::load(File.read('test.yaml'))
ArgumentError: syntax error on line 50, col 57: `C3\xBD, Denisi."
- ""
- "Z\xC3\xA1\xC5\x99n\xC3\xB'
from /usr/local/lib/ruby/1.8/yaml.rb:133:in `load'
from /usr/local/lib/ruby/1.8/yaml.rb:133:in `load'
from (irb):3
YAML::load_file('test.yaml')
ArgumentError: syntax error on line 50, col 57: `C3\xBD, Denisi."
- ""
- "Z\xC3\xA1\xC5\x99n\xC3\xB'
from /usr/local/lib/ruby/1.8/yaml.rb:133:in `load'
from /usr/local/lib/ruby/1.8/yaml.rb:133:in `load'
from /usr/local/lib/ruby/1.8/yaml.rb:144:in `load_file'
from /usr/local/lib/ruby/1.8/yaml.rb:143:in `load_file'
from (irb):5
YAML::load_file('test.yaml')
# => {:subtitles=>[ ... 49 elements ... ]}
YAML::load_file('test.yaml')
ArgumentError: syntax error on line 50, col 57: `C3\xBD, Denisi."
- ""
- "Z\xC3\xA1\xC5\x99n\xC3\xB'
from /usr/local/lib/ruby/1.8/yaml.rb:133:in `load'
from /usr/local/lib/ruby/1.8/yaml.rb:133:in `load'
from /usr/local/lib/ruby/1.8/yaml.rb:144:in `load_file'
from /usr/local/lib/ruby/1.8/yaml.rb:143:in `load_file'
from (irb):7
YAML::load_file('test.yaml')
# => {:subtitles=>[ ... 49 elements ... ]}
YAML::load_file('test.yaml')
# => {:subtitles=>[ ... 49 elements ... ]}
YAML::load_file('test.yaml')
# => {:subtitles=>[ ... 49 elements ... ]}
--
Ross Bamford - ro...@roscopeco.remove.co.uk
Frantisek Fuka
> It's fixed in cvs
Glad to hear that. How exactly does the updating/packaging work? When
can I expect this to be fixed in Ubuntu? Or can I use some sort of
workaround meanwhile? I plan to use this code on several Ubuntu
machines.
ts
F> Glad to hear that. How exactly does the updating/packaging work? When
F> can I expect this to be fixed in Ubuntu? Or can I use some sort of
F> workaround meanwhile? I plan to use this code on several Ubuntu
F> machines.
Well, you can install it from cvs :
cvs -d :pserver:anon...@cvs.ruby-lang.org:/src login
Just press RETURN when it ask for a password, then
cvs -z4 -d :pserver:anon...@cvs.ruby-lang.org:/src co -r ruby_1_8 ruby
(without -r ruby_1_8, it will retrieve ruby 1.9)
cd ruby
autoconf
./configure
make
sudo make install
it will be installed in /usr/local/bin, or you can run
./configure --prefix=some_dir
if you want to put it in some_dir
--
Guy Decoux
Frantisek Fuka
don't have development environment on them (e.g. cannot compile) by
people who don't have admin rights there. I guess I'll have to wait
until it's automatically updated by Ubuntu. In the meantime - is there
some way I can get around this bug in my Ruby code? Maybe redefining
some method in Yaml??
ts
F> Thanks. The thing is, the code has to be used on several machines that
F> don't have development environment on them (e.g. cannot compile) by
F> people who don't have admin rights there. I guess I'll have to wait
F> until it's automatically updated by Ubuntu. In the meantime - is there
F> some way I can get around this bug in my Ruby code? Maybe redefining
F> some method in Yaml??
Well, YAML::load_file call a C method. Perhaps the problem is in syck (the
C extension) I've not looked at it.
--
Guy Decoux
Francis Hwang
my experience, stable enough to hold large amounts of data, or data
that changes frequently. And _why's page on Syck (
http://whytheluckystiff.net/syck/ ) points out that Unicode support is
also not very good yet.
If you need to stick with YAML, I'd recommend getting Ruby from CVS
head; it's supposed to contain a newer version of Syck. If you can get
away with it, though, I'd suggest using Marshal instead. It's less cool
than YAML, but pretty solid otherwise.
Francis Hwang
http://fhwang.net/
wolfram
already, then you should simply be able to upgrade to the latest
packages with the Synaptics package manager.
I installed the Dapper Ruby packages on Breezy, because Breezy's Ruby
is 1.8.3 which was not compatible with Rails :-( To do that, add the
Dapper repository channels, update the package list and install only
the Ruby packages, then de-activate the Dapper channels again.
Wolfram
Frantisek Fuka
> Ubuntu Dapper comes with Ruby 1.8.4. If you've installed Dapper
> already, then you should simply be able to upgrade to the latest
> packages with the Synaptics package manager.
I'm not sure if I understand you. I have Dapper Drake and I update
daily. I was asking when can I expect this fix to appear in Dapper
repositories and will get to all "dumb users". I have no idea how often
the packages are updated but I'm afraid I don't remember my ruby
packages updating very often.