[libxml-devel] Memory leak with 0.9.6 and 0.9.7

[Joseph Palermo]

We are using some code that utilizes libxml and creates XML
documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the
more we use this code, the higher our memory usage gets.

Seems likely that a memory leak was introduced in 0.9.6. We are
seeing this same behavior on OS X and Gentoo.
_______________________________________________
libxml-devel mailing list
libxml...@rubyforge.org
http://rubyforge.org/mailman/listinfo/libxml-devel

[Charlie Savage]

Hi Joseph,

> We are using some code that utilizes libxml and creates XML
> documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the
> more we use this code, the higher our memory usage gets.
>
> Seems likely that a memory leak was introduced in 0.9.6. We are
> seeing this same behavior on OS X and Gentoo.

Can you be more specific? Do you know what code is causing the problem?
Can can put together a test case? If you can identify the code that
is causing the issue, then it should be easy enough to fix.

Charlie

[Joseph Palermo]

> smime.p7s
> 4KViewDownload
>

I'll see if I can put something together, we know it's in the
to_solr_doc from acts_as_solr somewhere.

[Joseph Palermo]

On Jan 7, 1:38 pm, Joseph Palermo <jpale...@pivotallabs.com> wrote:
> On Jan 7, 1:08 pm, Charlie Savage <c...@savagexi.com> wrote:
>
>
>
> > Hi Joseph,
>
> > > We are using some code that utilizes libxml and creates XML
> > > documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the
> > > more we use this code, the higher our memory usage gets.
>
> > > Seems likely that a memory leak was introduced in 0.9.6. We are
> > > seeing this same behavior on OS X and Gentoo.
>
> > Can you be more specific? Do you know what code is causing the problem?
> > Can can put together a test case? If you can identify the code that
> > is causing the issue, then it should be easy enough to fix.
>
> > Charlie
>
> > smime.p7s
> > 4KViewDownload
>
> I'll see if I can put something together, we know it's in the
> to_solr_doc from acts_as_solr somewhere.

Looks like the problem is in to_s on Node

10000.times do
node = XML::Node.new 'node'
node.to_s
end

If I run that on 0.9.5 my memory usage is fine, if I run it on 0.9.6
it balloons.

[Charlie Savage]

Hi Joseph,

>> I'll see if I can put something together, we know it's in the
>> to_solr_doc from acts_as_solr somewhere.
>
> Looks like the problem is in to_s on Node
>
> 10000.times do
> node = XML::Node.new 'node'
> node.to_s
> end

Nice work - that's a great test case.

So if I I run this code:

10000000.times do

node = XML::Node.new 'node'

#node.to_s
end

I see no issues.

Adding the .to_s does seem to cause a problem. Are you actually doing
so many to_s calls in your code?

Anyway, will have to look and see if there is anything obvious in the
part of the bindings.

Charlie

[Charlie Savage]

>> Looks like the problem is in to_s on Node
>>
>> 10000.times do
>> node = XML::Node.new 'node'
>> node.to_s
>> end
>
>

> Anyway, will have to look and see if there is anything obvious in the
> part of the bindings.

And fixed in trunk. It was caused by the addition of encoding support
in the last couple of releases. The new code forgot to release the
internal libxml output buffer.

Thanks for the heads-up on this.

Charlie

[Joseph Palermo]

> smime.p7s
> 4KViewDownload

We had a batch job where we had to update ~300,000 records which
caused a solr_save for each one which does the to_s. We don't have a
lot of extra memory on our hosting provider so it showed up pretty
quick.

Thanks for being so responsive.

[Aaron Patterson]

2009/1/7 Charlie Savage <cf...@savagexi.com>:

Looks like the xmlOutputBufferPtr never gets freed.

This might do the trick:

Index: ext/libxml/ruby_xml_node.c
===================================================================
--- ext/libxml/ruby_xml_node.c (revision 693)
+++ ext/libxml/ruby_xml_node.c (working copy)
@@ -570,10 +570,16 @@
xmlNodeDumpOutput(output, xnode->doc, xnode, level, indent, encoding);
xmlOutputBufferFlush(output);

+ VALUE string = Qnil;
+
if (output->conv)
- return rb_str_new2((const char*) output->conv->content);
+ string = rb_str_new2((const char*) output->conv->content);
else
- return rb_str_new2((const char*) output->buffer->content);
+ string = rb_str_new2((const char*) output->buffer->content);
+
+ xmlOutputBufferClose(output);
+
+ return string;
}

--
Aaron Patterson
http://tenderlovemaking.com/

[Charlie Savage]

Hey Aaron,

> Looks like the xmlOutputBufferPtr never gets freed.

Yup, exactly right. I posted a fix to trunk and sent another email to
the list, but somehow it never made it through.

> Index: ext/libxml/ruby_xml_node.c
> ===================================================================
> --- ext/libxml/ruby_xml_node.c (revision 693)
> +++ ext/libxml/ruby_xml_node.c (working copy)
> @@ -570,10 +570,16 @@
> xmlNodeDumpOutput(output, xnode->doc, xnode, level, indent, encoding);
> xmlOutputBufferFlush(output);
>
> + VALUE string = Qnil;
> +
> if (output->conv)
> - return rb_str_new2((const char*) output->conv->content);
> + string = rb_str_new2((const char*) output->conv->content);
> else
> - return rb_str_new2((const char*) output->buffer->content);
> + string = rb_str_new2((const char*) output->buffer->content);
> +
> + xmlOutputBufferClose(output);
> +
> + return string;
> }
>

Yup, trunk updated to do exactly this. Thanks for the patch.

Charlie