replace REXML with Hpricot

12 views
Skip to first unread message

AD

unread,
Jun 26, 2008, 3:39:36 PM6/26/08
to rubyonra...@googlegroups.com
Hi,

 is there any way to replace REXML with Hpricot for rendering XML? I am  using Jruby so cant use libxml, and need to significantly increase the speed of XML rendering.

Thanks
Adam

Phlip

unread,
Jun 26, 2008, 4:57:07 PM6/26/08
to rubyonra...@googlegroups.com
AD wrote:

What's wrong with Builder::XmlMarkup (iirc)?

Hpricot uses a C module to get fast - like libxml. So if you can't use one in
Jruby...

AD

unread,
Jun 26, 2008, 10:08:16 PM6/26/08
to rubyonra...@googlegroups.com
I think I actually meant XML::Builder, I was under the initial impression this was actually a REXML Library.

At any rate my performance metrics show that 60-70% is done via "Rendering" b/c of some complicated XML generation so I am looking for some optimization mechanisms here. HPricot is faster than Builder and REXML and is supported in JRuby (running it now).

So basically I am looking to replace XML::Builder with something faster , that is not libXML (like Hpricot)

Adam

Frederick Cheung

unread,
Jun 27, 2008, 4:05:41 AM6/27/08
to Ruby on Rails: Talk


On Jun 27, 3:08 am, AD <straightfl...@gmail.com> wrote:
> I think I actually meant XML::Builder, I was under the initial impression
> this was actually a REXML Library.
> At any rate my performance metrics show that 60-70% is done via "Rendering"
> b/c of some complicated XML generation so I am looking for some optimization
> mechanisms here. HPricot is faster than Builder and REXML and is supported
> in JRuby (running it now).
>
> So basically I am looking to replace XML::Builder with something faster ,
> that is not libXML (like Hpricot)
>
There is a jruby version of hpricot. Builder and Rexml are unrelated.
The only bit I've ever found to be slow about builder is that it tries
to be a bit too clever with its escaping and tries to work out all
sorts of stuff that I didn't need. Overriding the _escape method to
just replace &, <, > sped things up for me

Fred

Phlip

unread,
Jun 27, 2008, 5:02:34 AM6/27/08
to rubyonra...@googlegroups.com
AD wrote:

> So basically I am looking to replace XML::Builder with something faster
> , that is not libXML (like Hpricot)

Have you time-tested and profiled your code & tests to see what's fast? it's
usually a surprise...

AD

unread,
Jun 27, 2008, 8:34:44 AM6/27/08
to rubyonra...@googlegroups.com
well I am just noticing when the actions get rendered, its always like (Rendering (78%)) and some of the XML rendering just seems slow in relation to the time for the DB query.

Do you have any profile recommendations besides ruby-prof ?

AD

unread,
Jun 27, 2008, 8:50:55 AM6/27/08
to rubyonra...@googlegroups.com
here is some real world output.. Any feedback is welcome, notice the very high render time:

Rendering channels/index
Completed in 3.38310 (0 reqs/sec) | Rendering: 1.38117 (40%) | DB: 0.48588 (14%) | 200 OK [http://dev1.domain.com/users/1/2008/05/28/index.xml]
 [http://dev1.domain.com/users/1/2008/05/28/index.xml]

Thread ID: 3067917240
Total: 2.920000

 %self     total     self     wait    child    calls  name
  5.82      0.42     0.17     0.00     0.25    18281  Fixnum#xchr
  4.45      0.14     0.13     0.00     0.01       79  Array#select
  4.11      0.56     0.12     0.00     0.44     2288  Array#map
  3.08      0.10     0.09     0.00     0.01       29  ActiveRecord::Associations::ClassMethods#collection_accessor_methods
  2.74      0.11     0.08     0.00     0.03    54266  Kernel#===
  1.71      0.16     0.05     0.00     0.11      248  Hash#each
  1.71      0.08     0.05     0.00     0.03    54327  Hash#[]
  1.37      0.06     0.04     0.00     0.02      288  Module#define_method
  1.37      0.04     0.04     0.00     0.00       15  Mysql::Result#each_hash
  1.37      0.04     0.04     0.00     0.00      108  Array#-
  1.03      0.03     0.03     0.00     0.00    39128  Hash#default
  1.03      0.03     0.03     0.00     0.00      324  Module#constants
  1.03      0.03     0.03     0.00     0.00    56670  Fixnum#==


Thread ID: 3067917860
Total: 0.770000

Frederick Cheung

unread,
Jun 27, 2008, 8:54:23 AM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 13:50, AD wrote:

> here is some real world output.. Any feedback is welcome, notice the
> very high render time:
>

Fixnum.xchr is builder's escaping stuff. See my previous tip for
possible ways out.

Fred

AD

unread,
Jun 27, 2008, 9:09:56 AM6/27/08
to rubyonra...@googlegroups.com
I believe you are referencing this method

 def _escape(text)
      text.to_xs
 end

How do I go about overriding this ?

Frederick Cheung

unread,
Jun 27, 2008, 9:25:01 AM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 14:09, AD wrote:

> I believe you are referencing this method
>
> def _escape(text)
> text.to_xs
> end
>
> How do I go about overriding this ?
>

same way as you override any other method

class Builder::XmlMarkup
def _escape(text)
result = text.dup
result.gsub!("&", "&amp;")
result.gsub!("<", "&lt;")
result.gsub!(">", "&gt;")
end
end

in my case what i did was

xml = Builder::XmlMarkup.new(:target => $stdout, :indent => 2)
xml.instruct!

def xml.output(string)
result = string.dup
result.gsub!("&", "&amp;")
result.gsub!("<", "&lt;")
result.gsub!(">", "&gt;")
self << result
end

and then call output rather than text whenever i wanted to output stuff.
There was a thread a while back on one of the ruby list about the
fastest way to do this - do you do it in one pass fixing up each of
the 3 possible substitutions in one go, or in 3 passes like i did, and
concluded that the above version was the fastest. It might be
different if you had really long strings.

See also http://bogomips.org/fast_xs/ which is a c implementation of
the escaping thing (haven't tried this, but is probably the fastest of
all).


Fred

Frederick Cheung

unread,
Jun 27, 2008, 10:05:48 AM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 14:25, Frederick Cheung wrote:
>
> See also http://bogomips.org/fast_xs/ which is a c implementation of
> the escaping thing (haven't tried this, but is probably the fastest
> of all).
>
to expand on that, just gem install fast_xs. It's faster than what I
had previously

Fred

AD

unread,
Jun 27, 2008, 10:09:48 AM6/27/08
to rubyonra...@googlegroups.com
yea fast_xs looks better, but right now its native C only so no Jruby support.

Is it possible to generated XML without using builder ?  If the XML structure is not too complicated, and I am OK sacrificing some of the validation/substitution, could I just @results.each { |r| } and print some XML ?

Thx
Adam

Frederick Cheung

unread,
Jun 27, 2008, 10:30:18 AM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 15:09, AD wrote:

> yea fast_xs looks better, but right now its native C only so no
> Jruby support.
>
> Is it possible to generated XML without using builder ? If the XML
> structure is not too complicated, and I am OK sacrificing some of
> the validation/substitution, could I just @results.each { |r| } and
> print some XML ?
>

Course you can, it's just text after all! I got a significant speed
boost overwriting _escape so that might be enough for you.

Fred

AD

unread,
Jun 27, 2008, 10:40:05 AM6/27/08
to rubyonra...@googlegroups.com
Did you just drop this in application.rb ?


class Builder::XmlMarkup
  def _escape(text)
    result = text.dup
    result.gsub!("&", "&amp;")
    result.gsub!("<", "&lt;")
    result.gsub!(">", "&gt;")
  end
end

Frederick Cheung

unread,
Jun 27, 2008, 10:54:24 AM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 15:40, AD wrote:

> Did you just drop this in application.rb ?
>
> class Builder::XmlMarkup
> def _escape(text)
> result = text.dup
> result.gsub!("&", "&amp;")
> result.gsub!("<", "&lt;")
> result.gsub!(">", "&gt;")
> end
> end

just in a script i was writing. (but it should be ok in
application.rb), with the exception of one line which i left out :-)

it should be


def _escape(text)
result = text.dup
result.gsub!("&", "&amp;")
result.gsub!("<", "&lt;")
result.gsub!(">", "&gt;")

result
end

jruby is causing you to miss out on the fast_xs fun, but you might be
able to rewrite that method in java, with similar effects.

Fred

AD

unread,
Jun 27, 2008, 12:49:30 PM6/27/08
to rubyonra...@googlegroups.com
Frederick,

 Thanks for the help but i am getting an error.  I dropped this outside the ApplicationController but is throwing an error


class Builder::XmlMarkup
  def _escape(text)
    result = text.dup
    result.gsub!("&", "&amp;")
    result.gsub!("<", "&lt;")
    result.gsub!(">", "&gt;")
    result
  end
end

ActionView::TemplateError (You have a nil object when you didn't expect it!
The error occurred while evaluating nil.gsub) on line #1 of channels/system.xml.builder:
1: xml.instruct!
2: xml.response {
3:   xml.result {
4:     xml.status "Success"

Frederick Cheung

unread,
Jun 27, 2008, 1:10:19 PM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 17:49, AD wrote:

> Frederick,
>
> Thanks for the help but i am getting an error. I dropped this
> outside the ApplicationController but is throwing an error
>
> class Builder::XmlMarkup
> def _escape(text)
> result = text.dup
> result.gsub!("&", "&amp;")
> result.gsub!("<", "&lt;")
> result.gsub!(">", "&gt;")
> result
> end
> end

Weird, works well enough for me. the error you got looks like what you
would get with the version without the result on the last line.

Fred

AD

unread,
Jun 27, 2008, 1:37:56 PM6/27/08
to rubyonra...@googlegroups.com
do you have it in application.rb ?

AD

unread,
Jun 27, 2008, 1:50:33 PM6/27/08
to rubyonra...@googlegroups.com
I also noticed that the _escape function is in xmlbase.rb not xmlmarkup.rb.  Not sure if that is what is breaking here...

Frederick Cheung

unread,
Jun 27, 2008, 2:05:50 PM6/27/08
to rubyonra...@googlegroups.com

On 27 Jun 2008, at 18:50, AD wrote:

> I also noticed that the _escape function is in xmlbase.rb not
> xmlmarkup.rb. Not sure if that is what is breaking here...
>

XmlMarkup inherits from XmlBase so it shouldn't make any difference. I
didn't have it in application.rb but I doubt that would make a
difference.

AD

unread,
Jun 29, 2008, 12:25:55 AM6/29/08
to rubyonra...@googlegroups.com
updating this thread for everyone's sake.  I had to drop it into an initializers file in config/initializers and it worked.

however performance is not much better  36% of the time was just on gsub. 

Thread ID: 3063488800
Total: 2.750000

 %self     total     self     wait    child    calls  name
 36.36      1.00     1.00     0.00     0.00    38508  String#gsub!
 16.36      1.62     0.45     0.00     1.17     1008  Builder::XmlBase#method_missing-5
  3.64      0.10     0.10     0.00     0.00     1282  <Class::File>#expand_path
  2.55      1.15     0.07     0.00     1.08    12836  Builder::XmlBase#_escape
  2.18      0.06     0.06     0.00     0.00       16  Mysql::Result#each_hash
  1.82      2.14     0.05     0.00     2.09     1097  Array#each-1
  1.82      0.05     0.05     0.00     0.00    12836  String#initialize_copy
  1.82      0.07     0.05     0.00     0.02     1015  ActiveRecord::AttributeMethods#read_attribute
  1.45      0.04     0.04     0.00     0.00    31114  Fixnum#==
  1.45      0.04     0.04     0.00     0.00    14171  Symbol#to_s
  1.45      0.04     0.04     0.00     0.00     2159  Hash#initialize_copy
  1.45      0.04     0.04     0.00     0.00     4432  Kernel#instance_variable_set
  1.09      0.20     0.03     0.00     0.17     1008  Builder::XmlBase#method_missing-6
  1.09      0.03     0.03     0.00     0.00     8262  Builder::XmlMarkup#_text
  1.09      0.42     0.03     0.00     0.39     1126  Hash#each
  1.09      2.62     0.03     0.00     2.59      102  Array#each
  1.09      0.08     0.03     0.00     0.05     2154  <Class::ActiveRecord::Base>#instantiate
Reply all
Reply to author
Forward
0 new messages