http://segment7.net/projects/ruby/inline_optimization.html
--
Eric Hodel - drb...@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
Thanks so much! Writing up and sharing stuff like this is what makes
the net.mind smarter.
It confirmed two things for me:
1) RubyInline is a great argument for convincing naysayers that Ruby is
a good idea.
2) I'm really glad I don't program in C anymore. ;)
Charlie
Francis Cianfrocca wrote:
> On 9/1/06, Eric Hodel <drb...@segment7.net> wrote:
>>
>> I wrote an article on using RubyInline for optimization where I take
>> png.rb, sprinkle in a little profiling and a little C and make it go
>> over 100 times faster.
>
>
> I liked this so much I tried it out on Net::LDAP and immediately hit a bug
> in ext/ruby_prof.c, a missing case in figure_singleton_name. Do you care or
> shall I just tell Charlie Savage about it?
>
> I wrote an article on using RubyInline for optimization where I take
> png.rb, sprinkle in a little profiling and a little C and make it go
> over 100 times faster.
Nice article, but in this case it is possible to get almost the same
speedup in pure Ruby:
Base version:
def to_blob
blob = []
blob << [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
blob << PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
blob << PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
blob << PNG.chunk('IEND', '')
blob.join
end
$ time ruby -Ilib profile.rb
real 0m15.504s
user 0m15.119s
sys 0m0.309s
Avoiding flatten (and using a literal for the signature):
def to_blob
blob = []
blob << "\211PNG\r\n\032\n" # PNG signature
blob << PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| "\0" < row.map { |p| p.values.pack("C*")
}.join }
blob << PNG.chunk('IDAT', Zlib::Deflate.deflate(data.join, 9))
blob << PNG.chunk('IEND', '')
blob.join
end
$ time ruby -Ilib profile.rb
real 0m10.190s
user 0m10.081s
sys 0m0.043s
Using String#% instead of Array#pack:
format_str = "%c%c%c%c"
data = @data.map { |row| "\0" < row.map { |p| format_str % p.values
}.join }
instead of
data = @data.map { |row| "\0" < row.map { |p| p.values.pack("C*")
}.join }
$ time ruby -Ilib profile.rb
real 0m4.974s
user 0m4.911s
sys 0m0.031s
Caching the string representation of the values in PNG::Color (because
each pixel is the same instance of color in this case):
Add to PNG::Color
def values_str
@values_str ||= "%c%c%c%c" % @values
end
Use
data = @data.map { |row| "\0" < row.map { |p| p.values_str }.join }
instead of
format_str = "%c%c%c%c"
data = @data.map { |row| "\0" < row.map { |p| format_str % p.values
}.join }
$ time ruby -Ilib profile.rb
real 0m2.489s
user 0m2.463s
sys 0m0.013s
Improving PNG::Canvas#initialize:
Use
@data = Array.new(@width) { |x| Array.new(@height, background) }
instead of
@data = Array.new(@width) { |x| Array.new(@height) { background } }
$ time ruby -Ilib profile.rb
real 0m1.941s
user 0m1.914s
sys 0m0.014s
Representing the values in PNG::Color as String (instead of as Array) (see
complete patch below):
$ time ruby -Ilib profile.rb
real 0m1.492s
user 0m1.445s
sys 0m0.015s
So, it is ten times faster in pure Ruby.
Dominik
--- png-1.0.0/lib/png.rb 2006-08-31 22:57:13.000000000 +0200
+++ png-1.0.0_opt/lib/png.rb 2006-09-02 19:26:46.000000000 +0200
@@ -71,15 +71,21 @@
##
# Writes the PNG to +path+.
- def save(path)
- File.open(path, "w") do |f|
- f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
- f.write PNG.chunk('IHDR',
+ def to_blob
+ blob = []
+ blob << "\211PNG\r\n\032\n" # PNG signature
+ blob << PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0
].pack("N2C5"))
# 0 == filter type code "none"
- data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
- f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
- f.write PNG.chunk('IEND', '')
+ data = @data.map { |row| "\0" < row.map { |p| p.values }.join }
+ blob << PNG.chunk('IDAT', Zlib::Deflate.deflate(data.join, 9))
+ blob << PNG.chunk('IEND', '')
+ blob.join
+ end
+
+ def save(path)
+ File.open(path, "w") do |f|
+ f.write to_blob
end
end
@@ -94,7 +100,7 @@
# Creates a new color with values +red+, +green+, +blue+, and +alpha+.
def initialize(red, green, blue, alpha)
- @values = [red, green, blue, alpha]
+ @values = "%c%c%c%c" % [red, green, blue, alpha]
end
##
@@ -151,7 +157,7 @@
end
def inspect # :nodoc:
- "#<%s %02x %02x %02x %02x>" % [self.class, *@values]
+ "#<%s %02x %02x %02x %02x>" % [self.class, r, b, g, a]
end
end
@@ -179,7 +185,7 @@
def initialize(height, width, background = Color::White)
@height = height
@width = width
- @data = Array.new(@width) { |x| Array.new(@height) { background } }
+ @data = Array.new(@width) { |x| Array.new(@height, background) }
end
##
This change gives a broken PNG. You meant to call #<<, not #<.
I see no significant speedup when using #<<.
> [values array for string changes]
These are good.
> Improving PNG::Canvas#initialize:
>
> Use
>
> @data = Array.new(@width) { |x| Array.new(@height, background) }
>
> instead of
>
> @data = Array.new(@width) { |x| Array.new(@height)
> { background } }
>
> $ time ruby -Ilib profile.rb
>
> real 0m1.941s
> user 0m1.914s
> sys 0m0.014s
Not really worth optimizing, since the improvement is so small. In
the optimized RubyInline version only 20% of the time is spent here
with no image generation. Adding image generation makes this
optimization insignificant.
Using your changes from the pure-ruby version I went from about 30
seconds to about 4 seconds, or 7.5 times faster.
> On Sep 2, 2006, at 10:42 AM, Dominik Bathon wrote:
>> On Sat, 02 Sep 2006 05:26:04 +0200, Eric Hodel <drb...@segment7.net>
>> wrote:
>>> I wrote an article on using RubyInline for optimization where I take
>>> png.rb, sprinkle in a little profiling and a little C and make it go
>>> over 100 times faster.
>>
>> Nice article, but in this case it is possible to get almost the same
>> speedup in pure Ruby:
>>
>> $ time ruby -Ilib profile.rb
>>
>> real 0m15.504s
>> user 0m15.119s
>> sys 0m0.309s
>>
>>
>> Avoiding flatten (and using a literal for the signature):
>>
>> def to_blob
>> blob = []
>> blob << "\211PNG\r\n\032\n" # PNG signature
>> blob << PNG.chunk('IHDR',
>> [ @height, @width, @bits, 6, 0, 0, 0 ].pack
>> ("N2C5"))
>> # 0 == filter type code "none"
>> data = @data.map { |row| "\0" < row.map { |p| p.values.pack("C*")
>> }.join }
>> blob << PNG.chunk('IDAT', Zlib::Deflate.deflate(data.join, 9))
>> blob << PNG.chunk('IEND', '')
>> blob.join
>> end
>
> This change gives a broken PNG. You meant to call #<<, not #<.
Oops, of course I meant #<<. I actually had tested that my changes still
produce correct results when I did the optimizations originally. Then I
recreated them to get the times for each step and made that typo, oh well.
Just for reference, the time for the final version with #< changed to #<<
on my machine:
$ time ruby -Ilib profile.rb
real 0m1.901s
user 0m1.841s
sys 0m0.032s
> I see no significant speedup when using #<<.
>
>> [values array for string changes]
>
> These are good.
>
>> Improving PNG::Canvas#initialize:
>>
>> Use
>>
>> @data = Array.new(@width) { |x| Array.new(@height, background) }
>>
>> instead of
>>
>> @data = Array.new(@width) { |x| Array.new(@height) { background } }
>>
>> $ time ruby -Ilib profile.rb
>>
>> real 0m1.941s
>> user 0m1.914s
>> sys 0m0.014s
>
> Not really worth optimizing, since the improvement is so small. In the
> optimized RubyInline version only 20% of the time is spent here with no
> image generation. Adding image generation makes this optimization
> insignificant.
Yes, it's not much for the total runtime, but it makes this one line about
14 times faster:
$ time ruby -e "40.times { Array.new(400) { Array.new(400) { 0 } } }"
real 0m2.430s
user 0m2.402s
sys 0m0.017s
$ time ruby -e "40.times { Array.new(400) { Array.new(400, 0) } }"
real 0m0.165s
user 0m0.147s
sys 0m0.016s
Hi, I installed the gems but I immediately run into this:
C:\Inline\png-1.0.0>ruby -Ilib example/profile.rb
example/profile.rb:8:in `draw': undefined method `to_blob' for
#<PNG:0x28c7b28>
(NoMethodError)
from example/profile.rb:15
from example/profile.rb:15
.. on Windows.
I get the same thing in cygwin:
$ time ruby -Ilib example/profile.rb
example/profile.rb:8:in `draw': undefined method `to_blob' for
#<PNG:0x28c7b28>
(NoMethodError)
from example/profile.rb:15
from example/profile.rb:15
real 0m2.361s
user 0m0.015s
sys 0m0.046s
The rdoc does not show a a 'to_blob' method for PNG
Any idea?
-- Mike Berrow
--
Posted via http://www.ruby-forum.com/.
> Eric Hodel wrote:
>> I wrote an article on using RubyInline for optimization where I take
>> png.rb, sprinkle in a little profiling and a little C and make it go
>> over 100 times faster.
>>
>> http://segment7.net/projects/ruby/inline_optimization.html
>>
>
> Hi, I installed the gems but I immediately run into this:
>
> C:\Inline\png-1.0.0>ruby -Ilib example/profile.rb
> example/profile.rb:8:in `draw': undefined method `to_blob' for
> #<PNG:0x28c7b28>
> (NoMethodError)
>
> Any idea?
From the article:
== Play along at home!
If you want to play along at home, download and unpack png-1.0.0.tgz
and save the profile benchmark code into example/.
I also added PNG#to_blob to eliminate the need to write the file to
disk, it looks almost exactly like PNG#save:
def to_blob
blob = []
blob << [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
blob << PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0 ].pack
("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
blob << PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
blob << PNG.chunk('IEND', '')
blob.join
end
I added my RubyInline extensions in png.rb and modified png.rb as
appropriate, so while you play along you should do that too.
Thanks,
but it seems I am missing some setup needed for RubyInline itself
I tried running the hello.rb in
C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.5.0\demo
and got this ...
C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.5.0\demo>ruby hello.rb
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:64:in
`test': can't
convert nil into String (TypeError)
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:64:in `
rootdir'
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:73:in `
directory'
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:253:in
`so_name'
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:287:in
`load_cache'
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:601:in
`inline'
from hello.rb:7
In Cygwin I get ...
$ ruby hello.rb
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:396:in ``':
No such
file or directory - cl -nologo -LD -MD -Zi -O2b2xg- -G6 -I
c:/ruby/lib/ruby/1.8
/i386-mswin32 -o C:\cygwin\home\User1/.ruby_inline/Inline_Hello_5d41.so
C:
\cygwin\home\User1/.ruby_inline/Inline_Hello_5d41.c -link
/INCREMENTAL:no
/EXPORT:Init_Inline_Hello_5d41 (Errno::ENOENT)
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:396:in
`build'
from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:602:in
`inline'
from hello.rb:7
I know I have MingW32 on here since I installed Dev-C++ and have that
running.
At C:\Dev-Cpp\bin there is a gcc.exe
Is it it just a matter of hooking it up this right way in RubyInline ?
> Eric Hodel wrote:
>> == Play along at home!
>> ....
>
> Thanks,
> but it seems I am missing some setup needed for RubyInline itself
>
> I tried running the hello.rb in
> C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.5.0\demo
>
> and got this ...
>
> C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.5.0\demo>ruby hello.rb
> c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:64:in
> `test': can't
> convert nil into String (TypeError)
> from
You need to set INLINEDIR or HOME in your environment. This will be
fixed in the next release.
> In Cygwin I get ...
>
> $ ruby hello.rb
> c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:396:in
> ``':
> No such
> file or directory - cl -nologo -LD -MD -Zi -O2b2xg- -G6 -I
> c:/ruby/lib/ruby/1.8
> /i386-mswin32 -o C:\cygwin\home\User1/.ruby_inline/
> Inline_Hello_5d41.so
> C:
> \cygwin\home\User1/.ruby_inline/Inline_Hello_5d41.c -link
> /INCREMENTAL:no
> /EXPORT:Init_Inline_Hello_5d41 (Errno::ENOENT)
> from
> c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.5.0/./inline.rb:396:in
> `build'
You need the MS C compiler.
> I know I have MingW32 on here since I installed Dev-C++ and have that
> running.
> At C:\Dev-Cpp\bin there is a gcc.exe
Your ruby wasn't compiled with GCC. If you want inline to use GCC,
you need to compile ruby with it.
> Is it it just a matter of hooking it up this right way in RubyInline ?
RubyInline uses whatever C compiler Ruby was built with. If you
don't have the same compiler installed inline won't be able to do its
thing. For windows using the one-click installer you need to either
install the same compiler or build ruby from scratch with a GCC.
> Your ruby wasn't compiled with GCC. If you want inline to use GCC, you need
> to compile ruby with it.
>
>> Is it it just a matter of hooking it up this right way in RubyInline ?
>
> RubyInline uses whatever C compiler Ruby was built with. If you don't have
> the same compiler installed inline won't be able to do its thing. For
> windows using the one-click installer you need to either install the same
> compiler or build ruby from scratch with a GCC.
or campaign for an msys based ruby dist which would make this all go away ;-)
-a
--
what science finds to be nonexistent, we must accept as nonexistent; but what
science merely does not find is a completely different matter... it is quite
clear that there are many, many mysterious things.
- h.h. the 14th dalai lama
> On Tue, 12 Sep 2006, Eric Hodel wrote:
>
>> Your ruby wasn't compiled with GCC. If you want inline to use
>> GCC, you need to compile ruby with it.
>>
>>> Is it it just a matter of hooking it up this right way in
>>> RubyInline ?
>>
>> RubyInline uses whatever C compiler Ruby was built with. If you
>> don't have
>> the same compiler installed inline won't be able to do its thing.
>> For
>> windows using the one-click installer you need to either install
>> the same
>> compiler or build ruby from scratch with a GCC.
>
> or campaign for an msys based ruby dist which would make this all
> go away ;-)
That would be ideal, but seems unlikely. Is there any progress on
this front?
I wonder how hard it would be to adapt the OCI build process (rakefiles
and other scripts) to generate both mingw and mswin based installers?
I'm sure it's much more than s/nmake/make/ and so on, but maybe it's
feasible.
Not to suggest that Curt should take this on, but there seems to be a
critical mass of interested folks here who would really like to have an
windows OCI that contains a gnu toolchain and can load gcc-built extensions.
(I'm doing a svn co to see what goes into the OCI, and to find out how
naive I am....)
--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
maybe we should just make one? if it were me, i'd probably just compile
everything, zip it, and let people unpack it. a bat script to setup the
environment, associactions, etc. might be nice, but i'm a fan of keeping
stuff out of system space and letting people just point their env at it...
what do you think? basically i'm thinking
msys-ruby-1.0.0.tgz
(unpack)
cd c:\msys-ruby-1.0.0\
setup.bat # sets up env vars, file assoc, etc.
setup.bat, for that matter, need only configure %PATH and then spawn setup.rb.
thoughts?
-a
--
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei
The command line is fine for a Ruby developer. For an "average" end
user of some Ruby program, a one-click installer is essential on the
Windows platform.
Along those same lines, you'll also want to automagically install gem,
win32 stuff, etc. -- all the good things on the windows platform that
*NIX does not need (except for gem -- that's always good stuff).
TwP
-austin
--
Austin Ziegler * halos...@gmail.com * http://www.halostatue.ca/
* aus...@halostatue.ca * http://www.halostatue.ca/feed/
* aus...@zieglers.ca
No. I have, unfortunately, been busy with wedding planning and have
not yet had time to pull together a promised email. Soon, I hope. The
invitations are sent.
> On 9/11/06, ara.t....@noaa.gov <ara.t....@noaa.gov> wrote:
>> or campaign for an msys based ruby dist which would make this all go away
>> ;-)
> ...and introduce three dozen other problems that aren't worth using MSYS for.
i'll confess that i have no idea if your are right or not and that chances are
very good that you are - still, it seems like it may be a worthwhile
experiment.
cheers.
No matter what path is taken, someone will still be unhappy. My goal
is to make things as simple as possible for 80% of the Windows users.
Curt
The "average" user might not care about InLine Ruby as much as you do...
I have a lot more to learn before I need InLine and by then it might be worth
DIY to get there.
If an "average" developer uses a library that depends on RubyInline,
then it becomes very important that it be easy to deploy ruby with
RubyInline working out of the box.
Well, now I've downloaded and installed Visual C++ 2005 express
edition in order to get that. I also upgraded the RubyInline gem
to the latest, greatest 3.6.0 version.
I set INLINEDIR to
INLINEDIR=C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.6.0\demo
To verify that the C-compiler is operational, I commandline-compiled
the following example provided by Microsoft.
#include <iostream>
int main()
{
std::cout << "This is a native C++ program." << std::endl;
return 0;
}
with the line:
cl /EHsc simple.cpp
after running the env var setup script they refer you to.
That all works fine.
So, anyway, here is what RubyInline gives me now ...
C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.6.0\demo>ruby hello.rb
cl : Command line warning D9035 : option 'Og-' has been deprecated and
will be r
emoved in a future release
cl : Command line warning D9035 : option 'o' has been deprecated and
will be rem
oved in a future release
cl : Command line warning D9002 : ignoring unknown option '-G6'
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.6.0/./inline.rb:404:in
`build': erro
r executing cl -nologo -LD -MD -Zi -O2b2xg- -G6 -I
c:/ruby/lib/ruby/1.8/i386-ms
win32 -o
"C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.6.0\demo/.ruby_inline/Inli
ne_Hello_5d41.so"
"C:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3.6.0/demo/.ruby_in
line/Inline_Hello_5d41.c" -link /LIBPATH:"c:/ruby/lib"
/DEFAULTLIB:"msvcrt-ruby
18.lib" /INCREMENTAL:no /EXPORT:Init_Inline_Hello_5d41: 512
(CompilationError)
Renamed
C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.6.0\demo/.ruby_inline/Inline
_Hello_5d41.c to
C:\ruby\lib\ruby\gems\1.8\gems\RubyInline-3.6.0\demo/.ruby_inli
ne/Inline_Hello_5d41.c.bad from
c:/ruby/lib/ruby/gems/1.8/gems/RubyInline-3
6.0/./inline.rb:624:in `inline'
from hello.rb:7
Hmmm. Am I getting closer?
BTW. I am using the "One-Click Ruby Installer for Windows" as my Ruby
installation.
Thanks for any help,
So ... no takers on this?
It would be hard to believe that I am the only one
that has (recently) tried to use RubyInline on Windows.
-- Mike
I only got it working after I recompiled ruby from sources using a
combination of cygwin and mingw, and even then I still had to patch
rubyinline to work with mingw.
Are you able to compile extensions using mkmf? If you can do that you
can try comparing the compile commands and adding the missing flags to
rubyinline.
Did you build ruby with this compiler?
> BTW. I am using the "One-Click Ruby Installer for Windows" as my Ruby
> installation.
.. no
You need to use the same compiler for building both Ruby and any
extensions, inline or not.
The simple solution, now that you have a compiler, is to build and
install a new ruby.
RubyInline has the ability to create a gem that includes the built
library removing the need to install a compiler.
Sigh ... I get the picture.
I guess that means I have to abandon all the fine work done by
Curt Hibbs on the one-click installer and start from scratch.
I am not an ace with C makefiles etc. these days so I sense that
could be a tough road.
Would it be easier to track down the exact C compiler used by
Curt and install that instead?.
Thanks for your help,