Failing to build `rake ruby19`

297 views
Skip to first unread message

Jarmo Pertman

unread,
Nov 9, 2011, 4:06:32 PM11/9/11
to RubyInstaller
I'm not able to build Ruby. This is what i see:

C:\rubyinstaller>rake ruby19 NOGEMS=1 NOTK=1 LOCAL=ruby --trace
Loading 001_dkcompiler_init.rb
Initializing DevKit compilers
...
** Extracting C:/rubyinstaller/downloads/coreutils-5.97-3-
msys-1.0.13-ext.tar.lzma into sandbox/devkit
"C:/rubyinstaller/sandbox/extract_utils/basic-bsdtar.exe" -xf "C:/
rubyinstaller/downloads/coreutils-5.97-3-msys-1.0.13-ext.tar.lzma" >
NUL 2>&1
rake aborted!
Command failed with status (1): ["C:/rubyinstaller...]
C:/Ruby193/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in
create_shell_runner'
C:/Ruby193/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
C:/Ruby193/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
C:/Ruby193/lib/ruby/1.9.1/rake/file_utils_ext.rb:39:in `sh'
C:/rubyinstaller/rake/extracttask.rb:93:in `block in
bsd_tar_extract'
C:/rubyinstaller/rake/extracttask.rb:92:in `chdir'
C:/rubyinstaller/rake/extracttask.rb:92:in `bsd_tar_extract'
C:/rubyinstaller/rake/extracttask.rb:27:in `extract'
C:/rubyinstaller/recipes/devkit/msys.rake:32:in `block (4 levels)
in <top (required)>'
C:/rubyinstaller/recipes/devkit/msys.rake:30:in `each'
C:/rubyinstaller/recipes/devkit/msys.rake:30:in `block (3 levels)
in <top (required)>'
C:/rubyinstaller/rake/checkpoint.rb:6:in `block in checkpoint'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:203:in `call'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:203:in `block in execute'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:200:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:200:in `execute'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:158:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:176:in `block in
invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in
`invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:157:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:176:in `block in
invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in
`invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:157:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:176:in `block in
invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in
`invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:157:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:176:in `block in
invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in
`invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:157:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:176:in `block in
invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in
`invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:157:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:176:in `block in
invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:174:in
`invoke_prerequisites'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:157:in `block in
invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:151:in
`invoke_with_call_chain'
C:/Ruby193/lib/ruby/1.9.1/rake/task.rb:144:in `invoke'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:116:in `invoke_task'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:94:in `block (2
levels) in top_level'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:94:in `each'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:94:in `block in
top_level'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:133:in
`standard_exception_handling'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:88:in `top_level'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:66:in `block in run'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:133:in
`standard_exception_handling'
C:/Ruby193/lib/ruby/1.9.1/rake/application.rb:63:in `run'
C:/Ruby193/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/bin/rake:32:in
`<top (required)>'
C:/Ruby193/bin/rake:19:in `load'
C:/Ruby193/bin/rake:19:in `<main>'
Tasks: TOP => ruby19 => interpreter:ruby19:configure => compiler
=> devkit:activate => devkit:msys => devkit:msys:extract => C:/
rubyinstaller/sandbox/.checkpoints/.msys-extract

Jarmo

Jon

unread,
Nov 9, 2011, 4:15:03 PM11/9/11
to rubyin...@googlegroups.com

From the gist it sounds like the file is OK, but would you try to see if you can manually list it's contents via something like this from cmd.exe.

C:/rubyinstaller/sandbox/extract_utils/basic-bsdtar.exe -tf C:/rubyinstaller/downloads/coreutils-5.97-3-msys-1.0.13-ext.tar.lzma


Jon

---
Fail fast. Fail often. Fail publicly. Learn. Adapt. Repeat.
http://thecodeshop.github.com | http://jonforums.github.com/
twitter: @jonforums

Luis Lavena

unread,
Nov 9, 2011, 4:15:29 PM11/9/11
to rubyin...@googlegroups.com
On Wed, Nov 9, 2011 at 6:06 PM, Jarmo Pertman <jar...@gmail.com> wrote:
> I'm not able to build Ruby. This is what i see:
>
> C:\rubyinstaller>rake ruby19 NOGEMS=1 NOTK=1 LOCAL=ruby --trace
>    Loading 001_dkcompiler_init.rb
>    Initializing DevKit compilers
>    ...
>    ** Extracting C:/rubyinstaller/downloads/coreutils-5.97-3-
> msys-1.0.13-ext.tar.lzma into sandbox/devkit
>    "C:/rubyinstaller/sandbox/extract_utils/basic-bsdtar.exe" -xf "C:/
> rubyinstaller/downloads/coreutils-5.97-3-msys-1.0.13-ext.tar.lzma" >
> NUL 2>&1
>    rake aborted!
>    Command failed with status (1): ["C:/rubyinstaller...]
>    C:/Ruby193/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in
> create_shell_runner'

Can you provide more information about your environment?

I'm using Ruby 1.9.3-p0, Windows 7 x64, fresh checkout of
RubyInstaller and Ruby with patch applied, and it extracts properly.

This is how files looks like in my downloads folder:

https://gist.github.com/1353054

Can you verify the sizes of those?
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

Jon

unread,
Nov 9, 2011, 4:23:11 PM11/9/11
to rubyin...@googlegroups.com
> On Wed, Nov 9, 2011 at 6:06 PM, Jarmo Pertman <jar...@gmail.com> wrote:
> > I'm not able to build Ruby. This is what i see:
> >
> > C:\rubyinstaller>rake ruby19 NOGEMS=1 NOTK=1 LOCAL=ruby --trace
> >    Loading 001_dkcompiler_init.rb
> >    Initializing DevKit compilers
> >    ...
> >    ** Extracting C:/rubyinstaller/downloads/coreutils-5.97-3-
> > msys-1.0.13-ext.tar.lzma into sandbox/devkit
> >    "C:/rubyinstaller/sandbox/extract_utils/basic-bsdtar.exe" -xf "C:/
> > rubyinstaller/downloads/coreutils-5.97-3-msys-1.0.13-ext.tar.lzma" >
> > NUL 2>&1
> >    rake aborted!
> >    Command failed with status (1): ["C:/rubyinstaller...]
> >    C:/Ruby193/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in
> > create_shell_runner'
>
> Can you provide more information about your environment?

And I'd like the full path to where you locally checked out Ruby.

It doesn't appear related to your failure but I don't like this command line

C:\rubyinstaller>rake ruby19 NOGEMS=1 NOTK=1 LOCAL=ruby --trace

because of "LOCAL=ruby". I usually use build with

rake ruby19 local=c:\Users\Jon\Documents\RubyDev\ruby-git nogems=1 notk=1

Jarmo Pertman

unread,
Nov 9, 2011, 4:26:19 PM11/9/11
to RubyInstaller
I did `rake clobber` to start over and i am getting Timeout::Error now
when trying to download libffi-3.0.10.tar.gz. I guess i have to wait
another chance now.

I'm also on Ruby 1.9.3-p0, Win 7 x64 with fresh clone of rubyinstaller
and ruby with applied expand_path patch.

Jarmo

On Nov 9, 11:15 pm, Luis Lavena <luislav...@gmail.com> wrote:

Luis Lavena

unread,
Nov 9, 2011, 4:28:04 PM11/9/11
to rubyin...@googlegroups.com

Please read Jon comment about using LOCAL and the need of a full path for it to work.

Sent from mobile.

--
You received this message because you are subscribed to the Google Groups "RubyInstaller" group.
To post to this group, send email to rubyin...@googlegroups.com.
To unsubscribe from this group, send email to rubyinstalle...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyinstaller?hl=en.

Jon

unread,
Nov 9, 2011, 4:31:25 PM11/9/11
to rubyin...@googlegroups.com
> I did `rake clobber` to start over and i am getting Timeout::Error now
> when trying to download libffi-3.0.10.tar.gz. I guess i have to wait
> another chance now.
>
> I'm also on Ruby 1.9.3-p0, Win 7 x64 with fresh clone of rubyinstaller
> and ruby with applied expand_path patch.
>

When you do get all the deps downloaded and cached to `downloads` I suggest using `rake clean` as it will save your `downloads` cache.

`rake clobber` really does do a wonderful job of clobbering ;)

Jarmo Pertman

unread,
Nov 9, 2011, 4:51:02 PM11/9/11
to RubyInstaller
It seems that the clobbering helped since i'm now in the middle of
compiling.

LOCAL seems to be okay with relative path since i have cloned ruby/
ruby into rubyinstaller directory :)

Jarmo

On Nov 9, 11:31 pm, Jon <jon.for...@gmail.com> wrote:
> > I did `rake clobber` to start over and i am getting Timeout::Error now
> > when trying to download libffi-3.0.10.tar.gz. I guess i have to wait
> > another chance now.
>
> > I'm also on Ruby 1.9.3-p0, Win 7 x64 with fresh clone of rubyinstaller
> > and ruby with applied expand_path patch.
>
> When you do get all the deps downloaded and cached to `downloads` I suggest using `rake clean` as it will save your `downloads` cache.
>
> `rake clobber` really does do a wonderful job of clobbering ;)
>
> Jon
>
> ---
> Fail fast. Fail often. Fail publicly. Learn. Adapt. Repeat.http://thecodeshop.github.com|http://jonforums.github.com/
> twitter: @jonforums

Jarmo Pertman

unread,
Nov 9, 2011, 4:53:16 PM11/9/11
to RubyInstaller
Oh, forgot to mention that compile crashed before taking cmd.exe with
it and i can see this in Event Log:
Faulting application name: conhost.exe, version: 6.1.7601.17641, time
stamp: 0x4e03fdaa
Faulting module name: conhost.exe, version: 6.1.7601.17641, time
stamp: 0x4e03fdaa
Exception code: 0xc0000005
Fault offset: 0x0000000000002b11
Faulting process id: 0x159c
Faulting application start time: 0x01cc9f00da2bed87
Faulting application path: C:\Windows\system32\conhost.exe
Faulting module path: C:\Windows\system32\conhost.exe
Report Id: b06fd945-0b0c-11e1-bdbb-001e33247bf5

Not much to see there, but i uninstalled ANSICON since that hooks into
cmd and after that it seems that i'm having more luck. Can't say for
sure though that ANSICON had anything to do with the crash in the
first place, but if anyone else experiences similar issues, maybe
uninstalling it helps them too.

I'm gonna install it later again of course :)

Jarmo

On Nov 9, 11:31 pm, Jon <jon.for...@gmail.com> wrote:
> > I did `rake clobber` to start over and i am getting Timeout::Error now
> > when trying to download libffi-3.0.10.tar.gz. I guess i have to wait
> > another chance now.
>
> > I'm also on Ruby 1.9.3-p0, Win 7 x64 with fresh clone of rubyinstaller
> > and ruby with applied expand_path patch.
>
> When you do get all the deps downloaded and cached to `downloads` I suggest using `rake clean` as it will save your `downloads` cache.
>
> `rake clobber` really does do a wonderful job of clobbering ;)
>
> Jon
>
> ---
> Fail fast. Fail often. Fail publicly. Learn. Adapt. Repeat.http://thecodeshop.github.com|http://jonforums.github.com/
> twitter: @jonforums

Luis Lavena

unread,
Nov 9, 2011, 5:03:12 PM11/9/11
to rubyin...@googlegroups.com

Oh, is ansicon, that is a known issue. Remove it from autorun and should be fine.

Sent from mobile.

Jarmo Pertman

unread,
Nov 10, 2011, 1:32:47 PM11/10/11
to RubyInstaller
Okay, i got it compiled and here are my results.

For my small project with very few specs i got the specs running time
down about 1-2 seconds. I used this command to measure:
ruby -e "ts = []; 5.times {t = Time.now; system 'rspec spec'; ts <<
Time.now - t}; puts ts.reduce(&:+) / ts.size.to_f"

And when it comes to empty Rails 3 project (with `rails new empty-
ruby193 -d sqlite3`) running the command Luis also ran:
ruby -e "ts = []; 5.times {t = Time.now; system 'ruby script\rails
runner \"puts $LOADED_FEATURES.size\"'; ts << Time.now - t}; puts
ts.reduce(&:+) / ts.size.to_f"

I got it down about ~10 seconds from 18 to 8. And this only happened
when i used patched Ruby version with having "-rfenix/replace" as
RUBYOPT.

It's still nowhere near your 2.x seconds, but i guess that here's just
hardware factors to keep in mind. I'm using really average laptop
here. Anyway, the drop is almost 50% for rails and less for other
projects. It's still better than nothing :)

Thanks!

Jarmo

On Nov 10, 12:03 am, Luis Lavena <luislav...@gmail.com> wrote:
> Oh, is ansicon, that is a known issue. Remove it from autorun and should be
> fine.
>
> Sent from mobile.

Jon

unread,
Nov 10, 2011, 1:50:49 PM11/10/11
to rubyin...@googlegroups.com
> Okay, i got it compiled and here are my results.
>
> For my small project with very few specs i got the specs running time
> down about 1-2 seconds. I used this command to measure:
> ruby -e "ts = []; 5.times {t = Time.now; system 'rspec spec'; ts <<
> Time.now - t}; puts ts.reduce(&:+) / ts.size.to_f"
>
> And when it comes to empty Rails 3 project (with `rails new empty-
> ruby193 -d sqlite3`) running the command Luis also ran:
> ruby -e "ts = []; 5.times {t = Time.now; system 'ruby script\rails
> runner \"puts $LOADED_FEATURES.size\"'; ts << Time.now - t}; puts
> ts.reduce(&:+) / ts.size.to_f"
>
> I got it down about ~10 seconds from 18 to 8. And this only happened
> when i used patched Ruby version with having "-rfenix/replace" as
> RUBYOPT.
>
> It's still nowhere near your 2.x seconds, but i guess that here's just
> hardware factors to keep in mind. I'm using really average laptop
> here. Anyway, the drop is almost 50% for rails and less for other
> projects. It's still better than nothing :)


If you're still in the building and testing mood, would you try adding the fenix patch to the patches listed here and see what you get?

http://groups.google.com/group/thecodeshop/browse_thread/thread/b963c472e3c747a8

I have a belief/hope that the combination will lower both `user` and `system` time, but haven't had time yet to look into it.


Also, would you mind critically reviewing the following benchmark workloads?

https://github.com/jonforums/measurements/blob/master/workloads/core_require_empty.rb
https://github.com/jonforums/measurements/blob/master/workloads/core_require_nested.rb

and the `--extended` cmd line option

https://github.com/jonforums/measurements/blob/master/lib/inquisitor.rb#L79-103
https://github.com/jonforums/measurements/blob/master/config.yml#L2


I'd like to whip those workloads into shape so they're viewed as a valid way to emulate requiring a large number of files similar to enough to Rails apps without requiring that one build, or agree upon, a common test Rails app. I think an easy-to-use-and-valid require workload is important as we continue to test fenix and other require optimizations like Yura's patch and Xavier's work.

In a nutshell, once you've cloned the measurements repo and run `rci init`, the require workloads are meant to emulate requiring files, 1000 at a time. The `config.yml` setting and `--extended` try to make it easy to scale the emulation in 1000's. So to emulate requiring 4000 files, use `:extended_iterations: 4` and --extended.

https://github.com/jonforums/measurements/blob/master/README.mkd

Luis Lavena

unread,
Nov 10, 2011, 1:51:22 PM11/10/11
to rubyin...@googlegroups.com
On Thu, Nov 10, 2011 at 3:32 PM, Jarmo Pertman <jar...@gmail.com> wrote:
> Okay, i got it compiled and here are my results.
>
> [...]

>
> And when it comes to empty Rails 3 project (with `rails new empty-
> ruby193 -d sqlite3`) running the command Luis also ran:
> ruby -e "ts = []; 5.times {t = Time.now; system 'ruby script\rails
> runner \"puts $LOADED_FEATURES.size\"'; ts << Time.now - t}; puts
> ts.reduce(&:+) / ts.size.to_f"
>
> I got it down about ~10 seconds from 18 to 8. And this only happened
> when i used patched Ruby version with having "-rfenix/replace" as
> RUBYOPT.

That is correct, Fenix will only be enabled if RUBYOPT is set.

> It's still nowhere near your 2.x seconds, but i guess that here's just
> hardware factors to keep in mind. I'm using really average laptop
> here. Anyway, the drop is almost 50% for rails and less for other
> projects. It's still better than nothing :)

Yes, 2.x seconds are hardware factors. Core 2 Duo T7500 @ 2.2GHz
4GB of RAM and using a RAMDisk for both Ruby and the application just
to eliminate traditional HDD spinup or fragmentation factors that
could generate random results.

What is important is the difference percentage, in my case I did
obtain up to 40% boost in some cases (between empty and mid size
application).

Is worth to mention also the excessive amount of calls to
File.expand_path are done for every single require, at least 3 times
for every single element in $LOAD_PATH combining .rb, .so and .dll
extensions when looking up for files.

THAT needs to be improved beyond any API-usage improvement we do.

James Cowlishaw

unread,
Nov 10, 2011, 3:06:48 PM11/10/11
to rubyin...@googlegroups.com, rubyin...@googlegroups.com
On 10 Nov 2011, at 18:51, Luis Lavena <luisl...@gmail.com> wrote:
>
> Is worth to mention also the excessive amount of calls to
> File.expand_path are done for every single require, at least 3 times
> for every single element in $LOAD_PATH combining .rb, .so and .dll
> extensions when looking up for files.
>
>
Is that one of the append advantages posix systems get; they're not looking for .dll files, just .so and .rb?

James.

Luis Lavena

unread,
Nov 10, 2011, 3:31:24 PM11/10/11
to rubyin...@googlegroups.com

Normally Ruby doesn't reach to .dll, either it finds .rb or .so before.

The thing is that every element in $LOAD_PATH gets expanded *over and
over again* multiple times.

The following is the pseudo code of what happens:

https://gist.github.com/1356075

Now, replace name for anything you want to require that is in the
$LOAD_PATH, and do the following on top of the code:

https://gist.github.com/1356101

And see the results:

found: C:/Users/Luis/Tools/Ruby/ruby-1.9.3-p0-i386-mingw32/lib/ruby/site_ruby/1.9.1/readline.rb
size of $LOAD_PATH: 8
calls to File.expand_path: 9

9 calls for 1 file in a LOAD_PATH of 8 elements.

Rails average LOAD_PATH (due all the gems that needs to be loaded) is 68.

Attempting the same after doing a gem "rails" and require on top of
the script (before $count):

found: C:/Users/Luis/Tools/Ruby/ruby-1.9.3-p0-i386-mingw32/lib/ruby/site_ruby/1.9.1/readline.rb
size of $LOAD_PATH: 32
calls to File.expand_path: 57

And that still didn't actually load any code, mainly because things
are using autoload for lazy loading, but once they are added to the
$LOAD_PATH, the number of calls to File.expand_path increase.

This could be solved if the expanded_path is expanded only once before
the extension or even better, $LOAD_PATH knows which ones are expanded
from the ones they aren't and avoid the expensive expand_path.

The main blocker here is that everything around feature loading and
tracking in the already loaded features is written in C, inside load.c
and combined with file.c (find_file_* functions)

The code is not easy to read and has zero comments to be able to
understand the whole process.

Sorry, couldn't avoid the rant :-P

Jarmo Pertman

unread,
Nov 10, 2011, 3:41:51 PM11/10/11
to RubyInstaller
Here are my results https://gist.github.com/1356135

I used default settings in config.yml

What is this "Yura's updated `load.c` patch for sorted
$LOADED_FEATURES"? I remember that i saw some patch when this "slow
require problem" appeared some time ago where using an array was
replaced with using a hashmap internally. This makes the lookup almost
O(1) which ought to be a lot faster than using array as it used at the
moment in official Ruby. Or?

Anyway, as seeing from the results then the fastest one is Luis's
patch with fenix :)

Jarmo

On Nov 10, 8:51 pm, Luis Lavena <luislav...@gmail.com> wrote:

Luis Lavena

unread,
Nov 10, 2011, 3:49:37 PM11/10/11
to rubyin...@googlegroups.com
On Thu, Nov 10, 2011 at 5:41 PM, Jarmo Pertman <jar...@gmail.com> wrote:
>
> What is this "Yura's updated `load.c` patch for sorted
> $LOADED_FEATURES"?

http://redmine.ruby-lang.org/issues/5427

Is a non-complex patch that improves the require performance.

>I remember that i saw some patch when this "slow
> require problem" appeared some time ago where using an array was
> replaced with using a hashmap internally. This makes the lookup almost
> O(1) which ought to be a lot faster than using array as it used at the
> moment in official Ruby. Or?
>

That was Xavier Shay, didn't get accepted but a minor variance was
implemented in 1.9.3, that is why 1.9.3 is noticeable faster than
1.9.2. similar to 1.8.7

(but still slow as snail on Windows, that is another thing)

> Anyway, as seeing from the results then the fastest one is Luis's
> patch with fenix :)
>

Yura's + Dusan + Fenix does increase performance beyond just Fenix,
but that needs more experimentation. (don't have the results right now
but I believe I've commented that to Jon)

Thank you for volunteering to be guinea pig on this :o)

Jarmo Pertman

unread,
Nov 10, 2011, 4:03:30 PM11/10/11
to RubyInstaller
On Nov 10, 10:49 pm, Luis Lavena <luislav...@gmail.com> wrote:
> That was Xavier Shay, didn't get accepted but a minor variance was
> implemented in 1.9.3, that is why 1.9.3 is noticeable faster than
> 1.9.2. similar to 1.8.7

What's the reason for not getting accepted? Binary search is still
slower than lookup from hashmap...

Jarmo

Luis Lavena

unread,
Nov 10, 2011, 4:10:42 PM11/10/11
to rubyin...@googlegroups.com

You should ask Ruby-Core (errr, that is me).

Was not accepted because:

#1 - The changes to how $LOADED_FEATURES work was BIG and could
introduce newer bugs
(rant: there is no single test or spec that defines what is the right
behavior, nor there is a comment in the code, so go figure)

#2 - 1.9.3 was already in feature-freeze mode, so big changes are not accepted.

If you want to learn more instead of my transcript, please read the
entire thread here:

http://redmine.ruby-lang.org/issues/3924

Jarmo Pertman

unread,
Nov 10, 2011, 4:21:59 PM11/10/11
to RubyInstaller
So, there is still hope :)

On Nov 10, 11:10 pm, Luis Lavena <luislav...@gmail.com> wrote:

Jon

unread,
Nov 10, 2011, 4:32:02 PM11/10/11
to rubyin...@googlegroups.com
> Here are my results https://gist.github.com/1356135
>
> I used default settings in config.yml

Just got out of a meeting and saw your results. Finally, a decent inbox message today ;)


> Yura's + Dusan + Fenix does increase performance beyond just Fenix,
> but that needs more experimentation. (don't have the results right now
> but I believe I've commented that to Jon)

Take a look again Luis...it looks to me that Jarmo's data shows ruby_1_9_3 + fenix beating the tcs-ruby193_require_winio (with load.c patch) and fenix.

Did I understand correctly that everything below `RUBYOPT=-rfenix/replace` line has fenix support compiled in? If so, just ruby_1_9_3 + fenix is the current winner in both `user` and `system`.

Or at least on those benchmarks. Probably need to correlate on a real Rails app to see if these results sync up with Rails reality?

Looks like we also need to build ruby_1_9_3 + fenix + http://redmine.ruby-lang.org/attachments/2214/win_io_for_r33699.patch and see how that works.

And as you both say, there's Xavier's work to check out too.

Jarmo Pertman

unread,
Nov 11, 2011, 8:36:25 AM11/11/11
to RubyInstaller
On Nov 10, 11:32 pm, Jon <jon.for...@gmail.com> wrote:
> Did I understand correctly that everything  below `RUBYOPT=-rfenix/replace` line has fenix support compiled in?  If so, just ruby_1_9_3 + fenix is the current winner in both `user` and `system`.

I'm not sure if fenix support was compiled in to every Ruby below my
RUBYOPT setting - only place i compiled it personally in was with the
"# Ruby 1.9.3 built from ruby/ruby ruby_1_9_3 using Luis's patches.".
All other were already precompiled and i downloaded these from the
links you provided. If fenix was compiled in then it was compiled,
otherwise not. I had only RUBYOPT set to -rfenix/replace and had
fenix.so and replace.rb under these rubies.

>
> Or at least on those benchmarks. Probably need to correlate on a real Rails app to see if these results sync up with Rails reality?
>
> Looks like we also need to build ruby_1_9_3 + fenix +http://redmine.ruby-lang.org/attachments/2214/win_io_for_r33699.patchand see how that works.
Let me know if you've done that so i can test it out :)


> And as you both say, there's Xavier's work to check out too.
Especially let me know when Xavier's work has been incorporated into
current patch so i could try it out. I'm hoping that it could be
pretty much faster.


>
> Jon
Jarmo

Jarmo Pertman

unread,
Nov 15, 2011, 11:28:58 AM11/15/11
to RubyInstaller
Hi!

I tried to recompile with doing everything the same as before and
didn't notice any new changes in ruby/ruby during the meantime, but
i'm unable to - doesn't matter how much i retry. This is what i'm
getting:

checking for dup2... yes
checking for memmove... yes
../../../ruby/configure: command substitution: line 14002: unexpected
EOF while looking for matching `"'
../../../ruby/configure: command substitution: line 14003: syntax
error: unexpected end of file
checking for strerror"
if eval test "x$" = xyes; then :
printf %s\n #define... ../../../ruby/configure: eval: line 2205:
unexpected EOF while looking for matching `"'
../../../ruby/configure: eval: line 2206: syntax error: unexpected end
of file
../../../ruby/configure: command substitution: line 14008: unexpected
EOF while looking for matching `"'
../../../ruby/configure: command substitution: line 14009: syntax
error: unexpected end of file
../../../ruby/configure: line 14008: $as_tr_cpp`: command not found
../../../ruby/configure: eval: line 2252: unexpected EOF while looking
for matching ``'
../../../ruby/configure: eval: line 2253: syntax error: unexpected end
of file
../../../ruby/configure: eval: line 2257: unexpected EOF while looking
for matching ``'
../../../ruby/configure: eval: line 2258: syntax error: unexpected end
of file
yes
checking for isnan... (cached) yes
checking for finite... (cached) yes
checking for isinf... (cached) yes
checking for hypot... yes
checking for acosh... yes
checking for erf... yes
checking for tgamma... yes
checking for lgamma_r... no
checking for cbrt... yes
../../../ruby/configure: command substitution: line 14186: unexpected
EOF while looking for matching `"'
../../../ruby/configure: command substitution: line 14187: syntax
error: unexpected end of file
../../../ruby/configure: line 14270: syntax error near unexpected
token `('
../../../ruby/configure: line 14270: ` $as_echo_n "(cached) " >&6'
rake aborted!
Command failed with status (2): [sh -c "../../../ruby/configure --
enable-sh...]

Tasks: TOP => ruby19 => interpreter:ruby19:configure
(See full trace by running task with --trace)


Any ideas, what might cause it and why it started to happen?

Jarmo

On Nov 11, 3:36 pm, Jarmo Pertman <jarm...@gmail.com> wrote:
> On Nov 10, 11:32 pm, Jon <jon.for...@gmail.com> wrote:
>
> > Did I understand correctly that everything  below `RUBYOPT=-rfenix/replace` line has fenix support compiled in?  If so, just ruby_1_9_3 + fenix is the current winner in both `user` and `system`.
>
> I'm not sure if fenix support was compiled in to every Ruby below my
> RUBYOPT setting - only place i compiled it personally in was with the
> "# Ruby 1.9.3 built from ruby/ruby ruby_1_9_3 using Luis's patches.".
> All other were already precompiled and i downloaded these from the
> links you provided. If fenix was compiled in then it was compiled,
> otherwise not. I had only RUBYOPT set to -rfenix/replace and had
> fenix.so and replace.rb under these rubies.
>
>
>
> > Or at least on those benchmarks. Probably need to correlate on a real Rails app to see if these results sync up with Rails reality?
>
> > Looks like we also need to build ruby_1_9_3 + fenix +http://redmine.ruby-lang.org/attachments/2214/win_io_for_r33699.patchandsee how that works.

Jon

unread,
Nov 15, 2011, 11:37:52 AM11/15/11
to rubyin...@googlegroups.com

Short term, this may be interesting:

http://groups.google.com/group/thecodeshop/browse_thread/thread/741b810701b990e9

FWIW, I just built both ruby_1_9_3 and trunk on Win7 32bit and didn't run into the problem you're having.

Reply all
Reply to author
Forward
0 new messages