1.9.3 IO degradation?
rogerdpack
I ran a small...fake, useless test [1] against 1.9.3 versus 1.9.2
windows, and found that even binary mode seems slower in 1.9.3.
about 15% slower in ascii mode, but then in binary mode:
$ ruby -v bench_io.rb b
ruby 1.9.2p290 (2011-07-09) [i386-mingw32]
1.42756986618042
3.213285207748413
ruby.exe -v bench_io.rb b
ruby 1.9.3dev (2011-09-27 revision 33347) [i386-mingw32]
13.221287
13.765504
Anybody have some insight?
Not surprisingly, in ascii mode 1.9.2 and 1.9.3 take like 50s but
that's because I guess I'm too lazy to go in there and optimize for
the ASCII case or something.
Thanks!
-roger-
[1] https://gist.github.com/1265504
Jon
Ugh. FYI, from three months ago posted to ruby-core
http://redmine.ruby-lang.org/issues/5056#note-14
and most recently attempting to get additional ruby-core attention to the issue
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/39873
Forgetting the lack of ruby-core responses or the concerning higher level issues for the time, it would be great to get some additional eyes looking at this in a sustainable way.
But the problem we all know and all face is our that our real lives, real work, and other interests outside Ruby only allow so much time. And AFAIK none of us are being sponsored on this issue.
If you have time and interest Roger, would you eyeball the info at my first link and what I'd discovered and see how it meshes with what you've found? BTW 1.9.4dev back then is actually early 1.9.3 now.
Perhaps you and I (and anyone else interested..hint, hint) can spend some time on this while Luis continues to look into the area he's investigating. At some point it makes sense to combine these efforts, but divide-and-conquer may keep us moving forward, however slowly :)
BTW, we know have a good way to get decent profiling info
http://jonforums.github.com/cochise/2011/07/18/profiling-mingw-apps.html
and another good API tracing tool in addition to some of the sysinternals tools is
http://www.rohitab.com/apimonitor
Jon
---
blog: http://jonforums.github.com/
twitter: @jonforums
Most people die of a sort of creeping common sense, and discover when it
is too late that the only things one never regrets are one's mistakes.
- Oscar Wilde
rogerdpack
> Ugh. FYI, from three months ago posted to ruby-core
>
It's just this type of thing that keeps me on jruby for windows. At
least it's stable :)
-roger-
rogerdpack
http://blogs.msdn.com/b/brian_swan/archive/2011/10/12/why-is-php-5-3-on-windows-faster-than-previous-php-versions.aspx
http://dev.mysql.com/tech-resources/articles/5.5/mysql-55-faster-on-windows.html
If only somebody had enough time to move everything over to more
native calls (I know Luis has done some research in the past in this
regard)...
Thanks!
-r
Jon
Nice links Roger, thanks.
Implicit in these slides is the importance of profiling and tracing tools to determine _what_ to change. It's not always clear what tools exist that allow one to get this insight when building with MinGW.
If you're using recent Windows SDKs or VS, the built-in debugging and performance tools are very compelling. But sadly they can't be used with MinGW primarily due to differences in debugging symbols. Even more sad is that a time strapped Windows gurus may not have the time/desire to learn to use new development tools to scratch their OSS itch.
That said, I'm very excited with the following tools that work well with MinGW builds
http://smartbear.com/products/free-tools/aqtime-standard/
http://www.rohitab.com/apimonitor
http://technet.microsoft.com/en-us/sysinternals/bb545021
For me, two key issues remain:
1) Finding a way to enable and entice multiple people to invest their limited time to make part-time contributions (they need to be high quality contributions) to finding, updating, and testing performance mods. Given today's freely available collaboration tools, this really is _not_ a technical issue.
2) Changing deeply ingrained biases/beliefs regarding Ruby on the Windows platform. IMO, this is THE single biggest issue on which everything else depends. Outdated and just plain wrong beliefs affect resourcing decisions, set the tone for both users and project contributors, and impact perceptions as to the long-term viability of an implementation that consequently diminish sustainable development investment in a particular implementation. While it may be heroic to do the heavy lift once to solve a single issue, if you can't sustain the development investment, you ultimately lose to apathy, neglect, and finally bit rot. Some claim this a "technical marketing" challenge, but it's more complex and often more subtle than that.
Given your long-term involvement and interest in Ruby and experience with both MRI and JRuby, what do you (and others) think are creative, pragmatic, and most importantly, implementable ways to address the obvious MRI on Windows performance issues given resourcing constraints? At the end of the day, we all know it ultimately needs developer-mind-to-source-file work ;)
Or put another way, what's preventing MRI from emulating the fine multi-platform work being done by the https://github.com/joyent/libuv contributors?
Phillip Gawlowski
>
> Given your long-term involvement and interest in Ruby and experience with both
>MRI and JRuby, what do you (and others) think are creative, pragmatic, and
> most importantly, implementable ways to address the obvious MRI on Windows
> performance issues given resourcing constraints?
The most glaring issue is with Ruby's startup performance, but that's
already being tackled with the recent cascade of load.c patches for
Ruby.
What is lacking is Open Source tooling, like Visual Studio support
(hah!), Win32/64 API support (IIRC, that's been gemified, but it is
reasonable to include that in the RubyInstaller distribution, I
think), and MS Office scripting (via OLE, for example).
Oh, and color output. on the command line, via ANSIcon (which now
works wonderfully; thanks Luis!). Maybe a custom Ruby Shell would help
with that, similar to the Visual Studio command prompt.
I think making color output easier (or rather: available) would be the
easiest, and most visible, change to implement, right after including
Windows-specific gems.
--
Phillip Gawlowski
gplus.to/phgaw | twitter.com/phgaw
A method of solution is perfect if we can forsee from the start,
and even prove, that following that method we shall attain our aim.
-- Leibniz
Dušan D. Majkić
> work being done by the https://github.com/joyent/libuv contributors?
Node on Windows is supported by Joyent and Microsoft.
Oneclick Ruby on Windows is supported by Luis Lavena and Jon
Forums.
And RubyInstaller is, by far, better and more complete Windows
product than Node is. I can say that as a user, and as a user
who'll peek (and probably poke) into code repository.
But what am I, as a developer, missing from Ruby on Windows?
First thing I'm missing is endpoint. Rails apps written on Windows
will end on Linux server, and decent Windows GUI is absent.
I found a niche in Cucumber tests using Rails background and
(excellent!) SQL Server ActiveRecord adapter.
JRuby is in advantage here. It contains cross-platform Java GUI,
compiler and servers. Enabling users to compile Ruby code
to end-user deploy-able war/jar file. But Java is a beast I'd rather
avoid.
MacRuby is also in advantage here, it lays directly on native
ObjC platform, have a full support for XCode including GUI
designer and you can sell your app on AppStore.
There should be a "Windows Way" to deploy Ruby App to
end user. Being it a web or a desktop app. I like the MacRuby
idea of bundling MacRuby as framework dir in your app. And
I like exerb, but it stalled.
Second thing I'm missing is more info back from people
working on Windows Ruby Core. Looking at that code, I see
too much of the Win9x, and lack of API calls that recent
Windows have.
So, here I am. By iterating throughout Cucumber steps, I now
have a full model, and a lot of controller code. What are my
next options on Windows?
Luis Lavena
>> what's preventing MRI from emulating the fine multi-platform
>> work being done by the https://github.com/joyent/libuv contributors?
>
> Node on Windows is supported by Joyent and Microsoft.
> Oneclick Ruby on Windows is supported by Luis Lavena and Jon
> Forums.
>
Hello,
Chiming late to the party, but I wanted to share the following:
Node is financially backed by Joyent and Microsoft. This means they
get people paid to work on it, including work and improve it on any
platform Joyent and Microsoft are interested node works on.
Even if Microsoft wanted to dedicate resources for Ruby improvement,
that would never happen. Not due MS not willing to put dedicated
resources in it, but because the long and stale development way of
Ruby and it's structure.
If we can map Ruby development cycle to some timeline will be closer
to world periods been every release something between Paleolithic and
mesolithic.
To Ruby-Core, performance issues are not considered bugs, are
considered features, because of that, fixing has taken really low
priority.
There is nobody interested in clean up Ruby to be a more multi
platform friendly. Just the idea of change part of the 4K lines that
is file.c will raise rage on Ruby-Core, specially since maintainers
will no longer be familiar with the codebase and thus, no longer able
to maintain it.
1.9.3 is a bit faster than 1.9.2 in some aspect, yet still is slow as
snail on Windows, even worse than 1.9.2.
Things like multiple expansion of path, usage of obsolete API or due
things using inefficient code.
I can go for hours on this...
The team around Node.js, along with the help of Joyent and Microsoft
are proving that Windows is not the bottleneck of performance of your
language, that is just FUD I'm tired of listening to.
So, to be able to change this is not only about resources, but about
willingness to change it.
I can't change this by myself, nor to I have the knowledge compared to
experts like Microsoft engineers. I'm just a dumb guy who is very
stubborn.
While Rubinius might promise a pure-Ruby alternative to this, it's
state on Windows is still uncertain, mainly because things do not
compile yet.
I encourage you guys keep talking about this, but I'm going to step
out of the conversation if you don't mind.
Regards,
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Jon
Luis Lavena <luisl...@gmail.com> wrote:
> 2011/10/13 Dušan D. Majkić <dma...@gmail.com>:
> >> what's preventing MRI from emulating the fine multi-platform
> >> work being done by the https://github.com/joyent/libuv contributors?
> >
> > Node on Windows is supported by Joyent and Microsoft.
> > Oneclick Ruby on Windows is supported by Luis Lavena and Jon
> > Forums.
> >
>
> Hello,
>
> Chiming late to the party, but I wanted to share the following:
>
> Node is financially backed by Joyent and Microsoft. This means they
> get people paid to work on it, including work and improve it on any
> platform Joyent and Microsoft are interested node works on.
Yes, and that's great for both Node and it's underlying platform abstraction library libuv.
I'm more interested in *how* this relationship got started and what can be emulated to help MRI. The Joyent/Microsoft relationship didn't just magically happen; someone had an idea and took creative action in the face of pessimism and constraints.
More than likely someone thought it was crucial for the project to be a great multi-platform implementation. Maybe they saw a way to directly monetize the effort. Or maybe they recognized they would miss the opportunity to serve a sizeable chunk of the market by not building a multi-platform solution.
Then, they may have thought "Hey, we're *nix gurus, and while we're pretty good with the Windows API, we're not going to pull off true multi-platform support anytime soon by floundering about. We need to reach out and get some Windows gurus on board or our Windows implementation is going to be a zombie. What creative things we can we do?"
If so, that's true leadership. It's one thing to say "Yeh, that's a real need but I don't have the resources." and stop at "Patches welcome." but quite another to say "Ok, we don't currently have the resources, but this capability is *crucial* to the project. We're going to find another way to make it happen on a credible timeline." The latter shows craftsmanship and commitment to building great things.
I wonder how close this little tale is to the actual "magic" that started the ball rolling.
> To Ruby-Core, performance issues are not considered bugs, are
> considered features, because of that, fixing has taken really low
> priority.
>
> There is nobody interested in clean up Ruby to be a more multi
> platform friendly. Just the idea of change part of the 4K lines that
> is file.c will raise rage on Ruby-Core, specially since maintainers
> will no longer be familiar with the codebase and thus, no longer able
> to maintain it.
Sadly that's an example of a "language experimentation implementation" mindset rather than "best-of-breed production implementation" mindset. Both mindsets are important for innovation, but sadly for the MRI project, the language experimentation mindset seems to continue be the primary focus even after 15+ years.
Perhaps the mashup of ideas and real-world business needs from
http://blog.heroku.com/archives/2011/7/12/matz_joins_heroku
will lead to a better balance between the two mindsets and a focus on building an implementation with great GC, great tooling capabilities, and great multi-platform support.
> While Rubinius might promise a pure-Ruby alternative to this, it's
> state on Windows is still uncertain, mainly because things do not
> compile yet.
Windows support on Rubinius is a joke, and IMO, a strategic and purposeful joke.
I suspect EY sees very little short or medium term business potential in Rubinius Windows support and are focusing their limited resources on building and optimizing their infrastructure and other crown jewels. Being able to gracefully transition the bulk of their clients to Rubinius or JRuby rather than MRI may be Technical Job One. Bravo.
If you have a bit of a Machiavellian twist to you, you might see value in allowing Rubinius Windows support to slog along in zombie mode until you're able to determine whether there's an opportunity that you to capitalize on. Investing just enough to give yourself a future option, a "call" in the financial sense. Why do all the investing and make it freely available to your competitors but be unable to monetize it yourself because you're focused elsewhere?
But all this is speculation. And speculation is like the old saying regarding opinions. ;)
And the joke?
As Rubinius stares down 2.0.0, you still can't build a version for Windows while projects like libuv and Go are providing Windows support and binary builds *pre* v1.0! And I see no credible technical or marketing actions on EY's part to bring on additional resources and relationships to deliver a multiplatform solution in the short to medium term a la Joyent/Microsoft.
Regardless of whether the lack of Windows support is strategic or simply poor strategy and execution, I'm disappointed not to be able to kick the tires of the vaunted Rubinius VM, GC, and tooling capabilities by running it on Windows.
Uh oh, this has turned into a whale TL;DR, so here's my question relevant to MRI IO performance degradation.
Is there any value in cloning a git MRI repo with the sole focus to optimize for multi-platform (starting first with Windows) and can the effort attract experienced Windows devs and be sustained? A common meeting point for any work that you may already be doing in private. The focus would be similar to http://www.boost.org/ with the goal of folding all optimizations back upstream. Language changes would be verboten.
I don't like the idea for many reasons, but am interested in hearing other perspectives.
Jon
---
blog: http://jonforums.github.com/
twitter: @jonforums
I have always thought the actions of men are the best interpreters of their thoughts.
- John Locke
Dušan D. Majkić
> someone had an idea and took creative action in the face
> of pessimism and constraints.
Let's note a few things that "clicked into place" there.
* MS turned their focus to Cloud and JavaScript
* Node is server side JS with a lot of developer buzz
* Node is not in conflict with any of MS server products
* MS can resell Node (btw Apple iCloud is using Azure)
* Joyent is a company, with full-time devs
* Company makes easier to do business with.
* Company is viable target for potential acquisition.
This is a win-win-win for both of them.
If things go right, there will be Node on Azure, and MS support
both financial and in "manpower" from MS.
If things go wrong, project will fade, but it will be noted that
MS got involved and helped.
If things go great, Joyent will be Microsoft.
I can see the light (profits) at the end of that development tunnel.
Now let's look at "Ruby vs Ruby" case (eg. Heroku vs EY).
As a Ruby fan I find ruby divergence unpleasant.
But as a someone expecting to be paid, I do understand
that the best way to get development financed is to get
development close to end users.
This is what Ms/Joyent and Heroku/Matz and EY/JRuby/Rubinius
are doing. They all just keep close to the technology they sell.
They both see Windows devs as customers.
> can the effort attract experienced Windows devs and be sustained?
I have no idea.
In fact "experienced Windows dev" is propbably someone using C#
and EntityFramework in VisualStudio. And even them are hard to find.
We need inexperienced people, who want to tame the platform
and offer them a way to be more productive on WIndows. Those are
not hard to find.
> Is there any value in cloning a git MRI repo
Yes.
First value is centralized build repository. Even without patches.
Second value is to let Louis and you patch what you want in The Core.
Internal happiness got to have some value.
Third value is having repository for others to fork on Windows Ruby.
Even if nobody patch anything, it would be nice to have all about
Ruby on Windows in one public place.
Jon
>
>
> We need inexperienced people, who want to tame the platform
> and offer them a way to be more productive on WIndows. Those are
> not hard to find.
I strongly agree. For an optimization repo to succeed we need people with multiple experience levels who are passionate and persistent about building great things and not afraid to dive in and struggle for awhile while they gain experience.
While it will go faster if people can regularly contribute, the reality is real-life and other interests prevents this from happening. We're naive to think otherwise. One of our challenges is to set up the project structure that allows people to come and go and irregularly contribute while keeping things moving forward.
We also need experienced folks passionate in seeing a multi-platform MRI to help guide the efforts of those of us who are inexperienced in certain areas so we don't give up when things become too frustrating. Mentors. And not just Windows-only mentors even though the first target is solving the Windows 1.9 IO performance regression.
I hate to name names because I'm doing a disservice by not mentioning other people who've made significant contributions, but most recently Chuck Remes efforts with Shoes on Windows comes to mind as the type of experienced, passionate, and persistent person we'd like to attract.
> > Is there any value in cloning a git MRI repo
>
> Yes.
>
> First value is centralized build repository. Even without patches.
>
> Second value is to let Louis and you patch what you want in The Core.
> Internal happiness got to have some value.
>
> Third value is having repository for others to fork on Windows Ruby.
>
> Even if nobody patch anything, it would be nice to have all about
> Ruby on Windows in one public place.
In no particular order or priority, here are a few reasons why I'm not yet convinced an optimization repo is a good idea or sustainable. Are the concerns legitimate, and if so, how should they be addressed? What others should be considered?
1) Unrealistic expectations. Setting the bar too high or the timeline too short plants the seeds for a project to fizzle out. The bottom line is that MRI on Windows is good in many areas and much of the support structure is in place. Think of how many of the really useful native Gems are available in binary form *and* usable in source form via our DevKit. In two words, f'n awesome! And don't forget JRuby on Windows is fantastic and continually getting better.
2) "Forking" misperceptions. Ultimately the goal is to merge the optimizations upstream into ruby-core. As such, it's very important that the hard-working ruby-core developers view this repo positively rather than as a threat to be scorned and ignored. In our favor, these days with DVCS's like git (github) and hg (bitbucket) the term "fork" doesn't create as much bad blood as in the CVS/SVN days. That said, it's a very good idea to discuss this privately with Matz before any repo is made public.
3) Code divergence. It's highly likely that optimizations could dramatically change the source code layout as well as how the internals play together. Not only does this make it harder to merge upstream, but it increases the chances of breaking dependent code. For example, it may be hard to optimize these while maintaining compatibility: `file.c` at ~126K, `io.c` at ~276K, `transcode.c` at ~134K, and `win32/win32.c` and ~129K.
4) Enabling infrastructure. For example, at a minimum we need a common set of test/spec and benchmarking workloads so we're all focused on the key problem(s). I'd started https://github.com/jonforums/measurements with the intention of solving the benchmarking workload issue and optimizing for VTune profiling, but it's been AbandonWare for awhile. We'll need to create custom tooling similar to this right away. What other tooling is needed?
5) Enabling documentation. Clear documentation in the areas of compiling, debugging, and profiling tooling so that folks not familiar with the MinGW world don't waste a lot of time getting up to speed. Who's going to write this?
6) Attracting inexperienced and experienced people with the right mindset. At the end of the day nothing gets done without people committing tested and profiled code. And we can't expect just one person to do all the heavy lifting without burning out. What specific things should we do to attract the right people? ANNs to ruby-core and ruby-talk? ANNs to other MLs? Private emails to personal contacts? Approach MSFT? Approach Heroku/Salesforce? Other?
Frankly, all of this hinges upon whether enough people believe that investing in MRI on Windows is worth their time, and to some extent, the chances that ruby-core will merge the optimizations. I have nothing but the deepest respect for Brent's efforts at https://github.com/brentr/matzruby/tree/v1_8_7_352-mbari but my gut tells me that something similar may slow down attracting contributors to the optimization repo.
7) Specific goals and rules. We need just a few as too much ceremony is bad and just not any fun. But we absolutely need a laser-like focus on a very few key problems in order to make progress. Minimal dilution of efforts. And we need to be clear on areas in which contributions are not welcome, e.g. - language changes.
8) Project/repo naming. I'm *strongly* opposed to bringing a Ruby clone under our current GitHub https://github.com/oneclick/ organization. We're talking about multi-platform optimizations here and I see no value in taking the risk of creating confusion that the optimization repo is a "Windows-only" effort. I personally want all comers, regardless of whether their first OS love is OSX, Linux, or Windows. I want contributors who embrace todays multi-platform reality.
9) Funding, fund raising, and sponsorships. Free contributions of time, brainpower, and code are very much appreciated and respected. But we also need to find ways to sponsor people's efforts especially in the areas requiring hard-to-find expertise.
This is not only pragmatic, it's also very honorable on many levels. I'm not going to belabor the point, but a person's time is their life. Whether contributors would like to be "paid" in visibility, karma, prestige, personal learning, money, or something else, we'd like to be able to sponsor people's contributions in the currencies they value.
Jon
---
blog: http://jonforums.github.com/
twitter: @jonforums
Most people die of a sort of creeping common sense, and discover when it
Chuck Remes
On Oct 19, 2011, at 11:09 AM, Jon wrote:
>>> can the effort attract experienced Windows devs and be sustained?
>>
>> ...SNIP...
>>
>> We need inexperienced people, who want to tame the platform
>> and offer them a way to be more productive on WIndows. Those are
>> not hard to find.
>
> I strongly agree. For an optimization repo to succeed we need people with multiple experience levels who are passionate and persistent about building great things and not afraid to dive in and struggle for awhile while they gain experience.
>
> While it will go faster if people can regularly contribute, the reality is real-life and other interests prevents this from happening. We're naive to think otherwise. One of our challenges is to set up the project structure that allows people to come and go and irregularly contribute while keeping things moving forward.
>
> We also need experienced folks passionate in seeing a multi-platform MRI to help guide the efforts of those of us who are inexperienced in certain areas so we don't give up when things become too frustrating. Mentors. And not just Windows-only mentors even though the first target is solving the Windows 1.9 IO performance regression.
>
> I hate to name names because I'm doing a disservice by not mentioning other people who've made significant contributions, but most recently Chuck Remes efforts with Shoes on Windows comes to mind as the type of experienced, passionate, and persistent person we'd like to attract.
Thank you for the shout-out. I have been a long-time lurker on this list but only got involved with recipes and other RubyInstaller details recently. I finally got involved because I had an "itch to scratch" in the form of Shoes on Windows.
To attract more people as contributors, we need to find that "itch" and make it easy for them to scratch it.
I'll make a small pledge here to help with that effort, but first a little background. When I first started hacking on Shoes, the main requirement was to port some recipes from the old version to the new one. RubyInstaller had changed substantially since the last Shoes release, so the recipes were pretty far out of whack. The bulk of my time was spent on learning how the current recipes worked and translating the old ones to the new format.
So, the hill that I had to climb to be somewhat conversant with RubyInstaller (and trust me, I am *far* from an expert!) was to understand the recipes, the purpose of each task, what kind of magic was happening under the covers, etc. I will write some documentation for the wiki that will hopefully make that education simpler for future contributors. I had a few key "aha!" moments... anything I can do to reduce the time between "wtf?" and "aha!" will be useful.
> 1) Unrealistic expectations. Setting the bar too high or the timeline too short plants the seeds for a project to fizzle out. The bottom line is that MRI on Windows is good in many areas and much of the support structure is in place. Think of how many of the really useful native Gems are available in binary form *and* usable in source form via our DevKit. In two words, f'n awesome! And don't forget JRuby on Windows is fantastic and continually getting better.
Creating DevKit was a stroke of genius. By itself it has made life on Windows *so* much better. I use JRuby for all of my Windows production code, but I would still like there to be at least one other strong runtime on this platform.
> 2) "Forking" misperceptions. Ultimately the goal is to merge the optimizations upstream into ruby-core. As such, it's very important that the hard-working ruby-core developers view this repo positively rather than as a threat to be scorned and ignored. In our favor, these days with DVCS's like git (github) and hg (bitbucket) the term "fork" doesn't create as much bad blood as in the CVS/SVN days. That said, it's a very good idea to discuss this privately with Matz before any repo is made public.
I think getting patches upstream is easier now that Luis is a core committer for the Windows platform.
On your last point, I think a fork to make Ruby run better on Windows is actually *necessary.* I say that because of responses on ruby-core and tweets from core members oftentimes boil down to "show me why your patch is better." That is sometimes difficult to do in isolation. Plus, based on past ruby-core discussions I have seen quite a few Windows patches that were hundreds of lines long and touched a bunch of files. The core guys are very conservative and usually just reject those patches because they are too big to understand. What this says to me is that they don't trust their specs/tests (although they create lots of regressions with every patch release on unix... <sigh>).
A fork would allow some of these larger patches to "settle" and prove themselves. We could create a second RubyInstaller that delivers this patched version so folks in the wild can bang on it and help prove its worth.
> 3) Code divergence. It's highly likely that optimizations could dramatically change the source code layout as well as how the internals play together. Not only does this make it harder to merge upstream, but it increases the chances of breaking dependent code. For example, it may be hard to optimize these while maintaining compatibility: `file.c` at ~126K, `io.c` at ~276K, `transcode.c` at ~134K, and `win32/win32.c` and ~129K.
You have to break a few eggs to make an omelet. I think it is *ridiculous* that these files haven't been refactored already. Who can understand a file with over 100k lines in it? That's crazy.
This is where a project like Rubinius could potentially shine. It's a "green field" where the Windows platform code can be (hopefully) nicely factored into more intelligible and understandable units.
> 4) Enabling infrastructure. For example, at a minimum we need a common set of test/spec and benchmarking workloads so we're all focused on the key problem(s). I'd started https://github.com/jonforums/measurements with the intention of solving the benchmarking workload issue and optimizing for VTune profiling, but it's been AbandonWare for awhile. We'll need to create custom tooling similar to this right away. What other tooling is needed?
I know there was a long thread several months ago about the difficulty of creating debug builds for the runtime and C extensions. I don't know if this has been solved yet. If it has, then this is a great step forward on being able to debug SEGVs and other crashers on Windows.
Also, it may prove fruitful to get MRI building with LLVM. I have read that in certain cases it produces faster code than GCC and slower code in others. If LLVM continues to improve and produce more performant code, this may be an easy win for MRI. (I wouldn't bet the farm on this!)
> 5) Enabling documentation. Clear documentation in the areas of compiling, debugging, and profiling tooling so that folks not familiar with the MinGW world don't waste a lot of time getting up to speed. Who's going to write this?
I can and will help. I am far from expert here, but sometimes ignorance is a useful trait when writing documentation. Experts in a topic have a tendency to gloss over details that they assume everyone knows. The ignorant person will actually document these steps or details. :)
> 6) Attracting inexperienced and experienced people with the right mindset. At the end of the day nothing gets done without people committing tested and profiled code. And we can't expect just one person to do all the heavy lifting without burning out. What specific things should we do to attract the right people? ANNs to ruby-core and ruby-talk? ANNs to other MLs? Private emails to personal contacts? Approach MSFT? Approach Heroku/Salesforce? Other?
EY already supports JRuby & Rubinius by paying full-timers to work on those code bases. I would guess that they see their long-term future tied to the success of those runtimes and have waning interest in MRI.
Heroku is now one of several companies (!!) that employ Matz. I don't think that is going to help them. :)
I do believe that for Ruby on Windows to flourish that it needs some kind of corporate sponsorship. I don't know how to attract such a thing, however.
I think one of the most effective ways of drumming up interest is doing a short talk at a local user's group meetup. I have already spoken with the leader of the Chicago meetup about doing a presentation on *something* within the next few months. There's no reason it couldn't focus on Ruby on Windows.
> 7) Specific goals and rules. We need just a few as too much ceremony is bad and just not any fun. But we absolutely need a laser-like focus on a very few key problems in order to make progress. Minimal dilution of efforts. And we need to be clear on areas in which contributions are not welcome, e.g. - language changes.
Agreed 100%. I wish ruby-core would stop changing the language so much too!
> 8) Project/repo naming. I'm *strongly* opposed to bringing a Ruby clone under our current GitHub https://github.com/oneclick/ organization. We're talking about multi-platform optimizations here and I see no value in taking the risk of creating confusion that the optimization repo is a "Windows-only" effort. I personally want all comers, regardless of whether their first OS love is OSX, Linux, or Windows. I want contributors who embrace todays multi-platform reality.
I'm not certain I understand this point. How does attracting OSX & Linux users help with Windows optimization efforts?
> 9) Funding, fund raising, and sponsorships. Free contributions of time, brainpower, and code are very much appreciated and respected. But we also need to find ways to sponsor people's efforts especially in the areas requiring hard-to-find expertise.
My company actually paid Engine Yard to add the WIN32OLE support to JRuby, so corporate *paid* sponsorship is definitely helpful.
Perhaps we could pool money for bounties (or pledgies?). I'm not exactly flush with cash right now, but for something where my livelihood depends on good Ruby support I am certain I could scare up $500-$1000 a year for worthwhile efforts. If we can get a few dozen people to pledge some amount per year, perhaps we could steer some $$ towards programmers who have the skills we need.
> This is not only pragmatic, it's also very honorable on many levels. I'm not going to belabor the point, but a person's time is their life. Whether contributors would like to be "paid" in visibility, karma, prestige, personal learning, money, or something else, we'd like to be able to sponsor people's contributions in the currencies they value.
Good point. This may sound silly, but a quarterly (or annual) "trophy" that may or may not include $$ to recognize contributions and achievements in this space could prove useful. Maybe some of the money raised as pledges could be earmarked for such a thing.
cr
Jon
You're being modest and downplaying your experience and persistence.
> I will write some documentation for the wiki that will hopefully make that education simpler for future
> contributors. I had a few key "aha!" moments... anything I can do to reduce the time between "wtf?" and "aha!"
> will be useful.
Thank you for the offer and blank check...may need to take you up on it dependent on how this proceeds ;)
> > 2) "Forking" misperceptions. Ultimately the goal is to merge the optimizations upstream into ruby-core. As such, it's very important that the hard-working ruby-core developers view this repo positively rather than as a threat to be scorned and ignored. In our favor, these days with DVCS's like git (github) and hg (bitbucket) the term "fork" doesn't create as much bad blood as in the CVS/SVN days. That said, it's a very good idea to discuss this privately with Matz before any repo is made public.
>
> I think getting patches upstream is easier now that Luis is a core committer for the Windows platform.
>
> On your last point, I think a fork to make Ruby run better on Windows is actually *necessary.* I say that because of responses on ruby-core and tweets from core members oftentimes boil down to "show me why your patch is better." That is sometimes difficult to do in isolation. Plus, based on past ruby-core discussions I have seen quite a few Windows patches that were hundreds of lines long and touched a bunch of files. The core guys are very conservative and usually just reject those patches because they are too big to understand. What this says to me is that they don't trust their specs/tests (although they create lots of regressions with every patch release on unix... <sigh>).
>
> A fork would allow some of these larger patches to "settle" and prove themselves. We could create a second RubyInstaller that delivers this patched version so folks in the wild can bang on it and help prove its worth.
Makes sense. While I prefer *-experimental.7z archives for those who like wild-eyed adventures, I don't have a problem with *-experimental.exe installers. We'd likely need to tweak the recipes just a bit to ensure they're easy to build. I think Luis and I already have it set up to do this, but should check especially when it comes to setting id's in the registry to keep experimental installers from fouling the environment for supported installers.
> > 4) Enabling infrastructure. For example, at a minimum we need a common set of test/spec and benchmarking workloads so we're all focused on the key problem(s). I'd started https://github.com/jonforums/measurements with the intention of solving the benchmarking workload issue and optimizing for VTune profiling, but it's been AbandonWare for awhile. We'll need to create custom tooling similar to this right away. What other tooling is needed?
>
> I know there was a long thread several months ago about the difficulty of creating debug builds for the runtime and C extensions. I don't know if this has been solved yet. If it has, then this is a great step forward on being able to debug SEGVs and other crashers on Windows.
IIRC, our biggest headache was trying to build a gprof profiled version. I think we're OK with building debug versions, but should double check.
> Also, it may prove fruitful to get MRI building with LLVM. I have read that in certain cases it produces faster code than GCC and slower code in others. If LLVM continues to improve and produce more performant code, this may be an easy win for MRI. (I wouldn't bet the farm on this!)
I included this from the very beginning when I implemented the current DevKit functionality
https://github.com/oneclick/rubyinstaller/blob/master/config/compilers/001_dkcompiler_init.rb#L11
https://github.com/oneclick/rubyinstaller/blob/master/config/compilers/llvm.rb
and worked a couple build failures with Nobu IIRC so that you could build with LLVM v2.8. Sadly, I've not paid enough attention to keeping it fresh and I believe you can no longer build via something like
rake ruby19 local=... dkver=llvm-32-2.8
And the last time I looked, LLVM v2.9 didn't solve the issue. It likely just needs a bit of love and maybe a few tweaks to the current ruby-core configure.in, Makefile.in, cygwin/GNUmakefile.in trifecta. I've always been very happy with ruby-core's willingness to make these changes.
> > 5) Enabling documentation. Clear documentation in the areas of compiling, debugging, and profiling tooling so that folks not familiar with the MinGW world don't waste a lot of time getting up to speed. Who's going to write this?
>
> I can and will help. I am far from expert here, but sometimes ignorance is a useful trait when writing documentation. Experts in a topic have a tendency to gloss over details that they assume everyone knows. The ignorant person will actually document these steps or details. :)
Hehe...thanks and agreed. I think we tend to fight harder when we're first climbing the cliff.
> > 6) Attracting inexperienced and experienced people with the right mindset. At the end of the day nothing gets done without people committing tested and profiled code. And we can't expect just one person to do all the heavy lifting without burning out. What specific things should we do to attract the right people? ANNs to ruby-core and ruby-talk? ANNs to other MLs? Private emails to personal contacts? Approach MSFT? Approach Heroku/Salesforce? Other?
>
> ...SNIP...
>
> I think one of the most effective ways of drumming up interest is doing a short talk at a local user's group meetup. I have already spoken with the leader of the Chicago meetup about doing a presentation on *something* within the next few months. There's no reason it couldn't focus on Ruby on Windows.
Great idea. Hopefully you wouldn't have to bribe them with Ed Debevics or Uno's ;)
> > 8) Project/repo naming. I'm *strongly* opposed to bringing a Ruby clone under our current GitHub https://github.com/oneclick/ organization. We're talking about multi-platform optimizations here and I see no value in taking the risk of creating confusion that the optimization repo is a "Windows-only" effort. I personally want all comers, regardless of whether their first OS love is OSX, Linux, or Windows. I want contributors who embrace todays multi-platform reality.
>
> I'm not certain I understand this point. How does attracting OSX & Linux users help with Windows optimization efforts?
Technically, the diversity of experience and views helps minimize problems like Rubinius' vm/vm.exe (~187MB on Windows and not exporting required symbols causing build to fail, ~13MB on OSX and exporting symbols). While the first focus is on Windows IO performance, any mods absolutely *must* run great on OSX and Linux distros. This needs to be a non-negotiable mindset for anyone contributing to the repo.
That's not to say all platforms must be equivalent, but it's absolutely unacceptable to have regressions like
http://redmine.ruby-lang.org/issues/5056#note-14
For example, I care about Arch, Ubuntu, and Win7/8 but don't have the experience of say an Eric Wong. Even though he may not have specific Windows API expertise, I believe his experience and perspective would add tremendous value to IO optimizations. Similar for other OSX and Linux folks.
Politically, I'm strongly opposed to the repo being viewed as a Windows-only effort. It's a cross-platform optimization repo similiar in purpose to Boost. If the repo is ever viewed as Windows-only, the little attention ruby-core currently gives to Windows (a best efforts platform) issues and futures could diminish even more. Windows is not currently viewed as a platform important to MRI's success. Consequently, if one was looking to minimize support resources even further, one simply says "MRI Windows development and support is being done on another repo. Please take your issue there." Is that likely to happen? I doubt it, but I continue to be amazed by past and current decisions.
Jon
and can't seem to get above 45 MPH. The JRuby Mercedes is running
well, but may be a bit too much for some. Bob took the IronRuby Camry
on another Tijuana bender and we don't know if, or when, he'll return.
But we all cut Bob a little slack since the poor guy got named after
an OS. And no one's seen the mythical Rubinius Tesla talked about in
dimly lit, smoke filled back rooms.
Vinnie finally snapped. He just couldn't bear the thought of trading
in the old MRI and opened up the doors to his garage for any weekend
warrior mechanic wanting to tinker and get it back on the road.
http://thecodeshop.github.com/
http://groups.google.com/group/thecodeshop
Rumor has it that celebrity guest mechanic Linus Torvalds may stop by
with a hammer and see if he can help.
Jon