RFC: RubyInstaller, DevKit and all their challenges

[Luis Lavena]

Hello,

Over the past months I've been thinking on ways to improve current RubyInstaller

experience for users -- including DevKit -- aiming to make it more Windows-friendly.

While RubyInstaller in its current form has fulfilled my personal needs, I

acknowledge that, for newcomers or Windows-centric developers, its usage is

problematic and sometimes cryptic.

Below are the common list of complaints I often hear (via Twitter rants,

GitHub issues and personal hate emails delivered to my inbox).

I would appreciate your comments on such issues, specially if you had

experiences with other tools that have found balance to solve those issues.

---

RubyInstaller does not install inside `Program Files`

Also: installing inside a path with spaces.

The technicalities around this are big, ranging from UNIXism inherent to Ruby,

looks like `mkmf` (used to compile extensions inside gems), RubyGems and

Bundler, to name a few.

You can experience the same issues on Linux/OSX just by installing Ruby in

a directory with spaces.

Ruby, RubyGems and also Bundler have improved in this area, but 3rd party gems

might break due this.

Associated with it but no less important, is the nature of `Program Files`

and the need for elevation of privileges (UAC) dealing with it.

This forces most of all the users that had installed Ruby inside

`Program Files` to run *all* their commands as Administrators. We all

understand that such usage is not recommended.

Proposed changes

While we cannot fix all the gems that do not properly handle path with spaces,

we could attempt to put the basic foundation for those gems and make Ruby a

better Windows citizen.

• Recommend installation inside `Program Files`
  
• No longer install gems *along* Ruby's structure
  
• Adjust RubyGems defaults to use `CommonAppData` for gem installation
  
• Consider that platforms (`i386-mingw32` and `x64-mingw32`) can coexist
  
• Ensure both `Ruby\bin` and theoretical `Gems\bin` directories are in the  `PATH`
• Ensure *Start Command Prompt with Ruby* also uses those paths (for those with multiple versions)

Introducing those changes will allow installed gems be shared by different

users, avoid the need for elevation of privileges during installation and

last but not least, reusability across upgrades of Ruby.

As pointed by Rob Reynolds (ferventcoder), this will require some special

permissions to be applied to this shared location so both services and users

can access and write into this location.

Only drawback with this approach is compatibility with older (pre-Vista)

versions of Windows like XP or Server 2003, which uses path with spaces for

`CommonAppData` (not to mention Windows Server 2003 uses `AllUsersProfile`,

argh).

One last thing, there is only one issue with RubyGems, and is described in

the following pull request:

https://github.com/rubygems/rubygems/pull/942

---

DevKit, in all its versions and forms, is not an installer

DevKit installation experience has been the top complaint about RubyInstaller

usability issues.

Not being an installer, not knowing which one matches the version of Ruby

that was just installed, running cryptic commands in the command line and

not knowing what are we doing (or how to interpret the output).

This is just a summary of the common complaints about it.

Proposed changes

There are a few challenges around this and some compromises. Include the DevKit

on each release of RubyInstaller will bump our installers from 16MB to 120MB.

Not only that, will promote duplication of the DevKit itself by duplicating

the bytes transfered.

Instead of going that route, we should look into making the RubyInstaller

offer you the option to install the *Development Kit*, which will download a

defined package for that version of Ruby and proceed to install in a shared

location (similar to gems perhaps).

If the compiler was already installed (due a previous installation), then no

download and no installation is required.

At the time the Ruby package is generated, we can stamp the default

configuration (like we do with RubyGems), to include the specific version of

the DevKit required for this version.

We can attempt to load it and if is not found, then indicate about the issue

and propose the next command to be executed to solve it.

For those who want to built a portable Ruby that includes DevKit, we can

prioritize lookup by checking a `DevKit` folder relative to `Ruby\bin`

directory and use that instead.

If they wanted to use a custom/different DevKit, they could load it via

`devkitvars.bat` and avoid the entire check/lookup described above.

---

If you reached this point, thank you.

Above was the list and my initial thoughts on what believe are changes that

will improve this experience.

Please take your time to analyze it and let me know your thoughts.

Once again, thank you for your time, patience and interest in making Ruby for

Windows a better experience for everybody.

Regards.

PS: The same contents of this email are as Markdown available on this gist:

https://gist.github.com/luislavena/d59fe5daa71e5b4d9a99

-- 

Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[Luis Lavena]

Hello folks,

I want to clarify a bit the point related to DevKit.

Seems most of the comments I got around it has been in the lines of disk space usage and why bundling DevKit with RubyInstaller is necessary or what that could be a problem.

To better describe it, here is the answers to those comments:

Bundle DevKit with each RubyInstaller release is inefficient. Changes in DevKit versions are almost rare after a release, so bundling repeated packages between each version of RubyInstaller will just increase download time and bandwidth consumed by these downloads.

DevKit versions are also shared across versions. Ruby 1.8.7 and 1.9.3 shared DevKit based on GCC 4.5.2 while Ruby 2.0.0 (and possibly 2.1.x) will use DevKit 4.7.2.

While some might think this point is irrelevant, want to mention that *just* for latest 2.0.0-p481 release has been downloaded around 380000 (380K or 380 thousands times).

Installer size is 16MB, so I ask you do the math in relation to bandwidth cost to our providers if the installers went from that size to 120MB by including the duplicate DevKit on each version.

On the same subject, RubyInstaller's Amazon bill has been on a steady 100 USD/month for the past year. This amazon account is the one responsibly to offer alternate downloads for installers and currently DevKit releases (after RubyForge shutdown).

But all these details are trivial compared to the main issue: usability. The biggest complaint around DevKit has been the complexity of installation and the confusion to those that are not familiar with RubyInstaller, what it is and what its purpose.

While most of Ruby developers coming from environments like Linux or OSX are familiar with GCC, that cannot be translated to Windows developers outside the OpenSource area, even less what is MinGW or mingw-w64 and what they are different.

There are alternatives on how to obtain compiler toolchains based on GCC on Windows, but these are more obscure and complex than the ones DevKit currently offers.

I'm really sorry that my mention of the download size of the DevKit has blurred the point I was trying to make.

Thank you for your time.

[Fervent Coder]

Also, let's not forget the fun of the Windows File System Redirector. If you are curious on the background of what this means with Program Files vs redirection to Program Files (x86), see PUP-2984[1]. This may or may not affect RubyInstaller if it is always run inside a 64 bit process.

[1]: https://tickets.puppetlabs.com/browse/PUP-2984

[Roy Pardee]

-- 

FWIW, my opinion is that installing to 'Program Files' will not be worth the pain (especially all the "why doesn't this gem install/compile right?" questions you're sure to get).  My (no doubt naive) hope is that Microsoft is slowly repenting its spacey-directory-name ways (e.g., 'documents and settings' is now 'users'; 'Program Files' is now just 'programs'.

I love the idea of an integrated devkit download/install for rubyinstaller though--that would be very graceful, and likely to protect a lot of us from our own inattention.

Thanks for everything you do Luis!

[Luis Lavena]

Hello Roy,

Thank you for your comments.

On Mon, Aug 11, 2014 at 3:04 PM, Roy Pardee <rpa...@gmail.com> wrote:

FWIW, my opinion is that installing to 'Program Files' will not be worth the pain (especially all the "why doesn't this gem install/compile right?" questions you're sure to get).  My (no doubt naive) hope is that Microsoft is slowly repenting its spacey-directory-name ways (e.g., 'documents and settings' is now 'users'; 'Program Files' is now just 'programs'.

Please note that the suggestion is to install Ruby, not gems inside Program Files.

While most of us are quite used to install Ruby, Python or other tools/languages outside Program Files, most of the newcomers find that odd and force the installation location without knowing the consequences of those decisions.

I receive on a weekly basis reports of issues associated with installation inside Program Files.

For example, this is how non-Ruby experts see the installation recommendation:

https://twitter.com/mikehdt/status/497143020332924929

[Boško Ivanišević]

Even though I am not so active in this project lately I couldn't resist to give my two cents on this issue.

On Mon, Aug 11, 2014 at 9:13 PM, Luis Lavena <luisl...@gmail.com> wrote:

Hello Roy,

Thank you for your comments.

On Mon, Aug 11, 2014 at 3:04 PM, Roy Pardee <rpa...@gmail.com> wrote:

FWIW, my opinion is that installing to 'Program Files' will not be worth the pain (especially all the "why doesn't this gem install/compile right?" questions you're sure to get).  My (no doubt naive) hope is that Microsoft is slowly repenting its spacey-directory-name ways (e.g., 'documents and settings' is now 'users'; 'Program Files' is now just 'programs'.

Please note that the suggestion is to install Ruby, not gems inside Program Files.

While most of us are quite used to install Ruby, Python or other tools/languages outside Program Files, most of the newcomers find that odd and force the installation location without knowing the consequences of those decisions.

I receive on a weekly basis reports of issues associated with installation inside Program Files.

For example, this is how non-Ruby experts see the installation recommendation:

https://twitter.com/mikehdt/status/497143020332924929

I know that many non-Ruby experts and solely Windows based developers expect to find every installation in Program Files but I agree with Roy. It simply isn't worth the pain. Especially with Program Files and Program Fiels (x86) confusion.

Being solely developer on Windows only for years (before I moved to LInux and OS X beside Windows) I have never felt it as a problem if some application does not install in Program Files. If someone wants to to learn new programming language he should adopt rules that come with it. Doesn't golang recommend single GOPATH value to be set to keep all projects and external dependencies there even on OS X?

Luis, I know you constantly receive complains about this but if developer is so lazy to learn such a simple thing I really doubt he will adopt Ruby without expecting Ruby program to be compiled from VisualStudio. There are enough resources on the Web with detailed explanation about this topic, you have explained it numerous times here and everywhere else so everybody can learn and start using Ruby very quickly with really minimum effort compared to era before RubyInstaller.

Regards,

Bosko Ivanisevic

http://pragdevnotes.com

http://rubyonwindowsguides.github.io

[Justin Baker]

I'm going to skip the `Program Files` issue for now. I have some things to say, but I need to figure out how to say them in a relevant and constructive way.

I feel like the act of building that into an installer would require an enormous amount of effort. Unless maybe Inno has an easy way to build it and a failure fallback built in.

But I do think that it could be better. Downloading, installing, and using the DevKit all seem to have quite a few paths for error and frustration.

I do think that installing it in a way you're proposing installing gems would be a good idea, but I think that half of the problem is when people move things. >_<

I know we've talked about this before, but what about "gem"ifying all of those things instead of doing things inside of the installer?

I think it could be a more robust solution, hopefully completely replacing the solution we have now.

I'm pretty sure you already thought about this but I wrote a quick PoC on how it could be used to load the DevKit and build gems. 

I think building a gem to use config files would help reduce the install process by checking default locations for config and refusing to use DevKits that aren't the correct versions for the built Ruby could be robust. Plus it could provide path error-checking/sanitation while allowing the user to define the path.

The PoC is here: https://github.com/Azolo/poc-devkit

Justin

[Justin Baker]

On Sat, Aug 9, 2014 at 10:30 AM, Luis Lavena <luisl...@gmail.com> wrote:

RubyInstaller does not install inside `Program Files`

Also: installing inside a path with spaces.

The technicalities around this are big, ranging from UNIXism inherent to Ruby,

looks like `mkmf` (used to compile extensions inside gems), RubyGems and

Bundler, to name a few.

You can experience the same issues on Linux/OSX just by installing Ruby in

a directory with spaces.

Ruby, RubyGems and also Bundler have improved in this area, but 3rd party gems

might break due this.

Associated with it but no less important, is the nature of `Program Files`

and the need for elevation of privileges (UAC) dealing with it.

This forces most of all the users that had installed Ruby inside

`Program Files` to run *all* their commands as Administrators. We all

understand that such usage is not recommended.

Proposed changes

While we cannot fix all the gems that do not properly handle path with spaces,

we could attempt to put the basic foundation for those gems and make Ruby a

better Windows citizen.

• Recommend installation inside `Program Files`
  
• No longer install gems *along* Ruby's structure
  
• Adjust RubyGems defaults to use `CommonAppData` for gem installation
  
• Consider that platforms (`i386-mingw32` and `x64-mingw32`) can coexist
  
• Ensure both `Ruby\bin` and theoretical `Gems\bin` directories are in the  `PATH`

This may actually cause security issue. =/

Because of the nature of gems adding a `bin` folder where other services and users could write data to the `System PATH` could be a security risk if ever the user or program was compromised.

I feel like I don't have a good way to explain it without making it more confusing. I can say that it isn't generally a good idea to be able to add places with frequent written data without knowing exactly what is happening.

Of course, you could be talking about doing the same way node.js does it. The `System PATH` gets modified for the node `bin` folder and the `User PATH` gets modified for the npm `bin` folder (Which is just where ever `APPDATA` points). Since this varies by user it looks like npm doesn't have a fool proof solution. (If I'm wrong someone please correct me. I honestly feel like I'm missing something here)

 

• Ensure *Start Command Prompt with Ruby* also uses those paths (for those with multiple versions)

Introducing those changes will allow installed gems be shared by different

users, avoid the need for elevation of privileges during installation and

last but not least, reusability across upgrades of Ruby.

As pointed by Rob Reynolds (ferventcoder), this will require some special

permissions to be applied to this shared location so both services and users

can access and write into this location.

These last two paragraphs have some interesting consequences, but only with respect to services and user shared locations; not to upgrade reusability.

Just to reiterate, in **regards to usability across users and services**:

While being something that I really want, it goes back to the security risk of adding things to the `System PATH`. =/

Like I really want to say I know a way around it, but I don't.

There are certainly times and places to do it, I just don't want to do it to other people without them knowing the implications.

I think that if someone is looking for *global gem install* behavior that it should require administrator privileges or a UAC prompt.

Of course, if I'm the only one that feels this way then I know enough to change it for myself. =)

Personally, I think that something like a global install that requires administrator elevation and would install to the protected `Ruby\lib` and `Ruby\bin` directories would be great. Something that could be used as special use case while allowing the common case to be installing to user specific `GEM_X` directories.

But that would bring so many gem specific problems, not to mention weird changes to RubyGems defaults and probably slight changes to RubyGems itself. Probably not at all worth it, but worth it to bring up.

 

Only drawback with this approach is compatibility with older (pre-Vista)

versions of Windows like XP or Server 2003, which uses path with spaces for

`CommonAppData` (not to mention Windows Server 2003 uses `AllUsersProfile`,

argh).

 

One last thing, there is only one issue with RubyGems, and is described in

the following pull request:

https://github.com/rubygems/rubygems/pull/942

Justin

[Thierry]

Hello Luis,

First, I take this opportunity to thank you for your continued support for ruby on Windows, which saves my life every day. If you ever pass by Paris again, let me know: if you feel like it, I'll treat you to a good restaurant!

Regarding the changes your propose, I guess they are for the installer? I always use the 7-zip archives, so I do not know specifically what the installer experience is like. For what it's worth, I'll give here the way I see ruby on Windows: for me, it's like a Unix window in an MS-DOS world. Knowing somewhat Unix, I don't feel that odd: if fact, I like it. I even organize this "environment" differently from Windows: I set my HOME dir to /home/thierry, I install rubies in /ruby-xxx with a symlink from /ruby, I build the doc in HTML from the sources with RDoc, and I work extensively from the command line, with Rakefiles everywhere, and most of the time no spaces in paths. So for me, the current distribution is fine (in fact, fantastic). If you change it, would my way of downloading/working be still available? (I guess yes.)

Again, thanks for all your work.
Take care
-- Thierry

[Luis Lavena]

Hello Justin,

Much appreciated the feedback.

On Mon, Aug 11, 2014 at 6:50 PM, Justin Baker <azol...@gmail.com> wrote:

I feel like the act of building that into an installer would require an enormous amount of effort. Unless maybe Inno has an easy way to build it and a failure fallback built in.

But I do think that it could be better. Downloading, installing, and using the DevKit all seem to have quite a few paths for error and frustration.

InnoSetup have some extensions capable of downloading extra packages not bundled in the installer.

My idea around this was something like this:

Installer is *aware* of the DevKit it needs to download, and just download the compressed package if user wanted and no existing package was found in the common location defined to store those packages.

Offer at the command line (via a bundled gem or something), functionality to extract the package (if hasn't been extracted before) and perform the checks of the installation.

This gem can offer the RubyGems plugin mechanism to detect the installation of the DevKit and activation.

I do think that installing it in a way you're proposing installing gems would be a good idea, but I think that half of the problem is when people move things. >_<

It will be extremely hard to please everybody, the ida of the DevKits located in a common place will work for both cases, and those who want to bundle everything up (including the DevKit) could place it relatively to the Ruby installation.

 

I know we've talked about this before, but what about "gem"ifying all of those things instead of doing things inside of the installer?

What I had in my head was the following:

- Modify operating_system.rb to change defaults of gem installation (no longer deal with DevKit presence)

- Ship with RubyInstaller package a metadata file that indicates which DevKit version is used by the version.

- Build a gem that deals with both verification, extraction and post confirmation of DevKit installation

- Such gem will expand RubyGems via plugin to hook up DevKit lookup and activation

So this will be half/half: some modifications to the installer, but definitely will not be part of it (just bundled as a gem).

I think it could be a more robust solution, hopefully completely replacing the solution we have now.

I'm pretty sure you already thought about this but I wrote a quick PoC on how it could be used to load the DevKit and build gems. 

I think building a gem to use config files would help reduce the install process by checking default locations for config and refusing to use DevKits that aren't the correct versions for the built Ruby could be robust. Plus it could provide path error-checking/sanitation while allowing the user to define the path.

The PoC is here: https://github.com/Azolo/poc-devkit

Thank you again Justin, will look into this shortly and get back to you.

[Justin Baker]

On Saturday, August 9, 2014 10:30:33 AM UTC-5, Luis Lavena wrote:

Proposed changes

While we cannot fix all the gems that do not properly handle path with spaces,

we could attempt to put the basic foundation for those gems and make Ruby a

better Windows citizen.

• Recommend installation inside `Program Files`
  
• No longer install gems *along* Ruby's structure
  
• Adjust RubyGems defaults to use `CommonAppData` for gem installation
  
• Consider that platforms (`i386-mingw32` and `x64-mingw32`) can coexist
  
• Ensure both `Ruby\bin` and theoretical `Gems\bin` directories are in the  `PATH`
• Ensure *Start Command Prompt with Ruby* also uses those paths (for those with multiple versions)

Just had a thought about Gem installation. What about adjusting the defaults to use the `--user-install` flag, then adding that to the User `env:path`?

Then anyone could get back to the "old" behavior by adding `--no-user-install` to a `.gemrc`.

I don't know though, just an idea.

Justin

[Luis Lavena]

I like your idea, but I found a problem with default installation location.

Take the following example:

C:\Users\Luis>ruby -v

ruby 2.0.0p481 (2014-05-08) [i386-mingw32]

C:\Users\Luis>gem env path

C:/Users/Luis/.gem/ruby/2.0.0;C:/Users/Luis/Tools/Ruby/ruby-2.0.0-p481-i386-mingw32/lib/ruby/gems/2.0.0

Now, let's take same version, but for x64:

C:\Users\Luis>ruby -v

ruby 2.0.0p481 (2014-05-08) [x64-mingw32]

C:\Users\Luis>gem env path

C:/Users/Luis/.gem/ruby/2.0.0;C:/Users/Luis/Tools/Ruby/ruby-2.0.0-p481-x64-mingw32/lib/ruby/gems/2.0.0

In both cases, the user installation directory is .gem/ruby/2.0.0, independently of the platform.

If we make the default so all gems get installed into user directory, we need to take care that platform is also considered for it. Simply because ignoring it will cause binary extensions to produce issues when loaded.

I believe this (Windows) is the only platform that presents this issue where both i386 and x64 builds can coexist.

I've been playing with changes to operating_system.rb and usage of platform and ProgramData instead of Ruby installation directory:

https://gist.github.com/luislavena/093e79168e7608e15a62

In either case, both user-install directory *and* the gem directory inside ProgramData will need to be added to the PATH so executables in "bin" directory are accessible.

Having executables accesibles is the only thing that make this a bit more complicated (oh, and the binary extensions part too, hehe)

Cheers,

-- 

[Justin Baker]

That's actually really impressive. But you're right, setting the PATH in the installer seems a bit harder now. Any changes in the "assumed" environment may be hard to handle. I don't know though.

Overall, I approve. I don't see any surface problems.

Justin