Making the 64bit RubyInstaller a reality

[Jon]

While there's been activity, we've not yet had enough sustained focus to deliver a 64bit RubyInstaller.

The purpose of this thread is to identify and prioritize the details of what's left to finish, and ask for your help to make it happen in the next few months.

My hope is that once the tasks are identified and understood, those of you who are interested will be able to find some spare time in your busy days/nights/weekends to take ownership of one or more of the tasks and help to finally push 64bits out the door.

Please take a moment to review the topics, help prioritize and consolidate them down to specific, detailed tasks, and consider adopting one as your very own :)

First pass strawman in priority order:

1) Update the DevKit build recipes to create a reliable 64bit toolkit. The two current contenders are:

a) the v4.5.4 mingw-w64-1.0-bin_i686-mingw-* downloads from http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds/

b) the v4.6.2 downloads from http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/rubenvb/

Currently the (a) download is a cross-compiler requiring modifications before being useful. We either ensure the recipes use env vars to accomodated the prefixed executable names (UGLY), modify the recipes to build using `--host=x86_64-w64-mingw32` (NICER-BUT-STILL-UGLY), or remove the cross-compiling prefixes from the exes in the download's top-level bin/ dir (NICE).

Both of the downloads need to be enhanced with a working MSYS and other build tools similar to

https://github.com/oneclick/rubyinstaller/blob/master/config/devkit.rb#L27-64
https://github.com/oneclick/rubyinstaller/blob/master/config/compilers/mingw64.rb#L16-31

Tweaks likely needed to the `dk.rb` install script in the registry scanning section. Perhaps more.

2) Using the 64bit DevKit from (1) ensure the ruby_1_9_3 branch from https://github.com/ruby/ruby can be repeatedly and reliably built with no dependencies. Document and work any build errors on this list in preparation for submitting to ruby-core at http://redmine.ruby-lang.org/ if needed.

* Kou has submitted pull request https://github.com/oneclick/rubyinstaller/pull/70
* 64bit RubyInstallers will only be created for 1.9.3+ releases; no 64bit 1.8.x RubyInstallers.
* Ensure Ruby builds cleanly from its source repo using (1) and something similar to:

C:\>git clone https://github.com/ruby/ruby.git
C:\>cd ruby
C:\ruby>\DevKit64\devkitvars.bat
C:\ruby>git checkout -b ruby_1_9_3 origin/ruby_1_9_3
C:\ruby>sh -c "autoconf"
C:\ruby>mkdir build && cd build
C:\ruby\build>sh -c "../configure --enable-shared --disable-install-doc" # --host=x86_64-w64-mingw32 (??)
C:\ruby\build>make

3) Update dependency build recipes to use and create 64bit deps. Of the current dep recipes listed below, for 1.9 we build FFI, libyaml, OpenSSL, and Tcl/Tk from source while use pre-built binaries for the rest.

https://github.com/oneclick/rubyinstaller/tree/master/recipes/dependencies

While we could continue to build libraries from source, I'd prefer we find (or prebuild) correctly linked binary deps like those from

http://ftp.gnome.org/pub/gnome/binaries/win64/dependencies/

and archive them at https://github.com/oneclick/rubyinstaller/downloads similar to our current use of zlib.

That said, having the ability to build certain deps from source is a benefit to the project as it helps us control our own destiny wrt to updates. Perhaps we consider a RubyInstaller sub-project that simply builds deps from source. (mini_portile based as mentioned next?)

4) Update the main build recipes to support both 32 and 64bit builds. Currently, one can build 32bit packages similar to:

rake ruby19
rake ruby19 local=c:\ruby-git

While we have plenty of env var knobs (and a build override structure) to tweak the rake build process (eg - nogems=1, notk=1, nodeps=1, dkver=mingw64-32-4.5.4, etc) I'm not a fan of adding another env var. I prefer something like:

rake ruby19_64 local=c:\ruby-git

but could be convinced that just one more env var is OK for 64bit builds:

rake ruby19 local=c:\ruby-git x64=1

Updating the build recipes to ensure the top-level building CLI remains clean while allowing for both 32 and 64bit builds is likely a lot of work. We'll want to discuss this one in great detail to ensure we're not making the build recipes unstable or harder to maintain.

Luis is enhancing his https://github.com/luislavena/mini_portile project and it's unclear how much, if any, he wants to use from that project. 64bit support is a good opportunity to refactor some of our build processes, but I prefer 64bit over refactoring if refactoring means we overly complicate and delay 64bit availability. In that case, refactoring should occur later.

5) Update the Inno install scripts and corresponding recipes to provide separate 32 and 64bit Inno-based installers named *-x86.exe and *-x64.exe.

While the Inno installer is always a 32bit app, it's got settings used to properly install 64bit apps. I think the mods will be straight forward. The effort should mainly be testing and verification to ensure artifacts get placed in the correct directories, and the registry use is correct for the separate 32 and 64bit installers.

6) Update the http://rubyinstaller.org/downloads/ and http://rubyinstaller.org/downloads/archives pages to accomodate both 32 and 64bit downloads while cleaning up the layout to make things more clear.

I maintain a dev site at http://rubyinstallerdev.heroku.com/ for this purpose and will add anyone who takes ownership of this task as a collaborator. Luis tells me he's been playing with an idea on a revised layout for these pages, but we haven't had time to work on it further.

What else?

Jon

---
blog: http://jonforums.github.com/
twitter: @jonforums

"Anyone who can only think of one way to spell a word obviously lacks imagination." - Mark Twain

[Boško Ivanišević]

Jon than you for creating this detailed list of things that should be done in order to get x64 version of Ruby on Windows. At the moment I would like to give remark only for one of items - dependencies.

On Wed, Aug 17, 2011 at 6:42 PM, Jon <jon.f...@gmail.com> wrote:

3) Update dependency build recipes to use and create 64bit deps. Of the current dep recipes listed below, for 1.9 we build FFI, libyaml, OpenSSL, and Tcl/Tk from source while use pre-built binaries for the rest.

 https://github.com/oneclick/rubyinstaller/tree/master/recipes/dependencies

While we could continue to build libraries from source, I'd prefer we find (or prebuild) correctly linked binary deps like those from

 http://ftp.gnome.org/pub/gnome/binaries/win64/dependencies/

and archive them at https://github.com/oneclick/rubyinstaller/downloads similar to our current use of zlib.

That said, having the ability to build certain deps from source is a benefit to the project as it helps us control our own destiny wrt to updates. Perhaps we consider a RubyInstaller sub-project that simply builds deps from source. (mini_portile based as mentioned next?)

I would stick with building deps from the source. Relying on external sources for prebuilt dependencies will probably lead to dead-end if dependencies are not updated regularly. In that case we would be forced to get back to our recipes again. I would like to avoid this round trip. Probably there is a space for improvement of current build recipes so dependencies do not have to be built every time. Maybe we can cache binary versions somewhere? In case users/developers do not want to build all dependencies they can used cached versions, but if they want to upgrade version for any dependency they still can rebuild it from the scratch.

Moving these steps to sub-project increases complexity of build procedure. In that case users would have to first build all dependencies from that sub-project and then start building Ruby.

As I understand mini_portile project that Luis is making should address these issues so probably Luis will write few words about this too.

--
Regards,
Boško Ivanišević

http://pragdevnotes.com

[Phillip Gawlowski]

On Wed, Aug 17, 2011 at 6:42 PM, Jon <jon.f...@gmail.com> wrote:
>

> 2) Using the 64bit DevKit from (1) ensure the ruby_1_9_3 branch from https://github.com/ruby/ruby can be repeatedly and reliably built with no dependencies. Document and work any build errors on this list in preparation for submitting to ruby-core at http://redmine.ruby-lang.org/ if needed.

I'm willing to supply my little laptop as compilation node for a CI
system (or something similarly automated: Pull the source some time,
build, send logs to someone who can make sense of them), including
running ruby-spec tests.

Core2Duo @ 2x2GHz, 4GB RAM, Windows 7 Pro 64 Bit, in case it matters. :)

--
Phillip Gawlowski

phgaw.posterous.com | twitter.com/phgaw | gplus.to/phgaw

A method of solution is perfect if we can forsee from the start,
and even prove, that following that method we shall attain our aim.
              -- Leibniz

[Jon]

> Jon than you for creating this detailed list of things that should be done
> in order to get x64 version of Ruby on Windows. At the moment I would like
> to give remark only for one of items - dependencies.
>
> On Wed, Aug 17, 2011 at 6:42 PM, Jon <jon.f...@gmail.com> wrote:
>
> > 3) Update dependency build recipes to use and create 64bit deps. Of the
> > current dep recipes listed below, for 1.9 we build FFI, libyaml, OpenSSL,
> > and Tcl/Tk from source while use pre-built binaries for the rest.
> >
> >
> > https://github.com/oneclick/rubyinstaller/tree/master/recipes/dependencies
> >
> > While we could continue to build libraries from source, I'd prefer we find
> > (or prebuild) correctly linked binary deps like those from
> >
> > http://ftp.gnome.org/pub/gnome/binaries/win64/dependencies/
> >

> > and archive them at https://github.com/oneclick/rubyinstaller/downloadssimilar to our current use of zlib.

> >
> > That said, having the ability to build certain deps from source is a
> > benefit to the project as it helps us control our own destiny wrt to
> > updates. Perhaps we consider a RubyInstaller sub-project that simply builds
> > deps from source. (mini_portile based as mentioned next?)
> >
> > I would stick with building deps from the source. Relying on external
> sources for prebuilt dependencies will probably lead to dead-end if
> dependencies are not updated regularly. In that case we would be forced to
> get back to our recipes again. I would like to avoid this round trip.
> Probably there is a space for improvement of current build recipes so
> dependencies do not have to be built every time. Maybe we can cache binary
> versions somewhere? In case users/developers do not want to build all
> dependencies they can used cached versions, but if they want to upgrade
> version for any dependency they still can rebuild it from the scratch.

I like the idea of always being able to build as many of our deps from source as possible but using cached, pre-built binaries by default to speed up the build.

What do you think of:

1) add top-level visible (use `desc`) tasks `deps18` and `deps19` that use the existing internal `ruby18:dependencies` and `ruby19:dependencies` tasks (see deps trace via `rake -P | grep dependencies`) to build from source. Output artifacts placed in sandbox/ subdirs as done today.

2) we maintain pre-built binaries from (1) to https://github.com/oneclick/rubyinstaller/downloads

3) existing dep recipes/checkpoints modified to (a) check for existence of sandbox/<DEP-DIR> subdir (or a checkpoint file), and (b) if subdir/checkpoint doesn't exist, download pre-built binaries into downloads/ and extract them into sandbox/ similar to how we currently manage zlib. Like today, running `rake clean` will purge sandbox/ but leave the artifacts in downloads/ cached for a future use.

To build deps from source, run

rake deps19 ruby19 x64=1

and to use the pre-built binaries, run

rake ruby19 x64=1

For 32bit builds it's the same thing without the `x64` rake env var. Yes, I've changed my mind and think another env var for 64bit builds looks better than `rake deps19_64 ruby19_64`

[Boško Ivanišević]

I agree. More or less it will remain as it is now except adding top-level visible tasks, right?
 

2) we maintain pre-built binaries from (1) to https://github.com/oneclick/rubyinstaller/downloads

Exactly. That was my idea. Only concern is available disk space on GitHub. At the momen we are using approximately 25% and usage will rise once we start caching pre-built binaries. But for the start this certainly is good solution and we can use it till we reach allowed quota.
 

3) existing dep recipes/checkpoints modified to (a) check for existence of sandbox/<DEP-DIR> subdir (or a checkpoint file), and (b) if subdir/checkpoint doesn't exist, download pre-built binaries into downloads/ and extract them into sandbox/ similar to how we currently manage zlib. Like today, running `rake clean` will purge sandbox/ but leave the artifacts in downloads/ cached for a future use.

To build deps from source, run

 rake deps19 ruby19 x64=1

and to use the pre-built binaries, run

 rake ruby19 x64=1

For 32bit builds it's the same thing without the `x64` rake env var.  Yes, I've changed my mind and think another env var for 64bit builds looks better than `rake deps19_64 ruby19_64`

Like it ;-)

[Jon]

> On Aug 18 2011, 12:42 am, Jon <jon.for...@gmail.com> wrote:
> > While there's been activity, we've not yet had enough sustained focus to deliver a 64bit RubyInstaller.
> >
> > The purpose of this thread is to identify and prioritize the details of what's left to finish, and ask for your help to make it happen in the next few months.
>

> ...SNIP..
>
> What is the current progress on this? I think I have the necessary
> equipment to help but would need some clearer explanation in laymen
> terms. Also, Kou seemed to have built a 64 bit version successfully
> from what I can understand in the other thread. How does that relate
> to whatever problems that are outstanding now?

Hi,

Moving the discussion back to the RubyInstaller group...

Many of the original issues I summarized remain open and could use help from people interested in making a 64bit RubyInstaller happen. Regarding the 64bit DevKit issue, Luis has recently made a commit that allows one to build a 64bit compiler by using `dkver=tdm-64-4.6.1`. For example,

rake devkit dkver=tdm-64-4.6.1 7z=1

From my perspective, the highest priority issues are (a) get all the dependencies to reliably build as 64bit deps, and (b) update the build recipes to support both 32bit and 64bit. Once these are working, updating the Inno installer scripts come next then testing by multiple people in multiple scenarios.

Luis is the project lead and can better summarize the current status and plans for 64bit.

Would you provide a bit more detail on how you'd like to help, and which issues you think you could best help with?

Thanks for your interest and offer to help make it happen :)

Jon

---
Fail fast. Fail often. Fail publicly. Learn. Adapt. Repeat.
http://thecodeshop.github.com | http://jonforums.github.com/
twitter: @jonforums

[Luis Lavena]

Hello everyone, first apologies for top posting.

I've been silent on this topic mainly because wanted to flesh out a simpler alternative to our current recipe model.

For those who follow me on twitter, you saw a picture of me teasing about x64 ruby, including basic dependencies like zlib, openssl, libiconv, libffi and libyaml.
I haven't created recipes yet for tcl or tk, neither pdcurses because it seems do not compile from source, which is required for the 64bits build.

I promise will give and update over the weekend as I manage to package those builds.

More to come.

Sorry for top posting. Sent from mobile.

--
You received this message because you are subscribed to the Google Groups "RubyInstaller" group.
To post to this group, send email to rubyin...@googlegroups.com.
To unsubscribe from this group, send email to rubyinstalle...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyinstaller?hl=en.

[Pruss Wan]

Hi,

As a a primary windows user, I can help by attempting to build on Windows x64 machines that I have ready access to (XP, 7), or cygwin if that is called for. If any Visual Studio dependencies are needed, I should be able to get those as well.

[rogerdpack]

I guess the good news is that VLC has 64 builds these days:
http://nightlies.videolan.org/build/win64/
So hopefully the toolchain has improved in quality in recent years (/
dependency compatibility).