[ANN] dep_walker 1.0.0 Released
Boško Ivanišević
* <http://github.com/bosko/dep_walker>
The dep_walker is small utility gem that checks dependencies for native extensions
used by installed gems on Windows. If you are {RubyInstaller}[http://www.rubyinstaller.org]
user and have seen message box:
"This application has failed to start because <name_of_dll>.dll was not found.
Re-installing the application may fix this problem"
when you tried to use gem that has pre-built binariy extension, you've faced common
problem on Windows systems - missing dependency dll. Same error might occur even if
extension library was built during gem installation if all header files and libraries
are available to the build tools, but runtime dependencies are not present.
With dep_walker you can simply check all installed gems. Even more, if log is turned on,
gem will print out information where dependency is found on the system, so you can check
whether Ruby extension really uses correct version of required dll.
Changes:
### 1.0.0 / 2011-04-13
* 1 major enhancement
* Birthday!
--
Regards,
Boško Ivanišević
Luis Lavena
Seems that the generated gemspec has issues.
C:\Users\Luis>ruby -v
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-mingw32]
C:\Users\Luis>gem --version
1.8.1
C:\Users\Luis>gem install dep_walker
Building native extensions. This could take a while...
Invalid gemspec in
[C:/Users/Luis/.gem/ruby/x86-mingw32/1.8/specifications/dep_walker-1.0.0.gemspec]:
invalid date format in specification: "2011-05-07 00:00:00.000000000Z"
ERROR: While executing gem ... (NoMethodError)
undefined method `name' for nil:NilClass
C:\Users\Luis>gem spec dep_walker -r date
--- 2011-05-07 00:00:00 Z
Can you downgrade to 1.7.2 and re-release it as 1.0.1 ?
A bummer, but RubyGems is giving pretty annoying issues to lot of folks.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Luis Lavena
>
> Seems that the generated gemspec has issues.
>
http://rubyforge.org/tracker/index.php?func=detail&aid=29181&group_id=126&atid=575
--
Luis Lavena
Boško Ivanišević
2011/5/7 Boško Ivanišević <bosko.iv...@gmail.com>:
Seems that the generated gemspec has issues.
C:\Users\Luis>ruby -v
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-mingw32]
C:\Users\Luis>gem --version
1.8.1
C:\Users\Luis>gem install dep_walker
Building native extensions. This could take a while...
Invalid gemspec in
[C:/Users/Luis/.gem/ruby/x86-mingw32/1.8/specifications/dep_walker-1.0.0.gemspec]:
invalid date format in specification: "2011-05-07 00:00:00.000000000Z"
ERROR: While executing gem ... (NoMethodError)
undefined method `name' for nil:NilClass
C:\Users\Luis>gem spec dep_walker -r date
--- 2011-05-07 00:00:00 Z
Can you downgrade to 1.7.2 and re-release it as 1.0.1 ?
A bummer, but RubyGems is giving pretty annoying issues to lot of folks.
Thanks for reporting this issue with date format. I used hoe for creating gem and it looked really useful, but now I'm not sure whether hoe put date in this format or not. Currently I'm looking for the way to fix this (apart from manually editing gemspec file after .gem is made).
Luis Lavena
>
> I can make specific version for RubyGems 1.7.2 but problem is that, in that
> case, dep_walker cannot be used on version 1.8.1. While I was developing gem
> I was using soruce_index to get the list of all installed gems. Few days
> after that new RubyGems version is released and they completely broke
> dep_walker since source_index is depricated and all functionality was moved
> to Gem::Specification. That's why I decided to stick with newest RubyGems
> version, although I do not know whether it is the best decision.
>
Understood.
> Thanks for reporting this issue with date format. I used hoe for creating
> gem and it looked really useful, but now I'm not sure whether hoe put date
> in this format or not. Currently I'm looking for the way to fix this (apart
> from manually editing gemspec file after .gem is made).
>
Is a bug in Gem::Specification, is not you.
I'll see what can be done.
Boško Ivanišević
> I can make specific version for RubyGems 1.7.2 but problem is that, in thatUnderstood.
> case, dep_walker cannot be used on version 1.8.1. While I was developing gem
> I was using soruce_index to get the list of all installed gems. Few days
> after that new RubyGems version is released and they completely broke
> dep_walker since source_index is depricated and all functionality was moved
> to Gem::Specification. That's why I decided to stick with newest RubyGems
> version, although I do not know whether it is the best decision.
>
Is a bug in Gem::Specification, is not you.
> Thanks for reporting this issue with date format. I used hoe for creating
> gem and it looked really useful, but now I'm not sure whether hoe put date
> in this format or not. Currently I'm looking for the way to fix this (apart
> from manually editing gemspec file after .gem is made).
>
I'll see what can be done.
bbiker
On May 7, 8:57 am, Boško Ivanišević <bosko.ivanise...@gmail.com>
wrote:
told me that gem libxml-ruby-2.0.5 was missing libxml2-2.dll.
In fact it is present in libxml-ruby-2.0.5-x86-mingw32\lib\libs
together with libz.dll and libiconv-2.dll.
dep_walker --check libxml-ruby return that lib/1.8/libxml_ruby.so and
lib/1.9/libxml_ruby.so were missing
rather inconsistent searches and results ... depending on the query.
-a and -c did different searches -a produced an incorrect response
-c search for non existent paths
I think that it is time to go to the drawing board!
bbiker
ren...@nc.rr.com
bbiker
On May 8, 4:25 am, bbiker <ren...@nc.rr.com> wrote:
> On May 7, 8:57 am, Bo¹ko Ivani¹eviæ <bosko.ivanise...@gmail.com>
> Tested dep-walker on my gems : dep_walker -a
>
> told me that gem libxml-ruby-2.0.5 was missing libxml2-2.dll.
> In fact it is present in libxml-ruby-2.0.5-x86-mingw32\lib\libs
> together with libz.dll and libiconv-2.dll.
>
> dep_walker --check libxml-ruby return that lib/1.8/libxml_ruby.so and
> lib/1.9/libxml_ruby.so were missing
>
> rather inconsistent searches and results ... depending on the query.
> -a and -c did different searches -a produced an incorrect response
> -c search for non existent paths
>
> I think that it is time to go to the drawing board!
>
> bbiker
>
> ren...@nc.rr.com
were non-existent paths .. they do exist and are present
dep_walker stated that they missing when in fact they do exis6
Boško Ivanišević
1. Current process executable directory
2. Current directory
3. Windows system directory
4. Windows directory
5. Directories listed in the path
dep_walker currently checks directories in the path and if it doesn't find dll reports that extension library is missing dependencies.
libxml-ruby obviously uses explicit linking. In that case extension library explicitly loads dll and it would be very hard to find out from which directory extension library loads dll. dep_walker could recursively search all gem's folders but that would be more like guessing.
dep_walker --check libxml-ruby return that lib/1.8/libxml_ruby.so and
lib/1.9/libxml_ruby.so were missing
rather inconsistent searches and results ... depending on the query.
-a and -c did different searches -a produced an incorrect response
-c search for non existent paths
I think that it is time to go to the drawing board!
Luis Lavena
>
>
> On May 7, 8:57 am, Boško Ivanišević <bosko.ivanise...@gmail.com>
>
>
> told me that gem libxml-ruby-2.0.5 was missing libxml2-2.dll.
> In fact it is present in libxml-ruby-2.0.5-x86-mingw32\lib\libs
> together with libz.dll and libiconv-2.dll.
>
That is because these DLLs are not in the PATH, the gem is doing some
tricks to place them in PATH before loading the extension.
> dep_walker --check libxml-ruby return that lib/1.8/libxml_ruby.so and
> lib/1.9/libxml_ruby.so were missing
>
Because of the above DLL issue I'm mentioning above.
> I think that it is time to go to the drawing board!
>
?
Are you saying that dep_walker do not work? Or because that particular
gem Bosko will need to rework and consider something that is a bad
practice into account?
Sorry, but the tone of that statement sound harsh.
cfis
*not* put them on your path. I decided to leave that up to the user.
If you need to use the provided binaries then move them to someplace
on your path. The gem used to mess around with the PATH variable, but
I took that out in the 2.x series.
Thought I added that to the readme, will have to check.
Thanks,
Charlie
Luis Lavena
> Just to be clear, libxml-ruby provides the prebuilt binaries but does
> *not* put them on your path. I decided to leave that up to the user.
> If you need to use the provided binaries then move them to someplace
> on your path. The gem used to mess around with the PATH variable, but
> I took that out in the 2.x series.
Ahh, thank you for the clarification, I thought was still doing that.
> Thought I added that to the readme, will have to check.
>
You did:
"If you are running Windows, make sure to install the Win32 RubyGem
which includes prebuilt extensions for Ruby 1.8 and Ruby 1.9. These
extensions are built with MinGW32 against libxml2 version 2.7.8, iconv
version 1.13 and zlib version 1.2.5. Note these binaries are available
in the liblibs directory. To use them, put them someplace on your
path."
Is me not reading the instructions.
Thank you.
Charlie Savage
While on this subject, has the RubyInstaller adopted a "best practice"
for handling this issue? I'm happy to update the gem to handle this in
standard way (if there is a standard way).
Also, it makes sense for the RubyInstaller not to have libxml2 as part
of the base install (since obviously it comes in a gem), but I assume it
includes zlib and iconv binaries (sorry, haven't checked)? Should I
take those out of the libxml-ruby gem?
Thanks,
Charlie
On 5/9/2011 11:11 AM, Luis Lavena wrote:
> On Mon, May 9, 2011 at 1:24 PM, cfis<cf...@savagexi.com> wrote:
>> Just to be clear, libxml-ruby provides the prebuilt binaries but does
>> *not* put them on your path. I decided to leave that up to the user.
>> If you need to use the provided binaries then move them to someplace
>> on your path. The gem used to mess around with the PATH variable, but
>> I took that out in the 2.x series.
>
> Ahh, thank you for the clarification, I thought was still doing that.
>
>> Thought I added that to the readme, will have to check.
>>
>
> You did:
>
> "If you are running Windows, make sure to install the Win32 RubyGem
> which includes prebuilt extensions for Ruby 1.8 and Ruby 1.9. These
> extensions are built with MinGW32 against libxml2 version 2.7.8, iconv
> version 1.13 and zlib version 1.2.5. Note these binaries are available
> in the liblibs directory. To use them, put them someplace on your
> path."
>
> Is me not reading the instructions.
>
> Thank you.
--
Charlie Savage
http://cfis.savagexi.com
Jon
> includes zlib and iconv binaries (sorry, haven't checked)?
Ruby18.dependencies == [ :gdbm, :iconv, :openssl, :pdcurses, :zlib ]
Ruby19.dependencies == [ :ffi, :gdbm, :iconv, :openssl, :pdcurses, :yaml, :zlib ]
https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L35-36
https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L68-69
Jon
---
blog: http://jonforums.github.com/
twitter: @jonforums
Luis Lavena
> Luis,
Hello Charlie, good you asked.
> While on this subject, has the RubyInstaller adopted a "best practice" for
> handling this issue? I'm happy to update the gem to handle this in standard
> way (if there is a standard way).
>
As you know, there has been no standard way to handle DLL dependencies
except for WinSxS and manifest, something we are not using.
You also knew what was my reasoning against playing with PATH to load
these DLLs, and RubyGems didn't had a mechanism to support this (nor
it will ever have since is way too Windows-specific)
I've been recommending to gem authors use post_install_message to
advertise what do to after the gem installation (copy binaries, place
them somewhere, etc). But keep reading...
> Also, it makes sense for the RubyInstaller not to have libxml2 as part of
> the base install (since obviously it comes in a gem), but I assume it
> includes zlib and iconv binaries (sorry, haven't checked)? Should I take
> those out of the libxml-ruby gem?
>
zlib and iconv came with RubyInstaller because are part of the bundled
extensions of Ruby stdlib. If libxml was part of Ruby, then bundling
libxml2 would be part of it.
I've been against bundling DLLs for things that are not part of
RubyInstaller itself because that increases the burden of maintaining
the installers. As soon you release it (the installer) and a new
version of the DLL gets out, you need to update the installer again.
But, things slightly changed since then.
Bundler got massive adoption and bundler decided to silently chew all
your post_install messages, so people that uses any gem with their
projects on Windows will fail to get any message about what do to or
where to get these files.
Because of that, people using Bundler raised lot of questions on
Stackoverflow, this list and other places on the web (twitter) about
why things didn't work.
So, rant on a side, seems that rely on post_install_message do not
work, at least not reliably.
I'm been helping out the TinyTDS (SQL Server support based on FreeTDS
library) to provide binaries for Windows, and contrary to place DLLs
and advertise a post install message, I've recommended to them do a
static library build.
Which I know has it's own set of drawbacks (like building against
different version of libiconv and using them on the same executable)
but is the less worse scenario in the long run.
As for RubyInstaller own provided DLLs, since we are not shipping with
headers or link libraries for them, we might end killing them as
shared objects and move to the static model to reduce any possible
zlib1.dll or other DLL conflicts.
Thoughts?
Boško Ivanišević
Luis Lavena
> So this means you are not explicitly loading these dlls but leave to user to
> copy them somewhere to the path prior to using gem. Am I right, or I missed
> something?
>
Seems correct to me, is advertised in the README so the user of
libxml-ruby should have copied these DLLs prior checking with
dep_walker.
Charlie Savage
>> of the base install (since obviously it comes in a gem), but I assume it
>> includes zlib and iconv binaries (sorry, haven't checked)?
>
> Ruby18.dependencies == [ :gdbm, :iconv, :openssl, :pdcurses, :zlib ]
> Ruby19.dependencies == [ :ffi, :gdbm, :iconv, :openssl, :pdcurses, :yaml, :zlib ]
>
>
> https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L35-36
> https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L68-69
Thanks for the link.
So the zlib version is the same I'm including, but iconv is different.
I'm using the latest from gnu (1.13.1), but the ruby installer seems to
be using a really old version based on the above link (1.9.2-1, 2004??)
from gnuwin32.
Should I be worried? Building iconv on windows with msys/mingw works
well enough - can RubyInstaller move the latest version?
Sorry if this has all been discussed before....
Thanks,
Charlie
Charlie Savage
> user to copy them somewhere to the path prior to using gem. Am I right,
> or I missed something?
You are correct.
Charlie
Boško Ivanišević
2011/5/9 Boško Ivanišević <bosko.iv...@gmail.com>:
> So this means you are not explicitly loading these dlls but leave to user toSeems correct to me, is advertised in the README so the user of
> copy them somewhere to the path prior to using gem. Am I right, or I missed
> something?
>
libxml-ruby should have copied these DLLs prior checking with
dep_walker.
Jon
Good point. Do you have a link for a maintained iconv binary package? I'm only aware of the gnuwin32 one we currently use and what appears to be an older version at http://ftp.gnome.org/pub/gnome/binaries/win32/dependencies/
We should look at upgrading and that probably means creating a new dependency recipe to build from source.
Of course, now that you've started with this, you're tempting those of us of the Must-Have-The-Lastest-Revision-Clan down the slippery slope on the other deps ;)
Luis Lavena
>
> Thanks for the link.
>
> So the zlib version is the same I'm including, but iconv is different.
>
> I'm using the latest from gnu (1.13.1), but the ruby installer seems to be
> using a really old version based on the above link (1.9.2-1, 2004??) from
> gnuwin32.
>
> Should I be worried? Building iconv on windows with msys/mingw works well
> enough - can RubyInstaller move the latest version?
>
At the time libiconv support was added to RubyInstaller recipes, I
decided to opt for existing binaries than compile myself.
Nowdays, several libraries have been compiled from scratch, so the
modifications wouldn't be too much.
> Sorry if this has all been discussed before....
>
No need to apologize, actually hasn't been and wanted to bring the
topic long time ago.
Everybody here knows that I'm lazy, so at the time libiconv support
was added to RubyInstaller (to allow iconv to work) I opted for the
binaries instead of building from source, because was painful.
That was back in 2008...
Since then, lot of things improved on the build recipes structure to
better support building from source. All that thanks to contributions.
Nowdays, we build libffi and libyaml without a glitch.
Taking what we do for libyaml (that requires zero patching) and do the
same for libiconv is a matter of 20 minutes patch and 30 minutes test
that everything works.
Just take this:
https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L198-207
Change to something like this:
https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L168-181
And change the recipe from this:
https://github.com/oneclick/rubyinstaller/blob/master/recipes/dependencies/iconv.rake
To this:
https://github.com/oneclick/rubyinstaller/blob/master/recipes/dependencies/yaml.rake
But there is one small detail: introducing the libiconv change could
affect upgrade of the same installation.
Something to take into account when doing a release, but shouldn't be
a stopper for not going ahead and do it :)
But since then (2008) and all the iterations on the recipes, still
feels complicated.
A glimpse of what I've been working to improve this can be seen here:
https://github.com/rails-sqlserver/tiny_tds/blob/master/tasks/ports.rake#L23-32
But it is not ready to replace what we do for RubyInstaller, yet.
Pull requests/patches for newer iconv support are welcome :)
Cheers,
Charlie Savage
>
> Good point. Do you have a link for a maintained iconv binary package?
I just build it myself from source.
I'm only aware of the gnuwin32 one we currently use and what appears to
be an older version at
http://ftp.gnome.org/pub/gnome/binaries/win32/dependencies/
>
> We should look at upgrading and that probably means creating a new dependency recipe to build from source.
>
> Of course, now that you've started with this, you're tempting those of us of the Must-Have-The-Lastest-Revision-Clan down the slippery slope on the other deps ;)
Hah - so true...
Charlie
bbiker
xml2-2.dll.
I used libxml-ruby.
Now I tried it for myself and had no problem. No error message and no
notice about a read me. So how am I supposed to know that I need to
copy files from one location to another?
Then I read about dep_walker .. it sounded interesting and wanted to
check if I had any dependencies problem with my gems.
dep_walker -a (check all gems) returned that libxml-ruby was
missing a .dll file.
dep_walker -c libxml-ruby returned returned that libxml-ruby was
missing a .so file.
Now I expected the response from both queries .. definitely not two
different responses.
Now I am not for sure a Ruby guru.
Then I looked through the libxml-ruby files and found both of the
"missing" files were indeed present.
My conclusion was dep_walker was not up to par
1. Search for different files depending on how the query was posed.
2. Reported files were missing when actually they were present.
To me, the above stated dep_walker was not ready for prime time.
Perhaps a harsh but realistic conclusion from a first time user of
dep_walker. You tell me something is missing then it is missing.
Obviously I was not aware the problem resides in where the files were
located as opposed to where they were supposed to be.
The topic of this thread has changed from Dep_Walker to Handling
Binary Dependencies. So I am going to say my 2-cents. I am an
average user of Ruby and at best I consider myself at the intermediate
level ( I do have some ego).
I am sure that all are aware users will not RTFM unless their backs is
against a wall and a gun pointed to their head.
It is my understanding that a *.gemspec file has a space where post-
installation message can be posted. I looked at the libxml-ruby's
gemspec and there is none.
I just have now glanced at the README.rdoc. It is rather extensive and
does provide instructions that need to done after installation. It
does raise more questions that it answers.
Another thing I notice by just glancing at the README.rdoc, that
libxm-ruby is not out of the box ready to use. The README.rdoc needs
to be studied!!!!
Now as an average user, I do not make a point to dig into the innards
of a gem unless I have problem and the gem is written in Ruby ... I
have long forgotten C.
In the least, I would expect a post-installation message warning me of
the dire circumstances if I failed to read the README.doc and follow
its instructions.
A slightly off-topic but still related. ocra will have a hell of a
difficult time to package the libxml-ruby gem as part of an "exe"
appl with information provided in the .gemspec.
I relealize that you all are above my head and thank you your allowing
my 2 cents.
bbiker
ren...@nc.rr.com
Charlie Savage
> As you know, there has been no standard way to handle DLL dependencies
> except for WinSxS and manifest, something we are not using.
Yeah - I thought so but wanted to double check and see if anything has
changed.
> You also knew what was my reasoning against playing with PATH to load
> these DLLs, and RubyGems didn't had a mechanism to support this (nor
> it will ever have since is way too Windows-specific)
Yes.
> I've been against bundling DLLs for things that are not part of
> RubyInstaller itself because that increases the burden of maintaining
> the installers. As soon you release it (the installer) and a new
> version of the DLL gets out, you need to update the installer again.
I agree with all the above.
> But, things slightly changed since then.
>
> Bundler got massive adoption and bundler decided to silently chew all
> your post_install messages, so people that uses any gem with their
> projects on Windows will fail to get any message about what do to or
> where to get these files.
>
> Because of that, people using Bundler raised lot of questions on
> Stackoverflow, this list and other places on the web (twitter) about
> why things didn't work.
>
> So, rant on a side, seems that rely on post_install_message do not
> work, at least not reliably.
>
> I'm been helping out the TinyTDS (SQL Server support based on FreeTDS
> library) to provide binaries for Windows, and contrary to place DLLs
> and advertise a post install message, I've recommended to them do a
> static library build.
>
> Which I know has it's own set of drawbacks (like building against
> different version of libiconv and using them on the same executable)
> but is the less worse scenario in the long run.
Ok.
> As for RubyInstaller own provided DLLs, since we are not shipping with
> headers or link libraries for them, we might end killing them as
> shared objects and move to the static model to reduce any possible
> zlib1.dll or other DLL conflicts.
>
> Thoughts?
This is going to be a bit rambly - obviously there is no good answer
here. On one hand you could make everyone compile everything statically
and provide binaries with each gem. On the other hand, you could
provide a prepacked dev environment (Devkit) that has all the necessary
include files and libs and then you compile everything.
Or you could try the middle ground - provide a one click installer that
has some binaries but not all. But you end up with issues like this -
libxml is compiled against a newer version of iconv that the one click
installer doesn't support. So the middle ground seems hardest to me.
I would lean towards the full dev environment approach because I doubt
you'll get gem authors to provide statically compiled binaries for
windows users.
If there was just one mingw project, I'd adopt mingw-get and then have
the ruby one click installer basically:
* Run a bunch of mingw-get commands to install the base system (gcc,
zlib, msys, etc).
* mingw-get already has mingw version of iconv and zlib, so nothing to
do there but install them
* Package ruby up to be installable via mingw-get ruby18. or mingw-get
ruby19
So at least this way some work is offloaded to the mingw community. And
you have a well defined base system.
But then you of course end up at the same place - for libxml-ruby, what
does it do with libxml2.dll? Choices:
1. Tell user to go download, build and install libxml2 (many users won't
have the knowledge to do that, I don't thing its workable)
2. Create a mingw-get libxml2 package (best choice, but no gem author is
going to do this, not workable)
3. Provide a binary and munge paths (like libxml-ruby used to)
4. Provide a binary and copy to ruby base directory (like the old
version of the one click installer)
5. Provide a binary and don't munge paths, basically make users do #4
manually (like libxml-ruby does now)
None of #3, #4 or #5 are great - I can't say I really have a preference.
But it would be nice if the OneClickInstaller picked one and then
called it "best practices" so at least every user knows what to expect.
And then of course there are two Mingw project, making it even harder to
deal with (you have to stick with your own installer and lose all the
work that has gone into ming-get, bleh).
Thanks,
Charlie
Boško Ivanišević
I agree that resulting message is definitely confusing. I will fix it in the next version. However dep_walker is not searching for different files. In both cases it was checking same extension libraries and found that libxml2-2.dll is missing. Just the report message is obviously misleading.
2. Reported files were missing when actually they were present.
Present DLLs on the system doesn't mean that they will be used. Actually they will if they are on the path or explicitly loaded (see my previous message about rules for DLLs loading).
To me, the above stated dep_walker was not ready for prime time.
Perhaps a harsh but realistic conclusion from a first time user of
dep_walker. You tell me something is missing then it is missing.
Obviously I was not aware the problem resides in where the files were
located as opposed to where they were supposed to be.
--
Jon
> > headers or link libraries for them, we might end killing them as
> > shared objects and move to the static model to reduce any possible
> > zlib1.dll or other DLL conflicts.
> >
> > Thoughts?
To be clear, you're only talking about making the relevant lib/ruby/1.9.1/i386-mingw32/*.so Ruby built-in extensions static, correct? For example, new static i386-mingw32/{dbm.so,curses.so} would not be dependent upon {dbm3.dll,pdcurses.dll} in PATH.
If we move to static built-in Ruby extensions, is your major concern that we could break code and any 3rd-party native Ruby extensions that are explicitly depending upon these DLLs currently included in <RUBYINSTALLER>/bin/ and presumably on PATH?
C:\ruby192>dir bin\*.dll
05/01/2011 08:03 PM 48,640 dbm3.dll
05/01/2011 08:03 PM 62,464 gdbm3.dll
05/01/2011 08:03 PM 2,082,385 libeay32-1.0.0-msvcrt.dll
05/01/2011 08:03 PM 978,432 libiconv2.dll
05/01/2011 08:03 PM 2,030,592 msvcrt-ruby191.dll
05/01/2011 08:03 PM 114,688 pdcurses.dll
05/01/2011 08:03 PM 438,545 ssleay32-1.0.0-msvcrt.dll
05/01/2011 08:03 PM 104,448 zlib1.dll
> This is going to be a bit rambly - obviously there is no good answer
> here. On one hand you could make everyone compile everything statically
> and provide binaries with each gem.
Do you think a gem author would ever choose to provide the relevant dependencies as private DLLs? For example, would a porky binary gem having lib/blah/{1.8,1.9}/zlib1.dll alongside its lib/blah/{1.8,1.9}/blah.so work?
> On the other hand, you could
> provide a prepacked dev environment (Devkit) that has all the necessary
> include files and libs and then you compile everything.
Assuming RubyInstaller stays as-is with the dependency DLLs in <RUBYINSTALLER>/bin/, are you saying the headers for those dependency DLLs should be added to one of the DevKit's include/ (probably <DEVKIT>/mingw/include) subdirs and then a 3rd-party gem install would use the DevKit to compile against those headers and link against the dependency DLLs in <RUBYINSTALLER>/bin?
What do you see for the DevKit if RubyInstaller goes static for all the built-in Ruby extensions?
> Or you could try the middle ground - provide a one click installer that
> has some binaries but not all. But you end up with issues like this -
> libxml is compiled against a newer version of iconv that the one click
> installer doesn't support. So the middle ground seems hardest to me.
Isn't this the scenario we have today? If the libxml gem is linked against a newer libiconv (hypothetically, lets say libxml uses a new function not in the old libiconv) than the one on PATH from RubyInstaller, we should expect problems, right?
> I would lean towards the full dev environment approach because I doubt
> you'll get gem authors to provide statically compiled binaries for
> windows users.
Could rake-compiler be updated to make this a no-brainer for gem authors building Windows binary gems on Linux/OSX?
> If there was just one mingw project, I'd adopt mingw-get and then have
> the ruby one click installer basically:
>
> * Run a bunch of mingw-get commands to install the base system (gcc,
> zlib, msys, etc).
> * mingw-get already has mingw version of iconv and zlib, so nothing to
> do there but install them
> * Package ruby up to be installable via mingw-get ruby18. or mingw-get
> ruby19
Or update the existing installer to download and integrate all the DevKit artifacts. Currently we have a 2+ step process in which you download the installer/archive and install it, then you download the DevKit, do some basic YAML config and run a Ruby script to infect your roobies with DevKit gem install overrides. While not as nice as a smarter installer, the DevKit's dk.rd Ruby install script is a lot nicer (for maintainers) than InnoSetup's Pascal scripting. Here comes the "why autotools" point... ;)
> But then you of course end up at the same place - for libxml-ruby, what
> does it do with libxml2.dll? Choices:
>
> 1. Tell user to go download, build and install libxml2 (many users won't
> have the knowledge to do that, I don't thing its workable)
> 2. Create a mingw-get libxml2 package (best choice, but no gem author is
> going to do this, not workable)
> 3. Provide a binary and munge paths (like libxml-ruby used to)
> 4. Provide a binary and copy to ruby base directory (like the old
> version of the one click installer)
> 5. Provide a binary and don't munge paths, basically make users do #4
> manually (like libxml-ruby does now)
>
> None of #3, #4 or #5 are great - I can't say I really have a preference.
> But it would be nice if the OneClickInstaller picked one and then
> called it "best practices" so at least every user knows what to expect.
I prefer #5 and good documentation eventhough it's likely most people want the gem install process to automagically set up everything...a gem that's "ready to go" with no extra intervention. I'd always liked the post_install_message tradeoff eventhough Bundler has affected that.
We're always going to have tradeoffs. Currently, the punchline for me is:
1) RubyInstaller moves to static Ruby built-in extensions except for openssl. Main reason is to minimize version conflicts due to too much being on PATH.
2) DevKit stays as-is but we look to update the RubyInstaller to provide an option to download (but not install) the DevKit. Longer term, look at moving the DevKit from an archive to a full-blown installer.
3) Native gem authors provide dependent binaries and document any manual post-install steps as you're doing with libxml-ruby.
It's easy to back yourself into a corner by trying to do too much and end up causing a whole bunch of end-user frustration by not being able to address _all_ corner cases. As users, many of us have unique and clever setups that we know best how to accomodate. I'm all for the "80% solution + good documentation" even at the cost of additional manual intervention. I really think what frustrates most users is the historically sh*tty, hard-to-find, outdated, and unmaintained documentation containing the 7-Secret-Steps-to-a-Working-Install.
Luis Lavena
>> > As for RubyInstaller own provided DLLs, since we are not shipping with
>> > headers or link libraries for them, we might end killing them as
>> > shared objects and move to the static model to reduce any possible
>> > zlib1.dll or other DLL conflicts.
>> >
>> > Thoughts?
>
> To be clear, you're only talking about making the relevant lib/ruby/1.9.1/i386-mingw32/*.so Ruby built-in extensions static, correct? For example, new static i386-mingw32/{dbm.so,curses.so} would not be dependent upon {dbm3.dll,pdcurses.dll} in PATH.
>
That is correct.
> If we move to static built-in Ruby extensions, is your major concern that we could break code and any 3rd-party native Ruby extensions that are explicitly depending upon these DLLs currently included in <RUBYINSTALLER>/bin/ and presumably on PATH?
>
Hmn, is a mixture of answers for that. We don't distribute headers or
link libraries to these DLLs, so users can't compile an extension that
depend on them
However, user could install a gem that contain an extension that
depends on zlib1.dll, or a FFI library that depends on zlib1.dll or
openssl, just to mention a few.
We no longer providing the DLLs could break these scenarios, but
further thinking on that: it should break that, we never advertised
these DLLs for end-user usage, and is one of the reasons we are adding
msvcrt to OpenSSL dlls (the other being the MySQL and PostgreSQL DLL
clash).
>
>> I would lean towards the full dev environment approach because I doubt
>> you'll get gem authors to provide statically compiled binaries for
>> windows users.
>
> Could rake-compiler be updated to make this a no-brainer for gem authors building Windows binary gems on Linux/OSX?
>
rake-compiler + MiniPortile handle this scenario already.
Jon
> >
>
> Hmn, is a mixture of answers for that. We don't distribute headers or
> link libraries to these DLLs, so users can't compile an extension that
> depend on them
A nitpick, but someone could grab the headers and (if building with mingw) directly link against the DLLs to become dependent.
> However, user could install a gem that contain an extension that
> depends on zlib1.dll, or a FFI library that depends on zlib1.dll or
> openssl, just to mention a few.
>
> We no longer providing the DLLs could break these scenarios, but
> further thinking on that: it should break that, we never advertised
> these DLLs for end-user usage, and is one of the reasons we are adding
> msvcrt to OpenSSL dlls (the other being the MySQL and PostgreSQL DLL
> clash).
Regardless of whether we advertised the DLLs or not, it's possible, so someone may have become dependent upon them :(
IMO, these DLLs are an internal implementation detail of the RubyInstaller...depending upon them for external code or a native gem is unsupported; caveat emptor. I like that static linking would make this idea explicit. IIRC both fiddle and psych already static link against libffi and libyaml.