State of Ruby on Windows and Native Extensions, C++

[Joe N]

I'm hoping to find some answers to questions around the state of Ruby and Windows as it relates to native extensions.

First, my background is non-Windows, so that is why I'm trying to get an understanding.  I've been using Ruby since Rails came out around 2006, but always in a mac or linux environment.

We recently had a situation come up where we have a large amount of internal C++ code we are trying to integrate with our backend server code that is written in node.  The integration is not great, as the event-loop to c++ integration makes for an ugly, manual, painful wrapping style.  I thought ruby might be a more natural integration, and it was.  I was able to essentially wrap a large portion of our c++ codebase in an automated way, with fully generated documentation, and use it from ruby easily, almost magically.  I did this all under linux as that is where I'm familiar.

The problems began when trying to get this to work under Windows.  Our internal C++ code compiles under gcc and visual studio.  For linux/gcc, where I am comfortable, all of the above worked excellent.  However for Windows, we hit a major roadblock.  Because the dll's from our internal c++ code are built by visual studio, I was unable to link with them when trying to build my native extension.  I believe this is due to the name mangling differences between the compilers but this is not my area of expertise - I just know we hit a point where the "fix" would be very difficult.

I believe we had two options, both of which seemed to be too hard to pursue:

1.) Get our internal c++ code to compile with mingw.  This would require a significant effort due to 3rd party dependencies.  We went through this effort to get it to compile with gcc about a year ago, and my understanding was this took months of work.  But if we did this, then the output .lib/.so/.dll files would be compatible with ruby on windows which is built with mingw, and everything would "just work" - plus we could cross-compile for both platforms from linux.

2.) Build ruby itself with visual studio.  This seemed very difficult and as we have very limited ruby talent outside of myself (who has no Windows experience), it would basically be non-ruby ops people with Windows experience trying to understand how to build Ruby - not ideal in either direction!  And, even if we achieved this, my understanding is we would probably run into tons of problems with other 3rd part rubygems that probably don't support mswin ruby.  So in short this would be a constant pain point if we went off the "easy path".

Maybe there is a 3rd option I didn't consider, or options 1 or 2 aren't as bad as we think.  Ideally what we are building would be containerized, so this wouldn't matter - we would just package this up into linux docker containers and it would work everywhere.  However our largest customer base are primarily Windows users who probably don't even know what docker is, and so a big consideration is making it easy for them to "install" our product.  I think there is work happening now in this space with CNAB and duffle.sh, but it is still early.  Maybe some day this will be a reality.  For now, we probably need to stick with the lowest common denominator and make sure what we build works natively on both Windows and Linux.

If we could find a way to use Ruby for this, the solution is so beautiful compared to the way we have to integrate on the nodejs side.  Any feedback or suggestions anyone might have would be very much appreciated!

[Dušan D. Majkić]

Did you consider to reoragnise your dll so that it exports only C functions.

C dll can easily be imported in MinGW, or even direct to ruby ffi. 

That way you can distribute binary 32bit/64bit dll-s in gem.

--
You received this message because you are subscribed to the Google Groups "RubyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyinstalle...@googlegroups.com.
To post to this group, send email to rubyin...@googlegroups.com.
Visit this group at https://groups.google.com/group/rubyinstaller.
For more options, visit https://groups.google.com/d/optout.

[Lars Kanis]

Hi Joe,

this kind of issue isn't uncommon. I faced several such situations at work. In fact there's kind of a third option to the both you described. But let me start with the first two:

1. If you have access to the C++ source code and it's already compiling on Linux, then I doubt that it's a big amount of work to port it to MINGW. MSYS2 makes it possible to use all automake/autoconf or cmake scripts that already work on Linux with little to no changes. Have a look into the MSYS2 MINGW-packages repository: https://github.com/Alexpux/MINGW-packages There are hundreds of libraries which are patched as necessary and are built with a PKGBUILD description file, to make the process repeatable and to get a pacman installable package out. MSYS2 is a great foundation for libraries written in any language, especially if you have a lot of dependencies. This is how RubyInstaller2 is built as well.

2. MRI builds fine with MSVC. It is an official supported platform. However there is almost no support beyond the ruby core build. There's no standardized environment like RubyInstaller and most gems with C extensions don't build to my experience. Even the ruby stdlib is difficult to build due to external dependencies. I don't have any interest in maintaining a MSVC based ruby ecosystem for the following reasons: 1. There's no packaging system like MSYS2 with pacman. 2. Redistribution of MSVC is more difficult than gcc. 3. It's closed source, so it is hard to analyze, especially in the context of the new ruby MJIT implementation. 4. I had to work with MSVC and I don't like it due to it's bugs, missing features and bad error reporting.

3. The third option is to mix MSVC and MINGW. This usually works well with recent versions of gcc and MSVC for C interfaces. That means that both are compatible in regards to call conventions, memory management and memory layout. Linking can be done by gcc directly to the dll or through a import library file (both MSVC foo.lib and gcc foo.a are supported). Alternatively you can load the DLL per LoadLibrary through ffi/fiddle.

C++ on the other hand is typically not compatible. I didn't check this recently, but as far a I know, they use different name mangling, different call tables for virtual methods, incompatible template implementations and so on.

So wrapping C++ code behind a C interface is usually the easiest way to go if MINGW doesn't work. However there's no one who guarantees this compatibility. The pg gem can use a MSVC compiled libpq for instance. It usually links to the dll but in some seldom cases it can fail like here: https://github.com/ged/ruby-pg/commit/f7b006026a60491b66da04a3425f622f1ff0d891
The workaround here is to switch to the lib file instead. This kind of issue wouldn't appear with ffi/fiddle. And these have the advantage to release the GVL while the C function is called.

Regarding closed source third party MSVC libraries I usually ask the vendor to provide a fully static linked 64 bit DLL. This avoids hassles with the VC redistributable package. I have several such dlls running in production.

One time I had a reproducible segfault at LoadLibrary with a VC 2003 compiled 32 bit dll loaded per ffi into RubyInstaller-x86. It has been solved by using VC 2008 and switching to x64.

Hope this helps!
--
Kind Regards,
Lars

[Lars Kanis]

[Joe N]

Thank you both for the replies.  I have no experience wrapping a C++ library with a C interface, but when I've read about that before it seemed to me then I'd lose the ability to interact with the C++ code the same way.  For example now, if I wrap a C++ class called Foo and only wrap a couple member functions of Foo, I can use that basically as if it were a plain Ruby class called Foo.  I can call one of those member functions like a method: foo.do_something, and the C++ code may call its other member functions which I never wrapped, and it all just works.  If I somehow wrapped in a C interface first, I wouldn't have the classes at all.  I could imagine how this might work but I get the sense I'd lose most of the benefits I was hoping for.  But again, I've never done this so can't confirm.  I think I've read that SWIG does create a C interface around C++ code, so maybe that is something I will experiment with.  So far I had stuck to Rice/Rbplusplus and my own DSL on top, but if SWIG can do the C interface that might get around these problems.  The mingw route does sound appealing for all the reasons you mentioned.  I'll also take a look at how much actual work would be involved given we already have gcc working.

Thanks again!