Accessing Windows 64 dll's from within Julia

[Bill Hart]

Hi,

I'm currently up to working on building the dependencies for our Nemo computer algebra package for Julia:

https://github.com/wbhart/Nemo.jl

We have dependencies on the following C libraries:

libpari (Pari/GP)

Antic (https://github.com/wbhart/Antic)

Flint (https://github.com/wbhart/flint2)

mpfr >= 3.1.2

gmp >= 5.1

Whatever logic I had already coded up in our build.jl to build these on Windows 64 and use them from within Julia no longer works. We always knew that there were going to be changes in the way things worked, so this is not unexpected.

So I thought I would take a look at the OpenSUSE build service. But flint, pari and antic are not available on there. (I found the WebRPM package for Julia and have been using that.)

It looks like someone has tried to build flint on there, but their system doesn't cope with the flint build system (or vice versa if you prefer).

Pari only builds on Windows 64 with some coaxing (I have succeeded in doing that). It certainly won't build with OpenSUSE.

So this means I have to build these binaries myself perhaps and just download them from somewhere on the web, which I'm happy to do.

So I tried building binaries in both msys2 and mingw64. After much work I cannot figure out where Julia expects to find the binaries. I've tried setting PATH and Libdl.LD_LOAD_PATH appropriately. I've tried static and dynamic libraries, but under no circumstances can I get Julia to find the libraries. It just says the "module is not found" when trying to ccall them.

Moreover, OpenSUSE doesn't provide a "make" package, as far as I can see. And make is not available from the GitBash shell in Julia on Windows 64. So even building the software from within Julia currently seems impossible.

I also had problems with the gcc not wanting to build 64 bit binaries.

Given that OpenSUSE appears out of the question for us on Windows 64. can anyone shed any light on how to get Julia compatible libraries built (msys2, mingw64??) and what the current best practice is for ccall'ing those binaries (how to set the linker path?).

Note that flint and pari both have custom configure arguments that must be set and neither flint nor pari uses autotools, though both roughly emulate an autotools build with some extra bells and whistles.

Any help would be very much appreciated. We have everything working beautifully on Linux, but one of the big selling points for us is that the entire Julia stack works natively on Windows. This is an especially big thing for us given that IJulia also works on Windows (most computer algebra systems do not work on Windows).

Julia is working beautifully for us and we have 40-50,000 lines of Julia code locally for our computer algebra package (Nemo + Hecke). But getting it working on Windows just seems to be outside of my current skill set.

One other minor question: because we currently need to build the libraries ourselves, we have to (or would like to) run configure and make, etc., from the shell in Julia. This means directories have to be converted from the C:\\abc\\def style to /c/abc/def/ style. Is there a Julia function for doing this, or a best practice?

Bill.

[Bill Hart]

I forgot to ask: does Julia currently expect to find .a or .dll files on Windows 64? Is anything else needed?

Bill.

[Tony Kelman]

All Julia needs is dll's. Best to refer to them via absolute paths in ccall. See how SCS.jl handles these for a simple example: https://github.com/JuliaOpt/SCS.jl/blob/master/deps/build.jl

Requiring Windows users of your package to have an MSYS environment and MinGW-w64 toolchain installed on their computers is asking for trouble and not at all recommended. I don't see why any library that you could build on Windows with MinGW-w64 couldn't build in cross-compilation from the opensuse build service - generally it's a deficiency in that library's build system that should be fixable. Cross-compilation is something that autotools is very good at, CMake is decent at if you know what you're doing. Rolling your own non-standard build system is asking for trouble, you can see places where Julia's own makefiles have gone to extra lengths involving XC_HOST to make this possible.

If you have a working build setup from a local Windows machine that can produce dll's, most of what you need to do is upload those dll's somewhere (bintray works fairly well) then set up the BinDeps scripts to download and determine the install-time paths.

[Bill Hart]

Sorry I surely meant WinRPM not WebRPM.

[Tony Kelman]

On second thought ECOS.jl is a simpler example: https://github.com/JuliaOpt/ECOS.jl/blob/master/deps/build.jl

Or MbedTLS.jl, very similar but with multiple libraries: https://github.com/JuliaWeb/MbedTLS.jl/blob/master/deps/build.jl

[Bill Hart]

On 15 September 2015 at 02:54, Tony Kelman <to...@kelman.net> wrote:

On second thought ECOS.jl is a simpler example: https://github.com/JuliaOpt/ECOS.jl/blob/master/deps/build.jl

Or MbedTLS.jl, very similar but with multiple libraries: https://github.com/JuliaWeb/MbedTLS.jl/blob/master/deps/build.jl

On Monday, September 14, 2015 at 5:51:33 PM UTC-7, Tony Kelman wrote:

All Julia needs is dll's.

Great, that halves the number of permutations for me to try. Thanks.

 

Best to refer to them via absolute paths in ccall.

Ouch. We have by now thousands of ccalls. Anyway I'll give it a try.

 

See how SCS.jl handles these for a simple example: https://github.com/JuliaOpt/SCS.jl/blob/master/deps/build.jl

Thanks.

 

Requiring Windows users of your package to have an MSYS environment and MinGW-w64 toolchain installed on their computers is asking for trouble and not at all recommended.

I agree, and I'd prefer to avoid it. But we still need to build them ourselves. And I do have a working MSYS and MinGW toolchain (not without great cost to management of course).

 

I don't see why any library that you could build on Windows with MinGW-w64 couldn't build in cross-compilation from the opensuse build service - generally it's a deficiency in that library's build system that should be fixable. Cross-compilation is something that autotools is very good at, CMake is decent at if you know what you're doing. Rolling your own non-standard build system is asking for trouble, you can see places where Julia's own makefiles have gone to extra lengths involving XC_HOST to make this possible.

Pari is not our package so there's nothing we can do about their build system. 

Many Open Source projects in our area simply aren't invested in Windows and have existed long before autotools was a thing.

Very basic autotools builds have been produced for Flint a few times, even recently, but unfortunately autotools is way too slow and is not able to support a number of the features of our build system that we rely on. 

Autotools gives an approximation that is good enough for some simple applications of Flint, but not for us.

If you have a working build setup from a local Windows machine that can produce dll's, most of what you need to do is upload those dll's somewhere (bintray works fairly well) then set up the BinDeps scripts to download and determine the install-time paths.

Thanks, I'll look at BinDeps next.

I have dedicated web hosting for the dll's, so that isn't a problem. Just getting them running from within Julia will be enough.

Bill.

[Tony Kelman]

You can refer to the path to a library in ccall via a const variable. BinDeps helps with setting that variable's value based on the user-dependent install path, so everything is relocatable. You can get by without BinDeps if you'd like, it's not without its flaws. Here's an example from Blosc.jl that barely uses BinDeps, only for the purposes of download_cmd and unpack_cmd: https://github.com/stevengj/Blosc.jl/blob/master/deps/build.jl

[Bill Hart]

I tried two different dll's with absolute paths, built with recent MinGW64 and Julia's ccall does not work for either. It still says, "The specified module could not be found".

The file type is "gmp-6.0.0/.libs/libgmp-10.dll: PE32+ executable (DLL) (console) x86-64, for MS Windows" and it is just GMP, so it has no external dependencies.

I have verified that a simple program will link against the dll and run within msys2/MinGW64. So the dll is certainly built correctly.

I realise Julia already supplies its own GMP, I'm just using this as a test case because I know there are no other dependencies. The result is the same with libpari.dll, which Julia does not provide.

Needless to say I am using the latest Julia 0.4 release candidate.

Bill.

[Jameson Nash]

You can use a program called depends22.exe to check whether it has any odd dependencies. Then try calling just `dlopen("C:\\absolute\\path\\to\\lib.dll")` and see whether that works. ccall uses the same underlying dlopen call, so once you get it working for one, it should work for both.

[Bill Hart]

Thanks Jameson. depends22.exe gave the required hint.

Apparently msys2 makes gmp (and everything else) depend on msys-2.0.dll. Who knows why it does that. I thought the entire point of msys was to create native, standalone Windows applications.

I can now access the dll's.

I imagine it will be enough for me to simply supply the dll's I need, plus msys-2.0.dll for download somewhere, unless I can find some magic invocation which builds msys-2.0.dll into the other dlls I'm building.

Bill.

[Tony Kelman]

If your libraries depend on msys-2.0.dll, you're using the wrong gcc, and the resulting libraries are unlikely to work with Julia (they might, if you get lucky, but I wouldn't count on it). This is essentially the same as if you built gmp in cygwin. If you use the MinGW-w64 version of gcc, then your libgmp will depend on msvcrt.dll instead of msys-2.0.dll.

[Bill Hart]

I simply used the "MinGW-w64 Win64 shell" that msys2 provides. Is that not the correct thing?

If not, where do I obtain the correct gcc? Is it for example available via pacman in msys2?

Bill.

[Jameson Nash]

The shell is correct, but you will probably want to download mingw64 separately. The julia `README.windows` file has links, I think, to the right one to download.

[Bill Hart]

Thanks Jameson. I'll try this tomorrow.

I think msys2 somehow conflates what is available via its msys2 64 bit shell and via the MinGW64 shell. I think I did download MinGW separately once, but it probably got overwritten at some point when I used msys2 64 bit.

Bill.

[Tony Kelman]

Yes, though the 32-bit version of i686-w64-mingw32-gcc available via pacman uses a different exception-handling format than the toolchain we use to build Julia, so there's another potential source of incompatibility if you're also providing 32-bit binaries. For 64 bit it should be compatible, hopefully. You can refer to https://github.com/JuliaLang/julia/blob/master/README.windows.md#compiling-with-mingwmsys2, specifically the contrib/windows/get_toolchain.sh script to obtain a toolchain with the highest likelihood of being compatible with Julia binaries. Or you can cross-compile from Cygwin, but sounds like the build systems of the libraries you're building can't handle that.

[Bill Hart]

I've managed to get it working without msys.dll now.

One unfortunately problem is that Flint relies on pthreads and gcc introduces a dependency on libwin-pthreads1.dll, so I have to distribute this dll also with my project. I've already looked online and there seems to be no way to get around this. There's a lot of people scratching their heads about this one online. The msys developers usually suggest that people compile their libraries with -static. But I'm not sure that is a solution is this case.

Anyway, it's not a major problem.

Bill.