Julia as a matlab mex function (and the struggle for independence from cygwin)

[Tracy Wadleigh]

I have created an example embedding of julia into matlab via a mex function, available as a gist:

https://gist.github.com/twadleigh/387f7aeb9707226d7d18

which I have working from a nightly win64 install (patched with the extra libuv headers).

I'm building via cygwin using the mingw32 compiler.

I say it's working, but with one caveat: it only works when I launch MATLAB from the cygwin prompt. I presume that is because there are some extra linker flags that I'm not throwing. Is there a windows expert out there that might be able to shed some light on how to make this DLL happy regardless of how I launch matlab?

[Elliot Saba]

Can you post the error you're running into when you try to compile outside of Cygwin?

-E

[Isaiah Norton]

Does it tell you what DLL is missing? If not, try using Dependency Walker to find out what it is looking for but not finding.

http://www.dependencywalker.com/

[Tracy Wadleigh]

Elliot:

The (unhelpful) error:

>> mexjulia

Invalid MEX-file 'U:\prj\mexjulia\mexjulia.mexw64': Invalid access to memory location.

My understanding is that this is due to some DLL that fails to be found. I tried to step through with gdb (in cygwin) to see if I could get more information about what wasn't getting found, only to find that it worked in that case!

Isaiah:

I had tried dependency walker before I discovered that I could make it work by launching via cygwin. The unresolved DLLs that it reports are:

API-MS-WIN-APPMODEL-RUNTIME-L1-1-0.DLL

API-MS-WIN-CORE-WINRT-ERROR-L1-1-0.DLL

API-MS-WIN-CORE-WINRT-L1-1-0.DLL

API-MS-WIN-CORE-WINRT-ROBUFFER-L1-1-0.DLL

API-MS-WIN-CORE-WINRT-STRING-L1-1-0.DLL

API-MS-WIN-SHCORE-SCALING-L1-1-1.DLL

DCOMP.DLL

IESHIMS.DLL

My subsequent googles suggested that these were really red herrings, but any insight you may have would be very much appreciated.

[Tracy Wadleigh]

BTW, it's not a compile-time error I get, but a matlab runtime error.

[J Luis]

Tracy,

This dumb batch file is what I use to build all my MEX files. It's kind of dumb but with this approach I can build with whatever compiler I want and I don't even need the whole Matlab installed (only the necessary .lib and .h files)
See if it can help.

https://code.google.com/p/mirone/source/browse/trunk/mex/compile_one.bat

[Tracy Wadleigh]

To be more specific:

When I launch the actual matlab exe (in the bin/win64 directory and NOT the wrapper in the bin directory) from cygwin with:

> /cygdrive/c/Program\ Files/MATLAB/R2012a/bin/win64/MATLAB.exe

I get:

>> mexjulia

ans =

Hello, from julia!

But it doesn't work if I launch the bin/matlab.exe wrapper or if I launch from the windows command prompt.

[Tracy Wadleigh]

Yet more detective work:

I downloaded ListDLLs. I dumped the results for MATLAB.exe to files for both the successful launch and call via cygwin and the unsuccessful non-cygwin launch. I then diff'd the files. The only differences are the obvious ones:

> 0x0000000035da0000  0x25000   U:\prj\mexjulia\mexjulia.mexw64

> 0x0000000036730000  0x336d000  C:\Julia-0.4.0-dev\bin\libjulia.dll

> 0x0000000035dd0000  0x1c000   C:\Julia-0.4.0-dev\bin\libgcc_s_seh-1.dll

> 0x0000000035e80000  0xed000   C:\Julia-0.4.0-dev\bin\libstdc++-6.dll

> 0x000000007c190000  0x203c000  C:\Julia-0.4.0-dev\bin\libopenblas.DLL

> 0x000000006f600000  0x134000  C:\Julia-0.4.0-dev\bin\libgfortran-3.dll

> 0x0000000039aa0000  0x57000   C:\Julia-0.4.0-dev\bin\libquadmath-0.dll

> 0x000000005ac10000  0x182000  C:\Julia-0.4.0-dev\bin\libpcre.DLL

> 0x0000000039b00000  0x27000   C:\Julia-0.4.0-dev\bin\libdSFMT.DLL

> 0x000000003d7b0000  0x8b000   C:\Julia-0.4.0-dev\bin\libgmp.DLL

> 0x0000000064b90000  0x1ac000  C:\Julia-0.4.0-dev\bin\libmpfr.DLL

> 0x0000000039b30000  0x4e000   C:\Julia-0.4.0-dev\bin\libopenlibm.DLL

I double-checked to make sure that C:\Julia-0.4.0-dev\bin is on the system path, so that can't be it...

:-(

[Jameson Nash]

perhaps matlab is providing an older version of one of those dll files? if launched from cygwin, you might still end up pulling in an newer one (from cygwin), so julia is still happy. however, if launched directly, then julia is unable to load.

depends22 will throw up a warning dialog box if one of the required dependencies is missing. the other unresolved dlls are typically wrapped inside of one of the other dlls (advapi), and are unimportant.

[J Luis]

Ok, this is to report my failure as well. I managed to build the MEX with VS. For that I had to create an import lib from libjulia.dll, which I did both with MinGW and VS (2013) but Matlab fails with

>> mexjulia
Invalid MEX-file 'C:\SVN\mironeWC64\mexjulia.mexw64': A dynamic link library (DLL) initialization routine failed.

and I'm not able to find why with the Dependency Walker (libjulia.dll is from master built by me as well)

[Tracy Wadleigh]

I did some more poking around with gdb, and was able to isolate one issue.

When run outside of cygwin, `uv_guess_handle` returns `UV_UNKNOWN_HANDLE` such that `init_stdio_handle` errors out. When run inside of cygwin, `uv_guess_handle` returns `UV_NAMED_PIPE` to no ill effect.

How can this be? I'm hoping an expert knows. I opened an issue for this.

[Tracy Wadleigh]

Update: with a couple of patches pending acceptance upstream (https://github.com/JuliaLang/julia/pull/9251 & https://github.com/JuliaLang/julia/pull/9221), my julia-in-a-mex-function example is working like a charm.

[Tim Holy]

It's really cool. Have you considered making it a package? Long, long ago I
wrote this:
https://github.com/timholy/julia-matlab
but I frankly suspect that your MEX solution is simpler.

--Tim

[Tracy Wadleigh]

On Thursday, December 4, 2014 9:08:18 PM UTC-5, Tim Holy wrote:

It's really cool. Have you considered making it a package? Long, long ago I
wrote this:
https://github.com/timholy/julia-matlab
but I frankly suspect that your MEX solution is simpler. 

I'm plowing through writing long lists of ccalls as I write in preparation for making a package when I have rudimentary marshaling in place.

I am familiar with your julia-matlab effort. I don't know that my approach is any simpler, but the primary advantage I was after in mexification was eliminating the requirement to copy data when crossing the matlab/julia threshold.

The true, subversive purpose of this effort is to sneak Julia into my organization without my colleagues needing to modify their workflows if they don't want.

[Tracy Wadleigh]

Rather than doing my own package, I think it makes more sense to try to add this functionality to the existing MATLAB.jl package, where, for instance, they already have mxArray marshaling.

[Tim Holy]

Sounds good. It would be for interop in both ways, then.

--Tim

[Tim Holy]

Tracy, any progress on this? I have a sudden need for this myself, and I'd be happy to help if there's stuff that needs finishing.

--Tim

[Tracy Wadleigh]

Tim:

Unfortunately, I haven't made much real progress on this since my initial post of the gist.

The last pass I took at it I was making attempts to clean up the julia initialization (which led to at least one stillborn PR).

My motivation had flagged a bit, as:

 1. I have limited extra cycles that are already spread thin across a variety of personal and professional projects;

 2. I have been making an effort to improve my pure-julia workflow (still very much a work in progress);

 3. for me, matlab is an at work, on windows thing, and I've recently stopped using julia on windows; [It's a shame, but, unfortunately, the windows version is simply not good enough for everyday use: significantly slower startup times, partial or missing backtraces, unacceptably slow git underneath the package manager, etc. Now my professional uses of julia are done in an Ubuntu VM hosted on my windows machine. I'd volunteer to make it better, but see #1.]

 4. I haven't been able to justify to myself that I needed it enough to invest the time; and,

 5. no one else has needed it.

Of course, #5 can come off the list now. Besides, I'm more than happy to do what I can to help out one of the most prolific and generous members of the Julia community.

I've just created Mex.jl, a stub of a repo on github and added you as a contributor. It will have to wait until I get to work on Monday (alas, no remote access---against company policy), but I will populate it with whatever I have and can probably donate at least a couple of hours everyday until it's at the point where it meets your needs.

I was initially thinking that this stuff could be included in the MATLAB.jl package, but, looking into it more, there were a lot of choices made therein that are specific to the matlab-from-julia model. Still very usable for this effort, but probably better as a dependency.

We can continue this thread on github. 

I'm looking forward to the collaboration!

[Tim Holy]

Hi Tracy,

Thanks for getting this started! Given your constraints, I understand why
you'd be reluctant to put much effort into this. Still very nice to capture the
work you've done already.

I'll wait until the code dump happens before getting into specific on GitHub,
but I too am looking forward to the collaboration!

Best regards,
--Tim

On Saturday, February 28, 2015 02:31:25 PM Tracy Wadleigh wrote:
> Tim:
>
> Unfortunately, I haven't made much real progress on this since my initial
> post of the gist.
>
> The last pass I took at it I was making attempts to clean up the julia
> initialization (which led to at least one stillborn PR).
>
> My motivation had flagged a bit, as:
> 1. I have limited extra cycles that are already spread thin across a
> variety of personal and professional projects;
> 2. I have been making an effort to improve my pure-julia workflow (still
> very much a work in progress);
> 3. for me, matlab is an at work, on windows thing, and I've recently
> stopped using julia on windows; [It's a shame, but, unfortunately, the
> windows version is simply not good enough for everyday use: significantly
> slower startup times, partial or missing backtraces, unacceptably slow git
> underneath the package manager, etc. Now my professional uses of julia are
> done in an Ubuntu VM hosted on my windows machine. I'd volunteer to make it
> better, but see #1.]
> 4. I haven't been able to justify to myself that I needed it enough to
> invest the time; and,
> 5. no one else has needed it.
>
> Of course, #5 can come off the list now. Besides, I'm more than happy to do
> what I can to help out one of the most prolific and generous members of the
> Julia community.
>

> I've just created Mex.jl <https://github.com/twadleigh/Mex.jl>, a stub of a