Help needed with creating Julia package

[Bill Hart]

Hi,

I have been writing a new Julia package, which I have called Nemo (it's essentially a limited computer algebra system).

I have two specific problems:

1) The git and GitHub repository for Nemo already exists, but I haven't created a Julia Pkg yet.

https://github.com/wbhart/nemo

The documentation on creating a Julia Pkg seems to assume you are going to start with the Pkg then commit code to the git repository that it creates, not create a git/github project and then add the necessary stuff to turn it into a Julia package.

Is there documentation somewhere explaining how to do the latter? Or can someone help me with doing the latter?

(I have a couple of small build issues to fix in order for flint to work on Windows 64 before it will work there. But I will be working on those right away. I have managed to get it to work with Julia there, just not hacked the fixes into the flint build system yet. Other than this minor thing, I am quite ready to publish Nemo as a package right away (well, apart from a horrible 3x slowdown and excessive memory usage caused by gc, but I think I've given up on solving that problem for now).)

2) Nemo relies on mpir (or GMP), mpfr and flint, which are large external C/assembly libraries which need to get built or be available to run Nemo. I understand Julia has its own GMP and MPFR which I can probably link to if they are recent enough. 

Flint needs to be built when the package is installed. It takes a long time to build, e.g. 40 minutes or so on Windows, maybe a third of that on Linux.

I can't find any documentation explaining where to put the commands in a Pkg to actually git clone flint, build it, install it and set up paths for Nemo. Given the complexities of installing flint for the user, I'd like to have the Julia package manager do this automatically if at all possible. And I see it does seem to be possible. I just can't figure out how.

Flint is here:

https://github.com/wbhart/flint2

Can anyone help, or point me in the right direction?

Bill.

[Leah Hanson]

1) You can try looking at other packages to see the structure.

Code goes is `src/` (and you'll need a `Nemo.jl` in there that will be what Julia runs when you say `using Nemo`). Tests go in the `tests` folder; you should have a file `runtests.jl` in there that runs the tests for your package (if you want Julia's auto-testing stuff to work).

A file named REQUIRE is where any dependencies on other Julia go; it is also where you specify compatible versions of Julia. Every package has one of these, so you can look at them for examples to see the syntax. You probably want a line like `julia 0.3-` if you believe you package works for 0.3 and will work for 0.4, but not in 0.2.

Like most OSS projects: license in LICENSE.md, some documentation in README.md. (*.md means a plain text Markdown file; there's flexibility on plain text format/filename here; these are just what's made by default by `Pkg.generate`).

This is enough for people to get your package installed via the Julia package manager by running `Pkg.clone("https://github.com/wbhart/nemo")`.

Since Pkg.clone will put you package in the "right" place (i.e. where all your other Julia packages are installed), you'll be able to follow the normal package publishing (register with METADATA) instructions.

2) https://github.com/JuliaLang/BinDeps.jl is the package for installing binary dependencies.

You should also search this mailing list for past BinDeps questions, as there have been a number of them. Looking at other packages that use BinDeps can also be very helpful; Cairo.jl is an example, but anything that wraps C libraries would probably be helpful.

Best,

Leah

[Isaiah Norton]

Is there documentation somewhere explaining how to do the latter? Or can someone help me with doing the latter?

You could run `Pkg.generate("Nemo")` and then copy and commit (some of) the resulting files in your own Nemo git tree; there aren't very many.

I can't find any documentation explaining where to put the commands in a Pkg to actually git clone flint, build it, install it and set up paths for Nemo. Given the complexities of installing flint for the user, I'd like to have the Julia package manager do this automatically if at all possible. And I see it does seem to be possible. I just can't figure out how.

The Pkg manager will look for a file called `MYPKG/deps/build.jl` and run that if it exists. That's just a Julia file, so you can do whatever you want there (shell out, etc.). One option is to use the BinDeps package which provides primitives for interacting with various package managers and build systems:

https://github.com/JuliaLang/BinDeps.jl

A very advanced and fully-developed usage example can be found in the Cairo package, which has Autotools, Apt, Yum, and several other targets:

https://github.com/JuliaLang/Cairo.jl/blob/master/deps/build.jl

There are a number of other examples to draw from. Hopefully the above links will give you a sense of where to start. I can help out on Linux and Windows (@ihnorton on github).

[Bill Hart]

Thanks for the reply. Some follow-up questions inline below.

On Wednesday, 10 September 2014 15:57:21 UTC+2, Leah Hanson wrote:

1) You can try looking at other packages to see the structure.

Thanks, I see you mention Cairo below. I'll try to find that on GitHub for some clues.

 

Code goes is `src/`

 Do you mean src/ in the Julia directory, or do you mean the Nemo code should go in a subdirectory called /src within the git repository?

And test should be a subdirectory of src/ or a separate directory. (I guess I can look at an example for this.)

 

(and you'll need a `Nemo.jl` in there that will be what Julia runs when you say `using Nemo`).

I do have Nemo.jl, since I read I'd need this. But it is currently empty.

The problem, leading to another question, is that Nemo currently consists of two large modules called Rings and Fields (Fields uses Rings, but not vice versa). There will be more such modules in the future.

But I can't find a way for Nemo.jl to import everything from Rings and Fields and then export all those symbols out to the user if they type say `using Nemo`. Is there a way to do this without exporting all the symbols again individually from the Nemo module?

Tests go in the `tests` folder; you should have a file `runtests.jl` in there that runs the tests for your package (if you want Julia's auto-testing stuff to work).

Great. I wondered how to make that work. This should be easy to add. It will just call the two large sets of test functions I currently have.

 

A file named REQUIRE is where any dependencies on other Julia go; it is also where you specify compatible versions of Julia. Every package has one of these, so you can look at them for examples to see the syntax. You probably want a line like `julia 0.3-` if you believe you package works for 0.3 and will work for 0.4, but not in 0.2.

Ok.

 

Like most OSS projects: license in LICENSE.md, some documentation in README.md. (*.md means a plain text Markdown file; there's flexibility on plain text format/filename here; these are just what's made by default by `Pkg.generate`).

Ok.

 

This is enough for people to get your package installed via the Julia package manager by running `Pkg.clone("https://github.com/wbhart/nemo")`.

Yes, that is all I want for the time being.

 

Since Pkg.clone will put you package in the "right" place (i.e. where all your other Julia packages are installed), you'll be able to follow the normal package publishing (register with METADATA) instructions.

Ok.

 

2) https://github.com/JuliaLang/BinDeps.jl is the package for installing binary dependencies.

You should also search this mailing list for past BinDeps questions, as there have been a number of them. Looking at other packages that use BinDeps can also be very helpful; Cairo.jl is an example, but anything that wraps C libraries would probably be helpful.

Will do. That is the keyword I needed, thanks.

Bill.

[Bill Hart]

On Wednesday, 10 September 2014 15:57:56 UTC+2, Isaiah wrote:

Is there documentation somewhere explaining how to do the latter? Or can someone help me with doing the latter?

You could run `Pkg.generate("Nemo")` and then copy and commit (some of) the resulting files in your own Nemo git tree; there aren't very many.

This was what I thought of trying first. But I couldn't figure out how it worked out what GitHub repository to associate this with, or whether it would try to create one, possibly scrubbing my existing nemo repository on GitHub. Obviously I don't want to lose my commit history.

 

It also isn't clear where Julia creates the empty git repository. In the current directory? Or in some subdirectory of the Julia source tree?

I can't find any documentation explaining where to put the commands in a Pkg to actually git clone flint, build it, install it and set up paths for Nemo. Given the complexities of installing flint for the user, I'd like to have the Julia package manager do this automatically if at all possible. And I see it does seem to be possible. I just can't figure out how.

The Pkg manager will look for a file called `MYPKG/deps/build.jl` and run that if it exists. That's just a Julia file, so you can do whatever you want there (shell out, etc.).

Perfect. For the most part I can just run configure, make, make install for now and set some library paths (if I can figure out what kind of system I am on).

Finding the Julia installation on the system in order to link against the gmp/mpfr might be slightly more difficult.

 

One option is to use the BinDeps package which provides primitives for interacting with various package managers and build systems:

https://github.com/JuliaLang/BinDeps.jl

A very advanced and fully-developed usage example can be found in the Cairo package, which has Autotools, Apt, Yum, and several other targets:

https://github.com/JuliaLang/Cairo.jl/blob/master/deps/build.jl

There are a number of other examples to draw from. Hopefully the above links will give you a sense of where to start. I can help out on Linux and Windows (@ihnorton on github).

Thanks.

Bill.

[Isaiah Norton]

This was what I thought of trying first. But I couldn't figure out how it worked out what GitHub repository to associate this with, or whether it would try to create one, possibly scrubbing my existing nemo repository on GitHub. Obviously I don't want to lose my commit history.

For Pkg manager purposes, the association will be created later (when you register the package).

It also isn't clear where Julia creates the empty git repository. In the current directory? Or in some subdirectory of the Julia source tree?

Under `$HOME/.julia/v0.3` (or v0.4 if you are on git master)

For the most part I can just run configure, make, make install for now and set some library paths (if I can figure out what kind of system I am on).

There are some macros to help with this: @osx, @linux, @unix (both), and @windows. There is also a variable called OS_NAME with a platform-specific value (:Windows, :Linux, etc.) See:
http://docs.julialang.org/en/release-0.3/manual/calling-c-and-fortran-code/#handling-platform-variations

 

[Bill Hart]

Thanks very much for the information. I might have additional specific questions along the way, but this definitely helps massively for now.

[Kevin Squire]

You also might want to check out 
http://julia.readthedocs.org/en/latest/manual/packages/#package-development

Cheers,

   Kevin 

[Bill Hart]

I had a go at making a preliminary package which should vaguely build on Linux when "using Nemo" is run from within the src/ directory.

I checked this works on my machine at least.

I also checked that Pkg.clone("https://github.com/wbhart/nemo.git") clones the Nemo repository from within Julia. But this seems to only clone the repository and doesn't appear to issue "using Nemo" as I had expected.

Also typing "using Nemo" manually says that it can't find Nemo. What command to users have to issue to get "using Nemo" to actually do something?

Bill.

[Bill Hart]

It's ok, I got it. Pkg.build()

Bill.

[Bill Hart]

OK, I can build Nemo. But how do I load modules from Nemo now that it is installed and built.

For example "using Nemo", "using Rings", "using Fields" all fail, complaining that it can't find the modules.

I must be missing a step somewhere.

Bill.

[Leah Hanson]

Does your Nemo.jl contain 

~~~

module Nemo

<code goes here...>

end

~~~

?

[Bill Hart]

Yes.

I notice that after building Nemo it is in a directory called nemo, not Nemo.

At the moment Nemo.jl simply builds Nemo by including deps/build.jl, since this is what Cairo does.

The code the user will be interested in is in Rings.jl and Fields.jl. How do they load those?

Bill.

[Bill Hart]

I guess I figured it out.

[Johan Sigfrids]

Have you looked at Reexport.jl?

[Bill Hart]

Hi Isaiah,

Thanks for the offer of help on Windows. I think I am at the stage where I need to ask for it.

Here is the repo so far:

https://github.com/wbhart/Nemo

The problem I have now is as follows.

I have msys2 installed on my computer, with Ming64. Whilst each of the packages MPIR, MPFR and flint builds just fine from within the msys shell, the tools when called individually from Julia always seem to fail, exactly as the old Mingw 1.0 tools did. I've checked carefully that the versions of make, gcc, bash, etc that answer at the command prompt are all from msys2 and I've removed all old mingw and msys installations from my machine totally. I've tried building dozens of times, but it usually crashes early on in the MPIR install.

I have set MSYSTEM=MINGW64, which I notice the msys2-mingw64 shell does. But I cannot figure out what else it does to stop the crashes.

Do you have any insight into this?

I wasn't sure what you meant about contacting you on github, as they've removed the PM service. So I figured I'd just post here, also giving other knowledgeable individuals a chance to comment.

Bill.

On Wednesday, 10 September 2014 16:26:10 UTC+2, Isaiah wrote:

[Bill Hart]

One hint I have is that I keep getting the error message "couldnt make stderr distinct from stdout". 

Even if I build from a naked ,sys2 sh invoked without any decoration from the Windows command prompt, everything builds fine.

Although I can't detect any differences in the versions of gcc, make, ld, etc, I can detect one difference between the shell from Julia and the shell from msys2. When I do run(`ls /`) from Julia I am clearly in Git Bash. When I do it from msys2 I am clearly in msys2.

So it looks to me like some of the files distributed with Julia's GitBash are causing issues with building from Julia on Windows. Maybe there are some dll conflicts or something.

Either way, I don't know how to fix the problem. Any advice would be welcome.

Bill.

[Tony Kelman]

Bill,

Shelling out in Julia on Windows currently calls into the bundled Git-bash, for the purposes of the package manager. That will hopefully change in 0.4 by replacing the bundled Git-bash with libgit2 bindings, but PR 7339 is currently a bit stalled for lack of anyone who understands the LibGit2.jl / Pkg code digging into it.

We've generally tried to avoid the "force users to build packages from source" approach for Windows, since getting MSYS2 and the correct version of MinGW installed and working can be a bit challenging. It's been more effective to upload binaries containing the appropriate DLL's, and use BinDeps to download and extract them upon Pkg.add.

The question is how you want to build and where you want to upload those binaries. If your package is possible to cross-compile from Linux to Windows (using Linux versions of the MinGW compilers), then we can write a short spec file and submit it to the OpenSUSE build service, which has a large selection of several hundred cross-compiled libraries for 32 and 64 bit Windows: https://build.opensuse.org/project/show/windows:mingw:win64 . I had recent success doing this for Lapack, HDF5, Metis, and the stack of COIN-OR libraries used by various JuliaOpt packages. There's a Julia package called WinRPM that works like a package-manager provider with BinDeps, and handles all the dependencies properly.

Worrying about cross-compiling is a little confusing at first and more effort to set up initially, but using WinRPM and the build service makes future maintenance and interaction with dependencies much easier in the long term. You only need small tweaks to the spec file to update future versions or rebuild with a different set of options, for example.

If you can point me to the source of the package and provide representative configure invocations, I can take a look at working on a spec file with you.

-Tony

[Bill Hart]

Hi Tony,

I'm worried about the legality of downloading binaries for users, since the packages involved, GMP/MPIR, MPFR and FLINT are (L)GPL.

I see now that Julia adds things to the system PATH, which is why it is picking up things in the GitBash bundled in Julia. If I can figure out how to get PATH back to its original state temporarily, I think I can get the build to succeed (supposing the user has MSYS2 and Mingw64 correctly installed -- not easy, admittedly).

I agree binary distribution is the normal method on Windows. But I've already been forced to go through one FSF "license compliance review" because a GNU project reported us for being in breach of the license. I'm not doing it again.

If there is a way to do things in strict accordance with the LGPL and GPL I can go down that route. In that case, you can see the invocations here:

https://github.com/wbhart/Nemo/blob/master/deps/build.jl

Is that sufficient?

Bill.

[Bill Hart]

I should mention that there are some complications. Flint needs to be built in situ because it needs a large .txt file at runtime which contains data it uses (Conway polynomials). The only way to tell flint where it will be is through the build system, and the only way to know where it will be is to actually build flint.

Moreover, the system linker cannot find the dependencies MPIR and MPFR of flint unless the -rpath is set by flint during build. And again, the only way -rpath can be set is if flint knows where it is being built.

So I think source build is probably the only feasible option, actually.

Having said that, I cannot figure out how to revert the PATH to the system provided one in Julia. It just seems to clobber it.

Bill.

[Tony Kelman]

> I'm worried about the legality of downloading binaries for users, since the packages involved, GMP/MPIR, MPFR and FLINT are (L)GPL.

Quite a few packages rely on (L)GPL code. The GPL is a long confusing document, and limits commercial application of some code, that's for sure, but to the best of my (IANAL) understanding it's not too onerous to comply with for open-source projects. Mostly including the license in the documentation and being careful about bundling/derived works. The spec files on the build service have to clearly state the license, plenty of libraries there are GPL and/or have GPL dependencies.

> I see now that Julia adds things to the system PATH, which is why it is picking up things in the GitBash bundled in Julia. If I can figure out how to get PATH back to its original state temporarily, I think I can get the build to succeed (supposing the user has MSYS2 and Mingw64 correctly installed -- not easy, admittedly).

You can modify ENV["PATH"] as you like, but I recommend against this approach. Compare the relative difficulty of building Julia from source to downloading and using the binary installer. On a slightly smaller scale, having the package rely on MSYS2 and MinGW vs just downloading binaries will lead to a similar usability comparison for your package. On Linux there's much more variety in compiler and library versions so the best way to download binaries is going through the system package manager, but if your library isn't available in the package manager then building from source is easy because compilers are really easy to install. On Mac it's a little more annoying to install compilers, so a decent number of packages use Homebrew.jl to distribute binaries. (Just have to set up a short "formula" file with instructions on how to build, and bug Elliot, @staticfloat on github, to build binaries for you).

> I agree binary distribution is the normal method on Windows. But I've already been forced to go through one FSF "license compliance review" because a GNU project reported us for being in breach of the license. I'm not doing it again.

> 

> If there is a way to do things in strict accordance with the LGPL and GPL I can go down that route. In that case, you can see the invocations here:

I can imagine that experience would give you pause here. If you can link directly to the same version of GMP and MPFR as Julia is using (they are currently required dependencies, so this should be fairly safe - just have to figure out the best way to do it that would also work if someone is using their system's version of GMP and/or MPFR), then you don't need to worry about distributing anything except Flint itself. Would GMP be a drop-in replacement for MPIR for you? If we use WinRPM, I believe it would automatically detect GMP and MPFR (both are already available there) as linked DLL's and automatically download them, we just have to declare the dependency in the spec file and make sure configure picks them up or is told explicitly where they are located. They would be in a standard predictable location on the cross-compiling build machine.

> https://github.com/wbhart/Nemo/blob/master/deps/build.jl

> 

> Is that sufficient?

I can see from that script how you're configuring, yes. Downloading and building your own copy of MPIR and MPFR seems slightly redundant and possibly problematic to load them from within the same process as Julia's copies of GMP and MPFR.

> I should mention that there are some complications. Flint needs to be built in situ because it needs a large .txt file at runtime which contains data it uses (Conway polynomials). The only way to tell flint where it will be is through the build system, and the only way to know where it will be is to actually build flint.

Can that text file be included along with the binary?

> Moreover, the system linker cannot find the dependencies MPIR and MPFR of flint unless the -rpath is set by flint during build. And again, the only way -rpath can be set is if flint knows where it is being built.

You can't just provide a full -L path?

A few minor notes, standard convention for Julia packages is to name the Git repository include the ".jl" extension in its name. This helps visually identify Julia package repositories easily, and is nice to keep consistent.

You're also frequently doing "if on_windows", there's a pre-defined macro for that: @windows_only, or the opposite @unix_only.

-Tony

[Tony Kelman]

The library cross-compiles okay (at least at the last released version, I can try git master if you've made changes that are necessary), see https://build.opensuse.org/package/show/home:kelman/mingw32-flint and https://build.opensuse.org/package/show/home:kelman/mingw64-flint

It was made a little more challenging by the fact that flint uses a hand-rolled configure script that errors on unrecognized arguments. That configure script uses --build for what is conventionally meant by --host, the two differ in a cross compile. --host is where the compiled code is intended to run, --build is (in standard autotools) the machine that is doing the compiling.

There was an existing openSUSE Linux package for flint here https://build.opensuse.org/package/show/science/flint with a patch to use real autotools, though it's not completely usable for a cross compile at the moment since it doesn't handle using GMP instead of MPIR. MPIR is not available as a cross-compiled package yet, though making one would probably be easy. The existing cross-compiled version of MPFR already links to GMP so I think it would be redundant.

Flint has configure-time checks for popcnt and tls that run executables, neither of which work properly in a cross-compile. Locally on Windows they both succeed, but could those potentially be processor-dependent in any way? I could patch configure to override the results for the purposes of cross compiling, if you can recommend one way or the other.

On the text file, it's really bad for library relocatability to hard-code the compile-time location of required data. Is the content of the file ever expected to change? If not, I would just hard-code its contents into a string or more appropriate data structure. If so, then its location should really be made a runtime setting, either through a library initialization API or from an environment variable or something.

I could be wrong, but I don't think rpath does anything on Windows. Standard practice for MinGW libraries is to put dll files in bin/ rather than lib/ for this reason. If you were using libtool, it would create a .dll.a import library stub in lib/ to assist in linking to the dll if you wanted to compile a C/C++ application or library with -lflint. Luckily Julia doesn't need that stub to just ccall into a dll.

-Tony

[Bill Hart]

Lots of issues.

Firstly, GMP 5 is currently a drop-in replacement for MPIR for us. We prefer to use MPIR since I am the maintainer of the latter, and significant work has gone into making various features faster. 

For flint we definitely require the latest bleeding edge git version. Even the last official release from us, flint-2.4.4, will not work with Nemo.

The CP text file is absolutely enormous (nearly a megabyte). Encoding it as a string is certainly out of the question (we strictly adhere to the ANSI standard, which limits strings to something like 750 bytes). We've discussed many other ways of handling it, and numerous people have tried various things. It's a major pain in the backside, but totally necessary to have it (mathematically speaking).

Currently it is not too bad in that once flint is installed, so is the file, in a location with a fixed offset from the library (which will have been specified by --prefix). If the user chooses to build flint in the source tree, it is located in a fixed location within the source tree which flint can also find it. This is all fine, so long as nothing gets moved after compilation and installation, which is the usual case, it seems.

The problem with linking against Julia's GMP and MPFR was as follows. We can tell Julia where to find flint by pushing to DL_LOAD_PATH. But this only tells Julia where to find flint. The system linker is the one that tries to subsequently link flint with MPIR and MPFR, and it hasn't got a clue where to find them. What we push on DL_LOAD_PATH is irrelevant, as only Julia knows about that. This actually causes Julia to report that it can't find flint. I spent many, many hours figuring that one out.

Now it occurs to me that perhaps if we build flint against Julia's GMP and MPFR (assuming GMP is version 5 and MPFR version 3.1), then because Julia's libs are placed in the path by Julia, the system linker might just find them. That hadn't occurred to me as a possibility. The obvious problem is GMP is not MPIR and we are limited to using whatever version of GMP Julia chooses. Actually, maybe the git version of flint already supports GMP 6 nowadays too.

Also, flint uses libgcc_s and pthreads (and libm). We've currently set it up to statically build libgcc_s into flint, but we can't do that with pthreads. Initially at least we could disable pthreads when building flint (there is an option for it) but I guess in the long run we won't want to do that. I guess we can for now, but it seems like a compromise.

Regarding licenses, I would guess the GPL v3 and LGPL v3 license texts need to be distributed with Julia binaries, along with a written offer to provide the source code (or the actual source code). Certainly the LGPL allows Julia to dynamically link against LGPL libraries without becoming a derivative work. I don't want to make absolute statements about that issue, as IANAL, but I'd be interested in more details of how Julia is satisfying the conditions summarised here http://en.wikipedia.org/wiki/GNU_General_Public_License#Terms_and_conditions

Aside from those issues, I don't seem to easily be able to find out what version of GMP or MPFR Julia is currently using in v0.3. That could be helpful. 

Regarding the .jl extension, I did read that, but am not entirely sure I know how to change it on github. I will take a look and see if I can change it.

I didn't realise @unix_only was the opposite of @windows_only. I thought unix meant Free BSD, Net BSD, Unix, etc. And I couldn't see a way to combine Unix, Linux and OSX using the macros. Given that OSX is probably going to need something special, I'm not sure what the best combination of macros will be.

Popcnt is processor dependent. TLS is more OS dependent and I think available on Mingw. We can just disable hardware popcnt. It's not vitally important for more than about one noncritical function.

Thanks for all your comments and your assistance. Unfortunately flint is slightly complicated by various issues. 

For now I'm happy to go with GMP on Windows (ironic given that MPIR was originally a Windows-friendly fork of GMP). But for the simple reason that GMP is already there in Julia it shouldn't be a problem for now. In the long term we definitely want to go with MPIR. I will probably add some additional files to assure there is no question of compliance with licenses for Nemo at least.

Bill.

[Tony Kelman]

> Firstly, GMP 5 is currently a drop-in replacement for MPIR for us. We prefer to use MPIR since I am the maintainer of the latter, and significant work has gone into making various features faster. 

If the reverse is true, and MPIR would be a drop-in replacement for GMP for Julia, then it could also be a possibility that Julia could try to use MPIR instead of GMP in the future. I'm not aware of the specific differences other than MPIR intends to be more compatible with MSVC, and compiling Julia with MSVC is an ongoing experiment that I've done a fair amount of work on. Having the maintainer of a dependency library happen to be a user of Julia might be convenient for... reasons.

> For flint we definitely require the latest bleeding edge git version. Even the last official release from us, flint-2.4.4, will not work with Nemo.

We can experiment with bleeding edge git instead. We'll have to see how many more changes will be needed. Presumably you can tag a release of flint once we're happy things are working?

> The CP text file is absolutely enormous (nearly a megabyte). Encoding it as a string is certainly out of the question (we strictly adhere to the ANSI standard, which limits strings to something like 750 bytes). We've discussed many other ways of handling it, and numerous people have tried various things. It's a major pain in the backside, but totally necessary to have it (mathematically speaking).

> 

> Currently it is not too bad in that once flint is installed, so is the file, in a location with a fixed offset from the library (which will have been specified by --prefix). If the user chooses to build flint in the source tree, it is located in a fixed location within the source tree which flint can also find it. This is all fine, so long as nothing gets moved after compilation and installation, which is the usual case, it seems.

So is the path a relative path between the library and the text file, or an absolute path to the text file? When installing from a binary we can keep the relative path consistent pretty easily, absolute path definitely not. If it's an absolute path, would it be feasible to check the compile-time defined path first by default (as is done currently), but fall back to using an environment variable or other runtime path initialization method if the default location fails?

> The problem with linking against Julia's GMP and MPFR was as follows. We can tell Julia where to find flint by pushing to DL_LOAD_PATH. But this only tells Julia where to find flint. The system linker is the one that tries to subsequently link flint with MPIR and MPFR, and it hasn't got a clue where to find them. What we push on DL_LOAD_PATH is irrelevant, as only Julia knows about that. This actually causes Julia to report that it can't find flint. I spent many, many hours figuring that one out.

I'm not proposing to push anything to DL_LOAD_PATH for GMP or MPFR. Rather we tell flint's build system at compile time where to find Julia's GMP and MPFR via -L. Those libraries will always be either at a consistent relative path vs JULIA_HOME, or in standard system library paths if the user (or package manager) built Julia against system GMP/MPFR. I think it would be safe to check if joinpath(JULIA_HOME,"..","lib","libgmp.$(sys.dlext)") is openable via dlopen_e, and if not then falling back to find_library(["libgmp"]).

> Now it occurs to me that perhaps if we build flint against Julia's GMP and MPFR (assuming GMP is version 5 and MPFR version 3.1), then because Julia's libs are placed in the path by Julia, the system linker might just find them. That hadn't occurred to me as a possibility. The obvious problem is GMP is not MPIR and we are limited to using whatever version of GMP Julia chooses. Actually, maybe the git version of flint already supports GMP 6 nowadays too.

For Julia 0.3.0, GMP is version 5.1.3, MPFR is version 3.1.2. Check https://github.com/JuliaLang/julia/blob/768187890c2709bf2ff06818f40e1cdc79bd44b0/deps/Versions.make for the default version numbers used. On Julia master, GMP has been bumped to 6.0.0.

> Also, flint uses libgcc_s and pthreads (and libm). We've currently set it up to statically build libgcc_s into flint, but we can't do that with pthreads. Initially at least we could disable pthreads when building flint (there is an option for it) but I guess in the long run we won't want to do that. I guess we can for now, but it seems like a compromise.

You don't need to do this if you use WinRPM. If libgcc_s and/or pthreads are dynamically linked, the RPM packaging scripts on the build service detect that and mark the dll's as dependencies, so they get correctly downloaded when you install the package. Linking dynamically will make your binaries smaller, and many other things also link to these same libraries.

> Regarding licenses, I would guess the GPL v3 and LGPL v3 license texts need to be distributed with Julia binaries, along with a written offer to provide the source code (or the actual source code). Certainly the LGPL allows Julia to dynamically link against LGPL libraries without becoming a derivative work. I don't want to make absolute statements about that issue, as IANAL, but I'd be interested in more details of how Julia is satisfying the conditions summarised here http://en.wikipedia.org/wiki/GNU_General_Public_License#Terms_and_conditions

LICENSE.md was clarified recently, and DISTRIBUTING.md has some additional information on this too. I don't think any of Julia's required dependencies are GPL-3 only, it looks like all the GPL ones are either GPL-2 or GPL-2+. Binaries of flint would need to include its own license, but flint is not being distributed with Julia here, just available to easily download in a user-triggered fashion.

> I didn't realise @unix_only was the opposite of @windows_only. I thought unix meant Free BSD, Net BSD, Unix, etc. And I couldn't see a way to combine Unix, Linux and OSX using the macros. Given that OSX is probably going to need something special, I'm not sure what the best combination of macros will be.

OSX is a Unix. It is special and needs its own modified code much of the time, yes.

> Popcnt is processor dependent. TLS is more OS dependent and I think available on Mingw. We can just disable hardware popcnt. It's not vitally important for more than about one noncritical function.

The TLS check passes on a local MinGW configure. Should I enable TLS then? I'll leave popcnt disabled if you recommend as much.

[Bill Hart]

Hi Tony,

A quick reply as I am about to be unavailable for some hours.

On Saturday, 13 September 2014 09:19:27 UTC+2, Tony Kelman wrote:

> Firstly, GMP 5 is currently a drop-in replacement for MPIR for us. We prefer to use MPIR since I am the maintainer of the latter, and significant work has gone into making various features faster. 

If the reverse is true, and MPIR would be a drop-in replacement for GMP for Julia, then it could also be a possibility that Julia could try to use MPIR instead of GMP in the future. I'm not aware of the specific differences other than MPIR intends to be more compatible with MSVC, and compiling Julia with MSVC is an ongoing experiment that I've done a fair amount of work on. Having the maintainer of a dependency library happen to be a user of Julia might be convenient for... reasons.

> For flint we definitely require the latest bleeding edge git version. Even the last official release from us, flint-2.4.4, will not work with Nemo.

We can experiment with bleeding edge git instead. We'll have to see how many more changes will be needed. Presumably you can tag a release of flint once we're happy things are working?

We tend not to use tags. But there's no problem introducing a named tag to pin things.

 

> The CP text file is absolutely enormous (nearly a megabyte). Encoding it as a string is certainly out of the question (we strictly adhere to the ANSI standard, which limits strings to something like 750 bytes). We've discussed many other ways of handling it, and numerous people have tried various things. It's a major pain in the backside, but totally necessary to have it (mathematically speaking).

> 

> Currently it is not too bad in that once flint is installed, so is the file, in a location with a fixed offset from the library (which will have been specified by --prefix). If the user chooses to build flint in the source tree, it is located in a fixed location within the source tree which flint can also find it. This is all fine, so long as nothing gets moved after compilation and installation, which is the usual case, it seems.

So is the path a relative path between the library and the text file, or an absolute path to the text file? When installing from a binary we can keep the relative path consistent pretty easily, absolute path definitely not. If it's an absolute path, would it be feasible to check the compile-time defined path first by default (as is done currently), but fall back to using an environment variable or other runtime path initialization method if the default location fails?

The way it currently works is the user will specify --prefix. This is where libflint will be installed (in a subdirectory called lib). Obviously a default is chosen otherwise.

Flint is at first built inside its own source tree. Flint knows where to find the file relative to this source tree.

But when make install is issued, it will be moved across to --prefix.

As flint is compiled, --prefix is used to generate an absolute path where the text file will be stored, and this is baked into the library, which then also looks in this absolute path.

Of course the absolute path is computed at compile time to be some path relative to --prefix. It is passed to the relevant parts of flint via a -D define.

I don't know of a way to encode a path relative to where the library is installed. That would be very useful, if possible. But I'm not sure if it can be done.

 

> The problem with linking against Julia's GMP and MPFR was as follows. We can tell Julia where to find flint by pushing to DL_LOAD_PATH. But this only tells Julia where to find flint. The system linker is the one that tries to subsequently link flint with MPIR and MPFR, and it hasn't got a clue where to find them. What we push on DL_LOAD_PATH is irrelevant, as only Julia knows about that. This actually causes Julia to report that it can't find flint. I spent many, many hours figuring that one out.

I'm not proposing to push anything to DL_LOAD_PATH for GMP or MPFR. Rather we tell flint's build system at compile time where to find Julia's GMP and MPFR via -L. Those libraries will always be either at a consistent relative path vs JULIA_HOME,

Ah, I think JULIA_HOME should be useful. I didn't know about this.

 

or in standard system library paths if the user (or package manager) built Julia against system GMP/MPFR. I think it would be safe to check if joinpath(JULIA_HOME,"..","lib","libgmp.$(sys.dlext)") is openable via dlopen_e, and if not then falling back to find_library(["libgmp"]).

Ok.

 

> Now it occurs to me that perhaps if we build flint against Julia's GMP and MPFR (assuming GMP is version 5 and MPFR version 3.1), then because Julia's libs are placed in the path by Julia, the system linker might just find them. That hadn't occurred to me as a possibility. The obvious problem is GMP is not MPIR and we are limited to using whatever version of GMP Julia chooses. Actually, maybe the git version of flint already supports GMP 6 nowadays too.

For Julia 0.3.0, GMP is version 5.1.3, MPFR is version 3.1.2. Check https://github.com/JuliaLang/julia/blob/768187890c2709bf2ff06818f40e1cdc79bd44b0/deps/Versions.make for the default version numbers used. On Julia master, GMP has been bumped to 6.0.0.

Those are fine for now of course. If not I can easily push patches to flint to ensure that it is.

 

 

> Also, flint uses libgcc_s and pthreads (and libm). We've currently set it up to statically build libgcc_s into flint, but we can't do that with pthreads. Initially at least we could disable pthreads when building flint (there is an option for it) but I guess in the long run we won't want to do that. I guess we can for now, but it seems like a compromise.

You don't need to do this if you use WinRPM. If libgcc_s and/or pthreads are dynamically linked, the RPM packaging scripts on the build service detect that and mark the dll's as dependencies, so they get correctly downloaded when you install the package. Linking dynamically will make your binaries smaller, and many other things also link to these same libraries.

Great.

 

> Regarding licenses, I would guess the GPL v3 and LGPL v3 license texts need to be distributed with Julia binaries, along with a written offer to provide the source code (or the actual source code). Certainly the LGPL allows Julia to dynamically link against LGPL libraries without becoming a derivative work. I don't want to make absolute statements about that issue, as IANAL, but I'd be interested in more details of how Julia is satisfying the conditions summarised here http://en.wikipedia.org/wiki/GNU_General_Public_License#Terms_and_conditions

LICENSE.md was clarified recently, and DISTRIBUTING.md has some additional information on this too. I don't think any of Julia's required dependencies are GPL-3 only, it looks like all the GPL ones are either GPL-2 or GPL-2+. Binaries of flint would need to include its own license, but flint is not being distributed with Julia here, just available to easily download in a user-triggered fashion.

Since git is installed with Julia, I'd like to git clone flint as part of the install process, so that the user always has the source code of flint (and the license, etc). 

Flint is GPL, not LGPL. There is no linking exemption for flint. On the other hand, the FSF seems to be pretty isolated in its position that dynamic linking to a library results in a derived work. Fortunately there is almost no FSF code in flint and it is all LGPL'd. No one else is going to care.

 

> I didn't realise @unix_only was the opposite of @windows_only. I thought unix meant Free BSD, Net BSD, Unix, etc. And I couldn't see a way to combine Unix, Linux and OSX using the macros. Given that OSX is probably going to need something special, I'm not sure what the best combination of macros will be.

OSX is a Unix. It is special and needs its own modified code much of the time, yes.

> Popcnt is processor dependent. TLS is more OS dependent and I think available on Mingw. We can just disable hardware popcnt. It's not vitally important for more than about one noncritical function.

The TLS check passes on a local MinGW configure. Should I enable TLS then? I'll leave popcnt disabled if you recommend as much.

Sure. More when I return in a few hours.

I think I will get things going temporarily on Windows by modifying PATH first. Then I will promise a binary distribution method to come, which we can work on.

 

Bill. 

[Tony Kelman]

> We tend not to use tags. But there's no problem introducing a named tag to pin things.

Any reason why not? Your code is on github but your releases aren't, would be great if it was possible to track exactly which commits correspond to exactly which numbered release version.

> The way it currently works is the user will specify --prefix. This is where libflint will be installed (in a subdirectory called lib). Obviously a default is chosen otherwise.

> 

> Flint is at first built inside its own source tree. Flint knows where to find the file relative to this source tree.

> 

> But when make install is issued, it will be moved across to --prefix.

> 

> As flint is compiled, --prefix is used to generate an absolute path where the text file will be stored, and this is baked into the library, which then also looks in this absolute path.

> 

> Of course the absolute path is computed at compile time to be some path relative to --prefix. It is passed to the relevant parts of flint via a -D define.

None of this is going to work if you want the library to be usable on a different machine than it gets compiled on. That's a pretty big restriction to put on your users. A bit like using Gentoo - some people think it's fun out of masochism or something, but not many. Most users would rather save time and use binaries wherever possible (assuming the results are the same and you don't introduce any compatibility problems - if you use an automated build service and a package manager these are easy).

> I don't know of a way to encode a path relative to where the library is installed. That would be very useful, if possible. But I'm not sure if it can be done.

I'm actually not sure how fopen() with relative paths would work inside library code - it might depend on the current working directory of the calling process, which is also something you probably shouldn't be imposing restrictions on. This path is going to need to be runtime configurable, I don't see any other way around it. Unless you want to generate the file in such a way that it becomes valid C syntax assigning into one string per line or an array of integers per line, then use it during compilation as a .c or .h file.

> Since git is installed with Julia, I'd like to git clone flint as part of the install process, so that the user always has the source code of flint (and the license, etc). 

This isn't necessary. The flint source code is on github, it's pretty simple to document that and point people to the right place if they want to look at it. Keep in mind that not all Julia package users are going to understand or care about the source code of a C library. The Julia package existing and providing access to some of the functionality is enough for many users.

> Flint is GPL, not LGPL. There is no linking exemption for flint. On the other hand, the FSF seems to be pretty isolated in its position that dynamic linking to a library results in a derived work. Fortunately there is almost no FSF code in flint and it is all LGPL'd. No one else is going to care.

I don't know if we've gotten a clear legal intepretation of exactly where Julia's ccall falls with respect to licensing. I don't think it's exactly the same thing as linking. If your non-GPL Julia package depends on a GPL library and is going to automatically download that GPL library (either from source or from a binary) as part of its installation, then you should clearly document that fact. There are companies that forbid their employees from using or looking at GPL code. 

> I think I will get things going temporarily on Windows by modifying PATH first. Then I will promise a binary distribution method to come, which we can work on.

Alright. Just to warn you again, plenty of MinGW users refuse to add MinGW permanently to PATH, due to the compatibility issues that introduces. More often than not you need to work with several different subtly incompatible toolchains, and putting any of those on PATH permanently is counter-productive. So this installation method is something that you may get to work on your machine, but it's unlikely to work many other places.

[Bill Hart]

On 14 September 2014 06:14, Tony Kelman <to...@kelman.net> wrote:

> We tend not to use tags. But there's no problem introducing a named tag to pin things.

Any reason why not? Your code is on github but your releases aren't, would be great if it was possible to track exactly which commits correspond to exactly which numbered release version.

We have been using branches, not tags for that. We like to only push fixes to release branches, whilst work continues in trunk, and make those minor point releases, 2.4.0, 2.4.1. 2.4.2, etc. 

I suppose we could tag all the releases in their branches too.

 

> The way it currently works is the user will specify --prefix. This is where libflint will be installed (in a subdirectory called lib). Obviously a default is chosen otherwise.

> 

> Flint is at first built inside its own source tree. Flint knows where to find the file relative to this source tree.

> 

> But when make install is issued, it will be moved across to --prefix.

> 

> As flint is compiled, --prefix is used to generate an absolute path where the text file will be stored, and this is baked into the library, which then also looks in this absolute path.

> 

> Of course the absolute path is computed at compile time to be some path relative to --prefix. It is passed to the relevant parts of flint via a -D define.

None of this is going to work if you want the library to be usable on a different machine than it gets compiled on. That's a pretty big restriction to put on your users. A bit like using Gentoo - some people think it's fun out of masochism or something, but not many. Most users would rather save time and use binaries wherever possible (assuming the results are the same and you don't introduce any compatibility problems - if you use an automated build service and a package manager these are easy).

We'd like to fix the problem. We just don't know how. At some point, the library has to read from a large text file. There must be a relocatable way of doing that.

Of course you can pass an explicit directory to flint's configure to tell it where to put this text file. Perhaps there is some canonical place.

 

> I don't know of a way to encode a path relative to where the library is installed. That would be very useful, if possible. But I'm not sure if it can be done.

I'm actually not sure how fopen() with relative paths would work inside library code - it might depend on the current working directory of the calling process, which is also something you probably shouldn't be imposing restrictions on. This path is going to need to be runtime configurable, I don't see any other way around it.

That should be possible. I'll add a function to set it at runtime.

 

Unless you want to generate the file in such a way that it becomes valid C syntax assigning into one string per line or an array of integers per line, then use it during compilation as a .c or .h file.

> Since git is installed with Julia, I'd like to git clone flint as part of the install process, so that the user always has the source code of flint (and the license, etc). 

This isn't necessary. The flint source code is on github, it's pretty simple to document that and point people to the right place if they want to look at it. Keep in mind that not all Julia package users are going to understand or care about the source code of a C library. The Julia package existing and providing access to some of the functionality is enough for many users.

> Flint is GPL, not LGPL. There is no linking exemption for flint. On the other hand, the FSF seems to be pretty isolated in its position that dynamic linking to a library results in a derived work. Fortunately there is almost no FSF code in flint and it is all LGPL'd. No one else is going to care.

I don't know if we've gotten a clear legal intepretation of exactly where Julia's ccall falls with respect to licensing. I don't think it's exactly the same thing as linking.

The Free Software Foundation explicitly says that "bindings" from a programming language interpreter, not just linking, constitute a derived work and therefore the combination becomes GPL'd. They are very explicit about this on their website.

I also had someone write to me the other day and refuse to access the source code via Github because they "refused to indemnify Github against all damages". It's not clear if I am required under the GPL to send them the source code some other way...

We're also not yet fully in compliance with the GPL. Because it is an interpreted environment GPL v2 (which flint uses) requires a notice to be printed when Nemo starts up saying that the program comes with no warranty. I'll have to add that, if I can figure out how.

We also need to either include the full source code of flint with the binary, or a written offer to supply it upon request. Here the FSF claims a link to a website is not really good enough because the URL might change. As a result of this, I am personally very reluctant to not just supply the source code for flint. I also want people to have the git repository automatically so they can contribute to it if they want.

 

If your non-GPL Julia package depends on a GPL library and is going to automatically download that GPL library (either from source or from a binary) as part of its installation, then you should clearly document that fact. There are companies that forbid their employees from using or looking at GPL code. 

I know. We will do this too.

We tried to get rid of the GPL in flint recently, hoping we might even make it BSD licensed or compatible. We spent some time contacting copyright holders. Some umm'd and ahh'd and never gave us a response. Others we have not been able to track down at all. At least two have fairly famously "disappeared" and not been contactable from anyone in the Open Source community. The contributions of one are probably not legally significant. The other previously authorised us to use the LGPL.

But at this point, unfortunately we are stuck with the GPL in flint.

 

> I think I will get things going temporarily on Windows by modifying PATH first. Then I will promise a binary distribution method to come, which we can work on.

Alright. Just to warn you again, plenty of MinGW users refuse to add MinGW permanently to PATH, due to the compatibility issues that introduces. More often than not you need to work with several different subtly incompatible toolchains, and putting any of those on PATH permanently is counter-productive. So this installation method is something that you may get to work on your machine, but it's unlikely to work many other places.

Unfortunately it doesn't even work on my machine. It seems ok for some calls into the dll, but as soon as I try to say print something using a function in the MPIR dll it segfaults. I suppose it must be linked against subtly different Microsoft dll's than Julia and some kind of conflict results.

The dependency walker does show some missing dll's, but they are all obscure Microsoft ones. Perhaps that is normal, I don't know.

I have had it working on my machine, but only if I build all the dlls myself from msys2, not from within Julia (even if I pop the Julia paths off the PATH environment variable so that it really uses msys2).

I realise that binary distribution is going to be the only solution for many Windows users in the long run. I just thought I could quickly get something working for people who were prepared to uninstall old versions of msys, install msys2 and put it in their path. I have to admit to being quite personally disappointed that I haven't been able to get that to work.

Bill.

[Bill Hart]

If I substitute the mpir.dll produced with msys2 under Julia with one produced with msys2 standalone, everything works. Of course, nothing that I am using from within Nemo actually uses MPFR, so this is not surprising.

The problem must be what libtool is doing on mingw64. Make install doesn't even copy the generated dll across to the install directory, so you have to do this manually. 

I also can't build flint against the Julia mpir and mpfr since Julia doesn't seem to distribute the gmp.h and mpfr.h files, and flint requires these when building.

Bill.

[Tony Kelman]

> Of course you can pass an explicit directory to flint's configure to tell it where to put this text file. Perhaps there is some canonical place.

Julia has settled on some good default choices for where to put packages by now, the problem is users are free to override the default locations and can have good reasons for wanting to do so.

> The Free Software Foundation explicitly says that "bindings" from a programming language interpreter, not just linking, constitute a derived work and therefore the combination becomes GPL'd. They are very explicit about this on their website.

> 

> I also had someone write to me the other day and refuse to access the source code via Github because they "refused to indemnify Github against all damages". It's not clear if I am required under the GPL to send them the source code some other way...

Oh dear. Well what I've been doing when building binaries is including the Readme and the License files as part of the packaging, so those will always get installed along with the dll's.

> We also need to either include the full source code of flint with the binary, or a written offer to supply it upon request. Here the FSF claims a link to a website is not really good enough because the URL might change. As a result of this, I am personally very reluctant to not just supply the source code for flint. I also want people to have the git repository automatically so they can contribute to it if they want.

Okay. It's not the end of the world to checkout the source code even when downloading a binary. Though keep in mind that for Julia 0.4, the plan is to transition Julia's package manager from relying on shelling out to command-line git to using ccall bindings to libgit2, so it won't necessarily be a safe assumption to make that you can always shell out to git. That's still a ways out though, and you can always put a try-catch around the checkout since it's not an absolute requirement.

> We're also not yet fully in compliance with the GPL. Because it is an interpreted environment GPL v2 (which flint uses) requires a notice to be printed when Nemo starts up saying that the program comes with no warranty. I'll have to add that, if I can figure out how.

That should be pretty easy with a println in your Nemo.jl module file, should come up any time anyone calls "using Nemo". You can also write a function called __init__() that gets called on package startup. Not sure if/where in the docs that's listed.

> Unfortunately it doesn't even work on my machine. It seems ok for some calls into the dll, but as soon as I try to say print something using a function in the MPIR dll it segfaults. I suppose it must be linked against subtly different Microsoft dll's than Julia and some kind of conflict results.

Or maybe different GCC dll's. Exactly which version of MinGW-w64 are you using? Might not be compatible with what was used to build Julia. Or there could be some issue due to having gmp and mpir both loaded within the same process? We've seen sort-of-similar issues with blas and lapack and various packages, though not so much on Windows.

> The problem must be what libtool is doing on mingw64. Make install doesn't even copy the generated dll across to the install directory, so you have to do this manually. 

Hm, yeah, libtool can be very picky, especially when it comes to dll's. I think I've used the "subtle and quick to anger" quote on this list before when talking about libtool... I've found configuring with lt_cv_deplibs_check_method=pass_all can sometimes help.

> I also can't build flint against the Julia mpir and mpfr since Julia doesn't seem to distribute the gmp.h and mpfr.h files, and flint requires these when building.

Oh, right, damn. Sorry I forgot about that! That is an issue, maybe we should open a separate discussion on that general topic. It has come up before that various packages would like to link against the same libraries as Julia (in the Blas and Lapack cases we're building with an incompatible ABI which makes this actually not a good idea in most cases, but for GMP and MPFR I think we're configuring them in the default way), so installing the corresponding headers would actually be necessary.

Even though I'm not entirely sure what Nemo or Flint do or whether I would personally have a use for them, I have some strange desire to see more Julia packages be painless to install cross-platform and want to help here. Let me know how the runtime configuration of the text file location goes, then I can mock up what BinDeps would look like. It should simplify your deps/build.jl script by automating the standard download, extract, configure, make steps. There are also some Julia idioms like joinpath("a","b") that would be cleaner than what you're doing now with if statements to switch between forward slashes and backslashes, and things like

cd(newdir) do

    ...

end

that work in a nicer way including returning to the old directory even on failure.

[Bill Hart]

On 14 September 2014 17:46, Tony Kelman <to...@kelman.net> wrote:

> Unfortunately it doesn't even work on my machine. It seems ok for some calls into the dll, but as soon as I try to say print something using a function in the MPIR dll it segfaults. I suppose it must be linked against subtly different Microsoft dll's than Julia and some kind of conflict results.

Or maybe different GCC dll's. Exactly which version of MinGW-w64 are you using?

I've just updated the msys2 packages with pacman. Recently I updated just some of the packages and not others, and I thought this might have introduced some incompatibilities. I noticed that if I build MPIR even with msys2 and not via Julia, the resulting dll doesn't work any more.

After updating msys2 fully to msys 2.0.1, it still doesn't work. So it looks to me like some new issues have been introduced by the more recent msys or something.

Their gas also barfs on one of the assembly files which we now have to patch. This issue affects GMP too, not just MPIR since we use the same code for that file.

 

Might not be compatible with what was used to build Julia. Or there could be some issue due to having gmp and mpir both loaded within the same process?

Just to reiterate, it was working fine before. I even went back to the precise configure invocation I used before in my bash history to ensure I was building MPIR the same way as before, and I checked which alpha version of MPIR I used.

Admittedly I was using Julia 0.2.1 before, not Julia 0.3, which I have now upgraded to.

I can try downloading Julia 0.2 for Windows again and see if that fixes the issue I guess.

 

We've seen sort-of-similar issues with blas and lapack and various packages, though not so much on Windows.

> The problem must be what libtool is doing on mingw64. Make install doesn't even copy the generated dll across to the install directory, so you have to do this manually. 

Hm, yeah, libtool can be very picky, especially when it comes to dll's. I think I've used the "subtle and quick to anger" quote on this list before when talking about libtool... I've found configuring with lt_cv_deplibs_check_method=pass_all can sometimes help.

> I also can't build flint against the Julia mpir and mpfr since Julia doesn't seem to distribute the gmp.h and mpfr.h files, and flint requires these when building.

Oh, right, damn. Sorry I forgot about that! That is an issue, maybe we should open a separate discussion on that general topic. It has come up before that various packages would like to link against the same libraries as Julia (in the Blas and Lapack cases we're building with an incompatible ABI which makes this actually not a good idea in most cases, but for GMP and MPFR I think we're configuring them in the default way), so installing the corresponding headers would actually be necessary.

Even though I'm not entirely sure what Nemo or Flint do or whether I would personally have a use for them, I have some strange desire to see more Julia packages be painless to install cross-platform and want to help here.

I can understand that.

 

Let me know how the runtime configuration of the text file location goes,

It won't happen for a while. I'm afraid I'm currently 14 days behind schedule at work. I've spent days trying to get this to work on Windows and have just run out of time to keep working on it. I will have to come back to it when I catch up with everything else that has been piling up. Perhaps I can spend some more time on it next weekend.

 

then I can mock up what BinDeps would look like. It should simplify your deps/build.jl script by automating the standard download, extract, configure, make steps. There are also some Julia idioms like joinpath("a","b") that would be cleaner than what you're doing now with if statements to switch between forward slashes and backslashes, and things like

cd(newdir) do

    ...

end

that work in a nicer way including returning to the old directory even on failure.

Thanks, I will try to incorporate these suggestions in a later incarnation of the code.

Bill.

[Bill Hart]

I checked that the MPIR dll's produced with the updated msys2 also don't work with Julia 0.2.

I can't think of many other variables here. It has to be msys2 related. I could try not patching the assembly file it barfs on, but have it built from C fallback code. But I know for a fact that file is not being used in the functions I'm calling that cause it to segfault.

I can try uninstalling msys2 completely and reinstalling it and mingw64 and see if I get any joy.

[Bill Hart]

I traced through the precise calls to libflint and libgmp that make it segfault and wrote a few test programs from the msys2 command line, taking Julia right out of the loop.

At first I thought that it was only segfaulting when gmp was called from flint. But in fact I can make it segfault calling gmp directly from the program, though only in slightly strange circumstances.

If I take the integer a = 123456789, I can compute the square and cube of this without problems. But if I try to compute the fourth power it segfaults.

If I call gmp_printf it segfaults. And I checked that the mpz_t I'm trying to print contains valid data and that accessing it doesn't cause a segfault.

In fact, the MPIR test suite fails most of its tests. In particular I notice that C only functions pass their tests, but assembly ones don't.

I've got a feeling that msys have changed how it reports itself in a way that autotools hasn't kept up with, causing it to compile in linux assembly functions instead of Windows ones.

Given that patching the linux assembly file and not the Windows one causes msys2 to not barf on that file, I'd say that's a certainty.

Now I just have to hack autotools one more time to recognise msys2 and I should be ok. This is why I don't use autotools in flint.

Bill.

[Bill Hart]

That was the problem. I've added hack number 40001 to the MPIR autotools to fix this issue. Nemo now passes its tests on Window 64 for the first time ever.

Bill.

[Tony Kelman]

I've compiled many many dozens of libraries using MinGW-w64 on MSYS, and cross-compiling from Cygwin and various Linux distributions, and not encountered nearly as many problems as you seem to be. Pretending to be autotools but hacking it up yourself is the worst possible thing you could do - you're completely nonstandard and nonfunctional in cross-compilation for example, and you have as many or more dependencies - requiring bash for example is often not acceptable on FreeBSD and similar systems.

Part of the problem is the way MSYS2 identifies itself changes depending on what environment variables you have set (or which bat file you start it with). The MSYS uname is unrecognized by autotools because MSYS2 is new, and you almost never want to actually be compiling with the MSYS host, since that causes your application to link to the msys-2.0.dll posix compatibility layer. It's like using the standard gcc in Cygwin, any users would need Cygwin and the associated GPL posix runtime library to use the compiled code. The MINGW64 uname is also new and unrecognized, and totally nonstandard. This is why you should be querying the COMPILER for information about what system the generated code is supposed to run on, NOT the build environment. The accepted GNU triples for MinGW are:

i686-pc-mingw32: MinGW.org 32 bit compiler, outdated and obsolete, don't use this

i686-w64-mingw32: 32 bit compiler from MinGW-w64 project, use this for 32 bit host

x86_64-w64-mingw32: 64 bit compiler from MinGW-w64 project, use this for 64 bit host

Yes the names are confusing and make no sense, but this is the GNU standard that every other conforming project in the world is using. LLVM decided to go out on a limb and rearrange some of these names recently, adding to the confusion, but GCC, autotools, and MinGW are unlikely to follow their lead.

[Bill Hart]

On 15 September 2014 07:38, Tony Kelman <to...@kelman.net> wrote:

I've compiled many many dozens of libraries using MinGW-w64 on MSYS, and cross-compiling from Cygwin and various Linux distributions, and not encountered nearly as many problems as you seem to be. Pretending to be autotools but hacking it up yourself is the worst possible thing you could do - you're completely nonstandard and nonfunctional in cross-compilation for example, and you have as many or more dependencies - requiring bash for example is often not acceptable on FreeBSD and similar systems.

I take it you are referring to flint rather than MPIR here.

I was referring to hacking the MPIR autotools, which has been necessary long before MPIR even existed. It was inherited from GMP and certainly necessary for a multitude of reasons: making fat binaries, distinguishing microprocessor microarchitectures, linking C code with assembly code cross platform, compiling on Windows and OSX, excluding compilers which compile everything else just fine but miscompile bignum libraries, tuning, etc, etc, etc. 

Autotools has always been unsuitable for such things at some level.

 

Part of the problem is the way MSYS2 identifies itself changes depending on what environment variables you have set (or which bat file you start it with). The MSYS uname is unrecognized by autotools because MSYS2 is new, and you almost never want to actually be compiling with the MSYS host, since that causes your application to link to the msys-2.0.dll posix compatibility layer.

I will have to look into that. It doesn't sound like something we want. Though I don't really understand why changing the host should change the way gcc compiles code, unless it does so for cygwin itself.

 

It's like using the standard gcc in Cygwin, any users would need Cygwin and the associated GPL posix runtime library to use the compiled code. The MINGW64 uname is also new and unrecognized, and totally nonstandard. This is why you should be querying the COMPILER for information about what system the generated code is supposed to run on, NOT the build environment.

It was my understanding that the Mingw compiler could be used from Cygwin. Perhaps I am confused about it, but it seems like interrogating the compiler may not give the information we need.

 

The accepted GNU triples for MinGW are:

i686-pc-mingw32: MinGW.org 32 bit compiler, outdated and obsolete, don't use this

i686-w64-mingw32: 32 bit compiler from MinGW-w64 project, use this for 32 bit host

x86_64-w64-mingw32: 64 bit compiler from MinGW-w64 project, use this for 64 bit host

Yes the names are confusing and make no sense, but this is the GNU standard that every other conforming project in the world is using. LLVM decided to go out on a limb and rearrange some of these names recently, adding to the confusion, but GCC, autotools, and MinGW are unlikely to follow their lead.

We can't use the accepted gnu triples in MPIR and that has always been the case for GMP too. They don't give nearly enough information for our needs.

 

As far as MPIR is concerned the first part of the triple has to be the microarchitecture of the CPU for assembly language purposes. Then we have information about the platform/OS. If we reserve that for microarchitecture, clearly we can't use the triples given above.

Canonicalising triples in MPIR is a two step process. Firstly we call the standard FSF config.guess, plus any hacks necessary for systems that are known to be incorrectly identified by the standard config.guess. We update the standard config.guess every now and again.

We then have our own config.guess, which uses a completely different convention which gives us the information we require. This has also always been the case in GMP.

We also use a cpuid utility in conjunction with config.guess to give us information about the microarchitecture.

Personally I'd like to get rid of autotools in MPIR altogether. It is much more effort to maintain than it is worth, with thousands of lines of hacks. And our main autotools maintainer died in 2012. I know much less about it.

[Tony Kelman]

> I will have to look into that. It doesn't sound like something we want. Though I don't really understand why changing the host should change the way gcc compiles code, unless it does so for cygwin itself.

Changing the host changes which compiler gets used to create the binaries. In Cygwin or Linux, you can call the MinGW compilers by providing --host=x86_64-w64-mingw32 or similar, if you're using conventional autotools. It's also possible in Cmake but it's about 5 lines worth of flags (or a toolchain file with the same content) - it works, eventually, but it's really picky. 

> It was my understanding that the Mingw compiler could be used from Cygwin. Perhaps I am confused about it, but it seems like interrogating the compiler may not give the information we need.

Right, but when you configure with --host=x86_64-w64-mingw32 in Cygwin or Linux, you're using a cross-compiler that is prefixed by the host triple: x86_64-w64-mingw32-gcc and similar for binutils. So you can't just ask "gcc" for information - in a lot of ways MSYS(2) is a hack to avoid worrying about MinGW cross-compilation, and it leads to some bad ways of doing things. I think it's better to do the work to make a library possible to cross-compile, than to give up and say MSYS(2) is the only method of compiling it for Windows. I've been cross-compiling Julia itself and all of its dependencies from both Linux and Cygwin for months, I very rarely touch MSYS(2) any more.

> We can't use the accepted gnu triples in MPIR and that has always been the case for GMP too. They don't give nearly enough information for our needs.

Of course they don't provide a complete story when you're worrying about assembly and various SIMD instruction sets. But that's not the GNU triple's job. That information should be determined and represented separately, you're not going to include this kind of information in the host prefix of the name of a cross-compiler executable for example. GMP is also a pretty bad actor in terms of its autotools usage - making a symlink to libgmp.dll called libgmp.lib in lib/ is completely the wrong thing to do for MinGW. Not your fault I know, but sounds like you may have inherited some autotools abuses from GMP.

 

 

[Bill Hart]

On 15 September 2014 13:39, Tony Kelman <to...@kelman.net> wrote:

> I will have to look into that. It doesn't sound like something we want. Though I don't really understand why changing the host should change the way gcc compiles code, unless it does so for cygwin itself.

Changing the host changes which compiler gets used to create the binaries. In Cygwin or Linux, you can call the MinGW compilers by providing --host=x86_64-w64-mingw32 or similar, if you're using conventional autotools. It's also possible in Cmake but it's about 5 lines worth of flags (or a toolchain file with the same content) - it works, eventually, but it's really picky. 

Sorry, I think I misunderstood what you were referring to when you said "host". So my response doesn't make any sense in the way that you meant it. I understand what you are getting at now.

 

> It was my understanding that the Mingw compiler could be used from Cygwin. Perhaps I am confused about it, but it seems like interrogating the compiler may not give the information we need.

Right, but when you configure with --host=x86_64-w64-mingw32 in Cygwin or Linux, you're using a cross-compiler that is prefixed by the host triple: x86_64-w64-mingw32-gcc and similar for binutils. So you can't just ask "gcc" for information - in a lot of ways MSYS(2) is a hack to avoid worrying about MinGW cross-compilation, and it leads to some bad ways of doing things. I think it's better to do the work to make a library possible to cross-compile, than to give up and say MSYS(2) is the only method of compiling it for Windows. I've been cross-compiling Julia itself and all of its dependencies from both Linux and Cygwin for months, I very rarely touch MSYS(2) any more.

I think this is very common. But it means that people who really want their code to actually compile on Windows frequently have an inferior system with which to do it, because all the real hackers are cross compiling from Linux.

There's nothing wrong in that, it just leads to additional frustration for some people.

> We can't use the accepted gnu triples in MPIR and that has always been the case for GMP too. They don't give nearly enough information for our needs.

Of course they don't provide a complete story when you're worrying about assembly and various SIMD instruction sets. But that's not the GNU triple's job. That information should be determined and represented separately, you're not going to include this kind of information in the host prefix of the name of a cross-compiler executable for example.

Sure, but just to emphasise, we use the standard gnu triples in the first level of config.guess (except when it's broken). The hack I applied was to the second level, which uses the MPIR triples convention.

 

GMP is also a pretty bad actor in terms of its autotools usage - making a symlink to libgmp.dll called libgmp.lib in lib/ is completely the wrong thing to do for MinGW.

As I understand it, few of their core developers even have access to Windows boxes. They also reportedly don't support stack unwinding on Windows (I've never actually checked this) or provide Visual Studio project files. Their assembly code is just modified with lead-in and lead-out code so they can use the same assembly code for Windows and Linux, despite the differences in ABI. (It works relatively well, but it's a hack -- one that we use quite a bit too, I confess).

One enormous frustration for us is that MSVC has no inline assembler on Windows 64, and gas format assembly has not traditionally been available in the assemblers available for Windows. There are many, many problems that we work around that require many hacks, even on the linux side.

We could fix them all and make it less hackish, but you are talking about something like a year or two of full time work, just to fix all the build system issues alone!

Not your fault I know, but sounds like you may have inherited some autotools abuses from GMP.

 

 

Many of them are inherited from there. But remember that GMP has been in development since at least 1994! They did things back then that many other projects have not even thought about doing, to this day.

MPIR and GMP are *incredibly* complex projects and few people recognise the subtleties which simply don't arise for other types of software.

Bill.

[Tony Kelman]

> I think this is very common. But it means that people who really want their code to actually compile on Windows frequently have an inferior system with which to do it, because all the real hackers are cross compiling from Linux.

Or Cygwin. Part of the point is the exact same process should result in the same built binaries for cross-compilation from Linux or cross-compilation from Cygwin. Any complaint about Cygwin in terms of size or performance applies equally to MSYS2, and MSYS2's newness and multitude of build/run modes causes more trouble than it's worth in many cases.

> or provide Visual Studio project files

Can you blame them? Speaking of awful, impenetrable build systems that don't translate well to other platforms...

> MPIR and GMP are *incredibly* complex projects and few people recognise the subtleties which simply don't arise for other types of software.

I'm not familiar with writing this type of software, but plenty familiar with building it, and I really don't see a whole lot that goes into a bignum library that doesn't also happen in, say, an optimized BLAS/LAPACK implementation. Of course those are also notorious for having messy and fragile build systems.

[Bill Hart]

Sorry for the delay, but I have now added a function to flint which takes the location of the CPimport.txt file (which is in the flint source tree in the qadic directory) at runtime.

The function is void flint_set_cpimport(char * path); where path is a full path including filename of the (CPimport.txt) file.

This is now committed to the flint trunk branch in my github. We should be able to use this by calling it from Julia when initialising anything that requires Conway polynomials. I haven't made the necessary changes to Nemo yet, but most of Nemo works without the changes since only the finite field functionality actually use Conway polynomials.

This should allow us to move forward with a binary version of Nemo on Windows, as soon as we decide a good place to install the CPimport.txt file. I guess it could just sit in the Nemo directory and just be made part of the Nemo git checkout.

Bill.

[Tony Kelman]

Great. Bumping the git version to a51e15e53d636e38c107e48f1d8ec34a932590ab looks like it built okay: https://build.opensuse.org/package/show/home:kelman/mingw32-flint

We'll need to adjust Nemo.jl's deps/build.jl script to use BinDeps Autotools/WinRPM providers and library_dependency's instead of just running shell commands. Will also want to use the library location caching method instead of push!'ing to DL_LOAD_PATH.

[Bill Hart]

Thanks for checking that.

I'll try to work on those things this weekend. The amount of time I'll have to work on Nemo for the next two months will be smallish, but not completely zero. So we'll get it there eventually.

Bill.