Flint --with-gmp instruction

[Oscar Benjamin]

Hi all,

Thanks for the excellent library!

Looking at the instructions here:
http://flintlib.org/doc/building.html

It says to tell ./configure about where GMP and FMPR are installed and
gives the example:

./configure --with-gmp=/home/user1/local/lib --with-mpfr=/home/user1/local/lib

I've installed GMP and MPFR with --prefix=$(pwd)/local and giving
--with-gmp=$(pwd)/local/lib fails but it seems that
--with-gmp=$(pwd)/local works. Maybe that instruction should be
changed because it suggests to me that I should append /lib.

I think I have it working now. Just waiting for the compile... Why
does it take so long to compile flint? I'm also compiling GMP, FMPR,
Arb and python_flint and Flint takes way longer than the others.

(I'm trying to set up prebuilt wheels for python-flint at the moment
https://github.com/fredrik-johansson/python-flint/issues/1)

Oscar

[Bill Hart]

Thanks for reporting this. Indeed the instructions are incorrect.

Flint takes a long time to compile because it is around 500,000 lines
of code. It is a huge library!

Bill.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "flint-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to flint-devel...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAHVvXxTUaGkdLHJuqqQDDNZGb2qK6FGL45trVCBmsb_b3BjRxw%40mail.gmail.com.

[Fredrik Johansson]

First, a suggestion if you're not already doing this: make sure you configure all libraries with either --disable-static or --disable-shared. Building both static and shared libraries will double compilation times.

Part of why Flint is so slow to build is that we have thousands of small files, and each file compilation requires launching the compiler and processing thousands of lines of header files. For fun, I tried to see what happens if we concatenate most .c files in each module (ignoring those with #defines and static methods, which may conflict). Quick Python script to do this:

import os

# some exceptions
fp = open("fq_poly_templates/sqr_reorder.c"); s = fp.read()
s = s.replace("__", "__sqr_"); fp.close()
fp = open("fq_poly_templates/sqr_reorder.c", "w"); fp.write(s); fp.close()
fp = open("fq_poly_templates/mul_reorder.c"); s = fp.read()
s = s.replace("__", "__mul_"); fp.close()
fp = open("fq_poly_templates/mul_reorder.c", "w"); fp.write(s); fp.close()

for mod in os.listdir("."):
    if os.path.isdir(mod) and "templates" not in mod:
        module_files = []
        for srcfile in os.listdir(mod):
            if srcfile.endswith(".c") and "inlines" not in srcfile and "everything" not in srcfile:
                srcfile = os.path.join(mod, srcfile)
                file_ok = True
                for line in open(srcfile).readlines():
                    if line.startswith("#define") or line.startswith("static"):
                        if not (line.startswith("#define T ") or line.startswith("#define CAP_T")):
                            file_ok = False
                    if not file_ok:
                        break
                if file_ok:
                    module_files.append(srcfile)
        if module_files:
            dest = open(os.path.join(mod, "everything.c"), "w")
            data = ""
            for f in module_files:
                data2 = open(f, "r").read()
                if "sqr_reorder.c" in f:
                    data2 = data2.replace("__", "__sqr_")
                data += data2 + "\n\n"
                os.remove(f)
            dest.write(data)

Time needed to build Flint originally:

real 8m29,040s
user 7m30,750s
sys 0m57,446s

With concatenation:

real 3m59,677s
user 3m49,346s
sys 0m8,657s

More than a 2x speedup. Of course, I don't seriously propose building Flint this way :-) I wonder if we could achieve the same speedup by 1) building each module with one compiler invocation to process multiple source files (assuming that compilers support and optimize for this), and 2) turning on header precompilation in some form? If 1) is doable somehow, we might not want it for development, but it could be useful for remote testing and deployment.

Another idea worth looking into is manually reducing header file bloat. We probably have far more inline functions than we really need (this slows down compilation and bloats the built library file size). Many #includes probably just serve to include a typedef, so we could look into putting types in a flint_types.h. Etc.

Fredrik

To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAB0xFns1OcULkO-ZMK-geH2NgDz38irX6_uE7btoUZyqrUvFrw%40mail.gmail.com.

[Mahrud Sayrafi]

Another suggestion is to use ccache, which has significantly improved my life personally. A little known advantage is that CI services like github actions support keeping a persistent storage between builds which can be used to store ccache artifacts, hence speeding up remote builds by a substantial amount. For instance, the nightly builds of Macaulay2 on github actions build everything up to the executable in ~11 minutes normally but in ~17 seconds (not minutes!) with cmake+ninja+ccache.

Of course, this doesn't help with the first build. So for #2 I would suggest trying out cmake's target_precompile_headers command, though I haven't used it myself.

Mahrud

To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAJdUXTKt5zV_%3DE6NQhU301hGGG32wT1-zT_9BHF1pPYNWp%3Dt9A%40mail.gmail.com.

[Bill Hart]

The original documentation issue is now:

https://github.com/wbhart/flint2/issues/923

> To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAJdUXTKt5zV_%3DE6NQhU301hGGG32wT1-zT_9BHF1pPYNWp%3Dt9A%40mail.gmail.com.

[Oscar Benjamin]

On Tue, 6 Apr 2021 at 08:31, Fredrik Johansson
<fredrik....@gmail.com> wrote:
>
> First, a suggestion if you're not already doing this: make sure you configure all libraries with either --disable-static or --disable-shared. Building both static and shared libraries will double compilation times.

No, I haven't been doing this. I didn't realise it would literally
double the time.

I'm also wondering whether static or shared should be preferred in
this scenario. The wheel I built is:

python_flint-0.3.0-cp38-cp38-macosx_10_9_x86_64.whl

Internally it has:

$ ls -AR flint/
.dylibs __init__.py _flint.cpython-38-darwin.so

flint//.dylibs:
libarb-2.10.0.dylib libflint-15.dylib libgmp.10.dylib libmpfr.6.dylib

I'm not sure I know enough about how these things work but is there
any possibility that these could conflict with another Python
extension module using different versions of the same libraries (e.g.
gmpy2 using some other libgmp.x or a different version of libgmp.10).
I presume using an actual system gmp would be better in that scenario
because then it is shared but here I think I definitely want
everything bundled and isolated.


Oscar

[Bill Hart]

I believe Python will want shared libraries.

I don't know much about Python extension libraries, so can't answer
your other questions. However, I think it probably boils down to the
behaviour of the linker. Unfortunately I am not sure if Python manages
this themselves or relies on the standard behaviour of the system
linker.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "flint-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to flint-devel...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAHVvXxSfkaNFQU6LKwub2e1ax_FTN%2BvSs1n5M1M39q8sotFqJg%40mail.gmail.com.

[Oscar Benjamin]

On Tue, 6 Apr 2021 at 09:34, Mahrud Sayrafi <sayr...@gmail.com> wrote:
>
> Another suggestion is to use ccache, which has significantly improved my life personally. A little known advantage is that CI services like github actions support keeping a persistent storage between builds which can be used to store ccache artifacts, hence speeding up remote builds by a substantial amount. For instance, the nightly builds of Macaulay2 on github actions build everything up to the executable in ~11 minutes normally but in ~17 seconds (not minutes!) with cmake+ninja+ccache.

Using ccache and CI caching seems like a very good idea. I'll have to
give it a try...

I see this is where it happens for Macaulay2:
https://github.com/Macaulay2/M2/blob/5251daa59628663f9428c4eed062773b543b0624/.github/workflows/test_build.yml#L99-L119


Oscar

[Oscar Benjamin]

On Tue, 6 Apr 2021 at 11:06, 'Bill Hart' via flint-devel
<flint...@googlegroups.com> wrote:
>
> I believe Python will want shared libraries.
>
> I don't know much about Python extension libraries, so can't answer
> your other questions. However, I think it probably boils down to the
> behaviour of the linker. Unfortunately I am not sure if Python manages
> this themselves or relies on the standard behaviour of the system
> linker.

Python wants the main importable module to be a shared library. In
this case that means that I need to produce a _flint module that can
be loaded as a shared library. The question is more about the way that
module accesses its dependencies i.e. GMP, FMPR, Flint and Arb. I
think those could be statically linked into the main .so file. In any
case I have this working without static linking right now so this is
not a priority.

I have this partially sorted now in the sense that I can make sure
that wheels get built for OSX and Linux for PYthon 3.7 and 3.8. The
built wheels are in the artifact zip here:
https://github.com/oscarbenjamin/python-flint/actions/runs/727631490

I've sent that as a PR to python-flint:
https://github.com/fredrik-johansson/python-flint/pull/16

There are some issues to iron out such as newer Cython versions and
also making wheels for newer Python versions (these are related). The
big elephant in the room though is Windows. I don't have easy access
to a Windows machine and I see a bunch of problems with Windows:

1) I think that recent Python versions are compiled with MSVC in such
a way that extension modules also need to be compiled with MSVC in
order to link reliably (I'm not sure how contagious this effect is,
like do I need to use MSVC for *everything*?).

2) It doesn't look like GMP can be compiled with MSVC without patches.

3) It doesn't seem straight-forward to compile MPIR with MSVC either.

If anyone has any advice/experience with getting this sort of thing
set up on Windows then that would be great. I have looked at related
projects like gmpy but I don't see anything that fully automates
compiling these kinds of libraries in CI (for Windows). If anyone
wants to help with this then that would be fantastic.


Oscar

[Bill Hart]

Flint and its dependencies build just fine with MSVC. Check out the
CMake build in the repo, which does just this. The appveyor CI makes
use of it.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "flint-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to flint-devel...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAHVvXxT3H8pP0PEwaqDJmpGC-RWHNchODF-RfwMk%2BH543bW%2B3w%40mail.gmail.com.

[Oscar Benjamin]

On Thu, 8 Apr 2021 at 01:04, 'Bill Hart' via flint-devel
<flint...@googlegroups.com> wrote:
>
> Flint and its dependencies build just fine with MSVC. Check out the
> CMake build in the repo, which does just this. The appveyor CI makes
> use of it.

Thanks.

It looks like it's using conda to install prebuilt versions of the dependencies:
https://github.com/wbhart/flint2/blob/f7918785866e09fdab6ac23925dcf7bcb25c97c3/appveyor.yml#L49

Have I understood that correctly?


Oscar

[Bill Hart]

Oh yes, I forgot about that. Still you should be able to look at the
Conda scripts.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "flint-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to flint-devel...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/flint-devel/CAHVvXxSBtM%3D0_YBb2G4g3Pue%3D5noBet7DEdLv%3DfkemYY7D-%2BZg%40mail.gmail.com.