Build system

[Jason Moxham]

Hi

Doing the unix-like configure-make-make-check got me thinking about the MPIR
build system. Here are some thoughts, some of this is a rehash of older
conversations/ideas.

At the moment we have autotools to take care the unix systems and VC projects
to take care of the Windows side. It would be nice if we had one system to do
both . Clearly VC projects only work on windows so this is no good , why wont
autotools run on windows. It does under cygwin , but this has problems of 1)
License ? not lgpl2? 2) require installation 3) requires runtime cygwin.dll .
Mingw/MSYS fairs better in that we dont require a runtime dll , the
installation could be done at build time , like we do for YASM at the
moment. Dont know about the license , but I think it may be more suited, if
not perhaps we could mix and match our own( we need bash,binutils... all of
which have been ported, I think). Perhaps this solution would better suit a
bigger project like SAGE , if SAGE shipped with a cutdown MSYS/Mingw then
porting could be fast , we could wrap MSVC's cl.exe so it looks like a
version of cc/gcc.

Another problem of autotools is the complexity , if you look at reported
errors for any package then a large percentage are autotools related ,
strictly speaking they are not the fault of autotools , but rather the
scripts we write to configure and make our packages. Because autotools must
work with every version of unix since the big bang , you have to work to the
lowest common denominator, which is time consuming and error prone. Autotools
brings all developers done to the lowest level , whereas I think it should
brought all systems *up to* a modern level, it should be a set of patches and
tools that users install to bring their systems up to stuff . There have been
thoughts of writing a new python build system , sounds attractive , but the
more I think about it , the more like a black hole it becomes for our time .
eg we would need to support parallel make, cross-compilation. I think we
really have to use a ready made solution?

If the existing build system was simpler then perhaps the above tasks may
become clearer , and it hopefully encourage people to help. One aspect that
could be cleaned is the mpn cpu selection and fat-build. The current cpu
selection (ie the sym-links) is done at configure time and is restrictive (ie
it wont cope with a new K8 asm function). There is also no need for this to
be done at build time , we could do it when we do autoreconf for example. The
configure mechanism at build time would be then simpler . Not sure how clean
up the fat build !!

Jason

[Gonzalo Tornaria]

Have you had a look at qmake (from Qt)?. I'm not sure it can be
adapted to MPIR, but it is a pretty decent build system. It has a
decent configuration language, and many targets including:
- gnu makefile (unix/linux)
- osx makefile (xcode)
- windows mingw
- windows msvc nmake
- windows msvc projects

etc.

In principle, from a single qmake project one can generate (by running
qmake) all of these.

I don't think one needs to link (use/distribute) qt libraries at all
to use qmake. AFAIK, qmake being part of Qt should be GPL and now LGPL
(not sure).

Gonzalo

[Cactus]

On Jul 13, 11:39 pm, Jason Moxham <ja...@njkfrudils.plus.com> wrote:
> Hi
>
> Doing the unix-like configure-make-make-check got me thinking about the MPIR
> build system. Here are some thoughts, some of this is a rehash of older
> conversations/ideas.
>
> At the moment we have autotools to take care the unix systems and VC projects
> to take care of the Windows side. It would be nice if we had one system to do
> both . Clearly VC projects only work on windows so this is no good , why wont
> autotools run on windows. It does under cygwin , but this has problems of 1)
> License ? not lgpl2? 2) require installation 3) requires runtime cygwin.dll .
> Mingw/MSYS fairs better in that we dont require a runtime dll , the
> installation  could be done at build time , like we do for YASM at the
> moment. Dont know about the license , but I think it may be more suited, if
> not perhaps we could mix and match our own( we need bash,binutils... all of
> which have been ported, I think). Perhaps this solution would better suit a
> bigger project like SAGE , if SAGE shipped with  a cutdown MSYS/Mingw then
> porting could be fast , we could wrap MSVC's cl.exe so it looks like a
> version of cc/gcc.

For me cygwin is a non starter for several reasons. First, it is only
32-bits and won't support 64-bits for a long time (if ever). Second,
I don't want to add a further extensive collection of tools to my
Windows systems for just one purpose.

Mingw is a better bet and is now available in both 32 and 64 bit mode
on Windows. I use this although I have not tried it with MPIR and I am
far from sure that its library outputs will work with VC++. It is
probably ok in 32-bit mode but there may still be issues with stack
unwinding support in 64-bits (this may now be ok though).

Wrapping MSVC to work in the *nix build system should work for 32-bit
builds but it would require a lot of modifications for 64-bit builds
since all the assembler files have to be different. But it looks like
it might be the easiest solution. It would require changes to the
Windows assembler files to align some of the symbols that need to be
recognised during builds (GLOBAL_FUNC etc.) but I have been thinking
about this in looking at a Python based build system. The assembler
components on Windows won't ever line up with those on *nix though
since I don't have the time to do this.

> Another problem of autotools is the complexity , if you look at reported
> errors for any package then a large percentage are autotools related ,
> strictly speaking they are not the fault of autotools , but rather the
> scripts we write to configure and make our packages. Because autotools must
> work with every version of unix since the big bang , you have to work to the
> lowest common denominator, which is time consuming and error prone. Autotools
> brings all developers done to the lowest level , whereas I think it should
> brought all systems *up to* a modern level, it should be a set of patches and
> tools that users install to bring their systems up to stuff . There have been
> thoughts of writing a new python  build system , sounds attractive , but the
> more I think about it , the more like a black hole it becomes for our time .
> eg we would need to support parallel make, cross-compilation. I think we
> really have to use a ready made solution?

The problem being that there isn't one spanning 32 and 64 bit builds :-
(

Brian

[Jason Moxham]

I think if we simplify the current build system , so that it resembles a
more standard package then we can have a better/easier choice of build
system.When I say simplify I mean , currently a lot of complication is in
the form of "bash in configure" , we want to take it out of configure so we
have a more standard package , and rewrite the "bash in configure" in python
so windows etc can use it.

[jason]

On Jul 14, 10:18 am, Cactus <rieman...@googlemail.com> wrote:
> On Jul 13, 11:39 pm, Jason Moxham <ja...@njkfrudils.plus.com> wrote:
>
>
>
>
>
> > Hi
>
> > Doing the unix-like configure-make-make-check got me thinking about the MPIR
> > build system. Here are some thoughts, some of this is a rehash of older
> > conversations/ideas.
>
> > At the moment we have autotools to take care the unix systems and VC projects
> > to take care of the Windows side. It would be nice if we had one system to do
> > both . Clearly VC projects only work on windows so this is no good , why wont
> > autotools run on windows. It does under cygwin , but this has problems of 1)
> > License ? not lgpl2? 2) require installation 3) requires runtime cygwin.dll .
> > Mingw/MSYS fairs better in that we dont require a runtime dll , the
> > installation  could be done at build time , like we do for YASM at the
> > moment. Dont know about the license , but I think it may be more suited, if
> > not perhaps we could mix and match our own( we need bash,binutils... all of
> > which have been ported, I think). Perhaps this solution would better suit a
> > bigger project like SAGE , if SAGE shipped with  a cutdown MSYS/Mingw then
> > porting could be fast , we could wrap MSVC's cl.exe so it looks like a
> > version of cc/gcc.
>
> For me cygwin is a non starter for several reasons. First, it is only
> 32-bits and won't support 64-bits for a long time (if ever).   Second,
> I don't want to add a further extensive collection of tools to my
> Windows systems for just one purpose.
>

I was thinking more along the lines of using cygwin as a "driver" to
cl.exe , so whether it is 32 or 64 bit shouldn't matter , it would be
a local install , like mpir has a local install of yasm , and yasm has
a local install of re2c. I agree cygwin is too big for our purposes ,
it's overkill.

> Mingw is a better bet and is now available in both 32 and 64 bit mode
> on Windows. I use this although I have not tried it with MPIR and I am
> far from sure that its library outputs will work with VC++.  It is
> probably ok in 32-bit mode but there may still be issues with stack
> unwinding support in 64-bits (this may now be ok though).
>
> Wrapping MSVC to work in the *nix build system should work for 32-bit
> builds but it would require a lot of modifications for 64-bit builds
> since all the assembler files have to be different.  But it looks like
> it might be the easiest solution.  It would require changes to the
> Windows assembler files to align some of the symbols that need to be
> recognised during builds (GLOBAL_FUNC etc.) but I have been thinking
> about this in looking at a Python based build system.  The assembler
> components on Windows won't ever line up with those on *nix though
> since I don't have the time to do this.
>

Perhaps think of it as a type of cross compilation.
Conceptually think of it like this.

We start with a standard Mingw/MSYS build , like we do now. ie
download mingw and install it , download mpir and install it. Now
replace referances to gcc with cl.exe in all makefiles etc , so we are
just building with cl.exe rather than gcc. Next we ship mingw with
mpir . Then we change the build process of mpir so that it installs
mingw in a temp place then installs mpir and then deletes mingw. So as
far as the user is concerned , they have a native build.

> > Another problem of autotools is the complexity , if you look at reported
> > errors for any package then a large percentage are autotools related ,
> > strictly speaking they are not the fault of autotools , but rather the
> > scripts we write to configure and make our packages. Because autotools must
> > work with every version of unix since the big bang , you have to work to the
> > lowest common denominator, which is time consuming and error prone. Autotools
> > brings all developers done to the lowest level , whereas I think it should
> > brought all systems *up to* a modern level, it should be a set of patches and
> > tools that users install to bring their systems up to stuff . There have been
> > thoughts of writing a new python  build system , sounds attractive , but the
> > more I think about it , the more like a black hole it becomes for our time .
> > eg we would need to support parallel make, cross-compilation. I think we
> > really have to use a ready made solution?
>
> The problem being that there isn't one spanning 32 and 64 bit builds :-
> (
>

or linux and windows , bastards wont talk to each other

>      Brian- Hide quoted text -
>
> - Show quoted text -

[William Stein]

On Mon, Jul 13, 2009 at 3:39 PM, Jason Moxham<ja...@njkfrudils.plus.com> wrote:
>
> Hi
>
> Doing the unix-like configure-make-make-check got me thinking about the MPIR
> build system. Here are some thoughts, some of this is a rehash of older
> conversations/ideas.
>
> At the moment we have autotools to take care the unix systems and VC projects
> to take care of the Windows side. It would be nice if we had one system to do
> both . Clearly VC projects only work on windows so this is no good , why wont
> autotools run on windows. It does under cygwin , but this has problems of 1)
> License ? not lgpl2? 2) require installation 3) requires runtime cygwin.dll .
> Mingw/MSYS fairs better in that we dont require a runtime dll , the
> installation  could be done at build time , like we do for YASM at the
> moment. Dont know about the license , but I think it may be more suited, if
> not perhaps we could mix and match our own( we need bash,binutils... all of
> which have been ported, I think). Perhaps this solution would better suit a
> bigger project like SAGE , if SAGE shipped with  a cutdown MSYS/Mingw then
> porting could be fast , we could wrap MSVC's cl.exe so it looks like a
> version of cc/gcc.
>

For Sage-for-windows it would be very reasonable to ship with a
msys/mingw or whatever else along those lines makes porting work
easier.

-- William

[Cactus]

Thanks for the further detail Jason - how would this drive the Windows
build.

I can see how to do this by driving the VC++ build system as you have
already done - is that what you are suggsting?

Or are you suggesting that some new makefiles are created to drive the
Windows builds?

If we want to fully integrate the WIndows build so that, for example,
FAT builds become possible, I will end up with quite a bit of work. I
am certainly not objeccting to this but I do need to understand the
magnitude of the task to assess if I have the time and the ability to
do it.

Brian

Brian

[Jason Moxham]

I think I may give a go , or at least a test project , say PARI


>
>

[Jason Moxham]

Not quite , or not like the windows Pari build.

> Or are you suggesting that some new makefiles are created to drive the
> Windows builds?
>

Again not quite.

> If we want to fully integrate the WIndows build so that, for example,
> FAT builds become possible, I will end up with quite a bit of work. I
> am certainly not objeccting to this but I do need to understand the
> magnitude of the task to assess if I have the time and the ability to
> do it.
>

At the moment I'm just playing with ideas, so I'm not sure what is possible
and/or practical. Ideally we have one build system that covers all systems ,
I'm fairly convinced it's possible , but it may be more effort than it's
worth. I'm going to try a demo to get a better idea of what I think it will
involve.


> Brian
>
> Brian
>
>

[Bill Hart]

Worth checking that the 64 bit MSYS actually works, e.g. with MPIR...
or did you already do that?

Sorry I've not been keeping up too well. Between computer failure and
my decision to work on an additional small FLINT release, I've had
trouble getting back to MPIR. But I'm nearly done with FLINT now.

Bill.

2009/7/16 Jason Moxham <ja...@njkfrudils.plus.com>:

[Jason Moxham]

On Thursday 16 July 2009 00:35:07 Bill Hart wrote:
> Worth checking that the 64 bit MSYS actually works, e.g. with MPIR...
> or did you already do that?
>

No , I didn't get around to it , I'll try it later