More on ntl
François Bissey
It's been quite on debian-sage lately. So I am posting today
about some of my endeavours since the ntl on Solaris upsets.
It has been one of my hitches ever since to produce something
that would please everyone (tall order) and that I would hope
upstream would be happy with. So the list of requirements is:
*no gnu-ism unless we have ascertained they are available and
then as little as possible (more on that below)
*satisfy everyone - i.e. it is easy to target
*easily extensible for new targets in the future if possible
*elegant
I hope I have some success with that but I am posting here
(rather than on sage-devel) to test the waters.
So here is what I did:
*included a new PICFLAG option that can be configured at
configuration time. It default to -fPIC but in sage it should
be set to $SHAREDFLAGS so it will always be right.
*included a target option for shared libraries called
SHMAKE it can currently take the following values:
- non-gld (default it would work on linux and solaris
with gcc and sun ld)
-linux (some gnu-ism it add -Wl,soname ... to the
previous one)
-debian (with versionning for debian obviously)
-osx (current behavior for osx)
-osx-64 (current behavior for osx-64)
-cygwin (current behavior for cygwin)
SHMAKE is set during the configuration phase, to build the
shared library just issue 'make shared' and it will do the right
thing. This is one of the extensible bits the target is included
from a file called $SHMAKE.mk to add a new target just add
a .mk file with an appropriate name and content and you are
set. GNU-ism can appear in the .mk file as they should match
a target where they are available.
Other point the 'all' target could be changed to include shared
if desired (at upstream leisure I guess) and it wouldn't break
parallel make. The previous stuff we did with Timothy and that
has landed all over the place as "the ntl treatment" can break
parallel make - it happened to me with tachyon.
In attachment are the files for anyone who want to give it a
spin. The .mk files go in src and the patch is applied from that
folder too (it will touch DoConfig, mfile and Wizard).
Cheers,
Francois
Timothy G Abbott
without -fPIC but libntl.so would be built with -fPIC? That feature is
required by Debian policy.
I think -debian is the wrong name for the shared library versioning
feature, because any binary distribution requires it, and it's weird for
Fedora to be using an option called "debian". This is especially true
because library versioning really should be the same across all
distributions. It's harmless for source-based linux distributions like
Gentoo to be using versioning, and so I think whatever patch we submit
upstream should do correct library versioning on any linux platform.
The right solution to the problems we are facing is probably libtool,
something which was designed to do the right thing with libraries on every
platform. If you look at the libraries that we've packaged that don't
have library policy problems, essentially all of them are using it.
-Tim Abbott
François Bissey
On Sat, 28 Jun 2008, Timothy G Abbott wrote:
> So this patch is dropping the feature where libntl.a would be built
> without -fPIC but libntl.so would be built with -fPIC? That feature is
> required by Debian policy.
>
No you got it wrong. 'make all' will still build libntl.a correctly.
Now with this patch I have 2 kind of objects. The regular .o for
static linking, and .lo for dynamic linking (inspired from libtool I
admit) compiled with the appropriate PIC flag. To build the
dynamic object you have to call 'make shared'.
> I think -debian is the wrong name for the shared library versioning
> feature, because any binary distribution requires it, and it's weird for
> Fedora to be using an option called "debian". This is especially true
> because library versioning really should be the same across all
> distributions. It's harmless for source-based linux distributions like
> Gentoo to be using versioning, and so I think whatever patch we submit
> upstream should do correct library versioning on any linux platform.
>
Versioning is fine and useful in many case. But really in some situation
it is overkill. But yes it should be the same across distros. Anyway I am
not sure you looked at it right the "debian" option just does the versioning
we initially settled on. I guess we could rename debian.mk to linux.mk
and forget about the current linux.mk. Sage probably would go with non-gld
as the default as it is what is now used.
> The right solution to the problems we are facing is probably libtool,
> something which was designed to do the right thing with libraries on every
> platform. If you look at the libraries that we've packaged that don't
> have library policy problems, essentially all of them are using it.
You are right. My current solution is just a good band aid (I hope, I just
spotted a potential issue with using include but I think I have a work around
if it doesn't work with BSD make).
I guess we could auto/libtool (or possibly scons if it is capable) everything
in sight but Michael won't be very supportive and that will take time.
And it fells so overkill for stuff like zn_poly, although m4ri is also
relatively small and yet it is libtooled.
Anyway, like I said I am testing the water here. This is were we discuse
issue so I guess I got what I wanted.
Cheers,
Francois
François Bissey
the patch. The previous one had a problem with net-bsd make
that escaped me (but was OK on David Kirkby's machine). I
also don't need to patch "Wizard" anymore by being a little more
subtle with what I do.
Cheers
Francois
Timothy G Abbott
substitutions to generate SHOBJ from OBJ, i.e.
SHOBJ = $(OBJ:%.o=%.lo)
but I don't know whether that feature is specific to GNU make.
This is quite specific to NTL, but ctools.c and GetTime.c should just be
using the default .c.lo rule since their rules are precisely the same as
the default.
Where does $MAKE get set if it isn't in the user's environment?
-Tim Abbott
Timothy G Abbott
> Hi Tim,
>
> On Sat, 28 Jun 2008, Timothy G Abbott wrote:
>> So this patch is dropping the feature where libntl.a would be built
>> without -fPIC but libntl.so would be built with -fPIC? That feature is
>> required by Debian policy.
>>
> No you got it wrong. 'make all' will still build libntl.a correctly.
> Now with this patch I have 2 kind of objects. The regular .o for
> static linking, and .lo for dynamic linking (inspired from libtool I
> admit) compiled with the appropriate PIC flag. To build the
> dynamic object you have to call 'make shared'.
Ahh, OK. Makes sense. I would set things up so the dynamic object is
built by the "all" target, since that's what most libraries do. It's
silly to need to call "make" twice in order to build the entire package.
> Versioning is fine and useful in many case. But really in some situation
> it is overkill. But yes it should be the same across distros. Anyway I am
> not sure you looked at it right the "debian" option just does the versioning
> we initially settled on. I guess we could rename debian.mk to linux.mk
> and forget about the current linux.mk. Sage probably would go with non-gld
> as the default as it is what is now used.
Sounds reasonable.
> You are right. My current solution is just a good band aid (I hope, I just
> spotted a potential issue with using include but I think I have a work around
> if it doesn't work with BSD make).
> I guess we could auto/libtool (or possibly scons if it is capable) everything
> in sight but Michael won't be very supportive and that will take time.
> And it fells so overkill for stuff like zn_poly, although m4ri is also
> relatively small and yet it is libtooled.
I would like to hear Michael's thoughts on this sort of thing. My main
concern with automaking things is that the upstream developers won't know
automake and may not be receptive to replacing their build system with
something they don't understand.
-Tim Abbott
François Bissey
Good to know you had a look at it.
On Sun, 29 Jun 2008, Timothy G Abbott wrote:
> You should be able to make the patch more maintainable by using make
> substitutions to generate SHOBJ from OBJ, i.e.
>
> SHOBJ = $(OBJ:%.o=%.lo)
>
> but I don't know whether that feature is specific to GNU make.
>
As far as I know it is a GNU-ism which is why I have an old fashioned
.c.lo: rule and a .SUFFIXES (which is not needed with the kind of
construct you suggest).
> This is quite specific to NTL, but ctools.c and GetTime.c should just be
> using the default .c.lo rule since their rules are precisely the same as
> the default.
>
No. Read more carefully. ctools.c and GetTime.c are compiled with the
C compiler while the rule is for objects build with the C++ compiler. This
is why there is a separate rule for those, both for the static and my new
dynamic rules.
> Where does $MAKE get set if it isn't in the user's environment?
>
By default for all versions of make that I know that's the make running the
current makefile. This is needed for sanity, should definitely be picked up
upstream and should always be used if you write a makefile yourself.
A real life example that happened with ntl to David Kirkby on his solaris
machine. He tried to build ntl with our patch by using gmake. Because the
makefile directly issue call to 'make' at the first call of make, sun make was
called instead of GNU make and failed because of the introduced GNU-ism.
With $(MAKE) instead of 'make', gmake would have been called all the way.
As for Michael opinion on auto/libtool I recommend this message:
http://groups.google.com/group/sage-devel/msg/01d2e3c16a892934
Cheers,
Francois
Timothy G Abbott
> By default for all versions of make that I know that's the make running the
> current makefile. This is needed for sanity, should definitely be picked up
> upstream and should always be used if you write a makefile yourself.
> A real life example that happened with ntl to David Kirkby on his solaris
> machine. He tried to build ntl with our patch by using gmake. Because the
> makefile directly issue call to 'make' at the first call of make, sun make was
> called instead of GNU make and failed because of the introduced GNU-ism.
> With $(MAKE) instead of 'make', gmake would have been called all the way.
>
> As for Michael opinion on auto/libtool I recommend this message:
> http://groups.google.com/group/sage-devel/msg/01d2e3c16a892934
Well, if Michael is opposed to automake, we can try using libtool without
using automake, which might allow us to get everything working without
having to write any substantial custom code.
-Tim Abbott
François Bissey
can you show a working case with say, zn_poly for example?
Francois
mabshoff
On Jul 5, 12:50 pm, Timothy G Abbott <tabb...@MIT.EDU> wrote:
<SNIP>
Hi Tim,
> Well, if Michael is opposed to automake, we can try using libtool without
> using automake, which might allow us to get everything working without
> having to write any substantial custom code.
>
> -Tim Abbott
"autohell" in all possible cases. Since upstream has so far chosen not
to go that route I would prefer if for Debian you ended up patching
the build system as you build the deb. If upstream was happy to
accept an autohell version of the build system I would not oppose it
since as is the shell/Perl based on is far from ideal anyway :)
Cheers,
Michael
François Bissey
the auxilliary files and then the current system will do all we currently
want without too much of a hassle.
Francois
Timothy G Abbott
It's definitely possible. Basically, you replace all instances of
$(CC) -> $(LIBTOOL) --mode=compile $(CC)
$(LD) -> $(LIBTOOL) --mode=link $(LD)
$(INSTALL) -> $(LIBTOOL) --mode=install $(INSTALL)
rm -f -> $(LIBTOOL) --mode=clean rm -f
and you replace all foo.o targets with foo.lo targets, and foo.a targets
with foo.la targets.
One example of doing this for another project I'm involved with is here:
http://web.mit.edu/broder/Public/shared-moira.patch
I'll try to do a more relevant proof-of-concept when I get a chance, but
I've been very busy of late.
-Tim Abbott