error building barvinok (sage 8.9.beta8 + system NTL)

[Vincent Delecroix]

Dear all,

I am not able to build the optional package barvinok. Note that
my Sage is built such that it uses GMP and NTL from the system
(archlinux). I opened

https://trac.sagemath.org/ticket/28406


Vincent

[Victor Shoup]

I reviewed some comments which mentioned a problem with ntl and threads. I’m happy to fix that, but I don’t think I understand what the issue is. Can anyone explain? Thanks.

[Dima Pasechnik]

On Tue, 27 Aug 2019 17:25 Victor Shoup, <sh...@cs.nyu.edu> wrote:

I reviewed some comments which mentioned a problem with ntl and threads. I’m happy to fix that, but I don’t think I understand what the issue is. Can anyone explain?

if NTL is built with multithreading then it does not link in the corresponding library, e.g. libpthread, which is not standard thing to do (i.e. this is "underlinking") . It can be worked around by NTL users, by supplying extra -l (e.g. -lpthread) to the linker, but this is extra hassle, as it requires tricky extra tests at configure stage, extra flags to mess around with.

Thanks.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/498a5033-c89e-48bf-a520-6dd7f6cac6b0%40googlegroups.com.

[Antonio Rojas]

El martes, 27 de agosto de 2019, 16:25:12 (UTC+2), Victor Shoup escribió:

I reviewed some comments which mentioned a problem with ntl and threads. I’m happy to fix that, but I don’t think I understand what the issue is. Can anyone explain? Thanks.

Hi Victor,

 IIRC I reported this to you about a year ago. The problem is that you are using libtool as a build command, which calls the compiler with the -nostdlib flag, which in turn overrides the -pthread flag, so the binaries end up not being linked to libpthread. See eg. [1][2] for more info.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=661333

[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460  

[Victor Shoup]

Thanks. I guess what I'm asking for is a solution.  From what you say here, and what is said in the links, the problem seems to be a bug in libtool, not NTL.  So a solution would be, either:

1) a patch other type of libtool workaround, or

2) an alternative to libtool.

I though the whole point of libtool was to take care of all this nonsense, and if it's not doing that, then

it seems kind of pointless.

[Vincent Delecroix]

Victor, as far as I understand the main configuration script of ntl
is the perl DoConfig script. It has nothing to do with libtool. libtool
is robust if you let it handle the configuration. It will not try to
fix a given one.

In a libtool configure.ac script you would just have a directive
AC_CHECK_LIB for pthread.

In short, I would suggest

3) Replace DoConfig by a configure.ac script

Vincent

[Antonio Rojas]

El miércoles, 28 de agosto de 2019, 15:15:18 (UTC+2), Victor Shoup escribió:

Thanks. I guess what I'm asking for is a solution.  From what you say here, and what is said in the links, the problem seems to be a bug in libtool, not NTL.  So a solution would be, either:

1) a patch other type of libtool workaround, or

2) an alternative to libtool.

I though the whole point of libtool was to take care of all this nonsense, and if it's not doing that, then

it seems kind of pointless.

I'd say the easiest fix would be to treat libpthread just like any other library and link to it by passing -lpthread to the linker instead of relying on the -pthread gcc flag (I don't speak perl so I can't help with a patch, sorry) 

[John Cremona]

On Wed, 28 Aug 2019 at 14:55, Vincent Delecroix <20100.d...@gmail.com> wrote:

Victor, as far as I understand the main configuration script of ntl
is the perl DoConfig script. It has nothing to do with libtool. libtool
is robust if you let it handle the configuration. It will not try to
fix a given one.

In a libtool configure.ac script you would just have a directive
AC_CHECK_LIB for pthread.

In short, I would suggest

3) Replace DoConfig by a configure.ac script

Good luck with that.  Feel free to consult eclib's configure.ac which was 90% written by people more knowledgeable than me, and which ensures that NTL works for eclib.

John

 

Vincent

Le 28/08/2019 à 15:15, Victor Shoup a écrit :
> Thanks. I guess what I'm asking for is a solution.  From what you say here,
> and what is said in the links, the problem seems to be a bug in libtool,
> not NTL.  So a solution would be, either:
> 1) a patch other type of libtool workaround, or
> 2) an alternative to libtool.
> I though the whole point of libtool was to take care of all this nonsense,
> and if it's not doing that, then
> it seems kind of pointless.
>
> On Tuesday, August 27, 2019 at 1:42:51 PM UTC-4, Antonio Rojas wrote:
>>
>>
>>
>> El martes, 27 de agosto de 2019, 16:25:12 (UTC+2), Victor Shoup escribió:
>>>
>>> I reviewed some comments which mentioned a problem with ntl and threads.
>>> I’m happy to fix that, but I don’t think I understand what the issue is.
>>> Can anyone explain? Thanks.
>>
>>
>> Hi Victor,
>>   IIRC I reported this to you about a year ago. The problem is that you are
>> using libtool as a build command, which calls the compiler with the
>> -nostdlib flag, which in turn overrides the -pthread flag, so the binaries
>> end up not being linked to libpthread. See eg. [1][2] for more info.
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=661333
>> [2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460
>>
>

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/fa916b1e-c62c-d64d-485e-a84b736a4bff%40gmail.com.

[Dima Pasechnik]

Victor,
as you ship libtool, you can just patch it, essentially as in
https://lists.gnu.org/archive/html/libtool-patches/2013-11/msg00015.html
It needs to be moved to a slightly different place, but it works,
you'd get a properly linked libntl.so with the patch (attached), I
checked this on Linux at
least.
HTH
Dima

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/e48173fb-ec1f-4e69-af58-19dc6035b32d%40googlegroups.com.

[Antonio Rojas]

El miércoles, 28 de agosto de 2019, 23:47:43 (UTC+2), Dima Pasechnik escribió:

Victor,
as you ship libtool, you can just patch it, essentially as in
https://lists.gnu.org/archive/html/libtool-patches/2013-11/msg00015.html
It needs to be moved to a slightly different place, but it works,
you'd get a properly linked libntl.so with the patch (attached), I
checked this on Linux at
least.

Relying on patched bundled copies of dependencies is very much frowned upon by distros. It would be much preferable to make this work with unmodified libtool, which may be provided by the distribution. 

[Dima Pasechnik]

Well, libtool is just a shell script, more or less. NTL isn't unique
in shipping its own version.
Does Arch use its own libtool when building NTL, or the bundled one?

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/fe05da99-083f-4283-ab3f-14249d14dc15%40googlegroups.com.

[Antonio Rojas]

El jueves, 29 de agosto de 2019, 0:06:49 (UTC+2), Dima Pasechnik escribió:

> Relying on patched bundled copies of dependencies is very much frowned upon by distros. It would be much preferable to make this work with unmodified libtool, which may be provided by the distribution.

Well, libtool is just a shell script, more or less. NTL isn't unique
in shipping its own version.
Does Arch use its own libtool when building NTL, or the bundled one?

Regardless of how simple if may be, bundling dependencies increases the risk of introducing bugs and inconsistencies between distro packages. On Arch we are building ntl using system's libtool.

Anyway, whatever solution is adopted is not such a big deal since there is an easy workaround (pass LDLIBS='-lpthread' to configure) that distros can keep using - It would just be nicer to fix it without resorting to patched dependencies IMO. 

[Victor Shoup]

Thanks for all of the thoughtful replies.

I don't have time to rewrite the config script any time soon.

According to the libtool documentation, it is supposed to be usable

without relying on automake.

NTL's config script allows you to override the default libtool with a system libtool,

so that seems like the simplest way to deal with this for now.

If I rewrite any of NTL's config scripts, it will likely be to move it over to cmake.

[Dima Pasechnik]

On Thu, Aug 29, 2019 at 5:13 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> Thanks for all of the thoughtful replies.
>
> I don't have time to rewrite the config script any time soon.
> According to the libtool documentation, it is supposed to be usable
> without relying on automake.
> NTL's config script allows you to override the default libtool with a system libtool,
> so that seems like the simplest way to deal with this for now.

I must say I didn't verify that in my experiment the bundled libtool was used
(I thought it's the default, but I don't know).

If it's not the case then on the Debian 10 (buster) system I tried running this,
the system libtool is already fixed in some way to fix that `-pthread` issue.
(and not that the patch I provided actually does the job)

Dima

>
> If I rewrite any of NTL's config scripts, it will likely be to move it over to cmake.
>
>
> On Thursday, August 29, 2019 at 3:43:33 AM UTC-4, Antonio Rojas wrote:
>>
>>
>>
>> El jueves, 29 de agosto de 2019, 0:06:49 (UTC+2), Dima Pasechnik escribió:
>>>
>>>
>>> > Relying on patched bundled copies of dependencies is very much frowned upon by distros. It would be much preferable to make this work with unmodified libtool, which may be provided by the distribution.
>>>
>>> Well, libtool is just a shell script, more or less. NTL isn't unique
>>> in shipping its own version.
>>> Does Arch use its own libtool when building NTL, or the bundled one?
>>>
>>
>> Regardless of how simple if may be, bundling dependencies increases the risk of introducing bugs and inconsistencies between distro packages. On Arch we are building ntl using system's libtool.
>> Anyway, whatever solution is adopted is not such a big deal since there is an easy workaround (pass LDLIBS='-lpthread' to configure) that distros can keep using - It would just be nicer to fix it without resorting to patched dependencies IMO.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/43feca19-aecc-42f4-a8a3-0e15472ada37%40googlegroups.com.

[Victor Shoup]

No.  System libtool is not the default.

It is done the way it is because of advice I got from some other folks who

knew more about these things than I did.

Try configure with LIBTOOL=<system lib tool>

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Victor Shoup]

A clarification and a question...

NTL's config system does not use a pre-built ibtool script.  Rather, it builds a customized libtool by running a configure script on the host machine, so it should, in principle, be using a libtool script that is properly configured for the host machine.

The configure script itself was built on a linux machine with up-to-date autotools using the following configure.ac file:

AC_INIT(ntl-libtool, 1.0)

AM_INIT_AUTOMAKE([foreign])

AC_CONFIG_FILES([Makefile])

LT_INIT

AC_PROG_CXX

AC_PROG_CC

AC_PROG_LIBTOOL

AC_OUTPUT

I don't remember where I got this...but it was from somebody who seemed to know what they were talking about.

Maybe the logic of this configure.ac file is not right?

Any thoughts on this?

[Dima Pasechnik]

On Mon, Sep 2, 2019 at 11:37 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> A clarification and a question...
>
> NTL's config system does not use a pre-built ibtool script. Rather, it builds a customized libtool by running a configure script on the host machine, so it should, in principle, be using a libtool script that is properly configured for the host machine.
>
> The configure script itself was built on a linux machine with up-to-date autotools using the following configure.ac file:
>
> AC_INIT(ntl-libtool, 1.0)
> AM_INIT_AUTOMAKE([foreign])
> AC_CONFIG_FILES([Makefile])
> LT_INIT
> AC_PROG_CXX
> AC_PROG_CC
> AC_PROG_LIBTOOL
> AC_OUTPUT
>
> I don't remember where I got this...but it was from somebody who seemed to know what they were talking about.
>
> Maybe the logic of this configure.ac file is not right?
> Any thoughts on this?

it looks OK, and the libtool you get will ignore -pthread option for gcc/g++,
(as this is by design), unless you apply the patch I posted earlier before
buiding it.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/30264407-95c4-4057-a7a8-5dc0ea8b6437%40googlegroups.com.

[Victor Shoup]

OK.  So this is really a bug in libtool.

It seems to me that even if I used autotools for everything, the same problem would persist

without a patch to ltmain.sh. 

 Is that right?

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Dima Pasechnik]

On Tue, Sep 3, 2019 at 1:05 AM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> OK. So this is really a bug in libtool.

it's a feature :-)
It's not hard to explicitly pass extra libraries to link to libtool,
and thus it's still like this.

> It seems to me that even if I used autotools for everything, the same problem would persist
> without a patch to ltmain.sh.
>
> Is that right?
>

no, if you use autotools, you'd just add a couple of macros to deal with
threads to configure.ac, and the rest will be done correctly,
as autotools are aware of this bug/feature of libtool and know
what to do.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/77278251-5ad4-42a2-a9f0-c0bf322fb972%40googlegroups.com.

[John Cremona]

On Mon, 2 Sep 2019 at 23:24, Dima Pasechnik <dim...@gmail.com> wrote:

On Tue, Sep 3, 2019 at 1:05 AM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> OK.  So this is really a bug in libtool.
it's a feature :-)
It's not hard to explicitly pass extra libraries to link to libtool,
and thus it's still like this.

> It seems to me that even if I used autotools for everything, the same problem would persist
> without a patch to ltmain.sh.
>
>  Is that right?
>
no,  if you use autotools, you'd just add a couple of macros to deal with
threads to configure.ac, and the rest will be done correctly,
as autotools are aware of this bug/feature of libtool and know
what to do.

In eclib, in configure.ac (https://github.com/JohnCremona/eclib/blob/master/configure.ac) I have the line AX_PTHREAD but also put in some explicit fplags into the Makefile templates, e.g. https://github.com/JohnCremona/eclib/blob/master/libsrc/Makefile.am has the lines

AM_CPPFLAGS = $(FLINT_CFLAGS) $(NTL_CFLAGS) $(PARI_CFLAGS) $(BOOST_CPPFLAGS) $(MPFP_SWITCH) $(PTHREAD_CFLAGS)
AM_LDFLAGS  = $(FLINT_LDFLAGS) $(NTL_LDFLAGS) $(PARI_LDFLAGS) $(LDFLAGS_BOOST) $(PTHREAD_CFLAGS) $(PTHREAD_LDFLAGS)
LIBS        = $(FLINT_LIBS) $(NTL_LIBS) $(PARI_LIBS) $(BOOST_LIBS) $(PTHREAD_LIBS)

I would expect not to have to "manually" add these PTHREAD_* items if libtool worked as expected.

It took a lot of experimentation to get something which worked, and I don't know if it is right, but it does work.  I am *not* an expert on this!

John

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq1Bie-zogVmyK4rKBoLycQF9cqnEibvNrJGaophUeoVkA%40mail.gmail.com.

[Victor Shoup]

So, I'm willing to patch libtool using your patch, as you suggest.

Just to confirm: I just replace ltmain.sh in libtool-origin with the patched version,

and that's it.

I have another question...

I've installed NTL on Linux with libtool, and I've never had a problem.

When I compile a program that uses NTL, I always pass -pthread to g++.

Doesn't that automatically cause the necessary pthread-specific library to be linked in?

As such, I'm still kind of confused as to what the problem is.

Related to that, from what I read, -lpthread is less standard than -pthread:

again, the latter is supposed to add the necessary linker flags.

Thus, I'm still a bit hesitant to do this.

I guess I still don't entirely understand the model.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/77278251-5ad4-42a2-a9f0-c0bf322fb972%40googlegroups.com.

[Dima Pasechnik]

On Tue, Sep 3, 2019 at 2:21 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> So, I'm willing to patch libtool using your patch, as you suggest.
> Just to confirm: I just replace ltmain.sh in libtool-origin with the patched version,
> and that's it.

yes.

>
> I have another question...
> I've installed NTL on Linux with libtool, and I've never had a problem.
> When I compile a program that uses NTL, I always pass -pthread to g++.
> Doesn't that automatically cause the necessary pthread-specific library to be linked in?

it does, if you don't use libtool to link (and typically one would
link an executable with
an explicit call to g++).

> As such, I'm still kind of confused as to what the problem is.

the problem is that libtool strips away the -pthread flag.
As long as you don't use it, there is no problem. See
https://www.gnu.org/software/libtool/manual/libtool.html#Stripped-link-flags
"13.1 Why does libtool strip link flags when creating a library?

When creating a shared library, but not when compiling or creating a
program, libtool drops some flags from the command line provided by
the user. This is done because flags unknown to libtool may interfere
with library creation or require additional support from libtool, and
because omitting flags is usually the conservative choice for a
successful build."

>
> Related to that, from what I read, -lpthread is less standard than -pthread:
> again, the latter is supposed to add the necessary linker flags.
> Thus, I'm still a bit hesitant to do this.

well, see above. As you elect to explictly call libtool, you have no
choice but to use "-lpthread"

Dima

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/94761df3-02c7-4304-b218-81607c13aa21%40googlegroups.com.

[Victor Shoup]

Please see comments/questions below...

On Wednesday, September 4, 2019 at 6:04:42 AM UTC-4, Dima Pasechnik wrote:

On Tue, Sep 3, 2019 at 2:21 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>

> When I compile a program that uses NTL, I always pass -pthread to g++.
> Doesn't that automatically cause the necessary pthread-specific library to be linked in?

it does, if you don't use libtool to link (and typically one would
link an executable with
an explicit call to g++).

So, based on that comment, it seems that there is no problem if you link with g++

rather than libtool.  But who or what uses libtool to link?

I would never have thought of doing that, as I thought libtool was a tool

for creating libraries, not linking programs.

the problem is that libtool strips away the -pthread flag.
As long as you don't use it, there is no problem. See
https://www.gnu.org/software/libtool/manual/libtool.html#Stripped-link-flags
"13.1 Why does libtool strip link flags when creating a library?

When creating a shared library, but not when compiling or creating a
program, libtool drops some flags from the command line provided by
the user. This is done because flags unknown to libtool may interfere
with library creation or require additional support from libtool, and
because omitting flags is usually the conservative choice for a
successful build."

This is confusing. If you use libtool to link, then the above quote would suggest that

in that context, libtool should not strip away any linker flags.

But you seem to be saying that it does.

So...I am still confused.

Regarding the libtool that I am bundling with NTL:

1) is it stripping some flags during compilation that should not be stripped?

2) is it stripping some flags during linking that should not be stripped?

Again, when I build and install a shared NTL on a linux system, everything works fine.

So I'm not even sure what the problem is:

I can build it and link to it in the build directory using libtool.

I can install it using libtool somewhere else and later link to it using g++.

What am I missing?!?

[Antonio Rojas]

El jueves, 5 de septiembre de 2019, 6:07:13 (UTC+2), Victor Shoup escribió:

 

So, based on that comment, it seems that there is no problem if you link with g++

rather than libtool.  But who or what uses libtool to link?

I would never have thought of doing that, as I thought libtool was a tool

for creating libraries, not linking programs

But you do! This is from your mfile:

ntl.a: $(OBJ)

[...]

@{LSHAR} $(LIBTOOL) --tag=CXX --mode=link $(LINK) $(LIBTOOL_LINK_FLAGS) -o libntl.la $(OBJ:.o=.lo) $(GMP_OPT_LIBDIR) $(GMP_OPT_LIB) $(GF2X_OPT_LIBDIR) $(GF2X_OPT_LIB) $(LDLIBS) -rpath $(LIBDIR) -version-info `cat VERSION_INFO` #LSHAR

If you check the compilation output, this is the line where this is called:

libtool --tag=CXX --mode=link g++ -I../include -I.  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -pthread -Wl,-O1,--sort-common,--as-needed,-z,relro,-z
,now  -o libntl.la FFT.lo FacVec.lo GF2.lo GF2E.lo GF2EX.lo GF2EXFactoring.lo GF2X.lo GF2X1.lo GF2XFactoring.lo GF2XVec.lo GetTime.lo GetPID.lo HNF.lo cto
ols.lo LLL.lo LLL_FP.lo LLL_QP.lo LLL_RR.lo LLL_XD.lo RR.lo WordVector.lo ZZ.lo ZZVec.lo ZZX.lo ZZX1.lo ZZXCharPoly.lo ZZXFactoring.lo ZZ_p.lo ZZ_pE.lo ZZ
_pEX.lo ZZ_pEXFactoring.lo ZZ_pX.lo ZZ_pX1.lo ZZ_pXCharPoly.lo ZZ_pXFactoring.lo fileio.lo lip.lo lzz_p.lo lzz_pE.lo lzz_pEX.lo lzz_pEXFactoring.lo lzz_pX
.lo lzz_pX1.lo lzz_pXCharPoly.lo lzz_pXFactoring.lo mat_GF2.lo mat_GF2E.lo mat_RR.lo mat_ZZ.lo mat_ZZ_p.lo mat_ZZ_pE.lo mat_lzz_p.lo mat_lzz_pE.lo mat_pol
y_ZZ.lo mat_poly_ZZ_p.lo mat_poly_lzz_p.lo quad_float.lo quad_float1.lo tools.lo vec_GF2.lo vec_GF2E.lo vec_RR.lo vec_ZZ.lo vec_ZZ_p.lo vec_ZZ_pE.lo vec_l
zz_p.lo vec_lzz_pE.lo xdouble.lo G_LLL_FP.lo G_LLL_QP.lo G_LLL_XD.lo G_LLL_RR.lo thread.lo BasicThreadPool.lo MatPrime.lo pd_FFT.lo -L/usr/lib  -lgmp  -L/
usr/lib  -lgf2x  -lpthread -rpath /usr/lib -version-info `cat VERSION_INFO` #LSHAR

And, right below it, you have the g++ command that libtool actually runs:

libtool: link: g++  -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/crt
beginS.o  .libs/FFT.o .libs/FacVec.o .libs/GF2.o .libs/GF2E.o .libs/GF2EX.o .libs/GF2EXFactoring.o .libs/GF2X.o .libs/GF2X1.o .libs/GF2XFactoring.o .libs/
GF2XVec.o .libs/GetTime.o .libs/GetPID.o .libs/HNF.o .libs/ctools.o .libs/LLL.o .libs/LLL_FP.o .libs/LLL_QP.o .libs/LLL_RR.o .libs/LLL_XD.o .libs/RR.o .li
bs/WordVector.o .libs/ZZ.o .libs/ZZVec.o .libs/ZZX.o .libs/ZZX1.o .libs/ZZXCharPoly.o .libs/ZZXFactoring.o .libs/ZZ_p.o .libs/ZZ_pE.o .libs/ZZ_pEX.o .libs
/ZZ_pEXFactoring.o .libs/ZZ_pX.o .libs/ZZ_pX1.o .libs/ZZ_pXCharPoly.o .libs/ZZ_pXFactoring.o .libs/fileio.o .libs/lip.o .libs/lzz_p.o .libs/lzz_pE.o .libs
/lzz_pEX.o .libs/lzz_pEXFactoring.o .libs/lzz_pX.o .libs/lzz_pX1.o .libs/lzz_pXCharPoly.o .libs/lzz_pXFactoring.o .libs/mat_GF2.o .libs/mat_GF2E.o .libs/m
at_RR.o .libs/mat_ZZ.o .libs/mat_ZZ_p.o .libs/mat_ZZ_pE.o .libs/mat_lzz_p.o .libs/mat_lzz_pE.o .libs/mat_poly_ZZ.o .libs/mat_poly_ZZ_p.o .libs/mat_poly_lz
z_p.o .libs/quad_float.o .libs/quad_float1.o .libs/tools.o .libs/vec_GF2.o .libs/vec_GF2E.o .libs/vec_RR.o .libs/vec_ZZ.o .libs/vec_ZZ_p.o .libs/vec_ZZ_pE
.o .libs/vec_lzz_p.o .libs/vec_lzz_pE.o .libs/xdouble.o .libs/G_LLL_FP.o .libs/G_LLL_QP.o .libs/G_LLL_XD.o .libs/G_LLL_RR.o .libs/thread.o .libs/BasicThre
adPool.o .libs/MatPrime.o .libs/pd_FFT.o   -L/usr/lib -lgmp -lgf2x -lpthread -L/usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0 -L/usr/lib/gcc/x86_64-pc-linux-gnu/9
.1.0/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-pc-lin
ux-gnu/9.1.0/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/crtn.o  -march=x86-64 -mtune=generic -O2 -pthread -Wl,-O1 -Wl,--sort-common
-Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now   -pthread -Wl,-soname -Wl,libntl.so.40 -o .libs/libntl.so.40.0.0

As you can see, libtool has added the -nostdlib flag to g++, which in turn invalidates all subsequent -pthread flags. Change that line in mfile to call $(LINK) directly and everything will work fine.

 

 Again, when I build and install a shared NTL on a linux system, everything works fine.

So I'm not even sure what the problem is:

What do you mean by "everything works fine"? Is libntl.so linked to libpthread? Because if not, it's not fine, it's underlinked, which is what is causing these issues that are being reported. 

[Dima Pasechnik]

On Thu, 5 Sep 2019 at 05:07, Victor Shoup <sh...@cs.nyu.edu> wrote:

Please see comments/questions below...

On Wednesday, September 4, 2019 at 6:04:42 AM UTC-4, Dima Pasechnik wrote:

On Tue, Sep 3, 2019 at 2:21 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>

> When I compile a program that uses NTL, I always pass -pthread to g++.
> Doesn't that automatically cause the necessary pthread-specific library to be linked in?

it does, if you don't use libtool to link (and typically one would
link an executable with
an explicit call to g++).

So, based on that comment, it seems that there is no problem if you link with g++

rather than libtool.  But who or what uses libtool to link?

I would never have thought of doing that, as I thought libtool was a tool

for creating libraries, not linking programs.

Linking a program with g++ is easy, linking a shared library with g++ is hard.

This is why libtool was invented in the 1st place, to help the latter.

the problem is that libtool strips away the -pthread flag.
As long as you don't use it, there is no problem. See
https://www.gnu.org/software/libtool/manual/libtool.html#Stripped-link-flags
"13.1 Why does libtool strip link flags when creating a library?

When creating a shared library, but not when compiling or creating a
program, libtool drops some flags from the command line provided by
the user. This is done because flags unknown to libtool may interfere
with library creation or require additional support from libtool, and
because omitting flags is usually the conservative choice for a
successful build."

This is confusing. If you use libtool to link, then the above quote would suggest that

in that context, libtool should not strip away any linker flags.

But you seem to be saying that it does.

Sorry, I was not careful  in my last message. indeed, you are right, libtool only drops flag while making a shared library, but not while making a program.

So...I am still confused.

Regarding the libtool that I am bundling with NTL:

1) is it stripping some flags during compilation that should not be stripped?

2) is it stripping some flags during linking that should not be stripped?

during linking a dynamic library libtools strips flags it does not know about.

(and only in this case).

With the patch I posted, it will “know” about the -pthread flag and do the right thing.

Again, when I build and install a shared NTL on a linux system, everything works fine.

So I'm not even sure what the problem is:

I can build it and link to it in the build directory using libtool.

I can install it using libtool somewhere else and later link to it using g++.

What am I missing?!?

 libtool handles linking dylibs and programs differently, it honours flags in the latter case, but not in the former case.

Dima

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/15c39061-e49d-4c6b-951b-6763aaea5dd0%40googlegroups.com.

[Victor Shoup]

We seem to be talking past each other.

Here is my experience.

My OS: Red Hat Enterprise Linux Server 7.7 (Maipo)

My compiler: gcc 4.8.5 (yes, it's old!)

Here is how I built NTL:

$ ./configure SHARED=on PREFIX= /home/gid-shoupv/sw

$ make 

$ make install

Then, in another directory, where I am using NTL as a library in another program:

$ g++ -std=c++11 -pthread -g -O2   -DFHE_THREAD   -o Test_thinboot_x Test_thinboot.cpp fhe.a   -lntl -lgmp -lm

Then, ldd says:

$ ldd Test_thinboot_x

linux-vdso.so.1 =>  (0x00007ffef18a0000)

libntl.so.40 => /home/gid-shoupv/sw/lib/libntl.so.40 (0x00007f51044d4000)

libgmp.so.10 => /usr/local/lib/libgmp.so.10 (0x00007f510425e000)

libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5103f45000)

libm.so.6 => /lib64/libm.so.6 (0x00007f5103c43000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5103a2c000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5103810000)

libc.so.6 => /lib64/libc.so.6 (0x00007f5103442000)

/lib64/ld-linux-x86-64.so.2 (0x00007f5104978000)

This is what I mean by "everything just works".

Also, when I build NTL with SHARED=on, if I build test programs

in the build directory using the makefile there, then, for example, I get:

$ make ThreadTest

./libtool-build/libtool --tag=CXX --mode=link g++ -I../include -I.  -g -O2 -std=c++11 -pthread -march=native  -o ThreadTest ThreadTest.cpp libntl.la #LSHAR

libtool: link: g++ -I../include -I. -g -O2 -std=c++11 -pthread -march=native -o .libs/ThreadTest ThreadTest.cpp  ./.libs/libntl.so -lgmp -pthread -Wl,-rpath -Wl,/home/gid-shoupv/sw/lib

This works just fine: the program ThreadTest compiles, links, and runs just fine, but for some reason ldd says "not a dynamic executable".

I'm not sure what's up with that.  A quick google search indicates that ldd can sometimes be buggy in this way.

This may not mean anything regarding the issues we are discussing.

Now, Antonio posted some output that showed that when libtool is used to create the library, it uses g++ with -nostdlib.

Indeed, that is what I see as well.

But from where I'm sitting, this is not a problem.

When I actually build a program, either in the build directory using NTL's makefile with libtool,

or in another directory using a different makefile that uses g++, it works fine.

It is also true that when I run ldd on libntl.so, I do not see anything related to pthread.

From the comments I'm reading in  https://trac.sagemath.org/ticket/28406, that seems

to be what is "wrong".  Looking closer at the comments there, I see that the command

that does not as expected is this:

$ gcc -o conftest -g -O2 -L/opt/sage/local/lib -Wl,-rpath,/opt/sage/local/lib conftest.c -lntl -lgmp

That this should work is contrary to my understanding of the "-pthread" flag.

man gcc says:

       -pthread

           Adds support for multithreading with the pthreads library.  This

           option sets flags for both the preprocessor and linker.

So my understanding is that whenever you compile a multi-threaded program

you should pass -pthread to gcc.  To me, this has nothing to do with libtool.

Maybe, just maybe, the real "issue" is this: some build systems do not pass

"-pthread" to gcc, but rather, they depend on libntl.so to contain a reference

to libpthreads.so.

If that is true, then I guess this "issue" is really just one of what the proper

conventions are for who is responsible to passing the right flags to gcc.

There is never going to be a perfect solution to this.

For example, on some compiles, you have to pass "-std=c++11" to gcc

in order to compile a program that includes NTL header files.

That's just one more thing to keep straight.

So I'm not convinced that patching libtool in NTL is the right thing to do.

Before I fix anything, I'd like to know what it is I'm actually fixing.

But probably if your build system needs t follow specific conventions

not supported by libtool, then you can build NTL using a custom libtool

that supports your conventions.

That's probably where this ends.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Dima Pasechnik]

No, I am compiling a program that uses NTL, that's all. It need not use threads by itself. Why must I bother about -pthread? And if NTL is using threads, but not linked to pthread library (the current NTL setup), then my program will not link.

This won't happen with a patch I proposed to apply.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/9fe439d2-c87c-44e1-aa06-f2b856ff6bec%40googlegroups.com.

[Antonio Rojas]

El jueves, 5 de septiembre de 2019, 15:51:34 (UTC+2), Victor Shoup escribió:

We seem to be talking past each other.

Unfortunately it seems so.

It is also true that when I run ldd on libntl.so, I do not see anything related to pthread.

From the comments I'm reading in  https://trac.sagemath.org/ticket/28406, that seems

to be what is "wrong".

Do you seriously not see why this is wrong? This is a textbook case of underlinking. Your library is calling functions from the libpthread library, so it *must* link to it. This is not a matter of "conventions" or "choices", this is how dynamic linking works. If you refuse to acknowledge that this is a problem then there's certainly nothing else to discuss.

When I actually build a program, either in the build directory using NTL's makefile with libtool,

or in another directory using a different makefile that uses g++, it works fine.

And we already have two programs that don't work: latte-integrale (al least the old version which I reported to you a  year ago) and barvinok. And no, building those programs with -pthread is definitely not the solution: If a library A uses a function from library B and a program C uses a function from library A but nothing from library B then C does not have to and should not link to B: it's A's job to link to B.

So my understanding is that whenever you compile a multi-threaded program

you should pass -pthread to gcc.

Correct. But you should *not* have to pass -pthread when you compile a single-threaded program and happens to use a multi-threaded library, which is the case here.

[Victor Shoup]

Sigh.
"...a textbook case of underlinking..."

I guess I never read that textbook.

I guess I just didn't know there was a whole community of people and projects who don't

like putting -pthread in their makefiles.

I feel like this is becoming a religious war.

If it will keep the peace, I will patch my libbtool based on Dima's suggestion.

Someday, I will make NTL's build system more conventional...maybe I can

find a student who can help me out.

[Dima Pasechnik]

On Thu, Sep 5, 2019 at 9:48 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> Sigh.
> "...a textbook case of underlinking..."
> I guess I never read that textbook.
> I guess I just didn't know there was a whole community of people and projects who don't
> like putting -pthread in their makefiles.
>
> I feel like this is becoming a religious war.
> If it will keep the peace, I will patch my libbtool based on Dima's suggestion.

yes, please - this would save many people a lot of time and hassle if
it's done...

>
> Someday, I will make NTL's build system more conventional...maybe I can
> find a student who can help me out.

one might try to use Google Summer of Code for this purpose.
(this was already pointed out in
https://www.shoup.net/forum/viewtopic.php?f=1&t=14)
one might make this a part of SageMath organisation proposal for the next round,
if you don't feel adventurous enough to apply for GSoC funding directly.

Dima

>
> On Thursday, September 5, 2019 at 10:35:42 AM UTC-4, Antonio Rojas wrote:
>>
>>
>>
>> El jueves, 5 de septiembre de 2019, 15:51:34 (UTC+2), Victor Shoup escribió:
>>>
>>> We seem to be talking past each other.
>>
>>
>> Unfortunately it seems so.
>>
>>> It is also true that when I run ldd on libntl.so, I do not see anything related to pthread.
>>> From the comments I'm reading in https://trac.sagemath.org/ticket/28406, that seems
>>> to be what is "wrong".
>>
>>
>> Do you seriously not see why this is wrong? This is a textbook case of underlinking. Your library is calling functions from the libpthread library, so it *must* link to it. This is not a matter of "conventions" or "choices", this is how dynamic linking works. If you refuse to acknowledge that this is a problem then there's certainly nothing else to discuss.
>>
>>> When I actually build a program, either in the build directory using NTL's makefile with libtool,
>>> or in another directory using a different makefile that uses g++, it works fine.
>>
>>
>> And we already have two programs that don't work: latte-integrale (al least the old version which I reported to you a year ago) and barvinok. And no, building those programs with -pthread is definitely not the solution: If a library A uses a function from library B and a program C uses a function from library A but nothing from library B then C does not have to and should not link to B: it's A's job to link to B.
>>
>>> So my understanding is that whenever you compile a multi-threaded program
>>> you should pass -pthread to gcc.
>>
>>
>> Correct. But you should *not* have to pass -pthread when you compile a single-threaded program and happens to use a multi-threaded library, which is the case here.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/4c15db12-02a6-4ee3-ad39-8bf9245812f3%40googlegroups.com.

[Victor Shoup]

So NTL's configure script already provides a work-around.

Just pass LIBTOOL_LINKER_FLAGS=-lpthread to NTL's configure script,

and the problem goes away.

To the extent that NTL's configure interface is already a bit non-standard, asking

package managers to do this is probably OK...at least, it's not any worse than the

current situation.

At a minimum, I can update NTL's documentation.

I could also modify the configure script so that this setting is the default.

Or I can try to do something "automagical"...

I'd rather do this than patch libtool itself.

On Thursday, September 5, 2019 at 10:35:42 AM UTC-4, Antonio Rojas wrote:

[Dima Pasechnik]

On Thu, Sep 5, 2019 at 11:43 PM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> So NTL's configure script already provides a work-around.
> Just pass LIBTOOL_LINKER_FLAGS=-lpthread to NTL's configure script,
> and the problem goes away.

No, it won'd magically go away, as one cannot blindly stick in this `-lpthread`
there. This has already been explained, I think.

We'd need to find out first whether this needs to be passed to the configure
script. In particular as NTL provides no way for discovering whether
it is built with
pthreads or not.
E.g. for my task it means to do half a page of ugly hard to write and
maintain autoconf stuff
just because you refuse to apply the patch I already provided and
tested for you?

We've been sending these dozens of emails back and force here for no
good reason...

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/71bbcca2-89d0-4c97-b9fd-639cc695bee4%40googlegroups.com.

[Victor Shoup]

I'm really just trying to understand what's going on.

So there is a good reason: my own education...although maybe you don't like that reason :-)

In fact, I just tested my proposal, and it indeed doesn't work.

I am convinced that the patch is the way to go.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Victor Shoup]

I spent a couple of hours experimenting with your patch as well as with the solution of just changing NTL's makefile

to stick -lpthread in the invocation of libtool:

$(LIBTOOL) --tag=CXX --mode=link $(LINK) $(LIBTOOL_LINK_FLAGS) -o libntl.la $(OBJ:.o=.lo) $(GMP_OPT_LIBDIR) $(GMP_OPT_LIB) $(GF2X_OPT_LIBDIR) $(GF2X_OPT_LIB) $(LDLIBS) -lpthread -rpath $(LIBDIR) -version-info `cat VERSION_INFO`

Examining the make output, they both yield essentially the same result.

Your libtool patch ends up putting -lpthread in basically the same place.

I have tested both approaches in linking another program (without specifying -pthread or -lpthread)

and they both worked fine.

I wasted a lot of time investigating why both approaches fail when linking a program that uses a 

static library that depends on NTLI got the error:

   /usr/lib64/libpthread.so.0: error adding symbols: DSO missing from command line

I never really solved that puzzle. 

Instead of patching libtool (which is harder for me to maintain in the long run),

I will just modify NTL's configure script to do the same thing (at least that is something

under my direct control).

I will modify NTL's config to define a makefile variable PTHREAD_LIB which will

be set to -lpthread and passed to libtool when linking, if that seems like the right thing to do.

The user running config can manually override PTHREAD_LIB if need be.

This was a learning experience.  I'm very used to linking static libraries, where you must explicitly

list all libraries on the command line.

Now dynamic libs come along and offer the feature of listing their own dependencies,

which is a feature that build systems have apparently come to rely upon.

That was news to me.

Another bit of information: the only reason NTL relies on libpthread at all is because it

is working around buggy implementations of the C++11 "thread_local" feature.

It has been absolutely shocking to me that these bugs have been around for several years,

and I'm not sure if they are still around or not...I trust my simple pthread implementation

more than the buggy gcc/clang implementations.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Dima Pasechnik]

On Sat, Sep 7, 2019 at 4:28 AM Victor Shoup <sh...@cs.nyu.edu> wrote:
>
> I spent a couple of hours experimenting with your patch as well as with the solution of just changing NTL's makefile
> to stick -lpthread in the invocation of libtool:
>
> $(LIBTOOL) --tag=CXX --mode=link $(LINK) $(LIBTOOL_LINK_FLAGS) -o libntl.la $(OBJ:.o=.lo) $(GMP_OPT_LIBDIR) $(GMP_OPT_LIB) $(GF2X_OPT_LIBDIR) $(GF2X_OPT_LIB) $(LDLIBS) -lpthread -rpath $(LIBDIR) -version-info `cat VERSION_INFO`
>
> Examining the make output, they both yield essentially the same result.
> Your libtool patch ends up putting -lpthread in basically the same place.

anything that provides a correctly linked NTL (i.e. no underlinking,
no overlinking - do not link in -lpthread if it's not used) would be
fine.

Needless to say, unconditionally putting -lpthread there is not what
my patch does, so it won't lead to overlinking. Your alternative fix
would need to only add -lpthread if NTL is built with multi-threads
on.

>
> I have tested both approaches in linking another program (without specifying -pthread or -lpthread)
> and they both worked fine.
>
> I wasted a lot of time investigating why both approaches fail when linking a program that uses a
> static library that depends on NTLI got the error:
>
> /usr/lib64/libpthread.so.0: error adding symbols: DSO missing from command line
>
> I never really solved that puzzle.

this sort of error usually comes up due to incorrect ordering of
-l<...> options (and/or object files), i.e.

-lA -lB foo.o

might give an error, and to fix it you might need to re-order

foo.o -lB -lA

or, in case of circular dependencies, one might even need

foo.o -lB -lA -lB

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5d7683bd-dfa9-48de-9a93-3c566dab8bbb%40googlegroups.com.