Progress towards the release

[Bill Hart]

Today I finally fixed mpn_mulmod_2expp1 to use the FFT wraparound trick in the FFT region.

As mpn_mulmod_bnm1 uses this function, this means that GMP-ECM will now be able to use MPIR more efficiently.

I think that much of our mulmod_2expp1 and mulmod_bnm1 code needs rewriting, but this can wait until someone can do it properly.

I also added some tuning for the new division functions. I'm not convinced it is picking the right thresholds, based on my own timings. But it's coded correctly, so there's nothing I can do about it. At least the division code doesn't seem to slow down with the new tuning values.

Tomorrow I will add some tuning functions for the new GMP combinatorics functions that we added. I will also fix a bug in the fft tuning code which causes it to crash on some systems.

This should hopefully complete all the new features for MPIR 2.7 and we can move on to closing all the open tickets for the release.

I have essentially until the end of next week to do as much as I can, after which I will have to work weekends if it isn't done.

After tomorrow we will need to retune MPIR on as many platforms as we can. Help with that would be appreciated.

By the way, after May 31st we'll lose access to about half the machines we build on, due to one of the build farms closing down. So if anyone has access to lots of heterogeneous hardware with linux, git and recent gcc on it, please let us know.

Bill.

[Bill Hart]

I meant to say, "Today I finally fixed mpn_mulmod_2expp1 to use the Nussbaumer trick in the FFT region".

[Bill Hart]

I have now added all the tuning code we need for the release.

I just need to fix the fft tuning code on systems on which it still crashes.

Bill.

[Bill Hart]

I now fixed the bug which was causing the fft tuning code to crash on some machines. In theory it could also cause a crash in the fft on those machines. But in practice the FFT is never used in that range. So it's a theoretical bug only, in theory.

Bill.

[Bill Hart]

I have now tuned on the following architectures:

core2, penryn, k102, netburst, k8, k10, bulldozer, westmere, sandybridge, piledriver, nehalem, core2, ia64.

I don't have tuning values for:

x86 (any kind), bobcat, atom, arm, mips32, mips64, sparc32, sparc64, ppc32, ppc64.

If anyone can contribute any of the above, please do.

Bill.

[leif]

Bill Hart wrote:
> I have now tuned on the following architectures:
>
> core2, penryn, k102, netburst, k8, k10, bulldozer, westmere,
> sandybridge, piledriver, nehalem, core2, ia64.
>
> I don't have tuning values for:
>
> x86 (any kind), bobcat, atom, arm, mips32, mips64, sparc32, sparc64,
> ppc32, ppc64.
>
> If anyone can contribute any of the above, please do.
>
> Bill.

Fresh tuning values for Bobcat (AMD E-450) are on the way; I've
meanwhile run tuneup 20 times while the machine was otherwise idle...
(GCC 4.8.0 on Ubuntu 10.04.4 x86_64, FWIW).

I'll presumably provide tuning values for Pentium 4 Prescott (x86)
tomorrow; MPIR's current cpuid.c still ("mis"-)detects that CPU (family
15 model 3; same for model 4) as a stock Pentium 4 though. There's
currently no special assembly code for Prescott (using SSE3/PNI, say),
but GCC is fed with '-march=pentium4 -mtune=pentium4' instead of
'-march=prescott -mtune=prescott'.

If it's of any use, I could also try to revive a Pentium III
(Coppermine) and tune on that.


-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Bill Hart]

On 21 March 2014 21:26, leif <not.r...@online.de> wrote:

Bill Hart wrote:

I have now tuned on the following architectures:

core2, penryn, k102, netburst, k8, k10, bulldozer, westmere,
sandybridge, piledriver, nehalem, core2, ia64.

I don't have tuning values for:

x86 (any kind), bobcat, atom, arm, mips32, mips64, sparc32, sparc64,
ppc32, ppc64.

If anyone can contribute any of the above, please do.

Bill.

Fresh tuning values for Bobcat (AMD E-450) are on the way; I've meanwhile run tuneup 20 times while the machine was otherwise idle... (GCC 4.8.0 on Ubuntu 10.04.4 x86_64, FWIW).

Fantastic. However, we need to know one thing before getting bobcat tuning values: which is faster, mpn/x86_64/bobcat/sqr_basecase.as or mpn/x86_64/sqr_basecase.asm.

To find out, could you do the following:

./configure

make

cd tune

make speed

./speed -s 1-40 mpn_sqr_basecase

{delete mpn/x86_64/bobcat/sqr_basecase.as}

 ./configure

make clean

make

cd tune

make speed

./speed -s 1-40 mpn_sqr_basecase

If the second set of timings looks lower, leave it as it is when doing make tune, (otherwise -- unlikely -- put mpn/x86_64/bobcat/sqr_basecase.as back, ./configure; make clean; make).

Now it should be fine to do make tune.

This step only needs to be done for bobcat.

I'll presumably provide tuning values for Pentium 4 Prescott (x86) tomorrow; MPIR's current cpuid.c still ("mis"-)detects that CPU (family 15 model 3; same for model 4) as a stock Pentium 4 though.  There's currently no special assembly code for Prescott (using SSE3/PNI, say), but GCC is fed with '-march=pentium4 -mtune=pentium4' instead of '-march=prescott -mtune=prescott'.

Is it 64 bit? If not, we don't have any SSE3 code in MPIR. We only go up to SSE2 on the x86 branch.

If it is 64 bit, you can force it to build as netburst, that's about it, i.e. --build=netburst-unknown-linux

 

If it's of any use, I could also try to revive a Pentium III (Coppermine) and tune on that.

That would be fun. :-) But it's not important if you don't have time. We assume people are using more recent processors if they want performance.

Bill.

 

-leif

--
() The ASCII Ribbon Campaign
/\   Help Cure HTML E-Mail

--
You received this message because you are subscribed to the Google Groups "mpir-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpir-devel+unsubscribe@googlegroups.com.
To post to this group, send email to mpir-...@googlegroups.com.
Visit this group at http://groups.google.com/group/mpir-devel.
For more options, visit https://groups.google.com/d/optout.

[Bill Hart]

I have now assigned 36 open tickets to the 2.7 release. I'm sure more will be added, but at least we now have some measure of how far we are from a release.

Next I will finish off tuning on the few other machines I have available.

Bill.

[Bill Hart]

I have now tuned o x86/pentium4/sse2, sparc32, sparc64, ppc32, ppc64.

That just leaves  bobcat, atom, arm, mips32, mips64.

Bill.

[Bill Hart]

I was amazed MPIR built fine on an AIX IBM Power7 machine.

Bill.

[leif]

On 24.03.2014 17:17, Bill Hart wrote:
> I have now assigned 36 open tickets to the 2.7 release. I'm sure more
> will be added, but at least we now have some measure of how far we are
> from a release.

Just in case that's not yet on the todo list:

While --enable-cxx (and 'make check' with that) meanwhile works,
apparently the checks for the C++ compiler's capabilities are currently
missing, such that CXXFLAGS lack the ABI- and machine-specific flags
(-m32/-m64, -march=.../-mtune=...).

[Bill Hart]

I've added it to the list.

I'm puzzled as to how this bug has lasted until now to be reported. It must surely have broken things on a lot of C++ platforms, and it almost certainly has never been right.

Bill.

[Bill Hart]

I have dealt with all the nontrivial tickets I believe.

Here is the list of remaining tickets for this release:

https://github.com/wbhart/mpir/issues?milestone=1&state=open

I think essentially all of those are either invalid, will get moved to the next release or are trivial.

Bill.

[Bill Hart]

Actually, I don't see how this can be right. It would imply that autotools was broken, as CXXFLAGS is filled by autotools, not MPIR.

[Well, autotools *is* broken. You have no argument from me there (for example, nothing builds in MinGW64 because of this).]

Bill.

[Bill Hart]

Here is the relevant part of configure.ac. Who knows what it is supposed to have there if it isn't correct:

# Now actually check that the C++ compiler works if it is needed.

want_cxx=no

if test $enable_cxx != no; then

  test_CXXFLAGS=${CXXFLAGS+set}

  echo "CXXFLAGS chosen by autoconf: $CXXFLAGS" >&AC_FD_CC

  cxxflags_ac_prog_cxx=$CXXFLAGS

  cxxflags_list=ac_prog_cxx

  # If the user didn't specify $CXXFLAGS, then try $CFLAGS, with -g removed

  # if AC_PROG_CXX thinks that doesn't work.  $CFLAGS stands a good chance

  # of working, eg. on a GNU system where CC=gcc and CXX=g++.

  #

  if test "$test_CXXFLAGS" != set; then

    cxxflags_cflags=$CFLAGS

    cxxflags_list="cflags $cxxflags_list"

    if test "$ac_prog_cxx_g" = no; then

      cxxflags_cflags=`echo "$cxxflags_cflags" | sed -e 's/ -g //' -e 's/^-g //' -e 's/ -g$//'`

    fi

  fi

  # See if the C++ compiler works.  If the user specified CXXFLAGS then all

  # we're doing is checking whether AC_PROG_CXX succeeded, since it doesn't

  # give a fatal error, just leaves CXX set to a default g++.  If on the

  # other hand the user didn't specify CXXFLAGS then we get to try here our

  # $cxxflags_list alternatives.

  #

  # Automake includes $CPPFLAGS in a C++ compile, so we do the same here.

  #

  for cxxflags_choice in $cxxflags_list; do

    eval CXXFLAGS=\"\$cxxflags_$cxxflags_choice\"

    GMP_PROG_CXX_WORKS($CXX $CPPFLAGS $CXXFLAGS,

      [want_cxx=yes

      break])

  done

  # If --enable-cxx=yes but a C++ compiler can't be found, then abort.

  if test $want_cxx = no && test $enable_cxx = yes; then

    AC_MSG_ERROR([C++ compiler not available, see config.log for details])

  fi

fi

AM_CONDITIONAL(WANT_CXX, test $want_cxx = yes)

if test -z "$MPN_PATH"; then

  path="$add_path $path"

fi

[leif]

On 28.03.2014 19:36, Bill Hart wrote:
> I've added it to the list.
>
> I'm puzzled as to how this bug has lasted until now to be reported. It
> must surely have broken things on a lot of C++ platforms, and it almost
> certainly has never been right.

With MPIR 2.6.0 (and presumably [m]any earlier version[s]) this worked
for me (although I usally set CXXFLAGS manually, so wouldn't notice
unless I'm explicitly testing or compiling the C++ interface for a
non-default ABI):

...
checking ABI=64
checking whether gcc-4.7.0 is gcc... yes
checking compiler gcc-4.7.0 -m64 ... yes
checking compiler gcc-4.7.0 -m64 -O2... yes
checking compiler gcc-4.7.0 -m64 -O2 -march=btver1... yes
checking compiler gcc-4.7.0 -m64 -O2 -march=btver1 -mtune=btver1... yes
checking for gcc... gcc-4.7.0
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc-4.7.0 accepts -g... yes
checking for gcc-4.7.0 option to accept ISO C89... none needed
...
checking for gcc... (cached) gcc-4.7.0
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc-4.7.0 accepts -g... (cached) yes
checking for gcc-4.7.0 option to accept ISO C89... (cached) none needed
checking for gcc-4.7.0 option to accept ISO C99... -std=gnu99
checking for gcc-4.7.0 -std=gnu99 option to accept ISO Standard C...
(cached) -std=gnu99
checking how to run the C preprocessor... cpp-4.7.0
checking build system compiler gcc-4.7.0 -std=gnu99... yes
checking for build system preprocessor... gcc-4.7.0 -std=gnu99 -E
checking for build system executable suffix...
checking whether build system compiler is ANSI... yes
checking for build system compiler math library... -lm
checking whether we are using the GNU C++ compiler... yes
checking whether g++-4.7.0 accepts -g... yes
checking C++ compiler g++-4.7.0 -m64 -O2 -march=btver1 -mtune=btver1... yes
checking how to run the C++ preprocessor... g++-4.7.0 -E
using ABI="64"
CC="gcc-4.7.0 -std=gnu99"
CFLAGS="-m64 -O2 -march=btver1 -mtune=btver1"
CPPFLAGS=""
CXX="g++-4.7.0"
CXXFLAGS="-m64 -O2 -march=btver1 -mtune=btver1"
MPN_PATH=" x86_64/bobcat x86_64 generic"
...


-leif

> On 28 March 2014 18:42, leif <not.r...@online.de

> <mailto:not.r...@online.de>> wrote:

>
> On 24.03.2014 17 <tel:24.03.2014%2017>:17, Bill Hart wrote:
>
> I have now assigned 36 open tickets to the 2.7 release. I'm sure
> more
> will be added, but at least we now have some measure of how far
> we are
> from a release.
>
>
> Just in case that's not yet on the todo list:
>
> While --enable-cxx (and 'make check' with that) meanwhile works,
> apparently the checks for the C++ compiler's capabilities are
> currently missing, such that CXXFLAGS lack the ABI- and
> machine-specific flags (-m32/-m64, -march=.../-mtune=...).

[Bill Hart]

I think it's a new autotools bug. It certainly should set CXXFLAGS appropriately, but doesn't. And we've not changed that part of configure.ac. It also happens to be the same as in GMP, which does get the right CXXFLAGS.

Fortunately I have to issue the release with a different version of autotools anyway, as the Ubuntu version doesn't support Windows, etc.

So perhaps you can check again when I do that.

Bill.

[Bill Hart]

Hmm, autoconf 2.69 doesn't fix this problem either. Screwed if I know how CXXFLAGS gets populated.

Bill.

[leif]

On 29.03.2014 00:09, Bill Hart wrote:
> Hmm, autoconf 2.69 doesn't fix this problem either. Screwed if I know
> how CXXFLAGS gets populated.

I *think* this is pretty unrelated to the autotools version; you simply
moved AC_PROG_CXX up (in configure.ac), but not the prior test which is
supposed to record whether CXXFLAGS had been set (by the user!):

test_CXXFLAGS=${CXXFLAGS+set}

AC_PROG_CXX of course modifies / sets CXXFLAGS, such that test_CXXFLAGS
now always equals "set", so MPIR assumes *the user* had set them, and
hence never tries to modify them.

So it should be sufficient to either move AC_PROG_CXX back down, the
test for user-set CXXFLAGS up (right /before/ AC_PROG_CXX), or --
presumably less robust -- to use one of the autoconf-saved values,
ac_test_CXXFLAGS (instead of test_CXXFLAGS), or ac_save_CXXFLAGS
(instead of CXXFLAGS).


-leif

> On 28 March 2014 22:46, Bill Hart <goodwi...@googlemail.com

> <mailto:goodwi...@googlemail.com>> wrote:
>
> I think it's a new autotools bug. It certainly should set CXXFLAGS
> appropriately, but doesn't. And we've not changed that part of

> configure.ac <http://configure.ac>. It also happens to be the same

> as in GMP, which does get the right CXXFLAGS.
>
> Fortunately I have to issue the release with a different version of
> autotools anyway, as the Ubuntu version doesn't support Windows, etc.
>
> So perhaps you can check again when I do that.
>
> Bill.
>
>
> On 28 March 2014 21:43, leif <not.r...@online.de

> <mailto:not.r...@online.de <mailto:not.r...@online.de>>>

[Bill Hart]

Ah, bingo, well done.

I had completely ignored this because according to the autotools manual that script just checks for a C++ compiler and lets you specify an order to search for them. But I had forgotten about the fact that it sets CXXFLAGS with a preliminary value.

Oddly enough, when I reran autoconf -fiv and configure, this did not fix the problem.

So I deleted configure and tried again. This time success.

Don't ask me. Yet another bug in my opinion.

Bill.

On 29 March 2014 01:06, leif <not.r...@online.de> wrote:

On 29.03.2014 00:09, Bill Hart wrote:

Hmm, autoconf 2.69 doesn't fix this problem either. Screwed if I know
how CXXFLAGS gets populated.

I *think* this is pretty unrelated to the autotools version; you simply moved AC_PROG_CXX up (in configure.ac), but not the prior test which is supposed to record whether CXXFLAGS had been set (by the user!):

  test_CXXFLAGS=${CXXFLAGS+set}

AC_PROG_CXX of course modifies / sets CXXFLAGS, such that test_CXXFLAGS now always equals "set", so MPIR assumes *the user* had set them, and hence never tries to modify them.

So it should be sufficient to either move AC_PROG_CXX back down, the test for user-set CXXFLAGS up (right /before/ AC_PROG_CXX), or -- presumably less robust -- to use one of the autoconf-saved values, ac_test_CXXFLAGS (instead of test_CXXFLAGS), or ac_save_CXXFLAGS (instead of CXXFLAGS).


-leif

On 28 March 2014 22:46, Bill Hart <goodwi...@googlemail.com

[Bill Hart]

All tests now pass on MinGW, MinGW64, Cygwin and Cygwin64. For the first time ever!

Bill.

On 20 March 2014 00:44, Bill Hart <goodwi...@googlemail.com> wrote:

[leif]

On 29.03.2014 12:48, Bill Hart wrote:
> All tests now pass on MinGW, MinGW64, Cygwin and Cygwin64. For the first
> time ever!

Windows FTW! ;-)

The fix for issue #23 is incomplete though; you apparently missed the
patch to acinclude.m4:

--- src/acinclude.m4 2013-02-27 04:51:08.528733998 +0100
+++ src/acinclude.m4 2013-02-27 06:19:10.558735026 +0100
@@ -41,7 +41,7 @@
[[powerpc64-*-* | powerpc64le-*-* | powerpc620-*-* | powerpc630-*-* |
powerpc970-*-* | power[3-9]-*-*]])

define(X86_PATTERN,
-[[i?86*-*-* | k[5-8]*-*-* | pentium*-*-* | athlon-*-* | viac3*-*-*]])
+[[i?86*-*-* | k[5-8]*-*-* | pentium*-*-* | prescott-*-* | core-*-* |
athlon-*-* | viac3*-*-*]])

define(X86_64_PATTERN,
[[x86_64-*-* | netburst-*-* | netburstlahf-*-* | k8-*-* | k10-*-* |
k102-*-* | k103-*-* | core2-*-* | penryn-*-* | nehalem-*-* |
westmere-*-* | sandybridge-*-* | atom-*-* | nano-*-* | bobcat-*-* |
bulldozer-*-*]])

such that one still ends up with ABI="standard", an (almost) empty
MPN_PATH (just containing "generic") and incorrect / incomplete C- and
CXXFLAGS.


I have to admit I'm not sure whether I submitted the patches to the
autotools *source* files upstream last year; they're contained in the
Sage package (or nowadays the "package metadata"(?) for MPIR) though.


-leif


P.S.: In the long run, it may make sense to also have core-*-* and
prescott-*-* have their own gmp-mparam.h files (by adding some
subdirectories like you did for e.g. piledriver and bulldozer, and
prepending them to their MPN_PATH, although there's not yet specific
assembly code for these). My Prescott still doesn't get detected as
such (just as an ordinary Pentium 4), because cpuid.c considers every
model <=6 of family 15 a plain Pentium 4, although model 3 and model 4
are (or at least can be?) Prescotts as well. (Mine is a model 3.) But
that's a different story...

> On 20 March 2014 00:44, Bill Hart <goodwi...@googlemail.com

[Jean-Pierre Flori]

Dear all,

I think the fix in https://github.com/wbhart/mpir/commit/053476c4567874223180485ff0a4fa808d0ab4d1 is actually wrong.
What you did is basically go back to the previous state of configure.in/ac with regards to AC_PROG_CXX.
But note that it should not be run conditionally this way, whence my (not quite correct) fix.
In particular, running it conditionnally as it is now will completely break the autotools magic if mpir is built without C++ support at least on Cygwin.
See http://cygwin.com/ml/cygwin/2013-06/msg00580.html and http://cygwin.com/ml/cygwin/2013-06/msg00620.html

So we should come up with a better solution.

Best,
JP

On Saturday, March 29, 2014 1:22:35 AM UTC+1, Bill Hart wrote:

Ah, bingo, well done.

I had completely ignored this because according to the autotools manual that script just checks for a C++ compiler and lets you specify an order to search for them. But I had forgotten about the fact that it sets CXXFLAGS with a preliminary value.

Oddly enough, when I reran autoconf -fiv and configure, this did not fix the problem.

So I deleted configure and tried again. This time success.

Don't ask me. Yet another bug in my opinion.

Bill.

On 29 March 2014 01:06, leif <not.r...@online.de> wrote:

On 29.03.2014 00:09, Bill Hart wrote:

Hmm, autoconf 2.69 doesn't fix this problem either. Screwed if I know
how CXXFLAGS gets populated.

I *think* this is pretty unrelated to the autotools version; you simply moved AC_PROG_CXX up (in configure.ac), but not the prior test which is supposed to record whether CXXFLAGS had been set (by the user!):

  test_CXXFLAGS=${CXXFLAGS+set}

AC_PROG_CXX of course modifies / sets CXXFLAGS, such that test_CXXFLAGS now always equals "set", so MPIR assumes *the user* had set them, and hence never tries to modify them.

So it should be sufficient to either move AC_PROG_CXX back down, the test for user-set CXXFLAGS up (right /before/ AC_PROG_CXX), or -- presumably less robust -- to use one of the autoconf-saved values, ac_test_CXXFLAGS (instead of test_CXXFLAGS), or ac_save_CXXFLAGS (instead of CXXFLAGS).


-leif

On 28 March 2014 22:46, Bill Hart <goodwi...@googlemail.com

[Jean-Pierre Flori]

On Monday, March 31, 2014 9:15:15 PM UTC+2, Jean-Pierre Flori wrote:

Dear all,

I think the fix in https://github.com/wbhart/mpir/commit/053476c4567874223180485ff0a4fa808d0ab4d1 is actually wrong.
What you did is basically go back to the previous state of configure.in/ac with regards to AC_PROG_CXX.
But note that it should not be run conditionally this way, whence my (not quite correct) fix.
In particular, running it conditionnally as it is now will completely break the autotools magic if mpir is built without C++ support at least on Cygwin.
See http://cygwin.com/ml/cygwin/2013-06/msg00580.html and http://cygwin.com/ml/cygwin/2013-06/msg00620.html

So we should come up with a better solution.

Best,
JP

Maybe move "test_CXXFLAGS=${CXXFLAGS+set}" out of the conditional together with and before AC_PROG_CXX.

[Bill Hart]

I'll give it a try. It took me nearly a whole day to test on MinGW, MinGW32, Cygwin and Cygwin64. Now it'll have to be done again, with and without --enable-cxx. So that's two full days of testing.... if nothing breaks.

MinGW, Cygwin and autotools are such piles of turd. 

Besides, GMP doesn't do this. How does it build on Cygwin64? And why does it only cause a problem on Cygwin64 and not on any other OS?

Bill.

[Jean-Pierre Flori]

This seems to work (and I also reproduced the broken CXXFLAGS treatment when just AC_PROG_CXX was moved)..

[Jean-Pierre Flori]

(I mean: I tested that CXXFLAGS is now correctly treated on Linux only, though I'm quite confident it should be ok on more exotic platforms.)

[Bill Hart]

How on earth did you test it so fast? It takes 45 minutes just to run configure on my machine!

Bill.

[Jean-Pierre Flori]

On Monday, March 31, 2014 9:30:26 PM UTC+2, Bill Hart wrote:

How on earth did you test it so fast? It takes 45 minutes just to run configure on my machine!

Our mails crossed each other.

I'll be testing Cygwin and Cygwin64 now.

[Bill Hart]

Ok, I'll test MinGW and MinGW32 then.

I'm just about to commit it, so if you could use my version built with my autotools, that would be great.

Bill.

[Bill Hart]

OK, I've committed it now.

Bill.

[Jean-Pierre Flori]

On Monday, March 31, 2014 9:41:27 PM UTC+2, Bill Hart wrote:

OK, I've committed it now.

Bill.

Works fine on Cygwin32 (in a 32bit Windows XP VM) with and without C+ bindings.

Testing now on Cygwin64 (in a 64 bit Windows 7 VM).

[Jean-Pierre Flori]

On Monday, March 31, 2014 10:27:01 PM UTC+2, Jean-Pierre Flori wrote:

Testing now on Cygwin64 (in a 64 bit Windows 7 VM).

Built fine there as well with and without C++ support.

Now running the testsuite.

[Jean-Pierre Flori]

The testsuite (with C++ bindings) passed fine on Cygwin32 in the 32 bit Windows XP VM.

[Jean-Pierre Flori]

On Monday, March 31, 2014 11:49:38 PM UTC+2, Jean-Pierre Flori wrote:

On Monday, March 31, 2014 10:45:48 PM UTC+2, Jean-Pierre Flori wrote:

On Monday, March 31, 2014 10:27:01 PM UTC+2, Jean-Pierre Flori wrote:

Testing now on Cygwin64 (in a 64 bit Windows 7 VM).

Built fine there as well with and without C++ support.

Now running the testsuite.

 
I'm getting a bunch of segfaults on the 64 bit install...

Not sure why yet.

[Jean-Pierre Flori]

Playing with different arch choices, it seems that x86_64w/gmp-mparam.h lacks a definition for FFT_N_NUM.

[Bill Hart]

Indeed it does. I will commit one shortly. Just about done with another commit.

Bill.

[Brian Gladman]

On 31/03/2014 23:46, Jean-Pierre Flori wrote:
> Playing with different arch choices, it seems that x86_64w/gmp-mparam.h
> lacks a definition for FFT_N_NUM.

There might be several like that in mpn/x86w and mpn/x86_64w because I
haven't copied over the tuning files from the equivalent GCC directories
(tuning on Windows is unreliable so we just use the GCC tuning files).

I didn't want to do this piecemeal but it looks as if I should do it if
tuning is now close to complete.

Brian

[Bill Hart]

Tuning is certainly done. So now is as good a time as any I would say.

Bill.

[Brian Gladman]

On 01/04/2014 00:30, Bill Hart wrote:
> Tuning is certainly done. So now is as good a time as any I would say.

Done

Brian

[Jean-Pierre Flori]

Same problem on an actual 64 but Windows 7 system.

[Jean-Pierre Flori]

Building a static lib instead of a shared one, makes the issue disappear...

[Bill Hart]

I'll try building a shared library on MinGW64 and see what happens. Will take a couple of hours.

Bill.

[Bill Hart]

Oh, maybe the shared library uses all the PIC paths in the assembly code, and some of these are broken?

Bill.

[Jean-Pierre Flori]

On Tuesday, April 1, 2014 1:54:49 PM UTC+2, Bill Hart wrote:

Oh, maybe the shared library uses all the PIC paths in the assembly code, and some of these are broken?

Bill.

Maybe...
What's disturbing is that I seem to remember there were no problems a few months ago.
I'll try a previous revision later.

[Bill Hart]

Sure, the assembly files will have changed. For example, in mpn/x86_64w/sqr_basecase.asm the PIC path didn't even compile until Brian helped me change it. Maybe we still don't have it correct, though it didn't seem to work at all even for a static library when I tried. So I thought this was now fixed.

Someone did make a bug report about segfaults with a shared library build, ages ago.

Bill.

[Bill Hart]

Actually, it was you that reported this. You also reported it on the Cygwin64 list (they didn't seem to know anything about the segfaults).

Apparently there were problems within ld, which have presumably been resolved. 

On the other hand, you were able to build a shared library ok, but some (not all) of the tests segfaulted.

It looks like this just never got dealt with (perhaps we dealt with the ld problem and never did anything about the other).

Bill.

[Brian Gladman]

On 01/04/2014 12:59, Bill Hart wrote:
> Sure, the assembly files will have changed. For example, in
> mpn/x86_64w/sqr_basecase.asm the PIC path didn't even compile until
> Brian helped me change it. Maybe we still don't have it correct, though
> it didn't seem to work at all even for a static library when I tried. So
> I thought this was now fixed.

All is well when I turn on PIC and build a DLL on WIndows x64. All the
tests pass (except those that are supressed for a DLL).

Brian

[Bill Hart]

Which tests are suppressed for dll? Maybe we are supposed to suppress them on Linux too when building a shared library?

Bill.

   Brian

--
You received this message because you are subscribed to the Google Groups "mpir-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpir-devel+...@googlegroups.com.
To post to this group, send email to mpir-...@googlegroups.com.
Visit this group at http://groups.google.com/group/mpir-devel.
For more options, visit https://groups.google.com/d/optout.

[Bill Hart]

Also, what arch are you building for? Maybe it's a specific asm file that is broken in one of the arch dirs and Brian and I aren't seeing it.

Bill.

[Brian Gladman]

On 01/04/2014 13:25, Bill Hart wrote:
> Which tests are suppressed for dll? Maybe we are supposed to suppress
> them on Linux too when building a shared library?

misc.locale
mpf.reuse
mpz.reuse

Brian

[Brian Gladman]

On 01/04/2014 13:27, Bill Hart wrote:
> Also, what arch are you building for? Maybe it's a specific asm file
> that is broken in one of the arch dirs and Brian and I aren't seeing it.

Apart from whatever happens in the FAT directory (that I don't touch), I
believe that there is only one file in x86_64 that uses PIC
(sqr_basecase.asm). I have a feeling that I stripped out all the PIC
paths - sqr_basecase only has a PIC path because I converted it recently
and left it in place.

Brian

[Bill Hart]

No problem on MinGW64 with ./configure --disable-static --enable-cxx

On the other hand, I don't think it uses PIC. Then again, I'm not actually sure Cygwin64 is supposed to either.

I will now try a shared library on my machine with Cygwin64. It may be about 6 hours before I have the results.

Bill.

   Brian

[Jean-Pierre Flori]

On Tuesday, April 1, 2014 2:20:54 PM UTC+2, Bill Hart wrote:

Actually, it was you that reported this. You also reported it on the Cygwin64 list (they didn't seem to know anything about the segfaults).

Apparently there were problems within ld, which have presumably been resolved. 

On the other hand, you were able to build a shared library ok, but some (not all) of the tests segfaulted.

It looks like this just never got dealt with (perhaps we dealt with the ld problem and never did anything about the other).

Bill.

I do remember that.
IIRC I think the original problem was that the wrong assembly dir was used (i.e. x86_64 with the AMD64 ABI instead of x86_64w with the Win64 one).
I don't really remember what happened next.

[Bill Hart]

That apparently solved part of the problem, but not the segfaults with a shared lib.

Getting closer to being able to test it myself. We'll see if it segfaults on my machine.

Bill.

[Bill Hart]

Here are the functions which segfault for me:

t-dc_bdiv_q_n

t-hgcd

t-invert

t-mulmod_2expm1

t-mulmod_2expp1

t-neg

t-sb_bdiv_q

I suggest we start with t-neg, as that is by far the simplest function. There is no reason I can see for all of those to fail, unless one of the mpn functions is broken on Cygwin64.

The only other clue we have is this:

      0 [sig] -bash 25560! get_proc_lock: Couldn't acquire sync_proc_subproc for(5,1), last 7, Win32 error 0

  60308 [sig] -bash 25560! proc_subproc: couldn't get proc lock. what 5, val 1

It might be unrelated. It happened when I tried to open one of the log files with vim immediately after.

JP, are these the same failures for you?

Bill.

[Bill Hart]

On MinGW64 we pass  -DDLL_EXPORT -D PIC to yasm.

On Cygwin64 it's  -DDLL_EXPORT -DPIC.

mpn_neg_n calls mpn_com_n and mpn_add_1. The latter is a C macro. The former does not have a PIC path, but it does have SSE2 instructions. In particular is uses movdqu. Maybe there is some kind of alignment issue with malloc?

Bill.

[Jean-Pierre Flori]

I'd say I see the same issues indeed.
I tried going back to older commits but errors are still there.

--
Jean-Pierre Flori

[Jean-Pierre Flori]

t-neg seems to segfault when calling MPN_ZERO on line 41.

Same for t-hgcd.

[Jean-Pierre Flori]

On Tuesday, April 1, 2014 9:45:04 PM UTC+2, Jean-Pierre Flori wrote:

t-neg seems to segfault when calling MPN_ZERO on line 41.

Same for t-hgcd.

For some reason the produced binary gets for the MPN_ZERO call:
callq  0xff2746c0

[Jean-Pierre Flori]

And I get within the dll:
 $ nm .libs/cygmpir-16.dll |grep mpn_store
00000004ff2746c0 T __gmpn_store

[Bill Hart]

Yes, it calls mpn_store.

Bill.

[Bill Hart]

The assembly file is mpn/x86_64w/nehalem/store.asm.

I don't see any immediate problems. It only seems to destroy rcx, rdx and xmm0, the first two of which are parameters anyway.

Apart from possible unaligned access and the slightly suspicious comparison with -2, there isn't much to write home about.

Bill.

[Jean-Pierre Flori]

What look strange to me is the way the function is called.
All other function calls use rip or a different address range.

E.g. for the call in t-neg:
39   gmp_randinit_default(rands);

First it goes through the import lib:
   0x00000001004010fa <+42>:    mov    0x818f(%rip),%rax        # 0x100409290 <__imp___gmp_randinit_default>
   0x0000000100401101 <+49>:    callq  *%rax

And %rax looks saner:
(gdb) info reg
rax            0x4ff1e1550      21460030800

Just as in cygmpir-16.dll:
00000004ff1e1550 T __gmp_randinit_default

In particularn in the mpn_store direct call, the leading 4 is missing...

[Jean-Pierre Flori]

And:
 (gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /home/jp/mpir-flori.git/tests/mpn/.libs/t-neg
[New Thread 3836.0x408]
[New Thread 3836.0x26c]

Program received signal SIGSEGV, Segmentation fault.
0x00000000ff2746c0 in ?? ()
(gdb) x/i $pc
=> 0xff2746c0:  Cannot access memory at address 0xff2746c0
(gdb) x/i $pc + 0x400000000
   0x4ff2746c0 <__gmpn_store>:  lea    -0x20(%rcx),%rcx
(gdb)

[Bill Hart]

Surely a compiler error then. 

I can't seem to get libspeed or try to build on Cygwin64. It looses the include and lib directories in the gcc invocation.

But I think you've found the problem, for sure.

Bill. 

[Bill Hart]

On the other hand, should we be passing -fPIC to gcc?

Bill.

[Jean-Pierre Flori]

We shouldn't but on Windows it should make no difference to GCC, and
as far as I saw, there is no specific PIC code in the assembly files
chosen on my setup which could be triggered by the "-DPIC" define.

--
Jean-Pierre Flori

[Bill Hart]

It looks like we don't pass it anyway when compiling test programs (which makes sense). We only pass it when compiling library .c files.

We do seem to link libmpir twice for good measure. I'm amazed that works.

Bill.

[Bill Hart]

According to config.h we are certainly using the native mpn_store. I can't see this being anything other than a compiler bug.

Bill.

[Jean-Pierre Flori]

On Tuesday, April 1, 2014 11:01:25 PM UTC+2, Bill Hart wrote:

According to config.h we are certainly using the native mpn_store. I can't see this being anything other than a compiler bug.

That migh also explain my message here: https://groups.google.com/d/msg/mpir-devel/wkn3G-uur1Y/orsQOENXsC0J
That was surely another compiler version...

[Bill Hart]

I don't see the relation. You only speak about the ABI, which is not relevant, is it?

Bill.

[Jean-Pierre Flori]

On Tuesday, April 1, 2014 11:09:43 PM UTC+2, Bill Hart wrote:

I don't see the relation. You only speak about the ABI, which is not relevant, is it?

Sure... wrong link.
i thought about https://groups.google.com/d/msg/mpir-devel/jzCmaWf8sL8/deLYDTEjdBcJ where I mentionned all tests used to pass.

[Jean-Pierre Flori]

Anyway, I'll compile GCC 4.7.3 and test back.
That will need some time though.

[Bill Hart]

So I tried writing a small program to call MPN_ZERO, but I don't know how to set the library path on this stupid platform, so I can't get it to run.

Bill.

[Jean-Pierre Flori]

You have to put the path in PATH (not LD_LIBRARY_PATH).

--
Jean-Pierre Flori

[Bill Hart]

How? It doesn't seem to work.

[Jean-Pierre Flori]

On Tuesday, April 1, 2014 11:21:09 PM UTC+2, Bill Hart wrote:

How? It doesn't seem to work.

Something like:
PATH=<path to cygmpir>:$PATH ./<exename>

[Bill Hart]

Got it. And it in fact segfaults.

Bill.

[Bill Hart]

Even calling mpn_store directly segfaults.

Bill.

[Bill Hart]

Sadly I don't have gdb. But what on earth possesses it to miscompile a call to that function and not any other?

Bill. 

[Bill Hart]

The entries in the dll do look different:

d000331.o:

0000000000000000 i .idata$4

0000000000000000 i .idata$5

0000000000000000 i .idata$6

0000000000000000 i .idata$7

0000000000000000 t .text

0000000000000000 T __gmpn_sub

0000000000000000 I __imp___gmpn_sub

                 U _head_cygmpir_16_dll

d000330.o:

0000000000000000 i .idata$4

0000000000000000 i .idata$5

0000000000000000 i .idata$6

0000000000000000 i .idata$7

0000000000000000 t .text

0000000000000000 I __imp___gmpn_store

0000000000000000 I __nm___gmpn_store

                 U _head_cygmpir_16_dll

It looks like the symbol name is not exported properly.There's no T entry!

Bill.

[Bill Hart]

That doesn't seem to be the problem. Other functions that have no T entry work perfectly well.

Bill.

[Bill Hart]

Dropping the optimisation level doesn't help.

[Jean-Pierre Flori]

Or maybe that's an binutils/ld bug as I guess ld is in charge of putting the right address in the call[q] arg.

[Bill Hart]

It even segfaults with the second parameter 0, in which case the function does not touch memory. So it is definitely not a bug in our asm code.

[Bill Hart]

Yeah that seems more likely, doesn't it.

Why on earth for just that one function though. It's totally bizarre.

[Bill Hart]

It's nothing to do with the MPN_ZERO macro. A straight call to mpn_store segfaults.

Bill.

[Bill Hart]

I think it is the compiler, not the linker. For a call to mpn_add_n (also assembly) it emits:

        call    *__imp___gmpn_add_n(%rip)

For a call to mpn_store, it emits:

        call    __gmpn_store

The question is, why?

Bill.

[Bill Hart]

It compiles correctly if it is declared with __GMP_PROTO instead of _PROTO.

No idea why!!

Bill.

[Bill Hart]

Ah, no sorry, ignore that. It doesn't. Same problem.

[Bill Hart]

Even if I extern __gmpn_store and don't include gmp-impl.h, it still segfaults.

This is wild!

Bill.

[Bill Hart]

What does definitely work is this:

#define mpn_store __MPN(store)

__GMP_DECLSPEC mp_limb_t mpn_store __GMP_PROTO ((mp_ptr, mp_size_t, mp_limb_t));

Bill.

[Bill Hart]

Got it!

/* __GMP_DECLSPEC supports Windows DLL versions of libmpir, and is empty in

   all other circumstances.

   When compiling objects for libmpir, __GMP_DECLSPEC is an export directive,

   or when compiling for an application it's an import directive.  The two

   cases are differentiated by __GMP_WITHIN_GMP defined by the GMP Makefiles

   (and not defined from an application).

   __GMP_DECLSPEC_XX is similarly used for libmpirxx.  __GMP_WITHIN_GMPXX

   indicates when building libmpirxx, and in that case libmpirxx functions are

   exports, but libmpir functions which might get called are imports.

   libmp.la uses __GMP_DECLSPEC, just as if it were libmpir.la.  libmpir and

   libmp don't call each other, so there's no conflict or confusion.

   Libtool DLL_EXPORT define is not used.

   There's no attempt to support GMP built both static and DLL.  Doing so

   would mean applications would have to tell us which of the two is going

   to be used when linking, and that seems very tedious and error prone if

   using GMP by hand, and equally tedious from a package since autoconf and

   automake don't give much help.

   __GMP_DECLSPEC is required on all documented global functions and

   variables, the various internals in gmp-impl.h etc can be left unadorned.

   But internals used by the test programs or speed measuring programs

   should have __GMP_DECLSPEC, and certainly constants or variables must

   have it or the wrong address will be resolved.

   In gcc __declspec can go at either the start or end of a prototype.

   In Microsoft C __declspec must go at the start, or after the type like

   void __declspec(...) *foo()".  There's no __dllexport or anything to

   guard against someone foolish #defining dllexport.  _export used to be

   available, but no longer.

*/

[Bill Hart]

All the C tests pass now. Just checking the C++ ones.

[Jean-Pierre Flori]

Great!

By the way, this looks quite interesting:
http://www.cygwin.com/ml/cygwin/2002-01/msg00236.html

Maybe we could just get rid of the dllexport/import nightmare? at least on Cygwin (and MinGW?).

[Jean-Pierre Flori]

FTR: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/win32.html

[leif]

Jean-Pierre Flori wrote:
> On Wednesday, April 2, 2014 1:47:30 AM UTC+2, Bill Hart wrote:
>
> All the C tests pass now. Just checking the C++ ones.
>
>
> On 2 April 2014 00:49, Bill Hart <goodwi...@googlemail.com

> <javascript:>> wrote:
>
> Got it!
>
> /* __GMP_DECLSPEC supports Windows DLL versions of libmpir, and
> is empty in
> all other circumstances.

Just wonder how the correct declarations of the affected (called)
functions vanished.

> Great!
>
> By the way, this looks quite interesting:
> http://www.cygwin.com/ml/cygwin/2002-01/msg00236.html
>
> Maybe we could just get rid of the dllexport/import nightmare? at least
> on Cygwin (and MinGW?).

Yes. Just install Youbetcha... B)


-leif

P.S.: The "nice" aspect of import libs used to be the fact that you
didn't need the DLLs at all to build executables, just proper (partially
hand-[re]made) headers and the automatically generated, non-copyrighted
hence also freely distributable import libs. So you could easily
cross-compile for Windows on any *nix box (with GCC and binutils
configured accordingly).

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Bill Hart]

On 2 April 2014 11:34, leif <not.r...@online.de> wrote:

Jean-Pierre Flori wrote:

On Wednesday, April 2, 2014 1:47:30 AM UTC+2, Bill Hart wrote:

    All the C tests pass now. Just checking the C++ ones.


    On 2 April 2014 00:49, Bill Hart <goodwi...@googlemail.com

    <javascript:>> wrote:

        Got it!

        /* __GMP_DECLSPEC supports Windows DLL versions of libmpir, and
        is empty in
            all other circumstances.

Just wonder how the correct declarations of the affected (called) functions vanished.

We've never supported Cygwin64, so this has not been an issue until now. The correct declaration had never been added to this function.

[Brian Gladman]

On 02/04/2014 11:30, Bill Hart wrote:

[snip detail]

>
> We've never supported Cygwin64, so this has not been an issue until now.
> The correct declaration had never been added to this function.

It's a bit odd that this never showed up on the Visual Studio builds as
I have had to add these definitions in several places over the years.

Having to remeber the very obscurely named __GMP_WITHIN_GMP define is a
nuisance - it would be easier if we used a sensible name for this such
as BUILDING_DLL. Is there a wider significance to this name - if not I
vote to change it!

Brian