MPIR fails a bunch of tests on Cygwin64

[Jean-Pierre Flori]

Dear all,

I've been trying MPIR (after updating configfsf.* in src and config.* in yasm/config/) on Cygwin64 and a lot of tests (but not all of them) fail.
I had no time (and surely won't before a few days/weeks) to run gdb on them but any hint would be welcomed.

Best,
JP

[Bill Hart]

This is the first moment I have even heard of Cygwin64, so I don't have any clues for you unfortunately.

Bill.

--
You received this message because you are subscribed to the Google Groups "mpir-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpir-devel+...@googlegroups.com.
To post to this group, send email to mpir-...@googlegroups.com.
Visit this group at http://groups.google.com/group/mpir-devel.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jean-Pierre Flori]

I've reported the problem to the Cygwin folk at http://cygwin.com/ml/cygwin/2013-06/msg00539.html.

[leif]

Jean-Pierre Flori wrote:
> I've reported the problem to the Cygwin folk at
> http://cygwin.com/ml/cygwin/2013-06/msg00539.html.

Nice. Is it always 'ld' that crashes?

I'd guess that yasm is to blame (probably not yet fully prepared for
Cygwin64*), somehow generating invalid object files that cause (e.g.)
'ld' to crash.

Did you try to use some other tools (using the BFD library) on the
offending object files? (The archives in contrast are probably ok,
although perhaps not all object files they contain.)


-leif

____________
*(at least not the version currently shipped with MPIR)

> On 20 June 2013 10:40, Jean-Pierre Flori <jpf...@gmail.com

> <javascript:>> wrote:
>
> Dear all,
>
> I've been trying MPIR (after updating configfsf.* in src and
> config.* in yasm/config/) on Cygwin64 and a lot of tests (but
> not all of them) fail.
> I had no time (and surely won't before a few days/weeks) to run
> gdb on them but any hint would be welcomed.
>
> Best,
> JP

--

() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Jean-Pierre Flori]

On Friday, June 21, 2013 5:36:26 PM UTC+2, leif wrote:

Jean-Pierre Flori wrote:
> I've reported the problem to the Cygwin folk at
> http://cygwin.com/ml/cygwin/2013-06/msg00539.html.

Nice.  Is it always 'ld' that crashes?

When using the static lib, ld crashes when linking libmpir.a into the test exe.
When using the shared lib, its the test exe (a lot of them, not all) which segfaults.

So indeed the problem is surely the same.

[leif]

leif wrote:
> I'd guess that yasm is to blame (probably not yet fully prepared for
> Cygwin64*), somehow generating invalid object files that cause (e.g.)
> 'ld' to crash.

Just noticed both Cygwin(64) and MPIR 2.6.0 have yasm 1.2.0, but it
looks as if Cygwin would patch it (from the package version).

You may try to use Cygwin's yasm, or look whether they do anything
significant to its vanilla sources.

(At any rate, surely 'ld' shouldn't segfault even if yasm was the
culprit...)


Or experiment a little with --host=... (i.e., the CPU type, such that
other [assembly] files / a different MPN_PATH is chosen).


-leif

> Did you try to use some other tools (using the BFD library) on the
> offending object files? (The archives in contrast are probably ok,
> although perhaps not all object files they contain.)
>
>
> -leif
>
> ____________
> *(at least not the version currently shipped with MPIR)

[leif]

leif wrote:
> leif wrote:
>> I'd guess that yasm is to blame (probably not yet fully prepared for
>> Cygwin64*), somehow generating invalid object files that cause (e.g.)
>> 'ld' to crash.
>
> Just noticed both Cygwin(64) and MPIR 2.6.0 have yasm 1.2.0, but it
> looks as if Cygwin would patch it (from the package version).
>
> You may try to use Cygwin's yasm, or look whether they do anything
> significant to its vanilla sources.

Hmmm, doesn't look like, or at least I couldn't find anything relevant.

Perhaps GCC miscompiles it on your machine...


Could you post a configure+build[+make check] log somewhere (e.g. Sage
trac ticket #14648)?


-leif

[Jean-Pierre Flori]

I got the same result with Cygwin provided yasm, so it's ont Cygwin's GCC which miscompiles the MPIR provided yasm.

[Jean-Pierre Flori]

But it's definitely something asm related as passing MPN_PATH=generic to configure solves the tests segfault.
(The ld segfault was solved following http://www.cygwin.com/ml/cygwin/2013-06/msg00558.html)

[Jean-Pierre Flori]

On Friday, June 21, 2013 7:20:23 PM UTC+2, Jean-Pierre Flori wrote:

But it's definitely something asm related as passing MPN_PATH=generic to configure solves the tests segfault.
(The ld segfault was solved following http://www.cygwin.com/ml/cygwin/2013-06/msg00558.html)

Ok, the problem is that now the asm code used was the one targeted for System V calling convention...
The solution is basically to do the (almost) same thing on cygwin(64) as on mingw64, passing -f win64 passed to yasm,  using the x86_64w directories and renaming .asm to .as, using yasm_macwin.inc, expanding m4 macros, and so on.

Now I can build MPIR easily and it passes (nearly all of) its testsuite.

By the way it's stated one cannot build static and shared libraries at the same time because some headers are different but I cannot spot a difference except for one lind defining the value of a *DLL macro to 0 or 1...
I asked the same question on the GMP list in december 2012 and never got an answer.

[Bill Hart]

Sounds like progress!

By the way, I don't know the specifics of the static + shared thing. It's been a limitation for a very long time, including way back when very little worked properly on Windows. The build system would randomly hang, things would randomly crash, compilers were really broken, you had to use a specific version of MinGW and it was really difficult to install, everything was really, really slow, it would run out of memory when building, parallel make didn't work, antivirus flagged configure as malicious, autotools needed a special version to work with Windows, even the archiving tool could completely crash the machine when symlinks were present. So the shared + static limitation didn't seem like that much of an issue.

But perhaps these days some of these old problems have been worked out on Windows.

Bill.

--

[Jean-Pierre Flori]

Hi all,

I've upped a version of MPIR seemingly passing all its tests on Cygwin64 at https://github.com/jpflori/mpir.git (including the fix for t-scan from https://groups.google.com/forum/?hl=en#!topic/mpir-devel/oLk3gMULxu0 ).
Note that I've not included the complete fixes suggested at http://cygwin.com/ml/cygwin/2013-06/msg00580.html to remove the now obsolete hacks to shorten configure time.
I've dos2unix'ed some files though it does not seem to appear in any of the commits I pushed...
Finally some of the fixes I included do not make any difference between Cygwin32 and Cygwin64, this actually broke MPIR on Cygwin32; I'm working on it.

At some point I'll also try to make the build of yasm optional, and correct when cross compiling (see https://groups.google.com/forum/?hl=en#!topic/mpir-devel/oLk3gMULxu ).

I'll also give a shot to the static/shared limitation a shot.

Best,
JP

[Bill Hart]

Wow. That's really awesome!

Bill.

[Jean-Pierre Flori]

I've now fixed my trunk and performed basic testing (building and running the testsuite, sahred and static) on different flavours of MinGW64 (both 32 and 64 bits) and Cygwin (both 32 and 64 bits) and everything seems fine.

I've also (tried to) put back vanilla Yasm (1.2.0) sources, and added the possibility to use a system wide one.
The only modifications to these sources are now done at configure time (use updated config.(guess|sub) and a stub configure.gnu to discard superfluous configure options).
I've used "autoreconf -fiv --disable-recursive" to update the build system at the toplevel so that yasm subdirectory was not updated.
I'm not quite convinced that shipping the file generated by autoreconf (or equivalent commands) in the git trunk is such a good idea.
It seems to me that most projects only ship the input to autotools there and generate the build system at release time (and for testing/devel on one's computer of course!).

I've not struggled enough with git to manage to convince it to keep the CRLF line endings in the Yasm files having some (mostly in the VS build projects).
So these have been automatically converted to LF at commit time.
This should not be a problem as Yasm is not built with VS anyway.
Note that an old commit ensured that the x86*w files also had CRLF line endings.
This was still the case in the 2.6.0 tarball (and gave me troubles with MinGW at configure time) but was changed later (after the 2.6.0 release I'd say) in Bill's git trunk.
I have not tested anything with VS so I don't know if having assembly files with LF line endings is a problem for the VS projects.
Whatsoever, if VS requires CRLF line endings, then some magic translation will have to be performed at configure time so that both VS and MinGW are happy.

The final obscure point is about the GLOBAL_FUNC/PROLOGUE declarations in the comment parts of the x86_64w assembly code.
There are some commits mentionning adding the GLOBAL_FUNC piece and then converting it to PROLOGUE and then removing duplicate PROLOGUE declarations (always in the comments, so without real influence).
The strange thing is that when the GLOBAL_FUNC was introduced, it dropped the mpn_ prefix in the function names and that prevented the functions to be correctly detected by the configure scripts and so the correct HAVE_NATIVE_* to be defined.
In particular, that could lead to a problem already mentioned here: mpn_addmul_2 being defined twice on MinGW because redc_2 found that HAVE_NATIVE_mpn_addmul_2 != 1 whereas the function was already provided.

From what I understood, this GLOBAL_FUNC/PROLOGUE in the comments trick was only there to help the configure scripts on MinGW, so I've transformed it to produce correct results there (and on Cygwin 64).
I did not find any trace of it being used for VS, except for the g2y.py script which, if I understood correctly, is used to translate GAS asm to Yasm asm, and so is not used at build time.
If the VS project actually depend on these PROLOGUE declarations (within the top-level comments), then I might have broken the VS projects.

Best,
JP

[Brian Gladman]

On 05/08/2013 10:40, Jean-Pierre Flori wrote:

> I've now fixed my trunk and performed basic testing (building and running
> the testsuite, sahred and static) on different flavours of MinGW64 (both 32
> and 64 bits) and Cygwin (both 32 and 64 bits) and everything seems fine.

You are making good progress! It will be nice to have builds with the
32 and 64 bit versions of both mingw and cygwin.

> I've not struggled enough with git to manage to convince it to keep the
> CRLF line endings in the Yasm files having some (mostly in the VS build
> projects).
> So these have been automatically converted to LF at commit time.
> This should not be a problem as Yasm is not built with VS anyway.
> Note that an old commit ensured that the x86*w files also had CRLF line
> endings.
>
> This was still the case in the 2.6.0 tarball (and gave me troubles with
> MinGW at configure time) but was changed later (after the 2.6.0 release I'd
> say) in Bill's git trunk.
> I have not tested anything with VS so I don't know if having assembly files
> with LF line endings is a problem for the VS projects.
> Whatsoever, if VS requires CRLF line endings, then some magic translation
> will have to be performed at configure time so that both VS and MinGW are
> happy.

Visual Studio can cope with mixed line endings but I am much less
certain about YASM on Windows as I recall having problems with LF
endings in the *.asm files. I may be wrong but I think we ended up with
LF endings everywhere except for the dedicated Windows files in x86w,
x86_64w and the Windows build directories (basically all the stuff that
I manage for the Visual Studio/YASM builds).

> The final obscure point is about the GLOBAL_FUNC/PROLOGUE declarations in
> the comment parts of the x86_64w assembly code.
> There are some commits mentionning adding the GLOBAL_FUNC piece and then
> converting it to PROLOGUE and then removing duplicate PROLOGUE declarations
> (always in the comments, so without real influence).
> The strange thing is that when the GLOBAL_FUNC was introduced, it dropped
> the mpn_ prefix in the function names and that prevented the functions to
> be correctly detected by the configure scripts and so the correct
> HAVE_NATIVE_* to be defined.
> In particular, that could lead to a problem already mentioned here:
> mpn_addmul_2 being defined twice on MinGW because redc_2 found that
> HAVE_NATIVE_mpn_addmul_2 != 1 whereas the function was already provided.
>
> From what I understood, this GLOBAL_FUNC/PROLOGUE in the comments trick was
> only there to help the configure scripts on MinGW, so I've transformed it
> to produce correct results there (and on Cygwin 64).
> I did not find any trace of it being used for VS, except for the g2y.py
> script which, if I understood correctly, is used to translate GAS asm to
> Yasm asm, and so is not used at build time.
> If the VS project actually depend on these PROLOGUE declarations (within
> the top-level comments), then I might have broken the VS projects.

As you say, the commented PROLOGUE stuff in the files in the
x86w/x86_64w directories is there to support building with
mingw/mingw64. I don't use these lines anywhere in the Visual Studio
build as I scan the files for exported symbols using a different approach.

Brian

[Jean-Pierre Flori]

Hi,

Visual Studio can cope with mixed line endings but I am much less
certain about YASM on Windows as I recall having problems with LF
endings in the *.asm files.  I may be wrong but I think we ended up with
LF endings everywhere except for the dedicated Windows files in x86w,
x86_64w and the Windows build directories (basically all the stuff that
I manage for the Visual Studio/YASM builds).

Good to know.
It seems to indicate we'll have to resort to some trick to use the x86_64w directory both for VS and MinGW/Cygwin.
And carefully deal with git which has a tendency to change the line endings depending on the system where the changes are commited without really asking the user.

From what I've seen, you indeed commited a change to ensure that files in x86*w directories have CRLF line endings, but that was changed later by Bill.
If you could try to build what is currently available at https://github.com/jpflori/mpir with some assembly enabled we will quickly know if vsyasm can deal with the LF line endings files.
You should also make sure that git does not automagically convert line endings to CRLF when you clone the repo...
If you'd prefer I can provide a tarball, so that you'll be sure to get LF line endings.

 

As you say, the commented PROLOGUE stuff in the files in the
x86w/x86_64w directories is there to support building with
mingw/mingw64.  I don't use these lines anywhere in the Visual Studio
build as I scan the files for exported symbols using a different approach.

Thanks for the confirmation!
Note that the x86w dir is not used for MinGW/Cygwin in 32 bits mode.
It's the x86 dir which is and that seems fine (at least the ABI should be similar on Linux/Win32, and it builds and passes tests correctly (except for t-scanf.exe IIRC)).

Best,
JP

[Brian Gladman]

On 05/08/2013 14:03, Jean-Pierre Flori wrote:

[snip]

> If you could try to build what is currently available at
> https://github.com/jpflori/mpir with some assembly enabled we will quickly
> know if vsyasm can deal with the LF line endings files.
> You should also make sure that git does not automagically convert line
> endings to CRLF when you clone the repo...
> If you'd prefer I can provide a tarball, so that you'll be sure to get LF
> line endings.

I would need a tarball as I have LF to CRLF conversion set to on.

Brian

[Jean-Pierre Flori]

On Monday, August 5, 2013 4:32:52 PM UTC+2, Cactus wrote:

On 05/08/2013 14:03, Jean-Pierre Flori wrote:

[snip]
> If you could try to build what is currently available at
> https://github.com/jpflori/mpir with some assembly enabled we will quickly
> know if vsyasm can deal with the LF line endings files.
> You should also make sure that git does not automagically convert line
> endings to CRLF when you clone the repo...
> If you'd prefer I can provide a tarball, so that you'll be sure to get LF
> line endings.

There is one at http://boxen.math.washington.edu/home/jpflori/mpir/mpir-2.6.0+jp.tar.bz2

Best,
JP

[Brian Gladman]

On 05/08/2013 15:42, Jean-Pierre Flori wrote:
>
>
> On Monday, August 5, 2013 4:32:52 PM UTC+2, Cactus wrote:
>>
>> On 05/08/2013 14:03, Jean-Pierre Flori wrote:
>>
>> [snip]
>>> If you could try to build what is currently available at
>>> https://github.com/jpflori/mpir with some assembly enabled we will
>> quickly
>>> know if vsyasm can deal with the LF line endings files.
>>> You should also make sure that git does not automagically convert line
>>> endings to CRLF when you clone the repo...
>>> If you'd prefer I can provide a tarball, so that you'll be sure to get
>> LF
>>> line endings.
>>
> There is one at
> http://boxen.math.washington.edu/home/jpflori/mpir/mpir-2.6.0+jp.tar.bz2

I had no problems building this with Visual Studio and VSYASM without
changing the LF line endings to CRLF.

But I cannot guarantee that the line endings in the Windows files will
stay as LF whenever I edit them.

Brian

[Bill Hart]

Hi JP,

thanks for all your hard work with the above.

I've set up a todo list with all the issues I'm aware of, including some that are fixed by you already:

http://mathprogrammer.com/wiki/tiki-index.php?page=mpir-todo

Bill.

     Brian