MPIR support for 64-bit integers on Windows

[Cactus]

The issue of 32/64-bit integers on Windows comes up regularly and, after the recent discussion, I have been wondering if there is a relatively easy implementation strategy.

Since we already know from 64-bit Unix/Linux distributions that the existing *_ui and *_si functions work when signed and unsigned integers are 64 bit long, could we not:

 (a) define two _new_ integer types for signed and unsigned integers 

 (b) use these for _both_ the interfaces and internally in all *_ui and *_si functions

 (c) normally define these types as signed and unsigned integers

 (d) but allow alternative definitions, for example, using 64-bit types on Windows x64

We would need to check out all the ui/si function code for oddball cases but my guess is that a lot of it would work without changes.

This would not only cover the Windows problem but would also allow these defines to be set to intmax_t and uintmax_t where this was advantageous. 

It would also avoid a lot of new code where, for example, we duplicate all existing ui/si functions but with 64-bit integers (which seems to me very messy since we end up with a lot more code to test and maintain)

Would this work?  Would there be unintended consequences?  

If it's a stupid idea just say so!

[Bill Hart]

Do the internal library assembly functions work with 64 bit inputs, or
do they rely on extending 32 bit inputs?

> --
> You received this message because you are subscribed to the Google Groups
> "mpir-devel" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mpir-devel/-/MCgGFdS3qAsJ.
> To post to this group, send email to mpir-...@googlegroups.com.
> To unsubscribe from this group, send email to
> mpir-devel+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mpir-devel?hl=en.
>

[Cactus]

All the Windows x64 MPN assembler code now handles 64-bit integers (this was not always the case).

[Jason]

Sounds good and simple , the only possible drawback is weird things like this
from gmp-in.h

/* __GMP_USHRT_MAX is not "~ (unsigned short) 0" because short is promoted
to int by "~". */
#define __GMP_UINT_MAX (~ (unsigned) 0)
#define __GMP_ULONG_MAX (~ (unsigned long) 0)
#define __GMP_USHRT_MAX ((unsigned short) ~0)

although the integer type are always? promoted to bigger not smaller types

Jason

[Cactus]

I have now got a version of MPIR in which it is possible to set the types of the integers that are used in all the ui/si functions.  

As usual it was harder than I expected to get this to work but it now passes all the tests on win32 and x64 systems with the integer sizes set to be equal to the limb sizes.

Since this version involves a lot of changes, we need an extended testing period so we need volunteers who are willing to build it, test it and use it in some applications.

Although the testing needed is primarily on Windows x64 (where ui/si integers have changed from 32 to 64 bits) it also needs testing on Windows (win32) and on Unix/Linux to check that I have not messed anything up.

If anyone is willing to test it, I will work out how best to make this version available.  I have developed it outside the SVN as I am not familiar with creating branches.  It would make sense for me to add it as an SVN branch but, sadly, I don't have a clue how to do this.

   Brian

[Case Van Horsen]

Brian,

I'm willing to test.

casevh

>
> --
> You received this message because you are subscribed to the Google Groups
> "mpir-devel" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mpir-devel/-/h0iAqcgbx8EJ.

[David Cleaver]

On 12/14/2011 4:06 PM, Cactus wrote:
> If anyone is willing to test it, I will work out how best to make this version
> available. I have developed it outside the SVN as I am not familiar with
> creating branches. It would make sense for me to add it as an SVN branch but,
> sadly, I don't have a clue how to do this.
>
> Brian

I would be willing to test it out in my WinXPx64/Msys/Mingw64 environment.

-David C.

[Sisyphus]

----- Original Message -----
From: "Cactus"

> Since this version involves a lot of changes, we need an extended testing
> period so we need volunteers who are willing to build it, test it and use
> it in some applications.

I can do that.

Cheers,
Rob

[Cactus]

My experimental version of MPIR with support for 64-bit integer (ui/si) conversions is now available here:

http://www.mpir.org/mpir.experimental.zip

This ZIP file contains only files that need to change with respect to the MPIR trunk.  So after loading a version of MPIR from SVN trunk into a directory, use the ZIP file to overwrite the files that have changed. You will also ned to overwrite mpir.h with the version attached here (I overlooked this when making the ZIP file).

I have only tested the Visual Studio builds but I hope I have not disturbed other builds, which I hope will work.  

Please do NOT use this where results matter as there will be errors in it right now.  Pleasse report issues here.

   Brian

[Cactus]

I have now added this to the MPIR SVN as the branch mpir-exp (with some updates).

    Brian

[David Cleaver]

On 12/16/2011 3:01 PM, Cactus wrote:
> I have now added this to the MPIR SVN as the branch mpir-exp (with some updates).
>
> Brian

Hello all,

I am using mingw64 as my compiler inside an msys shell. I recently downloaded
the experimental branch from svn and got version 3752.
I ran './configure --enable-gmpcompat' and have come across a problem.
Basically nothing is defined inside of config.h. Everything has an #undef in
front of it. The 'configure' script is the same, line-for-line, as the one in
the mpir-2.5.0-rc1 directory. Even config.log is the same in both directories.

Can someone help me track down why nothing is defined in config.h? What
information would help track this down?

-David C.

P.S. I did try copying the config.h from the rc1 directory to the exp
directory, but 'make' eventually ran into a problem. I think if we can figure
out the config.h problem, it should help alleviate this later problem.

[Cactus]

Hi David,

I only support Visual Studio builds so I am afraid that we need someone who uses the cyggwin/mingw/mingw64 builds to help you with this.

I have just done a fairly major update to mpir-exp in SVN but I don't think this will help with your problem. For others I do recommend that this revised version is used.

To help with debugging and testing, here are the changes that I have made in mpir-exp relative to mpir (both in SVN):

1.  Two new types have been introduced for use as the integers in all MPIR ui/si functions. These are called gmp_ui and gmp_si

2. All ui/si functions have been changed to use these new types

3. These types are set to long types except on Windows x64 where they are set to long long types.

4. The ui/si functions have been reviewed to ensure that internal integer variables are of the right length

5: 4 new functions have been added mpf_fits_ui, mpf_fits_si, mpz_fits_ui and mpz_fits_si

I have also made an unrelated change, which is intended to ensure that the move to Visual Studio 11 will go smoothly:

1.  The output directories that used to be mpir\build.vc10\dll and mpir\build.vc10\lib are now mpir\dll and mpir\lib

Provided that build tools use my prebuild and postbuild batch files, this change should be transparent. 

I have also improved the python build generator to ensure that its output is compatible with Visual Studio 11.

    Brian

[Sisyphus]

----- Original Message -----
From: "Cactus"

> You
> will also ned to overwrite mpir.h with the version attached here (I
> overlooked this when making the ZIP file).

But isn't mpir.h built by the configure process ?
It's not in the top-level directory of the svn source that I downloaded, but
*does* appear there after./configure has been run.

If that's so, then we would need to introduce that mpir.h into the build
*after* we've run ./configure - which is a dubious practice. (Introducing it
prior to ./configure would be equally dubious - and I'm guessing it would
only get overwritten anyway.)

Not sure if that has anything to do with the problem that David is
experiencing, and I won't have time to take a look for a day or two.

(Sorry if I've missed something, and am being stupid.)

Cheers,
Rob

[David Cleaver]

On 12/17/2011 11:23 PM, David Cleaver wrote:
>
> On 12/16/2011 3:01 PM, Cactus wrote:
>> I have now added this to the MPIR SVN as the branch mpir-exp (with some updates).
>>
>> Brian
>
> Hello all,
>
> I am using mingw64 as my compiler inside an msys shell. I recently downloaded
> the experimental branch from svn and got version 3752.
> I ran './configure --enable-gmpcompat' and have come across a problem. Basically
> nothing is defined inside of config.h. Everything has an #undef in front of it.
> The 'configure' script is the same, line-for-line, as the one in the
> mpir-2.5.0-rc1 directory. Even config.log is the same in both directories.
>

All of the following is based on svn 3757.

Alright, I finally found out why nothing was defined in my config.h. It turns
out that the config.in file has CRLF line endings in this experimental
directory, but it only has the LF line endings in the 2.5.0-rc1 directory. This
caused my tools to not properly process the config.in file into a config.h file.
Brian, can you check to see if there are CRLF line endings in the svn? Or
maybe my svn client changed the line endings when I downloaded it? If it is
CRLF in the svn repository, can we change that to LF?

Then, when that was fixed, I ran 'make' which went until the mpq directory and
came to an error with the mpq/cmp_ui.c/mpq_cmp_ui function saying it was
different from the mpir.h definition (on my mpir.h line 1252).
mpir.h has:
__GMP_DECLSPEC int _mpq_cmp_ui __GMP_PROTO ((mpq_srcptr, gmp_si, gmp_si))
__GMP_ATTRIBUTE_PURE;
and mpq/cmp_ui.c has:
_mpq_cmp_ui (mpq_srcptr op1, gmp_ui num2, gmp_ui den2)

I think the definition in mpir.h should be changed to:
__GMP_DECLSPEC int _mpq_cmp_ui __GMP_PROTO ((mpq_srcptr, gmp_ui, gmp_ui))
__GMP_ATTRIBUTE_PURE;
To match what is in the mpq/cmp_ui.c file.

Then, once I changed mpir.h to match the mpq/cmp_ui.c, both 'make' and 'make
check' ran successfully to completion. [except for my printf issue, of course ;) ]

On a different, but related note, with both the 2.5.0-rc1 and this 2.5.0-exp, I
have seen various warning messages scroll by, such as:
warning: cast from pointer to integer of different size
warning: overflow in implicit constant conversion
warning: initialization from incompatible pointer type
warning: passing argument 1 of '__gmpz_get_d_2exp' from incompatible pointer type
warning: integer overflow in expression
warning: assignment from incompatible pointer type
warning: passing argument 1 of '__gmpf_get_d_2exp' from incompatible pointer type

I only see these messages when running 'make check', and all of the tests in
'make check' pass. So, maybe this isn't anything to worry about? But, if you'd
like, I can go through and find each one and write back with a list of changes
that would be needed.

-David C.

P.S. I didn't copy the mpir.h file from one of your previous posts. That isn't
needed any more, is it?

[Cactus]

Hi Rob,

The need to use this mpir.h version was only a temporary measure, one that is no longer necessary with the SVN version of mpir-exp.

The Visual Studio build doesn't use the Unix/Linux build infrastructure and the original mpir-exp version that I supplied via a link did not build mpir.h from gmp_h.in so no overwriting would have occurred.

The SVN version of mpir-exp does now build mpir.h from gmp_h.in but it does not do this using configure.

    Brian

[Cactus]

Hi David

Thank you for your efforts and for finding the bad declaration _mpq_cmp_ui, which I will shortly correct in the SVN.

THe CRLF vs LF issue is a nuisance but I don't really know what can be done about it.  

This is not the first time the issue has arisen - Jason and I spent some time working out why I could not build with mingw64 some months ago and it was this issue that caused the problem.  It's bizarre that tools doing native builds fail with native line endings but I am not sure whether anything is being done about this on mingw and/or mingw64.

But following the earlier problem with line endings, I set my local SVN client to do automatic translation so that my files have CRLF line endings but are (supposed to be) translated to and from LF line endings when transferred to and from the SVN.    But from what you have found, it seems that this does not behave as it is supposed to :-(

If you are working from SVN (and keeping up to date with my SVN changes), you don't need to use the mpir.h file I supplied earlier.

     Brian

[Cactus]

I have just found an easy solution for the CRLF issue. 

It occurred to me that the SVN automatic translation has to check file types in order to avoid messing up binary files.  

I looked at this and it turns out that the SVN configuration file contains a list of file extensions that specify which files are subject to line ending translation.  Although these already include *.txt, *.bat and other common fie name extensions, they did not include *.in so the gmp_h.in did not get translated.  

I have now added this extension to my SVN configuration file so my edits of gmp_h.in should not cause problems in future. And since I think I am probably the only Windows user who is going to edit this file, this ought to avoid the issue in future.

    Brian

[Sisyphus]

----- Original Message -----
From: "David Cleaver"

> Or maybe my svn client changed the line endings when I downloaded it?

That's what's happened. Cygwin's svn faithfully reproduces the files, if you
have it available.
If not, I would hope that there is a way to turn that "feature" off on your
version of svn, so that every file is downloaded "as is" (in binary mode,
not text mode).

Cheers,
Rob

[David Cleaver]

On 12/19/2011 2:46 AM, Cactus wrote:
>
> THe CRLF vs LF issue is a nuisance but I don't really know what can be done
> about it.
>

I found a way to update the configure file so that if it encounters CRLF line
endings in the config.in file, it will strip out the CR and then proceed
normally. Would this be an acceptable solution so that it doesn't matter what
svn client was used for the download?

I am including the diff between the original 'configure' and the new
'configure'. This diff is based off of svn 3759.

$ diff configure configure.new
29942c29942,29943
< /^[\t ]*#[\t ]*(define|undef)[\t ]+$ac_word_re([\t (]|\$)/ {
---
> /^[\t ]*#[\t ]*(define|undef)[\t ]+$ac_word_re([\t (]|\r\$|\$)/ {
> sub(/\r\$/,"")
32965c32966,32967
< /^[\t ]*#[\t ]*(define|undef)[\t ]+$ac_word_re([\t (]|\$)/ {
---
> /^[\t ]*#[\t ]*(define|undef)[\t ]+$ac_word_re([\t (]|\r\$|\$)/ {
> sub(/\r\$/,"")
35994c35996,35997
< /^[\t ]*#[\t ]*(define|undef)[\t ]+$ac_word_re([\t (]|\$)/ {
---
> /^[\t ]*#[\t ]*(define|undef)[\t ]+$ac_word_re([\t (]|\r\$|\$)/ {
> sub(/\r\$/,"")

I would be interested to know if this works for others in their environments.

-David C.

[Jason]

The distributed versions and rc* one are automatically fixed of LFCR problems , it's just the svn that isn't , although every time
certain autotools/scripts are run they get fixed

[Jason]

thanks , but configure is a machine generated file from configure.in and autotools , so this would
not be very useful , I think autotools will fix it soon , in the next mpir we will be upgrading to the latest
autotools anyway. the script regen in the develop directory will do this sort sort of correction and also fill
in missing files etc

Thanks
Jason