C++ wrappers don't support 64-bit integers (size_t, long long int) on Win64

[Joris Mooij]

Hi,

first of all, I would like to thank the developers of (GMP and) MPIR
for this great library. I am using it in my own open source library,
libDAI.

When using MPIR in a Win64 environment, I encountered the problem that
the C++ wrappers do not seem to handle 64-bit integers, like size_t's
(or long long ints). This is illustrated by the following code, which
compiles fine under 64-bit linux, but fails to compile in a Windows 64
environment:

---
#include <gmpxx.h>
#include <iostream>
using namespace std;
int main(void) {
size_t a = 5;
mpz_class b(a);
cout << b.get_ui() << endl;
return 0;
}
---

The reason seems to be that a size_t is a 64-bit integer in Win64,
whereas an unsigned long is only 32 bits in that environment, and the C
++ wrapper classes in MPIR only provide a constructor for mpz_class
from (up to) 32-bit integer types. The C part of the library does
provide such functionality by the function mpz_init_set_ux().
On 64-bit linux, however, an unsigned long is 64 bits, and therefore
the code runs fine.

Now to me, this seems to be an undesirable cross-platform
incompatability. On the MPIR homepage, I read that two of the primary
goals of MPIR are:
- To maintain full interface support with GMP - MPIR is a drop-in
replacement for GMP.
- Support for building MPIR using Microsoft Visual Studio 2010 for use
in both 32-bit and 64-bit versions of Windows.
Such a platform dependence as discussed above seems to conflict with
these goals.

So I have the following questions (sorry, I am new to this list, and
didn't read all the previous conversations, although I tried searching
for this issue on this list, without much success):
- Is this a known issue (I saw some other posts on this list and on
the internet which mention, so I suspect the answer would be "yes")?
- Is somebody working on it?
- Would it be difficult to add support for 64-bit integers to the C++
wrappers also for platforms where an unsigned long is only 32 bits?
- Is there a reason that this hasn't been dealt with yet?

Thanks for your time,
Joris

[Bill Hart]

Hi Joris,

It might be worth us including a 64 bit C++ constructor for mpz's on
Windows 64. At the moment the constructor basically mimics mpz_set_ui,
which explains the limitation.

Brian may be able to let us know whether it is even possible to do
this (I don't know enough C++ to say for sure).

Bill.

> --
> You received this message because you are subscribed to the Google Groups "mpir-devel" group.
> To post to this group, send email to mpir-...@googlegroups.com.
> To unsubscribe from this group, send email to mpir-devel+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mpir-devel?hl=en.
>
>

[Cactus]

This is a well known issue that derives from a fundamental incompatibility between the data models used by GMP (and MPIR) and Windows x64.  

GMP and MPIR use the LP64 model whereas Windows x64 uses the LLP64 model - in consequence integers and longs on Windows x64 integers are 32-bits whereas in most (all?) Linux/Unix 64-bit distributions, integers are 64-bits.   

And a consequence of this is that all the functions dealing with integers in GMP on Windows x64 (e.g. the get/set_ui functions) all handle 32 bit integers.

The only functions that are capable of dealing with 64 bit integers are the get_ux(), set_ux(), get_sx() and set_sx() functions that have been added to handle 64-bit integers.

As Bill says, we could add a 64-bit constructor to mpirxx.h - I'll look at this to see if it would be easy to do.

    Brian

[Cactus]

Following up my earlier response, size_t on Windows x64 is a 32-bit integer so the test program you used won't use a 64-bit integer.  To obtain 64-bit integers you can use intmax_t and uintmax_t 

I found it easy to add a 64-bit constructor to mpirxx.h (NOT gmpxx.h) and this program then gives the output '9223372036854775808'

#include <cstdint>

#include <iostream>

#include "mpirxx.h"

using namespace std;

int main(void) 

{

    uintmax_t a = 1ull << 63;

    mpz_class b(a);

    cout << b.get_ux() << endl;

    return 0;

}

I have not yet added this to mpirxx.h in the SVN but I can do so if you want to try it.

It seems like a useful addition - what do others think?

   Brian

[Joris Mooij]

Thanks for your answers so far.

Actually, adding a constructor seems to be a simple two-line patch to
mpixx.h:
At line 1565 of the file mpirxx.h (in release 2.4.0), insert:
__gmp_expr(intmax_t l) { mpz_init_set_sx(mp, l); }
__gmp_expr(uintmax_t l) { mpz_init_set_ux(mp, l); }

I realize that my question was not general enough. Although my code
example only illustrated the incompatability of the constructor, the
incompatibility also includes other ways in which a mpz_class can
interact with 64-bit integers, for example, operator<=(long long) or
operator+=(long long). So if you would only add the constructor, that
would deal with one incompatibility, but many other related ones would
remain. (At least that's what I believe - right now I am puzzled why
the compiler cannot just automatically upcast the long longs to
mpz_class and do the comparison or addition or whatever with those
instead...)

I've taken a look at the contents of mpixx.h, and it seems as if there
is a lot of code duplication, implementing special cases for all
possible integer types up to and including long ints. If one would
duplicate all this code once more, for intmax_t, would that work?
Because it looks like a lot of work, and I don't have Win64 myself
around to test it, I would like to know first whether that would solve
the problem, or whether there's a more fundamental underlying problem.
For example, in all these __gmp_binary_{plus/minus/multiplies/*}
structs, do the relevant corresponding mpz_* functions exist for
intmax_t's? If these don't exist, then I would not know how to fix
that without delving deeply into the internals of MPIR/GMP... so then
I wouldn't even start to try fixing this.

From Brian's answer:

On Oct 24, 7:14 pm, Cactus <rieman...@gmail.com> wrote:
> The only functions that are capable of dealing with 64 bit integers are the
> get_ux(), set_ux(), get_sx() and set_sx() functions that have been added to
> handle 64-bit integers.

I understand that only the constructor incompatibility can be solved
easily, and that the other operations supplied by the C++ wrapper
first need support in the C part of the library. Is that correct?

[Cactus]

As you have correctly surmise, solving this requires a full rewrite of about 50 functions in the underlying C library, all of which are written in a way that will use 32-bit integer types on Windows.

This is a large undertaking and, so far, nobody has been willing to embark on this.  With enough volunteers, we could do this in a reasonable time but I am not keen to start unless we get a group who are fully committed to help in the implementation. 

In the meantime, as you suggest, it is straightforward to add a few C++ calls that access the sx/ux functions (constructors, copy, conversion).  I just added these in the MPIR SVN repository.

   Brian

   

[Bill Hart]

To my knowledge there are not even mpz_add_ux, mpz_mul_ux functions,
etc. in the MPIR C library, let alone C++ analogues.

I think it is essentially assumed that a C/C++ program on Windows will
use unsigned longs and longs throughout, not 64 bit integers.

>
> From Brian's answer:
>
> On Oct 24, 7:14 pm, Cactus <rieman...@gmail.com> wrote:
>> The only functions that are capable of dealing with 64 bit integers are the
>> get_ux(), set_ux(), get_sx() and set_sx() functions that have been added to
>> handle 64-bit integers.
>
> I understand that only the constructor incompatibility can be solved
> easily, and that the other operations supplied by the C++ wrapper
> first need support in the C part of the library. Is that correct?
>

I think so.

Bill.

[Joris Mooij]

On Oct 24, 8:30 pm, Cactus <rieman...@gmail.com> wrote:
> Following up my earlier response, size_t on Windows x64 is a 32-bit integer
> so the test program you used won't use a 64-bit integer.  To obtain 64-bit
> integers you can use intmax_t and uintmax_t

Brian, I'm really surprised that size_t on Windows x64 is a 32-bit
integer!

When I google on "definition size_t msvc 2010 windows 64" it brings me
to this page:
http://msdn.microsoft.com/en-us/library/3b2e7499.aspx
which clearly states:
"size_t, time_t, and ptrdiff_t are 64-bit values on 64-bit Windows
operating systems."

Maybe I wasn't clear enough - when speaking about 64-bit Windows, I
actually meant on Windows x64 in MSVC
(I don't know about gcc on Win64, it might have similar issues, or
not, depending on which memory model it uses).

Joris

[Joris Mooij]

On Oct 24, 9:02 pm, Cactus <rieman...@gmail.com> wrote:
> As you have correctly surmise, solving this requires a full rewrite of about
> 50 functions in the underlying C library, all of which are written in a way
> that will use 32-bit integer types on Windows.

Ah... that's unfortunate. Good to know before I start patching
mpirxx.h myself :)

> This is a large undertaking and, so far, nobody has been willing to embark
> on this.  With enough volunteers, we could do this in a reasonable time but
> I am not keen to start unless we get a group who are fully committed to help
> in the implementation.

Yes, I can imagine this from my short exposure to the source code.

> In the meantime, as you suggest, it is straightforward to add a few C++
> calls that access the sx/ux functions (constructors, copy, conversion).  I
> just added these in the MPIR SVN repository.

Well, that's great, I hope it will solve at least a few of these
problems for some people. I am still puzzled about why the compiler
didn't automatically upcast given the constructor from long long, but
maybe the reason is that assignment or copy were still missing as
well.

Thanks a lot for the quick responses, Brian and Bill! I'll see whether
the SVN version fixes the incompatibilities in my case.

- Joris

[Cactus]

You are right - size_t is supposed to be 64-bits on Windows x64.  But to get this behaviour, you have to define _WIN64 during the build process.

I have been caught by this so many times because  _WIN64 is supposed to be a predefined macro, onr that is automatically defined for 64-bit builds.  

But, as far as I can see, it has to be user defined.

   Brian

[leif]

Bill Hart wrote:
> To my knowledge there are not even mpz_add_ux, mpz_mul_ux functions,
> etc. in the MPIR C library, let alone C++ analogues.

Well, is it worth adding such?

If you're using a platform which, despite its machine word width being
64 bits, doesn't support an LP64 programming environment, you have to
pay the price.

Obviously one can convert 64-bit long long ints to mpzs and compute with
these.

Adding more _ux and _sx functions would perhaps make more sense once we
get long longs that are, e.g., 128 bits on a variety of platforms.

Otherwise adding such functions would just break compatibility with GMP
[further], i.e., programs using these functions wouldn't work with GMP
on other platforms. Although it's "To maintain full interface support
with GMP - MPIR is a drop-in replacement for GMP", not the other way
around. ;-)

2ct,

-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML Email

[Bill Hart]

On 24 October 2011 20:31, leif <not.r...@online.de> wrote:
> Bill Hart wrote:
>> To my knowledge there are not even mpz_add_ux, mpz_mul_ux functions,
>> etc. in the MPIR C library, let alone C++ analogues.
>
> Well, is it worth adding such?
>
> If you're using a platform which, despite its machine word width being
> 64 bits, doesn't support an LP64 programming environment, you have to
> pay the price.
>
> Obviously one can convert 64-bit long long ints to mpzs and compute with
> these.
>
> Adding more _ux and _sx functions would perhaps make more sense once we
> get long longs that are, e.g., 128 bits on a variety of platforms.

That will never happen. 64 bits is enough for anyone.

>
> Otherwise adding such functions would just break compatibility with GMP
> [further], i.e., programs using these functions wouldn't work with GMP
> on other platforms.  Although it's "To maintain full interface support
> with GMP - MPIR is a drop-in replacement for GMP", not the other way
> around. ;-)
>
>

There are obviously competing project aims at work here.

Bill.

[leif]

Bill Hart wrote:
> On 24 October 2011 20:31, leif <not.r...@online.de> wrote:
>> Adding more _ux and _sx functions would perhaps make more sense once we
>> get long longs that are, e.g., 128 bits on a variety of platforms.
>
> That will never happen. 64 bits is enough for anyone.

I bet the same had been said about 32-bit ints (and pointers, or an
address space of 4GB) when they were introduced.

64 bits are enough, just like 32 bits are enough for IP addresses.

[terry...@lyonstech.net]

Sorry this is late in the day, but surely defining _Win64 is required internals to getting an x64 build?  In Intel MKL at least ILP64 is the usual notation for 64bit integers on x64 - while LP64 refers to using 32 bit integers in an x64 environment. This is orthogonal to the language used in this thread.

At the risk of opening old discussions, one point I never understood is the trouble over 32bit and 64 bit integers in GMP/MPIR. It seems this is still (2016) causing trouble in some parts of the testing code. The mkl solution is one way, with distinct ILP64, LP64, WIN32 libraries; but ?unless I am missing something, it seems that all MPIR wanted was integers that are 64 bit on 64 bit machines and 32 bits on 32 bit machines. Surely this configuration is always available from the original unix code via a global replace of unsigned int by std::size_t and int by std::ptrdiff_t in <cstddef>.  As far as I can see this is truly system independent and standard compliant.

If one wants precise bit length integers then int32_t and int64_t seem the way to go - as they are (rather elegantly) only defined in the standard if the OS actually has those lengths (which it may not since a byte may not be 8 bits).

The subtle flaws referred to in MPIR documentation really come neither from windows or unix but from using a non-standard defined behaviour of int when really this is not necessary (although understandable). The type int should probably not appear anywhere in the code of MPIR except for input and output to other api's (such as output) where it should be checked for overflow using safeint<> or similar. I worry that builds (at least of tests) still show some 64bit<->32bit conversion warnings. I suppose now that the underling library code is really OK. I certainly believe/assume that if I use strings to serialise and pass long integers between machines in MPI there is no danger of lost information. In fact I think MPIR::GMP is a great tool.

[Bill Hart]

MPIR is 250,000 lines of code. Does this possibly answer your questions?

We have 1 (ONE) Windows developer, and one hell of a lot of users. It would certainly help if people using MPIR on Windows were able to contribute back to the project in the form of patches (preferably in the form of GitHub Pull Requests).

It's my hope that Microsoft making Ubuntu available on Windows and projects like Visual Studio Code will increase the ratio of developers to users of Open Source software on Windows.

Bill.

--
You received this message because you are subscribed to the Google Groups "mpir-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mpir-devel+...@googlegroups.com.

To post to this group, send email to mpir-...@googlegroups.com.

Visit this group at https://groups.google.com/group/mpir-devel.
For more options, visit https://groups.google.com/d/optout.

[Brian Gladman]

Hi Terry,

As I am sure you know, there is an enormous amount of code in MPIR,
which for the most part works well in a *nix environment. We would not
want to have separate code bases for *nix/GCC and Windows/MSC so doing a
global replace would mean that code that works well on *nix/GCC would be
subject to significant changes that would in all probability lead to
quite a few bugs that would have to be tracked down and fixed. A lot of
this code is very mature and has worked without problems for a long time
so it really seems sensible to take the line that 'if it isn't broken
don't fix it.

> If one wants precise bit length integers then int32_t and int64_t seem
> the way to go - as they are (rather elegantly) only defined in the
> standard if the OS actually has those lengths (which it may not since a
> byte may not be 8 bits).
>
> The subtle flaws referred to in MPIR documentation really come neither
> from windows or unix but from using a non-standard defined behaviour of
> int when really this is not necessary (although understandable). The
> type int should probably not appear anywhere in the code of MPIR except
> for input and output to other api's (such as output) where it should be
> checked for overflow using safeint<> or similar. I worry that builds (at
> least of tests) still show some 64bit<->32bit conversion warnings. I
> suppose now that the underling library code is really OK. I certainly
> believe/assume that if I use strings to serialise and pass long integers
> between machines in MPI there is no danger of lost information. In fact
> I think MPIR::GMP is a great tool.

The code is as it is because of its origins and much of the code hasn't
changed for many years. So what you see is not what we would write if
we were to do it now.

In fact I think it is quite remarkable that we have got it working
pretty well on Windows x64 with 64-bit integer support. We do still
sometimes see 32/64 bit issues but they are, gladly, increasingly rare.

Brian