64-bit integers

[Jed Brown]

There are currently no prototypes for 64-bit integers in elemental/core/imports/mpi.hpp. Would you rather add them (possibly meaning that you have to translate 

to call BLAS) or have PETSc convert indices to 32-bit before passing them to Elemental?

[Jeff Hammond]

Do you want support for long or int64_t? The current support in
Elemental is for int, which has a different minimum size in C89 (16b)
and C99 (32b). Does PETSc require C99? Since Microsoft doesn't
provide that, it seems unlikely. I guess you guys have a configure
test for sizeof(int)==4 then.

It might make sense to add support for long as well as int32_t and
int64_t while we're at it, which is why I ask. Because long is 32b on
BGP, I think it is important to support int64_t as well (or instead).

Jeff

> --
>
>



--
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jham...@alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

[Jed Brown]

On Wed, Jan 23, 2013 at 9:14 AM, Jeff Hammond <jham...@alcf.anl.gov> wrote:

Do you want support for long or int64_t?  The current support in
Elemental is for int, which has a different minimum size in C89 (16b)
and C99 (32b).

Nonsense, see section 5.2.4.2.1.

 

 Does PETSc require C99?  

No

 

Since Microsoft doesn't
provide that, it seems unlikely.  I guess you guys have a configure
test for sizeof(int)==4 then.

It might make sense to add support for long as well as int32_t and
int64_t while we're at it, which is why I ask.  Because long is 32b on
BGP, I think it is important to support int64_t as well (or instead).

PETSc uses long long when possible and __int64 on Microsoft. Of course we would rather use stdint.h, but we need universal support before stripping out the configure tests.

 

Jeff

On Wed, Jan 23, 2013 at 8:47 AM, Jed Brown <j...@59a2.org> wrote:
> There are currently no prototypes for 64-bit integers in
> elemental/core/imports/mpi.hpp. Would you rather add them (possibly meaning
> that you have to translate
> to call BLAS) or have PETSc convert indices to 32-bit before passing them to
> Elemental?
>

> --
>
>

--
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jham...@alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

--

[Jeff Hammond]

In any case, supporting the intN_t is a good idea from my perspective.

Jeff

[Jed Brown]

On Wed, Jan 23, 2013 at 9:30 AM, Jeff Hammond <jham...@alcf.anl.gov> wrote:

In any case, supporting the intN_t is a good idea from my perspective.

Sure, but it's cosmetic at this point because we're doing the configure tests to determine that stuff. The value of stdint.h is ability to remove the configure tests. PETSc uses typedefs for everything anyway.

Apparently MS provides stdint.h as of MSVC 2010. Does anyone know how common it is for MS developers to have an older toolchain than that? Maybe we can remove the configure tests soon?

 

Jeff

--

[Jeff Hammond]

I should be clear that MSFT supports things in C99 that don't conflict
with C++. That's their issue - C++ is what they care about and they
will not do any extra work to support C language features that's are a
proper subset thereof.

Jeff

[Jed Brown]

On Wed, Jan 23, 2013 at 9:39 AM, Jeff Hammond <jham...@alcf.anl.gov> wrote:

I should be clear that MSFT supports things in C99 that don't conflict
with C++.  That's their issue - C++ is what they care about and they
will not do any extra work to support C language features that's are a
proper subset thereof.

It took them *10 years* to provide stdint.h. I don't know how they could have made it more clear that they don't give a shit about developers of portable software. It's clearly not a C versus C++ issue. Out of exasperation, several open source projects implement stdint.h for Microsoft.

http://code.google.com/p/msinttypes/

http://snipplr.com/view/18199/stdinth/

Anyway, they have been shipping a stdint.h for a couple years now so we just have to wait for it to become widely available.

 

Jeff

--

[Jack Poulson]

Hi Jed,

Are you asking about prototypes which use 64-bit indexing for specifying message properties, such as send/recv counts, or prototypes which have payloads which are arrays of 64-bit integers?

I suppose I am a bit hind on the MPI standard, as I was not aware of MPI supporting the former, and I don't see what would need to be translated for BLAS with the latter.

Jack

On Wed, Jan 23, 2013 at 6:47 AM, Jed Brown <j...@59a2.org> wrote:

There are currently no prototypes for 64-bit integers in elemental/core/imports/mpi.hpp. Would you rather add them (possibly meaning that you have to translate 

to call BLAS) or have PETSc convert indices to 32-bit before passing them to Elemental?

--
 
 

[Jeff Hammond]

MPI count arguments are "int". Whether or not to supplement the
standard in this respect was debated extensively by the MPI Forum and
we should not revisit it here. I recall there were a few new
functions added to the standard to workaround the few cases where MPI
datatypes were not a reasonable way to send messages of more than 2Bi
elements.

I believe that Jed is asking for the following type of addition:

void AllReduce( byte* buf, int count, Op op, Comm comm );
void AllReduce( int* buf, int count, Op op, Comm comm );
void AllReduce( long* buf, int count, Op op, Comm comm ); // NEW long
= MPI_LONG (not the same as MPI_LONG_INT!*)
void AllReduce( long long* buf, int count, Op op, Comm comm ); // NEW
long = MPI_LONG_LONG (the same as MPI_LONG_LONG_INT!*)
void AllReduce( float* buf, int count, Op op, Comm comm );
void AllReduce( double* buf, int count, Op op, Comm comm );
void AllReduce( scomplex* buf, int count, Op op, Comm comm );
void AllReduce( dcomplex* buf, int count, Op op, Comm comm );

* Yes, this asymmetry is stupid but that's what happens when your
standard isn't allowed to break backwards compatibility unless it is
absolutely necessary.

If Elemental is to support the former feature and allow count
arguments >2Bi, then it will have to implement some glue to turn e.g.
16Bi doubles into 16 <derived type of 1Bi doubles> (or equivalent). I
do not recommend this approach.

Jeff

On Wed, Jan 23, 2013 at 12:08 PM, Jack Poulson <jack.p...@gmail.com> wrote:
>
> Are you asking about prototypes which use 64-bit indexing for specifying
> message properties, such as send/recv counts, or prototypes which have
> payloads which are arrays of 64-bit integers?
>
> I suppose I am a bit hind on the MPI standard, as I was not aware of MPI
> supporting the former, and I don't see what would need to be translated for
> BLAS with the latter.
>
> Jack

[Jack Poulson]

Okay, this shouldn't require any thought to add. I can probably take care of it some time soon. Someone else would be welcome to submit the patch as well.

Jack

--

[Jeff Hammond]

I can do it. I'll just add support for all of the MPI build-ins that
make sense. That way, if you ever want to use them later, they will
already be there.

Jeff

> --

[Jeff Hammond]

Hi Jack,

Did you autogenerate the MPI wrappers, automate them using some Emacs
magic or did you do it by hand? If the first of these, then I should
just extend that tool. Otherwise, I will create it, because it's
silly to do it by hand. If you used Emacs magic, but that's not an
option for me*.

Thanks,

Jeff

* Everyone: please take this opportunity to pipe your VI-related
insults to /dev/null.

[Jed Brown]

On Wed, Jan 23, 2013 at 12:41 PM, Jeff Hammond <jham...@alcf.anl.gov> wrote:

MPI count arguments are "int".  Whether or not to supplement the
standard in this respect was debated extensively by the MPI Forum and
we should not revisit it here.  I recall there were a few new
functions added to the standard to workaround the few cases where MPI
datatypes were not a reasonable way to send messages of more than 2Bi
elements.

I believe that Jed is asking for the following type of addition:

void AllReduce( byte* buf, int count, Op op, Comm comm );

Yes, exactly, we have compilation errors like the following when attempting to build PETSc with 64-bit integers because we currently pass PetscInt to Elemental. The prototypes Jeff suggests will fix this problem. I don't know if you check sizes/cast appropriately before passing "int" to BLAS. If so, you might have to add an explicit conversion and raise an error if the sizes are too large (that may never happen for dense).

In file included from /sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/dist_matrix_impl.hpp:22:0,

                 from /sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core.hpp:78,

                 from /sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental.hpp:17,

                 from /sandbox/petsc/petsc-dev-3/include/../src/mat/impls/elemental/matelemimpl.h:4,

                 from /sandbox/petsc/petsc-dev-3/src/mat/impls/elemental/matelem.cxx:1:

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/dist_matrix/vc_star_impl.hpp: In member function ‘T elem::DistMatrix<T, (elem::distribution_wrapper::Distribution)3u, (elem::distribution_wrapper::Distribution)5u, Int>::Get(Int, Int) const [with T = long long int, Int = int]’:

/sandbox/petsc/petsc-dev-3/src/mat/impls/elemental/matelem.cxx:1141:1:   instantiated from here

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/dist_matrix/vc_star_impl.hpp:950:5: error: no matching function for call to ‘Broadcast(long long int*, int, int, elem::mpi::Comm)’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/dist_matrix/vc_star_impl.hpp:950:5: note: candidates are:

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:205:6: note: void elem::mpi::Broadcast(elem::byte*, int, int, elem::mpi::Comm)

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:205:6: note:   no known conversion for argument 1 from ‘long long int*’ to ‘elem::byte* {aka unsigned char*}’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:206:6: note: void elem::mpi::Broadcast(int*, int, int, elem::mpi::Comm)

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:206:6: note:   no known conversion for argument 1 from ‘long long int*’ to ‘int*’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:207:6: note: void elem::mpi::Broadcast(float*, int, int, elem::mpi::Comm)

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:207:6: note:   no known conversion for argument 1 from ‘long long int*’ to ‘float*’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:208:6: note: void elem::mpi::Broadcast(double*, int, int, elem::mpi::Comm)

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:208:6: note:   no known conversion for argument 1 from ‘long long int*’ to ‘double*’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:209:6: note: void elem::mpi::Broadcast(elem::scomplex*, int, int, elem::mpi::Comm)

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:209:6: note:   no known conversion for argument 1 from ‘long long int*’ to ‘elem::scomplex* {aka elem::Complex<float>*}’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:210:6: note: void elem::mpi::Broadcast(elem::dcomplex*, int, int, elem::mpi::Comm)

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/imports/mpi.hpp:210:6: note:   no known conversion for argument 1 from ‘long long int*’ to ‘elem::dcomplex* {aka elem::Complex<double>*}’

/sandbox/petsc/petsc-dev-3/arch-linux-pkgs-64idx/include/elemental/core/dist_matrix/vc_star_impl.hpp: In member function ‘typename elem::Base<F>::type elem::DistMatrix<T, (elem::distribution_wrapper::Distribution)3u, (elem::distribution_wrapper::Distribution)5u, Int>::GetRealPart(Int, Int) const [with T = long long int, Int = int, typename elem::Base<F>::type = long long int]’:

/sandbox/petsc/petsc-dev-3/src/mat/impls/elemental/matelem.cxx:1141:1:   instantiated from here

 

--

[Jack Poulson]

Hi Jeff,

They were generated by hand. If I was to write I tool to generate them, I would then have two problems. I'm not so sure such a tool would be a good idea, but you're welcome to make one. I promise to be objective.

Jack

--

[Jeff Hammond]

My style is to use as little magic as possible. This is why I write
in C instead of C++ :-) My tool will probably not do much more than
intelligent use of copy+paste+search+replace on top of your existing
code. While it would be easier if I could make whitespace changes,
I'll resist that urge.

I might not get to this in the next 48 hours so if you or Jed are in a
rush, I guess the minimal, manually solution will have to do.

Jeff

[Jed Brown]

Not in a rush, we found this when we added Elemental to a nightly build with 64-bit ints. We haven't had a user ask for it yet, but that day is likely coming.

--

[Jack Poulson]

Hi Jeff,

FYI, I just simplified the MPI wrappers using the technique described here:
http://jholewinski.org/blog/the-beauty-of-c-templates/

You can see the current version of the file here:
https://github.com/poulson/Elemental/blob/65f8134f6ab5e0dba017f51f4ccb55ea3c419c30/src/core/imports/mpi.cpp

It should now be straightforward to add support for more datatypes (it will only require extending the MpiMap struct and adding more explicit instantiations for each MPI wrapper).

Jack

--

[Jeff Hammond]

Hi Jack,

Indeed, templating is definitely the easy way to do this. Thanks for
implementing this. I'll work on the additional datatypes for the new
version as soon as I can.

Best,

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "elemental-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elemental-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jed Brown]

---
src/core/imports/mpi.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

[Jack Poulson]

Thanks!

https://github.com/poulson/Elemental/commit/6d157f5b485c6eaade5347d4ed8528643ab3bc0e

[Jeff Hammond]

Sorry for the delay. Here are all the C/C++ predefined types that I
could find in the MPI standard. Let me know if this is what you want
or not.

The patch file is the 'hg export' corresponding to the plain text below.

Best,

Jeff

template<>
MpiMap<_Bool>::MpiMap() : type(MPI_C_BOOL) { }
template<>
MpiMap<bool>::MpiMap() : type(MPI_CXX_BOOL) { }
template<>
MpiMap<byte>::MpiMap() : type(MPI_BYTE) { }
template<>
MpiMap<char>::MpiMap() : type(MPI_CHAR) { }
template<>
MpiMap<wchar_t>::MpiMap() : type(MPI_WCHAR) { }
template<>
MpiMap<signed char>::MpiMap() : type(MPI_SIGNED_CHAR) { }
template<>
MpiMap<unsigned char>::MpiMap() : type(MPI_UNSIGNED_CHAR) { }
template<>
MpiMap<short>::MpiMap() : type(MPI_SHORT) { }
template<>
MpiMap<short int>::MpiMap() : type(MPI_SHORT) { }
template<>
MpiMap<signed short>::MpiMap() : type(MPI_SHORT) { }
template<>
MpiMap<signed short int>::MpiMap() : type(MPI_SHORT) { }
template<>
MpiMap<unsigned short>::MpiMap() : type(MPI_UNSIGNED_SHORT) { }
template<>
MpiMap<unsigned short int>::MpiMap() : type(MPI_UNSINGED_SHORT) { }
template<>
MpiMap<int>::MpiMap() : type(MPI_INT) { }
template<>
MpiMap<signed int>::MpiMap() : type(MPI_INT) { }
template<>
MpiMap<unsigned>::MpiMap() : type(MPI_UNSIGNED) { }
template<>
MpiMap<unsigned int>::MpiMap() : type(MPI_UNSINGED) { }
template<>
MpiMap<long>::MpiMap() : type(MPI_LONG) { }
template<>
MpiMap<long int>::MpiMap() : type(MPI_LONG) { }
template<>
MpiMap<signed long>::MpiMap() : type(MPI_LONG) { }
template<>
MpiMap<signed long int>::MpiMap() : type(MPI_LONG) { }
template<>
MpiMap<unsigned long>::MpiMap() : type(MPI_UNSIGNED_LONG) { }
template<>
MpiMap<unsigned long int>::MpiMap() : type(MPI_UNSINGED_LONG) { }
template<>
MpiMap<long long>::MpiMap() : type(MPI_LONG_LONG) { }
template<>
MpiMap<long long int>::MpiMap() : type(MPI_LONG_LONG) { }
template<>
MpiMap<signed long long>::MpiMap() : type(MPI_LONG_LONG) { }
template<>
MpiMap<signed long long int>::MpiMap() : type(MPI_LONG_LONG) { }
template<>
MpiMap<unsigned long long>::MpiMap() : type(MPI_UNSIGNED_LONG_LONG) { }
template<>
MpiMap<unsigned long long int>::MpiMap() : type(MPI_UNSINGED_LONG_LONG) { }
template<>
MpiMap<float>::MpiMap() : type(MPI_FLOAT) { }
template<>
MpiMap<double>::MpiMap() : type(MPI_DOUBLE) { }
template<>
MpiMap<long double>::MpiMap() : type(MPI_LONG_DOUBLE) { }
template<>
MpiMap<Complex<float>>::MpiMap() : type(MPI_COMPLE) { }
template<>
MpiMap<Complex<double>>::MpiMap() : type(MPI_DOUBLE_COMPLEX) { }
template<>
MpiMap<Complex<long double>>::MpiMap() : type(MPI_LONG_DOUBLE_COMPLEX) { }
template<>
MpiMap<std::complex<float>>::MpiMap() : type(MPI_CXX_FLOAT_COMPLE) { }
template<>
MpiMap<std::complex<double>>::MpiMap() : type(MPI_CXX_DOUBLE_COMPLEX) { }
template<>
MpiMap<std::complex<long double>>::MpiMap() :
type(MPI_CXX_LONG_DOUBLE_COMPLEX) { }
template<>
MpiMap<ValueInt<short>>::MpiMap() : type(MPI_SHORT_INT) { }
template<>
MpiMap<ValueInt<short int>>::MpiMap() : type(MPI_SHORT_INT) { }
template<>
MpiMap<ValueInt<signed short>>::MpiMap() : type(MPI_SHORT_INT) { }
template<>
MpiMap<ValueInt<signed short int>>::MpiMap() : type(MPI_SHORT_INT) { }
template<>
MpiMap<ValueInt<int>>::MpiMap() : type(MPI_2INT) { }
template<>
MpiMap<ValueInt<singed int>>::MpiMap() : type(MPI_2INT) { }
template<>
MpiMap<ValueInt<long>>::MpiMap() : type(MPI_LONG_INT) { }
template<>
MpiMap<ValueInt<long int>>::MpiMap() : type(MPI_LONG_INT) { }
template<>
MpiMap<ValueInt<signed long>>::MpiMap() : type(MPI_LONG_INT) { }
template<>
MpiMap<ValueInt<signed long int>>::MpiMap() : type(MPI_LONG_INT) { }
template<>
MpiMap<ValueInt<FLOAT>>::MpiMap() : type(MPI_FLOAT_INT) { }
template<>
MpiMap<ValueInt<DOUBLE>>::MpiMap() : type(MPI_DOUBLE_INT) { }
template<>
MpiMap<ValueInt<LONG_DOUBLE>>::MpiMap() : type(MPI_LONG_DOUBLE_INT) { }
template<>
MpiMap<int8_t>::MpiMap() : type(MPI_INT8_T) { }
template<>
MpiMap<int16_t>::MpiMap() : type(MPI_INT16_T) { }
template<>
MpiMap<int32_t>::MpiMap() : type(MPI_INT32_T) { }
template<>
MpiMap<int64_t>::MpiMap() : type(MPI_INT64_T) { }
template<>
MpiMap<uint8_t>::MpiMap() : type(MPI_UINT8_T) { }
template<>
MpiMap<uint16_t>::MpiMap() : type(MPI_UINT16_T) { }
template<>
MpiMap<uint32_t>::MpiMap() : type(MPI_UINT32_T) { }
template<>
MpiMap<uint64_t>::MpiMap() : type(MPI_UINT64_T) { }

ALCF docs: http://www.alcf.anl.gov/user-guides

[Jack Poulson]

Hi Jeff,

Thanks!

Which datatypes do you want the MPI wrappers instantiated for? The function templates will need to be explicitly instantiated for those datatypes.

So far, the MPI wrappers have existed solely for the purpose of supporting Elemental and are nowhere near feature-complete. It would thus be best to only selective instantiate the datatypes that are needed.

If a feature-complete set of MPI wrappers is desired, it would be best to pull out the MPI wrappers from Elemental and turn it into a full-fledged external project (which Elemental could then import).

Jack

[Jeff Hammond]

Well, I'm not the one who had the problem with this, so I don't have
any needs w.r.t. instantiation. I suppose Jed can tell you what he
needs for the 64-bit PETSc build.

If you can't tell, I just mined the relevant bits from the MPI
standard and wrote a code generator for the MpiMap business, just to
ensure that that part was complete. There is no justification to
instantiation some of those types (WCHAR=WTF, at least in Elemental).

I agree that a full-service MPI C++ interface that doesn't suck would
be highly desirable, and I have some mild interest in that. However,
I have no time to devote to it right now. If/when that changes,
you'll be one of the first people to know.

Best,

Jeff

ALCF docs: http://www.alcf.anl.gov/user-guides

[Jed Brown]

Jeff Hammond <jham...@alcf.anl.gov> writes:

> Well, I'm not the one who had the problem with this, so I don't have
> any needs w.r.t. instantiation. I suppose Jed can tell you what he
> needs for the 64-bit PETSc build.

I think I stated that in my original email. Let me know if anything was
unclear.

[Jeff Hammond]

Did we ever resolve this?

I figure it is useful to have the full set of type maps even if they
aren't instantiated so that future devs merely need to do that and not
also add the typemaps.

Jeff

[Jack Poulson]

As far as I am aware, we have not. Are you proposing to take care of it?

Jack

[Jeff Hammond]

I was planning to push the complete list of type maps unless there
were objections.

I can also use my Python generator to generate all the instantiators as well.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "elemental-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elemental-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Jeff Hammond

jeff.s...@gmail.com