Process ordering in PMIx_Group_construct

[Ralph Castain]

A question arose at Tuesday’s (Oct 14) meeting regarding the ordering of processes passed to the PMIx_Group_construct API. Specifically, the issue was raised that the current API states that the input array of pmix_proc_t processes is not considered ordered - i.e., the order of the procs listed in the array is disregarded by the implementation.

The reason for this is that users had encountered situations where different participants would provide arrays that contained the same proc IDs, but in different order - and the implementation treated those as a request to construct different groups. In other words, the array itself was considered a unique signature of a group. This caused group construct operations to fail as they waited for members to participate.

The currently revised definition ignores the order of the procs in the membership array, using the provided group ID as the sole identifier of the group. The only guarantee relative to process ID ordering is that the final membership array provided to every group member at completion of the construct operation will be identical in both membership and order.

However, for some users such as MPI, the ordering of members in that final array must be set to a specific pattern as dictated by the original members. It cannot be left to the implementation to decide, even if the reported array is identical for all group members.

There are some possible remedies for this situation. For the case where at least one process knows all members and the desired final membership ordering, that process can provide the desired final membership array by adding it to its own call to PMIx_Group_construct using a new PMIX_GROUP_FINAL_MEMBERSHIP_ORDER attribute. Note that multiple procs providing this attribute must have the value agree between them - otherwise, an error will be returned. Note also that wildcard ranks would be supported, so one could use this to (for example) direct that all members from nspace A be placed at the beginning of the array, followed by all members from nspace B.

Of course, no process can know the full final membership when constructing the group via the “bootstrap” method, However, we could still use the prior PMIX_GROUP_FINAL_MEMBERSHIP_ORDER to specify the order of participants from different nspaces using wildcard ranks (as described above). If there are other patterns we need to support, we can try to define appropriate attributes for communicating them. It would be the responsibility of the host environment to properly assemble the final membership array based on these new attributes.


I propose that we accept the provisional group changes to the Standard as they currently are, and if folks think this extension worthy of consideration, then we create a new PR that would introduce the new attribute(s).

Thoughts?
Ralph

[Thomas Naughton]

I do not immediately have an opinion on the new attribute, but do support
pushing the current provisional group changes in PR to the Standard.

As mentioned during the meeting, the Provisional status is intended for
just this kind of try, and refine if needed approach.

my $0.02,
--tjn

_________________________________________________________________________
Thomas Naughton naug...@ornl.gov
Research Staff (865) 576-4184

> --
> You received this message because you are subscribed to the Google Groups "pmix-forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pmix-forum+...@googlegroups.com.
> To view this discussion visit https://urldefense.us/v2/url?u=https-3A__groups.google.com_d_msgid_pmix-2Dforum_F594FCF3-2D77F
> 5-2D4435-2DA48E-2DA494F41B5E6B-2540pmix.org&d=DwIFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=WeQv0iup0i3_Si4hSC4cpks
> 7-Efpe2AGJJE4EiqI3K0&m=JucrtWCT0lcADCDO5jcZFP_I7MRd3_ODIsHszaN-uNtmI2pxIUGc2G1g7vm9mSkM&s=nXBUvi6phK9uknSo4ZUA6rxaXczdZ05gAA
> 9DRh7cwd0&e=.
>
>