.. _group-implementation-label:

Group Operations: Theory and Implementation
===========================================

.. note::

   This is a *developer-level* specification. It describes how the
   ``PMIx_Group_*`` family is realized inside ``libpmix`` — the client-side
   and server-side machinery, the two-phase collective model, and the
   fault-tolerance accounting that keeps a group operation from hanging when
   a participant is lost. The user-facing semantics are documented in
   :ref:`Group Construction, Destruction, and Fault Tolerance
   <group-construction-label>`; the general server-side collective-tracking
   model that the group family reuses is specified in
   :doc:`../collectives/tracking_spec`.

Audience: contributors modifying the group code, the collective-tracking
service, or a host environment's ``pmix_server_module_t.group`` callback.


Two realizations of "group construction"
----------------------------------------

The group family is implemented as **two distinct mechanisms**, chosen per
API rather than per group:

* **Round-trip collectives.** ``PMIx_Group_construct`` and
  ``PMIx_Group_destruct`` are true collectives. Each participant's client
  library packs a command and sends it to its local server with
  ``PMIX_PTL_SEND_RECV``; the server assembles the local participants,
  forwards the aggregated operation to the host for global completion, and
  replies to each local participant. Commands: ``PMIX_GROUP_CONSTRUCT_CMD``
  and ``PMIX_GROUP_DESTRUCT_CMD``.

* **Server-run handshakes.** ``PMIx_Group_invite`` and ``PMIx_Group_join``
  send ``PMIX_GROUP_INVITE_CMD`` and ``PMIX_GROUP_JOIN_CMD`` to the local
  server, which runs the operation: membership expansion, response
  accounting, the timeout, the abort-or-announce decision, the context ID,
  and the completion broadcast. The negotiation still travels as events
  between the participants — there is no host *collective* — but the events
  are raised by each participant's server rather than by the participant.

  That is not a preference: a member's contribution to the group is what it
  has **committed**, and only its own server knows that (see the modex log
  on ``pmix_rank_info_t``). A client assembling its own contribution reads
  its whole store, committed or not, plus data that arrived by other
  routes. So the operation moved to where the right answer is known.

  ``PMIx_Group_leave`` is still realized purely as an event; its former
  ``PMIX_GROUP_LEAVE_CMD`` round-trip was never handled by any server
  switchyard and has been removed.

  A server raising an event on behalf of a client marks it **proxy** on the
  notify caddy. Three things in the event layer key off "am I the source",
  and all three are wrong for such an event: the up-call that carries it
  beyond this node, the discard of an event this process is not a target of
  (which would hide it from the library's own observers), and the skip that
  spares a source the event it generated. See ``src/event/AGENTS.md``.

The code lives in two files:

* ``src/client/pmix_client_group.c`` — all six client API pairs, the client
  group tracker, and the client-side fault handling (invite timeout/decline,
  construct abort, leader-failure watch).
* ``src/server/pmix_server_group.c`` — the server block/tracker structures,
  the local-phase assembly and completion predicate, host handoff, and the
  server-side fault accounting.

Shared membership helpers used by the collectives live in
``src/client/pmix_client_convert.c``.


Client side
-----------

The group tracker
^^^^^^^^^^^^^^^^^

A single caddy type, ``pmix_group_tracker_t`` (in
``pmix_client_group.c``), carries state across the progress-thread boundary
for every API in the family. It is deliberately overloaded — its meaning
shifts with the operation — but always carries the standard caddy machinery
(``ev``, ``lock``, ``status``, ``cbdata``, cached ``cbfunc``/``opcbfunc``)
plus the group identity (``grpid``, ``members``/``nmembers``). For the
event-driven paths it additionally holds the per-member bookkeeping used by
the invite/join handshake (``responded[]``, ``answered[]``, ``nanswered``,
``optional``), the timeout state (``timer_active``), a one-shot resolution
guard (``completed``), and the event-handler registration id (``ref``, seeded
to ``SIZE_MAX`` so teardown can tell whether a handler was ever registered).

Construct and destruct (round-trip)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``PMIx_Group_construct_nb`` does the real work; the blocking form is a thin
wrapper that supplies an internal callback and waits on the caddy lock. The
non-blocking form:

#. **Expands and validates membership.** When the participants are named
   explicitly (not the add-members or bootstrap methods), the ``procs`` array
   is run through ``pmix_client_convert_group_procs()`` so any embedded PMIx
   *group* identifier is replaced by its member processes, and the caller is
   checked for inclusion via ``pmix_client_proc_is_included()``. A caller
   that is not among the expanded participants gets ``PMIX_ERR_NOT_A_MEMBER``
   immediately. Both steps are skipped when ``PMIX_GROUP_ADD_MEMBERS`` or
   ``PMIX_GROUP_BOOTSTRAP`` is present, since those intentionally omit
   participants.

#. **Packs the message** (``construct_msg()``): the command, group ID,
   participant array, and info array, folding in the caller's own endpoint
   data (fetched at ``PMIX_REMOTE`` scope and packed as ``PMIX_PROC_INFO_ARRAY``)
   and prepending the caller's proc ID to any ``PMIX_GROUP_INFO`` array.

#. **Dispatches** with ``PMIX_PTL_SEND_RECV(..., construct_cbfunc, cb)`` and
   returns.

``construct_cbfunc`` unpacks the returned status, final membership, optional
context ID, and group info, registers the completed group in
``pmix_client_globals.groups`` (sorted for a consistent cross-participant
view), and delivers the results. ``PMIx_Group_destruct_nb`` mirrors this with
``PMIX_GROUP_DESTRUCT_CMD``, sending the membership it holds locally (the
server does not store it) and removing the group from the local list on
completion.

The persistent ``pmix_group_t`` also carries the group's failure policy across
the two collectives. ``PMIX_GROUP_NOTIFY_TERMINATION`` is specified at construct
time but governs the eventual destruct, and the server keeps no group state
between operations — so the blocking construct wrapper captures the flag into
the tracker, ``add_group()`` records it in the ``pmix_group_t`` (``notterm``),
and ``PMIx_Group_destruct_nb`` re-attaches ``PMIX_GROUP_NOTIFY_TERMINATION`` to
the destruct request (unless the caller passed it explicitly, which overrides).
The server then reads it off the aggregated destruct block via
``grp_notify_termination()``.

``pmix_client_proc_is_included()`` (in ``pmix_client_convert.c``) is the
shared "is the caller covered by this participant array?" predicate — it
matches the caller's namespace and accepts ``PMIX_RANK_WILDCARD``,
``PMIX_RANK_LOCAL_NODE``, ``PMIX_RANK_LOCAL_PEERS``, or an exact rank. It was
hoisted out of duplicate copies in the fence and connect/disconnect paths so
the collectives share one membership check. ``PMIx_Group_invite`` uses it
too: a process may not form a group it does not belong to, and an invitation
that excludes its own leader is refused with ``PMIX_ERR_NOT_A_MEMBER``
before anything is sent — the detection asked for in openpmix#3850, since
the failure it otherwise produces (the leader waiting out a completion event
addressed to a group it is not in) reports nothing and is very hard to
chase down.

Invite / join / leave (server-run)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``PMIx_Group_invite`` sends ``PMIX_GROUP_INVITE_CMD`` to its server with the
proposed membership, the timeout, and whether the invitation is optional or
wants a context ID. ``PMIx_Group_join`` sends ``PMIX_GROUP_JOIN_CMD`` with
the group id, the leader, and its accept/decline. Neither raises the
negotiation's events itself.

``pmix_server_group_invite()`` builds a ``pmix_server_invite_t``: it expands
wildcard ranks into concrete members (the accounting is by identity, so a
wildcard cannot be matched against an answer), seeds the leader's own slot
and its contribution from that rank's modex log, arms the timeout, registers
an internal observer for ``PMIX_GROUP_INVITE_ACCEPTED``,
``PMIX_GROUP_INVITE_DECLINED`` and ``PMIX_PROC_TERMINATED``, and raises
``PMIX_GROUP_INVITED`` at the invitees. The timer is armed *before* the
observer so every path that can resolve the invitation is downstream of it
and therefore cancels it.

``pmix_server_group_join()`` builds the joiner's contribution from *its*
modex log and raises the accept or decline naming the joiner as source, so
the leader's server sees exactly the event the client used to send.

Each answer runs the observer, which attributes it by identity — a
termination names its subject in ``PMIX_EVENT_AFFECTED_PROC`` rather than
being sourced from it — and records an acceptance along with the
contribution it carried. When every invitee has answered (or the timeout
fires), ``invite_complete()`` either aborts the whole construct, if anyone
failed to join and the invitation was not ``PMIX_GROUP_OPTIONAL``, or
reports each non-accepter to the leader with ``PMIX_GROUP_INVITE_FAILED``
and announces the group. A requested context ID is asked of the host
through ``pmix_host_server.job_control``; that is the one asynchronous step,
and ``invite_broadcast()`` runs from its callback.

There is no equivalent of the client's old wake-then-announce split. That
existed because a client's response handler runs on the progress thread and
cannot call the public ``PMIx_Notify_event`` from there. The server raises
its events through the internal path instead — the rule
``notify_local_members_of_loss()`` follows — so the decision and the
announcement happen in one place.

``PMIx_Group_leave_nb`` is unchanged: it finds the group locally, drops it
from the local list immediately, and generates a ``PMIX_GROUP_LEFT`` event
ranged to the membership excluding self (naming self in
``PMIX_EVENT_AFFECTED_PROC``). Per its contract it returns once the event is
*locally generated*. Remaining members update their local membership when
they receive ``PMIX_GROUP_LEFT``, handled in
``pmix_invoke_local_event_hdlr`` (``src/event/pmix_event_notification.c``)
alongside the existing group-construct-complete membership handling.

The leader-failure watch
^^^^^^^^^^^^^^^^^^^^^^^^^

When a process accepts an invitation, ``setup_leader_watch()`` registers a
private, non-blocking, prepended handler keyed on leader termination
(``PMIX_PROC_TERMINATED``, ``PMIX_ERR_PROC_ABORTED``,
``PMIX_ERR_PROC_TERM_WO_SYNC``) and on construct resolution
(``PMIX_GROUP_CONSTRUCT_COMPLETE`` / ``PMIX_GROUP_CONSTRUCT_ABORT``). If the
watched leader (held in ``members[0]``) is lost before the construct
resolves, the handler emits ``PMIX_GROUP_LEADER_FAILED`` to this process's own
handlers via **non-blocking** ``PMIx_Notify_event`` (the blocking form would
deadlock on the progress thread) and tears itself down. It always returns
``PMIX_EVENT_NO_ACTION_TAKEN`` so it never swallows the events the application
registered to see. Reselection is entirely application-driven: a process
nominates itself by returning ``PMIX_GROUP_LEADER`` in its handler's results,
and the outcome is announced with ``PMIX_GROUP_LEADER_SELECTED``.


Server side
-----------

Block and tracker structures
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Group collectives use a **two-level** tracker (unlike the flat
``pmix_server_trkr_t`` shared by fence/connect/disconnect), because a single
group construct can be assembled from several distinct call signatures
(leader, followers, bootstrap) and carries group-only state:

* ``grp_block_t`` — one per group-ID operation, on the server-global list
  ``pmix_server_globals.grp_collectives`` (declared in
  ``src/server/pmix_server_ops.h``). It owns: the operation kind ``grpop``
  (a ``pmix_group_operation_t``, initialized to ``PMIX_GROUP_NONE``); the
  aggregated membership (``pcs``/``npcs``) and directives (``info``/``ninfo``)
  built by ``aggregate_info``; the list ``mbrs`` of ``grp_trk_t``, **one per
  participant call** (its size is the number of local contributions); the
  list ``departed`` of processes lost before contributing; ``nlocal`` (the
  expected local count) and ``def_complete`` (all participating namespaces
  registered, so ``nlocal`` is final); ``host_called`` (the local phase is
  forwarded and frozen); ``need_cxtid``; and the local-phase timeout state
  (``ev``, ``event_active``).

* ``grp_trk_t`` — one per participant call, hung off a block's ``mbrs``. It
  holds the membership and directives *as passed by that participant*, a
  back-pointer to its block, and ``local_cbs`` — the list of
  ``pmix_server_caddy_t`` that both identifies **who actually contributed**
  (by peer name) and holds the reply destination for each local participant.

* ``grp_shifter_t`` — the thread-shift caddy carrying the host's completion
  result back onto the progress thread.

Local-phase assembly and completion
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``pmix_server_group()`` receives a construct/destruct command on the progress
thread. It requires a host ``group`` callback (else ``PMIX_ERR_NOT_SUPPORTED``),
unpacks the group ID / procs / directives, seeds a default
``PMIX_LOCAL_COLLECTIVE_STATUS = PMIX_SUCCESS`` slot as the last element of
the local info array (the slot the fault machinery later overwrites), and
locates or creates the block via ``get_tracker()``. A call with ``nprocs ==
0`` is a *follower*; a ``PMIX_GROUP_BOOTSTRAP`` call is a *bootstrap* leader —
both keep independent trackers (the leader is unknown) and are forwarded to
the host immediately, one call per participant, without local aggregation.

For the collective (fully-specified) method:

* ``check_definition_complete()`` walks all trackers' member procs, verifies
  every participating namespace is known and fully registered, and — once so
  — sets ``def_complete`` and the final ``nlocal`` (wildcards contribute the
  namespace's local process count). Until every namespace is registered the
  block stays pending, so a client that calls in before its local
  ``register_client`` event cannot complete the collective early.

* ``grp_blk_locally_complete()`` is the **single completion predicate**::

      def_complete  AND  (size(mbrs) + size(departed)) >= nlocal

  This is the group analog of ``pmix_server_trk_complete()`` used by the
  fence family. The ``>=`` (rather than ``==``) tolerates fork/exec clones
  that share a rank and over-count. The local phase is complete once every
  expected local participant has either **contributed** (a tracker on
  ``mbrs``) or **departed**.

* When the predicate holds and the block has not already been forwarded,
  ``aggregate_info()`` merges every tracker's procs/info into the block
  (deduplicating and special-casing ``PMIX_GROUP_ADD_MEMBERS``,
  ``PMIX_GROUP_BOOTSTRAP``, ``PMIX_PROC_INFO_ARRAY``, and ``PMIX_GROUP_INFO``), the
  fault decision below is applied, the local-phase timer is deleted,
  ``host_called`` is set, and the operation is forwarded to
  ``pmix_host_server.group()``.

The host's completion callback ``grpcbfunc`` thread-shifts into
``_grpcbfunc``, which extracts the context ID, final membership, and group
info from the host result, stores group info via
``pmix_server_process_grpinfo``, and — for every block matching the ID
(bootstrap can create several) — packs each tracker's ``local_cbs`` reply and
queues it, then removes and releases the block.


Fault tolerance: identity-based accounting
------------------------------------------

The group family reuses the identity-based collective-tracking model
specified in :doc:`../collectives/tracking_spec`. The essential idea is that
local participation must be tracked **by participant identity**, never by a
bare counter, so the lost-connection handler can answer the one question a
counter cannot: *"has this specific participant already contributed?"* The
completion predicate counts ``contributed ∪ departed`` against ``expected``;
the two lost-participant rules are:

* **Case A — the participant had already contributed.** Ignore the loss for
  this collective. The contribution stands, its data stays in the assembled
  result, and ``nlocal`` is not reduced. The dead peer's queued reply is
  harmless (its socket is closed). This is the rule that fixes the historical
  early-completion / data-loss bug.

* **Case B — the participant had not yet contributed.** It never will, so
  record it on ``departed`` (once, deduplicated). Completion is then
  re-evaluated; if the block is now locally complete it is forwarded exactly
  as on a normal final contribution.

Two entry points reach the same per-block routine ``account_departed()``:

* ``pmix_server_grp_peer_lost(peer)`` — called from ``lost_connection()`` in
  ``src/mca/ptl/base/ptl_base_sendrecv.c`` when a socket drops. A lost
  connection is not scoped to one group, so it applies ``account_departed``
  to **every** block on ``grp_collectives``. (Before this work the
  lost-connection handler walked only the fence-family ``collectives`` list,
  so a lost group member hung the construct forever.)

* ``pmix_server_grp_member_left(grpid, proc)`` — the deliberate cousin. A
  ``PMIx_Group_leave`` generates a ``PMIX_GROUP_LEFT`` event that the
  originating server intercepts in ``src/server/pmix_server_ops.c`` (before
  loop-detection, so only the originating server acts). It applies
  ``account_departed`` to the blocks for the **named** group only.

``account_departed()`` skips bootstrap/follower blocks (``nlocal == 0``) and
already-forwarded blocks (``host_called``); determines membership by matching
``proc`` against each tracker's ``pcs`` with ``PMIX_CHECK_NAMES`` and prior
contribution by matching against each tracker's ``local_cbs`` peer names
(Case A vs. Case B); and, if the loss completes the local phase, aggregates,
applies the fault decision, and forwards. Callers iterate with a
``FOREACH_SAFE`` macro because the block may be released on error.

The abort-vs-survive gate
^^^^^^^^^^^^^^^^^^^^^^^^^^

At each point where a *loss* (or leave) completes a block's local phase, the
server decides between aborting and surviving:

.. code-block:: c

   if (0 < pmix_list_get_size(&blk->departed)) {
       if (PMIX_GROUP_CONSTRUCT == op && !grp_ft_collective(blk)) {
           abort_construct(blk, PMIX_GROUP_CONSTRUCT_ABORT);
           return;
       }
       if (!(PMIX_GROUP_DESTRUCT == op && grp_notify_termination(blk))) {
           pmix_server_set_collective_status(blk->info, blk->ninfo,
                                             PMIX_ERR_LOST_CONNECTION);
       }
   }

* ``grp_ft_collective(blk)`` scans the aggregated directives for
  ``PMIX_GROUP_FT_COLLECTIVE`` (``"pmix.grp.ftcoll"``, bool, default false);
  ``grp_notify_termination(blk)`` scans them for
  ``PMIX_GROUP_NOTIFY_TERMINATION`` (``"pmix.grp.notterm"``, bool, default
  false).
* A **construct without the FT flag** aborts. Every other case survives on the
  survivors. A **destruct with termination notification** completes cleanly:
  the ``PMIX_LOCAL_COLLECTIVE_STATUS`` slot is left at ``PMIX_SUCCESS`` and the
  survivors are told which member was lost via the synthesized
  ``PMIX_GROUP_MEMBER_FAILED`` event (the event standing in place of an error).
  Every other survive path — a fault-tolerant **construct**, or a **destruct
  without notification** — records ``PMIX_ERR_LOST_CONNECTION`` into that slot
  so the host is told the local phase was degraded (clients still receive their
  own status through ``grpcbfunc``).

``abort_construct(blk, status)`` deletes any armed timer; if a host is
present it issues ``pmix_host_server.group(PMIX_GROUP_CANCEL, blk->id, ...)``
with ``cancel_cbfunc``; sets ``host_called``; and completes the local
participants directly by driving ``grpcbfunc`` with the abort status. The
``PMIX_GROUP_CANCEL`` call is essential in the cross-server case: other
servers that already forwarded their local phase are waiting on the host
collective for a contribution this server will never send, so the host must
be told to unstick that collective. Because the block is removed locally, the
host's resulting abort release is a no-op back here — no double completion.
``abort_construct`` is only ever called pre-forward, so no host completion can
race it.

Synthesizing ``PMIX_GROUP_MEMBER_FAILED``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

On the *survive* path, ``notify_local_members_of_loss()`` emits, for each
departed process, a ``PMIX_GROUP_MEMBER_FAILED`` event carrying
``PMIX_EVENT_AFFECTED_PROC`` (the lost member) and ``PMIX_GROUP_ID``. It is
delivered through the internal, progress-thread-safe
``pmix_server_notify_client_of_event`` — never the public ``PMIx_Notify_event``.
Because each server does this accounting for its own local survivors, a member
lost on another node is reported by that node's server performing the identical
logic; the feature therefore needs no cooperation from the host environment.

``_grpcbfunc`` calls it when the operation completed successfully (status is
``PMIX_SUCCESS``), has a non-empty ``departed`` list, and is either a construct
(which reaches ``_grpcbfunc`` with success only on the fault-tolerant survive
path — an aborted construct arrives with ``PMIX_GROUP_CONSTRUCT_ABORT``) or a
destruct for which ``grp_notify_termination()`` reports that
``PMIX_GROUP_NOTIFY_TERMINATION`` was requested. For the destruct survive path
the two host-forward points (``pmix_server_group`` and ``account_departed``)
deliberately leave the local-collective status at ``PMIX_SUCCESS`` when
notification was requested (so the operation returns success, the event standing
in place of an error) and record ``PMIX_ERR_LOST_CONNECTION`` otherwise.

The local-phase timeout
^^^^^^^^^^^^^^^^^^^^^^^^^

A ``PMIX_TIMEOUT`` directive arms a libevent timer over the local phase via
``PMIX_THREADSHIFT_DELAY(blk, group_timeout, tmo)``, setting ``blk->ev`` and
``event_active``. ``group_timeout()`` fires ``abort_construct(blk,
PMIX_ERR_TIMEOUT)``. The timer is deleted before every host-forward (and in
the block destructor ``gbdes`` as an error-path safety net), so it can never
fire on an already-forwarded — and possibly freed — block. As with the fence
family, this timer covers only the *local* phase; the host owns the
cross-server timeout.

The collective-status slot
^^^^^^^^^^^^^^^^^^^^^^^^^^^

``pmix_server_set_collective_status()`` (in ``src/server/pmix_server_fence.c``)
locates the ``PMIX_LOCAL_COLLECTIVE_STATUS`` entry **by key** (not by
position) and writes the status in place; an absent slot is a no-op. Finding
it by key matters for connect, which appends per-participant and job-level
info after the seeded slot — a positional write would corrupt that appended
data and leave the real status stale. The same helper is used by the
fence-family ``lost_connection`` path, both group host-forward points, and
the white-box unit test ``test/unit/collective_status.c``.


The host callback boundary
--------------------------

Group construct/destruct/cancel all cross the PMIx-to-host boundary through
one callback slot, ``pmix_server_module_t.group``, whose operation is a
``pmix_group_operation_t``:

.. code-block:: c

   typedef enum {
       PMIX_GROUP_CONSTRUCT,   /* 0 */
       PMIX_GROUP_DESTRUCT,    /* 1 */
       PMIX_GROUP_NONE,        /* 2 */
       PMIX_GROUP_CANCEL       /* 3 */
   } pmix_group_operation_t;

Because the enum's numeric values cross that boundary, new values **must** be
appended, never inserted — ``PMIX_GROUP_CANCEL`` was added at the end.
``PMIx_Group_operation_string()`` (in ``src/common/pmix_strings.c``)
stringifies each value. Two footguns to respect when editing this enum:

* Keep the body **free of comment lines**: the Python bindings generator
  ``bindings/python/construct.py`` treats every line inside a ``typedef
  enum`` as an enumerator, so an interleaved comment shifts every subsequent
  value. The explanatory comment therefore lives in a block above the
  ``typedef``.
* ``PMIX_GROUP_CANCEL`` is *not* carried on the client-to-server wire (it is
  purely a PMIx-to-host callback op), so no new ``bfrops`` support is needed.

Capability detection
^^^^^^^^^^^^^^^^^^^^^

The whole feature set above is advertised by the ``PMIX_CAP_GROUP_FT``
capability flag in ``include/pmix_version.h.in``. Its *presence* in the
installed ``pmix_version.h`` is the signal a consumer (e.g. PRRTE, via
``PRTE_CHECK_PMIX_CAP``) tests for; absence means the running PMIx predates
group fault tolerance. Edit the ``.in`` template, never the generated header.


Event and attribute reference
-----------------------------

Events (defined in ``include/pmix_common.h.in``):

============================================  ======  ================================================================
Event                                         Value   Role
============================================  ======  ================================================================
``PMIX_GROUP_INVITED``                        -159    Leader → invitees; carries ``PMIX_GROUP_ID``
``PMIX_GROUP_LEFT``                           -160    Departing proc → members; server intercepts to account the leave
``PMIX_GROUP_INVITE_ACCEPTED``                -161    Invitee → leader (``PMIx_Group_join`` accept)
``PMIX_GROUP_INVITE_DECLINED``                -162    Invitee → leader (``PMIx_Group_join`` decline)
``PMIX_GROUP_INVITE_FAILED``                  -163    Leader-local report of a non-accepter
``PMIX_GROUP_MEMBERSHIP_UPDATE``              -164    Membership change notification
``PMIX_GROUP_CONSTRUCT_ABORT``                -165    Construct aborted; all participants notified
``PMIX_GROUP_CONSTRUCT_COMPLETE``             -166    Group formed; carries final ``PMIX_GROUP_MEMBERSHIP``
``PMIX_GROUP_LEADER_SELECTED``                -167    App-driven reselection result
``PMIX_GROUP_LEADER_FAILED``                  -168    Leader lost; surfaced to a joining member's handlers
``PMIX_GROUP_MEMBER_FAILED``                  -170    Server-synthesized on a lost member (survive path)
============================================  ======  ================================================================

Key attributes / status codes:

* ``PMIX_GROUP_FT_COLLECTIVE`` (``"pmix.grp.ftcoll"``, bool) — the
  survive-vs-abort gate for a lost member during construct.
* ``PMIX_GROUP_OPTIONAL`` (bool) — all-or-nothing (default) vs. reduced group
  for the invite method.
* ``PMIX_GROUP_NOTIFY_TERMINATION`` (bool) — report vs. error a member lost
  mid-destruct.
* ``PMIX_GROUP_LEADER`` (bool) — declares the construct leader; also returned
  from a leader-failed handler to nominate a replacement.
* ``PMIX_LOCAL_COLLECTIVE_STATUS`` (``pmix_status_t``) — the per-tracker
  health slot the fault machinery updates by key.
* ``PMIX_ERR_NOT_A_MEMBER`` — returned when the caller is not among the
  expanded construct participants.
* ``PMIX_ERR_TIMEOUT`` — local-phase timeout expiry.
* ``PMIX_ERR_LOST_CONNECTION`` / ``PMIX_ERR_PARTIAL_SUCCESS`` — degraded
  local-collective status recorded for the host.


Testing
-------

The behaviors above are exercised at several levels:

* ``test/unit/collective_status.c`` — white-box coverage of the by-key status
  helper (connect/fence layouts, absent slot, degenerate inputs), wired into
  ``make check``.
* ``test/simple`` drivers — ``simpgrpdie`` (construct survives loss +
  ``PMIX_GROUP_MEMBER_FAILED``, with ``PMIX_GROUP_FT_COLLECTIVE``),
  ``simpgrpddie`` (destruct survives loss +
  ``PMIX_GROUP_MEMBER_FAILED`` and returns success, with
  ``PMIX_GROUP_NOTIFY_TERMINATION``), ``simpgrpcabort`` (default abort with
  ``PMIX_GROUP_CONSTRUCT_ABORT``), and ``simpgrpctimeout``
  (``PMIX_ERR_TIMEOUT`` on a hung live member).
* ``examples/`` exercisers driven by the dockerswarm harness
  (``contrib/dockerswarm/run-group-events.sh``) across a multi-node swarm, so
  the cross-server notification and cancel paths actually run:
  ``group_invite`` (happy path), ``group_invite_timeout`` /
  ``group_invite_decline`` / ``group_invite_abort`` (invite fault paths),
  ``group_destruct_die`` and ``group_die`` (loss mid-destruct/construct),
  ``group_construct_abort`` (lost member forces ``PMIX_GROUP_CANCEL`` to
  unstick remote servers), ``group_daemon_fail`` (whole-daemon loss), and
  ``group_leave``.
