Performance: hierarchical barrier, tiered huge pages, and OFI/CXI transport improvements#1237
Draft
bcmIntc wants to merge 14 commits into
Draft
Performance: hierarchical barrier, tiered huge pages, and OFI/CXI transport improvements#1237bcmIntc wants to merge 14 commits into
bcmIntc wants to merge 14 commits into
Conversation
Do not force hard polling when XPMEM is enabled. This conflicts with counter based polling when used with OFI transport. Issue Sandia-OpenSHMEM#1217 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
Replaced complex polling with simpler OFI counter wait Issue Sandia-OpenSHMEM#1217 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
Issue Sandia-OpenSHMEM#1221 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
Let the provider advertise its natural inject_size rather than requiring it to be at least sizeof(long double). The returned inject_size is adopted immediately after fi_getinfo.
Adds --enable-hierarchical-barrier, a three-phase barrier that keeps intranode traffic off the NIC by using CPU atomics over XPMEM for gather/fanout and restricts NIC puts to the internode phase (node roots only). Phase 1 (intranode gather): local PEs signal up a k-ary tree. Each PE writes to its OWN up-slot in local_pSync; the parent reads each child's slot individually. Slots are padded to one cache line (HIER_SLOT_STRIDE=8 longs, 64 bytes) so no two PEs share a line, eliminating the MESI serialization that would occur if all children wrote to a single counter. Signal values increment monotonically via hier_sense, avoiding explicit slot resets between calls (sense alternation). Phase 2 (internode dissemination): node roots run a put-based binary dissemination across the NIC. After each round the slot is reset via a CPU store rather than a self-put, saving ceil(log2(N_nodes)) NIC round-trips per barrier (12 at 4096 nodes). Phase 3 (intranode fanout): node root CPU-stores an ack into each child's down-slot; children relay down the k-ary tree with reset-before-signal ordering. Down-slots are in the upper half of local_pSync, laid out with the same per-PE cache-line padding as up-slots. AUTO selection activates when local PE count >= SHMEM_HIER_BARRIER_THRESHOLD (default 2). Also selectable via SHMEM_BARRIER_ALGORITHM=hierarchical. New infrastructure: - src/shr_transport.h — XPMEM CPU pointer mapping; self-access returns the address directly without an XPMEM lookup - src/runtime_util.c — global hostname exchange so each PE can identify its node root - configure.ac — --enable-hierarchical-barrier requires --with-xpmem and a network transport
…archical barrier Three coupled refinements to the hierarchical barrier: Per-team sense state. A static global hier_sense caused a signal mismatch when PEs participated in different barrier teams: only PEs in the active set incremented it, so PEs that skipped a subset barrier carried a stale sense value into the next TEAM_WORLD barrier. Move hier_sense into shmem_internal_team_t; TEAM_WORLD barriers (PE_start=0, PE_stride=1, PE_size=num_pes) use team-local state, and subset barriers use a static fallback. All PEs in TEAM_WORLD now stay synchronized across interleaved team/subset barrier sequences. Persistent active-set scratch. The barrier built its local_pes/root_pes arrays fresh on every call. An alloca-per-call risked stack overflow at scale (PE_size up to 100K+ is ~400KB per array), while malloc/free per-call serialized the hot path and tanked throughput. Allocate both once in shmem_internal_collectives_init, sized to num_pes, and reuse. Safe to share across calls because barriers are serialized -- a PE cannot enter a new barrier until it has exited the previous one, the same invariant the shared hierarchical_local_psync relies on. Dead-code removal. Drop three unused symbols that produced -Wunused warnings: shmem_internal_cpu_atomic_load_long (never called), tree_parent_shr (computed but never read -- the Phase 1 up signal writes to the PE's own up-slot, which the parent reads directly), and my_up_slot (only the down-slot is read in the spin loops; my_up_raw is still used for the parent store).
Cray SHMEM calls FI_CXI_DOM_OPS_3 enable_hybrid_mr_desc right after fi_domain(); SOS does not. Without it, the CXI provider performs internal memory registration on every fi_write/fi_writemsg/fi_fetch_ atomicmsg call where the desc field is non-NULL, even when the buffer already lives in a provider-registered region. Each MR cache lookup adds latency to large RDMA puts. With hybrid mode enabled, the provider trusts a non-NULL desc and skips its internal registration walk, dropping per-call overhead on the put data path. The call must occur before any endpoints are created; the domain only propagates the setting to child endpoints at creation time. Reintroduces the provider-name infrastructure to gate this: the provider name is captured via strdup in query_for_fabric (it must outlive fi_freeinfo) and freed in shmem_transport_fini. An inline shmem_transport_ofi_check_provider helper does an exact strcmp so that the call is attempted only on a plain cxi provider and layered providers do not inherit it. The dom_ops struct fields are mirrored locally so the build does not require rdma/fi_cxi_ext.h. Gated by SHMEM_OFI_CXI_HYBRID_MR_DESC=true (default) for A/B testing without rebuilding.
Reintroduce the symmetric heap huge page allocator gated on
SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES (default off). When enabled, the
heap is backed by explicitly reserved huge pages via two paths, tried
in order:
1. hugetlbfs file mapping (when a matching hugetlbfs mount exists)
2. anonymous MAP_HUGETLB with an explicit 2MB page size (works with
nr_overcommit_hugepages)
When neither reserved-huge-page path succeeds, the heap falls back to
regular pages. There is intentionally no transparent-huge-page
(madvise) tier: only explicitly reserved huge pages are used, otherwise
regular pages.
The regular-page fallback is a single fixed-address mmap at
requested_base, identical to the mapping used when huge pages are
disabled, so the symmetric heap keeps the same virtual address on every
PE whether or not the huge-page flag is set.
mmap_alloc now reports the actual mapped size back to the caller, since
the hugetlbfs path rounds the length up to a huge-page boundary; that
rounded size is used for munmap and transport registration.
Disabled by default, so the regular-page path is byte-for-byte the
prior behavior unless the flag is set.
…chical barrier Give the hierarchical barrier a per-team sense counter to eliminate the overlapping-team deadlock. shmem_internal_sync_hierarchical gains a hier_sense_ptr parameter: non-NULL supplies a caller-owned per-team counter; NULL retains the prior selection (TEAM_WORLD uses its own counter, other active sets share a static fallback). New shmem_internal_sync_for_team / shmem_internal_barrier_for_team wrappers pass &team->hier_sense, and shmem_team_sync, the team psync-recycle barrier, and the two team-split barriers route through them. sync_all / barrier_all pass &team_world.hier_sense explicitly. Add shmem_internal_sync_no_hier for the deprecated shmem_barrier / shmem_sync APIs: they carry no team context, so they never select the hierarchical algorithm and fall through to linear/tree by COLL_CROSSOVER. Build root active sets from a global node_id array instead of is_node_root. shmem_runtime_get_node_id(pe) returns the global PE number of the lowest-ranked PE on pe's node, computed once at init across all runtimes (mpi, pmi, pmi2, pmix). The root representative for each node is the lowest-ranked active PE on it (local_pes[0]), and the degenerate-case guard is reduced to local_count == PE_size.
Each hierarchical barrier rebuilt the active-set topology from scratch: the local PE set, the per-node root set, virtual indices, tree children (in shr-rank space), and the phase-2 dissemination round count. For a fixed team this is all invariant, yet it ran on every call -- including an O(PE_size^2) root-set scan, an O(PE_size) local scan, and a shmem_runtime_get_node_rank lookup per tree child. Cache it per team. Add shmem_internal_hier_cache_t to the team struct, built once on the first barrier (shmem_internal_hier_compute) and reused thereafter. shmem_internal_sync_hierarchical takes a hier_cache pointer: team-aware callers (sync_all, barrier_all, sync_for_team) pass &team->hier_cache; callers with no team context pass NULL and build into the shared per-call scratch as before. After the first call, per-barrier topology cost drops to a few field loads. Only topology is cached, not XPMEM mapped pointers: those depend on the local_pSync array (sync_all and barrier_all share a team but pass different arrays) and are cheap base+offset arithmetic to re-derive. Also make build_root_active_set O(PE_size) via a node_id-keyed seen-flag buffer (hier_node_seen) instead of the nested O(PE_size^2) scan, so even the one-time first-barrier build is linear at scale. The buffer is self-cleaning (resets only touched entries) to avoid an O(PE_size) clear per call. Cache persistence is allocation-failure safe: if any of the three exact-size buffers fails to allocate, the barrier falls back to the scratch topology for that call without persisting (valid stays 0), so it still completes correctly and retries the build next time. The topology cache is zeroed for the three predefined teams (world, shared, node; split teams are calloc'd) and freed in shmem_internal_team_destroy. configure.ac: --enable-hierarchical-barrier now requires XPMEM specifically, not "XPMEM or CMA". The intranode phase does CPU atomics on peers' slots through shmem_shr_transport_ptr, which only XPMEM provides; a CMA-only build would have configured successfully then failed at the first barrier.
… min PPN AUTO barrier selection chose the hierarchical algorithm whenever the local node had >= HIER_BARRIER_THRESHOLD PEs. Two problems: 1. On a single-node job the hierarchical barrier runs its degenerate intranode-only path, which is strictly worse than linear/tree: under USE_HIERARCHICAL_BARRIER the shr transport already routes single-node atomics through CPU atomics (shmem_shr_transport_use_atomic), so linear/tree run over shared memory with none of the hierarchical scaffolding. Now require the job to span more than one node (shr_size < num_pes) before AUTO selects hierarchical. 2. The threshold was checked against the local node size, which differs per PE on a heterogeneous-PPN job -- a large node could select hierarchical while a small node selects tree, diverging within one collective. Gate instead on a job-global minimum PEs-per-node, computed once in collectives_init from the global node_id array (already exchanged at init, no new communication): tally PEs per node, take the min over populated nodes. Both terms of the predicate are now job-global, so every PE makes the same selection. Consolidated into shmem_internal_hier_barrier_available(), used at all AUTO selection sites (sync, sync_all, barrier_all, sync_for_team) and the init-time barrier-algorithm diagnostic. Explicit BARRIER_ALGORITHM=hierarchical is unaffected and always honored.
bcmIntc
force-pushed
the
bcm_Performance
branch
2 times, most recently
from
June 26, 2026 11:56
a317318 to
eb627bf
Compare
configure checks for rdma/fi_cxi_ext.h and, when present, probes struct fi_cxi_dom_ops for the enable_hybrid_mr_desc member (added in a later libfabric). The hybrid local MR descriptor optimization is compiled in only when both HAVE_RDMA_FI_CXI_EXT_H and HAVE_STRUCT_FI_CXI_DOM_OPS_ENABLE_HYBRID_MR_DESC are defined, using the provider's real struct fi_cxi_dom_ops and the FI_CXI_DOM_OPS_3 macro. There is no hand-rolled struct mirror, so the build never risks an ABI mismatch against the provider's actual dom_ops layout. When support is absent and the runtime requests hybrid mode, PE 0 warns once and the request is ignored.
Collaborator
Author
Test config:--with-ofi=/opt/cray/libfabric/1.22.0/ --with-xpmem=/usr/lib --enable-pmi-mpi --disable-fortran --enable-ofi-mr=scalable --enable-ofi-manual-progress --disable-libtool-wrapper --enable-mr-endpoint --enable-manual-progress --enable-hierarchical-barrier CC=mpicc CXX=mpicxx Envt:export SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES=1 |
fi_cxi_ext.h references struct fid / uint64_t / bool / struct timespec without including the headers that define them. Whether those resolve during the configure probe depends on what a given libfabric's fabric.h pulls in transitively, which varies across clusters and versions, so the header could be present yet reported as unusable and silently disable the hybrid MR descriptor optimization. Supply the full prerequisite set (stdbool.h, time.h, stdint.h, rdma/fabric.h, rdma/fi_domain.h) in the AC_CHECK_HEADERS/AC_CHECK_MEMBER test bodies so the probe compiles independently of fabric.h's transitive includes. Also report CXI hybrid MR desc support in the configure summary so a misdetection is visible at configure time rather than only as a runtime warning on CXI hardware. Also lower the huge-page snprintf-failure message in symmetric_heap_c.c from a warning to a debug message, matching the other fallback-path messages in mmap_alloc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a NIC-offloading hierarchical barrier and reintroduces two CXI/heap optimizations in isolation, each independently gated.
Hierarchical barrier (XPMEM + NIC dissemination)
CXI hybrid MR descriptor (9eab54d, 3ba9445, 2cc39b2)
Enables FI_CXI_DOM_OPS_3 enable_hybrid_mr_desc after fi_domain() (as Cray SHMEM does) so the provider trusts a non-NULL desc and skips its internal MR-registration walk on large puts. Gated by SHMEM_OFI_CXI_HYBRID_MR_DESC and an exact-strcmp provider check. Compiled in only when a configure probe finds the real struct fi_cxi_dom_ops member (no hand-rolled struct mirror → no ABI-drift risk); the probe supplies a full prerequisite-include set so it's robust across libfabric versions/clusters, with a configure-summary line surfacing the result.
Symmetric heap huge pages (5285889)
Reintroduces tiered huge-page allocation gated on SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES (default off): hugetlbfs file → anonymous MAP_HUGETLB → regular pages. No THP/madvise tier. Fallback path is byte-for-byte the prior fixed-address mapping, preserving VA symmetry across PEs.
Misc #1236
Notes:
CXI hybrid MR and huge pages are both off/neutral by default. All optimizations are individually gated for safe A/B testing.