Speed up connect_the_chunks for large numbers of chunks

[jin-castle]

fields::connect_the_chunks() tests every not-owned boundary point of every
chunk against every other chunk, on every process. The expensive point scan
therefore grows as O(num_chunks² × boundary points), and workflows that rebuild
fields repeatedly pay this cost once per iteration.

This PR conservatively prunes candidate chunk pairs before the point scan.
The connection tables and per-point owns() checks are unchanged.

What changed

For the common case with no periodic boundaries or symmetry transforms,
chunk_candidates[i] is precomputed by intersecting chunk i's existing
one-grid-point-padded grid_volume with the other chunk volumes. Both point
scans then iterate only over those candidates.

Periodic or symmetry-reduced simulations use the original all-pairs scan.
This avoids adding new periodic-shift or symmetry-transform logic to the
already complicated boundary-connection code.

chunk_needed[i] also skips source chunks whose candidate pairs are all
remote-to-remote on the current process. The loop body previously rejected all
of those pairs individually with the same process-locality condition.

Communication-buffer allocation and lifetime are unchanged from master.

Why the pruning is conservative

Without periodic wrapping or symmetry transforms,
locate_component_point() either leaves a boundary point at the same location
or rejects it. grid_volume::pad() contains the one-cell not-owned halo, so if
that padded volume does not intersect chunk j, chunk j cannot own any point
from the scan. The existing per-point owns() test remains the final check, so
false candidates only cost extra comparisons.

For periodic or symmetry-reduced cases no pair is pruned, so their behavior is
the original behavior.

Measurements

Measured on a dual EPYC 9554 node using the 8×5×4 µm, resolution-24 3D PML
workload (approximately 2.2M voxels). Values are medians of three interleaved
runs with OMP_NUM_THREADS=1 and one MPI rank bound per physical core. The
metric is the maximum-rank wall time for the first fields::step() after
init_sim().

MPI ranks	actual chunks	master	patched	speedup
64	240	12.136 s	0.632 s	19.2×
128	360	21.506 s	0.786 s	27.4×

Serial forced-chunk measurements at resolution 16:

requested chunks	actual chunks	master	patched	speedup
16	96	4.524 s	1.939 s	2.33×
48	208	11.581 s	2.754 s	4.21×
96	312	19.734 s	3.420 s	5.77×
192	480	35.921 s	4.722 s	7.61×

The numerical Ez probe was exactly identical across all variants and repeats.
The pair enumeration remains O(num_chunks²), but the surviving box tests are
much cheaper than repeating the boundary-point scan for every chunk pair.

These measurements are from a separate, explicitly core-bound campaign and
should not be compared directly with absolute timings from the earlier
unbound campaign.

[stevengj]

This is a reasonable sort of optimization at first glance, but I'm a little worried about anything that increases the complexity of the code that computes the boundary connections, which is already extremely complicated.

How much speedup do real applications get from optimizing this code, which usually only executes once at the beginning of the simulation?

[jin-castle]

That is the right metric. For a conventional long, one-shot FDTD run, this only removes a fixed startup cost, so the end-to-end benefit may be small.

The motivating real application was a 2.2M-voxel 3D adjoint-optimization problem that reconstructs the fields once per iteration. The steady-state timings on a dual EPYC-9554 node were:

MPI ranks	iteration before	connection setup before	fraction of iteration	iteration after	connection setup after
64	35.2 s	12.3 s	35%	21.3 s	~1.7 s
128	49.9 s	40.9 s	82%	25.2 s	~2.5 s

Thus, in this repeated-initialization workload, connection construction accounted for 35–82% of the iteration time before the patch and approximately 8–10% afterward. The deployed build reduced the complete iteration time by 40% at 64 ranks and 50% at 128 ranks.

The patched-build iteration totals also include automatic caching of the adjoint eigenmode source, so I do not attribute the entire end-to-end reduction to this PR. The connection-setup timings are measured separately and isolate the change in this patch.

I agree that the tradeoff is much less compelling for long one-shot simulations. Given the maintenance concern, I can simplify this patch to retain only the conservative candidate-pair pruning if preferred.

[stevengj]

Go ahead and see if you can simplify the patch.

[jin-castle]

I tested two reduced versions before updating the PR.

The first retained only candidate-pair pruning. That preserved most of the
serial speedup, but it gave up a substantial part of the MPI improvement
because every process still scanned boundary points for chunks whose candidate
pairs were all remote-to-remote.

I then retained the small chunk_needed process-local skip while removing the
more complicated periodic/symmetry bounding-box logic and restoring the
original communication-buffer allocation. Periodic or symmetry-reduced cases
now fall back to the original all-pairs scan. The resulting diff is +25/-2
lines, down from +88/-10.

I rebuilt all variants from the same e7d46f2 baseline and measured the
2.2M-voxel 3D PML workload on a dual EPYC 9554 node. These are medians of three
interleaved runs with OMP_NUM_THREADS=1 and one MPI rank bound per physical
core. The metric is the maximum-rank wall time for the first fields::step()
after init_sim().

ranks	actual chunks	baseline	previous full patch	pruning only	reduced patch
64	240	12.136 s	0.640 s	2.955 s	0.632 s
128	360	21.506 s	0.903 s	4.204 s	0.786 s

Thus, candidate pruning alone is still 4–5× faster than the baseline, but the
process-local chunk skip is important for MPI. Retaining that skip while
removing the more complicated candidate construction and sparse allocation
preserved the full patch's setup-time improvement.

In the serial forced-chunk benchmark, candidate pruning alone was within 4% of
the full patch over 96–480 actual chunks and reduced setup time by 2.3–7.6×
relative to the baseline. This indicates that sparse buffer allocation is not
needed for the measured setup-time speedup.

The numerical Ez probe was exactly identical across all variants and repeats.
For periodic or symmetry-reduced simulations, the reduced patch uses the
original all-pairs path, so those cases no longer rely on a new bounding-box
argument.

These absolute timings are from a separate, explicitly core-bound server8
campaign and should not be compared directly with the earlier 12.3/40.9 s
figures; the relevant comparison is between variants within this table.

I have force-pushed the reduced version and updated the PR description.