Improve adjoint test hygiene

[smartalecH]

The tests we have for the adjoint solver (unit tests, end2tests, etc) are in a rough state. Here are some things we can do to improve things:

• Document which end2end cases we have as a matrix (e.g. objective quantity vs simulation dimensionality vs design grid dimensionality vs boundary condition). Use this to pick up on gaps.
• Consolidate end2end tests to a single driver that loop over different simulation/optimization setups. This makes it easier to audit the code. The above test matrix makes this really intuitive.
• Establish a unified tolerance for all single precision tests and all double precision tests (which, of course, are different from each other). Over the years, these tolerances have drifted higher and higher with the addition of new features as we continued to introduce more and more bugs.
• Clearly outline all the unit tests we have for the python front-end vs the C++ backend (in particular, the recombination step in meepgeom.cpp).

[jin-castle]

This is very much aligned with what I've been running into while working on
#3263/#3274 — happy to help here.

One suggestion for the consolidation step: if the unified driver emits a small
structured record per case (objective, simulation dim, design-grid dim,
boundary, precision, measured error, tolerance), the test matrix in point 1
stops being documentation and becomes a CI output. From that same record, CI
can render:

• the coverage matrix (tested / passing / missing combinations);
• measured error vs. allowed tolerance per case, which makes tolerance drift
  visible — passes that consume >50% of their tolerance are exactly the ones
  worth auditing before unifying thresholds;
• for failed or near-tolerance cases, a spatial residual map over the design
  grid (the chunk-partition bug in [1/2] Fix adjoint gradient chunk-dependence without gathering the DFT monitors #3274 shows up as a block pattern at chunk
  boundaries, which is far more diagnostic than a scalar L2 error).

Mockup of what this could look like (illustrative data):

Recording the measured error per case also forces the refactor that point 3
needs anyway — most current tests assert deep inside and never expose the
actual error.

Assertions would remain the sole source of pass/fail; reporting would be
opt-in via an environment variable so local make check-adjoint is unchanged,
and matplotlib is already in python/requirements.txt so there are no new
dependencies. A first PR could be just the record schema + a GitHub Actions
job summary, with the plots as a follow-up.

Do you already have a shape in mind for the unified driver? I'd rather fit
the record schema to that than design it separately.

[smartalecH]

Thanks! I think a visual dashboard like what you describe above would be fantastic. Some challenges we might hit (would like to hear your thoughts wrt how we could tackle them):

• Is there a clean way to integrate this within our CI? How can we access a dashboard like this with the current setup?
• Adjoint tests are really expensive (especially the 3D) and we don't run them all by default in CI. Is there a way we can optionally trigger a full "dashboard" run on a PR or branch (e.g. right before we merge)?