Skip to content

feat(spectroscopy): Wave 2 — composition → auto-populated model - #84

Open
CSSFrancis wants to merge 5 commits into
feat/0.3.0-wave0-wave1from
feat/0.3.0-wave2-spectroscopy
Open

feat(spectroscopy): Wave 2 — composition → auto-populated model#84
CSSFrancis wants to merge 5 commits into
feat/0.3.0-wave0-wave1from
feat/0.3.0-wave2-spectroscopy

Conversation

@CSSFrancis

Copy link
Copy Markdown
Owner

Closes #62. Part of tracker #48.

Stacked on #82 (needs ModelSpec + the engine) — base retargets to main once that lands.

Elements in, model out

Type which elements are present, get a populated model. exspy owns the spectroscopy — edge and line energies, GOS edge shapes, family relationships — because reimplementing it would mean diverging from the numbers the community publishes. This module owns the plumbing:

  • one call for EELS and EDS alike returning a ModelSpec, so the result feeds the batched engine, the wizard and the renderer through one type;
  • pruning to the measured range;
  • an honest report of whether the batched engine can fit the result.

The asymmetry worth knowing

components batched engine
EDS Polynomial + Gaussian fully supported — gets the GPU path now
EELS EELSCLEdge (GOS table) ❌ no batched port yet (#63) → falls back to hyperspy

The fallback is correct, just slower. It's reported in the returned info rather than left as a surprise.

Why pruning isn't housekeeping

exspy will happily add a Cu-L line at 0.93 keV to a spectrum whose useful range starts at 2 keV. A component with no data under it is not harmless: its amplitude is unconstrained, so the optimiser trades it against everything else and degrades the parameters that are measurable.

Two exspy behaviours that don't do what their names suggest

  1. add_lines appends — restricting needs set_lines.
  2. Even then, the EDS model expands each element into its whole family (selecting Fe_Ka still builds Fe_Kb, Fe_La, Fe_Ln, …) regardless of metadata.Sample.xray_lines. So honouring only_lines means filtering the built spec by name.

A real bug this shook out

Polynomial and Offset shaped their output with expand_as/expand, which forces the parameter block's P down to the axis's leading 1 and raises for any batch bigger than one spectrum. Found by fitting a real EDS model, whose background is a Polynomial — not by the component tests, because every one of them used a single spectrum, where expand_as happens to work.

That's the actual defect: a batched component library whose tests never batched. Closed with TestEveryComponentBatches, which runs every component at P=4 and asserts value shape, gradient shape, and that row 1 reflects its own parameters — a broadcasting slip returning row 0 for everything would still have the right shape.

The fix commit is included here since this is the first PR that exercises it; it's a fix to #82's code.

Notes

  • The source signal is never mutated — a user may be exploring several compositions.
  • The whole test module importorskips exspy, so the no-extras CI job skips cleanly rather than erroring. That's the point of the extra.

15 composition tests + the batching suite; 147 green on this branch. Still open in Wave 2: #61, #63, #64, #65, #66, #67.

@CSSFrancis

Copy link
Copy Markdown
Owner Author

Three bugs found by round-tripping a real composition model

I went to start #63 (tabulated EELS edges) and couldn't — ModelSpec.to_model could not rebuild an EELS or EDS model at all. Worth flagging separately because two of the three fail silently.

to_model rebuilt every component bare, which breaks for anything taking constructor arguments:

  1. EELSCLEdge raiseselement_subshell is required. No EELS model could go spec → hyperspy.
  2. Polynomial is worse, because it does not raise. Rebuilt bare it comes back at its default order, so exspy's order-6 EDS background silently lost a3..a6to_model skipped the parameters that no longer existed and returned a different model that still fits and still looks plausible.
  3. Fixing those surfaced a third: EELSCLEdge.fine_structure_coeff is a vector with 8 elements, and ParameterSpec stored a single float, so hyperspy rejected the assignment on length.

Why the existing tests missed it

The round-trip tests only ever went model → spec, never spec → model with a component of this kind. A round-trip test that only goes one way isn't one.

Fixes

  • ComponentSpec.init_args, captured by from_model via a small _INIT_ARGS table. An unrebuildable component now gets an error naming the table to add to, rather than a TypeError from deep inside hyperspy.
  • Vector parameters round-trip faithfully (value becomes a list) but are excluded from the packed vector via ComponentSpec.scalar_parameters — the solver puts one scalar per column, and no component with such a parameter is fittable by the batched engine anyway (components.supports already says no). components.py uses the same scalar view so the two can't drift.

198 tests green on this branch.

#63 itself is not done — this was the blocker in front of it, not the work. Next up.

Both shaped their output with `expand_as` / `expand`, which forces the
parameter block's P down to the axis's leading 1 and raises for P > 1:

    RuntimeError: The expanded size of the tensor (1) must match the
    existing size (4) at non-singleton dimension 0

Plain broadcasting (`p + 0.0 * x`) gives (P, C) for both a shared axis and a
per-position one.

Found by fitting a real EDS composition model, whose background is a
Polynomial — not by the component tests, because every one of them used a
SINGLE spectrum, where expand_as happens to work. That is the actual defect
here: a batched component library whose tests never batch.

Closed with TestEveryComponentBatches, which runs every component (and
Polynomial) with P=4 and asserts the value shape, the gradient shape, and that
row 1 reflects ITS OWN parameters — a broadcasting slip that returns row 0 for
everything would still have the right shape.
Wave 2's headline: type which elements are present, get a populated model.

exspy owns the spectroscopy — edge and line energies, GOS edge shapes, family
relationships — because reimplementing it would mean diverging from the numbers
the community publishes. This module owns the plumbing:

- one call for EELS and EDS alike returning a ModelSpec, so the result feeds the
  batched engine, the wizard and the renderer through one type;
- pruning to the measured range. A component with no data under it is NOT
  harmless: its amplitude is unconstrained, so the optimiser trades it against
  everything else and degrades the parameters that ARE measurable. exspy will
  happily add a Cu-L line at 0.93 keV to a spectrum whose useful range starts
  at 2 keV;
- an honest report of whether the batched engine can fit the result.

That last point is asymmetric today and worth knowing:

  EDS   -> Polynomial + Gaussian  -> fully supported, gets the GPU path NOW
  EELS  -> EELSCLEdge (GOS table) -> no batched port yet (#63), falls back to
                                     hyperspy, which is correct but slower

Reported in the returned info rather than left as a surprise.

Two exspy behaviours that do not do what their names suggest:

- `add_lines` APPENDS, so restricting needs `set_lines`.
- Even then, the EDS model expands each ELEMENT into its whole family
  (selecting Fe_Ka still builds Fe_Kb, Fe_La, Fe_Ln, ...) regardless of
  metadata.Sample.xray_lines — so honouring only_lines means filtering the
  built spec by name.

The source signal is never mutated; a user may be exploring several
compositions.
Two bugs of the same shape, found by round-tripping a composition model.
to_model() rebuilt every component BARE, which breaks for anything that takes
constructor arguments:

- EELSCLEdge raises outright (element_subshell is required), so no EELS model
  could be turned back into hyperspy at all.
- Polynomial is worse because it does NOT raise. Rebuilt bare it comes back at
  its DEFAULT order, so exspy's order-6 EDS background silently lost a3..a6 —
  to_model skipped the parameters that no longer existed and returned a
  different model that still fits and still looks plausible.

ComponentSpec now carries init_args, captured by from_model via a small
_INIT_ARGS table and used when rebuilding. An unrebuildable component gets an
error naming the table to add to, rather than a bare TypeError from inside
hyperspy.

The original round-trip tests missed both because they only ever went
model -> spec, never spec -> model with a component of this kind.

Fixing that surfaced a third: EELSCLEdge.fine_structure_coeff is a VECTOR
parameter with 8 elements, and ParameterSpec stored a single float, so
hyperspy rejected the assignment on length. Vector parameters now round-trip
faithfully (value becomes a list) but are excluded from the PACKED vector via
ComponentSpec.scalar_parameters — the solver puts one scalar per column, and no
component with such a parameter is fittable by the batched engine anyway
(components.supports already says no). components.py uses the same scalar view
so the two cannot drift apart.
…oo (#63)

An EELS core-loss edge is not a formula — EELSCLEdge looks its shape up in a
GOS table, which is why it had no batched port and why an EELS model fell back
to hyperspy's one-pixel-at-a-time fitting while an EDS model (gaussians) got
the GPU.

The way out is to notice what is actually being fitted. Across a spectrum image
the edge SHAPE is the same everywhere — atomic physics, fixed by the element and
the microscope. What varies pixel to pixel is HOW MUCH of the element there is
and exactly WHERE its edge starts. So the edge is sampled once on the signal
axis and replaced by a tabulated component fitting intensity (linear) and
onset_shift, batched like everything else.

The approximation, stated rather than buried: fine structure and effective
angle are FROZEN at the values the table was sampled with. They are what make
an edge a lookup in the first place, and refitting them would mean re-sampling
the table every iteration — the per-item Python work the batched engine exists
to avoid. tabulate_model() reports what it froze. Right trade for
quantification; wrong one for fine-structure analysis, which is why it is an
explicit call and not automatic.

Two deliberate choices in the component:

- Outside the table the value HOLDS at the end sample rather than
  extrapolating. Extrapolating a GOS tail would invent signal where the
  measurement has none.
- The onset-shift derivative is a central difference over one channel, not the
  exact piecewise-linear slope. The exact one is a step function that jumps at
  every segment boundary, so LM chatters as the shift crosses a channel. It
  differs from autodiff by ~1.5% of the gradient's own scale, only within a
  channel of a kink, and the test states that as the property rather than
  asserting an equality that is not true.

Tested end to end: the table reproduces hyperspy's edge to 1e-9, a fit recovers
both a known intensity and a known 12 eV onset shift, and on the synthetic SI
the fitted O_K intensity tracks the concentration map the data was built from.
@CSSFrancis

Copy link
Copy Markdown
Owner Author

#63 done — EELS now gets the GPU path

An EELS core-loss edge is not a formula: EELSCLEdge looks its shape up in a GOS table, which is why it had no batched port and why EELS fell back to hyperspy's one-pixel-at-a-time fitting while EDS (gaussians) got the GPU.

The way out is noticing what is actually being fitted. Across a spectrum image the edge shape is the same everywhere — atomic physics, fixed by element and microscope. What varies pixel to pixel is how much of the element there is and exactly where its edge starts. So the edge is sampled once on the signal axis and replaced by a tabulated component fitting intensity (linear) and onset_shift, batched like everything else.

model_for_composition(...)  ->  engine_supported: False
tabulate_model(...)         ->  engine_supported: True

The approximation, stated rather than buried

Fine structure and effective angle are frozen at the values the table was sampled with. They're what make an edge a lookup in the first place, and refitting them would mean re-sampling the table every iteration — the per-item Python work the batched engine exists to avoid.

tabulate_model() returns what it froze. Right trade for quantification; wrong one for fine-structure analysis — which is why it's an explicit call and not automatic.

Two deliberate choices

  • Outside the table the value holds at the end sample rather than extrapolating. Extrapolating a GOS tail would invent signal where the measurement has none.
  • The onset-shift derivative is a central difference over one channel, not the exact piecewise-linear slope. The exact one is a step function that jumps at every segment boundary, so LM chatters as the shift crosses a channel. It differs from autodiff by ~1.5% of the gradient's own scale, only within a channel of a kink — and the test states that as the property rather than asserting an equality which isn't true.

Verified

  • the table reproduces hyperspy's edge to 1e-9
  • a fit recovers a known intensity, and a known 12 eV onset shift
  • on the synthetic SI the fitted O_K intensity tracks the concentration map the data was built from

218 tests green. I also rebased this branch onto the current #82 tip so the stack doesn't hand you a components.py conflict at merge time — 2-D support and the tabulated component now coexist and are tested together.

@CSSFrancis
CSSFrancis force-pushed the feat/0.3.0-wave2-spectroscopy branch from 10b4f9c to 3105585 Compare July 27, 2026 06:24
Two steps, kept separate because they fail differently.

element_intensity_maps() is the BRIDGE: fitted parameters -> one intensity map
per element, reading the component names the model was built with (Fe_Ka, O_K)
and summing a family's lines. Pure array work, no physics, no optional extra.
One bridge serves EDS and EELS because the naming shape is the same and the
intensity parameter is looked up per component kind (Gaussian's A,
TabulatedShape's intensity) — otherwise the EELS path silently produces no maps.

quantify() is the PHYSICS: relative, Cliff-Lorimer, or EELS cross-sections.

The split matters because the bridge is where a naming or ordering mistake
sends iron's intensity into copper's map, and that is SILENT — every downstream
number stays plausible. A wrong k-factor is at least a number a microscopist
can sanity-check. They deserve separate tests, and the bridge gets more of them.

Three things that would otherwise be quiet:

- Negative fitted intensities are clamped at SOURCE. A fit can drive a line
  slightly negative on noise, and a negative "amount of an element" is not just
  unphysical for that element — it corrupts the normalisation for every other
  one.
- Missing k-factors default to 1.0 and are REPORTED, because defaulting
  silently makes an uncalibrated number look calibrated.
- The result records that composition is RELATIVE to the fitted elements.
  Cliff-Lorimer cannot know about an element you did not fit, so leaving one
  out does not cause a small error — it redistributes that element's share
  across everything else.

End to end on the synthetic EDS SI: composition -> model -> batched fit ->
quantify recovers the generator's concentration maps at r > 0.9 per element.
"Quantification works" means it recovers what we put in, not that the numbers
sum to 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant