feat(spectroscopy): Wave 2 — composition → auto-populated model - #84
feat(spectroscopy): Wave 2 — composition → auto-populated model#84CSSFrancis wants to merge 5 commits into
Conversation
Three bugs found by round-tripping a real composition modelI went to start #63 (tabulated EELS edges) and couldn't —
Why the existing tests missed itThe round-trip tests only ever went model → spec, never spec → model with a component of this kind. A round-trip test that only goes one way isn't one. Fixes
198 tests green on this branch. #63 itself is not done — this was the blocker in front of it, not the work. Next up. |
Both shaped their output with `expand_as` / `expand`, which forces the
parameter block's P down to the axis's leading 1 and raises for P > 1:
RuntimeError: The expanded size of the tensor (1) must match the
existing size (4) at non-singleton dimension 0
Plain broadcasting (`p + 0.0 * x`) gives (P, C) for both a shared axis and a
per-position one.
Found by fitting a real EDS composition model, whose background is a
Polynomial — not by the component tests, because every one of them used a
SINGLE spectrum, where expand_as happens to work. That is the actual defect
here: a batched component library whose tests never batch.
Closed with TestEveryComponentBatches, which runs every component (and
Polynomial) with P=4 and asserts the value shape, the gradient shape, and that
row 1 reflects ITS OWN parameters — a broadcasting slip that returns row 0 for
everything would still have the right shape.
Wave 2's headline: type which elements are present, get a populated model. exspy owns the spectroscopy — edge and line energies, GOS edge shapes, family relationships — because reimplementing it would mean diverging from the numbers the community publishes. This module owns the plumbing: - one call for EELS and EDS alike returning a ModelSpec, so the result feeds the batched engine, the wizard and the renderer through one type; - pruning to the measured range. A component with no data under it is NOT harmless: its amplitude is unconstrained, so the optimiser trades it against everything else and degrades the parameters that ARE measurable. exspy will happily add a Cu-L line at 0.93 keV to a spectrum whose useful range starts at 2 keV; - an honest report of whether the batched engine can fit the result. That last point is asymmetric today and worth knowing: EDS -> Polynomial + Gaussian -> fully supported, gets the GPU path NOW EELS -> EELSCLEdge (GOS table) -> no batched port yet (#63), falls back to hyperspy, which is correct but slower Reported in the returned info rather than left as a surprise. Two exspy behaviours that do not do what their names suggest: - `add_lines` APPENDS, so restricting needs `set_lines`. - Even then, the EDS model expands each ELEMENT into its whole family (selecting Fe_Ka still builds Fe_Kb, Fe_La, Fe_Ln, ...) regardless of metadata.Sample.xray_lines — so honouring only_lines means filtering the built spec by name. The source signal is never mutated; a user may be exploring several compositions.
Two bugs of the same shape, found by round-tripping a composition model. to_model() rebuilt every component BARE, which breaks for anything that takes constructor arguments: - EELSCLEdge raises outright (element_subshell is required), so no EELS model could be turned back into hyperspy at all. - Polynomial is worse because it does NOT raise. Rebuilt bare it comes back at its DEFAULT order, so exspy's order-6 EDS background silently lost a3..a6 — to_model skipped the parameters that no longer existed and returned a different model that still fits and still looks plausible. ComponentSpec now carries init_args, captured by from_model via a small _INIT_ARGS table and used when rebuilding. An unrebuildable component gets an error naming the table to add to, rather than a bare TypeError from inside hyperspy. The original round-trip tests missed both because they only ever went model -> spec, never spec -> model with a component of this kind. Fixing that surfaced a third: EELSCLEdge.fine_structure_coeff is a VECTOR parameter with 8 elements, and ParameterSpec stored a single float, so hyperspy rejected the assignment on length. Vector parameters now round-trip faithfully (value becomes a list) but are excluded from the PACKED vector via ComponentSpec.scalar_parameters — the solver puts one scalar per column, and no component with such a parameter is fittable by the batched engine anyway (components.supports already says no). components.py uses the same scalar view so the two cannot drift apart.
…oo (#63) An EELS core-loss edge is not a formula — EELSCLEdge looks its shape up in a GOS table, which is why it had no batched port and why an EELS model fell back to hyperspy's one-pixel-at-a-time fitting while an EDS model (gaussians) got the GPU. The way out is to notice what is actually being fitted. Across a spectrum image the edge SHAPE is the same everywhere — atomic physics, fixed by the element and the microscope. What varies pixel to pixel is HOW MUCH of the element there is and exactly WHERE its edge starts. So the edge is sampled once on the signal axis and replaced by a tabulated component fitting intensity (linear) and onset_shift, batched like everything else. The approximation, stated rather than buried: fine structure and effective angle are FROZEN at the values the table was sampled with. They are what make an edge a lookup in the first place, and refitting them would mean re-sampling the table every iteration — the per-item Python work the batched engine exists to avoid. tabulate_model() reports what it froze. Right trade for quantification; wrong one for fine-structure analysis, which is why it is an explicit call and not automatic. Two deliberate choices in the component: - Outside the table the value HOLDS at the end sample rather than extrapolating. Extrapolating a GOS tail would invent signal where the measurement has none. - The onset-shift derivative is a central difference over one channel, not the exact piecewise-linear slope. The exact one is a step function that jumps at every segment boundary, so LM chatters as the shift crosses a channel. It differs from autodiff by ~1.5% of the gradient's own scale, only within a channel of a kink, and the test states that as the property rather than asserting an equality that is not true. Tested end to end: the table reproduces hyperspy's edge to 1e-9, a fit recovers both a known intensity and a known 12 eV onset shift, and on the synthetic SI the fitted O_K intensity tracks the concentration map the data was built from.
#63 done — EELS now gets the GPU pathAn EELS core-loss edge is not a formula: The way out is noticing what is actually being fitted. Across a spectrum image the edge shape is the same everywhere — atomic physics, fixed by element and microscope. What varies pixel to pixel is how much of the element there is and exactly where its edge starts. So the edge is sampled once on the signal axis and replaced by a tabulated component fitting The approximation, stated rather than buriedFine structure and effective angle are frozen at the values the table was sampled with. They're what make an edge a lookup in the first place, and refitting them would mean re-sampling the table every iteration — the per-item Python work the batched engine exists to avoid.
Two deliberate choices
Verified
218 tests green. I also rebased this branch onto the current #82 tip so the stack doesn't hand you a |
10b4f9c to
3105585
Compare
Two steps, kept separate because they fail differently. element_intensity_maps() is the BRIDGE: fitted parameters -> one intensity map per element, reading the component names the model was built with (Fe_Ka, O_K) and summing a family's lines. Pure array work, no physics, no optional extra. One bridge serves EDS and EELS because the naming shape is the same and the intensity parameter is looked up per component kind (Gaussian's A, TabulatedShape's intensity) — otherwise the EELS path silently produces no maps. quantify() is the PHYSICS: relative, Cliff-Lorimer, or EELS cross-sections. The split matters because the bridge is where a naming or ordering mistake sends iron's intensity into copper's map, and that is SILENT — every downstream number stays plausible. A wrong k-factor is at least a number a microscopist can sanity-check. They deserve separate tests, and the bridge gets more of them. Three things that would otherwise be quiet: - Negative fitted intensities are clamped at SOURCE. A fit can drive a line slightly negative on noise, and a negative "amount of an element" is not just unphysical for that element — it corrupts the normalisation for every other one. - Missing k-factors default to 1.0 and are REPORTED, because defaulting silently makes an uncalibrated number look calibrated. - The result records that composition is RELATIVE to the fitted elements. Cliff-Lorimer cannot know about an element you did not fit, so leaving one out does not cause a small error — it redistributes that element's share across everything else. End to end on the synthetic EDS SI: composition -> model -> batched fit -> quantify recovers the generator's concentration maps at r > 0.9 per element. "Quantification works" means it recovers what we put in, not that the numbers sum to 1.
Closes #62. Part of tracker #48.
Stacked on #82 (needs
ModelSpec+ the engine) — base retargets tomainonce that lands.Elements in, model out
Type which elements are present, get a populated model. exspy owns the spectroscopy — edge and line energies, GOS edge shapes, family relationships — because reimplementing it would mean diverging from the numbers the community publishes. This module owns the plumbing:
ModelSpec, so the result feeds the batched engine, the wizard and the renderer through one type;The asymmetry worth knowing
EELSCLEdge(GOS table)The fallback is correct, just slower. It's reported in the returned
inforather than left as a surprise.Why pruning isn't housekeeping
exspy will happily add a Cu-L line at 0.93 keV to a spectrum whose useful range starts at 2 keV. A component with no data under it is not harmless: its amplitude is unconstrained, so the optimiser trades it against everything else and degrades the parameters that are measurable.
Two exspy behaviours that don't do what their names suggest
add_linesappends — restricting needsset_lines.Fe_Kastill buildsFe_Kb,Fe_La,Fe_Ln, …) regardless ofmetadata.Sample.xray_lines. So honouringonly_linesmeans filtering the built spec by name.A real bug this shook out
PolynomialandOffsetshaped their output withexpand_as/expand, which forces the parameter block'sPdown to the axis's leading1and raises for any batch bigger than one spectrum. Found by fitting a real EDS model, whose background is a Polynomial — not by the component tests, because every one of them used a single spectrum, whereexpand_ashappens to work.That's the actual defect: a batched component library whose tests never batched. Closed with
TestEveryComponentBatches, which runs every component at P=4 and asserts value shape, gradient shape, and that row 1 reflects its own parameters — a broadcasting slip returning row 0 for everything would still have the right shape.The fix commit is included here since this is the first PR that exercises it; it's a fix to #82's code.
Notes
importorskips exspy, so the no-extras CI job skips cleanly rather than erroring. That's the point of the extra.15 composition tests + the batching suite; 147 green on this branch. Still open in Wave 2: #61, #63, #64, #65, #66, #67.