Post processing EBSD data - #8
Conversation
Explains why EBSD acquisition parameters take the values recorded in standards (ASTM E2627 / ISO 13067), grounded in backscatter physics and sampling statistics rather than deferring to the standards themselves. Addresses issue #2. Co-authored-by: Ronnie Guymon <244881888+ronnie-guymon@users.noreply.github.com>
|
@claude what are the methods of EBSD post processing data used most commonly by research professionals? Are there any where I can upload the raw patterns, then constantly tweak parameters in the software to get the best grain characterization possible? |
|
Claude Code is working… I'll analyze this and get back to you. |
|
@claude what are the methods of EBSD post processing data used most commonly by research professionals? Are there any where I can upload the raw patterns, then constantly tweak parameters in the software to get the best grain characterization possible? |
|
Claude finished @ronnie-guymon's task in 1m 34s —— View job EBSD Post-Processing — Common Methods & Iterative-Tweaking ToolsShort answer to your second question: yes — this workflow exists and it's exactly where the field is heading. It's called pattern reprocessing / re-indexing, and it requires that you save the raw EBSPs (diffraction patterns) during acquisition, not just the indexed orientation map. Once you have the raw patterns on disk, you can re-run indexing and grain reconstruction as many times as you like, tweaking parameters each pass, without ever going back to the microscope. 1. The two stages of EBSD post-processingProfessionals treat post-processing as two distinct stages. It helps to keep them separate because your "tweak parameters" question lives mostly in Stage A. Stage A — Indexing (raw pattern → crystal orientation). Three approaches dominate:
Stage B — Microstructure analysis (orientation map → grains, texture, GND, etc.): grain reconstruction (misorientation-threshold flood-fill, typically ~5°), clean-up (neighbour-orientation correlation, grain dilation, CI/MAD/band-contrast thresholding), grain-size statistics, texture/ODF, KAM/GND, phase maps. This is the downstream analysis your existing 2. "Upload raw patterns, then constantly tweak parameters" — the tools that do thisThis is the re-indexing / pattern-matching workflow. Prerequisite: acquire with "save patterns" enabled (Oxford Commercial (GUI, interactive tweaking):
Open-source (scriptable — best fit for a Python project like CALIBER):
3. Recommendation for CALIBERGiven CALIBER is Python and the goal is sample-specific parameter recommendation with an iterate-to-best loop, the strongest foundation is:
This maps cleanly onto CALIBER's premise: keep raw patterns, then let the software sweep the trade-off curves documented in Happy to draft a short Sources: AZtecCrystal 4.0 · kikuchipy (pattern matching) · PyEBSDIndex · EMsoft · EMSphInx · MTEX grain reconstruction · DI hands-on tutorial
This was a question, so no code changes were made. |
@claude it looks like this job failed, if anything is still running for this job stop it. |
|
Claude finished @ronnie-guymon's task in 1m 33s —— View job EBSD post-processing: common methods & re-indexable "tweak-the-parameters" softwareOn the failed job: nothing is still running that I can stop. GitHub Actions runs are ephemeral — the two earlier runs ( Short answer to your key question: yes — the capability you're describing (upload the raw patterns, then keep re-tuning parameters until grain characterization looks right) is a real, well-established workflow. The enabling requirement is that the raw Kikuchi patterns (EBSPs) are saved to disk (usually an HDF5 The most common post-processing methods (in workflow order)
Software that lets you re-index stored patterns and iterateCommercial (pattern-matching re-indexing built in):
Open-source (fully scriptable — best fit for CALIBER):
Recommendation for CALIBERThe key architectural choice is to save raw patterns during acquisition (HDF5), which decouples indexing from the microscope and makes every downstream parameter tunable and re-runnable. For an automated, parameter-optimizing tool the strongest foundation is the open-source Python stack — kikuchipy (dictionary/hybrid indexing) + EMsoft (master patterns) + MTEX or orix (grain/texture analysis) — because it's fully scriptable, so CALIBER can programmatically sweep indexing/clean-up/reconstruction parameters and score grain-characterization quality automatically. AZtecCrystal/MapSweeper is the best turnkey GUI equivalent if you'd rather tune interactively before automating. I've left the committed doc as-is (it covers acquisition parameter selection); if you'd like, I can add a companion Sources: kikuchipy pattern matching · kikuchipy hybrid indexing · A Dictionary Approach to EBSD Indexing (arXiv) · EMsoft DI wiki · AZtecCrystal · MapSweeper pattern matching · Parent grain reconstruction · EBSD platform considerations
|
|
@sgbaird is there software you have used or are aware of that I should look into using for our own post processing? If not that's ok, I'll just experiment and find one. |
@claude I have raw patterns I can use to try this out. give me a simple workflow I can try to follow, and for this run make sure the required software is free |
|
Claude finished @ronnie-guymon's task in 2m 22s —— View job A free, end-to-end EBSD re-indexing workflow you can run todayEverything below is free and open source (BSD/GPL, The core idea: your raw patterns make indexing an offline, repeatable step. You re-run it with different settings as often as you like and score the result automatically — that's the loop CALIBER is meant to automate. Step 0 — Check what you actually havekikuchipy reads raw patterns from all the common vendor containers, so check your file extension first (supported formats):
If you only have a Step 1 — Install (all free)pip install kikuchipy pyebsdindex orix matplotlib
Step 2 — Load, correct background, indexThis is the whole first pass. Adapt the phase and the pattern-centre guess to your material/geometry. import kikuchipy as kp
from orix.crystal_map import PhaseList
from orix import io
s = kp.load("patterns.h5oina", processed=False) # or .ebsp / .up1 / .h5
s.remove_static_background()
s.remove_dynamic_background()
phase_list = PhaseList(names=["ni"], space_groups=[225],
structures=[...]) # your phase
det = kp.detectors.EBSDDetector(s.axes_manager.signal_shape[::-1],
sample_tilt=70)
indexer = det.get_indexer(
phase_list,
[[1, 1, 1], [2, 0, 0], [2, 2, 0], [3, 1, 1]], # reflectors
nBands=10, tSigma=2, rSigma=2,
)
# Calibrate the pattern centre on a small grid of patterns, not the whole map
s_grid, idx = s.extract_grid((5, 5), return_indices=True)
det = s_grid.hough_indexing_optimize_pc(
pc0=[0.42, 0.22, 0.50], indexer=indexer,
batch=True, method="PSO", search_limit=0.05,
)
indexer = det.get_indexer(phase_list, nBands=10, tSigma=2, rSigma=2)
xmap = s.hough_indexing(phase_list=phase_list, indexer=indexer, verbose=2)
io.save("run01.ang", xmap)PC calibration is the step that most often makes or breaks a first run — a pattern-centre error maps almost linearly into orientation error (see Step 3 — Score the run, then sweep
score = {
"indexed_frac": (xmap.fit < 1.4).mean(),
"mean_fit": xmap[xmap.is_indexed].fit.mean(),
}Then loop the knobs that matter, keeping the best score:
This sweep is the "constantly tweak parameters" experience, just scripted instead of clicked — which is strictly better, because every run is reproducible and you get a number to rank them by. Step 4 — Refine, and escalate only if you need toCheap refinement of the Hough solution: xmap_ref = s.refine_orientation(xmap=xmap, detector=det, master_pattern=mp,
energy=20, method="minimize")If Hough indexing plateaus at a poor success rate (deformed material, noisy or low-symmetry patterns), escalate to dictionary indexing — matching each pattern against simulated master patterns. Also free, but heavier: it needs a dynamical master pattern from EMsoft, which is a compile-from-source step. Don't start here; start with Hough. from orix import sampling
R = sampling.get_sample_fundamental(method="cubochoric", resolution=3, point_group=ph.point_group)
sim = mp.get_patterns(rotations=R, detector=det, energy=20, compute=True)
xmap = s.dictionary_indexing(sim, metric="ncc", keep_n=20, signal_mask=signal_mask)Step 5 — Grains and statistics from the
|
|
Using a Jupyter Lab notebook allows me to see the entire scanned area, as well as the pattern for that area. I took these steps from https://kikuchipy.org/en/stable/tutorials/load_save_data.html. Apparently the widget is supposed to allow me to click anywhere on the scanned area to select which pattern I want to see, but I can only see the default (0,0) pattern unless I use s.inav[x,y].plot(). So the widget pretty much acts the same as if I did %matplotlib inline. The tool bars work for the widget, like zooming in or moving the map/pattern around, but actually clicking on the map doesn't navigate to a new pattern like it's supposed to. I tried %matplotlib qt as well which is supposed to work be interactive too, but it gave the exact same results as widget just with the extra step of opening two new windows, on for the scanned area and one for the pattern. Chat thinks that there might be some incompatibility with me using Python 3.14. @claude any ideas on how I could get this to be interactive? I'm doing all this in VS code btw %matplotlib widget import dask.array as da import hyperspy.api as hs data_path = Path(r"C:\Users\rongu\Box\Baird_S_EBSD\Ronnie Guymon\AlSi10Mg EBSD\AlSi10Mg Part 2\Area 2\map20260707104751409.up2") |
|
Claude finished @ronnie-guymon's task in 1m 28s —— View job Why clicking on the map doesn't navigate — and how to fix itTL;DR: plain clicking has never been the navigation gesture in HyperSpy 2.x. Use The interaction model (this is the actual fix)Since HyperSpy 2.0, the navigator interactions are (visualization docs):
One extra gotcha: if a toolbar tool (zoom or pan) is active, it captures mouse drags, so deselect it before trying to drag the marker. The symptom you describe — toolbar zoom/pan works, On the Python 3.14 theoryChat's guess is reasonable but likely wrong here: HyperSpy 2.4.0 (released 2026-01-26) added explicit Python 3.14 support (changelog), and 2.4.0 also added ipympl fixes for newer ipykernel. Worth a quick check that you're actually on it: import hyperspy, kikuchipy, ipympl
print(hyperspy.__version__, kikuchipy.__version__, ipympl.__version__)If VS Code specifics (only if Shift+click still doesn't work)
Suggested sanity check%matplotlib widget
import kikuchipy as kp
s = kp.load(data_path)
s.plot()
# Now: hold Shift and click anywhere on the scanned-area mapIf that jumps the pattern view, you're done — and this modifier-click detail is worth remembering for CALIBER docs later, since everyone hits it the first time. Sources: HyperSpy visualization docs · HyperSpy changelog (2.4.0: Python 3.14 support; 2.0: Shift+click navigation) · HyperSpy install guide
No code changes — this was a question. |
|
@ronnie-guymon @claude see vertical-cloud-lab/byu-vcl#95 (comment) @claude, look around thereabouts for additional context. kikuchipy probably makes sense to use. But, I haven't done a lot of this so it's very much about testing out tools and finding the right ones. If we have real data, that means we can do a lot of empirical testing of different methods and settings, too. |
|
Claude finished @sgbaird's task in 1m 39s —— View job Context from byu-vcl PR 95 — and why it strengthens the kikuchipy pickTodo
What I found "thereabouts"The linked comment (byu-vcl#95, comment 4272328728) is the open-source EDS stack recommendation: HyperSpy + eXSpy as the core, with sigma/SIGMA2 for phase clustering and pyEDS for map denoising. The surrounding thread ("Quantitative compositional analysis") adds a lot of directly relevant history:
Why this context reinforces kikuchipy
Concrete empirical tests your existing data already supportsThese use the Area 2
That set is basically CALIBER's proof-of-concept: same data, varied method/settings, objective score. Happy to commit test No. 1 and No. 2 as a runnable script against the Area 2 file when you want to start — that's the natural first commit beyond docs. · branch |
|
That's why the "public share link" is nice. Claude can access that directly. Alternatively, it can probably use what comes via globus too |
|
@claude help me understand what all I can do using kikuchipy, and at what point I would want to move on to MTEX. For me, everything is being compared to TEAM software. So what can kikuchipy do that is comparable to TEAM, and what does MTEX do that is comparable? Now that I have my patterns uploaded and I'm able to navigate around thanks to your shift+click advice, how do I go through with detecting patterns, seeing the CI and fit angle of what patterns are being detected, etc? |
|
Claude finished @ronnie-guymon's task in 2m 5s —— View job kikuchipy vs MTEX vs TEAM — who does what, and your next concrete stepsTodo
The mental modelTEAM does three jobs in one program: it acquires patterns, indexes them live (band detection → CI, fit, IQ), and gives you basic maps (IPF, IQ, phase). Offline, that splits into two tools:
The Feature-by-feature vs TEAM
MTEX caveat as before: MTEX is free but runs on MATLAB — as BYU folks you almost certainly have a campus MATLAB license, so this is likely a non-issue. (Free alternatives if not: DREAM3D-NX GUI, or orix for basic orientation analysis in the same notebook.) Your next steps: index the Area 2 map and see CI/fitYou're at # pip install pyebsdindex (kikuchipy calls it under the hood)
from diffpy.structure import Atom, Lattice, Structure
from orix.crystal_map import Phase, PhaseList
from orix import io, plot
import matplotlib.pyplot as plt
s.remove_static_background()
s.remove_dynamic_background()
# Your custom AlSi10Mg phase — use the lattice parameter from your TEAM phase (~4.05 Å fcc)
phase_list = PhaseList(Phase(
name="alsi10mg", space_group=225,
structure=Structure(atoms=[Atom("Al", [0, 0, 0])],
lattice=Lattice(4.05, 4.05, 4.05, 90, 90, 90)),
))
sig_shape = s.axes_manager.signal_shape[::-1]
det = kp.detectors.EBSDDetector(sig_shape, sample_tilt=70)
indexer = det.get_indexer(phase_list, nBands=10, tSigma=2, rSigma=2)
# Calibrate PC on a small grid first — pc0 guess can come from TEAM's calibration
s_grid, _ = s.extract_grid((5, 5), return_indices=True)
det = s_grid.hough_indexing_optimize_pc(
pc0=[0.5, 0.5, 0.6], indexer=indexer, batch=True, method="PSO", search_limit=0.1)
indexer = det.get_indexer(phase_list, nBands=10, tSigma=2, rSigma=2)
xmap = s.hough_indexing(phase_list=phase_list, indexer=indexer, verbose=2)Seeing CI and fit — the TEAM-style quality maps: fig, axes = plt.subplots(2, 2, figsize=(10, 8))
for ax, metric in zip(axes.ravel(), ["cm", "fit", "pq", "nmatch"]):
im = ax.imshow(xmap.get_map_data(metric))
fig.colorbar(im, ax=ax, label=metric)
print("CI > 0.1 fraction:", (xmap.cm > 0.1).mean()) # compare to TEAM's ~77%
print("mean fit (deg):", xmap.fit.mean()) # want < ~1°, per docs §9That IPF map (TEAM's orientation map): ckey = plot.IPFColorKeyTSL(xmap.phases[0].point_group)
xmap.plot(ckey.orientation2color(xmap.orientations), overlay="cm")Handoff to MTEX once the metrics look right: io.save("area2_run01.ang", xmap)ebsd = EBSD.load('area2_run01.ang', 'convertEuler2SpatialReferenceFrame', 'setting 2');
ebsd = ebsd(ebsd.ci > 0.1);
[grains, ebsd.grainId] = calcGrains(ebsd('indexed'), 'angle', 5*degree);
grains = grains(grains.grainSize >= 10); % the ≥10-points-per-grain rule, docs §4
plot(grains, grains.meanOrientation)Practical notes
Sources: kikuchipy Hough indexing · kikuchipy PC optimization · PyEBSDIndex · orix · MTEX grain reconstruction · MTEX EBSD import · No code changes — this was a question. |
@ronnie-guymon could you either get a public share/download link to the globus transferred items or copy the patterns over to this folder |
I had to combine the 3 pattern zip files into one zip file since it didn't like me using files ending with ".zip.001" and just wanted ".zip". The combined .zip file, as well as a the folder containing the files from the zip file, are now uploaded. I did both to see how well claude would handle the zip file. In the future, I will probably take the saved patterns and combine them into one file, or at least rename the files in a way that ends in .zip before uploading through globus so we can just give claude the public share/download link. @claude how well do you handle zip files? and could you combine zip files together in a way that would be compliant with how kikuchipy wants them named? try some testing of what you mentioned above with kikuchipy using https://byu.app.box.com/folder/403652824587?s=hz48nsx7ciln1jifm3u0yo5vqpge2lma |
|
Claude finished @ronnie-guymon's task in 14m 37s —— View job Zip handling + a real kikuchipy test on your Area 2 patternsI could reach the Box link directly, so I ran an actual test end-to-end. Short version: zip handling is a non-issue, and kikuchipy indexed your real AlSi10Mg patterns with 99% CI > 0.1 on the slice I pulled. 1. How I handle zips (I never downloaded the 2 GB)Box serves the file with HTTP range support, so I read the zip's central directory from the tail and pulled only the bytes I needed — no 2 GB download. The archive
Because the 2. Combining zips + "kikuchipy-compliant naming" — clearing up the confusionThe naming issue you hit isn't a kikuchipy thing at all:
3. I ran your
|
| Metric (kikuchipy → TEAM equivalent) | Value |
|---|---|
cm (≈ CI) mean / median |
0.62 / 0.67 |
| fraction CI > 0.1 | 99.2% |
fit (mean angular deviation), median |
0.53° |
| fraction fit < 1.4° | 99.5% |
| points indexed | 4,876 / 4,900 |
| speed (CPU, no GPU) | ~433 patterns/s |
That 99.2% CI > 0.1 is above the ~77% TEAM baseline from the byu-vcl thread — with the caveats that this is only the top 25 of 446 rows (~6% of the map) and the pattern centre was fit on that same slice, so full-map numbers will differ. But it confirms the offline pipeline reproduces (and here beats) the on-scope result on your actual data.
The exact recipe that worked on your file (the two gotchas folded in):
import numpy as np, kikuchipy as kp
from diffpy.structure import Atom, Lattice, Structure
from orix.crystal_map import Phase, PhaseList
s = kp.load(r"...\Area 2\map20260707104751409.up2") # nav_shape read from header (196x446)
s.static_background = s.data.mean(axis=(0, 1)).astype(s.data.dtype) # up2 has none
s.remove_static_background(operation="subtract")
s.remove_dynamic_background()
pl = PhaseList(Phase(name="alsi10mg", space_group=225,
structure=Structure(atoms=[Atom("Al", [0, 0, 0])],
lattice=Lattice(4.05, 4.05, 4.05, 90, 90, 90))))
sig = s.axes_manager.signal_shape[::-1]
det = kp.detectors.EBSDDetector(sig, sample_tilt=70.0)
indexer = det.get_indexer(pl, nBands=9, tSigma=2, rSigma=2)
s_grid, _ = s.extract_grid((5, 5), return_indices=True)
det = s_grid.hough_indexing_optimize_pc(pc0=[0.5, 0.5, 0.6], indexer=indexer,
batch=False, method="PSO", search_limit=0.2)
indexer = det.get_indexer(pl, nBands=9, tSigma=2, rSigma=2)
xmap = s.hough_indexing(phase_list=pl, indexer=indexer)
print((xmap.cm > 0.1).mean(), np.median(xmap.fit)) # 0.99, ~0.5 deg on my sliceNote batch=False in the PC step — batch=True currently trips a reshape bug in 0.13 when the extracted grid isn't perfectly rectangular.
Two things worth doing next
- Run the full map, not a slice, and add Si as a second phase (diamond-cubic) to test the eutectic-Si indexing-noise hypothesis from byu-vcl PR 95 — that's the phase-list experiment that directly probes the CI 0.1 → 0.94 story.
- This is exactly the loop CALIBER should wrap:
load → bg-correct → sweep {PC, nBands, tSigma/rSigma, phase list} → score by cm/fit. I can commit a runnablescripts/reindex_sweep.pydoing this (parameterized so it runs on any.up2) plus a short doc capturing the verified.up2spec above — say the word and I'll add it to this PR.
I didn't commit anything (PR is docs-only and the 2 GB data doesn't belong in git). No secrets touched; I accessed the Box link anonymously via its public share token.
Todo: ✅ access Box · ✅ enumerate zip remotely · ✅ extract small files + pattern slice · ✅ install free stack · ✅ Hough-index real patterns (CI/fit) · ✅ answer zip/naming questions
· branch claude/issue-2-20260724-1601
|
@claude download the full 2 GB and process in-full, then report back to @ronnie-guymon |
|
Claude finished @sgbaird's task in 2m 53s —— View job Processing the full 2 GB Area 2 dataset… Todo
|
|
@claude also, with the same portion of the patterns you processed in the first run, try it three more times. One time do just an Al phase instead of AlSi10Mg, one with AlSi10Mg and Si phase, and one with Al and Si phase. Building off of this, let's say that the phase makes a difference. Is it possible to build your own phases in kikuchipy? We can figure out how useful that would be after seeing the results of this test. |
|
Claude finished @ronnie-guymon's task in 12m 27s —— View job Phase-list experiment on the same 4,900-pattern slice — result: the phase list changed nothing, and that's the most useful findingTodo
The four runs you asked forSame slice, same background correction, same pattern centre (optimized once at PC = (0.528, 0.337, 0.741) and reused, so the phase list is the only variable):
Every number is identical to full float precision — not "similar," identical. So I ran two diagnostics to find out why. Diagnostic 1 — Si-only control: also identicalIndexing with only Si (diamond cubic, a = 5.4307 Å) gave the exact same CI (0.683) and fit (0.476°), just with every pixel labeled "si." The reason: kikuchipy's Diagnostic 2 — physically correct reflector lists don't rescue itI rebuilt the two-phase indexer with proper per-phase reflectors (fcc Al: {111, 200, 220, 311}; diamond Si, where {200} is structure-factor-forbidden: {111, 220, 311, 004}). Now the map "split" — 70% Al / 29% Si, with equally high CI on both (0.68 vs 0.70). That 29% Si is not real; it's a coin flip. The {004} and {002} poles are parallel, so even these two libraries are angularly identical, and phase assignment comes down to numerical noise. (It also ran 9× slower: 51 vs 451 patterns/s, since every pattern is voted against both libraries.) What this means
Can you build your own phases in kikuchipy? Yes — we've been doing it all alongEvery phase in these tests was custom-built. The recipe is three free building blocks ( from diffpy.structure import Atom, Lattice, Structure
from orix.crystal_map import Phase
si = Phase(
name="si", space_group=227,
structure=Structure(
atoms=[Atom("Si", [0, 0, 0]), Atom("Si", [0.25, 0.25, 0.25])], # full basis
lattice=Lattice(5.4307, 5.4307, 5.4307, 90, 90, 90),
),
)Any space group, any lattice parameters (e.g. straight from a PXRD refinement of a novel alloy), full atomic basis with occupancies. And crucially, When custom phases actually pay off, given today's result:
Suggested next stepThe decisive Si test is a small dictionary-indexing run on this same slice with Al and Si master patterns (EMsoft). Slower than Hough but runs offline on the patterns we already have. Happy to set that up, and/or commit the experiment script as No code committed. Data accessed anonymously via the public Box share token; nothing echoed from secrets. |
|
@claude I'm pretty sure we ruled out the phase being a defining factor of greatly increasing the CI, it was increasing the beam current by about 4x if I remember right that gave better patterns, but I'll have to double check. Just wanted to double check that. What would the next step be in actually getting the grain map? is that where MTEX comes in? Also it looks like the job Sterling asked for didn't work correctly. Figure out what happened and try indexing the entire 2GB pattern file again.
|
|
Claude finished @ronnie-guymon's task in 18m 53s —— View job Full 2 GB map indexed successfully + answers on beam current and grain mapsDone. The full 87,416-pattern Area 2 map is indexed, and the plots/ 1. Why Sterling's job didn't deliver — and it's fixed nowThe previous run (30292360697) succeeded as a workflow (exit 0) but ended before the compute finished. Timeline from its log: the Claude step ran only ~3 minutes (18:07:52 → 18:10:58), posted the "Processing…" comment with the download/PC steps still marked running, then the turn ended and the runner immediately hit This run avoided that by doing the whole pipeline as a single blocking foreground step (download → background-correct → PC-optimize → index → plot → export), so the runner can't be torn down mid-compute. Total wall-clock ~7 min. 2. Beam current vs phase — you're right, and our own data backs you upYour memory is correct: the ~4× beam-current increase (1.6 → 6.4 nA) is what drove CI up (that's the ~40% → ~77% CI > 0.1 jump from the byu-vcl thread), not the phase definition. The phase-list experiment from the last run corroborates this from the other direction: swapping Al ↔ AlSi10Mg ↔ Si gave bit-for-bit identical CI and fit, because for cubic phases the Hough band angles carry no lattice-parameter information. So phase choice can't move CI here, but beam current changes the actual pattern signal-to-noise, which is exactly what CI measures. Both observations point the same way. (Worth double-checking your notes as you said, but nothing here contradicts the beam-current explanation.) 3. Full-map resultsCustom AlSi10Mg fcc phase (a = 4.05 Å), mean-pattern static + dynamic background, PC auto-optimized to (0.518, 0.336, 0.753),
The full-map 99.5% CI > 0.1 confirms the slice result held up across the whole scan — and it's well above the ~77% on-scope TEAM baseline, validating the offline pipeline on your real data. Quality maps (CI, fit, IQ-equivalent, bands matched): IPF-Z orientation map — the elongated, directional grains are the classic AM melt-pool solidification texture: CI distribution: Committed to 4. Getting the grain map — yes, this is where MTEX comes inYou now have MTEX (MATLAB) — the field standard, closest to OIM Analysis: ebsd = EBSD.load('area2_full.ang','convertEuler2SpatialReferenceFrame','setting 2');
ebsd = ebsd(ebsd.fit < 1.4*degree); % CI col is 0 here, so gate on fit
[grains, ebsd.grainId] = calcGrains(ebsd('indexed'),'angle',5*degree);
grains = grains(grains.grainSize >= 10); % ≥10 px/grain, docs §4
plot(grains, grains.meanOrientation) % grain map
mean(grains.equivalentRadius)*2 % mean grain sizeMTEX is free but needs MATLAB — as BYU folks you almost certainly have a campus license, so this is likely your path. It gives grain size/shape, ODF/texture, KAM/GND, boundary character — everything past indexing. Free alternatives if you'd rather not touch MATLAB:
The natural loop: iterate indexing in kikuchipy (seconds per re-index), reconstruct grains in MTEX/DREAM3D, check grain-count stability — that back-and-forth is the CALIBER proof-of-concept in miniature. Suggested next stepReconstruct grains from this Todo: ✅ diagnosed prior job (ended mid-compute, runner torn down) · ✅ downloaded full 2.18 GB · ✅ indexed all 87,416 patterns (99.5% CI>0.1) · ✅ CI/fit/IQ/IPF plots + stats + .ang committed · ✅ answered beam-current + MTEX grain-map questions |
Full 87,416-pattern map indexed with kikuchipy + PyEBSDIndex: 99.5% CI>0.1, median fit 0.49 deg. Includes CI/fit/IQ/bands quality maps, IPF-Z orientation map, CI histogram, stats, and gzipped .ang. Co-authored-by: Ronnie Guymon <244881888+ronnie-guymon@users.noreply.github.com>
|
@claude help me understand dictionary indexing better, the benefits it has, and run a test with dictionary indexing and show me the script with everything you did. |
|
Claude finished @ronnie-guymon's task in 32m 19s —— View job Dictionary indexing: what it is, why it matters, and a real test on your Area 2 patternsEverything ran end-to-end on the same 4,900-pattern slice (first 25 rows) used in the earlier experiments. The script and all outputs are committed: 1. What dictionary indexing actually isHough indexing reduces each pattern to ~9 detected band positions and solves orientation from the angles between them. Everything else in the pattern — band widths, intensities, the fine structure between bands — is thrown away. Dictionary indexing keeps all of it. The recipe:
The benefits, concretely:
2. What the test showed on your dataYour patterns are extremely noisy — and that made this a genuinely informative test. Look at the leftmost panel: to the eye there are almost no bands. Hough survives this because the Radon transform integrates intensity along whole lines. For DI we first applied neighbour-pattern averaging (
That rightmost panel is the refined best-match simulation — DI's answer is literally "here is the pattern the physics predicts; it correlates at NCC 0.39 with what you measured." DI vs Hough agreement (stats.json):
The two methods independently reconstruct the same microstructure — mutual validation from completely different physics:
The ~17% disagreeing pixels are scattered speckle, concentrated at low-CI/boundary points (compare bottom-left vs bottom-right):
The Al-vs-Si phase test — DI succeeds where Hough couldn't. Remember the earlier experiment: Hough gave identical results for Al, AlSi10Mg, and Si phases, because cubic band angles carry no lattice information. We indexed the same slice against an Al dictionary and a Si dictionary (both refined, same settings):
The refined orientation map is committed as 3. The scriptFull runnable version (with the data download built in): import kikuchipy as kp
from orix import sampling
from orix.quaternion import symmetry
# patterns loaded + static/dynamic background removed as before, then:
s.average_neighbour_patterns() # the key step for noisy data
mp = kp.data.ebsd_master_pattern("al", allow_download=True,
projection="lambert", energy=20, hemisphere="upper")
R = sampling.get_sample_fundamental(method="cubochoric", resolution=2.5,
point_group=symmetry.Oh) # 52,607 orientations
sim = mp.get_patterns(rotations=R, detector=det, energy=20, compute=True)
signal_mask = ~kp.filters.Window("circular", det.shape).astype(bool)
xmap_di = s.dictionary_indexing(sim, metric="ncc", keep_n=5, signal_mask=signal_mask)
xmap_ref = s.refine_orientation(xmap=xmap_best, detector=det, master_pattern=mp,
energy=20, signal_mask=signal_mask,
method="minimize", trust_region=[2, 2, 2])4. Hard-won practical notes (worth remembering for CALIBER)
When to reach for DI in your workflow: Hough at 99.3% CI > 0.1 is clearly sufficient for routine grain mapping here. DI earns its cost when you need the low-CI pixels resolved (boundaries, deformed regions), when you want phase discrimination between cubic phases (the eutectic-Si question — which would need a finer-step scan to resolve spatially), or when you want a physics-based quality score for CALIBER's parameter-optimization loop. Todo: ✅ explain DI + benefits · ✅ install stack · ✅ fetch slice + Al/Si master patterns · ✅ Hough baseline (99.3% CI>0.1) · ✅ full DI + refinement + Si phase test · ✅ commit script + results (4a760d6) |
|
Claude finished @ronnie-guymon's task in 2m 53s —— View job Your staged Hough→DI idea is sound — here are the holes and how to close themFirst, validation: the architecture you sketched (cheap Hough first, escalate to DI only when needed) is not just reasonable — it's essentially what the field calls hybrid indexing, and kikuchipy ships a tutorial implementing exactly it: Hough-index everything, flag badly-indexed points by quality metrics, dictionary-index only those points, refine, merge. So you independently arrived at the community's answer. That said, there are five real holes in the version you described, and each one changes the design in a concrete way. Todo
Hole 1 — A CI plateau tells you Hough stalled, not why it stalledYour trigger ("5 runs, CI fraction only moved 0.40→0.43 → switch to DI") treats every plateau the same. But a stalled CI has at least four distinct causes, and only one of them is fixed by DI:
So the loop needs a diagnosis step between plateau and escalation: compute where the low-CI pixels live (uniform vs boundary-correlated vs patchy) and branch on that, not just on the plateau. This is exactly the "different functions depending on what outcomes still need improvement" part of your idea — the table above is the dispatch logic, and every input to it (CI map, fit map, IQ map, a quick boundary mask) is already computed. Hole 2 — CI can't score the DI stage, so your loop's objective breaks at the handoff"Move to DI and see if CI improves" is ill-posed: DI doesn't produce a CI. It produces NCC (pattern-to-simulation correlation), a different quantity on a different scale — our Area 2 runs gave median CI 0.69 from Hough and median NCC 0.285 from refined DI on the same pixels, and those numbers are not comparable. Two fixes, use both:
Related: on good data the CI > 0.1 fraction saturates (ours sits at 99.5%) and stops providing gradient for the loop. Use continuous versions (median CI, mean fit, median NCC) as the optimization signal and keep the threshold fractions only as pass/fail gates. Hole 3 — "Map 15% first" needs a sampling design, and contiguity is the constraintTwo failure modes with a naive 15%:
The fix is distributed tiles: several contiguous patches (say 4–6 tiles of ~30×50 pixels) spread across the map — corners plus center. Each tile supports the grain-adjacent metrics internally, the union approximates the global distributions, and per-tile PC fits reveal whether PC drift across the map is significant (if it is, that's its own finding: fit a PC plane, not a point). You likely need less than 15%, too — 5% in tiles was enough for our slice results (99.3%) to predict the full map (99.5%) almost exactly. One honest caveat from our timings: for Hough, sampling barely matters — the full 87k-pattern map indexes in ~3.3 min, the slice in 11 s. Sampling is really protecting the DI tier (about 14 min per full map at 2.5°, hours at finer spacings) and the many-iteration sweep, which is where your instinct is right. Hole 4 — DI is not an automatic rescue; neighbour averaging is the missing rungYour ladder goes Hough → DI, but our own data shows a step in between. On the raw Area 2 patterns, DI failed (best NCC ≈ 0.09, pure noise matches) while Hough sailed through — the Radon transform integrates along whole lines, so it's more noise-tolerant than naive per-pixel correlation. DI only became viable after Hough sweep → PC re-optimization → averaging + Hough → averaging + DI + refinement (residue only) …with the caveat we've flagged before: averaging blurs boundary pixels, so once it's on, the boundary-integrity and speckle metrics must stay in the gate to catch the damage. Also note DI has its own (smaller) parameter loop — dictionary resolution, energy, signal mask — so "move to DI and go from there" is a new inner loop, not a terminal state. Hole 5 — Don't switch to DI wholesale; escalate only the failing pixelsThis is the biggest cost win and it's exactly what the hybrid-indexing tutorial does. After your best Hough pass, typically 95%+ of pixels are already well-indexed — re-doing them with DI wastes nearly all the DI compute to reproduce answers you already trust. Instead: threshold on CI/fit, run DI only on the flagged residue, refine both populations, merge. If 5% of an 87k map fails, that's ~4,400 patterns — DI at 2.5° takes well under a minute instead of ~14, which means you can afford finer dictionaries (1.5–2°) on the residue, where they actually matter. On the beam-current point — agreed, with one design consequenceYou're right that re-acquisition (vent, reload, re-navigate) is the most expensive iteration loop imaginable, and right to push everything possible offline. The consequence to design in: offline processing cannot create information the detector never captured. There is an acquisition-quality floor below which even averaging + DI won't reach your targets — and when the loop terminates in that state, that's not a failure of the loop, it's the measurement that feeds your future SEM-parameter section: "this material at this current/dwell cannot support the requested characterization; the minimum acceptable acquisition is X." Two practical additions:
The assembled loop (phases 1–4, as you scoped)
On "5 iterations": make it a convergence rule rather than a fixed count. Hough evaluations are so cheap relative to DI (one DI + refinement run ≈ 150 s ≈ 14 Hough sweeps on the slice) that the economics say exhaust Hough thoroughly — a plateau detector ("best-so-far improved < 2% over the last 8 evals") spends compute exactly as long as it's earning, and it's also the natural seam to swap in Bayesian optimization later without changing the architecture. Net assessment: the skeleton is right and matches field practice; the five fixes are (1) diagnose before escalating, (2) use a cross-method score, (3) sample in distributed tiles, (4) insert the averaging rung, (5) escalate the residue, not the map. With those, phases 1–4 form a defensible base for grain reconstruction — and the terminal-failure case doubles as the empirical bridge to the SEM-parameter section you're planning. Happy to turn the tier table into a runnable No code changes — this was a design discussion. |
|
@claude we would also want to include Bayesian optimization in order to determine what parameters to try next for a specific outcome. |
|
Claude finished @ronnie-guymon's task in 2m 7s —— View job Bayesian optimization as CALIBER's parameter-selection brain — how it slots inAgreed, and the design from the last two comments was quietly shaped for exactly this: Todo
1. Why BO beats a grid hereA grid over just Tier 1 (
2. Library: Ax — with a Honegumi assistAx (Meta, open source, BoTorch/PyTorch under the hood) is the standard Python BO platform and the one I'd wire in:
Alternatives, for completeness: BayBE (Merck, chemistry-flavored), Optuna (great TPE-based generalist, weaker multi-objective GP support). Nothing about the loop locks you in — the score dict is library-agnostic. 3. The wiring — CALIBER's Tier-1 loop with Ax choosing parametersEverything except the Ax calls is code already validated on Area 2 in this PR ( from ax.api.client import Client
from ax.api.configs import RangeParameterConfig, ChoiceParameterConfig
client = Client()
client.configure_experiment(
name="area2_hough_tier1",
parameters=[
RangeParameterConfig(name="nBands", bounds=(7, 12), parameter_type="int"),
RangeParameterConfig(name="tSigma", bounds=(1.0, 3.0), parameter_type="float"),
RangeParameterConfig(name="rSigma", bounds=(1.0, 3.0), parameter_type="float"),
RangeParameterConfig(name="bg_std", bounds=(5.0, 15.0), parameter_type="float"),
ChoiceParameterConfig(name="nbr_avg", values=[False, True], parameter_type="bool"),
],
)
# "For a specific outcome": objective + guardrails on the others
client.configure_optimization(
objective="ci_frac_01", # maximize CI>0.1 fraction...
outcome_constraints=[
"mean_fit_deg <= 1.0", # ...but only where fit stays acceptable
"speckle_frac <= 0.05", # ...and averaging can't game CI by blurring boundaries
],
)
def evaluate(p): # one trial = one re-index of the slice
s2 = s_raw.deepcopy()
s2.static_background = s2.data.mean(axis=(0, 1)).astype(s2.data.dtype)
s2.remove_static_background(operation="subtract")
s2.remove_dynamic_background(std=p["bg_std"])
if p["nbr_avg"]:
s2.average_neighbour_patterns()
indexer = det.get_indexer(phase_list, nBands=p["nBands"],
tSigma=p["tSigma"], rSigma=p["rSigma"])
xmap = s2.hough_indexing(phase_list=phase_list, indexer=indexer, verbose=0)
return {**score_indexing(xmap), **quick_speckle_metric(xmap)} # from the previous comment
for _ in range(30): # 30 trials ≈ 6–10 min on the slice
for trial_index, params in client.get_next_trials(max_trials=1).items():
client.complete_trial(trial_index=trial_index, raw_data=evaluate(params))
best_params, best_metrics, _, _ = client.get_best_parameterization()When you genuinely care about two outcomes at once and want the trade-off curve instead of a single winner, switch one line: 4. How BO upgrades your Hough→DI switching ruleYour previous comment proposed: index ~15% of patterns, run 5 Hough feedback iterations, and switch to DI if CI improves by less than ~0.03. BO gives you a more principled version of each piece:
5. Practical notes before v1
Suggested next stepCommit a runnable Sources: Ax · Ax 1.0 quickstart (Client API) · Ax multi-objective tutorial · ax-platform on PyPI · Honegumi · Ax GitHub No code changes — this was a design question; the BO script is comment-only until requested. |
|
@claude that feedback is super helpful. More thoughts/ideas based on the feedback:
After you mentioned this is basically what the hybrid indexing already does, I'm having trouble seeing what novelty CALIBER brings, at least for EBSD. It seems like these systems already use a feedback loop when indexing. Would some sort of BO/Honegumi incorporation add more novelty? To what point does existing software already do what I am proposing? |
|
Claude finished @ronnie-guymon's task in 3m 56s —— View job Answers to the five questions + an honest novelty assessment (with new prior art found)Todo
1. Tile grid: yes, evenly spaced — with one twist worth addingYour instinct is right: no placement optimization is needed, just coverage. Three design notes:
2. "DI at 2.5°" — it's the spacing of the dictionary grid, not a match toleranceNot 2.5% and not a search radius around a guess. The dictionary is built by taking a uniform grid over all possible crystal orientations (the cubic fundamental zone) with neighboring grid points about 2.5° of misorientation apart — that's the 52,607 simulated patterns. Every experimental pattern is compared against all of them and the highest-NCC entry wins. The consequence of the spacing: the true orientation of your pixel almost never sits exactly on a grid point, so the raw DI answer can be off by up to roughly half the grid spacing (about 1.2–1.7°). That's why refinement follows — a continuous local optimizer starts from the winning grid point and polishes the orientation until the simulated pattern best matches the measured one, removing most of the grid error. Map-search analogy: pins every 2.5 km, find the nearest pin, then walk from the pin to the exact address. Finer spacing = more pins = better starting points (mainly for very noisy pixels) at proportionally more compute — that's the whole trade in the earlier timing table. 3. NCC threshold equivalent to CI > 0.1: there isn't a universal one — derive it per datasetThis is the key subtlety: NCC's absolute scale shifts with pattern noise, masking, resolution, and especially neighbour averaging. Our own data shows it: raw Area 2 patterns topped out near NCC 0.09 (indistinguishable from chance), while after averaging the refined median was 0.285. A hardcoded "NCC > X" transplanted between datasets is meaningless — but that's fine, because the threshold can be calibrated automatically, which fits the CALIBER loop perfectly:
4. Quick beam-current check in TEAM/APEX: yes — a small test scan is all it takesNo special feature needed; both TEAM and APEX let you drag the mapping region down to a small box and/or coarsen the step, so a few-hundred-point scan runs in well under a minute, with live CI/IQ statistics as it collects (APEX also has an explicit Quick scan mode among SD/HD/Manual). Practical session protocol:
5. Boundary-correlated vs arbitrary low CI: fully automatic, no eyeballing neededYou don't have to look at the map — the diagnosis is a few lines of array math, because after the first Hough pass you already hold an orientation map:
One honest caveat: the boundary mask comes from the very map you're diagnosing, so it's imperfect where indexing is worst — but for a coarse dispatch decision ("which escalation branch?") a rough segmentation is entirely sufficient. This is what makes the Hole-1 table an automated dispatcher rather than a manual checklist; the human only reviews the verdict. 6. The novelty question — the honest answer, including prior art I just foundFirst, the uncomfortable part, because you should scope claims correctly: while checking this I found a January 2026 paper (arXiv 2601.09627, ferroelectric polarization mapping via EBSD reindexing) that already uses Gaussian-process Bayesian optimization to auto-tune kikuchipy pattern-processing parameters (dynamic background, adaptive histogram equalization, FFT filters), maximizing NCC. So "apply BO to EBSD processing knobs" by itself is no longer a novel claim — though it's also encouraging: it validates that this direction is real and publishable, and it appeared months ago, meaning the field is just opening. Now the layered answer to "what do existing systems already do":
So the important reframe: "a feedback loop when indexing" does not exist in current systems — the feedback loop is the operator. Hybrid indexing is a recipe a person executes once; MapSweeper is a person turning dials. What no software does, and what your last several comments have effectively specified, is:
My suggestion: before writing this into a proposal or paper outline, run an Edison literature search specifically on closed-loop/automated EBSD parameter optimization to map the prior-art boundary properly (the arXiv paper above was found with a quick web search; a systematic sweep may find more, or confirm the gap). Say the word and I'll send it. Sources: kikuchipy hybrid indexing tutorial · arXiv 2601.09627 — BO-tuned pattern processing for EBSD reindexing · kikuchipy Hough indexing · EDAX APEX for EBSD · TEAM EBSD manual (v4.5, PDF) · prior Area 2 numbers from stats.json and dictionary-indexing/stats.json No code changes — this was a design discussion. |
Run the exact Edison search you suggested. I'm not understanding how the paper using BO doesn't use a feedback loop, doesn't BO use the previous data to choose the next best point? Which is a feedback loop? And you mentioned optimizing against grain level outcomes. Does that not come from the pattern level changes? Or are there other parameters for the grain-level software that can be messed wtih? I'm a lot more familiar with the pattern-level changes. |
|
Claude finished @ronnie-guymon's task in 15m 59s —— View job Edison prior-art survey results + answers on the noise-floor rule, the BO "feedback loop" question, and grain-level parametersTodo
1. Tile sizing — agreed, and the adaptive version is nearly freeYour split is right: for a fixed material class (LPBF Al alloys) one calibrated tile size will hold; for arbitrary metals, size tiles from a rough first pass. The nice part is the rough pass costs almost nothing — a full-map default-settings Hough pass is ~3 min on Area 2, and even a handful of tiles indexed with defaults gives a usable grain-scale estimate. So "measure grain size → set tile size" just becomes part of the Tier-0 pre-flight that was already in the design, not a new stage. Rule of thumb to encode: tile edge ≥ ~5× the mean grain diameter (in pixels), so each tile holds several whole grains and the boundary/speckle metrics computed inside it are meaningful. 3. The noise-floor rule — you have it right, with two directional clarificationsDirection of the test: a pixel passes when its NCC is ≥ (noise floor + 3σ); it fails when its best match is statistically indistinguishable from correlating against random orientations. So yes — failed = not above the noise floor. Direction of the DI trigger: one subtle flip from how you phrased it. You don't wait until the failed fraction shrinks to 5–10% before moving to DI — the failed fraction is whatever it is once Hough is exhausted (plateau detected + Hole-1 diagnosis run). At that point:
So the failed fraction sets the cost of the DI tier, not the decision to run it; the decision comes from the plateau + diagnosis. One practical note: during the Hough tiers the working definition of "failed" is CI/fit (Hough's native metrics, free per pixel); the NCC noise-floor definition takes over once you're scoring in the shared NCC currency. The percentile-matching calibration from the last comment is what keeps those two definitions consistent so pixels don't flip categories at the handoff. 4. Saved test-scan patterns — noted, it's in the designNothing to add — enabling save-patterns on the beam-current test scans gives the offline scorer real data during the session and doubles as the pre-flight gate. The Edison search — five-area prior-art verdictThe exact search ran successfully (Edison LITERATURE/PaperQA, task
Edison's closing assessment is worth quoting because it is nearly a CALIBER spec: the unexplored combination is "repeated reindexing of archived raw patterns under optimizer-selected settings; joint pattern-, pixel-, and grain-level objectives with explicit cleanup/topology constraints; confidence- and grain-context-based escalation from Hough to dictionary matching; and translation of the optimized result into acquisition recommendations." No publication combines those elements. One reconciliation caveat: Edison's corpus did not surface the January 2026 arXiv paper (2601.09627) I found by web search, which does apply GP-based BO to kikuchipy pattern-processing parameters — likely because it's an arXiv preprint too recent for the indexed corpus. So treat Area 1 as "open in the peer-reviewed literature, with one very recent preprint doing the narrow single-objective version." Areas 2–5 verdicts stand: that paper tunes processing knobs against NCC only — no indexing-method escalation, no grain-level objectives, no acquisition bridge. "Doesn't BO use previous data to pick the next point? Isn't that a feedback loop?"You're completely right, and my earlier wording was sloppy. BO is a feedback loop by definition — evaluate parameters → update the surrogate model → pick the next trial from everything seen so far. The arXiv paper absolutely closes a loop, and so does CALIBER. The distinction I was actually drawing is what's inside the loop and what it optimizes:
So the accurate novelty claim is not "we add a feedback loop" — loops exist — it's "we change what the loop measures and decides." The Edison survey confirms the fuller version is unclaimed in the peer-reviewed literature. "Don't grain-level outcomes come from pattern-level changes? Or are there grain-level parameters too?"Both, and this is exactly the right question. Two separate mechanisms:
So the full CALIBER search space is two coupled layers — pattern-level knobs (change what orientations you get) and grain-level knobs (change how orientations become grains) — with grain-level metrics as the shared objective across both. Nobody has published the joint optimization; Polonsky covers a one-parameter sweep of the second layer only. Committed this session (3d238c0): the complete Edison trajectory under |
Co-authored-by: Ronnie Guymon <244881888+ronnie-guymon@users.noreply.github.com>
|
Texture as I understand it: seeing if a certain grain orientation is preferred. If yes, then there will be anisotropy in the properties. |
|
@claude using the full map from #8 (comment), use DREAM3DMX to do grain reconstruction. Change different parameters by a large amount so I can understand and see the effect changing the parameters has on the grain map. Show me the script you use, and explain how each changed parameter changed the outcome. If these can be changed in DREAM3DMX, change them:
Even if changing a parameter will result in a worse outcome, do it anyways to show me what it looks like. In fact, I would like you to make the quality worse and then try to make it better, again showing me the code used and explaining what it does. |
|
Claude finished @ronnie-guymon's task in 11m 2s —— View job DREAM3D-NX grain reconstruction on the full Area 2 map — 14 reconstructions, 4 parameter sweeps, worse-then-better demoAll done and committed (fdb3910). The script is Todo
Setup and the scriptDREAM3D-NX's Python bindings aren't on PyPI ( conda create -y -n nx -c bluequartzsoftware -c conda-forge python=3.12 dream3dnx numpy matplotlib-base
conda run -n nx python scripts/dream3dnx_grain_reconstruction.pyThe pipeline in the script (full version here) is six DREAM3D-NX filters, mirroring what OIM Analysis does internally: import simplnx as nx
import orientationanalysis as oa
# 1. Read the .ang from the kikuchipy Hough run (gunzip area2_full.ang.gz first)
oa.ReadAngDataFilter.execute(data_structure=ds, input_file="area2_full.ang",
output_image_geometry_path=GEOM)
# 2. Pre-segmentation quality filter -> boolean Mask
# (kikuchipy's .ang writer zeroed CI/IQ, so we gate on the fit column, in degrees)
t = nx.ArrayThreshold(); t.array_path = FIT; t.comparison = LessThan; t.value = fit_max
nx.MultiThresholdObjectsFilter.execute(..., array_thresholds_object=ts)
# 3. Euler angles -> quaternions (what the segmenter compares)
oa.ConvertOrientationsFilter.execute(..., input_representation_index=0, # Euler
output_representation_index=2) # Quaternion
# 4. THE grain reconstruction: flood-fill; a boundary = neighbor misorientation > tol
oa.EBSDSegmentFeaturesFilter.execute(..., misorientation_tolerance=5.0, use_mask=True)
# 5. Minimum-points-per-grain cutoff
nx.ComputeFeatureSizesFilter.execute(...) # counts px per grain
nx.RequireMinimumSizeFeaturesFilter.execute(..., min_allowed_features_size=10)
# 6. Cleanup: grow grains into unassigned (id 0) pixels
nx.ErodeDilateBadDataFilter.execute(..., operation_index=1, # 1 = erode bad data
num_iterations=2)
nx.FillBadDataFilter.execute(..., min_allowed_defect_size=1000)Each run reports: raw grain count (straight out of segmentation), kept count, retained pixel fraction, % of pixels rewritten by cleanup, number-weighted and area-weighted mean equivalent diameter, speckle fraction, largest grain. All 14 runs: Sweep A — misorientation tolerance (what counts as a boundary)
Two neighboring pixels join the same grain if their crystal misorientation is below the tolerance. Lower it toward the orientation noise floor (Hough precision is roughly 0.5°) and real grains shatter: at 1° the raw count triples to 34,274 because intra-grain orientation noise now counts as "boundaries." Raise it past real boundary angles and grains merge: at 15° the largest grain balloons from 544 to 1,391 px as genuinely distinct neighbors (misoriented 5–15°) fuse. The subtle and important part: the kept counts (1,426 / 1,930 / 1,905 / 1,583) are non-monotonic and look deceptively similar, because the min-size cull deletes most of the 1° fragments. The raw count is the honest signal — which is exactly why Polonsky 2019's stability curve is plotted against raw segmentation output. In your future feedback loop, track raw count vs tolerance and look for the plateau (here roughly 2–5°, consistent with the Edison recommendation for AM material). Sweep B — minimum-points-per-grain cutoff
With the cutoff off, 82% of "grains" are sub-10-px speckle — mostly misindexing noise — and the number-weighted mean grain size crashes to 4.7 µm while the area-weighted mean stays at 14.7 µm. That divergence between the two means is the fingerprint of a speckle-contaminated map, and it's the same mechanism as Staněk 2020's million-grains-to-75k example. At 100 px the opposite failure: real small grains are deleted, only 109 survive, and "mean grain size" inflates to 50 µm. At 400 px the map is reduced to 2 absurd grains. Two DREAM3D-NX behaviors discovered here worth knowing:
Sweep C — pre-segmentation quality filter
This filter excludes untrusted pixels before segmentation (the DREAM3D equivalent of TEAM's CI filter; here it gates on fit since the CI column was zeroed on export). Because your indexing is good — 99.7% of pixels have fit < 1.4° — the mild filter barely differs from no filter at all. But fit < 0.5° sits at the median of the fit distribution (0.49°), so it discards half the map: grains become moth-eaten fragments, the count halves, and mean size shrinks 26%. Lesson for the feedback loop: quality thresholds must be set relative to the dataset's metric distribution, not as absolute constants — a threshold that's mild on one map is destructive on another. Sweep D — cleanup passes (this is the "fabricated data" knob)
Starting from the crippled fit < 0.5° map (49% of pixels are holes) to make the effect visible:
Each erode iteration grows every grain one pixel into adjacent unassigned space. At ×10 the map looks "complete" (98% retained) — but 72% of its pixels are invented, and look at the morphology: the recovered grains are round blobs, while the real microstructure (baseline panel in the figures above) has the elongated, directional melt-pool grains. The cleanup didn't recover the microstructure; it manufactured a plausible-looking wrong one, doubling the apparent mean grain size along the way. Worse, then better
Same orientations in, ~18× difference in reported grain count — the entire swing is reconstruction parameters. That's the empirical version of the point from the earlier design discussions: Stage 5 has its own parameter layer worth optimizing (and guarding), independent of indexing quality. What this means for CALIBER's loopConcrete, computable signals this experiment validated: raw-grain-count-vs-tolerance plateau (stability), number-weighted vs area-weighted mean divergence (speckle detector), retained fraction after filtering (threshold sanity), and % pixels rewritten by cleanup (fabrication penalty). All four came straight out of Natural next step: wire these grain-level metrics into the staged feedback loop as the guardrail objectives alongside CI/fit — say the word and I'll extend the script into that combined loop. · branch |
Co-authored-by: Ronnie Guymon <244881888+ronnie-guymon@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@claude a few questions: It looks like there are hundreds of grains, not the thousands you said we would need in order to find the texture. Would we need to scan a larger sample area if we wat to understand the texture? For sweep A, why are there progressively fewer and fewer holes? Why are there holes at all? For Sweep C, would we want to keep the fit cutoff at <1.5°? or would 2° work (Mike has said we want to be at least under 2°, ideally less than 1°)? This would also help with the questions from Sweep D, as there would be much fewer grains to reconstruct if we filtered at <1.5° instead of <0.5°, kind of like was done with the "worse, then better" portion? If only 1.2% of pixels had to be rewritten with a fit filter as low as <1.4°, then I guess that works. What we could do, and lmk if this is unnecessary, is start at fit <0.5°, see the percentage of pixels that were rewritten, and then increase the fit cutoff by a few tenths of a degree until only say 5% (or some other optimal percentage) of pixels are rewritten? This way we could reach a good balance between keeping the best quality pixels without having to rewrite a large quantity of grains. |
|
Claude encountered an error after 3m 25s —— View job I'll analyze this and get back to you. |
|
Super cool. Love what you're doing here, Ronnie. @claude pick back up on #8 (comment) |
|
Claude finished @sgbaird's task in 3m 42s —— View job Answers: texture statistics, the Sweep A holes, and choosing the fit cutoff(Picking up the errored run — the previous job died before answering.) Todo
1. Texture: you have ~1,900 grains, and the fix is a coarser step, not more scan timeFirst a small correction to the premise: the baseline reconstruction kept 1,905 grains (stats.json), not hundreds — the 109-grain number was the deliberately-broken 100 px cutoff run. So you're at roughly 2k. Is 2k enough? The statistical unit for texture is the grain, not the pixel — the 87k pixels don't help, because pixels inside one grain are copies of the same orientation, so your effective sample size is ~1,900 independent orientations. What that buys you:
Now the important part: you don't need a bigger sample or longer scan — you need a coarser step over a larger area. Texture needs exactly one orientation per grain; it does not need the ≥10 pixels per grain that size statistics need. At your 2 µm step you're spending ~50 pixels on each grain to answer a one-pixel question. Triple the step to 6 µm and the same scan time covers 9× the area ≈ 9× the grains (~17k). Alternatives: stitch several maps, or just combine the orientation data from multiple areas when computing the ODF (MTEX merges datasets trivially). This is worth pausing on for CALIBER, because it's the cleanest demonstration yet that the optimal parameter set depends on the objective: grain-size statistics want fine steps (points-per-grain floor), texture wants coarse steps (max grain count per unit time). Same sample, same instrument, opposite step-size recommendations — exactly the kind of trade-off the tool should surface rather than hide. 2. Sweep A holes: two sources, and one of them scales with the cullHoles = black pixels = pixels that ended the pipeline assigned to no grain (FeatureId 0). They come from two places in the script:
Why the holes shrink as tolerance rises is then just arithmetic on the cull:
At 1°, the tolerance sits near the Hough orientation noise floor (~0.5°), so intra-grain noise gets segmented as "boundaries," the map shatters into 34k fragments, ~96% of them get culled, and whole shattered regions have no surviving grain nearby to absorb them → 7.5% holes. As the tolerance rises past the noise, fragments stop forming, the cull touches less area, and holes converge toward the 0.35% mask floor. Useful side effect: post-cull hole fraction is itself a free diagnostic. A spike in holes as you lower the tolerance is the signature that you've crossed below the orientation noise floor — one more automatic signal for the feedback loop, complementing the raw-grain-count plateau. 3. Fit cutoff: on this map 1.4° and 2° are the same choice — and your adaptive scheme is right, with one simplification(Small note: Sweep C used fit < 1.4°, not 1.5°.) I computed the actual fit distribution from the committed
Three conclusions fall straight out:
Two guardrails before adopting "exclude 5%" as the policy, though:
So: not unnecessary at all — your scheme, restated as "set the cutoff from the fit-distribution percentile, subject to a boundary-enrichment veto," is the right general policy for CALIBER and drops into the loop as two lines of numpy. No code changes — these were questions about the existing sweep results. |
|
@ronnie-guymon how do you feel about the "DI + refinement on the residue only, finer dictionary as budget allows", from what I understand, the highest quality analysis only being run on a subset of patterns? Claude seems to keep suggesting this, but it didn't seem like running everything with DI + refinement would be computationally intractable or even that burdensome, which gets me wondering how much we need to push back on Claude or if we're not actually losing out on anything. Just seems to me like we should be using the highest quality analysis on everything, not just a subset. @claude see above. Keep response concise |
|
Claude finished @sgbaird's task in 44s —— View job Residue-only DI was a loop-economics choice, not a quality ceiling — full-map DI is fine as a final pass
You're right that full-map DI + refinement is tractable. Measured on Area 2 (87k patterns, 4 CPU cores, 2.5° dictionary): DI ≈ 14 min + refinement ≈ 26 min ≈ 40 min total, ~2.6 GB dictionary in RAM. That's an easy overnight-or-lunch job, and a GPU or finer-but-residue-only combos make it faster still. Nothing intractable. The residue-only suggestion was about where in the workflow the cost sits, not whether it's payable once:
Two quality caveats before calling DI "the highest quality analysis on everything," though — from our own runs:
Suggested policy: Hough (+ refinement) drives the parameter loop; final deliverable map = full-map refinement, with DI on everything if budget allows or on the residue if not — then verify the grain-level metrics didn't move. If full-map DI and the hybrid map give the same grain statistics (my expectation), that's empirical license to use the cheap path routinely; easy experiment to run on Area 2 if you want the numbers. |
|
Notes from meeting Xavier: @claude I don't need any feedback, just keep this in mind. |
|
Claude finished @ronnie-guymon's task in 45s —— View job Noted — plan recorded, no feedback given per your request
Captured for future work, in order:
I'll treat this as the standing plan for the next implementation step on this branch (starting point: --- · branch |
@sgbaird yeah as Claude said, I think doing Hough indexing will be ok as long as the patterns are intense enough it can read the Kikuchi patterns. If the lines aren't intense enough the we can move to DI. I can keep this in mind though and see the difference it would make to do all DI right off the bat. |
|
Claude finished @ronnie-guymon's task in 37s —— View job Agreed — with one caveat on "if the lines aren't intense enough, move to DI"
Your summary matches the plan: Hough drives the loop while band intensity supports it, escalate when it can't. Two brief refinements from our Area 2 runs worth keeping in mind:
No code changes — nothing was requested. --- · branch |
|
@claude in response to your commentary here #8 (comment): I think you and I are saying the same thing for the texture. I said we need to scan a larger area of the sample, is that not also the same thing you're saying just with the addition that when we do scan the larger area, we use a coarser step size? In terms of workflow, that means when doing the imaging session, we should make sure that we have enough time to get both a smaller area for the microstructure analysis, and a larger scan for our texture analysis. We will probably have different functions for the two forms of analysis as well, or possibly even different objectives for each o f them. Something to keep in mind We will probably just stay around the 5° misorientation tolereance, I don't see any reason to change from that. If we do decide to change that, we can use the hole fraction as our metric to decide whether or not a threshold has been reached. I will finish commenting/questioning on the last half later |
|
Claude finished @ronnie-guymon's task in 52s —— View job Confirmed — same page on texture; two notes filed for the workflow
Texture: yes, we're saying the same thingLarger scan area is the substance of it; the coarser step is just what makes the larger area affordable in one session. Scan time scales roughly with area ÷ step², so a bigger area at the same 2 µm step costs quadratically more time — while tripling the step to 6 µm buys about 9× the area (≈ 9× the grains) in the same scan time. So your workflow conclusion is exactly right:
So yes: two scans per imaging session, and two objective functions in CALIBER keyed to which deliverable the scan serves. Two practical notes for the session plan: (1) coarsening the step doesn't degrade per-pattern quality — dwell/current still set CI, so the beam-current pre-flight applies to both scans; (2) keep save-patterns on for both, per the standing raw-patterns policy — the texture scan's patterns are cheap insurance and reindexable like everything else. 5° tolerance: agreed as the default, one asymmetry to remember about the hole-fraction metric5° sits on the stability plateau from Sweep A (2–5°) and is the field convention — no reason to move for grain characterization. The one nuance if it's ever revisited: hole fraction only guards the low side. It spikes when the tolerance drops below the orientation noise floor (map shatters → mass culling → orphaned pixels), but merging at too-high tolerance produces no holes at all — that failure shows up in the raw grain count and the largest-grain size instead (Sweep A: largest grain 544 px at 5° → 1,391 px at 15°, holes nearly unchanged). So the pair of signals is: hole fraction for "too low," raw-count/largest-grain drift for "too high." One flag for later, not now: the Edison synthesis recommended treating 2–5° as a sweep in AM material because cellular/subgrain boundaries are physically real in that range. Fixed 5° is the right call while the outcome is grain characterization — just note that it deliberately merges subgrain structure, so if KAM/subgrain analysis ever becomes a CALIBER deliverable, the tolerance becomes an output again rather than a constant. Noted on the rest — I'll pick up your remaining questions on the last half whenever you post them. No code changes — this was a design confirmation. --- · branch |
|
@claude I think my knowledge on the subject of EBSD is still lacking. Find papers that will help me directly understand the subjects we have been talking about, especially when it comes to how parameters will alter the outcome. I want to understand the grain orientation as well and the implications of it. Provide any other papers you thin would be helpful, such as the ones you cited that are similar to CALIBER, and provide a reasoning for why they are chosen and their subject. Also claude, Xavier thought that measuring variance would be the best outcome to start with measuring and trying to reduce and see if that causes a decrease in the failed pixels. Is this a good place to start, or are there any other outcomes we could use and possibly have a dual-outcome function. This would be done through probably minimum points per cutoff, fit filter, and regrowth in the function, as well as any other parameters we might need to change. @sgbaird I feel like most of what I have been learning on this subject is from what claude has summarized, and after Xavier's comment yesterday about how he has been best able to gain understanding from reading other papers, I feel like that might be the best thing for me to do right now. I don't really understand what the parameters do or what specific outcomes I should be looking for, nor the implications of those outcomes. Is there a better way you can think of for me to go about CALIBER? I feel like before I can try and optimize parameters to get better outcomes, I need to have an understanding of why I even want those outcomes and why they are useful. Am I trying to get too deep of an understanding for the scope of this project? I just want to have the understanding we were talking about yesterday during our meeting. |
|
Claude finished @ronnie-guymon's task in 4m 51s —— View job A reading curriculum for the EBSD topics in this thread + assessment of variance as the starting objectiveTodo
Part 1 — The reading listOrdered as a curriculum, not by importance — each tier assumes the one before it. OA = open access (readable without library login; for the rest, BYU library access will get you in). Reasoning for each pick is the second sentence. Tier 1 — Foundations (read these first):
Tier 2 — How parameters alter the outcome (the core of your question):
Tier 3 — Indexing methods:
Tier 4 — Your material: grain orientation and its implications in LPBF AlSi10Mg:
Tier 5 — CALIBER-adjacent (the prior-art papers, and why each matters to the project):
The full Edison-generated bibliographies (with more, e.g. Wanni 2024 on texture representation) are committed at Suggested order: 1 → 3 → 10 (one sitting each — orientation, motivation, your material), then 2 and 4 carefully (they're the parameter→outcome core), then 6–8 alongside the committed sweep figures, then 9 and the Tier-5 set when you're designing rather than learning. Part 2 — Is variance the right starting objective?Short answer: variance is a reasonable candidate, but only as one half of a pair — on its own it has a degenerate minimum that the optimizer will find immediately. And one causality correction first. The causality correction. "Reduce variance and see if that causes a decrease in the failed pixels" has the arrow backwards for the Stage-5 knobs you listed. Failed pixels (excluded/unassigned) are directly set by two of the three parameters in the function: the fit filter decides how many pixels are excluded, and regrowth decides how many holes get filled back in. So variance and failed-pixel count won't be cause and effect — they're two symptoms co-driven by the same knobs. That's fine (Xavier's plan is to check whether objectives correlate with map quality, not to establish causation), but it means the loop should treat "failed pixels" as a second measured outcome, not as the downstream effect of variance. Why variance alone is gameable — with numbers from our own sweeps (stats.json):
The fix is exactly the dual-outcome function you asked about. Pair a quality objective with a fabrication/coverage penalty so the degenerate solutions become losing moves:
That structure drops straight into Honegumi/Ax as a two-objective (Pareto) problem with outcome constraints. Two measurement details that will matter:
So: start where Xavier suggested, but as CV of grain size + % regrown as a pair, computed alongside the rest of the panel in a single Sobol campaign. Happy to extend Part 3 — On the learning-approach question (addressed to Sterling, but two cents)You are not going too deep. Objective design is the intellectual core of CALIBER — an optimizer pointed at a metric nobody understands produces maps nobody trusts, and Part 2 above is a live example of why (variance has a failure mode you can only see if you know what a good map looks like). The calibration I'd suggest: aim for working understanding, not theory — for each parameter, can you (a) predict the direction of its effect, and (b) name the failure mode it guards against? The reading list is scoped to that altitude: you don't need dynamical diffraction physics (treat EMsoft as a black box), but Tiers 1–2 you want cold. One study tip that fits how this branch has evolved: read Humphreys and Wright 2015 with the committed sweep figures open — the papers explain the mechanisms, and the sweeps show the same mechanisms on your own Area 2 data, which is a faster way to make it stick than either alone. No code changes — this was a literature/design question. All paper links above were validated this session. |
|
@claude are you there? |
|
Claude encountered an error after 1s —— View job I'll analyze this and get back to you. |
|
@claude+opus you there? |
|
Claude finished @ronnie-guymon's task in 32s —— View job Here — and the two errored runs didn't lose anything
Both of the failed runs (32505468129, 1 s; 31621260719, 3 m 25 s) died in the harness before doing any work — no partial commits, no half-finished state. Branch Where the thread stands: the last substantive item was your Aug 20 question, answered with the reading curriculum and the assessment of variance as the starting objective (short version: use CV of grain size + % pixels regrown as a pair, with Standing next step, ready when you want it — Xavier's plan from the Aug 13 meeting notes:
That's an extension of No code changes — this was a check-in. |
2 participants
Learn about the most prevalent methods of post processing EBSD data and which one(s) would be best to use for CALIBER