Conversation
…dex for the darks (eg below the first 10 time bins use that as the dark spectrum)
…ization also IRF fitting that is dubious right now but fits something
merging master to the current experiment to capture the recent PR for the dark shot changes.
…rence_spectrum-should-have-a-feature-to-make-difference-based-on-indices 21 xspectvisualizationplot 2d difference spectrum should have a feature to make difference based on indices
…. Scaled fast_delay_key values by 1e12 in time_binning method in XSpect_Analysis.py. Created a current clean and working notebook XSpect_XES_dehe_on_demand_05022025.ipynb
…n_demand, etc. Renamed the notebooks so that they are unique and they don't conflict with merges from others. It would be good to come up with a system/protocol for git and notebook.
…heritance chain. It doesn't seem to break anything. I just think it is neater that way.
Merging code changes from Dehe for Follmer experiment
Mfx10089224
…eys are unique and not duplicates in the final set of attributed.
This commit introduces a new method, `primary_analysis_static_laser`, specifically designed for analyzing data with static laser conditions. This method now separates shots based on laser presence and applies hitfinding and spatial reduction steps individually to the resulting shot subsets: 'epix', 'epix_simultaneous_laser', and 'epix_xray_not_laser'. This allows for a more granular analysis of laser-related effects. Key changes: * Added `primary_analysis_static_laser` method for laser analysis. * Modified the laser analysis method to filter shots and handle hitfinding, transposition, patching, rotation and spatial reduction for different shot types to enable laser-specific filtering and analysis. * Updated `keys_to_save` to include results from laser-specific analysis.
…tra (i.e. no laser)
…his at the object level
…zation taking the absolute value of the trapz because the energy axis direction changes the values of the trapezoidal integrals.
…d_rotation_angle patch_pixels gains auto_detect mode for ASIC panel-gap columns. A two-method approach handles the full range of defect severity: Method 1 (ratio): compares column profile against median-filtered baseline. Columns with ratio > threshold (spikes) or < 1/threshold within the signal band (dead gaps) are flagged. Catches extreme charge-sharing spikes that dwarf the spectral signal. Method 2 (z-score + width filter): computes a robust z-score using MAD-based local sigma. Bright and dark outliers above nsigma are separately clustered and kept only when cluster width <= max_gap_width. Separating bright/dark prevents merging a dark gap with its bright charge-sharing neighbors into a single wide cluster that would be rejected as a signal gradient. This catches subtle 50%-reduced dead columns that the ratio method misses. The polynomial fit is vectorised: instead of calling np.polyfit for every (shot, row) pair, the Vandermonde matrix is solved once via np.linalg.solve to get a projection vector, then all rows are evaluated with a single dot product. The fit region now also excludes the ±patch_range neighborhood of every known bad column, not just the target pixel, so elevated charge-sharing neighbors don't bias the polynomial anchor. find_rotation_angle wraps XSpectDetectorProcessor to auto-detect the tilt of a dispersed spectral signal via Canny edge detection → DBSCAN clustering → per-cluster PCA. Stores the angle on the run for use by rotate_detector. XSpectDetectorProcessor.find_optimal_rotation_angle: fix arctan2 argument order (was arctan2(dy,dx), should be arctan2(dx,dy) for row-dominant principal vector); weight cluster angles by cluster size so small edge-detection artifacts don't bias the result. filter_detector_adu: guard getattr with a None check so the step skips gracefully when the detector key is absent on the pre-pipeline pass.
…gies length reduce_detector_ccm previously only handled 1D (scalar) and 2D (spectrum) detectors. A new 3D branch accumulates full 2D images per energy bin — producing a (n_energy, rows, cols) stack — enabling RIXS-plane pipelines where the detector is a 2D spectrometer image not yet collapsed spatially. Off-by-one in bin count: both reduce_detector_ccm and reduce_detector_ccm_temporal were allocating n_bins = len(ccm_bins) where ccm_bins is the n+1-element edge array from make_ccm_axis. They now use len(ccm_energies) (= n edges - 1) so the output axis length matches the energy axis. NaN-guarded accumulation: shots that produce NaN values (e.g. empty bins, missing HDF5 rows) are now skipped rather than poisoning the running sum. For 2D/3D detectors np.where(np.isnan(...), 0.0, ...) replaces NaN pixels element-wise so a single bad shot doesn't zero an entire spectrum row. reduce_detector_ccm_temporal gets the same ccm_energies length fix and NaN guarding for both 1D and 2D detector dims. make_energy_axis and normalize_xes: minor formatting only.
…erance
Four new test cases cover the full auto_detect surface:
- test_auto_detect_spike: confirms a 100× bright column is flagged and
patched via the ratio method (Method 1)
- test_auto_detect_subtle_gap: confirms a 50%-reduced narrow 2-col dip
is caught by the z-score method (Method 2) which the ratio threshold misses
- test_auto_detect_ignores_wide_gradient: verifies that a smooth
intensity gradient across 200 columns produces zero flagged pixels
- test_auto_detect_merges_manual: checks that auto-detected and
manually specified pixel lists are both applied
test_patch_interpolate tolerance relaxed from exact equality to atol=1e-3
because the vectorized polynomial fit (Vandermonde + np.linalg.solve)
differs from np.polyfit in numerical precision by ~1e-10 — the test was
comparing against a specific polyfit rounding, not a scientific threshold.
…nalysis notebook
Three YAML pipeline configs for the polariton / FeNO pump-probe XAS
experiment at XCS (LCLS run 26, runs 187-216 and 339-347):
xcs101591326_ultrafast_xas.yaml — static DCCM energy scan, auto CCM
axis from setpoints, laser-on vs laser-off, 1D XAS with reduce_detector_ccm
xcs101591326_temporal_xas.yaml — time-delay scan at fixed DCCM energy,
enc/lasDelay binning (-10 to 25 ps, 35 bins), reduce_detector_temporal,
combine_runs reduction across 7 runs with uncertainty propagation
xcs101591326_2d_xas.yaml — simultaneous DCCM energy × time-delay scan,
21-bin time axis (-2 to 8 ps), 2D binning via reduce_detector_ccm_temporal,
produces transient absorption map Δμ(E,t)
Analysis notebook (xcs101591326_temporal_xas_analysis.ipynb) computes
mu = If/I0 per delay bin from the pipeline output and plots normalised
difference spectra for kinetic analysis.
…eline configs
mfx101609126_pershot_xes.yaml — stochastic RIXS input preparation:
filters to xray shots, ADU thresholds both detectors, patches ASIC
gap columns (manual on epix100_0; auto_detect on epix100_1 SEER),
rotates -2.0° (spec) / -1.6° (SEER), projects rows → 1D per-shot
spectra for spook correlation. Produces:
xrt_hproj (N×2048) and epix_spec_ROI_1 / epix_seer_ROI_1 (N×n_cols)
mfx101609126_seer_xrt.yaml — static comparison of both spectrometers
across runs 78-82: sums all xray shots to 2D image then collapses
rows → 1D spectrum for each detector independently.
mfx101609126_static_rixs.yaml — RIXS plane for run 97 DCCM scan:
reduce_detector_ccm accumulates full 2D epix images at each incident
energy → (n_energy, 60, 300) stack, then reduce_detector_spatial
collapses the cross-dispersion axis → (n_energy, 300) RIXS plane.
mfx101609126_droplet_pershot_xes.yaml — per-shot XES using photon-
counting via droplet_reconstruction (epix100_0 fixed-length sparse
format), same downstream steps as pershot_xes.
mfx101609126_droplet_recon_test.yaml — 100-shot smoke test for
droplet_reconstruction; max_shots: 100 avoids materialising the full
~36k-shot array during development.
rotation_diagnostic — finds optimal detector tilt angle by running the pipeline on a subset of shots and inspecting the 2D sum image with overlaid rotation angle estimates from XSpectDetectorProcessor. Fixed angles (-2.0° / -1.6°) were determined here and hardcoded into YAMLs. pixel_patch_diagnostic — compares raw vs patched column profiles for epix100_0 and epix100_1, visualises the ASIC gap positions and the effect of auto_detect patching on the baseline ratio and z-score. seer_shot_browser — per-shot SEER image browser: scan forward/backward through individual shots to inspect photon distributions and confirm the ASIC gap correction is working correctly. seer_vs_xrt — quantifies SEER as an alternative incident-energy monitor for stochastic RIXS. Computes correlation between SEER and XRT per- shot total-intensity and center-of-mass energy. Conclusion: SEER tracks intensity well (r=0.90) but has low COM energy correlation (0.41) because it encodes sample transmission/emission, not purely the incident beam. Gap patching dramatically improves the spook AtA condition number. spook_rixs — stochastic RIXS extraction using the spook library: builds A (XRT hproj) and B (epix_spec per-shot spectra) matrices and solves for the RIXS cross-section X via regularised least squares. droplet_spook_rixs — same as spook_rixs but using photon-counted spectra from droplet_reconstruction instead of raw ADU integration. droplet_spook_transmission — stochastic RIXS using SEER (transmission geometry) as the B matrix; photon-counted for improved S/N. rixs_plane — static RIXS plane visualisation for the DCCM scan run.
regenerate_run0079.py — runs mfx101609126_pershot_xes.yaml (standard ADU integration) and saves per-shot spectra to examples/results/; used as the baseline to compare against droplet-reconstructed spectra. regenerate_droplet_pershot.py — runs mfx101609126_droplet_pershot_xes.yaml for runs 78-82 sequentially, saving one HDF5 per run to results/. Measures reconstruction throughput (shots/sec) and reports memory usage. droplet_reconstruct.py — standalone utility script that reconstructs a single run's sparse photon data without the full pipeline framework; used for rapid iteration and debugging of the reconstruction step. droplet_reconstruction.ipynb — step-by-step walkthrough of the sparse format: reads raw HDF5, shows fixed-length vs variable-length layouts, runs _scatter and _scatter_roi helpers manually, compares output images against the pre-processed ROI_area dataset for bit-identity verification. droplet_recon_pipeline_eval.ipynb — end-to-end comparison of ADU integration vs photon-counting via the pipeline API. Runs both YAMLs on run 79, overlays 1D emission spectra, and evaluates peak S/N ratios to quantify the photon-counting improvement for spook inputs.
…c assets mfx101080524_static_xes.yaml: remove hitfinding step. The hit-finder was removing shots that had low per-frame median signal, which is expected for single-photon-level XES data. Removing it recovers the correct shot count and avoids discarding weak-signal runs. seer_full_detector_run78.png: full-panel SEER detector image from run 78 showing the spatial distribution of signal across all ASIC tiles; used in rotation_diagnostic.ipynb and seer_shot_browser.ipynb to orient the cross-dispersion ROI selection. c.ipynb: scratch notebook for static RIXS pipeline diagnostics (run 97); shows RIXS plane shape, energy range, and 2D sum-image ROI inspection. xcs101591326.ipynb, Untitled.ipynb: updated experiment analysis notebooks.
…MFX pipelines Droplet reconstruction, patch_pixels auto-detect, XAS fixes, and XCS/MFX pipelines
…, and analysis notebooks
Adds the full mfx102101026 (ferricyanide) static XES analysis workflow and the
supporting XSpect engine changes needed to run it at scale, plus the earlier
mfx100895324 static-XES artifacts.
XSpect engine changes
----------------------
* model/run.py, controller/{config_parser,pipeline,batch_manager}.py:
Add an import-time row-range crop for detector keys (`row_range: [start, end]`
in the YAML `detector_keys` block). The slice is applied at HDF5 read time in
both the single-run (`load_run_key_delayed`) and batched (`_load_batch_data`)
paths, so the full detector frame is never materialized. For epix100_0 this
cuts per-batch memory ~5-7x and was necessary to stop the pipeline dying in
silent OOM. The crop origin is recorded on the run (`run._row_offset`) so that
downstream ROIs stay in absolute (full-frame) coordinates.
* analysis/spectroscopy.py:
- reduce_detector_spatial now auto-translates absolute ROI coordinates into
the cropped-array frame using the recorded row_range offset (stripping
_reduced/_ROI_* suffixes so derived keys inherit the parent offset). This
fixes ROIs silently reading out of bounds (Ka summing to zero) after a crop.
- hitfinding gains an absolute `min_sum` mode to reject genuinely dark shots
(sum == 0 after ADU threshold); the old relative median-cutoff mode is kept
as a fallback for signal-dominated data.
* controller/pipeline.py, pipeline_runner.py, __init__.py:
Add a real `logging`-based progress system on the "XSpect" logger, exposed via
`enable_logging(level, log_file=None)` (stderr and/or file). The batched path
now reports per-batch completion via imap_unordered and raises an explicit
OOM-hint error if a worker dies, replacing the previous silent hang. Per-step
debug logging and full tracebacks are emitted from run_pipeline.
mfx102101026 experiment (experiments/mfx102101026/)
---------------------------------------------------
* mfx102101026_static_xes.yaml: static Fe Ka/Kb pipeline on epix100_0
(transpose=true, row_range crop, hitfinding min_sum, ferricyanide ROIs).
* mfx102101026_static_xes_visualization.ipynb: per-run pipeline + diagnostics.
* mfx102101026_aggregate_xes.ipynb: cross-run aggregation from saved results.
* mfx102101026_ferricyanide_reference.ipynb: aggregates runs 9+10, minimum
subtraction then area normalization, saves the ferric (Fe3+) reference.
* mfx102101026_run9_baseline_speciation.ipynb: crop -> baseline -> normalize ->
speciation on the fixed windows (Ka 6380-6410, Kb 7031-7075 eV); baseline
subtraction currently disabled (passthrough) pending a robust routine.
* mfx102101026_speciation_simple.ipynb: non-expert runner. User lists runs;
runs are XSpect-processed only if not already cached (per-run HDF5), then
combined -> min-subtract -> normalize -> speciation vs the ferric reference,
with a difference spectrum and an eLog beamline-summary section (compiles the
figures, labeled by run, into stats/summary/XSpect_speciation via SummaryPost).
* mfx102101026_droplet_visualization.ipynb: droplet sparse-data viewer. Reads
only ROI droplets by streaming contiguous slabs of the first N sparse columns
(avoids the 28 GB full-array read that hung the notebook); documents that
droplet coordinates are in the raw, non-transposed detector frame.
* runs.csv: run bookkeeping; runs 4/5/8 flagged as no-beam, run 9 first 30 Hz.
* results/*.h5: cached per-run and aggregate spectra, ferric reference.
* FeIIICN6_*.dat: exported Ka/Kb reference spectra.
mfx100895324 (experiments/mfx100895324/) and examples/
------------------------------------------------------
Adds the static-XES notebooks, YAML, runs.csv, manifest/README, and saved
results that the mfx102101026 work was derived from, plus a spike-finder
example notebook.
Note: the LUTE smalldata configs and epix common-mode diagnosis for
mfx102101026 live in the experiment data area (not this repo); summary_post.py
lives alongside the notebooks in results/lbgee/ for the same reason.
…r mfx102101026 Adds a repository skill documenting how to stand up a new XES experiment end-to-end, plus the droplet-reconstruction pieces it references. skills/XSpect-setup-XES/SKILL.md - Phased guide (experiment-type triage -> LUTE smalldata pipeline -> geometry discovery -> XSpect YAML -> diagnostic/analysis notebooks -> issue tracking). - Decision questions to classify the analysis: static, time-resolved pump-probe, CCM-scanned/RIXS, or droplet/photon-counted XES. - YAML templates for each type, memory (row_range) guidance, the ePix100 common-mode gotcha, droplet vs droplet2photon distinction, and the LUTE droplet2photon template gap (issue #97). - Guardrails: never invent step names, verify units (ADU vs keV) and the transpose/ROI coordinate frame, confirm beam-on before debugging. - README links to the skill and the YAML pipeline guide. experiments/mfx102101026/ - mfx102101026_droplet_recon_xes.yaml: native photon-counted workflow using the droplet_reconstruction step (ROI applied on sparse arrays) -> union -> sum -> rotate -> reduce, modeled on the working mfx101609126 droplet pipeline. - droplet_visualization.ipynb: Section 7 reconstruction now uses the native XSpect droplet_reconstruction step instead of an external helper script.
…ing conventions Adds Phase 0.5 (Identify the HDF5 keys) and strengthens the guardrails so the skill enumerates datasets and asks the user when a role is ambiguous rather than guessing paths. - Role -> naming conventions: emission = ePix100 (epix100_0/1, epix_0/1 and variants); scattering = epix10k2m / jungfrau16m (usually not the XES signal); I0 monitor = ipm<N>/sum in standard hutches but MfxDg*BmMon/... at MFX; lightStatus masks, timing keys, CCM scan variable, von Hamos geometry PVs. - Ambiguity protocol: 0 candidates -> ask / offer to skip; >1 -> list with shapes and ask; always echo the final key->role mapping for confirmation. - Notes that with two ePix100s (spectrometer + SEER) the agent must ask which is which instead of assuming _0 is the spectrometer.
…ring Zeros low-variance detector pixels via sklearn VarianceThreshold instead of a hand-tuned ADU cutoff. Pixels that barely change across shots (dead, static hot, constant background) are dropped; signal-bearing pixels are kept. Writes back to the same detector key and stores the retained mask as <key>_variance_mask. Handles 3D and 2D detectors. Closes #36
…itecture Brings in the mfx102101026 static + droplet Fe Kα/Kβ XES analysis and the XSpect engine features developed for it: - Import-time detector row_range crop with automatic ROI offset translation (model/run.py, controller/batch_manager.py, analysis/spectroscopy.py) — the memory fix that prevents OOM in batched runs. - hitfinding min_sum absolute mode to reject dark shots. - logging system with enable_logging() and per-batch progress (controller). - experiments/mfx102101026/: static XES YAML, droplet recon YAML, and visualization/aggregate/reference/speciation/droplet notebooks. - experiments/mfx100895324/: predecessor static-XES artifacts. - skills/XSpect-setup-XES/: repository skill for standing up a new XES experiment (LUTE + XSpect YAML + diagnostics), with HDF5 key identification. Run 26 tracking: experiment #98, tracked under #99; open dev items #97, #96, #70, #55, #45.
…ne-architecture # Conflicts: # XSpect_XES_mfx101080524.ipynb
YAML-driven pipeline ar Here's an extended merge description you can paste into the GitHub merge box. Merge YAML-driven pipeline architecture into master (#100) Replaces the monolithic controller with a declarative pipeline. Analysis is defined in a YAML config as an ordered list of steps; a registry maps each step name to a stateless function step(run, **kwargs). Model, controller, and analysis layers are separated so the same steps run across XES, XAS, RIXS, and droplet workflows. Architecture (issue #82, Phase 1A) - Step and reduction registry (XSpect/analysis/registry.py) with decorator registration and lookup by name. - Config parser that reads the data and pipeline sections of a YAML file. - Batch manager with multiprocessing, scalar pre-pipeline pass, max_shots limit, and absolute HDF5 index handling. - Pipeline runner that executes steps in order and writes results onto the run. Registered steps (Phases 1B/1C and time-resolved XES: #85, #86, #87) - Loading: load_run_keys, load_detector (ROI, transpose, combine). - Shot and pixel filtering: filter_shots, filter_detector_adu, filter_detector_variance. - Combination: union_shots, separate_shots. - Spatial and temporal reduction, time binning, CCM binning (3D detector support, NaN handling). - Energy calibration: make_energy_axis, make_ccm_axis. - Normalization, rotation detection (find_rotation_angle), rotate_detector. New capabilities - droplet_reconstruction for sparse photon-counting HDF5 layouts. - patch_pixels with auto-detection (ratio and z-score methods) and vectorized polynomial fitting. - filter_detector_variance: sklearn VarianceThreshold as a data-driven alternative to a hand-tuned ADU cutoff. Zeros pixels whose value barely changes across shots and stores the retained mask as <key>_variance_mask. Closes #36. Examples and docs - YAML configs and analysis notebooks for mfx101080524, mfx101609126, mfxl1027922, xcs101591326, and mfx102101026 beamtimes. - User guide at docs/YAML_PIPELINE_GUIDE.md. - New XSpect-setup-XES skill for HDF5 key identification and detector naming. Testing - Regression suite with saved reference arrays for static and ultrafast XES. - Unit tests for pipeline steps (32 pass locally for the step and spectroscopy suites). - Pre-existing failures in test_xas_steps.py (CCM off-by-one) and test_batch_manager.py are unrelated to this branch and predate the merge. Compatibility and conflict resolution - Old imports still work (from XSpect.XSpect_Analysis import spectroscopy_run). - origin/master (XAS foil-scan and derivative-analyzer work, PR #90) was merged into the branch first. One conflict resolved: the root-level XSpect_XES_mfx101080524.ipynb was deleted per master's reorg; the canonical copy remains at examples/XSpect_XES_mfx101080524.ipynb. XSpect_Analysis.py auto-merged and should get a review pass.chitecture with MVC separation
Quarterly review (2026-04-16 to 2026-07-16) covering the YAML pipeline overhaul, new analysis steps, bug fixes, and LUTE work for discussion. README now describes the model/controller/analysis layers, the step registry pattern, data flow, and per-mode use cases with example configs.
Move the four pipeline-YAML templates out of SKILL.md into templates/, reached by a Phase 3 pointer keyed to the experiment type, so a run reads only the one it needs. Thin the Guardrails section to a failure-mode checklist that points at each rule's canonical phase instead of restating it. Trim the description to identity plus triggers and add the droplet type.
Port the MkDocs Material + mkdocstrings site off the stranded jtb branch onto master and retire the parallel Sphinx/Read the Docs setup, so there is one documentation source. - mkdocs.yml: mkdocstrings pointed at XSpect.model/controller/analysis via new source pages, plus the legacy top-level modules; nav includes the YAML pipeline guide. Drop the logo/favicon refs (files never existed in source). - .github/workflows/docs.yml: build and gh-deploy on push to master (the old workflow only triggered on jtb, so the live site froze at 2025-10-10). Install requirements.txt so mkdocstrings can import the package in CI. - Remove docs/_build, conf.py, *.rst, Makefile/make.bat, docs/requirements.txt, and .readthedocs.yaml. - Fix index.md links (.html -> .md) and a pipeline.py docstring that griffe read as a bogus parameter. Build verified locally. Refs #101
common_mode_correction subtracts per-row, per-column, or per-bank electronic baseline estimated from a signal-free reference band, on 3D detector data before shot reduction, preserving shape. subtract_polynomial_background fits a low-order polynomial to signal-free regions along the spatial axis and subtracts it, writing a non-destructive <key>_bkgsub. peak_mask accepts multiple ranges so several emission lines dispersed on one detector can be masked at once. Per-row weighting excludes NaNs from the fit. Reuses the vectorized weighted-polyfit approach from patch_pixels. Adds unit tests for both steps and documents them in the YAML guide and README step lists. Closes #102
…kgsub Add common_mode_correction and subtract_polynomial_background steps
Hand-written reference for all 28 steps plus combine_runs. One index page (step contract, array-shape vocabulary, master table) and four category pages: loading & filtering, detector & spatial, binning, spectra & reductions. Each entry lists what the step reads, what it writes, and every tunable parameter with its default. Wire a Step reference nav section into mkdocs.yml after the YAML guide; keep source_analysis.md as the source-linked API.
Add detailed step reference docs
Each step and the combine_runs reduction now carries a short YAML snippet showing a typical invocation, so every entry has one.
…gs page Group the guide, a new Example configs page, the step reference, and the source-linked API under one "YAML pipeline" nav section. The example configs page embeds the six analysis-mode YAML files live via snippets, so the docs track the real files in examples/.
Add YAML example to every step reference entry
Nest pipeline docs and add example configs page
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs are out of date compared to master. Need to merge master changes into jtb to build docs.