[essimaging] Orca normalization by proton charge per frame by nvaytet · Pull Request #663 · scipp/ess

nvaytet · 2026-07-01T19:32:37Z

We change the way we use the proton charge to normalize the Orca data.

We normalize each frame in the Dark, OpenBeam, and Sample runs by the corresponding proton charge.
After that we take the mean of the Dark along the time dimension.
We then remove the MeanDark from the OpenBeam and Sample data.
Finally, we divide the (Sample - MeanDark) / (OpenBeam - MeanDark).mean('time')

Fixes #636

…both sample and open beam, then divide by open beam

…at covers the whole experiment range

nvaytet · 2026-07-01T19:36:34Z

-        "ymir_lego_dark_run.hdf": "md5:c0ed089dd7663986042e29fb47514130",
-        "ymir_lego_openbeam_run.hdf": "md5:00375becd54d2ed3be096413dc30883c",
-        "ymir_lego_sample_run.hdf": "md5:ae56a335cf3d4e87ef090ec4e51da69c",
+        "ymir_lego_dark_run.hdf": "md5:2d9ae7d6f1d0502c17b0030bcb3dad1d",


I had to update the data because the proton charge in the old files started exactly at the start of the time axis for the camera data. However, when doing sc.lookup in 'previous' mode, the first data point was not found (probably because of numerical precision errors) and a NaN was returned.

So the first data slice in the dark lookup was NaN. We then took the mean which gave NaN, and then all data was NaN.
I now updated the proton charge logs to start a bit before the data starts.

jl-wynen · 2026-07-03T05:29:15Z

+    # not the same time as the proton charge (which is when the protons hit the
+    # target). We need to shift the proton charge time to account for the time it takes
+    # for neutrons to travel from the target to the detector. Does this mean we cannot
+    # do the normalization without computing time of flight?


Yes, I think that is what this implies. Or is ODIN short enough that we can just use the 'time' coord from the nexus data?

What is the 'time' coord in data here? Is that the time at the source?

The comment here was already present in the code (in the _compute_proton_charge_per_exposure function). This is not new, and I still have not figured out how to resolve this.
Fixing it should not be part of this PR.

nvaytet · 2026-07-03T09:27:32Z

 def subtract_background_openbeam(
    data: FluxNormalizedDetector[OpenBeamRun],
-    background: FluxNormalizedDetector[DarkBackgroundRun],
+    background: MeanDarkFrame,


Note: Should we subtract the mean dark from each OB frame or should we subtract it from the mean OB? Are the resulting variances the same?

It's also a question about the scales of float values. Are there significant truncation errors from subtracting the (large?) mean dark from each individual OB frame?

I think we can safely assume that counts in the dark frame will always be lower than in open beam and sample. Open beam will probably have the highest counts.

But is sum(open_beam) >> sum(dark_frame)? In that case, you might get better accuracy by subtracting frame by frame.

I also think we expect the dark frame to be more stable than the open beam. The dark is the inherent noise of the chip which is always there. The open beam can fluctuate due to external factors. So I would vote to subtract the dark from each OB frame and then take the mean OB after that.

nvaytet · 2026-07-03T09:28:28Z



+def average_dark_frames(
+    dark_frames: FluxNormalizedDetector[DarkBackgroundRun],


Should we actually normalize the dark measurement by proton charge? If it never sees the neutrons, we could be introducing fluctuations that aren't actually there?

If there is background from e.g. a neighouring instrument bleeding in through the cave walls, then fluctuations could be related to the proton charge after all?

-> assume to begin with that we don't need to normalize by proton charge but keep the option open to add it back in later.

Sounds to me like you don't want to normalise by proton charge but by measurement duration. You need something that corresponds to an event rate, not a count. The proton charge norm does this but it depends on the source. So if you would measure a dark frame while the source is off, it wouldn't count under proton charge norm. But that makes no sense.

For the measurement duration, the conclusion was that all 3 measurements will be recorded with the same frame rate (fps). So every frame has the exact same duration. That duration cancels out in the normalization so we actually don't need the exposure time of each frame.

For my understanding: Does this mean that every frame is normalised by the duration such that sum(dark_frame) does not depend on the number of dark frames recorded?

We don't take the sum of dark frames, we take the mean. The same for the open beam.
I think as long as every frame has been normalized by the corresponding proton charge, and everything is in units of counts/C (or something like that), then this should work fine?

Sounds ok. But I don't know the underlying equation you're solving here.

So if we decide not to normalize the dark frames by the proton charge, we have a units issue.
The sample frame is in counts/C while the dark frame is just counts.

I think this implies that we need to subtract the dark frame first (as it should never depend on proton charge) and then normalize the dark-subtracted sample and OB runs by proton charge?

jl-wynen · 2026-07-09T13:14:58Z

The latest changes look good. But what about #663 (comment) ?

nvaytet · 2026-07-09T13:18:34Z

The latest changes look good. But what about #663 (comment) ?

See reply

nvaytet added 5 commits June 22, 2026 16:57

normalize each frame by proton charge, subtract mean dark frame from …

ce8d233

…both sample and open beam, then divide by open beam

use mode nearest

43aa61f

Merge branch 'main' into orca-normalization-fix

0acc532

fix unit tests and udpate data files for docs with a proton charge th…

2a5f64f

…at covers the whole experiment range

update notebook to create ymir images

62c33d5

github-actions Bot added the essimaging Issues for essimaging. label Jul 1, 2026

nvaytet commented Jul 1, 2026

View reviewed changes

nvaytet requested review from YooSunYoung and jl-wynen July 1, 2026 19:37

nvaytet added 3 commits July 2, 2026 23:38

finish removing ExposureTime

48d192c

remove commented code in notebook

5f724a3

restore pixi lock file

88c46ed

jl-wynen reviewed Jul 3, 2026

View reviewed changes

nvaytet commented Jul 3, 2026

View reviewed changes

nvaytet and others added 3 commits July 9, 2026 13:41

subtract dark frames before normalizing to proton charge

242c393

update tests

373283f

Merge branch 'main' into orca-normalization-fix

2aa33b6

jl-wynen approved these changes Jul 9, 2026

View reviewed changes

nvaytet added this pull request to the merge queue Jul 9, 2026

Merged via the queue into main with commit aaf3a93 Jul 9, 2026
6 checks passed

nvaytet deleted the orca-normalization-fix branch July 9, 2026 13:39



		def average_dark_frames(
		dark_frames: FluxNormalizedDetector[DarkBackgroundRun],

Uh oh!

Conversation

nvaytet commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jl-wynen commented Jul 9, 2026

Uh oh!

nvaytet commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nvaytet commented Jul 1, 2026 •

edited

Loading