Aij/photometry by capetillo · Pull Request #123 · LCOGT/ptr_datalab

capetillo · 2026-06-02T23:41:06Z

FEATURE: Aperture Photometry Operation

Background:
Rachel requested aperture photometry as an operation. In adding this operation, centroiding was refactored.

Implementation:

Adds an aperture photometry data operation that generates light curve rows, comp star metadata, diagnostics, and diagnostic JPEG for each input file with overlays where the target and comp stars are circled
Aperture photometry logic is broken up into utils for light curve generation, comp star selection, photometry logic, geometry, and diagnostics
Ranks comp stars using brightness similarity and proximity to the target (following AstroImageJ's weighting logic)
Refactors centroiding logic into a shared utility so both the centroiding endpoint and aperture photometry use the same centroiding logic

…otometry file

…rture_photometry to use the logic instead

jnation3406

The only things that must be changed are removing fits_utils.py and using the already opened header/data in the InputDataHandler, and then some code reuse/removal stuff from the AI. More docstring comments would be nice, even if they are AI generated. I've also added some AI suggestions to some of the math heavy functions to simplify them and speed them up using numpy for math vectorization but you can ignore that if you are not interested.

jnation3406 · 2026-06-05T18:55:14Z

@@ -0,0 +1,170 @@
+from __future__ import annotations


This isn't part of the main code path right? Just for your testing? If that is the case, I think you should put it in a separate subpackage like diagnostics or scripts or something.

jnation3406 · 2026-06-05T22:12:50Z

@@ -0,0 +1,53 @@
+from typing import Any, Mapping, Sequence


This file should not exist, we already have this functionality elsewhere. Instead of passing around the InputDataHander.fits_file, pass the data handler itself. It contains an already open SCI header in .sci_hdu or SCI data in .sci_data. And you can also get the cat data out of the open file there with .get_hdu('CAT').data

jnation3406 · 2026-06-05T22:17:15Z

+log = logging.getLogger()
+log.setLevel(logging.INFO)
+
+PIXELCENTER = 0.5


I would call this HALF_PIXEL

jnation3406 · 2026-06-05T22:25:47Z

+  return source_max
+
+## Fits a plane to the given points using least squares. Returns None if the points are degenerate.
+def _fit_plane(points: list[tuple[float, float, float]]) -> PlaneModel | None:


Could just use numpy functions - pass in a numpy array of points instead:

def _fit_plane(points: list[tuple[float, float, float]]) -> PlaneModel | None: if len(points) < 4: return None pts = np.asarray(points, dtype=float) x, y, z = pts[:, 0], pts[:, 1], pts[:, 2] design = np.column_stack((np.ones_like(x), x, y)) coeffs, _residuals, rank, _sv = np.linalg.lstsq(design, z, rcond=None) if rank < 3: # degenerate / collinear -> matches the old pivot check return None return PlaneModel(*coeffs)

jnation3406 · 2026-06-06T00:04:35Z

+    return float(ra), float(dec)
+
+
+def _header_float(header: Mapping[str, Any], keys: tuple[str, ...], default: float) -> float:


This same function is defined in comparison_stars.py as well

jnation3406 · 2026-06-06T00:04:55Z

+    pass
+
+
+def _world_to_pixel(header: Mapping[str, Any], ra_deg: float, dec_deg: float) -> tuple[float, float]:


This same function is defined in comparison_stars.py as well

jnation3406 · 2026-06-06T00:07:20Z

+        raise LightCurveError("comparison_strategy must be 'auto' or 'variability_first'.")
+
+
+def _load_and_validate_frames(fits_paths: Sequence[str]) -> list[FrameContext]:


The image data already exists and is open in the InputDataHandler, as does the SCI header. Use those so we don't open them twice. You can also get the CAT header from it like I've described in another comment. This function is only used in one place so maybe doesn't need to be a function once its simplified.

jnation3406 · 2026-06-06T00:10:28Z

+    return frames
+
+
+def _parse_date_obs(value: Any, fits_path: str) -> datetime:


If you need to add date parsing I would just use dateutil.parse from the dateutil library. It covers many corner cases.

jnation3406 · 2026-06-06T00:20:19Z

+                log.warning(f"rejected comparison candidate in {frame.fits_path}: {exc}")
+        rejected_for_target = 0
+        rejected_for_edge = 0
+        for row in rows:


I feel like this function, if it is a bottleneck, could be improved with numpy vectorization by pre-calculating the angular_distance for all rows and filtering them in/out by proximity to target with masks and stuff.

This is definitely worth considering when we've got the algorithm stable - but maybe this should be its own later PR.

…inputdatahandler instead of opening them twice. and uses numpy vectorization to build field star catalog

…pixel

jnation3406

Looks good, but I see a few of my original comments were never changed.

jnation3406 · 2026-06-12T20:46:44Z

+    return brightness_weight * norm_brightness + distance_weight * norm_distance
+
+
+_astroimagej_comparison_weight = astroimagej_comparison_weight


I still don't understand this

markBowman · 2026-06-15T20:43:06Z

+class CentroidResult:
+  x: float
+  y: float
+  background: float


the background and peak are fields of the BackgroundModel so don't need to be included in the CentroidResult twice.

markBowman · 2026-06-15T21:08:10Z

+
+    iteration -= 1
+
+  return CentroidResult(


It's easier to see here that you are including information twice

markBowman · 2026-06-15T21:19:43Z

+        "net_source_counts": net_source,
+        "source_uncertainty": source_uncertainty,
+        "mean_background_per_pixel": mean_background_per_pixel,
+        "peak_pixel_value": background_model.source_peak,


Is this right? I would assume peak pixel is for the target not the background model

Or maybe we have information in the background model that does not belong

… and make calibration fixes. This commit refines the aperture photometry feature: - Target measurement never drops a frame: on centroid failure or a recenter drift beyond ~6px from the WCS position, measure at the WCS position (with the background re-estimated there). Keeps supernovae embedded in host-galaxy light. - Per-frame arcsec aperture option: radii can be given in arcsec and are converted to pixels from each frame's plate scale, so a fixed on-sky aperture works across mixed telescopes and instruments. - Comparison stars: require presence in >=80% of frames (not all) so heterogeneous multi-telescope sets still yield an ensemble; rank candidates by their own measured instrumental magnitude against the measured target (shared zero point). Drop inconsistent (e.g. blended) catalog matches before they bias the calibration. - Parametrise background star rejection convergence for potential future use. Further PR Review cleanups: remove unused imports, extract float-coercion helper, drop the redundant CentroidResult.background field, convert leftover comment blocks to docstrings, delete stray debug JPEGs from test_outputs. Reproduces the NGC 7331 SN rp light curve to <=0.016 mag (pixel aperture) / <=0.0045 mag (arcsec) across single- and multi-telescope frames. Confirmed that aperture-photometry, utils, and analysis test suites pass.

- Reuse comparison-star measurements from ranking instead of re-measuring - Guard zero target counts in calibration (was a ZeroDivisionError) - Remove the unused comparison_strategy parameter - Refactor gain/read-noise header lookups into fits_metadata helpers

Copilot

Pull request overview

Adds an Aperture Photometry feature to the DataLab session pipeline, including a new data operation that produces calibrated light-curve rows, comparison-star metadata, and per-frame diagnostics/overlay JPEGs, while refactoring centroiding into shared utilities.

Changes:

Introduces an aperture photometry light-curve generation pipeline (target measurement, comparison-star selection, calibration, diagnostics).
Refactors centroiding into a shared utils.centroiding module and updates the existing centroiding analysis endpoint to use it.
Adds new utilities (photometry, geometry, FITS metadata helpers) plus extensive unit/integration tests for the new behavior.

Reviewed changes

Copilot reviewed 13 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
datalab/datalab_session/utils/photometry.py	Adds aperture photometry measurement + fractional pixel overlap helper.
datalab/datalab_session/utils/photometry_diagnostics.py	Adds diagnostic overlay JPEG generation and comparison-star validation diagnostics.
datalab/datalab_session/utils/geometry.py	Adds pixel/sky distance helpers used by comparison selection and clustering.
datalab/datalab_session/utils/fits_metadata.py	Adds WCS/header helpers and aperture unit scaling (px vs arcsec).
datalab/datalab_session/utils/comparison_stars.py	Adds comparison-star measurement and selection/ranking logic.
datalab/datalab_session/utils/centroiding.py	Adds shared centroiding + background model implementation for reuse.
datalab/datalab_session/utils/aperture_light_curve.py	Implements end-to-end aperture photometry light-curve pipeline + diagnostics output.
datalab/datalab_session/tests/test_utils.py	Adds unit tests for new geometry/photometry/plane-fit helpers.
datalab/datalab_session/tests/test_operations.py	Adds operation-level tests for the new Aperture Photometry operation wiring.
datalab/datalab_session/tests/test_aperture_photometry.py	Adds comprehensive pipeline tests (synthetic frames + real FITS fixtures).
datalab/datalab_session/tests/test_analysis.py	Updates centroiding tests for the refactored centroiding result shape.
datalab/datalab_session/data_operations/aperture_photometry.py	Adds new data operation that runs the pipeline and formats output for the frontend.
datalab/datalab_session/analysis/centroiding.py	Switches analysis endpoint to call the shared centroiding utility.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sfoale · 2026-07-06T13:20:23Z

+    """
+        Builds comp star candidates from the source catalogs across valid frames.
+
+        Returns candidates that are present in all frames and are not too close to the target or the edge of the image.
+    """


sfoale · 2026-07-06T13:20:37Z

+                "second_hdu_magnitude": float(np.median(np.asarray(cluster["mags"], dtype=float))),
+                "source_catalog_by_frame": dict(cluster["source_catalog_by_frame"]),
+                "frame_coverage": len(cluster["frame_paths"]),
+                "isolation_px": isolation,


sfoale · 2026-07-06T13:20:50Z

+        self.set_output(output, is_raw=True)
+        self.set_operation_progress(AperturePhotometry.PROGRESS_STEPS['OUTPUT_PERCENTAGE_COMPLETION'])
+        self.set_status('COMPLETED')
+        log.info(f"Aperture Photometry output: {output}")


sfoale · 2026-07-06T13:20:56Z

+    bck_cnt = float(max(int(background_model.effective_pixels), 1))
+    if background_model.effective_pixels <= 0.0:
+        raise error_class("Background annulus does not contain any valid pixels.")
+


sfoale · 2026-07-06T13:22:21Z

+    if background_model.effective_pixels <= 0.0:
+        raise error_class("Background annulus does not contain any valid pixels.")
+
+    mean_background_per_pixel = max(background_model.mean, 0.0)


documented only

- Rename isolation_px -> isolation_arcsec (field held arcsec) - Guard non-positive/non-finite gain in measure_aperture - Log an output summary instead of the full payload - Fix stale coverage docstring, document constant background, typo

capetillo added 10 commits May 5, 2026 11:16

adds pipeline and tests

78f245d

updates tests and module

9013fa2

adds aperture photometry and test. still have to clean up aperture ph…

433ff27

…otometry file

adds aij way of weighting

59d568d

Refactors centroiding

982b0eb

adds aperture photometry operation and utils

731b147

adds logic to find comparison stars and rank them like AIJ does

6a6d8c8

adds utils functions necessary to perform aperture photometry

1c155c6

adds tests and updates them

e54c67c

deletes unused pipeline that was used for testing

ed519ca

capetillo requested review from jnation3406 and markBowman June 2, 2026 23:48

capetillo added 3 commits June 3, 2026 12:26

deletes serialization and wcs_utils and updates tests, utils, and ape…

30bc4c6

…rture_photometry to use the logic instead

updates colors of circles in jpeg output and flips jpeg on its y axis

99bf5d1

removes black text

f471bbc

jnation3406 requested changes Jun 6, 2026

View reviewed changes

capetillo added 7 commits June 11, 2026 09:04

applies Jon's feedback

b6c4d5e

uses np functions

416f743

removes repeated functions--> moved to fits_utils. opens images with …

fb0c3cc

…inputdatahandler instead of opening them twice. and uses numpy vectorization to build field star catalog

removes unused code

774811b

uses centroiding from utils to remove background

00761bb

updates tests

1965e41

deletes fits_utils file and adds fits_metadata utils to use world to …

7d3f3a8

…pixel

capetillo requested a review from jnation3406 June 11, 2026 16:36

jnation3406 approved these changes Jun 12, 2026

View reviewed changes

markBowman reviewed Jun 15, 2026

View reviewed changes

capetillo added 2 commits June 22, 2026 17:26

removes source peak and peak from background model

7911230

updates tests

fc00309

capetillo and others added 5 commits June 23, 2026 15:54

Adds pydocs

693678f

vectorizes generate_light_curve() and updates test

52aa41a

Improve comments etc

6f628c7

sfoale requested a review from Copilot July 6, 2026 12:21

Copilot started reviewing on behalf of sfoale July 6, 2026 12:21 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Address Copilot review:

6041d60

- Rename isolation_px -> isolation_arcsec (field held arcsec) - Guard non-positive/non-finite gain in measure_aperture - Log an output summary instead of the full payload - Fix stale coverage docstring, document constant background, typo

capetillo merged commit 0687e1a into main Jul 7, 2026
3 checks passed

		return float(ra), float(dec)


		def _header_float(header: Mapping[str, Any], keys: tuple[str, ...], default: float) -> float:

		pass


		def _world_to_pixel(header: Mapping[str, Any], ra_deg: float, dec_deg: float) -> tuple[float, float]:

		raise LightCurveError("comparison_strategy must be 'auto' or 'variability_first'.")


		def _load_and_validate_frames(fits_paths: Sequence[str]) -> list[FrameContext]:

		return frames


		def _parse_date_obs(value: Any, fits_path: str) -> datetime:

		return brightness_weight * norm_brightness + distance_weight * norm_distance


		_astroimagej_comparison_weight = astroimagej_comparison_weight

Uh oh!

Conversation

capetillo commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

FEATURE: Aperture Photometry Operation

Uh oh!

jnation3406 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnation3406 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

capetillo commented Jun 2, 2026 •

edited

Loading