Skip to content

Optimize the aperture photometry operation#129

Open
jnation3406 wants to merge 5 commits into
mainfrom
fix/optimize_aperture_photometry
Open

Optimize the aperture photometry operation#129
jnation3406 wants to merge 5 commits into
mainfrom
fix/optimize_aperture_photometry

Conversation

@jnation3406

Copy link
Copy Markdown
Contributor

The aperture photometry was killing the production pods with OOM when trying to do it with 10 QHY full frame images. This PR is mostly Claude implementing a few optimizations to prevent this:

  1. Decouple measuring and ranking candidate stars so first we measure stars for each image one by one, closing the image when we are done. Then rank all the candidates
  2. Don't upcast to data to float64 in aperture_light_curve.py - keep it as float32
  3. Downsample the diagnostic jpegs created to a max dimension of 1500x1500 - a reasonable size to be displayed on the screen.

I also threw in some optimization of the wcs and source catalog analysis task, to not load the unused pixel data into those tasks.

Claudes description of changes:
aperture_light_curve.py — the core restructure. generate_light_curve now takes local FITS paths instead of eagerly-loaded handlers, and runs in three phases: a metadata pass that validates frames reading only the SCI header and CAT table (no pixels), the candidate catalog build (headers only), then a streamed pixel pass (_measure_frame_pixels) that loads one frame as float32, measures the target plus every candidate on it, captures a downsampled preview, and releases the pixels before the next frame loads. FrameContext no longer has an .image field, and CAT rows are trimmed to the six columns the pipeline actually reads.

comparison_stars.py — the candidate-outer/frame-inner measurement loop (which forced all frames to coexist) is gone. candidate_stars_from_catalog builds candidates up front, the pixel pass measures them frame-by-frame, and select_comparison_stars is now pure math over the collected measurements, with identical drop/rank semantics.

photometry_diagnostics.py — overlays are drawn on a block-mean-downsampled FramePreview (capped at your 2000 px) captured during the pixel pass. That removes the GB-scale full-frame render temporaries and shrinks each frame's base64 JPEG in the operation output (and Redis, and the API response) from tens of MB to well under 1 MB. Small frames like the 80×80 test fixtures are untouched (scale 1.0).

aperture_photometry.py — operate() resolves inputs to file-cache paths via FileCache directly; no more list of eager InputDataHandlers each holding a decompressed frame.

centroiding.py:260 — removed the dtype=float upcast that would otherwise have silently copied the whole frame to float64 on every centroid call (once per candidate per frame).

@jnation3406
jnation3406 requested review from capetillo and sfoale July 15, 2026 05:38

@sfoale sfoale left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done, Claude.

@capetillo capetillo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks okay but I'm confused if the ranking changed. Anyway, good job Jon & Fable

sfoale and others added 4 commits July 16, 2026 06:45
Cache per-frame WCS and aperture geometry for light-curve measurement
…photometry again. There are still further optimizations that can be done, but vectorizing the centroiding stuff, or by using a spatial hash for the cluster matching to only compute cluster angular distance for close clusters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants