42 update tom alertstreams for lsst#52
Draft
phycodurus wants to merge 33 commits into
Draft
Conversation
The FIFOQueueMixin class turns the Model's db table into a FIFO queue of the most recent alerts. The Alert model is just a generic data structure for an alert from a stream. Register new model with admin pages
Uses integration points to add URL patterns and navbar item. Defines a basic HTMX table View for the Alerts model.
Adds pydanic dependency to pyproject.toml. Defines a normalization mechanism for alerts. Adds a handler that works for any properly implemented and configured AlertStream subclass, demonstrating normalization mechanism. (`save_alert_to_database` adds alerts to the models.Alert(FIFOQueueMixin) db table.
this introduces the AlertStreamPresenter class and it's stream-specific subclasses. This allows the AlertTable to create links to <objectID>s and <alertID>s as appropriate for each alert stream. (And the AlertStream class itself doesn't have to know about any of that).
This had to do with creating links to display in an alert table that is separate from reading the stream, which is what these classes do. (Single Responsibilty Principle ftw).
LSST streams need this field to distinguish nanojansky fluxesfrom ZTF magnitudes
- AntaresAlertStream: abstract base implements shared listen() with StreamingClient using context manager- AntaresZtfAlertStream: extracts newest_alert_magnitude, ztf_object_id, newest_alert_observation_time (MJD)- AntaresLsstAlertStream: forward-compatible stub for LSST properties (survey.lsst.dia_object_id); inactive until LSST topics are confirmed - AntaresMockAlertStream: generates fake alerts for demos without credentials (This is just for Recent Alerts table testing). Fixes normalize_alert to use correct property keys discovered from a live ZTF locus (newest_alert_magnitude, not the nonexistent ztf_magpsf).
Replaces the AmpelAlertStream stub with a three-class hierarchy: - AmpelMockAlertStream: renamed from AmpelAlertStream, generates fake alerts while real AMPEL packages cannot be installed (pymongo bson conflict with antares_client) - AmpelZtfAlertStream: consumes ZTF alerts from UW Kafka broker via ampel-ztf ZiAlertSupplier/UWAlertLoader stack - AmpelLsstAlertStream: consumes LSST alerts via ampel-lsst LSSTAlertSupplier/KafkaAlertLoader with SASL_SSL auth Real AMPEL imports are commented out due to the bson conflict -- the classes are structurally complete but untested against live data. Also adds LsstPresenter to tables.py for RSP deep-link URLs on ampel-lsst alerts.
Test coverage: - FIFOQueueMixin partition truncation and global limit - NormalizedAlert Pydantic validation (required fields, optional fields) - save_alert_to_database handler (success and error paths) - RecentAlertsView GET (200, stream dashboard context, empty state) - URL auto-registration via AppConfig.include_url_paths - AlertStreamConfig and AlertStream base class validation
There are differences (e.g. ZTF has magnitude, while LSST has flux), but those can be handled in normalize_alerts() when we examine an LSST alert. (This commit as code in there to do the examination that will be removed later).
15 tasks
15 tasks
15 tasks
combinations of specified fields define what contributes to the FIFO_MAX limit. So, with (stream+topic), for a given Alert stream, the number of alerts within a topic count toward FIFO_MAX (and each topic gets it's own budget).
- `observation_time` is the time of the observation that the alert is about. - `published_time` - is when the broker published the alert. - `created` is when the alert was received and saved by the consumer.
Google Pub/Sub streams are configured slightly differently than kafka streams in that a subscription binds one topic only. So, there's one config dict for each topic. This confuses the way we instrospect the streams for the AlertFilterSet choices: we see a Pub/Sub broker one for each of it's topics, but we only want to list it once (it's one broker). This removes the duplicate choices.
in addition to General search, populate a topics drop-down according to the stream that is selected in the stream drop-down.
just put general search, stream, and topic filters on the same line. Override the `form()` method to do this.
with mock alerts.
This branch is now archival. We're about to remove the Recent Alerts page and associated functionality from tom_alertstreams and refactor it into a new repo and INSTALLED_APP: tom_lsst_alertstreams. This keeps tom_alertstream dependecies to a minimum, separates the alert ingestion examples, and allows TOM developers to incorporate what they need from there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.