feat(observability): ordered structured log-level filtering (Loki detected_level + pino map) - #3
Merged
Merged
Conversation
… + pino mapping Replace the cross-format substring-regex Min-level stopgap in the Logs (Loki) dashboard (uid solid-stats-logs) with a real ordered level filter. The dropdown now emits LogQL label filters over the level / detected_level fields instead of matching message substrings, so an info line containing the word "error" no longer matches Warning+. Loki 3.6.x already attaches a query-time detected_level for the text-format apps (rabbitmq/postgres/glitchtip), but maps server-2's pino NUMERIC "level":30/40/50 to detected_level: unknown, hiding our main app under any level filter. Add an Alloy pipeline stage (80-alloy.yaml) scoped to app="server-2" that parses the pino JSON level, maps the numeric severity to a normalized text level (info/warn/error/fatal), and exposes it as STRUCTURED METADATA named level — not a new stream label, so the Phase 15 5-key label allowlist and stream cardinality are unchanged. The dashboard filter ORs both fields (| level=~... or detected_level=~...) so a single Min-level dropdown works across all apps. Closes the logs-level-filtering-todo brief (option 1 detected_level + option 2 Alloy mapping fallback for pino). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the substring-regex stopgap
Min levelfilter in the Logs (Loki) dashboard with a real ordered level, and makes server-2 (pino numeric levels) filterable instead of resolving todetected_level: unknown.Change
k8s/observability/61-grafana-logs-dashboard.yaml:Min levelvariable now emits real LogQL label filters (| level=~"warn|error|fatal|critical" or detected_level=~"..."), panel exprs switched from|~ $levelsubstring to$level. "All" = empty.k8s/observability/80-alloy.yaml: inloki.processfor{app="server-2"}, parse the pino JSONlevel, map numeric severity (10/20/30/40/50/60 → trace/debug/info/warn/error/fatal) and expose it as structured metadatalevel(not a label — preserves the 5-key allowlist + stream cardinality).Validation
solid-stats-logs),kubectl apply --dry-run=clientOK. Alloy/LogQL syntax cross-checked against current docs (stage.template,stage.structured_metadata, LogQLorlabel filters).Activation (staging, manual)
Caveat: structured-metadata
levelattaches only to logs ingested after Alloy reloads; historical server-2 lines stayunknown.🤖 Generated with Claude Code