feat(api-server): add OpenTelemetry instrumentation for HTTP and gRPC - #141
Closed
JuanmaBM wants to merge 1 commit into
Closed
feat(api-server): add OpenTelemetry instrumentation for HTTP and gRPC#141JuanmaBM wants to merge 1 commit into
JuanmaBM wants to merge 1 commit into
Conversation
Add OTel tracing and metrics to the API server via a new plugin that registers HTTP middleware and gRPC interceptors through the framework's pre-auth hooks. Instrumentation is opt-in: when OTEL_EXPORTER_OTLP_ENDPOINT is unset the plugin is a no-op with zero overhead. - OTel SDK init with TracerProvider + MeterProvider (OTLP gRPC exporters) - HTTP middleware via otelhttp with W3C Trace Context propagation - gRPC unary + streaming interceptors with semantic convention attributes - Graceful shutdown flushes buffered telemetry on SIGTERM - Jaeger all-in-one manifest for Kind (KIND_JAEGER=true) - Spec: specs/platform/api-server-observability.spec.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5 tasks
jsell-rh
added a commit
that referenced
this pull request
Aug 19, 2026
…ase tracing (#158) * [HYPERSHELL-26] docs(specs): add API server observability spec Author the desired state for OpenTelemetry instrumentation of the HyperShell API server (HTTP + gRPC): OTel SDK bootstrap and env-driven configuration, HTTP server spans with templated route names, gRPC server spans, W3C cross-service trace continuation from the web-console BFF, OTLP request metrics, telemetry privacy with operation-id correlation, and development trace export via KIND_JAEGER. This is the API-server counterpart to web-console/tracing.spec.md (HYPERSHELL-27) and closes the trace chain browser -> BFF -> API. It fixes the static-span-name gap observed in the reference draft (PR #141) by requiring method + templated route span names, and aligns with main's reality (Jaeger v2 on 4317, existing operation_id error envelope). Register the spec in the Spec Registry for autonomous reconciliation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(api-server): add OpenTelemetry tracing and metrics plugin Add an opt-in cross-cutting `otel` plugin that instruments the API server's HTTP and gRPC surfaces, implementing HYPERSHELL-26 per specs/platform/api-server-observability.spec.md. - Bootstrap the OTel SDK from standard OTEL_* env vars; enabled only when OTEL_EXPORTER_OTLP_ENDPOINT is set. Trace and metric export go over OTLP/gRPC with a parent-based trace-id-ratio sampler and a W3C TraceContext + Baggage propagator. Setup failures degrade rather than crash, and a bounded shutdown flushes buffered telemetry on signal. - HTTP: one server span per request via otelhttp (registered pre-auth so inbound W3C context is extracted before routing), refined in-router to a templated "METHOD /route" span name with http.route recorded. This closes the static span-name gap from the web-console tracing work. - gRPC: pre-auth unary and stream interceptors that continue an inbound trace from metadata, name spans by full method, record rpc.* semconv attributes and status, and emit an rpc.server.duration histogram. - Record the framework operation id on HTTP spans for trace-to-support correlation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(dev): export API server traces to Jaeger in KIND_JAEGER dev clusters Wire the api-server deployment's OTEL_EXPORTER_OTLP_ENDPOINT to the in-cluster Jaeger OTLP/gRPC endpoint when KIND_JAEGER=true, mirroring the existing BFF wiring, and unset it when Jaeger is disabled. This lets the `otel` plugin export API server traces during local e2e verification. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(api-server): honor OTEL_METRICS_EXPORTER=none to disable metric export Add opt-out support for the standard OTEL_METRICS_EXPORTER=none env var: when set, the plugin exports traces but skips the OTLP metric exporter and meter provider, leaving the global no-op meter (recordRPCDuration then no-ops). A trace-only backend such as Jaeger has no OTLP metrics service and answers metric uploads with "Unimplemented", producing a periodic upload error. The KIND_JAEGER dev wiring now sets OTEL_METRICS_EXPORTER=none on the API server so local trace verification is clean; production points at a full collector that accepts both signals and leaves metrics enabled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(specs): document OTEL_METRICS_EXPORTER in api-server observability spec Record the OTEL_METRICS_EXPORTER=none opt-out in the configuration env table so the spec matches the implemented behavior for trace-only backends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(web-console): accept same-origin browser telemetry via Fetch Metadata The BFF CSRF hook rejected any mutating request whose Origin header did not parse to the Host header, returning 403. Firefox sends `Origin: null` on same-origin beacons under this app's no-referrer policy, so the browser OTLP telemetry POST to /telemetry/v1/traces was rejected and every browser span was dropped. That left BFF traces headless: no browser root span and no hypershell-web-console service in the collector, even though the gateway faithfully preserves the Host header and the browser reports the request as same-origin. Check the browser-set, unforgeable Sec-Fetch-Site header first: allow same-origin and user-initiated (none) requests and reject cross-site. Fall back to the existing Origin/Host comparison for non-browser clients that omit Fetch Metadata. Cross-site attacks remain rejected, preserving the CSRF protection required by WEB-AUTH-02 and WEB-TRACE-02. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(specs): add API-OBS-08 database client spans requirement Extend the API server observability spec with a database-tracing requirement: each query issued while handling a request emits a client span nested under the request span, installed once as a cross-cutting GORM plugin on the base connection (not per data-access object) so plugin queries are traced without per-plugin changes. Span names stay bounded by operation and table; the parameterized statement may be recorded but bound values, credentials, and secrets are excluded per API-OBS-06. Instrumentation is opt-in on the same collector-endpoint presence as the rest of the SDK and inherits the parent-based sampling decision, so an unsampled or telemetry-disabled request produces no database spans. Records the plugin-registry design decision for wiring through the rh-trex-ai SessionFactory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(api-server): trace database queries via otelgorm (API-OBS-08) Register the otelgorm GORM plugin through the framework's new gorm-plugin registry (openshift-online/rh-trex-ai#51) so every database query issued while handling a request produces a client span nested under that request's span. This is a cross-cutting concern applied once on the base connection, transparent to plugin authors, mirroring the HTTP middleware and gRPC interceptors. registerDBTracing runs from the otel plugin init() after setupOTel installs the global TracerProvider, because otelgorm captures the provider when the plugin is constructed. WithoutQueryVariables masks bound parameter values in db.statement so no identifier, secret, or literal is recorded (API-OBS-06); WithoutMetrics keeps this to tracing, leaving database metrics on the framework's existing Prometheus collector (API-OBS-05). Gated on the collector-endpoint presence like the rest of the SDK, so with no endpoint no plugin is registered and the database layer runs with no overhead. Bump rh-trex-ai to the merged registry commit and refresh its dependency-age allowlist entry accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: user <u@example.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
Closing — superseded by #158 which was merged with a more complete implementation (database tracing, templated route span names, metrics opt-out, framework upstream changes). |
Collaborator
Author
|
Already implemented on #158 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
plugins/otelplugin that registers HTTP middleware (otelhttp) and gRPC interceptors through the framework's pre-auth hooksOTEL_EXPORTER_OTLP_ENDPOINTis unset the plugin is a no-op with zero overheadKIND_JAEGER=truespecs/platform/api-server-observability.spec.mdWhat's included
plugins/otel/plugin.gocmd/hypershell/main.godeploy/kind/jaeger.yamlscripts/kind/up.shKIND_JAEGER=truespecs/platform/api-server-observability.spec.mdspecs/index.spec.mdspecs/platform/local-development.spec.mdKIND_JAEGERandJAEGER_VERSIONenv varsEnvironment Variables
OTEL_EXPORTER_OTLP_ENDPOINTOTEL_TRACES_SAMPLER_ARG1.0OTEL_SERVICE_NAMEhypershell-api-serverKIND_JAEGERtrueto deploy Jaeger in KindJAEGER_VERSION2.6Test plan
go build ./...andgo vet ./...pass (verified)KIND_JAEGER=true make kind-upOTEL_EXPORTER_OTLP_ENDPOINTis unset🤖 Generated with Claude Code