Hash PII in the TS, Go, Python and .NET SDKs, not just Rust

.changeset/pii-hashing-polyglot.md

@@ -0,0 +1,50 @@
+---
+'@smooai/observability': minor
+---
+
+TypeScript, Go, Python and .NET: hash PII instead of leaking it, matching the Rust SDK.
+
+All four SDKs scrubbed **credentials only** — `Bearer`, `password=`,
+`token`/`api_key`/`secret=`, `sk-…` — while their module docs claimed "PII
+scrubbing". Emails, phone numbers and street addresses passed through to the
+backend untouched. Rust fixed this in #82; this brings the other four to parity
+with byte-identical output.
+
+Personal identifiers are now **hashed, not dropped**: `a@b.com` →
+`[email:9f2a41c8]`. The type prefix stays visible, so "are these two spans the
+same person?" stays answerable while nothing reversible is stored. The hash is
+**HMAC-SHA256, keyed** — not a bare digest, which a rainbow table reverses in
+seconds for a space as small as email addresses — and the org id is mixed into
+the message, so identical PII hashes differently in different orgs. Phone
+numbers normalize to digits and emails to lowercase before hashing, so
+`(415) 555-0142` and `415-555-0142` correlate.
+
+Credentials are still **dropped**, never hashed, and are matched first: a hash of
+a live token is a token oracle, and PII inside a secret (`token=a@b.com`) goes
+with the secret. With no key configured (`SMOOAI_OBSERVABILITY_PII_HASH_KEY`, or
+the per-SDK setter), personal identifiers are fully redacted (`[email:redacted]`)
+rather than hashed under a guessable one — fail closed, never fail open.
+
+New API, same shape in every SDK. The org-less entry points keep working
+unchanged (they hash under the empty org salt):
+
+- TypeScript: `setPiiHashKey`, `piiToken`, `scrubStringForOrg`,
+  `scrubHeadersForOrg`, `PiiKind` — now exported from the package entry
+- Go: `SetPiiHashKey`, `PiiToken`, `ScrubStringForOrg`, `ScrubHeadersForOrg`,
+  `PiiKind`, `BootstrapEnv.PiiHashKey`
+- Python: `set_pii_hash_key`, `pii_token`, `scrub_string_for_org`,
+  `scrub_headers_for_org`, `PiiKind`, `BootstrapEnv.pii_hash_key`
+- .NET: `Pii.SetPiiHashKey`, `Pii.PiiToken`, `Pii.ScrubStringForOrg`,
+  `Pii.ScrubHeadersForOrg`, `PiiKind`, `BootstrapEnv.PiiHashKey`
+
+`piiToken(kind, raw, orgId)` is the search seam: hash a typed query term the same
+way and match the stored token.
+
+⚠️ **The key is load-bearing — rotate never.** Rotating it silently forks
+correlation with every hash already stored. Supply it once at startup; the
+setters are set-once and refuse a second key.
+
+The TypeScript SDK ships a small synchronous SHA-256/HMAC (`hmac-sha256.ts`)
+rather than taking a dependency: `scrubString` is sync and runs in the browser
+bundle, where `node:crypto` is unavailable and WebCrypto is async-only. It is
+pinned by the RFC 4231 and FIPS 180-4 vectors.

README.md

@@ -34,15 +34,15 @@
 - 🗺️ **Source maps** — uploaded to S3 at build time, applied lazily on view
 - 🚪 **Beacon flush** — events queued at `pagehide` ship via `navigator.sendBeacon`
 - 💾 **Offline queue** — events captured while offline persist in `IndexedDB` and retry on focus
-- 🔐 **PII scrub** — `password`, `token`, `Bearer ...`, and friends are redacted before transport
+- 🔐 **PII scrub** — credentials (`password`, `token`, `Bearer ...`) are dropped; emails / phones / addresses are HMAC-hashed per-org (`a@b.com` → `[email:9f2a41c8]`) so traces stay correlatable without storing the value
 
 **Node**
 
 - 🛑 **`uncaughtException` + `unhandledRejection`** with full stack
 - 🪢 **Hono middleware** — captures errors propagating to the global `onError` handler
 - 🧠 **AsyncLocalStorage scope** — per-request user, tags, breadcrumbs without leaking across requests
 - 📦 **Batched transport** — `undici` with retry / backoff
-- 🔐 **Same PII scrub policy** as the browser
+- 🔐 **Same PII scrub policy** as the browser — key from `SMOOAI_OBSERVABILITY_PII_HASH_KEY`
 
 **React / Next.js**
 
@@ -55,7 +55,8 @@
 
 - `console.log` / `console.info` / `console.warn` — only `console.error` is tapped, and that's opt-out
 - HTTP request **bodies** — only method, path, status, and duration appear in breadcrumbs
-- Anything matching the PII scrub regex unless you explicitly allowlist it
+- Credentials matching the PII scrub regex — dropped outright, never hashed
+- Raw emails / phones / street addresses — replaced by a keyed per-org hash, never stored in the clear
 
 ## 📦 Install
 
@@ -200,7 +201,7 @@ Known divergences: TypeScript, Python, Go and .NET emit `gen_ai.tool.names` as a
 
 ## 📖 Architecture
 
-The SDK is intentionally thin. It captures, batches, redacts PII, and POSTs to a Smoo ingest endpoint. All of the heavy lifting — fingerprint grouping, source-map symbolication, dashboards, alerts, retention — lives in the Smoo platform.
+The SDK is intentionally thin. It captures, batches, redacts credentials, hashes personal identifiers, and POSTs to a Smoo ingest endpoint. All of the heavy lifting — fingerprint grouping, source-map symbolication, dashboards, alerts, retention — lives in the Smoo platform.
 
 ```mermaid
 %%{init: {'theme':'base','themeVariables':{
@@ -236,7 +237,7 @@ This SDK is opinionated about privacy:
 - We never capture form bodies, request bodies, or response bodies by default
 - We never capture cookies
 - We never send anything to a third-party service — your events go to **your** Smoo backend only
-- PII scrubbing is enabled by default and can be tuned per-tenant
+- PII scrubbing is enabled by default and can be tuned per-tenant. Personal identifiers are hashed with HMAC-SHA256 under a key you supply (`SMOOAI_OBSERVABILITY_PII_HASH_KEY`), salted by org id — identical across the TypeScript, Rust, Go, Python and .NET SDKs. **With no key configured they are fully redacted, never hashed under a guessable one.**
 
 ## 📖 Status
 

dotnet/src/SmooAI.Observability/Bootstrap.cs

@@ -47,6 +47,13 @@ public sealed class BootstrapEnv
 
     /// <summary>Skip bootstrap entirely.</summary>
     public bool? Disabled { get; set; }
+
+    /// <summary>
+    /// HMAC key used to hash emails / phones / addresses in scrubbed strings
+    /// (see <see cref="Pii"/>). Unset means personal identifiers are fully
+    /// redacted rather than hashed.
+    /// </summary>
+    public string? PiiHashKey { get; set; }
 }
 
 /// <summary>
@@ -109,6 +116,11 @@ public static async Task<BootstrapResult> Run(BootstrapEnv? overrides = null)
 
         var env = ResolveEnv(overrides);
 
+        // Before anything can emit: a scrubbed string written without this key
+        // redacts PII outright, so installing it late would silently produce a
+        // window of uncorrelatable spans rather than an error.
+        Pii.SetPiiHashKey(env.PiiHashKey);
+
         if (env.Disabled == true)
         {
             return Cache(new BootstrapResult { Installed = false, Exporting = false, Otel = null });
@@ -242,6 +254,7 @@ internal static BootstrapEnv ResolveEnv(BootstrapEnv? overrides)
             Release = overrides?.Release ?? Env("SMOOAI_OBSERVABILITY_RELEASE") ?? Env("GIT_SHA") ?? Env("LAMBDA_FUNCTION_VERSION") ?? "dev",
             Dsn = overrides?.Dsn ?? Env("SMOOAI_OBSERVABILITY_DSN") ?? Env("OBSERVABILITY_DSN"),
             Disabled = overrides?.Disabled ?? Truthy(Env("SMOOAI_OBSERVABILITY_DISABLED")),
+            PiiHashKey = overrides?.PiiHashKey ?? Env("SMOOAI_OBSERVABILITY_PII_HASH_KEY"),
         };
     }