Hash PII in the TS, Go, Python and .NET SDKs, not just Rust

[brentrager]

Problem

The TypeScript, Go, Python and .NET SDKs scrubbed credentials only — Bearer, password=, token/api_key/secret=, sk-… — while their module docs claimed "PII scrubbing". Emails, phone numbers and street addresses went to the backend untouched. That's the false-advertising half of the bug, and it matters now: telemetry that was previously going nowhere is actually being exported (chat-ws was just wired up), so this data starts landing for real.

Rust fixed this in #82. This brings the other four to parity — with byte-identical output, not just similar behavior.

Solution

Hash, don't drop. [redacted] destroys the ability to ask "is this the same user as that other trace?". A deterministic keyed hash keeps correlation while storing nothing reversible, and the type prefix stays visible so you can still see what kind of value was there:

a@b.com   ->  [email:9f2a41c8]
555-0142  ->  [phone:3b7e0d92]

• HMAC-SHA256, keyed — not a bare digest. Emails and phones are a small enumerable space a rainbow table reverses in seconds.
• Per-org salt — the org id goes into the HMAC message, so identical PII hashes differently across orgs. The kind is in there too, NUL-separated, so values that normalize alike cannot collide.
• Fail closed — no key ⇒ [email:redacted]. Never plaintext, never a hash under a guessable key.
• Credentials matched FIRST and dropped entirely, never hashed — a hash of a live token is still a token oracle, and PII inside a secret (token=a@b.com) is dropped with the secret.
• Normalized by kind — phone → digits, email → lowercase — so (415) 555-0142 and 415-555-0142 correlate.
• Truncated to 8 hex chars.

Key supply

SMOOAI_OBSERVABILITY_PII_HASH_KEY, read at bootstrap in every SDK, plus a per-SDK setter. Set-once: the setters refuse a second key, because rotating it silently forks every correlation already stored. Rotate never.

SDK	Setter	Bootstrap field
TypeScript	setPiiHashKey() (exported from the package entry — the browser bundle has no env, so it must call this explicitly)	BootstrapEnv.piiHashKey
Go	SetPiiHashKey()	BootstrapEnv.PiiHashKey
Python	set_pii_hash_key()	BootstrapEnv.pii_hash_key
.NET	Pii.SetPiiHashKey()	BootstrapEnv.PiiHashKey

Compatibility

The existing org-less signatures keep working unchanged (they hash under the empty org salt). scrubStringForOrg / scrubHeadersForOrg / piiToken are additive. piiToken(kind, raw, orgId) is the search seam: hash a typed query term the same way and match the stored token.

TypeScript: why a hand-rolled HMAC

scrubString is synchronous and runs in both the browser and Node bundles. node:crypto breaks the browser build and WebCrypto's subtle.sign is async-only, so neither can back a sync scrubber. Rather than add a dependency to a published SDK, packages/core/src/hmac-sha256.ts is ~130 lines pinned by the published RFC 4231 and FIPS 180-4 vectors (including the 55/56/64-byte padding boundaries and the million-a long vector).

Verification

Every SDK's suite mirrors the Rust test names case-for-case: same input + same org ⇒ same token; different org ⇒ different token; no key ⇒ redacted and never the raw value; credentials dropped not hashed; type prefix survives; a raw email/phone never appears in output; piiToken matches what scrubbing wrote.

Cross-SDK byte parity. All five SDKs — including Rust — assert the same cross_sdk_parity_vectors, computed independently. If any SDK's message framing, normalization or truncation drifts, exactly one suite breaks.

[email:02ea437f]  email  a@b.com / "A@B.COM "  org-1
[email:fd96f7dc]  email  a@b.com               org-2
[phone:415a9aea]  phone  (415) 555-0142 / 415-555-0142
[address:c5351f4a] address  "1600  Pennsylvania   Ave"

Mutation-checked. Each guard removed one at a time in all four languages — 24 mutations, every one caught by a specific named test, no survivors. Files verified byte-identical after restore.

Mutation	Example test that catches it
email pattern removed	hashes_emails_keeping_the_type_prefix
phone pattern removed	hashes_phone_numbers
address pattern removed	hashes_street_addresses
per-org salt removed	same_value_different_org_hashes_differently + cross_sdk_parity_vectors
credential-drop removed	credentials_are_dropped_not_hashed
no-key fallback removed	no_key_redacts_rather_than_hashing

Gates, run as CI runs them (all exit 0): pnpm typecheck / test (295) / build / format:check (oxfmt); gofmt -l / go vet / go test per-module (80); uvx ruff@0.16.2 check + format --check / pytest (98); dotnet build / test (105) / format --verify-no-changes; cargo fmt --all --check / clippy -D warnings / test --all-features.

Unrelated fixes included

Two pre-existing format failures on main that this PR's CI lanes would otherwise turn red, both verified against a clean tree first: CrashChild.cs whitespace (dotnet format) and a stray blank line in bootstrap.rs (cargo fmt).

A pre-existing flaky test, OtelSetupTests.Setup_IsIdempotent, fails intermittently on clean origin/main under the full-suite run and is unrelated to this change.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 8be7f17

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@smooai/observability	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR