Feature/semantic type classification

[IvannKurchenko]

Changes

Introduce opt-in semantic-aware profiling in DQProfiler. A lightweight classification stage runs between metric collection and check generation, so each column receives one consistent family of checks instead of contradictory
overlaps (e.g. vehicle_type no longer gets both is_in([...]) and min_max(...)).

The feature is fully backward compatible: when a DQProfiler is constructed without a semantic_registry, no detection runs and the generated profiles are byte-identical to today.

New public surface (all in src/databricks/labs/dqx/profiler/semantic.py):

• DQSemanticType — Pydantic v2 model naming a column's semantic meaning (e.g. enum, key, measurement, text) plus optional properties.
• DQSemanticTypeDetector — named callable (DQProfileContext) -> DQSemanticType | None.
• DQProfileContext — frozen context passed to detectors and contextual builders. Carries df, column_name, column_type, metrics, options, metadata, and semantic_type.
• SemanticRegistry — immutable, name-unique, ordered chain of detectors. Provides default(), prepend(...), and replace([...]), all returning new instances (uniqueness enforced via a model_validator).
• default_semantic_detectors() and four built-in detectors: DEFAULT_ENUM_DETECTOR, DEFAULT_KEY_DETECTOR, DEFAULT_MEASUREMENT_DETECTOR, DEFAULT_TEXT_DETECTOR plus threshold constants (ENUM_MAX_CARDINALITY_RATIO, KEY_MIN_DENSITY_RATIO, KEY_MIN_LENGTH_STABILITY_RATIO).

Profiler wiring (profiler/profiler.py, profiler/profile_builder.py,profiler/profile.py):

• DQProfiler.__init__ gains a keyword-only semantic_registry: SemanticRegistry | None = None. Presence of the argument opts the run into semantic detection — there is no per-call override.
• DQProfile gains an optional semantic_type: str | None = None field so generated profiles record why a check was emitted (survives YAML/JSON round-trip; defaults to None).
• DQProfileBuilder supports two mutually-exclusive callback shapes: the legacy 5-argument builder (unchanged) and the new contextual_builder(ctx: DQProfileContext). @register_profile_builder gains a kind="context" opt-in for the new shape; existing legacy registrations keep working without changes.
• The four library-native builders (null_or_empty, is_in, min_max, has_no_outliers) are migrated to the contextual form. min_max and has_no_outliers now skip emission unless ctx.semantic_type is None or "measurement"; is_in reuses the enum detector's already-collected distinct values so no second Spark .distinct().collect() runs.
• profile_table fetches Unity Catalog table_name, table_comment, and column_comment and threads them into ctx.metadata so custom detectors can consult them. Failures degrade gracefully (warning + empty metadata, never a crash). Tags are intentionally out of scope this release.

Not in scope (documented as user-supplied examples only): specialised format detectors such as UUID, email, H3 hash, geo coord, date_int. See the docs for a copy-paste DQSemanticTypeDetector example.

Linked issues

Resolves #1343

Tests

• added unit tests — tests/unit/profiler/test_semantic.py covers each detector (positive + negative), SemanticRegistry immutability and uniqueness invariants, chain semantics (first-match-wins), enum cardinality guard, numeric-density and string-length-stability key guards including the empty-string edge case. tests/unit/test_profile_builder.py covers kind="context" vs legacy registration, mutually-exclusive callback validation, and per-builder positive/negative behaviour through the new contextual path.
• added integration tests — tests/integration/test_profile_semantic.py covers the default-registry classification of the grounded design columns (vehicle_type, cargo_weight, deal_value, user_id, order_id, user_name, work_description), the no-registry byte-identical default, custom-chain composition via prepend/replace, UC metadata plumbing via a spy detector against a real Delta table, and the enum-value reuse optimisation (single .distinct() invocation per enum column).
• added end-to-end tests
• added performance tests

Documentation and Demos

• added/updated demos
• added/updated docs — new "Semantic-aware profiling" section in docs/dqx/docs/reference/profiler.mdx tagged
  <FeatureLifecycleStage stage="beta"> / <AvailableSinceVersion version="0.17.0">.
  Documents the opt-in model, the four built-in detectors and their applicability
  rules and thresholds, the immutable SemanticRegistry composition patterns, the UC metadata plumbing on profile_table, the @register_profile_builder(kind="context") opt-in, and includes a copy-paste example specialised (UUID) detector.
• added/updated agent skills

[mwojtyczka]

Automated code review (5 findings). Comments posted inline below.

[mwojtyczka]

ShortType columns lose all range/enum checks. _detect_enum accepts ShortType here, so a low-cardinality ShortType column (e.g. a status code 1..5) is classified as enum. But make_is_in_profile returns None because _supports_distinct (profile_builder.py:325) only accepts IntegerType/LongType/text — not ShortType — so no is_in is emitted. make_min_max_profile is then also suppressed because the semantic type is enum, not measurement. Net result: the column gets neither is_in nor min_max. In the legacy/no-registry path the same column still gets a min_max profile (it is a NumericType), so enabling the default registry silently drops checks. Either add ShortType to _supports_distinct, or don't classify ShortType as enum.

[mwojtyczka]

min/max write-back never reaches contextual builders. builder_ctx is created once here with metrics=metrics, but pydantic materializes ctx.metrics as a new dict (ctx.metrics is metrics is False). The min/max write-back below (~lines 633/635) mutates the outer metrics dict, not ctx.metrics. A contextual builder registered after min_max that reads ctx.metrics['min']/['max'] — the exact scenario the docstring promises support for — receives the pre-resolution summary values, not the outlier-adjusted resolved ones. No built-in triggers it today, so it's a silent latent contract break for custom contextual builders.

[mwojtyczka]

Enum cardinality gate is far more permissive than the is_in builder. This ratio ceiling (ENUM_MAX_CARDINALITY_RATIO = 0.95) is much looser than the legacy is_in builder's distinct_ratio (default 0.05). On a small table (e.g. 12 rows, 9 distinct strings): legacy is_in is suppressed because 9/12 = 0.75 is not < 0.05; but semantic profiling passes here (cardinality 9 < max_in_count 10 and 9/12 = 0.75 <= 0.95) and emits is_in with 9 allowed values. The enum branch never consults distinct_ratio, so semantic profiling generates is_in rules that ordinary profiling deliberately suppresses on low-repetition columns. Consider aligning the gate with the builder's distinct_ratio.

[mwojtyczka]

Non-DatabricksError aborts profiling. This only catches DatabricksError. If ws.tables.get(location) raises a client-side ValueError (e.g. a storage path or otherwise malformed/non-UC three-part name fails SDK name validation), it propagates and crashes profile_table — contradicting the docstring's "never block profiling" guarantee. Broaden the except (e.g. also catch ValueError, or Exception) so best-effort metadata fetch stays non-blocking.

[mwojtyczka]

Copy-paste example raises TypeError. The decorator keyword is kind, not type — register_profile_builder(profile_type, *, kind=...). As written, this example (and the prose mention on line 271) raises TypeError: register_profile_builder() got an unexpected keyword argument 'type'. Should be @register_profile_builder("my_custom", kind="context").

[mwojtyczka]

Suggested change

	The feature is fully opt-in: it runs iff a `semantic_registry` is supplied to the
	The feature is fully opt-in: it runs if a `semantic_registry` is supplied to the

[mwojtyczka]

a comment would be good here:

• prepend(detector) → puts one detector at the front (highest priority)
• replace(detectors) → swaps the entire chain

[mwojtyczka]

replace doesn't mean "replace a detector" — it means "set the whole chain. I would use something like with_detectors(...)

how about if i want to append or remove detector?

[mwojtyczka]

They way this is constructed is different than registry approach like @register_profile_builder(...) . Profile builders are customized with a decorator against a global registry; semantic detectors are customized by constructing an immutable value object and threading it through a constructor. Two different mental models for "add my thing to the chain."

[mwojtyczka]

Follow-up: ergonomics of the SemanticRegistry customization surface.

[mwojtyczka]

The customization surface (prepend + whole-chain replace) is thin, asymmetric, and slightly mis-named. The underlying model is sound — immutable, name-unique, first-match-wins, functional (returns new instances). The concern is scope, not correctness:

1. replace(detectors) doesn't replace a detector — it sets the entire chain. The name is misleading: a reader expects replace(name, detector) to swap one entry, but this discards everything. with_detectors(...) / of(...) would signal "this is the whole chain."

2. Common operations don't exist, so they all degrade into a manual replace([...]). The docs' own example proves it — to add one detector the user must re-spread the defaults by hand:

   SemanticRegistry.default().replace([uuid_detector, *default_semantic_detectors()])

   There's no append (lower-priority fallback), no insert_after(name, ...) (e.g. after enum but before key — impossible without hand-rebuilding), no remove(name), and no replace-one-by-name. prepend only covers the single "make mine win first" case.

3. Inconsistent with the sibling extension point. Profile builders use a @register_profile_builder(...) decorator against a global registry; semantic detectors use an immutable value object threaded through a constructor — two mental models for "add my thing to the chain."

Suggested fix: rename whole-chain replace → with_detectors/of, and add append, remove(name), and either insert_after(name, detector) or a real replace(name, detector), so realistic customizations don't require re-spreading default_semantic_detectors().