Add ODCS type: library quality metric support

[Christopher-Lawford]

Summary

• Adds native mapping from ODCS type: library quality entries to DQX checks for the five supported metrics: rowCount, nullValues, missingValues, invalidValues, and duplicateValues.
• Each metric maps its eight shared ODCS threshold fields (mustBe, mustNotBe, mustBeGreaterOrEqualTo, mustBeLessOrEqualTo, mustBeGreaterThan, mustBeLessThan, mustBeBetween, mustNotBeBetween) onto exact-fit DQX aggregate checks where possible, with a dataset-level sql_query fallback for strict inequalities and the (both-bounds-exclusive, per ODCS) mustBeBetween/mustNotBeBetween forms.
• Malformed or unrecognized type: library entries (missing/unknown metric, no recognized threshold field, malformed arguments, unrecognized unit, misplaced property/schema-level entries) are warned-and-skipped per entry rather than failing the whole contract; this processing is unconditional, with no opt-out flag.
• Adds end-to-end integration coverage generating and applying rules from a contract exercising all five metrics.
• Adds a new "Metric Rule Generation" guide section (and updates the "Metadata Fields" table) documenting the feature for end users.

Test plan

• tests/unit/test_datacontract_generator.py::TestDataContractGeneratorLibraryRules — unit coverage per metric/threshold-field combination
• tests/unit/test_checks_semantic_validator.py — updated coverage for the semantic validator changes
• tests/integration/test_datacontract_integration.py — end-to-end generation + apply_checks_by_metadata against a real DataFrame for all five metrics
• make test / make lint run clean on this branch (please confirm in CI)

🤖 Generated with Claude Code

[github-actions]

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[CLAassistant]

All committers have signed the CLA.

[mwojtyczka]

Out of scope for this PR (should be a follow up): I would implement the missing functions and replace the sql query:

• mustBeGreaterThan (enforce metric > X): new function is_aggr_greater_than(limit=X)
• mustBeLessThan (enforce metric < X): new function is_aggr_less_than(limit=X)
• mustBeBetween (enforce lo < metric < hi): new function is_aggr_in_range(lo, hi)
• mustNotBeBetween (enforce metric ≤ lo OR metric ≥ hi): new function is_aggr_not_in_range(lo, hi)

Can you please create a follow up issue for this?

[mwojtyczka]

Automated code review — ODCS type: library quality metrics

Verified against head d2b6471. One serious correctness bug plus a serialization/validation cluster and a few lower-severity consistency issues. Details are in the inline comments; summary:

Severity	Finding
🔴 High	duplicateValues (any non-zero threshold) nests COUNT(*) OVER (...) inside a SUM/AVG aggregate → Spark AnalysisException at apply time. Only mustBe: 0 is execution-tested.
🟠 Med	nullValues percent and missingValues forbidden embed live F.when/F.lit Column objects in the generated rule dicts → not YAML/JSON-serializable (save_checks fails). Other percent paths already use SQL strings to avoid this.
🟠 Med	Because of the above, ChecksSemanticValidator silently skips conflict detection for those rules (unhashable Column in the key → TypeError swallowed).
🟡 Low	RLIKE string literals don't escape backslashes → regex patterns mangled vs the row-level regex_match path.
🟡 Low	Numeric validValues stringified into NOT IN ('..') → string vs numeric comparison mismatch with is_in_list.
🟡 Low	mustBe == 0 matches boolean False, misses string "0".
🟡 Low	Multiple rowCount entries on one schema collide on rule name.

Cleared: the sql_query fallback for mustBeGreaterThan/mustBeLessThan/mustBeBetween/mustNotBeBetween is correct — DQX has no strict-inequality or between aggregate check (only is_aggr_not_greater_than/not_less_than/equal/not_equal). SQL-injection via threshold interpolation is not viable — those fields are typed float | int.

[mwojtyczka]

Window function nested inside an aggregate — Spark rejects this at apply time.

_duplicate_indicator_sql builds CASE WHEN … COUNT(*) OVER (PARTITION BY …) > 1 …, and _duplicate_values_aggregate_check (just below) wraps it in is_aggr_equal/etc. with aggr_type sum/avg. That produces SUM(CASE WHEN … COUNT(*) OVER (PARTITION BY …) …), which Spark forbids: AnalysisException: It is not allowed to use window functions inside aggregate functions.

Every duplicateValues threshold except mustBe: 0 (which routes to is_unique) hits this path, and it is only asserted as a generated string in unit tests — never executed. Recommend (a) an integration test that actually applies a non-zero duplicateValues rule, and (b) a rewrite that derives the duplicate count without nesting a window inside an aggregate (e.g. aggregate over a pre-grouped count).

[mwojtyczka]

Non-serializable Column embedded in a metadata rule dict.

This F.when(...) Column is passed straight into _nullvalues_percent_aggregate_check as the check's column argument, so the generated rule dict carries a live PySpark Column. It cannot round-trip through YAML/JSON, so DQEngine.save_checks() (or any serialization of the generated rules) fails.

The missingValues/invalidValues percent paths deliberately use an AVG(CASE WHEN …) SQL string for exactly this reason (see the class comment near line 2508). This path — and the missingValues forbidden list (line ~2607) — were missed. Use the same SQL-string indicator here so the rule stays serializable.

[mwojtyczka]

Same serialization issue as the nullValues percent path.

forbidden holds F.lit(...) Column objects, so the generated is_not_in_list rule dict is not YAML/JSON-serializable. Pass plain scalar literals instead (the F.lit/resolver conversion happens at apply time), so the rule can round-trip through save_checks().

[mwojtyczka]

Conflict detection is silently skipped for Column-valued arguments.

When a generated check carries a Column as its column argument (the nullValues-percent path in contract_rules_generator.py), _make_hashable returns the Column unchanged (it is neither list/tuple/dict), so the _conflict_key tuple contains an unhashable Column. In _conflict_issue, conflict_key not in seen then raises TypeError, which detect_conflicts catches and skips — so two genuinely conflicting generated rules on the same column are never flagged.

(Duplicate detection via _full_key is unaffected — it stringifies through json.dumps(default=str).) Fixing the root cause — keeping generated column args as SQL strings rather than Column objects — resolves this as well.

[mwojtyczka]

RLIKE patterns with backslashes get mangled.

_invalid_values_sql_literal only doubles single quotes; it does not escape backslashes. With Spark's default spark.sql.parser.escapedStringLiterals=false, a backslash in the literal acts as an escape character, so a regex like \d+ delivered via col RLIKE '\d+' differs from the raw pattern the row-level regex_match check (the mustBe: 0 path) uses. The aggregate/percent invalid-count then disagrees with the row-level rule. Escape backslashes too, or build the RLIKE via a bound parameter rather than string interpolation.

[mwojtyczka]

Numeric validValues are stringified into the NOT IN list.

_invalid_values_sql_literal wraps every value in single quotes, so a numeric allow-list becomes col NOT IN ('1','2','3') (string comparison), while the row-level is_in_list path (mustBe: 0) passes the numbers through unchanged. On numeric columns the two paths can classify rows differently, so the aggregate count/percentage disagrees with the row-level rule. Preserve numeric literals unquoted.

[mwojtyczka]

mustBe == 0 special-cases boolean False and misses numeric strings.

mustBe is Any-typed. False == 0 is True, so mustBe: false is routed into the is_unique fast path; conversely mustBe: "0" (string) fails == 0 and silently drops into the aggregate path with a string limit. Consider validating/normalizing the threshold type before this comparison. (Low severity.)

[mwojtyczka]

Colliding rule names for multiple rowCount entries.

The name is always f"{schema_name}_rowCount", so two type: library rowCount entries on one schema (e.g. a lower and an upper bound) both get named <schema>_rowCount. Downstream tooling keyed on rule name (dedup, reporting, user_metadata joins) can't distinguish them. Include the threshold field or an index in the name. (Low severity.)

[mwojtyczka]

Going in the right direction. Left some comments