Add --metadata-only to datacontract test and ci

[royroyroys]

As discussed with @jschoedl on Slack: add --metadata-only, align --checks naming with the ODCS terms, and document that schema checks can read data.

Problem

datacontract test --checks schema runs every check derived from the contract's properties. Some checks read only the schema, while others read row values.

Row-value checks can scan large warehouse tables and add cost. There was no way to check field presence and types without also running checks such as required, unique, ranges, patterns, row counts, service levels, and custom SQL.

The command also used schema and servicelevel as category names instead of the ODCS terms properties and slaProperties.

Solution

Add --metadata-only to datacontract test and datacontract ci.

With the flag:

• Field presence and type checks run.
• Checks that read row values show as skipped.
• Text, JSON, and JUnit results include the skipped checks and their reason.
• JUnit writes skipped checks as <skipped> test cases.

The flag treats field presence and type metrics as schema reads. It treats all other metrics as row reads, so new metrics are skipped by default.

--checks now accepts the ODCS terms properties and slaProperties.
schema and servicelevel remain as legacy aliases. Category and dimension names now match without regard to case.

Behaviour worth reviewing

• --metadata-only applies after --checks, --dimension, --quality-id, and --tag, so these options combine.
• JSON file and API record validation still reads every record.
• If every selected check is skipped, the run stays unknown. As before, test exits with code 1 for an unknown run, while ci follows --fail-on and exits with code 0 by default.
• Azure Blob metadata checks keep their current behaviour because they do not scan table rows.

Notes

• _set_result renamed to public set_result (with ->None and a docstring) because the metadata-only partition in data_contract_test.py calls it from another module.

Tests

Added tests for:

• Default row-check behaviour without the flag.
• Skipped required, unique, range, pattern, and row-count checks.
• Field presence and type checks that still run.
• Skipped freshness, retention, and custom SQL checks.
• Runs where every selected check is skipped.
• JSON record validation under the flag.
• Text and JUnit skipped output.
• End-to-end test and ci options.
• ODCS category names, legacy aliases, and case handling.

Also checked the CLI with CSV and Parquet data, including text, JSON, JUnit, help output, aliases, and exit codes.

Full test suite:

• 1,646 passed
• 25 skipped
• 0 failed

Docs

Updated the testing guide to explain:

• The ODCS category names and legacy aliases.
• Why selecting properties is not metadata-only.
• Which checks --metadata-only skips.
• Why JSON file and API validation still reads records.

Regenerated the test and ci command pages from CLI help, and added the entries to the release notes page.

• Tests pass (uv run pytest)
• Code formatted (uv run ruff check --fix && uv run ruff format)
• Docs updated (if relevant)
• CHANGELOG.md entry added

[jochenchrist]

Wouldn't it rather be --checks metadata instead of a new --metadata-only option ?

[royroyroys]

Wouldn't it rather be --checks metadata instead of a new --metadata-only option ?

Wow thanks for the quick feedback! 😮

I thought it seemed like the --checks was to handle the various ODCS sections and having a flag for --metadata-only then could let people toggle something that is outside of those ODCS sections and does something slightly different.
eg. --checks being "which rules to run"
and --metadata-only being "of the selected checks, which may read data?" (aka just run the 'cheap' tests)

And could help be a bit of a blanket safety net for testing where the user might want to say 'don't spend money in this job' but still see what the job would be doing if they just let it run.

Also partly because Jakob agreed to this on Slack 😄

But I can also see how it could be part of just checks if we're happy to break that convention and if we don't care about having a 'only run cheap tests' toggle option that highlights it.

Happy to update if needs be, but I thought it's also nice to see that it's skipping certain checks in the reporting now with this. Would you prefer it to be --checks metadata and just omit the non-matching checks?