Skip to content

feat(metrics): add JSON schema conformance metric - #370

Merged
himanshu231204 merged 1 commit into
OpenAgentHQ:mainfrom
aayushi-sing:feat/schema-conformance
Sep 18, 2026
Merged

himanshu231204 merged 1 commit into
OpenAgentHQ:mainfrom
aayushi-sing:feat/schema-conformance

Conversation

@aayushi-sing

Copy link
Copy Markdown
Contributor

Summary

Adds a deterministic JSON Schema conformance metric for evaluating structured model outputs.

Changes

  • Add SchemaConformance generation metric.
  • Parse generated answers as JSON and validate them against JSON Schema Draft 2020-12.
  • Return 1.0 for fully conformant outputs and 0.0 for invalid JSON or non-conformant outputs.
  • Add deterministic top-level field-level partial scoring for object schemas.
  • Pass dataset metadata through the evaluation pipeline so metrics can access the expected JSON schema.
  • Register schema_conformance in METRIC_REGISTRY.
  • Add jsonschema dependency and update uv.lock.
  • Add unit tests covering valid, invalid, partial, missing-schema, empty-schema, and extra-property cases.

Testing

  • uv run pytest tests/unit/test_metrics/test_generation.py -q — 41 passed
  • uv run ruff check openagent_eval/metrics/generation/schema_conformance.py tests/unit/test_metrics/test_generation.py — passed
  • uv run ruff format --check openagent_eval/metrics/generation/schema_conformance.py tests/unit/test_metrics/test_generation.py — passed
  • git diff --check — passed

The full test suite was also run locally. It had one unrelated Windows-specific failure in test_oaeval_help_runs_without_pytest caused by a cp1252 Unicode encoding error; no changes were made to that unrelated test.
Closes #347

@himanshu231204
himanshu231204 merged commit 0ba23e1 into OpenAgentHQ:main Sep 18, 2026
10 checks passed
@github-actions

Copy link
Copy Markdown

🎉 Congratulations @aayushi-sing!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a structured-output / JSON-schema conformance metric

2 participants