Skip to content

[ai-projects] Use typed EvaluatorGenerationJob in rubric samples#47271

Open
aprilk-ms wants to merge 2 commits into
mainfrom
users/aprilk/sample-evaluator-job
Open

[ai-projects] Use typed EvaluatorGenerationJob in rubric samples#47271
aprilk-ms wants to merge 2 commits into
mainfrom
users/aprilk/sample-evaluator-job

Conversation

@aprilk-ms
Copy link
Copy Markdown
Member

The four rubric-evaluator-generation samples were passing flat dicts to create_generation_job. The service contract nests the job inputs under an inputs field on EvaluatorGenerationJob; the SDK was tolerating the flat form, but the rolling-out service change requires the nested form.

Changes (all under sdk/ai/azure-ai-projects/samples/evaluations/):

  • sample_rubric_evaluator_generation_basic.py — convert single-prompt call site.
  • sample_rubric_evaluator_generation_iterate.py — convert single-prompt call site.
  • sample_rubric_evaluator_generation_lifecycle.py — convert shared job_body (used for create + idempotent replay).
  • sample_rubric_evaluator_generation_all_sources.py — convert multi_job (prompt + agent + dataset) and traces_job (traces + agent companion). For the traces source, switch from int(time.time()) arithmetic to datetime + timedelta (the SDK model serializes start_time / end_time as format="unix-timestamp").

All samples now use the typed models from azure.ai.projects.models:

  • EvaluatorGenerationJob(inputs=EvaluatorGenerationInputs(model=..., evaluator_name=..., evaluator_display_name=..., evaluator_description=..., sources=[...]))
  • Sources: PromptEvaluatorGenerationJobSource, AgentEvaluatorGenerationJobSource, DatasetEvaluatorGenerationJobSource, TracesEvaluatorGenerationJobSource (discriminator is auto-set in __init__).

The stale top-level name field (which had no home in either EvaluatorGenerationJob or EvaluatorGenerationInputs — the SDK was silently dropping it) is removed.

Verified

  • AST parses for all four files.
  • Repo black config (eng/black-pyproject.toml) reports no changes needed.

Not yet verified

  • Live run — the service contract change is still rolling out. Will live-test once the service requires the nested inputs form.

The service contract nests the job inputs under an inputs field on

EvaluatorGenerationJob. The four rubric-evaluator-generation samples were

passing flat dicts that the SDK was tolerating but the rolling-out service

change requires the nested form. Convert all four samples to use the typed

EvaluatorGenerationJob / EvaluatorGenerationInputs / *EvaluatorGenerationJobSource

models, and drop the stale top-level `name` field which has no home in

the new contract. For the traces source, switch from int unix timestamps

to datetime values (the SDK model serializes them as unix-timestamp).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…samples

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@aprilk-ms aprilk-ms marked this pull request as ready for review June 1, 2026 23:25
Copilot AI review requested due to automatic review settings June 1, 2026 23:25
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the azure-ai-projects rubric evaluator generation samples to align with the service’s EvaluatorGenerationJob contract by nesting job parameters under inputs and using the typed model classes throughout, preventing failures as the service rolls out stricter request validation.

Changes:

  • Updated rubric evaluator generation samples to pass EvaluatorGenerationJob(inputs=EvaluatorGenerationInputs(...)) instead of flat dict payloads.
  • Converted all source payloads to typed *EvaluatorGenerationJobSource models (Prompt/Agent/Dataset/Traces), including switching the traces window calculation to datetime/timedelta.
  • Documented the sample update in sdk/ai/azure-ai-projects/CHANGELOG.md.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_basic.py Uses typed EvaluatorGenerationJob / EvaluatorGenerationInputs and typed Prompt source for a single-prompt generation flow.
sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_iterate.py Uses typed job + inputs + Prompt source for the initial generation step in the iteration workflow.
sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_lifecycle.py Replaces the shared dict “job body” with a reusable typed EvaluatorGenerationJob request object for create + idempotent replay.
sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_all_sources.py Uses typed sources for prompt/agent/dataset/traces and computes trace window with datetime/timedelta for unix-timestamp serialization.
sdk/ai/azure-ai-projects/CHANGELOG.md Notes the rubric evaluator generation sample updates and the inputs nesting change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants