[ai-projects] Use typed EvaluatorGenerationJob in rubric samples#47271
Open
aprilk-ms wants to merge 2 commits into
Open
[ai-projects] Use typed EvaluatorGenerationJob in rubric samples#47271aprilk-ms wants to merge 2 commits into
aprilk-ms wants to merge 2 commits into
Conversation
The service contract nests the job inputs under an inputs field on EvaluatorGenerationJob. The four rubric-evaluator-generation samples were passing flat dicts that the SDK was tolerating but the rolling-out service change requires the nested form. Convert all four samples to use the typed EvaluatorGenerationJob / EvaluatorGenerationInputs / *EvaluatorGenerationJobSource models, and drop the stale top-level `name` field which has no home in the new contract. For the traces source, switch from int unix timestamps to datetime values (the SDK model serializes them as unix-timestamp). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the azure-ai-projects rubric evaluator generation samples to align with the service’s EvaluatorGenerationJob contract by nesting job parameters under inputs and using the typed model classes throughout, preventing failures as the service rolls out stricter request validation.
Changes:
- Updated rubric evaluator generation samples to pass
EvaluatorGenerationJob(inputs=EvaluatorGenerationInputs(...))instead of flat dict payloads. - Converted all source payloads to typed
*EvaluatorGenerationJobSourcemodels (Prompt/Agent/Dataset/Traces), including switching the traces window calculation todatetime/timedelta. - Documented the sample update in
sdk/ai/azure-ai-projects/CHANGELOG.md.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_basic.py | Uses typed EvaluatorGenerationJob / EvaluatorGenerationInputs and typed Prompt source for a single-prompt generation flow. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_iterate.py | Uses typed job + inputs + Prompt source for the initial generation step in the iteration workflow. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_lifecycle.py | Replaces the shared dict “job body” with a reusable typed EvaluatorGenerationJob request object for create + idempotent replay. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_rubric_evaluator_generation_all_sources.py | Uses typed sources for prompt/agent/dataset/traces and computes trace window with datetime/timedelta for unix-timestamp serialization. |
| sdk/ai/azure-ai-projects/CHANGELOG.md | Notes the rubric evaluator generation sample updates and the inputs nesting change. |
howieleung
approved these changes
Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four rubric-evaluator-generation samples were passing flat dicts to
create_generation_job. The service contract nests the job inputs under aninputsfield onEvaluatorGenerationJob; the SDK was tolerating the flat form, but the rolling-out service change requires the nested form.Changes (all under
sdk/ai/azure-ai-projects/samples/evaluations/):sample_rubric_evaluator_generation_basic.py— convert single-prompt call site.sample_rubric_evaluator_generation_iterate.py— convert single-prompt call site.sample_rubric_evaluator_generation_lifecycle.py— convert sharedjob_body(used for create + idempotent replay).sample_rubric_evaluator_generation_all_sources.py— convertmulti_job(prompt + agent + dataset) andtraces_job(traces + agent companion). For the traces source, switch fromint(time.time())arithmetic todatetime+timedelta(the SDK model serializesstart_time/end_timeasformat="unix-timestamp").All samples now use the typed models from
azure.ai.projects.models:EvaluatorGenerationJob(inputs=EvaluatorGenerationInputs(model=..., evaluator_name=..., evaluator_display_name=..., evaluator_description=..., sources=[...]))PromptEvaluatorGenerationJobSource,AgentEvaluatorGenerationJobSource,DatasetEvaluatorGenerationJobSource,TracesEvaluatorGenerationJobSource(discriminator is auto-set in__init__).The stale top-level
namefield (which had no home in eitherEvaluatorGenerationJoborEvaluatorGenerationInputs— the SDK was silently dropping it) is removed.Verified
eng/black-pyproject.toml) reports no changes needed.Not yet verified
inputsform.