Sync analysis-orchestrator evals with amd/skills#193 - #2
Merged
Conversation
Replace the interpolated gemm-01-repeatability prompt with a self-contained one that fetches and unpacks the standalone unit-test trace itself, widen the expected artifact list, add an executive-summary behavior expectation, and drop the hooks.py clone/venv setup the new prompt no longer needs. Co-authored-by: Daniel Holanda <holand.daniel@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the
analysis-orchestratoreval definitions in line with amd/skills#193, making thegemm-01-repeatabilitycase self-contained so it no longer needs an out-of-band Python setup hook.What changed
evals/evals.json— thegemm-01-repeatabilitycase is reworked:{trace_path},{platform},{output_dir},{venv_path}and{tracelens_dir}interpolation. It now tells the agent where to fetch and unpackunit_tests_standalone.tar.gzand which trace inside it to analyze, and to write results toanalysis_output/in the current directory.files_existis widened from justanalysis_output/analysis.mdto also coverperf_report.xlsx,category_data/category_manifest.json,priority_data.json,metadata/model_info.jsonandperf_improvement.png, so a run that produces a report but skips the supporting artifacts no longer passes.expected_behaviorassertion that the Executive Summary ofanalysis.mdcontains measured numbers rather than leftover template placeholders such asX ms,Y%or<Brief Title>.evals/hooks.py— deleted (178 lines). It existed only to supply the values the old interpolated prompt consumed: cloning TraceLens into a cache dir, extracting the unit-test archive, building a virtualenv, and then scoring the result by shelling out toworkflow_scripted_evals.py. With the prompt now fetching its own inputs and the expectations expressed declaratively inevals.json, none of that setup is reachable.Notes for reviewers
hooks.pywas the only provider of the{...}placeholders, and those placeholders were its only consumer. A search across the tree confirms nothing else referenceshooks.py,setup_session, or the removed placeholders.workflow_scripted_evals.pyinvocation to the dataset's ownfiles_exist/expected_behaviorchecks. That is the intended trade in Simplify Tracelens Tests amd/skills#193, but it does mean the standalone comparison-scope checks that script performed are no longer run as part of this case.raw.githubusercontent.comat eval time, so this case requires network egress where the previous version worked from a local clone.