Skip to content

Sync analysis-orchestrator evals with amd/skills#193 - #2

Merged
danielholanda merged 1 commit into
mainfrom
dholanda/tl_ss
Aug 29, 2026
Merged

Sync analysis-orchestrator evals with amd/skills#193#2
danielholanda merged 1 commit into
mainfrom
dholanda/tl_ss

Conversation

@danielholanda

Copy link
Copy Markdown
Owner

Brings the analysis-orchestrator eval definitions in line with amd/skills#193, making the gemm-01-repeatability case self-contained so it no longer needs an out-of-band Python setup hook.

What changed

evals/evals.json — the gemm-01-repeatability case is reworked:

  • The prompt no longer relies on {trace_path}, {platform}, {output_dir}, {venv_path} and {tracelens_dir} interpolation. It now tells the agent where to fetch and unpack unit_tests_standalone.tar.gz and which trace inside it to analyze, and to write results to analysis_output/ in the current directory.
  • files_exist is widened from just analysis_output/analysis.md to also cover perf_report.xlsx, category_data/category_manifest.json, priority_data.json, metadata/model_info.json and perf_improvement.png, so a run that produces a report but skips the supporting artifacts no longer passes.
  • Adds an expected_behavior assertion that the Executive Summary of analysis.md contains measured numbers rather than leftover template placeholders such as X ms, Y% or <Brief Title>.

evals/hooks.py — deleted (178 lines). It existed only to supply the values the old interpolated prompt consumed: cloning TraceLens into a cache dir, extracting the unit-test archive, building a virtualenv, and then scoring the result by shelling out to workflow_scripted_evals.py. With the prompt now fetching its own inputs and the expectations expressed declaratively in evals.json, none of that setup is reachable.

Notes for reviewers

  • The two files change together: hooks.py was the only provider of the {...} placeholders, and those placeholders were its only consumer. A search across the tree confirms nothing else references hooks.py, setup_session, or the removed placeholders.
  • Scoring moves from the external workflow_scripted_evals.py invocation to the dataset's own files_exist / expected_behavior checks. That is the intended trade in Simplify Tracelens Tests amd/skills#193, but it does mean the standalone comparison-scope checks that script performed are no longer run as part of this case.
  • The trace is now fetched over the network from raw.githubusercontent.com at eval time, so this case requires network egress where the previous version worked from a local clone.

Note to AMDers:
This is a public repository. Please do not upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead. — No customer or confidential data is included; this change touches only eval definitions and public trace URLs.

Open in Web Open in Cursor 

Replace the interpolated gemm-01-repeatability prompt with a self-contained
one that fetches and unpacks the standalone unit-test trace itself, widen the
expected artifact list, add an executive-summary behavior expectation, and drop
the hooks.py clone/venv setup the new prompt no longer needs.

Co-authored-by: Daniel Holanda <holand.daniel@gmail.com>
@danielholanda
danielholanda marked this pull request as ready for review August 29, 2026 19:36
@danielholanda
danielholanda merged commit 53cb50b into main Aug 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants