Skip to content

Harden serving recipe cache freshness and provenance - #196

Open
amd-mkarvir wants to merge 2 commits into
amd:mainfrom
amd-mkarvir:fix/serving-skill-recipe-freshness
Open

Harden serving recipe cache freshness and provenance#196
amd-mkarvir wants to merge 2 commits into
amd:mainfrom
amd-mkarvir:fix/serving-skill-recipe-freshness

Conversation

@amd-mkarvir

@amd-mkarvir amd-mkarvir commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • make recipe-cache freshness a mandatory, fail-closed preflight
  • write refreshed data to a writable runtime cache and record source provenance plus an immutable ROCm image digest
  • select the highest stable semantic vLLM tag across every Docker Hub result page
  • resolve runtime and CLI cache paths consistently, including environment-variable expansion
  • require version checks and safer handling when a requested model is absent from the refreshed cache
  • fix UTF-8 recipe parsing on Windows and add CI-backed regression tests

Motivation

The bundled recipe snapshot is a fallback and can become stale. The previous helper always exited successfully when refresh failed, allowing callers to continue without a reliable freshness signal. It also selected an older Docker version simply because it was updated recently, and it did not record the image digest needed to reproduce the exact container.

This change makes refresh failures observable and keeps stale fallback use explicit.

Validation

  • repository skill validation passes
  • eval dataset validation passes
  • ten recipe-cache unit tests pass
  • workflow YAML and Python compilation pass
  • live synchronization smoke test passes against the public vLLM recipes repository and Docker Hub

This complements #126: that PR expands behavioral coverage, while this change hardens the deterministic cache and provenance path.

@amd-mkarvir

amd-mkarvir commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator Author

CI note: All non-secret-dependent validation passed, including the new recipe-cache unit-test job, skill validation, eval-schema validation, manifests, external references, and SkillSpector.

@amd-mkarvir
amd-mkarvir marked this pull request as draft August 30, 2026 23:57
@amd-mkarvir
amd-mkarvir marked this pull request as ready for review August 31, 2026 00:02
@amd-mkarvir

amd-mkarvir commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Local routing-eval result

Reproduced the failed Routing (whole catalog) job locally against commit 9afb63059df13f7c118cc6b9d1fca895cc28a20a.

python eval/run_evals.py --mode routing --min-accuracy 0 --no-extended \
  --output routing-report-individual-local.json \
  --keep-logs routing-logs-individual-local
  • Claude Code: 2.1.167
  • Model / effort: opus / high
  • Result: 48/52 correct (92.3%)
  • Errors: 0
  • Wrong-skill / false-trigger cases: 0 / 0
  • Positive prompts: 25/29 (86%)
  • Near misses: 18/18 (100%)
  • Unrelated prompts: 5/5 (100%)

For the skill changed by this PR:

Skill Expected Correct Recall Precision
serving-llms-on-instinct 5 5 100% 100%

All five serving cases passed: serve-tiny-model-on-instinct, qwen-on-mi300x, vllm-rocm-node, openai-compatible-mi355x, and deploy-datacenter-gpu.

The four catalog-wide misses were outside this PR''s skill:

  • hyperloom-launcher-gates
  • hyperloom-workload-intake
  • lemond-health-check
  • local-mode-without-api-key

This local result demonstrates that the PR''s serving-skill routing cases pass.

@amd-mkarvir
amd-mkarvir marked this pull request as draft August 31, 2026 01:07
@danielholanda

Copy link
Copy Markdown
Collaborator

@amd-mkarvir Please let us know when this is ready to be reviewed

@amd-mkarvir
amd-mkarvir marked this pull request as ready for review August 31, 2026 17:46
@amd-mkarvir
amd-mkarvir marked this pull request as draft September 1, 2026 05:06
@amd-mkarvir
amd-mkarvir force-pushed the fix/serving-skill-recipe-freshness branch from cfba46f to 43159c7 Compare September 1, 2026 05:37
@amd-mkarvir
amd-mkarvir marked this pull request as ready for review September 1, 2026 05:39

@danielholanda danielholanda left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are in the process of significantly refactoring the skill testing harness. If you would like to proceed with the PR, please remove changes made outside the skill folder.

Comment on lines +91 to +107
# The serving recipe cache controls model compatibility and the container
# image agents deploy. Exercise its fail-closed freshness and immutable-image
# selection without making network requests.
test-serving-recipe-cache:
name: Test serving recipe cache
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4

- name: Set up uv
uses: astral-sh/setup-uv@v7

- name: Test recipe cache
run: >-
uv run --with pyyaml python -m unittest discover
-s skills/serving-llms-on-instinct/scripts/tests -v

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All workflows are skill-agnostic. Please do not modify anything inside .github/workflows to enable skill-specific tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants