Harden serving recipe cache freshness and provenance - #196
Conversation
|
CI note: All non-secret-dependent validation passed, including the new recipe-cache unit-test job, skill validation, eval-schema validation, manifests, external references, and SkillSpector. |
Local routing-eval resultReproduced the failed Routing (whole catalog) job locally against commit
For the skill changed by this PR:
All five serving cases passed: The four catalog-wide misses were outside this PR''s skill:
This local result demonstrates that the PR''s serving-skill routing cases pass. |
|
@amd-mkarvir Please let us know when this is ready to be reviewed |
cfba46f to
43159c7
Compare
danielholanda
left a comment
There was a problem hiding this comment.
We are in the process of significantly refactoring the skill testing harness. If you would like to proceed with the PR, please remove changes made outside the skill folder.
| # The serving recipe cache controls model compatibility and the container | ||
| # image agents deploy. Exercise its fail-closed freshness and immutable-image | ||
| # selection without making network requests. | ||
| test-serving-recipe-cache: | ||
| name: Test serving recipe cache | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Check out repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up uv | ||
| uses: astral-sh/setup-uv@v7 | ||
|
|
||
| - name: Test recipe cache | ||
| run: >- | ||
| uv run --with pyyaml python -m unittest discover | ||
| -s skills/serving-llms-on-instinct/scripts/tests -v |
There was a problem hiding this comment.
All workflows are skill-agnostic. Please do not modify anything inside .github/workflows to enable skill-specific tests.
Summary
Motivation
The bundled recipe snapshot is a fallback and can become stale. The previous helper always exited successfully when refresh failed, allowing callers to continue without a reliable freshness signal. It also selected an older Docker version simply because it was updated recently, and it did not record the image digest needed to reproduce the exact container.
This change makes refresh failures observable and keeps stale fallback use explicit.
Validation
This complements #126: that PR expands behavioral coverage, while this change hardens the deterministic cache and provenance path.