Let reusable.yml authenticate through an API gateway - #24
danielholanda wants to merge 3 commits into
Conversation
Add api_base_url and api_custom_headers inputs, matching skill-evals.yml, so a key issued by a gateway can reach it instead of failing with a 401. Co-authored-by: Cursor <cursoragent@cursor.com>
johnl-amd
left a comment
There was a problem hiding this comment.
Approving. I reviewed the workflow change, the documentation, and how this compares with skill-evals.yml. I also reproduced the claims in the description: 226 passed on the PR head, and the local gateway run exporting the base URL and the substituted header.
Four non-blocking notes.
The docs don't mention the new failure. Setting api_base_url or api_custom_headers without a key now stops the run with a clear message. That's the one genuinely new behaviour here, since those inputs used to be inert. docs/usage.md documents the inputs and the example but not this, so someone reading only the docs won't know it exists. One sentence would cover it.
Nothing tests the masking on this path. The key ends up inside a header. I think it stays masked, because it entered through ${{ secrets.* }} and GitHub masks every copy of that literal. But test_the_minted_token_is_masked_before_anything_can_log_it only covers the federation path. There's no equivalent that runs main() with a key plus gateway and asserts what lands in GITHUB_ENV and stdout. Given this PR is specifically about putting a secret in a header, that feels like the test worth having.
Worth a line in the docs too: a caller who hardcodes a literal secret into api_custom_headers instead of using $API_KEY gets no masking at all, since GitHub never saw that value.
The credential step is now in five places. Three in skill-evals.yml, two here. I diffed the two new copies: 39 lines each, one differing line, and that difference is the correct routing / behavioral swap. So they're in sync today. They already disagree on what they forward though, skill-evals.yml passes the federation variables and SECRET_NAME and this doesn't, and nothing tests parity. Fine as is, just easy to drift.
reusable.yml still can't do federation. Pre-existing, not this PR, and resolve() falls back to the key path cleanly. Noting it because this closes the gateway gap between the two workflows and leaves that one open.
Things I'd call out as good: the new guard turns a silent no-op into a hard error, extending the environment rather than replacing it is the right call and would have failed subtly otherwise, and trimming the top-of-file comment about never learning your gateway keeps it true.
rominf
left a comment
There was a problem hiding this comment.
I agree with the review by John. And I would not merge it as is, but address non-blocking comments first. Also, I would move Python code out of GitHub Action YAML, and put it into separate script to make it reusable.
Both jobs now call credentials.py --reusable instead of carrying their own inline copy. Adds a test that a gateway key reaches GITHUB_ENV but never stdout, and documents the no-key error and why the key belongs in a header as $API_KEY. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks both! Addressed the feedback in the last commit
|
rominf
left a comment
There was a problem hiding this comment.
Almost perfect, but please have a look at my comment about Python.
| - name: Resolve the model credentials | ||
| shell: python | ||
| env: | ||
| EXPECT_KEY: ${{ inputs.routing != 'off' && (inputs.api_key_secret != '' || secrets.api_key != '') }} | ||
| API_BASE_URL: ${{ inputs.api_base_url }} | ||
| API_CUSTOM_HEADERS: ${{ inputs.api_custom_headers }} | ||
| run: | | ||
| import os | ||
| import subprocess | ||
| import sys | ||
|
|
||
| if os.environ.get("EXPECT_KEY") != "true": | ||
| sys.exit(0) | ||
| if not os.environ.get("ANTHROPIC_API_KEY", "").strip(): | ||
| sys.exit( | ||
| "error: the model API key resolved to an empty value. GitHub " | ||
| "withholds secrets from pull requests opened from a fork, so " | ||
| "re-run this from a branch in the repository; otherwise check " | ||
| "that the secret is set and that the caller maps it onto " | ||
| "'api_key' (or passes 'secrets: inherit')." | ||
| ) | ||
| completed = subprocess.run( | ||
| [sys.executable, ".skillscope-action/skillscope/credentials.py", "--reusable"] | ||
| ) | ||
| raise SystemExit(completed.returncode) |
There was a problem hiding this comment.
It's possible to avoid python here and in other places at all, since it's just calls subprocess. :-)
The credential steps only ran the script as a subprocess, so run it as a plain command. Workflow steps may now leave shell unset; action.yml still has to use Python. Co-authored-by: Cursor <cursoragent@cursor.com>
Problem
reusable.ymlcan only send a key straight to the model provider. Teams whose key comes from an API gateway need two more things: the gateway's base URL, and often the key in a header of the gateway's own (for exampleOcp-Apim-Subscription-Keyon Azure API Management).reusable.ymlhad no way to set either, so their runs failed with a 401.skill-evals.ymlalready supports both.Fix
Two optional inputs on
reusable.yml, named and behaving the same as inskill-evals.yml:api_base_url: where to send requests (setsANTHROPIC_BASE_URL).api_custom_headers: extra headers, one per line, where$API_KEYis replaced with the key (setsANTHROPIC_CUSTOM_HEADERS).The routing and behavioral jobs pass these to the existing
skillscope/credentials.py, so the gateway logic stays in one place. Callers that set neither input see no change. Setting them without a key fails early with a clear message instead of a 401.docs/usage.mddocuments the two inputs.Testing
pytest tests: 226 passed.