feat: versioned metadata-only environment discovery (RFC 011) - #1138
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@burtenshaw tagging you for review of the RFC 011 catalog producer and discovery CLI, especially source identity, snapshot completeness, and the declaration-only profile. The independent public consumer is in huggingface/hf-discover#46. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b1b0ed0. Configure here.
burtenshaw
left a comment
burtenshaw
left a comment
There was a problem hiding this comment.
Looks good. Just left some turbo nits.
| text = source.read(path) | ||
| if text is None or not text.startswith("---\n"): | ||
| return {} | ||
| lines = text.splitlines() |
There was a problem hiding this comment.
Need to watch different line ending formats because this will be run in the CLI.
There was a problem hiding this comment.
Fixed in 89774a9. Frontmatter now recognizes LF, CRLF and CR openers, with the existing splitlines-based body/closing-marker parsing. The real Git-to-CLI regression uses a description available only in README frontmatter, so CRLF/CR cannot pass via package-metadata fallback. Unclosed frontmatter remains an error for all three formats. Git autocrlf is disabled within the test fixtures, not in user repositories.
Address huggingface#1138 feedback on line endings, custom license identity and schema validation. Document consumer validation and source retrieval boundaries.
There was a problem hiding this comment.
Re-reviewed exact head 89774a99 after the Ben nits follow-up.
- Exact-head CI is fully green (Tests 3.11/3.12, Package CI, lint, docs, locks, ThinkingBox, discovery snapshot, Bugbot).
- The three review items look addressed: LF/CRLF/CR frontmatter parsing, source-aware custom-license comparison, and generated JSON Schema parity with the Python model, including focused tests.
- This remains a public discovery API / architecture change implementing a Draft RFC. It is not a routine merge for the release automation.
Please re-approve on this head if the design is accepted (burtenshaw, zkwentz). Until then it stays out of the 0.4.3 candidate.
Sent by Cursor Automation: Release
There was a problem hiding this comment.
Recorded burtenshaw APPROVED on exact head a41a4cd9 (main merge-up after the nits fix).
Next gate for release automation: wait for exact-head CI to finish green on this head, then squash-merge if repository policy allows. zkwentz is still requested — if that review is required for merge, please resolve it; otherwise we will proceed once checks are terminal green.
This PR remains a public discovery API / Draft RFC change and will stay out of the 0.4.3 notes/version decision until maintainers confirm whether it ships as part of that cut or with a later minor.
Sent by Cursor Automation: Release
|
2 participants
Summary
Implements the first metadata-only repository discovery profile from RFC 011,
merged in #1093.
source/path/revision and interface-role invariants.
pinned Git tree. It reuses authored manifest, package and README metadata,
records provenance, and never imports candidate code.
openenv catalog build,openenv catalog inspect, and a read-onlyopenenv discover --cataloglexical baseline.same-snapshot identifier lookup, and correction/supersession/withdrawal
comparison.
Reasoning Gym, plus a revision-named CI snapshot artifact.
No runtime endpoint, environment installation, image pull, candidate probing,
tool invocation, reset/step, or execution-policy boolean is added.
Profile decisions
This draft implements
0.1-draft: GitHub source definitions, explicit publisherconfiguration, complete inline snapshots, and declared interfaces only. It does
not pretend to verify RFC 008 runtime evidence or duplicate the normalized
validation manifest. Unsupported profiles and validated claims fail explicitly.
The initial inventory is the configured Git tree's direct-child
openenv.yamldefinitions, not every deployment or community environment. Proposed source
revision
338577885c00d69a92e652991b367fa246918093produces 39 records.A failed eligible record still makes the inventory incomplete; logging it does
not make the build complete.
The CLI requires an explicit snapshot. Automatic remote retrieval and a central
registry are not introduced. GitHub Actions artifacts are reviewable outputs,
not an assertion that every finder already indexes them.
Task discovery beyond Echo
Six environments now have authored descriptions, tags and representative queries:
Calendar's five declared event tools are evidenced in its tracked tool-definition
file. The other new entries do not infer an agent-tool interface from a framework
or harness name. Runtime, reward and training code are unchanged.
The maintained-inventory tests use actual repository metadata in a temporary Git
tree, keep all eligible environment paths, and fail if candidate packages are
imported. Five held-out task queries are different from the authored hints.
Validation
89774a991c77c780e45df767fe95730ac7f06b4d.The PR check list records its Python 3.11/3.12 CI, packaging, docs, lint and
automated review status.
rebuilds, uncommitted-file isolation, symlink rejection, duplicate metadata,
cross-revision mismatches, missing wire claims, malformed snapshots, exact
lookup, lexical selection, and lifecycle failures. Wrong-root regressions
recompute valid digests before asserting scope rejection.
schema path cases were added; all 103 final discovery tests then passed.
package and repository scope, conflicting file/text tables, and matching query
cardinality across declaration/entry models and their packaged JSON schemas.
the independently implemented HF reader. Every complete card round-tripped
unchanged through six HTTP pages and six MCP pages. All five held-out task
queries also selected the intended environment through HTTP and the HF CLI.
parsers. Their complete
datais unchanged in the public catalog proposal.for unavailable optional runtime dependencies. These are not claims of live
browser, database, simulator or training validation.
lint, documentation sync, and whitespace checks completed.
Maintainer review fixes
frontmatter-only descriptions, so package fallback cannot hide the regression.
Unclosed frontmatter fails under each line-ending format.
otherresult as the same license. Different or unspecified custom declarationsbecome
unknownwith a warning; matching text and SPDX aliases retain evidence.presence/absence rules, conditional license evidence and exactly one
orchestration interface. The same negative cases exercise Python and JSON
Schema, and all schema patterns compile in both ECMAScript and Go.
generated schema. This is sampled coverage, not a claim that JSON Schema
captures every semantic rule.
explicitly identifies the additional URI, SPDX, package-requirement,
source/artifact and tool-revision checks. Source retrieval is not installation.
corrected producer gives the same digest and zero corrected, added or withdrawn
cards. The six catalog proposals remain valid and unchanged.
An earlier Python 3.12 job missed an unchanged rubric test's 100ms wall-clock
bound. That test and its runtime files match upstream. The target passed
locally alongside the discovery and CLI tests, and one unchanged CI rerun
passed. No timing threshold, assertion or reward logic was changed.
The repository lint hook reformats and checks out modified files; the equivalent
non-mutating formatter/import-sort/lint checks were used instead to preserve the
change set. No assertions were removed and no runtime tests were replaced by
success-shaped fallbacks.
Review focus
Please review the explicit repository profile, publisher namespace contract,
metadata precedence, and the local snapshot publication/consumption boundary.
Finder integrations are separate consumers, not implementation details of the
OpenEnv runtime.
Public companions:
Environment records from this snapshot.
discovery rather than a prerequisite for catalog admission.
These PRs do not assert that an external finder already indexes the records.
Catalog admission, presentation and rollout remain separate owner decisions.
Note
Medium Risk
New CLI surface and metadata contract affect publication and consumer trust boundaries (licenses, identifiers, completeness), but the path does not install or run environments and fails closed on incomplete catalogs.
Overview
Adds RFC 011’s
0.1-draftprofile: revision-bound, metadata-only environment catalogs built from a pinned Git tree—separate fromAutoEnvruntime resolution.A new
openenv.discoverylibrary scans direct-childenvs/*/openenv.yamlat a commit (viagit ls-tree/cat-file, no Python imports), mergesdiscovery.json, manifests,pyproject.toml, and README frontmatter into ARD-style Environment Cards, and emits complete snapshots with digest, provenance, license rules, and strict invariants (artifact binding, inventory accounting, incomplete builds on errors).CLI:
openenv catalog build/inspectand top-levelopenenv discoverfor lexical search, exact filters, and identifier lookup against a local snapshot only—no remote fetch or execution.Also ships packaged JSON schemas +
generate_discovery_schemas.py --check,discovery.jsonfor Echo, Coding, BrowserGym, Calendar, Chess, and Reasoning Gym, a Discovery catalog GitHub Actions artifact workflow, docs (catalog-discovery.md), and broad contract/integration tests (tests/discovery/).Reviewed by Cursor Bugbot for commit a41a4cd. Bugbot is set up for automated code reviews on this repo. Configure here.