Skip to content

Absorb the validation gaps left by retiring jdh-agents' marketplace CLI #22

Description

@jdh313

Retiring jdh-agents' repository-owned Python marketplace CLI in favour of the
pinned agentforge binary (v0.4.0) left a set of checks with no home. Each one
below was verified by that repo's own validator and is now verified by
nothing — agentforge does not implement it, and jdh-agents no longer can.

This is a tracking issue for the whole set. None of it blocks the jdh-agents
migration, which is otherwise complete and drift-clean; the point is that the
gaps are written down somewhere actionable rather than living only in a
consumer's compatibility doc.

Context, with the reasoning behind each disposition:
docs/agentforge-compatibility.md
in jdh-agents.

1. Codex materialized-tree assertions

Codex has no non-interactive plugin validate, so there is no runtime-native
gate to delegate to the way --claude-native delegates to
claude plugin validate --strict. jdh-agents carried a hand-written substitute
(scripts/marketplace/codex_validate.py, 429 lines, now deleted). The Codex
publication is currently gated by agentforge check alone: manifest parity,
declared plugin path resolution, and skill frontmatter against the Codex schema.

Four assertions the deleted validator made are not re-verified by anything:

  • Declared-vs-materialized package parity — every package the manifest
    declares actually materialized into the tree, and nothing else did.
  • Companion-script reference resolution — a hook or skill that references
    a sibling script resolves to a file that exists in the compiled output.
  • Hook event names — validated against the events the target actually
    supports (see item 3, which is the structural version of this).
  • SessionEnd timeout cap — Codex enforces a ceiling; exceeding it is
    accepted at compile time today.

These are checks on a materialized tree, which is what check already walks,
so they plausibly belong as additional check gates rather than compile-time
ones — consistent with compilation staying total (ndr:tfee0d).

2. Agent model: / effort: pinning is invisible

The construct detector reads only tools: from agent frontmatter. Nothing reads
model: or effort:, so an author who tiers agents by model loses that tiering
on Codex with no diagnostic naming it.

The inferred-artifact-projection note does say "Claude model, turn, and tool
constraints … are not enforced by Codex," but that is fixed boilerplate emitted
for every agent — it reads identically whether an agent pins opus or inherits.

Because a losses entry's construct is validated against a closed enum, there
is no agent-model-pin construct to declare, so this cannot even be recorded as
a declared loss. It can only be written into some other entry's prose, which is
where it currently lives.

Surfaced by shake-tune, which tiers opus/high for its three interpretation
agents and inherit/low for its two mechanical checks. Applies equally to
coach, debate, librarian, skillsmith, and spec-flow.

  • Add an agent-model-pin construct (detector + capability row + enum entry)
    so the loss is declarable and the diagnostic names the pinned value.

3. Hook events are outside the capability table

ConstructSurface admits only skill and prompt, so supportFor cannot be
asked whether a hook event exists on a target. The answer instead lives in a
hardcoded set inside the Codex marketplace adapter whose only citation is a code
comment — outside the per-row doc-citation discipline every other capability fact
is held to.

The set is accurate as of codex 0.146.0 (verified against the binary's embedded
JSON schemas), but nothing structural keeps it that way, and supportFor's
supported/unsupported/unknown contract is exactly the mechanism this should
be using.

  • Add a hook surface to ConstructSurface and move the Codex hook-event
    set into the capability table as doc-cited rows.

Not transferring

For the record, so a future reader does not re-litigate these:

  • The repo-wide secret scan stays in jdh-agents. agentforge only ever sees
    files a publication declares, so it structurally cannot scan a whole repo.
  • test_attention_workflow.py's state-and-guards layer stays in jdh-agents.
    It exercises plugin runtime behavior (grant immutability, supersession,
    PreToolUse guards) as subprocesses against temporary repos. A structural
    compiler cannot cover this, and should not try. Its publication-layer half
    is now covered by agentforge check and was correctly dropped.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions