Skip to content

benchmark: run_case.sh misses manifests nested by Claude (e.g. cargo new subdir) #44

Description

@algomaster99

For `fresh` cases, `run_case.sh` only checks the exact `.manifest` path from the case definition (e.g. `Cargo.toml` at the workdir root). If Claude nests the crate in a subdirectory (e.g. `cargo new --lib my-proc-macro`), the real manifest ends up at `my-proc-macro/Cargo.toml` and is never found, so `final_manifest` is wrongly recorded as `MANIFEST_NOT_WRITTEN` even though a manifest was written and edited.

Repro: `cargo-top-06-proc-macro2`, `nohook` condition — transcript shows `cargo new --lib my-proc-macro`, `cargo add proc-macro2`, and an `Edit` on `my-proc-macro/Cargo.toml`, but `final_manifest` says `MANIFEST_NOT_WRITTEN`.

Likely generalizes beyond cargo — any ecosystem where Claude can plausibly scaffold into a subdirectory (e.g. `npm init` into a named folder, Go module in a subdir) has the same risk if `.manifest` is a fixed root-relative string.

Suggest: for `fresh` cases, search recursively for the manifest filename (e.g. `find . -name Cargo.toml`) instead of assuming a fixed path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions