Skip to content

Commit e3a6aec

Browse files
speak-agentspeak-agent
committed
a target that cannot be run could only be measured by not building it
`mcpp test` for a target this host can neither execute nor reach through a runner leaves every test `not run` and exits 2. That is the correct answer to "do these tests pass" — mcpp did not find out — but 2 is also what a broken runner returns, so a caller that wanted only the build cannot tell the two apart and falls back to `mcpp build`. AND `mcpp build` BUILDS THE PACKAGE. For a package whose only sources are under `tests/` it compiles nothing of it at all. Measured on mcpp-index's `archive` member, whose sources are two files under `tests/`: $ mcpp build --target aarch64-macos # exits 0 Compiling compat.lz4 / compat.xz / compat.zlib / compat.zstd … $ find target -name '*compression*' -o -name '*versions*' (only musl's versionsort.o) The member's dependencies compiled; not one line of the member did. A compatibility sweep reading that exit code records the member as building on macOS, which is a reading about the dependencies with the member's name on it. `--no-run` gives the narrower claim its own answer: every selected test is compiled and linked for the target, none is executed, and the result says so. `built` is counted apart from `not_run`, in the human summary and in the machine interface, because `not_run` means mcpp tried and could not — the question is open, the exit code is 2 — while `built` means it was told not to, so the build was the whole question and the exit code is 0. The criterion is `745_no_run_builds_the_tests_and_says_so.sh`, four legs. Its runner is a name that is not a program: an unexecutable target would make the test need a cross toolchain and a host that cannot run it, while a runner that cannot be found produces the same situation on every host, for the native target, with nothing installed. Run against a binary without the flag it fails at leg B.
1 parent 65bf675 commit e3a6aec

9 files changed

Lines changed: 294 additions & 10 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,41 @@
55

66
## [Unreleased]
77

8+
### `mcpp test --no-run`:为一个跑不了的目标构建测试,并把这当成答案
9+
10+
一个本机既不能执行、也没有 runner 可达的目标,会让每个测试停在 `not run`,命令退出 2。
11+
这是对「这些测试通过吗」的正确回答——mcpp 没有查明。但 **2 同样是 runner 坏掉时的
12+
退出码**,于是一个只想要「构建」的调用方无法区分这两者,只能退回去用 `mcpp build`。
13+
14+
而 `mcpp build` 构建的是**包**。对一个源码只在 `tests/` 下的包,它一行都不编译。
15+
实测 mcpp-index 的 `archive` 成员(源码是 `tests/` 下两个文件):
16+
17+
```
18+
$ mcpp build --target aarch64-macos # 退出 0
19+
Compiling compat.lz4 / compat.xz / compat.zlib / compat.zstd ...
20+
$ find target -name '*compression*' -o -name '*versions*'
21+
(只有 musl 的 versionsort.o)
22+
```
23+
24+
退出 0,编译了这个成员的**依赖**,而**该成员自己的代码一行都没有编过**。一个读这个
25+
退出码的兼容性测量,会把它记成「这个成员在 macOS 上构建通过」。
26+
27+
`--no-run` 让那个更窄的断言有了自己的答案:被选中的每个测试都为该目标编译并链接,
28+
没有任何一个被执行,结果也这么写:
29+
30+
```
31+
test result ok. 0 passed; 0 failed; 2 built, not run
32+
```
33+
34+
`built` 与 `not_run` 分开计数,机器接口也一样(`docs/50`):`not_run` 是「试过而做不到,
35+
问题悬着,退出 2」,`built` 是「被要求不要执行,构建就是问题的全部,退出 0」。编译不过
36+
的测试仍然是失败。`--no-run` 与 `--no-runner` 同时给出会被拒绝——两个名字只差一个字符
37+
而含义相反,不存在应当优先的那一个读法。
38+
39+
判据是 `tests/e2e/745_no_run_builds_the_tests_and_says_so.sh`,四条腿。其中 runner 用的是
40+
一个**不存在的程序名**:用一个执行不了的目标会让这个测试需要交叉工具链和特定宿主,而
41+
一个找不到的 runner 在每台宿主上、对本机目标、不装任何东西,就能造出同一个局面。
42+
843
### `builtins = "iso"` 发的那个 token 是静默空操作,已换成 `-fno-builtin`
944

1045
`[c-abi] builtins = "iso"` 声明 C 库只提供 ISO 函数、没有厂商扩展。Apple 目标上

‎docs/08-testing.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ mcpp performs with the test binary appended.
105105
```bash
106106
mcpp test --target thumbv7em-none-eabihf # built for the board, run through its runner
107107
mcpp test --no-runner # ignore the runner and execute directly
108+
mcpp test --target aarch64-macos --no-run # build the tests for the target and stop
108109
```
109110

110111
Nothing about the test changes. The same `tests/**/*.cpp` compiles for the
@@ -114,6 +115,23 @@ or a QEMU exit code is what a bare-metal runner is chosen to produce.
114115
`--no-runner` exists for a host that can execute the binaries natively and
115116
should not pay for an emulator.
116117

118+
`--no-run` makes the narrower claim, and it has to be asked for. Without it, a
119+
target this host can neither execute nor reach through a runner leaves every
120+
test not run and the command exits 2: mcpp did not establish whether the tests
121+
pass, and reporting that as success is the false reading this repository has
122+
recorded most often. But 2 is also what a broken runner returns, so a caller
123+
that wanted only the build could not tell the two apart. Under `--no-run` every
124+
selected test is compiled and linked for the target, none is executed, and the
125+
result says so:
126+
127+
```
128+
test result ok. 0 passed; 0 failed; 2 built, not run
129+
```
130+
131+
A test that does not compile is still a failure, and `--no-run` with
132+
`--no-runner` is refused rather than resolved: one says to run the binaries
133+
without the declared runner, the other says not to run them.
134+
117135
A test program carries the files it reads beside it: the runner receives
118136
`MCPP_RUNTIME_FILES`, the list of its deployed files and the shared libraries it
119137
loads, and a runner that moves the program to a device copies them with it. A

‎docs/50-machine-output.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -523,8 +523,8 @@ Per test:
523523
|---|---|
524524
| `member` | the workspace member, or `""` outside a workspace |
525525
| `test` | the path-based test name (`tests/00-a/0.cpp` → `00-a/0`) |
526-
| `status` | `pass`, `compile_fail`, `run_fail`, or `not_run` |
527-
| `exit_code` | the test's exit status; `0` for `not_run` |
526+
| `status` | `pass`, `compile_fail`, `run_fail`, `not_run`, or `built` |
527+
| `exit_code` | the test's exit status; `0` for `not_run` and `built` |
528528
| `signal` | the signal number when the status encodes one, else `null` |
529529
| `duration_ms` | build+run wall time of this test |
530530
| `timed_out` | `true` when `--timeout` killed it (`run_fail`) |
@@ -538,8 +538,17 @@ Summary record, `{"summary": {...}}`:
538538
| `member`, `passed`, `failed` | counts |
539539
| `not_run` | tests that were built and not executed |
540540
| `not_run_reason` | the reason shared by all of them, or `""` |
541+
| `built` | tests built under `--no-run`, which were not to be executed |
541542
| `elapsed_ms`, `build_ms`, `run_ms` | wall time, split |
542543

544+
**`built` and `not_run` are different answers and are counted apart.** Both
545+
describe a test that was compiled and not executed, and that is where the
546+
resemblance ends: `not_run` means mcpp tried and could not, so the question
547+
is open and the exit code is 2; `built` means `--no-run` said not to, so the
548+
build was the whole question and the exit code is 0. A consumer that added
549+
the two together would report a run it never asked for as one that could not
550+
be performed.
551+
543552
**`not_run` is neither `pass` nor `run_fail`, and the exit code says so
544553
(2026.9.2.1).** A test is `not_run` when this host cannot load its artifact
545554
(`Exec format error` on a cross target with no runner declared), or when the

‎docs/zh/08-testing.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,13 +96,27 @@ runner 是板级支持包提供的一串 argv,mcpp 把测试二进制附加在
9696
```bash
9797
mcpp test --target thumbv7em-none-eabihf # 为板子构建,经它的 runner 运行
9898
mcpp test --no-runner # 忽略 runner,直接执行
99+
mcpp test --target aarch64-macos --no-run # 只为目标构建测试,不执行
99100
```
100101

101102
测试本身一个字都不用改。同样的 `tests/**/*.cpp` 为设备编译,判据仍然是退出码 ——
102103
这正是裸机 runner 被选成「能产生 semihosting 退出码或 QEMU 退出码」的原因。
103104

104105
`--no-runner` 是给「本机就能原生执行这些二进制、不该为模拟器付代价」的宿主准备的。
105106

107+
`--no-run` 做的是更窄的那个断言,而它必须被显式要求。没有它时,一个本机既不能执行、
108+
也没有 runner 可达的目标,会让每个测试都停在 not run,命令退出 2:mcpp 没有查明这些
109+
测试是否通过,而把它报成成功是本仓记录得最多的一种假读数。但 2 同样是 runner 坏掉时
110+
的退出码,于是一个只想要「构建」的调用方无法区分这两者。在 `--no-run` 下,每个被选中
111+
的测试都为该目标编译并链接,没有任何一个被执行,结果也这么写:
112+
113+
```
114+
test result ok. 0 passed; 0 failed; 2 built, not run
115+
```
116+
117+
编译不过的测试仍然是失败;`--no-run` 与 `--no-runner` 同时给出会被拒绝,而不是在两者
118+
之间挑一个:一个说的是「不经声明的 runner 直接执行」,另一个说的是「不要执行」。
119+
106120
测试程序把它读取的文件带在身边:runner 收到 `MCPP_RUNTIME_FILES`,即它部署的文件与
107121
它加载的共享库的清单,把程序移到设备上的 runner 连同这些文件一起复制。测试按相对于
108122
自身所在目录的路径定位这类文件。在 Android 行上,除非 `cxx_runtime` 另有声明,测试

‎docs/zh/50-machine-output.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -455,8 +455,8 @@ mcpp test [pattern] [--workspace] --message-format json
455455
|---|---|
456456
| `member` | workspace 成员;workspace 之外为 `""` |
457457
| `test` | 按路径命名的测试名(`tests/00-a/0.cpp` → `00-a/0`) |
458-
| `status` | `pass`、`compile_fail`、`run_fail` 或 `not_run` |
459-
| `exit_code` | 测试的退出状态;`not_run` 时为 `0` |
458+
| `status` | `pass`、`compile_fail`、`run_fail`、`not_run` 或 `built` |
459+
| `exit_code` | 测试的退出状态;`not_run` 与 `built` 时为 `0` |
460460
| `signal` | 状态编码了信号时是信号号,否则 `null` |
461461
| `duration_ms` | 这个测试构建+运行的墙钟时间 |
462462
| `timed_out` | 被 `--timeout` 杀掉时为 `true`(`run_fail`) |
@@ -470,8 +470,14 @@ mcpp test [pattern] [--workspace] --message-format json
470470
| `member`、`passed`、`failed` | 计数 |
471471
| `not_run` | 已构建但没有执行的测试数 |
472472
| `not_run_reason` | 它们共同的原因,或 `""` |
473+
| `built` | 在 `--no-run` 下构建、本就不打算执行的测试数 |
473474
| `elapsed_ms`、`build_ms`、`run_ms` | 墙钟时间,分段 |
474475

476+
**`built` 与 `not_run` 是两个不同的答案,分开计数。** 两者描述的都是「编译了但没有
477+
执行」的测试,相似之处到此为止:`not_run` 意味着 mcpp 试过而做不到,因此问题仍然悬着,
478+
退出码是 2;`built` 意味着 `--no-run` 要求不要执行,因此构建就是问题的全部,退出码是 0。
479+
把两者相加的消费方,会把一次它从未要求过的运行,报成一次无法完成的运行。
480+
475481
**`not_run` 既不是 `pass` 也不是 `run_fail`,退出码也这么说(2026.9.2.1)。**
476482
本机无法加载测试产物(交叉目标未声明 runner 时的 `Exec format error`),或声明的
477483
`[target.<triple>].runner` 找不到、启动不了时,测试为 `not_run`。这是关于整次调用的

‎src/build/execute.cppm‎

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2179,6 +2179,22 @@ export struct TestOptions {
21792179
// runner — the operator on this host stating that the triple is native
21802180
// here, a fact the manifest has no axis for (#544, D3).
21812181
bool noRunner = false;
2182+
// `--no-run`: build the tests for `--target` and stop. THE CLAIM IT MAKES
2183+
// IS NARROWER THAN A PASS, AND IT IS STATED RATHER THAN INFERRED.
2184+
//
2185+
// Without it, a target this host cannot execute leaves every test `NotRun`
2186+
// and the command exits 2, which is correct: mcpp did not establish
2187+
// whether the tests pass, and a zero there is the false reading #544
2188+
// records. But `2` is also what a broken runner returns, so a caller that
2189+
// wanted only the build --- a compatibility sweep measuring a target no
2190+
// runner exists for --- cannot tell "the tests built" from "the tests
2191+
// built and the runner is missing" and must settle for `mcpp build`, which
2192+
// builds the package and, for a package whose only sources are under
2193+
// `tests/`, compiles NOTHING of it at all.
2194+
//
2195+
// `--no-run` makes the narrower claim available as its own answer: every
2196+
// selected test compiled and linked for the target, and none was executed.
2197+
bool noRun = false;
21822198
// Per-test RUN deadline. The default is deliberately non-zero: `mcpp test`
21832199
// is something CI runs unattended, and an unbounded default makes a single
21842200
// hung test able to consume the whole job with nothing to show for it.
@@ -2211,6 +2227,10 @@ export struct TestRunSummary {
22112227
// test did not run — and not a pass either: the exit code is 2.
22122228
int notRun = 0;
22132229
std::string notRunReason;
2230+
// Built and deliberately not executed (`--no-run`). Counted apart from
2231+
// `notRun` so the workspace total cannot add a stated build-only result to
2232+
// a run that mcpp could not perform.
2233+
int built = 0;
22142234
long long buildMs = 0; // Phase A + bulk pass + per-test drives
22152235
long long runMs = 0; // the test binaries' own execution
22162236
long long elapsedMs = 0; // wall clock for the whole member
@@ -2404,7 +2424,10 @@ export int run_tests(std::span<const std::string> passthrough,
24042424
// Reporting that as `RunFail (exit 127)` states that the test ran and
24052425
// returned 127, which is false and indistinguishable from a missing
24062426
// program; reporting it as a pass would be read as one.
2407-
enum class St { Pass, CompileFail, RunFail, NotRun } status;
2427+
// `Built` (`--no-run`): compiled and linked, and deliberately not
2428+
// executed. Distinct from `NotRun`, which means mcpp tried and could
2429+
// not --- the difference is whether anything was left unanswered.
2430+
enum class St { Pass, CompileFail, RunFail, NotRun, Built } status;
24082431
int exitCode = 0;
24092432
std::string compileOutput;
24102433
std::string runOutput;
@@ -2422,6 +2445,7 @@ export int run_tests(std::span<const std::string> passthrough,
24222445
const char* st = r.status == TestResult::St::Pass ? "pass"
24232446
: r.status == TestResult::St::CompileFail ? "compile_fail"
24242447
: r.status == TestResult::St::NotRun ? "not_run"
2448+
: r.status == TestResult::St::Built ? "built"
24252449
: "run_fail";
24262450
std::string signal = (r.exitCode > 128 && r.exitCode < 128 + 65)
24272451
? std::to_string(r.exitCode - 128) : "null";
@@ -2891,7 +2915,18 @@ export int run_tests(std::span<const std::string> passthrough,
28912915

28922916
// Pass 2: run them. Concurrently unless there is exactly one — see
28932917
// `runJobs` for why the single-test case is deliberately different.
2894-
run_tests_now(runnable);
2918+
//
2919+
// UNDER `--no-run` THE LIST IS THE ANSWER. Everything that reaches
2920+
// `runnable` compiled and linked; a test that did not is already a
2921+
// `CompileFail` in `results` and keeps that status, so this path reports
2922+
// what was built without also reporting anything about what it does.
2923+
if (testOpts.noRun) {
2924+
for (auto& r : runnable)
2925+
results.push_back({r.name, TestResult::St::Built, 0, {}, {}, 0,
2926+
false, {}});
2927+
} else {
2928+
run_tests_now(runnable);
2929+
}
28952930
summary.elapsedMs = member_ms();
28962931

28972932
// 7. Summary.
@@ -2900,8 +2935,10 @@ export int run_tests(std::span<const std::string> passthrough,
29002935
int notRun = 0;
29012936
std::string notRunReason;
29022937
std::vector<std::string> failures;
2938+
int built = 0;
29032939
for (auto& r : results) {
29042940
if (r.status == TestResult::St::Pass) ++passed;
2941+
else if (r.status == TestResult::St::Built) ++built;
29052942
else if (r.status == TestResult::St::NotRun) {
29062943
++notRun;
29072944
if (notRunReason.empty()) notRunReason = r.reason;
@@ -2913,6 +2950,7 @@ export int run_tests(std::span<const std::string> passthrough,
29132950
summary.failed = failed;
29142951
summary.notRun = notRun;
29152952
summary.notRunReason = notRunReason;
2953+
summary.built = built;
29162954

29172955
// "build X + run Y" rather than one merged number: on a member whose tests
29182956
// are cheap but whose link is not, those two are three orders of magnitude
@@ -2932,9 +2970,10 @@ export int run_tests(std::span<const std::string> passthrough,
29322970
if (json) {
29332971
std::println("{{\"summary\":{{\"member\":\"{}\",\"passed\":{},\"failed\":{},"
29342972
"\"not_run\":{},\"not_run_reason\":\"{}\","
2973+
"\"built\":{},"
29352974
"\"elapsed_ms\":{},\"build_ms\":{},\"run_ms\":{}}}}}",
29362975
test_json_escape(memberName), passed, failed,
2937-
notRun, test_json_escape(notRunReason),
2976+
notRun, test_json_escape(notRunReason), built,
29382977
summary.elapsedMs, summary.buildMs, summary.runMs);
29392978
std::fflush(stdout);
29402979
return rc;
@@ -2944,6 +2983,7 @@ export int run_tests(std::span<const std::string> passthrough,
29442983
// its reason: a quiet skip is read as a pass. First line of the reason
29452984
// only — the full text was printed when it was established.
29462985
auto counts = std::format("{} passed; {} failed", passed, failed);
2986+
if (built) counts += std::format("; {} built, not run", built);
29472987
if (notRun) {
29482988
auto firstLine = notRunReason.substr(0, notRunReason.find('\n'));
29492989
counts += std::format("; {} not run ({})", notRun, firstLine);

‎src/cli.cppm‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ void print_usage() {
6161
std::println(" mcpp new <name> Create a new package skeleton");
6262
std::println(" mcpp build [options] Build the current package");
6363
std::println(" mcpp run [target] [-- args...] Build + run a binary target");
64-
std::println(" mcpp test [pattern] [-- args...] Build + run tests/**/*.cpp (--list, --timeout, --build-timeout, --message-format json, --no-runner)");
64+
std::println(" mcpp test [pattern] [-- args...] Build + run tests/**/*.cpp (--list, --no-run, --timeout, --build-timeout, --message-format json, --no-runner)");
6565
std::println(" mcpp clean [--stale] [--bmi-cache] Remove target/ (or, with --stale, only its non-current fingerprint dirs)");
6666
std::println(" mcpp add [ns.]pkg@ver Add an exact dependency to mcpp.toml");
6767
std::println(" mcpp remove [ns.]pkg Remove an exact dependency from mcpp.toml");
@@ -502,6 +502,8 @@ int run(int argc, char** argv) {
502502
.help("List (filtered) tests without building or running them"))
503503
.option(cl::Option("no-runner")
504504
.help("Run test binaries directly, ignoring any [target.<triple>].runner (a host that runs them natively)"))
505+
.option(cl::Option("no-run")
506+
.help("Build the tests for --target and stop; a build that succeeds is the result (for a target this host cannot execute)"))
505507
.option(cl::Option("timeout").takes_value().value_name("SECS")
506508
.help("Kill a test still RUNNING after SECS seconds (default 300; 0 = no limit)"))
507509
.option(cl::Option("build-timeout").takes_value().value_name("SECS")

‎src/cli/cmd_build.cppm‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -539,6 +539,16 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed,
539539
if (parsed.positional_count() > 0) to.filter = parsed.positional(0);
540540
to.list = parsed.is_flag_set("list");
541541
to.noRunner = parsed.is_flag_set("no-runner"); // see cmd_run
542+
to.noRun = parsed.is_flag_set("no-run");
543+
// The two read alike and mean opposite things: `--no-runner` says to run
544+
// the binaries WITHOUT the declared runner, `--no-run` says not to run
545+
// them at all. Asking for both is not a preference to resolve.
546+
if (to.noRun && to.noRunner) {
547+
mcpp::ui::error("--no-run and --no-runner cannot be combined: "
548+
"--no-runner runs the test binaries directly, "
549+
"--no-run does not run them.");
550+
return 2;
551+
}
542552
// The three deadlines share one parser: they differ only in what they
543553
// bound, not in how they are spelled. 0 always means "no limit" — for
544554
// --timeout that now has to be asked for rather than being the default.
@@ -630,9 +640,15 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed,
630640
std::format("member '{}' ({}/{}) FAILED — {} passed, {} failed in {:.2f}s",
631641
mp, idx, members->size(), sum.passed, sum.failed, secs));
632642
} else {
643+
// Under `--no-run` nothing passed and nothing was meant to:
644+
// reporting "0 passed" for a member whose tests all built is
645+
// the same sentence a member with no tests would produce.
633646
mcpp::ui::status("Workspace",
634-
std::format("member '{}' ({}/{}) ok — {} passed in {:.2f}s",
635-
mp, idx, members->size(), sum.passed, secs));
647+
sum.built
648+
? std::format("member '{}' ({}/{}) ok — {} built, not run in {:.2f}s",
649+
mp, idx, members->size(), sum.built, secs)
650+
: std::format("member '{}' ({}/{}) ok — {} passed in {:.2f}s",
651+
mp, idx, members->size(), sum.passed, secs));
636652
}
637653
}
638654

0 commit comments

Comments
 (0)