The Behat suite occasionally reports a single failed scenario and then passes unchanged on re-run. Observed on 8.4 during the PHP 8.5 work:
PHP 8.4.4 behat=1 433 scenarios (432 passed, 1 failed)
# immediate re-run, no changes
PHP 8.4.4 behat=0 433 scenarios (433 passed)
It predates the 8.5 work and is not tied to a particular scenario. The suspected cause is timing in ReplProcessManager, which drives a real REPL subprocess over pipes: ReplSteps::definesUserType() switches to that manager when input declares a class/trait/interface/enum/function or uses spread syntax, and output is read with the timeouts in .env (REPL_OUTPUT_TIMEOUT, REPL_PROMPT_TIMEOUT). A slow spawn or a partial read looks like missing output.
Worth fixing because it makes CI red at random, which trains people to re-run rather than read failures.
Suggested starting point: have ReplOutputReader wait for the prompt sentinel rather than a wall-clock timeout, and log the raw buffer on timeout so an intermittent failure carries enough evidence to diagnose without reproducing it.
The Behat suite occasionally reports a single failed scenario and then passes unchanged on re-run. Observed on 8.4 during the PHP 8.5 work:
It predates the 8.5 work and is not tied to a particular scenario. The suspected cause is timing in
ReplProcessManager, which drives a real REPL subprocess over pipes:ReplSteps::definesUserType()switches to that manager when input declares a class/trait/interface/enum/function or uses spread syntax, and output is read with the timeouts in.env(REPL_OUTPUT_TIMEOUT,REPL_PROMPT_TIMEOUT). A slow spawn or a partial read looks like missing output.Worth fixing because it makes CI red at random, which trains people to re-run rather than read failures.
Suggested starting point: have
ReplOutputReaderwait for the prompt sentinel rather than a wall-clock timeout, and log the raw buffer on timeout so an intermittent failure carries enough evidence to diagnose without reproducing it.