feat(suite): line-range skipping for Lua 5.3 official tests#249
Merged
Conversation
Replace whole-file `@tag :skip` with per-file line ranges. Each suite file can now run partially, with only the failing lines skipped, so every assertion outside the skipped range contributes to the green signal. The skip list itself becomes the conformance backlog ahead of 1.0.0. - `Lua.SuiteRunner.run_file/2` accepts `:skip_ranges`, replacing each skipped line with a one-line comment to preserve line numbers (load-bearing for `Lua.VM.AssertionError`). - `test/lua53_skips.exs` holds the per-file skip map. Each entry pairs a line range with a category, one-line reason, and optional GitHub issue number. `lines: :all` marks files awaiting initial triage. - `test/lua53_suite_test.exs` generates one ExUnit test per suite file from the skip map. Test names carry the conformance debt count (e.g. `pm.lua (47 lines skipped, 3 ranges)`). - `mix lua.suite --status` summarises the conformance backlog (totals by category and tracking issue). - `mix lua.suite --audit` re-runs each file with each entry omitted to surface stale ranges and promotion candidates. - Triage skill rewritten so §6.C produces a skip-range entry instead of a whole-file skip. Documents range edge cases (multi-line strings, local-decl coupling, if/then/end). Zero behavior change for the existing suite: same 6 ready files pass, the 19 previously-skipped files keep `lines: :all` (still tagged `:skip`), and the 4 permanently-deferred files (filesystem I/O, subprocess) stay in their own `@deferred_permanent` attribute.
- Validate skip file: raise when a file mixes `lines: :all` with specific ranges. The test driver treats `:all` as a hard skip and would silently drop the ranges. Loader lives on `Lua.SuiteRunner` so both the Mix task and the test module enforce it. - Audit labels exceptions without a `:line` field (e.g. `Lua.VM.InternalError`, `Lua.CompilerException`) as `unknown line (StructName)` instead of bare `?`. - Collapse four passes over the skip map in `--status` into one reducer; extract per-file printing for clarity. - `range_to_string/1` handles non-default step ranges.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@tag :skipfor the Lua 5.3 official suite. Suite files now run with specific line ranges commented out, so every assertion outside those ranges contributes to the green signal.test/lua53_skips.exsas the central conformance backlog. Each entry pairs aRangewith a category, one-line reason, and optional GitHub issue number. The skip list answers "what's left before 1.0.0?" from one file.mix lua.suite --status(fast summary) andmix lua.suite --audit(re-runs each file with each entry omitted to surface stale ranges and promotion candidates).triage-suite-failureskill to produce line-range entries instead of whole-file skips.Zero behavior change in this PR: same 6 ready files pass, the 19 previously-skipped files keep
lines: :all(still tagged:skip), and the 4 permanently-deferred files stay in@deferred_permanent. The follow-up triage of the 19:allfiles happens in subsequent PRs — themix lua.suite --auditoutput already gives first-failure line numbers for each as a starting point.Why
Direction A's goal (ROADMAP.md) is pushing the suite from 6/29 ready to a healthier pass rate before
1.0.0-rc.1. The file-level skip mechanism throws away signal: a single failing assertion inpm.luablocks the rest of the file from running. With line-range skipping we get incremental progress per assertion, andmix lua.suite --auditautomatically detects when fixes elsewhere have made a skip range obsolete.Key design choices
apply_skip_ranges/2replaces each skipped line with-- skipped (suite triage): was line N, preserving line numbers. This is load-bearing —Lua.VM.AssertionErrorauto-captures source position viaLua.VM.Executor.current_position/0, so the line in the error must match the line in the file. A regression test intest/suite_runner_test.exspins this invariant.lines: :allas a pressure-release valve. Files that need deeper triage stay at:all(still@tag :skip'd) without blocking the mechanism PR.main.lua,files.lua,attrib.lua,verybig.luaremain in@deferred_permanent— they're "we will never fix" (filesystem I/O, subprocess), categorically different from the new "fix before 1.0" backlog.Test plan
mix test— 1898 passed, 30 skipped (was 1894/30 before adding the 9 new tests in this PR)mix test test/lua53_suite_test.exs --include lua53— same 6 passing, 23 skippedmix format --check-formattedmix lua.suite --status— prints per-file summary, totals by category and issuemix lua.suite --audit --timeout 8000— reports ACTIVE/CANDIDATE/STALE/MOVED for each entry1..3skip range appliedFollow-ups
:allentries — one PR per file (or small group), turning whole-file skips into specific ranges with reasons and (where applicable) tracking issues.mix lua.suite --auditto surface stale entries automatically.