Skip to content

fix(plugin): decode diff Git paths as UTF-8 - #543

Open
sylvesterkaczmarek wants to merge 3 commits into
openai:mainfrom
sylvesterkaczmarek:fix/diff-rank-git-utf8
Open

fix(plugin): decode diff Git paths as UTF-8#543
sylvesterkaczmarek wants to merge 3 commits into
openai:mainfrom
sylvesterkaczmarek:fix/diff-rank-git-utf8

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Decode Git filename lists explicitly as UTF-8 in diff ranking instead of using Python's process-locale decoder.

Fixes #533.

Reproduction / evidence

Current upstream main at 37bf87a692fc72d41f7312cc48808d699d204fba has two direct text-mode Git subprocesses in generate_rank_input.py without an explicit encoding:

  • git diff --name-status -z for committed, staged, and unstaged changed paths;
  • git ls-files --others --exclude-standard -z for working-tree untracked paths.

With text=True, Python chooses the process locale encoding. On Windows with a non-UTF-8 ANSI code page, Git's UTF-8 filename output can therefore decode incorrectly or fail before ranking begins. The repository path itself can be ASCII; a changed filename containing non-ASCII characters is sufficient.

This exact call site was independently identified in the earlier closed #293 investigation while addressing #192. That PR documented failures for non-ASCII Git filenames and explicitly noted this helper as a second call site. The current helper still contained both locale-dependent subprocesses.

The regression in this PR instruments subprocess.run inside the actual Python helper and refuses every Git text invocation unless encoding == "utf-8". It exercises both the changed-path and untracked-path flows using synthetic 測試.py and 新規.py names. On the unfixed helper the assertion fails because encoding is absent.

Root cause

The diff-ranking helper launches Git directly rather than routing these path-list operations through the already-hardened workbench Git helper. Its text-mode subprocesses therefore inherited Python's locale decoder.

Fix

Add encoding="utf-8" to both text-mode Git filename-list subprocesses:

  • changed-path enumeration;
  • untracked-file enumeration.

No error handler or replacement decoding is added. Invalid UTF-8 still fails rather than silently changing a filename.

Tests / validation

Added diff-rank-git-encoding.test.ts. The test imports the real bundled Python helper, replaces only its subprocess boundary, and verifies:

  • every text-mode Git call requests UTF-8;
  • committed non-ASCII paths are parsed correctly;
  • working-tree changed and untracked non-ASCII paths are preserved;
  • synthetic path output is normalized for cross-platform CI.

The branch is based directly on current upstream main at 37bf87a692fc72d41f7312cc48808d699d204fba and is not behind it. Production change: exactly 2 additions, 0 deletions. The remainder is focused regression coverage.

Full repository tests cannot be run in this execution environment because the repository cannot be cloned here. Pushed-head CI remains the authoritative full-suite validation.

Risk

Low. Git filename decoding becomes deterministic UTF-8 instead of locale-dependent. ASCII paths are unchanged, and binary Git operations are untouched.

@github-actions github-actions Bot added the bug Something isn't working label Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Diff ranking still decodes Git filenames with the Windows locale

1 participant