Skip to content

fix(code_indexer): honor ensure_ascii config instead of inverting it#139

Open
Osamaali313 wants to merge 1 commit into
HKUDS:mainfrom
Osamaali313:fix/indexer-ensure-ascii-inverted
Open

fix(code_indexer): honor ensure_ascii config instead of inverting it#139
Osamaali313 wants to merge 1 commit into
HKUDS:mainfrom
Osamaali313:fix/indexer-ensure-ascii-inverted

Conversation

@Osamaali313

Copy link
Copy Markdown

Problem

Three JSON-writing paths in tools/code_indexer.py invert the ensure_ascii config flag:

json_indent = output_config.get("json_indent", 2)
ensure_ascii = not output_config.get("ensure_ascii", False)   # <-- stray `not`
...
json.dump(index_data, f, indent=json_indent, ensure_ascii=ensure_ascii)

(at lines 1131 / 1325 / 1341 — build_all_indexes, generate_statistics_report, generate_summary_report).

This is clearly meant to be a pass-through:

  • the sibling option one line above, json_indent, is read verbatim with no not;
  • the config field is literally named ensure_ascii — the same keyword it feeds to json.dump;
  • json.dump is used with ensure_ascii=False directly elsewhere in the file (line 445).

The shipped default is ensure_ascii: false (tools/indexer_config.yaml:100), i.e. keep literal UTF-8. But not FalseTrue, so the code escapes every non-ASCII character (Chinese text, accented identifiers in LLM-generated file summaries/concepts) as \uXXXX in the index/statistics/summary JSON — the opposite of the configured behavior.

Reproduction

Config {"ensure_ascii": false}, data {"summary": "推荐 café"}:

output
before (not inverts) {"summary": "推荐 café"}
after (pass-through) {"summary": "推荐 café"}

These functions are live: workflows/codebase_index_workflow.py invokes build_all_indexes(), generate_summary_report(), and generate_statistics_report().

Fix

Drop the not at all three sites:

ensure_ascii = output_config.get("ensure_ascii", False)

No test suite currently covers code_indexer.py. python -m py_compile tools/code_indexer.py passes.

Three JSON-writing paths computed
  ensure_ascii = not output_config.get("ensure_ascii", False)
and passed it to json.dump. The stray `not` inverts the configured flag: the
sibling option on the line above (json_indent) is a direct pass-through, the
config field name equals the json.dump parameter, and the shipped default in
indexer_config.yaml is `ensure_ascii: false`. So with the default config the
code sent ensure_ascii=True, escaping every non-ASCII char (Chinese text,
accented identifiers) as \uXXXX in the index/statistics/summary JSON. Drop
the `not` so the flag is passed through as configured.
Copilot AI review requested due to automatic review settings July 14, 2026 18:32

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants