diff --git a/CHANGELOG.md b/CHANGELOG.md index e9a140d9..e0605793 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [Unreleased] + +### Fixed + +- **A `--lang-bias-config` YAML file with two or more languages now steers deterministically** (#1267). The `bias:` block deserialized into a `HashMap`, whose iteration order `RandomState` randomizes per map instance, and that iteration order was pushed straight into `LangBiasSet.ordered`, which is the priority order `TokenLanguageIndex::to_token_bias` resolves conflicts with under first-language-wins. Han is shared by `ja`, `zh` and `ko`, so a config naming two or more CJK languages assigned a different bias to every shared Han token on every run, with no error and no warning, and the one place the order surfaced was the `languages` field of a DEBUG-level tracing event that is off by default. The shipped schema example is itself a three-CJK config, so copying it was enough to hit this. The `bias:` block is now collected through `MapAccess` into an ordered `Vec`, so index 0 is the first language written in the file, matching what `--lang-bias ja=-inf,zh=-10.0,ko=+5.0` and the `LLAMA_ARG_LANG_BIAS` env fallback have always done. **The accepted YAML syntax is unchanged**: `bias:` is still a plain mapping and existing config files keep working, though a file relying on the previous random behavior will now steer consistently toward its first-listed language. +- **A language code repeated inside one YAML `bias:` block is rejected** (#1267). `serde_yaml` resolves a repeated key into a typed `HashMap` last-wins with no diagnostic, which made the duplicate check in the resolve path unreachable and let the YAML path silently accept input the `--lang-bias` parser has always rejected. The ordered representation delivers both occurrences, so both entry points now fail with `duplicate language code '' in language bias entries (ambiguous priority)`. This is a behavior change for any config file that repeats a language code: it previously took the last occurrence and now errors out. + ## [v0.5.2] - 2026-08-18 ### Changed diff --git a/TECHNICAL_REPORTS/1269-lang-bias-yaml-order-20260822.en.md b/TECHNICAL_REPORTS/1269-lang-bias-yaml-order-20260822.en.md new file mode 100644 index 00000000..821ed547 --- /dev/null +++ b/TECHNICAL_REPORTS/1269-lang-bias-yaml-order-20260822.en.md @@ -0,0 +1,70 @@ +# Technical Report: PR #1269 - Deterministic language priority from a YAML bias block + +## Executive Summary + +`LangBiasSet.ordered` is documented as "pairs in priority order (index 0 = highest priority)" and its consumer `to_token_bias` resolves shared tokens under "first-language-wins". The YAML config path built that order by iterating a `HashMap`, so `RandomState` decided the priority. Han script is shared by `ja`, `zh` and `ko`, which means a `--lang-bias-config` file naming two or more CJK languages assigned a different bias to every shared token on every run, silently. + +This is the same root-cause class as #1265 (`HashMap` iteration order leaking into ordered state), but in production code rather than a test fixture. The fix collects the `bias:` block through `MapAccess` into an ordered `Vec`, so the author's document order becomes the priority order, matching what `--lang-bias` and the `LLAMA_ARG_LANG_BIAS` fallback have always done. + +## 1. Problem Statement + +Three entry points produce a `LangBiasSet` and only one of them was wrong. + +`parse_lang_bias_entries` walks `s.split(',')` in document order, uses a `seen` map purely as a membership set, and rejects duplicates with `CliError::DuplicateLanguageCode`. `env_fallback_lang_bias` routes `LLAMA_ARG_LANG_BIAS` through that same parser. The YAML path deserialized `bias:` into `Option>` and pushed the map's iteration order straight into `ordered`. + +Two consequences followed, and the second was invisible. Priority became random per run. And because `serde_yaml` resolves a repeated key into a typed `HashMap` last-wins with no diagnostic, the duplicate check in the resolve loop was unreachable, so the YAML path silently accepted input the `--lang-bias` parser has always rejected. + +The trigger is not exotic: the schema example in the `LangBiasYamlConfig` doc comment is itself a three-CJK config, so copying the documented example was enough. + +## 2. Technical Decisions + +### 2.1 A hand-written `Deserialize` rather than an order-preserving map type + +Three candidates were considered. `IndexMap` as the field type preserves order but resolves duplicates last-wins, which would leave the dead check dead and keep the YAML path accepting what the CLI rejects. `serde_yaml::Mapping` is `IndexMap`-backed and does reject duplicates, but with serde_yaml's own error rather than `CliError::DuplicateLanguageCode`, so the two entry points would still disagree on the error surfaced. + +The chosen shape is a `BiasEntries` newtype over `Vec<(String, BiasValueStr)>` with a hand-written `Deserialize` that collects through `MapAccess::next_entry`. Document order survives, a repeated key arrives as a second entry that the existing resolve loop rejects with the repository's own error, and no new dependency is added. `indexmap` is present in `Cargo.lock` but not in `Cargo.toml`, so both map-type candidates would have promoted a transitive dependency to a direct one. + +### 2.2 The accepted YAML schema does not change + +The visitor implements `visit_map` and the entry point is `deserialize_map`, so `bias:` remains a plain mapping. `#[serde(deny_unknown_fields)]` is untouched, an absent or empty block still resolves to an empty set, and a sequence-shaped block is still a parse error. Existing config files keep working. A fix that required users to rewrite `bias:` as a list would have been the wrong shape for a bug report about ordering. + +### 2.3 The duplicate error message names both surfaces + +Making the check reachable from YAML made the existing text, which named only `--lang-bias`, actively wrong for a user who wrote a YAML file. The variant and its `code` field are unchanged; only the message broadened to name both surfaces. + +## 3. Change Summary + +| File | Change | +| --- | --- | +| `src/lang_bias.rs` | `BiasEntries` newtype plus its `Deserialize`; `LangBiasYamlConfig::bias` retyped; duplicate error message broadened; new tests | +| `CHANGELOG.md` | `## [Unreleased]` entries for the two user-visible behavior changes | + +## 4. Review Findings + +The requirement that carried the most weight was not a review finding but a precondition: the regression tests had to be demonstrated failing against the unfixed code. This bug class is unusually good at producing tests that pass before and after, because a single `resolve()` returns the correct order by luck a fair fraction of the time. + +The demonstration reverted only the field type (and the one pre-existing test whose membership assertions would not compile against the ordered type), ran the suite, and restored from a copy rather than using `git stash`, so untracked work was never at risk. Four tests failed, and the output showed three distinct permutations of the same three-key file within a single process run: + +``` +iteration 0: [(Zh, -10.0), (Ko, 5.0), (Ja, -inf)] +iteration 2: [(Zh, -10.0), (Ja, -inf), (Ko, 5.0)] + : [(Ja, -inf), (Zh, -10.0), (Ko, 5.0)] +``` + +That independently reproduces the measurement taken while filing #1267: `RandomState` randomizes per `HashMap` instance, not merely per process (ten maps from the same five keys gave nine distinct orders in one process). It is also why each ordering test runs 32 full `resolve()` calls rather than one. + +## 5. Validation + +Measured on GB10 (DGX Spark, CUDA sm_121, Linux aarch64). + +- `cargo test --profile test-fast --features cuda --lib lang_bias`: 30 passed, exit 0. Against the pre-fix field type: 26 passed, 4 failed, exit 101. +- `cargo fmt --all -- --check`, `cargo clippy --lib --tests --features cuda -- -D warnings`, `cargo check --lib --tests --features cuda`, `cargo check --bins --features cuda`: all exit 0. +- `make verify-test-cuda`: recorded in the PR thread. + +## 6. Related Work + +- #1267: the issue this closes, filed from the review sweep on PR #1268. +- #1265 and PR #1266: the same root-cause class in four test fixtures, and the origin of the sweep. +- #1277 and #1276: two further instances found by the same sweep, in the distributed registry accessors and in the RT-DETRv2 checkpoint layout sniffer. + +Four independent instances of one pattern in a single sweep is the finding that outlives this PR. The pattern is a `HashMap` iteration result becoming an ordered or order-sensitive decision, and nothing in the toolchain flags it: the types are correct, the code compiles, and the tests pass most of the time. diff --git a/TECHNICAL_REPORTS/1269-lang-bias-yaml-order-20260822.ko.md b/TECHNICAL_REPORTS/1269-lang-bias-yaml-order-20260822.ko.md new file mode 100644 index 00000000..8ee82e98 --- /dev/null +++ b/TECHNICAL_REPORTS/1269-lang-bias-yaml-order-20260822.ko.md @@ -0,0 +1,70 @@ +# 기술 보고서: PR #1269 - YAML bias 블록에서 결정적인 언어 우선순위 + +## 요약 + +`LangBiasSet.ordered`는 "pairs in priority order (index 0 = highest priority)"로 문서화돼 있고, 소비자 `to_token_bias`는 공유 토큰을 first-language-wins로 해소한다. 그런데 YAML 설정 경로가 그 순서를 `HashMap` 순회로 만들고 있었다. 즉 우선순위를 `RandomState`가 정했다. Han 문자는 `ja`, `zh`, `ko`가 공유하므로, CJK 언어를 둘 이상 적은 `--lang-bias-config` 파일은 공유 토큰마다 실행할 때마다 다른 바이어스를 배정했다. 조용히. + +#1265와 같은 근본 원인 계열(`HashMap` 순회 순서가 순서 의존 상태로 새는 것)이지만, 테스트 픽스처가 아니라 프로덕션 코드다. 수정은 `bias:` 블록을 `MapAccess`로 훑어 순서 있는 `Vec`에 모으는 것이고, 그러면 작성자가 파일에 쓴 순서가 곧 우선순위가 된다. `--lang-bias`와 `LLAMA_ARG_LANG_BIAS` 폴백이 처음부터 해오던 것과 같아진다. + +## 1. 문제 + +`LangBiasSet`을 만드는 진입점이 셋인데 틀린 것은 하나뿐이었다. + +`parse_lang_bias_entries`는 `s.split(',')`를 문서 순서로 훑고, `seen` 맵을 멤버십 집합으로만 쓰며, 중복을 `CliError::DuplicateLanguageCode`로 거부한다. `env_fallback_lang_bias`는 `LLAMA_ARG_LANG_BIAS`를 같은 파서로 보낸다. YAML 경로만 `bias:`를 `Option>`로 역직렬화하고 맵의 순회 순서를 그대로 `ordered`에 밀어 넣었다. + +결과가 둘인데 두 번째는 보이지도 않았다. 우선순위가 실행마다 무작위가 됐다. 그리고 `serde_yaml`이 반복된 키를 타입 지정 `HashMap`에 last-wins로 넣고 아무 진단도 내지 않기 때문에, resolve 루프의 중복 검사가 도달 불가가 됐다. YAML 경로가 `--lang-bias` 파서라면 계속 거부해 온 입력을 조용히 받아들이고 있었다는 뜻이다. + +트리거는 이국적이지 않다. `LangBiasYamlConfig` 문서 주석의 스키마 예시 자체가 CJK 3종 설정이라, 문서화된 예시를 복사하는 것으로 충분했다. + +## 2. 기술적 판단 + +### 2.1 순서 보존 맵 타입이 아니라 손으로 쓴 `Deserialize` + +후보 셋을 놓고 골랐다. 필드 타입을 `IndexMap`으로 하면 순서는 지키지만 중복은 last-wins라 죽은 검사가 계속 죽어 있고 YAML 경로는 CLI가 거부하는 입력을 계속 받는다. `serde_yaml::Mapping`은 `IndexMap` 기반이라 중복을 거부하긴 하는데 `CliError::DuplicateLanguageCode`가 아니라 serde_yaml 자체 오류를 내므로 두 진입점이 내는 오류가 여전히 어긋난다. + +택한 형태는 `Vec<(String, BiasValueStr)>` 위의 `BiasEntries` 뉴타입이고, `MapAccess::next_entry`로 모으는 `Deserialize`를 손으로 썼다. 문서 순서가 살아남고, 반복된 키는 두 번째 항목으로 도착해 기존 resolve 루프가 저장소 자체 오류로 거부하며, 새 의존성이 없다. `indexmap`은 `Cargo.lock`에는 있지만 `Cargo.toml`에는 없어서, 맵 타입 후보 둘은 전이 의존성을 직접 의존성으로 승격시켰을 것이다. + +### 2.2 허용되는 YAML 스키마는 바뀌지 않는다 + +visitor가 `visit_map`을 구현하고 진입점이 `deserialize_map`이라 `bias:`는 여전히 평범한 매핑이다. `#[serde(deny_unknown_fields)]`는 그대로고, 블록이 없거나 비어 있으면 여전히 빈 집합으로 풀리며, 시퀀스 모양 블록은 여전히 파싱 오류다. 기존 설정 파일은 계속 동작한다. 사용자에게 `bias:`를 리스트로 다시 쓰라고 요구하는 수정이었다면 순서에 관한 버그 리포트에 대한 답으로는 형태가 틀렸을 것이다. + +### 2.3 중복 오류 메시지가 두 표면을 모두 지목한다 + +YAML에서 검사가 도달 가능해지자, `--lang-bias`만 지목하던 기존 문구가 YAML 파일을 쓴 사용자에게는 적극적으로 틀린 말이 됐다. variant와 `code` 필드는 그대로 두고 메시지만 두 표면을 다 언급하도록 넓혔다. + +## 3. 변경 요약 + +| 파일 | 변경 | +| --- | --- | +| `src/lang_bias.rs` | `BiasEntries` 뉴타입과 `Deserialize`, `LangBiasYamlConfig::bias` 타입 변경, 중복 오류 메시지 확장, 신규 테스트 | +| `CHANGELOG.md` | 사용자 가시 동작 변경 2건에 대한 `## [Unreleased]` 항목 | + +## 4. 리뷰 지적사항 + +가장 무겁게 잡은 요건은 리뷰 지적이 아니라 전제조건이었다. 회귀 테스트가 수정 전 코드에서 실제로 실패하는 것을 증명해야 했다. 이 버그 계열은 수정 전후로 다 통과하는 테스트를 만들어내는 데 특히 능하다. `resolve()` 한 번은 상당한 확률로 우연히 맞는 순서를 낸다. + +증명은 필드 타입만 되돌리고(그리고 순서 있는 타입에서 컴파일되지 않는 기존 테스트 하나의 멤버십 단언만), 스위트를 돌린 뒤 `git stash`가 아니라 복사본에서 복원하는 방식으로 했다. 미추적 작업이 위험해지지 않는다. 테스트 4건이 실패했고, 출력에 같은 3키 파일의 서로 다른 순열이 **한 프로세스 안에서 세 가지** 나왔다. + +``` +iteration 0: [(Zh, -10.0), (Ko, 5.0), (Ja, -inf)] +iteration 2: [(Zh, -10.0), (Ja, -inf), (Ko, 5.0)] + : [(Ja, -inf), (Zh, -10.0), (Ko, 5.0)] +``` + +#1267을 발행하며 잰 측정을 독립적으로 재현한 결과다. `RandomState`는 프로세스가 아니라 **`HashMap` 인스턴스마다** 무작위화한다(같은 다섯 키로 만든 맵 10개가 한 프로세스에서 고유 순서 9개). 각 순서 테스트가 `resolve()`를 1회가 아니라 32회 도는 이유도 그것이다. + +## 5. 검증 + +GB10(DGX Spark, CUDA sm_121, Linux aarch64)에서 실측. + +- `cargo test --profile test-fast --features cuda --lib lang_bias`: 30 통과, exit 0. 수정 전 필드 타입 대비: 26 통과, 4 실패, exit 101. +- `cargo fmt --all -- --check`, `cargo clippy --lib --tests --features cuda -- -D warnings`, `cargo check --lib --tests --features cuda`, `cargo check --bins --features cuda`: 전부 exit 0. +- `make verify-test-cuda`: PR 스레드에 기록. + +## 6. 관련 작업 + +- #1267: 이 PR이 닫는 이슈. PR #1268 리뷰 스윕에서 나왔다. +- #1265, PR #1266: 테스트 픽스처 네 곳의 같은 근본 원인이자 스윕의 출발점. +- #1277, #1276: 같은 스윕이 찾은 추가 인스턴스 둘. 분산 레지스트리 접근자와 RT-DETRv2 체크포인트 레이아웃 판별. + +한 번의 스윕에서 같은 패턴이 독립적으로 넷 나온 것이 이 PR보다 오래 남을 발견이다. 패턴은 `HashMap` 순회 결과가 순서 있는 또는 순서에 민감한 결정이 되는 것이고, 툴체인 어느 것도 이걸 잡지 못한다. 타입은 맞고, 컴파일되고, 테스트는 대체로 통과한다. diff --git a/src/lang_bias.rs b/src/lang_bias.rs index dea7ea74..87db33be 100644 --- a/src/lang_bias.rs +++ b/src/lang_bias.rs @@ -52,7 +52,12 @@ pub enum CliError { entry: String, reason: String, }, - #[error("duplicate language code '{code}' in --lang-bias (ambiguous priority)")] + /// Raised by both language-bias entry points: the `--lang-bias` string + /// parser and the YAML `bias:` block, which reject a repeated language code + /// identically because the repeat makes the priority order ambiguous. + #[error( + "duplicate language code '{code}' in language bias entries (ambiguous priority); check --lang-bias and the YAML bias: block" + )] DuplicateLanguageCode { code: String }, #[error("failed to read lang-bias config file '{path}': {source}")] ConfigReadError { @@ -171,17 +176,105 @@ pub fn parse_lang_bias_entries(s: &str) -> Result { /// ``` /// /// Unknown top-level keys produce a parse error via `#[serde(deny_unknown_fields)]`. +/// +/// The order of the `bias:` entries is significant and is preserved exactly as +/// written: index 0 is the highest priority. `to_token_bias` resolves a token +/// claimed by several languages with first-language-wins, so in the example +/// above the Han tokens shared by `ja`, `zh` and `ko` all receive `ja`'s +/// `-inf`. Writing the same three languages in a different order is a +/// different configuration, not a cosmetic difference. +/// +/// A language code repeated inside one `bias:` block is rejected with +/// [`CliError::DuplicateLanguageCode`], the same error the equivalent +/// `--lang-bias` string produces. #[derive(Debug, Deserialize)] #[serde(deny_unknown_fields)] pub struct LangBiasYamlConfig { #[serde(default)] pub policy: Option, #[serde(default)] - pub bias: Option>, + pub bias: Option, #[serde(default)] pub exceptions: Option, } +/// The `bias:` block of a YAML config, in document order. +/// +/// Deserializing that block into a `HashMap` (what this field used to be) threw +/// away the two properties the resolve loop depends on. `HashMap` iteration +/// order is randomized by `RandomState` per map instance, so the priority order +/// handed to `to_token_bias` was a fresh random permutation on every load, and +/// the resulting bias assigned to a shared Han token changed from run to run for +/// one unchanged config file. `HashMap` also collapses repeated keys during +/// deserialization (serde_yaml resolves them last-wins with no diagnostic), +/// which made the resolve loop's duplicate check unreachable and let the YAML +/// path silently accept input the `--lang-bias` parser rejects. See issue #1267. +/// +/// Collecting the entries through `MapAccess` into a `Vec` keeps both: the +/// author's order survives, and a repeated key arrives as a second entry that +/// the resolve loop can reject. The accepted YAML syntax is unchanged, `bias:` +/// is still a plain mapping. +#[derive(Debug, Default)] +pub struct BiasEntries(Vec<(String, BiasValueStr)>); + +impl BiasEntries { + /// The `(language code, bias)` pairs in the order they appear in the document. + pub fn as_slice(&self) -> &[(String, BiasValueStr)] { + &self.0 + } + + /// Number of entries, counting a repeated language code once per occurrence. + pub fn len(&self) -> usize { + self.0.len() + } + + /// Returns `true` when the `bias:` block is present but empty. + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } +} + +impl IntoIterator for BiasEntries { + type Item = (String, BiasValueStr); + type IntoIter = std::vec::IntoIter<(String, BiasValueStr)>; + + fn into_iter(self) -> Self::IntoIter { + self.0.into_iter() + } +} + +impl<'de> Deserialize<'de> for BiasEntries { + fn deserialize(deserializer: D) -> Result + where + D: serde::Deserializer<'de>, + { + struct BiasEntriesVisitor; + + impl<'de> serde::de::Visitor<'de> for BiasEntriesVisitor { + type Value = BiasEntries; + + fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("a mapping of language code to bias value") + } + + fn visit_map(self, mut access: M) -> Result + where + M: serde::de::MapAccess<'de>, + { + let mut entries = Vec::with_capacity(access.size_hint().unwrap_or(0)); + // Deliberately a Vec push per entry rather than a map insert: + // repeated keys must reach the caller so it can reject them. + while let Some(entry) = access.next_entry::()? { + entries.push(entry); + } + Ok(BiasEntries(entries)) + } + } + + deserializer.deserialize_map(BiasEntriesVisitor) + } +} + /// Wraps a YAML `policy:` string value with custom deserialization. #[derive(Debug, Deserialize)] #[serde(rename_all = "lowercase")] @@ -402,6 +495,10 @@ impl LangBiasCliArgs { } if let Some(yaml_bias) = yaml.bias { + // `BiasEntries` yields the `bias:` block in document order and + // keeps repeated keys, so `ordered` ends up in the priority + // order the author wrote and the duplicate check below actually + // fires. This mirrors `parse_lang_bias_entries` entry for entry. let mut ordered = Vec::new(); let mut seen: HashMap = HashMap::new(); for (code_str, BiasValueStr(bias)) in yaml_bias { @@ -598,10 +695,12 @@ exceptions: let config: LangBiasYamlConfig = serde_yaml::from_str(yaml_str).unwrap(); assert!(matches!(config.policy, Some(PolicyStr::Conservative))); let bias = config.bias.unwrap(); - assert!(bias.contains_key("ja")); - assert_eq!(bias["ja"].0, f32::NEG_INFINITY); - assert_eq!(bias["zh"].0, -10.0_f32); - assert_eq!(bias["ko"].0, 5.0_f32); + // Assert the order, not just membership. Asserting membership alone is + // what let the randomized `HashMap` ordering of issue #1267 stay hidden. + let codes: Vec<&str> = bias.as_slice().iter().map(|(c, _)| c.as_str()).collect(); + assert_eq!(codes, ["ja", "zh", "ko"]); + let values: Vec = bias.as_slice().iter().map(|(_, v)| v.0).collect(); + assert_eq!(values, [f32::NEG_INFINITY, -10.0_f32, 5.0_f32]); let ex = config.exceptions.unwrap(); assert!(!ex.include_special); assert!(!ex.include_numeric); @@ -632,6 +731,227 @@ unknown_field: value ); } + // ------------------------------------------------------------------------- + // YAML `bias:` ordering (issue #1267) + // + // The `bias:` block used to deserialize into a `HashMap`, whose iteration + // order `RandomState` randomizes per map instance. Because + // `TokenLanguageIndex::to_token_bias` resolves a token claimed by several + // languages with first-language-wins, the bias landing on a shared Han + // token changed from run to run for one unchanged config file. These tests + // resolve repeatedly inside one process, which is what makes them + // load-bearing: a single resolve can pass by luck, and the randomization is + // per map instance rather than per process, so a fresh resolve inside the + // same process draws a fresh order. + // ------------------------------------------------------------------------- + + /// Number of repeated resolves the ordering tests perform. + /// + /// With three languages there are six possible orders, so a single resolve + /// against the broken code had a good chance of coming out right and + /// proving nothing. Repeating the resolve makes an accidental pass + /// vanishingly unlikely. Measured against the pre-fix code, all three + /// ordering tests here failed at iteration 0 or 2. + const ORDER_RESOLVE_ITERATIONS: usize = 32; + + /// Write `contents` to a temp file and return the handle plus its path. + /// + /// The handle must stay alive for as long as the path is used: dropping a + /// `NamedTempFile` deletes the file. + fn write_temp_yaml(contents: &str) -> (tempfile::NamedTempFile, PathBuf) { + use std::io::Write; + + let mut tmpfile = tempfile::NamedTempFile::new().unwrap(); + tmpfile.write_all(contents.as_bytes()).unwrap(); + tmpfile.flush().unwrap(); + let path = tmpfile.path().to_path_buf(); + (tmpfile, path) + } + + /// The three-CJK config from the `LangBiasYamlConfig` schema doc comment. + /// + /// Han is shared by all three languages (`scripts_for`: Japanese includes + /// `Han` under both policies, Chinese is `{Han}`, Korean Conservative + /// includes `Han`), so this is exactly the case where the priority order + /// decides the outcome. + const THREE_CJK_YAML: &str = + "policy: conservative\nbias:\n ja: -inf\n zh: -10.0\n ko: +5.0\n"; + + #[test] + fn yaml_multi_cjk_bias_keeps_document_order_across_repeated_resolves() { + let (_tmpfile, path) = write_temp_yaml(THREE_CJK_YAML); + + let expected = [ + (LanguageCode::Ja, f32::NEG_INFINITY), + (LanguageCode::Zh, -10.0_f32), + (LanguageCode::Ko, 5.0_f32), + ]; + + for iteration in 0..ORDER_RESOLVE_ITERATIONS { + let args = LangBiasCliArgs { + lang_bias_config: Some(path.clone()), + ..Default::default() + }; + // A full `resolve()` per iteration, so each one re-reads and + // re-deserializes the file and gets a fresh map instance. + let config = args.resolve().unwrap().unwrap(); + let ordered = &config.bias_set.ordered; + + assert_eq!( + ordered.len(), + expected.len(), + "iteration {iteration}: expected {} entries, got {ordered:?}", + expected.len() + ); + for (index, (expected_code, expected_bias)) in expected.iter().enumerate() { + let (code, bias) = ordered[index]; + assert_eq!( + code, *expected_code, + "iteration {iteration}: entry {index} should be {expected_code:?} but the \ + resolved order was {ordered:?}; YAML bias: order must be the priority order" + ); + assert_eq!( + bias, *expected_bias, + "iteration {iteration}: entry {index} carried the wrong bias value" + ); + } + } + } + + #[test] + fn yaml_and_cli_paths_agree_on_multi_cjk_order() { + let (_tmpfile, path) = write_temp_yaml(THREE_CJK_YAML); + + let cli_args = LangBiasCliArgs { + lang_bias: Some("ja=-inf,zh=-10.0,ko=+5.0".to_owned()), + ..Default::default() + }; + let cli_ordered = cli_args.resolve().unwrap().unwrap().bias_set.ordered; + + for iteration in 0..ORDER_RESOLVE_ITERATIONS { + let yaml_args = LangBiasCliArgs { + lang_bias_config: Some(path.clone()), + ..Default::default() + }; + let yaml_ordered = yaml_args.resolve().unwrap().unwrap().bias_set.ordered; + assert_eq!( + yaml_ordered, cli_ordered, + "iteration {iteration}: the YAML path and the --lang-bias path must resolve \ + equivalent input to the same LangBiasSet" + ); + } + } + + #[test] + fn yaml_multi_cjk_first_language_wins_on_shared_han_tokens() { + use mlxcel_core::lang_analyzer::{ + CURRENT_VERSION, Script, TokenLanguageIndex, TokenScriptInfo, + }; + + // A three-token synthetic vocabulary: one pure-Han token claimed by ja, + // zh and ko alike, one Hiragana token only ja claims, and one Hangul + // token only ko claims. Building the index by hand keeps the assertion + // on `to_token_bias` without needing a real tokenizer. + let token = |token_id: i32, scripts: Vec