Skip to content

fix: clamp speaker mix weights to [0,1] to keep the frame mix convex - #6

Merged
LiuYunPlayer merged 1 commit into
LiuYunPlayer:masterfrom
KakaruHayate:fix/speaker-mix-weight-clamp
Jul 26, 2026
Merged

fix: clamp speaker mix weights to [0,1] to keep the frame mix convex#6
LiuYunPlayer merged 1 commit into
LiuYunPlayer:masterfrom
KakaruHayate:fix/speaker-mix-weight-clamp

Conversation

@KakaruHayate

Copy link
Copy Markdown
Contributor

Problem

DiffSingerSpeakerMix.Create accumulates each mix:<suffix> sample with only a NaN check — the value is never clamped to the track's declared range:

double v = f < sampled.Length ? sampled[f] : double.NaN;
if (!double.IsNaN(v)) w[f] += (float)v;   // no clamp

The track is declared Continuous(color, 0, 0, 1) and the host UI clamps on draw, but the data layer has no hard contract — the codebase already says exactly this in CombineVariance:

用户值钳到编辑量程:宿主 UI 落笔时钳制,但数据层无硬契约(锚点+默认值合成、跨引擎同 key 复用、手改工程皆可越界)

Negative values are the damaging case. They pull Σ back down to ≤ 1, which defeats the sum > 1 guard below: the weights are not normalized, and the default suffix is topped up by 1 - Σ > 1. The result leaves the convex hull — extrapolation instead of interpolation, i.e. an embedding outside the training distribution.

Reproduced against this repo's master (bcd2396), default speaker Alice, Alice=[1,0], Bob=[0,1], Carol=[.5,.5]:

input current in convex hull?
Bob=+0.5 [0.500, 0.500] ✅ correct interpolation
Bob=-0.5 [1.500, -0.500]
Bob=-2.0 [3.000, -2.000]
Bob=2, Carol=-1 [-0.500, 1.500]

The last row is the subtle one: Σ is exactly 1, so the normalization branch never fires.

Fix

Clamp the sampled value to [0,1] at accumulation. The sibling phoneme_mix track — declared with the identical Continuous(color, 0, 0, 1) shape — is already clamped at its sampling site in DiffSingerSynthesisSession:

br[f] = (float)Math.Clamp(double.IsNaN(sampled[f]) ? 0 : sampled[f], 0, 1);

so this only brings the two same-shaped tracks into line.

In-range behaviour is bit-for-bit unchanged: values within [0,1] are unaffected by the clamp, and the normalization / default top-up logic is untouched.

Tests

  • Added DiffSingerSpeakerMix.cs to the test project's Compile items (zero external dependencies, same treatment as DiffSingerFrames), so it still runs without the TuneLab SDK checked out.
  • New DiffSingerSpeakerMixTests covering existing behaviour (default top-up, over-unity normalization, NaN-as-unedited, short track) plus 6 out-of-range regressions (negative ×3, above-one, mixed-sign, per-frame clamping). The invariants asserted are the ones that actually matter: per-frame weights all in [0,1] and summing to 1, and the resulting embedding staying within the per-dimension convex hull of the speaker embeddings.

I stashed the fix and ran the suite first to confirm those 6 genuinely fail without it (the other 38 pass either way), then restored it for 44/44 — otherwise "weights sum to 1" is the kind of assertion that can silently hold no matter what.

Local run of this repo's CI sequence (plugin / tests / smoke tool, Release): 0 warnings, 0 errors.


Found while reviewing an unrelated feature branch in my fork; confirmed independent of it (git diff master...<that branch> -- DiffSingerSpeakerMix.cs is empty), so it's sent on its own.

DiffSingerSpeakerMix.Create 对 mix:<suffix> 采样值只判 NaN、不钳量程。
轨本身声明为 Continuous(color, 0, 0, 1)、宿主 UI 落笔时也钳制,但数据层无硬契约
(锚点+默认值合成、跨引擎同 key 复用、手改工程皆可越界)——与 CombineVariance
早已记录的兜底理由相同。

负权重危害最大:它把 Σ 拉回 ≤1,令下方「Σ>1 才归一」的判据失效,于是既不归一、
默认 suffix 又补上 1-Σ>1,混出凸包外的 embedding(外插而非插值,落到训练分布之外):

  default=Alice, Alice=[1,0] Bob=[0,1]
    Bob=+0.5 -> [0.5, 0.5]    正常插值
    Bob=-0.5 -> [1.5, -0.5]   修复前:越界
    Bob=-2.0 -> [3.0, -2.0]   修复前:越界
  正负混合 Bob=2 / Carol=-1(Σ 恰为 1、不触发归一)-> [-0.5, 1.5] 修复前:越界

同形态的 phoneme_mix 轨在 DiffSingerSynthesisSession 采样处本就 clamp 到 [0,1],
此处对齐,消除两条同类轨的行为不一致。

测试:
- 把零依赖的 DiffSingerSpeakerMix.cs 加入测试项目(无 SDK 亦可跑)。
- 新增 DiffSingerSpeakerMixTests:默认补权、超额归一、NaN 视作未编辑等既有行为,
  加 6 个越界回归(负权重、超 1、正负混合、逐帧钳制、凸包内断言)。
  已确认这 6 个在修复前失败、修复后通过;其余 38 个基线测试前后皆通过。

44 tests pass;插件/测试/冒烟工具 Release 均 0 warning 0 error。
@LiuYunPlayer
LiuYunPlayer merged commit 2b73048 into LiuYunPlayer:master Jul 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants