Move lossless DPCM check inside prune_intra_y_mode

[mudassir-sg]

This change moves condition checks related to lossless DPCM modes inside prune_intra_y_mode() and updates its return type from int to bool.

No stats changed.

[mudassir-sg]

@yunqingwang1 , @urvangjoshi
cc: @harishdm @ranjit-tulabandu @cherma-a @deepa-kg @RemyaPrakasan @vikasprasad10

[yunqingwang1]

Can we only pass in (xd->lossless[mbmi->segment_id] && mbmi->use_dpcm_y != 0) to this function instead of xd?

[mudassir-sg]

Since mbmi->use_dpcm_y=1 is applicable for lossless mode, checking xd->lossless is redundant. We have updated the function to take mbmi->use_dpcm_y directly.

[yunqingwang1]

"mbmi->use_dpcm_y=1 is applicable for lossless mode". Is there any code to guarantee this? Otherwise, could we at least add an assertion before calling prune_intra_y_mode()?

[mudassir-sg]

mbmi->use_dpcm_y = 1 is applicable only for lossless mode.

For example, in the Decoder (decodemv.c, line 1654), mbmi->use_dpcm_y is read from the bitstream only when xd->lossless[mbmi->segment_id] is set:

if (xd->lossless[mbmi->segment_id]) {
  mbmi->use_dpcm_y = read_dpcm_mode(ec_ctx, r);

Similar handling is present on the encoder side as well (in bitstream writing and intra mode search).

To make this invariant explicit, we have two options to add an assertion:

Option 1: Add assert(IMPLIES(mbmi->use_dpcm_y, lossless)) outside prune_intra_y_mode() at all 4 call sites.
Option 2: Pass lossless (or xd->lossless[mbmi->segment_id]) to prune_intra_y_mode() and add the assertion inside prune_intra_y_mode().
Please let us know which option would be preferred.

[yunqingwang1]

Either way is fine. Since one condition is being removed, let's add a check to ensure this logic isn't mistakenly broken in the future. Thanks.

[mudassir-sg]

Added the assertion inside prune_intra_y_mode() to guarantee that use_dpcm_y is set only for the lossless case.

During further testing, we noticed a bit-mismatch on the Vertical_Bayshore_270x480_2997 sequence in the speed=0 Low-Delay (LD) configuration.
Root Cause: When evaluating lossless DPCM mode (use_dpcm_y != 0), the function returned early before updating best_model_rd and top_intra_model_rd[] . However, best_model_rd is later being used in:

• model_intra_yrd_and_prune() (for palette mode search)
• rd_pick_intra_dip_sby() (as an input feature to the ML model that prunes DIP modes)

(Note: While top_intra_model_rd[] is also bypassed during the early return for lossless DPCM, this is not an issue since top_intra_model_rd[] is reset prior to its usage).
Fix: Moved the best_model_rd update before the early exit check for use_dpcm_y.