Optimize AV2 encoder by removing redundant memset calls - #5194
Merged
Conversation
Eliminates unnecessary memset calls in av2_quantize_skip, highbd_quantize_dc, and av2_xform_dc_only where eob is set to 0 or coefficients are written directly. TAG=agy CONV=dfbcd5f8-b1c1-4b21-904b-b14f69c835cc
Removes "= { 0 };" zero-initialization from the 16 KB stack buffer
"int buf[MAX_TX_SQUARE]" in fwd_txfm_c and fwd_txfm_avx2. The entire
buffer is written by the column 1D transform before being read by the
row 1D transform.
Includes an explicit assertion (assert(width >= 4 && height >= 4)) to
document valid transform dimensions and hint to Clang Static Analyzer
during symbolic execution, preventing false positive uninitialized memory
warnings.
TAG=agy
CONV=dfbcd5f8-b1c1-4b21-904b-b14f69c835cc
yeqing-wu
reviewed
Aug 4, 2026
- Sort config/ and project header includes alphabetically in hybrid_fwd_txfm.c and highbd_fwd_txfm_avx2.c. - Place system header <assert.h> before project headers in hybrid_fwd_txfm.c. - Clean up av2_quantize_skip signature to void av2_quantize_skip(uint16_t *eob_ptr), removing unused n_coeffs, qcoeff_ptr, and dqcoeff_ptr parameters. - Remove unused n_coeffs parameter from static inline helper highbd_quantize_dc. TAG=agy CONV=f1b0a382-a1eb-48a9-9834-f89940af07ff
TAG=agy CONV=f1b0a382-a1eb-48a9-9834-f89940af07ff
leolzhao
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eliminate redundant memset and buffer-zeroing overhead across quantization and forward transform routines:
Remove unnecessary memset calls in av2_quantize_skip, highbd_quantize_dc, and av2_xform_dc_only where eob is set to 0 or coefficients are directly written.
Remove = { 0 } zero-initialization from the 16 KB stack buffer int buf[MAX_TX_SQUARE]` in fwd_txfm_c and fwd_txfm_avx2, as the buffer is unconditionally overwritten by the column 1D transform before being read.
Anchor:
dd6f21f
Tests: 33 frame CTC against dd6f21f
Speed 1:
| Set | Y | U | V | YUV | Enc-time |
|---------+-------+-------+-------+-------+----------|
| A1 | 0.00% | 0.00% | 0.00% | 0.00% | 97.6% |
| A2 | 0.00% | 0.00% | 0.00% | 0.00% | 96.7% |
Speed 4:
| Set| Y | U | V | YUV | Enc-time |
|---------+-------+-------+-------+-------+----------|
| A1 | 0.00% | 0.00% | 0.00% | 0.00% | 98.0% |
| A2 | 0.00% | 0.00% | 0.00% | 0.00% | 97.3% |