[PW_SID:1160924] riscv: word-at-a-time: improve find_zero() - #2626
Open
linux-riscv-bot wants to merge 3 commits into
Open
[PW_SID:1160924] riscv: word-at-a-time: improve find_zero()#2626linux-riscv-bot wants to merge 3 commits into
linux-riscv-bot wants to merge 3 commits into
Conversation
Current find_zero() heavily depends on fls64() for calculation. This
bring non-optimal code when !RISCV_ISA_ZBB.
But in word-at-a-time case, we don't have to go with fls64() code path,
instead, we can fallback to the generic word-at-a-time implementaion.
What's more, the fls64() brings non-necessary zero bits couting for
RV32. In fact, fls() is enough.
Before the patch:
0000000000000000 <find_zero>:
0: c529 beqz a0,4a <.L1>
2: 577d li a4,-1
4: 9301 srli a4,a4,0x20
6: 03f00793 li a5,63
a: 00a76463 bltu a4,a0,12 <.L3>
e: 1502 slli a0,a0,0x20
10: 47fd li a5,31
0000000000000012 <.L3>:
12: 577d li a4,-1
14: 8341 srli a4,a4,0x10
16: 00a76463 bltu a4,a0,1e <.L4>
1a: 37c1 addiw a5,a5,-16
1c: 0542 slli a0,a0,0x10
000000000000001e <.L4>:
1e: 577d li a4,-1
20: 8321 srli a4,a4,0x8
22: 00a76463 bltu a4,a0,2a <.L5>
26: 37e1 addiw a5,a5,-8
28: 0522 slli a0,a0,0x8
000000000000002a <.L5>:
2a: 577d li a4,-1
2c: 8311 srli a4,a4,0x4
2e: 00a76463 bltu a4,a0,36 <.L6>
32: 37f1 addiw a5,a5,-4
34: 0512 slli a0,a0,0x4
0000000000000036 <.L6>:
36: 577d li a4,-1
38: 8309 srli a4,a4,0x2
3a: 00a76463 bltu a4,a0,42 <.L7>
3e: 37f9 addiw a5,a5,-2
40: 050a slli a0,a0,0x2
0000000000000042 <.L7>:
42: 00054563 bltz a0,4c <.L12>
46: 4037d51b sraiw a0,a5,0x3
000000000000004a <.L1>:
4a: 8082 ret
000000000000004c <.L12>:
4c: 2785 addiw a5,a5,1
4e: 4037d51b sraiw a0,a5,0x3
52: 8082 ret
After the patch:
0000000000000000 <find_zero>:
0: 102037b7 lui a5,0x10203
4: 0792 slli a5,a5,0x4
6: 40578793 addi a5,a5,1029 # 10203405 <.L4+0x102033c5>
a: 07c2 slli a5,a5,0x10
c: 60878793 addi a5,a5,1544
10: 02f50533 mul a0,a0,a5
14: 9161 srli a0,a0,0x38
16: 8082 ret
33 instructions vs 8 instructions!
And this kind of instructions reducing dramatically improves the
performance of below micro-benchmark:
$ cat tt.c
#inlcude <stdio.h>
#inlcude "word-at-a-time.h" // copy and modify, eg. remove other headers
int main()
{
int i;
unsigned long ret = 0;
for (i = 0; i < 100000000; i++)
ret |= find_zero(0xabcd123 + i);
printf("%ld\n", ret);
}
$ gcc -O tt.c
$ time ./a.out
Per my test, the above micro-benchmark is improved by about 1150%!
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Previous commit improved the find_zero() performance for !RISCV_ISA_ZBB. What about RISCV_ISA_ZBB=y but the HW doesn't support Zbb? We have the same heavy generic fls64() issue. Let's improve this situation by checking Zbb extension and fall back to generic count_masked_bytes() if Zbb isn't supported. To remove non-necessary zero bits couting on RV32, we also replace the 'fls64(mask) >> 3' with '!mask ? 0 : ((__fls(mask) + 1) >> 3);' We will get similar performance improvement as previous commit for RISCV_ISA_ZBB=y but HW doesn't support Zbb. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
In commit f915a3e ("arm64: word-at-a-time: improve byte count calculations for LE"), Linus improved the find_zero() for arm64 LE. Do the same optimization as he did: "do __ffs() on the intermediate value that found whether there is a zero byte, before we've actually computed the final byte mask.", so that we share the similar improvements: "The difference between the old and the new implementation is that "count_zero()" ends up scheduling better because it is being done on a value that is available earlier (before the final mask). But more importantly, it can be implemented without the insane semantics of the standard bit finding helpers that have the off-by-one issue and have to special-case the zero mask situation." Before the patch: 0000000000000000 <find_zero>: 0: c909 beqz a0,12 <.L1> 2: 60051793 clz a5,a0 6: 03f00513 li a0,63 a: 8d1d sub a0,a0,a5 c: 2505 addiw a0,a0,1 e: 4035551b sraiw a0,a0,0x3 0000000000000012 <.L1>: 12: 8082 ret After the patch: 0000000000000000 <find_zero>: 0: 60151513 ctz a0,a0 4: 810d srli a0,a0,0x3 6: 8082 ret 7 instructions vs 3 instructions! As can be seen, on RV64 w/ Zbb, the new "find_zero()" ends up just "ctz" plus the shift right that then ends up being subsumed by the "add to final length". But I have no HW platform which supports Zbb, so I can't get the performance improvement numbers by the last patch, only built and tested the patch on QEMU. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR for series 1160924 applied to workflow__riscv__fixes
Name: riscv: word-at-a-time: improve find_zero()
URL: https://patchwork.kernel.org/series/1160924/
Version: 3