[Nexthop] Fix ARS spray test by checking deviation from fair share - #1595
Open
vvasavada-nexthop wants to merge 1 commit into
Open
vvasavada-nexthop wants to merge 1 commit into
vvasavada-nexthop wants to merge 1 commit into
Conversation
Signed-off-by: Vishrant Vasavada <vvasavada@nexthop.ai>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist
pip install -r requirements-dev.txt && pre-commit installpre-commit runSummary
Issue:
AgentLoadBalancerTestV6ArsSpray.EcmpLoadBalanceFullHashCpuTrafficfails intermittently on Wedge800BACT at[0/1619] LoadBalancerTestUtils.cpp: Value of: isLoadBalanced() Actual: false Expected: true.In one of the failures, this is what is seen:
Note that the deviation in this case is
26.57%which is slightly above the strict test threshold assert of25%.Four member ports are at ~0.92× the fair share, two at ~1.16×. All 200k packets egressed (17,000,000 bytes exactly), all six members active — spray works. The expected skew of this mode is ~25% against a 25% tolerance, so the test sits exactly on its own pass/fail boundary and flakes indefinitely.
On some platforms (e.g. TH5), we need ports to be on same core for load balancing spray to work roughly evenly. However, port/platform mapping is different across different platforms and it is not guaranteed that first six logical ports would end up sitting in same core. So the fix in 1212ea9 is loosely ended and works only for few platforms (e.g. minipack3).
Fix:
Instead of strictly asserting on equality of quality spray, we verify the participation. That is, every member must carry 25% to 200% of the fair share (
total / numMembers).The floor catches a starved or dead member and a group that never sprayed; the ceiling catches one member hogging while the rest sit just above the floor.
Test Plan
Passes consistently.