Skip to content

[Nexthop] Fix ARS spray test by checking deviation from fair share - #1595

Open
vvasavada-nexthop wants to merge 1 commit into
facebook:mainfrom
nexthop-ai:fix-ars-spray-test-by-checking
Open

vvasavada-nexthop wants to merge 1 commit into
facebook:mainfrom
nexthop-ai:fix-ars-spray-test-by-checking

Conversation

@vvasavada-nexthop

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Issue:

AgentLoadBalancerTestV6ArsSpray.EcmpLoadBalanceFullHashCpuTraffic fails intermittently on Wedge800BACT at [0/1619] LoadBalancerTestUtils.cpp: Value of: isLoadBalanced() Actual: false Expected: true.

In one of the failures, this is what is seen:

I0826 20:48:39.506009  4683 LoadBalancerTestUtils.cpp:416] NOS-13295: member PortID(1) outBytes: 2616555
I0826 20:48:39.506014  4683 LoadBalancerTestUtils.cpp:416] NOS-13295: member PortID(3) outBytes: 2619700
I0826 20:48:39.506017  4683 LoadBalancerTestUtils.cpp:416] NOS-13295: member PortID(5) outBytes: 2605335
I0826 20:48:39.506018  4683 LoadBalancerTestUtils.cpp:416] NOS-13295: member PortID(13) outBytes: 2613580
I0826 20:48:39.506019  4683 LoadBalancerTestUtils.cpp:416] NOS-13295: member PortID(22) outBytes: 3247170
I0826 20:48:39.506021  4683 LoadBalancerTestUtils.cpp:416] NOS-13295: member PortID(24) outBytes: 3297660
I0826 20:48:39.506023  4683 LoadBalancerTestUtils.cpp:419] NOS-13295: members: 6 highest bytes: 3297660 lowest bytes: 2605335 deviation: 26.57%

Note that the deviation in this case is 26.57% which is slightly above the strict test threshold assert of 25%.

Four member ports are at ~0.92× the fair share, two at ~1.16×. All 200k packets egressed (17,000,000 bytes exactly), all six members active — spray works. The expected skew of this mode is ~25% against a 25% tolerance, so the test sits exactly on its own pass/fail boundary and flakes indefinitely.

On some platforms (e.g. TH5), we need ports to be on same core for load balancing spray to work roughly evenly. However, port/platform mapping is different across different platforms and it is not guaranteed that first six logical ports would end up sitting in same core. So the fix in 1212ea9 is loosely ended and works only for few platforms (e.g. minipack3).

Fix:

Instead of strictly asserting on equality of quality spray, we verify the participation. That is, every member must carry 25% to 200% of the fair share (total / numMembers ).

The floor catches a starved or dead member and a group that never sprayed; the ceiling catches one member hogging while the rest sit just above the floor.

Test Plan

Passes consistently.

Running all tests took 0:00:40.420734 between 2026-08-26 21:53:57.553296 and 2026-08-26 21:54:37.974030
[ PASSED ] cold_boot.AgentLoadBalancerTestV6ArsSpray.EcmpLoadBalanceFullHashCpuTraffic (9328 ms)
[ PASSED ] warm_boot.AgentLoadBalancerTestV6ArsSpray.EcmpLoadBalanceFullHashCpuTraffic (10293 ms)
Summary:
   PASSED : 2
   FAILED : 0
   SKIPPED : 0
   TIMEOUT : 0

Signed-off-by: Vishrant Vasavada <vvasavada@nexthop.ai>
@vvasavada-nexthop
vvasavada-nexthop requested a review from a team as a code owner September 3, 2026 22:25
@meta-cla meta-cla Bot added the CLA Signed label Sep 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant