infra,l2,ip: fix flow hashing for ports without RSS

[hcaldicott]

Several datapaths read m->hash.rss directly for ECMP and load-balancing
decisions (fib4/fib6 lookups, vxlan underlay routing and source port
selection). On ports without RSS — TAP and other virtual devices — that
field holds stale or zero data, so every flow shares one value: ECMP
collapses onto a single nexthop and all vxlan flows share one UDP source
port. After SRv6 decapsulation, the outer flow label is copied over the
hash while a hardware valid flag survives; with a zero label (as grout's
own encap emits) every inner flow shares hash 0.

The series first extracts the bond Toeplitz hashing into a shared helper
and fixes two fragment-handling bugs in it: packets with the DF flag set
were treated as fragments and hashed L3-only, and first fragments (offset
zero, MF set) were hashed with their L4 ports unlike the rest of their
datagram. It then caches one canonical per-packet hash by storing the
software result in m->hash.rss and marking it valid with
RTE_MBUF_F_RX_RSS_HASH, exactly as a hardware driver would — no new
mbuf metadata. The bond, vxlan and L3 consumers read it through one
helper. vxlan decapsulation leaves the outer hash in place: a VTEP
following RFC 7348 derives the outer UDP source port from the inner
flow, so an outer RSS value is already a function of the inner flow and
stays valid after decapsulation. SRv6 decapsulation keeps using a
non-zero outer flow label as the flow entropy, now stored through the
shared helper so the valid flag is coherent, and invalidates the hash
when the label is zero so the first consumer hashes the inner packet.

For packets whose driver does provide RSS, the datapath is unchanged:
the same flag test and the same m->hash.rss read as before the series,
with no recomputation after encap or decap.

Found while prototyping EVPN all-active multihoming (#698), where bridged
flows must keep per-flow path affinity across bond and vxlan ECMP, but
everything above is reproducible on plain upstream.

Testing

• New flow_hash unit tests: per-flow stability, DF/MF fragment handling,
  single computation and caching, hardware RSS precedence, hash
  invalidation.
• New srv6 decap unit test: a non-zero outer flow label becomes the flow
  hash with the valid flag set; a zero label drops a stale outer RSS
  value.
• Extended smoke/vxlan_test.sh: a second bridged port injects 32
  distinct UDP flows and the test asserts the encapsulated packets use
  several distinct source ports.
• Extended smoke/ip_loadbalance_test.sh: 64 distinct UDP flows through
  the ECMP route, asserting both group members carry traffic.
• Without the fixes (tests-only applied to main): the vxlan test fails
  with "32 flows shared 1 vxlan source ports", the load-balance test
  fails with "member p1 carried 2 of 64 distinct flows", the MF fragment
  unit test fails, and the srv6 decap test fails with the stale outer
  hash still marked valid.
• With the series: full unit suite and the vxlan, vxlan6 and
  ip_loadbalance smoke tests pass (AlmaLinux 9 container, arm64).

Related: #698

[qodo-free-for-open-source-projects]

1. Flow-hash init race 🐞 Bug ☼ Reliability

The new flow-hash mbuf dynfield/dynflag is registered in a module with no dependency ordering, but
worker module init starts datapath threads (and loads graphs) during modules_init. If the
topo-sorted init order runs worker before flow_hash, gr_mbuf_flow_hash_set/get can execute with
gr_mbuf_flow_hash_offset still -1, causing out-of-bounds writes via RTE_MBUF_DYNFIELD().

Agent Prompt

### Issue description
`flow_hash` registers an mbuf dynfield and dynflag in `flow_hash_init()`, but the module definition has no `depends_on`. `worker_init()` starts datapath threads and reloads graphs during `modules_init()`. Because `modules_init()` orders modules only via `depends_on`, `flow_hash_init()` can legally run after `worker_init()`, allowing datapath code paths (including `gr_mbuf_flow_hash_set/get`) to run while `gr_mbuf_flow_hash_offset` is still `-1`.

This can turn `*RTE_MBUF_DYNFIELD(m, -1, ...) = ...` into memory corruption.

### Issue Context
- `gr_mbuf_flow_hash_offset` defaults to `-1` until `flow_hash_init()` runs.
- `worker_init()` calls `worker_queue_distribute()`, which creates pthreads (`gr_datapath_loop`) and calls `worker_graph_reload_all()`, enabling `rte_graph_walk()` in worker threads while `modules_init()` may still be iterating.

### Fix Focus Areas
- Ensure `flow_hash_init()` runs before any worker threads can run graphs:
  - modules/infra/datapath/flow_hash.c[211-214]
  - modules/infra/control/worker.c[612-616]
- (Optional hardening) add `assert(gr_mbuf_flow_hash_offset >= 0)` in `gr_mbuf_flow_hash_set()` / getters to fail fast if used pre-init:
  - modules/infra/datapath/flow_hash.h[26-33]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[rjarry]

There is a bit too much code churn in the successive commits. Could you try and make this more incremental?

Did you test if this series introduces any performance change? I am concerned about the re-computation of RSS after decap/encap even with hardware drivers that support L4 UDP ports.

Also, please drop the Assisted-by: <LLM> trailers.

[hcaldicott]

Thanks for the review - I will re-work and clean this up. This was cherry-picked from other development work on our MH implementation, so it could use some more scrutiny - sit tight.

[hcaldicott]

There is a bit too much code churn in the successive commits. Could you try and make this more incremental?

Could you confirm if the commit history is acceptable now? I updated Patch 1 so that it uses the final shape from the beginning to remove most of the churn. This was due to cherry-picking from my upstream MH codebase - apologies.

Did you test if this series introduces any performance change? I am concerned about the re-computation of RSS after decap/encap even with hardware drivers that support L4 UDP ports.

Fair call - I have reworked so that decap no longer recomputes anything. vxlan_input just clears the RSS flag; a software hash for the inner frame is computed lazily at the first consumer that needs one, then cached. Packets with genuine NIC RSS follow the exact upstream code path (same flag test, same field read), so hardware fast paths are untouched. I am going to see how I can benchmark this.

Also, please drop the Assisted-by: <LLM> trailers.

Done.

[hcaldicott]

Okay, I have thrown together some AI tests and sanity checked them, seems good to me.

I benchmarked the series against v0.17.1 on a pair of Xeon D-2143IT boxes (X722 10GbE SFP+, back to back), using the per-node cycle counters from grcli stats software under ~226 kpps of 8-flow UDP.

Two topologies: plain routed forwarding (kernel TAP in, 10G wire, TAP out) and a bridged VXLAN overlay mirroring a production EVPN setup. 30 s measured runs after warmup, 3 reps per build per scenario, medians below. X722 RSS was confirmed active (flows spread across both RX queues).

path	node	v0.17.1	this series	delta
routed ingress, NIC RSS present	ip_input	67.9 c/pkt	~68 c/pkt	none — one flag test
routed ingress, no RSS (TAP)	ip_input	70.3	206.6	+136 c/pkt (~62 ns): the software Toeplitz
vxlan encap from bridged no-RSS port	vxlan_output	84.6	245.7	+161 c/pkt: same hash over the inner frame
vxlan decap	vxlan_input	47.6	51.5	+4 c/pkt: clearing the flag
all other nodes				within noise (±5%)

Notes:

• c/pkt is CPU cycles per packet - lower being better.
• Where the driver provides RSS, the datapath is unchanged: same flag test, same m->hash.rss read, nothing recomputed after encap. An earlier revision of this series showed +5 c/pkt here from the getters being extern functions (call overhead on a test-and-return path); they are now static inline and that is gone.
• The +136/+161 only appear on paths that previously fed stale or zero data into the fib lookups and the source port — the bug being fixed. The hash is computed once and cached, so fib lookup, vxlan source port and bond selection on the same packet share one computation.
• After decap nothing is recomputed unless a consumer actually needs a hash for the inner frame.

Throughput was generator-limited and identical on both builds. The fixed build showed lower loss in the vxlan scenario (<=0.01% vs up to 0.77%): per-flow source ports let the receiving NIC spread flows across both RSS queues instead of piling them onto one.

Incidental finding from the same rig: the X722 PF firmware rejects rte_eth_allmulticast_enable() with -ENOTSUP, so interface add port on these NICs fails hard on current main — I had to apply #702 to both builds to run the benchmark, so statistics here might be slightly skewed by that change also.

Crux of this: I see about a 62ns (at 2.2ghz) performance decrease with the corrected hashing function in place. However I feel like it could be worthwhile for the correctness aspect of this fix?

[christophefontaine]

With the introduction of gr_mbuf_flow_hash_get to get the mbuf->hash.rss, we should also update process_behav_decap in modules/srv6/datapath/srv6_local.c to set the hash.
WDYT ?

And to update modules/ip/datapath/icmp_local_send.c as well as icmp6_local_send.c for consistency.

[christophefontaine]

vlan = PAYLOAD(eth);

[christophefontaine]

Outside of this commit, and eligible for a future discussion / PR.
As we only look at ip6->proto, we do not take into account the extension headers which could be before a valid IPPROTO_*.
So, as part of the hash computation, what should we do ?

[christophefontaine]

I'm not comfortable in always invalidating the hash, as it may have been computed by the hardware on the inner packet.
Also, relying on the PTYPE may not be good either: as you said, even if the hash isn't computed on the inner, the remote VTEP may have spread the source UDP port.

  // keep a hardware-provided inner-flow hash; only invalidate an outer one
  if (!(m->packet_type & RTE_PTYPE_INNER_L4_MASK))
      gr_mbuf_flow_hash_invalidate(m);

And I know that @david-marchand is not fan of using packet_type as well :)

[david-marchand]

@hcaldicott

Hum, I did not look too much in detail at the full PR sorry.
I am trying to go back to the actual problem you are facing.

The issue seems to be that the rss hash is not a function of the inner packet.
This should be the case for routers implementing the recommendation in RFC7348.

   Outer UDP Header:  This is the outer UDP header with a source port
      provided by the VTEP and the destination port being a well-known
      UDP port.
...
-  Source Port:  It is recommended that the UDP source port number
         be calculated using a hash of fields from the inner packet --
         one example being a hash of the inner Ethernet frame's headers.
         This is to enable a level of entropy for the ECMP/load-
         balancing of the VM-to-VM traffic across the VXLAN overlay.
         When calculating the UDP source port number in this manner, it
         is RECOMMENDED that the value be in the dynamic/private port
         range 49152-65535 [[RFC6335](https://datatracker.ietf.org/doc/html/rfc6335)].

If you are receiving traffic from another grout instance, the rss hash should have a source port that depends on the inner packet RSS (https://github.com/DPDK/grout/blob/main/modules/l2/datapath/vxlan_output.c#L90).

Is it that in your usecase the inner (meaning, before encapsulation on the transmitter side) packet rss 0 because it is received from a net/virtio (including virtio-user) port?

If this is the case, you may want to compute a RSS when it is not set at the RX side (see rx_virtio_process/rx_bond_virtio_process).

[christophefontaine]

(not introduced by this commit, but to open the discussion for a future PR)
The fib4_lookup gets the nh based on the hash. Yet, this hash is still the one computed by the hardware, pre-nat.
Shouldn't we update the RSS field to reflect the modified dst_ip & dsp_port?

[christophefontaine]

same comment as dnat44_dynamic