Description
Grout already supports single-homed EVPN/VXLAN bridging with FRR (evpn_vxlan_frr_test.sh). This issue proposes extending that to all-active EVPN multihoming (RFC 7432 / RFC 8365): two grout PEs presenting one 802.3ad LAG to a multihomed device, with FRR providing the EVPN-MH control plane.
We have a working prototype of the complete feature (validated in a three-node namespace lab with a split-chassis LACP peer, including IPv6 underlays, daemon restarts and failure injection) and intend to upstream it as a series of PRs. I am filing this issue so the individual PRs, including bugfixes in existing code that we found along the way, can link back to it.
Context and Use Cases
A device (host, VM host, or access switch) is attached with a LAG whose members land on two different grout instances. Both PEs forward simultaneously (all-active): remote PEs load-balance unicast toward the segment across both VTEPs, BUM traffic is forwarded onto the segment by exactly one elected PE, and traffic received from the segment is never reflected back to it through the other PE.
Division of labour
FRR owns BGP EVPN, Ethernet Segment state, Designated Forwarder election and route signalling (Type-1/Type-4). Grout owns LACP, VXLAN, bridging and all packet forwarding; the ES-facing ports stay bound to DPDK.
Required grout changes
1. L2 VTEP nexthops and L2 nexthop groups. A new L2-VTEP nexthop type, reusing the existing RCU-safe weighted group representation, plus an optional NHG ID on external FDB entries. The bridge datapath resolves the group by ID and selects a member per flow, giving MAC ECMP / aliasing toward a remote all-active segment. Mixed L2/L3 groups and cross-type route/FDB references are rejected; deletes are type/origin-conditional so a rejected install cannot later remove another owner's object.
2. Bridge-port policy for DF and split horizon. Consume DPLANE_OP_BR_PORT_UPDATE in the FRR provider and apply it atomically in the bridge/VXLAN graph: suppress overlay BUM on a non-DF port, filter unicast and BUM received from an ES peer VTEP, and redirect a local-ES hairpin through FRR's backup NHG (local bias). State is visible via grcli bridge-port show.
3. LACP member protodown. Consume the interface-state part of DPLANE_OP_INTF_UPDATE for FRR uplink tracking. Protodown is implemented as LACP member suppression rather than admin-down: the member keeps exchanging LACPDUs but clears sync/collecting/distributing, leaves the transmit hash and drops ordinary ingress, so the peer withdraws the member and recovery is a normal LACP renegotiation.
4. Canonical flow-hash metadata. All-active load-balancing needs one stable flow decision across bridge forwarding, EVPN NHG selection, VXLAN source-port selection, underlay ECMP and LACP member selection — including for packets from TAP/vhost-user ports with no hardware RSS. The prototype adds a grout-owned dynamic mbuf field plus validity flag (gr_mbuf_flow_hash_*()): a genuine NIC RSS value is imported, otherwise a software Toeplitz hash is computed lazily and cached, without forging RTE_MBUF_F_RX_RSS_HASH. VXLAN decap re-bases the metadata on the inner frame so outer-tunnel RSS is never used as tenant flow identity. This also fixes existing consumers that read m->hash.rss unconditionally and could act on stale metadata after mbuf reuse.
5. Bridge-domain metadata for FRR. FRR's ES-EVI association needs VLAN/VNI metadata, while grout's bridge model is VLAN-abstracted. Each applicable bridge domain is presented to FRR as a VLAN-aware bridge with one synthetic access VLAN, translated back at the FDB API boundary. Unsupported tagged configurations fail visibly.
6. FRR provider lifecycle hardening. Provider rejection, timeouts and partial replay fail closed and are repaired by a clean replay; interface-linked local MACs are flushed when an ES is attached or detached so stale non-ESI MAC entries are relearned with the correct ESI.
FRR dependency
FRR's zebra_evpn_mh.c installs EVPN-MH L2 nexthops and MAC-ECMP groups by calling the kernel netlink helpers directly (kernel_upd_mac_nh() etc.), bypassing the dataplane provider abstraction. We carry an FRR patch that queues these as typed L2 objects through the regular dplane queue while preserving the Linux NHA_FDB behaviour, and plan to propose it to FRR separately. Grout's provider consumes those operations. Until it is accepted, this feature requires a patched FRR; the grout-side changes are designed to not depend on anything FRR-specific beyond the dplane operations.
Testing
The prototype is covered by unit tests (flow hash, bridge-port policy ordering) and smoke tests: a direct L2 NH/NHG lifecycle test, a split-chassis LACP/protodown test, and a three-node EVPN-MH test covering IPv4/IPv6 underlays, DF election and preference change, split-horizon with injected VXLAN traffic, uplink-tracking protodown, MAC reconciliation, provider failure injection and FRR daemon/stack restarts. We would upstream these alongside the corresponding PRs.
Description
Grout already supports single-homed EVPN/VXLAN bridging with FRR (evpn_vxlan_frr_test.sh). This issue proposes extending that to all-active EVPN multihoming (RFC 7432 / RFC 8365): two grout PEs presenting one 802.3ad LAG to a multihomed device, with FRR providing the EVPN-MH control plane.
We have a working prototype of the complete feature (validated in a three-node namespace lab with a split-chassis LACP peer, including IPv6 underlays, daemon restarts and failure injection) and intend to upstream it as a series of PRs. I am filing this issue so the individual PRs, including bugfixes in existing code that we found along the way, can link back to it.
Context and Use Cases
A device (host, VM host, or access switch) is attached with a LAG whose members land on two different grout instances. Both PEs forward simultaneously (all-active): remote PEs load-balance unicast toward the segment across both VTEPs, BUM traffic is forwarded onto the segment by exactly one elected PE, and traffic received from the segment is never reflected back to it through the other PE.
Division of labour
FRR owns BGP EVPN, Ethernet Segment state, Designated Forwarder election and route signalling (Type-1/Type-4). Grout owns LACP, VXLAN, bridging and all packet forwarding; the ES-facing ports stay bound to DPDK.
Required grout changes
1. L2 VTEP nexthops and L2 nexthop groups. A new L2-VTEP nexthop type, reusing the existing RCU-safe weighted group representation, plus an optional NHG ID on external FDB entries. The bridge datapath resolves the group by ID and selects a member per flow, giving MAC ECMP / aliasing toward a remote all-active segment. Mixed L2/L3 groups and cross-type route/FDB references are rejected; deletes are type/origin-conditional so a rejected install cannot later remove another owner's object.
2. Bridge-port policy for DF and split horizon. Consume DPLANE_OP_BR_PORT_UPDATE in the FRR provider and apply it atomically in the bridge/VXLAN graph: suppress overlay BUM on a non-DF port, filter unicast and BUM received from an ES peer VTEP, and redirect a local-ES hairpin through FRR's backup NHG (local bias). State is visible via grcli bridge-port show.
3. LACP member protodown. Consume the interface-state part of DPLANE_OP_INTF_UPDATE for FRR uplink tracking. Protodown is implemented as LACP member suppression rather than admin-down: the member keeps exchanging LACPDUs but clears sync/collecting/distributing, leaves the transmit hash and drops ordinary ingress, so the peer withdraws the member and recovery is a normal LACP renegotiation.
4. Canonical flow-hash metadata. All-active load-balancing needs one stable flow decision across bridge forwarding, EVPN NHG selection, VXLAN source-port selection, underlay ECMP and LACP member selection — including for packets from TAP/vhost-user ports with no hardware RSS. The prototype adds a grout-owned dynamic mbuf field plus validity flag (gr_mbuf_flow_hash_*()): a genuine NIC RSS value is imported, otherwise a software Toeplitz hash is computed lazily and cached, without forging RTE_MBUF_F_RX_RSS_HASH. VXLAN decap re-bases the metadata on the inner frame so outer-tunnel RSS is never used as tenant flow identity. This also fixes existing consumers that read m->hash.rss unconditionally and could act on stale metadata after mbuf reuse.
5. Bridge-domain metadata for FRR. FRR's ES-EVI association needs VLAN/VNI metadata, while grout's bridge model is VLAN-abstracted. Each applicable bridge domain is presented to FRR as a VLAN-aware bridge with one synthetic access VLAN, translated back at the FDB API boundary. Unsupported tagged configurations fail visibly.
6. FRR provider lifecycle hardening. Provider rejection, timeouts and partial replay fail closed and are repaired by a clean replay; interface-linked local MACs are flushed when an ES is attached or detached so stale non-ESI MAC entries are relearned with the correct ESI.
FRR dependency
FRR's zebra_evpn_mh.c installs EVPN-MH L2 nexthops and MAC-ECMP groups by calling the kernel netlink helpers directly (kernel_upd_mac_nh() etc.), bypassing the dataplane provider abstraction. We carry an FRR patch that queues these as typed L2 objects through the regular dplane queue while preserving the Linux NHA_FDB behaviour, and plan to propose it to FRR separately. Grout's provider consumes those operations. Until it is accepted, this feature requires a patched FRR; the grout-side changes are designed to not depend on anything FRR-specific beyond the dplane operations.
Testing
The prototype is covered by unit tests (flow hash, bridge-port policy ordering) and smoke tests: a direct L2 NH/NHG lifecycle test, a split-chassis LACP/protodown test, and a three-node EVPN-MH test covering IPv4/IPv6 underlays, DF election and preference change, split-horizon with injected VXLAN traffic, uplink-tracking protodown, MAC reconciliation, provider failure injection and FRR daemon/stack restarts. We would upstream these alongside the corresponding PRs.