Evaluate Increased Light Node Connection Limits

[bcsorvasi]

Context

We want to investigate the impact of increasing the connection capacity of Light Nodes and better understand the practical limits of the current Bee and go-libp2p implementation.

There are two potential approaches:

Option 1 Safe Mode: Increase Light Refresh Rate

Keep the current connection limit unchanged, but increase the allowed light refresh rate by approximately 4x.

This would allow us to increase the effective capacity of Light Nodes without changing the underlying connection limit.

This approach should have a relatively low development cost, requiring approval from Viktor and changes to a few configuration variables in the Bee code.

Option 2 Increase the Light Connection Limit

A more interesting approach from a testing perspective would be to increase the Light Node connection limit from 100 to 400 connections.

The main goal is to understand how Bee and go-libp2p behave when maintaining a significantly higher number of concurrent connections.

Currently, we don't know:

• What practical connection limits or constraints go-libp2p imposes.
• How maintaining an additional ~300 connections affects Bee node performance.
• Whether 400 concurrent connections have any significant impact on stability or resource usage.
• Whether there are other unexpected side effects of increasing the connection limit.

[mfw78]

I think that in order to process this, it would be advisable to define what are the resource requirement expectations for running a single-neighbourhood staked node (CPU / RAM / storage space / storage IOPS / bandwidth).

Without setting a baseline for expectations, and ensuring that the node is meeting these requirements to start with, changing connection resource consumption parameters seems to be a bit of "wandering in the dark". The network right now is certainly nowhere near the resource limitations due to inactivity, so without any goals, any measurements that are done are likely premature / incomplete and may create issues as network utilisation increases (and these being difficult to walk back from).

[LevilkTheReal]

We run a measurement lab for a video streaming project on Swarm at Solar Punk, and we drove this question on ordinary hardware because we needed the answer ourselves. Sharing what we measured in case it is useful.

Everything below is off a node running v2.8.1 with #5559 backported as a 13-line patch, so the flag under test is yours rather than ours. One full node on a 64-core box, up to 400 penned ultra-light bees as light peers, each attached with an explicit POST /connect rather than as a bootnode. The whole /metrics surface (1,432 series) was captured either side of every arm rather than a chosen handful.

Three of the four questions

the issue asks	what we measured
practical limits go-libp2p imposes	none that fired. libp2p_rcmgr_blocked_resources never emitted, at any rung
how ~300 extra connections affect performance	goroutines +6.0 per peer, heap peak +43 MB across a 4x rise in peer count, 3.8% of eight cores
whether 400 concurrent connections affect stability	stable at 400 attached and idle, and stable under sustained request load, below
other unexpected side effects	the eviction regime stops once the limit is above demand, but please see the correction

A correction we had to publish against ourselves

We first attributed the peer churn at the 200 rung to slot eviction, on the strength of bee_libp2p_kickedout_peers_count going 0 to 11 to 19.

That attribution was wrong. In the same scrape, bee_accounting_disconnects_reconnect_count stood at 59,975 against 60,282 inbound handshakes, and each of those blocklists the peer for (refreshRate + paymentThreshold) / refreshRate = 4 seconds (accounting.go:1509). Slot eviction is about three thousandths of the churn on that node. An A-B-A on the limit itself, 400 → 1000 → 400, moved the churn by 4.9%.

So raising the limit removes a real constraint, and it is not the constraint that dominates peer churn on a busy node. We think that is worth knowing before Option 1 is priced.

On Option 1, raising the light refresh rate

The free allowance is 450,000 units/s per light peer against 320,000 units per chunk, which is 1.406 chunks/s or about 41 kbit/s each.

We then drove readers from 28% to 171% of that allowance. The delivered ceiling is ~1.2 chunks/s per peer, and it arrives without touching the median at all: p50 stayed at 9-10 ms across a six-fold rise in demand, while p90 went from 18 ms to 611 ms and the fraction of requests over 400 ms went from 1-3% to 21-26%.

611 ms is overDraftRefresh = time.Millisecond * 600 (retrieval.go:127, used at :224) plus the read.

⭐ And that per-peer ceiling is additive. We ran it again at 125 concurrent peers and each one still got 1.131-1.169 chunks/s, against 1.158-1.279 at 5 peers. Twenty-five times the peers, the same rate each, median unmoved, stalls on the same constant. So a node's free serving capacity really is

light-node-limit x ~1.15 chunks/s

which at 400 is about 460 chunks/s, roughly 15 Mbit/s, and both terms are configuration. The limit and the refresh rate multiply, so raising one without the other moves the ceiling proportionally and no further.

One honest caveat on that number: per-peer went 1.24 to 1.15 across the 25-fold rise, which is inside our predicted band and also inside arm-to-arm variance, so we are claiming additive to within about 10% rather than exactly additive.

Happy to share raw rows, per-arm metric diffs, or to run a specific rung if it would help.

[gacevicljubisa]

What limits go-libp2p actually imposes. The resource manager is built on rcmgr.InfiniteLimits (pkg/p2p/libp2p/libp2p.go:237), so go-libp2p imposes no connection-count, memory, or FD limits at any scope. The only constraints we configure are:

• 15k streams system-wide - 5,000 inbound + 10,000 outbound (

  bee/pkg/p2p/libp2p/libp2p.go

  Lines 89 to 90 in ae1ad4e

  	IncomingStreamCountLimit = 5_000
  	OutgoingStreamCountLimit = 10_000

  ).
• 200 connections per source IP (IPv4 /32, IPv6 /56;

  bee/pkg/p2p/libp2p/libp2p.go

  Lines 247 to 250 in ae1ad4e

  	limitPerIp := rcmgr.WithLimitPerSubnet(
  	[]rcmgr.ConnLimitPerSubnet{{PrefixLength: 32, ConnCount: 200}}, // IPv4 /32 (Single IP) -> 200 conns
  	[]rcmgr.ConnLimitPerSubnet{{PrefixLength: 56, ConnCount: 200}}, // IPv6 /56 subnet -> 200 conns
  	)

  ). Raising --light-node-limit to 400 still caps any single NAT/CGNAT IP at 200 light peers.
• Dial-rate limiting — 10 attempts/s, burst 40, per IP (

  bee/pkg/p2p/libp2p/libp2p.go

  Lines 255 to 281 in ae1ad4e

  	connLimiter := &libp2prate.Limiter{
  	// Allow unlimited local connections (same as default)
  	NetworkPrefixLimits: []libp2prate.PrefixLimit{
  	{Prefix: netip.MustParsePrefix("127.0.0.0/8"), Limit: libp2prate.Limit{}},
  	{Prefix: netip.MustParsePrefix("::1/128"), Limit: libp2prate.Limit{}},
  	},
  	GlobalLimit: libp2prate.Limit{}, // Unlimited global
  	SubnetRateLimiter: libp2prate.SubnetLimiter{
  	IPv4SubnetLimits: []libp2prate.SubnetLimit{
  	{
  	PrefixLength: 32, // Apply limits per individual IPv4 address (/32)
  	// Allow 10 connection attempts per second per IP, burst up to 40
  	Limit: libp2prate.Limit{RPS: 10.0, Burst: 40},
  	},
  	},
  	IPv6SubnetLimits: []libp2prate.SubnetLimit{
  	{
  	PrefixLength: 56, // Apply limits per /56 IPv6 subnet
  	// Allow 10 connection attempts per second per IP, burst up to 40
  	// Subnet-level limiting prevents flooding from multiple addresses in the same block.
  	Limit: libp2prate.Limit{RPS: 10.0, Burst: 40},
  	},
  	},
  	// Duration to retain state for an IP or subnet after it becomes inactive.
  	GracePeriod: 10 * time.Second,
  	},
  	}

  ).

None of these bind at 400 light peers spread across distinct IPs, consistent with libp2p_rcmgr_blocked_resources never emitting in the measurements above.