Skip to content

marilib: make the metrics probe a usable measurement instrument - #169

Open
geonnave wants to merge 18 commits into
DotBots:developfrom
geonnave:marilib-metrics-instrument
Open

marilib: make the metrics probe a usable measurement instrument#169
geonnave wants to merge 18 commits into
DotBots:developfrom
geonnave:marilib-metrics-instrument

Conversation

@geonnave

@geonnave geonnave commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Preparing marilib to measure latency and PDR for a paper campaign turned up
several things that made the numbers wrong or unreadable rather than merely
inconvenient. This fixes them and adds the firmware tooling the campaign needs.

The probe stream had to stop scaling with the fleet

The probe interval was per node, so the offered probe load grew linearly with
the node count: a fixed 5 s/node is 20 probes/s at 100 nodes but 2 at 10. That
makes the instrument itself the variable under test when the fleet size or the
schedule changes.

--probe-every now cadences in slotframes instead of seconds, so the probe
share of downlink stays roughly constant across schedules (N/(K x d_down) is
within a few percent of 4.6/K for tiny through huge). -i still sets a
wall-clock interval when that is what you want. The cadence is derived lazily,
once the gateway has actually reported its schedule, because deriving it up
front meant a fixed deadline that could abort a run before the first beacon.

--load now means total downlink occupancy: the load generator subtracts
the measured probe rate, so passing 75 puts 75% on the link rather than 75%
plus whatever the probes were already using. Recomputed per packet, since the
probe stream grows with the node count.

Also removes an upper bound in set_interval that compared a value in seconds
against a deque length, and so silently clamped long intervals.

Late probe replies were being discarded after being counted

The edge stamped edge_rx_ts_us only after matching a reply against the
pending-probe table, so a reply arriving after its entry had been evicted was
dropped without a timestamp - censoring exactly the slow tail that a latency
distribution is about. Stamping happens before the match now, so a late reply
still reaches the cloud carrying usable data.

The logs could not reconstruct a run

node_metrics.csv gained the raw counters (gw_tx_count, gw_rx_count,
node_tx_count, node_rx_count, edge_tx_count, edge_rx_count) and the
per-hop latency halves, so PDR can be computed after the fact from counter
deltas rather than trusted from a running average. This matters because a node
rejoining zeroes the gateway's counters but not its own, and only the raw pairs
let an analysis segment around that.

metrics_setup.csv records the run's parameters, the write-once guard on it is
gone (it prevented recording a cadence derived after startup), and the metrics
CSVs are flushed at the sampling rate so a run can be analysed while it is still
going.

TUI and CLI

The TUI shows the probe budget as a share of downlink, the measured probe rate
with a retry indicator, uplink slot utilisation, and raw link saturation in both
directions - enough to see a saturating link during a run instead of at the
analysis. MQTT credentials now come from the environment.

Firmware tooling

flash.sh lists only the role's own device family, reports other-family boards
separately so a missing board is distinguishable from one plugged in under the
wrong role, and gains --erase-only and a both role.

build-schedules.sh builds one gateway net-core image per schedule. It edits a
single line to select the schedule and restores the file byte-for-byte on exit,
including through a failure, because a bench tree legitimately carries local
edits that must survive the build.

Validation

The Python changes have run continuously on the bench: 100 nodes at 2 m and
10 m, four traffic loads, and a schedule sweep across tiny/medium/big/huge, with
the counters cross-checked against the sliding-window PDR marilib reports
independently. hatch fmt --check and hatch test pass; firmware changes are
scripts only, no compiled code is touched.

geonnave added 18 commits July 31, 2026 07:38
edge_rx_ts_us must be written before the pending-probe match, not
after: a reply that arrives past its timeout has no pending entry, but
the frame is still forwarded to the cloud, where an unstamped 0 reads
as a large negative round trip. Keep the stamp ahead of the match if
this function is ever reordered.

AI-assisted: Claude Opus 5
log_setup_parameters rewrites metrics_setup.csv on every call by
design. It previously wrote once, which silently dropped every field
added after construction - including schedule_name, which is only
knowable when the first GATEWAY_INFO arrives.

AI-assisted: Claude Opus 5
The metrics probes are unicast downlink packets competing for the same
D slots as the filler traffic, so a flag that ignored them put 90% on
the link when asked for 75%. The probe rate is now subtracted from the
target, recomputed per packet since it grows with the node count.

AI-assisted: Claude Opus 5
The interval that keeps the measurement footprint constant depends on
the schedule (1.0 s on tiny, 7.9 on huge at 15%), so a period is the
wrong thing to hand an operator: it makes the instrument scale with
the node count unless recomputed by hand per run. --probe-load takes
the invariant instead and derives the period from the schedule the
gateway reports. -i still overrides, for reproducing a fixed cadence.

AI-assisted: Claude Opus 5
The measured rate counts retries, so it reads above target exactly when
probes are timing out - the case where the link carries more than the
requested load and the latency series is being censored. Also carries
the RSSI colouring on the per-node table.

AI-assisted: Claude Opus 5
Same variable names as the dotbot CLI. MQTTAdapter.from_url already
took the arguments and its docstring said callers thread env vars
through it, but none of them did, so authenticating against a broker
meant putting credentials in the URL and therefore in shell history.

AI-assisted: Claude Opus 5
from_url takes is_edge positionally, so splatting the credentials with
a single star landed one of them on it. They go through as keyword
arguments now.

The cadence derivation no longer blocks at startup with a deadline:
GATEWAY_INFO arrives asynchronously over serial and can take longer
than any timeout worth aborting a run over. It is applied from the main
loop the moment the schedule is known, which costs nothing beforehand
since the tester skips every cycle while no node has joined.

AI-assisted: Claude Opus 5
Erasing a gateway meant running the two nrfjprog recovers by hand, one
per core, since the script would only erase on the way to programming.

AI-assisted: Claude Opus 5
Once --load began subtracting the probe rate, total offered downlink
became constant by construction, so a share-derived cadence bought
nothing and cost sampling resolution: 15% meant 7.9 s per node on the
huge schedule. --probe-load stays for holding the footprint constant.

AI-assisted: Claude Opus 5
The slotframe is the network's own clock, so one K samples every
schedule at the same rate relative to its own dynamics. It also holds
the probe's downlink share near-constant for free: that share is
N/(K*d_down), and N/d_down is 4.13 to 5.00 across the four schedules,
so it is ~4.6/K on any of them. A wall-clock cadence gave an 8x spread
in share and a 10x spread in sample count over the same sweep.

set_interval had been rejecting any interval above
MARI_PROBE_STATS_MAX_LEN, comparing a duration against a deque length.
Nothing breaks at a long interval; the bound worth having is at the
short end, where the probe stream outruns downlink capacity, and that
depends on the schedule and node count. The TUI reports it instead.

AI-assisted: Claude Opus 5
test_state.rate was only ever set by LoadTester, which does not start
when no load is generated - so the one case where the probe budget is
the whole story was the case where the TUI hid it.

AI-assisted: Claude Opus 5
Each node owns exactly one uplink slot per slotframe, so its budget is
1/sf_duration packets per second and the number worth seeing is how
much of it is in use. Measured from received frames rather than derived
from the node app's send rates: the 500 ms status packet alone is 51%
of that budget on the huge schedule, and that is a firmware constant
the host should not be asserting.

AI-assisted: Claude Opus 5
What --load and --probe-every were set to is what was asked for; this
is what the link is actually carrying, counted from frames. Downlink is
measured against d_down slots per slotframe, uplink against one slot
per connected node, since unassigned U slots are idle by construction
and counting them would hide node saturation behind an empty schedule.

AI-assisted: Claude Opus 5
log_events.csv already flushed per row; the gateway and node CSVs only
at close. So a run was unreadable while in progress, and a process that
died without close() left empty files behind.

AI-assisted: Claude Opus 5
The gateway's schedule is a compile-time pointer, so switching it needs
a rebuild; nodes do not, since they adopt what the beacon advertises.
Building all of them once turns a campaign's schedule changes into
flashes. main.c is restored by an EXIT trap, and the script refuses to
start if the file is already modified rather than reverting someone
else's edit.

AI-assisted: Claude Opus 5
The refusal to run on a modified main.c blocked the normal case: the
bench net id lives in that file and is never committed. The snapshot
and restore already preserve whatever state the file was in, so the
guard bought nothing; the restore is now verified instead, and says
where the copy is if it could not be put back.

AI-assisted: Claude Opus 5
AI-assisted: Claude Opus 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant