Skip to content

Scale FMoW to 200 Satellites - #77

Open
Nova-David wants to merge 3 commits into
dg-fork-mainfrom
dc/FMoW
Open

Scale FMoW to 200 Satellites#77
Nova-David wants to merge 3 commits into
dg-fork-mainfrom
dc/FMoW

Conversation

@Nova-David

Copy link
Copy Markdown
Collaborator

Scaling to 200 satellites

  • Fixed chunk transport to tolerate out-of-order delivery under concurrency, added a 30s stale-transfer timeout, doubled chunk size to 4MB
  • Fixed RandomSelector never resetting selected_ends/all_selected between rounds which made agg_goal unreachable and triggered false SIM_STARVATION after the first round
  • Reduced per-round payload size: frozen-layer parameters excluded from delta_weights, empty-dataset trainers now send a status-only message instead of a full payload, added send jitter to avoid synchronized traffic bursts

Evaluation

  • Final round now always evaluates (previously skippable), runs synchronously so the run doesn't exit before it finishes

Guards ahead of a dg-fork-main merge

  • eval_every_n_commits set 1 to neutralize a new change that would otherwise silently drop ~half of scheduled evaluations
  • heartbeats config access now uses getattr with a default, since the field is being removed from the shared schema upstream

@Nova-David Nova-David changed the title FMoW Full Run Changes Scale FMoW to 200 Satellites Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants