EPIC: Issues following API Keyper set transition deployment 1002 to eon 13

[ylembachar]

We run into several issues during the API Keyper set transition of deployment 1002 to eon 13.

Corresponding issues are linked as sub-issues.

Suggested fix to get the API Keyper set to a working state:

• 0- Hardcode eon 13 in the Shutter API so that the API is still operational
• 1- Update internal Keypers with Fork Override
• 2- Fix issue with the fork update / test it
• 3- Find a fix for the DKG issue, to fix eon mismatch between Keyper and Shuttermint chain
• 4- Fix the malformed config 14 pipeline block so later valid configs can be picked up
• 5- Create a release for shutter-api-gnosis-1002 with override and new assets version with updated dkg variables (correct phase length and delta) and fork fix
• 6- Ask external Keypers to update
• 7- Create a script so Keypers can manually delete the scheduled Shuttermint messages
• 8- Write steps to delete the Shuttermint messages
• 9- Ask corresponding Keypers to update
• 10- Make sure with the above fixes that the validator set can transition, otherwise check what the issue is
• 11- Fix the Shutter API to use the latest eon
• 12- Deploy the fork fix to Gnosis Keypers as well

[ylembachar]

Document to send to the Keypers: https://docs.google.com/document/d/1wSvkUBy2v3aVFd2LFXqlcNjnJRn5vGPmS2P045T_Jzs/edit?tab=t.0

[ylembachar]

PR to review: shutter-network/shutter-keyper-deployment#61

[ylembachar]

Asked Keypers to delete the Shuttermint outgoing messages for config 14: https://github.com/brainbot-com/Shutter-PM/issues/1655

We're planning a DKG tomorrow to transition to our API Keypers and we can then reschedule another Keyepr set transition to the external Keypers once they the following issues:

1. all have upgraded their nodes to Fusaka
2. all with a consensus failure have resynched the chain

The validator not transitioning issue should be fixed when 2/3+1 send the block seen message, which wasn't the case in the last transition, but we would have to check that this works next time.

Then we can ask to inject the ETH keys.

[ylembachar]

When transitioning to config 15, we observed that the validator set finally transitioned to the validator set of config 13, after having been stuck on config 12, which in turn failed the DKG for config 15.

The validator transition to config 13 is due to two reasons:

1. The Keyper code currently only sends blockSeen for configs that include the local Keyper. This creates an implicit overlap
   requirement between consecutive keyper sets, since a config can only gather enough blockSeen votes if enough keypers from the previous config are also present in the next one.
   https://github.com/shutter-network/rolling-shutter/blob/main/rolling-shutter/keyper/keyper.go#L231
   https://github.com/shutter-network/rolling-shutter/blob/main/rolling-shutter/app/app.go#L775

2. BlocksSeen is tracked per sender, not per config. As a result, later blockSeen messages sent by keypers for config 15 also counted toward starting config 13, because their reported block numbers were already greater than or equal to activationBlockNumber(13).
   https://github.com/shutter-network/rolling-shutter/blob/main/rolling-shutter/app/app.go#L780

Concretely, config 12 had internal keypers 1,2,3,4,5. Config 13 had only internal Keypers 4 and 5 plus other Keypers (mostly external). Config 15 again had internal keypers 1,2,3,4,5. Because those internal Keypers were present again in config 15, their later blockSeen messages caused Shuttermint to consider config 13 started, which triggered the validator transition.

Once the validator set transitioned to config 13, the DKG did not continue successfully as the Shuttermint chain stalled due to external Keypers on the consensus failure.

Point 1 creates an implicit overlap requirement between consecutive Keypers sets, which probably shouldn't be enforced. A possible fix is to allow Keypers to send blockSeen for upcoming configs even if they are not part of those configs. @jannikluhn perhaps you have some thoughts on this?

As a workaround, we found that in this specific case we can issue a new keyper-set transition to config 16 with the same set of Keypers 1,2,3,4,5, once enough Keypers from config 13 have re-synced their chain.

To make that possible, we first need to fix the re-sync process for the case where the state file is missing. Otherwise, operators would have to back up the state file, manually set its height to 0, and then re-sync the chain. If you could please create a corresponding issue for this, @blockchainluffy, since you're working on it. thank you.

So basically next steps are:

1. Fix the resynch process
2. Ask Keypers on the consensus failure to resynch their chain
3. Transition to config 16 with internal Keypers
4. Transition to the external Keyper set
5. Inject the time capsule keys (outside the scope of this EPIC)
6. Decide how to go about the keyper set overlap and fix it (outside the scope of this EPIC)

[ylembachar]

Released v1.4.2 to the API keypers and asked the docker ones with a consensus failure to re-synch first.
Related comms:
https://github.com/brainbot-com/Shutter-PM/issues/1664
https://github.com/brainbot-com/Shutter-PM/issues/1665
https://github.com/brainbot-com/Shutter-PM/issues/1666

Next steps:
We don't have a process for the DAppNode Keypers with the consensus failure, the idea being to transition to the internal Keyper set and ask them to start a Keyper from scratch.

[ylembachar]

Additional note on eon 13:

Eon 13 DKG appears to have failed on the same keypers that experienced the Shuttermint consensus/app-state divergence.

Eon 13 started at height 159198, with check-ins around 159201, poly commitments around 159202, and poly evals around 159203-159204. The affected keypers also logged a consensus error at height 159202:

2026-03-25 02:03:33.9932026/03/25 01:03:33.993429 ERR [       state.go:1290] prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.LastResultsHash.  Expected 4CE605E896D6525A09900F870434EE0B43F000B40D4E96EF0D053960F1A06BA2, got EC5BD2D781D5C4501F6D712508C1CE0573B921C6DEB043EA47E5A2EB887BE13A" height=159202 module=consensus round=0

Those same keypers later replayed the canonical chain after resync but finalized eon 13 as failed:

DKG process failed error="corrupt keyper 5 not considered corrupt" eon=13 success=false

This suggests the failed DKG result is related to the consensus divergence on those keypers.