Restore typesense volume in prod - #4
Open
fastfadingviolets wants to merge 3 commits into
Open
fastfadingviolets wants to merge 3 commits into
fastfadingviolets wants to merge 3 commits into
Conversation
Prod has been on typesense/typesense:0.24.1 since June 2023. Dev is already on 30.2 and the upgrade is a documented single hop: the server rebuilds its in-memory indices from the existing data directory on start, so no reindex is required on the way up. It is a one-way door, though. Typesense 0.25 changed the on-disk format and v30 downgrades no lower than v27, so once the prod container starts on 30.2 the data directory can never be read by 0.24.1 again. Rollback means restoring the data directory, not re-pinning the tag. Also give prod the 0/100 deployment percentages dev already has, in this same commit rather than a later one. Prod was deliberately left on CDK's 50/200 default so that shipping the dev upgrade left the prod service byte-identical to what was deployed, but 200% lets ECS start the replacement task before stopping the old one. Both tasks land on the single container instance and mount the same shared Docker volume, and Typesense holds an exclusive RocksDB lock on /app/data, so the replacement cannot open the data directory and exits. Neither service configures a deployment circuit breaker -- cdk synth warns about this -- so ECS would retry rather than fail fast. That hazard fires on the first change to the prod task definition, which is this one. The new test pins both services at 0/100 so the asymmetry cannot return silently. Verified by synthesizing all three stacks against HEAD: of 80 resources in the shared stack exactly two differ, the prod task definition's image and the prod service's DeploymentConfiguration. Both admin key secrets, both dev search resources, and the cluster launch configuration are untouched, so the instance is not replaced and neither Typesense volume is at risk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVeWYTERpQWo3TpoyRuGtD
Prepared rollback for the 30.2 prod upgrade, to stay UNMERGED unless a rollback is actually needed. Same attach-and-copy design as the dev branch: a cdk.context.json key gates a second block device created from a snapshot, plus instance user data that stops the ECS agent, copies the two Typesense docker volumes off it, and starts the agent again. Merging changes the launch configuration, which replaces the instance, which comes up carrying the restored indexes. Reverting the merge is the undo. Written fresh rather than carried over from typesense-dev-restore, which is based on 2f1b282 and so predates c3d899d -- merging that branch today would revert the launch-configuration byte-stability test and the keyword-only deployment percentages. ARMING THIS IS TWO COUPLED EDITS. Set search_restore_snapshot_id to the pre-upgrade snapshot AND put typesense_image_prod back to 0.24.1. Typesense 0.25 changed the on-disk format and v30 downgrades no lower than v27, so a restored 0.24 data directory cannot be served by 30.2, and re-pinning the tag without restoring the data cannot read the directory 30.2 has migrated. Either edit alone leaves prod broken. The snapshot to use does not exist yet -- take it immediately before the prod deploy; snap-0bf402788b535f7cf predates the dev upgrade and is stale. Both volumes are copied even though this is a prod rollback. The restore works by replacing the instance and both volumes live on it, so dev's data dies either way; copying prod alone would leave dev empty and needing a full reindex, while copying both puts dev back where it already was. That holds only if the snapshot is fresh, which is the other reason not to reuse the August one. Per #10, an ASG block device created from a snapshot reseeds on every instance launch rather than persisting. Put the key back to null immediately after a successful restore, or the next unrelated instance replacement silently reverts prod to the snapshot. Verified by synthesis: with the key null all three stacks are byte-identical to e58377c across 100 resources, so the mechanism is inert as committed. Armed with a snapshot id, exactly one resource changes -- the launch configuration -- in exactly two properties, BlockDeviceMappings and UserData. 10 tests pass, black clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVeWYTERpQWo3TpoyRuGtD
Sets both halves of the gate together, which is the only correct way to set either. search_restore_snapshot_id points at the pre-upgrade snapshot taken 2026-09-09T14:40:08Z of vol-0d73d8c1016d4a6bb, and typesense_image_prod goes back to 0.24.1 to match the data directory that snapshot contains. The snapshot was taken with both services running, so both copies are crash-consistent and prod's RocksDB may want WAL recovery on start. Dev was cleanly stopped for the August capture; it was not stopped for this one, because stopping a service to improve a rollback that may never be used is a worse trade than a WAL replay. Restructured the launch-configuration tests, which is not cosmetic: arming deliberately breaks the byte-stability the old assertion pinned, and since c3d899d the pipeline's Synth step runs `uv run pytest` as a deploy gate. A rollback branch that fails its own gate cannot deploy, so the rollback would have been unusable in exactly the situation it exists for. The user data literal is now split into a base and a restore tail, the expected value is derived from whether the gate is set, and the two states get a fixture each. Verified the same suite passes both armed and disarmed, so this survives the key going back to null after a restore. Synthesized and compared against e58377c, the upgrade this undoes: of 80 resources exactly two differ -- the launch configuration gains BlockDeviceMappings and the restore user data, and the prod task definition's image returns to 0.24.1. Nothing else moves. 10 tests pass, black clean. DO NOT MERGE unless rolling back. After a successful restore, put search_restore_snapshot_id back to null: an ASG block device created from a snapshot reseeds on every instance launch rather than persisting, so leaving it set silently reverts prod on the next instance replacement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVeWYTERpQWo3TpoyRuGtD
fastfadingviolets
force-pushed
the
typesense-prod-restore
branch
from
September 9, 2026 15:38
fd4b3c7 to
06d1b05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only needed if #3 goes awry and we need to roll back.