fix(yaml_loader): refresh stale table cache on miss during config apply - #2151
fix(yaml_loader): refresh stale table cache on miss during config apply#2151ddavignon-hadrian wants to merge 1 commit into
Conversation
change_retentions resolve source/destination tables against the tables list cached on the postgres_databases record, which is only refreshed on database create and by the 6-hourly EnqueueDatabaseUpdateWorker cron. For an existing database, the config-apply update path never refreshes tables, so a table created since the last refresh is absent during apply. Because apply runs at container start, this fails the boot and crash-loops the deployment, which in turn prevents the cron from ever healing it. Refresh the referenced database's tables from the live database on a cache miss (and retry the lookup) while parsing change_retention attrs. Cached hits keep the existing in-memory fast path, and genuinely missing tables still return not_found, so this only adds a round-trip when a referenced table is absent from the cache. Fixes sequinstream#2150.
|
Quick heads up on the failing Looks like the runner is missing the postgres client. Could a maintainer re-run it once that's sorted? In the meantime I ran it locally on Elixir 1.19.4 / OTP 28.3 against Postgres 17 with pgvector and logical replication. The project compiles clean and the new test passes. I also checked that the test fails without the fix, with the same "No table found matching params: name=sequin_events, schema=public" error from the linked issue, so it reproduces the bug and the change resolves it. Happy to change anything on my side if it's needed. |
Problem
Fixes #2150.
When applying declarative YAML config,
change_retentionsresolve their source/destination tables against thetableslist cached on thepostgres_databasesrecord. That cache is only refreshed when a database is created and by the 6-hourlyEnqueueDatabaseUpdateWorkercron — the config-apply update path (update_db/2) never refreshes it. So if a table is created after the last refresh,apply_configfails with:Because config apply runs at container start (before
start_application), this fails the boot and crash-loops the container, which in turn prevents the 6-hourly cron from ever running — so the deployment can't self-heal. (Full analysis in #2150.)Fix
Refresh the referenced database's tables from the live database on a cache miss while parsing
change_retentionattrs, then retry the lookup (option 1 from the issue). A newensure_table_cached/3helper:Databases.update_tables/1once to refresh from the live DB, preserving thereplication_slotpreload;not_foundif the table is genuinely absent after refreshing (no behavior change for real misses).A round-trip is only added when a referenced table is missing from the cache, so applies that don't reference new tables are unchanged.
Test
Adds a
change_retentionstest that creates a database, clears its cachedtablesto simulate a stale cache, then applies a config whosechange_retentionreferences those tables and asserts the apply succeeds and the pipeline is created.