Skip to content

fix(yaml_loader): refresh stale table cache on miss during config apply - #2151

Open
ddavignon-hadrian wants to merge 1 commit into
sequinstream:mainfrom
ddavignon-hadrian:fix/yaml-config-stale-table-cache
Open

fix(yaml_loader): refresh stale table cache on miss during config apply#2151
ddavignon-hadrian wants to merge 1 commit into
sequinstream:mainfrom
ddavignon-hadrian:fix/yaml-config-stale-table-cache

Conversation

@ddavignon-hadrian

Copy link
Copy Markdown

Problem

Fixes #2150.

When applying declarative YAML config, change_retentions resolve their source/destination tables against the tables list cached on the postgres_databases record. That cache is only refreshed when a database is created and by the 6-hourly EnqueueDatabaseUpdateWorker cron — the config-apply update path (update_db/2) never refreshes it. So if a table is created after the last refresh, apply_config fails with:

No `table` found matching params: `name=..., schema=public`

Because config apply runs at container start (before start_application), this fails the boot and crash-loops the container, which in turn prevents the 6-hourly cron from ever running — so the deployment can't self-heal. (Full analysis in #2150.)

Fix

Refresh the referenced database's tables from the live database on a cache miss while parsing change_retention attrs, then retry the lookup (option 1 from the issue). A new ensure_table_cached/3 helper:

  • returns the database unchanged on a cache hit (existing in-memory fast path — no extra round-trip);
  • on a miss, calls Databases.update_tables/1 once to refresh from the live DB, preserving the replication_slot preload;
  • still returns not_found if the table is genuinely absent after refreshing (no behavior change for real misses).

A round-trip is only added when a referenced table is missing from the cache, so applies that don't reference new tables are unchanged.

Test

Adds a change_retentions test that creates a database, clears its cached tables to simulate a stale cache, then applies a config whose change_retention references those tables and asserts the apply succeeds and the pipeline is created.

change_retentions resolve source/destination tables against the tables list
cached on the postgres_databases record, which is only refreshed on database
create and by the 6-hourly EnqueueDatabaseUpdateWorker cron. For an existing
database, the config-apply update path never refreshes tables, so a table
created since the last refresh is absent during apply. Because apply runs at
container start, this fails the boot and crash-loops the deployment, which in
turn prevents the cron from ever healing it.

Refresh the referenced database's tables from the live database on a cache
miss (and retry the lookup) while parsing change_retention attrs. Cached hits
keep the existing in-memory fast path, and genuinely missing tables still
return not_found, so this only adds a round-trip when a referenced table is
absent from the cache.

Fixes sequinstream#2150.
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. bug Something isn't working labels Jul 30, 2026
@ddavignon-hadrian

ddavignon-hadrian commented Jul 30, 2026

Copy link
Copy Markdown
Author

Quick heads up on the failing signoff check: it died at the Configure PostgreSQL step with psql: command not found (exit 127) on the warp-ubuntu-latest-arm64-32x runner. That's a setup step, so nothing after it ran and the Elixir compile and tests were skipped entirely. I don't think it's related to this PR, which only changes lib/sequin/yaml_loader.ex and its test.

Looks like the runner is missing the postgres client. Could a maintainer re-run it once that's sorted?

In the meantime I ran it locally on Elixir 1.19.4 / OTP 28.3 against Postgres 17 with pgvector and logical replication. The project compiles clean and the new test passes. I also checked that the test fails without the fix, with the same "No table found matching params: name=sequin_events, schema=public" error from the linked issue, so it reproduces the bug and the change resolves it.

Happy to change anything on my side if it's needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

YAML config apply resolves change_retentions against a stale table cache, crash-looping self-hosted deployments when tables are new

2 participants