Skip to content

feat(store): migrate the response-store filter to registry-only resolution - #1284

Closed
hexfusion wants to merge 29 commits into
praxis-proxy:mainfrom
hexfusion:feat/1259b-filter-migration
Closed

hexfusion wants to merge 29 commits into
praxis-proxy:mainfrom
hexfusion:feat/1259b-filter-migration

Conversation

@hexfusion

@hexfusion hexfusion commented Sep 22, 2026 •

Copy link
Copy Markdown

built on #1279

Summary

Migrate the response-store filter off its own connection-pool lifecycle onto the shared registry, and populate that registry on the serving runtime. The filter resolves the store only through the owner-scoped registry handle (get_scoped) that the rehydrate filter already uses, and holds no OnceCell, build_store path, or StorageBackend enum. A Pingora background service on the serving runtime provisions the configured backends once and registers them into the per-listener registry. A request that arrives before provisioning completes, or against an unreachable backend, gets a backend-unavailable error rather than a per-request pool build. Backend config is validated fatally at pipeline construction, so a malformed or unknown-backend config fails startup. Part of the store-layer split (#1257), building on the contracts crate (#1258).

Related issue

Closes #1259
Also lands the store-injection portion of #1261: the response-store filter is now registry-only. The remaining Responses service-layer extraction stays tracked in #1261.
Part of #1257.

Validation

  • Unit tests
  • Integration or functional tests
  • make lint

Checklist

  • I reviewed every changed line and can explain the change.
  • No new user-facing capability: a behavior-preserving migration.
  • No user-facing behavior or generated-doc change.
  • No hot-path change.
  • Commits are signed and include a Signed-off-by trailer.

Breaking changes

No public API or runtime behavior change. The response-store filter no longer constructs its own SQL pool; backend provisioning moves to a serving-runtime background service. A store-configuration change is now restart-required rather than hot-reloaded, consistent with the existing restart-required configuration surfaces.

@leseb leseb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 — Reloaded store configuration is ignored. Reload swaps the new pipeline while reusing startup registries. Changing the database silently keeps the old backend; adding a store filter produces an empty registry and 500s. This directly misses #1259’s reload lifecycle requirement. reload.rs:69

P1 — Branch-scoped store filters are never provisioned. The scan only checks listeners’ top-level chains, while the pipeline supports reachable named/inline branch_chains. Taking such a branch returns store-unavailable. store_provision.rs:131

P1 — Readiness becomes stale after reload. The readiness endpoint retains the startup health registry, while reload creates a replacement. It can report 200 for degraded current clusters. server.rs:117

P2 — Permanent provisioning failures retry forever. SQLite permanent failures and exhausted PostgreSQL retries are both caught by the unconditional outer retry loop. The normal listener is already accepting traffic, and readiness is opt-in, so an invalid deployment can remain alive returning 500s indefinitely. store_provision.rs:229

P2 — Partial provisioning does not roll back registry entries. If a later store reference fails, earlier entries remain registered even though their leases are released. Retrying then fails as a duplicate. lib.rs:193

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

Well-structured migration. The refcounted BackendCache with dedup-by-effective-key, the BackendLease lifecycle, and the readiness-gated provisioning background service are all cleanly designed. One medium finding on config-time validation coverage.

Comment thread apis/src/store/provisioning.rs
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…n, lint)

Rebased onto the praxis-proxy#1258 tip. Thread response-store compression (praxis-proxy#1182) through
the provisioning factory (new compression field on the Sqlite/Postgres factory
configs, passed to the constructors, fingerprinted in the dedup key). Restore
the store-feature gate on DEFAULT_STORE_NAME, turbofish four Arc-to-dyn clones
for clone_on_ref_ptr, and expect too_many_lines where the policy-connector
setup lands.

Known-incomplete: the server threads store unconditionally (77 refs, 7 files)
but current main made the store feature optional, so this does not yet build
under --features standard. Needs a store-optionality decision before merge.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
Current main makes the store feature optional (a standard/FIPS build carries no
SQL crypto), but praxis-proxy#1284 threaded the store unconditionally through serve, reload,
watch, and pipeline resolution. Introduce a StoreRegistries alias (the real map
under store, a placeholder otherwise) so the serve and watch signatures stay
feature-free, gate the two resolver call sites and the store-config-change
detector, and route a no-store build through resolve_pipelines. The server now
builds under --features standard with no store or SQL crypto.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…SSRF)

Address the outstanding review on this PR:

- validate_config runs revalidate_postgres_host so a private/loopback DB host
  fails fatally at startup instead of passing construction and then looping in
  async provisioning (the praxis-bot SSRF finding).
- Permanent provisioning errors (unknown backend, config) stop retrying and
  leave readiness failed, rather than looping forever; transient errors still
  self-heal with backoff.
- provision_into deregisters the names it registered when a later reference
  fails, so a partial attempt does not leave entries that make a retry a
  duplicate (adds StoreRegistry::deregister).
- Store provisioning follows inline and named branch chains, so a store
  configured only inside a branch is provisioned, not left store-unavailable.
- A store-config change rejects the reload loudly (keeping the running pipeline
  and its provisioned backends) instead of swapping in one whose registry is
  stale or empty and 500s.
- Readiness reads a shared, reload-updated health registry rather than the
  startup snapshot, so it cannot report ready for degraded current clusters.

Also regate is_explicit_compact_request to openai-compact, its only remaining
caller after the registry-only filter migration removed the store-filter use.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
@hexfusion
hexfusion force-pushed the feat/1259b-filter-migration branch from 632014f to e3b2bfb Compare September 25, 2026 03:02
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…n, lint)

Rebased onto the praxis-proxy#1258 tip. Thread response-store compression (praxis-proxy#1182) through
the provisioning factory (new compression field on the Sqlite/Postgres factory
configs, passed to the constructors, fingerprinted in the dedup key). Restore
the store-feature gate on DEFAULT_STORE_NAME, turbofish four Arc-to-dyn clones
for clone_on_ref_ptr, and expect too_many_lines where the policy-connector
setup lands.

Known-incomplete: the server threads store unconditionally (77 refs, 7 files)
but current main made the store feature optional, so this does not yet build
under --features standard. Needs a store-optionality decision before merge.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
Current main makes the store feature optional (a standard/FIPS build carries no
SQL crypto), but praxis-proxy#1284 threaded the store unconditionally through serve, reload,
watch, and pipeline resolution. Introduce a StoreRegistries alias (the real map
under store, a placeholder otherwise) so the serve and watch signatures stay
feature-free, gate the two resolver call sites and the store-config-change
detector, and route a no-store build through resolve_pipelines. The server now
builds under --features standard with no store or SQL crypto.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…SSRF)

Address the outstanding review on this PR:

- validate_config runs revalidate_postgres_host so a private/loopback DB host
  fails fatally at startup instead of passing construction and then looping in
  async provisioning (the praxis-bot SSRF finding).
- Permanent provisioning errors (unknown backend, config) stop retrying and
  leave readiness failed, rather than looping forever; transient errors still
  self-heal with backoff.
- provision_into deregisters the names it registered when a later reference
  fails, so a partial attempt does not leave entries that make a retry a
  duplicate (adds StoreRegistry::deregister).
- Store provisioning follows inline and named branch chains, so a store
  configured only inside a branch is provisioned, not left store-unavailable.
- A store-config change rejects the reload loudly (keeping the running pipeline
  and its provisioned backends) instead of swapping in one whose registry is
  stale or empty and 500s.
- Readiness reads a shared, reload-updated health registry rather than the
  startup snapshot, so it cannot report ready for degraded current clusters.

Also regate is_explicit_compact_request to openai-compact, its only remaining
caller after the registry-only filter migration removed the store-filter use.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…proxy#1284

Import DEFAULT_PAGE_LIMIT from the ungated input_items path rather than the
openai-conversations-gated re-export, so the store-postgres unit tests build
without the conversations feature. Copy the store-lifecycle crate into both
Containerfiles so the FIPS ubi-image workspace resolves the member this PR
adds.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
@hexfusion
hexfusion force-pushed the feat/1259b-filter-migration branch from e3b2bfb to 5525aa5 Compare September 25, 2026 04:08
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…n, lint)

Rebased onto the praxis-proxy#1258 tip. Thread response-store compression (praxis-proxy#1182) through
the provisioning factory (new compression field on the Sqlite/Postgres factory
configs, passed to the constructors, fingerprinted in the dedup key). Restore
the store-feature gate on DEFAULT_STORE_NAME, turbofish four Arc-to-dyn clones
for clone_on_ref_ptr, and expect too_many_lines where the policy-connector
setup lands.

Known-incomplete: the server threads store unconditionally (77 refs, 7 files)
but current main made the store feature optional, so this does not yet build
under --features standard. Needs a store-optionality decision before merge.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
Current main makes the store feature optional (a standard/FIPS build carries no
SQL crypto), but praxis-proxy#1284 threaded the store unconditionally through serve, reload,
watch, and pipeline resolution. Introduce a StoreRegistries alias (the real map
under store, a placeholder otherwise) so the serve and watch signatures stay
feature-free, gate the two resolver call sites and the store-config-change
detector, and route a no-store build through resolve_pipelines. The server now
builds under --features standard with no store or SQL crypto.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…SSRF)

Address the outstanding review on this PR:

- validate_config runs revalidate_postgres_host so a private/loopback DB host
  fails fatally at startup instead of passing construction and then looping in
  async provisioning (the praxis-bot SSRF finding).
- Permanent provisioning errors (unknown backend, config) stop retrying and
  leave readiness failed, rather than looping forever; transient errors still
  self-heal with backoff.
- provision_into deregisters the names it registered when a later reference
  fails, so a partial attempt does not leave entries that make a retry a
  duplicate (adds StoreRegistry::deregister).
- Store provisioning follows inline and named branch chains, so a store
  configured only inside a branch is provisioned, not left store-unavailable.
- A store-config change rejects the reload loudly (keeping the running pipeline
  and its provisioned backends) instead of swapping in one whose registry is
  stale or empty and 500s.
- Readiness reads a shared, reload-updated health registry rather than the
  startup snapshot, so it cannot report ready for degraded current clusters.

Also regate is_explicit_compact_request to openai-compact, its only remaining
caller after the registry-only filter migration removed the store-filter use.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…proxy#1284

Import DEFAULT_PAGE_LIMIT from the ungated input_items path rather than the
openai-conversations-gated re-export, so the store-postgres unit tests build
without the conversations feature. Copy the store-lifecycle crate into both
Containerfiles so the FIPS ubi-image workspace resolves the member this PR
adds.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
@hexfusion
hexfusion force-pushed the feat/1259b-filter-migration branch from 5525aa5 to 090b7a5 Compare September 25, 2026 07:21
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…proxy#1284

Import DEFAULT_PAGE_LIMIT from the ungated input_items path rather than the
openai-conversations-gated re-export, so the store-postgres unit tests build
without the conversations feature. Copy the store-lifecycle crate into both
Containerfiles so the FIPS ubi-image workspace resolves the member this PR
adds.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
@hexfusion
hexfusion force-pushed the feat/1259b-filter-migration branch from 090b7a5 to f35cb97 Compare September 25, 2026 11:56
Extract the persistence contracts from praxis-ai-apis into a new
publish=false praxis-ai-store crate with no sqlx/TLS in its dependency
graph: the StateOwner identity (from_trusted_parts stays the only
constructor, so the crate cannot mint a request-forged owner), the record
types and StoreError, the ResponseStore and ConversationItemStore traits,
and a PersistedStateBackend supertrait with a blanket impl over any type
implementing both halves.

Part of praxis-proxy#1258 (epic praxis-proxy#1257). The SQL backends adopt these in a follow-up.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…t suite

StoreRegistry holds Arc<dyn PersistedStateBackend> so a resolved backend
provably implements both trait halves; a Conversations-requiring caller
can never receive a response-only backend. OwnerScopedStore binds every
request-driven access to one validated owner. The registry stays
transport-free (no praxis_filter); the pipeline binding lives in the
transport layer.

InMemoryStore is a deterministic double for service unit tests: a single
mutex makes the multi-step operations (all-or-nothing approval
consumption, compare-and-swap, create/delete-items-and-sync) atomic, as
the SQL backends achieve with a transaction. The test-support
contract_tests suite is the shared conformance harness the in-memory and
SQL backends run against.

Part of praxis-proxy#1258 (epic praxis-proxy#1257).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…fter the response

The transport layer (apis) validates an individual owner component as it
parses trusted headers, before it has all three parts for
from_trusted_parts, so validate_component is now public. The contract
suite writes the owning response before recording its pending approvals,
so a backend with a foreign key from approvals to responses accepts them.

Part of praxis-proxy#1258 (epic praxis-proxy#1257).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Depend on praxis-ai-store and re-export its traits, records, owner, and
errors from crate::store and crate root, so existing paths keep compiling.
Remove the moved trait_def and types modules; split StateOwner out of
state_owner.rs (the transport filter and projection stay). The SQL
backends implement the moved traits and the ResponseStoreRegistry becomes
a transport-bound wrapper over the SQL-free StoreRegistry, holding a
combined Arc<dyn PersistedStateBackend> so a resolved backend always has
both halves; the PipelineExtension binding lives here, keeping the store
crate transport-free.

Prove adoption by running the shared contract suite against the SQLite
backend. The conversations-filter store path is unchanged (deferred to

Part of praxis-proxy#1258 (epic praxis-proxy#1257).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…d test-support

Nothing in non-test src uses tokio (InMemoryStore is std::sync::Mutex;
the tokio::test macros are covered by the dev-dependency), so remove it
from the normal dependencies to keep tokio out of every downstream
consumer's non-dev closure. Gate pub mod memory behind test-support like
contract_tests, so the test double is not built or exported in the
default build.

Part of praxis-proxy#1258 (epic praxis-proxy#1257).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…i-store

The registry that used DashMap moved into praxis-ai-store, so apis no
longer uses dashmap. Remove it from the dependencies and the store
feature; cargo machete flagged it as unused.

Part of praxis-proxy#1258 (epic praxis-proxy#1257).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…ants

The reference double diverged from what the SQL backends enforce, so the
shared contract suite could pass a backend the SQL side would reject.

- Key responses globally by id; a colliding id from another owner is
  rejected, not shadowed (SQL PRIMARY KEY (id) + owner-guarded upsert).
- Override persist_response_with_pending_approvals to hold one lock across
  the response and approval writes, so a concurrent delete cannot orphan an
  approval (the default impl locks twice and leaves that window open).
- Reject item writes that orphan, cross owners, or reuse an id within one
  batch (SQL parent-scoped insert + item PRIMARY KEY).
- Expose the conversation-item surface owner-scoped on OwnerScopedStore,
  alongside the responses surface.

Extend the shared contract suite to assert each invariant against both the
in-memory and SQLite backends.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Relocate PoolConfig, SslMode, and table-name validation from apis/src/store
into a new backend_config module in the SQL-free praxis-ai-store crate, so
the backend-free build no longer links sqlx or its crypto (sqlx-postgres,
sqlite, hkdf, stringprep, and the prohibited md-5).

The sqlx conversions stay behind with the SQL backends: apply_pool_config in
pool.rs and the SslMode-to-PgSslMode mapping in postgres.rs. The mapping
becomes a free function because both types are foreign once SslMode moves, so
the From impl would violate the orphan rule.

praxis-ai-store stays SQL-free and crypto-free: cargo tree shows no sqlx,
rustls, ring, openssl, or md-5 under default or --all-features.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…ession

main landed response-store compression (praxis-proxy#1182) inside the store region this
branch extracts. Carry it across the rebase: add the compression argument to
the SQLite contract-suite constructor call, and point the compression benchmark
at a now-public to_pg_ssl_mode. The From<SslMode> for PgSslMode impl became an
orphan once SslMode moved to praxis-ai-store, so the conversion is a function
the benchmark can call.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
The workspace gained the praxis-ai-store member, but the container and FIPS
Containerfiles COPY crate dirs selectively and omitted it, so cargo could not
load the workspace manifest in the ubi-image (FIPS) build. Copy store/ into
both, matching each file's pattern (manifest+src for the cached build, whole
dir for the FIPS build).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Add StoreBackendFactory (backend_id, effective_key, build), BackendError with
a distinct backend-unavailable variant (and a transient variant for bounded
retry), a StoreCapability descriptor, an EffectiveConfigKey dedup key that
redacts its Debug, and a RetireBackend hook plus ProvisionedBackend. SQL-free
and crypto-free: concrete factories own their config type and any SQL/TLS.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…e crate

New publish=false praxis-ai-store-lifecycle crate depending only on the
praxis-ai-store interface (crypto-free, no store-backends). It routes inline
config to injected StoreBackendFactory implementations, builds and validates
backends eagerly at pipeline construction (SQLite permanent-init fails the
build; a transient failure retries within a bounded budget then fails as
unavailable), deduplicates identical effective configs onto one pooled backend,
and reuses or retires backends across a reload via a process-wide refcounted
cache with explicit retirement. Nine unit tests cover the six behaviors:
initial load, reload reuse, retire on last release and on changed config,
unavailable-at-build, transient-then-build, and single-pool dedup.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Wrap the existing SQLite backend as a StoreBackendFactory the lifecycle layer
provisions: it parses the inline config, computes the effective dedup key,
classifies a SQLite init failure as permanent (BackendError::Unavailable so the
build fails closed), and closes the pool on retirement via a new close() on the
SQL stores. Add a redaction helper that scrubs the connection string and any
embedded credentials from an error before it surfaces. Five end-to-end tests
bind the SQLite-applicable behaviors against a real file-backed SQLite backend
(initial load, reload reuse, changed-config second backend, unavailable-at-build
fails closed distinct from unknown-backend, single-pool dedup).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Mirror the SQLite factory for Postgres: parse inline config (incl. ssl_mode,
ssl_root_cert, allow_private_database_url), re-run revalidate_postgres_host on
every build attempt (guards DNS rebinding), build over PgTlsConfig, and close
the pool on retirement. A connect failure is classified BackendError::Transient
so the lifecycle cache retries it within budget, preserving the tested
Postgres-transient vs SQLite-permanent semantics. Redaction and the pool
fingerprint are shared across both factories. No runnable Postgres here, so the
transient-then-build behavior remains machinery-covered (9 lifecycle tests).

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…esolution

Add resolve_pipelines_with_stores, which attaches a caller-provided
ResponseStoreRegistry per listener. resolve_pipelines stays a thin
wrapper delegating with empty registries, so the validation, CLI, and
test paths are unchanged. The serve and reload paths provision backends
and pass populated registries here.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Add StoreBackendFactory::validate_config (default over effective_key, no I/O)
and BackendCache::validate, a synchronous pass that fails a malformed or
unknown-backend store config at pipeline construction rather than at first
traffic. Pool creation stays lazy on the serving runtime: sqlx pools bind to
the runtime that opens them and the server owns that runtime, unreachable at
sync construction, so eager here is eager config validation.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…e factory injection

The Postgres backend factory now models ssl_client_cert, ssl_client_key, and
require_certificate_authentication, matching the response-store filter config, so
factory validation and build cover client-cert mutual TLS and no longer reject a
config the filter accepts. Add store_backend_factories() so a binary can inject
the compiled factories into the lifecycle cache.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…ifecycle state

Run the filter's TLS/auth validation from the factory (connect and
validate_config), so a client cert under a non-verifying ssl_mode and the
certificate-authentication compliance profile fail closed on this path as they
already do in the filter. Key the Postgres dedup on the cert-path values rather
than their presence, so distinct trust anchors or client identities at
different paths do not collide onto one pool.

Drop the unused StoreCapability descriptor and StoreRef.capability field (no
runtime reads them), privatize redact_connection_error, correct the build-retry
pacing comment, and fix the cert-field docs to say path, not PEM material.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…ime background service

Install an empty per-listener ResponseStoreRegistry into each pipeline via a
From<StoreRegistry> bridge, and provision the configured backends into that
same shared map from a Pingora background service that runs on the serving
runtime. sqlx pools bind to the runtime that opens them, so provisioning must
run there rather than on the config-watcher runtime a reload uses.

BackendCache gains provision_into so a caller can populate a shared registry
handed to it before the serving runtime exists. Store configuration is
validated eagerly at startup (fatal), so a malformed or unknown-backend config
fails before serving; connectivity surfaces when the pools open.

Reload reuses the serving-runtime-provisioned registries: the reloaded pipeline
keeps the live backends, and a changed store config is restart-required rather
than re-provisioned on the watcher runtime.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…ution

The response-store filter no longer builds or caches its own backend. It
resolves an owner-scoped handle from the per-request ResponseStoreRegistry via
get_scoped(DEFAULT_STORE_NAME, owner), mirroring the rehydrate filter, so the
tenant-isolation seam has one public request-path read. The OnceCell, the
build/init/registration helpers, and the StorageBackend match are removed; the
filter is a stateless unit struct that validates config at construction and
rejects with 500 when a request that needs the store finds none provisioned.

OwnerScopedStore gains an owner-checked persist_response_with_pending_approvals
so the write path stays inside the facade. Permanent-vs-transient init policy
now lives in the factory's BackendError classification and the lifecycle cache,
not the filter.

Connection-error redaction is threaded into the Postgres backend's runtime
StoreError::Database sites: the store carries its URL and redacts it (and any
embedded credentials) through a shared helper, matching the connect-time
redaction the provisioning factories apply. SQLite URLs carry no credentials.

The filter test suite is rewritten to the provisioning model: tests install a
store into the context registry (or seed one) rather than the filter's OnceCell,
and the obsolete lazy-init / self-registration / permanent-failure tests are
dropped since that behavior moved to the factory and cache.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
The integration/recording harness (praxis-test-utils build_pingora_server)
built pipelines with an empty ResponseStoreRegistry and never ran the
store-provision background service, so after the filter migration a store-filter
request through the harness found no provisioned backend and rejected with 500.

Reuse the production wiring: build_store_wiring and StoreProvisionService are now
public on the praxis_ai crate, and build_pingora_server installs the per-listener
registries into the pipelines and registers the background service on the
harness's own Pingora server, exactly as boot_server does. No second provisioning
path is introduced.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
The provisioning module doc links StoreBackendFactory unqualified, but the
trait moved to praxis-ai-store in praxis-proxy#1258 and the module imports only
BackendError (cfg-gated). The always-compiled module doc left the link
unresolved, failing cargo doc under RUSTDOCFLAGS="-D warnings". Qualify it to
praxis_ai_store::StoreBackendFactory.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…ignal

The store-provision background service opens backend pools asynchronously after
the HTTP listener already accepts, so a store-filter request that arrives during
the provisioning window finds no provisioned backend and returns 500. The
integration harness only waited for HTTP readiness, so the two Postgres store
tests failed deterministically (500 vs 200): a Postgres pool open, and its TLS
handshake for the mTLS variant, reliably lose the race against HTTP readiness.

Add a StoreReadiness watch owned by StoreProvisionService and a cloneable
StoreReadinessHandle observers read or await. Signal Ready only once every
configured listener holds a lease. On a provisioning failure, keep the state
non-Ready and retry with bounded backoff so a transient database or TLS failure
self-heals rather than being logged and abandoned. The harness awaits Ready
before the first request.

The handle is reusable: the readiness endpoint and sibling workstreams read the
same one.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Expose the store readiness signal as an HTTP endpoint so an orchestrator probe
gates traffic on store provisioning. It runs on its own listener because the
protocol admin service owns its route set, and it composes store readiness with
cluster health: ready (200) only when store provisioning is Ready and cluster
health is ready, 503 otherwise, so a provisioning failure stops reporting
healthy.

It reads the same StoreReadinessHandle the provisioning service drives, so one
state serves both the endpoint and the harness wait. The listen address comes
from PRAXIS_STORE_READINESS_ADDR; enabling it is opt-in via that variable
(default value 0.0.0.0:9200, path /ready) so no new port binds unless a
deployment asks for it. praxis_core is untouched.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…n, lint)

Rebased onto the praxis-proxy#1258 tip. Thread response-store compression (praxis-proxy#1182) through
the provisioning factory (new compression field on the Sqlite/Postgres factory
configs, passed to the constructors, fingerprinted in the dedup key). Restore
the store-feature gate on DEFAULT_STORE_NAME, turbofish four Arc-to-dyn clones
for clone_on_ref_ptr, and expect too_many_lines where the policy-connector
setup lands.

Known-incomplete: the server threads store unconditionally (77 refs, 7 files)
but current main made the store feature optional, so this does not yet build
under --features standard. Needs a store-optionality decision before merge.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Current main makes the store feature optional (a standard/FIPS build carries no
SQL crypto), but praxis-proxy#1284 threaded the store unconditionally through serve, reload,
watch, and pipeline resolution. Introduce a StoreRegistries alias (the real map
under store, a placeholder otherwise) so the serve and watch signatures stay
feature-free, gate the two resolver call sites and the store-config-change
detector, and route a no-store build through resolve_pipelines. The server now
builds under --features standard with no store or SQL crypto.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…SSRF)

Address the outstanding review on this PR:

- validate_config runs revalidate_postgres_host so a private/loopback DB host
  fails fatally at startup instead of passing construction and then looping in
  async provisioning (the praxis-bot SSRF finding).
- Permanent provisioning errors (unknown backend, config) stop retrying and
  leave readiness failed, rather than looping forever; transient errors still
  self-heal with backoff.
- provision_into deregisters the names it registered when a later reference
  fails, so a partial attempt does not leave entries that make a retry a
  duplicate (adds StoreRegistry::deregister).
- Store provisioning follows inline and named branch chains, so a store
  configured only inside a branch is provisioned, not left store-unavailable.
- A store-config change rejects the reload loudly (keeping the running pipeline
  and its provisioned backends) instead of swapping in one whose registry is
  stale or empty and 500s.
- Readiness reads a shared, reload-updated health registry rather than the
  startup snapshot, so it cannot report ready for degraded current clusters.

Also regate is_explicit_compact_request to openai-compact, its only remaining
caller after the registry-only filter migration removed the store-filter use.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
…proxy#1284

Import DEFAULT_PAGE_LIMIT from the ungated input_items path rather than the
openai-conversations-gated re-export, so the store-postgres unit tests build
without the conversations feature. Copy the store-lifecycle crate into both
Containerfiles so the FIPS ubi-image workspace resolves the member this PR
adds.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
@hexfusion

Copy link
Copy Markdown
Author

Superseded by #1369, which carries the whole store-layer split (#1257) as one canonical PR. Closing to consolidate to a single source of truth.

@hexfusion hexfusion closed this Sep 25, 2026
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…n, lint)

Rebased onto the praxis-proxy#1258 tip. Thread response-store compression (praxis-proxy#1182) through
the provisioning factory (new compression field on the Sqlite/Postgres factory
configs, passed to the constructors, fingerprinted in the dedup key). Restore
the store-feature gate on DEFAULT_STORE_NAME, turbofish four Arc-to-dyn clones
for clone_on_ref_ptr, and expect too_many_lines where the policy-connector
setup lands.

Known-incomplete: the server threads store unconditionally (77 refs, 7 files)
but current main made the store feature optional, so this does not yet build
under --features standard. Needs a store-optionality decision before merge.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
Current main makes the store feature optional (a standard/FIPS build carries no
SQL crypto), but praxis-proxy#1284 threaded the store unconditionally through serve, reload,
watch, and pipeline resolution. Introduce a StoreRegistries alias (the real map
under store, a placeholder otherwise) so the serve and watch signatures stay
feature-free, gate the two resolver call sites and the store-config-change
detector, and route a no-store build through resolve_pipelines. The server now
builds under --features standard with no store or SQL crypto.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…SSRF)

Address the outstanding review on this PR:

- validate_config runs revalidate_postgres_host so a private/loopback DB host
  fails fatally at startup instead of passing construction and then looping in
  async provisioning (the praxis-bot SSRF finding).
- Permanent provisioning errors (unknown backend, config) stop retrying and
  leave readiness failed, rather than looping forever; transient errors still
  self-heal with backoff.
- provision_into deregisters the names it registered when a later reference
  fails, so a partial attempt does not leave entries that make a retry a
  duplicate (adds StoreRegistry::deregister).
- Store provisioning follows inline and named branch chains, so a store
  configured only inside a branch is provisioned, not left store-unavailable.
- A store-config change rejects the reload loudly (keeping the running pipeline
  and its provisioned backends) instead of swapping in one whose registry is
  stale or empty and 500s.
- Readiness reads a shared, reload-updated health registry rather than the
  startup snapshot, so it cannot report ready for degraded current clusters.

Also regate is_explicit_compact_request to openai-compact, its only remaining
caller after the registry-only filter migration removed the store-filter use.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
hexfusion added a commit to hexfusion/ai that referenced this pull request Sep 25, 2026
…proxy#1284

Import DEFAULT_PAGE_LIMIT from the ungated input_items path rather than the
openai-conversations-gated re-export, so the store-postgres unit tests build
without the conversations feature. Copy the store-lifecycle crate into both
Containerfiles so the FIPS ubi-image workspace resolves the member this PR
adds.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define persisted-state backend factories, configuration, and reload lifecycle

3 participants