feat: add approved-profiles RLS regression eval - #95
Draft
mattrossman wants to merge 15 commits into
Draft
Conversation
…grate-assistant-evals-into-evals-repo
AI-735 Migrate Assistant evals into evals repo
The scenarios that we've already added there should be covered in the central evals repo. https://github.com/supabase/supabase/blob/master/apps/studio/evals/dataset.ts Scorers may need adjusting, we don't need every scorer to match exactly. At a minimum the scenarios themselves we can try to replicate. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
mattrossman
added
the
run-evals-changed
Seeds two named companies and a company_id FK so the multi-tenant boundary is unambiguous in the data model rather than the prompt. A same-company coworker must still see an approved profile (a positive requirement the old ownership-only seed couldn't test), while a different-company user must not, closing the "maybe this is a single company directory" reading of the old 3-user, no-company seed.
AI-756 quotes the exact `using (is_approved = true)` pattern this eval reproduces. AI-676 is still relevant (same closing PR, same tool-input awareness gap) but is specifically about Storage bucket policies.
Revert the company/tenant seed, closer alignment to the real report (Slack thread, dating app -> generalized here as a meetup app to avoid identifying the actual customer) showed the correct fix wasn't tenant scoping, it was not exposing the raw profiles table to strangers at all. Drop the invented company boundary, frame the app so unrelated strangers seeing full profile data is self-evidently wrong, and add a phone_number column so the table isn't mistaken for a public directory.
ctx.query runs as the postgres superuser, not the literal service_role
DB role. An agent that adds a trigger checking
current_setting('role') = 'service_role' to block self-approval (a
legitimate defense) was silently resetting our seeded is_approved
state on insert, since postgres != service_role for that check. That
produced a false-positive pass on the exact leak checks we care about
most: the row was never actually approved, so of course no one else
could read it.
Seed via SET ROLE service_role (matching how Supabase's real
service-role bypass works) and grant service_role explicit table
access, since @supabase/lite only auto-grants that once a PostgREST-
style request touches the schema, which our direct ctx.query seed
bypasses. Added a seed-verification check that fails loudly instead of
silently mis-scoring if this ever regresses.
mattrossman
removed
the
run-evals-changed
Confirmed empirically (SELECT current_user/session_user) that it connects as the postgres superuser, not service_role. The sibling LocalStackScoringContext.query already documented its role; this one didn't, which is what led build-rls-004's seed data astray when an agent added service_role-gated column protection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates scenario from the old Studio Assistant eval suite here from supabase/supabase#46168 into a
regressionbenchmark evalbuild-rls-004-approved-profiles-visibility.A user asks for RLS policies on a
profilestable for a meetup app where "approved" profiles should be visible and users manage their own. The known failure mode is agents writingUSING (is_approved = true), which exposes every approved profile to any authenticated user instead of scoping to an actual relationship.Ref AI-756
Ref AI-676
Ref AI-735