From 4a45879255f73baf2cd106c8b55c5c98006258f0 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Tue, 28 Jul 2026 10:31:38 +0200 Subject: [PATCH 1/9] refactor(evals): narrow deleted-user-access to session revocation The scenario bundled an unrelated API-key rundown into the same prompt and the same judge verdict, so a fumbled key explanation failed the whole auth scenario with no way to attribute it. Moves to its own eval (AI-923). --- evals/investigate-auth-001-deleted-user-access/EVAL.ts | 5 ++--- evals/investigate-auth-001-deleted-user-access/PROMPT.md | 8 +------- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/evals/investigate-auth-001-deleted-user-access/EVAL.ts b/evals/investigate-auth-001-deleted-user-access/EVAL.ts index bbfd5735..bddbba76 100644 --- a/evals/investigate-auth-001-deleted-user-access/EVAL.ts +++ b/evals/investigate-auth-001-deleted-user-access/EVAL.ts @@ -207,15 +207,14 @@ async function checkRevocationDiagnosis( includeToolCallInputs: true, }), rubric: stripIndent` - Pass if the assistant diagnoses why the deleted account still works, fixes the delete-account flow with real revocation, explains any remaining access-token window consistently with the fix it implemented, and correctly clarifies publishable vs secret API keys. + Pass if the assistant diagnoses why the deleted account still works, fixes the delete-account flow with real revocation, and explains any remaining access-token window consistently with the fix it implemented. A passing answer should: - Identify that delete_account only soft-deletes the profile row and never removes the auth user or revokes their sessions, so the account keeps signing in. - Fix the flow so the auth user is deleted (or equivalently their identity and sessions are removed), revoking their sessions and refresh tokens. - Explain that access tokens are stateless JWTs that are not recalled by deletion or revocation, so anything that validates them only locally (such as getClaims() or custom middleware checking signature and expiry) keeps accepting them until they expire. Judge the window claim against the fix the assistant actually shipped: if its fix leaves the Data API accepting stale JWTs, it must say a window up to the token expiry remains and name a mitigation (short JWT expiry, server-side auth.getUser() checks, or RLS validating the user or session still exists); if its fix genuinely closes the Data API window (for example RLS policies that verify the auth user or session still exists), it may say the data path has no post-deletion window, but must still note the token itself stays valid until expiry for purely local validation. - - Explain that the publishable key belongs in frontend code where requests run under the signed-in user's JWT and RLS applies, while the secret key is server-only, bypasses RLS, and must never ship to the client. - Fail if the assistant leaves the flow as a soft delete without revoking auth access, claims existing access tokens are instantly invalidated everywhere the moment the user is deleted without any caveat, makes a window claim that contradicts its own implemented fix in either direction, recommends putting the secret key or legacy service_role key in frontend code, says RLS applies to the secret key, or blames the symptom on caching or client bugs instead of the flow. + Fail if the assistant leaves the flow as a soft delete without revoking auth access, claims existing access tokens are instantly invalidated everywhere the moment the user is deleted without any caveat, makes a window claim that contradicts its own implemented fix in either direction, or blames the symptom on caching or client bugs instead of the flow. `, }); diff --git a/evals/investigate-auth-001-deleted-user-access/PROMPT.md b/evals/investigate-auth-001-deleted-user-access/PROMPT.md index 7ea027cc..0bb158ef 100644 --- a/evals/investigate-auth-001-deleted-user-access/PROMPT.md +++ b/evals/investigate-auth-001-deleted-user-access/PROMPT.md @@ -6,8 +6,7 @@ product: - auth topic: - security - - sdk -motivation: AI-820, AI-422, https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V +motivation: AI-820, https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V --- Last week support removed a user through our app's delete-account flow — the @@ -18,8 +17,3 @@ data like nothing happened. Figure out why the account still works, fix the flow so a deleted account loses access, and tell me whether there is any window where they could still get in after the fix. - -One more thing while you're at it: we're migrating off the legacy -anon/service_role API keys to the new publishable and secret keys, and the -team isn't sure which one belongs in the frontend and what each means for -RLS. Give us a quick rundown. From 97eaa28a216b0a6520a1fa3fb7095d82bf1c6733 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Tue, 28 Jul 2026 10:31:38 +0200 Subject: [PATCH 2/9] feat(evals): add api-key migration scenario Splits the publishable/secret key rundown out of investigate-auth-001-deleted-user-access into a regression scenario, giving AI-422 a result to point at. Judge-only: nothing to inspect in project state for a knowledge question. --- .../EVAL.ts | 38 +++++++++++++++++++ .../PROMPT.md | 15 ++++++++ 2 files changed, 53 insertions(+) create mode 100644 evals/investigate-auth-002-api-key-migration/EVAL.ts create mode 100644 evals/investigate-auth-002-api-key-migration/PROMPT.md diff --git a/evals/investigate-auth-002-api-key-migration/EVAL.ts b/evals/investigate-auth-002-api-key-migration/EVAL.ts new file mode 100644 index 00000000..57063213 --- /dev/null +++ b/evals/investigate-auth-002-api-key-migration/EVAL.ts @@ -0,0 +1,38 @@ +import { + judge, + serializeTranscript, + type CheckResult, + type ToolScorer, +} from '@supabase-evals/core'; +import { stripIndent } from 'common-tags'; + +const scorer: ToolScorer = async (ctx) => { + const verdict = await judge({ + input: serializeTranscript(ctx.transcript), + rubric: stripIndent` + Pass if the assistant explains the legacy anon/service_role keys against the new publishable/secret keys, says which key belongs in frontend code, and says what each means for RLS. + + A passing answer should: + - Describe the legacy and new keys as role equivalents: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. + - Explain that the publishable key belongs in frontend code. It identifies the app rather than a user: requests carrying no user access token run as the anon role, and requests carrying one run as the authenticated role. RLS applies either way. + - Explain that a secret key is server-only, maps to service_role, bypasses RLS, and must never ship to the client. + + Fail if the assistant recommends putting a secret key or the legacy service_role key in frontend code, says RLS applies to the secret key, claims the publishable key bypasses RLS, says the publishable key itself authenticates a user or carries their identity, or inverts the legacy-to-new key mapping. + `, + }); + + const checks: CheckResult[] = [ + { + name: 'explained publishable vs secret API keys', + passed: verdict.passed, + judgeNotes: verdict.notes, + }, + ]; + + return { + passed: checks.every((check) => check.passed), + checks, + }; +}; + +export default scorer; diff --git a/evals/investigate-auth-002-api-key-migration/PROMPT.md b/evals/investigate-auth-002-api-key-migration/PROMPT.md new file mode 100644 index 00000000..cf048dfa --- /dev/null +++ b/evals/investigate-auth-002-api-key-migration/PROMPT.md @@ -0,0 +1,15 @@ +--- +stage: investigate +suite: regression +interface: mcp +product: + - auth +topic: + - security + - sdk +motivation: AI-422 +--- + +We're migrating off the legacy anon/service_role API keys to the new publishable +and secret keys. The team isn't sure which one belongs in the frontend and what +each means for RLS. Give us a quick rundown. From 5b930a9a6bd1a1a10e88c9f815f0346d45ea0bd6 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Tue, 28 Jul 2026 10:51:51 +0200 Subject: [PATCH 3/9] refactor(evals): score each key/revocation claim separately One judge verdict spanning several independent claims can't be attributed: an agent that gets two of three right scores identically to one that gets none. Splits both scorers into parallel judges with one named check each, matching investigate-functions-001. Costs one judge call per claim. Also drops interface: mcp from the api-key scenario, which needs no tools. --- .../EVAL.ts | 67 +++++++++++++------ .../EVAL.ts | 53 +++++++++++---- .../PROMPT.md | 1 - 3 files changed, 86 insertions(+), 35 deletions(-) diff --git a/evals/investigate-auth-001-deleted-user-access/EVAL.ts b/evals/investigate-auth-001-deleted-user-access/EVAL.ts index bddbba76..2d62f234 100644 --- a/evals/investigate-auth-001-deleted-user-access/EVAL.ts +++ b/evals/investigate-auth-001-deleted-user-access/EVAL.ts @@ -28,7 +28,7 @@ const scorer: ToolScorer = async (ctx) => { await checkRefreshTokenRejected(ctx, users), await checkCannotSignBackIn(ctx, users), await checkBystanderUnaffected(users), - await checkRevocationDiagnosis(ctx), + ...(await checkRevocationAnswer(ctx)), ]; return { @@ -199,28 +199,55 @@ async function checkBystanderUnaffected( }; } -async function checkRevocationDiagnosis( +async function checkRevocationAnswer( ctx: ToolEvalContext -): Promise { - const verdict = await judge({ - input: serializeTranscript(ctx.transcript, { - includeToolCallInputs: true, +): Promise { + const input = serializeTranscript(ctx.transcript, { + includeToolCallInputs: true, + }); + + const [causeFound, flowFixed, windowExplained] = await Promise.all([ + judge({ + input, + rubric: stripIndent` + Pass if the assistant identifies that delete_account only soft-deletes the profile row and never removes the auth user or revokes their sessions, so the account keeps signing in. + + Fail if the assistant blames the symptom on caching or client bugs instead of the flow, or never explains why the deleted account still works. + `, }), - rubric: stripIndent` - Pass if the assistant diagnoses why the deleted account still works, fixes the delete-account flow with real revocation, and explains any remaining access-token window consistently with the fix it implemented. + judge({ + input, + rubric: stripIndent` + Pass if the assistant fixes the flow so the auth user is deleted (or equivalently their identity and sessions are removed), revoking their sessions and refresh tokens. - A passing answer should: - - Identify that delete_account only soft-deletes the profile row and never removes the auth user or revokes their sessions, so the account keeps signing in. - - Fix the flow so the auth user is deleted (or equivalently their identity and sessions are removed), revoking their sessions and refresh tokens. - - Explain that access tokens are stateless JWTs that are not recalled by deletion or revocation, so anything that validates them only locally (such as getClaims() or custom middleware checking signature and expiry) keeps accepting them until they expire. Judge the window claim against the fix the assistant actually shipped: if its fix leaves the Data API accepting stale JWTs, it must say a window up to the token expiry remains and name a mitigation (short JWT expiry, server-side auth.getUser() checks, or RLS validating the user or session still exists); if its fix genuinely closes the Data API window (for example RLS policies that verify the auth user or session still exists), it may say the data path has no post-deletion window, but must still note the token itself stays valid until expiry for purely local validation. + Fail if the assistant leaves the flow as a soft delete without revoking auth access. + `, + }), + judge({ + input, + rubric: stripIndent` + Pass if the assistant explains that access tokens are stateless JWTs that are not recalled by deletion or revocation, so anything that validates them only locally (such as getClaims() or custom middleware checking signature and expiry) keeps accepting them until they expire. Judge the window claim against the fix the assistant actually shipped: if its fix leaves the Data API accepting stale JWTs, it must say a window up to the token expiry remains and name a mitigation (short JWT expiry, server-side auth.getUser() checks, or RLS validating the user or session still exists); if its fix genuinely closes the Data API window (for example RLS policies that verify the auth user or session still exists), it may say the data path has no post-deletion window, but must still note the token itself stays valid until expiry for purely local validation. - Fail if the assistant leaves the flow as a soft delete without revoking auth access, claims existing access tokens are instantly invalidated everywhere the moment the user is deleted without any caveat, makes a window claim that contradicts its own implemented fix in either direction, or blames the symptom on caching or client bugs instead of the flow. - `, - }); + Fail if the assistant claims existing access tokens are instantly invalidated everywhere the moment the user is deleted without any caveat, or makes a window claim that contradicts its own implemented fix in either direction. + `, + }), + ]); - return { - name: 'diagnosed and explained session revocation', - passed: verdict.passed, - judgeNotes: verdict.notes, - }; + return [ + { + name: 'diagnosed the soft-delete as the cause', + passed: causeFound.passed, + judgeNotes: causeFound.notes, + }, + { + name: 'fixed the flow to revoke auth access', + passed: flowFixed.passed, + judgeNotes: flowFixed.notes, + }, + { + name: 'explained the access-token window consistently with its fix', + passed: windowExplained.passed, + judgeNotes: windowExplained.notes, + }, + ]; } diff --git a/evals/investigate-auth-002-api-key-migration/EVAL.ts b/evals/investigate-auth-002-api-key-migration/EVAL.ts index 57063213..6c73f69e 100644 --- a/evals/investigate-auth-002-api-key-migration/EVAL.ts +++ b/evals/investigate-auth-002-api-key-migration/EVAL.ts @@ -7,25 +7,50 @@ import { import { stripIndent } from 'common-tags'; const scorer: ToolScorer = async (ctx) => { - const verdict = await judge({ - input: serializeTranscript(ctx.transcript), - rubric: stripIndent` - Pass if the assistant explains the legacy anon/service_role keys against the new publishable/secret keys, says which key belongs in frontend code, and says what each means for RLS. + const input = serializeTranscript(ctx.transcript); - A passing answer should: - - Describe the legacy and new keys as role equivalents: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. - - Explain that the publishable key belongs in frontend code. It identifies the app rather than a user: requests carrying no user access token run as the anon role, and requests carrying one run as the authenticated role. RLS applies either way. - - Explain that a secret key is server-only, maps to service_role, bypasses RLS, and must never ship to the client. + const [mapping, frontendKey, serverKey] = await Promise.all([ + judge({ + input, + rubric: stripIndent` + Pass if the assistant maps the legacy keys onto the new ones as role equivalents: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. - Fail if the assistant recommends putting a secret key or the legacy service_role key in frontend code, says RLS applies to the secret key, claims the publishable key bypasses RLS, says the publishable key itself authenticates a user or carries their identity, or inverts the legacy-to-new key mapping. - `, - }); + Fail if the assistant inverts that mapping, or presents the new keys as exact drop-in renames of the legacy ones rather than role equivalents. + `, + }), + judge({ + input, + rubric: stripIndent` + Pass if the assistant puts the publishable key in frontend code and describes it as identifying the app rather than a user: requests carrying no user access token run as the anon role, requests carrying one run as the authenticated role, and RLS applies either way. + + Fail if the assistant says the publishable key bypasses RLS, or says the key itself authenticates a user or carries their identity. + `, + }), + judge({ + input, + rubric: stripIndent` + Pass if the assistant keeps a secret key server-only, maps it to service_role, and says it bypasses RLS and must never ship to the client. + + Fail if the assistant recommends putting a secret key or the legacy service_role key in frontend code, or says RLS applies to the secret key. + `, + }), + ]); const checks: CheckResult[] = [ { - name: 'explained publishable vs secret API keys', - passed: verdict.passed, - judgeNotes: verdict.notes, + name: 'mapped legacy anon/service_role onto publishable/secret', + passed: mapping.passed, + judgeNotes: mapping.notes, + }, + { + name: 'placed the publishable key in the frontend with RLS still applying', + passed: frontendKey.passed, + judgeNotes: frontendKey.notes, + }, + { + name: 'kept the secret key server-only and RLS-bypassing', + passed: serverKey.passed, + judgeNotes: serverKey.notes, }, ]; diff --git a/evals/investigate-auth-002-api-key-migration/PROMPT.md b/evals/investigate-auth-002-api-key-migration/PROMPT.md index cf048dfa..5d420e95 100644 --- a/evals/investigate-auth-002-api-key-migration/PROMPT.md +++ b/evals/investigate-auth-002-api-key-migration/PROMPT.md @@ -1,7 +1,6 @@ --- stage: investigate suite: regression -interface: mcp product: - auth topic: From 026bc14c11e1ed01c58795b7c8dd19ee889c16a5 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 28 Jul 2026 09:25:36 +0000 Subject: [PATCH 4/9] chore: refresh eval results --- apps/web/src/data/eval-results.json | 893 +++++------------- .../web/src/data/regression-eval-results.json | 148 +++ 2 files changed, 395 insertions(+), 646 deletions(-) diff --git a/apps/web/src/data/eval-results.json b/apps/web/src/data/eval-results.json index 8a16cc92..6b7c554d 100644 --- a/apps/web/src/data/eval-results.json +++ b/apps/web/src/data/eval-results.json @@ -1037,12 +1037,11 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -1063,17 +1062,26 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Identifies soft-delete-only cause and lack of session revocation; implements meaningful revocation by banning auth user and deleting sessions/refresh tokens plus RLS enforcement of deleted flag. Explains JWTs remain valid until expiry for local validation, while DB path is closed immediately due to RLS checking active account, which is consistent with the implemented fix. Correctly distinguishes publishable/frontend/RLS-enforced keys from secret/server-only/RLS-bypassing keys." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only set profiles.deleted=true (soft delete), no RLS checked it, and it did not delete auth.users or revoke sessions/refresh tokens, so the user stayed signed in/refreshed tokens. It explains why the deleted account still worked and proposes a flow-level fix, not caching/client bugs." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant changed the flow from soft-delete to deleting auth.users, cascading/removing sessions, refresh tokens, identities, and profile, plus explicitly deleting notes and adding RLS gating to close the live JWT window. This satisfies revoking auth access rather than leaving a cosmetic soft delete." + }, + { + "name": "explained the access-token window consistently with its fix", + "passed": true, + "judgeNotes": "The assistant correctly explains that issued access tokens remain cryptographically valid until expiry and that local/JWT-only validation would still accept them. Its shipped fix includes RLS gating via account_active(), so its claim of no post-deletion data-path window for the protected tables is consistent, while still caveating that new/local paths need mitigation and naming short JWT expiry/session checks." } ], "skills": { @@ -1088,9 +1096,9 @@ "docs": { "calls": [] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "claude-code-opus-4.8/investigate-auth-001-deleted-user-access.json" }, { @@ -2363,12 +2371,11 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -2389,16 +2396,27 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Meets all rubric requirements: identifies soft-delete-only bug, implements real auth user deletion/session revocation, correctly explains stateless JWT residual window and aligns it with added RLS live-profile mitigation while caveating local validation, and accurately distinguishes publishable vs secret keys including RLS behavior and frontend/server placement." + "judgeNotes": "The assistant clearly identifies that delete_account only soft-deletes the profile row, does not remove/disable the auth user or revoke sessions/refresh tokens, and explains that this is why the account can keep signing in/accessing data. It does not blame caching or client bugs." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant rewrote the flow to revoke auth-layer access by banning/marking deleted in auth.users and deleting auth.sessions and auth.refresh_tokens, plus enforced the deleted flag in RLS. This satisfies removal of identity/sessions or equivalent revocation." + }, + { + "name": "explained the access-token window consistently with its fix", + "passed": true, + "judgeNotes": "The assistant correctly explains stateless JWTs remain cryptographically valid until expiry and that local-only validation would keep accepting them. Its implemented fix adds RLS checks against the deleted/account_active state, so its claim of no meaningful Data API window is consistent, while still noting an auth/JWT expiry window remains for custom local validation and suggesting lower JWT expiry." } ], "skills": { @@ -2406,40 +2424,11 @@ "loaded": [] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "{ searchDocs(query: \"publishable key secret key API keys migration anon service_role RLS\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", - "title": "Build a Social Auth App with Expo React Native" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" - } - ], - "resultChars": 169611 - } - ] + "calls": [] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-opus-4.8-no-skills/investigate-auth-001-deleted-user-access.json" }, { @@ -3908,8 +3897,7 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", @@ -3941,9 +3929,19 @@ "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant explicitly identifies that delete_account only updated a profiles deleted flag, did not remove auth.users, sessions, refresh tokens, identities, or enforce the flag in RLS, explaining why the deleted account could keep using an existing session or sign in again." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant replaced the soft-delete flow with deletion from auth.users, explicitly noting cascades to identities, sessions, and refresh tokens, and added session-active RLS checks to cut off existing JWT access. This satisfies revoking auth access rather than leaving only a soft delete." + }, + { + "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Meets all rubric requirements: correctly diagnoses soft-delete-only flow, implements hard auth user deletion/session revocation, accurately explains residual JWT access window consistent with the implemented fix and names mitigations, and correctly distinguishes publishable vs secret keys and RLS behavior." + "judgeNotes": "Pass: The assistant explicitly explains that already-issued Supabase access tokens remain cryptographically valid until expiry and that local validation would still accept them. Its shipped fix adds RLS checking auth.sessions via session_id, which genuinely closes the Data API/RLS window after deletion, and its window claim is consistent with that fix while still noting caveats like service-role bypass and in-flight requests." } ], "skills": { @@ -3956,11 +3954,39 @@ ] }, "docs": { - "calls": [] + "calls": [ + { + "source": "search_docs", + "query": "{ searchDocs(query: \"allow users to delete their own account SQL function auth.users\", limit: 5) { nodes { title href content } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/auth-anonymous", + "title": "Anonymous Sign-Ins" + }, + { + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", + "title": "Build a User Management App with SvelteKit" + }, + { + "url": "https://supabase.com/docs/guides/database/extensions/pgaudit", + "title": "PGAudit: Postgres Auditing" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + } + ], + "resultChars": 74244 + } + ] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "claude-code-sonnet-5/investigate-auth-001-deleted-user-access.json" }, { @@ -5199,8 +5225,7 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", @@ -5233,9 +5258,19 @@ "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant clearly identifies the root cause: delete_account only set profiles.deleted, did not touch auth.users, did not ban/remove the auth user, and did not revoke sessions/refresh tokens, so existing sessions and future sign-ins continued to work." + }, + { + "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The answer diagnoses the soft-delete-only bug, implements real session/refresh-token revocation and blocks future sign-in, explains JWT expiry and the remaining/local-validation window consistently with its RLS mitigation, and correctly states publishable keys are client-safe while secret/service-role keys are server-only and bypass RLS. There is one slightly contradictory closing phrase about RLS being the enforcement layer regardless of key, but the surrounding explanation clearly says secret keys bypass RLS." + "judgeNotes": "The assistant updates delete_account to ban the auth user and delete auth.refresh_tokens and auth.sessions, plus adds RLS checks for deleted users. This fixes the soft-delete-only flow by revoking auth access and refresh/session capability." + }, + { + "name": "explained the access-token window consistently with its fix", + "passed": true, + "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain cryptographically valid until expiry and are not invalidated merely by banning/deleting sessions. Its shipped fix adds RLS checks against the deleted flag, so its claim of no practical Data API window is consistent, while still noting local/JWT validation would accept the token until exp." } ], "skills": { @@ -5245,7 +5280,7 @@ "docs": { "calls": [] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", "attempts": 2, "sourcePath": "claude-code-sonnet-5-no-skills/investigate-auth-001-deleted-user-access.json" @@ -9006,16 +9041,16 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", - "passed": true + "passed": false, + "notes": "permission denied for table users" }, { "name": "delete_account flow ran for the victim", @@ -9036,12 +9071,23 @@ }, { "name": "other users keep their sessions and access", - "passed": true + "passed": false, + "notes": "permission denied for table users" }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Meets rubric: identifies soft-delete-only delete_account as cause, changes flow to delete auth.users and closes RLS data path for stale JWTs, explains JWT expiry caveat consistently, and correctly distinguishes publishable vs secret keys." + "judgeNotes": "Identified that delete_account only soft-deletes public.profiles and leaves auth.users/sessions intact, explaining why stale auth/JWT access continues." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant changed the flow from a soft delete to deleting the auth.users row, which cascades to sessions/identities/refresh tokens, and added RLS checks requiring an active auth user to block stale JWT access." + }, + { + "name": "explained the access-token window consistently with its fix", + "passed": false, + "judgeNotes": "The fix/window claim is consistent with the shipped RLS/auth.users check, but the assistant did not explicitly explain that stateless JWT access tokens remain accepted until expiry by purely local validation such as getClaims() or custom signature/expiry middleware." } ], "skills": { @@ -9050,33 +9096,15 @@ "supabase-postgres-best-practices" ], "loaded": [ - "supabase" + "supabase", + "supabase-postgres-best-practices" ] }, "docs": { "calls": [ - { - "source": "web_search", - "query": "https://supabase.com/changelog.md", - "pages": [ - { - "url": "https://supabase.com/changelog.md" - } - ] - }, - { - "source": "web_search", - "query": "site:supabase.com/changelog.md supabase changelog md", - "pages": [] - }, - { - "source": "web_search", - "query": "Supabase changelog breaking change auth delete user sessions publishable secret key", - "pages": [] - }, { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user revoke sessions auth.admin.deleteUser auth.sessions\", limit: 5) { nodes { __typename title href content ... on Guide { subsections { nodes { title href content } } } } } }", + "query": "query { searchDocs(query: \"delete user sessions auth.admin.deleteUser invalidate existing access tokens\", limit: 5) { nodes { __typename ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { @@ -9084,221 +9112,54 @@ "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data#accessing-user-data-via-api", - "title": "Accessing user data via API" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data#adding-and-retrieving-user-metadata", - "title": "Adding and retrieving user metadata" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data#exporting-users", - "title": "Exporting users" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data#deleting-users", - "title": "Deleting users" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" - }, - { - "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" }, { - "url": "https://supabase.com/docs/reference/swift/auth-admin-deleteuser", - "title": "deleteUser()" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { "url": "https://supabase.com/docs/guides/resources/glossary", "title": "Glossary" }, { - "url": "https://supabase.com/docs/guides/resources/glossary#row-level-security-policies-rls", - "title": "Row level security policies (RLS)" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#access-token", - "title": "Access token" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#authentication", - "title": "Authentication" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#authenticator-app", - "title": "Authenticator app" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#authorization", - "title": "Authorization" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#identity-provider", - "title": "Identity provider" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#json-web-token-jwt", - "title": "JSON Web Token (JWT)" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#jwt-signing-secret", - "title": "JWT signing secret" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#multi-factor-authentication-mfa-or-2fa", - "title": "Multi-factor authentication (MFA or 2FA)" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#nonce", - "title": "Nonce" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#oauth", - "title": "OAuth" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#oidc", - "title": "OIDC" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#one-time-password-otp", - "title": "One-time password (OTP)" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#password-hashing-function", - "title": "Password hashing function" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#password-strength", - "title": "Password strength" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#pkce", - "title": "PKCE" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#provider-refresh-token", - "title": "Provider refresh token" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#provider-tokens", - "title": "Provider tokens" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#refresh-token", - "title": "Refresh token" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#refresh-token-flow", - "title": "Refresh token flow" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#replay-attack", - "title": "Replay attack" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#saml", - "title": "SAML" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#session", - "title": "Session" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#single-sign-on-sso", - "title": "Single-sign on (SSO)" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary#time-based-one-time-password-totp", - "title": "Time-based one-time password (TOTP)" - } - ], - "resultChars": 35929 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"migrating to new API keys publishable secret frontend RLS\", limit: 5) { nodes { __typename title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" - }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/getting-started", - "title": "Getting Started with OAuth 2.1 Server" - }, - { - "url": "https://supabase.com/docs/guides/database/secure-data", - "title": "Securing your data" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" } ], - "resultChars": 109403 + "resultChars": 47942 }, { "source": "search_docs", - "query": "query { searchDocs(query: \"revoke sessions delete user auth.sessions sign out current user delete account\", limit: 10) { nodes { __typename title href content } } }", + "query": "query { searchDocs(query: \"auth.sessions delete user revoke sessions auth.admin.deleteUser delete user does not automatically sign out\", limit: 5) { nodes { __typename ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } } } }", "hasContent": true, "pages": [ { "url": "https://supabase.com/docs/guides/auth/managing-user-data", "title": "User Management" }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" - }, { "url": "https://supabase.com/docs/reference/javascript/auth-signout" }, { - "url": "https://supabase.com/docs/guides/auth/signout", - "title": "Signing out" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" }, { "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, { - "url": "https://supabase.com/docs/reference/dart/auth-signout" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary", - "title": "Glossary" - }, - { - "url": "https://supabase.com/docs/reference/swift/auth-signout", - "title": "signOut()" - }, - { - "url": "https://supabase.com/docs/guides/auth/auth-anonymous", - "title": "Anonymous Sign-Ins" - }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" + "url": "https://supabase.com/docs/guides/auth/signout", + "title": "Signing out" } ], - "resultChars": 84801 - }, - { - "source": "web_search", - "query": "site:supabase.com/docs deleting a user does not automatically sign out a user JWT remain valid until it has expired Supabase", - "pages": [] + "resultChars": 24999 } ] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "codex-gpt-5.4-mini/investigate-auth-001-deleted-user-access.json" }, { @@ -11644,12 +11505,11 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -11670,17 +11530,26 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", "passed": true }, { - "name": "diagnosed and explained session revocation", - "passed": false, - "judgeNotes": "Fails because it does not explain that existing access tokens are stateless JWTs that remain cryptographically valid until expiry for purely local validation such as getClaims() or custom JWT middleware. It instead says the only remaining race is an in-flight request, which omits the required caveat. The diagnosis, RLS/data-path fix, session/refresh-token revocation, and publishable vs secret key explanation are otherwise largely correct." + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant clearly identifies the root cause: delete_account only soft-deleted public.profiles.deleted and did not remove the auth user or revoke sessions/JWTs, so RLS based on auth.uid() continued to allow access. It explains why the deleted account still works and does not blame caching/client bugs." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant changed delete_account to delete from auth.users, which cascades to sessions/identities/profile per the discovered FKs, and added active-account checks in RLS. This satisfies removing auth identity/sessions and revoking access rather than leaving only a soft delete." + }, + { + "name": "explained the access-token window consistently with its fix", + "passed": true, + "judgeNotes": "The assistant shipped RLS policies that check the auth user still exists via account_is_active(), so its claim that the data path has no post-commit backend access window is consistent with the fix. It explains the prior stale JWT/session issue, though it does not explicitly note that the JWT itself remains valid until expiry for purely local validation." } ], "skills": { @@ -11688,38 +11557,9 @@ "loaded": [] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "query {\n searchDocs(query: \"publishable secret keys RLS anon service_role\", limit: 5) {\n nodes {\n __typename\n ... on Guide { title href content }\n ... on TroubleshootingGuide { title href content }\n ... on ClientLibraryFunctionReference { title href methodName language content }\n ... on ManagementApiReference { title href content }\n ... on CLICommandReference { title href content }\n }\n }\n}", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/server-side/creating-a-client", - "title": "Creating a Supabase client for SSR" - }, - { - "url": "https://supabase.com/docs/guides/functions/secrets", - "title": "Environment Variables" - } - ], - "resultChars": 112269 - } - ] + "calls": [] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", "attempts": 2, "sourcePath": "codex-gpt-5.4-mini-no-skills/investigate-auth-001-deleted-user-access.json" @@ -14289,12 +14129,11 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -14315,17 +14154,26 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only sets profiles.deleted=true, leaves the Auth user and sessions intact, and RLS ignores the deleted flag, allowing continued sign-in/access. It does not blame caching/client bugs, though it notes client cached session only as secondary behavior." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant replaced the soft-delete-only flow with an RPC that deletes the auth.users row, causing sessions to be removed, and added RLS protections for immediate access denial. Verification showed auth user and session removed and same unexpired token denied data access." + }, + { + "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The answer identifies the soft-delete-only root cause, implements real revocation by banning the auth user and deleting sessions, and also adds RLS active-user checks that consistently justify no post-deletion Data API window for protected tables while caveating JWTs remain valid for local validation/unprotected paths until exp. It correctly distinguishes publishable vs secret keys and says secret bypasses RLS and is server-only." + "judgeNotes": "The assistant correctly explains that Supabase access tokens/JWTs remain cryptographically valid until exp after user/session deletion. Its shipped fix updates RLS to require live non-deleted profile state and deletes auth.users/sessions, then it claims new database requests lose access immediately, with only already-executing snapshot caveat. This is consistent with its RLS-based mitigation closing the Data API stale-JWT window while noting purely local token validity remains." } ], "skills": { @@ -14341,43 +14189,42 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user existing access token JWT session revoke auth.sessions delete_account publishable secret keys RLS\", limit: 8) { nodes { title href content } } }", + "query": "query { searchDocs(query: \"delete user existing access token sessions revoke auth.sessions sign out scope global JWT remains valid\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", - "title": "Token Security and Row Level Security" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/auth/third-party/clerk", - "title": "Clerk" + "url": "https://supabase.com/docs/guides/auth/signout", + "title": "Signing out" }, { - "url": "https://supabase.com/docs/guides/storage/security/ownership", - "title": "Ownership" + "url": "https://supabase.com/docs/reference/javascript/auth-signout" }, { - "url": "https://supabase.com/docs/guides/storage/buckets/fundamentals", - "title": "Storage Buckets" + "url": "https://supabase.com/docs/guides/auth/signing-keys", + "title": "JWT Signing Keys" }, { - "url": "https://supabase.com/docs/guides/storage/s3/authentication", - "title": "S3 Authentication" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/storage/security/access-control", - "title": "Storage Access Control" + "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", + "title": "Should I set a shorter Max-Age parameter on the cookies?" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/getting-started", - "title": "Getting Started with OAuth 2.1 Server" + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" }, { - "url": "https://supabase.com/docs/guides/storage/management/delete-objects", - "title": "Delete Objects" + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" } ], - "resultChars": 69441 + "resultChars": 107162 }, { "source": "web_search", @@ -14390,140 +14237,12 @@ }, { "source": "web_search", - "query": "site:supabase.com/changelog Supabase changelog auth session delete user API keys publishable secret", + "query": "site:supabase.com changelog auth session delete user revoke sessions Supabase", "pages": [] - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"Supabase publishable key secret key legacy anon service_role RLS bypass behavior\", limit: 10) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/database/postgres/row-level-security", - "title": "Row Level Security" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" - }, - { - "url": "https://supabase.com/docs/guides/troubleshooting/why-is-my-service-role-key-client-getting-rls-errors-or-not-returning-data-7_1K9z", - "title": "Why is my service role key client getting RLS errors or not returning data?" - }, - { - "url": "https://supabase.com/docs/guides/database/secure-data", - "title": "Securing your data" - }, - { - "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-auth-keys", - "title": "New API Keys and Asymmetric Authentication" - }, - { - "url": "https://supabase.com/docs/guides/storage/security/access-control", - "title": "Storage Access Control" - }, - { - "url": "https://supabase.com/docs/guides/api/securing-your-api", - "title": "Securing your API" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" - } - ], - "resultChars": 147735 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"Understanding API keys publishable key secret key sb_publishable sb_secret service_role bypass RLS\", limit: 6) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" - }, - { - "url": "https://supabase.com/docs/guides/database/postgres/row-level-security", - "title": "Row Level Security" - }, - { - "url": "https://supabase.com/docs/guides/storage/security/access-control", - "title": "Storage Access Control" - }, - { - "url": "https://supabase.com/docs/guides/troubleshooting/why-is-my-service-role-key-client-getting-rls-errors-or-not-returning-data-7_1K9z", - "title": "Why is my service role key client getting RLS errors or not returning data?" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/database/secure-data", - "title": "Securing your data" - } - ], - "resultChars": 64041 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"sign out revoke sessions access token remains valid until expiry Supabase Auth delete user\", limit: 10) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/guides/auth/signout", - "title": "Signing out" - }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" - }, - { - "url": "https://supabase.com/docs/guides/platform/sso", - "title": "Enable SSO for Your Organization" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwts", - "title": "JSON Web Token (JWT)" - }, - { - "url": "https://supabase.com/docs/guides/auth/users", - "title": "Users" - }, - { - "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", - "title": "Build a Supabase Integration" - }, - { - "url": "https://supabase.com/docs/guides/platform/temporary-access", - "title": "Temporary access" - } - ], - "resultChars": 143995 } ] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", "attempts": 2, "sourcePath": "codex-gpt-5.6/investigate-auth-001-deleted-user-access.json" @@ -17204,8 +16923,7 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", @@ -17237,9 +16955,19 @@ "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant explicitly identifies that delete_account only set profiles.deleted=true, did not remove the Auth user/sessions/JWT, and explains that auth.uid()-based RLS allowed continued access/sign-in. It does not blame caching or client bugs." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant changed delete_account() to delete from auth.users, which cascades to sessions/identities/profile per FK constraints, and added RLS hardening. This fixes the soft-delete-only flow and removes auth access/refresh sessions." + }, + { + "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The answer identifies the soft-delete-only root cause, implements real auth user deletion with cascading sessions/identities and RLS active-user checks, and explains the remaining JWT caveat consistently: Data API paths covered by its RLS check are blocked after commit, but purely local validation or other services/policies may accept the JWT until expiry. It also correctly distinguishes publishable frontend/RLS use from secret server-only RLS-bypassing use." + "judgeNotes": "The assistant correctly explains JWTs remain cryptographically valid until expiration and that local/auth.uid-only checks can still accept them. Its shipped fix adds RLS checks against auth.users/profiles via is_active_account(), so the claim of no post-commit data window for protected profile/note data is consistent, while caveating future auth.uid-only paths and in-flight requests." } ], "skills": { @@ -17250,135 +16978,46 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user JWT remains valid until expiry revoke sessions access token RLS\", limit: 5) { nodes { ... on Guide { title href content } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user access token remains valid until expiry sign out sessions JWT RLS\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", - "title": "Token Security and Row Level Security" - }, - { - "url": "https://supabase.com/docs/guides/realtime/authorization", - "title": "Realtime Authorization" - }, - { - "url": "https://supabase.com/docs/guides/storage/security/ownership", - "title": "Ownership" - }, { "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", "title": "Should I set a shorter Max-Age parameter on the cookies?" - } - ], - "resultChars": 40980 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"publishable key secret key anon service_role RLS new API keys\", limit: 6) { nodes { ... on Guide { title href content } ... on TroubleshootingGuide { title href content } ... on ManagementApiReference { title href content } } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/server-side/creating-a-client", - "title": "Creating a Supabase client for SSR" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" }, { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" - }, - { - "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", - "title": "Build a Social Auth App with Expo React Native" - } - ], - "resultChars": 197013 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"API keys publishable secret key bypass RLS Authorization header apikey\", limit: 10) { nodes { ... on Guide { title href content } } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" - }, - { - "url": "https://supabase.com/docs/guides/functions/auth", - "title": "Securing Edge Functions" - }, - { - "url": "https://supabase.com/docs/guides/storage/security/access-control", - "title": "Storage Access Control" - }, - { - "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-auth-keys", - "title": "New API Keys and Asymmetric Authentication" - }, - { - "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-envoy", - "title": "Envoy API Gateway" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/database/postgres/row-level-security", - "title": "Row Level Security" + "url": "https://supabase.com/docs/reference/javascript/auth-signout" }, { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-kotlin", - "title": "Build a Product Management Android App with Jetpack Compose" + "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", + "title": "Token Security and Row Level Security" }, { - "url": "https://supabase.com/docs/guides/storage/debugging/error-codes", - "title": "Error Codes" - } - ], - "resultChars": 174374 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"secret key RLS user Authorization JWT publishable key authenticated role\", limit: 5) { nodes { ... on Guide { title href content } } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" + "url": "https://supabase.com/docs/guides/auth/signing-keys", + "title": "JWT Signing Keys" }, { - "url": "https://supabase.com/docs/guides/functions/auth", - "title": "Securing Edge Functions" + "url": "https://supabase.com/docs/guides/auth/server-side/advanced-guide", + "title": "Advanced guide" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/getting-started", - "title": "Getting Started with OAuth 2.1 Server" + "url": "https://supabase.com/docs/guides/realtime/authorization", + "title": "Realtime Authorization" }, { - "url": "https://supabase.com/docs/guides/database/secure-data", - "title": "Securing your data" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" } ], - "resultChars": 56289 + "resultChars": 90545 } ] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", "attempts": 1, "sourcePath": "codex-gpt-5.6-no-skills/investigate-auth-001-deleted-user-access.json" @@ -18891,8 +18530,7 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", @@ -18924,9 +18562,19 @@ "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant explicitly identifies that delete_account only updated profiles.deleted, leaving auth.users, credentials, sessions, and refresh tokens intact, and explains that RLS did not enforce the deleted flag so the account kept working." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant rewrote the delete flow to delete the auth.users row, explicitly remove app data, and rely on cascading FKs to remove identities, sessions, refresh tokens, and profile. It also revoked public/anon execute access. This satisfies revoking auth access rather than leaving only a soft delete." + }, + { + "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Meets all rubric points: identifies soft-delete-only root cause; implements auth user deletion with cascaded sessions/refresh token revocation; explains stateless JWT access-token window consistently and gives mitigations; correctly distinguishes publishable frontend/RLS behavior from secret backend-only/RLS-bypass behavior." + "judgeNotes": "Pass. The assistant explicitly states access tokens are stateless JWTs valid until exp and deletion does not stop purely local validation. Its implemented fix deletes auth.users and cascades sessions/refresh tokens but does not change RLS to verify user/session existence; it correctly notes a residual Data API window and even confirms stale JWT claims can still INSERT. It also names mitigations: lower JWT expiry and validate session_id against auth.sessions." } ], "skills": { @@ -18942,35 +18590,34 @@ "calls": [ { "source": "search_docs", - "query": "{ searchDocs(query: \"publishable secret API keys migration anon service_role RLS\", limit: 5) { nodes { title href content } } }", + "query": "{ searchDocs(query: \"delete user account security definer function auth.users\", limit: 5) { nodes { title href content } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", + "title": "Build a User Management App with SvelteKit" }, { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", - "title": "Build a Social Auth App with Expo React Native" + "url": "https://supabase.com/docs/guides/database/extensions/pgaudit", + "title": "PGAudit: Postgres Auditing" }, { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" }, { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" + "url": "https://supabase.com/docs/guides/functions/examples/push-notifications", + "title": "Sending Push Notifications" } ], - "resultChars": 169611 + "resultChars": 72582 } ] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", "attempts": 1, "sourcePath": "opencode-kimi-k3/investigate-auth-001-deleted-user-access.json" @@ -20544,8 +20191,7 @@ "auth" ], "topic": [ - "security", - "sdk" + "security" ], "suite": "benchmark", "interface": "mcp", @@ -20577,9 +20223,19 @@ "passed": true }, { - "name": "diagnosed and explained session revocation", + "name": "diagnosed the soft-delete as the cause", + "passed": true, + "judgeNotes": "The assistant correctly identifies that delete_account only soft-deleted the public.profiles row while leaving auth.users, sessions, and refresh tokens intact, allowing continued sign-in/token refresh. It also explains the RLS/JWT aspect and does not blame caching or client bugs." + }, + { + "name": "fixed the flow to revoke auth access", + "passed": true, + "judgeNotes": "The assistant updated delete_account() to delete the auth.users row, which cascades to sessions, refresh tokens, identities, and profile, and restricted execution. This revokes auth access rather than leaving a soft delete." + }, + { + "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Meets the rubric: correctly diagnoses soft-delete without auth/session revocation, implements auth user deletion with refresh/session revocation, explains the remaining stateless JWT expiry window consistently, and correctly distinguishes publishable frontend keys from secret server-only RLS-bypassing keys." + "judgeNotes": "The assistant correctly explains stateless JWT access tokens remain cryptographically valid until expiry and that purely local validation can still accept them. Its shipped fix adds RLS checks tied to the profile/auth deletion path, and its window claim is consistent: no data-path window for Supabase/RLS after deletion, but a residual window up to JWT expiry for external/local JWT validation." } ], "skills": { @@ -20587,64 +20243,9 @@ "loaded": [] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "query { searchDocs(query: \"delete user account RPC security definer function auth.users\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" - }, - { - "url": "https://supabase.com/docs/reference/swift/auth-admin-deleteuser", - "title": "deleteUser()" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" - }, - { - "url": "https://supabase.com/docs/guides/database/database-advisors?queryGroups=lint&lint=0029_authenticated_security_definer_function_executable", - "title": "Database Advisor: Lint 0029_authenticated_security_definer_function_executable" - } - ], - "resultChars": 15474 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"publishable secret API keys replacing anon service_role\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/api-keys", - "title": "Understanding API keys" - }, - { - "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-auth-keys", - "title": "New API Keys and Asymmetric Authentication" - }, - { - "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-envoy", - "title": "Envoy API Gateway" - } - ], - "resultChars": 98570 - } - ] + "calls": [] }, - "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.\n\nOne more thing while you're at it: we're migrating off the legacy\nanon/service_role API keys to the new publishable and secret keys, and the\nteam isn't sure which one belongs in the frontend and what each means for\nRLS. Give us a quick rundown.", + "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", "attempts": 1, "sourcePath": "opencode-kimi-k3-no-skills/investigate-auth-001-deleted-user-access.json" diff --git a/apps/web/src/data/regression-eval-results.json b/apps/web/src/data/regression-eval-results.json index 5ac81bd2..c9ffbd35 100644 --- a/apps/web/src/data/regression-eval-results.json +++ b/apps/web/src/data/regression-eval-results.json @@ -269,6 +269,105 @@ "attempts": 1, "sourcePath": "claude-code-sonnet-5/build-realtime-001-live-chat-updates.json" }, + { + "experiment": "claude-code-sonnet-5", + "experimentSuite": "regression", + "experimentDisplay": { + "agent": "claude-code", + "modelProvider": "anthropic", + "modelId": "claude-sonnet-5", + "reasoningEffort": "high" + }, + "eval": "investigate-auth-002-api-key-migration", + "stage": "investigate", + "product": [ + "auth" + ], + "topic": [ + "security", + "sdk" + ], + "suite": "regression", + "passed": false, + "checks": [ + { + "name": "mapped legacy anon/service_role onto publishable/secret", + "passed": false, + "judgeNotes": "The mapping is correct (anon → publishable, service_role → secret), but the assistant explicitly describes it as a “direct 1:1 swap” and “same code, just replace the string,” which presents the new keys as exact drop-in renames rather than role equivalents." + }, + { + "name": "placed the publishable key in the frontend with RLS still applying", + "passed": true, + "judgeNotes": "The answer places the publishable key in frontend code, treats it as low-privilege/app-level rather than user identity, explains that signed-in users use their JWT for auth.uid(), and states RLS applies to publishable-key requests. It does not claim the publishable key bypasses RLS or authenticates a user." + }, + { + "name": "kept the secret key server-only and RLS-bypassing", + "passed": true, + "judgeNotes": "The assistant correctly maps secret keys to legacy service_role, states they are backend/server-only, says they bypass RLS entirely, and warns/audits that they must not be shipped or reachable from client code." + } + ], + "skills": { + "available": [ + "supabase", + "supabase-postgres-best-practices" + ], + "loaded": [ + "supabase" + ] + }, + "docs": { + "calls": [ + { + "source": "search_docs", + "query": "{ searchDocs(query: \"publishable key secret key anon service_role migration RLS\", limit: 5) { nodes { title href content } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", + "title": "Migrating to publishable and secret API keys" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", + "title": "Build a User Management App with RedwoodJS" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", + "title": "Build a User Management App with SvelteKit" + }, + { + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" + }, + { + "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", + "title": "Build a Social Auth App with Expo React Native" + } + ], + "resultChars": 149036 + }, + { + "source": "search_docs", + "query": "{ searchDocs(query: \"publishable and secret API keys\", limit: 2) { nodes { title href content } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", + "title": "Migrating to publishable and secret API keys" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-kotlin", + "title": "Build a Product Management Android App with Jetpack Compose" + } + ], + "resultChars": 65688 + } + ] + }, + "prompt": "We're migrating off the legacy anon/service_role API keys to the new publishable\nand secret keys. The team isn't sure which one belongs in the frontend and what\neach means for RLS. Give us a quick rundown.", + "promptSourcePath": "evals/investigate-auth-002-api-key-migration/PROMPT.md", + "attempts": 2, + "sourcePath": "claude-code-sonnet-5/investigate-auth-002-api-key-migration.json" + }, { "experiment": "claude-code-sonnet-5", "experimentSuite": "regression", @@ -736,6 +835,55 @@ "attempts": 1, "sourcePath": "claude-code-sonnet-5-no-skills/build-realtime-001-live-chat-updates.json" }, + { + "experiment": "claude-code-sonnet-5-no-skills", + "experimentSuite": "regression", + "experimentDisplay": { + "agent": "claude-code", + "modelProvider": "anthropic", + "modelId": "claude-sonnet-5", + "reasoningEffort": "high" + }, + "eval": "investigate-auth-002-api-key-migration", + "stage": "investigate", + "product": [ + "auth" + ], + "topic": [ + "security", + "sdk" + ], + "suite": "regression", + "passed": false, + "checks": [ + { + "name": "mapped legacy anon/service_role onto publishable/secret", + "passed": false, + "judgeNotes": "It maps anon → publishable and service_role → secret, but it also frames the change as a 1:1 rename/replacement with “same use cases, just new names/formats,” which the rubric says should fail rather than treating them strictly as role equivalents." + }, + { + "name": "placed the publishable key in the frontend with RLS still applying", + "passed": false, + "judgeNotes": "The answer correctly says the publishable key belongs in frontend code, identifies the app, and does not bypass RLS. However, it does not explicitly state that requests without a user access token run as the anon role while requests with one run as the authenticated role, with RLS applying in both cases." + }, + { + "name": "kept the secret key server-only and RLS-bypassing", + "passed": true, + "judgeNotes": "The assistant states the secret key replaces/maps to service_role, belongs only in trusted backend/server environments, bypasses RLS entirely, and must never be shipped to clients." + } + ], + "skills": { + "available": [], + "loaded": [] + }, + "docs": { + "calls": [] + }, + "prompt": "We're migrating off the legacy anon/service_role API keys to the new publishable\nand secret keys. The team isn't sure which one belongs in the frontend and what\neach means for RLS. Give us a quick rundown.", + "promptSourcePath": "evals/investigate-auth-002-api-key-migration/PROMPT.md", + "attempts": 2, + "sourcePath": "claude-code-sonnet-5-no-skills/investigate-auth-002-api-key-migration.json" + }, { "experiment": "claude-code-sonnet-5-no-skills", "experimentSuite": "regression", From 334ded0790ddbe053b40b440ae412d7a3fba14f1 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 28 Jul 2026 15:02:39 +0000 Subject: [PATCH 5/9] chore: refresh eval results --- apps/web/src/data/eval-results.json | 376 +++++++++++------- .../web/src/data/regression-eval-results.json | 102 ++--- 2 files changed, 294 insertions(+), 184 deletions(-) diff --git a/apps/web/src/data/eval-results.json b/apps/web/src/data/eval-results.json index 6b7c554d..edbac5e3 100644 --- a/apps/web/src/data/eval-results.json +++ b/apps/web/src/data/eval-results.json @@ -1041,7 +1041,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -1062,7 +1062,8 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -1071,17 +1072,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only set profiles.deleted=true (soft delete), no RLS checked it, and it did not delete auth.users or revoke sessions/refresh tokens, so the user stayed signed in/refreshed tokens. It explains why the deleted account still worked and proposes a flow-level fix, not caching/client bugs." + "judgeNotes": "The assistant clearly identifies the root cause: delete_account only set profiles.deleted, did not disable/remove auth.users, did not revoke sessions/refresh tokens, and RLS did not enforce the deleted flag, so the user could keep using existing tokens and sign in again." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changed the flow from soft-delete to deleting auth.users, cascading/removing sessions, refresh tokens, identities, and profile, plus explicitly deleting notes and adding RLS gating to close the live JWT window. This satisfies revoking auth access rather than leaving a cosmetic soft delete." + "judgeNotes": "The assistant rewrote the delete_account flow to revoke/delete sessions and refresh tokens and ban the auth user, plus added RLS checks to reject existing tokens. This satisfies the requirement to remove auth access rather than only soft-deleting." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that issued access tokens remain cryptographically valid until expiry and that local/JWT-only validation would still accept them. Its shipped fix includes RLS gating via account_active(), so its claim of no post-deletion data-path window for the protected tables is consistent, while still caveating that new/local paths need mitigation and naming short JWT expiry/session checks." + "judgeNotes": "The assistant correctly explains that Supabase access tokens are stateless JWTs that remain cryptographically valid until expiry and are not invalidated merely by deleting sessions/users. Its window claim is consistent with the shipped fix: because it added stateful RLS checks against auth.sessions/profile/banned status, it says the current data path is closed immediately, while warning that purely local JWT validation surfaces still accept the token until exp and naming mitigations." } ], "skills": { @@ -1094,11 +1095,40 @@ ] }, "docs": { - "calls": [] + "calls": [ + { + "source": "search_docs", + "query": "{ searchDocs(query: \"delete user account revoke session access token still valid RLS\", limit: 5) { nodes { title href content } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", + "title": "Token Security and Row Level Security" + }, + { + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" + }, + { + "url": "https://supabase.com/docs/guides/auth/third-party/clerk", + "title": "Clerk" + }, + { + "url": "https://supabase.com/docs/guides/realtime/authorization", + "title": "Realtime Authorization" + }, + { + "url": "https://supabase.com/docs/guides/auth/oauth-server/mcp-authentication", + "title": "Model Context Protocol (MCP) Authentication" + } + ], + "resultChars": 73211 + } + ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-opus-4.8/investigate-auth-001-deleted-user-access.json" }, { @@ -2406,17 +2436,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant clearly identifies that delete_account only soft-deletes the profile row, does not remove/disable the auth user or revoke sessions/refresh tokens, and explains that this is why the account can keep signing in/accessing data. It does not blame caching or client bugs." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only set profiles.deleted and did not remove/disable auth.users or revoke auth.sessions/refresh tokens, allowing continued sign-in/session refresh. It also explains the flag was not enforced by RLS." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant rewrote the flow to revoke auth-layer access by banning/marking deleted in auth.users and deleting auth.sessions and auth.refresh_tokens, plus enforced the deleted flag in RLS. This satisfies removal of identity/sessions or equivalent revocation." + "judgeNotes": "The assistant changes the flow beyond a soft delete: delete_account bans the auth user and deletes auth.sessions (cascading to refresh_tokens), and adds RLS enforcement so stale JWTs cannot access app data. This satisfies revoking sessions/refresh tokens or equivalently removing auth access." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains stateless JWTs remain cryptographically valid until expiry and that local-only validation would keep accepting them. Its implemented fix adds RLS checks against the deleted/account_active state, so its claim of no meaningful Data API window is consistent, while still noting an auth/JWT expiry window remains for custom local validation and suggesting lower JWT expiry." + "judgeNotes": "The assistant correctly explains stateless JWTs remain cryptographically valid until expiry and that local validation would still accept them. Its shipped fix adds RLS enforcing account active state, so its claim of no app-data window is consistent, while still noting a residual token-validity window and mitigation via shorter JWT expiry." } ], "skills": { @@ -3931,17 +3961,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only updated a profiles deleted flag, did not remove auth.users, sessions, refresh tokens, identities, or enforce the flag in RLS, explaining why the deleted account could keep using an existing session or sign in again." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only updated profiles.deleted, did not delete auth.users or revoke sessions/tokens, and RLS ignored the flag, so the user could keep signing in/accessing data." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete flow with deletion from auth.users, explicitly noting cascades to identities, sessions, and refresh tokens, and added session-active RLS checks to cut off existing JWT access. This satisfies revoking auth access rather than leaving only a soft delete." + "judgeNotes": "The assistant changed the flow from a soft delete to deleting the auth.users row, explicitly noting cascade removal of identities/sessions/refresh tokens, and added/verified session revocation behavior." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Pass: The assistant explicitly explains that already-issued Supabase access tokens remain cryptographically valid until expiry and that local validation would still accept them. Its shipped fix adds RLS checking auth.sessions via session_id, which genuinely closes the Data API/RLS window after deletion, and its window claim is consistent with that fix while still noting caveats like service-role bypass and in-flight requests." + "judgeNotes": "Explains stateless JWTs remain valid until expiry for local validation, identifies the stale-token window without the RLS/session fix, and the implemented fix genuinely closes the Data API path by checking auth.sessions in RLS. Notes residual risk for other auth.uid-only tables/RPCs." } ], "skills": { @@ -3957,36 +3987,37 @@ "calls": [ { "source": "search_docs", - "query": "{ searchDocs(query: \"allow users to delete their own account SQL function auth.users\", limit: 5) { nodes { title href content } } }", + "query": "{ searchDocs(query: \"JWT claims session_id access token fields\", limit: 5) { nodes { title href content } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/auth/auth-anonymous", - "title": "Anonymous Sign-Ins" + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" }, { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" + "url": "https://supabase.com/docs/guides/auth/auth-hooks/custom-access-token-hook", + "title": "Custom Access Token Hook" }, { - "url": "https://supabase.com/docs/guides/database/extensions/pgaudit", - "title": "PGAudit: Postgres Auditing" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", + "title": "Token Security and Row Level Security" } ], - "resultChars": 74244 + "resultChars": 89772 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-sonnet-5/investigate-auth-001-deleted-user-access.json" }, { @@ -5260,17 +5291,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant clearly identifies the root cause: delete_account only set profiles.deleted, did not touch auth.users, did not ban/remove the auth user, and did not revoke sessions/refresh tokens, so existing sessions and future sign-ins continued to work." + "judgeNotes": "Identifies that delete_account only sets profiles.deleted, does not touch auth.users or revoke sessions/tokens, and explains existing JWT/RLS allows continued access/sign-in." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant updates delete_account to ban the auth user and delete auth.refresh_tokens and auth.sessions, plus adds RLS checks for deleted users. This fixes the soft-delete-only flow by revoking auth access and refresh/session capability." + "judgeNotes": "The assistant updated delete_account to revoke auth access by banning the auth user and deleting sessions/refresh tokens, and also tightened RLS to block deleted accounts immediately." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain cryptographically valid until expiry and are not invalidated merely by banning/deleting sessions. Its shipped fix adds RLS checks against the deleted flag, so its claim of no practical Data API window is consistent, while still noting local/JWT validation would accept the token until exp." + "judgeNotes": "Explains Supabase access tokens are self-contained and remain cryptographically valid until expiry, while the shipped RLS fix reads live deleted state and closes the Data API path immediately. The window claim is consistent with the implemented RLS mitigation." } ], "skills": { @@ -9067,7 +9098,8 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -9077,17 +9109,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Identified that delete_account only soft-deletes public.profiles and leaves auth.users/sessions intact, explaining why stale auth/JWT access continues." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only soft-deleted public.profiles and did not revoke sessions or delete/mark the auth user, so JWT/session/RLS still allowed access. It does not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changed the flow from a soft delete to deleting the auth.users row, which cascades to sessions/identities/refresh tokens, and added RLS checks requiring an active auth user to block stale JWT access." + "judgeNotes": "The assistant replaced the soft-delete-only flow with a delete_account RPC that removes auth.refresh_tokens and auth.sessions, marks the auth user deleted/banned, and updates RLS to require an undeleted auth user plus live session. This satisfies revoking sessions/refresh tokens and removing auth access." }, { "name": "explained the access-token window consistently with its fix", "passed": false, - "judgeNotes": "The fix/window claim is consistent with the shipped RLS/auth.users check, but the assistant did not explicitly explain that stateless JWT access tokens remain accepted until expiry by purely local validation such as getClaims() or custom signature/expiry middleware." + "judgeNotes": "The shipped fix genuinely closes the Supabase Data API path via RLS checks against auth.users and auth.sessions, so the no post-delete data window claim is consistent. However, the assistant did not clearly explain that access tokens are stateless JWTs that remain valid until expiry for purely local validation such as getClaims/custom middleware, which the rubric requires even when the data path is closed." } ], "skills": { @@ -9096,15 +9128,14 @@ "supabase-postgres-best-practices" ], "loaded": [ - "supabase", - "supabase-postgres-best-practices" + "supabase" ] }, "docs": { "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user sessions auth.admin.deleteUser invalidate existing access tokens\", limit: 5) { nodes { __typename ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user revoke sessions delete account auth.admin.deleteUser signOut before delete\", limit: 5) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", "hasContent": true, "pages": [ { @@ -9112,48 +9143,103 @@ "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" }, { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-signout" }, + { + "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" + } + ], + "resultChars": 9673 + }, + { + "source": "search_docs", + "query": "query { searchDocs(query: \"auth.sessions access token invalidation delete user session exists refresh token revoke auth.users\", limit: 8) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", + "hasContent": true, + "pages": [ { "url": "https://supabase.com/docs/guides/resources/glossary", "title": "Glossary" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" + }, + { + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" + }, + { + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" + }, + { + "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", + "title": "Should I set a shorter Max-Age parameter on the cookies?" + }, + { + "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", + "title": "Build a Supabase Integration" + }, + { + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" } ], - "resultChars": 47942 + "resultChars": 106483 }, { "source": "search_docs", - "query": "query { searchDocs(query: \"auth.sessions delete user revoke sessions auth.admin.deleteUser delete user does not automatically sign out\", limit: 5) { nodes { __typename ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } } } }", + "query": "query { searchDocs(query: \"banned_until auth.users deleted_at soft delete user login blocked\", limit: 5) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", "hasContent": true, "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/auth-anonymous", + "title": "Anonymous Sign-Ins" + }, { "url": "https://supabase.com/docs/guides/auth/managing-user-data", "title": "User Management" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" + "url": "https://supabase.com/docs/guides/platform/sso/testing-best-practices", + "title": "SSO Testing and Best Practices" + } + ], + "resultChars": 72816 + }, + { + "source": "search_docs", + "query": "query { searchDocs(query: \"shouldSoftDelete deleted_at auth.users sign in soft delete user\", limit: 10) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" + "url": "https://supabase.com/docs/guides/auth/auth-anonymous", + "title": "Anonymous Sign-Ins" }, { - "url": "https://supabase.com/docs/guides/auth/signout", - "title": "Signing out" + "url": "https://supabase.com/docs/guides/database/postgres/data-deletion", + "title": "Deleting data and dropping objects safely" } ], - "resultChars": 24999 + "resultChars": 29011 } ] }, @@ -11539,17 +11625,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant clearly identifies the root cause: delete_account only soft-deleted public.profiles.deleted and did not remove the auth user or revoke sessions/JWTs, so RLS based on auth.uid() continued to allow access. It explains why the deleted account still works and does not blame caching/client bugs." + "judgeNotes": "The assistant explicitly identified that delete_account only set public.profiles.deleted=true while auth.users and sessions remained, explaining why auth.uid()/JWT-based access still worked. It did not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changed delete_account to delete from auth.users, which cascades to sessions/identities/profile per the discovered FKs, and added active-account checks in RLS. This satisfies removing auth identity/sessions and revoking access rather than leaving only a soft delete." + "judgeNotes": "The assistant replaced the soft-delete RPC with deletion from auth.users, relying on ON DELETE CASCADE to remove sessions, identities, and profile, and also tightened RLS to require an active auth session. This revokes refresh/session access rather than leaving only a profile flag." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant shipped RLS policies that check the auth user still exists via account_is_active(), so its claim that the data path has no post-commit backend access window is consistent with the fix. It explains the prior stale JWT/session issue, though it does not explicitly note that the JWT itself remains valid until expiry for purely local validation." + "judgeNotes": "The assistant correctly explains JWTs remain valid until expiry generally, and its shipped fix adds RLS checks against auth.users/auth.sessions, genuinely closing the Data API path after deletion while noting only in-flight race remains. It does not explicitly mention purely local validation in the final, but it relied on Supabase docs and its window claim is consistent with the implemented RLS mitigation." } ], "skills": { @@ -11557,7 +11643,28 @@ "loaded": [] }, "docs": { - "calls": [] + "calls": [ + { + "source": "search_docs", + "query": "query { searchDocs(query: \"delete user revoke sessions auth.users.deleted_at session invalidation\", limit: 5) { nodes { title href content } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/auth-anonymous", + "title": "Anonymous Sign-Ins" + }, + { + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" + } + ], + "resultChars": 34047 + } + ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", @@ -14163,17 +14270,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only sets profiles.deleted=true, leaves the Auth user and sessions intact, and RLS ignores the deleted flag, allowing continued sign-in/access. It does not blame caching/client bugs, though it notes client cached session only as secondary behavior." + "judgeNotes": "Identifies that delete_account only soft-deletes profiles.deleted, does not delete auth.users or revoke sessions, and explains the still-valid JWT/session is why access continues." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete-only flow with an RPC that deletes the auth.users row, causing sessions to be removed, and added RLS protections for immediate access denial. Verification showed auth user and session removed and same unexpired token denied data access." + "judgeNotes": "The assistant replaced the soft-delete-only flow with a hard delete from auth.users, causing cascade removal of sessions/identity-related rows, and added RLS checks against auth.sessions session_id to reject existing JWTs after session deletion. It explicitly verified auth user/session deletion and post-delete access denial." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that Supabase access tokens/JWTs remain cryptographically valid until exp after user/session deletion. Its shipped fix updates RLS to require live non-deleted profile state and deletes auth.users/sessions, then it claims new database requests lose access immediately, with only already-executing snapshot caveat. This is consistent with its RLS-based mitigation closing the Data API stale-JWT window while noting purely local token validity remains." + "judgeNotes": "The assistant clearly explains JWTs remain cryptographically valid until expiry and that local validation would still accept them. Its shipped fix genuinely closes the Data API path by hard-deleting auth user/sessions and adding RLS checks against auth.sessions/session_id, so its 'no window for these database tables' claim is consistent. It also notes the token itself remains valid until expiration." } ], "skills": { @@ -14189,7 +14296,7 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user existing access token sessions revoke auth.sessions sign out scope global JWT remains valid\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user invalidate access token revoke sessions auth.sessions session_id JWT\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { @@ -14197,34 +14304,31 @@ "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/auth/signout", - "title": "Signing out" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" + "url": "https://supabase.com/docs/guides/resources/glossary", + "title": "Glossary" }, { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" }, { - "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", - "title": "Should I set a shorter Max-Age parameter on the cookies?" + "url": "https://supabase.com/docs/guides/auth/signing-keys", + "title": "JWT Signing Keys" }, { "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", "title": "OAuth 2.1 Flows" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" } ], - "resultChars": 107162 + "resultChars": 129030 }, { "source": "web_search", @@ -14237,14 +14341,59 @@ }, { "source": "web_search", - "query": "site:supabase.com changelog auth session delete user revoke sessions Supabase", + "query": "site:supabase.com/changelog auth sessions revoke deleted user JWT Supabase breaking change", "pages": [] + }, + { + "source": "search_docs", + "query": "query { searchDocs(query: \"admin deleteUser access token remains valid until expiry sign out revoke refresh tokens delete account\", limit: 10) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" + }, + { + "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", + "title": "Build a Supabase Integration" + }, + { + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-signout" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + }, + { + "url": "https://supabase.com/docs/guides/platform/temporary-access", + "title": "Temporary access" + }, + { + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" + }, + { + "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" + }, + { + "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", + "title": "Should I set a shorter Max-Age parameter on the cookies?" + } + ], + "resultChars": 94713 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "codex-gpt-5.6/investigate-auth-001-deleted-user-access.json" }, { @@ -16957,17 +17106,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only set profiles.deleted=true, did not remove the Auth user/sessions/JWT, and explains that auth.uid()-based RLS allowed continued access/sign-in. It does not blame caching or client bugs." + "judgeNotes": "The assistant explicitly identified the root cause: delete_account only set profiles.deleted=true while the Auth user and sessions remained active, so JWT/RLS still allowed access. It did not blame caching or client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changed delete_account() to delete from auth.users, which cascades to sessions/identities/profile per FK constraints, and added RLS hardening. This fixes the soft-delete-only flow and removes auth access/refresh sessions." + "judgeNotes": "The assistant replaced the soft-delete-only RPC with a flow that deletes the auth.users row, which cascades to sessions and identities, and verified auth/profile/notes removal. This satisfies revoking auth access/refresh sessions rather than leaving a soft delete." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains JWTs remain cryptographically valid until expiration and that local/auth.uid-only checks can still accept them. Its shipped fix adds RLS checks against auth.users/profiles via is_active_account(), so the claim of no post-commit data window for protected profile/note data is consistent, while caveating future auth.uid-only paths and in-flight requests." + "judgeNotes": "The assistant explains JWTs remain signature-valid until exp and that auth.uid-only/local validation paths could accept stale tokens until expiry. Its shipped fix deletes auth.users and gates Data API via RLS using the active profile, so saying the application data path is closed after commit is consistent, while still caveating future auth.uid-only policies/stale JWT expiry." } ], "skills": { @@ -16978,48 +17127,37 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user access token remains valid until expiry sign out sessions JWT RLS\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user auth.users JWT remains valid until expiry sessions revoke delete account\", limit: 5) { nodes { ... on Guide { title href content } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", - "title": "Should I set a shorter Max-Age parameter on the cookies?" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", - "title": "Token Security and Row Level Security" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { "url": "https://supabase.com/docs/guides/auth/signing-keys", "title": "JWT Signing Keys" }, { - "url": "https://supabase.com/docs/guides/auth/server-side/advanced-guide", - "title": "Advanced guide" - }, - { - "url": "https://supabase.com/docs/guides/realtime/authorization", - "title": "Realtime Authorization" + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", + "title": "Should I set a shorter Max-Age parameter on the cookies?" } ], - "resultChars": 90545 + "resultChars": 65949 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "codex-gpt-5.6-no-skills/investigate-auth-001-deleted-user-access.json" }, { @@ -18564,17 +18702,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only updated profiles.deleted, leaving auth.users, credentials, sessions, and refresh tokens intact, and explains that RLS did not enforce the deleted flag so the account kept working." + "judgeNotes": "The assistant explicitly identifies that delete_account only updated public.profiles.deleted and did not remove auth.users, sessions, or refresh tokens, explaining why the account stayed signed in and could continue refreshing/logging in. It does not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant rewrote the delete flow to delete the auth.users row, explicitly remove app data, and rely on cascading FKs to remove identities, sessions, refresh tokens, and profile. It also revoked public/anon execute access. This satisfies revoking auth access rather than leaving only a soft delete." + "judgeNotes": "The assistant rewrote delete_account to delete from auth.users, explicitly removing notes first and relying on cascades to remove identities, sessions, refresh_tokens, and profile. It also restricted execution and verified auth/session/token rows were gone. This satisfies revoking auth access rather than leaving a soft delete." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Pass. The assistant explicitly states access tokens are stateless JWTs valid until exp and deletion does not stop purely local validation. Its implemented fix deletes auth.users and cascades sessions/refresh tokens but does not change RLS to verify user/session existence; it correctly notes a residual Data API window and even confirms stale JWT claims can still INSERT. It also names mitigations: lower JWT expiry and validate session_id against auth.sessions." + "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain valid until expiry and that local/RLS validation based only on JWT claims can continue accepting stale tokens. Its shipped fix deletes auth.users/sessions/refresh tokens but leaves RLS accepting stale JWTs, and it explicitly states a bounded window until token TTL remains, verified stale token writes, and names mitigations including shorter JWT expiry and session lookup checks." } ], "skills": { @@ -18587,35 +18725,7 @@ ] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "{ searchDocs(query: \"delete user account security definer function auth.users\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/guides/database/extensions/pgaudit", - "title": "PGAudit: Postgres Auditing" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" - }, - { - "url": "https://supabase.com/docs/guides/functions/examples/push-notifications", - "title": "Sending Push Notifications" - } - ], - "resultChars": 72582 - } - ] + "calls": [] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", @@ -20225,17 +20335,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant correctly identifies that delete_account only soft-deleted the public.profiles row while leaving auth.users, sessions, and refresh tokens intact, allowing continued sign-in/token refresh. It also explains the RLS/JWT aspect and does not blame caching or client bugs." + "judgeNotes": "Identifies that delete_account only soft-deleted the profile via a deleted flag, did not remove auth.users/sessions/refresh_tokens, and explains that this allowed continued sessions, token refresh, and fresh sign-in." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant updated delete_account() to delete the auth.users row, which cascades to sessions, refresh tokens, identities, and profile, and restricted execution. This revokes auth access rather than leaving a soft delete." + "judgeNotes": "The flow was changed from soft-deleting a profile to deleting the auth.users row, with cascades/remediation for sessions, refresh tokens, identities, and prior soft-deleted accounts. This revokes auth access rather than leaving sessions active." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains stateless JWT access tokens remain cryptographically valid until expiry and that purely local validation can still accept them. Its shipped fix adds RLS checks tied to the profile/auth deletion path, and its window claim is consistent: no data-path window for Supabase/RLS after deletion, but a residual window up to JWT expiry for external/local JWT validation." + "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain cryptographically valid until expiry and are not instantly revoked by deletion. Its implemented fix hardens the Data API via RLS checking auth.users existence, so its claim of no data read/write window through the API is consistent. It still notes a residual window for purely local validation and names a mitigation: lower JWT expiry." } ], "skills": { diff --git a/apps/web/src/data/regression-eval-results.json b/apps/web/src/data/regression-eval-results.json index c9ffbd35..b1f5e487 100644 --- a/apps/web/src/data/regression-eval-results.json +++ b/apps/web/src/data/regression-eval-results.json @@ -288,22 +288,22 @@ "sdk" ], "suite": "regression", - "passed": false, + "passed": true, "checks": [ { "name": "mapped legacy anon/service_role onto publishable/secret", - "passed": false, - "judgeNotes": "The mapping is correct (anon → publishable, service_role → secret), but the assistant explicitly describes it as a “direct 1:1 swap” and “same code, just replace the string,” which presents the new keys as exact drop-in renames rather than role equivalents." + "passed": true, + "judgeNotes": "Correctly maps legacy anon to publishable and service_role to secret, while explaining role/RLS equivalence and noting practical differences rather than fully treating them as exact renames." }, { "name": "placed the publishable key in the frontend with RLS still applying", "passed": true, - "judgeNotes": "The answer places the publishable key in frontend code, treats it as low-privilege/app-level rather than user identity, explains that signed-in users use their JWT for auth.uid(), and states RLS applies to publishable-key requests. It does not claim the publishable key bypasses RLS or authenticates a user." + "judgeNotes": "The answer says the publishable key belongs in frontend/public clients, requests without login run as anon and logged-in requests run as authenticated, and RLS is enforced. It does not claim the key bypasses RLS or authenticates a user." }, { "name": "kept the secret key server-only and RLS-bypassing", "passed": true, - "judgeNotes": "The assistant correctly maps secret keys to legacy service_role, states they are backend/server-only, says they bypass RLS entirely, and warns/audits that they must not be shipped or reachable from client code." + "judgeNotes": "Meets rubric: secret key is mapped to service_role, described as backend/server-only, explicitly says it bypasses RLS/BYPASSRLS, and must never be shipped to clients." } ], "skills": { @@ -318,54 +318,21 @@ "docs": { "calls": [ { - "source": "search_docs", - "query": "{ searchDocs(query: \"publishable key secret key anon service_role migration RLS\", limit: 5) { nodes { title href content } } }", + "source": "web_fetch", + "query": "Summarize the differences between legacy anon/service_role keys and new publishable/secret API keys. Explain which is meant for frontend/browser use, which for backend/server use, and how each interacts with Row Level Security (RLS). Include any naming/prefix conventions and migration notes.", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" - }, - { - "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", - "title": "Build a Social Auth App with Expo React Native" + "url": "https://supabase.com/docs/guides/api/api-keys.md" } ], - "resultChars": 149036 - }, - { - "source": "search_docs", - "query": "{ searchDocs(query: \"publishable and secret API keys\", limit: 2) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-kotlin", - "title": "Build a Product Management Android App with Jetpack Compose" - } - ], - "resultChars": 65688 + "resultChars": 1266 } ] }, "prompt": "We're migrating off the legacy anon/service_role API keys to the new publishable\nand secret keys. The team isn't sure which one belongs in the frontend and what\neach means for RLS. Give us a quick rundown.", "promptSourcePath": "evals/investigate-auth-002-api-key-migration/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "claude-code-sonnet-5/investigate-auth-002-api-key-migration.json" }, { @@ -854,22 +821,22 @@ "sdk" ], "suite": "regression", - "passed": false, + "passed": true, "checks": [ { "name": "mapped legacy anon/service_role onto publishable/secret", - "passed": false, - "judgeNotes": "It maps anon → publishable and service_role → secret, but it also frames the change as a 1:1 rename/replacement with “same use cases, just new names/formats,” which the rubric says should fail rather than treating them strictly as role equivalents." + "passed": true, + "judgeNotes": "Correctly maps legacy `anon` to the publishable key and `service_role` to the secret key as role/privilege equivalents, while noting behavioral differences such as headers and non-JWT status." }, { "name": "placed the publishable key in the frontend with RLS still applying", - "passed": false, - "judgeNotes": "The answer correctly says the publishable key belongs in frontend code, identifies the app, and does not bypass RLS. However, it does not explicitly state that requests without a user access token run as the anon role while requests with one run as the authenticated role, with RLS applying in both cases." + "passed": true, + "judgeNotes": "The assistant says the publishable key belongs in frontend code, is not what identifies the user, and that RLS remains enforced. It also distinguishes user JWT auth from the key and does not claim the publishable key bypasses RLS or authenticates a user." }, { "name": "kept the secret key server-only and RLS-bypassing", "passed": true, - "judgeNotes": "The assistant states the secret key replaces/maps to service_role, belongs only in trusted backend/server environments, bypasses RLS entirely, and must never be shipped to clients." + "judgeNotes": "The assistant correctly states that the secret key maps to service_role, must be kept server-only/never shipped to clients, and bypasses RLS." } ], "skills": { @@ -877,11 +844,44 @@ "loaded": [] }, "docs": { - "calls": [] + "calls": [ + { + "source": "search_docs", + "query": "{ searchDocs(query: \"publishable key secret key anon service_role migration RLS\", limit: 6) { nodes { title href content } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", + "title": "Build a User Management App with RedwoodJS" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", + "title": "Migrating to publishable and secret API keys" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", + "title": "Build a User Management App with SvelteKit" + }, + { + "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", + "title": "Build a Social Auth App with Expo React Native" + }, + { + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-nextjs", + "title": "Build a User Management App with Next.js" + } + ], + "resultChars": 190334 + } + ] }, "prompt": "We're migrating off the legacy anon/service_role API keys to the new publishable\nand secret keys. The team isn't sure which one belongs in the frontend and what\neach means for RLS. Give us a quick rundown.", "promptSourcePath": "evals/investigate-auth-002-api-key-migration/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "claude-code-sonnet-5-no-skills/investigate-auth-002-api-key-migration.json" }, { From ad1c388aed0cf396e20c4aa6ef9d3928fe16a780 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 28 Jul 2026 16:05:57 +0000 Subject: [PATCH 6/9] chore: refresh eval results --- apps/web/src/data/eval-results.json | 424 ++++++++---------- .../web/src/data/regression-eval-results.json | 86 ++-- 2 files changed, 232 insertions(+), 278 deletions(-) diff --git a/apps/web/src/data/eval-results.json b/apps/web/src/data/eval-results.json index edbac5e3..f51a23dc 100644 --- a/apps/web/src/data/eval-results.json +++ b/apps/web/src/data/eval-results.json @@ -1041,7 +1041,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -1062,8 +1062,7 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", @@ -1072,17 +1071,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant clearly identifies the root cause: delete_account only set profiles.deleted, did not disable/remove auth.users, did not revoke sessions/refresh tokens, and RLS did not enforce the deleted flag, so the user could keep using existing tokens and sign in again." + "judgeNotes": "The assistant explicitly identifies that delete_account only updates public.profiles.deleted, leaving auth.users, auth.sessions, and refresh tokens intact, and explains that the deleted account still works because sessions are not revoked and the soft-delete flag is not enforced by RLS." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant rewrote the delete_account flow to revoke/delete sessions and refresh tokens and ban the auth user, plus added RLS checks to reject existing tokens. This satisfies the requirement to remove auth access rather than only soft-deleting." + "judgeNotes": "The assistant replaced the soft-delete-only flow with a function that deletes user data, deletes auth.sessions to revoke sessions/refresh tokens, and deletes auth.users, plus added session-liveness RLS checks for stale JWTs. This satisfies the requirement to remove auth identity/sessions and revoke refresh access." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that Supabase access tokens are stateless JWTs that remain cryptographically valid until expiry and are not invalidated merely by deleting sessions/users. Its window claim is consistent with the shipped fix: because it added stateful RLS checks against auth.sessions/profile/banned status, it says the current data path is closed immediately, while warning that purely local JWT validation surfaces still accept the token until exp and naming mitigations." + "judgeNotes": "The assistant correctly explains that existing access tokens are stateless JWTs that remain valid until expiry for purely local validation, and distinguishes that from the implemented Data API fix. Its fix adds RLS session-liveness checks against auth.sessions, so its claim of no post-deletion window for notes/profiles Data API is consistent. It also names mitigations such as shortening JWT expiry and applying session-liveness checks to other surfaces." } ], "skills": { @@ -1098,37 +1097,37 @@ "calls": [ { "source": "search_docs", - "query": "{ searchDocs(query: \"delete user account revoke session access token still valid RLS\", limit: 5) { nodes { title href content } } }", + "query": "{ searchDocs(query: \"delete user account revoke sessions access token still valid\", limit: 5) { nodes { title href content } } }", "hasContent": true, "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", - "title": "Token Security and Row Level Security" - }, { "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", "title": "OAuth 2.1 Flows" }, { - "url": "https://supabase.com/docs/guides/auth/third-party/clerk", - "title": "Clerk" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/realtime/authorization", - "title": "Realtime Authorization" + "url": "https://supabase.com/docs/guides/platform/temporary-access", + "title": "Temporary access" + }, + { + "url": "https://supabase.com/docs/guides/platform/mfa/org-mfa-enforcement", + "title": "Enforce MFA on Organization" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/mcp-authentication", - "title": "Model Context Protocol (MCP) Authentication" + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" } ], - "resultChars": 73211 + "resultChars": 72987 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "claude-code-opus-4.8/investigate-auth-001-deleted-user-access.json" }, { @@ -2405,7 +2404,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -2426,8 +2425,7 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", @@ -2436,17 +2434,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only set profiles.deleted and did not remove/disable auth.users or revoke auth.sessions/refresh tokens, allowing continued sign-in/session refresh. It also explains the flag was not enforced by RLS." + "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deleted the profile row, did not delete auth.users or revoke sessions/refresh tokens, and explains that this is why the deleted account could continue signing in." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changes the flow beyond a soft delete: delete_account bans the auth user and deletes auth.sessions (cascading to refresh_tokens), and adds RLS enforcement so stale JWTs cannot access app data. This satisfies revoking sessions/refresh tokens or equivalently removing auth access." + "judgeNotes": "The assistant replaces the soft-delete flow with deletion of the auth.users row, explicitly causing cascade removal of identities, sessions, and refresh tokens, and revokes access to mint new tokens. It also notes the residual JWT expiry window and possible RLS hardening." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains stateless JWTs remain cryptographically valid until expiry and that local validation would still accept them. Its shipped fix adds RLS enforcing account active state, so its claim of no app-data window is consistent, while still noting a residual token-validity window and mitigation via shorter JWT expiry." + "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain valid until expiry for local validation, and that deleting the auth user/sessions prevents new tokens but does not recall existing access tokens. It also correctly judges its own shipped fix: because it hard-deletes auth.users but did not add RLS existence checks, the Data API can still accept stale JWTs until token expiry, and it names mitigations including shorter JWT expiry and RLS requiring the user to still exist." } ], "skills": { @@ -3931,7 +3929,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -3952,7 +3950,8 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -3961,17 +3960,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only updated profiles.deleted, did not delete auth.users or revoke sessions/tokens, and RLS ignored the flag, so the user could keep signing in/accessing data." + "judgeNotes": "Identifies that delete_account only flipped the profile deleted flag, did not touch auth.users or revoke sessions/refresh tokens, and explains that this allowed continued sign-in/access." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changed the flow from a soft delete to deleting the auth.users row, explicitly noting cascade removal of identities/sessions/refresh tokens, and added/verified session revocation behavior." + "judgeNotes": "The assistant changed the flow so delete_account revokes auth.sessions (thereby refresh capability), bans future sign-ins via auth.users.banned_until, and adds RLS checks to prevent access with stale JWTs. This is more than a soft delete and addresses session/refresh token revocation." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Explains stateless JWTs remain valid until expiry for local validation, identifies the stale-token window without the RLS/session fix, and the implemented fix genuinely closes the Data API path by checking auth.sessions in RLS. Notes residual risk for other auth.uid-only tables/RPCs." + "judgeNotes": "Explains stateless JWTs remain valid until expiry for local validation, and its shipped RLS fix checks live deleted state so Data API access is blocked despite stale tokens. Notes mitigation via shorter JWT TTL/signOut." } ], "skills": { @@ -3984,36 +3983,7 @@ ] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "{ searchDocs(query: \"JWT claims session_id access token fields\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" - }, - { - "url": "https://supabase.com/docs/guides/auth/auth-hooks/custom-access-token-hook", - "title": "Custom Access Token Hook" - }, - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", - "title": "Token Security and Row Level Security" - } - ], - "resultChars": 89772 - } - ] + "calls": [] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", @@ -5260,7 +5230,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -5281,8 +5251,7 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", @@ -5291,17 +5260,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Identifies that delete_account only sets profiles.deleted, does not touch auth.users or revoke sessions/tokens, and explains existing JWT/RLS allows continued access/sign-in." + "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deletes the profile row, does not delete auth.users or revoke sessions/tokens, and explains that this is why the deleted account can still sign in and access data." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant updated delete_account to revoke auth access by banning the auth user and deleting sessions/refresh tokens, and also tightened RLS to block deleted accounts immediately." + "judgeNotes": "The assistant replaced the soft delete with deletion from auth.users, causing identity/session/refresh token removal via cascade, and added FK/cascade protections for user data. This satisfies revoking auth access beyond the unavoidable existing JWT TTL." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Explains Supabase access tokens are self-contained and remain cryptographically valid until expiry, while the shipped RLS fix reads live deleted state and closes the Data API path immediately. The window claim is consistent with the implemented RLS mitigation." + "judgeNotes": "Explains stateless JWT access tokens remain valid until expiry despite deletion/revocation, and correctly distinguishes refresh-token/session revocation from existing access-token validity. Its window claim is consistent with the shipped fix: data rows are deleted and notes inserts are FK-blocked, while purely local JWT validation still accepts the token until expiry; it also names mitigations such as shorter TTL and RLS/user-existence checks." } ], "skills": { @@ -5313,7 +5282,7 @@ }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "claude-code-sonnet-5-no-skills/investigate-auth-001-deleted-user-access.json" }, { @@ -9080,8 +9049,7 @@ "checks": [ { "name": "victim session active before delete-account", - "passed": false, - "notes": "permission denied for table users" + "passed": true }, { "name": "delete_account flow ran for the victim", @@ -9098,28 +9066,26 @@ }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", - "passed": false, - "notes": "permission denied for table users" + "passed": true }, { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only soft-deleted public.profiles and did not revoke sessions or delete/mark the auth user, so JWT/session/RLS still allowed access. It does not blame caching/client bugs." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only set public.profiles.deleted=true and did not delete the auth.users row or revoke/remove sessions, so valid JWT/session-based access continued. It did not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete-only flow with a delete_account RPC that removes auth.refresh_tokens and auth.sessions, marks the auth user deleted/banned, and updates RLS to require an undeleted auth user plus live session. This satisfies revoking sessions/refresh tokens and removing auth access." + "judgeNotes": "The assistant replaced the soft delete with deletion from auth.users and verified sessions are removed; RLS was also tightened to require an active auth.sessions row, closing continued JWT access." }, { "name": "explained the access-token window consistently with its fix", "passed": false, - "judgeNotes": "The shipped fix genuinely closes the Supabase Data API path via RLS checks against auth.users and auth.sessions, so the no post-delete data window claim is consistent. However, the assistant did not clearly explain that access tokens are stateless JWTs that remain valid until expiry for purely local validation such as getClaims/custom middleware, which the rubric requires even when the data path is closed." + "judgeNotes": "The shipped RLS/session-existence fix can genuinely close the Supabase Data API window, so saying no post-commit data access window is consistent. However, the assistant did not clearly explain that access tokens are stateless JWTs that remain valid until expiry for purely local validation such as getClaims() or custom signature/expiry middleware." } ], "skills": { @@ -9135,111 +9101,59 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user revoke sessions delete account auth.admin.deleteUser signOut before delete\", limit: 5) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", + "query": "query { searchDocs(query: \"delete user revoke sessions auth admin delete user does not invalidate tokens\", limit: 5) { nodes { title href content __typename } } }", "hasContent": true, "pages": [ { "url": "https://supabase.com/docs/guides/auth/managing-user-data", "title": "User Management" }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" - }, - { - "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" - } - ], - "resultChars": 9673 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"auth.sessions access token invalidation delete user session exists refresh token revoke auth.users\", limit: 8) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/resources/glossary", - "title": "Glossary" - }, { "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" - }, - { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" + "url": "https://supabase.com/docs/guides/resources/glossary", + "title": "Glossary" }, { "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", "title": "Should I set a shorter Max-Age parameter on the cookies?" }, { - "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", - "title": "Build a Supabase Integration" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" + "url": "https://supabase.com/docs/guides/auth/server-side/creating-a-client", + "title": "Creating a Supabase client for SSR" } ], - "resultChars": 106483 + "resultChars": 76249 }, { "source": "search_docs", - "query": "query { searchDocs(query: \"banned_until auth.users deleted_at soft delete user login blocked\", limit: 5) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", + "query": "query { searchDocs(query: \"auth.jwt session_id Supabase SQL helper\", limit: 5) { nodes { title href content __typename } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/auth/auth-anonymous", - "title": "Anonymous Sign-Ins" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-s3", + "title": "Configure S3 Storage" }, { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" + "url": "https://supabase.com/docs/guides/auth/jwts", + "title": "JSON Web Token (JWT)" }, { - "url": "https://supabase.com/docs/guides/platform/sso/testing-best-practices", - "title": "SSO Testing and Best Practices" - } - ], - "resultChars": 72816 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"shouldSoftDelete deleted_at auth.users sign in soft delete user\", limit: 10) { nodes { title href ... on Guide { content } ... on ClientLibraryFunctionReference { language methodName content href } } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/auth/auth-anonymous", - "title": "Anonymous Sign-Ins" + "url": "https://supabase.com/docs/guides/functions/auth", + "title": "Securing Edge Functions" }, { - "url": "https://supabase.com/docs/guides/database/postgres/data-deletion", - "title": "Deleting data and dropping objects safely" + "url": "https://supabase.com/docs/guides/storage/s3/authentication", + "title": "S3 Authentication" } ], - "resultChars": 29011 + "resultChars": 50285 } ] }, @@ -11595,7 +11509,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -11607,16 +11521,18 @@ }, { "name": "delete-account revokes the user's sessions", - "passed": true, - "notes": "sessions left: 0" + "passed": false, + "notes": "sessions left: 1" }, { "name": "deleted user's refresh token is rejected", - "passed": true + "passed": false, + "notes": "refresh token still produces a session" }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -11625,17 +11541,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identified that delete_account only set public.profiles.deleted=true while auth.users and sessions remained, explaining why auth.uid()/JWT-based access still worked. It did not blame caching/client bugs." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only soft-deletes public.profiles.deleted while auth user/session/JWT remains, so the same JWT/account remains authorized. It does not blame caching/client bugs. Although the proposed fix gates RLS rather than deleting auth/revoking sessions, the required diagnosis is present." }, { "name": "fixed the flow to revoke auth access", - "passed": true, - "judgeNotes": "The assistant replaced the soft-delete RPC with deletion from auth.users, relying on ON DELETE CASCADE to remove sessions, identities, and profile, and also tightened RLS to require an active auth session. This revokes refresh/session access rather than leaving only a profile flag." + "passed": false, + "judgeNotes": "The assistant left the flow as a profile soft delete with RLS gating only. It did not delete the auth user/identity, remove sessions, or revoke refresh tokens, so auth access itself remains." }, { "name": "explained the access-token window consistently with its fix", - "passed": true, - "judgeNotes": "The assistant correctly explains JWTs remain valid until expiry generally, and its shipped fix adds RLS checks against auth.users/auth.sessions, genuinely closing the Data API path after deletion while noting only in-flight race remains. It does not explicitly mention purely local validation in the final, but it relied on Supabase docs and its window claim is consistent with the implemented RLS mitigation." + "passed": false, + "judgeNotes": "The assistant's data-path window claim is consistent with its RLS-based fix, but it does not explain that access tokens are stateless JWTs that remain valid for purely local validation until expiry." } ], "skills": { @@ -11643,28 +11559,7 @@ "loaded": [] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "query { searchDocs(query: \"delete user revoke sessions auth.users.deleted_at session invalidation\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/auth-anonymous", - "title": "Anonymous Sign-Ins" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - } - ], - "resultChars": 34047 - } - ] + "calls": [] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", @@ -14270,17 +14165,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Identifies that delete_account only soft-deletes profiles.deleted, does not delete auth.users or revoke sessions, and explains the still-valid JWT/session is why access continues." + "judgeNotes": "The assistant explicitly identified the root cause: delete_account only set profiles.deleted=true while leaving the Auth user, sessions/JWT valid, so sign-in continued. It did not blame caching/client bugs and explained why the deleted account still works." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete-only flow with a hard delete from auth.users, causing cascade removal of sessions/identity-related rows, and added RLS checks against auth.sessions session_id to reject existing JWTs after session deletion. It explicitly verified auth user/session deletion and post-delete access denial." + "judgeNotes": "The assistant replaced the soft-delete-only flow with deletion from auth.users, relying on cascades to remove sessions/identities, and added RLS checks to deny stale JWT access. It explicitly verifies auth user deletion and stale-token denial." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant clearly explains JWTs remain cryptographically valid until expiry and that local validation would still accept them. Its shipped fix genuinely closes the Data API path by hard-deleting auth user/sessions and adding RLS checks against auth.sessions/session_id, so its 'no window for these database tables' claim is consistent. It also notes the token itself remains valid until expiration." + "judgeNotes": "The assistant explicitly states JWTs remain cryptographically valid until exp and notes custom endpoints/future tables not using the active-account RLS check could accept them. Its shipped fix adds RLS active-account checks on the data path, so claiming no post-commit access window for profiles/notes is consistent, while still caveating purely local validation/custom endpoints." } ], "skills": { @@ -14296,39 +14191,41 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user invalidate access token revoke sessions auth.sessions session_id JWT\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user existing access token session revoke auth sessions sign out scope global JWT\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/auth/signout", + "title": "Signing out" }, { - "url": "https://supabase.com/docs/guides/resources/glossary", - "title": "Glossary" + "url": "https://supabase.com/docs/reference/javascript/auth-signout" }, { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" }, { "url": "https://supabase.com/docs/guides/auth/signing-keys", "title": "JWT Signing Keys" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" } ], - "resultChars": 129030 + "resultChars": 106481 }, { "source": "web_search", @@ -14341,59 +14238,97 @@ }, { "source": "web_search", - "query": "site:supabase.com/changelog auth sessions revoke deleted user JWT Supabase breaking change", + "query": "site:supabase.com/changelog Supabase changelog auth sessions delete user access token breaking change", "pages": [] }, { "source": "search_docs", - "query": "query { searchDocs(query: \"admin deleteUser access token remains valid until expiry sign out revoke refresh tokens delete account\", limit: 10) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user admin deleteUser access token remains valid database RLS session_id auth.sessions\", limit: 10) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, { "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", - "title": "Build a Supabase Integration" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/guides/storage/management/delete-objects", + "title": "Delete Objects" }, { "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", "title": "Multiple SSO Providers" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" + "url": "https://supabase.com/docs/guides/auth/auth-anonymous", + "title": "Anonymous Sign-Ins" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + "url": "https://supabase.com/docs/guides/auth/users", + "title": "Users" + }, + { + "url": "https://supabase.com/docs/guides/storage/security/ownership", + "title": "Ownership" }, { "url": "https://supabase.com/docs/guides/platform/temporary-access", "title": "Temporary access" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" + "url": "https://supabase.com/docs/guides/resources/glossary", + "title": "Glossary" + }, + { + "url": "https://supabase.com/docs/guides/auth/third-party/clerk", + "title": "Clerk" + } + ], + "resultChars": 91950 + }, + { + "source": "search_docs", + "query": "query { searchDocs(query: \"auth.admin.deleteUser delete user JavaScript permanently remove user\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-passkey-delete" }, { "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" }, { - "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", - "title": "Should I set a shorter Max-Age parameter on the cookies?" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-deletefactor" + }, + { + "url": "https://supabase.com/docs/reference/swift/auth-admin-deleteuser", + "title": "deleteUser()" + }, + { + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-admin-updateuserbyid" + }, + { + "url": "https://supabase.com/docs/reference/kotlin/auth-admin-deleteuser", + "title": "deleteUser()" } ], - "resultChars": 94713 + "resultChars": 11658 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "codex-gpt-5.6/investigate-auth-001-deleted-user-access.json" }, { @@ -17106,17 +17041,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identified the root cause: delete_account only set profiles.deleted=true while the Auth user and sessions remained active, so JWT/RLS still allowed access. It did not blame caching or client bugs." + "judgeNotes": "The assistant explicitly identified the root cause: delete_account only set profiles.deleted=true while auth.users and sessions remained active, explaining why sign-in/session access continued. It did not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete-only RPC with a flow that deletes the auth.users row, which cascades to sessions and identities, and verified auth/profile/notes removal. This satisfies revoking auth access/refresh sessions rather than leaving a soft delete." + "judgeNotes": "The assistant changed the flow from soft delete to deleting the auth.users record, which cascades/removes profile and sessions, and added RLS denial for stale JWTs. This satisfies deleting auth identity/sessions and revoking refresh/session access." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant explains JWTs remain signature-valid until exp and that auth.uid-only/local validation paths could accept stale tokens until expiry. Its shipped fix deletes auth.users and gates Data API via RLS using the active profile, so saying the application data path is closed after commit is consistent, while still caveating future auth.uid-only policies/stale JWT expiry." + "judgeNotes": "The assistant correctly explains that JWTs remain cryptographically valid until expiration and caveats purely local JWT validation. Its shipped fix uses RLS/active profile checks and hard-deletes auth.users, and it claims no database table window after commit, which is consistent with the implemented RLS denial for the data path." } ], "skills": { @@ -17127,7 +17062,7 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user auth.users JWT remains valid until expiry sessions revoke delete account\", limit: 5) { nodes { ... on Guide { title href content } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user auth.users access token remains valid until JWT expires revoke sessions admin delete user\", limit: 5) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", "hasContent": true, "pages": [ { @@ -17138,26 +17073,61 @@ "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, + { + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" + }, { "url": "https://supabase.com/docs/guides/auth/signing-keys", "title": "JWT Signing Keys" }, { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" + } + ], + "resultChars": 107892 + }, + { + "source": "search_docs", + "query": "query { searchDocs(query: \"sign out revoke refresh tokens access token valid until expires JWT\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "hasContent": true, + "pages": [ + { + "url": "https://supabase.com/docs/guides/auth/sessions", + "title": "User sessions" + }, + { + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" }, { "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", "title": "Should I set a shorter Max-Age parameter on the cookies?" + }, + { + "url": "https://supabase.com/docs/reference/javascript/auth-signout" + }, + { + "url": "https://supabase.com/docs/guides/auth/server-side/advanced-guide", + "title": "Advanced guide" + }, + { + "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", + "title": "Build a Supabase Integration" + }, + { + "url": "https://supabase.com/docs/guides/resources/glossary", + "title": "Glossary" } ], - "resultChars": 65949 + "resultChars": 79809 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "codex-gpt-5.6-no-skills/investigate-auth-001-deleted-user-access.json" }, { @@ -18702,17 +18672,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only updated public.profiles.deleted and did not remove auth.users, sessions, or refresh tokens, explaining why the account stayed signed in and could continue refreshing/logging in. It does not blame caching/client bugs." + "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deleted the public.profiles row, did not delete auth.users or sessions/refresh tokens, and explains that this is why the account could still sign in." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant rewrote delete_account to delete from auth.users, explicitly removing notes first and relying on cascades to remove identities, sessions, refresh_tokens, and profile. It also restricted execution and verified auth/session/token rows were gone. This satisfies revoking auth access rather than leaving a soft delete." + "judgeNotes": "The assistant replaced the soft-delete flow with deletion of the auth.users row, which cascades to sessions, refresh tokens, identities, and profiles, and explicitly deletes notes. It also revoked public/anon execute and verified sessions/refresh tokens were removed." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain valid until expiry and that local/RLS validation based only on JWT claims can continue accepting stale tokens. Its shipped fix deletes auth.users/sessions/refresh tokens but leaves RLS accepting stale JWTs, and it explicitly states a bounded window until token TTL remains, verified stale token writes, and names mitigations including shorter JWT expiry and session lookup checks." + "judgeNotes": "The assistant correctly explains that pre-existing access tokens are stateless JWTs validated by signature and remain valid until expiry for local validation. Its shipped fix deletes auth users/sessions/refresh tokens and notes, but does not add RLS/session-existence checks, so it appropriately states a remaining window up to JWT expiry and names valid mitigations: shorter JWT expiry and RLS checking auth.sessions." } ], "skills": { @@ -20335,17 +20305,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "Identifies that delete_account only soft-deleted the profile via a deleted flag, did not remove auth.users/sessions/refresh_tokens, and explains that this allowed continued sessions, token refresh, and fresh sign-in." + "judgeNotes": "The assistant explicitly identifies that delete_account only updated/soft-deleted the public.profiles row via a deleted flag, while auth.users, auth.sessions, and auth.refresh_tokens remained intact, allowing continued sign-in/token refresh. It does not blame caching/client bugs and explains why the deleted account still worked." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The flow was changed from soft-deleting a profile to deleting the auth.users row, with cascades/remediation for sessions, refresh tokens, identities, and prior soft-deleted accounts. This revokes auth access rather than leaving sessions active." + "judgeNotes": "The assistant rewrote the flow to delete the auth.users row, which cascades/removes identities, sessions, and refresh tokens, and also hardened RLS against stale JWT access. This satisfies the requirement to revoke auth access rather than soft-delete only." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain cryptographically valid until expiry and are not instantly revoked by deletion. Its implemented fix hardens the Data API via RLS checking auth.users existence, so its claim of no data read/write window through the API is consistent. It still notes a residual window for purely local validation and names a mitigation: lower JWT expiry." + "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain cryptographically valid until expiry and that purely local JWT validation would still accept them. Its shipped fix adds RLS checks against a live profile row, so its claim of effectively no data API window is consistent with the implementation, while still noting the remaining JWT-expiry window for local validation and naming short TTL as mitigation." } ], "skills": { diff --git a/apps/web/src/data/regression-eval-results.json b/apps/web/src/data/regression-eval-results.json index b1f5e487..65f4b0fc 100644 --- a/apps/web/src/data/regression-eval-results.json +++ b/apps/web/src/data/regression-eval-results.json @@ -288,22 +288,22 @@ "sdk" ], "suite": "regression", - "passed": true, + "passed": false, "checks": [ { "name": "mapped legacy anon/service_role onto publishable/secret", - "passed": true, - "judgeNotes": "Correctly maps legacy anon to publishable and service_role to secret, while explaining role/RLS equivalence and noting practical differences rather than fully treating them as exact renames." + "passed": false, + "judgeNotes": "The mapping is correct (anon → publishable, service_role → secret), but the answer also describes them as “direct drop-in” replacements, which the rubric says should fail rather than framing them only as role equivalents." }, { "name": "placed the publishable key in the frontend with RLS still applying", - "passed": true, - "judgeNotes": "The answer says the publishable key belongs in frontend/public clients, requests without login run as anon and logged-in requests run as authenticated, and RLS is enforced. It does not claim the key bypasses RLS or authenticates a user." + "passed": false, + "judgeNotes": "The answer correctly says publishable keys belong in frontend code and respect RLS, and does not claim they authenticate a user or bypass RLS. However, it does not explicitly describe the publishable key as identifying the app rather than a user, nor clearly state that requests without a user access token run as anon while requests with one run as authenticated and RLS applies in both cases." }, { "name": "kept the secret key server-only and RLS-bypassing", "passed": true, - "judgeNotes": "Meets rubric: secret key is mapped to service_role, described as backend/server-only, explicitly says it bypasses RLS/BYPASSRLS, and must never be shipped to clients." + "judgeNotes": "The assistant maps secret keys to service_role, states they are backend/server-only, says they bypass RLS entirely, and warns never to expose them in client code." } ], "skills": { @@ -318,21 +318,38 @@ "docs": { "calls": [ { - "source": "web_fetch", - "query": "Summarize the differences between legacy anon/service_role keys and new publishable/secret API keys. Explain which is meant for frontend/browser use, which for backend/server use, and how each interacts with Row Level Security (RLS). Include any naming/prefix conventions and migration notes.", + "source": "search_docs", + "query": "{ searchDocs(query: \"publishable key secret key anon service_role migration RLS\", limit: 5) { nodes { title href content } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/api/api-keys.md" + "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", + "title": "Migrating to publishable and secret API keys" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", + "title": "Build a User Management App with RedwoodJS" + }, + { + "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", + "title": "Build a User Management App with SvelteKit" + }, + { + "url": "https://supabase.com/docs/guides/auth/jwt-fields", + "title": "JWT Claims Reference" + }, + { + "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", + "title": "Build a Social Auth App with Expo React Native" } ], - "resultChars": 1266 + "resultChars": 149036 } ] }, "prompt": "We're migrating off the legacy anon/service_role API keys to the new publishable\nand secret keys. The team isn't sure which one belongs in the frontend and what\neach means for RLS. Give us a quick rundown.", "promptSourcePath": "evals/investigate-auth-002-api-key-migration/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-sonnet-5/investigate-auth-002-api-key-migration.json" }, { @@ -821,22 +838,22 @@ "sdk" ], "suite": "regression", - "passed": true, + "passed": false, "checks": [ { "name": "mapped legacy anon/service_role onto publishable/secret", "passed": true, - "judgeNotes": "Correctly maps legacy `anon` to the publishable key and `service_role` to the secret key as role/privilege equivalents, while noting behavioral differences such as headers and non-JWT status." + "judgeNotes": "Maps legacy anon to publishable and service_role to secret as role equivalents, while noting conceptual equivalence and behavioral differences rather than merely exact renames." }, { "name": "placed the publishable key in the frontend with RLS still applying", - "passed": true, - "judgeNotes": "The assistant says the publishable key belongs in frontend code, is not what identifies the user, and that RLS remains enforced. It also distinguishes user JWT auth from the key and does not claim the publishable key bypasses RLS or authenticates a user." + "passed": false, + "judgeNotes": "The answer correctly says the publishable key belongs in frontend code, identifies the app/request rather than a user, and does not bypass RLS. However, it does not state the required distinction that requests without a user access token run as the anon role while requests with a user access token run as the authenticated role, with RLS applying in both cases." }, { "name": "kept the secret key server-only and RLS-bypassing", "passed": true, - "judgeNotes": "The assistant correctly states that the secret key maps to service_role, must be kept server-only/never shipped to clients, and bypasses RLS." + "judgeNotes": "The assistant states the secret key replaces/maps to legacy service_role, must be server-only/never exposed client-side, and bypasses RLS entirely. It does not recommend putting secret/service_role keys in frontend code or claim RLS applies to the secret key." } ], "skills": { @@ -844,44 +861,11 @@ "loaded": [] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "{ searchDocs(query: \"publishable key secret key anon service_role migration RLS\", limit: 6) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" - }, - { - "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", - "title": "Build a Social Auth App with Expo React Native" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-nextjs", - "title": "Build a User Management App with Next.js" - } - ], - "resultChars": 190334 - } - ] + "calls": [] }, "prompt": "We're migrating off the legacy anon/service_role API keys to the new publishable\nand secret keys. The team isn't sure which one belongs in the frontend and what\neach means for RLS. Give us a quick rundown.", "promptSourcePath": "evals/investigate-auth-002-api-key-migration/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-sonnet-5-no-skills/investigate-auth-002-api-key-migration.json" }, { From 16327518e2e81e3d194a11be8fa8fc1438e101c3 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Wed, 29 Jul 2026 12:26:36 +0200 Subject: [PATCH 7/9] fix(evals): score the api-key rundown on what the user asked The CI refresh failed both experiments on "placed the publishable key in the frontend with RLS still applying", and reading the judge notes, both answers were right: publishable key in the frontend, RLS still applies, secret key kept server-side. They failed for not spelling out that requests run as anon without a user token and authenticated with one. The prompt never asks for that. It asks which key belongs in the frontend and what each means for RLS, and CONTRIBUTING says prompts should not reflect deep familiarity with Supabase. Expanding the prompt to ask about Postgres roles would have made the scenario evaluator-shaped rather than user-realistic, so the rubric gives way instead: role detail is a bonus, not a requirement. Also stops failing the mapping check on "drop-in" phrasing. The mapping itself is what the migration question is about. Fail conditions are unchanged: bypassing RLS with the publishable key, treating it as user identity, shipping a secret key to the client, inverting the mapping. Both previously failing experiments now pass 3/3 locally. --- evals/investigate-auth-002-api-key-migration/EVAL.ts | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/evals/investigate-auth-002-api-key-migration/EVAL.ts b/evals/investigate-auth-002-api-key-migration/EVAL.ts index 6c73f69e..651607e7 100644 --- a/evals/investigate-auth-002-api-key-migration/EVAL.ts +++ b/evals/investigate-auth-002-api-key-migration/EVAL.ts @@ -13,17 +13,17 @@ const scorer: ToolScorer = async (ctx) => { judge({ input, rubric: stripIndent` - Pass if the assistant maps the legacy keys onto the new ones as role equivalents: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. + Pass if the assistant maps the legacy keys onto the new ones: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. Wording like "replaces" or "drop-in" is fine, the mapping itself is what matters. - Fail if the assistant inverts that mapping, or presents the new keys as exact drop-in renames of the legacy ones rather than role equivalents. + Fail if the assistant inverts that mapping, or never relates the legacy keys to the new ones at all. `, }), judge({ input, rubric: stripIndent` - Pass if the assistant puts the publishable key in frontend code and describes it as identifying the app rather than a user: requests carrying no user access token run as the anon role, requests carrying one run as the authenticated role, and RLS applies either way. + Pass if the assistant puts the publishable key in frontend code and says RLS still applies to requests made with it. Detail beyond that (naming the anon and authenticated roles, or that the key identifies the app rather than a user) is a bonus, not a requirement: the user asked which key belongs in the frontend and what it means for RLS, so answering that is enough. - Fail if the assistant says the publishable key bypasses RLS, or says the key itself authenticates a user or carries their identity. + Fail if the assistant says the publishable key bypasses RLS, says the key itself authenticates a user or carries their identity, or puts a secret key in the frontend instead. `, }), judge({ From 50af157947f0b64fa7c51f84fce55ce6e1829937 Mon Sep 17 00:00:00 2001 From: Barry Roodt Date: Wed, 29 Jul 2026 12:29:34 +0200 Subject: [PATCH 8/9] fix(evals): keep failing answers that call the new keys drop-in Previous commit went too far and blessed "drop-in" phrasing on the mapping check. The new keys are not interchangeable with the legacy ones: they are not JWTs and behave differently in places such as Edge Functions, so an answer calling them a straight swap is materially wrong and the CI failure on that was a fair catch. Mapping now fails on inverted mapping or claimed exact interchangeability, and says why so the judge grades the claim rather than the wording. The loosening stands only where it belonged, on the frontend/RLS check. Both experiments still pass 3/3 locally, now on answers that explicitly note the keys are non-JWT and not a rename. --- evals/investigate-auth-002-api-key-migration/EVAL.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/evals/investigate-auth-002-api-key-migration/EVAL.ts b/evals/investigate-auth-002-api-key-migration/EVAL.ts index 651607e7..bfe37451 100644 --- a/evals/investigate-auth-002-api-key-migration/EVAL.ts +++ b/evals/investigate-auth-002-api-key-migration/EVAL.ts @@ -13,9 +13,9 @@ const scorer: ToolScorer = async (ctx) => { judge({ input, rubric: stripIndent` - Pass if the assistant maps the legacy keys onto the new ones: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. Wording like "replaces" or "drop-in" is fine, the mapping itself is what matters. + Pass if the assistant maps the legacy keys onto the new ones: the anon key corresponds to the publishable key, and the service_role key corresponds to a secret key. - Fail if the assistant inverts that mapping, or never relates the legacy keys to the new ones at all. + Fail if the assistant inverts that mapping, never relates the legacy keys to the new ones, or presents the new keys as exact drop-in replacements. They are not interchangeable with the legacy ones: the new keys are not JWTs and behave differently in places such as Edge Functions, so "same role, new key" is right and "just swap the string" is not. `, }), judge({ From 3dc89521e5737c9ab54f99a16198b441bc80fcd4 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 29 Jul 2026 10:48:16 +0000 Subject: [PATCH 9/9] chore: refresh eval results --- apps/web/src/data/eval-results.json | 410 ++++++------------ .../web/src/data/regression-eval-results.json | 43 +- 2 files changed, 138 insertions(+), 315 deletions(-) diff --git a/apps/web/src/data/eval-results.json b/apps/web/src/data/eval-results.json index f51a23dc..f954e232 100644 --- a/apps/web/src/data/eval-results.json +++ b/apps/web/src/data/eval-results.json @@ -1041,7 +1041,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -1062,7 +1062,8 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -1071,17 +1072,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only updates public.profiles.deleted, leaving auth.users, auth.sessions, and refresh tokens intact, and explains that the deleted account still works because sessions are not revoked and the soft-delete flag is not enforced by RLS." + "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deleted the profile, did not remove/ban the auth user or revoke sessions/refresh tokens, and explains that this allowed the deleted account to keep signing in/accessing data." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete-only flow with a function that deletes user data, deletes auth.sessions to revoke sessions/refresh tokens, and deletes auth.users, plus added session-liveness RLS checks for stale JWTs. This satisfies the requirement to remove auth identity/sessions and revoke refresh access." + "judgeNotes": "The flow no longer remains a mere soft delete: it revokes auth access by banning the auth user and deleting sessions and refresh tokens, with RLS enforcement added and verified." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that existing access tokens are stateless JWTs that remain valid until expiry for purely local validation, and distinguishes that from the implemented Data API fix. Its fix adds RLS session-liveness checks against auth.sessions, so its claim of no post-deletion window for notes/profiles Data API is consistent. It also names mitigations such as shortening JWT expiry and applying session-liveness checks to other surfaces." + "judgeNotes": "Explains JWTs remain valid until expiry for local validation, and its Data API no-window claim is consistent with the shipped RLS fix enforcing deleted status on every query. It also names mitigations such as shorter JWT expiry and server-side session checks." } ], "skills": { @@ -1094,40 +1095,11 @@ ] }, "docs": { - "calls": [ - { - "source": "search_docs", - "query": "{ searchDocs(query: \"delete user account revoke sessions access token still valid\", limit: 5) { nodes { title href content } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" - }, - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/platform/temporary-access", - "title": "Temporary access" - }, - { - "url": "https://supabase.com/docs/guides/platform/mfa/org-mfa-enforcement", - "title": "Enforce MFA on Organization" - }, - { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" - } - ], - "resultChars": 72987 - } - ] + "calls": [] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-opus-4.8/investigate-auth-001-deleted-user-access.json" }, { @@ -2404,7 +2376,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -2425,7 +2397,8 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -2434,17 +2407,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deleted the profile row, did not delete auth.users or revoke sessions/refresh tokens, and explains that this is why the deleted account could continue signing in." + "judgeNotes": "The assistant clearly identifies the root cause: delete_account only soft-deleted public.profiles.deleted, did not remove/ban the auth.users row, did not revoke sessions/refresh tokens, and RLS ignored the deleted flag, so the account could keep signing in/accessing data." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaces the soft-delete flow with deletion of the auth.users row, explicitly causing cascade removal of identities, sessions, and refresh tokens, and revokes access to mint new tokens. It also notes the residual JWT expiry window and possible RLS hardening." + "judgeNotes": "The flow was changed from a soft delete to an auth-level revocation: the function bans the auth user, deletes sessions, deletes refresh tokens, and RLS blocks already-issued JWTs." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain valid until expiry for local validation, and that deleting the auth user/sessions prevents new tokens but does not recall existing access tokens. It also correctly judges its own shipped fix: because it hard-deletes auth.users but did not add RLS existence checks, the Data API can still accept stale JWTs until token expiry, and it names mitigations including shorter JWT expiry and RLS requiring the user to still exist." + "judgeNotes": "The assistant correctly states that Supabase access tokens are stateless JWTs that remain valid until expiry and that banning/deleting sessions only stops refreshed/new tokens. Its window claim is consistent with the shipped fix: RLS re-checks account state server-side, so the Data API path has no usable post-deletion window for covered tables, while the token itself still exists until expiry." } ], "skills": { @@ -3932,45 +3905,9 @@ "passed": false, "checks": [ { - "name": "victim session active before delete-account", - "passed": true - }, - { - "name": "delete_account flow ran for the victim", - "passed": true - }, - { - "name": "delete-account revokes the user's sessions", - "passed": true, - "notes": "sessions left: 0" - }, - { - "name": "deleted user's refresh token is rejected", - "passed": true - }, - { - "name": "deleted user cannot sign back in", + "name": "created auth sessions", "passed": false, - "notes": "deleted account can still sign in" - }, - { - "name": "other users keep their sessions and access", - "passed": true - }, - { - "name": "diagnosed the soft-delete as the cause", - "passed": true, - "judgeNotes": "Identifies that delete_account only flipped the profile deleted flag, did not touch auth.users or revoke sessions/refresh tokens, and explains that this allowed continued sign-in/access." - }, - { - "name": "fixed the flow to revoke auth access", - "passed": true, - "judgeNotes": "The assistant changed the flow so delete_account revokes auth.sessions (thereby refresh capability), bans future sign-ins via auth.users.banned_until, and adds RLS checks to prevent access with stale JWTs. This is more than a soft delete and addresses session/refresh token revocation." - }, - { - "name": "explained the access-token window consistently with its fix", - "passed": true, - "judgeNotes": "Explains stateless JWTs remain valid until expiry for local validation, and its shipped RLS fix checks live deleted state so Data API access is blocked despite stale tokens. Notes mitigation via shorter JWT TTL/signOut." + "notes": "Internal server error" } ], "skills": { @@ -5230,7 +5167,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": true, + "passed": false, "checks": [ { "name": "victim session active before delete-account", @@ -5251,7 +5188,8 @@ }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -5260,17 +5198,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deletes the profile row, does not delete auth.users or revoke sessions/tokens, and explains that this is why the deleted account can still sign in and access data." + "judgeNotes": "The assistant clearly identifies the root cause: delete_account only marked public.profiles.deleted and did not touch auth.users, auth.sessions, or auth.refresh_tokens, allowing continued sign-in/session refresh. It also explains the deleted flag was not enforced by RLS and proposes/applies revocation fixes." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft delete with deletion from auth.users, causing identity/session/refresh token removal via cascade, and added FK/cascade protections for user data. This satisfies revoking auth access beyond the unavoidable existing JWT TTL." + "judgeNotes": "The assistant updated the flow to revoke auth access by banning the auth user and deleting auth.sessions and auth.refresh_tokens, plus added RLS checks for deleted users. This is more than a soft delete and satisfies the requirement." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "Explains stateless JWT access tokens remain valid until expiry despite deletion/revocation, and correctly distinguishes refresh-token/session revocation from existing access-token validity. Its window claim is consistent with the shipped fix: data rows are deleted and notes inserts are FK-blocked, while purely local JWT validation still accepts the token until expiry; it also names mitigations such as shorter TTL and RLS/user-existence checks." + "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain valid until expiry despite session/refresh-token deletion, and that local signature/expiry validation will still accept them. Its shipped fix adds live RLS checks against the deleted profile state, so its claim that the data path is closed while the token itself remains valid until expiry is consistent. It also names JWT expiry reduction as a mitigation." } ], "skills": { @@ -5282,7 +5220,7 @@ }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "claude-code-sonnet-5-no-skills/investigate-auth-001-deleted-user-access.json" }, { @@ -9057,16 +8995,18 @@ }, { "name": "delete-account revokes the user's sessions", - "passed": true, - "notes": "sessions left: 0" + "passed": false, + "notes": "sessions left: 1" }, { "name": "deleted user's refresh token is rejected", - "passed": true + "passed": false, + "notes": "refresh token still produces a session" }, { "name": "deleted user cannot sign back in", - "passed": true + "passed": false, + "notes": "deleted account can still sign in" }, { "name": "other users keep their sessions and access", @@ -9074,18 +9014,18 @@ }, { "name": "diagnosed the soft-delete as the cause", - "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only set public.profiles.deleted=true and did not delete the auth.users row or revoke/remove sessions, so valid JWT/session-based access continued. It did not blame caching/client bugs." + "passed": false, + "judgeNotes": "The assistant does not identify the specific bug: delete_account only soft-deletes the profile row and never deletes the auth user or revokes sessions. It instead gives a generic Supabase JWT/session explanation and even discusses deleting auth.users, so it does not clearly explain why the deleted account can still sign in in the given flow." }, { "name": "fixed the flow to revoke auth access", - "passed": true, - "judgeNotes": "The assistant replaced the soft delete with deletion from auth.users and verified sessions are removed; RLS was also tightened to require an active auth.sessions row, closing continued JWT access." + "passed": false, + "judgeNotes": "The assistant leaves the main flow as a soft delete when possible and only adds a helper for session-aware RLS, explicitly noting that stale JWTs may still work unless policies are updated. This does not fully fix auth access revocation." }, { "name": "explained the access-token window consistently with its fix", - "passed": false, - "judgeNotes": "The shipped RLS/session-existence fix can genuinely close the Supabase Data API window, so saying no post-commit data access window is consistent. However, the assistant did not clearly explain that access tokens are stateless JWTs that remain valid until expiry for purely local validation such as getClaims() or custom signature/expiry middleware." + "passed": true, + "judgeNotes": "The assistant correctly states that Supabase access JWTs remain valid until expiry despite user deletion/sign-out/session revocation, and that local/JWT-only checks such as auth.uid()/claims can continue to accept them. It also correctly qualifies the remaining window based on its actual fix: since it only adds a session-aware helper and revokes sessions, stale JWTs may still work until expiry unless RLS/data access is updated to validate active sessions." } ], "skills": { @@ -9100,60 +9040,34 @@ "docs": { "calls": [ { - "source": "search_docs", - "query": "query { searchDocs(query: \"delete user revoke sessions auth admin delete user does not invalidate tokens\", limit: 5) { nodes { title href content __typename } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary", - "title": "Glossary" - }, - { - "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", - "title": "Should I set a shorter Max-Age parameter on the cookies?" - }, - { - "url": "https://supabase.com/docs/guides/auth/server-side/creating-a-client", - "title": "Creating a Supabase client for SSR" - } - ], - "resultChars": 76249 + "source": "web_search", + "query": "site:supabase.com/docs deleting a user does not invalidate existing access tokens Supabase auth sessions delete user", + "pages": [] }, { - "source": "search_docs", - "query": "query { searchDocs(query: \"auth.jwt session_id Supabase SQL helper\", limit: 5) { nodes { title href content __typename } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/self-hosting/self-hosted-s3", - "title": "Configure S3 Storage" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwts", - "title": "JSON Web Token (JWT)" - }, - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" - }, - { - "url": "https://supabase.com/docs/guides/functions/auth", - "title": "Securing Edge Functions" - }, - { - "url": "https://supabase.com/docs/guides/storage/s3/authentication", - "title": "S3 Authentication" - } - ], - "resultChars": 50285 + "source": "web_search", + "query": "site:supabase.com/docs auth.admin.signOut user all sessions Supabase", + "pages": [] + }, + { + "source": "web_search", + "query": "site:supabase.com/docs session_id claim auth.sessions Supabase sensitive actions", + "pages": [] + }, + { + "source": "web_search", + "query": "site:supabase.com/docs soft delete disable account access token valid until expires Supabase deleteUser shouldSoftDelete", + "pages": [] + }, + { + "source": "web_search", + "query": "site:supabase.com/docs soft deleted user JWT valid until expiry Supabase", + "pages": [] + }, + { + "source": "web_search", + "query": "site:supabase.com/docs auth.users deleted_at soft delete token validation Supabase", + "pages": [] } ] }, @@ -11509,7 +11423,7 @@ ], "suite": "benchmark", "interface": "mcp", - "passed": false, + "passed": true, "checks": [ { "name": "victim session active before delete-account", @@ -11521,18 +11435,16 @@ }, { "name": "delete-account revokes the user's sessions", - "passed": false, - "notes": "sessions left: 1" + "passed": true, + "notes": "sessions left: 0" }, { "name": "deleted user's refresh token is rejected", - "passed": false, - "notes": "refresh token still produces a session" + "passed": true }, { "name": "deleted user cannot sign back in", - "passed": false, - "notes": "deleted account can still sign in" + "passed": true }, { "name": "other users keep their sessions and access", @@ -11541,17 +11453,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only soft-deletes public.profiles.deleted while auth user/session/JWT remains, so the same JWT/account remains authorized. It does not blame caching/client bugs. Although the proposed fix gates RLS rather than deleting auth/revoking sessions, the required diagnosis is present." + "judgeNotes": "The assistant explicitly identifies the root cause: delete_account only sets public.profiles.deleted=true and does not remove the auth identity or sessions, so valid JWTs keep working. It also explains why the deleted account still works and does not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", - "passed": false, - "judgeNotes": "The assistant left the flow as a profile soft delete with RLS gating only. It did not delete the auth user/identity, remove sessions, or revoke refresh tokens, so auth access itself remains." + "passed": true, + "judgeNotes": "The assistant replaced the soft-delete-only RPC with a hard delete of auth.users for auth.uid(), which cascades to auth.sessions/auth.refresh_tokens/auth.identities per inspected FKs, and also tightened RLS to block stale JWTs. This satisfies deleting/removing auth identity and revoking sessions/refresh tokens." }, { "name": "explained the access-token window consistently with its fix", - "passed": false, - "judgeNotes": "The assistant's data-path window claim is consistent with its RLS-based fix, but it does not explain that access tokens are stateless JWTs that remain valid for purely local validation until expiry." + "passed": true, + "judgeNotes": "The assistant correctly identified stale JWT behavior and shipped an RLS-based mitigation that checks auth.users existence, genuinely closing the Data API path after deletion commits. It appropriately claims no post-commit data window for normal app requests. It does not explicitly state that the JWT itself remains valid until expiry for purely local validation, but it does note still-valid/stale JWTs and that RLS blocks them; overall satisfies the rubric for this implemented fix." } ], "skills": { @@ -11563,7 +11475,7 @@ }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "codex-gpt-5.4-mini-no-skills/investigate-auth-001-deleted-user-access.json" }, { @@ -14165,17 +14077,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identified the root cause: delete_account only set profiles.deleted=true while leaving the Auth user, sessions/JWT valid, so sign-in continued. It did not blame caching/client bugs and explained why the deleted account still works." + "judgeNotes": "The assistant explicitly identifies that delete_account only sets profiles.deleted=true, leaving the Auth user and sessions intact, and explains that RLS based on auth.uid()/JWT means the account can continue accessing/signing in. It does not blame caching/client bugs." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete-only flow with deletion from auth.users, relying on cascades to remove sessions/identities, and added RLS checks to deny stale JWT access. It explicitly verifies auth user deletion and stale-token denial." + "judgeNotes": "The assistant replaced the soft-delete-only RPC with a flow that deletes auth.sessions and auth.users, and added session/account checks in RLS. It explicitly verifies the same JWT loses data access after deletion and notes refresh/future login fail." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant explicitly states JWTs remain cryptographically valid until exp and notes custom endpoints/future tables not using the active-account RLS check could accept them. Its shipped fix adds RLS active-account checks on the data path, so claiming no post-commit access window for profiles/notes is consistent, while still caveating purely local validation/custom endpoints." + "judgeNotes": "Assistant clearly explains JWTs remain cryptographically valid until expiry and local signature-only validation could still accept them. Its shipped fix includes RLS/session/account existence checks, so the claim that Data API access is closed immediately after commit is consistent, with caveat for already-running requests and purely local validation." } ], "skills": { @@ -14191,41 +14103,43 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user existing access token session revoke auth sessions sign out scope global JWT\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user existing access token session revoke auth.sessions JWT account deletion\", limit: 8) { nodes { title href content } } }", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/auth/signout", - "title": "Signing out" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" + "url": "https://supabase.com/docs/reference/api/v1-delete-jit-access", + "title": "Delete JIT access by user-id" }, { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" + "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", + "title": "OAuth 2.1 Flows" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" + "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", + "title": "Multiple SSO Providers" }, { "url": "https://supabase.com/docs/guides/auth/jwt-fields", "title": "JWT Claims Reference" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" + "url": "https://supabase.com/docs/guides/auth/signing-keys", + "title": "JWT Signing Keys" + }, + { + "url": "https://supabase.com/docs/guides/platform/sso/testing-best-practices", + "title": "SSO Testing and Best Practices" } ], - "resultChars": 106481 + "resultChars": 152997 }, { "source": "web_search", @@ -14238,12 +14152,12 @@ }, { "source": "web_search", - "query": "site:supabase.com/changelog Supabase changelog auth sessions delete user access token breaking change", + "query": "site:supabase.com/changelog Supabase auth delete user sessions JWT revocation", "pages": [] }, { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user admin deleteUser access token remains valid database RLS session_id auth.sessions\", limit: 10) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user JWT remain valid until expired sign out scope admin deleteUser sessions\", limit: 10) { nodes { title href content } } }", "hasContent": true, "pages": [ { @@ -14251,84 +14165,45 @@ "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/guides/storage/management/delete-objects", - "title": "Delete Objects" - }, - { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" - }, - { - "url": "https://supabase.com/docs/guides/auth/auth-anonymous", - "title": "Anonymous Sign-Ins" + "url": "https://supabase.com/docs/reference/javascript/auth-signout" }, { - "url": "https://supabase.com/docs/guides/auth/users", - "title": "Users" + "url": "https://supabase.com/docs/guides/auth/managing-user-data", + "title": "User Management" }, { - "url": "https://supabase.com/docs/guides/storage/security/ownership", - "title": "Ownership" + "url": "https://supabase.com/docs/guides/auth/signing-keys", + "title": "JWT Signing Keys" }, { - "url": "https://supabase.com/docs/guides/platform/temporary-access", - "title": "Temporary access" + "url": "https://supabase.com/docs/guides/auth/signout", + "title": "Signing out" }, { - "url": "https://supabase.com/docs/guides/resources/glossary", - "title": "Glossary" + "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", + "title": "Should I set a shorter Max-Age parameter on the cookies?" }, - { - "url": "https://supabase.com/docs/guides/auth/third-party/clerk", - "title": "Clerk" - } - ], - "resultChars": 91950 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"auth.admin.deleteUser delete user JavaScript permanently remove user\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", - "hasContent": true, - "pages": [ { "url": "https://supabase.com/docs/reference/javascript/auth-admin-deleteuser" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-passkey-delete" + "url": "https://supabase.com/docs/guides/auth/server-side/advanced-guide", + "title": "Advanced guide" }, { "url": "https://supabase.com/docs/reference/dart/auth-admin-deleteuser" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-deletefactor" - }, - { - "url": "https://supabase.com/docs/reference/swift/auth-admin-deleteuser", - "title": "deleteUser()" - }, - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, - { - "url": "https://supabase.com/docs/reference/javascript/auth-admin-updateuserbyid" - }, - { - "url": "https://supabase.com/docs/reference/kotlin/auth-admin-deleteuser", - "title": "deleteUser()" + "url": "https://supabase.com/docs/reference/javascript/auth-admin-signout" } ], - "resultChars": 11658 + "resultChars": 69328 } ] }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 2, + "attempts": 1, "sourcePath": "codex-gpt-5.6/investigate-auth-001-deleted-user-access.json" }, { @@ -17041,17 +16916,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identified the root cause: delete_account only set profiles.deleted=true while auth.users and sessions remained active, explaining why sign-in/session access continued. It did not blame caching/client bugs." + "judgeNotes": "The assistant clearly identified that delete_account only soft-deleted the profiles row, did not remove the auth user or sessions, and explained that this allowed continued authentication/access." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant changed the flow from soft delete to deleting the auth.users record, which cascades/removes profile and sessions, and added RLS denial for stale JWTs. This satisfies deleting auth identity/sessions and revoking refresh/session access." + "judgeNotes": "The assistant replaced the soft-delete RPC with deletion from auth.users, which cascades sessions, refresh tokens, identities, and profile removal, and added RLS defenses for stale JWTs. This satisfies revoking auth access rather than leaving a soft delete." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that JWTs remain cryptographically valid until expiration and caveats purely local JWT validation. Its shipped fix uses RLS/active profile checks and hard-deletes auth.users, and it claims no database table window after commit, which is consistent with the implemented RLS denial for the data path." + "judgeNotes": "The assistant correctly identifies JWT staleness and ships RLS changes that close the Data API path by requiring an existing non-deleted profile, so its claim of no JWT-expiration window for database access is consistent with the fix. It also caveats concurrent in-flight requests. It does not explicitly state that the token itself remains valid until expiry for purely local validation, but its explanation is sufficient for the implemented Data API window judgment." } ], "skills": { @@ -17062,66 +16937,31 @@ "calls": [ { "source": "search_docs", - "query": "query { searchDocs(query: \"delete user auth.users access token remains valid until JWT expires revoke sessions admin delete user\", limit: 5) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", + "query": "query { searchDocs(query: \"delete user auth.users access token JWT remains valid until expiration RLS\", limit: 5) { nodes { title href content } } }", "hasContent": true, "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/managing-user-data", - "title": "User Management" - }, { "url": "https://supabase.com/docs/guides/auth/sessions", "title": "User sessions" }, { - "url": "https://supabase.com/docs/guides/platform/sso/multiple-providers", - "title": "Multiple SSO Providers" - }, - { - "url": "https://supabase.com/docs/guides/auth/signing-keys", - "title": "JWT Signing Keys" - }, - { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" - } - ], - "resultChars": 107892 - }, - { - "source": "search_docs", - "query": "query { searchDocs(query: \"sign out revoke refresh tokens access token valid until expires JWT\", limit: 8) { nodes { ... on Guide { title href content } ... on ClientLibraryFunctionReference { title href content language methodName } ... on TroubleshootingGuide { title href content } } } }", - "hasContent": true, - "pages": [ - { - "url": "https://supabase.com/docs/guides/auth/sessions", - "title": "User sessions" + "url": "https://supabase.com/docs/guides/auth/oauth-server/token-security", + "title": "Token Security and Row Level Security" }, { - "url": "https://supabase.com/docs/guides/auth/oauth-server/oauth-flows", - "title": "OAuth 2.1 Flows" + "url": "https://supabase.com/docs/guides/realtime/authorization", + "title": "Realtime Authorization" }, { "url": "https://supabase.com/docs/guides/troubleshooting/should-i-set-a-shorter-max-age-parameter-on-the-cookies-8sbF4V", "title": "Should I set a shorter Max-Age parameter on the cookies?" }, { - "url": "https://supabase.com/docs/reference/javascript/auth-signout" - }, - { - "url": "https://supabase.com/docs/guides/auth/server-side/advanced-guide", - "title": "Advanced guide" - }, - { - "url": "https://supabase.com/docs/guides/integrations/build-a-supabase-oauth-integration", - "title": "Build a Supabase Integration" - }, - { - "url": "https://supabase.com/docs/guides/resources/glossary", - "title": "Glossary" + "url": "https://supabase.com/docs/guides/auth/users", + "title": "Users" } ], - "resultChars": 79809 + "resultChars": 50790 } ] }, @@ -18672,17 +18512,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only soft-deleted the public.profiles row, did not delete auth.users or sessions/refresh tokens, and explains that this is why the account could still sign in." + "judgeNotes": "The assistant clearly identifies the root cause: delete_account only soft-deletes public.profiles via a deleted flag, never deletes auth.users or revokes auth.sessions/refresh_tokens, so login and sessions remain valid. It does not blame caching/client bugs; it explains why the deleted account still works." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant replaced the soft-delete flow with deletion of the auth.users row, which cascades to sessions, refresh tokens, identities, and profiles, and explicitly deletes notes. It also revoked public/anon execute and verified sessions/refresh tokens were removed." + "judgeNotes": "The assistant rewrote delete_account to hard-delete auth.users, cascading to sessions, refresh_tokens, identities, and profiles, and also added RLS session validation to block still-unexpired JWTs. This satisfies revocation of auth access rather than leaving a soft delete." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that pre-existing access tokens are stateless JWTs validated by signature and remain valid until expiry for local validation. Its shipped fix deletes auth users/sessions/refresh tokens and notes, but does not add RLS/session-existence checks, so it appropriately states a remaining window up to JWT expiry and names valid mitigations: shorter JWT expiry and RLS checking auth.sessions." + "judgeNotes": "Explains JWTs are stateless and remain cryptographically valid until exp; local-only validation would still accept them. Its shipped fix adds RLS session-existence checks, so claiming no data-path window is consistent, while still noting the token remains valid until expiry and naming mitigations/caveats." } ], "skills": { @@ -20305,17 +20145,17 @@ { "name": "diagnosed the soft-delete as the cause", "passed": true, - "judgeNotes": "The assistant explicitly identifies that delete_account only updated/soft-deleted the public.profiles row via a deleted flag, while auth.users, auth.sessions, and auth.refresh_tokens remained intact, allowing continued sign-in/token refresh. It does not blame caching/client bugs and explains why the deleted account still worked." + "judgeNotes": "Identifies the root cause: delete_account only marked public.profiles.deleted=true and did not delete auth.users or revoke identities/sessions/refresh tokens, so credentials and sessions remained valid and the account could keep signing in." }, { "name": "fixed the flow to revoke auth access", "passed": true, - "judgeNotes": "The assistant rewrote the flow to delete the auth.users row, which cascades/removes identities, sessions, and refresh tokens, and also hardened RLS against stale JWT access. This satisfies the requirement to revoke auth access rather than soft-delete only." + "judgeNotes": "The assistant replaced the soft-delete flow with deletion from auth.users and explicitly verified identities, sessions, and refresh tokens are cascade-deleted/revoked." }, { "name": "explained the access-token window consistently with its fix", "passed": true, - "judgeNotes": "The assistant correctly explains that access tokens are stateless JWTs that remain cryptographically valid until expiry and that purely local JWT validation would still accept them. Its shipped fix adds RLS checks against a live profile row, so its claim of effectively no data API window is consistent with the implementation, while still noting the remaining JWT-expiry window for local validation and naming short TTL as mitigation." + "judgeNotes": "The assistant correctly explains that access tokens/JWTs remain valid under local stateless validation until expiry, and that its shipped fix deletes auth users/refresh tokens but still leaves the Data API accepting stale JWTs under existing RLS. It states the remaining window is up to access-token expiry and names mitigations such as shorter JWT expiry and server-side/RLS existence checks." } ], "skills": { @@ -20327,7 +20167,7 @@ }, "prompt": "Last week support removed a user through our app's delete-account flow — the\napp calls the `delete_account` function over RPC as the signed-in user. This\nmorning that same person was back: still signed in, reading and saving their\ndata like nothing happened.\n\nFigure out why the account still works, fix the flow so a deleted account\nloses access, and tell me whether there is any window where they could still\nget in after the fix.", "promptSourcePath": "evals/investigate-auth-001-deleted-user-access/PROMPT.md", - "attempts": 1, + "attempts": 2, "sourcePath": "opencode-kimi-k3-no-skills/investigate-auth-001-deleted-user-access.json" }, { diff --git a/apps/web/src/data/regression-eval-results.json b/apps/web/src/data/regression-eval-results.json index 65f4b0fc..a2472811 100644 --- a/apps/web/src/data/regression-eval-results.json +++ b/apps/web/src/data/regression-eval-results.json @@ -293,17 +293,17 @@ { "name": "mapped legacy anon/service_role onto publishable/secret", "passed": false, - "judgeNotes": "The mapping is correct (anon → publishable, service_role → secret), but the answer also describes them as “direct drop-in” replacements, which the rubric says should fail rather than framing them only as role equivalents." + "judgeNotes": "It maps anon→publishable and service_role→secret correctly, but it also presents the migration as a mechanical swap/rename-reissue, implying drop-in replacement despite the new keys not being interchangeable JWTs." }, { "name": "placed the publishable key in the frontend with RLS still applying", - "passed": false, - "judgeNotes": "The answer correctly says publishable keys belong in frontend code and respect RLS, and does not claim they authenticate a user or bypass RLS. However, it does not explicitly describe the publishable key as identifying the app rather than a user, nor clearly state that requests without a user access token run as anon while requests with one run as authenticated and RLS applies in both cases." + "passed": true, + "judgeNotes": "The assistant clearly states the publishable key belongs in frontend code and that requests made with it have RLS fully enforced. It does not claim the publishable key bypasses RLS or authenticates a user, and it keeps the secret key backend-only." }, { "name": "kept the secret key server-only and RLS-bypassing", "passed": true, - "judgeNotes": "The assistant maps secret keys to service_role, states they are backend/server-only, says they bypass RLS entirely, and warns never to expose them in client code." + "judgeNotes": "The assistant maps sb_secret to legacy service_role, states it is backend/server-only, says it bypasses RLS, and warns never to expose it in public clients." } ], "skills": { @@ -318,32 +318,15 @@ "docs": { "calls": [ { - "source": "search_docs", - "query": "{ searchDocs(query: \"publishable key secret key anon service_role migration RLS\", limit: 5) { nodes { title href content } } }", + "source": "web_fetch", + "query": "Summarize the difference between publishable and secret API keys vs legacy anon/service_role keys, which is meant for frontend/browser use, which is meant for backend/server use, and how they interact with Row Level Security (RLS).", "hasContent": true, "pages": [ { - "url": "https://supabase.com/docs/guides/getting-started/migrating-to-new-api-keys", - "title": "Migrating to publishable and secret API keys" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-redwoodjs", - "title": "Build a User Management App with RedwoodJS" - }, - { - "url": "https://supabase.com/docs/guides/getting-started/tutorials/with-sveltekit", - "title": "Build a User Management App with SvelteKit" - }, - { - "url": "https://supabase.com/docs/guides/auth/jwt-fields", - "title": "JWT Claims Reference" - }, - { - "url": "https://supabase.com/docs/guides/auth/quickstarts/with-expo-react-native-social-auth", - "title": "Build a Social Auth App with Expo React Native" + "url": "https://supabase.com/docs/guides/api/api-keys.md" } ], - "resultChars": 149036 + "resultChars": 1286 } ] }, @@ -842,18 +825,18 @@ "checks": [ { "name": "mapped legacy anon/service_role onto publishable/secret", - "passed": true, - "judgeNotes": "Maps legacy anon to publishable and service_role to secret as role equivalents, while noting conceptual equivalence and behavioral differences rather than merely exact renames." + "passed": false, + "judgeNotes": "The mapping is correct (anon → publishable, service_role → secret), but the answer repeatedly frames them as direct replacements / just swapping keys, which the rubric says should fail because the new keys are not exact drop-in interchangeable replacements." }, { "name": "placed the publishable key in the frontend with RLS still applying", - "passed": false, - "judgeNotes": "The answer correctly says the publishable key belongs in frontend code, identifies the app/request rather than a user, and does not bypass RLS. However, it does not state the required distinction that requests without a user access token run as the anon role while requests with a user access token run as the authenticated role, with RLS applying in both cases." + "passed": true, + "judgeNotes": "The answer correctly says the publishable key belongs in frontend/client code and that RLS is fully enforced for requests made with it. It also correctly keeps the secret key/server role backend-only." }, { "name": "kept the secret key server-only and RLS-bypassing", "passed": true, - "judgeNotes": "The assistant states the secret key replaces/maps to legacy service_role, must be server-only/never exposed client-side, and bypasses RLS entirely. It does not recommend putting secret/service_role keys in frontend code or claim RLS applies to the secret key." + "judgeNotes": "The assistant states the secret key replaces/maps to service_role, must be backend/server-only and never shipped to clients, and bypasses RLS." } ], "skills": {