Skip to content

chore: refresh eval results #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
supabase-evals-releaser wants to merge 1 commit into from
Closed

chore: refresh eval results #116

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions apps/web/src/data/regression-eval-results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,17 +305,17 @@
{
"name": "identified video-thumbnails and the 546 resource-limit pattern",
"passed": true,
"judgeNotes": "The assistant correctly identified `video-thumbnails` as the affected function and explicitly stated the failures are `status_code: 546` due to CPU/resource limits, while ruling out the unrelated `welcome-email` 500."
"judgeNotes": "Identified `video-thumbnails` as affected and correctly recognized failures as HTTP 546 due to CPU/resource limit, while distinguishing unrelated `welcome-email` 500."
},
{
"name": "attributed the 546s to CPU time exhaustion",
"passed": true,
"judgeNotes": "The assistant specifically attributes the 546 failures to CPU time exhaustion and grounds it in log evidence: shutdown reason CPUTime plus cpu_time_used 2000ms at cpu_time_limit 2000ms. It also explicitly rules out unrelated errors."
"judgeNotes": "The assistant specifically attributes the 546 failures to CPU time exhaustion, citing the log signature `shutdown (reason: CPUTime, cpu_time_used: 2000ms, cpu_time_limit: 2000ms)` and distinguishes it from wall-clock time, memory, code exceptions, and the unrelated welcome-email error."
},
{
"name": "recommended reducing/offloading CPU work as the fix",
"passed": true,
"judgeNotes": "The assistant correctly identifies CPUTime limit as root cause and recommends reducing/offloading CPU-intensive thumbnail generation: rejecting/downscaling large inputs, moving work to a background/external worker, and optimizing/profiling the thumbnailing code. It also notes the CPU ceiling cannot be raised."
"judgeNotes": "Recommended moving CPU-heavy thumbnail generation out of the Edge Function and reducing per-call CPU work if kept there."
}
],
"skills": {
Expand Down Expand Up @@ -972,17 +972,17 @@
{
"name": "identified video-thumbnails and the 546 resource-limit pattern",
"passed": true,
"judgeNotes": "Identified `video-thumbnails` as the affected function and correctly recognized HTTP 546 worker/resource/CPU limit errors, not 500/503 server errors."
"judgeNotes": "Identified `video-thumbnails` as the affected function and explicitly described failures as HTTP 546 due to worker CPU/resource limit, not 500/503."
},
{
"name": "attributed the 546s to CPU time exhaustion",
"passed": true,
"judgeNotes": "The assistant specifically attributes the 546 failures to CPU time exhaustion, citing log evidence including `shutdown (reason: CPUTime, cpu_time_used: 2000ms, cpu_time_limit: 2000ms)` and explicitly distinguishing it from wall-clock timeout."
"judgeNotes": "The assistant specifically attributes the 546 failures to CPU time exhaustion, citing log evidence including `shutdown (reason: CPUTime, cpu_time_used: 2000ms, cpu_time_limit: 2000ms)`, and explicitly rules out wall-clock timeout and memory."
},
{
"name": "recommended reducing/offloading CPU work as the fix",
"passed": true,
"judgeNotes": "The assistant explicitly recommends offloading thumbnail generation to a background worker/external container and reducing per-call CPU cost, rather than retrying or raising limits."
"judgeNotes": "The assistant correctly identifies the CPU-time limit and recommends reducing/offloading CPU-intensive thumbnail generation: moving work to a background/external worker, keeping the Edge Function as a thin trigger, and reducing per-call CPU work via seeking/downscaling/rejecting large inputs."
}
],
"skills": {
Expand Down