Skip to content

fix: don't count failing phase as completed in dax benchmark#195

Merged
HeyGarrison merged 1 commit into
masterfrom
fix/dax-phase-counting
Jul 17, 2026
Merged

fix: don't count failing phase as completed in dax benchmark#195
HeyGarrison merged 1 commit into
masterfrom
fix/dax-phase-counting

Conversation

@HeyGarrison

Copy link
Copy Markdown
Collaborator

The script's phase() function emits BENCH_PHASE even for the failing phase (it prints the timing before checking the exit code). When there's an error, the last phase that emitted a BENCH_PHASE line is the one that failed, so we now subtract 1 from phasesCompleted.

Before: E2B shows 7/7 phases when typecheck fails (0% success)
After: E2B shows 6/7 phases when typecheck fails

The script's phase() function emits BENCH_PHASE even for the failing
phase (it prints timing before checking exit code). When there's an
error, subtract 1 from phasesCompleted so e.g. E2B shows 6/7 instead
of 7/7 when typecheck fails.
@open-cla

open-cla Bot commented Jul 17, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.7 0.01s 0.05s 0.05s 10/10
2 northflank 98.8 0.08s 0.18s 0.18s 10/10
3 declaw 98.6 0.08s 0.23s 0.23s 10/10
4 daytona 98.1 0.13s 0.28s 0.28s 10/10
5 archil 98.0 0.15s 0.28s 0.28s 10/10
6 blaxel 97.3 0.20s 0.38s 0.38s 10/10
7 upstash 96.5 0.30s 0.43s 0.43s 10/10
8 lightning 96.4 0.24s 0.53s 0.53s 10/10
9 createos 96.3 0.27s 0.53s 0.53s 10/10
10 vercel 95.6 0.35s 0.58s 0.58s 10/10
11 e2b 94.5 0.40s 0.78s 0.78s 10/10
12 superserve 94.3 0.40s 0.84s 0.84s 10/10
13 modal 94.0 0.49s 0.75s 0.75s 10/10
14 runloop 93.5 0.51s 0.87s 0.87s 10/10
15 cloud-run 92.4 0.49s 1.17s 1.17s 10/10
16 tensorlake 91.7 0.78s 0.90s 0.90s 10/10
17 beam 91.3 0.40s 1.57s 1.57s 10/10
18 cloudflare 77.3 1.91s 2.80s 2.80s 10/10
19 codesandbox 73.5 2.56s 2.78s 2.78s 10/10
20 hopx 28.6 4.88s 5.77s 5.77s 6/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.03s 0.03s 10/10
2 northflank 99.1 0.08s 0.11s 0.11s 10/10
3 declaw 98.9 0.10s 0.13s 0.13s 10/10
4 archil 98.5 0.13s 0.19s 0.19s 10/10
5 daytona 98.1 0.10s 0.31s 0.31s 10/10
6 blaxel 97.8 0.21s 0.24s 0.24s 10/10
7 createos 97.4 0.25s 0.28s 0.28s 10/10
8 lightning 97.0 0.21s 0.42s 0.42s 10/10
9 beam 96.2 0.37s 0.39s 0.39s 10/10
10 upstash 95.9 0.33s 0.53s 0.53s 10/10
11 superserve 95.7 0.38s 0.49s 0.49s 10/10
12 vercel 94.8 0.36s 0.75s 0.75s 10/10
13 runloop 94.4 0.54s 0.59s 0.59s 10/10
14 modal 94.1 0.50s 0.71s 0.71s 10/10
15 e2b 91.9 0.40s 1.43s 1.43s 10/10
16 tensorlake 91.7 0.76s 0.94s 0.94s 10/10
17 cloud-run 89.5 0.54s 1.81s 1.81s 10/10
18 cloudflare 80.7 1.62s 2.41s 2.41s 10/10
19 codesandbox 71.5 2.53s 3.33s 3.33s 10/10
20 hopx 16.8 7.12s 9.09s 9.09s 8/10

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.02s 0.02s 0.02s 10/10
2 northflank 98.8 0.12s 0.12s 0.12s 10/10
3 daytona 98.6 0.13s 0.16s 0.16s 10/10
4 archil 98.0 0.19s 0.23s 0.23s 10/10
5 declaw 97.7 0.21s 0.26s 0.26s 10/10
6 lightning 97.6 0.23s 0.25s 0.25s 10/10
7 blaxel 97.6 0.22s 0.27s 0.27s 10/10
8 superserve 96.2 0.37s 0.39s 0.39s 10/10
9 vercel 95.7 0.34s 0.56s 0.56s 10/10
10 e2b 95.0 0.45s 0.57s 0.57s 10/10
11 upstash 94.8 0.47s 0.58s 0.58s 10/10
12 beam 94.6 0.42s 0.72s 0.72s 10/10
13 runloop 94.0 0.57s 0.64s 0.64s 10/10
14 modal 93.8 0.55s 0.74s 0.74s 10/10
15 createos 93.2 0.28s 1.27s 1.27s 10/10
16 cloud-run 92.9 0.53s 0.98s 0.98s 10/10
17 tensorlake 92.0 0.80s 0.80s 0.80s 10/10
18 codesandbox 68.8 2.68s 3.77s 3.77s 10/10
19 cloudflare 64.0 3.24s 4.14s 4.14s 10/10
20 hopx 0.0 12.85s 14.11s 14.11s 7/10

Sandbox Dax

Provider Metric Status
archil 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
beam 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
blaxel 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
cloud-run 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
cloudflare 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
codesandbox 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
createos 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
daytona 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
declaw -- phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/1 OK
e2b 6/7 phases · total 18.88s · prepare 12.68s · clone 2.88s · install 1.10s · typecheck 0.03s 6/7 phases
hopx -- phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/1 OK
isorun 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
modal 7/7 phases · total 56.60s · prepare 7.23s · clone 2.58s · install 19.88s · typecheck 18.90s 7/7 phases
northflank 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
runloop 6/7 phases · total 81.00s · prepare 24.13s · clone 4.03s · install 27.65s · typecheck 10.91s 6/7 phases
superserve -- phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/1 OK
tensorlake 6/7 phases · total 19.62s · prepare 12.52s · clone 2.17s · install 3.50s · typecheck 0.03s 6/7 phases
upstash 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases
vercel 0/7 phases · total 0.00s · prepare 0.00s · clone 0.00s · install 0.00s · typecheck 0.00s 0/7 phases

View full run · SVGs available as build artifacts

@HeyGarrison
HeyGarrison merged commit 8b87c0b into master Jul 17, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant