Skip to content

fix: standardize dax benchmark resources to 8 vCPU / 16 GiB#196

Open
HeyGarrison wants to merge 9 commits into
masterfrom
fix/dax-standardize-resources
Open

fix: standardize dax benchmark resources to 8 vCPU / 16 GiB#196
HeyGarrison wants to merge 9 commits into
masterfrom
fix/dax-standardize-resources

Conversation

@HeyGarrison

@HeyGarrison HeyGarrison commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Standardizes sandbox sizing across providers for fair comparison. Previously Modal was bursting to 17 vCPUs while Tensorlake had 1 vCPU, making results incomparable.

Sized providers (8 vCPU / 16 GiB)

Provider CPU param Memory param
E2B cpuCount=8 memoryMB=16384
Modal cpu=4 (4 physical = 8 vCPUs) memoryMiB=16384
Tensorlake cpus=8 memoryMb=16384
isorun vcpus=8 memMiB=16384
runloop customCpuCores=8 customMemoryGb=16
daytona resources.cpu=8 resources.memory=16 (GiB)
upstash size=large (8 cores, 16 GB preset) -
vercel resources.vcpus=8 no memory control
blaxel memory=16384 (CPU derived: 16384/2048=8) -

Providers without CPU/memory control (use defaults)

archil, cloudflare, northflank, beam, declaw, hopx, codesandbox

@HeyGarrison
fix: standardize dax benchmark resources to 8 vCPU / 16 GiB
ad8d3b0 
Each provider gets 8 vCPUs and 16 GiB RAM for fair comparison:
- E2B: cpuCount=8, memoryMB=16384
- Modal: cpu=4 (4 physical = 8 vCPUs), memoryMiB=16384
- Tensorlake: cpus=8, memoryMb=16384
- isorun: vcpus=8, memMiB=16384
- runloop: customCpuCores=8, customMemoryGb=16
- daytona: resources.cpu=8, resources.memory=16 (GiB)
- upstash: size=large (8 cores, 16 GB)
- vercel: resources.vcpus=8 (no memory control)
- blaxel: memory=16384 (CPU derived: 16384/2048=8 cores)

Providers without CPU/memory control (archil, cloudflare, northflank,
beam, declaw, hopx, codesandbox) use their defaults.
@open-cla

open-cla Bot commented Jul 17, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@github-actions

github-actions Bot commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.04s 0.04s 10/10
2 declaw 99.3 0.05s 0.10s 0.10s 10/10
3 northflank 98.9 0.08s 0.15s 0.15s 10/10
4 archil 98.4 0.14s 0.18s 0.18s 10/10
5 daytona 98.3 0.10s 0.28s 0.28s 10/10
6 lightning 97.5 0.20s 0.32s 0.32s 10/10
7 upstash 97.3 0.26s 0.29s 0.29s 10/10
8 createos 96.8 0.26s 0.41s 0.41s 10/10
9 blaxel 96.3 0.29s 0.48s 0.48s 10/10
10 vercel 95.8 0.32s 0.56s 0.56s 10/10
11 modal 94.5 0.47s 0.68s 0.68s 10/10
12 superserve 94.4 0.44s 0.74s 0.74s 10/10
13 runloop 93.9 0.50s 0.78s 0.78s 10/10
14 cloud-run 93.3 0.49s 0.93s 0.93s 10/10
15 tensorlake 93.2 0.63s 0.75s 0.75s 10/10
16 e2b 91.0 0.84s 1.01s 1.01s 10/10
17 beam 89.3 0.39s 2.11s 2.11s 10/10
18 cloudflare 78.7 1.93s 2.43s 2.43s 10/10
19 hopx 64.4 3.24s 4.04s 4.04s 10/10
20 codesandbox 45.9 2.36s 19.72s 19.72s 10/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 declaw 99.2 0.06s 0.12s 0.12s 10/10
2 northflank 98.9 0.08s 0.15s 0.15s 10/10
3 archil 98.3 0.14s 0.23s 0.23s 10/10
4 daytona 98.2 0.13s 0.27s 0.27s 10/10
5 lightning 97.6 0.20s 0.29s 0.29s 10/10
6 isorun 97.4 0.01s 0.64s 0.64s 10/10
7 createos 97.3 0.26s 0.29s 0.29s 10/10
8 blaxel 96.8 0.29s 0.35s 0.35s 10/10
9 upstash 96.2 0.28s 0.52s 0.52s 10/10
10 beam 96.1 0.37s 0.43s 0.43s 10/10
11 superserve 95.6 0.40s 0.50s 0.50s 10/10
12 vercel 94.5 0.33s 0.89s 0.89s 10/10
13 modal 94.3 0.47s 0.70s 0.70s 10/10
14 tensorlake 92.9 0.63s 0.83s 0.83s 10/10
15 cloud-run 91.7 0.68s 1.06s 1.06s 10/10
16 e2b 90.8 0.87s 0.99s 0.99s 10/10
17 runloop 86.9 0.60s 2.36s 2.36s 10/10
18 cloudflare 78.4 1.81s 2.69s 2.69s 10/10
19 codesandbox 72.8 2.52s 3.03s 3.03s 10/10
20 hopx 35.0 6.41s 6.63s 6.63s 10/10

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.02s 0.02s 0.02s 10/10
2 northflank 98.8 0.11s 0.15s 0.15s 10/10
3 archil 98.3 0.15s 0.20s 0.20s 10/10
4 declaw 98.2 0.16s 0.21s 0.21s 10/10
5 daytona 98.1 0.12s 0.29s 0.29s 10/10
6 lightning 97.9 0.21s 0.22s 0.22s 10/10
7 blaxel 96.6 0.32s 0.35s 0.35s 10/10
8 beam 96.4 0.34s 0.39s 0.39s 10/10
9 createos 96.3 0.28s 0.49s 0.49s 10/10
10 vercel 96.2 0.34s 0.43s 0.43s 10/10
11 upstash 95.8 0.37s 0.50s 0.50s 10/10
12 superserve 95.2 0.47s 0.50s 0.50s 10/10
13 modal 94.7 0.52s 0.53s 0.53s 10/10
14 runloop 94.2 0.55s 0.64s 0.64s 10/10
15 tensorlake 93.1 0.67s 0.72s 0.72s 10/10
16 cloud-run 92.3 0.55s 1.10s 1.10s 10/10
17 e2b 90.3 0.79s 1.24s 1.24s 10/10
18 cloudflare 68.0 2.68s 3.97s 3.97s 10/10
19 codesandbox 66.5 3.08s 3.76s 3.76s 10/10
20 hopx 27.7 7.03s 7.52s 7.52s 10/10

View full run · SVGs available as build artifacts

@github-actions

github-actions Bot commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
beam 7/7 55.18s 21.78s 1.78s 9.34s 11.95s 1/1 OK
blaxel 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
cloud-run 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
daytona 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
declaw -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
e2b 7/7 94.68s 15.55s 3.19s 20.12s 46.67s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
lightning -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
modal 7/7 104.13s 7.25s 2.30s 21.35s 65.33s 1/1 OK
namespace 7/7 28.22s 3.34s 1.23s 7.29s 12.54s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
runloop 7/7 84.48s 22.58s 3.33s 16.02s 37.00s 1/1 OK
superserve -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
tensorlake 7/7 89.99s 14.99s 1.80s 12.56s 52.42s 1/1 OK
upstash -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
vercel 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK

View full run

@HeyGarrison
fix: remove E2B from resource options - CPU/mem set at template build…
27f9ee4 
… time

E2B's Sandbox.create() accepts SandboxOpts which does not include
cpuCount/memoryMB. Those are template build options only.
@shivam-declaw

shivam-declaw commented Jul 18, 2026

Copy link
Copy Markdown

For declaw, the cpu, memory and disk are property of the template, you can create a custom template with the required size or we can have a node-large template with above spec

@shivam-declaw shivam-declaw mentioned this pull request Jul 18, 2026
Merged
HeyGarrison and others added 5 commits July 18, 2026 01:56
@HeyGarrison
fix: standardize dax resource specs for all providers to 8 vCPU / 16 GiB
bca8ec7 
- Fix runloop: nest resource params in launch_parameters with snake_case
- Add beam: cpu=8, memory=16384 MiB
- Add codesandbox: vmTier=VMTier.Small (8 CPU / 16 GiB)
- Add northflank: deploymentPlan resolved via API at runtime
- Add E2B template build prereq (base-8cpu-16gb with 8 vCPU / 16 GiB)
- Add Northflank plan discovery prereq script
@HeyGarrison
feat: local benchmark script with apt/dnf and arch detection
19740f5 
- Add scripts/provider-benchmark.sh: vendored from upstream with dnf
  support (RHEL/Fedora) alongside apt-get (Debian/Ubuntu) and aarch64
  architecture detection for Node.js and Bun downloads
- Update dax.ts to load script from local filesystem instead of curl,
  eliminating curl dependency for providers that don't ship it
@HeyGarrison
rename: provider-benchmark.sh to dax-benchmark.sh
5f71355
@HeyGarrison
feat: add namespace provider to dax benchmark (16 vCPU / 32 GiB)
d6497ad
@shivam-declaw @claude
dax: standardize declaw resources via node-large template (#197)
f0efe46 
Adds declaw to DAX_RESOURCE_OPTIONS so it targets the same 8 vCPU / 16 GiB profile as the other providers in the map. The node-large template provisions 8 vCPU / 16 GiB RAM with an 8 GiB disk overlay.

Co-authored-by: shivam-declaw <272314190+shivam-declaw@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
superagent-security[bot]
superagent-security Bot reviewed Jul 18, 2026
View reviewed changes
Comment thread scripts/find-northflank-plan.ts
superagent-security[bot]
superagent-security Bot reviewed Jul 18, 2026
View reviewed changes
Comment thread scripts/find-northflank-plan.ts
@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jul 18, 2026
@HeyGarrison
fix: bun zip path, northflank API fallback, daytona snapshot, namespa…
c7f4c47 
…ce node image

- Fix BUN_INTERNAL_PATH: remove .zip extension from zip-internal path
  (was bun-linux-x64-baseline.zip/bun, should be bun-linux-x64-baseline/bun)
- Northflank: robust API response parsing, graceful fallback to nf-compute-50,
  sanitize plan ID before writing to GITHUB_ENV
- Daytona: remove from DAX_RESOURCE_OPTIONS (resources not supported with
  snapshot-based sandbox creation)
- Namespace: add sandboxOptions image node:22 (dax benchmark requires node)
@superagent-security superagent-security Bot removed the pr:flagged PR flagged for review by security analysis. label Jul 18, 2026
@HeyGarrison
fix: northflank smallest plan selection + E2B command timeout workaround
58b9d5e 
- Northflank: pick smallest suitable plan instead of closest to reduce
  chance of exceeding project resource allowance
- E2B: workaround for 60s command timeout by setting
  defaultProcessConnectionTimeout=0 via getInstance() while waiting
  for upstream wrapper fix to forward timeoutMs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@superagent-security superagent-security[bot] superagent-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@HeyGarrison @shivam-declaw