[WIP] Scenic Parallelization and Simulator Stepping by Eric-Vin · Pull Request #422 · BerkeleyLearnVerify/Scenic

Eric-Vin · 2025-11-21T20:52:00Z

Description

Adds parallelization to scene sampling and simulation.

Issue Link

N/A

Checklist

I have tested the changes locally via pytest and/or other means
I have added or updated relevant documentation
I have autoformatted the code with black and isort
I have added test cases (if applicable)

Additional Notes

N/A

Eric-Vin · 2025-11-21T20:52:47Z

Benchmark results for parallelized scene sampling on Macbook with M4 Pro:

Base Performance (`generate`, numWorkers=0):
   3.09 | ('adjacentOpposingPair.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   5.95 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  16.80 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   6.12 | ('city_intersection.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   7.84 | ('enclosed_visible.scenic', {})
  80.35 | ('fully_occluded.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   1.23 | ('fully_visible.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
 248.27 | ('narrowGoalNew.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   6.42 | ('narrowGoalOld.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  13.92 | ('pedestrian_02.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  12.85 | ('vacuum.scenic', {'numToys': 0, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  19.26 | ('vacuum.scenic', {'numToys': 1, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  22.96 | ('vacuum.scenic', {'numToys': 2, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  36.70 | ('vacuum.scenic', {'numToys': 4, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  86.58 | ('vacuum.scenic', {'numToys': 8, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})

Base + Overhead Performance (`generateBatch`, numWorkers=1):
   4.09 | ('adjacentOpposingPair.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   6.86 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  15.14 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   6.53 | ('city_intersection.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   7.97 | ('enclosed_visible.scenic', {})
  81.75 | ('fully_occluded.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   1.74 | ('fully_visible.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
 215.58 | ('narrowGoalNew.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   7.09 | ('narrowGoalOld.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  14.74 | ('pedestrian_02.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  19.33 | ('vacuum.scenic', {'numToys': 0, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  23.52 | ('vacuum.scenic', {'numToys': 1, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  31.18 | ('vacuum.scenic', {'numToys': 2, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  44.50 | ('vacuum.scenic', {'numToys': 4, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  98.84 | ('vacuum.scenic', {'numToys': 8, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})

Batch Performance (`generateBatch`, numWorkers=8):
   2.62 | ('adjacentOpposingPair.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   2.92 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   4.49 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   2.17 | ('city_intersection.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   2.08 | ('enclosed_visible.scenic', {})
  14.17 | ('fully_occluded.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   0.90 | ('fully_visible.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  48.10 | ('narrowGoalNew.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   2.18 | ('narrowGoalOld.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   4.34 | ('pedestrian_02.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   4.19 | ('vacuum.scenic', {'numToys': 0, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   5.21 | ('vacuum.scenic', {'numToys': 1, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   6.15 | ('vacuum.scenic', {'numToys': 2, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  10.69 | ('vacuum.scenic', {'numToys': 4, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  18.74 | ('vacuum.scenic', {'numToys': 8, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})

codecov · 2025-11-21T21:10:03Z

Codecov Report

❌ Patch coverage is 83.62573% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.68%. Comparing base (2fc1634) to head (069bec8).

Files with missing lines	Patch %	Lines
src/scenic/core/simulators.py	84.91%	35 Missing ⚠️
src/scenic/core/scenarios.py	78.35%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #422      +/-   ##
==========================================
- Coverage   89.75%   89.68%   -0.07%     
==========================================
  Files          48       48              
  Lines       13350    13598     +248     
==========================================
+ Hits        11982    12196     +214     
- Misses       1368     1402      +34

Files with missing lines	Coverage Δ
src/scenic/__init__.py	`100.00% <100.00%> (ø)`
src/scenic/core/utils.py	`77.51% <100.00%> (+0.66%)`	⬆️
src/scenic/syntax/translator.py	`93.67% <100.00%> (+0.09%)`	⬆️
src/scenic/core/scenarios.py	`90.29% <78.35%> (-4.05%)`	⬇️
src/scenic/core/simulators.py	`90.79% <84.91%> (-5.05%)`	⬇️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Eric-Vin · 2025-12-04T23:02:25Z

Preliminary benchmark results for parallelized simulation on Macbook with M4 Pro with Metadrive:

Base Performance (`generate`, numWorkers=0):
 139.95 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
 251.33 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})

Base + Overhead Performance (`generateBatch`, numWorkers=1):
 137.52 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
 140.97 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})

Batch Performance (`generateBatch`, numWorkers=8):
  16.59 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  17.21 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})

Eric-Vin added 4 commits November 6, 2024 16:49

Initial simulator stepping implementation.

ef9d735

Added stepped simulation test.

a452067

Scene sampling parallelization prototype.

07e773b

Scene sampling parallelization.

99f4ac0

Eric-Vin marked this pull request as draft November 21, 2025 20:52

Simulation parallelization

cf3c647

Eric-Vin added 2 commits December 24, 2025 20:55

Added documentation for setSeed.

349727f

Added documentation and tests for simulation parallelization.

335b437

Eric-Vin marked this pull request as ready for review December 25, 2025 19:17

Merge branch 'main' into ScenicParallelization

e5048b4

Eric-Vin changed the title ~~[WIP] Scenic Parallelization~~ Scenic Parallelization Dec 25, 2025

Eric-Vin added 2 commits December 25, 2025 14:10

Merge branch 'main' into SimulatorStepping

61e61b8

xMerge branch 'SimulatorStepping' into ScenicParallelization

9689729

Eric-Vin changed the title ~~Scenic Parallelization~~ Scenic Parallelization and Simulator Stepping Dec 26, 2025

Eric-Vin mentioned this pull request Dec 26, 2025

[WIP] Simulator Stepping Functionality #312

Closed

4 tasks

Eric-Vin mentioned this pull request Apr 15, 2026

Add scenic.setSeed() public API #454

Closed

5 tasks

Eric-Vin added 2 commits April 18, 2026 13:09

Minor documentation and test additions.

f673bb4

Simplified test.

24e5ee9

Eric-Vin changed the title ~~Scenic Parallelization and Simulator Stepping~~ [WIP] Scenic Parallelization and Simulator Stepping Apr 29, 2026

Eric-Vin requested a review from dfremont April 29, 2026 21:49

Eric-Vin added 3 commits July 20, 2026 17:41

Merge branch 'main' into ScenicParallelization

79965e9

Merge branch 'main' into ScenicParallelization

786e48f

Simulation cleanup fix and tests.

069bec8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Scenic Parallelization and Simulator Stepping - #422

[WIP] Scenic Parallelization and Simulator Stepping#422
Eric-Vin wants to merge 15 commits into
mainfrom
ScenicParallelization

Eric-Vin commented Nov 21, 2025

Uh oh!

Eric-Vin commented Nov 21, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Nov 21, 2025 •

edited

Loading

Uh oh!

Eric-Vin commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Eric-Vin commented Nov 21, 2025

Description

Issue Link

Checklist

Additional Notes

Uh oh!

Eric-Vin commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Eric-Vin commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Eric-Vin commented Nov 21, 2025 •

edited

Loading

codecov Bot commented Nov 21, 2025 •

edited

Loading