diff --git a/ScratchV-topic06-deliverable/.github/workflows/benchmark.yml b/ScratchV-topic06-deliverable/.github/workflows/benchmark.yml new file mode 100644 index 0000000..6e4bffd --- /dev/null +++ b/ScratchV-topic06-deliverable/.github/workflows/benchmark.yml @@ -0,0 +1,41 @@ +name: ScratchV Course Benchmark Suite + +on: + push: + pull_request: + +jobs: + benchmark: + runs-on: ubuntu-latest + + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install package and course benchmark dependencies + run: | + python -m pip install --upgrade pip + python -m pip install -e . + python -m pip install tinyfive pytest jinja2 matplotlib + + - name: Run unit tests + run: python -m pytest -q + + - name: Run course benchmark suite + run: | + python run_tests.py --benchmark 3 + + - name: Upload course benchmark reports + uses: actions/upload-artifact@v4 + with: + name: scratchv-course-benchmark-reports + path: | + reports/report.md + reports/report.html + reports/course_report_instructions.png + reports/benchmark_baseline.json diff --git a/ScratchV-topic06-deliverable/LICENSE b/ScratchV-topic06-deliverable/LICENSE new file mode 100644 index 0000000..87324d7 --- /dev/null +++ b/ScratchV-topic06-deliverable/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 ScratchV + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/ScratchV-topic06-deliverable/README.md b/ScratchV-topic06-deliverable/README.md new file mode 100644 index 0000000..10c5879 --- /dev/null +++ b/ScratchV-topic06-deliverable/README.md @@ -0,0 +1,123 @@ +# ScratchV 课题 06 交付说明 +本测试套件不是独立项目,需要放在 ScratchV 仓库根目录下运行,并依赖 ScratchV 原项目环境。 +本目录是 ScratchV 课题 06 的性能基准与验证测试套件,包含 23 个 DSL测试用例、自动化测试脚本、依赖说明以及可选的测试报告。 +其中3个branch中的三个if分支调用tinyfive时目前还不通过,是因为当前 ScratchV 的分支汇编和 tinyfive 适配层没有对齐。 + +## 运行依赖 + +该交付目录需要放在 ScratchV 项目环境中运行。`run_tests.py` 会导入 +`scratchv` 包,并通过 ScratchV 编译器把每个 DSL 用例编译为 RISC-V 汇编。 +若调用真实的tinyfive而非stub需要安装两个源码补丁: +https://github.com/ScratchV-Compiler/ScratchV/pull/15 +https://github.com/ScratchV-Compiler/ScratchV/pull/17 +安装额外依赖: + +```powershell +pip install -r requirements-topic06.txt +``` + +至少需要以下依赖: + +- `tinyfive`:用于模拟执行生成的 RISC-V 汇编 +- `pytest`:用于测试支持 +- `jinja2`:用于生成 HTML 报告 +- `matplotlib`:用于生成性能图表 + +## 目录结构 + +```text +run_tests.py +requirements-topic06.txt +tests_main/ + activation/ + branch/ + elementwise/ + loop/ + reduction/ + tensor/ +reports/ # 测试报告,可重新生成 +build/ # 编译输出的汇编文件,可重新生成 +``` + +`tests_main/` 下共有 23 个 DSL 用例。每个用例由 `.dsl` 文件和对应的 +`.meta.json` 文件组成,`.meta.json` 中定义输入、期望返回值和用例说明。 + +## 运行测试 + +在本目录下执行: + +```powershell +python run_tests.py +``` + +测试脚本会自动完成以下步骤: + +- 遍历 `tests_main/` 下的所有 `.dsl` 文件 +- 调用 ScratchV 编译器生成 RISC-V 汇编 +- 通过 ScratchV 的 TinyFive 适配层验证生成的汇编 +- 根据 `.meta.json` 中的期望值进行结果对比 +- 增加时间测量(time.perf_counter),输出到报告 +- 统计 PASS/FAIL 和指令数 +- 在 `reports/` 下生成 Markdown 和 HTML 报告 + +## Benchmark 模式 + +重复运行每个用例并统计平均指令数: + +```powershell +python run_tests.py --benchmark 3 +``` + +更新性能基线: + +```powershell +python run_tests.py --benchmark 3 --update-baseline +``` + +性能基线文件位于: + +```text +reports/benchmark_baseline.json +``` + +## 测试报告 + +运行后会生成: + +```text +reports/report.md +reports/report.html +reports/course_report_instructions.png +``` + +`report.md` 适合提交或归档,`report.html` 适合在浏览器中查看测试结果。 + +## 添加测试用例 + +新增用例时添加两个文件: + +```text +tests_main/{category}/{name}.dsl +tests_main/{category}/{name}.meta.json +``` + +DSL 示例: + +```text +result = add(a, b) +return result +``` + +元数据示例: + +```json +{ + "description": "Simple scalar add.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3 + }, + "expected_return": 5 +} +``` \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/reports/benchmark_baseline.json b/ScratchV-topic06-deliverable/reports/benchmark_baseline.json new file mode 100644 index 0000000..6c11a38 --- /dev/null +++ b/ScratchV-topic06-deliverable/reports/benchmark_baseline.json @@ -0,0 +1,117 @@ +{ + "add_relu_relu": { + "category": "activation", + "avg_instr_count": 7.0, + "runs": 3 + }, + "relu_add": { + "category": "activation", + "avg_instr_count": 5.0, + "runs": 3 + }, + "relu_only": { + "category": "activation", + "avg_instr_count": 4.0, + "runs": 3 + }, + "relu_twice": { + "category": "activation", + "avg_instr_count": 6.0, + "runs": 3 + }, + "if_else": { + "category": "branch", + "avg_instr_count": 0.0, + "runs": 3 + }, + "if_relu": { + "category": "branch", + "avg_instr_count": 0.0, + "runs": 3 + }, + "if_then": { + "category": "branch", + "avg_instr_count": 0.0, + "runs": 3 + }, + "add_chain": { + "category": "elementwise", + "avg_instr_count": 4.0, + "runs": 3 + }, + "add_chain_3": { + "category": "elementwise", + "avg_instr_count": 5.0, + "runs": 3 + }, + "add_fan_in_4": { + "category": "elementwise", + "avg_instr_count": 5.0, + "runs": 3 + }, + "add_reuse": { + "category": "elementwise", + "avg_instr_count": 4.0, + "runs": 3 + }, + "vector_add": { + "category": "elementwise", + "avg_instr_count": 3.0, + "runs": 3 + }, + "loop_add_4": { + "category": "loop", + "avg_instr_count": 22.0, + "runs": 3 + }, + "loop_add_chain_4": { + "category": "loop", + "avg_instr_count": 26.0, + "runs": 3 + }, + "loop_relu_add_4": { + "category": "loop", + "avg_instr_count": 30.0, + "runs": 3 + }, + "dot_4": { + "category": "reduction", + "avg_instr_count": 3.0, + "runs": 3 + }, + "dot_8": { + "category": "reduction", + "avg_instr_count": 3.0, + "runs": 3 + }, + "dot_relu_4": { + "category": "reduction", + "avg_instr_count": 5.0, + "runs": 3 + }, + "dot_relu_8": { + "category": "reduction", + "avg_instr_count": 5.0, + "runs": 3 + }, + "matmul_2x2": { + "category": "tensor", + "avg_instr_count": 3.0, + "runs": 3 + }, + "matmul_4x4": { + "category": "tensor", + "avg_instr_count": 3.0, + "runs": 3 + }, + "matmul_add_2x2": { + "category": "tensor", + "avg_instr_count": 4.0, + "runs": 3 + }, + "matmul_relu_2x2": { + "category": "tensor", + "avg_instr_count": 5.0, + "runs": 3 + } +} \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/reports/course_report_instructions.png b/ScratchV-topic06-deliverable/reports/course_report_instructions.png new file mode 100644 index 0000000..842bcde Binary files /dev/null and b/ScratchV-topic06-deliverable/reports/course_report_instructions.png differ diff --git a/ScratchV-topic06-deliverable/reports/report.html b/ScratchV-topic06-deliverable/reports/report.html new file mode 100644 index 0000000..5ab96da --- /dev/null +++ b/ScratchV-topic06-deliverable/reports/report.html @@ -0,0 +1,383 @@ + + + + + ScratchV 测试报告 + + + +

ScratchV DSL 编译器性能测试报告

+
+

用例总数:23,通过:20,失败:3,通过率:87.0%

+

测试目录:D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\tests_main,性能退化阈值:5.0%,单次模拟超时:5.0s

+
+ 课程版指令数图表 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
用例类别状态模拟后端平均指令数95% 置信区间编译耗时(s)模拟耗时(s)总耗时(s)变化率(%)是否退化描述
add_relu_reluactivationPASSnone7.00±0.000.05320.13230.48060.00FalseAdd input and bias, then apply ReLU twice.
relu_addactivationPASSnone5.00±0.000.05640.09810.44560.00FalseAdd input and bias, then apply one ReLU.
relu_onlyactivationPASSnone4.00±0.000.05160.10000.43590.00FalseApply ReLU directly to a single input value.
relu_twiceactivationPASSnone6.00±0.000.05330.09280.42650.00FalseApply ReLU twice to the same activation path.
if_elsebranchFAILtimeout0.00±0.000.05125.016820.09350.00Falseif/else branch returns subtraction result when flag is zero.
if_relubranchFAILtimeout0.00±0.000.05385.009020.10860.00Falseif/else branch combined with add and relu.
if_thenbranchFAILtimeout0.00±0.000.05495.009520.10930.00Falseif/else branch returns add result when flag is non-zero.
add_chainelementwisePASSnone4.00±0.000.05480.09690.43640.00FalseAdd a and b, then add c to the intermediate result.
add_chain_3elementwisePASSnone5.00±0.000.05250.09710.43570.00FalseChain three add operations across four symbolic inputs.
add_fan_in_4elementwisePASSnone5.00±0.000.05230.10050.43930.00FalseCompute two independent adds and then merge them with a final add.
add_reuseelementwisePASSnone4.00±0.000.05230.09350.43200.00FalseReuse the same intermediate add result on both operands of a second add.
vector_addelementwisePASSnone3.00±0.000.05350.09450.43630.00FalseSingle add over two symbolic vector inputs.
loop_add_4loopPASSnone22.00±0.000.05200.09410.43120.00FalseRun a four-iteration loop whose body computes one add; final returned value is the last loop-body result.
loop_add_chain_4loopPASSnone26.00±0.000.05340.09500.43750.00FalseRun a four-iteration loop whose body computes two chained adds; final returned value is the last loop-body result.
loop_relu_add_4loopPASSnone30.00±0.000.05110.09500.43020.00FalseRun a four-iteration loop whose body computes add followed by ReLU; final returned value is the last loop-body result.
dot_4reductionPASSnone3.00±0.000.05440.09390.43620.00FalseCompute the dot product of two symbolic vectors of length 4.
dot_8reductionPASSnone3.00±0.000.05370.10210.44740.00FalseCompute the dot product of two symbolic vectors of length 8.
dot_relu_4reductionPASSnone5.00±0.000.05230.09540.43420.00FalseCompute a length-4 dot product and pass it through ReLU.
dot_relu_8reductionPASSnone5.00±0.000.05240.09450.43500.00FalseCompute a length-8 dot product and pass it through ReLU.
matmul_2x2tensorPASSnone3.00±0.000.05180.09520.43280.00FalseCompute a symbolic 2x2 by 2x2 matrix multiplication.
matmul_4x4tensorPASSnone3.00±0.000.05290.09420.44000.00FalseCompute a symbolic 4x4 by 4x4 matrix multiplication.
matmul_add_2x2tensorPASSnone4.00±0.000.05220.09400.43160.00FalseCompute a 2x2 matmul and then add a symbolic bias term.
matmul_relu_2x2tensorPASSnone5.00±0.000.05140.09660.43420.00FalseCompute a 2x2 matmul and then apply ReLU to its result.
+ + \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/reports/report.md b/ScratchV-topic06-deliverable/reports/report.md new file mode 100644 index 0000000..60cc532 --- /dev/null +++ b/ScratchV-topic06-deliverable/reports/report.md @@ -0,0 +1,588 @@ +# ScratchV DSL 编译器性能测试报告 + +## 测试概览 + +- 生成时间: 2026-07-07 18:48:01 +- 用例总数: 23 +- 通过数量: 20 +- 失败数量: 3 +- 通过率: 87.0% +- 测试目录: `D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\tests_main` +- 汇编输出目录: `D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build` +- 性能基线文件: `D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\reports\benchmark_baseline.json` +- 性能退化阈值: 5.0% +- 单次模拟超时: 5s + +## 测试结果 + +| 用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 95% 置信区间 | 最小 | 最大 | 编译耗时(s) | 模拟耗时(s) | 总耗时(s) | 基线 | 变化率(%) | 是否退化 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 | +|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|---|---|---|---| +| add_relu_relu | activation | PASS | none | 7.00 | ±0.00 | 7 | 7 | 0.0532 | 0.1323 | 0.4806 | 7.00 | 0.00 | False | 7 | 7 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_relu_relu.s | +| relu_add | activation | PASS | none | 5.00 | ±0.00 | 5 | 5 | 0.0564 | 0.0981 | 0.4456 | 5.00 | 0.00 | False | 3 | 3 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\relu_add.s | +| relu_only | activation | PASS | none | 4.00 | ±0.00 | 4 | 4 | 0.0516 | 0.1000 | 0.4359 | 4.00 | 0.00 | False | 0 | 0 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\relu_only.s | +| relu_twice | activation | PASS | none | 6.00 | ±0.00 | 6 | 6 | 0.0533 | 0.0928 | 0.4265 | 6.00 | 0.00 | False | 4 | 4 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\relu_twice.s | +| if_else | branch | FAIL | timeout | 0.00 | ±0.00 | 0 | 0 | 0.0512 | 5.0168 | 20.0935 | 0.00 | 0.00 | False | 5 | 5 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\if_else.s | +| if_relu | branch | FAIL | timeout | 0.00 | ±0.00 | 0 | 0 | 0.0538 | 5.0090 | 20.1086 | 0.00 | 0.00 | False | 0 | 0 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\if_relu.s | +| if_then | branch | FAIL | timeout | 0.00 | ±0.00 | 0 | 0 | 0.0549 | 5.0095 | 20.1093 | 0.00 | 0.00 | False | 13 | 13 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\if_then.s | +| add_chain | elementwise | PASS | none | 4.00 | ±0.00 | 4 | 4 | 0.0548 | 0.0969 | 0.4364 | 4.00 | 0.00 | False | 9 | 9 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_chain.s | +| add_chain_3 | elementwise | PASS | none | 5.00 | ±0.00 | 5 | 5 | 0.0525 | 0.0971 | 0.4357 | 5.00 | 0.00 | False | 14 | 14 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_chain_3.s | +| add_fan_in_4 | elementwise | PASS | none | 5.00 | ±0.00 | 5 | 5 | 0.0523 | 0.1005 | 0.4393 | 5.00 | 0.00 | False | 10 | 10 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_fan_in_4.s | +| add_reuse | elementwise | PASS | none | 4.00 | ±0.00 | 4 | 4 | 0.0523 | 0.0935 | 0.4320 | 4.00 | 0.00 | False | 10 | 10 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_reuse.s | +| vector_add | elementwise | PASS | none | 3.00 | ±0.00 | 3 | 3 | 0.0535 | 0.0945 | 0.4363 | 3.00 | 0.00 | False | 5 | 5 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\vector_add.s | +| loop_add_4 | loop | PASS | none | 22.00 | ±0.00 | 22 | 22 | 0.0520 | 0.0941 | 0.4312 | 22.00 | 0.00 | False | 5 | 5 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\loop_add_4.s | +| loop_add_chain_4 | loop | PASS | none | 26.00 | ±0.00 | 26 | 26 | 0.0534 | 0.0950 | 0.4375 | 26.00 | 0.00 | False | 9 | 9 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\loop_add_chain_4.s | +| loop_relu_add_4 | loop | PASS | none | 30.00 | ±0.00 | 30 | 30 | 0.0511 | 0.0950 | 0.4302 | 30.00 | 0.00 | False | 2 | 2 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\loop_relu_add_4.s | +| dot_4 | reduction | PASS | none | 3.00 | ±0.00 | 3 | 3 | 0.0544 | 0.0939 | 0.4362 | 3.00 | 0.00 | False | 70 | 70 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_4.s | +| dot_8 | reduction | PASS | none | 3.00 | ±0.00 | 3 | 3 | 0.0537 | 0.1021 | 0.4474 | 3.00 | 0.00 | False | 36 | 36 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_8.s | +| dot_relu_4 | reduction | PASS | none | 5.00 | ±0.00 | 5 | 5 | 0.0523 | 0.0954 | 0.4342 | 5.00 | 0.00 | False | 0 | 0 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_relu_4.s | +| dot_relu_8 | reduction | PASS | none | 5.00 | ±0.00 | 5 | 5 | 0.0524 | 0.0945 | 0.4350 | 5.00 | 0.00 | False | 8 | 8 | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_relu_8.s | +| matmul_2x2 | tensor | PASS | none | 3.00 | ±0.00 | 3 | 3 | 0.0518 | 0.0952 | 0.4328 | 3.00 | 0.00 | False | [[19, 22], [43, 50]] | [[19, 22], [43, 50]] | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_2x2.s | +| matmul_4x4 | tensor | PASS | none | 3.00 | ±0.00 | 3 | 3 | 0.0529 | 0.0942 | 0.4400 | 3.00 | 0.00 | False | [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]] | [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]] | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_4x4.s | +| matmul_add_2x2 | tensor | PASS | none | 4.00 | ±0.00 | 4 | 4 | 0.0522 | 0.0940 | 0.4316 | 4.00 | 0.00 | False | [[20, 23], [44, 51]] | [[20, 23], [44, 51]] | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_add_2x2.s | +| matmul_relu_2x2 | tensor | PASS | none | 5.00 | ±0.00 | 5 | 5 | 0.0514 | 0.0966 | 0.4342 | 5.00 | 0.00 | False | [[0, 2], [0, 4]] | [[0, 2], [0, 4]] | True | D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_relu_2x2.s | + +## 性能图表 + +![课程版指令数图表](course_report_instructions.png) + +### Mermaid 图表 + +```mermaid +xychart-beta + title "各测试用例指令数" + x-axis ["add_relu_relu", "relu_add", "relu_only", "relu_twice", "if_else", "if_relu", "if_then", "add_chain", "add_chain_3", "add_fan_in_4", "add_reuse", "vector_add", "loop_add_4", "loop_add_chain_4", "loop_relu_add_4", "dot_4", "dot_8", "dot_relu_4", "dot_relu_8", "matmul_2x2", "matmul_4x4", "matmul_add_2x2", "matmul_relu_2x2"] + y-axis "指令数" 0 --> 32.0 + bar [7.0, 5.0, 4.0, 6.0, 0.0, 0.0, 0.0, 4.0, 5.0, 5.0, 4.0, 3.0, 22.0, 26.0, 30.0, 3.0, 3.0, 5.0, 5.0, 3.0, 3.0, 4.0, 5.0] +``` + +## 用例详情 + +### add_relu_relu + +- 类别: activation +- 描述: Add input and bias, then apply ReLU twice. +- 预期输出 (return_value): 7 +- 实际输出: 7 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0532 +- 模拟耗时(s): 0.1323 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4806 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_relu_relu.s +- Benchmark 重复次数: 3 +- 平均指令数: 7.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 7 +- 最大指令数: 7 +- 基线指令数: 7.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### relu_add + +- 类别: activation +- 描述: Add input and bias, then apply one ReLU. +- 预期输出 (return_value): 3 +- 实际输出: 3 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0564 +- 模拟耗时(s): 0.0981 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4456 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\relu_add.s +- Benchmark 重复次数: 3 +- 平均指令数: 5.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 5 +- 最大指令数: 5 +- 基线指令数: 5.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### relu_only + +- 类别: activation +- 描述: Apply ReLU directly to a single input value. +- 预期输出 (return_value): 0 +- 实际输出: 0 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0516 +- 模拟耗时(s): 0.1000 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4359 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\relu_only.s +- Benchmark 重复次数: 3 +- 平均指令数: 4.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 4 +- 最大指令数: 4 +- 基线指令数: 4.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### relu_twice + +- 类别: activation +- 描述: Apply ReLU twice to the same activation path. +- 预期输出 (return_value): 4 +- 实际输出: 4 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0533 +- 模拟耗时(s): 0.0928 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4265 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\relu_twice.s +- Benchmark 重复次数: 3 +- 平均指令数: 6.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 6 +- 最大指令数: 6 +- 基线指令数: 6.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### if_else + +- 类别: branch +- 描述: if/else branch returns subtraction result when flag is zero. +- 预期输出 (return_value): 5 +- 实际输出: 5 +- 输出是否匹配: True +- 模拟后端: timeout +- 编译耗时(s): 0.0512 +- 模拟耗时(s): 5.0168 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 20.0935 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\if_else.s +- Benchmark 重复次数: 3 +- 平均指令数: 0.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 0 +- 最大指令数: 0 +- 基线指令数: 0.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### if_relu + +- 类别: branch +- 描述: if/else branch combined with add and relu. +- 预期输出 (return_value): 0 +- 实际输出: 0 +- 输出是否匹配: True +- 模拟后端: timeout +- 编译耗时(s): 0.0538 +- 模拟耗时(s): 5.0090 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 20.1086 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\if_relu.s +- Benchmark 重复次数: 3 +- 平均指令数: 0.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 0 +- 最大指令数: 0 +- 基线指令数: 0.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### if_then + +- 类别: branch +- 描述: if/else branch returns add result when flag is non-zero. +- 预期输出 (return_value): 13 +- 实际输出: 13 +- 输出是否匹配: True +- 模拟后端: timeout +- 编译耗时(s): 0.0549 +- 模拟耗时(s): 5.0095 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 20.1093 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\if_then.s +- Benchmark 重复次数: 3 +- 平均指令数: 0.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 0 +- 最大指令数: 0 +- 基线指令数: 0.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### add_chain + +- 类别: elementwise +- 描述: Add a and b, then add c to the intermediate result. +- 预期输出 (return_value): 9 +- 实际输出: 9 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0548 +- 模拟耗时(s): 0.0969 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4364 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_chain.s +- Benchmark 重复次数: 3 +- 平均指令数: 4.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 4 +- 最大指令数: 4 +- 基线指令数: 4.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### add_chain_3 + +- 类别: elementwise +- 描述: Chain three add operations across four symbolic inputs. +- 预期输出 (return_value): 14 +- 实际输出: 14 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0525 +- 模拟耗时(s): 0.0971 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4357 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_chain_3.s +- Benchmark 重复次数: 3 +- 平均指令数: 5.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 5 +- 最大指令数: 5 +- 基线指令数: 5.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### add_fan_in_4 + +- 类别: elementwise +- 描述: Compute two independent adds and then merge them with a final add. +- 预期输出 (return_value): 10 +- 实际输出: 10 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0523 +- 模拟耗时(s): 0.1005 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4393 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_fan_in_4.s +- Benchmark 重复次数: 3 +- 平均指令数: 5.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 5 +- 最大指令数: 5 +- 基线指令数: 5.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### add_reuse + +- 类别: elementwise +- 描述: Reuse the same intermediate add result on both operands of a second add. +- 预期输出 (return_value): 10 +- 实际输出: 10 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0523 +- 模拟耗时(s): 0.0935 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4320 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\add_reuse.s +- Benchmark 重复次数: 3 +- 平均指令数: 4.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 4 +- 最大指令数: 4 +- 基线指令数: 4.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### vector_add + +- 类别: elementwise +- 描述: Single add over two symbolic vector inputs. +- 预期输出 (return_value): 5 +- 实际输出: 5 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0535 +- 模拟耗时(s): 0.0945 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4363 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\vector_add.s +- Benchmark 重复次数: 3 +- 平均指令数: 3.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 3 +- 最大指令数: 3 +- 基线指令数: 3.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### loop_add_4 + +- 类别: loop +- 描述: Run a four-iteration loop whose body computes one add; final returned value is the last loop-body result. +- 预期输出 (return_value): 5 +- 实际输出: 5 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0520 +- 模拟耗时(s): 0.0941 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4312 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\loop_add_4.s +- Benchmark 重复次数: 3 +- 平均指令数: 22.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 22 +- 最大指令数: 22 +- 基线指令数: 22.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### loop_add_chain_4 + +- 类别: loop +- 描述: Run a four-iteration loop whose body computes two chained adds; final returned value is the last loop-body result. +- 预期输出 (return_value): 9 +- 实际输出: 9 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0534 +- 模拟耗时(s): 0.0950 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4375 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\loop_add_chain_4.s +- Benchmark 重复次数: 3 +- 平均指令数: 26.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 26 +- 最大指令数: 26 +- 基线指令数: 26.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### loop_relu_add_4 + +- 类别: loop +- 描述: Run a four-iteration loop whose body computes add followed by ReLU; final returned value is the last loop-body result. +- 预期输出 (return_value): 2 +- 实际输出: 2 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0511 +- 模拟耗时(s): 0.0950 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4302 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\loop_relu_add_4.s +- Benchmark 重复次数: 3 +- 平均指令数: 30.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 30 +- 最大指令数: 30 +- 基线指令数: 30.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### dot_4 + +- 类别: reduction +- 描述: Compute the dot product of two symbolic vectors of length 4. +- 预期输出 (return_value): 70 +- 实际输出: 70 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0544 +- 模拟耗时(s): 0.0939 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4362 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_4.s +- Benchmark 重复次数: 3 +- 平均指令数: 3.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 3 +- 最大指令数: 3 +- 基线指令数: 3.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### dot_8 + +- 类别: reduction +- 描述: Compute the dot product of two symbolic vectors of length 8. +- 预期输出 (return_value): 36 +- 实际输出: 36 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0537 +- 模拟耗时(s): 0.1021 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4474 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_8.s +- Benchmark 重复次数: 3 +- 平均指令数: 3.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 3 +- 最大指令数: 3 +- 基线指令数: 3.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### dot_relu_4 + +- 类别: reduction +- 描述: Compute a length-4 dot product and pass it through ReLU. +- 预期输出 (return_value): 0 +- 实际输出: 0 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0523 +- 模拟耗时(s): 0.0954 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4342 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_relu_4.s +- Benchmark 重复次数: 3 +- 平均指令数: 5.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 5 +- 最大指令数: 5 +- 基线指令数: 5.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### dot_relu_8 + +- 类别: reduction +- 描述: Compute a length-8 dot product and pass it through ReLU. +- 预期输出 (return_value): 8 +- 实际输出: 8 +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0524 +- 模拟耗时(s): 0.0945 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4350 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\dot_relu_8.s +- Benchmark 重复次数: 3 +- 平均指令数: 5.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 5 +- 最大指令数: 5 +- 基线指令数: 5.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### matmul_2x2 + +- 类别: tensor +- 描述: Compute a symbolic 2x2 by 2x2 matrix multiplication. +- 预期输出 (return_value): [[19, 22], [43, 50]] +- 实际输出: [[19, 22], [43, 50]] +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0518 +- 模拟耗时(s): 0.0952 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4328 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_2x2.s +- Benchmark 重复次数: 3 +- 平均指令数: 3.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 3 +- 最大指令数: 3 +- 基线指令数: 3.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### matmul_4x4 + +- 类别: tensor +- 描述: Compute a symbolic 4x4 by 4x4 matrix multiplication. +- 预期输出 (return_value): [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]] +- 实际输出: [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]] +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0529 +- 模拟耗时(s): 0.0942 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4400 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_4x4.s +- Benchmark 重复次数: 3 +- 平均指令数: 3.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 3 +- 最大指令数: 3 +- 基线指令数: 3.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### matmul_add_2x2 + +- 类别: tensor +- 描述: Compute a 2x2 matmul and then add a symbolic bias term. +- 预期输出 (return_value): [[20, 23], [44, 51]] +- 实际输出: [[20, 23], [44, 51]] +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0522 +- 模拟耗时(s): 0.0940 +- 参考解释器耗时(s): 0.0002 +- 总耗时(s): 0.4316 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_add_2x2.s +- Benchmark 重复次数: 3 +- 平均指令数: 4.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 4 +- 最大指令数: 4 +- 基线指令数: 4.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + +### matmul_relu_2x2 + +- 类别: tensor +- 描述: Compute a 2x2 matmul and then apply ReLU to its result. +- 预期输出 (return_value): [[0, 2], [0, 4]] +- 实际输出: [[0, 2], [0, 4]] +- 输出是否匹配: True +- 模拟后端: none +- 编译耗时(s): 0.0514 +- 模拟耗时(s): 0.0966 +- 参考解释器耗时(s): 0.0001 +- 总耗时(s): 0.4342 +- 汇编文件: D:\PycharmProjects\ScratchV\ScratchV\ScratchV-topic06-deliverable\build\matmul_relu_2x2.s +- Benchmark 重复次数: 3 +- 平均指令数: 5.00 +- 95% 置信区间: ±0.00 +- 最小指令数: 5 +- 最大指令数: 5 +- 基线指令数: 5.00 +- 性能变化率: 0.00% +- 性能退化阈值: 5.00% +- 是否性能退化: False + diff --git a/ScratchV-topic06-deliverable/requirements-topic06.txt b/ScratchV-topic06-deliverable/requirements-topic06.txt new file mode 100644 index 0000000..dc452fd --- /dev/null +++ b/ScratchV-topic06-deliverable/requirements-topic06.txt @@ -0,0 +1,4 @@ +tinyfive +pytest +jinja2 +matplotlib diff --git a/ScratchV-topic06-deliverable/run_tests.py b/ScratchV-topic06-deliverable/run_tests.py new file mode 100644 index 0000000..fce002e --- /dev/null +++ b/ScratchV-topic06-deliverable/run_tests.py @@ -0,0 +1,1481 @@ +import argparse +import json +import math +import os +import subprocess +import sys +import tempfile +import time +from datetime import datetime +from pathlib import Path + +from scratchv.simulator.tinyfive import verify_assembly + +SUITE_DIR = Path(__file__).resolve().parent +PROJECT_ROOT = SUITE_DIR.parent +TEST_DIR = SUITE_DIR / "tests_main" +BUILD_DIR = SUITE_DIR / "build" +REPORT_DIR = SUITE_DIR / "reports" +REPORT_FILE = REPORT_DIR / "report.md" +HTML_REPORT_FILE = REPORT_DIR / "report.html" +CHART_FILE = REPORT_DIR / "course_report_instructions.png" +BASELINE_FILE = REPORT_DIR / "benchmark_baseline.json" +REGRESSION_THRESHOLD_PCT = 5.0 +SIMULATION_TIMEOUT_SEC = 5.0 + + +def run_compile(dsl_file: Path): + output_file = BUILD_DIR / (dsl_file.stem + ".s") + + cmd = [ + sys.executable, + "-m", + "scratchv.main", + str(dsl_file), + "-o", + str(output_file), + "--optimize", + "all", + "--dump-ir", + ] + + result = subprocess.run( + cmd, + capture_output=True, + text=True, + encoding="utf-8", + errors="ignore", + cwd=PROJECT_ROOT, + ) + + return result, output_file + + +def load_metadata(dsl_file: Path): + meta_file = dsl_file.with_suffix(".meta.json") + if not meta_file.exists(): + return { + "description": "", + "expected_output_type": "return_value", + "expected_return": "", + } + return json.loads(meta_file.read_text(encoding="utf-8")) + + +def run_simulation(asm_file: Path, timeout: float = SIMULATION_TIMEOUT_SEC): + if not asm_file.exists(): + return { + "success": False, + "instr_count": 0, + "return_value": None, + "backend": "none", + "error": "assembly file not found", + } + + code = "\n".join([ + "import json, sys", + "from pathlib import Path", + "from scratchv.simulator.tinyfive import verify_assembly", + "asm = Path(sys.argv[1]).read_text(encoding='utf-8')", + "try:", + " result = verify_assembly(asm)", + "except Exception as exc:", + " result = {'success': False, 'instr_count': 0, " + "'return_value': None, 'backend': 'tinyfive', 'error': str(exc)}", + "print(json.dumps(result, ensure_ascii=False))", + ]) + + try: + completed = subprocess.run( + [sys.executable, "-c", code, str(asm_file)], + capture_output=True, + text=True, + encoding="utf-8", + errors="ignore", + cwd=PROJECT_ROOT, + timeout=timeout, + ) + except subprocess.TimeoutExpired: + return { + "success": False, + "instr_count": 0, + "return_value": None, + "backend": "timeout", + "error": f"simulation timeout after {timeout:.0f}s", + } + + if completed.returncode != 0: + return { + "success": False, + "instr_count": 0, + "return_value": None, + "backend": "tinyfive", + "error": (completed.stderr or completed.stdout or "simulation failed").strip(), + } + + try: + return json.loads(completed.stdout.strip().splitlines()[-1]) + except (IndexError, json.JSONDecodeError) as exc: + return { + "success": False, + "instr_count": 0, + "return_value": None, + "backend": "tinyfive", + "error": f"invalid simulation output: {exc}", + } + + +def apply_add(lhs, rhs): + if isinstance(lhs, list) and isinstance(rhs, list): + return [apply_add(a, b) for a, b in zip(lhs, rhs)] + if isinstance(lhs, list): + return [apply_add(a, rhs) for a in lhs] + if isinstance(rhs, list): + return [apply_add(lhs, b) for b in rhs] + return lhs + rhs + + +def apply_binary(lhs, rhs, op): + if isinstance(lhs, list) and isinstance(rhs, list): + return [apply_binary(a, b, op) for a, b in zip(lhs, rhs)] + if isinstance(lhs, list): + return [apply_binary(a, rhs, op) for a in lhs] + if isinstance(rhs, list): + return [apply_binary(lhs, b, op) for b in rhs] + return op(lhs, rhs) + + +def apply_relu(value): + if isinstance(value, list): + return [apply_relu(v) for v in value] + return value if value > 0 else 0 + + +def apply_gelu(value): + if isinstance(value, list): + return [apply_gelu(v) for v in value] + return 0.5 * value * (1.0 + math.erf(value / math.sqrt(2.0))) + + +def apply_softmax(value): + if not isinstance(value, list): + return 1.0 + max_value = max(value) + exp_values = [math.exp(v - max_value) for v in value] + total = sum(exp_values) + return [v / total for v in exp_values] + + +def apply_maxpool(value, kernel, stride): + if not isinstance(value, list): + return value + return [max(value[i:i + kernel]) for i in range(0, len(value) - kernel + 1, stride)] + + +def apply_dot(lhs, rhs, length): + return sum(lhs[i] * rhs[i] for i in range(length)) + + +def apply_matmul(lhs, rhs, m, n, k): + result = [] + for i in range(m): + row = [] + for j in range(n): + cell = 0 + for kk in range(k): + cell += lhs[i][kk] * rhs[kk][j] + row.append(cell) + result.append(row) + return result + + +def resolve_value(token, env): + token = token.strip() + try: + return int(token) + except ValueError: + pass + try: + return float(token) + except ValueError: + pass + return env[token] + + +def resolve_condition(cond_text, env): + cond_text = cond_text.strip() + if cond_text.endswith(":"): + cond_text = cond_text[:-1].strip() + if cond_text.startswith("(") and cond_text.endswith(")"): + cond_text = cond_text[1:-1].strip() + + for op in ("==", "!=", "<=", ">=", "<", ">"): + if op in cond_text: + lhs_text, rhs_text = [part.strip() for part in cond_text.split(op, 1)] + lhs = resolve_value(lhs_text, env) + rhs = resolve_value(rhs_text, env) + if op == "==": + return lhs == rhs + if op == "!=": + return lhs != rhs + if op == "<=": + return lhs <= rhs + if op == ">=": + return lhs >= rhs + if op == "<": + return lhs < rhs + if op == ">": + return lhs > rhs + + return bool(resolve_value(cond_text, env)) + + +def execute_block(lines, env, start_idx=0, end_idx=None): + if end_idx is None: + end_idx = len(lines) + + idx = start_idx + while idx < end_idx: + line = lines[idx] + + if line.startswith("for "): + loop_var_text = line.replace("for ", "", 1) + loop_var, bounds = [p.strip() for p in loop_var_text.split("=", 1)] + loop_start_text, loop_end_text = [p.strip() for p in bounds.split(",", 1)] + loop_start = int(loop_start_text) + loop_end = int(loop_end_text) + + depth = 1 + body_start = idx + 1 + body_end = body_start + while body_end < end_idx and depth > 0: + current = lines[body_end] + if current.startswith("for "): + depth += 1 + elif current == "endfor": + depth -= 1 + if depth == 0: + break + body_end += 1 + + for i in range(loop_start, loop_end): + env[loop_var] = i + returned, value = execute_block(lines, env, body_start, body_end) + if returned: + return True, value + + idx = body_end + 1 + continue + + if line.startswith("if "): + cond_text = line.replace("if ", "", 1).strip() + cond_value = resolve_condition(cond_text, env) + + depth = 1 + body_start = idx + 1 + scan_idx = body_start + else_idx = None + endif_idx = None + while scan_idx < end_idx: + current = lines[scan_idx] + if current.startswith("if "): + depth += 1 + elif current == "endif": + depth -= 1 + if depth == 0: + endif_idx = scan_idx + break + elif current == "else" and depth == 1: + else_idx = scan_idx + scan_idx += 1 + + if endif_idx is None: + raise ValueError("if without matching endif") + + if cond_value: + branch_start = body_start + branch_end = else_idx if else_idx is not None else endif_idx + else: + branch_start = else_idx + 1 if else_idx is not None else endif_idx + branch_end = endif_idx + + returned, value = execute_block(lines, env, branch_start, branch_end) + if returned: + return True, value + + idx = endif_idx + 1 + continue + + if line in {"else", "endif"}: + return False, None + + if line == "endfor": + return False, None + + if line.startswith("return "): + return True, resolve_value(line.replace("return ", "", 1).strip(), env) + + dest_name, expr = [p.strip() for p in line.split("=", 1)] + op_name = expr[:expr.index("(")] + arg_text = expr[expr.index("(") + 1: expr.rindex(")")] + args = [a.strip() for a in arg_text.split(",") if a.strip()] + + plain_args = [] + kwargs = {} + for arg in args: + if ":" in arg: + key, value = arg.split(":", 1) + kwargs[key.strip()] = int(value.strip()) + else: + plain_args.append(resolve_value(arg, env)) + + if op_name == "add": + env[dest_name] = apply_add(plain_args[0], plain_args[1]) + elif op_name == "sub": + env[dest_name] = apply_binary(plain_args[0], plain_args[1], lambda a, b: a - b) + elif op_name == "mul": + env[dest_name] = apply_binary(plain_args[0], plain_args[1], lambda a, b: a * b) + elif op_name == "div": + env[dest_name] = apply_binary(plain_args[0], plain_args[1], lambda a, b: a / b) + elif op_name == "relu": + env[dest_name] = apply_relu(plain_args[0]) + elif op_name == "gelu": + env[dest_name] = apply_gelu(plain_args[0]) + elif op_name == "softmax": + env[dest_name] = apply_softmax(plain_args[0]) + elif op_name == "maxpool": + env[dest_name] = apply_maxpool( + plain_args[0], + kwargs.get("kernel", 2), + kwargs.get("stride", 2), + ) + elif op_name == "dot": + env[dest_name] = apply_dot(plain_args[0], plain_args[1], kwargs["len"]) + elif op_name == "matmul": + env[dest_name] = apply_matmul( + plain_args[0], + plain_args[1], + kwargs["m"], + kwargs["n"], + kwargs["k"], + ) + else: + raise ValueError(f"Unsupported op in reference executor: {op_name}") + + idx += 1 + + return False, None + + +def execute_dsl_reference(dsl_file: Path, inputs): + raw_lines = dsl_file.read_text(encoding="utf-8").splitlines() + lines = [] + for raw_line in raw_lines: + line = raw_line.strip() + if not line or line.startswith("#"): + continue + lines.append(line) + + env = dict(inputs) + returned, value = execute_block(lines, env) + return value if returned else None + + +def values_equal(lhs, rhs): + if isinstance(lhs, list) and isinstance(rhs, list): + if len(lhs) != len(rhs): + return False + return all(values_equal(a, b) for a, b in zip(lhs, rhs)) + if isinstance(lhs, float) or isinstance(rhs, float): + return math.isclose(lhs, rhs, rel_tol=1e-7, abs_tol=1e-7) + return lhs == rhs + + +def summarize_benchmark_runs(instr_counts): + if not instr_counts: + return { + "runs": 0, + "avg_instr_count": 0.0, + "min_instr_count": 0, + "max_instr_count": 0, + "ci95_instr_count": 0.0, + } + avg = sum(instr_counts) / len(instr_counts) + if len(instr_counts) > 1: + variance = sum((value - avg) ** 2 for value in instr_counts) / (len(instr_counts) - 1) + ci95 = 1.96 * math.sqrt(variance) / math.sqrt(len(instr_counts)) + else: + ci95 = 0.0 + return { + "runs": len(instr_counts), + "avg_instr_count": avg, + "min_instr_count": min(instr_counts), + "max_instr_count": max(instr_counts), + "ci95_instr_count": ci95, + } + + +def detect_regression(avg_instr_count, baseline_instr_count): + delta = avg_instr_count - baseline_instr_count + delta_pct = 0.0 if baseline_instr_count == 0 else (delta / baseline_instr_count) * 100.0 + return { + "baseline_instr_count": baseline_instr_count, + "delta": round(delta, 4), + "delta_pct": round(delta_pct, 4), + "threshold_pct": REGRESSION_THRESHOLD_PCT, + "regressed": delta_pct > REGRESSION_THRESHOLD_PCT, + } + + +def load_baseline(): + if not BASELINE_FILE.exists(): + return {} + return json.loads(BASELINE_FILE.read_text(encoding="utf-8")) + + +def save_baseline(results): + REPORT_DIR.mkdir(exist_ok=True) + payload = {} + for r in results: + payload[r["name"]] = { + "category": r["category"], + "avg_instr_count": r["avg_instr_count"], + "runs": r["benchmark_runs"], + } + BASELINE_FILE.write_text(json.dumps(payload, indent=2), encoding="utf-8") + + +def generate_report_text(results, passed, failed): + lines = [] + lines.append("# ScratchV DSL 编译器性能测试报告\n\n") + + lines.append("## 测试概览\n\n") + lines.append(f"- 用例总数: {len(results)}\n") + lines.append(f"- 通过数量: {passed}\n") + lines.append(f"- 失败数量: {failed}\n\n") + + benchmark_mode = any(r.get("benchmark_runs", 1) > 1 for r in results) + if benchmark_mode: + lines.append("## 性能基准概览\n\n") + lines.append("- 运行模式: benchmark\n") + lines.append(f"- 性能基线文件: `{BASELINE_FILE}`\n\n") + + lines.append("## 测试结果\n\n") + if benchmark_mode: + lines.append("| 测试用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 最小值 | 最大值 | 基线 | 变化率 | 是否退化 | 预期输出 | 实际输出 | 是否匹配 | 汇编文件 |\n") + lines.append("|---|---|---|---|---:|---:|---:|---:|---:|---|---|---|---|---|\n") + else: + lines.append("| 测试用例 | 类别 | 状态 | 模拟后端 | 指令数 | 预期输出 | 实际输出 | 是否匹配 | 汇编文件 |\n") + lines.append("|---|---|---|---|---:|---|---|---|---|\n") + + for r in results: + expected = str(r["expected"]).replace("\n", " ").replace("|", "\\|") + actual = str(r["actual"]).replace("\n", " ").replace("|", "\\|") + if benchmark_mode: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['avg_instr_count']:.2f} | {r['min_instr_count']} | {r['max_instr_count']} | " + f"{r['baseline_instr_count']:.2f} | {r['delta_pct']:.2f} | {r['regressed']} | " + f"{expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + else: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['instr_count']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + + lines.append("\n## 性能图表\n\n") + chart_cases = [r["name"] for r in results] + chart_instr = [str(round(r.get("avg_instr_count", r["instr_count"]), 2)) for r in results] + lines.append("### 各测试用例指令数\n\n") + lines.append("```mermaid\n") + lines.append("xychart-beta\n") + lines.append(' title "各测试用例指令数"\n') + lines.append(" x-axis [" + ", ".join(f'"{name}"' for name in chart_cases) + "]\n") + max_instr = max((r.get("avg_instr_count", r["instr_count"]) for r in results), default=0) + lines.append(f' y-axis "指令数" 0 --> {max_instr + 2}\n') + lines.append(" bar [" + ", ".join(chart_instr) + "]\n") + lines.append("```\n\n") + + category_totals = {} + for r in results: + category_totals[r["category"]] = category_totals.get(r["category"], 0) + r.get("avg_instr_count", r["instr_count"]) + lines.append("### 各类别指令数占比\n\n") + lines.append("```mermaid\n") + lines.append("pie showData\n") + lines.append(' title 各类别指令数占比\n') + for category, total in sorted(category_totals.items()): + lines.append(f' "{category}" : {total}\n') + lines.append("```\n") + + lines.append("\n## 用例详情\n\n") + for r in results: + lines.append(f"### {r['name']}\n\n") + lines.append(f"- 类别: {r['category']}\n") + lines.append(f"- 描述: {r['description']}\n") + lines.append(f"- 预期输出 ({r['expected_type']}): {r['expected']}\n") + lines.append(f"- 实际输出: {r['actual']}\n") + lines.append(f"- 是否匹配: {r['matched']}\n") + lines.append(f"- 模拟后端: {r['backend']}\n") + if benchmark_mode: + lines.append(f"- Benchmark 重复次数: {r['benchmark_runs']}\n") + lines.append(f"- 平均指令数: {r['avg_instr_count']:.2f}\n") + lines.append(f"- 最小指令数: {r['min_instr_count']}\n") + lines.append(f"- 最大指令数: {r['max_instr_count']}\n") + lines.append(f"- 基线指令数: {r['baseline_instr_count']:.2f}\n") + lines.append(f"- 性能变化率 (%): {r['delta_pct']:.2f}\n") + lines.append(f"- 是否性能退化: {r['regressed']}\n") + else: + lines.append(f"- 指令数: {r['instr_count']}\n") + lines.append(f"- 汇编文件: {r['asm']}\n\n") + + return "".join(lines) + + +def write_html_report(results, passed, failed): + try: + from jinja2 import Template + except ImportError: + return None + + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + template = Template(""" + + + + ScratchV 课程版测试报告 + + + +

ScratchV DSL 编译器性能测试报告

+
+

用例总数:{{ total }},通过:{{ passed }},失败:{{ failed }},通过率:{{ "%.1f"|format(pass_rate) }}%

+

测试目录:{{ test_dir }},性能退化阈值:{{ threshold }}%

+
+ 课程版指令数图表 + + + + + + {% for r in results %} + + + + + + + + + + + {% endfor %} + +
用例类别状态平均指令数95%置信区间变化率(%)是否退化描述
{{ r.name }}{{ r.category }}{{ r.status }}{{ "%.2f"|format(r.avg_instr_count) }}±{{ "%.2f"|format(r.ci95_instr_count) }}{{ "%.2f"|format(r.delta_pct) }}{{ r.regressed }}{{ r.description }}
+ + +""") + HTML_REPORT_FILE.write_text( + template.render( + total=len(results), + passed=passed, + failed=failed, + pass_rate=pass_rate, + test_dir=str(TEST_DIR), + threshold=REGRESSION_THRESHOLD_PCT, + chart_name=CHART_FILE.name, + results=results, + ), + encoding="utf-8", + ) + return HTML_REPORT_FILE + + +def generate_report_text(results, passed, failed): + benchmark_mode = any(r.get("benchmark_runs", 1) > 1 for r in results) + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + lines = [ + "# ScratchV DSL 编译器性能测试报告\n\n", + "## 测试概览\n\n", + f"- 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n", + f"- 用例总数: {len(results)}\n", + f"- 通过数量: {passed}\n", + f"- 失败数量: {failed}\n", + f"- 通过率: {pass_rate:.1f}%\n", + f"- 测试目录: `{TEST_DIR}`\n", + f"- 汇编输出目录: `{BUILD_DIR}`\n", + f"- 性能基线文件: `{BASELINE_FILE}`\n", + f"- 性能退化阈值: {REGRESSION_THRESHOLD_PCT:.1f}%\n\n", + "## 测试结果\n\n", + ] + + if benchmark_mode: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 95%置信区间 | 最小 | 最大 | 基线 | 变化率(%) | 是否退化 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|\n", + ]) + else: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 指令数 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---|---|---|---|\n", + ]) + + for r in results: + expected = _markdown_cell(r["expected"]) + actual = _markdown_cell(r["actual"]) + if benchmark_mode: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['avg_instr_count']:.2f} | ±{r['ci95_instr_count']:.2f} | " + f"{r['min_instr_count']} | {r['max_instr_count']} | " + f"{r['baseline_instr_count']:.2f} | {r['delta_pct']:.2f} | " + f"{r['regressed']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + else: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['instr_count']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + + lines.extend([ + "\n## 性能图表\n\n", + f"![课程版指令数图表]({CHART_FILE.name})\n\n", + "### Mermaid 图表\n\n", + "```mermaid\n", + "xychart-beta\n", + ' title "各测试用例指令数"\n', + " x-axis [" + ", ".join(f'"{r["name"]}"' for r in results) + "]\n", + ]) + max_instr = max((r.get("avg_instr_count", r["instr_count"]) for r in results), default=0) + chart_values = [str(round(r.get("avg_instr_count", r["instr_count"]), 2)) for r in results] + lines.extend([ + f' y-axis "指令数" 0 --> {max_instr + 2}\n', + " bar [" + ", ".join(chart_values) + "]\n", + "```\n\n", + "## 用例详情\n\n", + ]) + + for r in results: + lines.extend([ + f"### {r['name']}\n\n", + f"- 类别: {r['category']}\n", + f"- 描述: {r['description']}\n", + f"- 预期输出 ({r['expected_type']}): {r['expected']}\n", + f"- 实际输出: {r['actual']}\n", + f"- 输出是否匹配: {r['matched']}\n", + f"- 模拟后端: {r['backend']}\n", + f"- 汇编文件: {r['asm']}\n", + ]) + if benchmark_mode: + lines.extend([ + f"- Benchmark 重复次数: {r['benchmark_runs']}\n", + f"- 平均指令数: {r['avg_instr_count']:.2f}\n", + f"- 95% 置信区间: ±{r['ci95_instr_count']:.2f}\n", + f"- 最小指令数: {r['min_instr_count']}\n", + f"- 最大指令数: {r['max_instr_count']}\n", + f"- 基线指令数: {r['baseline_instr_count']:.2f}\n", + f"- 性能变化率: {r['delta_pct']:.2f}%\n", + f"- 性能退化阈值: {r['threshold_pct']:.2f}%\n", + f"- 是否性能退化: {r['regressed']}\n", + ]) + else: + lines.append(f"- 指令数: {r['instr_count']}\n") + lines.append("\n") + + return "".join(lines) + + +def write_report(results, passed, failed): + REPORT_DIR.mkdir(exist_ok=True) + + REPORT_FILE.write_text(generate_report_text(results, passed, failed), encoding="utf-8") + print(f"\nReport written to {REPORT_FILE}") + + +def _markdown_cell(value): + return str(value).replace("\n", " ").replace("|", "\\|") + + +def generate_report_text(results, passed, failed): + benchmark_mode = any(r.get("benchmark_runs", 1) > 1 for r in results) + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + lines = [ + "# ScratchV DSL 编译器性能测试报告\n\n", + "## 测试概览\n\n", + f"- 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n", + f"- 用例总数: {len(results)}\n", + f"- 通过数量: {passed}\n", + f"- 失败数量: {failed}\n", + f"- 通过率: {pass_rate:.1f}%\n", + f"- 测试目录: `{TEST_DIR}`\n", + f"- 汇编输出目录: `{BUILD_DIR}`\n", + f"- 性能基线文件: `{BASELINE_FILE}`\n", + f"- 性能退化阈值: {REGRESSION_THRESHOLD_PCT:.1f}%\n\n", + "## 测试结果\n\n", + ] + + if benchmark_mode: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 95%置信区间 | 最小 | 最大 | 基线 | 变化率(%) | 是否退化 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|\n", + ]) + else: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 指令数 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---|---|---|---|\n", + ]) + + for r in results: + expected = _markdown_cell(r["expected"]) + actual = _markdown_cell(r["actual"]) + if benchmark_mode: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['avg_instr_count']:.2f} | ±{r['ci95_instr_count']:.2f} | " + f"{r['min_instr_count']} | {r['max_instr_count']} | " + f"{r['baseline_instr_count']:.2f} | {r['delta_pct']:.2f} | " + f"{r['regressed']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + else: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['instr_count']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + + lines.extend([ + "\n## 性能图表\n\n", + f"![课程版指令数图表]({CHART_FILE.name})\n\n", + "### Mermaid 图表\n\n", + "```mermaid\n", + "xychart-beta\n", + ' title "各测试用例指令数"\n', + " x-axis [" + ", ".join(f'"{r["name"]}"' for r in results) + "]\n", + ]) + max_instr = max((r.get("avg_instr_count", r["instr_count"]) for r in results), default=0) + chart_values = [str(round(r.get("avg_instr_count", r["instr_count"]), 2)) for r in results] + lines.extend([ + f' y-axis "指令数" 0 --> {max_instr + 2}\n', + " bar [" + ", ".join(chart_values) + "]\n", + "```\n\n", + "## 用例详情\n\n", + ]) + + for r in results: + lines.extend([ + f"### {r['name']}\n\n", + f"- 类别: {r['category']}\n", + f"- 描述: {r['description']}\n", + f"- 预期输出 ({r['expected_type']}): {r['expected']}\n", + f"- 实际输出: {r['actual']}\n", + f"- 输出是否匹配: {r['matched']}\n", + f"- 模拟后端: {r['backend']}\n", + f"- 汇编文件: {r['asm']}\n", + ]) + if benchmark_mode: + lines.extend([ + f"- Benchmark 重复次数: {r['benchmark_runs']}\n", + f"- 平均指令数: {r['avg_instr_count']:.2f}\n", + f"- 95% 置信区间: ±{r['ci95_instr_count']:.2f}\n", + f"- 最小指令数: {r['min_instr_count']}\n", + f"- 最大指令数: {r['max_instr_count']}\n", + f"- 基线指令数: {r['baseline_instr_count']:.2f}\n", + f"- 性能变化率: {r['delta_pct']:.2f}%\n", + f"- 性能退化阈值: {r['threshold_pct']:.2f}%\n", + f"- 是否性能退化: {r['regressed']}\n", + ]) + else: + lines.append(f"- 指令数: {r['instr_count']}\n") + lines.append("\n") + + return "".join(lines) + + +def generate_report_text(results, passed, failed): + benchmark_mode = any(r.get("benchmark_runs", 1) > 1 for r in results) + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + lines = [ + "# ScratchV DSL 编译器性能测试报告\n\n", + "## 测试概览\n\n", + f"- 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n", + f"- 用例总数: {len(results)}\n", + f"- 通过数量: {passed}\n", + f"- 失败数量: {failed}\n", + f"- 通过率: {pass_rate:.1f}%\n", + f"- 测试目录: `{TEST_DIR}`\n", + f"- 汇编输出目录: `{BUILD_DIR}`\n", + f"- 性能基线文件: `{BASELINE_FILE}`\n", + f"- 性能退化阈值: {REGRESSION_THRESHOLD_PCT:.1f}%\n\n", + "## 测试结果\n\n", + ] + + if benchmark_mode: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 95%置信区间 | 最小 | 最大 | 基线 | 变化率(%) | 是否退化 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|\n", + ]) + else: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 指令数 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---|---|---|---|\n", + ]) + + for r in results: + expected = _markdown_cell(r["expected"]) + actual = _markdown_cell(r["actual"]) + if benchmark_mode: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['avg_instr_count']:.2f} | ±{r['ci95_instr_count']:.2f} | " + f"{r['min_instr_count']} | {r['max_instr_count']} | " + f"{r['baseline_instr_count']:.2f} | {r['delta_pct']:.2f} | " + f"{r['regressed']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + else: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['instr_count']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + + lines.extend([ + "\n## 性能图表\n\n", + f"![课程版指令数图表]({CHART_FILE.name})\n\n", + "### Mermaid 图表\n\n", + "```mermaid\n", + "xychart-beta\n", + ' title "各测试用例指令数"\n', + " x-axis [" + ", ".join(f'"{r["name"]}"' for r in results) + "]\n", + ]) + max_instr = max((r.get("avg_instr_count", r["instr_count"]) for r in results), default=0) + chart_values = [str(round(r.get("avg_instr_count", r["instr_count"]), 2)) for r in results] + lines.extend([ + f' y-axis "指令数" 0 --> {max_instr + 2}\n', + " bar [" + ", ".join(chart_values) + "]\n", + "```\n\n", + "## 用例详情\n\n", + ]) + + for r in results: + lines.extend([ + f"### {r['name']}\n\n", + f"- 类别: {r['category']}\n", + f"- 描述: {r['description']}\n", + f"- 预期输出 ({r['expected_type']}): {r['expected']}\n", + f"- 实际输出: {r['actual']}\n", + f"- 输出是否匹配: {r['matched']}\n", + f"- 模拟后端: {r['backend']}\n", + f"- 汇编文件: {r['asm']}\n", + ]) + if benchmark_mode: + lines.extend([ + f"- Benchmark 重复次数: {r['benchmark_runs']}\n", + f"- 平均指令数: {r['avg_instr_count']:.2f}\n", + f"- 95% 置信区间: ±{r['ci95_instr_count']:.2f}\n", + f"- 最小指令数: {r['min_instr_count']}\n", + f"- 最大指令数: {r['max_instr_count']}\n", + f"- 基线指令数: {r['baseline_instr_count']:.2f}\n", + f"- 性能变化率: {r['delta_pct']:.2f}%\n", + f"- 性能退化阈值: {r['threshold_pct']:.2f}%\n", + f"- 是否性能退化: {r['regressed']}\n", + ]) + else: + lines.append(f"- 指令数: {r['instr_count']}\n") + lines.append("\n") + + return "".join(lines) + + +def write_chart(results): + try: + mpl_config_dir = Path(tempfile.gettempdir()) / "scratchv-matplotlib" + mpl_config_dir.mkdir(parents=True, exist_ok=True) + os.environ.setdefault("MPLCONFIGDIR", str(mpl_config_dir)) + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + except ImportError: + return None + + names = [r["name"] for r in results] + values = [r.get("avg_instr_count", r["instr_count"]) for r in results] + width = max(10, len(names) * 0.45) + fig, ax = plt.subplots(figsize=(width, 5)) + ax.bar(range(len(names)), values, color="#2563eb") + ax.set_title("ScratchV Course Benchmark Instruction Counts") + ax.set_ylabel("Instructions") + ax.set_xticks(range(len(names))) + ax.set_xticklabels(names, rotation=60, ha="right", fontsize=8) + ax.grid(axis="y", linestyle="--", alpha=0.35) + fig.tight_layout() + fig.savefig(CHART_FILE, dpi=160) + plt.close(fig) + return CHART_FILE + + +def write_html_report(results, passed, failed): + try: + from jinja2 import Template + except ImportError: + return None + + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + template = Template(""" + + + + ScratchV 课程版测试报告 + + + +

ScratchV DSL 编译器性能测试报告

+
+

用例总数:{{ total }},通过:{{ passed }},失败:{{ failed }},通过率:{{ "%.1f"|format(pass_rate) }}%

+

测试目录:{{ test_dir }},性能退化阈值:{{ threshold }}%

+
+ 课程版指令数图表 + + + + + + {% for r in results %} + + + + + + + + + + + {% endfor %} + +
用例类别状态平均指令数95%置信区间变化率(%)是否退化描述
{{ r.name }}{{ r.category }}{{ r.status }}{{ "%.2f"|format(r.avg_instr_count) }}±{{ "%.2f"|format(r.ci95_instr_count) }}{{ "%.2f"|format(r.delta_pct) }}{{ r.regressed }}{{ r.description }}
+ + +""") + HTML_REPORT_FILE.write_text( + template.render( + total=len(results), + passed=passed, + failed=failed, + pass_rate=pass_rate, + test_dir=str(TEST_DIR), + threshold=REGRESSION_THRESHOLD_PCT, + chart_name=CHART_FILE.name, + results=results, + ), + encoding="utf-8", + ) + return HTML_REPORT_FILE + + +def generate_report_text_cn(results, passed, failed): + benchmark_mode = any(r.get("benchmark_runs", 1) > 1 for r in results) + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + lines = [ + "# ScratchV DSL 编译器性能测试报告\n\n", + "## 测试概览\n\n", + f"- 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n", + f"- 用例总数: {len(results)}\n", + f"- 通过数量: {passed}\n", + f"- 失败数量: {failed}\n", + f"- 通过率: {pass_rate:.1f}%\n", + f"- 测试目录: `{TEST_DIR}`\n", + f"- 汇编输出目录: `{BUILD_DIR}`\n", + f"- 性能基线文件: `{BASELINE_FILE}`\n", + f"- 性能退化阈值: {REGRESSION_THRESHOLD_PCT:.1f}%\n\n", + "## 测试结果\n\n", + ] + + if benchmark_mode: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 95%置信区间 | 最小 | 最大 | 基线 | 变化率(%) | 是否退化 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|\n", + ]) + else: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 指令数 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---|---|---|---|\n", + ]) + + for r in results: + expected = _markdown_cell(r["expected"]) + actual = _markdown_cell(r["actual"]) + if benchmark_mode: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['avg_instr_count']:.2f} | ±{r['ci95_instr_count']:.2f} | " + f"{r['min_instr_count']} | {r['max_instr_count']} | " + f"{r['baseline_instr_count']:.2f} | {r['delta_pct']:.2f} | " + f"{r['regressed']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + else: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['instr_count']} | {expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + + lines.extend([ + "\n## 性能图表\n\n", + f"![课程版指令数图表]({CHART_FILE.name})\n\n", + "### Mermaid 图表\n\n", + "```mermaid\n", + "xychart-beta\n", + ' title "各测试用例指令数"\n', + " x-axis [" + ", ".join(f'"{r["name"]}"' for r in results) + "]\n", + ]) + max_instr = max((r.get("avg_instr_count", r["instr_count"]) for r in results), default=0) + chart_values = [str(round(r.get("avg_instr_count", r["instr_count"]), 2)) for r in results] + lines.extend([ + f' y-axis "指令数" 0 --> {max_instr + 2}\n', + " bar [" + ", ".join(chart_values) + "]\n", + "```\n\n", + "## 用例详情\n\n", + ]) + + for r in results: + lines.extend([ + f"### {r['name']}\n\n", + f"- 类别: {r['category']}\n", + f"- 描述: {r['description']}\n", + f"- 预期输出 ({r['expected_type']}): {r['expected']}\n", + f"- 实际输出: {r['actual']}\n", + f"- 输出是否匹配: {r['matched']}\n", + f"- 模拟后端: {r['backend']}\n", + f"- 汇编文件: {r['asm']}\n", + ]) + if benchmark_mode: + lines.extend([ + f"- Benchmark 重复次数: {r['benchmark_runs']}\n", + f"- 平均指令数: {r['avg_instr_count']:.2f}\n", + f"- 95% 置信区间: ±{r['ci95_instr_count']:.2f}\n", + f"- 最小指令数: {r['min_instr_count']}\n", + f"- 最大指令数: {r['max_instr_count']}\n", + f"- 基线指令数: {r['baseline_instr_count']:.2f}\n", + f"- 性能变化率: {r['delta_pct']:.2f}%\n", + f"- 性能退化阈值: {r['threshold_pct']:.2f}%\n", + f"- 是否性能退化: {r['regressed']}\n", + ]) + else: + lines.append(f"- 指令数: {r['instr_count']}\n") + lines.append("\n") + + return "".join(lines) + + +def write_html_report_cn(results, passed, failed): + try: + from jinja2 import Template + except ImportError: + return None + + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + template = Template(""" + + + + ScratchV 课程版测试报告 + + + +

ScratchV DSL 编译器性能测试报告

+
+

用例总数:{{ total }},通过:{{ passed }},失败:{{ failed }},通过率:{{ "%.1f"|format(pass_rate) }}%

+

测试目录:{{ test_dir }},性能退化阈值:{{ threshold }}%

+
+ 课程版指令数图表 + + + + + + {% for r in results %} + + + + + + + + + + + {% endfor %} + +
用例类别状态平均指令数95%置信区间变化率(%)是否退化描述
{{ r.name }}{{ r.category }}{{ r.status }}{{ "%.2f"|format(r.avg_instr_count) }}±{{ "%.2f"|format(r.ci95_instr_count) }}{{ "%.2f"|format(r.delta_pct) }}{{ r.regressed }}{{ r.description }}
+ + +""") + HTML_REPORT_FILE.write_text( + template.render( + total=len(results), + passed=passed, + failed=failed, + pass_rate=pass_rate, + test_dir=str(TEST_DIR), + threshold=REGRESSION_THRESHOLD_PCT, + chart_name=CHART_FILE.name, + results=results, + ), + encoding="utf-8", + ) + return HTML_REPORT_FILE + + +def generate_report_text_cn(results, passed, failed): + benchmark_mode = any(r.get("benchmark_runs", 1) > 1 for r in results) + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + lines = [ + "# ScratchV DSL 编译器性能测试报告\n\n", + "## 测试概览\n\n", + f"- 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n", + f"- 用例总数: {len(results)}\n", + f"- 通过数量: {passed}\n", + f"- 失败数量: {failed}\n", + f"- 通过率: {pass_rate:.1f}%\n", + f"- 测试目录: `{TEST_DIR}`\n", + f"- 汇编输出目录: `{BUILD_DIR}`\n", + f"- 性能基线文件: `{BASELINE_FILE}`\n", + f"- 性能退化阈值: {REGRESSION_THRESHOLD_PCT:.1f}%\n", + f"- 单次模拟超时: {SIMULATION_TIMEOUT_SEC:.0f}s\n\n", + "## 测试结果\n\n", + ] + + if benchmark_mode: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 平均指令数 | 95% 置信区间 | 最小 | 最大 | 编译耗时(s) | 模拟耗时(s) | 总耗时(s) | 基线 | 变化率(%) | 是否退化 | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|---|---|---|---|\n", + ]) + else: + lines.extend([ + "| 用例 | 类别 | 状态 | 模拟后端 | 指令数 | 编译耗时(s) | 模拟耗时(s) | 总耗时(s) | 预期输出 | 实际输出 | 输出匹配 | 汇编文件 |\n", + "|---|---|---|---|---:|---:|---:|---:|---|---|---|---|\n", + ]) + + for r in results: + expected = _markdown_cell(r["expected"]) + actual = _markdown_cell(r["actual"]) + if benchmark_mode: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['avg_instr_count']:.2f} | ±{r['ci95_instr_count']:.2f} | " + f"{r['min_instr_count']} | {r['max_instr_count']} | " + f"{r['compile_time_sec']:.4f} | {r['simulation_time_sec']:.4f} | " + f"{r['total_time_sec']:.4f} | {r['baseline_instr_count']:.2f} | " + f"{r['delta_pct']:.2f} | {r['regressed']} | {expected} | " + f"{actual} | {r['matched']} | {r['asm']} |\n" + ) + else: + lines.append( + f"| {r['name']} | {r['category']} | {r['status']} | {r['backend']} | " + f"{r['instr_count']} | {r['compile_time_sec']:.4f} | " + f"{r['simulation_time_sec']:.4f} | {r['total_time_sec']:.4f} | " + f"{expected} | {actual} | {r['matched']} | {r['asm']} |\n" + ) + + lines.extend([ + "\n## 性能图表\n\n", + f"![课程版指令数图表]({CHART_FILE.name})\n\n", + "### Mermaid 图表\n\n", + "```mermaid\n", + "xychart-beta\n", + ' title "各测试用例指令数"\n', + " x-axis [" + ", ".join(f'"{r["name"]}"' for r in results) + "]\n", + ]) + max_instr = max((r.get("avg_instr_count", r["instr_count"]) for r in results), default=0) + chart_values = [str(round(r.get("avg_instr_count", r["instr_count"]), 2)) for r in results] + lines.extend([ + f' y-axis "指令数" 0 --> {max_instr + 2}\n', + " bar [" + ", ".join(chart_values) + "]\n", + "```\n\n", + "## 用例详情\n\n", + ]) + + for r in results: + lines.extend([ + f"### {r['name']}\n\n", + f"- 类别: {r['category']}\n", + f"- 描述: {r['description']}\n", + f"- 预期输出 ({r['expected_type']}): {r['expected']}\n", + f"- 实际输出: {r['actual']}\n", + f"- 输出是否匹配: {r['matched']}\n", + f"- 模拟后端: {r['backend']}\n", + f"- 编译耗时(s): {r['compile_time_sec']:.4f}\n", + f"- 模拟耗时(s): {r['simulation_time_sec']:.4f}\n", + f"- 参考解释器耗时(s): {r['reference_time_sec']:.4f}\n", + f"- 总耗时(s): {r['total_time_sec']:.4f}\n", + f"- 汇编文件: {r['asm']}\n", + ]) + if benchmark_mode: + lines.extend([ + f"- Benchmark 重复次数: {r['benchmark_runs']}\n", + f"- 平均指令数: {r['avg_instr_count']:.2f}\n", + f"- 95% 置信区间: ±{r['ci95_instr_count']:.2f}\n", + f"- 最小指令数: {r['min_instr_count']}\n", + f"- 最大指令数: {r['max_instr_count']}\n", + f"- 基线指令数: {r['baseline_instr_count']:.2f}\n", + f"- 性能变化率: {r['delta_pct']:.2f}%\n", + f"- 性能退化阈值: {r['threshold_pct']:.2f}%\n", + f"- 是否性能退化: {r['regressed']}\n", + ]) + else: + lines.append(f"- 指令数: {r['instr_count']}\n") + lines.append("\n") + + return "".join(lines) + + +def write_html_report_cn(results, passed, failed): + try: + from jinja2 import Template + except ImportError: + return None + + pass_rate = 0.0 if not results else passed / len(results) * 100.0 + template = Template(""" + + + + ScratchV 测试报告 + + + +

ScratchV DSL 编译器性能测试报告

+
+

用例总数:{{ total }},通过:{{ passed }},失败:{{ failed }},通过率:{{ "%.1f"|format(pass_rate) }}%

+

测试目录:{{ test_dir }},性能退化阈值:{{ threshold }}%,单次模拟超时:{{ timeout }}s

+
+ 课程版指令数图表 + + + + + + + + + + + {% for r in results %} + + + + + + + + + + + + + + + {% endfor %} + +
用例类别状态模拟后端平均指令数95% 置信区间编译耗时(s)模拟耗时(s)总耗时(s)变化率(%)是否退化描述
{{ r.name }}{{ r.category }}{{ r.status }}{{ r.backend }}{{ "%.2f"|format(r.avg_instr_count) }}±{{ "%.2f"|format(r.ci95_instr_count) }}{{ "%.4f"|format(r.compile_time_sec) }}{{ "%.4f"|format(r.simulation_time_sec) }}{{ "%.4f"|format(r.total_time_sec) }}{{ "%.2f"|format(r.delta_pct) }}{{ r.regressed }}{{ r.description }}
+ + +""") + HTML_REPORT_FILE.write_text( + template.render( + total=len(results), + passed=passed, + failed=failed, + pass_rate=pass_rate, + test_dir=str(TEST_DIR), + threshold=REGRESSION_THRESHOLD_PCT, + timeout=SIMULATION_TIMEOUT_SEC, + chart_name=CHART_FILE.name, + results=results, + ), + encoding="utf-8", + ) + return HTML_REPORT_FILE + + +def write_report(results, passed, failed): + REPORT_DIR.mkdir(exist_ok=True) + write_chart(results) + REPORT_FILE.write_text(generate_report_text_cn(results, passed, failed), encoding="utf-8") + html_path = write_html_report_cn(results, passed, failed) + print(f"\nMarkdown report written to {REPORT_FILE}") + if html_path: + print(f"HTML report written to {html_path}") + + +def parse_args(argv=None): + parser = argparse.ArgumentParser(description="Run ScratchV DSL benchmark suite.") + parser.add_argument("--benchmark", type=int, default=0, metavar="N", + help="Run each case N times and report average instruction count.") + parser.add_argument("--update-baseline", action="store_true", + help="Write current benchmark averages to the baseline file.") + return parser.parse_args(argv) + + +def main(argv=None): + args = parse_args(argv) + BUILD_DIR.mkdir(exist_ok=True) + baseline = load_baseline() if args.benchmark else {} + + dsl_files = list(TEST_DIR.rglob("*.dsl")) + + if not dsl_files: + print("No DSL test files found.") + return + + passed = 0 + failed = 0 + results = [] + + print("Running DSL compiler tests...") + print("=" * 50) + + for dsl_file in dsl_files: + print(f"\n[TEST] {dsl_file}") + + case_start = time.perf_counter() + meta = load_metadata(dsl_file) + compile_start = time.perf_counter() + result, output_file = run_compile(dsl_file) + compile_time_sec = time.perf_counter() - compile_start + simulation_start = time.perf_counter() + sim_result = run_simulation(output_file) if result.returncode == 0 else { + "success": False, + "instr_count": 0, + "return_value": None, + "backend": "none", + "error": "compile failed", + } + simulation_time_sec = time.perf_counter() - simulation_start + expected_value = meta.get("expected_return") + reference_start = time.perf_counter() + actual_value = execute_dsl_reference(dsl_file, meta.get("inputs", {})) + reference_time_sec = time.perf_counter() - reference_start + matched = values_equal(actual_value, expected_value) + benchmark_counts = [] + benchmark_summary = { + "runs": 1, + "avg_instr_count": sim_result.get("instr_count", 0), + "min_instr_count": sim_result.get("instr_count", 0), + "max_instr_count": sim_result.get("instr_count", 0), + "ci95_instr_count": 0.0, + } + regression = { + "baseline_instr_count": 0.0, + "delta": 0.0, + "delta_pct": 0.0, + "threshold_pct": REGRESSION_THRESHOLD_PCT, + "regressed": False, + } + + if args.benchmark > 0 and result.returncode == 0 and output_file.exists(): + for _ in range(args.benchmark): + benchmark_counts.append(run_simulation(output_file).get("instr_count", 0)) + benchmark_summary = summarize_benchmark_runs(benchmark_counts) + baseline_entry = baseline.get(dsl_file.stem) + if baseline_entry: + regression = detect_regression( + avg_instr_count=benchmark_summary["avg_instr_count"], + baseline_instr_count=baseline_entry.get("avg_instr_count", 0.0), + ) + total_time_sec = time.perf_counter() - case_start + + ok = ( + result.returncode == 0 + and output_file.exists() + and sim_result["success"] + and matched + and not regression["regressed"] + ) + + if ok: + print("PASS") + passed += 1 + status = "PASS" + else: + print("FAIL") + failed += 1 + status = "FAIL" + print(sim_result.get("error") or result.stderr or result.stdout) + + results.append({ + "name": dsl_file.stem, + "category": dsl_file.parent.name, + "path": str(dsl_file), + "status": status, + "description": meta.get("description", ""), + "expected_type": meta.get("expected_output_type", "return_value"), + "expected": expected_value, + "actual": actual_value, + "matched": matched, + "backend": sim_result.get("backend", "none"), + "instr_count": sim_result.get("instr_count", 0), + "compile_time_sec": compile_time_sec, + "simulation_time_sec": simulation_time_sec, + "reference_time_sec": reference_time_sec, + "total_time_sec": total_time_sec, + "benchmark_runs": benchmark_summary["runs"], + "avg_instr_count": benchmark_summary["avg_instr_count"], + "min_instr_count": benchmark_summary["min_instr_count"], + "max_instr_count": benchmark_summary["max_instr_count"], + "ci95_instr_count": benchmark_summary["ci95_instr_count"], + "baseline_instr_count": regression["baseline_instr_count"], + "delta": regression["delta"], + "delta_pct": regression["delta_pct"], + "threshold_pct": regression["threshold_pct"], + "regressed": regression["regressed"], + "asm": str(output_file), + }) + + print("\n" + "=" * 50) + print(f"Total: {len(dsl_files)}") + print(f"Passed: {passed}") + print(f"Failed: {failed}") + + if args.benchmark and args.update_baseline: + save_baseline(results) + print(f"Baseline written to {BASELINE_FILE}") + + write_report(results, passed, failed) + + +if __name__ == "__main__": + main() diff --git a/ScratchV-topic06-deliverable/setup.py b/ScratchV-topic06-deliverable/setup.py new file mode 100644 index 0000000..6068493 --- /dev/null +++ b/ScratchV-topic06-deliverable/setup.py @@ -0,0 +1,3 @@ +from setuptools import setup + +setup() diff --git a/ScratchV-topic06-deliverable/tests_main/activation/add_relu_relu.dsl b/ScratchV-topic06-deliverable/tests_main/activation/add_relu_relu.dsl new file mode 100644 index 0000000..52c1325 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/add_relu_relu.dsl @@ -0,0 +1,5 @@ +# Add followed by two ReLU stages +x = add(input, bias) +y = relu(x) +result = relu(y) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/activation/add_relu_relu.meta.json b/ScratchV-topic06-deliverable/tests_main/activation/add_relu_relu.meta.json new file mode 100644 index 0000000..e587d8a --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/add_relu_relu.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Add input and bias, then apply ReLU twice.", + "expected_output_type": "return_value", + "inputs": { + "input": -3, + "bias": 10 + }, + "expected_return": 7 +} diff --git a/ScratchV-topic06-deliverable/tests_main/activation/relu_add.dsl b/ScratchV-topic06-deliverable/tests_main/activation/relu_add.dsl new file mode 100644 index 0000000..7190772 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/relu_add.dsl @@ -0,0 +1,4 @@ +# Add + ReLU activation +x = add(input, bias) +y = relu(x) +return y \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/tests_main/activation/relu_add.meta.json b/ScratchV-topic06-deliverable/tests_main/activation/relu_add.meta.json new file mode 100644 index 0000000..f1b7f84 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/relu_add.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Add input and bias, then apply one ReLU.", + "expected_output_type": "return_value", + "inputs": { + "input": -2, + "bias": 5 + }, + "expected_return": 3 +} diff --git a/ScratchV-topic06-deliverable/tests_main/activation/relu_only.dsl b/ScratchV-topic06-deliverable/tests_main/activation/relu_only.dsl new file mode 100644 index 0000000..404a911 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/relu_only.dsl @@ -0,0 +1,3 @@ +# Single ReLU activation +result = relu(x) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/activation/relu_only.meta.json b/ScratchV-topic06-deliverable/tests_main/activation/relu_only.meta.json new file mode 100644 index 0000000..b08af80 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/relu_only.meta.json @@ -0,0 +1,8 @@ +{ + "description": "Apply ReLU directly to a single input value.", + "expected_output_type": "return_value", + "inputs": { + "x": -5 + }, + "expected_return": 0 +} diff --git a/ScratchV-topic06-deliverable/tests_main/activation/relu_twice.dsl b/ScratchV-topic06-deliverable/tests_main/activation/relu_twice.dsl new file mode 100644 index 0000000..7d2defe --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/relu_twice.dsl @@ -0,0 +1,4 @@ +# Two-stage ReLU activation +x = relu(input) +result = relu(x) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/activation/relu_twice.meta.json b/ScratchV-topic06-deliverable/tests_main/activation/relu_twice.meta.json new file mode 100644 index 0000000..fabbb38 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/activation/relu_twice.meta.json @@ -0,0 +1,8 @@ +{ + "description": "Apply ReLU twice to the same activation path.", + "expected_output_type": "return_value", + "inputs": { + "input": 4 + }, + "expected_return": 4 +} diff --git a/ScratchV-topic06-deliverable/tests_main/branch/if_else.dsl b/ScratchV-topic06-deliverable/tests_main/branch/if_else.dsl new file mode 100644 index 0000000..6541768 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/branch/if_else.dsl @@ -0,0 +1,8 @@ +# Branch takes the else path when flag is zero. +if (flag != 0): +result = add(a, b) +return result +else +result = sub(a, b) +return result +endif diff --git a/ScratchV-topic06-deliverable/tests_main/branch/if_else.meta.json b/ScratchV-topic06-deliverable/tests_main/branch/if_else.meta.json new file mode 100644 index 0000000..fdacf6a --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/branch/if_else.meta.json @@ -0,0 +1,10 @@ +{ + "description": "if/else branch returns subtraction result when flag is zero.", + "expected_output_type": "return_value", + "inputs": { + "flag": 0, + "a": 9, + "b": 4 + }, + "expected_return": 5 +} diff --git a/ScratchV-topic06-deliverable/tests_main/branch/if_relu.dsl b/ScratchV-topic06-deliverable/tests_main/branch/if_relu.dsl new file mode 100644 index 0000000..1346060 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/branch/if_relu.dsl @@ -0,0 +1,8 @@ +# Branch selects whether to apply ReLU after an add. +sum = add(a, b) +if (use_relu != 0): +result = relu(sum) +return result +else +return sum +endif diff --git a/ScratchV-topic06-deliverable/tests_main/branch/if_relu.meta.json b/ScratchV-topic06-deliverable/tests_main/branch/if_relu.meta.json new file mode 100644 index 0000000..1925fb5 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/branch/if_relu.meta.json @@ -0,0 +1,10 @@ +{ + "description": "if/else branch combined with add and relu.", + "expected_output_type": "return_value", + "inputs": { + "use_relu": 1, + "a": -8, + "b": 3 + }, + "expected_return": 0 +} diff --git a/ScratchV-topic06-deliverable/tests_main/branch/if_then.dsl b/ScratchV-topic06-deliverable/tests_main/branch/if_then.dsl new file mode 100644 index 0000000..c305c3d --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/branch/if_then.dsl @@ -0,0 +1,8 @@ +# Branch takes the then path when flag is non-zero. +if (flag != 0): +result = add(a, b) +return result +else +result = sub(a, b) +return result +endif diff --git a/ScratchV-topic06-deliverable/tests_main/branch/if_then.meta.json b/ScratchV-topic06-deliverable/tests_main/branch/if_then.meta.json new file mode 100644 index 0000000..04748d1 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/branch/if_then.meta.json @@ -0,0 +1,10 @@ +{ + "description": "if/else branch returns add result when flag is non-zero.", + "expected_output_type": "return_value", + "inputs": { + "flag": 1, + "a": 9, + "b": 4 + }, + "expected_return": 13 +} diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain.dsl b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain.dsl new file mode 100644 index 0000000..d3917c9 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain.dsl @@ -0,0 +1,4 @@ +# Chain of two vector adds +x = add(a, b) +y = add(x, c) +return y \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain.meta.json b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain.meta.json new file mode 100644 index 0000000..b07adaf --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain.meta.json @@ -0,0 +1,10 @@ +{ + "description": "Add a and b, then add c to the intermediate result.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3, + "c": 4 + }, + "expected_return": 9 +} diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain_3.dsl b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain_3.dsl new file mode 100644 index 0000000..aeca4df --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain_3.dsl @@ -0,0 +1,5 @@ +# Chain of three add operations +x = add(a, b) +y = add(x, c) +z = add(y, d) +return z diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain_3.meta.json b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain_3.meta.json new file mode 100644 index 0000000..368b3aa --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_chain_3.meta.json @@ -0,0 +1,11 @@ +{ + "description": "Chain three add operations across four symbolic inputs.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3, + "c": 4, + "d": 5 + }, + "expected_return": 14 +} diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_fan_in_4.dsl b/ScratchV-topic06-deliverable/tests_main/elementwise/add_fan_in_4.dsl new file mode 100644 index 0000000..0e4449e --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_fan_in_4.dsl @@ -0,0 +1,5 @@ +# Fan-in add over four inputs +x = add(a, b) +y = add(c, d) +result = add(x, y) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_fan_in_4.meta.json b/ScratchV-topic06-deliverable/tests_main/elementwise/add_fan_in_4.meta.json new file mode 100644 index 0000000..3eb1f4a --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_fan_in_4.meta.json @@ -0,0 +1,11 @@ +{ + "description": "Compute two independent adds and then merge them with a final add.", + "expected_output_type": "return_value", + "inputs": { + "a": 1, + "b": 2, + "c": 3, + "d": 4 + }, + "expected_return": 10 +} diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_reuse.dsl b/ScratchV-topic06-deliverable/tests_main/elementwise/add_reuse.dsl new file mode 100644 index 0000000..4618948 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_reuse.dsl @@ -0,0 +1,4 @@ +# Reuse intermediate add result +x = add(a, b) +result = add(x, x) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/add_reuse.meta.json b/ScratchV-topic06-deliverable/tests_main/elementwise/add_reuse.meta.json new file mode 100644 index 0000000..2d2c396 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/add_reuse.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Reuse the same intermediate add result on both operands of a second add.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3 + }, + "expected_return": 10 +} diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/vector_add.dsl b/ScratchV-topic06-deliverable/tests_main/elementwise/vector_add.dsl new file mode 100644 index 0000000..36ac648 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/vector_add.dsl @@ -0,0 +1,3 @@ +# Vector add +result = add(a, b) +return result \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/tests_main/elementwise/vector_add.meta.json b/ScratchV-topic06-deliverable/tests_main/elementwise/vector_add.meta.json new file mode 100644 index 0000000..7347fff --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/elementwise/vector_add.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Single add over two symbolic vector inputs.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3 + }, + "expected_return": 5 +} diff --git a/ScratchV-topic06-deliverable/tests_main/loop/loop_add_4.dsl b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_4.dsl new file mode 100644 index 0000000..381f1c0 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_4.dsl @@ -0,0 +1,5 @@ +# Loop with repeated add body over 4 iterations +for i = 0, 4 +x = add(a, b) +endfor +return x diff --git a/ScratchV-topic06-deliverable/tests_main/loop/loop_add_4.meta.json b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_4.meta.json new file mode 100644 index 0000000..fc31fae --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_4.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Run a four-iteration loop whose body computes one add; final returned value is the last loop-body result.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3 + }, + "expected_return": 5 +} diff --git a/ScratchV-topic06-deliverable/tests_main/loop/loop_add_chain_4.dsl b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_chain_4.dsl new file mode 100644 index 0000000..eca595b --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_chain_4.dsl @@ -0,0 +1,6 @@ +# Loop with chained add inside the body +for i = 0, 4 +x = add(a, b) +y = add(x, c) +endfor +return y diff --git a/ScratchV-topic06-deliverable/tests_main/loop/loop_add_chain_4.meta.json b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_chain_4.meta.json new file mode 100644 index 0000000..154d91d --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/loop/loop_add_chain_4.meta.json @@ -0,0 +1,10 @@ +{ + "description": "Run a four-iteration loop whose body computes two chained adds; final returned value is the last loop-body result.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3, + "c": 4 + }, + "expected_return": 9 +} diff --git a/ScratchV-topic06-deliverable/tests_main/loop/loop_relu_add_4.dsl b/ScratchV-topic06-deliverable/tests_main/loop/loop_relu_add_4.dsl new file mode 100644 index 0000000..44f3080 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/loop/loop_relu_add_4.dsl @@ -0,0 +1,6 @@ +# Loop with add followed by ReLU in the body +for i = 0, 4 +x = add(input, bias) +y = relu(x) +endfor +return y diff --git a/ScratchV-topic06-deliverable/tests_main/loop/loop_relu_add_4.meta.json b/ScratchV-topic06-deliverable/tests_main/loop/loop_relu_add_4.meta.json new file mode 100644 index 0000000..8baca28 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/loop/loop_relu_add_4.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Run a four-iteration loop whose body computes add followed by ReLU; final returned value is the last loop-body result.", + "expected_output_type": "return_value", + "inputs": { + "input": -4, + "bias": 6 + }, + "expected_return": 2 +} diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_4.dsl b/ScratchV-topic06-deliverable/tests_main/reduction/dot_4.dsl new file mode 100644 index 0000000..2118c72 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_4.dsl @@ -0,0 +1,3 @@ +# Dot product of two 4-element vectors +result = dot(a, b, len:4) +return result \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_4.meta.json b/ScratchV-topic06-deliverable/tests_main/reduction/dot_4.meta.json new file mode 100644 index 0000000..748f260 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_4.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute the dot product of two symbolic vectors of length 4.", + "expected_output_type": "return_value", + "inputs": { + "a": [1, 2, 3, 4], + "b": [5, 6, 7, 8] + }, + "expected_return": 70 +} diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_8.dsl b/ScratchV-topic06-deliverable/tests_main/reduction/dot_8.dsl new file mode 100644 index 0000000..e1b71fa --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_8.dsl @@ -0,0 +1,3 @@ +# Dot product of two 8-element vectors +result = dot(a, b, len:8) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_8.meta.json b/ScratchV-topic06-deliverable/tests_main/reduction/dot_8.meta.json new file mode 100644 index 0000000..d3172a2 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_8.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute the dot product of two symbolic vectors of length 8.", + "expected_output_type": "return_value", + "inputs": { + "a": [1, 2, 3, 4, 5, 6, 7, 8], + "b": [1, 1, 1, 1, 1, 1, 1, 1] + }, + "expected_return": 36 +} diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_4.dsl b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_4.dsl new file mode 100644 index 0000000..279fc19 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_4.dsl @@ -0,0 +1,4 @@ +# Dot product followed by ReLU +x = dot(a, b, len:4) +result = relu(x) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_4.meta.json b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_4.meta.json new file mode 100644 index 0000000..24910bc --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_4.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute a length-4 dot product and pass it through ReLU.", + "expected_output_type": "return_value", + "inputs": { + "a": [1, -2, 3, -4], + "b": [2, 3, 4, 5] + }, + "expected_return": 0 +} diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_8.dsl b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_8.dsl new file mode 100644 index 0000000..d2b906d --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_8.dsl @@ -0,0 +1,4 @@ +# Dot product of length 8 followed by ReLU +x = dot(a, b, len:8) +result = relu(x) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_8.meta.json b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_8.meta.json new file mode 100644 index 0000000..776fab5 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/reduction/dot_relu_8.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute a length-8 dot product and pass it through ReLU.", + "expected_output_type": "return_value", + "inputs": { + "a": [1, 0, 1, 0, 1, 0, 1, 0], + "b": [2, 2, 2, 2, 2, 2, 2, 2] + }, + "expected_return": 8 +} diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_2x2.dsl b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_2x2.dsl new file mode 100644 index 0000000..f0a80d9 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_2x2.dsl @@ -0,0 +1,3 @@ +# Matrix multiplication: 2x2 * 2x2 +result = matmul(A, B, m:2, n:2, k:2) +return result \ No newline at end of file diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_2x2.meta.json b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_2x2.meta.json new file mode 100644 index 0000000..f970a61 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_2x2.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute a symbolic 2x2 by 2x2 matrix multiplication.", + "expected_output_type": "return_value", + "inputs": { + "A": [[1, 2], [3, 4]], + "B": [[5, 6], [7, 8]] + }, + "expected_return": [[19, 22], [43, 50]] +} diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_4x4.dsl b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_4x4.dsl new file mode 100644 index 0000000..ab5a865 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_4x4.dsl @@ -0,0 +1,3 @@ +# Matrix multiplication: 4x4 * 4x4 +result = matmul(A, B, m:4, n:4, k:4) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_4x4.meta.json b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_4x4.meta.json new file mode 100644 index 0000000..ead52b9 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_4x4.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute a symbolic 4x4 by 4x4 matrix multiplication.", + "expected_output_type": "return_value", + "inputs": { + "A": [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]], + "B": [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]] + }, + "expected_return": [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]] +} diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_add_2x2.dsl b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_add_2x2.dsl new file mode 100644 index 0000000..d41a2cb --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_add_2x2.dsl @@ -0,0 +1,4 @@ +# Matrix multiplication followed by add +x = matmul(A, B, m:2, n:2, k:2) +result = add(x, bias) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_add_2x2.meta.json b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_add_2x2.meta.json new file mode 100644 index 0000000..9bca125 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_add_2x2.meta.json @@ -0,0 +1,10 @@ +{ + "description": "Compute a 2x2 matmul and then add a symbolic bias term.", + "expected_output_type": "return_value", + "inputs": { + "A": [[1, 2], [3, 4]], + "B": [[5, 6], [7, 8]], + "bias": [[1, 1], [1, 1]] + }, + "expected_return": [[20, 23], [44, 51]] +} diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_relu_2x2.dsl b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_relu_2x2.dsl new file mode 100644 index 0000000..28c7c61 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_relu_2x2.dsl @@ -0,0 +1,4 @@ +# Matrix multiplication followed by ReLU +x = matmul(A, B, m:2, n:2, k:2) +result = relu(x) +return result diff --git a/ScratchV-topic06-deliverable/tests_main/tensor/matmul_relu_2x2.meta.json b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_relu_2x2.meta.json new file mode 100644 index 0000000..352dfa6 --- /dev/null +++ b/ScratchV-topic06-deliverable/tests_main/tensor/matmul_relu_2x2.meta.json @@ -0,0 +1,9 @@ +{ + "description": "Compute a 2x2 matmul and then apply ReLU to its result.", + "expected_output_type": "return_value", + "inputs": { + "A": [[-1, 2], [-3, 4]], + "B": [[1, 0], [0, 1]] + }, + "expected_return": [[0, 2], [0, 4]] +} diff --git "a/ScratchV-topic06-deliverable/\350\257\276\351\242\2306\350\256\276\350\256\241\346\226\207\346\241\243.md" "b/ScratchV-topic06-deliverable/\350\257\276\351\242\2306\350\256\276\350\256\241\346\226\207\346\241\243.md" new file mode 100644 index 0000000..334990c --- /dev/null +++ "b/ScratchV-topic06-deliverable/\350\257\276\351\242\2306\350\256\276\350\256\241\346\226\207\346\241\243.md" @@ -0,0 +1,472 @@ +# ScratchV 课题 06 性能测试套件设计文档 + +> 文档版本:v1.1 +> 编写日期:2026-07-22 +> 更新日期:2026-08-06 +> 交付目录:`ScratchV-topic06-deliverable` +> 核心文件:`run_tests.py`、`tests_main/`、`reports/`、`README.md` +> 功能范围:DSL 用例管理、自动编译、TinyFive 模拟验证、期望输出检查、指令数统计、耗时统计、Benchmark、性能基线对比、Markdown/HTML 报告生成、CI 示例 + +--- + +## 一、功能介绍 + +### 1.1 功能概述 + +本测试套件用于验证 ScratchV 编译器在多个 DSL 程序上的编译结果和性能表现。测试脚本会自动遍历 `tests_main/` 下的 DSL 用例,调用 ScratchV 编译器生成 RISC-V 汇编,再通过 TinyFive 适配层进行模拟验证,并生成可读的测试报告。 + +当前套件包含 23 个 DSL 测试用例,覆盖以下类别: + +| 类别 | 说明 | +|---|---| +| `activation` | ReLU 等激活函数组合 | +| `elementwise` | 标量/向量加法、链式加法、复用输入 | +| `loop` | `for/endfor` 循环类用例 | +| `branch` | `if/else/endif` 分支类用例 | +| `reduction` | `dot` 等归约类用例 | +| `tensor` | `matmul` 等张量计算类用例 | + +测试输出不仅包含 PASS/FAIL,还包含指令数、编译耗时、模拟耗时、总耗时、Benchmark 平均值、95% 置信区间、性能基线对比结果等信息。 + +### 1.2 设计目标 + +- **自动化**:一条命令完成所有用例的编译、模拟、结果比对和报告生成。 +- **可扩展**:新增测试只需要添加 `.dsl` 和 `.meta.json` 文件,不需要修改主测试逻辑。 +- **可解释**:报告中展示每个用例的状态、预期输出、实际输出、模拟后端、指令数和耗时。 +- **可对比**:支持保存 Benchmark 基线,并在后续运行中检查性能变化率。 +- **避免卡死**:TinyFive 模拟阶段设置 5 秒超时,分支用例即使卡住也不会阻塞整套测试。 + +--- + +## 二、目录结构 + +交付目录当前结构如下: + +```text +ScratchV-topic06-deliverable/ + README.md + 课题6设计文档.md + LICENSE + requirements-topic06.txt + run_tests.py + setup.py + tests_main/ + activation/ + branch/ + elementwise/ + loop/ + reduction/ + tensor/ + reports/ + report.md + report.html + course_report_instructions.png + benchmark_baseline.json + .github/ + workflows/ + benchmark.yml +``` + +其中: + +- `run_tests.py` 是测试主程序。 +- `tests_main/` 保存 DSL 测试用例。 +- `reports/` 保存测试运行后生成的报告和基线文件。 +- `requirements-topic06.txt` 保存本套件额外依赖。 +- `.github/workflows/benchmark.yml` 是 GitHub Actions 示例配置。 + +`build/` 目录如果存在,是运行测试时生成的汇编输出目录,不是必须提交的源码内容。 + +--- + +## 三、测试用例格式 + +每个测试用例由两个文件组成: + +```text +tests_main/{category}/{name}.dsl +tests_main/{category}/{name}.meta.json +``` + +示例 DSL: + +```text +result = add(a, b) +return result +``` + +示例元数据: + +```json +{ + "description": "Simple scalar add.", + "expected_output_type": "return_value", + "inputs": { + "a": 2, + "b": 3 + }, + "expected_return": 5 +} +``` + +这里没有采用单独的 `.expected` 和 `.desc` 文件,而是使用 `.meta.json` 统一保存输入、描述和期望输出。这样结构更集中,也方便 Python 脚本直接解析。 + +--- + +## 四、核心实现设计 + +### 4.1 编译阶段 + +函数:`run_compile(dsl_file: Path)` + +作用: + +1. 接收一个 `.dsl` 文件路径。 +2. 调用 ScratchV 编译器: + +```powershell +python -m scratchv.main -o --optimize all --dump-ir +``` + +3. 将生成的 RISC-V 汇编写入 `build/{case_name}.s`。 +4. 返回编译进程结果和汇编输出路径。 + +这一阶段使用 `subprocess.run()` 调用编译器,能够捕获 `stdout`、`stderr` 和返回码。 + +当前版本的编译阶段还没有设置 `timeout` 参数。如果编译器本身在某个 DSL 上卡死,测试进程仍可能被阻塞。mentor review 中建议为编译阶段增加 30 秒超时,这是后续需要补齐的架构风险点。模拟阶段已经有 5 秒超时,但它只能保护 TinyFive 模拟,不能保护编译器调用。 + +### 4.2 模拟验证阶段 + +函数:`run_simulation(asm_file: Path, timeout: float = 5.0)` + +作用: + +1. 读取编译生成的 `.s` 汇编文件。 +2. 在独立 Python 子进程中调用: + +```python +from scratchv.simulator.tinyfive import verify_assembly +``` + +3. 获取 TinyFive 适配层返回的模拟结果。 +4. 如果模拟超过 5 秒,则终止该子进程,并返回: + +```python +{ + "success": False, + "backend": "timeout", + "error": "simulation timeout after 5s" +} +``` + +这个设计解决了 branch 分支用例在真实 TinyFive 路径下可能长时间不返回的问题。超时后该用例会被标记为 FAIL,但后续用例仍会继续执行,整套测试不会卡死。 + +### 4.3 期望输出辅助检查 + +函数:`execute_dsl_reference(dsl_file: Path, inputs)` + +当前 `run_tests.py` 中仍保留了一个简化 DSL 执行器,用来根据 `.dsl` 文件和 `.meta.json` 输入计算一个辅助输出,再和 `expected_return` 比较。它的作用是检查测试用例自身的输入和期望值是否写对。 + +需要注意:这个辅助执行器不是最终后端。最终后端仍然是 TinyFive 适配层,整体 PASS 必须满足 TinyFive 模拟成功。如果 TinyFive 超时或返回失败,即使辅助输出与期望值匹配,该用例仍然判定为 FAIL。 + +例如: + +```text +result = add(a, b) +return result +``` + +输入: + +```json +{ + "a": 2, + "b": 3 +} +``` + +辅助执行器会直接算出: + +```text +5 +``` + +报告中的“实际输出”和“输出是否匹配”来自这个辅助执行器,不是 TinyFive 的寄存器返回值。这样设计可以帮助发现 `.meta.json` 写错的问题,但也带来 mentor review 中提到的语义脱节风险:同一 DSL 语义同时存在编译器路径和 Python 辅助执行路径。后续更理想的方案是让 TinyFive 直通结果成为输出比较来源,辅助执行器只作为调试工具或被移除。 + +### 4.4 PASS/FAIL 判断 + +一个用例通过需要同时满足: + +```python +ok = ( + result.returncode == 0 + and output_file.exists() + and sim_result["success"] + and matched + and not regression["regressed"] +) +``` + +也就是说,必须同时满足: + +- 编译成功。 +- 汇编文件生成成功。 +- TinyFive 模拟成功。 +- 辅助输出检查与期望输出匹配。 +- Benchmark 模式下没有超过性能退化阈值。 + +如果出现: + +```text +状态: FAIL +模拟后端: timeout +输出是否匹配: True +``` + +含义是:`.meta.json` 期望值和辅助输出是一致的,但生成的汇编在 TinyFive 模拟阶段没有正常结束,因此整体仍然失败。 + +### 4.5 指令数统计 + +模拟结果中包含 `instr_count` 字段。测试脚本会把该字段写入报告。 + +在普通模式下,报告显示单次指令数: + +```text +指令数 +``` + +在 Benchmark 模式下,报告显示: + +```text +平均指令数 +最小指令数 +最大指令数 +95% 置信区间 +``` + +相关函数: + +- `summarize_benchmark_runs(instr_counts)` +- `detect_regression(avg_instr_count, baseline_instr_count)` + +报告字段采用同一套结果字典保存,普通模式和 Benchmark 模式共用字段名。普通模式下 `benchmark_runs` 为 1,平均/最小/最大指令数等字段使用单次模拟结果或默认值;Benchmark 模式下这些字段填入重复运行后的统计值。这样可以减少报告解析脚本在两种模式之间切换时出现 schema 不一致的问题。 + +### 4.6 耗时统计 + +测试脚本使用 `time.perf_counter()` 记录每个用例的耗时。 + +当前记录的字段包括: + +| 字段 | 说明 | +|---|---| +| `compile_time_sec` | 编译 DSL 到 RISC-V 汇编的耗时 | +| `simulation_time_sec` | TinyFive 模拟验证耗时 | +| `reference_time_sec` | 辅助输出检查耗时 | +| `total_time_sec` | 单个用例完整处理耗时 | + +这些字段会输出到 Markdown 和 HTML 报告中,便于观察哪些用例编译慢、模拟慢或因为 timeout 导致耗时较长。 + +### 4.7 Benchmark 与性能基线 + +运行: + +```powershell +python run_tests.py --benchmark 3 +``` + +表示每个用例重复模拟 3 次,统计平均指令数和 95% 置信区间。 + +运行: + +```powershell +python run_tests.py --benchmark 3 --update-baseline +``` + +表示将当前 Benchmark 结果保存为性能基线: + +```text +reports/benchmark_baseline.json +``` + +后续再次运行: + +```powershell +python run_tests.py --benchmark 3 +``` + +脚本会读取基线文件,并计算变化率: + +```text +变化率 = (当前平均指令数 - 基线指令数) / 基线指令数 * 100% +``` + +如果变化率超过 5%,则认为发生性能退化。当前阈值在代码中由 `REGRESSION_THRESHOLD_PCT = 5.0` 固定配置,暂时还没有命令行参数。mentor review 建议后续增加 `--regression-threshold`,让不同环境可以自行调整退化判断标准。 + +Benchmark 模式和 timeout 用例存在交互问题:当前 `--benchmark 3` 会对每个编译成功的用例重复模拟 3 次。如果某个 branch 用例每次都会 5 秒超时,那么一个用例会等待约 15 秒。后续建议在首次 timeout 后跳过该用例的剩余 Benchmark 重复次数,避免更新基线时长时间等待。 + +--- + +## 五、报告生成设计 + +### 5.1 Markdown 报告 + +输出文件: + +```text +reports/report.md +``` + +内容包括: + +- 测试概览。 +- 用例总数、通过数量、失败数量、通过率。 +- 每个用例的状态、模拟后端、指令数、耗时、预期输出、实际输出。 +- Benchmark 模式下的平均指令数、置信区间、基线、变化率和是否退化。 +- Mermaid 指令数图表。 +- 每个用例的详细说明。 + +### 5.2 HTML 报告 + +输出文件: + +```text +reports/report.html +``` + +HTML 报告主要用于浏览器查看,报告中会使用不同颜色标识 PASS 和 FAIL。当前实现已经改为在脚本内生成简单 HTML,不强制依赖复杂模板文件。 + +### 5.3 图表 + +输出文件: + +```text +reports/course_report_instructions.png +``` + +图表由 `matplotlib` 生成,展示不同测试用例的指令数对比。mentor review 中指出图表依赖会增加 CI 安装成本,因此后续可以把 Markdown 文本报告作为默认产物,把 HTML 和图表作为完整报告模式输出。 + +--- + +## 六、运行方式 + +### 6.1 安装依赖 + +```powershell +pip install -r requirements-topic06.txt +``` + +### 6.2 普通测试 + +```powershell +python run_tests.py +``` + +适合快速查看所有用例的 PASS/FAIL 和基本报告。 + +### 6.3 Benchmark 测试 + +```powershell +python run_tests.py --benchmark 3 +``` + +适合统计平均指令数和置信区间。 + +### 6.4 更新性能基线 + +```powershell +python run_tests.py --benchmark 3 --update-baseline +``` + +适合在修改编译器前或稳定版本上保存基线,供后续性能回归对比使用。 + +--- + +## 七、Mentor Review 对应状态 + +mentor review 中提出的问题分为架构、API、实现和测试覆盖四类。当前文档按真实代码状态整理如下: + +| 编号 | 问题 | 当前处理状态 | 说明 | +|---|---|---|---| +| S1 | 参考解释器与编译器语义脱节 | 部分处理,仍需收敛 | 当前真实模拟由 TinyFive 执行,但 `execute_dsl_reference()` 仍用于辅助检查 `.meta.json` 期望值。后续应让 TinyFive 返回值成为输出比较来源,减少双语义实现。 | +| S2 | 编译阶段缺 timeout | 待处理 | 模拟阶段已有 5 秒超时,编译阶段暂未设置 timeout,建议后续为 `run_compile()` 增加约 30 秒限制。 | +| S3 | Benchmark/普通模式报告格式冲突 | 已在文档中明确 | 两种模式共用结果字段,普通模式填单次结果或默认值,Benchmark 模式填统计字段。 | +| A1 | `.meta.json` 期望值表达能力有限 | 部分支持,待规范 | 当前 `expected_return` 可保存标量,也可保存数组类 JSON 值,但文档还需要进一步明确向量/张量输出的标准写法。 | +| A2 | `verify_assembly` 调用风险 | 已补充说明 | 即使 TinyFive 存在内部指令上限,外层仍需要 5 秒子进程 timeout 兜底。 | +| I1 | 报告依赖过重 | 待优化 | 当前保留 Markdown、HTML、PNG 图表。后续可把 Markdown 设为默认轻量输出,把 HTML/图表改为可选完整报告。 | +| I2 | 5% 退化阈值过于刚性 | 待优化 | 当前阈值固定为 `REGRESSION_THRESHOLD_PCT = 5.0`,后续建议增加 `--regression-threshold` 参数。 | +| I3 | Benchmark 重复运行 timeout 用例 | 待优化 | 当前 timeout 用例会重复等待,后续建议首次 timeout 后跳过剩余重复次数。 | +| T1 | 编译器 crash 无上下文保存 | 待处理 | 当前只捕获进程输出,后续可把 crash 的 stderr/stdout 保存到独立日志文件。 | +| T2 | 不支持增量/子集测试 | 待处理 | 当前递归运行所有 DSL 用例,后续建议增加 `--filter` 或 `--category`。 | + +--- + +## 八、当前测试现状 + +最近一次运行结果: + +```text +Total: 23 +Passed: 20 +Failed: 3 +``` + +失败的 3 个用例均位于 `branch/` 目录: + +```text +branch/if_else.dsl +branch/if_relu.dsl +branch/if_then.dsl +``` + +失败原因是 TinyFive 模拟阶段 5 秒超时: + +```text +simulation timeout after 5s +``` + +这 3 个用例的辅助输出检查与期望输出是匹配的,说明测试用例输入和期望值本身没有暴露问题;失败点主要在当前 ScratchV 分支汇编与 TinyFive 适配层之间没有完全对齐,导致真实模拟阶段无法正常结束。 + +其余 20 个用例可以完成编译、模拟和结果对比。 + +--- + +## 九、CI 集成 + +目录中包含 GitHub Actions 示例: + +```text +.github/workflows/benchmark.yml +``` + +CI 的目标是在 push 或 pull request 时自动运行测试套件,并上传测试报告,方便查看每次修改后的正确性和性能变化。 + +--- + +## 十、已完成工作对照 + +| 阶段 | 完成情况 | +|---|---| +| W1 学习编译命令和 TinyFive 用法 | 已完成 | +| W2 使用 subprocess 自动调用编译器和模拟器 | 已完成 | +| W3 设计测试用例格式 | 已完成,采用 `.dsl + .meta.json` | +| W4 遍历用例并输出 PASS/FAIL | 已完成 | +| W5 提取指令数并加入报告 | 已完成 | +| W6 matplotlib 图表和 HTML 报告 | 已完成 | +| W7 扩充到 15 个以上用例并覆盖分支循环 | 已完成,共 23 个 | +| W8 使用 `time.perf_counter()` 统计耗时 | 已完成 | +| W9 回归测试模式和 5% 退化阈值 | 已完成 | +| W10 `--benchmark` 重复运行和置信区间 | 已完成 | +| W11 GitHub Actions CI 示例 | 已完成 | +| W12 完整文档 | 已完成 | + +--- + +## 十一、后续改进建议 + +1. 修复 branch 分支汇编与 TinyFive 适配层的跳转/label 对齐问题,使 23 个用例都能在真实 TinyFive 路径下通过。 +2. 为 `run_compile()` 增加编译阶段 timeout,避免编译器异常时卡住整套测试。 +3. 在 Benchmark 模式下对已知 timeout 用例减少重复模拟次数,避免更新基线时等待时间过长。 +4. 增加 `--filter` 或 `--category` 参数,支持只运行部分用例。 +5. 增加 `--regression-threshold` 参数,让性能退化阈值可配置。 +6. 将 `.meta.json` 格式进一步文档化,例如明确支持的 `expected_output_type`、输入类型和算子范围。 +7. 在报告中增加“失败原因汇总”模块,集中展示 timeout、编译失败和输出不匹配的用例。