SimAI_analytical hangs / becomes extremely slow for tp=2 configs

[habby1012]

Hi, I ran into a strange performance issue with SimAI_analytical.

Environment

• SimAI commit: f5efb5a
• Ubuntu 24.04.4 LTS
• GCC 13.3.0
• Built with ./scripts/build.sh -c analytical

I'm running a 256-GPU Megatron training workload.

For example:

• dp=8, tp=4, pp=8 finishes in about 1 second
• dp=8, tp=2, pp=16 did not finish after 2–3 hours

Other configs with tp >= 4 also finish within a few seconds. So far I only see this behavior with tp=2.

The slow case does not have a larger workload either:

• tp=2: 781 workload entries, 776 ALLREDUCE ops
• tp=4: 1421 workload entries, 1416 ALLREDUCE ops
• tp=8: 3085 workload entries, 3080 ALLREDUCE ops

Memory usage stays around 5 MB during the slow run, while one CPU core stays busy.

I traced it to the event loop in:

astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.cc

queue<struct CallTask> call_list;
uint64_t tick = 0;

void AnaSim::Run() {
    while (!call_list.empty()) {
        CallTask calltask = call_list.front();

        while (true) {
            if (calltask.time != tick) {
                tick++;
            } else {
                break;
            }
        }

        call_list.pop();
        calltask.fun_ptr(calltask.fun_arg);
    }
}

A couple of things here look suspicious to me:

1. tick is incremented one by one until it reaches calltask.time. If the timestamp is large, this can result in a very large number of iterations.

2. call_list is a FIFO queue. If an event with calltask.time < tick is popped, tick++ can no longer reach that value.

I also noticed that Sys::generate_time() takes int cycles:

timespec_t generate_time(int cycles);

I'm not sure if this is related, but some delay values seem to come from 64-bit values.

Has anyone seen this behavior before? Is this expected in the analytical backend?

Thanks for taking a look!

[tianhao909]

【Manual reply】【人工回复】
Q1: Has anyone seen this behavior before? Is this expected?
No, this is not expected behavior; it is a bug.

Q2: Does incrementing tick one by one cause excessive iterations?
yes agree

Q3: Does popping an event with time < tick from the FIFO queue cause tick++ to never match?
yes agree

Q4: Is generate_time(int cycles) related to 64-bit delays?
yes agree

Minimal Fix Plan :
Change A (mandatory): eliminate the 32-bit truncation
Change B (recommended): O(1) jump + out-of-order guard in the AnaSim loop
Change C (consistency): widen the event-registration interfaces to 64-bit

[tianhao909]

[Reply from Agent A] [For reference only]

en

SimAI Issue #300 Analysis and Fix Plan

Issue: SimAI_analytical hangs / becomes extremely slow for tp=2 configs
Status: open | Reporter environment: Ubuntu 24.04.4, GCC 13.3.0, commit f5efb5a, built with ./scripts/build.sh -c analytical

---

1. Background Summary

While simulating a 256-GPU Megatron training workload with SimAI_analytical (analytical backend), the user observed:

Config	Workload entries	ALLREDUCE ops	Behavior
dp=8, tp=2, pp=16	781	776	Does not finish after 2–3 hours
dp=8, tp=4, pp=8	1421	1416	Finishes in ~1 second
Other tp≥4 configs	—	—	All finish within seconds

During the slow run: memory stays at ~5 MB while one CPU core stays at 100% — the classic signature of an infinite loop / astronomically large iteration count, not a data-volume problem (the tp=2 workload is actually smaller).

The user traced the event loop in AnaSim.cc and raised three suspicions:

1. tick is incremented one by one to catch up with event timestamps — huge iteration counts for large timestamps;
2. call_list is a FIFO — once an event with time < tick is popped, tick++ can never reach it again → infinite loop;
3. Sys::generate_time() takes int cycles while some delay values originate from 64-bit values.

This analysis confirms: suspicion #3 is the direct root cause of the hang, while #1/#2 are amplifiers and structural hazards. This is not expected behavior — it is a real defect in the analytical backend.

---

2. Affected Files / Modules

File	Role
astra-sim-alibabacloud/astra-sim/system/Sys.cc / Sys.hh	Core defect: 32-bit truncation in generate_time(int); int cycles parameters of register_event / zero_latecy_register_event
astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.cc / AnaSim.h	Structural defect: FIFO queue + monotonic tick incremented one-by-one
astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalNetwork.cc	sim_schedule() → AnaSim::Schedule(delta.time_val, ...); time_val is passed directly as the delay
astra-sim-alibabacloud/astra-sim/workload/Workload.cc	try_register_event(this, Workload_Wait, NULL, counter); counter (Tick) comes from compute times in the workload file
astra-sim-alibabacloud/astra-sim/workload/Layer.cc	register_event(..., weight_grad_update_time) (one of the Tick → int truncation sites); under ANALYTI mode issue_*_comm produces no communication event stream
astra-sim-alibabacloud/astra-sim/system/MemBus.cc / LogGP.cc	Other register_event call sites (ANALYTI mode uses small fixed delays; under ns3/phynet, LogGP's G*(size-1) can likewise exceed 2^31)

The ANALYTICAL-mode event chain (#ifdef ANALYTI, built via ./scripts/build.sh -c analytical):
Workload::fire() → for each layer/state counter = compute_time (from the workload file, in ns, CLOCK_PERIOD=1) → Sys::try_register_event(Tick&) → event_queue[tick+cycles] + generate_time(cycles) → AnalyticalNetWork::sim_schedule() → AnaSim::Schedule() (FIFO push) → consumed by AnaSim::Run().
Communication produces no event stream in ANALYTI mode (Layer::issue_*_comm returns immediately; communication time is only used for reporting), so all delays on the event chain come from workload compute-time values.

---

3. Root-Cause Analysis

3.1 Direct root cause: 32-bit truncation in generate_time(int cycles)

// Sys.hh:376
timespec_t generate_time(int cycles);

// Sys.cc:2142-2147
AstraSim::timespec_t Sys::generate_time(int cycles) {
  timespec_t tmp = NI->sim_get_time();
  double addition = cycles * ((double)CLOCK_PERIOD);   // CLOCK_PERIOD = 1
  tmp.time_val = addition;                              // double → uint64_t
  return tmp;
}

Two lossy conversions sit on this path:

1. Sys::try_register_event(Callable*, EventType, CallData*, Tick& cycles) receives the full 64-bit delay (e.g. a workload entry's compute time), and the event_queue[boostedTick() + cycles] key is the correct 64-bit value; but the subsequent call generate_time(cycles) performs an implicit Tick(uint64) → int truncation.
2. The resulting negative number then flows through a double → uint64_t conversion (out-of-range is UB; on x86-64 it wraps per cvttsd2si) producing an astronomical value of ~1.8e19.

Measured semantics on x86-64/GCC (matching the reporter's environment):

Delay in workload (ns)	After (int) truncation	AnaSim event time	Consequence
7,956,367 (normal layer compute)	unchanged	tick + 7,956,367	OK
1,000,000,000	unchanged	OK	OK
2,147,483,647 (2^31−1)	unchanged	OK	OK (boundary)
2,147,483,648 (2^31)	−2,147,483,648	≈ 1.8446744e19	tick++ needs ~1.8e19 iterations → never finishes
2,500,000,000	−1,794,967,296	≈ 1.8446744e19	Same (hang)
4,302,923,663 (2^32+x)	+x (smaller)	time corrupted	May trigger the FIFO out-of-order infinite loop

For an event time of ~1.8e19, the while (true) { if (time != tick) tick++; else break; } loop in AnaSim::Run() must iterate ~1.8e19 times (~hundreds of years at ~1e9 iterations/sec on one core). Observed behavior: one core at 100%, flat memory, never terminates — an exact match for the user's report.

3.2 Why only tp=2 triggers it

In AICB/Megatron workloads, each entry's compute time is the per-GPU compute time, roughly inversely proportional to tp (tensor-parallel sharding). For the same model:

• At tp=4, a single step's compute is ~1.2e9 ns (< 2^31, safe);
• At tp=2, the same step doubles to ~2.4e9 ns (≥ 2^31, out of range).

That is, tp=2 pushes one or more large compute entries (e.g. embedding/lm_head, long-sequence entries, or a large operator profiled by AIOB) just past the 2^31 threshold, while all tp≥4 values stay below it. This explains the counter-intuitive "smaller workload but slower/hung" observation — the decisive factor is not the number of entries, but whether any single value is ≥ 2^31.

The in-repo example example/workload_analytical.txt (header model_parallel_NPU_group: 2 — itself a tp=2 config) already contains several values ≥ 2^31 (lines 3/4/7/8, comm_size values 2807758848, 5615517696, etc.). comm_size itself does not enter the event chain, but it shows such magnitudes are common in real workloads; the reporter's workload should contain compute times ≥ 2^31 (see Section 8).

3.3 Structural defects (the user's suspicions #1/#2 are valid)

1. One-by-one tick increment (performance): AnaSim::Run() catches up with event timestamps via tick++, giving O(total simulated time) complexity. Even without truncation, a total simulated time of ~seconds (~1e9 ticks) means ~1e9 idle loop iterations — which is why tp=4 still takes ~1 second.
2. FIFO + monotonic tick (infinite-loop hazard): call_list is consumed in push order, not time order, and tick never decreases. As soon as a front event has time < tick (e.g. when a truncated delay > 2^32 becomes smaller, or via future out-of-order enqueueing across multiple chains), the inner while(true) spins forever. The current ANALYTI chain is serial (single monotonic chain), so out-of-order insertion is unlikely to occur naturally — but the truncation bug can manufacture exactly this situation.
3. Schedule semantics: time = tick + delay is itself correct; generate_time's naming is misleading (it returns a relative delay, not an absolute time). The ns3 backend (Simulator::Schedule(NanoSeconds(t.schTime))) also treats the return value as a delay, so both backends agree semantically — meaning the truncation bug also affects the ns3/phynet backends' large-delay paths (e.g. LogGP's G*(size-1) for large messages).

3.4 Point-by-point response to the user's questions (suggested reply)

• Q1: Has anyone seen this behavior before? Is this expected in the analytical backend?
  It is not expected behavior — it is a bug. The hang is triggered by 32-bit truncation in Sys::generate_time(int): when any workload entry's compute time is ≥ 2^31 ns (~2.1 s), the scheduled event timestamp wraps to ~1.8e19 and AnaSim::Run()'s tick loop can never catch up. tp=2 doubles per-GPU compute times versus tp=4, crossing the 2^31 threshold.
• Q2: tick incremented one by one causes a very large number of iterations?
  Correct — this is an O(total simulated time) performance issue; the fix is to jump directly (plan B below). It only explains "seconds", not "hours" — the latter is the truncation-induced hang.
• Q3: If an event with calltask.time < tick is popped, tick++ can no longer reach that value?
  The analysis is exactly right — this is a potential infinite loop. The current ANALYTI event chain is serial (single monotonic chain), so out-of-order events do not naturally occur; however, the truncation bug produces wrapped timestamps (a delay > 2^32 becomes smaller) that trigger precisely the dead loop you describe. A defensive clamp for time < tick is recommended.
• Q4: generate_time(int cycles) related to 64-bit delay values?
  This is the root cause. The downstream generate_time(int) of register_event/try_register_event implicitly truncates Tick(uint64); any delay ≥ 2^31 corrupts the timeline. Fix: widen the parameter to Tick.

---

4. Minimal Fix Plan

Principles: minimal diff, preserve existing code style, no new dependencies. The three changes are independent; change A is mandatory (fixes the hang), B is strongly recommended (performance + defense), C is for interface consistency.

Change A (mandatory): eliminate the 32-bit truncation

astra-sim/system/Sys.hh (line 376) and Sys.cc (line 2142):

- timespec_t generate_time(int cycles);
+ timespec_t generate_time(Tick cycles);

- AstraSim::timespec_t Sys::generate_time(int cycles) {
+ AstraSim::timespec_t Sys::generate_time(Tick cycles) {
    timespec_t tmp = NI->sim_get_time();
-   double addition = cycles * ((double)CLOCK_PERIOD);
-   tmp.time_val = addition;
+   tmp.time_val = cycles * CLOCK_PERIOD;
    return tmp;
  }

This also removes the misleading "fetch current time then overwrite it" pattern (behaviorally equivalent: with CLOCK_PERIOD=1, time_val = cycles; Tick is unsigned long long, so the multiplication no longer truncates).

Change B (strongly recommended): O(1) jump + out-of-order guard in the AnaSim loop

astra-sim/network_frontend/analytical/AnaSim.cc (lines 22–40):

  void AnaSim::Run() {
    while (!call_list.empty())
    {
      CallTask calltask = call_list.front();
-     while (true) {
-       if (calltask.time != tick) {
-         tick++;
-       } else {
-         break;
-       }
-     }
+     if (calltask.time > tick) {
+       tick = calltask.time;   // O(1) jump to the event's timestamp
+     }
+     // calltask.time <= tick: execute immediately at the current tick
+     // (guards against out-of-order events; avoids the infinite loop)
      call_list.pop();
      calltask.fun_ptr(calltask.fun_arg);
    }
  }

On the normal monotonic path this is exactly equivalent to the original; on the anomalous path it turns "spin forever" into "execute immediately". It also reduces the idle loop from O(total time) to O(number of events).

Change C (consistency): widen the event-registration interfaces to 64-bit

Sys.hh (lines 189–198) / Sys.cc (lines 1929–1974):

  void zero_latecy_register_event(
      Callable* callable, EventType event, CallData* callData,
-     int cycles);
+     Tick cycles);
  void register_event(
      Callable* callable, EventType event, CallData* callData,
-     int cycles);
+     Tick cycles);

No call sites need modification (existing arguments are int literals 1/10/30, int communication_delay, Tick weight_grad_update_time, Tick counter, etc.; int→Tick implicit conversion is lossless; for the double argument in LogGP.cc, positive-value behavior is unchanged and large values change from "sign-truncating" to "lossless").

Impact assessment: for all existing delays < 2^31 the three changes are numerically identical (no regression); for the generate_time/register_event shared with the ns3/phynet backends they are pure fixes (large delays are no longer corrupted).

---

5. Test Prerequisites

No extra data required — the in-repo material suffices:

1. Reproduction case (hangs before the fix / passes after):
   • Copy example/workload_analytical.txt (it is itself a tp=2 config) and change any layer's compute time to ≥ 2^31 (e.g. 3000000000);
   • Run SimAI_analytical: before the fix, expect one core at 100% forever; after the fix, completion within seconds with normal ./results/ output.
2. Regression case: run example/workload_analytical.txt unmodified and confirm identical runtimes and outputs (EndToEnd.csv etc.) before and after the fix.
3. User-scenario validation (optional): ask the user to provide their tp=2 256-GPU workload (or regenerate it with AICB using the same parameters) and confirm that entries ≥ 2^31 no longer hang after the fix. Also suggest the user run
   awk '{for(i=1;i<=NF;i++) if($i+0>=2147483648) print NR": "$i}' workload.txt
   on their original workload to quickly locate the offending entries as final confirmation of the root cause.
4. Build: ./scripts/build.sh -c analytical (same as the issue report); platform note: negative double→uint64 wraps to a huge value on x86-64 (hang) but saturates to 0 on ARM64 (timeline corruption) — both are manifestations of the defect.

---

6. Does the README Need Updating?

No. This is an internal defect fix; it changes no usage, CLI parameters, configuration formats, or output formats, so the README/docs need no update.

Optional: add an entry to CHANGELOG.md, e.g.:

• Fixed: SimAI_analytical hangs when a workload entry's compute time reaches 2^31 ns (~2.1 s). Root cause: 32-bit truncation in Sys::generate_time() wrapped the scheduled event timestamp; AnaSim::Run() then spins forever. Also made the analytical event loop jump to event timestamps in O(1) and guard against out-of-order FIFO events. (SimAI_analytical hangs / becomes extremely slow for tp=2 configs #300)

---

7. Suggested Issue Reply (English)

Thanks for the detailed report — all three suspicions point in the right direction, and #3 (generate_time(int cycles)) is the direct root cause:

Mechanism: in analytical mode, all delays on the event chain come from workload compute times (ns). Sys::try_register_event books the delay in full 64-bit Tick, but the downstream generate_time(int cycles) truncates it to 32 bits. When a compute-time entry is ≥ 2^31 (~2.1 s), the truncated negative value goes through a double→uint64 conversion (wrapping on x86-64) and becomes an event timestamp of ~1.8e19, which AnaSim::Run()'s tick++ can never reach — exactly matching your observation of one busy core, flat memory, and no completion after hours.

Why only tp=2: per-GPU compute time scales roughly as 1/tp. The same step is ~1.2e9 ns at tp=4 (safe) but doubles to ~2.4e9 ns at tp=2, just crossing the 2^31 threshold; all tp≥4 values stay below it. You can confirm with
awk '{for(i=1;i<=NF;i++) if($i+0>=2147483648) print NR": "$i}' <your-workload>
— the tp=2 file should contain out-of-range entries while the tp=4 file does not.

Fix (PR in preparation): (1) widen the int cycles parameter of generate_time/register_event/zero_latecy_register_event to Tick to remove the truncation; (2) make AnaSim::Run() jump directly to event timestamps (O(1)) and clamp out-of-order events with time < tick, which also resolves the per-tick busy-wait performance issue and the FIFO dead-loop hazard you pointed out.

Workaround before the fix: check your workload for compute-time entries ≥ 2147483648; if such a value comes from an AIOB-profiled large operator, temporarily split it (or wait for the PR to land).

---

8. Missing Information and Assumptions

#	Missing information	Assumption made	How to verify
1	The user's tp=2 workload file was not provided	Assume it contains compute-time entries ≥ 2^31 ns (per-GPU compute doubles at tp=2)	Scan with the awk command in Section 5.3; or ask the user for the file
2	Model scale / GPU type / sequence length unknown	Assume an AICB Megatron training workload (consistent with 776 ALLREDUCE ops and the dp/tp/pp structure)	Ask the user for the AICB generation command
3	SimCCL/aicb submodules could not be fetched (network restrictions); only partial SimCCL sources were checked via CDN	Assume the flow-model generation path is unrelated to this analysis (in ANALYTI mode Layer::issue_*_comm returns directly under #ifdef ANALYTI, producing no communication event stream — verified in local code)	Re-check once submodules are available
4	No stdout from the slow run (e.g. whether "counter > 0" was printed repeatedly)	Assume the normal Workload_Wait chain	Ask the user for run logs
5	No full local build/repro (submodules missing; sandbox cannot execute compiled binaries)	Root-cause conclusion based on code walkthrough + exact simulation of x86-64 conversion semantics (table in Section 3.1)	Reproduce on Linux per Section 5 cases

---

Analysis based on the local SimAI repository (including astra-sim-alibabacloud sources), SimCCL/aicb GitHub master sources, and simulated verification of x86-64 conversion semantics.

cn

SimAI Issue #300 分析与修复方案

Issue: SimAI_analytical hangs / becomes extremely slow for tp=2 configs
状态: open | 报告环境: Ubuntu 24.04.4, GCC 13.3.0, commit f5efb5a, ./scripts/build.sh -c analytical

---

1. 需求背景总结

用户使用 SimAI_analytical（analytical 后端）模拟 256 GPU Megatron 训练 workload 时发现：

配置	workload 条目数	ALLREDUCE 数	运行表现
dp=8, tp=2, pp=16	781	776	2–3 小时不结束
dp=8, tp=4, pp=8	1421	1416	约 1 秒完成
其他 tp≥4 配置	—	—	均几秒内完成

慢速运行期间：内存稳定在 ~5 MB，单核 CPU 持续 100% —— 典型的"死循环/天文数字迭代"特征，而非数据规模问题（tp=2 的 workload 反而更小）。

用户自行定位到 AnaSim::Run() 的事件循环并提出三个疑点：

1. tick 逐次 +1 追赶事件时间戳，时间戳大时迭代次数巨大；
2. call_list 是 FIFO 队列，一旦弹出 time < tick 的事件，tick++ 永远无法匹配 → 死循环；
3. Sys::generate_time() 接收 int cycles，而部分延迟来自 64 位值，疑似相关。

本分析确认：疑点 3 是触发挂死的直接根因，疑点 1/2 是放大器与结构性隐患。这不是预期行为，是 analytical 后端的两个真实缺陷。

---

2. 涉及文件 / 模块

文件	角色
astra-sim-alibabacloud/astra-sim/system/Sys.cc / Sys.hh	核心缺陷：generate_time(int) 32 位截断；register_event/zero_latecy_register_event 的 int cycles 参数
astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.cc / AnaSim.h	结构性缺陷：FIFO 队列 + 单调 tick 逐次递增的事件循环
astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalNetwork.cc	sim_schedule() → AnaSim::Schedule(delta.time_val, ...)，time_val 直接作为 delay
astra-sim-alibabacloud/astra-sim/workload/Workload.cc	try_register_event(this, Workload_Wait, NULL, counter)，counter（Tick）来自 workload 文件的 compute time
astra-sim-alibabacloud/astra-sim/workload/Layer.cc	register_event(..., weight_grad_update_time)（Tick → int 截断点之一）；ANALYTI 模式下 issue_*_comm 不产生通信事件流
astra-sim-alibabacloud/astra-sim/system/MemBus.cc / LogGP.cc	其他 register_event 调用点（ANALYTI 模式走固定小延迟；ns3/phynet 模式下 LogGP 的 G*(size-1) 同样可能超 2^31）

ANALYTICAL 模式事件链（#ifdef ANALYTI，./scripts/build.sh -c analytical）：
Workload::fire() → 每层每状态 counter = compute_time（来自 workload 文件，单位 ns，CLOCK_PERIOD=1）→ Sys::try_register_event(Tick&) → event_queue[tick+cycles] + generate_time(cycles) → AnalyticalNetWork::sim_schedule() → AnaSim::Schedule()（FIFO push）→ AnaSim::Run() 消费。
通信在 ANALYTI 模式下不产生事件流（Layer::issue_*_comm 直接 return，通信时间仅用于 report），因此事件链上的 delay 完全来自 workload 的 compute time 数值。

---

3. 根因分析

3.1 直接根因：generate_time(int cycles) 的 32 位截断

// Sys.hh:376
timespec_t generate_time(int cycles);

// Sys.cc:2142-2147
AstraSim::timespec_t Sys::generate_time(int cycles) {
  timespec_t tmp = NI->sim_get_time();
  double addition = cycles * ((double)CLOCK_PERIOD);   // CLOCK_PERIOD = 1
  tmp.time_val = addition;                              // double → uint64_t
  return tmp;
}

调用链存在两次有损转换：

1. Sys::try_register_event(Callable*, EventType, CallData*, Tick& cycles) 收到完整的 64 位 delay（如 workload 单条 compute time），event_queue[boostedTick() + cycles] 的 key 是正确的 64 位值；但随后调用 generate_time(cycles) 时 Tick(uint64) → int 隐式截断。
2. 截断产生的负数经 double → uint64_t 转换（超范围为 UB，x86-64 上按 cvttsd2si 回绕）得到 ~1.8e19 的天文数字。

在 x86-64/GCC 下实测语义（与用户环境一致）：

workload 中的 delay（ns）	(int) 截断后	AnaSim 事件时间	后果
7,956,367（正常层 compute）	正常	tick + 7,956,367	正常
1,000,000,000	正常	正常	正常
2,147,483,647（2^31−1）	正常	正常	正常（临界）
2,147,483,648（2^31）	−2,147,483,648	≈ 1.8446744e19	tick++ 需 ~1.8e19 次迭代 → 永远跑不完
2,500,000,000	−1,794,967,296	≈ 1.8446744e19	同上（挂死）
4,302,923,663（2^32+x）	+x（变小）	时间错乱	可能触发 FIFO 乱序死循环

AnaSim::Run() 中 while (true) { if (time != tick) tick++; else break; } 对 ~1.8e19 的事件时间需迭代约 1.8e19 次（单核 ~1e9 次/秒 → 需数百年），表现为：单核 100%、内存不变、永远不结束 —— 与用户观察完全吻合。

3.2 为什么只有 tp=2 触发

AICB/Megatron workload 中每条 compute time 是单卡计算时间，与 tp 大致成反比（张量并行切分）。同一模型下：

• tp=4 时单步 compute 约 1.2e9 ns（< 2^31，安全）；
• tp=2 时同一步 compute 翻倍到 ~2.4e9 ns（≥ 2^31，越界）。

即 tp=2 恰好把某条（或若干条）大 compute 条目（如 embedding/lm_head、长序列条目、或 AIOB profiling 出的大算子）推过 2^31 阈值，而 tp≥4 全部低于阈值。这解释了"workload 更小却更慢/挂死"的反直觉现象——决定性因素不是条目数，而是单条数值是否 ≥ 2^31。

仓库自带示例 example/workload_analytical.txt（首行 model_parallel_NPU_group: 2，恰为 tp=2 配置）已含多个 ≥ 2^31 的数值（第 3/4/7/8 行的 comm_size：2807758848、5615517696 等）。comm_size 本身不进事件链，但说明该量级数值在真实 workload 中常见；用户的 workload 中应存在 ≥ 2^31 的 compute time（见第 8 节"缺失信息与假设"）。

3.3 结构性缺陷（用户疑点 1/2 属实）

1. tick 逐次递增（性能）：AnaSim::Run() 用 tick++ 逐拍追赶事件时间，复杂度 O(模拟总时长)。即使无截断，总模拟时间 ~秒级（~1e9 ticks）时循环要空转 ~1e9 次——这是 tp=4 也要约 1 秒的原因。
2. FIFO + 单调 tick（死循环隐患）：call_list 按 push 顺序而非时间排序消费，tick 只增不减。一旦出现 time < tick 的队首事件（如 3.1 中 delay > 2^32 时截断变小、或多链路未来可能的乱序入队），内层 while(true) 立即死循环。当前 ANALYTI 串行链下乱序本身不易发生，但截断 bug 恰好能制造出这种局面。
3. Schedule 语义：time = tick + delay 本身正确；generate_time 命名误导（返回的是相对 delay 而非绝对时间），ns3 后端 Simulator::Schedule(NanoSeconds(t.schTime)) 同样把返回值当 delay 用，两后端语义一致——因此该截断 bug 同样影响 ns3/phynet 后端下的大延迟路径（如 LogGP 的 G*(size-1)、大消息传输时间）。

3.4 对用户三个疑点的逐条回应（建议回复要点）

• Q1: Has anyone seen this behavior before? Is this expected?
  不是预期行为，是缺陷。挂死由 Sys::generate_time(int) 的 32 位截断触发：当 workload 中任何一条 compute time ≥ 2^31 ns（约 2.1 秒）时，事件时间戳回绕成 ~1.8e19，AnaSim::Run() 的 tick 循环永远追不上。tp=2 使单卡 compute time 相比 tp=4 翻倍，恰好越过 2^31 阈值。
• Q2: tick 逐次递增导致大量迭代？
  正确，这是 O(模拟总时长) 的性能问题，建议改为直接跳跃（见方案 B）。它只解释"秒级"耗时，不解释"数小时不结束"——后者是截断导致的挂死。
• Q3: FIFO 弹出 time < tick 的事件导致 tick++ 无法匹配？
  分析完全正确，这是潜在死循环。当前 ANALYTI 事件链是串行的（单链单调），乱序不易自然发生；但截断 bug 会制造出回绕的时间戳（delay > 2^32 时变小），从而触发你描述的死循环。建议对 time < tick 做防御性 clamp。
• Q4: generate_time(int cycles) 与 64 位延迟相关？
  正是根因。register_event/try_register_event 的下游 generate_time(int) 对 Tick(uint64) 做隐式截断，任何 ≥ 2^31 的 delay 都会破坏时间线。修复：参数改为 Tick。

---

4. 最小修改方案

原则：最小 diff、保持现有代码风格、不引入依赖。三处修改相互独立，修改 A 为必选（修复挂死），B 为强烈建议（性能+防御），C 为顺带一致性。

修改 A（必选）：消除 32 位截断

astra-sim/system/Sys.hh（376 行）与 Sys.cc（2142 行）：

- timespec_t generate_time(int cycles);
+ timespec_t generate_time(Tick cycles);

- AstraSim::timespec_t Sys::generate_time(int cycles) {
+ AstraSim::timespec_t Sys::generate_time(Tick cycles) {
    timespec_t tmp = NI->sim_get_time();
-   double addition = cycles * ((double)CLOCK_PERIOD);
-   tmp.time_val = addition;
+   tmp.time_val = cycles * CLOCK_PERIOD;
    return tmp;
  }

顺带消除原实现"先取当前时间又被覆盖"的误导写法（行为等价：CLOCK_PERIOD=1 时 time_val = cycles；Tick 为 unsigned long long，乘法无截断）。

修改 B（强烈建议）：AnaSim 事件循环跳跃 + 乱序防御

astra-sim/network_frontend/analytical/AnaSim.cc（22–40 行）：

  void AnaSim::Run() {
    while (!call_list.empty())
    {
      CallTask calltask = call_list.front();
-     while (true) {
-       if (calltask.time != tick) {
-         tick++;
-       } else {
-         break;
-       }
-     }
+     if (calltask.time > tick) {
+       tick = calltask.time;   // O(1) 跳跃到事件时刻
+     }
+     // calltask.time <= tick：按当前时刻立即执行（防御乱序，避免死循环）
      call_list.pop();
      calltask.fun_ptr(calltask.fun_arg);
    }
  }

正常单调路径下与原实现完全等价；异常路径下从"死循环"变为"立即执行"。同时把 O(总时长) 的空转降为 O(事件数)。

修改 C（一致性）：事件注册接口参数对齐 64 位

Sys.hh（189–198 行）/ Sys.cc（1929–1974 行）：

  void zero_latecy_register_event(
      Callable* callable, EventType event, CallData* callData,
-     int cycles);
+     Tick cycles);
  void register_event(
      Callable* callable, EventType event, CallData* callData,
-     int cycles);
+     Tick cycles);

调用点无需改动（现有实参为 int 字面量 1/10/30、int communication_delay、Tick weight_grad_update_time、Tick counter 等，int→Tick 隐式转换无损；LogGP.cc 中 double 实参的转换行为对正值不变、对大值由"符号截断"变为"无损"）。

影响面评估：三处修改对现有 < 2^31 的延迟路径数值完全一致（回归无差异）；对 ns3/phynet 后端共享的 generate_time/register_event 同样是纯修复（大延迟不再被破坏）。

---

5. 测试前置条件

无需额外数据，仓库自带材料即可覆盖：

1. 复现用例（修复前挂死 / 修复后通过）：
   • 基于 example/workload_analytical.txt（恰为 tp=2 配置）复制一份，将任意一层的 compute time 改为 ≥ 2^31（如 3000000000）；
   • 运行 SimAI_analytical：修复前应观察到单核 100% 永不结束；修复后秒级完成且 ./results/ 正常产出。
2. 回归用例：原样运行 example/workload_analytical.txt，确认修复前后运行时间与输出（EndToEnd.csv 等）一致。
3. 用户场景验证（可选）：请用户提供其 tp=2 的 256 GPU workload 文件（或用 AICB 以相同参数重新生成），确认其中 ≥ 2^31 的条目在修复后不再挂死。建议同时请用户在原 workload 上执行：
   awk '{for(i=1;i<=NF;i++) if($i+0>=2147483648) print NR": "$i}' workload.txt
   快速定位越界条目，作为根因的最终佐证。
4. 构建方式：./scripts/build.sh -c analytical（与 issue 报告一致）；平台差异提示：负 double→uint64 在 x86-64 回绕成巨大值（挂死），在 ARM64 饱和为 0（时间错乱），均属缺陷表现。

---

6. README 是否需要修改及建议内容

不需要。本修复为内部缺陷修复，不改变使用方式、CLI 参数、配置文件格式或输出格式，README/文档无需更新。

可选项：在 CHANGELOG.md 追加一条，例如：

• Fixed: SimAI_analytical hangs when a workload entry's compute time reaches 2^31 ns (~2.1 s). Root cause: 32-bit truncation in Sys::generate_time() wrapped the scheduled event timestamp; AnaSim::Run() then spins forever. Also made the analytical event loop jump to event timestamps in O(1) and guard against out-of-order FIFO events. (SimAI_analytical hangs / becomes extremely slow for tp=2 configs #300)

---

7. 建议的 Issue 回复（中文要点）

感谢详细报告，三个疑点方向都对，其中第 3 点（generate_time(int cycles)）正是直接根因：

机制：analytical 模式下事件链上的 delay 全部来自 workload 的 compute time（ns）。Sys::try_register_event 虽以 64 位 Tick 记账，但下游 generate_time(int cycles) 将其截断为 32 位。当某条 compute time ≥ 2^31（约 2.1 秒）时，截断出的负数经 double→uint64 转换（x86-64 回绕）变成 ~1.8e19 的事件时间戳，AnaSim::Run() 的 tick++ 永远追不上——这正是"单核 100%、内存不变、数小时不结束"的原因。

为何只有 tp=2：单卡 compute time 与 tp 大致成反比。同一模型 tp=4 时单步约 1.2e9 ns（安全），tp=2 时翻倍至 ~2.4e9 ns，恰好越过 2^31 阈值；tp≥4 全部低于阈值。可用
awk '{for(i=1;i<=NF;i++) if($i+0>=2147483648) print NR": "$i}' <你的workload> 验证 tp=2 文件中存在越界条目（而 tp=4 文件没有）。

修复（PR 准备中）：① generate_time/register_event/zero_latecy_register_event 的 int cycles 参数改为 Tick，消除截断；② AnaSim::Run() 改为直接跳跃到事件时间戳（O(1)），并对 time < tick 的乱序事件做 clamp 防御，同时解决你指出的逐拍递增性能问题与 FIFO 死循环隐患。

修复前 workaround：请检查 workload 中 ≥ 2147483648 的 compute time 条目；如该数值来自 AIOB profiling 的单步大算子，可暂时将超过 2^31 的单条值拆分（或等 PR 合入）。

---

8. 缺失信息与假设

#	缺失信息	分析所做假设	验证方法
1	用户 tp=2 workload 文件未提供	假设其中存在 ≥ 2^31 ns 的 compute time 条目（tp=2 单卡计算翻倍所致）	用第 5.3 节 awk 命令扫描；或请用户提供文件
2	用户的模型规模/GPU 型号/序列长度未知	假设为 AICB Megatron 训练 workload（与 776 ALLREDUCE、dp/tp/pp 结构一致）	用户提供 AICB 生成命令
3	SimCCL/aicb 子模块未能拉取（网络受限），MockNcclGroup 仅通过 CDN 获取部分源码核对	假设 flow model 生成路径与 astra-sim 侧分析无关（ANALYTI 模式下 Layer::issue_*_comm 在 #ifdef ANALYTI 分支直接 return，不产生通信事件流，已由本地代码证实）	子模块可用后复核
4	用户未提供慢速运行时的 stdout（如是否大量打印 "counter > 0"）	假设走正常 Workload_Wait 链	请用户补充运行日志
5	本地未做完整构建复现（子模块缺失 + 沙箱限制无法执行编译产物）	根因结论基于代码走读 + x86-64 转换语义的精确模拟验证（3.1 节表格）	按第 5 节用例在 Linux 环境复现

---

分析基于本地 SimAI 仓库（含 astra-sim-alibabacloud 源码）、SimCCL/aicb 的 GitHub master 分支源码，以及 x86-64 转换语义模拟验证。

[tianhao909]

[Reply from Agent B] [For reference only]

---

English Version

1. Background Summary

SimAI_analytical does not finish for the 256-GPU dp=8, tp=2, pp=16 configuration while continuously consuming one CPU core. Configurations with tp>=4 complete within seconds.

This is not expected analytical-backend behavior. It is a correctness and performance defect in the event scheduler.

---

2. Affected Files and Modules

Primary locations:

• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.cc
  • AnaSim::Run()
  • AnaSim::Schedule()
• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.h
  • CallTask
  • FIFO call_list
• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalNetwork.cc
  • Passes timespec_t::time_val to AnaSim::Schedule()
• astra-sim-alibabacloud/astra-sim/system/Sys.hh
• astra-sim-alibabacloud/astra-sim/system/Sys.cc
  • register_event(..., int cycles)
  • generate_time(int cycles)
• Potential narrowing call sites:
  • astra-sim-alibabacloud/astra-sim/workload/Layer.cc
  • astra-sim-alibabacloud/astra-sim/system/LogGP.cc
  • astra-sim-alibabacloud/astra-sim/system/MemBus.cc

Roles of the referenced repositories:

• AICB generates Megatron and other workloads; it does not implement the analytical event loop.
• SimCCL provides collective communication functionality; it does not implement the AnaSim scheduler.
• ns-3-alibabacloud uses time-ordered schedulers and therefore does not have the same FIFO ordering defect.

---

3. Root Cause and Implementation Approach

3.1 Linear tick advancement

The current implementation advances simulated time with:

while (calltask.time != tick) {
    tick++;
}

Its runtime is proportional to the simulated time gap. A delay of hundreds of millions or billions of ticks can therefore consume substantial CPU time even when the loop eventually terminates.

A discrete-event simulator does not need to iterate through idle ticks. It should jump directly to the next event:

tick = calltask.time;

3.2 FIFO does not preserve chronological event order

call_list is a std::queue, so events are processed in insertion order instead of timestamp order.

For example:

1. Events for times 100 and 200 are queued;
2. The callback at time 100 schedules a new event for time 110;
3. The FIFO queue becomes 200, 110;
4. The event at 200 executes first and moves tick to 200;
5. The event at 110 is then popped;
6. Because tick only increases, it can no longer reach 110.

The loop will continue until uint64_t wraps around. This matches the reported symptoms: stable memory usage, one busy CPU core, and no progress for hours.

3.3 64-bit to 32-bit narrowing

Tick is a 64-bit type:

typedef unsigned long long Tick;

However, these interfaces use int:

void register_event(..., int cycles);
timespec_t generate_time(int cycles);

try_register_event() carries the delay as a 64-bit Tick, but passing it to generate_time() implicitly narrows it to a 32-bit signed integer.

For delays above INT_MAX:

• The value may be truncated;
• It may become negative;
• A later conversion to uint64_t may produce a very large timestamp;
• This can amplify both the linear-time loop and out-of-order scheduling defects.

A tp=2 configuration normally has larger per-rank communication payloads, making large delays more likely. However, the original workload sizes and calculated delay values are unavailable, so narrowing cannot yet be proven to be the only TP-specific trigger.

3.4 Conclusion

Both concerns raised in the issue are valid:

1. Incrementing tick one unit at a time makes large timestamps extremely slow;
2. A FIFO queue can produce an event whose timestamp is earlier than the current tick, causing a practically infinite loop.

Additionally, generate_time(int cycles) and register_event(..., int cycles) introduce a separate 64-bit-to-32-bit narrowing risk.

---

4. Minimum Viable Change

Only four files need to be modified.

4.1 AnaSim.h

Replace the FIFO std::queue with a timestamp-ordered container.

A suitable option is:

std::multimap<uint64_t, CallTask>

Reasons:

• It matches the ordered-map approach already used by Sys::event_queue;
• It automatically orders events by timestamp;
• It supports multiple events at the same timestamp;
• It provides deterministic handling of equal-time events.

A std::priority_queue with a custom comparator would also work, but stable ordering for equal timestamps would require additional handling.

4.2 AnaSim.cc

Update AnaSim::Run() to:

• Retrieve the earliest event;
• Remove the linear tick loop;
• Assign:

tick = calltask.time;

• Remove the event before invoking its callback, preserving the current ability for callbacks to schedule new events.

4.3 Sys.hh

Change the cycle arguments to Tick:

void register_event(..., Tick cycles);
timespec_t generate_time(Tick cycles);

4.4 Sys.cc

Update the corresponding definitions and remove the implicit:

Tick -> int -> Tick

conversion path.

4.5 Optional defensive validation

The analytical scheduling entry point may additionally reject:

• Negative delays;
• Non-finite time_val values;
• Time values that cannot be safely converted.

This would make future timing errors fail explicitly instead of causing a silent hang. It is useful but not required for the minimum fix.

4.6 Recommended scope

Do not:

• Add third-party dependencies;
• Refactor the ns-3 or physical backends;
• Change the workload format;
• Modify SimCCL or AICB;
• Perform a broad redesign of the time representation.

---

5. Test Prerequisites

5.1 Tests requiring no external data

A minimal scheduler regression test can:

1. Schedule events for times 100 and 200;
2. Schedule an event with delay 10 from the callback at time 100;
3. Verify that the execution order is:

100, 110, 200

Additional coverage should include:

• Out-of-order insertion;
• Multiple events with the same timestamp;
• Very large timestamps;
• A Tick delay greater than INT_MAX;
• Runtime that is independent of the simulated time gap;
• Events scheduled from inside callbacks.

The analytical module does not have an obvious standalone unit-test framework. To keep the change minimal, the existing build target and a small end-to-end workload can be used without adding a new test dependency.

5.2 Additional data needed for exact reproduction

Exact reproduction of dp=8, tp=2, pp=16 requires:

• The reporter's workload file;
• The complete command line;
• Model name and model parameters;
• Batch size;
• Sequence length;
• Bus-bandwidth parameters;
• Matching tp=4 and tp=8 workloads;
• Ideally, the maximum calculated delay value.

These details are not currently attached to the issue.

If the original workload is unavailable, it can be regenerated with AICB, provided that all original model and parallelism parameters are known.

5.3 Suggested validation flow

Build the analytical backend:

./scripts/build.sh -c analytical

Then run:

1. A smoke test using example/workload_analytical.txt;
2. The original tp=2 reproduction workload;
3. Matching tp=4 and tp=8 workloads;
4. A comparison of simulation output before and after the fix;
5. A runtime check confirming that tp=2 completes in bounded time;
6. A correctness check confirming that event ordering does not change the expected final statistics.

This pass was analysis-only. No code was changed, and no build or tests were run.

---

6. README Impact

No README update is required.

Reasons:

• This is an implementation defect, not a documented analytical-backend limitation;
• The fix does not change command-line arguments;
• The fix does not change the workload format;
• The fix does not change user configuration.

If the project wants to record the behavioral change, a changelog or release-note entry would be more appropriate:

Fixed an analytical-backend issue that could cause extremely slow execution or a hang with large delays and out-of-order event insertion.

===========

SimAI Issue #300 分析与修复建议

中文版

1. 需求背景总结

SimAI_analytical 在 256 GPU、dp=8, tp=2, pp=16 配置下长时间占用一个 CPU 核心，但无法完成模拟；而 tp>=4 的配置可以在数秒内结束。

该行为不是 analytical 后端的预期表现，而是事件调度器存在性能和正确性问题。

---

2. 涉及文件/模块

主要相关文件：

• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.cc
  • AnaSim::Run()
  • AnaSim::Schedule()
• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.h
  • CallTask
  • FIFO call_list
• astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalNetwork.cc
  • 将 timespec_t::time_val 传给 AnaSim::Schedule()
• astra-sim-alibabacloud/astra-sim/system/Sys.hh
• astra-sim-alibabacloud/astra-sim/system/Sys.cc
  • register_event(..., int cycles)
  • generate_time(int cycles)
• 可能发生类型窄化的调用位置：
  • astra-sim-alibabacloud/astra-sim/workload/Layer.cc
  • astra-sim-alibabacloud/astra-sim/system/LogGP.cc
  • astra-sim-alibabacloud/astra-sim/system/MemBus.cc

相关仓库的作用：

• AICB：负责生成 Megatron 等训练 workload，不负责 analytical 事件循环。
• SimCCL：提供集合通信算法相关能力，不实现 AnaSim 调度器。
• ns-3-alibabacloud：使用按事件时间排序的 scheduler，因此不会出现相同的 FIFO 乱序问题。

---

3. 根因分析或实现思路

3.1 tick 逐个递增导致性能问题

当前实现通过以下循环推进模拟时间：

while (calltask.time != tick) {
    tick++;
}

该操作的复杂度与事件时间差成正比。如果事件时间差达到数亿或数十亿 tick，即使没有进入死循环，也会消耗大量 CPU 时间。

离散事件模拟器不需要遍历中间的空闲 tick，而应直接将模拟时间跳转到下一个事件：

tick = calltask.time;

3.2 FIFO 队列不能保证事件按时间执行

当前 call_list 使用 std::queue，事件按照插入顺序执行，而不是按照时间戳执行。

可能出现以下情况：

1. 队列中已有时间为 100 和 200 的事件；
2. 执行时间 100 的回调时，新调度了时间 110 的事件；
3. FIFO 队列变成 200, 110；
4. 时间 200 的事件先执行，tick 变为 200；
5. 随后取出时间 110 的事件；
6. 因为 tick 只能增加，循环无法再到达 110。

此时程序会持续占用一个 CPU 核心，直到 uint64_t 回绕。该现象与 issue 中“内存稳定、一个 CPU 核心繁忙、数小时没有完成”完全一致。

3.3 64 位到 32 位的类型窄化

Tick 定义为 64 位：

typedef unsigned long long Tick;

但以下接口使用 int：

void register_event(..., int cycles);
timespec_t generate_time(int cycles);

try_register_event() 内部使用的是 64 位 Tick，但调用 generate_time() 时会隐式转换为 32 位 int。

如果延迟超过 INT_MAX：

• 延迟值可能被截断；
• 可能转换成负数；
• 负数经过 double 再转换为 uint64_t 时，可能产生极大的时间戳；
• 进一步放大逐 tick 循环和事件乱序问题。

tp=2 通常意味着更大的单 rank 通信量，因此更可能产生较大的延迟值。不过 issue 没有提供原始 workload 中的消息大小及实际计算出的 delay，所以目前不能确认类型溢出是 tp=2 特有问题的唯一触发条件。

3.4 结论

Issue 中指出的两个问题都成立：

1. 逐 tick 增加会导致大时间戳场景极慢；
2. FIFO 队列可能执行已经落后于当前 tick 的事件，从而进入无法结束的循环。

此外，generate_time(int cycles) 和 register_event(..., int cycles) 确实存在独立的 64 位到 32 位窄化风险。

---

4. 最小修改方案

建议只修改以下四个文件。

4.1 AnaSim.h

将 FIFO std::queue 替换为按事件时间排序的容器。

推荐使用：

std::multimap<uint64_t, CallTask>

理由：

• 与现有 Sys::event_queue 使用有序 map 的方式一致；
• 自动按照时间戳排序；
• 支持相同时间戳的多个事件；
• 相同时间戳事件可以保持确定的处理顺序。

也可以使用带比较器的 std::priority_queue，但需要额外处理相同时间戳事件的稳定顺序。

4.2 AnaSim.cc

调整 AnaSim::Run()：

• 每次取时间最早的事件；
• 删除逐 tick 增加的循环；
• 直接设置：

tick = calltask.time;

• 在执行回调前从容器中移除事件，保持回调可以继续添加新事件的现有行为。

4.3 Sys.hh

将以下接口的 cycles 类型改为 Tick：

void register_event(..., Tick cycles);
timespec_t generate_time(Tick cycles);

4.4 Sys.cc

同步修改函数定义，避免出现：

Tick -> int -> Tick

的往返类型转换。

4.5 可选防御措施

可以在 analytical 调度入口增加断言或检查，拒绝：

• 负延迟；
• 非有限的 time_val；
• 无法安全转换的时间值。

这有助于未来的类型错误快速失败，而不是静默进入长时间循环。不过该项不是解决 issue 的最小必要修改。

4.6 修改范围建议

不建议：

• 引入新的第三方依赖；
• 重构 ns-3 或 physical 后端；
• 修改 workload 格式；
• 修改 SimCCL 或 AICB；
• 对整个时间系统做大范围重构。

---

5. 测试前置条件

5.1 不依赖额外数据的测试

可以构造一个最小事件调度测试：

1. 调度时间为 100 和 200 的事件；
2. 时间 100 的回调中再调度延迟为 10 的事件；
3. 验证最终执行顺序是：

100, 110, 200

还应覆盖：

• 非时间顺序插入；
• 多个相同时间戳事件；
• 很大的事件时间戳；
• 大于 INT_MAX 的 Tick 延迟；
• 验证运行时间不再与事件时间差线性相关；
• 验证回调中新建事件仍能正确执行。

当前 analytical 模块没有明显的独立单元测试框架。为保持最小改动，可以使用现有构建目标配合小型端到端 workload 验证，不需要引入新的测试依赖。

5.2 完整复现依赖的额外数据

精确复现 issue 中的 dp=8, tp=2, pp=16 场景，还需要：

• issue 提交者使用的 workload 文件；
• 完整运行命令；
• 模型名称及模型参数；
• batch size；
• sequence length；
• bus bandwidth 参数；
• 对应的 tp=4 和 tp=8 对照 workload；
• 最好提供实际计算出的最大 delay 值。

这些信息目前没有附在 issue 中。

如果无法获得原始文件，可以使用 AICB 重新生成 workload，但必须使用相同的模型和并行参数。

5.3 建议验证流程

构建 analytical 后端：

./scripts/build.sh -c analytical

然后执行：

1. 仓库自带的 example/workload_analytical.txt 冒烟测试；
2. 原始 tp=2 复现 workload；
3. tp=4 和 tp=8 对照 workload；
4. 比较修复前后的输出结果；
5. 确认 tp=2 可以在合理时间内完成；
6. 确认模拟时间和最终统计结果没有因事件排序修改而出现异常。

本次按要求仅做分析，没有修改代码，也没有运行构建或测试。

---

6. README 是否需要修改及建议内容

不需要修改 README。

原因：

• 这是实现缺陷，不是 analytical 后端的预期限制；
• 修复不会改变命令行参数；
• 修复不会改变 workload 格式；
• 修复不会改变用户配置方式。

如果项目希望记录该行为变化，更适合在 changelog 或 release notes 中加入：

修复 analytical 后端在大延迟及事件乱序场景下可能极慢或无法完成的问题。