Add sglang sglang request perf gate

[ghangz]

Summary

• Adds a focused sglang request perf gate improvement for MetaX-MACA/sglang.
• The change targets MetaX MACA development and validation workflows, with emphasis on earlier diagnostics, reproducible logs, or safer benchmark tooling.
• Existing default behavior is kept compatible; the new logic is scoped to explicit checks, helper tools, or validation metadata.

Validation

• Verified on Gitee.AI MetaX GPU resources: sglang_SGLang image batch, 12/12 PASS; PyTorch-MACA batch also covered SGLang tooling.
• Branch validation command: python tools/request_perf_gate.py --self-test x y
• Pull request text is intentionally ASCII-only to avoid encoding issues on web forms and API clients.

Review notes

• Source branch: ghangz:mengz/sglang-request-perf-gate
• Target branch: MetaX-MACA/sglang:main
• Maintainers can modify this branch if follow-up adjustments are needed.

[gemini-code-assist]

Code Review

This pull request introduces a new performance gating script, tools/request_perf_gate.py, which compares baseline and current performance metrics to detect regressions. The reviewer provided valuable feedback to enhance the script's robustness and flexibility, including implementing defensive parsing for JSON and JSON Lines (NDJSON) formats, parameterizing the compare function, expanding the self-test coverage to include regression and missing-case scenarios, and adding command-line arguments for custom metrics and tolerance thresholds.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The current load function is fragile and will crash with KeyError, TypeError, or ValueError if the JSON structure is slightly different than expected (e.g., missing keys, non-numeric values, or unexpected types). Additionally, since benchmark tools in SGLang (like bench_offline_throughput.py) append results to a file, the output file can be in JSON Lines (NDJSON) format, which json.loads() on the entire file content will fail to parse.

To make the performance gate robust and prevent CI/CD pipeline failures, we should implement defensive parsing that handles both standard JSON and JSON Lines, safely retrieves keys with fallbacks, and ignores malformed entries gracefully.

def load(path: Path, metric: str = METRIC) -> dict[str, float]:
    try:
        content = path.read_text(encoding="utf-8").strip()
    except Exception as e:
        raise RuntimeError(f"Failed to read file {path}: {e}")

    if not content:
        return {}

    try:
        data = json.loads(content)
    except json.JSONDecodeError:
        try:
            data = [json.loads(line) for line in content.splitlines() if line.strip()]
        except json.JSONDecodeError as e:
            raise ValueError(f"Failed to parse JSON from {path}: {e}")

    result = {}
    if isinstance(data, list):
        for idx, item in enumerate(data):
            if not isinstance(item, dict):
                continue
            name = item.get("name") or item.get("model") or f"case_{idx}"
            val = item.get(metric)
            if val is not None:
                try:
                    result[str(name)] = float(val)
                except (ValueError, TypeError):
                    pass
    elif isinstance(data, dict):
        for k, v in data.items():
            if isinstance(v, dict):
                val = v.get(metric)
            else:
                val = v
            if val is not None:
                try:
                    result[str(k)] = float(val)
                except (ValueError, TypeError):
                    pass
    return result

[gemini-code-assist]

To support configurable metrics and tolerances via the CLI, update the compare function to accept metric and tolerance as parameters instead of relying on hardcoded global constants.

Suggested change

	def compare(baseline: dict[str, float], current: dict[str, float]) -> dict[str, object]:
	rows: list[dict[str, object]] = []
	failed = False
	for name, old in sorted(baseline.items()):
	if name not in current:
	rows.append({"name": name, "status": "missing-current"})
	failed = True
	continue
	new = current[name]
	ratio = (new - old) / old if old else 0.0
	status = "regression" if ratio < -TOLERANCE else "ok"
	failed = failed or status != "ok"
	rows.append({"name": name, "baseline": old, "current": new, "delta_ratio": ratio, "status": status})
	return {"ok": not failed, "metric": METRIC, "rows": rows}
	def compare(
	baseline: dict[str, float],
	current: dict[str, float],
	metric: str = METRIC,
	tolerance: float = TOLERANCE,
	) -> dict[str, object]:
	rows: list[dict[str, object]] = []
	failed = False
	for name, old in sorted(baseline.items()):
	if name not in current:
	rows.append({"name": name, "status": "missing-current"})
	failed = True
	continue
	new = current[name]
	ratio = (new - old) / old if old else 0.0
	status = "regression" if ratio < -tolerance else "ok"
	failed = failed or status != "ok"
	rows.append({"name": name, "baseline": old, "current": new, "delta_ratio": ratio, "status": status})
	return {"ok": not failed, "metric": metric, "rows": rows}

[gemini-code-assist]

Expand the self_test function to cover regression and missing-case scenarios, ensuring that the regression detection logic is thoroughly validated.

Suggested change

	def self_test() -> None:
	data = compare({"case": 100.0}, {"case": 99.0})
	assert data["ok"]
	print(json.dumps({"ok": True, "rows": len(data["rows"])}, ensure_ascii=False))
	def self_test() -> None:
	# Test OK case
	data_ok = compare({"case": 100.0}, {"case": 99.0})
	assert data_ok["ok"]
	# Test regression case
	data_reg = compare({"case": 100.0}, {"case": 90.0})
	assert not data_reg["ok"]
	assert data_reg["rows"][0]["status"] == "regression"
	# Test missing case
	data_missing = compare({"case": 100.0}, {})
	assert not data_missing["ok"]
	assert data_missing["rows"][0]["status"] == "missing-current"
	print(json.dumps({"ok": True, "rows": len(data_ok["rows"])}, ensure_ascii=False))

[gemini-code-assist]

Add --metric and --tolerance command-line arguments to allow users to customize the target metric (e.g., request_throughput, output_throughput) and the regression threshold (e.g., 0.05 for 5%) without modifying the script.

def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("baseline")
    parser.add_argument("current")
    parser.add_argument("--metric", default=METRIC, help="Metric to compare (default: %(default)s)")
    parser.add_argument("--tolerance", type=float, default=TOLERANCE, help="Regression tolerance ratio (default: %(default)s)")
    parser.add_argument("--self-test", action="store_true")
    args = parser.parse_args()
    if args.self_test:
        self_test()
        return 0
    metric = args.metric
    tolerance = args.tolerance
    result = compare(
        load(Path(args.baseline), metric=metric),
        load(Path(args.current), metric=metric),
        metric=metric,
        tolerance=tolerance,
    )
    print(json.dumps(result, ensure_ascii=False, indent=2))
    return 0 if result["ok"] else 1