Conversation
What: Add a helper API to compute quantized matmul work_area requirements. Why: Avoid per-invocation allocator overhead by enabling deterministic caller preallocation in hot loops. Expected impact: Lower tail latency and fewer malloc/free calls; additive opt-in API. Tests: add sizing helper coverage (tests/testDriver_quantized_matmul_work_area_size_apis.c). Signed-off-by: Kaitlyn Davis <k8ika0s@gmail.com> Signed-off-by: Kaitlyn Davis <kaitlyn.davis@ibm.com>
|
H @k8ika0s, thanks for putting this together! I'm a recent member to the zDNN group, I support the goal, however there is just a few thoughts and concerns I have re: the abstraction and need of this. As I understand it, this API would expose zDNN's internal scratch-memory requirement so consumers can manage the allocation themselves. My concern is whether we want to make that implementation detail part of the public API contract. Do we have any evidence that the internal allocation is a significant bottleneck for quantized matmul? If there are benchmarks or workload data showing that quantized matmul is commonly on a hot path and that the associated allocation overhead has a measurable performance impact, I'd be interested in seeing that. I think having that data would help us determine whether the direction of this! |
| return 0; | ||
| } | ||
|
|
||
| if (!input_c || !input_c->transformed_desc) { |
There was a problem hiding this comment.
I think an error/status return would be more appropriate here to avoid a silent fail. Since 0 is also a legitimate result when no scratch memory is required (e.g. pre_computed).
|
|
||
| // The internal temporary tensor (qc_tilde) reuses input_c's transformed | ||
| // dimensions/layout/format but promotes the element type to DLFLOAT16. | ||
| zdnn_tensor_desc qc_tilde_desc; |
There was a problem hiding this comment.
Could we avoid duplicating this sizing logic here?
The actual qc_tilde allocation path in aiu_quantized_matmul.c (L 2343) already constructs the qc_tilde_desc and derives qc_tilde.buffer_size using zdnn_getsize_ztensor().
qc_tilde.buffer_size = zdnn_getsize_ztensor(&qc_tilde_desc);This new API appears to reproduce that logic separately.
Could we reuse/expose this existing sizing logic for maintenance purposes
There was a problem hiding this comment.
I no longer have any access to validate or test any of this. It was submitted months ago.
2 participants
Fixes #49
Summary
Why
Changes
qe/issue-10-quantized-matmul-work-area-size.Validation
artifacts/final-validation-20260212T173158Z.Notes