How to query a backend's native/recommended groupsize?

[jonas-schulze]

My use cases evolve around sparse linear algebra. As always, there are big differences in performance between CPU and GPU (-like). As a naive solution, I used isgpu(backend) to choose a groupsize. However, isgpu is no longer around, and I have (once) observed the PoCL backend on the CPU to be fastest for a groupsize of 1, whereas the CPU performed better for ~4 and CUDABackend much better for 32 (or 64). I've used smth like this, which I specialized using package extensions:

clamp_groupsize(groupsize::Int, backend)::Int

Technically, one can probably only recommend prod(@groupsize) but not @groupsize directly. It would be nice if KA would provide an abstraction that, in the long term, could make use of CUDA's occupancy API. Therefore, I would recommend smth like

recommended_groupsize(backend, kernel)::Int

where the user then has to recover the actual groupsize for N > 1 dimensional kernels.

• If CUDA, use the occupancy API
• If old CPU backend, use number of cores
• If new PoCL CPU backend, use one (more research needed!)

I'm sometimes confused on whether I should focus on groupsize or the hardware's vector width. Please let me know what you think. 🙂

[jonas-schulze]

I might still have a misconception about what to recommend for the CPU. For my use case, the typical "large enough to keep a core busy" did not work well. Using smth closer to the native vector width worked better, but that depends on the floating point type involved (but putting that in the API here is not ideal). And for PoCL I was surprised that goupsize = 1 worked best. Unfortunately, I can't share the code, yet.