Jobs: document that GPU performance counters are unavailable

[pjmalandrino]

Adds a note under Hardware flavor in jobs-configuration.md: NVIDIA GPU performance counters are not accessible from inside a Job.

The failure mode is confusing. Nsight Compute installs and even attaches to the target process, then fails:

==PROF== Connected to process 9012 (/usr/local/bin/my-benchmark)
==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0.
==PROF== Disconnected from process 9012

This looks like a problem with the image, so you keep trying variants (different CUDA base, different ncu version, --target-processes all, running as root). None of that can work: counter access is gated by the host's NVIDIA kernel module (NVreg_RestrictProfilingToAdminUsers), and nothing inside the job can change it. A note in the docs would have saved me that time.

For the record: observed on l40sx1, driver 580.159.03, nsight-compute-2024.1.1 installed from the CUDA apt repo. On the same job, nvidia-smi queries and CUDA event timing work fine.

I only hit the counter restriction, so that is all the note claims. Happy to reword this or move it if another spot fits better.

---

Note

Low Risk
Documentation-only change with no runtime, security, or API impact.

Overview
Adds a [!NOTE] under Hardware flavor in jobs-configuration.md (right after the hf jobs hardware table) explaining that NVIDIA GPU performance counters cannot be used inside Jobs.

The note calls out the ERR_NVGPUCTRPERM failure mode for profilers like Nsight Compute, states the limit is host-enforced (not fixable via image, command, or env), links NVIDIA’s docs, and clarifies that nvidia-smi and CUDA event timing still work.

^{Reviewed by Cursor Bugbot for commit 3bc07a0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[julien-c]

Maybe a bit too much of a edge case to be mentioned in the main docs?

[pjmalandrino]

Yeah, it's rare for sure.

The annoying part is that when you hit it, the error points to a driver setting you can't touch on Jobs, so you end up debugging your own image on paid GPU time for nothing.
That's all I wanted to save people from.

Happy to cut it to one sentence or move it somewhere less visible.
Also fine to just close it, the PR showing up in search already helps :)