User Issue: Running on large number of targets is slow - need option to override default QPS and Burst

[grzesuav]

Describe the bug
Tried to run dirsuption on 6k pods for 10 minutes. Chaos controller during those 10 minutes were able to schedule 1.5k pods and then cleanup up. It was too slow. Want to adjust QPS and Burst options but not possible, old controller runtime v0.17.2 is used, which have defaults to 20/30 and there is no option in chaos controller to override this values.

To Reproduce
Run i.e. latency disruption targeting 6k pods.

Expected behavior
I can tune up QPS and Burst to allow rapid pod creation.

Screenshots

Environment:

• Kubernetes version: 1.31.x
• Controller version: 12.4.4
• Cloud provider (or local): N/A
• Base OS for Kubernetes: N/A

Additional context

[grzesuav]

Actually setting higher QPS and burst do not help, it wasn't a bottleneck here

[aymericDD]

Hi @grzesuav
Any idea where the problem is coming from?

[grzesuav]

@aymericDD yes,basically there is. one synchronous loop which is creating chaos pods one by one with waiting for confirmation, which takes few seconds. On 6k pods it was very slow. I did a hack to ran this part in goroutines which significantly speed up the process.

One unresolved case for me was updating the CR. status in this approach, so I ignored that until. reconcile is finished - it was ok. in my case as I needed those test to run

[aymericDD]

I will be happy to see the changes you made to improve the performance. Do you have a commit where I can take a look at the changes you made?

What is exactly the issue you encounter regarding the CR? I will be happy to take a deeper look.

[aymericDD]

Hi @grzesuav We did some great performance improvement. Could you retry with the latest version of chaos-controller?