Handle kubernetes watcher exception

[stv0g]

In GitLab by @iripiri on Mar 24, 2022, 17:35

seems to occur occasionally

Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/villas/controller/components/managers/kubernetes.py", line 93, in _run_event_watcher
    for e in w.stream(c.list_namespaced_event,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/watch/watch.py", line 182, in stream
    raise client.rest.ApiException(
kubernetes.client.exceptions.ApiException: (500)
Reason: InternalError: Internal error occurred: etcdserver: no leader

[stv0g]

Hi @iripiri,

this error occurs if the Kubernetes API server is unhealthy due to some issues with the etcdserver.
In most cases the Kubernetes API server should recover after some time automatically.

So our watch request should be-retried after some time automatically. Ideally with some exponential back-off.