Default operator metrics secure by default

[squakez]

Requirement

Right now we use:

Metrics:                       metricsserver.Options{BindAddress: ":" + strconv.Itoa(int(monitoringPort))},

we should default instead to some stronger configuration.

[dshane1903]

Hi, I'd like to work on this. Is the issue still available? Before I start, could you confirm which metricsserver.Options you want enabled for the secure-by-default configuration? Once confirmed, I can add the configuration and corresponding tests

[squakez]

Thanks for taking care of this. The goal of the issue is to analyze and propose a "secure by default" behavior. No prescription about the options to use. It should be simple and documented. If it breaks previous behavior we may introduce some environment variable to the operator to let the user change it.

[dshane1903]

I took a look at the current metrics setup. Enabling SecureServing: true seems like the smallest change. But the existing PodMonitor assumes HTTP, and controller runtime will use a self signed cert unless certs are explicitly configured. I also checked filters.WithAuthenticationAndAuthorization, but that would make this a bigger change since it needs additional TokenReview/SubjectAccessReview permissions. Also the /metrics reader role based access control, bearer-token handling, and the PodMonitor updates. I would prefer to keep this issue scoped to HTTPS by default, update the monitoring config, and then add an environment variable to go back into HTTP if we need to preserver the old behavior. If you want the full authorization and authentication setup included here instead, I can do that too. Does the TLS only scope sound right?

[squakez]

If you mean the PodMonitor available in base/config/prometheus, that one is an example. We can change such an example according to the new secure setting if it is needed. Ideally we should be able to configure http/https as you're suggesting via env var as well as doing full auth, also, controlled via env var. Before doing the final implementation it would be nice to have a list those new variables that would enable the feature to validate and understand the feasibility.