Replace fluentd with fluent-bit in the operator

[hjiawei]

Description

New feature (Calico Enterprise): the LogCollector controller now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, completing the log-collector migration on the operator side.

• Resource identity migration: namespace tigera-fluentd → calico-system; DaemonSet/ServiceAccount fluentd-node → calico-fluent-bit; TLS secret → calico-fluent-bit-tls; metrics port 9081 → 2020 (fluent-bit's built-in HTTP server).
• Configuration is rendered in fluent-bit's YAML schema into per-OS ConfigMaps (calico-fluent-bit-conf/-windows), subPath-mounted on Linux and directory-mounted on Windows, started with -c. The render loads the Go plugins via plugins_file, defines parsers inline, applies the record-transform Lua filter, and inlines user-provided fluent-bit YAML filter lists. A rendered-config hash annotation rolls the pods on config-only changes.
• Tail inputs use the producing components' real log paths, with SQLite offset DBs and filesystem buffering under /var/log/calico/calico-fluent-bit; a pos-migrator init container (Linux and Windows) seeds offsets from the legacy fluentd .pos files and pre-creates the tailed directories. Windows tails the same log types the fluentd Windows variant shipped.
• The linseed output matches only Linseed-bound tags, authenticates with mTLS + the pod's ServiceAccount token, and retries without limit against bounded filesystem storage. S3/Splunk/Syslog outputs mirror fluentd's per-type fan-out (standard AWS credential env vars, endpoint scheme honored, syslog ships the whole record as JSON via a per-output Lua processor with TLS properly enabled).
• NonClusterHost renders the :9880 HTTP input with client-certificate verification; the input Service is cleaned up when the resource is removed.
• eks-log-forwarder runs the fluent-bit image with a rendered in_eks → linseed pipeline and health probes (no startup init container; the input plugin resolves its resume point from Linseed).
• Probes hit :2020/api/v1/health; the ServiceMonitor scrapes plain HTTP (fluent-bit's monitoring server has no TLS) with access restricted by the component NetworkPolicy, and legacy fluentd monitors are deleted.
• The LogCollector controller no longer creates/owns the calico-system namespace (deleting the LogCollector must not garbage-collect it); the deprecated fluentdDaemonSet override is honored as an alias of the new calicoFluentBitDaemonSet field (with container-name translation); legacy tigera-fluentd resources — the namespace last — are cleaned up idempotently.

Testing: render, controller, and monitor unit suites updated/extended (ConfigMap-content assertions replacing the env-var assertions); the rendered configuration was validated against the real fluent-bit binary; the full migration was validated end-to-end on a test cluster — all log types flowing to Linseed/Elasticsearch, fluentd resources fully removed, tail-offset handover without re-shipping, NonClusterHost ingestion with client-certificate enforcement, and EKS/Windows render shapes verified.

Release Note

The LogCollector now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, with operator-rendered configuration and automatic migration of fluentd tail positions.

Breaking changes and behavior changes in this migration:
- The tigera-fluentd namespace is removed; all log-collector resources are renamed and move to calico-system (DaemonSet/container calico-fluent-bit, TLS secret calico-fluent-bit-tls). Tooling referencing the old names (e.g. kubectl logs -c fluentd) must be updated. The LogCollector fluentdDaemonSet override field is deprecated in favor of calicoFluentBitDaemonSet; existing overrides (including legacy container names) keep applying for one release.
- User flow/DNS log filters: the fluentd-filters ConfigMap is no longer read. Filters must be recreated under the new fluent-bit-filters name as fluent-bit YAML filter lists; unparseable content raises a TigeraStatus warning while log shipping continues.
- Log-collector metrics are now fluent-bit's native Prometheus metrics (fluentbit_* metric names, plain HTTP on port 2020 at /api/v2/metrics/prometheus, guarded by NetworkPolicy). Dashboards and alerts keyed on fluentd_* metric names or the 9081 mTLS endpoint must be updated.
- S3 archive object keys change from fluentd's flat layout (<bucketPath>/flows20260101_<n>.gz) to directory-style keys (<bucketPath>/flows/20260101_<uuid>.gz), with non-cluster-host flows archived under their own non_cluster_flows/ directory. Downstream tooling anchored to the old flat patterns needs a one-time update.
- Log buffering moves from fluentd's in-memory buffers to fluent-bit filesystem storage under /var/log/calico/calico-fluent-bit/ on each node: buffered-but-unsent chunks now survive pod restarts, and host disk usage grows accordingly (capped per output by storage.total_limit_size).
- The direct Elasticsearch log output is removed (Linseed is the sole structured-log sink), and the Splunk output no longer has an L7 log type (deployed fluentd never enabled it).
- Non-cluster hosts can now forward DNS logs in addition to flow and policy-activity logs.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files
• If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug if this is a bugfix.
  • kind/enhancement if this is a a new feature.
  • enterprise if this PR applies to Calico Enterprise only.

[Copilot]

Pull request overview

This PR completes the operator-side migration of the LogCollector from fluentd to fluent-bit (Calico Enterprise), including updated resource identities (namespace/workload names), rendered configuration shape, and updated policy/monitoring expectations across the rendering and controllers.

Changes:

• Replace fluentd-specific rendering/controller wiring with fluent-bit equivalents (names, ports, RBAC, ServiceMonitor behavior).
• Update expected Calico policies/selectors to reflect fluent-bit running in calico-system and using port 2020.
• Extend the LogCollector CRD/API to add calicoFluentBitDaemonSet and treat the deprecated fluentdDaemonSet as an alias during migration.

Reviewed changes

Copilot reviewed 47 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/render/tiers/tiers_test.go	Updates tier rendering tests to stop expecting a separate LogCollector namespace.
pkg/render/testutils/expected_policies/node_local_dns_ipv6.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/node_local_dns_ipv4.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/node_local_dns_dual.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/linseed.json	Updates Linseed ingress policy sources from fluentd labels/namespace to fluent-bit in calico-system.
pkg/render/testutils/expected_policies/linseed_ocp.json	Same as above for OpenShift policy expectations.
pkg/render/testutils/expected_policies/linseed_ocp_dpi_enabled.json	Same as above for OpenShift + DPI-enabled scenario.
pkg/render/testutils/expected_policies/linseed_dpi_enabled.json	Same as above for DPI-enabled scenario.
pkg/render/testutils/expected_policies/guardian.json	Updates Guardian policy expectations to allow fluent-bit instead of fluentd.
pkg/render/testutils/expected_policies/guardian_ocp.json	Same as above for OpenShift policy expectations.
pkg/render/testutils/expected_policies/fluentbit_unmanaged.json	Updates fluent-bit “allow metrics” policy expectations (name/namespace/selector/port).
pkg/render/testutils/expected_policies/fluentbit_unmanaged_ocp.json	Same as above for OpenShift.
pkg/render/testutils/expected_policies/fluentbit_managed.json	Updates managed-cluster policy expectations for fluent-bit metrics access.
pkg/render/testutils/expected_policies/es-gateway.json	Updates ES gateway ingress policy sources from fluentd to fluent-bit.
pkg/render/testutils/expected_policies/es-gateway_ocp.json	Same as above for OpenShift.
pkg/render/testutils/expected_policies/dns.json	Removes legacy tigera-fluentd namespace from DNS policy selector expectations.
pkg/render/testutils/expected_policies/dns_ocp.json	Same as above for OpenShift DNS policy expectations.
pkg/render/nonclusterhost/nonclusterhost.go	Adds Linseed RBAC permissions for DNS logs in non-cluster-host flow.
pkg/render/nonclusterhost/nonclusterhost_test.go	Updates RBAC test expectations to include dnslogs.
pkg/render/monitor/monitor.go	Renames fluentd monitoring constants to fluent-bit and switches ServiceMonitor to plain HTTP fluent-bit metrics endpoint.
pkg/render/monitor/monitor_test.go	Updates monitor rendering tests for renamed ServiceMonitor and the removal of TLS/scheme relabeling.
pkg/render/manager.go	Updates manager-rendered NetworkPolicy destination service name and cluster-wide namespace list (includes a problematic entry noted in comments).
pkg/render/manager_test.go	Updates manager tests to expect fluent-bit service names.
pkg/render/logstorage/linseed/linseed.go	Updates Linseed policy source entity rule reference to fluent-bit.
pkg/render/logstorage/esgateway/esgateway.go	Updates ES gateway policy source entity rule reference to fluent-bit.
pkg/render/intrusion_detection.go	Updates comments to reflect fluent-bit consuming IDS logs.
pkg/render/guardian.go	Updates Guardian policy source entity rule reference to fluent-bit.
pkg/render/fluentd.go	Removes the fluentd renderer implementation.
pkg/render/fluentd_test.go	Removes fluentd rendering unit tests.
pkg/render/common/elasticsearch/service.go	Renames isFluentd flag to isFluentBit and updates related commentary for managed-cluster Linseed routing.
pkg/render/applicationlayer/applicationlayer.go	Updates comment about WAF logs being consumed by fluent-bit.
pkg/imports/crds/operator/operator.tigera.io_logcollectors.yaml	Adds calicoFluentBitDaemonSet and updates fluentdDaemonSet docs/enums for the migration (some descriptions still say “Fluentd”, noted in comments).
pkg/controller/tiers/tiers_controller.go	Stops treating LogCollector as its own namespace when computing tier namespace lists.
pkg/controller/monitor/prometheus.go	Watches fluent-bit ServiceMonitor name instead of fluentd.
pkg/controller/monitor/monitor_controller.go	Switches monitored TLS secret from fluentd to fluent-bit.
pkg/controller/monitor/monitor_controller_test.go	Updates controller tests to reference fluent-bit ServiceMonitor and secrets.
pkg/controller/logstorage/secrets/secret_controller.go	Updates upstream cert collection references from fluentd TLS secret to fluent-bit TLS secret.
pkg/controller/logstorage/secrets/secret_controller_test.go	Updates tests to create/check fluent-bit TLS secret scenarios.
pkg/controller/logcollector/logcollector_controller.go	Rewires controller from fluentd to fluent-bit rendering (secrets, filters, services, namespace ownership behavior).
pkg/controller/logcollector/logcollector_controller_test.go	Updates controller tests for fluent-bit DaemonSet naming, config-driven outputs, and non-creation of calico-system.
pkg/components/enterprise.go	Renames enterprise component definitions from fluentd to fluent-bit (linux/windows).
hack/gen-versions/enterprise.go.tpl	Updates version generation template keys/structs for fluent-bit components.
config/enterprise_versions.yml	Replaces fluentd component entries with fluent-bit equivalents.
api/v1/zz_generated.deepcopy.go	Adds deepcopy support for CalicoFluentBitDaemonSet on LogCollectorSpec.
api/v1/logcollector_types.go	Adds calicoFluentBitDaemonSet field and documents fluentdDaemonSet as deprecated alias.
api/v1/fluentd_daemonset_types.go	Updates override container/initContainer enums to fluent-bit naming (backward-compat concerns noted in comments).

Files not reviewed (1)

• api/v1/zz_generated.deepcopy.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• api/v1/zz_generated.deepcopy.go: Generated file

Comments suppressed due to low confidence (2)

pkg/controller/monitor/monitor_controller.go:156

• Monitor no longer scrapes fluent-bit over mTLS (serviceMonitorFluentBit is plain HTTP), so the controller shouldn't watch or require the fluent-bit TLS secret. Keeping it here creates an unnecessary dependency on the LogCollector controller and can degrade Monitor reconciliation when LogCollector isn't installed (secret not found).

	for _, secret := range []string{
		certificatemanagement.CASecretName,
		esmetrics.ElasticsearchMetricsServerTLSSecret,
		monitor.PrometheusServerTLSSecretName,
		render.FluentBitTLSSecretName,
		render.NodePrometheusTLSServerSecret,
		kubecontrollers.KubeControllerPrometheusTLSSecret,
		render.EKSLogForwarderTLSSecretName,
	} {

pkg/controller/monitor/monitor_controller.go:345

• Since fluent-bit metrics scraping is now plain HTTP, Monitor's trusted bundle no longer needs the fluent-bit certificate. Including it makes reconciliation fail with NotFound if LogCollector hasn't created the secret yet (or isn't installed).

	for _, certificateName := range []string{
		esmetrics.ElasticsearchMetricsServerTLSSecret,
		render.FluentBitTLSSecretName,
		render.NodePrometheusTLSServerSecret,
		render.CalicoAPIServerTLSSecretName,
		kubecontrollers.KubeControllerPrometheusTLSSecret,
	} {

[pasanw]

Consider watching the configmap we render for fluent-bit if we want to prevent users from tampering with it

[hjiawei]

Done — the controller now watches the rendered calico-fluent-bit-conf, calico-fluent-bit-conf-windows and eks-log-forwarder-conf ConfigMaps in calico-system, so tampering (or deletion) triggers reconciliation and the operator restores the rendered content.

[pasanw]

I think we should extract out a component that renders out the resources that are shared between the linux and windows instances of this component. As I understand it, some resources are unique to the specific instance via OS naming convention.

However, some resources are not unique. This means that each instance of the component needs to compute the same definition of such resources, otherwise we have contention like you described. This pattern is confusing and seems like it could be the cause bugs in the future like the one you described

[hjiawei]

Done — added logcollector.FluentBitShared, rendered exactly once per reconcile, owning everything that is not OS-specific: the allow-calico-fluent-bit NetworkPolicy, the GKE ResourceQuota, the S3/Splunk credential copies, the managed-cluster tigera-linseed Service/RoleBinding, the PacketCapture Role/Binding, and the legacy-fluentd cleanup. The per-OS components now render only their own metrics Service, conf ConfigMap, RBAC/SA, DaemonSet (+ EKS forwarder and the NCH input service on Linux). That removes the render-the-same-object-twice contention class outright — the Windows cfg no longer needs NonClusterHost at all, and there's a test asserting the Linux and Windows renders produce disjoint object sets.

[pasanw]

Does fluent-bit require tcp:// instead of http:// or https://? If so should we enforce this on the user input?

[hjiawei]

Yes — fluent-bit's syslog mode only accepts tcp/udp (TLS comes from the encryption field, which selects mode tls on top of tcp framing), and an http:// endpoint would have rendered a broken output. The controller now rejects non-tcp/udp schemes with a ResourceValidationError degraded status, with a test. (Same behavior fluentd had implicitly via SYSLOG_PROTOCOL.)

[pasanw]

nit: remove this comment, it's rationale that only benefits the author of this PR and its reviewers

[hjiawei]

Removed.

[pasanw]

nit: would prefer if the renderFluentBitConf was the first function definition after this one.

[hjiawei]

All taken: renderFluentBitConf now leads the file and reads as service+parsers → addInputs → addFilters → addOutputs, and the Linseed output helpers moved into outputs.go with the store outputs.

[pasanw]

Do we have E2Es for S3, Splunk and Syslog that can serve as regression testing?

[hjiawei]

Not operator E2Es. The calico-private side has the container fv (fluent-bit/fv/container/, design verification step 3) which drops fixture log lines per type and asserts delivery against local S3/Splunk/Syslog mocks using the real fluent-bit binary and the same config shapes the operator renders. On the operator side the golden-config suite added in this PR pins the exact rendered YAML per scenario. There is no e2e against real S3/Splunk/Syslog endpoints today (that was true of fluentd too).

[pasanw]

I think we need a test suite that has the exact level of coverage provided by test.sh for the fluentd equivalent of this code, especially given that we are performing a migration here. My ideal would be to see that we have each case from test.sh represented as a test in this repo, with the expectation as a raw YAML file that has a comment showing the fluentd equivalent.

[hjiawei]

Added golden_test.go + testdata/rendered-configs/: 17 goldens of the exact rendered config, byte-compared (UPDATE_RENDERED_CONFIGS=1 regenerates), each table entry naming its test.sh ancestor, reusing test.sh's endpoint/bucket/log-type values so the goldens read side-by-side with the fluentd .cfg files. The README maps all 20 test.sh cases: the six ES-era cases are n/a (ES output retired with fluentd per EV-6164), -nch cases map to hostScope NonClusterOnly, and the honest divergences are documented — the operator's Audit syslog type always enables both audit kinds (fluentd's env could do kube-only), the "all log types" case is capped at the CRD's legal enum (Audit;DNS;Flows;IDSEvents — the L7/runtime/WAF syslog envs were never operator-exposed), and the dual-scheme splunk case is split into splunk-http/splunk-https since the CRD takes one endpoint.

[pasanw]

Might be worth touching base with @caseydavenport on the steps taken in this PR to remove direct ES connections - just to make sure you are aligned with what he had in mind

[hjiawei]

Will do — the ES retirement steps here follow the EV-6164 design (ES output dropped with fluentd, Linseed as sole structured-log sink). Casey, the specific surface in this PR: fluentd's ES env plumbing and the ElasticsearchEksLogForwarderUserSecret/cluster-config watches are removed from the LogCollector controller; nothing else touches ES.

[pasanw]

Claude says:

Bug: fluent-bit's S3 object keys don't match fluentd's, despite a code comment in outputs.go claiming the archive layout is "preserved."

• fluentd (out-s3-flows.conf): used fluent-plugin-s3's default s3_object_key_format = %{path}%{time_slice}_%{index}… with path "flows" — path and date concatenated, no separator → key flows20260101_0.gz.
• fluent-bit (outputs.go): s3_key_format = "/flows/%Y%m%d_$INDEX.gz" — a / inserted between flows and the date → key flows/20260101_0.gz.

Impact: the extra / turns flowsYYYYMMDD… into flows/YYYYMMDD…. Any downstream tooling globbing or parsing the old flat prefix silently reads nothing after upgrade — directly contradicting the comment that says the segments preserve the layout so "existing downstream consumers keep reading the same prefixes."

[hjiawei]

Good catch that the layout differed while the comment claimed parity. After weighing exact-parity against fluent-bit's directory-style keys, we're keeping the directory layout as a deliberate, release-noted change rather than inheriting fluentd's flat concatenation: keys are now <bucketPath>/<type>/%Y%m%d_<uuid>.gz (one directory per log type, non-cluster flows in their own non_cluster_flows/ directory instead of being mixed into flows/). Rationale: full parity was impossible anyway — the _<index> suffix had to become $UUID because out_s3 tracks its index in a store_dir that doesn't survive pod restarts, so a restarted pod would silently overwrite same-day objects (fluent-plugin-s3 avoided this by checking object existence, which out_s3 doesn't) — and given a documented break either way, the directory layout is materially better for the S3 console, lifecycle tooling and Athena/Glue partitioning. Prefix filters (<bucketPath>/flows) keep matching under both layouts; only patterns anchored to the flat form (flows2026*) need a one-time update, which the release note calls out. The code comment now states the break explicitly, and the goldens show the exact keys per scenario.

[pasanw]

Claude says:

Bug: the fluent-bit :9880 HTTP input — which receives non-cluster-host logs relayed by voltron — omits the body-size limit fluentd's HTTP source had, so large POSTs are silently rejected/truncated.

• fluentd (fluent_sources.conf, port 9880): body_size_limit 100m.
• fluent-bit (config.go, [INPUT] Name http port 9880): no size set → inherits in_http's much smaller default buffer_max_size (~4M).

Impact: a voltron-relayed non-cluster-host batch larger than ~4M is rejected → those flow/DNS/policy-activity records silently never reach Linseed. Nothing in the CRD or operator surfaces an error.

Fix: set buffer_max_size 100M (and a matching buffer_chunk_size) on the input.

[hjiawei]

Confirmed and fixed — fluentd's :9880 source had body_size_limit 100m; the fluent-bit input now sets buffer_max_size 100M (plus buffer_chunk_size 5M so a full-size relayed batch doesn't grow 512K at a time), with a render test asserting both.

[Brian-McM]

I know this wasn't there before, but you might want to add a daemonset watch to trigger re reconciliation if the daemonset was removed. This is missing in a lot of places currently.

[Brian-McM]

It seems like, and I might be wrong, that this is all in service of just the config and every function that's called in this chain has nothing to do with rendering (which the component represents).

If this is true, consider attaching all the methods to the FluentBitConfiguration, or reversing the relationship so it's all attached to the fluendBitConfig struct and you pass the specific config options from the FluentBitConfiguration that pertain just to fluentd configuration.

This would be in further service of separating fluentd specific configuration (which is complex) from the component rendering.

[Brian-McM]

I'm imagining (please correct me if I'm wrong) that this just needs to be done once. If that's true, are we able to detect if we've already done the migration and not attach the init container?

I'm thinking that if we can skip this init container it's one less thing in the way of getting fluentd starting up. Might not be a huge deal in prod, but cutting out time to setup in the CI would be helpful.