Event emission doesn't work with webhook provider

[tommasopozzetti]

What happened: Events are not getting emitted on external-dns when using a webhook provider with the following debug statement

{"level":"debug","msg":"skipping event for resources as the name is not generated yet"}

What you expected to happen: Events are correctly emitted for the supported sources

How to reproduce it (as minimally and precisely as possible):

• install external-dns v0.21.0 with flags --events-emit=RecordReady --events-emit=RecordDeleted --events-emit=RecordError and using a webhook provider
• observe that events are not emitted for supported sources like services even if they are deleted and recreated and the corresponding DNS entries correctly deleted and recreated

Anything else we need to know?:

I believe that the issue lies in the serialization and deserialization of the Endpoint object when using the webhook provider. The event emission logic uses refObject in the Endpoint to emit the event correctly, however that field is dropped during serialization to the webhook and back https://github.com/kubernetes-sigs/external-dns/blob/v0.21.0/endpoint/endpoint.go#L271, leaving that field to being null after deserializing the response from the webhook and calling the event emission logic.

Environment: Kubernetes 1.33

• External-DNS version (use external-dns --version): 0.21.0
• DNS provider: custom

[ivankatliarchuk]

You most likely missing information. Any chance to provide more information https://github.com/kubernetes-sigs/external-dns/blob/master/docs/contributing/bug-report.md

Not all sources do support event emit yet https://github.com/kubernetes-sigs/external-dns/blob/master/docs/sources/index.md#available-sources

Technically, should work for any provider, as long as provider supports plan https://github.com/kubernetes-sigs/external-dns/blob/master/controller/controller.go#L120-L128. So there is no serialisation or desirialization

[tommasopozzetti]

Thanks @ivankatliarchuk
We have a custom provider built in-house to communicate with a custom DNS implementation. However it does support the plan.

The resource I am testing on is a service which should support events

Here is the full list of arguments to external-dns

    - --log-level=debug
    - --log-format=json
    - --interval=2m
    - --events
    - --source=service
    - --source=ingress
    - --source=gateway-grpcroute
    - --source=gateway-httproute
    - --source=crd
    - --policy=sync
    - --registry=txt
    - --txt-owner-id=owner-1234
    - --txt-prefix=k8s-owned-
    - --domain-filter=.myzone
    - --provider=webhook
    - --events-emit=RecordReady
    - --events-emit=RecordDeleted
    - --events-emit=RecordError
    - --service-type-filter=LoadBalancer

Here is the service

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: myservice.mydomain.kube.myzone
  creationTimestamp: "2026-04-07T20:05:32Z"
  labels:
    app.kubernetes.io/name: myapp
  name: myapp-frontend
  namespace: default
  resourceVersion: "23868816"
  uid: cf19e2a2-11c5-49e0-9912-29a87b7ffcc2
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 172.31.148.184
  clusterIPs:
  - 172.31.148.184
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 30191
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: myapp
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.37.223.237
      ipMode: VIP

I do see the DNS entry correctly created and deleted in our DNS provider. The issue is just on the events in k8s.

Technically, should work for any provider, as long as provider supports plan https://github.com/kubernetes-sigs/external-dns/blob/master/controller/controller.go#L120-L128. So there is no serialisation or desirialization

This is true, emitChangeEvent operates purely on in-memory Go objects, however for webhook providers the serialization and deserialization happens earlier than that which wipes out the ref.

The bug is in provider/webhook/webhook.go in AdjustEndpoints.As far as I can tell, this is the exact sequence:

• Endpoint.refObject is tagged json:"-" (by design, at endpoint/endpoint.go)
• WebhookProvider.AdjustEndpoints performs json.Encode then json.Decode (at provider/webhook/webhook.go)
• The returned endpoints have refObject == nil
• These nil-refObject endpoints flow into plan.Desired, then into plan.Changes
• emitChangeEvent gets endpoints with nil refObject -> NewEventFromEndpoint returns empty Event{} -> event() logs "skipping event for resources as the name is not generated yet"

[ivankatliarchuk]

Yeah. I did not looked at webhook implementation. This may take a while to understand what the approach could be

[tommasopozzetti]

@ivankatliarchuk I think a simple, low impact change that brings the webhook to behave more similar to in-tree providers could be to make the following:

• In WebhookProvider.AdjustEndpoints, before serializing the endpoints to JSON, build a map from endpoint key (DNSName, RecordType, SetIdentifier) to refObject
• After deserializing the response from the webhook, look up each returned endpoint's key in that map and re-attach the corresponding refObject
• Endpoints that don't match (new ones added by the webhook) simply get no refObject, which is the correct behavior as they have no source object to emit events against. This keeps backward compatibility with any webhook that might add or remove endpoints in adjustendpoints

I'd be happy to put up a PR with an attempt to implement this if this plan seems reasonable

[ivankatliarchuk]

Make sense to open a PR, we gonna need some evidences of this thingy working. With map approach you may get hard time identifying deleted records. Same time create/update events better then zero events ;-)

Yeah, something basic, like

refObjects := make(map[endpoint.EndpointKey]*endpoint.ObjectRef, len(e))
	for _, ep := range e {
		if ref := ep.RefObject(); ref != nil {
			refObjects[endpointKey(ep)] = ref
		}
	}
	for _, ep := range endpoints {
		if ref, ok := refObjects[endpointKey(ep)]; ok {
			ep.WithRefObject(ref)
		}
	}

[tommasopozzetti]

@ivankatliarchuk do delete events work at all even with in-tree providers? Because it seems like for deleted records, they would come from calling the Records() function of the provider which will build them from the DNS provider directly and then during planning they would be compared with the list gathered from the cluster and the "extra ones" deleted. But the ones built from the DNS provider will never have a refObject so I would expect that those delete events really would never work even for non-webhook providers as there is no easy way to trace them back to the k8s resource, unless there is additional logic built to recover that info from the TXT record content

[tommasopozzetti]

For what concerns creation and update events, I linked a PR with the above logic and added a test for it. I built and ran in my environment and creation and update events are working as expected for the webhook provider now