[Design] ate.actor.incarnation

[krisztianfekete]

Living design doc for easier collaboration, please use this for comments: https://docs.google.com/document/d/1tpUZNqH-xaSXzMVJf95CpWtKbIBD-mH93nqlDcEBOe8/edit?tab=t.0

Problem

An actor occupies many workers throughout its life. It runs on worker A for an hour, suspends, resumes on worker B three days later, crashes, comes back. Each time it occupies a worker we emit spans and logs, but we have nothing to refer to these intervals/periods, or "residencies" by. So for example "show me everything this actor did on Tuesday afternoon" is not a query we can run today.

Proposal: create an id per "residency", and set it on the signals where high cardinality is allowed, so spans and logs.

High level mechanism

ateapi creates a ULID/UUID (having it sortable by creation timestamp is nice) when it commits an actor into RUNNING, stores it on the Actor record, and it propagates trough the restore path (ateapi -> atelet -> ateom), for which we already have all the plumbing in place.
Where it goes

• spans: lifecycle operations plus the router resume span
• logs: through ateattr, next to the actor identity keys we set in (chore): consolidate logging to ateattr #1035
• usage events later (Feature Request: Resource-usage telemetry at actor / ActorTemplate granularity #550 / Actor resource telemetry Phase 1: atelet reader and OTel emission #896): a residency is the natural scope for "this cost X".
• NOT on metrics
• entities later, if we want to adopt them. its own type, incarnation_of the actor, runs_on the pod, see https://opentelemetry.io/docs/specs/otel/entities/entity-events/

We should consider storing the previous residency's suspend span context on the Actor record, and link each resume span back to it.

Benefits

• the actor timeline becomes one query instead of lots of joins across pod names and time windows
• usage and cost can be split into residency beyond actor and request
• we can make crash loops become visible as incarnation rate is increasing
• for [Design] Actor Otel Telemetry Relay #853, it seems to be the proper writer identity for the relayed metric stream. One incarnation is exactly one writer by design, which is a cleaner than a worker pod

Open questions

1. What ends an incarnation apart from SUSPEND? What about PAUSE, where the actor may come back to the same worker? Is an incarnation one running interval or one placement? This will affect the design.
2. Proto field or telemetry-internal? On the proto it's more useful and clients can correlate against it, but it's an API change, and we have multiple changes in-flgiht for the API already

cc. Da Huang (@git286), Julian Gutierrez Oschmann (@juli4n), Max Smythe (@maxsmythe)