Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract by pavlokhrebto · Pull Request #7580 · DataDog/system-tests

pavlokhrebto · 2026-08-25T13:07:11Z

Motivation

FFL-2695's Go agentless configuration-source implementation has now landed in dd-trace-go (DataDog/dd-trace-go#5223, #5226, #5346, #5363), so this PR brings the Go parametric app up to the /ffe/start and /ffe/evaluate contract already implemented for Python (#7411) and declares the coverage as supported.

Changes

main.go — lazy activation: skip eager OpenFeature provider init when any DD_FEATURE_FLAGS_* configuration variable is set, leaving setup to /ffe/start, since the contract forbids configuration delivery before the provider is accessed. Tests predating that contract keep the eager path, which stays asynchronous on purpose: it runs for every parametric test that sets no Feature Flagging variable, and blocking there would add the 10s DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS to each container start.

ffe.go

/ffe/start now waits for Init before answering. Plain SetProvider returns first, so the endpoint used to answer 200 with no configuration and the next evaluation got the default.
The wait carries our own 15s deadline via SetProviderWithContextAndWait rather than SetProviderAndWait's background context, so tracers that only bound Init cannot hang the suite — this is what keeps the prod job green against released tracers.
PROVIDER_NOT_READY is treated as a successful start: the provider is registered and evaluations return defaults until configuration arrives.
Start state is guarded by a mutex plus a success flag instead of sync.Once, so a failed attempt is not cached as success.
/ffe/evaluate uses the *ValueDetails APIs per variationType and returns {value, reason, errorCode}. Details are preserved on the error path too — the SDK returns them alongside the error, and _ffe_evaluate_with_rc_retry keys on errorCode == PROVIDER_NOT_READY to decide whether to retry.

tests/parametric/test_ffe/test_configuration_sources.py — _remote_config_products no longer filters on post_only. dd-trace-go polls /v0.7/config with GET where every other library uses POST, so all of its RC requests were discarded and FFE_FLAGS never appeared in the products set. The verb says nothing about which products the body advertises, and the capability assertion beside it was already unfiltered.

manifests/golang.yml — declare v2.11.0-dev for tests/parametric/test_ffe/test_configuration_sources.py (replacing the file-level block and ten class/test entries) and for tests/ffe/test_agentless_configuration.py. All 22 configuration-source tests pass on Linux; the end-to-end test was verified enforced and green.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

…entless configuration contract

github-actions · 2026-08-25T13:07:38Z

CODEOWNERS have been resolved as:

manifests/golang.yml                                                    @DataDog/system-tests-reviewers
tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk
utils/build/docker/golang/parametric/ffe.go                             @DataDog/system-tests-reviewers
utils/build/docker/golang/parametric/main.go                            @DataDog/system-tests-reviewers

…e and endpoint (#5218) ### What does this PR do? Lays the groundwork for Agentless Feature Flags delivery (polling configuration directly from Datadog over HTTPS instead of via the Agent's Remote Config): five new `DD_FEATURE_FLAGS_*` environment variables, `internal/config` fields/accessors for them, the source-precedence resolution logic (`internal/openfeature.ResolveSettings`/`RemoteConfigSourceSelected`), and Agentless endpoint URL resolution (`openfeature.buildAgentlessEndpoint`, managed vs. custom). No behavior change yet — nothing is wired up to the provider or issues any network requests. This is Phase 1 of a multi-PR rollout; the poll loop, JSON:API envelope parsing, and provider wiring/activation follow in subsequent PRs. ### Stack 1. #5218 — settings, source precedence, endpoint (this PR) 2. #5222 — JSON:API parser, poll loop 3. #5223 — provider wiring, activation, lifecycle, RC gating 4. #5226 — init outcomes, provider events ### Motivation dd-trace-go is one of the last server SDKs to support Agentless Feature Flags delivery (Python, Node, and Java have shipped it; .NET is in review; Ruby is also pending). Agentless removes the Agent as a dependency and becomes the default delivery mechanism. FFL-2695. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: DataDog/system-tests#7580 - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: pavlo.khrebto <pavlo.khrebto@datadoghq.com>

### What does this PR do? Adds the Agentless configuration delivery mechanism itself, stacked on #5218 (source resolution/endpoint): a JSON:API envelope parser for the Universal Flags Configuration payload (`openfeature/jsonapi.go`), and the poll loop that fetches it (`openfeature/agentless_source.go`) — per-poll retries with backoff, ETag/304 handling with last-known-good semantics, gzip decoding, and per-category log-once for persistent failures. Includes an `httptest.Server`-based fake backend mirroring the system-tests mock (`utils/mocked_backend/ffe.py`) covering the full behavior matrix: ETag not advanced on parse failure, retry-within-poll, non-retryable failures, no-overlap between polls, and no log line ever containing the endpoint URL. Still not wired up to the provider — nothing issues a request until Phase 3 lands. ### Stack 1. #5218 — settings, source precedence, endpoint 2. #5222 — JSON:API parser, poll loop (this PR) 3. #5223 — provider wiring, activation, lifecycle, RC gating 4. #5226 — init outcomes, provider events ### Motivation Continuation of Agentless Feature Flags delivery (FFL-2695); see #5218 for the overall context. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: see DataDog/system-tests#7580 (referenced from #5218). - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: pavlo.khrebto <pavlo.khrebto@datadoghq.com>

…ss delivery (#5223) ### What does this PR do? Wires up the provider itself, stacked on #5222 (poll loop/parser): `NewDatadogProvider` now resolves the delivery source and branches to Remote Config, Agentless, or a `NoopProvider`, replacing the old `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED`-gated check. Adds the Agentless activation critical section (`startWithAgentless`) with a `shutdownCalled` re-check so a poller can never be registered after `Shutdown` — that would otherwise leak a billable poller for the process lifetime. Reorders `ShutdownWithContext` so teardown runs without holding the provider's lock (needed because `agentless.Stop` joins the poll goroutine, which itself calls `updateConfiguration` and takes the same lock). Gates the tracer's eager Remote Config subscribe on the resolved source (`ddtrace/tracer/remote_config.go`). Also fixes a real pre-existing bug found by a new concurrency test: concurrent `Init` calls could double-start a writer and panic; `Init` is now idempotent. Updates `openfeature/doc.go` with the new environment variables and the delivery-source model. ### Stack 1. #5218 — settings, source precedence, endpoint 2. #5222 — JSON:API parser, poll loop 3. #5223 — provider wiring, activation, lifecycle, RC gating (this PR) 4. #5226 — init outcomes, provider events ### Motivation Continuation of Agentless Feature Flags delivery (FFL-2695); see #5218 for the overall context. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: see DataDog/system-tests#7580 (referenced from #5218). - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: kakkoyun <kakkoyun@users.noreply.github.com> Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>

…5226) ### What does this PR do? Final PR in the Agentless Feature Flags delivery series (stacked on #5223). Adds `DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS` (default 10s) replacing the hardcoded 30s init timeout, and implements the three `InitWithContext` outcomes: a permanent delivery failure returns a `*openfeature.ProviderInitError` immediately, configuration already present returns `nil`, and a timeout with delivery still running returns `nil` + a warning instead of the raw `ctx.Err()` (a deliberate change from the old Remote-Config-only behavior — see below). Adds `openfeature/events.go`: `EventChannel()` returns the same stored channel on every call (required by the SDK's listener), emitting `ProviderReady` once on the first configuration, `ProviderConfigChange` on later ones, and `ProviderStale` when configuration goes nil (e.g. an empty Remote Config set) so evaluation status reflects reality. `ProviderReady` is guaranteed to survive a full event buffer via a non-blocking drain-and-retry. ### Stack 1. #5218 — settings, source precedence, endpoint 2. #5222 — JSON:API parser, poll loop 3. #5223 — provider wiring, activation, lifecycle, RC gating 4. #5226 — init outcomes, provider events (this PR) ### Motivation Completes Agentless Feature Flags delivery (FFL-2695); see #5218 for the overall context. **Behavior change to flag for cross-language sign-off:** on an `Init` timeout with delivery still running, the provider now returns `nil` instead of `ctx.Err()`. Go's `ErrorState` doesn't block flag evaluation, so the only customer-visible difference is the reported status/event — but this could affect a health check that inspected the old error return. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: see DataDog/system-tests#7580 (referenced from #5218). - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: leoromanovsky <leo.romanovsky@datadoghq.com>

…ation

…ebto/FFL-2695/go-agentless-configuration

…polls by verb

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

datadog-official · 2026-09-11T21:05:23Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a21571b | Docs | View more details | Give us feedback!}

…ebto/FFL-2695/go-agentless-configuration

…/start

…ation

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b27857e0b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…fe/start to retry

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

… v2.11.0-dev

Guard eager OpenFeature init and add detailed /ffe/evaluate for Go ag…

0c4b674

…entless configuration contract

pavlokhrebto changed the title ~~Guard eager OpenFeature init and add detailed /ffe/evaluate for Go ag…~~ Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract Aug 25, 2026

pavlokhrebto added 4 commits September 11, 2026 08:37

Merge branch 'main' into pavlo.khrebto/FFL-2695/go-agentless-configur…

ddc8e97

…ation

Merge branch 'main' of github.com:DataDog/system-tests into pavlo.khr…

7bfaebc

…ebto/FFL-2695/go-agentless-configuration

fix(golang): block /ffe/start on provider init and stop filtering RC …

3d5729f

…polls by verb

Merge branch 'pavlo.khrebto/FFL-2695/go-agentless-configuration' of g…

f258148

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

chore: re-run CI against the tracer fix branch

136dd5e

pavlokhrebto added 3 commits September 14, 2026 09:27

Merge branch 'main' of github.com:DataDog/system-tests into pavlo.khr…

f5cc36b

…ebto/FFL-2695/go-agentless-configuration

fix(golang/parametric): treat PROVIDER_NOT_READY as a successful /ffe…

b9bbe44

…/start

fix(golang/parametric): bound the /ffe/start wait with our own deadline

b56ca81

Merge branch 'main' into pavlo.khrebto/FFL-2695/go-agentless-configur…

6b27857

…ation

pavlokhrebto marked this pull request as ready for review September 15, 2026 06:38

pavlokhrebto requested review from a team as code owners September 15, 2026 06:38

pavlokhrebto requested review from danyal002 and sameerank and removed request for a team September 15, 2026 06:38

pavlokhrebto requested review from Anilm3 and leoromanovsky and removed request for danyal002 September 15, 2026 06:38

chatgpt-codex-connector Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread utils/build/docker/golang/parametric/ffe.go Outdated

Comment thread utils/build/docker/golang/parametric/ffe.go Outdated

pavlokhrebto added 3 commits September 15, 2026 10:21

fix(golang/parametric): keep evaluation details on error and allow /f…

d0f849d

…fe/start to retry

Merge branch 'pavlo.khrebto/FFL-2695/go-agentless-configuration' of g…

30eaae6

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

feat(golang): declare FFE agentless configuration-source support from…

a21571b

… v2.11.0-dev

Navigation Menu

Uh oh!

Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract - #7580

Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract#7580
pavlokhrebto wants to merge 13 commits into
mainfrom
pavlo.khrebto/FFL-2695/go-agentless-configuration

pavlokhrebto commented Aug 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

pavlokhrebto commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pavlokhrebto commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading

datadog-official Bot commented Sep 11, 2026 •

edited

Loading