Cloud env configs are not visible across processes and trajectory rows fall back to job_id="gateway"

[MillionMillionLi]

Summary

In cloud storage mode, trajectory payloads are persisted successfully, but the Gateway can fail to resolve the environment metadata created by the launcher. When this happens, landing rows are written with the hard-coded fallback job_id="gateway" instead of the rollout batch ID.

The observed failure has two contributing causes:

1. Cloud environment synchronization globally replaces the shared evaluation_env_config table by calling clean_all_configs() before every launcher run.
2. A long-lived Gateway may not see environment configs committed by a launcher process started later. wt_sdk.EnvConfigManager.get_env_configs() uses session.filter() without a checkout_latest option.

The fallback then masks the lookup failure and produces valid trajectory content with incorrect ownership and partition metadata.

Identifier contract

The intended SAfactory identifiers are:

• job_id: one rollout batch and the landing HASH partition key
• session_id = env_id: one episode
• step_id: one model inference within the episode

job_id and session_id are not expected to be equal. The bug is that the landing job_id becomes gateway rather than the actual rollout batch ID.

Environment

• SAfactory base: v2 at dca7c465614af38f408db01130ff49bdcaf8f345
• Gateway trajectory correlation changes from open PR [WIP]Feat: normalize anthropic messages #49 were integrated locally for the observed run
• wt-data-platform-sdk / wt_sdk: 0.4.0
• Separate long-lived Gateway and launcher processes
• Cloud landing table and cloud environment-config table

Reproduction

1. Start a cloud-backed Gateway.
2. Start a cloud-backed launcher later with a new rollout job_id and multiple environment rows.
3. Observe the launcher report successful environment synchronization.
4. Send episode requests through the Gateway.
5. Query the landing table by the real rollout job_id and then by session_id.
6. Start another cloud launcher and inspect the shared environment-config table again.

Observed evidence

In one live rollout:

• The launcher reported 20/30 environment configs committed, followed by the remaining 10/10 completed.
• The first Gateway environment lookup missed about three minutes later, before any subsequent launcher had replaced the table.
• Across 22 local episode sessions, Gateway logs contained 3,252 environment lookup misses and zero successful resolutions.
• A read-only SDK checkout found 1,609 landing inference rows across 20 sessions, all with job_id="gateway".
• For retained local request logs, 920/920 overlapping requests and 927/927 terminal responses matched the S3 rows after applying the same redaction rule. This indicates that the trajectory payload itself was preserved; the association metadata was wrong.
• A later read of evaluation_env_config returned 1,400 rows, all belonging to a different, later job. None of the original 30 rows remained.

The first lookup miss occurring before a later table replacement suggests a stale-view problem in the long-lived Gateway. Later launcher runs then make the problem permanent by deleting all rows in the shared table.

Relevant code paths

Global full-table replacement

core/data_manager/yaml_aggregator.py::_sync_cloud() currently does:

env_manager.clean_all_configs()
data_manager.strategy._env_configs.clear()

This makes concurrent and historical jobs mutually destructive even though each environment row has a job_id.

Environment lookup visibility

wt_sdk.EnvConfigManager.get_env_configs() calls session.filter() without checkout_latest. A Gateway initialized before the launcher write can continue missing newly committed environment rows.

Fallback trajectory ownership

In the Gateway storage path observed with PR #49 integrated, the storage namespace is initialized as gateway. If environment lookup fails, session creation continues with that namespace and cloud landing records persist session.job_id unchanged.

Expected behavior

• Environment rows for multiple rollout jobs coexist in the shared table.
• Deletes and refreshes are scoped by job_id.
• A long-lived Gateway can read configs committed by another process after Gateway startup.
• Every landing row is partitioned by the real rollout batch job_id.
• Missing environment metadata does not silently create apparently valid rows under job_id="gateway".

Suggested changes

1. Replace clean_all_configs() with job-scoped upsert and cleanup semantics.
2. Add latest-version refresh support to EnvConfigManager reads, such as checkout_latest=True, or explicitly reopen/refresh the table before cross-process lookup.
3. Propagate job_id, env_name, and group_id directly when registering or creating a Gateway session, rather than relying exclusively on a secondary environment-table lookup.
4. In cloud mode, fail closed, queue for later reconciliation, or explicitly mark rows unresolved when environment metadata is unavailable. Avoid silently using gateway as the landing partition key.
5. Add an integration test with a long-lived Gateway plus two concurrent/sequential launcher jobs. Verify that both jobs remain queryable and that every trajectory row has the correct batch job_id.

Impact

• Trajectories cannot be queried through their real rollout partition.
• Evaluators and training consumers may see missing or incomplete jobs even though inference payloads exist in S3.
• Concurrent launcher jobs overwrite each other in the environment-config table.
• Storage success logs can give a false impression that a rollout is a correctly associated data asset.