Alignment Observability Framework ERATO Emprical Reasoning and Alignment Trajectory Observatory

A reproducible research platform for observing, measuring, and explaining how post-training interventions change language-model behaviour across checkpoints.

Project status: The research engine, plugin system, longitudinal audit harness, public package contracts, synthetic fixtures, and CPU-side validation are implemented. Policy Transition Analysis (PTA) and Realised Preference Margin Shift (RPMS) are proposed diagnostics under active empirical validation. Full multi-adapter model-native analysis remains GPU-dependent.

One-Command Demo

For a public review from a clone or extracted longitudinal_template package:

uv run --locked erato-demo

The command generates public-safe synthetic longitudinal inputs, runs CPU-side adapter inventory and registry construction through the normal research engine, validates the engine manifest and required artifacts, and writes a Markdown and JSON summary under outputs/demo/<run-id>/. It performs no model inference, does not probe or use a GPU, and requires no model weights, external adapters, private datasets, or inference service.

The default uv environment contains only the standard-library project. On a first run, uv may still need network access to obtain Python or the setuptools build requirement if they are not already installed or cached; the repository does not claim that initial environment creation is network-free.

List the installed research tasks:

uv run erato task-list

Run the existing longitudinal inventory profile against inputs in the normal repository locations:

uv run erato task-run \
  --task longitudinal_alignment \
  --profile inventory \
  --repo-root .

Public reviewer

Install uv, clone the repository, and run the one-command demo above. Each run uses a unique directory and an isolated public-code workspace so ignored local or historical assets cannot become demo inputs.

Developer

uv sync --locked --group dev
uv run --locked --group dev pytest -q
uv lock --check
uv build

The dev group contains test tooling plus pandas and Matplotlib, which are imported by the existing preference-movement tests and shared utilities. It is intentionally excluded from the default reviewer environment.

Researcher with external assets

Supply adapters, curricula, audit data, and model paths outside the public packages, then select the appropriate task profile. The CPU inventory profile does not load adapter tensors:

uv run erato task-run \
  --task longitudinal_alignment \
  --profile inventory \
  --repo-root . \
  --data-dir /path/to/external/inventory-inputs

Install the numerical and plotting stack when running analysis phases that use it:

uv sync --locked --extra analysis

GPU researcher

Model-native audit and PTA/RPMS phases require user-supplied assets and an appropriate CUDA runtime. Install those opt-in dependencies explicitly:

uv sync --locked --extra analysis --extra gpu

The GPU extra includes PyTorch, Transformers, PEFT, Accelerate, and related model tooling; it is never installed by the public demo or ordinary CPU CI.

---

Why This Project Exists

Modern language models are rarely trained once and evaluated once. They move through a sequence of interventions:

• supervised fine-tuning;
• preference optimisation;
• reinforcement learning;
• curriculum refinement;
• rejection sampling;
• safety tuning;
• model repair;
• targeted data augmentation.

Each intervention creates a new checkpoint or adapter. Standard evaluation usually compares only the final score:

checkpoint A → benchmark score
checkpoint B → benchmark score

That tells us whether performance changed, but not:

• which behaviours changed;
• which errors were repaired;
• which capabilities regressed;
• whether fixes transferred to unseen prompts;
• whether improvements were durable;
• whether the model learned a broad preference or fitted exact examples;
• whether an apparent gain came from chosen-response reinforcement, rejected-response suppression, formatting regularisation, or evaluator-specific behaviour;
• whether related reasoning trajectories changed together.

This project treats post-training as a longitudinal alignment trajectory rather than a set of isolated checkpoints.

The central research question is:

How can we observe, quantify, and explain alignment-relevant behavioural and policy changes across successive post-training interventions?

The objective is not to claim direct access to hidden motives or human-like reasoning. The objective is to build converging evidence about how observable behaviour, model-native preference margins, and eventually internal representations change over time.

---

Research Vision

The long-term goal is an Alignment Observability Framework that can reconstruct and analyse the trajectory induced by iterative post-training.

Instead of asking only:

Did this adapter improve accuracy?

the framework asks:

• What changed across the population of prompts?
• Which changes were local and which generalised?
• Did related problem families move together?
• Did the model become more stable or more volatile?
• Were gains caused by stronger preferred behaviour or weaker rejected behaviour?
• Did a proxy objective emerge?
• Did behaviour become evaluator-conditioned?
• Can internal representation changes predict behavioural transitions?
• Can these observations guide the next intervention?

The framework is designed to support controlled empirical research into:

• alignment stress testing;
• objective transfer;
• proxy optimisation;
• policy drift;
• evaluator-specific behaviour;
• durability of learned preferences;
• reasoning-strategy change;
• longitudinal post-training dynamics.

---

Two Coupled Systems

The project separates the system that changes the model from the system that observes the change.

flowchart LR
    subgraph Training["Optimisation loop"]
        A[Base model or parent adapter]
        B[SFT / DPO / GRPO]
        C[Child adapter]
        D[Prediction audit]
        E[Failure analysis]
        F[New curriculum]
        A --> B --> C --> D --> E --> F --> B
    end

    subgraph Observability["Alignment observability loop"]
        G[Adapters]
        H[Curricula]
        I[Prediction audits]
        J[Run metadata]
        K[Research engine]
        L[Behavioural transitions]
        M[Policy transitions]
        N[Representation analysis]
        O[Research report]
        G --> K
        H --> K
        I --> K
        J --> K
        K --> L
        K --> M
        K --> N
        L --> O
        M --> O
        N --> O
    end

    C -. checkpoint evidence .-> G
    D -. audit evidence .-> I
    F -. intervention evidence .-> H
    O -. validated hypothesis .-> E

Loading

The optimisation loop improves or alters the model.

The observability loop asks what changed, how it changed, whether it transferred, and what evidence best explains the transition.

The observability framework begins as a retrospective scientific instrument. Only validated diagnostics should later become inputs to an optimisation controller.

---

Alignment Trajectories

A sequence of checkpoints is represented as:

flowchart LR
    M0[Base model] --> M1[Adapter 1]
    M1 --> M2[Adapter 2]
    M2 --> M3[Adapter 3]
    M3 --> MN[Adapter N]

    M0 -. Δ0→1 .-> M1
    M1 -. Δ1→2 .-> M2
    M2 -. Δ2→3 .-> M3
    M3 -. Δ3→N .-> MN

Loading

Each transition may be associated with:

• a curriculum;
• an optimiser;
• hyperparameters;
• a preference dataset;
• a frozen audit;
• fixes and regressions;
• model-native likelihood shifts;
• changes in reasoning strategy;
• representation-level changes.

An alignment trajectory is the sequence of behavioural, policy, and representation changes induced by these interventions.

The framework analyses both:

1. point states — what each checkpoint does;
2. transition vectors — how the child differs from its parent.

---

Observability Layers

The framework is organised into four analytical layers.

flowchart TD
    A[Checkpoint lineage] --> B[Behavioural observability]
    B --> C[Policy observability]
    C --> D[Representation observability]
    D --> E[Objective-level hypotheses]

    B --> B1[Fixes and regressions]
    B --> B2[Durability and volatility]
    B --> B3[Generalisation]

    C --> C1[PTA]
    C --> C2[RPMS]
    C --> C3[Conflict and evaluator probes]

    D --> D1[Hidden-state deltas]
    D --> D2[Activation trajectories]
    D --> D3[Behaviour–representation correlation]

Loading

1. Behavioural observability

Measures observable change across checkpoints:

• stable correct;
• fixed;
• regressed;
• persistent wrong;
• fallback removed or introduced;
• format repaired or degraded;
• calibration changes;
• consistency under paraphrase;
• durability across descendants;
• correctness volatility;
• family-level co-movement.

The goal is to move beyond a single accuracy delta and describe the structure of change.

2. Policy observability

Measures model-native preference change:

• exact-pair preference margins;
• held-out preference margins;
• chosen-response reinforcement;
• rejected-response suppression;
• objective-conflict margins;
• evaluator disagreement;
• memorisation gaps;
• cross-family transfer.

3. Representation observability

Planned work will examine:

• layerwise hidden-state deltas;
• residual-stream trajectories;
• token-level entropy;
• logit-margin changes;
• LoRA-induced activation shifts;
• whether internal changes predict behavioural transitions;
• whether related prompt families share a reproducible internal direction.

This layer is not yet validated and is deliberately separated from completed behavioural claims.

4. Objective-level hypotheses

The strongest future use is controlled testing of whether observations are consistent with:

• intended objective acquisition;
• narrow pair fitting;
• shallow proxy learning;
• evaluator gaming;
• policy drift;
• unstable competing objectives.

The framework does not infer a hidden motive from one output. It seeks converging evidence across diverse prompts, evaluators, checkpoints, and internal signals.

---

Policy Transition Analysis

Policy Transition Analysis (PTA) is a proposed optimiser-independent module within the wider observability framework.

It studies how policy preferences and observable behaviour change between parent and child checkpoints.

PTA can analyse checkpoints produced by:

• SFT;
• DPO;
• GRPO;
• ORPO;
• PPO-style reinforcement learning;
• other post-training approaches.

For pairwise methods, preference pairs may come directly from training. For non-pairwise methods, evaluation pairs must be defined independently after training.

PTA combines:

• checkpoint lineage;
• frozen behavioural audits;
• fixes and regressions;
• durability;
• curriculum exposure;
• model-native preference margins;
• exact versus held-out transfer;
• objective-conflict probes;
• evaluator-conditioned behaviour.

PTA is a diagnostic framework, not a replacement for a training optimiser.

---

Realised Preference Margin Shift

For prompt (x_i), preferred completion (y_i^+), rejected completion (y_i^-), and checkpoint (t):

[ m_{i,t}

\log \pi_t(y_i^+ \mid x_i)

\log \pi_t(y_i^- \mid x_i) ]

The Realised Preference Margin Shift is:

[ RPMS_i

m_{i,t+1}

m_{i,t} ]

Interpretation:

• positive RPMS: the child checkpoint prefers the chosen completion more strongly;
• near-zero RPMS: little movement along the tested preference axis;
• negative RPMS: movement towards the rejected completion.

RPMS should be decomposed into:

[ \Delta^+

\log\pi_{child}(y^+\mid x)

\log\pi_{parent}(y^+\mid x) ]

[ \Delta^-

\log\pi_{child}(y^-\mid x)

\log\pi_{parent}(y^-\mid x) ]

[ RPMS = \Delta^+ - \Delta^- ]

This distinguishes:

• chosen reinforcement;
• rejected suppression;
• both;
• apparent margin growth caused by both likelihoods declining.

That distinction matters because suppressing an undesirable output is not necessarily the same as learning a better reasoning strategy.

Exact-pair versus held-out RPMS

Exact-pair RPMS	Held-out RPMS	Candidate interpretation
Positive	Positive	Transferable preference movement
Positive	Near zero	Pair-specific fitting
Positive	Negative	Training fit with adverse held-out movement
Near zero	Positive	Broad indirect transfer or probe mismatch
Near zero	Near zero	Little measurable policy movement

These interpretations are hypotheses to test, not conclusions encoded into the metric.

---

Current Empirical Result

The project began with an external embedding-space hypothesis:

successful curriculum-guided post-training would produce more coherent local geometry, stronger repair-vector consistency, and semantic movement aligned with the preferred response direction.

That hypothesis was tested using:

• prompt-only embeddings;
• output-only embeddings;
• prompt-plus-output embeddings;
• local k-nearest-neighbour purity;
• neighbourhood distance;
• failure entropy;
• repair-vector coherence;
• semantic preference-transition cosine;
• matched controls.

The best-supported conclusion was:

The external semantic embedding-space explanation was not supported in this setting.

Weak positive local-purity shifts were uncertain, neighbourhood distances increased rather than contracted, failure entropy increased at larger neighbourhood sizes, repair vectors lacked coherent direction, and semantic preference-transition cosine did not distinguish fixes from regressions.

This negative result motivated the shift from external text geometry towards:

• model-native likelihood measurements;
• longitudinal behavioural transitions;
• exact versus held-out transfer;
• representation-level analysis.

See docs/policy_transition_analysis_research_note.md for the full research note and limitations.

---

Research Engine

The observability framework is executed by a generic, task-agnostic offline research engine.

The engine owns:

• task discovery;
• runtime profiles;
• dependency-aware step execution;
• CPU/GPU preflight;
• input and step hashing;
• resumability;
• artifact contracts;
• provenance;
• logs and manifests;
• deterministic packaging;
• offline installation;
• public/private package boundaries.

Task plugins own:

• scientific assumptions;
• domain-specific inputs;
• experiment steps;
• metrics;
• statistical tests;
• figures;
• scientific reports.

flowchart TD
    A[CLI] --> B[Task registry]
    B --> C[Task plugin]
    C --> D[Runtime profile]
    D --> E[Execution planner]
    E --> F[Step DAG]
    F --> G[CPU or GPU runner]
    G --> H[Artifact validation]
    H --> I[Run manifest]
    H --> J[Tables]
    H --> K[Figures]
    H --> L[Reports]

Loading

This separation keeps the engine reusable while allowing the longitudinal alignment study to remain the first reference plugin.

---

Plugin Model

Each task plugin defines:

• task_id;
• supported profiles;
• required external inputs;
• step dependencies;
• CPU/GPU requirements;
• expected artifacts;
• task-specific runners;
• report generation.

The current source checkout exposes:

• dummy;
• longitudinal_inventory;
• longitudinal_alignment.

Package discovery is capability-aware:

• engine_public exposes only dummy;
• longitudinal_template exposes the longitudinal tasks;
• unavailable tasks produce a clear task not installed in this package message;
• the historical geometry profile remains source-only until its package contract is repaired.

Example task flow

flowchart LR
    A[Task plugin] --> B[Profile]
    B --> C[Preflight]
    C --> D[Inventory]
    D --> E[Audit]
    E --> F[PTA]
    F --> G[Reports]

Loading

---

Longitudinal Alignment Workflow

The reference task reconstructs a post-training history from:

• adapter archives;
• curricula;
• frozen audits;
• preference pairs;
• run metadata;
• public/private evaluation metadata;
• optional internal model signals.

flowchart TD
    A[Adapter archives] --> E[Canonical run registry]
    B[Curricula] --> E
    C[Audit set] --> E
    D[Run metadata] --> E

    E --> F[Lineage reconstruction]
    F --> G[Frozen behavioural audit]
    G --> H[Transition classification]
    H --> I[Exact-pair scoring]
    H --> J[Held-out scoring]
    I --> K[PTA]
    J --> K
    K --> L[Statistics and reports]

Loading

Public profiles

• inventory
• smoke
• audit
• pta
• full

inventory is CPU-capable.

Model-native profiles require external model assets, adapters, dependencies, and suitable GPU resources.

---

Quick Start

Source checkout

List tasks:

python3 -m researchpkg.cli task-list

Run the generic validation task:

python3 -m researchpkg.cli task-run \
  --task dummy \
  --repo-root .

Run CPU-side longitudinal inventory:

python3 -m researchpkg.cli task-run \
  --task longitudinal_inventory \
  --repo-root .

Run the alignment task’s inventory profile:

python3 -m researchpkg.cli task-run \
  --task longitudinal_alignment \
  --profile inventory \
  --repo-root .

Expected inventory behaviour

Inventory can:

• discover adapter ZIPs;
• verify archive structure;
• normalise score-bearing filenames;
• inspect adapter metadata;
• discover curriculum candidates;
• build a run registry;
• report unresolved mappings.

It does not require model loading.

Model-dependent phases should stop at preflight when the base model, dependencies, GPU capability, or disk budget are insufficient.

---

Public Packages

engine_public

A minimal generic package containing:

• the core engine;
• CLI;
• packaging and extraction utilities;
• deterministic manifests;
• the fully runnable dummy task;
• generic tests;
• approved public documentation.

Its installed task list reports only:

dummy

longitudinal_template

A public-safe research template containing:

• the generic engine;
• longitudinal task plugins;
• Experiments 04–08;
• synthetic fixtures;
• public input contracts;
• CPU-side inventory support;
• GPU-aware preflight for model-native phases.

It does not contain:

• base-model weights;
• LoRA adapters;
• private datasets;
• raw historical outputs;
• credentials;
• runtime caches.

---

Build Public Packages

bash scripts/package_colab_bundle.sh \
  --package engine_public

bash scripts/package_colab_phase56_gpu_bundle.sh \
  --package longitudinal_template

Outputs:

dist/engine_public.tar.gz
dist/engine_public.tar.gz.manifest.json
dist/engine_public.tar.gz.sha256

dist/longitudinal_template.tar.gz
dist/longitudinal_template.tar.gz.manifest.json
dist/longitudinal_template.tar.gz.sha256

The package builder produces deterministic archives with embedded and sidecar manifests plus SHA-256 checksums.

---

Offline Installation

mkdir -p /tmp/research-engine
tar -xzf dist/engine_public.tar.gz \
  -C /tmp/research-engine

cd /tmp/research-engine

python3 -m pip install \
  --no-index \
  --no-build-isolation \
  --no-deps \
  .

Then:

python3 -m researchpkg.cli task-list

python3 -m researchpkg.cli task-run \
  --task dummy \
  --repo-root .

---

Adding a New Experiment

A new experiment can be added in two ways.

Add a profile to an existing task

Use this when the experiment shares the same domain and inputs.

Example:

longitudinal_alignment
└── objective_conflict profile

The new profile defines:

• steps;
• dependencies;
• scale limits;
• resource requirements;
• expected artifacts.

Add a new task plugin

Use this when the study is a different domain.

Example:

researchpkg/tasks/
├── longitudinal_alignment.py
├── sensitive_topic_probe.py
└── model_organism_evaluation.py

Recommended development sequence:

research question
→ input contract
→ expected artifacts
→ experiment script
→ step definitions
→ runtime profile
→ synthetic fixture
→ tests
→ clean package validation
→ external GPU execution

---

Repository Structure

researchpkg/          Generic runtime, packaging, extraction, and CLI
researchpkg/tasks/    Task contracts, registry, and task plugins
experiments/          Scientific experiment implementations
scripts/              Runtime, fixture, and package entry points
config/               Public package and task configuration
tests/                Engine, plugin, artifact, and package tests
tests/fixtures/       Synthetic public fixtures
docs/                 Research notes, audits, validation, and guides
colab/                Public Colab runners and operating guides
archive/              Unsupported historical code retained for provenance

Large datasets, model weights, adapter archives, generated embeddings, caches, and private outputs remain outside Git.

---

Testing

Current CPU-side validation includes:

python3 -m compileall -q \
  researchpkg experiments scripts archive

pytest -q

Validated behaviours include:

• task discovery;
• generic dummy execution;
• package-specific task contracts;
• archive safety;
• deterministic package generation;
• clean extraction;
• offline installation;
• synthetic longitudinal inventory;
• required-artifact failure;
• optional-artifact handling;
• path-neutral manifests;
• checksum verification.

No model was loaded during public package validation.

---

Reproducibility and Provenance

Each run records:

• resolved task and profile;
• run ID;
• step ID;
• commands;
• input hashes;
• dependency hashes;
• step hashes;
• start and finish times;
• exit codes;
• warnings;
• expected artifacts;
• produced artifacts;
• software environment.

A completed step is reusable only when:

• its inputs are unchanged;
• its configuration is unchanged;
• dependency hashes match;
• required artifacts still validate.

This avoids treating a stale .phase_complete marker as sufficient evidence of reproducibility.

---

Safety and Public Release Boundary

Public packages deliberately exclude:

• model and adapter weights;
• raw training datasets;
• private audit datasets;
• historical outputs with personal paths;
• prediction caches;
• embedding arrays;
• generated packages;
• unreadable private notebooks;
• credentials and key files.

See:

• docs/PUBLIC_RELEASE_AUDIT.md
• docs/RELEASE_VALIDATION.md
• docs/PACKAGE_CONTRACT_REVIEW.md
• docs/LONGITUDINAL_TEMPLATE.md

---

Research Status

Completed

• Generic offline research engine
• Task registry and plugin boundary
• Deterministic public packages
• Longitudinal adapter inventory
• Frozen behavioural-audit harness
• External embedding-geometry investigation
• Public release safety audit
• Synthetic fixture validation

Current conclusion

The external semantic embedding geometry tested in the initial study did not robustly explain the observed post-training changes.

Under empirical validation

• Policy Transition Analysis
• Realised Preference Margin Shift
• Exact-pair versus held-out transfer
• Chosen reinforcement versus rejected suppression
• Behavioural durability
• Cross-family policy movement
• Full multi-adapter analysis

Future work

• Representation observability
• Objective-conflict probes
• Controlled proxy-objective adapters
• Evaluator-shift analysis
• Model-organism experiments
• Multi-seed replication
• Uncertainty-aware intervention selection
• Outer-loop optimisation control

---

Alignment-Relevant Research Directions

The next experiments are designed around a stronger alignment question:

Can longitudinal observability distinguish intended objective acquisition from pair fitting, proxy learning, evaluator-specific optimisation, and unstable policy drift?

Candidate controlled experiments include adapters trained from the same parent model:

1. intended-objective adapter;
2. shallow-proxy adapter;
3. weak-evaluator-gaming adapter;
4. neutral or composition-matched control.

The framework can then test whether behavioural, policy, and representation-level measurements distinguish these known conditions.

This is more informative than interpreting leaderboard movement without ground-truth knowledge of the training objective.

---

Limitations

The framework does not currently establish:

• hidden motives;
• deceptive intent;
• human-like understanding;
• a unique latent reasoning state;
• that RPMS predicts generalisation;
• that PTA is a validated alignment method;
• causal attribution from historical adapter comparisons;
• that alignment has been solved.

Additional limitations:

• the historical archive contains a small number of heterogeneous runs;
• some parent-child relationships require manual confirmation;
• held-out pair quality depends on the verifier;
• preference margins measure selected axes rather than the entire policy;
• full scoring is GPU-intensive;
• model-specific dependencies may affect reproducibility;
• representation-level observability is future work;
• controlled multi-seed replication has not yet been completed.

These boundaries are part of the research design, not hidden implementation details.

---

Roadmap

flowchart TD
    A[Phase 1: Behavioural observability] --> B[Phase 2: Policy observability]
    B --> C[Phase 3: Representation observability]
    C --> D[Phase 4: Objective-conflict observability]
    D --> E[Phase 5: Alignment-aware intervention control]

    A:::done
    B:::current

    classDef done fill:#dff5df,stroke:#2f7d32
    classDef current fill:#fff4cc,stroke:#9a7b00

Loading

Phase 1 — Behavioural observability

• fixes;
• regressions;
• durability;
• volatility;
• curriculum exposure.

Phase 2 — Policy observability

• PTA;
• RPMS;
• held-out transfer;
• chosen/rejected decomposition.

Phase 3 — Representation observability

• hidden-state trajectories;
• activation deltas;
• internal change correlated with behaviour.

Phase 4 — Objective-conflict observability

• proxy objectives;
• evaluator shifts;
• monitor-conditioned behaviour;
• known model organisms.

Phase 5 — Alignment-aware intervention control

• predict intervention utility;
• select curriculum;
• choose SFT, DPO, or GRPO;
• apply proximal constraints;
• re-audit and repeat.

---

Long-Term Vision

The long-term objective is a reproducible scientific observatory for post-training model evolution.

Rather than evaluating isolated checkpoints, the framework aims to reconstruct how successive interventions alter:

• behaviour;
• policy preferences;
• generalisation;
• internal representations;
• alignment-relevant failure modes.

The engine provides the experimental substrate. The observability framework defines the measurements. The alignment trajectory is the object of study.

The intended contribution is not a claim that one metric reveals a model’s true internal objective. It is a disciplined system for gathering converging evidence about how post-training changes a model, which changes transfer, which regress, and which hypotheses deserve stronger causal tests.

---

Licence and Citation

This project is licensed under the MIT License. You may use, copy, modify, merge, publish, distribute, sublicense, and sell copies of the software subject to the conditions of that licence.

Copyright © 2026 Graham Bonomo-Jackson.

No DOI, formal publication, paper acceptance, or validated novelty claim is currently asserted.