Add per-tag agent error-rate figures for the error analysis section - #16
Open
vissutagunawan wants to merge 13 commits into
Open
Add per-tag agent error-rate figures for the error analysis section#16vissutagunawan wants to merge 13 commits into
vissutagunawan wants to merge 13 commits into
Conversation
…L2 simulation guidelines
Adopts the seatau --seatau-experiment crosslingual approach from main and
removes the parallel, stale --crosslingual/--language implementation (and its
bundled utf-8/litellm patches) so there is a single way to run crosslingual.
- Deleted data/tau2/user_simulator/crosslingual/{simulation_guidelines,simulation_guidelines_tools}.md
- Took main's version of all 13 branch-touched files (cli, simulation config,
runner build/helpers/batch/checkpoint, user_simulator, llm_agent, io/llm utils, run)
Aggregates the 13 judge error tags (source: agent/user, severity: critical/benign) from results_reviewed.json across all four scenarios (English Baseline, L2 Interaction, L2 Tool, L2 Domain), pooling within each scenario x language x domain cell before averaging unweighted over the 3 domains and 5 L2 languages. Supports two denominators (per-simulation, per-turn) computed from the same cell counts.
Two-panel horizontal bar chart, one panel per role, showing all 13 judge tags with no threshold filtering -- including the near-zero tool-related tags for the user role and the always-zero interruption_error, both of which are meaningful nulls (the user only has tools in the dual-control Telecom domain; interruption_error only applies to full-duplex simulations, which this dataset never uses). Bars give average error occurrences per 100 turns of the relevant role, stacked by severity (critical/benign) and shaded by scenario. Palette matches src/seatau/plot/config.py's SEA_COLORS (blue/red), with per-scenario alpha spanning [0.5, 1.0] so English Baseline renders at the same shade Figure 5 uses and L2 Domain renders at the raw config hex.
Same aggregation as the appendix figure, restricted to the agent role and rendered single-column (3.35 x 3.2in, native width matches ACL's column width) so it fits the main text without a two-column figure* spanning the page. Drops interruption_error (structurally zero, since it only applies to full-duplex simulations); the remaining 12 tags, including both tool-schema tags, are kept so the figure can support the "policy vs. tool-schema vs. comprehension" root-cause argument in the discussion section.
figs/ on origin/main is PNG-only (matching the paper's \includegraphics targets), so track just the PDFs here for print quality and drop the PNGs -- they're regenerated by the script for quick preview and don't need to be in history. error_tags_agent_col.pdf -> avg_error_tags_occ_agent.pdf error_tags_per_turn.pdf -> avg_error_tags_occ_per_100_turns.pdf
scripts/plot_error_tags.py predated this branch's merge of main's plotting package (src/seatau/plot/), so it duplicated SEA_COLORS by hand and wrote its own rcParams/save loop instead of using config.py/plot_utils.py. It also ran its entire aggregation and both figure builds at module level, which would double-execute under the CLI dispatcher's import-then-call-main() pattern. Move to src/seatau/plot/error_tag_rates.py, following the package's conventions: - SEA_COLORS, DEFAULT_FIG_DIR, EXPORT_FORMATS imported from config.py rather than duplicated - apply_style()/save_figure() from plot_utils.py rather than a manual rcParams block and try/except savefig loop - aggregation and figure-building wrapped in real functions (collect_cells, aggregate, build_*_figure), with main()/parse_args() matching the pattern every other module in this package uses - registered in registry.py under COUPLED_FIGURE_MODULES (one module, two figure stems), matching language_drift.py's precedent for a script that produces more than one output - output moves from scripts/figures/ to figs/ (DEFAULT_FIG_DIR), matching where every other figure in this package writes and where the paper's \includegraphics calls already expect to find these two files Verified end-to-end via `python -m seatau.plot avg_error_tags_occ_agent`; the regenerated PDFs replace the placeholder content from the old script's manual rcParams path (font fallback and style-reset differences only, no data or layout change). Deliberately unchanged in this pass: still reads results_reviewed.json directly rather than a precomputed data/analyses/*.csv like sibling modules (no such recap exists yet for this data), and TAGS is still a flat per-role list rather than importing the narrower USER_VALID_TAGS/AGENT_VALID_TAGS split from seatau.utils.error_tags.
The four scenarios were distinguished only by alpha (0.5/0.67/0.83/1.0) over a single hue -- four shades of one colour are hard to tell apart at a glance. Switch to one hatch pattern per scenario, drawn at full saturation, and flip crit/benign to red/blue so the alarming half reads as red. - SCENARIO_HATCH is used verbatim for the bars and the legend swatches, so the key's hatch density matches the plot. The repeat counts are denser than a bar alone needs because matplotlib's hatch tile is a fixed device size and a sparser pattern renders blank on a swatch-sized patch. - the scenario legend lists top-to-bottom in the same order the bars stack within each cluster (L2 Domain first), so the key reads without mentally reversing it. - hatch.linewidth pinned to 0.6; the 1.0 default floods the dense patterns into a near-solid block. - _scenario_legend() takes fontsize/handlelength/bbox/linewidth and is now reused by the main-text figure instead of a duplicated inline legend. - both figures get more row height so thin bars keep their dot hatch. - module comments trimmed to the load-bearing rationale. Regenerated figs/*.pdf; PNGs stay untracked per 2f92271.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the two error-tag-rate figures used by the Error Analysis section, plus the module that builds them.
What this adds
src/seatau/plot/error_tag_rates.py— aggregates per-tag error rates straight fromresults_reviewed.jsonand builds both figures from one aggregation pass:avg_error_tags_occ_agent— main text, agent only, single columnavg_error_tags_occ_per_100_turns— appendix, agent + user, all 13 tagssrc/seatau/plot/registry.py— registers both stems underCOUPLED_FIGURE_MODULES(one module, two outputs), followinglanguage_drift.py's precedentfigs/*.pdf— the rendered figuresRates are occurrences per 100 role-matched turns, pooled within each scenario x language x domain cell, then averaged unweighted over the 3 domains and 5 L2 languages.
Encoding
Scenario is carried by hatch pattern rather than colour shade — four alpha levels of one hue were hard to tell apart, especially for colour-blind readers. Severity stays on colour (red = critical, blue = benign).
The legend uses the same
SCENARIO_HATCHstrings as the bars, and lists top-to-bottom in the order the bars stack within each cluster, so the key needs no mental reversal.Notes
results_reviewed.jsondirectly rather than a precomputeddata/analyses/*.csvlike sibling modules — no such recap exists for this data yet.