Conversation
The R0 models were not learning. Two separate degenerate modes were found and fixed, one objective was made learnable by modelling the right quantity, and the three-round campaign path now runs end to end. Model - Retire `default_current` to `legacy_matern_no_prior` and register `dim_scaled_prior` as the default. The old contract is kept byte-identical so archived Step 2B/2C runs stay reproducible; it must now be asked for by name. - BoTorch 0.15.1 already supplies a dimension-scaled LogNormal lengthscale prior; MOBO-Kit was discarding it by passing an explicit covar_module. Restored. Median ARD lengthscale 1121 -> 0.88, flat directions 6/10 -> 0/10. - The lengthscale prior alone opens a second degenerate mode: the outputscale collapses and the model calls the data pure noise (10/15 thickness folds, latent sd 1e-4 against fitted noise 0.93). Both priors are now required, and `_assert_signal_not_collapsed` checks it numerically on every fit, because a config name cannot prevent a degeneracy on refit. - Predictive interval coverage moves toward nominal on all three objectives. Objectives - Thickness trains on nanometres, not on its score. The score is a peaked Gaussian on 650 nm, so the map is 2-to-1 and destroys learnable signal. LOO R2 goes -0.503 (score) to +0.384 (nm with a structured mean). - Physics-informed mean functions, declared per objective in config. Thickness needs log(speed_1)+log(precur_conc); optoelectronic needs a single linear anneal_temp term. Opposite shapes; neither generalises. Optoelectronic's plain GP sat below the null at -0.342. - `ObjectiveSpec.model_link` records that a GP output is log-space. Link decode happens in exactly one place, so the sampling and quadrature paths cannot disagree; a test asserts they match to 4e-3. - Lognormal utility expectations use Gauss-Hermite quadrature. Moment-matching is ~500x less accurate and its bias changes sign across the range, which reorders candidates rather than shifting them. - Reference point declared in utility space. The old raw-scale point gave the optoelectronic axis 4.01x the uniformity axis. Campaign - `campaign.py`: run_r0_lhs / run_r1_ucb(5) / run_r2_qlognehvi(3), orchestrating the existing modules. 23 distinct conditions, three replicate films each. - The canonical config is runnable: resolved objectives, fixed scales, declared mean functions. - Debug/production approval tiers replaced by one validity check (count, uniqueness, on-grid, finite, spacing). Approval is a human decision recorded outside the code. - `assert_scaling_is_campaign_fixed` runs inside `build_objective_transform`, so no transform can exist without it. Data-derived scaling would make hypervolume incomparable between rounds. Workbook - `workbook_io.py` reads the source and writes candidates to a sibling file. openpyxl discards cached formula values on save, and Uniformity score is a formula column, so adding sheets to the source would blank a training column for every non-Excel reader. Verified directly. - Entry columns come from `model_source_columns(config)`, so thickness-in-nm is collected. Round detection fails closed on a partly scored sheet. - UTF-8 forced at every boundary; utf-8-sig for Excel-facing CSVs. Docs: GP_MODEL_DECISION.md records the numbers, both resolution floors, and the decision to keep sample 1. CAMPAIGN_STATUS.md orients collaborators and lists open issues. Not validated for fabrication. See CAMPAIGN_STATUS.md: an unexplained 0.089 discrepancy on optoelectronic, column AA is a stale pasted literal rather than a formula, and no proposed batch has been reviewed by a human. Tests: 517 passed, 2 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs a synthetic 3-objective, 10-input problem with a known Pareto front through run_r0_lhs -> run_r1_ucb(5) -> run_r2_qlognehvi(3). Nothing touches the experimental data, so it answers "does the algorithm work" separately from "are the measurements right". Two sign conventions fail silently and are now covered: - DTLZ2 minimises by default; negate=True is mandatory or the test would measure the opposite of optimisation. - BoTorch's Hypervolume assumes maximisation and silently DROPS points that do not dominate the reference -- no warning, no exception, just a smaller number or 0.0. The helper asserts at least one point dominates first. On the optimisation claim: cumulative hypervolume rises monotonically by construction, so that alone would pass for random sampling. The test therefore compares against a random baseline at equal budget, and asserts the MEAN gain rather than a per-seed win: measured 5 of 8 seeds, mean gain +0.075 vs +0.056, a ratio of 1.35x. With 8 added points in 10 dimensions that is the honest expectation; asserting a per-seed win would be flaky and false. Structural invariants covered: exact batch sizes (5 and 3, 23 distinct conditions), uniqueness, on-grid, in-bounds, finiteness, batch spacing well above the configured floor (0.735 and 0.859 against 0.15), proposals distinct from observed points, and determinism at a fixed seed. Pool sizes are shrunk for runtime; R2 is the bottleneck and mc_samples is the cheapest lever. Fast tests ~12 s, the multi-seed comparison ~33 s behind a new `slow` marker. Also records a finding from the recon: metrics.compute_ref_pareto_hv builds its automatic reference as mins - 1e-8, giving a degenerate 6e-8 hypervolume against 1.448 from botorch's infer_reference_point on the same data, and recomputes it per call so values are not comparable across iterations. The production path passes an explicit reference and is unaffected. Tests: 528 passed, 2 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
That machinery existed to audit a GP model this branch has since replaced. Its findings are recorded in docs/GP_MODEL_DECISION.md, so the code itself no longer earns its place: it was 16,442 lines that nothing on the campaign path imports. Removed 50 files -- 14 source modules, their tests, 5 examples, 4 configs, 9 docs and the Step 2B notebook -- plus stale demo/experiment output. Source drops from ~24k to ~9.6k lines; the campaign path itself is 7,073 lines across 16 modules. Everything is recoverable: tag pre-cleanup-2026-07-29 holds the full tree. git show pre-cleanup-2026-07-29:src/mobo_kit/<file>.py Kept despite not being on the campaign path yet, because they are genuinely useful rather than historical: discrete_refinement (exact-grid local search), sobol_pool (nested pools), candidate_diagnostics and plotting (both wanted for the visualization work). production_gate.py went with the ceremony. Its one valuable check -- that objective scales are never re-derived from observed data, which would make hypervolume incomparable between rounds -- was already extracted into campaign.assert_scaling_is_campaign_fixed, where it runs inside build_objective_transform and cannot be bypassed. main.py's propose_candidates branch was permanently blocked by that gate, i.e. dead. It now raises a message pointing at mobo_kit.campaign, which is the real proposal path and carries the objective contract and batch validity checks that the legacy path never had. Naming made consistent: configs are campaign_d2d_perovskite.yaml, example_demo.yaml, example_from_csv.yaml. No file is named after a step number any more. README rewritten around the campaign loop, the DTLZ2 acceptance evidence, the current parameter values, and where the removed history lives. Tests: 280 passed (was 528; the difference is tests for removed modules). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading order, the open issues in the order I would work them, the two resolution floors (-0.148 null and +/-0.236 resolution sd) that this project argued inside twice, the settled questions not to reopen (sample 1 stays, speed_2 does not explain the low-speed contradiction, uniformity has no learnable signal), and the two tooling traps that fail silently (openpyxl dropping cached formula values, BoTorch Hypervolume dropping non-dominating points). Also records the working advice: develop against DTLZ2, not the campaign workbook, since anything data-specific lives in config. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three of the workbook's derived score cells are pasted literals rather than formulas, so they do not update when the measurements behind them are edited. That is the failure that produced the original uniformity discrepancy, and an audit of all 15 R0 rows found the same divergence already present between AB and Y: Y evaluates the thickness Gaussian on the rounded X, AB was pasted from the same Gaussian on the unrounded T mean, and the two disagree by up to 1.7e-3. So the polarity is inverted. scores.py computes each objective from the raw measurement columns via a recipe declared in config, and the stored cells become cross-checks that warn on disagreement with a per-column tolerance -- a live formula and a deliberately rounded literal do not deserve the same one. On the R0 rows the recomputation reproduces Z to 1.1e-16, AA/R to 1.8e-15, and X to the 0.5 nm its rounding allows, so no campaign number changed except that thickness now reaches the GP unrounded. The formulas come from the removed d2d_scores.py with the polarity inverted; recover the original with git show pre-cleanup-2026-07-29:src/mobo_kit/d2d_scores.py Also here, because they share the read path: - read_candidate_results aggregates a filled-in candidate sheet to design points, so R1 -> R2 can advance at all. Thickness averages across replicate films in LOG space, matching the space the GP trains in and the train_Yvar pooling policy; within-film T1..T4 stays arithmetic, being a spatial average over one film. replicate_spread keeps the scatter that aggregation discards. - The candidate sheet now asks for raw measurement columns instead of derived scores, and round detection knows two thickness readings is complete while a missing Coverage is not. - fit_campaign_models and normalise_inputs are public, so callers stop reaching for _fit_models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compute_ref_pareto_hv defaulted to Y.min(dim=0) - 1e-8 when handed no reference, essentially the nadir of whatever data it was given, and re-derived it on every call. Two consequences: every hypervolume slab could be 1e-8 thick, and two rounds were measured against two different reference points, which is exactly what tracking hypervolume across rounds is supposed to rule out. Passing no reference now raises and names reference_point_utility. The precise condition, since the old issue text overstated it: mins - 1e-8 is harmless while some dominated point sets the per-objective minima, and collapses once the Pareto set itself sets them -- each point best in one objective and worst in another, which is what a genuine trade-off front is. So the default was worst exactly where the function mattered most. Pinned in a test. A reference that nothing dominates now also raises rather than reporting 0.0. BoTorch's Hypervolume assumes maximisation and silently drops non-dominating points, so an unreachable or wrongly signed reference is indistinguishable from a campaign that has not beaten its baseline yet. There were no tests on this function at all, which is how it survived: it returned a plausible number, and plausible numbers do not get questioned. main.py already passed an explicit reference. The demo notebook had one bare call, in a cell that then declared a different reference on the next line; its cached output came from the degenerate path and has been cleared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fifteen films is a real cost, and nothing here showed a human what a batch meant -- only that it passed its validity checks. batch_review.py writes a Review sheet into the candidate workbook and echoes the same text into the launcher pane, so the artifact can be forwarded to the experimental group on its own. Per candidate: inputs in physical units; predicted utility and sd per objective through posterior_utility_moments, the function the acquisition itself called; the prediction decoded into the measurement's own units; normalised distance to the nearest observed point; and which coordinates sit at a range edge, not only how many. For a log-link objective the decoded value is the posterior MEDIAN with a multiplicative interval, because exp of a mean of logs is not a mean. Labelling it as such matters: the two get quoted back later. Probes are declared in config under review:, because which counterfactual is worth asking is campaign knowledge. A probe holds every other coordinate and forces one input to a value worth interrogating. The informative comparison is the sd, not the mean: UCB pays for uncertainty, so a region skipped while the model still calls it uncertain is losing a trade-off, whereas a region skipped while the model calls it certain has been resolved -- possibly into an average. SD_MATERIALITY_RATIO makes that threshold explicit and the ratio is printed either way, because on the campaign's own R0 fit the probed sd came out 1-8% above the batch's on all three objectives, which a bare > reads as more uncertain while predicted thickness utility falls from 0.79 to 0.22. On the R0-trained batch the low-speed probe reports thickness utility 0.223 against 0.786 at an sd ratio of 1.02: skipped as known and bad, not unexplored. And because speed_1 is a feature of the thickness mean function, that confidence is a fitted global trend extrapolating to its range edge rather than a local average of the two contradictory observations there -- which the verdict says, derived from config rather than hardcoded. The launcher is a tkinter shell over inspect_campaign, gather_observations and generate_next_round, all testable without a display. It refuses an existing sheet before the ten seconds of model fitting rather than after, shows plain sentences for expected failures and a traceback only for unexpected ones, and approves nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he Step 1 spec Stale claims, all verified against the code before changing: - GP_MODEL_DECISION said the structured mean was not yet wired into campaign.py. It is; fit_campaign_models builds the mean modules. The same claim appeared as an open item and is now under Closed. - CAMPAIGN_STATUS listed removal of the legacy debug ceremony as not started. production_gate.py and 22 other modules went in 33f101f, and test_validity_report_carries_no_approval_flags holds the tiers out. - The config comment said to pool thickness replicate variance in NANOMETRES while claiming to match what the GP trains on. With response: log the GP trains on log T, so an nm^2 variance is wrong by a factor of T^2 -- 1.3e5 to 1.7e6 across the observed thicknesses, not even a constant rescaling. docs/D2D_CAMPAIGN_SPEC.md is deleted. Its grid rules are enforced by design.py and campaign.validate_batch, its thirteen unresolved decisions blocking real R1 are resolved in config, and its workbook audit describes a revision of the summary workbook that no longer exists -- it reports duplicate Uniformity score headers at Q/T, which the current workbook does not have. The one fact that lived nowhere else, the 177,816,994,740-recipe design space and why it must never be materialised, moved to the README. Recover with git show 19591cc:docs/D2D_CAMPAIGN_SPEC.md Also recorded: the workbook column audit and its numbers; sample 12's thickness being ROUND(mean(1600, 709)); the within-film log-T spread of 0.244 over 24 dof as a floor rather than an estimate of between-film variance; and a newly found issue -- the signal-collapse guard cannot distinguish a collapsed GP from a mean function that works, because it measures posterior variance, which a mean module does not enter, then reports that the posterior mean is effectively constant. Doc snippets now use the public fit_campaign_models and normalise_inputs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_assert_signal_not_collapsed compared the GP's latent sd against the fitted noise and stopped there. A mean module is not part of the covariance and so never enters posterior().variance, which means a structured mean that explains most of the data drove the residual GP's latent sd to ~0 and the fit was refused -- with a message asserting that the posterior mean is effectively constant, which is verifiably false in exactly that case. Two situations share the signature and now get different answers. True collapse -- zero-mean GP, outputscale to zero, posterior mean genuinely flat, nothing rankable -- still raises. The mean function having done its job now warns and lets the round proceed: refusing there dead-ends the campaign at the moment the physics model starts working, with no remedy available, because better data cannot be collected without first proposing conditions. The review artifact is the designed gate for a suspect batch. The warning is not a formality and says why: UCB's exploration term reads the latent posterior that just collapsed, and the mean module's coefficients are frozen buffers with no uncertainty of their own, so the narrow intervals such a model reports are understated rather than earned. It surfaces above the numbers in the launcher pane and the Review sheet, and in RoundResult.diagnostics[model_fit_warnings]. Two calibration decisions worth keeping: - Near-constant is measured against the objective's observed spread, floor 5%, not against the fitted noise sd. Noise-relative was the first attempt and is wrong: the noise is inflated precisely in the degenerate case, so the test co-varies with what it is trying to detect. Measured instance -- a linear mean on anneal_temp against a forced noise of 0.9 scored 0.38 on the noise yardstick and would have been called constant while it was tracking the data. - Only the guard's own warnings reach a human. record.warnings also collects every Python warning raised while fitting, about 18 numpy-2.0 deprecation notices per fit on this stack, and putting those in front of someone reviewing a batch is how people learn to ignore warnings. Whether a dataset trips the collapse is knife-edge: measured across residual magnitudes from 0 to 0.3 it fires at 0, 1e-4, 0.01 and 0.03 but not at 0.001 or 0.1, depending on where the MLL optimiser lands. So the guard's decision is tested directly on both branches, and the propagation tests force the condition rather than hoping data produces it. No fit on the current R0 data warns; the live campaign is unchanged. fit_campaign_models now returns (model, warnings). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_the_window_shows_a_readable_error_rather_than_a_traceback failed in a full-suite run and passed on its own. Two real problems, both mine: The launcher schedules a check() 200 ms after construction when it remembers a workbook. An earlier test in the same file checks a real workbook, which writes that path to ~/.mobo_kit/launcher.json. If pytest's tmp_path from that run still exists, the next window auto-checks it, and that result races the explicit check() the test performs -- overwriting the pane with a successful status where the test expected an error. Order-dependent, so it only appeared once the suite grew. Second, and true regardless of the race: a test suite has no business writing to the user's home directory. Both fixed by an isolated_settings fixture that stubs remembered_workbook to None and remember_workbook to a no-op. Verified with three consecutive runs of the file and two consecutive full-suite runs. Noted for the record: the previous commit was made in the same shell invocation as its verification run and did not stop when that run reported this failure. Verify first, commit second. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rnings Two hardenings. The launcher runs its work off the main thread and reports back through a queue, so a reply could paint the pane with an answer to a question the user had moved on from. Two ways in: the auto-check scheduled 200 ms after startup against the remembered workbook, and a second press while the first was still running. Human reaction times made it unlikely rather than impossible, and the previous commit only papered over it in the test fixture. Now each dispatch takes a request id and a reply carrying a stale one is dropped; a status reply also names the workbook it examined and is dropped if the selection has changed since. A dropped reply still clears the busy state -- without that the fix would trade a rare wrong answer for a window that disables its own buttons forever. Pending auto-checks are cancelled as soon as the user browses or presses anything. The drop rules are tested by putting messages on the queue, not by racing two real threads. The first version of those tests did race them, passed alone, and failed intermittently in a full-suite run. A flaky test of a race-condition fix is worse than no test, because it teaches people to re-run until green. Second: the unfiltered fit-warning list is retained under diagnostics[fit_warnings_raw], unsurfaced. The human-facing channel stays guard-only -- 18 numpy deprecation notices per fit is how people learn to ignore the one warning that matters -- but a BoTorch or scipy convergence warning the filter dropped is exactly what someone will want when a fit looks strange weeks from now. Entries carry objective, stage and category. Also noted next to the guard's spread-based threshold: it is a diagnostic and never touches utility space, so it does not fall under the campaign-fixed-scaling rule, which governs the objective scales feeding hypervolume. Written down so it is not later "corrected" to a fixed constant. Verified: three consecutive runs of tests/test_launcher.py and three consecutive full-suite runs, 399 passed each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each proposed condition is run in triplicate, and the scatter across those films is the only direct measurement this campaign has of its own reproducibility. replicate_variance.py pools it and hands it to the GP as train_Yvar, so the marginal likelihood no longer has to guess the noise from 15 points in 10 dimensions. Wired now, enabled later: model.observation_noise stays fit_from_marginal_likelihood until the R1 triplicates land, and turning it on is then one config key. That is the point of building it before the data exists -- arrival should be a data event, not a code event -- so it is tested against synthetic replicates end to end. Four things it is careful about, three of which fail silently otherwise: - The variance handed over is of the MEAN, pooled / n_films, because the observation is an average of n films. Passing the single-film variance understates it threefold on a triplicate. - BoTorch accepts BOTH train_Yvar and an explicit likelihood and then silently ignores the variance: the likelihood wins, stays single-element, and the replicate information is dropped with no error. Verified on 0.15.1. _build_single_task_gp now passes one or the other, never both. - Standardize rescales train_Yvar along with the targets, so it must arrive in the target's own units -- and in the model's space, which for thickness is log T, not nanometres. Aggregation and pooling therefore share one space by construction. - Zero pooled variance is refused. Replicate films agreeing to the last digit are a transcription, not a measurement, and a zero train_Yvar tells the model the observation is exact. Between-film and within-film stay distinct, in the config comment and in the module docstring. Between-film is the quantity train_Yvar needs. The within-film 0.0593 on log T (24 dof) contains no run-to-run variation, so it is a FLOOR: sanity_floor_findings reports a pooled estimate below it, because films cannot be more reproducible than points on a single film. The collapse guard now averages the noise vector rather than reading its first element, since a FixedNoiseGaussianLikelihood carries one value per observation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two operator-facing scripts, plus the results of running the first one. scripts/dtlz2_parameter_sweep.py sweeps UCB beta against the local-penalization radius on DTLZ2, whose Pareto front is known, so the answer does not depend on whether the campaign's measurements are right. The decision rule is written in the script above the code and was committed before any number existed: keep 4.0 / 0.25 unless a cell beats its mean hypervolume gain by more than the per-seed sd of gains without reducing spacing. Result, 8 seeds per cell: NO CHANGE. The default scores +0.0780 with a per-seed sd of 0.0428, so a challenger needed +0.1208; the whole grid spans +0.0776 to +0.0961 against sds of 0.043 to 0.074. Seven cells have a higher mean and none is close. Flat within noise, which is the outcome that says the default was not a lucky pick. BO beats the random baseline on the mean in 9 of 9 cells, and the edge-coordinate count is flat at 16-17 of 80 everywhere -- so neither knob is what drove the live campaign's batches onto range edges. That was the monotone anneal_temp mean function, as the review artifact said. Stated as a limit rather than buried: radius is NOT properly tested by this sweep. DTLZ2 batches land 0.72-0.98 apart, far outside every radius tried, so local penalization rarely has two candidates close enough to penalise -- visible in beta=8 giving bit-identical results at radius 0.15 and 0.25. The sweep validates beta and says little about radius. scripts/intake_new_data.py is the one command to run when the group returns re-measured or corrected data. It audits the read, checks the anchors still span the data and that the campaign-fixed scaling guard passes, recomputes BOTH floors at the new N rather than reusing the N=15 values, and gives a per-objective verdict: keep the mean function only if it beats plain by more than the resolution floor, and name the exact config block to delete otherwise. It also reports the fit guard's status per objective, which exercises the warning path added in f528f84 -- cleaner re-measured thickness is exactly the data that would trip it. On the current 15 rows it reports uniformity failing to beat the null (-0.6809), optoelectronic keeping its mean function (-0.3420 -> +0.2670) and thickness keeping its (+0.1160 -> +0.3806), with a clean guard for both. Also repaired: scripts/plot_dtlz2_report.py called the private _fit_models, whose return type changed when fit warnings started travelling with the model. It now uses the public fit_campaign_models. The CAMPAIGN_STATUS snippet had the same staleness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 0.089 optoelectronic discrepancy is closed as a numerical artifact. The decisive experiment -- disable Standardize in both pipelines -- was run: the gap does NOT collapse, moving only 0.0881 to 0.0715. So the standardization scale is about a fifth of it, not the cause. The rest is the MLL optimiser. Without the outcome transform the two pipelines are mathematically the same model: a zero-mean GP on (y - trend) and a fixed-mean GP on y with mean trend have identical marginal likelihoods, because a fixed mean only shifts the data. Yet the fits land in different places -- across folds the outputscale differs by up to 2.7%, the noise by 2.7%, the median lengthscale by 9.6% -- and at N=15 that is worth 0.07 of LOO R2. Deterministic, not stochastic: the earlier seed sweep was bit-identical across four seeds. There was never a modelling question to answer, and 0.0881 sits well inside the +-0.236 resolution floor, so it was never evidence of anything. Second: the intake command and the decision record disagreed on plain thickness, +0.116 against +0.183. Same 15 rows and no resampling, so per this project's own rule that had to be explained rather than absorbed. It is entirely the DATA, not the method: rounded X two-stage +0.1830 mean module +0.1830 unrounded two-stage +0.1160 mean module +0.1160 The pipeline contributes exactly nothing -- with no mean function the two routes are the same code -- and the whole 0.067 comes from 7 of 15 rows changing by at most 0.50 nm. Half a nanometre on seven rows moves LOO R2 by 0.067, which is the same fragility that produced the 0.089. The structured numbers move by 0.006 and the swing that justifies the mean function clears the floor either way, so no conclusion changes. intake_new_data.py is now declared canonical, and GP_MODEL_DECISION says at the top which instrument produced its tables and that it reads a column the model no longer trains on. Third: radius is now verified by construction. The DTLZ2 sweep could not test it -- batches land 0.72-0.98 apart, outside every radius tried -- and the live campaign's R1 spacing of 0.921 says the same. Inert is fine for a safety knob, but its function should not be inferred from a campaign that never exercised it, so there is now a test with three candidates crowded 0.02 apart scoring above an isolated fourth: greedy takes the two best, penalization pushes the second pick past the radius. A companion pins the inert case. Also: the single-film variance comment had its direction inverted -- passing it would OVERSTATE the mean's variance threefold, not understate it. The implemented choice was right, the sentence was not. And the plot script's unused import is gone with its blanket warnings filter scoped, so a fit-guard message can still get through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project had two floors to check before comparing LOO R2 values. It has three. Two measurements this week, from completely different directions, each moved LOO R2 by about 0.07 at N=15 without changing anything meaningful: the MLL optimiser landing elsewhere on an IDENTICAL likelihood surface (0.0715, from outputscale and noise differing by under 2.7% and the median lengthscale by 9.6%), and rounding the thickness input to whole nanometres (0.0670, from at most 0.5 nm on 7 of 15 rows). Neither is sampling noise -- both are deterministic and reproducible. So a second-decimal difference in LOO R2 at this N is below what the metric can reproduce even on identical data with identical models. Where the sampling floor says a difference may be luck, the numerical floor says it may not be a difference at all. Both were learned by walking into them. A corollary worth its own line: under the same rounding the plain GP moved 0.067 while the structured model moved 0.006. A model whose answer turns on half a nanometre is reporting arithmetic; one that ignores it is reporting a trend. That is independent evidence for the mean function, arrived at without looking at either model's score. Also in this pass: - The 0.089 closure now leads with the mechanism and cites the floor as a corollary, so the record shows this project's "explained, not absorbed" rule was satisfied rather than sidestepped. - Confirmed and strengthened the sentence stating that the thickness mean function's evidentiary weight is the rank permutation (p = 0.0350, CI [0.0270, 0.0446]), with the R2 swing merely consistent -- it survived the reconciliation edits, and now says explicitly that the reconciliation could not have touched it, being about ranks rather than a regression score. - CAMPAIGN_STATUS's model-state table now carries the canonical intake numbers (+0.116 -> +0.381 thickness, -0.342 -> +0.267 optoelectronic) with the older figures explained rather than silently contradicted. - HANDOFF rewritten as a conclusion: where the project stands, what is actually open, the three floors, the four tooling traps, and which instrument produced which numbers. Its old "open issues" list was mostly closed items. - Stale text removed: issue 3 still described the 0.089 as unexplained with untested suspects; issue 8 was headed "Not started" for work that is done; a cross-reference pointed at an issue number that had shifted. - README gained a runnable snippet showing where Y_model comes from -- the computed values, not the workbook's stored score cells -- and a note that the two tuned parameters were checked under a pre-committed rule. - Sampling floor and resolution floor are the same number under two names; the floors section now says so rather than leaving a reader to guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audited the notebook and the requirements, neither of which had been checked against this arc's changes. The requirements are clean: all 17 pins match what is installed, and every pyproject range contains its pin. Its header still pointed at docs/STEP1_HANDOFF.md, removed in the cleanup commit, and said nothing about why linear_operator is pinned when pyproject does not declare it. Both fixed -- the pin matters because it is a gpytorch/botorch transitive dependency whose version affects fitted numbers, and a second-decimal change in LOO R2 is inside the numerical floor. The notebook is structurally sound: every mobo_kit import resolves, no call passes a keyword the current signature does not accept, and it no longer reads stored score columns, reaches for private helpers, or calls compute_ref_pareto_hv without a reference. What it does do is build its GPs with models.fit_gp_models and models.loocv_select_models -- the prior-free ScaleKernel(MaternKernel(nu=2.5, ard)) construction that GP_MODEL_DECISION.md documents as degenerate on small data: ARD lengthscales from 0.13 to 38,000, six to nine of ten directions switched off, noise pinned at its floor so the model believed the data were noiseless. That is the model this project spent a session replacing, and a reader following the notebook as the recommended path would rebuild it. Rather than rewrite a demo that works and cannot be re-executed here, the notebook now opens with a scope note: it is the general toolkit API on an arbitrary CSV, not the campaign path; the campaign uses fit_model_variant with dim_scaled_prior via campaign.fit_campaign_models; objective values are computed rather than read; and compute_ref_pareto_hv now requires an explicit reference. HANDOFF says the same in one paragraph, and that models.py is legacy-but-live rather than dead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs R0 -> R1 -> R2 end to end with a GP fitted to the 15 real films standing in for the measurements, so the loop can be exercised before the R1 triplicates exist. 13 OFAT cells (9 radii at beta 4, 5 betas at radius 0.25), 45 input pairs as views of one 10-D run per cell, and a manifest of batch hashes that answers whether a knob changes the proposed batch at all. Derived from Annie Xu's examples/round_simulations.py on ax_plots_simulation. Her directory convention, slug rule, round legend and overlaid-boxplot figure are preserved. More importantly, her branch already carried the fix for the R1 baseline mis-encoding that colin still has: run_r1_ucb hands the objective transform measurement-space nanometres, which it exponentiates a second time, pinning every observation's thickness utility to exactly 0.0. Her _physical_to_model_output is that fix, written before we knew the bug existed. This script uses the corrected encoding and records both baseline hypervolumes (0.436442 against 0.004659) on every manifest row. campaign.py is deliberately untouched here; the fix lands as its own change. Two corrections recorded in docs/ROUND_SIM_DELTA.md are corrections to the brief, not to her code: her exp(mu + v/2) was the correct lognormal mean and her physical-mean label was accurate for it, and her slice fixing already used the median with a grid snap. The oracle here still reports the median, for a different reason -- a variance-dependent oracle bulges wherever R0 is sparse, so the simulated ground truth would encode sampling history rather than belief. Main finding: radius is not inert on this landscape. Achieved R1 spacing staircases 0.455 -> 0.921 as radius goes 0.05 -> 0.45, saturating around 0.30, and buys that spacing at a cost of 11 -> 15 range-edge coordinates. DTLZ2's not-exercised conclusion does not transfer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_r1_ucb handed its observed HVI baseline to ObjectiveTransform.transform in MEASUREMENT space. That transform is a model-output decoder: it applies exp() to a log-link objective before computing utility, so thickness in nanometres was exponentiated a second time. exp(360..1303) saturates the 650 nm Gaussian to exactly 0.0 -- finite, so neither the transform's own finiteness check nor the caller's fired -- and every observation's thickness utility was zero. Measured on the 15 R0 rows: baseline hypervolume 0.004659 against a correct 0.436442, a factor of 94, with the baseline Pareto set collapsing from 5 points to 2. Every candidate's improvement was scored against a front with no thickness axis. On the live campaign's own R1 step this moves one of the five proposed conditions (speed_1 4000 -> 2500, precur_conc 1.70 -> 1.45) and the batch's minimum spacing from 0.9209 to 0.6337 -- so the 0.921 on record was itself an artifact of the defect. The fix is ObjectiveTransform.encode_measurements, with transform_measurements as the one-call safe route, and run_r1_ucb encoding before it proposes. The acquisition modules are untouched: ucb_hvi.py stays byte-identical and the defect was in campaign.py orchestration. Annie Xu found this independently on ax_plots_simulation and fixed it there as _physical_to_model_output, before we knew it existed. This promotes her fix to the public contract. Audit of every ObjectiveTransform.transform call site: objectives.py 403/481/516 and ucb_hvi.py:342 all operate on posterior samples, which are already in model space; batch_review.py never routes measurements through the transform at all (it decodes exp(mu) itself and labels it a median). Exactly one call site was defective, ucb_hvi.py:698, reached only from run_r1_ucb. R2 was never affected: qLogNEHVI takes train_X_norm and derives its baseline through the model. Two things about why it survived 446 tests, both now closed. The baseline was invisible. It was a plausible finite number that nothing reproduced independently -- the same shape as the hypervolume auto-reference and the swallowed train_Yvar. run_r1_ucb now reports observed_baseline_hypervolume and observed_baseline_pareto_size, and the tests recompute them by a separate route. Reverting the encoding while keeping the diagnostic fails on the value, not merely on a missing key; that was checked rather than assumed. The synthetic acceptance test could not have caught it. Every DTLZ2 objective is affine, and for an affine objective measurement space and model space are the same numbers, so a link-encoding mistake is invisible by construction. DTLZ2 now also runs with its third objective reached through a log link -- reported as exp(f2), modelled with response: log, so the GP trains on the same latent quantity by a different route -- and asserts that no measurement strictly inside its anchors scores exactly zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The R1 batch was built on the mis-encoded UCB-HVI baseline fixed in 4b76670, so it is withdrawn. Regenerated through launcher.generate_next_round -- the same path the double-click launcher uses -- into local_inputs/Summary Table_R1_Candidates.xlsx. Four of the five conditions are unchanged. One is replaced: speed_1 4000 / precur_conc 1.70 out, speed_1 2500 / precur_conc 1.45 in. Baseline hypervolume 0.004659 -> 0.436442, baseline Pareto set 2 -> 5 points, minimum spacing 0.9209 -> 0.6337. NO FILMS WERE FABRICATED from the withdrawn batch; the human review gate caught it, which is what it is for. There was no prior candidate workbook on disk to archive: the withdrawn batch had been described in CAMPAIGN_STATUS and echoed to the launcher pane but never written. The withdrawal is declared in configs under review.notes, so it travels with the Review sheet if that is forwarded on its own. Delete that note once R1 is measured. Full diff in docs/R1_BATCH_WITHDRAWAL.md. Two things the defect had written into the docs, now corrected. The claim that radius is "probably inert" on the live campaign rested on the R1 batch's minimum spacing of 0.921 -- a figure the mis-encoding itself inflated. Corrected, the live batch spaces at 0.6337, and the round simulation measures a monotone staircase: achieved R1 spacing 0.455 / 0.455 / 0.455 / 0.543 / 0.720 / 0.921 / 0.921 / 0.921 / 0.921 across radius 0.05 to 0.45, with nine cells producing six distinct batches. radius binds below about 0.30 and saturates above it, and buys spacing at a cost of 11 -> 15 range-edge coordinates. That trade-off is a policy choice for the group. 0.25 stays the default as a declared choice; the hypervolumes are single-seed and cannot rank cells, so re-run the arm at ~5 seeds before moving it on performance grounds. Issue 4's probe numbers turn out NOT to be artifact-born, and are kept rather than voided. The 0.786 -> 0.223 fall with an sd ratio of 1.02 describes R1_C01, one of the four conditions that survived the reissue, and re-reads identically from the regenerated artifact. The batch-level figures are added alongside: 0.791 -> 0.299, mean sd ratio 1.10. The speed_1 = 1000 corner is still skipped (minimum proposed speed_1 is still 1500) and anneal_temp still pins at 100-105. The bug is recorded as CAMPAIGN_STATUS issue 9, and as the third plausible-finite-number failure after the hypervolume auto-reference and the swallowed train_Yvar: all three were finite, ordinary-looking wrong answers that no test reproduced independently. GP_MODEL_DECISION files it deliberately OUTSIDE the three floors -- a floor says when a difference is too small to argue about, and never licenses accepting one that is simply wrong. plot_round_simulation.py now calls the public run_r1_ucb instead of its own corrected copy, verified to reproduce it hash-for-hash on three cells, so the 13-cell manifest already on disk stays valid. Its manifest carries the baseline tripwire: reported (from the acquisition) must equal independent (recomputed by another route) or run_cell raises, with the unencoded value kept beside them as the size of the historical mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…aims README now separates the original design from the reconstruction: initially designed by Ethan Schwartz, Daniel Abdoue, Nicky Evans and Tonio Buonassisi, updated and reconstructed by Ziyang (Colin) Qi and Annie Xu. Three stale claims the first doc sweep missed, all traceable to the same defect or to work that has since landed. test_batch_selection.py still carried "the knob is probably inert there too" in a docstring, resting on the 0.921 spacing the mis-encoded baseline produced. The test itself is unaffected -- it verifies the mechanism by construction, which is exactly why it kept its value when the campaign evidence turned out to be wrong, and the docstring now says so. HANDOFF listed plotting as "the obvious next build"; it landed in f191730. Its "settled, do not reopen" entry for radius = 0.25 is split: beta = 4.0 stays settled, radius = 0.25 stays as a DECLARED POLICY CHOICE with the measured staircase and the diversity-versus-edge-pinning trade-off, because the evidence that made it settled was itself an artifact. CAMPAIGN_STATUS's plotting section now points at the implementation, and its round-comparison colours are corrected to the palette actually shipped (R0 #2a78d6 / R1 #eb6834 / R2 #1baf7a, shared with plot_dtlz2_report.py) rather than the grey/blue/orange it had described. Also strips a UTF-8 BOM accidentally written into plot_round_simulation.py by a PowerShell Set-Content during an earlier refactor. Python's loader tolerates it, so nothing failed, but it is the only such file in the repo and would confuse any tool reading the source as plain UTF-8. Not done deliberately: `black` would reformat 22 files here, including pre-existing core modules, so the repo is not black-clean as a baseline and reformatting now would bury this work in unrelated churn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… decisions scripts/plot_shap_attribution.py explains E[utility] per objective through ObjectiveTransform.expected_transform, so thickness goes through the lognormal quadrature rather than a transformed mean and every value is in utility units. 1000 on-grid instances, background = the training conditions, seed 73 on every figure. At ten inputs KernelExplainer enumerates all 2**10 coalitions, so the attributions are exact and reproducible rather than sampled. That is asserted rather than assumed: the suite checks additivity, base_value + sum(shap) == f(x), which holds to 1.1e-16 and is an independent comparator for the attributions rather than a plausibility check. THREE PRE-REGISTERED VERDICTS. Thickness is dominated by its declared mean-function features, decisively: precur_conc 0.1553 and speed_1 0.1463 rank 1-2, together 3.7x above the third. Optoelectronic behaves the same way with anneal_temp. Those are consistency checks, not discoveries -- the config told the model those relationships -- and in_mean_function marks the affected rows. Uniformity is NOT near zero, and the expectation that it would be was wrong. Its top attribution is 0.0621 (R0-only) and 0.0890 (final), the latter exceeding optoelectronic's. The expectation conflated "no predictive signal" with "flat surface": uniformity fails the leave-one-out null (LOO R2 -0.681, permutation p = 0.82) yet its GP still fits ARD lengthscales and has a posterior mean that varies. The resulting figure looks like a textbook result -- clean monotone gradient on time_1, orderly ranking, magnitudes of +-0.15 -- and is entirely fitted noise. It is kept and captioned rather than suppressed, because a reader who saw only the beeswarm would conclude the opposite. qLogNEHVI and qNEHVI propose the IDENTICAL R2 batch, verified across four cells (default, radius 0.05, radius 0.45, beta 25). So the two "final" model states are one model, three of the nine planned figures would have been bit-identical duplicates, and the deliverable is six beeswarms with the fact stamped on each. src/mobo_kit/research_qnehvi.py carries the variant; campaign.py and the frozen acquisition modules are untouched. One real difference is handled explicitly: qLogNEHVI returns a log and qNEHVI returns the improvement, while the local penalization selector penalises in log space, so a raw qNEHVI value must be logged before it enters the selector or the two are penalised on different scales. Extreme cells: no sweep needed. Across radius 0.05, radius 0.45 and beta 25 no objective changed its top feature, and the largest shift was 0.0996 of that objective's largest attribution -- clearing the pre-registered 0.10 threshold by 0.0004. Worth saying plainly: that near-miss was on uniformity, whose attributions are noise anyway; the two objectives carrying real structure moved by at most 6.3%. The attributions describe the fitted model, not the search. A performance defect found and fixed while measuring: batching the explainer's synthetic rows at 20000 split a 23506-row block in two, and the same rows cost 0.03 s in one call against 0.59 s in two. That 20x penalty turned each final-state run into 470 s; at 65536 it is 51 s, with identical values. Two smaller corrections to what the brief asked for, both because the figure would otherwise state something false. A beeswarm colours each row against that feature's own min-max, so one colorbar cannot carry physical units for ten inputs measured in rpm, seconds, molarity and microlitres -- the per-feature range goes in the row label instead. And the footer's first line states the model's provenance (real measurements or oracle) rather than an oracle caveat on the R0-only anchor, which is fitted entirely to measured data. plotting.py::plot_shap already existed and is called only by the demo notebook. It explains the RAW posterior mean -- log(nm) for thickness, so the 650 nm target is never applied -- with nsamples=300 where ten inputs need 1024 for exact enumeration, over the training points alone. A scope note now says so and points at the campaign-correct script, matching how fit_gp_models was handled. GROUP DECISIONS RECORDED. Samples 8/12/15: the within-film variation is real and the mean is the intended summary, so no re-derivation, exclusion or re-weighting. The findings are KEPT, not retracted -- what was settled is the action, not the fact, and spread_warning_ratio keeps firing because a row whose readings split 2.3-fold is still a different kind of observation. T anom exclusion confirmed on the same basis. constraints: is empty DELIBERATELY, which is why anneal_temp is free to sit at its lower bound. HANDOFF now routes anticipated workbook and formula changes through the score recipes, a contract_version bump and one intake run, and narrows the re-measurement advice the group's decision undercut. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ng designs
scripts/plot_boxplot_sweep.py runs the same boxplot construction as the round
simulation -- utility by round, one panel per objective, every raw point drawn
over its box -- across beta in {9, 25, 36, 49} x radius in {0.05 .. 0.45}, for
three trials. 108 cells, each a full R0 -> R1 -> R2 campaign at production
settings; pools and sample counts are NOT reduced, so these stay comparable with
the existing 13-cell manifest.
The three trials differ in exactly one thing: R0. The workbook's 15 real recipes,
then two fresh Latin hypercubes at seeds 101 and 202. The oracle, the acquisition
seed and therefore the 32768-recipe candidate pool are identical throughout, so a
difference between trials is attributable to the starting design and to nothing
else. That is a sensitivity check on R0, not three independent replicates of the
pipeline, and the doc and the tests both say so.
108 cells at ~195 s is about six hours in one process, so the work shards:
--shard i --num-shards N writes one .npz per cell, and --compose renders the
pages. Twelve workers finished in about half an hour. The sharding is the one
place a silent bug would be expensive -- a dropped cell becomes a blank panel
after six hours -- so tests assert that cells[shard::n] partitions the grid
exactly for several n, and that shards stay balanced within one cell.
Output is 12 pages (3 trials x 4 betas), each 9 radii x 3 objectives, plus a
combined 12-page PDF and a 972-row summary CSV. Y-limits are shared per objective
across ALL pages: without that a flat cell and a good cell look alike and no
cross-page comparison is possible, which is the entire point.
WHAT THE SWEEP SAYS. No cell can be called best on this evidence. Mean hypervolume
gain by beta is 0.0691 / 0.0756 / 0.0753 / 0.0726 for beta 9 / 25 / 36 / 49, a
spread of 0.0065, against a between-trial standard deviation of 0.010 to 0.027.
The best cell is a different (beta, radius) in each of the three trials -- (25,
0.35) on real, (36, 0.20) on lhs_a, (25, 0.05) on lhs_b -- which is what noise
looks like, not signal. Three trials is not enough to rank 36 cells.
What the deterministic bookkeeping does show: radius still binds, producing 3 to 8
distinct R1 batches per beta, and it binds LESS as beta rises. At beta 49 only 3
to 4 distinct batches survive across all nine radii, because achieved spacing is
already 0.94 to 1.40 -- above every radius tested. High exploration spreads the
batch beyond the penalisation radius and leaves the knob nothing to do.
Standing caveat, restated on every page: the oracle is noiseless and belongs to
the same model class the optimiser fits, so the model is correctly specified by
construction and exploration has unusually little to earn. beta 36 and 49 mean
kappa of 6 and 7. This instrument systematically undervalues exactly the
parameters being swept, so "highest median utility wins" is the wrong way to read
these pages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tests/test_launcher.py::test_a_result_for_a_workbook_the_user_left_is_discarded failed 6 runs in 9 when run alone and passed inside a full-file run, which read as a race in the launcher's stale-reply handling. It is neither a race nor a launcher defect: it dies inside _tkinter.create(), reporting that it cannot read its own init.tcl with the errno string No error. pytest's default --capture=fd swaps file descriptors 1 and 2 for temp files. A Tk interpreter built while that is in force holds descriptors that are gone by the time the next one is built, and the next Tk() then cannot read init.tcl. Measured, outside this repo as well as in it: - one import-time Tk() fails the next one in 4 runs out of 5; - twenty consecutive Tk() calls inside one test body all pass; - the whole effect disappears under -s, --capture=sys and --capture=tee-sys. That explains the intermittency: whether the stale descriptors are still valid depends on what file I/O ran in between, so the same test failed alone and passed in company. Two changes, both needed. The six skipif decorators each built and destroyed a real interpreter during collection; an open_window fixture now builds the window in the test body instead. And addopts gains --capture=sys, because moving construction out of collection was not sufficient on its own - pytest re-swaps those descriptors between tests, and the third window in one process still lost its interpreter. Only capsys is used in this suite, never capfd, so nothing here depends on fd-level capture. 492 passed with 0 skipped, against 23 passed / 1 skipped per launcher-module run before. The 28-warning tail is unchanged and no extra console output appears. A source-level guard pins the collection rule, because a reinstated skipif(not _tk_available()) is exactly how it comes back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…traints A new workbook arrived with a new column layout and two objectives computed differently. Uniformity is now the MEAN of Coverage, 1 - clamped Uniformity and Phase purity, where the first campaign took their product; optoelectronic is the mean of min(Voc, 1.4)/1.4 and a supplied normalised photoconductance, where the first campaign took log10 of a product. Utility space is what hypervolume is measured in, so an objective that keeps its name while changing its construction makes every cross-campaign number incomparable while every plot still renders. So this is a new config on a new contract_version (d2d-objectives-v3-test) rather than an edit, and campaign_d2d_perovskite.yaml is marked archived. It stays complete and loadable: every number in GP_MODEL_DECISION.md is about that contract. All three recipes reproduce the stored score columns on all 15 rows, worst disagreement 2.3e-13. Per the group, the stored scores are authoritative here and the recompute is the cross-check, so a disagreement is a warning finding rather than a block. scores.py gains a `mean` recipe and two threshold transforms. `mean` does the same arithmetic as `mean_of_present` and differs only on a blank cell, which is the whole reason it exists separately: a missing Coverage is a hole in the row, while three thickness readings instead of four is a normal film. `clamped_complement` reproduces the sheet's clamp of uniformity readings above 1 to 0.99, strictly above so an exact 1.0 keeps its own value. `capped_ratio` reproduces the Voc normalisation and adds a ceiling the sheet does not have - a declared divergence, dormant at the observed maximum of 1.135, which the cross-check will announce the day a reading exceeds 1.4. A new `agreement_check` reports rank agreement between a supplied normalised column and the raw one it summarises. It exists because `Normalized photoconductance` has no derivation in the workbook and does not track its own measurement: Spearman -0.5484, p = 0.0343 over the 15 rows, with the strongest film normalising to the lowest value in the column. It is a finding and never a gate - which column the model trains on is the group's decision, and a diagnostic that refused to run would make that decision by refusing. Three constraints, the first this project has carried. `zero_coupled` couples speed_2 and time_2 as an iff, so a one-step film stays reachable while a stage that spins at 0 rpm for 30 s does not. `sum_upper_strict` keeps anti_time strictly below time_1 + time_2. `nonzero_minimum` declares the hole at time_2 = 5: the grid is arithmetic, so reaching 0 with step 5 also reaches a value the first campaign excluded and no film has run. Enforcement stays in the campaign layer. The candidate pool is filtered before any acquisition sees it, and validate_batch re-checks the batch by an independent route - deliberately redundant, because all three of this project's silent-failure bugs were quantities nothing recomputed. The acquisition modules are untouched. discrete_refinement is not constraint-aware and its docstring now says so. Diagnostics carry the pool survival rate, since the sampler draws until it has the requested pool size and a mis-specified constraint would otherwise produce a normal-looking pool drawn from a sliver of the space. Two grid edits, both forced by the measured rows: time_2 reaches 0 so the one-step film is on-grid, and anti_time steps by 1 so sample 1's anti_time = 12 stops being a declared off-grid exception. All 15 rows land on the grid and satisfy all three constraints. The intake verdicts, measured rather than inherited, against a null of -0.1480: uniformity -0.6447 exploration_only optoelectronic -0.5842 exploration_only, mean function DELETED thickness +0.5227 -> +0.6630 learnable, mean function kept The optoelectronic mean function was the first campaign's linear anneal_temp trend, worth -0.342 to +0.355 on its own score. Here it makes the fit worse, -0.5842 to -0.6977: the target was redefined underneath it. Deleting it is the designed outcome of the intake rule, not a setback. The prime suspect for that objective being unlearnable at all is the normalisation above - no model can learn a column that ranks backwards against its own measurement. Thickness keeps its block on weaker grounds than before and the config says so: the swing of +0.1403 is inside the 0.236 resolution floor, so it is inconclusive rather than demonstrated. What changed is that the plain GP now learns thickness on its own, +0.5227 against +0.116 in the first campaign. Also recomputed rather than inherited: the replicate-variance sanity floor is 0.006374 on log(T) over 37 dof, against 0.0593 on the first campaign's films. These films are about ten times more internally consistent, and reusing the old constant would have set the floor an order of magnitude too high. compute_measurements and row_completeness now compare column names stripped, so a caller reading the sheet with pandas directly resolves the two headers that end in a space. It skips the rename entirely when it would merge two distinct labels. 558 passed, 0 failed, 28 warnings - 66 new tests, and the same warning tail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CAMPAIGN_STATUS.md opens with a "Second campaign" section, because everything below it describes the first one and the two contracts are easy to confuse: a table of what changed, the intake verdicts, the two grid edits with the question they leave open for the group, and the three constraints. The intake numbers, measured on the new rows rather than inherited, against a null of -0.1480: uniformity -0.6447 below the null, exploration only optoelectronic -0.5842 / -0.6977 below the null, mean function deleted thickness +0.5227 / +0.6630 learnable, swing inside the floor Two of three axes carry no signal, so a batch is chosen on one informative axis and two uninformative ones. That is a legitimate exploration round and it is not a three-objective optimisation; the docs say so rather than letting the predicted numbers imply otherwise. Issue 10 is the photoconductance normalisation: Spearman -0.5484 at p = 0.0343 against the raw measurement it summarises, with the strongest film carrying the column minimum. It is very likely why optoelectronic is unlearnable at all - no model can learn a column that ranks backwards against its own measurement. Closure path is one recipe edit plus one intake run, and the -0.5484 is pinned by a test so that the test failing is the signal to update the record. GP_MODEL_DECISION.md gains a banner rather than edits. It remains the first campaign's record; what carries over to the second is the METHOD - the three floors, the null, refitting the trend inside every fold, the two degenerate fitting modes - and not any of its LOO numbers, which are about quantities that have since been redefined. HANDOFF.md gains a "where to start today" section, the corrected expected count (558, was 478), and the reason --capture=sys is load-bearing. README updates the live config path, the test count, the design-space size after the two grid edits (396,945,008,460, was 177,816,994,740), and documents constraints as configuration. 558 passed, 0 failed, 28 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…on rank Six figures now land beside the workbook whenever a round is proposed, under <stem>_reports/<round>_<UTC timestamp>/, and a second launcher button renders the four that need no batch. scripts/generate_round_report.py is the same thing from a terminal. The source workbook is still never opened for writing. Every figure writes the CSV behind it, plus a manifest.json carrying the contract version, seed, git describe, reference point, runtime and the active notices. A PNG whose numbers cannot be re-derived is the next plausible-finite-number bug and this project has had three, so two equalities are asserted by tests rather than left to convention: the parity numbers ARE intake's numbers, and figure 03's numbers ARE the Review sheet's. The first of those is by construction rather than by tolerance. The leave-one-out fold loop had quietly become three implementations - the intake script, the round report and the new permutation test - so it moved into mobo_kit.loocv and all three now call it. A test asserts they are the same function object, not that they agree. The SHAP explainer moved to mobo_kit.attribution for the same reason; plot_shap_attribution.py keeps its beeswarms and imports the explainer. Each panel carries its caveats on its face, because a PNG gets pasted into a slide and a caption does not travel with it. The two no-signal axes are labelled NO LEARNABLE SIGNAL inside the axes where cropping cannot remove it - their scatter looks exactly as convincing as thickness's and means nothing. Three notebook conventions are deliberately not ported, and the docs say why: in- sample parity measures memorisation, plot-time sign flips make a figure disagree with the optimiser, and an auto-derived hypervolume reference already produced 6e-8 against 1.448 on this data once. Two figures needed their first draft rejected. Tracing the 3-objective Pareto set across a 2-D projection draws a self-crossing zigzag that is not a front in any space, so each pairwise panel now computes its own front and rings the 3-D members separately. And the reference point at (-0.01, -0.01) with all data above 0.35 spent 40% of every panel on empty space; axes are cropped to the data and the reference is named in text rather than drawn where it reads as an observation. THE THICKNESS MEAN FUNCTION IS SETTLED, and kept. Intake had left it inconclusive on R2 - plain +0.5227 against structured +0.6630, a swing of +0.1403 inside the 0.236 floor - which is a statement that R2 cannot resolve it at N=15 rather than a verdict. scripts/permutation_rank_test.py adjudicates on rank, which is what the acquisition actually consumes: observed rank rho +0.7250 null mean -0.1917 (sd 0.2937) exceedances 4 of 1800 p 0.0028 95% CI [0.0003, 0.0052] Stronger than the first campaign's p = 0.0350 on its own films. Intake now prints the rule in two parts and names the permutation when it lands in the middle one. The permutation script is config-driven and new rather than an edit to thickness_permutation_and_mean.py, which hard-codes the first campaign's columns and is kept as that campaign's reproducible record. beta 4.0 -> 36.0 and radius 0.25 -> 0.35, per the group's reading of the 108-cell sweep with Aleks. Recorded as a declared policy choice about how much to explore rather than a measured optimum: that sweep could not rank cells, and its noiseless same-model-class oracle systematically undervalues exploration. Heavy exploration is the right posture when two of three objectives carry no learnable signal. One measurement in this change was wrong and was caught before it shipped. The fold loop runs single-threaded because tiny matrices do not benefit from intra-op threading, and the first comment claimed 51 s against 117 s - measured while sixteen permutation workers were saturating the CPU. Re-measured idle: 9.8 s at one thread against 15.1 s at the default of 12, bit-identical either way. A timing under contention is an unreproduced number like any other. A report failure never costs a batch: the worklist and Review sheet are written first, and Generated.report_error says what happened. Inside the report, one failed figure is recorded in the manifest and the rest still render. Full run on the live workbook: six figures, 15 CSV and PNG files, 14.7 s. 575 passed, 0 failed, 28 warnings - the warning tail is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Double-clicking launch_mobo_kit.bat and browsing to the new workbook failed with
Sheet1 is missing required column(s): ['PL - Implied Voc (Max)']
which is a v2 column name. The workbook is intact; the launcher was reading it
against the ARCHIVED first-campaign config. DEFAULT_CONFIG still pointed there
after campaign_d2d_perovskite.yaml was archived - the intake script's default was
moved at the time and this one was not.
The launcher is the one path an experimentalist reaches without writing code, so
its default is a product decision rather than a constant, and it is now pinned by
a test that also asserts the config it names is active.
The error message was the second half of the problem. "Sheet1 is missing required
column(s)" reads as a broken workbook, and the realistic cause is the opposite: an
intact workbook read against another campaign's contract. It now names the
configuration, its status and its contract version, says plainly when that
configuration is archived, and lists the headers that look like the same
measurement under a different name. On this case it offers
'PL - Implied Voc (Max) Raw', which is the answer.
The near-miss search is prefix and containment rather than edit distance, because
the failure being diagnosed is a renamed column between campaigns, not a typo.
Also: a column-level finding rendered as "sample ?", which reads as a row whose
identity was lost rather than as a finding that has no row. The rank-agreement
warning is about a column across all fifteen rows and now says "all rows".
Verified end to end on the live workbook: the launcher reports "Ready to propose
R1" against Summary Table Test.xlsx.
578 passed, 0 failed, 28 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e old one Both simulation instruments now run on the v3 contract at radius 0.35 / beta 36 without sweeping, because the cell is a decision and running 108 of them to look at one is 36x the compute for the same answer: scripts/plot_round_simulation.py --cell 0.35,36 (heatmaps, HV, boxplots) scripts/plot_boxplot_sweep.py --betas 36 --radii 0.35 (three starting designs) Both default to the active config. --betas/--radii filter the knobs and never the trials: the trials are what turn three numbers per round into a distribution worth boxing, so a one-cell run is still three campaigns from three starting designs. Measured on the new data at seed 73: HV 0.7929 -> 0.7929 -> 0.8053. R1 adds nothing and R2 adds +0.0124. That is what beta = 36 looks like against a noiseless oracle of the same model class the optimiser fits: the batch spends its budget on exploration the instrument cannot repay. It understates the case for the policy it is testing, and the docs say so rather than letting the number imply otherwise. THE CROSS-INSTRUMENT IDENTITY HOLDS. The simulated R1 hashes to 60d1682aa055ca97, the same as the live run_r1_ucb proposal from the measured rows, so the simulation describes the batch the campaign would actually ship rather than a similar one. batch_hash moved into candidate_diagnostics for this, shared with the report's new worklist drift check. Two things the instruments were reporting wrongly, both of the same kind - a fact about the FIRST campaign presented as a fact about this one: The boxplot footer carried "uniformity carries no validated signal (permutation p = 0.82)". That is the first campaign's uniformity score on the first campaign's films. On v3 that objective is a different construction and optoelectronic is dead as well, so the constant put the wrong evidence under the right warning - worse than no caveat, because it looks checked. It is now read from the config's signal_status and names both axes with this contract's version. The pre-registered expectations reported FAILED twice on a single cell, once for an empty radius arm and once for an empty beta arm. A rule with no data now reports NOT APPLICABLE; two red lines under a run that did exactly what was asked are worse than silence. And the no-signal rule, written when uniformity was the only dead axis, said "uniformity moves least of the three" - which FAILED for the entirely correct reason that the two dead axes move by similar small amounts. It now reads the dead set from the config: both no-signal axes must move less than every learnable one. HELD, +0.1047 and +0.0842 against thickness's +0.3967. Review items folded in. Figure 04's footer and the manifest printed "np.float64(-0.01)" for the reference point, which tells an experimentalist about numpy rather than about the campaign. The --shap-instances help claimed to be the runtime knob; the leave-one-out refits are about two thirds of the cost and are not optional. And the report's default mode now compares its re-derived proposal against the worklist on disk by batch hash, printing MATCH or DRIFT - if they differ, the figures describe a different batch than the one on the bench. CAMPAIGN_STATUS gains "are beta = 36 and radius = 0.35 defensible?": the sweep's inability to rank (0.0065 against a trial sd of 0.010-0.027), the posture rationale, the two measured consequences - penalization inert at spacing 1.091, and 21 of 50 coordinates at a range edge - and the two revisit triggers. 580 passed, 0 failed, 28 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…is which The repository holds two campaigns and the documentation did not consistently say so. The first one was ALGORITHM TESTING - its 15 rows and its d2d-objectives-v2-nm-thickness contract existed to prove the loop worked, not to run an experiment. The second, on Summary Table Test.xlsx, is the real campaign. Two of the three objectives are computed differently there, so none of the first campaign's fitted numbers transfer; they describe quantities that were redefined. Six documents carried first-campaign numbers with nothing saying so, which is the failure this pass exists to catch: a fact about one campaign presented as a fact about the other looks checked and is not. R1_BATCH_WITHDRAWAL, ROUND_SIM_DELTA, ROUND_SIM_MANIFEST, SHAP_SUMMARY and the two figure READMEs now open with a banner naming the campaign they describe and pointing at the live one. GP_MODEL_DECISION's banner gains the same framing. CAMPAIGN_STATUS gains a divider - everything below it is the first campaign - and its numbered issues say that 1-9 are that campaign's while 10 is live and open. HANDOFF is rewritten for a cold reader in the post-push state rather than the July 30 world: the two campaigns and which config is active, the reading order, the expected test count, the four instruments with one command each, where the live campaign stands with its intake verdicts and the permutation result, what is open, the three floors as method that crosses both campaigns, the settled decisions, the recurring failure shape, and the tooling facts that bite. README drops the Step 1 / 2A / 2B / 2C history section, which described an apparatus removed a year of sessions ago and helps nobody now. It gains a two-campaign table at the top and a section stating how beta and radius were chosen: a sweep of box plots and heat maps - 2-D slices through the higher-dimensional Gaussian-process model - across beta from 9 to 49 and radius from 0.05 to 0.45, with local penalization inert at the current beta. The sweep outputs stay local; they are how the group picks a setting, not a result about the chemistry. Two instrument defects of the same class, both fixed: The round simulation's legend read "R0 LHS (GP_exp scored)". R0 in this script is the REAL measured recipes re-scored by the oracle; the LHS label came from the branch this script was adapted from, where R0 was a fresh draw. A reader has no way to tell those apart from the figure, so the legend now says which. The manifest recorded only the oracle-scored R0 hypervolume. Both belong there: hv_r0 is the baseline the simulation used, hv_r0_measured is what the campaign starts from, and anyone comparing a simulated trajectory against a live round needs to know which one they hold. .gitignore gains *_reports/ so the launcher's report directories are ignored wherever a user keeps their workbook, not only under local_inputs/. 580 passed, 0 failed, 28 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The final workbook arrived. Its columns moved again - the three scores are now AJ / AK / AL and the thickness readings AC..AG - and its sheets are named by round, so the measured rows live on R0 rather than Sheet1. THE SCORES ARE FROZEN, per the group. Uniformity and optoelectronic are read from the workbook as stored, with no recomputation in Python, via a new `stored` recipe. That is a deliberate reversal of this project's usual polarity, which is "Python computes and the stored cell is demoted to a cross-check". The reason is that both objectives have now been renormalised twice and the group is still revising them. Reimplementing a formula that is about to change means the code and the sheet disagree at exactly the moment someone edits the sheet, and the disagreement looks like a bug in whichever was checked second. What freezing COSTS is the part worth writing down, because it removes the cross-check this project otherwise insists on: there is no independent recomputation of those two objectives, so a stale pasted literal in AJ or AK cannot be caught by comparing it against anything. Intake and every round report now print that in one line rather than leaving it in a config comment. What partly replaces it is `formula_fingerprint`. The config records each frozen column's formula TEXT and the read compares it - reading the formula, never evaluating it - so a changed DEFINITION is audible even though a changed value is not. Comparison is row- and whitespace-independent, so one fingerprint covers all fifteen rows instead of reporting fifteen changes for one edit. A test asserts what it cannot catch, so nobody later mistakes it for a value check. Recorded: AJ =(L2+O2+P2)/3 AK =(S2+((0.75*Y2)+(0.25*AB2)))/2 Thickness is NOT frozen. Its definition has been stable across all three contracts and the recomputation is what lets an operator-flagged reading be excluded from the mean and reported rather than silently dropped. The v3 recipes stay in scores.py, unwired and still tested, so unfreezing is a config edit rather than a rebuild. `campaign.source_sheet` makes the sheet configuration; older contracts declare nothing and default to Sheet1. The workbook's own R1 sheet is deliberately not read - the round contract is unchanged, each round's worklist goes to a new file beside the workbook and the source is never opened for writing. Two defects the new contract exposed, both fixed: The agreement check listed only its `raw` column as optional, on the assumption that `normalized` was a recipe input. True while optoelectronic was computed; false the moment it was frozen and its only input became the score column. The check then reported "column absent" on a sheet that had it. `read_campaign_workbook` hard-coded Sheet1 in four places including the read-only digest, so the invariant that the source workbook is never modified would have been checked against the wrong sheet. INTAKE ON THE FINAL WORKBOOK, null -0.1480, floor 0.236: uniformity -0.4778 exploration only optoelectronic -0.7038 exploration only thickness +0.5814 -> +0.7422 learnable, swing inside the floor Thickness keeps its mean function on the rank permutation, measured fresh rather than inherited: observed rho +0.6500, null mean -0.1892 (sd 0.2944), 9 exceedances of 1800, p = 0.0056, 95% CI [0.0021, 0.0090]. BETA STAYS AT 36, and the rationale is re-earned rather than assumed. The recorded trigger to revisit it was "when the photoconductance normalisation is fixed and optoelectronic may become learnable". It fired - the inversion is fixed, Spearman +1.0000 against -0.5484 on v3, closing issue 10 - but optoelectronic did not become learnable. It is further below the null than before. Still one informative axis, so heavy exploration is still the right posture. Had two or more axes become learnable the recommendation would have been to return toward beta = 4. All 15 rows are on-grid and satisfy all three constraints; sample 2's `speed_2 = 0, time_2 = 60`, which broke the first rule, is corrected to `time_2 = 0`. Docs name three contracts now, because two was already ambiguous: v2 algorithm testing, v3 the dry run on a workbook literally called Test, v4 the campaign that produces films. v3's config is archived and its documentation banded like v2's. 601 passed, 0 failed, 28 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ey were The within-film thickness floor was still v3's constant. Recomputed on the final workbook: 0.003007 on log(T) over 36 dof, sd 0.0548, against v3's 0.006374. Half, and the reason is visible in the sheet - sample 1's 418.5 reading moved into `T anom`, which took that film's sd_log from 0.218 to 0.085. It is a FLOOR on between-film variance, so inheriting the old constant would have set it twice too high and called an ordinary pooled variance a pooling mistake. Naming, per Colin: the earlier two contracts were both TEST DATA and only v4 is the real campaign. The docs had been drawing a distinction between "algorithm testing" and "dry run" that is finer than the group needs and that invites the reading that v3 was somehow half-real. Every table and every historical banner now says test data, and points at v4 as the campaign that produces films. plot_shap_attribution.py still defaulted to the v2 config and workbook. It is a live instrument - its explainer was promoted into the package - so it now defaults to v4 like the rest. gp_diagnostic.py and validate_structured_means.py keep their v2 defaults deliberately: they hard-code that workbook's columns and are its reproducible record. Box plots and heat maps regenerated on the final data at the ratified cell, and they read correctly under v4: the footer names d2d-objectives-v4-final and the two dead axes it actually has, and the round-simulation legend says "R0 measured recipes (oracle-scored)" rather than the LHS label it inherited. Outputs stay local. The simulated loop on the final data, seed 73: HV 0.6322 -> 0.6322 -> 0.6443, R1 spacing 1.213. R1 adds nothing on a noiseless same-class oracle and R2 adds +0.0121, which is what beta = 36 looks like against a landscape with no surprises in it - the instrument understates the case for the posture it is testing. The pre-registered expectation holds: the two no-signal axes move +0.0261 and +0.1160 against thickness's +0.3527. 601 passed, 0 failed, 28 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request establishes the tested D2D MOBO workflow through Step 2C:
Why
Step 2B showed that candidate selection is sensitive to control inclusion, boundary behavior, and GP assumptions. Step 2C adds the validation and convergence evidence needed to assess those sensitivities reproducibly before any experimental batch is considered.
Safety and privacy
Validation
mainand zero commits behindReviewer focus
Please review the public/private configuration boundary, objective and reference-point contracts, GP validation gates, robust-region consensus criteria, and whether the current debug-only evidence is sufficient to proceed to a later experimental approval step.