Optionally, write an IIS when an xhatter hits an infeasible scenario (#356)#754
Open
DLWoodruff wants to merge 6 commits into
Open
Optionally, write an IIS when an xhatter hits an infeasible scenario (#356)#754DLWoodruff wants to merge 6 commits into
DLWoodruff wants to merge 6 commits into
Conversation
…yomo#356) Adds doc/designs/iis_on_xhat_infeasible_design.md proposing an opt-in --xhatter-write-iis option that, the first time an incumbent-finder rejects a candidate because a scenario subproblem is infeasible, writes an IIS via pyomo.contrib.iis to diagnose models that should have complete recourse but don't. Locked decisions captured in the doc: flag --xhatter-write-iis; method default auto (write_iis .ilp for commercial solvers, else compute_infeasibility_explanation); emit for exactly one infeasible scenario per cylinder (per MPI rank); output naming mirrors the --solver-log-dir convention; dedicated doc/src/iis.rst with the option help text referencing it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When an xhatter (incumbent-finder) rejects a candidate because a scenario subproblem is infeasible, optionally write an IIS for the offending subproblem via pyomo.contrib.iis, to diagnose models that should have (relatively) complete recourse but don't. - SPOpt.write_iis_on_xhatter_infeasible(): fires AT MOST ONCE per cylinder (per MPI rank) via a guard set before the work, so the expensive IIS computation cannot repeat; fail-soft (never perturbs the run). Targets the first infeasible local scenario, or an explicit model (used for the multistage stage2-EF path). _emit_iis dispatches on --xhatter-iis-method: 'ilp' (write_iis, .ilp, commercial persistent solver), 'explanation' (compute_infeasibility_explanation, any solver), 'auto' (ilp for cplex/gurobi/xpress, else explanation). The ilp path falls back to letting write_iis auto-pick a persistent solver when the configured one's persistent interface is absent. - Call sites: XhatBase._try_one (two-stage/multistage + stage2-EF), _PreLoopXhatMixin._evaluate_xhat, Xhat_Eval.calculate_incumbent -- each before any _restore_nonants so the model still carries the infeasible configuration the IIS re-solve needs. - Output naming mirrors the --solver-log-dir convention via a shared _subproblem_file_stem helper: <cylinder>_<scenario>.ilp / .iis.log, under --xhatter-iis-dir (default cwd). - Config: --xhatter-write-iis / --xhatter-iis-method / --xhatter-iis-dir in popular_args, forwarded in cfg_vanilla.shared_options. - Tests: test_iis_on_infeasible.py (config registration, forwarding, control-flow/guard/fail-soft via a stub, helpers, and solver-gated end-to-end ilp + explanation); wired into run_coverage.bash and test_pr_and_main.yml. Updated the _evaluate_xhat stubs in test_feasible_xhat.py / test_jensens.py for the new opt method. - Docs: dedicated doc/src/iis.rst (run-once semantics prominent), added to index, cross-referenced from generic_cylinders.rst and xhat_from_file.rst. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an incomplete-recourse two-stage model (x >= dmin per scenario, no recourse var to repair it) and drives the production path end to end: fix the nonant at an xhat that is optimal for scen0 but infeasible for scen1, run the real Xhat_Eval.calculate_incumbent (the method slam_heuristic / lshaped spokes use), and assert the IIS file actually lands on disk. Covers: the real solve_loop -> solve_one (solution_available=False on the genuinely infeasible scenario) -> no_incumbent_prob -> the new call site -> write_iis / compute_infeasibility_explanation chain; correct offending- scenario selection (Xhat_Eval_scen1.*); the run-once guard across repeated infeasibilities; and feasible-xhat-writes-nothing. Solver-gated (ilp also gated on a persistent commercial interface). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #754 +/- ##
==========================================
+ Coverage 72.98% 73.05% +0.07%
==========================================
Files 165 165
Lines 21004 21071 +67
==========================================
+ Hits 15329 15394 +65
- Misses 5675 5677 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The IIS-on-infeasible tests were wired only into the no-solver unit-tests job, where the solver-gated end-to-end classes are always skipped. That left both _emit_iis branch bodies (ilp / explanation) uncovered, which is what codecov/patch flagged, and meant the real pyomo.contrib.iis code never ran in CI at all. - Add TestEmitBranchesMocked: exercises both branch bodies (file naming, IIS-solver derivation, logger routing/teardown) with only the external write_iis / compute_infeasibility_explanation call mocked, so they run in the no-solver job and close the coverage gap. - Also run test_iis_on_infeasible.py in the ph-extensions job (cplex + xpress present) so TestEndToEnd / TestRealXhatterPath exercise the real write_iis (.ilp) and compute_infeasibility_explanation (.iis.log) paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "PH extension tests" job runs test_iis_on_infeasible.py via `-m pytest` (to exercise the real pyomo.contrib.iis paths where a commercial solver is present), but the job installed only pyomo/xpress/cplex/coverage. pytest was missing, so the step failed with "No module named 'pytest'" after the test_ph_extensions.py tests had already passed. Add pytest to the install step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
write_iis returns the path the solver writer actually produced; xpress (>=38) appends ".lp", yielding "<stem>.ilp.lp", so the requested name is not always the file on disk. Report the returned path in _emit_iis so the message is truthful, and make the solver-gated .ilp existence tests match "<stem>.ilp*" instead of an exact ".ilp" (fixes the PH-extension CI job, which runs xpress). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #356.
Motivation
An xhatter (incumbent-finder) fixes the first-stage (non-anticipative)
variables at a candidate
xhatand solves every scenario subproblem. If ascenario is infeasible at that
xhat, the candidate is silently rejected andthe search moves on. That silence is right almost always — but it is the wrong
behavior when a model is supposed to have (relatively) complete recourse and,
because of a modeling subtlety or data issue, doesn't. The run can then spin a
long time finding no incumbent, with nothing to explain why.
As requested in the issue, this adds an opt-in option that, the first time an
xhatter rejects a candidate due to scenario infeasibility, writes an IIS
(irreducible infeasible set) for the offending subproblem via
pyomo.contrib.iis— "with these first-stage decisions, scenario X isinfeasible because of this handful of constraints."
What it does
A new
SPOpt.write_iis_on_xhatter_infeasible()is called from the xhatinfeasibility-detection sites (
XhatBase._try_onetwo-stage + stage2-EFbranches,
_PreLoopXhatMixin._evaluate_xhat,Xhat_Eval.calculate_incumbent),each before any
_restore_nonantsso the model still carries the infeasibleconfiguration the IIS re-solve needs. Calling it only from xhat paths scopes
the feature to incumbent-finders by construction; the PH/APH main loop (where
infeasibility is routine) is untouched.
It fires at most once per cylinder (per MPI rank). A guard, set before the
work, ensures the expensive IIS computation cannot repeat — and a failed or
expensive attempt is never retried. The emission is fail-soft: any error is
reported and the run continues. In a parallel run you may get up to one IIS
file per rank that hit an infeasible local scenario, each named by its
scenario (so concurrent writers never collide). Output naming mirrors the
--solver-log-dirconvention via a shared_subproblem_file_stemhelper:<cylinder>_<scenario>.ilp/.iis.log.Options (in
popular_args, forwarded incfg_vanilla.shared_options)--xhatter-write-iis--xhatter-iis-methodautoilp(write_iis, .ilp, commercial persistent solver) /explanation(compute_infeasibility_explanation, any solver) /auto(ilp for cplex/gurobi/xpress, else explanation)--xhatter-iis-dirThe
ilppath falls back to lettingwrite_iisauto-pick a persistent solverwhen the configured one's persistent interface isn't installed.
Tests
New
mpisppy/tests/test_iis_on_infeasible.py(wired intorun_coverage.bashand
test_pr_and_main.yml):cfg_vanillaforwarding;selection, fail-soft) — no solver needed;
autoresolution, suffix stripping, file stem);Xhat_Eval.calculate_incumbentpath on an incomplete-recourse model: an xhat optimal for one scenario is
infeasible for another, and the test asserts the IIS file actually lands,
the run-once guard holds across repeated infeasibilities, and a feasible
xhat writes nothing. (Solver-gated; the
.ilpcase also gated on apersistent commercial interface.)
Docs
Dedicated
doc/src/iis.rst(run-once semantics front and center), added to theindex and cross-referenced from
generic_cylinders.rstandxhat_from_file.rst. Design doc atdoc/designs/iis_on_xhat_infeasible_design.md.🤖 Generated with Claude Code