Skip to content

Optionally, write an IIS when an xhatter hits an infeasible scenario (#356)#754

Open
DLWoodruff wants to merge 6 commits into
Pyomo:mainfrom
DLWoodruff:iis-on-xhatter-infeasible
Open

Optionally, write an IIS when an xhatter hits an infeasible scenario (#356)#754
DLWoodruff wants to merge 6 commits into
Pyomo:mainfrom
DLWoodruff:iis-on-xhatter-infeasible

Conversation

@DLWoodruff

Copy link
Copy Markdown
Collaborator

Resolves #356.

Motivation

An xhatter (incumbent-finder) fixes the first-stage (non-anticipative)
variables at a candidate xhat and solves every scenario subproblem. If a
scenario is infeasible at that xhat, the candidate is silently rejected and
the search moves on. That silence is right almost always — but it is the wrong
behavior when a model is supposed to have (relatively) complete recourse and,
because of a modeling subtlety or data issue, doesn't. The run can then spin a
long time finding no incumbent, with nothing to explain why.

As requested in the issue, this adds an opt-in option that, the first time an
xhatter rejects a candidate due to scenario infeasibility, writes an IIS
(irreducible infeasible set) for the offending subproblem via
pyomo.contrib.iis"with these first-stage decisions, scenario X is
infeasible because of this handful of constraints."

What it does

A new SPOpt.write_iis_on_xhatter_infeasible() is called from the xhat
infeasibility-detection sites (XhatBase._try_one two-stage + stage2-EF
branches, _PreLoopXhatMixin._evaluate_xhat, Xhat_Eval.calculate_incumbent),
each before any _restore_nonants so the model still carries the infeasible
configuration the IIS re-solve needs. Calling it only from xhat paths scopes
the feature to incumbent-finders by construction; the PH/APH main loop (where
infeasibility is routine) is untouched.

It fires at most once per cylinder (per MPI rank). A guard, set before the
work, ensures the expensive IIS computation cannot repeat — and a failed or
expensive attempt is never retried. The emission is fail-soft: any error is
reported and the run continues. In a parallel run you may get up to one IIS
file per rank that hit an infeasible local scenario, each named by its
scenario (so concurrent writers never collide). Output naming mirrors the
--solver-log-dir convention via a shared _subproblem_file_stem helper:
<cylinder>_<scenario>.ilp / .iis.log.

Options (in popular_args, forwarded in cfg_vanilla.shared_options)

Flag Default Meaning
--xhatter-write-iis off enable the feature
--xhatter-iis-method auto ilp (write_iis, .ilp, commercial persistent solver) / explanation (compute_infeasibility_explanation, any solver) / auto (ilp for cplex/gurobi/xpress, else explanation)
--xhatter-iis-dir cwd output directory

The ilp path falls back to letting write_iis auto-pick a persistent solver
when the configured one's persistent interface isn't installed.

Tests

New mpisppy/tests/test_iis_on_infeasible.py (wired into run_coverage.bash
and test_pr_and_main.yml):

  • config registration + cfg_vanilla forwarding;
  • control flow against a stub (option gating, run-once guard, target
    selection, fail-soft) — no solver needed;
  • helper logic (auto resolution, suffix stripping, file stem);
  • synthetic end-to-end through the real Xhat_Eval.calculate_incumbent
    path on an incomplete-recourse model: an xhat optimal for one scenario is
    infeasible for another, and the test asserts the IIS file actually lands,
    the run-once guard holds across repeated infeasibilities, and a feasible
    xhat writes nothing. (Solver-gated; the .ilp case also gated on a
    persistent commercial interface.)

Docs

Dedicated doc/src/iis.rst (run-once semantics front and center), added to the
index and cross-referenced from generic_cylinders.rst and
xhat_from_file.rst. Design doc at doc/designs/iis_on_xhat_infeasible_design.md.

🤖 Generated with Claude Code

DLWoodruff and others added 3 commits June 14, 2026 08:12
…yomo#356)

Adds doc/designs/iis_on_xhat_infeasible_design.md proposing an opt-in
--xhatter-write-iis option that, the first time an incumbent-finder
rejects a candidate because a scenario subproblem is infeasible, writes
an IIS via pyomo.contrib.iis to diagnose models that should have complete
recourse but don't.

Locked decisions captured in the doc: flag --xhatter-write-iis; method
default auto (write_iis .ilp for commercial solvers, else
compute_infeasibility_explanation); emit for exactly one infeasible
scenario per cylinder (per MPI rank); output naming mirrors the
--solver-log-dir convention; dedicated doc/src/iis.rst with the option
help text referencing it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When an xhatter (incumbent-finder) rejects a candidate because a
scenario subproblem is infeasible, optionally write an IIS for the
offending subproblem via pyomo.contrib.iis, to diagnose models that
should have (relatively) complete recourse but don't.

- SPOpt.write_iis_on_xhatter_infeasible(): fires AT MOST ONCE per
  cylinder (per MPI rank) via a guard set before the work, so the
  expensive IIS computation cannot repeat; fail-soft (never perturbs the
  run). Targets the first infeasible local scenario, or an explicit model
  (used for the multistage stage2-EF path). _emit_iis dispatches on
  --xhatter-iis-method: 'ilp' (write_iis, .ilp, commercial persistent
  solver), 'explanation' (compute_infeasibility_explanation, any solver),
  'auto' (ilp for cplex/gurobi/xpress, else explanation). The ilp path
  falls back to letting write_iis auto-pick a persistent solver when the
  configured one's persistent interface is absent.
- Call sites: XhatBase._try_one (two-stage/multistage + stage2-EF),
  _PreLoopXhatMixin._evaluate_xhat, Xhat_Eval.calculate_incumbent --
  each before any _restore_nonants so the model still carries the
  infeasible configuration the IIS re-solve needs.
- Output naming mirrors the --solver-log-dir convention via a shared
  _subproblem_file_stem helper: <cylinder>_<scenario>.ilp / .iis.log,
  under --xhatter-iis-dir (default cwd).
- Config: --xhatter-write-iis / --xhatter-iis-method / --xhatter-iis-dir
  in popular_args, forwarded in cfg_vanilla.shared_options.
- Tests: test_iis_on_infeasible.py (config registration, forwarding,
  control-flow/guard/fail-soft via a stub, helpers, and solver-gated
  end-to-end ilp + explanation); wired into run_coverage.bash and
  test_pr_and_main.yml. Updated the _evaluate_xhat stubs in
  test_feasible_xhat.py / test_jensens.py for the new opt method.
- Docs: dedicated doc/src/iis.rst (run-once semantics prominent), added
  to index, cross-referenced from generic_cylinders.rst and
  xhat_from_file.rst.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an incomplete-recourse two-stage model (x >= dmin per scenario, no
recourse var to repair it) and drives the production path end to end:
fix the nonant at an xhat that is optimal for scen0 but infeasible for
scen1, run the real Xhat_Eval.calculate_incumbent (the method
slam_heuristic / lshaped spokes use), and assert the IIS file actually
lands on disk.

Covers: the real solve_loop -> solve_one (solution_available=False on the
genuinely infeasible scenario) -> no_incumbent_prob -> the new call site
-> write_iis / compute_infeasibility_explanation chain; correct offending-
scenario selection (Xhat_Eval_scen1.*); the run-once guard across repeated
infeasibilities; and feasible-xhat-writes-nothing. Solver-gated (ilp also
gated on a persistent commercial interface).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@DLWoodruff DLWoodruff changed the title Write an IIS when an xhatter hits an infeasible scenario (#356) Optionally, write an IIS when an xhatter hits an infeasible scenario (#356) Jun 14, 2026
@codecov

codecov Bot commented Jun 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.52941% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.05%. Comparing base (a01c36c) to head (ae8f999).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
mpisppy/extensions/xhatbase.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #754      +/-   ##
==========================================
+ Coverage   72.98%   73.05%   +0.07%     
==========================================
  Files         165      165              
  Lines       21004    21071      +67     
==========================================
+ Hits        15329    15394      +65     
- Misses       5675     5677       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

DLWoodruff and others added 3 commits June 14, 2026 09:53
The IIS-on-infeasible tests were wired only into the no-solver unit-tests
job, where the solver-gated end-to-end classes are always skipped. That
left both _emit_iis branch bodies (ilp / explanation) uncovered, which is
what codecov/patch flagged, and meant the real pyomo.contrib.iis code
never ran in CI at all.

- Add TestEmitBranchesMocked: exercises both branch bodies (file naming,
  IIS-solver derivation, logger routing/teardown) with only the external
  write_iis / compute_infeasibility_explanation call mocked, so they run
  in the no-solver job and close the coverage gap.
- Also run test_iis_on_infeasible.py in the ph-extensions job (cplex +
  xpress present) so TestEndToEnd / TestRealXhatterPath exercise the real
  write_iis (.ilp) and compute_infeasibility_explanation (.iis.log) paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "PH extension tests" job runs test_iis_on_infeasible.py via `-m pytest`
(to exercise the real pyomo.contrib.iis paths where a commercial solver is
present), but the job installed only pyomo/xpress/cplex/coverage. pytest was
missing, so the step failed with "No module named 'pytest'" after the
test_ph_extensions.py tests had already passed. Add pytest to the install step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
write_iis returns the path the solver writer actually produced; xpress
(>=38) appends ".lp", yielding "<stem>.ilp.lp", so the requested name is
not always the file on disk. Report the returned path in _emit_iis so the
message is truthful, and make the solver-gated .ilp existence tests match
"<stem>.ilp*" instead of an exact ".ilp" (fixes the PH-extension CI job,
which runs xpress).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add IIS emission to incumbent-finders

1 participant