Add LLMRecommender for LLM-based experimental design

[tobiasploetz]

Add LLMRecommender for LLM-based experimental design

Summary

Adds an LLMRecommender that queries a large language model (via LiteLLM) to propose experiments. It is primarily intended for warm-starting a campaign: before enough data exists to train a surrogate, the model can use parameter semantics and domain/literature priors to suggest more sensible points than random sampling. It slots into the recommender hierarchy like any other pure recommender and is typically combined with a Bayesian recommender via a TwoPhaseMetaRecommender.

Ported from #561 and substantially reworked for the current codebase.

What it does

• Builds a natural-language prompt from the search space (parameter names, types, ranges/allowed values, and any metadata descriptions/units), the textual experiment_description / objective_description, the structured objective, and any collected measurements / pending_experiments.
• Sends the prompt to the configured model through LiteLLM, parses the returned JSON into candidate experiments, and validates them against the search space.
• Recovers from malformed responses via a one-shot correction prompt, and tolerates models that wrap JSON in Markdown fences or surrounding prose.

Key design decisions

• Inherits directly from PureRecommender. It legitimately consumes measurements/pending_experiments for the prompt but is neither non-predictive nor Bayesian. recommend() is fully overridden and builds the result directly from the model response, so the recommend* hooks are intentionally not implemented.
• Credentials are not a field. LiteLLM reads provider keys from environment variables (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY) selected by the model prefix — so there is deliberately no API-key attribute.
• Scope kept tight. There were some fields that were not really necessary for the first version like related_data or a feasibility check. These have been removed.
• Module layout. Prompt construction (_prompts.py) and response parsing/validation (_parsing.py) live in dedicated submodules; llm.py holds the recommender class. litellm/jinja2 are lazy-imported so import baybe stays light.

Testing

• Unit tests are fully mocked — no real LLM calls in CI (@pytest.mark.skipif(not LLM_INSTALLED)); they skip cleanly under coretest.
• Covers success paths, prompt inclusion of measurements/pending/objective, the full set of parse-error branches, code-fence/prose stripping, error wrapping, and a serialization roundtrip.
• LLMRecommender is excluded from the automatic get_subclasses test enumerations (it has required constructor args) via an explicit, documented helper in conftest.py.

Docs & example

• New "LLM Recommenders" section in docs/components/recommenders.md.
• Runnable example (examples/LLM/llm_recommender.py) comparing an LLM warm-start (then Bayesian) against BayBE's default recommender and a random baseline on the direct-arylation dataset; it mocks the LLM when no credentials are present, and the committed comparison images are pre-generated (the docs build is mocked).

[tobiasploetz]

Converting discussion comments into review threads for easier inline discussion.

[AVHopp]

@Scienfitz @AdrianSosic @kalama-ai I will do the first round of reviews/feedback, feel free to ignore until we move it out of draft :)

[AVHopp]

First round of comments. I hope that impementing those will solve the seralization issues as well as the most important design choice such that we can then see the CI in action. Note that I did not yet investigate the tests or the inner workings of the class in detail, but I think that doing small iterations like this is better than me providing too many comments :)

[Copilot]

Pull request overview

This PR introduces an LLMRecommender to BayBE’s pure recommender stack, enabling LLM-driven experiment suggestions (via LiteLLM) for warm-starting campaigns before surrogate-based Bayesian optimization becomes effective.

Changes:

• Add LLMRecommender implementation with prompt construction and robust JSON response parsing (including one-shot recovery).
• Add optional dependency group llm (Jinja2 + LiteLLM) plus optional-import plumbing and mypy config.
• Add tests, documentation, and an end-to-end example (with pre-generated comparison plots) and adjust iteration tests to avoid auto-instantiating non-default-constructible recommenders.

Reviewed changes

Copilot reviewed 19 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
baybe/recommenders/pure/llm/llm.py	Implements LLMRecommender with LiteLLM call, recovery flow, and prompt/parse integration.
baybe/recommenders/pure/llm/_prompts.py	Builds Jinja2 prompts from search space + descriptions + optional measurements/objective.
baybe/recommenders/pure/llm/_parsing.py	Extracts/validates JSON suggestions into a DataFrame.
baybe/recommenders/pure/llm/__init__.py	Exposes LLMRecommender from the llm subpackage.
baybe/recommenders/pure/__init__.py	Re-exports LLMRecommender in pure recommenders.
baybe/recommenders/__init__.py	Re-exports LLMRecommender at the recommender package level.
baybe/_optional/llm.py	Adds optional imports for Jinja2 Template + LiteLLM completion.
baybe/_optional/info.py	Adds LLM_INSTALLED check based on Jinja2 + LiteLLM presence.
baybe/exceptions.py	Adds LLMResponseError and LLMResponseWarning.
pyproject.toml	Adds baybe[llm] extra with jinja2 + litellm.
mypy.ini	Adds mypy ignore section for litellm.*.
tests/test_llm_recommender.py	Adds mocked unit tests for prompting, parsing branches, recovery, and warnings.
tests/serialization/test_llm_recommender_serialization.py	Adds serialization roundtrip test via meta recommender wrapper.
tests/conftest.py	Adds helper to exclude non-default-constructible recommenders from subclass enumeration tests.
tests/test_iterations.py	Switches subclass enumeration to the new default-constructible helper.
docs/components/recommenders.md	Documents the new LLM recommender and the baybe[llm] extra.
examples/LLM/llm_recommender.py	Adds runnable warm-start example with mock fallback for CI/offline runs.
examples/LLM/LLM_Header.md	Adds header page for the LLM examples section.
examples/LLM/llm_recommender_light.svg	Adds pre-generated plot asset (light theme).
examples/LLM/llm_recommender_dark.svg	Adds pre-generated plot asset (dark theme).
CHANGELOG.md	Adds changelog entry for LLMRecommender.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Scienfitz]

currently does not use all available metadata

[Scienfitz]

any particular reason why this si going via TwoPhaseMetaRecommender and not hjust smply via LLMRecommender?

normally Id expect a hypothesis strat and a serialziaiton test absed on that, why was it don different here?

[AVHopp]

Yes - none of our recommenders can be serialized directly as none of them uses the SerialMixin. Not sure if intentional, and until this has been made clear, I'll just wrap it inside the meta recommender, but already added a hypothesis strategy for the LLMRecommender.

[Scienfitz]

hmmm im clearly missing something here but if the pure reocmmenders cant be serialzied how can the meta-recommender usign them be serialized?

the fact that pure recommenders cannot be serialized is proably

• a bug or forgotten thing
• on purpose due to something only @AdrianSosic remembers