Narwhalify recommendation layer

[AdrianSosic]

Narwhalifies the recommendation layer, focusing on the core logic. That is, some methods/functions are narwhalified only at their API boundary, keeping an internal conversion to pd.DataFrame. These internals can be narhwhalified at any later point in time in the form of isolated PRs.

[Copilot]

Pull request overview

This PR narwhalifies the recommendation layer by widening recommender APIs to accept Narwhals-compatible inputs (IntoDataFrame*) while keeping key internals (notably BoTorch-facing paths) operating on pd.DataFrame. It also removes the expectation that pandas Series/DataFrame indices carry semantic meaning, updating code and tests accordingly.

Changes:

• Update recommender protocols/base classes and implementations to accept/return Narwhals IntoDataFrame* types, with explicit boundary conversions where needed.
• Add backend inference/conversion helpers and use Settings(default_dataframe_backend=...) scoping to keep constructed outputs consistent with the inferred backend.
• Remove index-preservation behavior in transforms and update tests, examples, and changelog to reflect that indices are no longer guaranteed to be preserved.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_targets.py	Removes index-preservation assertion for NumericalTarget.transform.
tests/constraints/test_cardinality_constraint_continuous.py	Narwhals-ifies validation helper to handle IntoDataFrame and avoids pandas-only ops.
examples/Custom_Hooks/probability_of_improvement.py	Widens hook signature to accept IntoDataFrameT.
examples/Custom_Hooks/campaign_stopping.py	Widens hook signature to accept IntoDataFrameT.
CHANGELOG.md	Documents removal of index semantics and dataframe-based recommendation selection.
baybe/utils/validation.py	Narwhals boundary conversion in preprocess_dataframe, preserving backend on return.
baybe/utils/sampling_algorithms.py	Accepts IntoDataFrame for sampling helpers and normalizes to pandas internally.
baybe/utils/dataframe.py	Adds _infer_backend and _df_with_backend helpers for backend routing/conversion.
baybe/targets/numerical.py	Removes pandas-index preservation in NumericalTarget.transform.
baybe/settings.py	Adds converter for default_dataframe_backend to normalize backend inputs.
baybe/recommenders/pure/nonpredictive/sampling.py	Narwhals-ifies outputs/combination logic and drops index-based alignment.
baybe/recommenders/pure/nonpredictive/clustering.py	Narwhals-ifies return type and normalizes discrete selections to default index.
baybe/recommenders/pure/nonpredictive/base.py	Narwhals-ifies API and replaces .empty with row-count check via Narwhals.
baybe/recommenders/pure/bayesian/botorch/hybrid.py	Narwhals-ifies hybrid recommenders and replaces index-based merge alignment.
baybe/recommenders/pure/bayesian/botorch/discrete.py	Narwhals-ifies return type and replaces merge-based matching with tensor equality matching.
baybe/recommenders/pure/bayesian/botorch/core.py	Narwhals-ifies return types and replaces pandas construction with Narwhals construction.
baybe/recommenders/pure/bayesian/botorch/continuous.py	Uses Narwhals dataframe construction for constraint checks.
baybe/recommenders/pure/bayesian/base.py	Narwhals-ifies API; converts to pandas for BoTorch setup internals.
baybe/recommenders/pure/base.py	Narwhals-ifies protocol surface; infers backend and scopes Settings for consistent outputs.
baybe/recommenders/naive.py	Narwhals-ifies API; scopes Settings and uses Narwhals concat for final output.
baybe/recommenders/meta/sequential.py	Narwhals-ifies meta recommender API signatures.
baybe/recommenders/meta/base.py	Narwhals-ifies meta recommender API signatures.
baybe/recommenders/base.py	Narwhals-ifies RecommenderProtocol to use IntoDataFrameT.
baybe/parameters/base.py	Removes pandas-index preservation in discrete parameter transformation.
baybe/objectives/base.py	Removes pandas-index preservation in objective transformation and uses Narwhals backend selection.
baybe/constraints/utils.py	Accepts IntoDataFrame and normalizes to pandas internally for checks.
baybe/campaign.py	Normalizes returned recommendation to pandas before caching logic.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[fabianliebig]

Hi @AdrianSosic, I unfortunately overlooked that you set the PR to draft yesterday. Will post my comments anyway, but feel free to resolve or ignore until the PR is ready.

[fabianliebig]

Is the potencial conversion to pandas if it is not the set backend in the settings paragamtic or is there another reason to have it? Could it e.g. also be written in plain narwhals?

[AdrianSosic]

Yes, the body is absolutely narwhalifiable. The reason why I haven't done it yet: there will be plenty of isolated code pieces that require the same. If I do all of them as part of the dev branch, the PR will become a never-ending story and put unnecessary burden on the reviewers. So I'm:

• truly narwhalifying (signature and body) the core recommendation chain
• only narwhalifying the API boundaries of helpers and keeping backend conversion shims in the bodies (e.g. like this function, but laters also constraint internals, simulation functionality, specific alternative surrogates etc)

Narwhalifying the latter can follow in isolated follow-up PRs any later point in time and, importantly, on main instead of the dev branch 🙃

[fabianliebig]

I see. Why is it different for the recommender? Or is it just that you wanted the integration to be in every relevant part up to a certain extent so that the rest can be done, e.g., simultaneously? Just for my understanding :D

[fabianliebig]

General question regarding that automated inference of backends: Currently we often set the backend from the settings if a dataframe is created without context. Can it then happen that we allow multiple backends simultaneously? If that can be the case (also by accident), code like _infer_backend would hide a lot of errors, wouldn't it?

[AdrianSosic]

Let us tear this question into its separated pieces so that I a) better understand your concern and b) see what can be fixed.

1. Yes, calling something like _infer_backend(pd.DataFrame([1, 2, 3]), pl.DataFrame([1, 2, 3])) would sort of silently swallow the pl backend. Perhaps not great but strictly speaking not violating what it promises to do (i.e. its docstring explicitly says it would do so). We could improve the situation with

   • static typing, but we'd need to turn the *frames approach into a container type argument for that
   • runtime checks
   • --> for both, not sure if worth it because, as said, it's actually not breaking the contract

2. For me the real question if there is a significant risk that we'd ever even run into the multi-backend situation in the first place. Of course, I could construct a scenario where I change the settings along the way or explicitly use two different modules as a user at the same time. But that's more a question of pragmatism/convenience vs 100% case coverage. Can you share one realistic pattern that you see where it could happen?

3. Is there something else you are concerned about? Or what did you actually have in mind when writing the comment?

[fabianliebig]

I probably have to rephrase. Sorry if the question/concern is a bit misleading. I'm also trying to get my head around the changes, and I think the main issue is that we cannot use an agnostic backend jet, so we have to build workarounds to still use Narwhals at the moment. That is completely fine, and the only thing that you can do, which I absolutely appreciate. I'm just wondering if that is the best approach for this transition or if we should take it differently, e.g., saying, "While the backend is changed, we still stick to pandas until we can provide an actual agnostic backend and figure out how to resolve all the edge cases" (thanks for directly working with the narwhals team on that, which has become a huge work package, but you manage). The reason I'm suggesting that is that the current code doesn't work as expected because for many examples we still have to use pandas so that it can happen that your resolved backend is ignored. A small example is this, where in the second round pandas is returned although polars was provided (I think the problem is in get_canidates but I couldn't validate):

import polars as pl

from baybe import active_settings
from baybe.objectives import SingleTargetObjective
from baybe.parameters import NumericalDiscreteParameter
from baybe.recommenders import (
    BotorchRecommender,
    RandomRecommender,
    TwoPhaseMetaRecommender,
)
from baybe.searchspace import SearchSpace
from baybe.targets import NumericalTarget

N_DOE_ITERATIONS =  3
BATCH_SIZE = 2

active_settings.default_dataframe_backend = "polars"

parameters = [
    NumericalDiscreteParameter("Temperature", values=(90, 105, 120), tolerance=2),
    NumericalDiscreteParameter(
        "Concentration", values=(0.057, 0.1, 0.153), tolerance=0.005
    ),
]

searchspace = SearchSpace.from_product(parameters=parameters)
objective = SingleTargetObjective(target=NumericalTarget(name="yield"))

recommender = TwoPhaseMetaRecommender(
    initial_recommender=RandomRecommender(),
    recommender=BotorchRecommender(),
)

measurements: pl.DataFrame | None = None

for _ in range(N_DOE_ITERATIONS):
    recommendation = recommender.recommend(
        batch_size=BATCH_SIZE,
        searchspace=searchspace,
        objective=objective,
        measurements=measurements,
    )
    assert type(recommendation) is pl.DataFrame, f"Expected a polars DataFrame, but got {type(recommendation)}."

    recommendation = recommendation.with_columns(
        pl.Series("yield", np.random.uniform(0, 100, size=len(recommendation)))
    )
    measurements = (
        recommendation
        if measurements is None
        else pl.concat([measurements, recommendation])
    )

The only alternative would be to actually convert all dataframes in the API parts that directly interact with the user, but I'm afraid that it continues to hide the problem, and we thereby actually duplicate code because @narwhalify does the same thing as _inferbackend, only without providing the backend to us. I hope that made it a bit clearer what I meant; my concern is not the approach but its implications, if that makes sense 😬

[fabianliebig]

Why IntoDataFrameT in this case? Doesn't that requiere (as a TypeVar) a concection between input and output which is not given as the function returns noting?

Suggested change

	measurements: IntoDataFrameT | None = None,
	measurements: IntoDataFrame | None = None,

[AdrianSosic]

Damn it, I've hoped I could sneak this one in without someone noticing 😅 Run it yourself and you'll see the exact problem 😬 Truth is: this entire hook approach that we're selling here has severe downsides. Back at the time when we conceptualized it, it looked nice on paper, but the involved monkeypatching really causes problems in many places. This is just one of them, and there's severl others (like bad interaction with slots, subprocesses etc). So I think in the long run we need to replace it with some explicit hooks that we offer to the users, or some other system that I haven't yet conceptualized – general input welcome!

For now, I have no better idea than pretending the problem did not exist 😄 Thoughts?

[fabianliebig]

Not under my watch ;) But I see the issue now. I was not so familiar with the hook concept tbh. One could also only go through the boundaries if it's a TypVar, but I guess we would then just add an extra if for every edge case. If the design should be kept, then the only possibility is to reduce the safeguards and only pare by name. Then the user is in charge. Everything else that I can think of is a bigger and probably unreasonable change. Many cases that I know use observer patterns that define the states in which a hook can happen but that can result in to generous or a lot of interfaces.