Statistics and trades for each combination of optimization

[xyffar]

When I run an optimization through bt.optimize(), I get the list of all the tried combinations of parameters along with the related value of the objective function but not all the other statistics. The full set of statistics and trades is available only for the best parameter combinations.

My question is: is there any way to get statistics and trades for each combination of parameters? So far, I've had to run a bt.run() for each parameter combination but it is a waste of time since they were already calculated during the optimization.

[initial-d]

Short answer: I do not think Backtest.optimize() exposes the full stats/trades for every tried combination.

What it can return is the best run plus, with return_heatmap=True, a pd.Series indexed by parameter combinations containing the optimization objective value. The individual stats objects produced during optimization are used to fill that heatmap, but they are not retained as full result objects. That is probably intentional: keeping _trades, equity curves, and other per-run objects for a large grid would get expensive quickly, especially with multiprocessing.

A practical pattern is to use optimization to find the interesting region, then rerun only the candidates you want to inspect:

best_stats, heatmap = bt.optimize(
    n1=[5, 10, 15],
    n2=[20, 40, 80],
    maximize="SQN",
    return_heatmap=True,
)

# Pick the top N parameter combinations by the optimized metric
top = heatmap.dropna().sort_values(ascending=False).head(20)

full_stats = {}
trades = {}

for key in top.index:
    # key is a tuple if there are multiple optimized parameters
    if not isinstance(key, tuple):
        key = (key,)

    params = dict(zip(top.index.names, key))
    stats = bt.run(**params)

    full_stats[key] = stats
    trades[key] = stats["_trades"].copy()

If you truly need full stats for every admissible combination, then a manual loop is usually clearer:

from itertools import product
import pandas as pd

rows = []
all_trades = {}

for n1, n2 in product([5, 10, 15], [20, 40, 80]):
    if n1 >= n2:
        continue

    stats = bt.run(n1=n1, n2=n2)
    key = (n1, n2)

    rows.append({
        "n1": n1,
        "n2": n2,
        "Return [%]": stats["Return [%]"],
        "Sharpe Ratio": stats["Sharpe Ratio"],
        "Max. Drawdown [%]": stats["Max. Drawdown [%]"],
        "# Trades": stats["# Trades"],
    })
    all_trades[key] = stats["_trades"].copy()

summary = pd.DataFrame(rows).set_index(["n1", "n2"])

So the tradeoff is:

• use optimize(..., return_heatmap=True) when you only need the objective value for each combination;
• rerun the top candidates when you need detailed diagnostics;
• use a manual loop when every combination's full stats / _trades matters more than optimization convenience.

[kernc]

In addition to the excellent answer by @initial-d above, you can always pass your own maximize= func that does the stats logging. E.g.:

my_stats_log = []

def my_eval_func(stats):
    my_stats_log.append(stats)  # Keep a reference on all stats
    return stats['SQN']

...

bt.optimize(..., maximize=my_eval_func)

# stats log contains stats, including `_trades`, from all the runs
print(len(my_stats_log))

Your PC might explode, though.

[initial-d]

Thanks, this is a much cleaner hook for this use case. I especially like that it keeps the normal optimize API intact while still letting the user collect the full stats objects.

The memory warning is worth emphasizing: keeping every stats object can become expensive quickly if the parameter grid is large, especially because each stats object may retain _trades and _equity_curve. For large sweeps I would probably log only the scalar fields needed for ranking first, then rerun a small shortlisted set to collect full trade/equity diagnostics.