Things this benchmark turned up that are worth recording separately: measurement defects that produced published numbers, and cost results that contradict the reason people reach for these models.
Both TabNet and FT-Transformer were characterised in earlier notes on numbers generated by a run in which they were largely not training.
The wrappers split off a validation set for early stopping with
rng.permutation seeded from a constant, unstratified. On any dataset where a class
missed that split, the logloss metric raised — and because the seed was fixed, it
raised on every trial, every fold, every time.
The failure was silent, and worse than silent: a dataset whose every fit raises completes fast. Optuna caught the exception, marked the trial failed, and moved on in milliseconds. So the dataset finished quickly, recorded a NaN or a score from whatever partial folds survived, and the run looked healthy.
The damage:
| model | datasets with no score | what the published numbers meant |
|---|---|---|
| TabNet | 26 of 146 | cost measured on mostly-failing fits |
| FT-Transformer | 79 of 146 | mean over the 67 that happened to work |
The clearest single case is abalone-7class: the log records 1736.9 s of TabNet
time with zero completed fits — no early-stopping line, no max-epochs line. That
1736.9 s is pure wrapper overhead for 324 fits that each raised on entry.
Consequence for TabNet's headline number: its published 107.3 h total made it look like a cheap model that simply performs badly. With the split stratified and every fit actually training, the first two datasets cost 11.47 h and 7.48 h against historical 0.53 h and 0.38 h — 20x. Projected over 146 datasets that is 824-1372 h, against ResNet's measured 197 h. TabNet is not a cheap weak model; it is an expensive one that was broken.
The lesson is about coverage, not about these two models. A mean is only meaningful next to the count it was taken over. Check the all-NaN count before trusting any aggregate — the run that produces the worst numbers is the one that looks fastest.
pr_auc_score reads y_prob[:, 1] on binary problems, so which class counts as
positive follows label ordering. The classical and foundation paths encoded labels
differently, and on the affected binary datasets the two families were scored
against opposite classes.
This did not produce obviously wrong numbers. It produced interesting ones, which is worse. It generated an entire published finding: a table of 13 datasets where foundation models supposedly collapsed, and two successive explanations for it — first "severe class imbalance", then a correction to "small binary data". Both described an artefact.
| dataset | published gap | actual |
|---|---|---|
blood-transfusion-service |
+0.3661 | −0.0122 (foundation wins) |
appendicitis |
+0.2361 | −0.0314 (foundation wins) |
seismic-bumps (TabICL vs CatBoost) |
+0.73 | +0.0026 |
thoracic-surgery (TabICL vs CatBoost) |
+0.66 | +0.0002 |
After the fix, exactly 2 of 146 datasets have any classical model ahead of every foundation model by more than 0.02.
A finding that is large, clean, and explainable deserves more suspicion than a messy one, not less. Both explanations fit the bad data comfortably.
The AutoML runs cover 142 of the 146 datasets. The figure notebooks sliced
evaluated_datasets positionally, which assumes the four missing datasets are last.
They are not — the gaps fall at indices 61, 62, 68 and 69 (movement-libras-10,
movement-libras, ozone-eighthr, ozone-onehr).
81 of 142 rows were attributed to the wrong dataset. Recovered by DP sequence alignment against the score vectors: the aligned mapping costs 12.797 / 12.729 with correlation 0.80 between the two frameworks, against 19.889 / 19.798 and correlation 0.37 for the positional assumption. The names are now written into the joblib files and joined by name.
The corrected numbers move AutoML from "comparable to individual gradient boosters" to the top of the benchmark by a clear margin.
ResNet and FT-Transformer use _get_device(), which selects MPS. TabNet is built
through pytorch-tabnet, whose define_device("auto") is:
if device_name == "auto":
if torch.cuda.is_available():
return "cuda"
else:
return "cpu"MPS is not in it. On Apple silicon "auto" silently means CPU, and no warning is
emitted. Forcing device_name="mps" measured 3.9-4.3x faster with no cost in
score: over 5 seeds per device, abalone-3class 0.5398 ± 0.0763 on CPU against
0.5733 ± 0.0245 on MPS, volcanoes-a3 0.8243 ± 0.0060 against 0.8140 ± 0.0265.
A single-seed comparison first looked alarming (0.6252 CPU vs 0.4670 MPS), but CPU's own five-seed range on that dataset is [0.4165, 0.6414]. Device comparisons on a stochastic model need distributions, not one run each.
That same experiment: on abalone-3class, TabNet's PR AUC across 5 seeds spans
0.4165 to 0.6414 on CPU, a standard deviation of 0.076.
For scale, the entire classical block of this benchmark — CatBoost, LightGBM, LightGBM-linear, XGBoost, Random Forest, HistGradientBoosting, SVC — spans 0.015 in mean PR AUC. TabNet's run-to-run noise on one dataset is five times the spread the benchmark is trying to resolve between seven different algorithms.
Nested CV over 4 outer folds damps this, but not to nothing, and nothing in the published single-number-per-model format shows it.
SVC originally recorded 96.3 h, the second most expensive model in the benchmark. Re-run
with identical scoring code on an otherwise idle machine it took 2.41 h, and
arcene went from 22 886.9 s to 9.8 s — a factor of 2300 on one dataset.
The difference was not the algorithm. The original run shared the machine with processes that drove it into swap. The binding resource in this benchmark is RAM, not cores: 24 GB, and a single TabFM dataset uncapped allocated 13.55 GB.
A separate real defect surfaced during the investigation: libsvm's SMO solver is
unbounded by default and genuinely never terminates on some (config, fold) pairs.
Four orphaned libsvm workers were found running 14 h 08 m with ppid=1, six hours
after their parent was killed. max_iter=2_000_000 bounds it; verified
non-binding — scores are byte-identical on all comparable datasets.
Partway through the ResNet run its rate drifted from 1.31x to 1.58x of historical over several days. The obvious reading was that the remaining datasets were more expensive. That was wrong.
Ten orphaned VSCode helper processes, each burning 90-125 % CPU with an RSS under 200 KB, had been running for 33 days and were taking roughly ten of the machine's fourteen cores. Load average was 82. After killing them the rate returned to 1.15x, better than the run had ever managed.
The run itself had no way to see this. Every in-process signal — its own CPU percentage, its memory, its progress per dataset — looked normal. Wall-clock per dataset was the only affected number, and it is exactly the number a benchmark publishes.
Every model stores per-fold probability matrices, so combining them costs arithmetic rather than compute. 126 datasets covered by all nine models tested; probability, logit and rank averaging.
| ensemble | best combiner | mean | vs best single | win rate |
|---|---|---|---|---|
| TabFM + TabICL + TabPFN-3 | logit | 0.8654 | +0.0001 | 38.9 % |
| 3 foundation + CatBoost | logit | 0.8647 | −0.0006 | 42.1 % |
| TabICL + TabPFN-3 | logit | 0.8636 | −0.0017 | 34.1 % |
| all 9 models | logit | 0.8620 | −0.0033 | 30.2 % |
| TabICL + CatBoost | logit | 0.8612 | −0.0041 | 29.4 % |
| CatBoost + LightGBM-linear + XGBoost | logit | 0.8491 | −0.0162 | 17.5 % |
Best single model is TabFM at 0.8653. Nothing beats it. The strongest ensemble ties it to within 0.0001 and wins on 39 % of datasets — worse than a coin flip.
Two specifics worth stating. Adding CatBoost to the foundation trio makes it worse (−0.0006), which is the second independent refutation of the "always co-train a cheap classical baseline" advice these notes used to give; the first was that the blind-spot table justifying it turned out to be a scoring bug. And logit averaging beats probability averaging beats rank averaging, consistently, in every combination — but the ordering does not matter much when none of them wins.
Total wall clock for the full 146-dataset nested CV, against mean PR AUC over each model's own coverage:
| model | hours | median/dataset | mean PR AUC | coverage |
|---|---|---|---|---|
| SGD | 0.9 | 11 s | 0.7729 | 146 |
| SVC | 2.4 | 10 s | 0.8248 | 146 |
| TabFM | 4.3 | 27 s | 0.8653 | 126 |
| LogReg | 5.9 | 24 s | 0.7772 | 146 |
| XGBoost | 7.4 | 53 s | 0.8328 | 146 |
| Random Forest | 7.6 | 105 s | 0.8359 | 146 |
| LightGBM | 11.3 | 118 s | 0.8345 | 146 |
| LightGBM-linear | 13.2 | 145 s | 0.8374 | 146 |
| TabICL | 34.7 | 364 s | 0.8574 | 142 |
| HistGradientBoosting | 36.5 | 384 s | 0.8303 | 146 |
| CatBoost | 75.7 | 268 s | 0.8386 | 146 |
| TabPFN-3 | 73.6 | 1199 s | 0.8591 | 146 |
| ResNet | 207.7 | 2597 s | 0.8234 | 146 |
| TabPFN 2.6 | 236.3 | 678 s | 0.8246 | 129 |
The two most expensive models in the benchmark rank eleventh and twelfth of fourteen. Together they cost 433.6 h — more than every other model combined — to land below Random Forest, which costs 7.6 h.
TabFM reaches the highest mean of any single model for 4.3 h, though on 126 datasets and with a GPU.
UCI++ reuses the same underlying data in different configurations. The figure notebooks silently exclude 15 of them:
wine-quality-white-5class, waveform-v2, volcanoes-a3, volcanoes-b3,
volcanoes-b4, volcanoes-b5, volcanoes-b6, volcanoes-d3, volcanoes-d4,
statlog-german-credit-numeric, thyroid-allhyper, thyroid-allhypo,
thyroid-allrep, thyroid-hypothyroid, thyroid-dis
They are still computed — 7 of the 9 volcanoes variants cost ResNet over 2 h each
— so about 10 % of the benchmark's compute goes to datasets that are then dropped
from every figure. Worth filtering before the run rather than after.
A timed-out (dataset, model) pair records time = NaN, so the hours it burned never
enter the cost figures. Both pairs were re-run with the cap raised to a week. Both
answers were worth the machine time.
CatBoost on plant-species-leaves-shape (100 classes, 1600 rows, 64 features)
needs 25.16 h. It had recorded NaN twice before — 14 h with the timeout defeated,
then a clean 12 h cut — so the number simply did not exist. What it buys:
| model | PR AUC | time |
|---|---|---|
| TabICL | 0.8998 | 0.34 h |
| TabPFN-3 | 0.8987 | 0.60 h |
| ResNet | 0.8250 | 1.57 h |
| SVC | 0.7620 | 0.01 h |
| CatBoost | 0.7321 | 25.16 h |
| Random Forest | 0.7203 | 0.17 h |
| LightGBM | 0.6665 | 0.55 h |
CatBoost places fifth. SVC scores higher in 36 seconds — 2500x cheaper for +0.03 PR AUC. TabICL scores +0.17 higher for 1/74th of the cost. This one dataset is a third of CatBoost's entire 75.7 h benchmark cost, and it is the single clearest case in the benchmark of compute buying nothing.
The failure mode is many-class data: CatBoost trains one-vs-all across 100 classes inside a 50-trial nested search. Nothing about the data is hard — every other model finishes in under two hours.
ResNet on multiple-features (2000 x 649) needs 10.40 h and scores 0.9987.
Above 500 features _train_rtdl_on_device falls back from MPS to CPU. Historically
1.20 h at 20 trials; 2.48x more fits predicts 2.98 h on the GPU, so the measured
10.40 h puts the cost of losing MPS at 3.49x — close to the 3.5x estimated from
the TabNet device experiment, and enough, multiplied by the trial change, to push a
one-hour dataset past a twelve-hour cap.
Both models now cover all 146 datasets.