Skip to content

Latest commit

 

History

History
258 lines (199 loc) · 12.1 KB

File metadata and controls

258 lines (199 loc) · 12.1 KB

Findings that are not about models

Things this benchmark turned up that are worth recording separately: measurement defects that produced published numbers, and cost results that contradict the reason people reach for these models.

The bug that produced two published results

Both TabNet and FT-Transformer were characterised in earlier notes on numbers generated by a run in which they were largely not training.

The wrappers split off a validation set for early stopping with rng.permutation seeded from a constant, unstratified. On any dataset where a class missed that split, the logloss metric raised — and because the seed was fixed, it raised on every trial, every fold, every time.

The failure was silent, and worse than silent: a dataset whose every fit raises completes fast. Optuna caught the exception, marked the trial failed, and moved on in milliseconds. So the dataset finished quickly, recorded a NaN or a score from whatever partial folds survived, and the run looked healthy.

The damage:

model datasets with no score what the published numbers meant
TabNet 26 of 146 cost measured on mostly-failing fits
FT-Transformer 79 of 146 mean over the 67 that happened to work

The clearest single case is abalone-7class: the log records 1736.9 s of TabNet time with zero completed fits — no early-stopping line, no max-epochs line. That 1736.9 s is pure wrapper overhead for 324 fits that each raised on entry.

Consequence for TabNet's headline number: its published 107.3 h total made it look like a cheap model that simply performs badly. With the split stratified and every fit actually training, the first two datasets cost 11.47 h and 7.48 h against historical 0.53 h and 0.38 h — 20x. Projected over 146 datasets that is 824-1372 h, against ResNet's measured 197 h. TabNet is not a cheap weak model; it is an expensive one that was broken.

The lesson is about coverage, not about these two models. A mean is only meaningful next to the count it was taken over. Check the all-NaN count before trusting any aggregate — the run that produces the worst numbers is the one that looks fastest.

Label ordering silently changed the metric

pr_auc_score reads y_prob[:, 1] on binary problems, so which class counts as positive follows label ordering. The classical and foundation paths encoded labels differently, and on the affected binary datasets the two families were scored against opposite classes.

This did not produce obviously wrong numbers. It produced interesting ones, which is worse. It generated an entire published finding: a table of 13 datasets where foundation models supposedly collapsed, and two successive explanations for it — first "severe class imbalance", then a correction to "small binary data". Both described an artefact.

dataset published gap actual
blood-transfusion-service +0.3661 −0.0122 (foundation wins)
appendicitis +0.2361 −0.0314 (foundation wins)
seismic-bumps (TabICL vs CatBoost) +0.73 +0.0026
thoracic-surgery (TabICL vs CatBoost) +0.66 +0.0002

After the fix, exactly 2 of 146 datasets have any classical model ahead of every foundation model by more than 0.02.

A finding that is large, clean, and explainable deserves more suspicion than a messy one, not less. Both explanations fit the bad data comfortably.

AutoML rows were joined by position

The AutoML runs cover 142 of the 146 datasets. The figure notebooks sliced evaluated_datasets positionally, which assumes the four missing datasets are last. They are not — the gaps fall at indices 61, 62, 68 and 69 (movement-libras-10, movement-libras, ozone-eighthr, ozone-onehr).

81 of 142 rows were attributed to the wrong dataset. Recovered by DP sequence alignment against the score vectors: the aligned mapping costs 12.797 / 12.729 with correlation 0.80 between the two frameworks, against 19.889 / 19.798 and correlation 0.37 for the positional assumption. The names are now written into the joblib files and joined by name.

The corrected numbers move AutoML from "comparable to individual gradient boosters" to the top of the benchmark by a clear margin.

TabNet was on the CPU the whole time

ResNet and FT-Transformer use _get_device(), which selects MPS. TabNet is built through pytorch-tabnet, whose define_device("auto") is:

if device_name == "auto":
    if torch.cuda.is_available():
        return "cuda"
    else:
        return "cpu"

MPS is not in it. On Apple silicon "auto" silently means CPU, and no warning is emitted. Forcing device_name="mps" measured 3.9-4.3x faster with no cost in score: over 5 seeds per device, abalone-3class 0.5398 ± 0.0763 on CPU against 0.5733 ± 0.0245 on MPS, volcanoes-a3 0.8243 ± 0.0060 against 0.8140 ± 0.0265.

A single-seed comparison first looked alarming (0.6252 CPU vs 0.4670 MPS), but CPU's own five-seed range on that dataset is [0.4165, 0.6414]. Device comparisons on a stochastic model need distributions, not one run each.

TabNet's seed variance exceeds most between-model gaps

That same experiment: on abalone-3class, TabNet's PR AUC across 5 seeds spans 0.4165 to 0.6414 on CPU, a standard deviation of 0.076.

For scale, the entire classical block of this benchmark — CatBoost, LightGBM, LightGBM-linear, XGBoost, Random Forest, HistGradientBoosting, SVC — spans 0.015 in mean PR AUC. TabNet's run-to-run noise on one dataset is five times the spread the benchmark is trying to resolve between seven different algorithms.

Nested CV over 4 outer folds damps this, but not to nothing, and nothing in the published single-number-per-model format shows it.

SVC's 96 hours were memory thrash, not compute

SVC originally recorded 96.3 h, the second most expensive model in the benchmark. Re-run with identical scoring code on an otherwise idle machine it took 2.41 h, and arcene went from 22 886.9 s to 9.8 s — a factor of 2300 on one dataset.

The difference was not the algorithm. The original run shared the machine with processes that drove it into swap. The binding resource in this benchmark is RAM, not cores: 24 GB, and a single TabFM dataset uncapped allocated 13.55 GB.

A separate real defect surfaced during the investigation: libsvm's SMO solver is unbounded by default and genuinely never terminates on some (config, fold) pairs. Four orphaned libsvm workers were found running 14 h 08 m with ppid=1, six hours after their parent was killed. max_iter=2_000_000 bounds it; verified non-binding — scores are byte-identical on all comparable datasets.

A month of stolen cores, invisible from inside

Partway through the ResNet run its rate drifted from 1.31x to 1.58x of historical over several days. The obvious reading was that the remaining datasets were more expensive. That was wrong.

Ten orphaned VSCode helper processes, each burning 90-125 % CPU with an RSS under 200 KB, had been running for 33 days and were taking roughly ten of the machine's fourteen cores. Load average was 82. After killing them the rate returned to 1.15x, better than the run had ever managed.

The run itself had no way to see this. Every in-process signal — its own CPU percentage, its memory, its progress per dataset — looked normal. Wall-clock per dataset was the only affected number, and it is exactly the number a benchmark publishes.

Ensembles over the stored predictions never help

Every model stores per-fold probability matrices, so combining them costs arithmetic rather than compute. 126 datasets covered by all nine models tested; probability, logit and rank averaging.

ensemble best combiner mean vs best single win rate
TabFM + TabICL + TabPFN-3 logit 0.8654 +0.0001 38.9 %
3 foundation + CatBoost logit 0.8647 −0.0006 42.1 %
TabICL + TabPFN-3 logit 0.8636 −0.0017 34.1 %
all 9 models logit 0.8620 −0.0033 30.2 %
TabICL + CatBoost logit 0.8612 −0.0041 29.4 %
CatBoost + LightGBM-linear + XGBoost logit 0.8491 −0.0162 17.5 %

Best single model is TabFM at 0.8653. Nothing beats it. The strongest ensemble ties it to within 0.0001 and wins on 39 % of datasets — worse than a coin flip.

Two specifics worth stating. Adding CatBoost to the foundation trio makes it worse (−0.0006), which is the second independent refutation of the "always co-train a cheap classical baseline" advice these notes used to give; the first was that the blind-spot table justifying it turned out to be a scoring bug. And logit averaging beats probability averaging beats rank averaging, consistently, in every combination — but the ordering does not matter much when none of them wins.

Cost does not track performance

Total wall clock for the full 146-dataset nested CV, against mean PR AUC over each model's own coverage:

model hours median/dataset mean PR AUC coverage
SGD 0.9 11 s 0.7729 146
SVC 2.4 10 s 0.8248 146
TabFM 4.3 27 s 0.8653 126
LogReg 5.9 24 s 0.7772 146
XGBoost 7.4 53 s 0.8328 146
Random Forest 7.6 105 s 0.8359 146
LightGBM 11.3 118 s 0.8345 146
LightGBM-linear 13.2 145 s 0.8374 146
TabICL 34.7 364 s 0.8574 142
HistGradientBoosting 36.5 384 s 0.8303 146
CatBoost 75.7 268 s 0.8386 146
TabPFN-3 73.6 1199 s 0.8591 146
ResNet 207.7 2597 s 0.8234 146
TabPFN 2.6 236.3 678 s 0.8246 129

The two most expensive models in the benchmark rank eleventh and twelfth of fourteen. Together they cost 433.6 h — more than every other model combined — to land below Random Forest, which costs 7.6 h.

TabFM reaches the highest mean of any single model for 4.3 h, though on 126 datasets and with a GPU.

Fifteen datasets are duplicates

UCI++ reuses the same underlying data in different configurations. The figure notebooks silently exclude 15 of them:

wine-quality-white-5class, waveform-v2, volcanoes-a3, volcanoes-b3, volcanoes-b4, volcanoes-b5, volcanoes-b6, volcanoes-d3, volcanoes-d4, statlog-german-credit-numeric, thyroid-allhyper, thyroid-allhypo, thyroid-allrep, thyroid-hypothyroid, thyroid-dis

They are still computed — 7 of the 9 volcanoes variants cost ResNet over 2 h each — so about 10 % of the benchmark's compute goes to datasets that are then dropped from every figure. Worth filtering before the run rather than after.

Two pairs the timeout could not measure — now measured

A timed-out (dataset, model) pair records time = NaN, so the hours it burned never enter the cost figures. Both pairs were re-run with the cap raised to a week. Both answers were worth the machine time.

CatBoost on plant-species-leaves-shape (100 classes, 1600 rows, 64 features) needs 25.16 h. It had recorded NaN twice before — 14 h with the timeout defeated, then a clean 12 h cut — so the number simply did not exist. What it buys:

model PR AUC time
TabICL 0.8998 0.34 h
TabPFN-3 0.8987 0.60 h
ResNet 0.8250 1.57 h
SVC 0.7620 0.01 h
CatBoost 0.7321 25.16 h
Random Forest 0.7203 0.17 h
LightGBM 0.6665 0.55 h

CatBoost places fifth. SVC scores higher in 36 seconds — 2500x cheaper for +0.03 PR AUC. TabICL scores +0.17 higher for 1/74th of the cost. This one dataset is a third of CatBoost's entire 75.7 h benchmark cost, and it is the single clearest case in the benchmark of compute buying nothing.

The failure mode is many-class data: CatBoost trains one-vs-all across 100 classes inside a 50-trial nested search. Nothing about the data is hard — every other model finishes in under two hours.

ResNet on multiple-features (2000 x 649) needs 10.40 h and scores 0.9987. Above 500 features _train_rtdl_on_device falls back from MPS to CPU. Historically 1.20 h at 20 trials; 2.48x more fits predicts 2.98 h on the GPU, so the measured 10.40 h puts the cost of losing MPS at 3.49x — close to the 3.5x estimated from the TabNet device experiment, and enough, multiplied by the trial change, to push a one-hour dataset past a twelve-hour cap.

Both models now cover all 146 datasets.