Implement interpolation and balancing for numeric variables (e.g., income)

[vahid-ahmadi]

In our process, we need to ensure that numeric variables (such as income) are properly balanced and interpolated when necessary.

Interpolation techniques:

• Linear Interpolation (pandas.DataFrame.interpolate(method='linear'))
• Polynomial Interpolation (numpy.polyfit)
• Spline Interpolation (scipy.interpolate.spline or scipy.interpolate.interp1d)
• KNN-based Imputation (sklearn.impute.KNNImputer)

Balancing techniques:

• Standardization (sklearn.preprocessing.StandardScaler)
• Quantile Normalization (scipy.stats.mstats.rankdata)
• SMOTER (Synthetic Minority Over-sampling Technique for Regression) for balancing underrepresented numeric values (imblearn.over_sampling.SMOTER)

[MaxGhenis]

Why? Is there evidence this would improve predictive performance in nonparametric models?

[vahid-ahmadi]

There is a large literature on balancing impacts for ML methods (not only classification, but also non-parametric methods). I include some main papers here.

I suggest: For the robustness check section, use interpolation and balancing methods to show that our results remain consistent even for the top quintile where we have fewer data points.

Main papers:

• "SMOTE for Regression" – Torgo et al. (2013): Adapting SMOTE to regression improves accuracy in predicting rare extreme target values.
• "SMOGN: A Pre-processing Approach for Imbalanced Regression" – Branco et al. (2017): It outperforms earlier methods, especially when using nonparametric models like Random Forests, by enhancing predictive performance on minority cases.

Other papers:

• "SMOTEBoost for Regression: Improving the Prediction of Extreme Values" – Moniz et al. (2018): It improves the prediction of extreme values compared to standard boosting.
• "DistSMOGN: A Distributed SMOGN Framework for Imbalanced Regression in Big Data" – Song et al. (2022): Developed a distributed SMOGN (DistSMOGN) for big data, confirming that SMOGN’s resampling benefits scale up and improve predictive performance on rare cases even in large-scale regression tasks.

[MaxGhenis]

ChatGPT predicts that SMOTE would worsen performance in this setting by introducing noise. Low priority, and if we do it we should thoroughly evaluate it.

[vahid-ahmadi]

Ok, we can place it in the robustness check section later.

By the way, SMOTE is for classification. Some adaptive methods, such as SMOGN and SMOTER, could be useful for us.