The paper Imputation Using Training Labels and Classification via Label Imputation introduces two novel machine learning algorithms designed to efficiently handle missing values, a common issue in practical datasets. The first approach, Classification Based on MissForest Imputation (CBMI), reframes classification as an imputation task, predicting test labels by treating them as missing values that are solved simultaneously with feature imputation using a unified framework.
The authors also propose the Imputation Using Labels (IUL) strategy, which leverages existing training labels by stacking them with the input data to significantly enhance the accuracy and quality of feature imputation. Experimental results demonstrate that IUL consistently achieves a lower mean squared error and improved downstream performance compared to traditional imputation methods that ignore label data. Furthermore, CBMI shows competitive classification accuracy, particularly for imbalanced and categorical datasets where missing values are present in the test set. The research highlights the significance of actively integrating label information during the imputation stage to improve overall data analysis and model performance.
Advantages
- Unified Classification and Imputation: Unlike traditional methods that treat imputation and classification as separate phases, CBMI reframes the classification task itself as a missing value problem,. By predicting test labels essentially as ‘missing values’ in a single step, it allows the complex, non-linear relationships between features and labels to inform one another simultaneously.
- Superior Performance on Imbalanced and Categorical Data: Experiments indicate that CBMI is particularly effective for imbalanced datasets (such as Parkinson, Heart, and Glass) and categorical datasets (such as Soybean),. When test sets contain missing values, CBMI generally yields better classification accuracy and F1-scores compared to standard methods like XGBoost or Random Forest,.
- Robustness to Missing Test Data: CBMI maintains stable and reliable performance even as the rate of missing data in the test set increases.
- Semi-Supervised Capability: Because CBMI stacks training and testing data together, it can handle training datasets that have missing labels, allowing it to function effectively in semi-supervised learning scenarios,.
Disadvantages
- Computational Cost: CBMI is computationally more expensive and requires more time for training compared to methods like IClf (Imputation then Classification) and Random Forest. This is largely due to the iterative nature of the MissForest algorithm it employs.
- Requirement for Data Availability: The method is primarily applicable to scenarios where all data (training and testing) is collected in advance. It operates similarly to K-Nearest Neighbours as an instance-based method, meaning it does not build a portable model that can easily predict outcomes for new, incoming data streams without re-running the imputation process,.
- Scalability Limitations: Due to its reliance on MissForest, CBMI faces computational limitations with very large or high-dimensional datasets, necessitating future investigation into more scalable variants.
Imputation Using Labels (IUL)
Advantages
- Enhanced Imputation Quality: By including the target label alongside input features during the imputation process, IUL significantly improves imputation quality, consistently achieving lower Mean Squared Error (MSE) compared to Direct Imputation (DI),.
- Improved Downstream Accuracy: The inclusion of labels aids the imputation of input features, which in turn leads to better classification and regression accuracy in downstream tasks,.
- Mechanism Alignment (MAR): Stacking labels with input features may reveal hidden dependencies regarding why data is missing. This increases the likelihood that the missing data mechanism aligns with the Missing At Random (MAR) assumption, under which algorithms like MissForest and MICE perform best.
- Increased Information for Tree-Based Models: When used with Random Forest-based imputation, IUL increases the number of available features (from $p$ to $p+1$). This exponentially increases the number of feature combinations available for building trees, potentially capturing more information than relying on input features alone.
Analogy
To understand the difference between the traditional approach and CBMI, imagine trying to solve a crossword puzzle.
- Traditional Approach (Train-then-Predict): You study a completed crossword (the training set) to learn the rules and patterns. Then, you put that away and try to fill in a new, empty crossword (the test set) based solely on your memory of the patterns you studied.
- CBMI Approach: You take the completed crossword and the empty crossword and tape them together into one giant puzzle. You solve the blanks in the new section by looking directly at the words and patterns in the completed section simultaneously. Because you are solving the whole grid at once, the “clues” from the finished section directly help you fill the blanks in the new section, and the surrounding letters in the new section help confirm if your fit is correct.