Data Preprocessing and Missing Data Handling for Predicting High School Academic Outcomes

Abstract

The challenge of accurately estimating student academic results based on complicated education records, including HSLC outcomes, where rigorous preprocessing and attentive handling of multi-field information, including demographics, institution characteristics, and topic-level marks, is frequently afflicted by non-existent values and discrepancies that undermine the independence of models (Hunt et al., 2021; Konda, 2022). It is a paper on an end-to-end preprocessing system harmonised to the educational marks data, comprising schema auditing, type enforcement, categorical harmonisation, the scale of numeric, outlier treatment, feature engineering, and imputation strategies which avoid information leakage on validation. The framework combines a mean imputation of statistical attributes, constant-token imputation of categorical variables with informative missing-indicators, with optional correlation elimination and a Principal Component Analysis (PCA) to provide a concise representation where necessary. Structured gaps (unused fields and systematically empty components) are given particular attention, and non-applicable practical are treated, wholly empty columns are not included. Our framework also dramatically increases the predictive algorithm performance and stability of predictive algorithms such as Random Forests, k-NN, and SVMs by making them work on clean and well-structured data. Moreover, it makes missingness consciously encoded, increasing the interpretability of the model, which is more informative to educational policy analysis. The work emphasises preparing an accurate, clean, and reliable dataset with regard to various approaches that are used as a subset of data preprocessing, which is considered paramount important in converting live, raw and messy large educational data into a trustworthy dataset to make decisions.