Multicollinearity In Machine Learning: Detection, Impact, And Remedial Techniques
DOI:
https://doi.org/10.29284/sg8jdw24Keywords:
Multicollinearity, Machine Learning, Feature Selection, Regularization, Variance Inflation Factor, Principal Component Analysis, Model Interpretability.Abstract
Machine learning and statistical models commonly experience multicollinearity effects because two or more independent variables within datasets shows strong interconnections. When this phenomenon occurs it weakens predictive models and their interpretability which results in untrustworthy coefficient estimates and larger standard error margins and decreased predictive capability. This paper investigates the factors that create multicollinearity together with detection methods and available solutions for this problem in the machine learning field. The effects on linear regression together with logistic regression and additional models such as decision trees and neural networks receive analysis. Researchers employ three approaches to identify multicollinearity that include Variance Inflation Factor (VIF) and correlation matrices as well as principal component analysis (PCA). We analyze multiple approaches for handling this problem which includes utilizing feature selection as well as implementing regularization methods such as Ridge and Lasso regression and performing dimensionality reduction.
The objectives of this research are three fold: (1) to analyze the significance of multicollinearity in statistical inference and machine learning models, with emphasis on its impact on error variances, feature selection, and overall predictive performance; (2) to evaluate and compare the effectiveness of detection methods for multicollinearity such as Variance Inflation Factor (VIF), correlation matrices, Principal Component Analysis (PCA), and eigenvalue decomposition across different modeling contexts; and (3) to validate through empirical analysis, using real-world datasets, how addressing multicollinearity improves model accuracy and interpretability, while also highlighting the trade-offs of methods like PCA in terms of interpretability.
The research employs bench marked datasets through empirical assessment to show model performance impacts from multicollinearity alongside practical real-world solutions for its management. The research intends to establish thorough knowledge of multicollinearity while providing practitioners with practical methods to stabilize model accuracy and clarity.
References
1. James, G., Witten, D., Hastie, T., and Tibshirani, R., 2013. An introduction to statistical learning. New York: Springer.
2. Jolliffe, I.T., 2002. Principal component analysis. 2nd ed. New York: Springer.
3. Farrar, D.E. and Glauber, R.R., 1967. Multicollinearity in regression analysis: the problem revisited. The Review of Economics and Statistics, 49(1), pp.92-107.
4. Hawkins, D.M., 2004. The problem of over fitting. Journal of Chemical Information and Computer Sciences, 44(1), pp.1-12.
5. Abdi, H. and Williams, L.J., 2010. Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4), pp.433-459.
6. Shlens, J., 2014. A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100.
7. Zou, H., Hastie, T., and Tibshirani, R., 2006. Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15(2), pp.265-286.
8. Bro, R. and Smilde, A.K., 2014. Principal component analysis. Analytical Methods, 6(9), pp.2812-2831.
9. Tibshirani, R., 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), pp.267-288.
10. Ringnér, M., 2008. What is principal component analysis? Nature Biotechnology, 26(3), pp.303-304.
11. Ledoit, O. and Wolf, M., 2003. Improved estimation of the covariance matrix of stock returns with an application to portfolio selection. Journal of Empirical Finance, 10(5), pp.603-621.
12. Alter, O., Brown, P.O., and Botstein, D., 2000. Singular value decomposition for genome-wide expression data processing and modeling. Proceedings of the National Academy of Sciences, 97(18), pp.10101-10106.
13. Van den Berg, R.A., Hoefsloot, H.C., Westerhuis, J.A., Smilde, A.K., and van der Werf, M.J., 2006. Centering, scaling, and transformations: improving the biological information content of metabolomics data. BMC Genomics, 7(1), pp.1-15.
14. Zou, H. and Hastie, T., 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), pp.301-320.
15. Wold, S., Esbensen, K., and Geladi, P., 1987. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), pp.37-52.
16. Hotelling, H., 1933. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), pp.417-441.
17. Gujarati, D.N., Porter, D.C. and Gunasekar, S., 2022. Basic econometrics. 6th Ed. New York: McGraw-Hill.
18. Wooldridge, J.M., 2020. Introductory econometrics: A modern approach. 7th ed. Cengage Learning.
19. James, G., Witten, D., Hastie, T. and Tibshirani, R., 2021. An introduction to statistical learning: with applications in R. 2nd ed. Springer.
20. Kutner, M.H., Nachtsheim, C.J. and Neter, J., 2020. Applied linear statistical models. 5th ed. McGraw-Hill.
21. Montgomery, D.C., Peck, E.A. and Vining, G.G., 2021. Introduction to linear regression analysis. 6th ed. Wiley.
22. Hair, J.F., Black, W.C., Babin, B.J. and Anderson, R.E., 2020. Multivariate data analysis. 8th ed. Pearson.
23. Tibshirani, R., 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), pp.267-288.
24. Dormann, C.F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G. and Lautenbach, S., 2013. Collinearity: a review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1), pp.27-46.
25. Aiken, L.S., West, S.G. and Reno, R.R., 1991. Multiple regression: Testing and interpreting interactions. Sage.
26. O’Brien, R.M., 2007. A caution regarding rules of thumb for variance inflation factors. Quality & Quantity, 41(5), pp.673-690.
27. Ghosh, S. and Vogt, A., 2012. Outlier detection and robustness in penalized regression. Journal of Statistical Planning and Inference, 142(5), pp.1428-1438.
28. Zou, H. and Hastie, T., 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), pp.301-320.
29. Jolliffe, I.T. and Cadima, J., 2016. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), p.20150202.
30. Fan, J. and Li, R., 2001. Variable selection via non concave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), pp.1348-1360.
31. Friedman, J., Hastie, T. and Tibshirani, R., 2010. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), pp.1-22.
32. Hoerl, A.E. and Kennard, R.W., 1970. Ridge regression: Biased estimation for non-orthogonal problems. Technometrics, 12(1), pp.55-67.
33. Montgomery, D.C., Peck, E.A. and Vining, G.G., 2021. Introduction to linear regression analysis. 6th ed. Wiley.
34. Wooldridge, J.M., 2020. Introductory econometrics: A modern approach. 7th ed. Cengage Learning.
35. Gujarati, D.N., Porter, D.C. and Gunasekar, S., 2022. Basic econometrics. 6th ed. McGraw-Hill.
36. James, G., Witten, D., Hastie, T. and Tibshirani, R., 2021. An introduction to statistical learning: with applications in R. 2nd ed. Springer.
37. Kutner, M.H., Nachtsheim, C.J. and Neter, J., 2020. Applied linear statistical models. 5th ed. McGraw-Hill.
38. O’Brien, R.M., 2007. A caution regarding rules of thumb for variance inflation factors. Quality & Quantity, 41(5), pp.673-690.
39. Hair, J.F., Black, W.C., Babin, B.J. and Anderson, R.E., 2020. Multivariate data analysis. 8th ed. Pearson.
40. Belsley, D.A., Kuh, E. and Welsch, R.E., 2005. Regression diagnostics: Identifying influential data and sources of collinearity. John Wiley & Sons.
41. Dormann, C.F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G. and Lautenbach, S., 2013. Collinearity: a review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1), pp.27-46.
42. Farrar, D.E. and Glauber, R.R., 1967. Multicollinearity in regression analysis: The problem revisited. The Review of Economics and Statistics, 49(1), pp.92-107.
43. Klein, L.R., 1955. A textbook of econometrics. Row, Peterson and Company.
44. Greene, W.H., 2018. Econometric analysis. 8th ed. Pearson.
45. Jolliffe, I.T. and Cadima, J., 2016. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), p.20150202.
46. Zou, H. and Hastie, T., 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), pp.301-320.
47. Hoerl, A.E. and Kennard, R.W., 1970. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), pp.55-67.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
