Comparative Analysis of Machine Learning Algorithms for Software Fault Prediction
DOI:
https://doi.org/10.57152/malcom.v6i4.2659Keywords:
Class Imbalance, Feature Selection, Machine Learning, Parameter Optimization, Software Fault PredictionAbstract
Software fault prediction helps development teams prioritize quality assurance by identifying software components that are likely to contain faults before release. The purpose of this study is to determine whether three recommended optimization strategies: parameter optimization, feature selection, and class-imbalance handling produce consistent performance improvements across different machine learning algorithms. This study compares five machine learning algorithms for software fault prediction, namely Random Forest, K-Nearest Neighbor, AdaBoost, Support Vector Machine (SVM), and a fully connected neural network, using historical change features derived from five open-source software projects. The study evaluates three optimization strategies: parameter optimization, feature selection using Least Absolute Shrinkage and Selection Operator (LASSO) regression, and class-imbalance handling through oversampling and undersampling. Model performance is assessed using five-fold stratified cross-validation with Accuracy, Receiver Operating Characteristic–Area Under the Curve (ROC-AUC), and F1-score. The results show that the parameter-optimized baseline models achieved comparable performance across algorithms, with accuracy values above 0.83 and ROC-AUC values modestly above random performance. Feature selection produced limited overall gains and improved all three performance measures only for AdaBoost, while class-imbalance handling generally reduced accuracy although it improved ROC-AUC in several cases.
Downloads
References
T. Mende, “Replication of defect prediction studies: problems, pitfalls and recommendations,” presented at the Proceedings of the 6th International Conference on Predictive Models in Software Engineering, 2010, pp. 1–10.
C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “The Impact of Automated Parameter Optimization on Defect Prediction Models,” IIEEE Trans. Software Eng., vol. 45, no. 7, pp. 683–711, Jul. 2019, doi: 10.1109/TSE.2018.2794977.
T. Zhou, X. Sun, X. Xia, B. Li, and X. Chen, “Improving defect prediction with deep forest,” Information and Software Technology, vol. 114, pp. 204–216, Oct. 2019, doi: 10.1016/j.infsof.2019.07.003.
F. Matloob et al., “Software Defect Prediction Using Ensemble Learning: A Systematic Literature Review,” IEEE Access, vol. 9, pp. 98754–98771, 2021, doi: 10.1109/ACCESS.2021.3095559.
Q. Song, Y. Guo, and M. Shepperd, “A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction,” IIEEE Trans. Software Eng., vol. 45, no. 12, pp. 1253–1269, Dec. 2019, doi: 10.1109/TSE.2018.2836442.
B. Ghotra, S. McIntosh, and A. E. Hassan, “Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Florence, Italy: IEEE, May 2015, pp. 789–800. doi: 10.1109/ICSE.2015.91.
T.-S. Quah and M. M. T. Thwin, “Application of neural networks for software quality prediction using object-oriented metrics,” presented at the International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings., IEEE, 2003, pp. 116–125.
X. Yu et al., “Improving Ranking-Oriented Defect Prediction Using a Cost-Sensitive Ranking SVM,” IEEE Trans. Rel., vol. 69, no. 1, pp. 139–153, Mar. 2020, doi: 10.1109/TR.2019.2931559.
K. Li, Z. Xiang, T. Chen, S. Wang, and K. C. Tan, “Understanding the automated parameter optimization on transfer learning for cross-project defect prediction: an empirical study,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, Seoul South Korea: ACM, Jun. 2020, pp. 566–577. doi: 10.1145/3377811.3380360.
S. Goyal, “Handling class-imbalance with KNN (neighbourhood) under-sampling for software defect prediction,” Artificial Intelligence Review, vol. 55, no. 3, pp. 2023–2064, 2022.
L. Miao, M. Liu, and D. Zhang, “Cost-sensitive feature selection with application in software defect prediction,” presented at the Proceedings of the 21st international conference on pattern recognition (ICPR2012), IEEE, 2012, pp. 967–970.
A. O. Balogun, S. Basri, S. J. Abdulkadir, V. E. Adeyemo, A. A. Imam, and A. O. Bajeh, “Software defect prediction: analysis of class imbalance and performance stability,” J. Eng. Sci. Technol, vol. 14, no. 6, pp. 3294–3308, 2019.
L. Zhao, Z. Shang, L. Zhao, A. Qin, and Y. Y. Tang, “Siamese Dense Neural Network for Software Defect Prediction With Small Data,” IEEE Access, vol. 7, pp. 7663–7677, 2019, doi: 10.1109/ACCESS.2018.2889061.
L. Sikic, A. S. Kurdija, K. Vladimir, and M. Silic, “Graph Neural Network for Source Code Defect Prediction,” IEEE Access, vol. 10, pp. 10402–10415, 2022, doi: 10.1109/ACCESS.2022.3144598.
J. Li, W. Zhang, and K. Li, “A Novel Semi-supervised SVM Based on Tri-training for Intrusition Detection,” JCP, vol. 5, no. 4, pp. 638–645, Apr. 2010, doi: 10.4304/jcp.5.4.638-645.
S. Ding, Z. Zhu, and X. Zhang, “An overview on semi-supervised support vector machine,” Neural Comput & Applic, vol. 28, no. 5, pp. 969–978, May 2017, doi: 10.1007/s00521-015-2113-7.
C. Pornprasit and C. Tantithamthavorn, “JITLine: A Simpler, Better, Faster, Finer-grained Just-In-Time Defect Prediction,” Mar. 16, 2021, arXiv: arXiv:2103.07068. Accessed: May 27, 2022. [Online]. Available: http://arxiv.org/abs/2103.07068
X. Zhu, C. Yan, E. J. Whitehead, B. Niu, L. Zhu, and L. Pan, “Just?in?time defect prediction for software hunks,” Softw: Pract Exper, vol. 52, no. 1, pp. 130–153, Jan. 2022, doi: 10.1002/spe.3001.
X. Yang, D. Lo, X. Xia, and J. Sun, “TLEL: A two-layer ensemble learning approach for just-in-time defect prediction,” Information and Software Technology, vol. 87, pp. 206–220, Jul. 2017, doi: 10.1016/j.infsof.2017.03.007.
H. Tong, B. Liu, and S. Wang, “Kernel Spectral Embedding Transfer Ensemble for Heterogeneous Defect Prediction,” IIEEE Trans. Software Eng., pp. 1–1, 2019, doi: 10.1109/TSE.2019.2939303.
K. E. Bennin, J. Keung, P. Phannachitta, A. Monden, and S. Mensah, “Mahakil: Diversity based oversampling approach to alleviate the class imbalance issue in software defect prediction,” IEEE Transactions on Software Engineering, vol. 44, no. 6, pp. 534–550, 2017.
C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “Automated parameter optimization of classification techniques for defect prediction models,” presented at the Proceedings of the 38th international conference on software engineering, 2016, pp. 321–332.
T. M. Khoshgoftaar, K. Gao, A. Napolitano, and R. Wald, “A comparative study of iterative and non-iterative feature selection techniques for software defect prediction,” Information Systems Frontiers, vol. 16, no. 5, pp. 801–822, 2014.
S. Liu, X. Chen, W. Liu, J. Chen, Q. Gu, and D. Chen, “FECAR: A feature selection framework for software defect prediction,” presented at the 2014 IEEE 38th Annual Computer Software and Applications Conference, IEEE, 2014, pp. 426–435.
Q. Yu, J. Qian, S. Jiang, Z. Wu, and G. Zhang, “An Empirical Study on the Effectiveness of Feature Selection for Cross-Project Defect Prediction,” IEEE Access, vol. 7, pp. 35710–35718, 2019, doi: 10.1109/ACCESS.2019.2895614.
Q. Cai, H. He, and H. Man, “Imbalanced evolving self-organizing learning,” Neurocomputing, vol. 133, pp. 258–270, 2014.
H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Transactions on knowledge and data engineering, vol. 21, no. 9, pp. 1263–1284, 2009.
S. Wang and X. Yao, “Diversity analysis on imbalanced data sets by using ensemble models,” presented at the 2009 IEEE symposium on computational intelligence and data mining, IEEE, 2009, pp. 324–331.
T. Mende and R. Koschke, “Revisiting the evaluation of defect prediction models,” presented at the Proceedings of the 5th International Conference on Predictor Models in Software Engineering, 2009, pp. 1–10.
C. Strobl, A.-L. Boulesteix, A. Zeileis, and T. Hothorn, “Bias in random forest variable importance measures: Illustrations, sources and a solution,” BMC bioinformatics, vol. 8, no. 1, pp. 1–21, 2007.
R. Genuer, J.-M. Poggi, and C. Tuleau-Malot, “Variable selection using random forests,” Pattern recognition letters, vol. 31, no. 14, pp. 2225–2236, 2010.
U. Grömping, “Variable importance assessment in regression: linear regression versus random forest,” The American Statistician, vol. 63, no. 4, pp. 308–319, 2009.
B. A. Goldstein, E. C. Polley, and F. B. Briggs, “Random forests for genetic association studies,” Statistical applications in genetics and molecular biology, vol. 10, no. 1, 2011.
H. Parvin, H. Alizadeh, and B. Minaei?Bidgoli, “Validation Based Modified K?Nearest Neighbor,” presented at the AIP Conference Proceedings, American Institute of Physics, 2009, pp. 153–161.
J. Huang, Y. Wei, J. Yi, and M. Liu, “An improved kNN based on class contribution and feature weighting,” presented at the 2018 10th international conference on measuring technology and mechatronics automation (ICMTMA), IEEE, 2018, pp. 313–316.
S. Zhang, X. Li, M. Zong, X. Zhu, and R. Wang, “Efficient kNN classification with different numbers of nearest neighbors,” IEEE transactions on neural networks and learning systems, vol. 29, no. 5, pp. 1774–1785, 2017.
R. Gao and Z. Liu, “An improved adaboost algorithm for hyperparameter optimization,” presented at the Journal of Physics: Conference Series, IOP Publishing, 2020, p. 012048.
H. Bhavsar and M. H. Panchal, “A review on support vector machine for data classification,” International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), vol. 1, no. 10, pp. 185–189, 2012.
C.-Y. Yeh, C.-W. Huang, and S.-J. Lee, “A multiple-kernel support vector regression approach for stock market price forecasting,” Expert Systems with Applications, vol. 38, no. 3, pp. 2177–2186, 2011.
W. Pannakkong, K. Thiwa-Anont, K. Singthong, P. Parthanadee, and J. Buddhakulsomsiri, “Hyperparameter Tuning of Machine Learning Algorithms Using Response Surface Methodology: A Case Study of ANN, SVM, and DBN,” Mathematical Problems in Engineering, vol. 2022, 2022.
J. Wu, X.-Y. Chen, H. Zhang, L.-D. Xiong, H. Lei, and S.-H. Deng, “Hyperparameter optimization for machine learning models based on Bayesian optimization,” Journal of Electronic Science and Technology, vol. 17, no. 1, pp. 26–40, 2019.
A. M. Karande and D. Kalbande, “Weight assignment algorithms for designing fully connected neural network,” International Journal of Intelligent Systems and Applications, vol. 10, no. 6, p. 68, 2018.
M. D’Ambros, M. Lanza, and R. Robbes, “An extensive comparison of bug prediction approaches,” presented at the 2010 7th IEEE working conference on mining software repositories (MSR 2010), IEEE, 2010, pp. 31–41.
M. D’Ambros, M. Lanza, and R. Robbes, “Evaluating defect prediction approaches: a benchmark and an extensive comparison,” Empir Software Eng, vol. 17, no. 4–5, pp. 531–577, Aug. 2012, doi: 10.1007/s10664-011-9173-9.
Y. Kim and J. Kim, “Gradient LASSO for feature selection,” presented at the Proceedings of the twenty-first international conference on Machine learning, 2004, p. 60.
F. Li, Y. Yang, and E. Xing, “From lasso regression to feature vector machine,” Advances in neural information processing systems, vol. 18, 2005.
R. Muthukrishnan and R. Rohini, “LASSO: A feature selection technique in predictive modeling for machine learning,” presented at the 2016 IEEE international conference on advances in computer applications (ICACA), IEEE, 2016, pp. 18–20.
F. Li, L. Lai, and S. Cui, “On the adversarial robustness of LASSO based feature selection,” IEEE Transactions on Signal Processing, vol. 69, pp. 5555–5567, 2021.
J. K. Sethi and M. Mittal, “An efficient correlation based adaptive LASSO regression method for air quality index prediction,” Earth Science Informatics, vol. 14, no. 4, pp. 1777–1786, 2021.
D. Ueno, H. Kawabe, S. Yamasaki, T. Demura, and K. Kato, “Feature selection for RNA cleavage efficiency at specific sites using the LASSO regression model in Arabidopsis thaliana,” BMC bioinformatics, vol. 22, no. 1, pp. 1–17, 2021.
S. Wattanakriengkrai, P. Thongtanunam, C. Tantithamthavorn, H. Hata, and K. Matsumoto, “Predicting Defective Lines Using a Model-Agnostic Technique,” IIEEE Trans. Software Eng., pp. 1–1, 2020, doi: 10.1109/TSE.2020.3023177.
F. Rahman and P. Devanbu, “How, and why, process metrics are better,” presented at the 2013 35th International Conference on Software Engineering (ICSE), IEEE, 2013, pp. 432–441.
J. A. Hanley and B. J. McNeil, “The meaning and use of the area under a receiver operating characteristic (ROC) curve.,” Radiology, vol. 143, no. 1, pp. 29–36, 1982.
E. Arisholm, L. C. Briand, and E. B. Johannessen, “A systematic and comprehensive investigation of methods to build and evaluate fault prediction models,” Journal of Systems and Software, vol. 83, no. 1, pp. 2–17, Jan. 2010, doi: 10.1016/j.jss.2009.06.055.
T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, “Cross-project defect prediction: a large scale experiment on data vs. domain vs. process,” presented at the Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, 2009, pp. 91–100.
R. Malhotra and S. Kamal, “An empirical study to investigate oversampling methods for improving software defect prediction using imbalanced data,” Neurocomputing, vol. 343, pp. 120–140, 2019.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Amirul Iqbal, Mugi Praseptiawan, Aidil Afriansyah, Hafiz Budi Firmansyah, Andre Febrianto
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Copyright © by Author; Published by Institut Riset dan Publikasi Indonesia (IRPI)
This Indonesian Journal of Machine Learning and Computer Science is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
