Machine Learning-Assisted Code Generation: Exploring The Role Of Large Language Models In Automating Software Development Tasks
DOI:
https://doi.org/10.29284/p2qjhb56Keywords:
Large Language Models (LLMs), Machine Learning-Assisted Code Generation, Software Engineering Automation, Human-in-the-Loop Evaluation, Developer Productivity and Trust.Abstract
This paper explores the capabilities of large language models (LLMs) to automate software development tasks in combination with human-in-the-loop evidence to quantitatively benchmark performance. Background: LLMs hold the promise of translating natural-language intent into code, but concerns remain over reliability, maintainability and trust in actual workflows. Procedure: We performed a mixed-methods evaluation of MBPP (Python) and a curated C++ dataset with GPT-4, Code Llama, and StarCoder, comparing Pass apartment at k accuracy, execution proficiency, and unit-goal coverage, and surveyed 21 items (n=100) on adoption, perceived productivity, belief, and experience effects. Results: GPT-4 had the highest Pass@5 and execution success both on the two datasets consistently; Code Llama was in the middle; StarCoder fell behind. Adoption is still patchy (51% had not used AI coding tools at all). The perceived gains in productivity were modest (median 3/5; mean 3.2/5), trust was concentrated at “Medium” (~38%), and those with 1-3 years of experience saw the largest benefits (mean approx. 3.58), indicating that LLMs are likely to be helpful to juniors on routine work but to seniors on complex design work. Recommendation: LLMs are optimal as a supplemental, facilitator of existing hybrid test-gated workflows; training should focus on problem development, trouble shooting, and AI critiquing. Disadvantages are the presence of only function-level benchmarks, Python/C++ scope, and self-reported survey data; future research must address industrial-scale repositories, other languages, longitudinal user studies, and iterative human-in-the-loop debugging, and improved levels of explainability and provenance controls.
References
1. E. Dehaerne, B. Dey, S. Halder, S. De Gendt, and W. Meert, “Code generation using machine learning: A systematic review,” IEEE Access, vol. 10, pp. 82434–82455, Aug. 2022.
2. L. Qu, Z. Qu, Q. Hu, M. Liu, and Z. Ren, “Precipitation forecasting with radar echo maps based on interactive spatiotemporal context with self-attention and the MIM model,” Geogr. Ann. Ser. A, Phys. Geogr., vol. 105, no. 2–3, pp. 166–178, Jul. 2023.
3. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol. (NAACL-HLT), 2019, pp. 4171–4186.
4. Z. Wang, “Empowering few-shot recommender systems with large language models-enhanced representations,” IEEE Access, vol. 12, pp. 29144–29153, Feb. 2024.
5. A. Chowdhery et al., “PaLM: Scaling language modeling with pathways,” J. Mach. Learn. Res., vol. 24, no. 240, pp. 1–113, 2023.
6. M. Chen et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, Jul. 2021.
7. F. Eshraghian, N. Hafezieh, F. Farivar, and S. de Cesare, “AI in software programming: Understanding emotional responses to GitHub Copilot,” Inf. Technol. People, vol. 38, no. 4, pp. 1659–1685, May 2025.
8. J. Austin et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, Aug. 2021.
9. E. Dehaerne, B. Dey, S. Halder, S. De Gendt, and W. Meert, “Code generation using machine learning: A systematic review,” IEEE Access, vol. 10, pp. 82434–82455, Aug. 2022.
10. A. Margolis, “Model geometries of finitely generated groups,” arXiv preprint arXiv:2207.10509, Jul. 2022.
11. M. Barenkamp, J. Rebstadt, and O. Thomas, “Applications of AI in classical software engineering,” AI Perspect., vol. 2, no. 1, p. 1, Jul. 2020.
12. F. Alet et al., “A large-scale benchmark for few-shot program induction and synthesis,” in Proc. Int. Conf. Mach. Learn. (ICML), 2021, pp. 175–186.
13. Y. Li et al., “Competition-level code generation with AlphaCode,” Science, vol. 378, no. 6624, pp. 1092–1097, Dec. 2022.
14. M. Valový and A. Buchalcevova, “The psychological effects of AI-assisted programming on students and professionals,” in Proc. IEEE Int. Conf. Softw. Maintenance Evol. (ICSME), Oct. 2023, pp. 385–390.
15. P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Proc. CHI Conf. Human Factors Comput. Syst. Extended Abstracts (CHI EA), Apr. 2022, pp. 1–7.
16. P. Patel and E. F. Melcer, “Exploring student developers’ trust, adoption, and usage of AI-powered coding assistants for web accessibility compliance,” in Proc. 18th ACM Int. Conf. Pervasive Technol. Relat. Assistive Environ. (PETRA), Jun. 2025, pp. 298–301.
17. A. Sergeyuk, I. Zakharov, E. Koshchenko, and M. Izadi, “Human-AI experience in integrated development environments: A systematic literature review,” arXiv preprint arXiv:2503.06195, Mar. 2025.
18. D. Lo, “Trustworthy and synergistic artificial intelligence for software engineering: Vision and roadmaps,” in Proc. IEEE/ACM Int. Conf. Softw. Eng.: Future Softw. Eng. (ICSE-FoSE), May 2023, pp. 69–85.
19. H. Touvron et al., “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, Feb. 2023.
20. J. Achiam et al., “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, Mar. 2023.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
