A Review of Research on the Development of Artificial Intelligence Programming Agent-Assisted Software

Main Article Content

Yuzhe Li
School of Microelectronics, Hefei University of Technology, Hefei 230601, China

DOI: https://doi.org/10.70267/cai.v3n4.127145

Keywords

artificial intelligence programming, large language models, programming agents, software engineering, code generation

Abstract

The development of large language models has enabled AI-assisted programming to gradually expand from early code completion to requirements understanding, code generation, testing, debugging, and project maintenance. The programming intelligence agent formed on this basis is no longer just returning a piece of code based on user input, but can read project files, break down tasks, call terminals and testing tools, and repeatedly modify the program based on the running results. Based on 60 publicly searchable representative papers, this paper adopts a structured narrative review method to review the research progress from the aspects of code-based models, agent control mechanisms, repository-level systems, evaluation benchmarks, and human-computer collaboration evidence. It also compares the research scope and analysis focus with existing reviews on software engineering agents. This paper further utilizes method classification, performance statistics, and mechanism diagrams to analyze different technical approaches, focusing on issues such as reliability, contextual understanding in large-scale projects, security and privacy, evaluation and reproducibility, operating costs, and accountability and governance. Research shows that programming agents can reduce the cost of repetitive coding and help users lacking professional programming experience complete prototype development. However, its efficiency gains are significantly affected by task size, project familiarity, quality standards, and manual review costs, making it difficult to consistently understand requirements and guarantee overall quality in complex projects. Future research should shift from simply pursuing the quantity of code generated to building verifiable, explainable, traceable, and collaborative software development processes.

Abstract 11 | PDF Downloads 5

References

  • [1] Chen, M., Tworek, J., Jun, H., et al. (2021). Evaluating large language models trained on code. arXiv. https://arxiv.org/abs/2107.03374.
  • [2] Austin, J., Odena, A., Nye, M., et al. (2021). Program synthesis with large language models. arXiv. https://arxiv.org/abs/2108.07732.
  • [3] Hou, X., Zhao, Y., Liu, Y., et al. (2024). Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8), 1–79.
  • [4] Jin, H., Huang, L., Cai, H., et al. (2024). From LLMs to LLM-based agents for software engineering: A survey of current, challenges and future. arXiv. https://arxiv.org/abs/2408.02479.
  • [5] Liu, J., Wang, K., Chen, Y., et al. (2024). Large language model-based agents for software engineering: A survey. arXiv. https://arxiv.org/abs/2409.02977.
  • [6] Wang, Y., Zhong, W., Huang, Y., et al. (2024). Agents in software engineering: Survey, landscape, and vision. arXiv. https://arxiv.org/abs/2409.09030.
  • [7] Jimenez, C. E., Yang, J., Wettig, A., et al. (2024). SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR).
  • [8] Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv. https://arxiv.org/abs/2302.06590.
  • [9] Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv. https://arxiv.org/abs/2507.09089.
  • [10] Dong, Y., Jiang, X., Jin, Z., et al. (2025). A survey on code generation with LLM-based agents. arXiv. https://arxiv.org/abs/2508.00083.
  • [11] Wang, H., et al. (2025). AI agentic programming: A survey of techniques, challenges, and opportunities. arXiv. https://arxiv.org/abs/2508.11126.
  • [12] Feng, Z., Guo, D., Tang, D., et al. (2020). CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1536–1547).
  • [13] Guo, D., Ren, S., Lu, S., et al. (2021). GraphCodeBERT: Pre-training code representations with data flow. In International Conference on Learning Representations (ICLR).
  • [14] Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K. W. (2021). Unified pre-training for program understanding and generation. In Proceedings of NAACL-HLT (pp. 2655–2668).
  • [15] Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021). CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of EMNLP (pp. 8696–8708).
  • [16] Nijkamp, E., Pang, B., Hayashi, H., et al. (2023). CodeGen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations (ICLR).
  • [17] Fried, D., Aghajanyan, A., Lin, J., et al. (2023). InCoder: A generative model for code infilling. In International Conference on Learning Representations (ICLR).
  • [18] Li, Y., Choi, D., Chung, J., et al. (2022). Competition-level code generation with AlphaCode. Science, 378(6624), 1092–1097.
  • [19] Wang, Y., Le, H., Gotmare, A., et al. (2023). CodeT5+: Open code large language models for code understanding and generation. In Proceedings of EMNLP (pp. 1069–1088).
  • [20] Rozière, B., Gehring, J., Goyal, F., et al. (2023). Code Llama: Open foundation models for code. arXiv. https://arxiv.org/abs/2308.12950.
  • [21] Li, R., Allal, L. B., Zi, Y., et al. (2023). StarCoder: May the source be with you! arXiv. https://arxiv.org/abs/2305.06161.
  • [22] Allal, L. B., Muennighoff, N., Kocetkov, D., et al. (2023). SantaCoder: Don’t reach for the stars! arXiv. https://arxiv.org/abs/2301.03988.
  • [23] Lozhkov, A., Li, R., Allal, L. B., et al. (2024). StarCoder 2 and The Stack v2: The next generation. arXiv. https://arxiv.org/abs/2402.19173.
  • [24] Guo, D., Zhu, Q., Yang, D., et al. (2024). DeepSeek-Coder: When the large language model meets programming—The rise of code intelligence. arXiv. https://arxiv.org/abs/2401.14196.
  • [25] Yao, S., Zhao, J., Yu, D., et al. (2023). ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  • [26] Shinn, N., Cassano, F., Gopinath, A., et al. (2023). Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (Vol. 36, pp. 8634–8652).
  • [27] Yao, S., Yu, D., Zhao, J., et al. (2023). Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems (Vol. 36, pp. 11809–11822).
  • [28] Schick, T., Dwivedi-Yu, J., Dessì, R., et al. (2023). Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (Vol. 36, pp. 68539–68551).
  • [29] Parisi, A., Zhao, Y., & Fiedel, N. (2022). TALM: Tool augmented language models. arXiv. https://arxiv.org/abs/2205.12255.
  • [30] Qin, Y., Liang, S., Ye, Y., et al. (2024). ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations (ICLR).
  • [31] Liu, X., Yu, H., Zhang, H., et al. (2024). AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations (ICLR).
  • [32] Wu, Q., Bansal, G., Zhang, J., et al. (2023). AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv. https://arxiv.org/abs/2308.08155.
  • [33] Hong, S., Zhuge, M., Chen, J., et al. (2024). MetaGPT: Meta programming for multi-agent collaborative framework. In International Conference on Learning Representations (ICLR).
  • [34] Qian, C., Cong, X., Yang, C., et al. (2024). ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (pp. 15174–15186).
  • [35] Li, G., Hammoud, H. A. A. K., Itani, H., et al. (2023). CAMEL: Communicative agents for mind exploration of large scale language model society. In Advances in Neural Information Processing Systems (Vol. 36, pp. 51991–52008).
  • [36] Yang, J., Jimenez, C. E., Wettig, A., et al. (2024). SWE-agent: Agent-computer interfaces enable automated software engineering. arXiv. https://arxiv.org/abs/2405.15793.
  • [37] Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024). Agentless: Demystifying LLM-based software engineering agents. arXiv. https://arxiv.org/abs/2407.01489.
  • [38] Zhang, Y., Raghothaman, M., Liu, Y., et al. (2024). AutoCodeRover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 1592–1604).
  • [39] Wang, X., Li, B., Song, Y., et al. (2024). OpenHands: An open platform for AI software developers as generalist agents. arXiv. https://arxiv.org/abs/2407.16741.
  • [40] Liu, T., Xu, C., & McAuley, J. (2023). RepoBench: Benchmarking repository-level code auto-completion systems. arXiv. https://arxiv.org/abs/2306.03091.
  • [41] Ding, Y., Wang, Z., Ahmad, W. U., et al. (2023). CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion. arXiv. https://arxiv.org/abs/2310.11248.
  • [42] Zhang, F., Chen, B., Zhang, Y., et al. (2023). RepoCoder: Repository-level code completion with iterative retrieval and generation. In Proceedings of EMNLP (pp. 2471–2484).
  • [43] Guo, D., Xu, C., Du, N., et al. (2023). LongCoder: A long-range pre-trained language model for code completion. In Proceedings of the 40th International Conference on Machine Learning (pp. 12098–12107).
  • [44] Pearce, H., Ahmad, B., Tan, B., et al. (2022). Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (pp. 754–768).
  • [45] Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). Do users write more insecure code with AI assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (pp. 2785–2799).
  • [46] Khoury, R., Avila, A. R., Brunelle, J., & Camara, B. M. (2023). How secure is code generated by ChatGPT? In 2023 IEEE International Conference on Systems, Man, and Cybernetics (pp. 2445–2451).
  • [47] Sandoval, G., Pearce, H., Noor, N., et al. (2023). Lost at C: A user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium (pp. 2205–2222).
  • [48] Hendrycks, D., Basart, S., Kadavath, S., et al. (2021). Measuring coding challenge competence with APPS. In Advances in Neural Information Processing Systems (Vol. 34, pp. 12674–12687).
  • [49] Lai, Y., Li, C., Wang, Y., et al. (2023). DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning (pp. 18319–18345).
  • [50] Cassano, F., Gou, A., Nguyen, M., et al. (2023). MultiPL-E: A scalable and extensible approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7), 3675–3691.
  • [51] Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023). Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems (Vol. 36, pp. 21558–21572).
  • [52] Muennighoff, N., Liu, Q., Zhuo, T. Y., et al. (2023). OctoPack: Instruction tuning code large language models. arXiv. https://arxiv.org/abs/2308.07124.
  • [53] Jain, N., Han, K., Gu, A., et al. (2024). LiveCodeBench: Holistic and contamination-free evaluation of large language models for code. arXiv. https://arxiv.org/abs/2403.07974.
  • [54] Zhuo, T. Y., Vu, M. C., Chim, J., et al. (2024). BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. arXiv. https://arxiv.org/abs/2406.15877.
  • [55] Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022). Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1–7).
  • [56] Barke, S., James, M. B., & Polikarpova, N. (2023). Grounded Copilot: How programmers interact with code-generating models. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1–27.
  • [57] Ziegler, A., Kalliamvakou, E., Li, X. A., et al. (2022). Productivity assessment of neural code completion. arXiv. https://arxiv.org/abs/2205.06537.
  • [58] Nguyen, N., & Niu, S. (2022). An empirical study of GitHub Copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (pp. 1–5).
  • [59] Dakhel, A. M., Majdinasab, V., Nikanjam, A., et al. (2023). GitHub Copilot AI pair programmer: Asset or liability? Journal of Systems and Software, 203, 111734.
  • [60] Yetiştiren, B., Özsoy, I., & Tüzün, E. (2022). Assessing the quality of GitHub Copilot’s code generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (pp. 62–71).

Article Sidebar

PDF
Published
Sep 19, 2026
Issue
Vol. 3 No. 4 (2026)
Section
Research Articles
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

Li, Y. (2026). A Review of Research on the Development of Artificial Intelligence Programming Agent-Assisted Software. Computers and Artificial Intelligence, 3(4), 127-145. https://doi.org/10.70267/cai.v3n4.127145