Details
| Original language | English |
|---|---|
| Article number | 107737 |
| Journal | Information and Software Technology |
| Volume | 183 |
| Early online date | 8 Apr 2025 |
| Publication status | E-pub ahead of print - 8 Apr 2025 |
Abstract
Context: With the integration of generative artificial intelligence (GenAI) tools such as GitHub Copilot into development processes, developers can be supported when writing code.
Objectives: As GitHub Copilot has a feature to provide up to ten solutions at once, we explore, how developers should approach those solutions with the goal of providing recommendations to achieve suitable trade-offs in finding correct solutions and checking solutions.
Methods: In this study, we analyze a total of 2025 coding problems provided by LeetCode and 17 048 solutions to solve these problems generated by GitHub Copilot in Python. We focus on three key issues: firstly, whether it is beneficial to consider multiple solutions; secondly, the impact of the position of a solution; and thirdly, the number of solutions that should be checked by a developer.
Results: Overall, our results point to the following observations: (1) solutions are not less likely to be correct if they appear at later positions; (2) when looking for a solution to a common problem, checking four to five solutions is generally enough; (3) novel or difficult problems are unlikely to be solved by GitHub Copilot; (4) skipping the first solution is advised when considering only one solution, as the first solution is less likely to be correct; and (5) checking all solutions is necessary to not miss correct solutions, but the effort is usually not justified.
Conclusion: Based on our study, we conclude that there is potential for improvement in better supporting developers. For instance, there are few cases where ten generated solutions provide more value than fewer solutions. Depending on the use scenario, it could be more useful if GitHub Copilot allowed developers to request a single, comprehensive solution.
Keywords
- Code generation, Empirical study, GenAI, Generative AI, GitHub Copilot
ASJC Scopus subject areas
- Computer Science(all)
- Software
- Computer Science(all)
- Information Systems
- Computer Science(all)
- Computer Science Applications
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: Information and Software Technology, Vol. 183, 107737, 07.2025.
Research output: Contribution to journal › Article › Research › peer review
}
TY - JOUR
T1 - Don't settle for the first! How many GitHub Copilot solutions should you check?
AU - Oertel, Julian
AU - Klünder, Jil
AU - Hebig, Regina
N1 - Publisher Copyright: © 2025 The Authors
PY - 2025/4/8
Y1 - 2025/4/8
N2 - Context: With the integration of generative artificial intelligence (GenAI) tools such as GitHub Copilot into development processes, developers can be supported when writing code. Objectives: As GitHub Copilot has a feature to provide up to ten solutions at once, we explore, how developers should approach those solutions with the goal of providing recommendations to achieve suitable trade-offs in finding correct solutions and checking solutions. Methods: In this study, we analyze a total of 2025 coding problems provided by LeetCode and 17 048 solutions to solve these problems generated by GitHub Copilot in Python. We focus on three key issues: firstly, whether it is beneficial to consider multiple solutions; secondly, the impact of the position of a solution; and thirdly, the number of solutions that should be checked by a developer. Results: Overall, our results point to the following observations: (1) solutions are not less likely to be correct if they appear at later positions; (2) when looking for a solution to a common problem, checking four to five solutions is generally enough; (3) novel or difficult problems are unlikely to be solved by GitHub Copilot; (4) skipping the first solution is advised when considering only one solution, as the first solution is less likely to be correct; and (5) checking all solutions is necessary to not miss correct solutions, but the effort is usually not justified. Conclusion: Based on our study, we conclude that there is potential for improvement in better supporting developers. For instance, there are few cases where ten generated solutions provide more value than fewer solutions. Depending on the use scenario, it could be more useful if GitHub Copilot allowed developers to request a single, comprehensive solution.
AB - Context: With the integration of generative artificial intelligence (GenAI) tools such as GitHub Copilot into development processes, developers can be supported when writing code. Objectives: As GitHub Copilot has a feature to provide up to ten solutions at once, we explore, how developers should approach those solutions with the goal of providing recommendations to achieve suitable trade-offs in finding correct solutions and checking solutions. Methods: In this study, we analyze a total of 2025 coding problems provided by LeetCode and 17 048 solutions to solve these problems generated by GitHub Copilot in Python. We focus on three key issues: firstly, whether it is beneficial to consider multiple solutions; secondly, the impact of the position of a solution; and thirdly, the number of solutions that should be checked by a developer. Results: Overall, our results point to the following observations: (1) solutions are not less likely to be correct if they appear at later positions; (2) when looking for a solution to a common problem, checking four to five solutions is generally enough; (3) novel or difficult problems are unlikely to be solved by GitHub Copilot; (4) skipping the first solution is advised when considering only one solution, as the first solution is less likely to be correct; and (5) checking all solutions is necessary to not miss correct solutions, but the effort is usually not justified. Conclusion: Based on our study, we conclude that there is potential for improvement in better supporting developers. For instance, there are few cases where ten generated solutions provide more value than fewer solutions. Depending on the use scenario, it could be more useful if GitHub Copilot allowed developers to request a single, comprehensive solution.
KW - Code generation
KW - Empirical study
KW - GenAI
KW - Generative AI
KW - GitHub Copilot
UR - http://www.scopus.com/inward/record.url?scp=105002640406&partnerID=8YFLogxK
U2 - 10.1016/j.infsof.2025.107737
DO - 10.1016/j.infsof.2025.107737
M3 - Article
AN - SCOPUS:105002640406
VL - 183
JO - Information and Software Technology
JF - Information and Software Technology
SN - 0950-5849
M1 - 107737
ER -