Report: ChatGPT-5 Coding Gains Come at a Higher Cost
A recent report by Sonar highlights the capabilities and challenges associated with OpenAI's GPT-5 platform. The platform is noted for its improved code generation capabilities, albeit at an increased cost.
A recent report by Sonar highlights the capabilities and challenges associated with OpenAI's GPT-5 platform. The platform is noted for its improved code generation capabilities, albeit at an increased cost.
The report analyzed over 4,400 Java tasks, revealing that the quality of code, particularly regarding vulnerabilities, improves with higher reasoning levels offered by OpenAI. However, this enhancement comes with a substantial increase in the volume of code produced per task, presenting additional maintenance challenges for developers.
The minimal edition of GPT-5 generates over twice the lines of code compared to the previous GPT-4o edition.
While higher reasoning levels reduce common vulnerabilities such as path-traversal and injection attacks, they introduce more subtle and harder-to-detect flaws. For example, vulnerabilities related to inadequate I/O error-handling increase from 30% in minimal reasoning mode to 44% in high reasoning mode.
A recent report by Sonar highlights the capabilities and challenges associated with OpenAI's GPT-5 platform.
The analysis shows a decrease in fundamental control-flow mistake bugs with increased reasoning. However, the incidence of advanced concurrency/threading bugs rises from 20% in minimal mode to approximately 38% in high mode.
Pricing and Operational Considerations
There is a cost associated with the different reasoning levels: $22 per month per developer for minimal level reasoning and $189 per month per developer for the highest level. Organizations are advised to evaluate the quality of the output against these costs to make informed decisions.
The report builds on a previous assessment comparing LLMs from OpenAI, Anthropic, and Meta. It was found that while these models can effectively translate code concepts and provide solutions for defined problems, they also introduce critical flaws such as hard-coded credentials and path-traversal injections.
There remains uncertainty regarding the extent to which DevOps teams trust and adopt AI coding tools for production environments, despite noted productivity gains.
Based on reporting by devops.com.