Research Study: Comparing GenAI Coding Tools
GenAI is a paradigm shift for software development. But most of the claims around AI improving developer productivity today are qualitative — based on surveys and anecdotes. Measuring the full impact of GenAI can be difficult or impossible without empirical data.
The goal of this research study is to compare GenAI coding tools using quantitative data. The study will evaluate the impact of AI-assisted development across a range of key metrics, including productivity, velocity, and quality. Data will be aggregated and standardized, and compared with global benchmarks.
There’s no cost to participate, and data collection is highly automated with minimal configuration required.
Benefits of Participating
Participants will get access to empirical data they can use to select or improve their GenAI tooling, including:
- Top GenAI tools across key productivity metrics
- Productivity impact of their specific GenAI tools
- Global reference data and benchmarks
All participants will receive the final report at no cost.
How to Participate
Development teams must use GitHub, GitLab, Bitbucket, or Azure DevOps. The study supports the use of any GenAI coding tools, including:
Security
Data collection and analysis are securely managed by Software.com. Software.com does not access or read source code and is SOC 2 Type 2 compliant. A SOC 2 Type 2 report, latest security and policy documents, continuous monitoring status, and list of subprocessors can all be found in the Trust Center.
A Security Policy and Privacy Policy are also available for reference.