Embracing the Future with AI

Embracing the Future

Our Journey with AI-Assisted Development

“I think it gave me confidence... feel good about taking tasks and not feeling like I was engineer

"I love Copilot. It helps complex

"It has allowed me to focus a lot more on hand by responding to queries

"It has helped me stay focused on the task, advice and code completion as I

“I have been able to write tests quicker than before I was using Copilot.”

“It has enhanced root cause analysis. When investigating issues, I can quickly identify potential problem areas and find solutions.”

“It's suggesting lots of edge cases for my unit tests which I normally don't think of.”

"It quickly gets an idea of what I want to do."

confidence... in that I was able to taking on more challenging was bugging the senior engineer all the time.”

helps me a lot coding complex functions."

on the task at in the IDE."

task, getting work." than Copilot.”

These quotes are just some of the enthusiastic responses from participants of our comprehensive trial of AI assisted development tools, conducted inside Version 1 between May and July of this year.

AI, both as disruptor and everyday tool, has begun to have an influence on many diverse job roles and disciplines - software development is no exception. While new innovations in this area continue to evolve, the predominant utilisation for now are AI Assisted Development tools which deeply integrate into developer toolsets to create software applications such as Microsoft Visual Studio and JetBrains IntelliJ. Rather than seeking to replace developers, these AI assistants aim to augment their capabilities, boost productivity and enhance job satisfaction.

The most well known in this area is GitHub Copilot, an enterprise grade tool that aids with bug fixing, code explanations and real-time coding suggestions. With over 1.8 million paying customers, it markets itself as “Your AI Pair Programmer” and studies by GitHub claim that 88% of developers who used the tool are more productive, 60% feel more fulfilled in their role and 73% have greater concentration/flow.*

Research from Gartner also highlights an undeniable trend - the cognitive demands on enterprise developers are escalating. We can see this in the growing complexity of codebases, the intricate web of dependencies to be managed on projects and the constant context switching between coding and sourcing information.

These challenges can disrupt developers' flow, affecting not just productivity but also their satisfaction with work.

Our expert teams are committed to innovation, delivering the best solutions for our customers and equipping our teams with the tools needed to do their jobs brilliantly.

Earlier this year, we decided to test AI supported development tools within our own delivery teams. Our primary objective was to conduct a thorough evaluation of the tools potential to enhance developer productivity and satisfaction within our operational environment.

In the context of AI assisted development tools, we aimed to:

Assess the impact of GitHub Copilot on reducing the cognitive load for developers

Measure the improvements in productivity and job satisfaction

Understand GitHub Copilot's potential to provide our software delivery teams with a competitive edge

Our initial, tightly controlled trial involved 52 participants across 12 customer projects over a threemonth period. We carefully selected participants to ensure we had people with varying levels of experience and a diverse mix of people from different locations, customers and technology landscapes (i.e. greenfield, brownfield). We also sought to limit the AI assisted tools to Github Copilot and Amazon Q Developer.

Finally, as a responsible user of AI, we sought permission from our customers before using the tool on their projects.

* (source: https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/)

Success with any tool hinges on knowing how to use it effectively, and this principle

guided our approach throughout the trial. To ensure participants were fully equipped, we provided each with a detailed training guide.

This covered not just the core functionality of the AI-assisted development tool, but also stepby-step instructions on installing it within their Integrated Developer Environment (IDE), whether they were using Visual Studio Code, JetBrains, or another platform.

The guide also included tailored examples illustrating how to use the tool in specific coding scenarios, such as refactoring legacy code, generating boilerplate, or automating repetitive tasks - alongside clear warnings about inappropriate use cases, such as over-reliance on the tool for complex algorithm design or security-sensitive code.

Throughout the trial, we hosted weekly drop-in clinics, where participants could discuss their experiences and raise any technical or workflow-related issues.

For instance, if developers encountered problems with the tool misinterpreting their intent or generating incorrect suggestions, our experts were on hand to provide real-time solutions and best practice advice. Additionally, we conducted one-on-one check-ins with key team members, providing targeted feedback on how they were using the tool in their specific projectswhether it was speeding up unit test generation or streamlining documentation writing.

In order to collect qualitative and quantitative feedback participants were asked to participate in regular surveys throughout the trial period. These surveys included questions about how they were using the tool, their satisfaction with the tool and the impact they observed on their productivity.

Key Findings

We saw strong engagement and personal commitment from our participants throughout the trial.

Over the course of the trial, we collected daily, weekly and monthly survey responses which allowed us to build a comprehensive picture of how our teams were using AI assisted development in their workflow.

We analysed the impact of the tool through the lens of the SPACE Framework which examines developer productivity across 5 different dimensions including Satisfaction and wellbeing, Performance, Activity, Communication and collaboration and Efficiency and flow. The trial confirmed that GitHub Copilot had a significant impact on developer productivity and satisfaction in each of these areas.

Performance:

90% of respondents to our surveys reported that the tool had a positive impact on their productivity.

79% of respondents to our weekly survey agreed (30% strongly, 49% agree) that using the tool allowed them to complete tasks faster. Participants were also asked the following survey question over the course of the trial: “Over the past week, on average, how many minutes per day have you saved by using AI assisted development?"

Satisfaction:

76% of respondents indicated that the tool helped them feel a greater sense of satisfaction in their work.

The tool helped them complete mundane and repetitive tasks more quickly, freeing them up to work on more challenging and satisfying work. In addition, 73% of respondents reported that they had a more enjoyable coding experience while using the tool.

Communication and Collaboration:

Developers reported using the tool to review their work and identify improvements, streamlining the code review process.

The tool's ability to explain code in simple terms also has the potential to enhance knowledge sharing within teams which is particularly beneficial for onboarding new team members or getting started with an unfamiliar codebase.

Trial participants reported they were able to work with the tool to identify areas of improvement in their code, with many learning new techniques and methods as a result. In addition, 72% agreed (23% strongly, 49% agreed) that after using AI assisted development they learned from the suggestions it showed them.

Responses

Responses to the statement "After using Copilot, I learn from the suggestions it shows me."

Efficiency and Flow:

79% of respondents reported completing tasks faster with the tool - proving particularly efficient in areas such as code generation, debugging, and unit test creation.

One of the most interesting observations was how it lowered the barrier to creating tests which directly impacts on code quality. One team of frontend developers reported that unit tests which previously took 2 hours could now be completed in just 15 minutes using the tool.

Others mentioned that it was also helpful in identifying edge cases for tests that they may not have considered previously. 76% of respondents agreed (28% strongly, 48% agreed) that they spent less time searching for information and 75% agreed (29% strongly, 46% agreed) that they spent less mental effort on repetitive tasks.

Activity:

The tool helped them complete mundane and repetitive tasks more quickly, freeing them up to work on more challenging and satisfying work.

In addition, 73% of respondents reported that they had a more enjoyable coding experience while using the tool.

While the overall feedback was overwhelmingly positive, we did observe some challenges and areas for further consideration:

Impact on Skill Development

While AI assisted development tools can enhance productivity and has shown itself to be an effective way to learn new skills, we need to be mindful of how this impacts skill development, especially for junior developers.

Many participants in our trial reported learning new coding patterns and best practices through the tools suggestions. However, ensuring that AI assistance complements rather than replaces the learning process will be a key consideration for maintaining sustainable software development practices.

Overreliance Risk

We don’t yet know the full effect of the long term use of these AI assisted coding tools. There is a risk that developers become overly dependent on AI-generated code without fully understanding it or without performing adequate peer review of the code.

Responsible usage of these tools must be borne in mind to minimise the potential for unintended negatives consequences.

Accuracy of Suggestions

The suggestions from AI assisted development tools are not always correct and we had reported instances of hallucinations where it confidently recommended code that was either incorrect or referenced non-existent features in a target framework.

For example, one participant noted, "Sometimes it would confidently predict something that would look correct but there was a subtle logic issue with it." As with any AI system, it does makes mistakes and the output must always be reviewed and verified for correctness. This reinforces the need for robust code review processes and the importance of training teams in how to use AI tools effectively and safely.

Summary

In conclusion, our trial of AI assisted development tools has revealed significant potential to enhance developer productivity, code quality and employee satisfaction.

The use of AI to streamline coding tasks, assist with problem-solving and facilitate learning were key highlights of the trial. However, as with any transformative technology, its adoption presents both opportunities and challenges. The key to maximising the benefits of AI-assisted development tools like GitHub Copilot or Amazon Q Developer lies in thoughtful implementation, ongoing education and a careful balance between utilising AI capabilities and maintaining core coding skills.

Going forward, we will need to monitor the long-term impacts of these tools, continuously refine our usage guidelines and adapt our development practices to ensure that we harness the power of AI while maintaining the highest possible standards of code quality and developer expertise that we exhibit today. By doing so, we can position ourselves at the forefront of the next generation of software development and continue to drive innovation and superior results for our customers.