Use Personalized AI Agents to Speed Up Software Development & Improve Code Quality [Testμ 2024]

AI code assistants are now a standard part of developers’ toolchains, but their effectiveness often comes into question. In this session, Eran Yahav, CTO of Tabnine, addresses these issues by offering practical tips on how to enhance AI tools. He explains how Retrieval-Augmented Generation (RAG) improves AI suggestions by incorporating external information, leading to more relevant recommendations.

He also highlights the importance of integrating AI with the existing codebase, such as through IDEs and code repositories, to boost performance. He discusses how to customize and fine-tune AI assistants for their specific needs, ensuring they are both effective and secure. The session provides valuable insights into optimizing AI tools for better productivity and code quality.

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the LambdaTest YouTube Channel.

AI’s Growing Impact on Software Development

Erhan opened the session by highlighting the significant shift in software development, where AI is beginning to play a pivotal role. He illustrated this by demonstrating how, in the near future, developers might simply point to a JIRA ticket in their IDE, and AI would autonomously generate a fully functional application—like a trading app—based on those requirements. While we’re not entirely there yet, we’re moving toward a reality where AI assists with an increasing number of tasks in the software development life cycle (SDLC).

He emphasized that AI is already delivering substantial productivity gains, ranging from 20% to 50%, by supporting various stages of the SDLC. Currently, human engineers drive the process, with AI acting as an assistant across different steps such as coding, testing, code review, and deployment. However, the future holds a more streamlined process involving three key phases: ideation, generation, and validation.

In this evolving landscape, the human engineer collaborates with AI during the ideation phase, defining designs and requirements. The generation phase sees AI taking over the code creation entirely, leaving humans to focus on refining tasks. The final phase—validation—becomes crucial as the bottleneck in ensuring software quality. As AI-generated code scales rapidly, so must the validation tools and processes, making the role of quality assurance more vital than ever.

Delegating Tasks to AI: The Fundamental Theorem of Generativity

Continuing from his initial discussion, Erhan delved deeper into the concept of delegating tasks to AI within the SDLC, framing it as a “delegation problem.” He introduced what he called the “Fundamental Theorem of Generativity.” This theorem suggests that it only makes sense to delegate a task to an AI agent if the combined cost of instructing the agent and verifying its output is significantly lower than the cost of doing the task manually.

The graphic shared by @yahave is enough to prove that AI is instrumental in realising tasks at every stage of the SDLC lifecycle! pic.twitter.com/vSlPUwDKoE

— LambdaTest (@lambdatesting) August 21, 2024

Erhan explained that for AI to be truly useful in the SDLC, two critical aspects need to be addressed:

• Efficient Communication: The process of telling the AI agent what you want and how you want it must be straightforward and time-efficient. If it takes too long to communicate these instructions, the benefits of using AI diminish.
• Trust in Results: Once the AI completes the task, its output must be reliable in terms of security, compliance, and performance. If there’s any doubt about the quality or accuracy of the AI’s work, developers may prefer to handle the task manually.

However, AI test agents like KaneAI by LambdaTest addresses these challenges head-on by simplifying the creation, debugging, and management of tests through natural language commands.

KaneAI is a smart AI test agent, guiding teams through the entire testing lifecycle. From creating and debugging tests to managing complex workflows, KaneAI uses natural language to simplify the process. It enables faster, more intuitive test automation, allowing teams to focus on delivering high-quality software with reduced manual effort.

Building Trust Through Human-AI Collaboration

Erhan expanded on the concept of trust in AI-assisted software development by emphasizing the importance of “tight loop integration” between the human engineer and the AI assistant.

He described this collaboration as a continuous feedback loop where the human provides lightweight specifications, and the AI generates code based on those inputs. The human engineer then reviews the generated code, applying their judgment to decide whether to accept, modify, or reject it. He highlighted two key points on Human-AI collaboration:

• Lightweight Specification: Erhan explained that the specification process in AI-assisted development is often minimal. It can involve simple actions like typing a small comment, code completion as you type, or brief instructions in a chat, which the AI then uses to generate a portion of the code.
• Human Review: Once the AI generates the code, typically in small chunks (e.g., 50-200 lines), the human engineer reviews it. This review process is crucial as it involves applying common sense and technical judgment to determine whether the generated code is suitable for integration into the project.

Erhan provided a practical example to illustrate this process. Suppose a developer writes a function signature, such as get_tweet. The AI might offer two possible completions: one using a well-known library like Tweepy, and another performing an HTTP request directly using the Requests library. The human engineer then evaluates these options, considering factors like the reliability and quality of the libraries involved, and makes a decision based on their judgment.

Enhancing Code with AI—Human Judgment Remains Key

Erhan continued by discussing scenarios where a developer might ask the AI to improve or refactor existing code. For instance, a simple function that converts temperatures from Fahrenheit to Celsius could be refactored by the AI to include a common conversion function, making the code more concise and maintainable.

For instance, Erhan presented the example of an AI-generated refractor where he replaced repetitive code with a common function. However, he acknowledged that in many cases, the decision to accept AI-generated refactoring may come down to personal preference or specific project requirements.

Erhan emphasized that while AI can significantly assist in enhancing code, the final judgment always rests with the human developer, ensuring that the code meets the project’s standards and requirements. This balance between AI assistance and human oversight is essential for maintaining trust in the development process.

Building Trust with AI Engineers: Strategies and Risks

In this segment, Erhan dived into the critical aspects of establishing trust in AI engineers, highlighting three key strategies that can help organizations confidently integrate AI into their software development processes.

If done correct, AI engineer will always bag “Employee of the month award” 👇🏼 pic.twitter.com/iEld7KKxzO

— LambdaTest (@lambdatesting) August 21, 2024

Personalization—Onboarding the AI Engineer

Erhan emphasized that for an AI engineer to be effective, it must be thoroughly personalized to understand the company’s specific code base and corporate standards. Without proper onboarding, an AI engineer might generate code that doesn’t align with the organization’s best practices, leading to issues such as:

• Incompatible Code Practices: The AI might use libraries or frameworks that are not standard within the company, or it could bypass crucial mechanisms like logging and authentication.
• Duplication of Efforts: Instead of utilizing existing libraries or microservices, an AI might regenerate code, creating unnecessary redundancy and technical debt.

Control Over AI Output

The second crucial strategy is ensuring that those responsible for quality—whether they are architects, development managers, or test engineers—have full control over the AI’s output. This control allows them to:

• Define AI Behavior: The quality leads must be able to set the scope of tasks the AI can perform and dictate the behaviors and decisions it should follow.
• Monitor and Adjust: They should also have the ability to review and adjust the AI-generated code, ensuring it meets the required standards before integration.

Increasing AI Autonomy with Clear Boundaries

The final strategy Erhan discussed involves gradually increasing the autonomy of AI engineers while maintaining clear delegation boundaries. The goal is to make it easy for developers to:

• Specify Tasks Clearly: Developers should be able to communicate tasks to the AI in a straightforward manner.
• Efficiently Review Results: The AI should return results that are easy to review and integrate, even as it takes on more complex and autonomous roles.

By successfully implementing these strategies, Erhan suggested that the AI engineer could effectively become the “employee of the month” consistently, knowing the codebase, adhering to company standards, and producing high-quality output that requires minimal oversight.

Specific Tasks for AI in the Software Development Life Cycle

Erhan also addressed a question regarding which tasks in the SDLC can be effectively handled by AI to save time and improve accuracy. He identified two primary areas:

• Unit Test Generation: AI is particularly suited for generating unit tests, where the risk of introducing bad code into production is low since the generated code is for testing purposes only.
• Automated Code Reviews: AI can automate many aspects of code reviews, particularly the more mundane tasks, providing insights that go beyond what traditional linters or static analysis tools offer. AI-driven code reviews tend to have lower false positive rates and can identify issues that might otherwise be overlooked.

Potential Risks of Relying on AI for Code Generation

Erhan acknowledged that there are risks associated with relying on AI for code generation, especially when human oversight is not sufficiently rigorous. These risks include:

• Quality Risks: If human engineers accept AI-generated code without a thorough review, there is a risk of introducing subpar or even faulty code into the production environment.
• Duplication and Technical Debt: AI might generate duplicate or redundant code if it is not properly onboarded and familiarized with the existing codebase, leading to unnecessary technical debt.

To mitigate these risks, Erhan highlighted the importance of maintaining tight human-AI integration and ensuring that the AI is personalized and well-controlled within the organization.

The Role of Generation and Validation Agents in Enhancing AI Efficiency

In discussing the future of AI in software development, Erhan emphasized a crucial separation within the system: the roles of generation and validation agents.

Here’s a closer look at these components:

• Generation Agents: These agents are responsible for creating code or unit tests. They automate the production of new code, handling tasks like coding and generating tests based on given specifications.
• Validation Agents: Their role is to review and validate the output from generation agents. They also ensure the generated code meets standards for security, compliance, performance, and quality.
• The Context Engine: The context engine gathers data from various sources to provide a comprehensive understanding of the organization. It offers visibility similar to that of a human engineer, utilizing historical support tickets, stack traces, and best practices to inform the validation process.
• Control Layer: This layer allows users to set parameters and guidelines for the AI agents. It further defines best practices, standards, and policies for what the validation agents should check, ensuring that the AI operates within defined boundaries and maintains quality.

As AI-generated code scales, so must the validation processes. This balance between generation and validation is critical for maintaining high standards and efficiency in software development.

The Evolving Role of Test Generation Agents in AI-Driven Development

Erhan shared some valuable insights into how test-generation agents are transforming the software development landscape.

Here’s a summary of what he discussed:

What Test Generation Agents Do

Here is how test generation agents work:

1. Planning First: These agents start by creating a test plan. They map out what needs to be tested before diving into the actual test creation.
2. Generating Tests: After the plan is set, the agent generates the individual unit tests based on that plan.
3. Tailored to Your Needs: The agents are personalized to understand your existing test setup, allowing them to adjust the test plan as needed.

What They’re Good At

Here are some of the benefits of test-generation agents:

• Boosting Coverage: Test agents are great for increasing your test coverage quickly. They can take you from a modest 20% coverage to 80% with minimal effort.
• Handling Edge Cases: They can come up with unexpected test cases that might not be obvious to human testers, like handling future dates in a date-of-birth test scenario.
• Simple Integration Tests: For straightforward integration tests with clear scenarios, these agents can handle the task effectively.

Challenges and What’s Next

The challenges associated with test generation agents include:

• Validating Tests: One challenge is ensuring the tests generated are correct. Since the agents infer test requirements from the code itself rather than from higher-level specs, there can be gaps.
• Better Specifications: In the future, integrating high-level specs from tools like JIRA might help the agents generate tests that align better with defined requirements.

Erhan’s insights underscore the strengths of test generation agents in expanding test coverage and managing routine tasks while also highlighting the need for ongoing human oversight, especially for complex testing scenarios.

Building the Ideal AI Engineer: Erhan’s Wishlist

In his talk, Erhan laid out what he envisions as the ideal AI engineer.

Here’s a breakdown of the key qualities he hopes for when hiring an AI agent:

• Personalized Onboarding: The AI engineer should be tailored to fit seamlessly into the organization. Without proper onboarding, the AI won’t reach its full potential.
• Concise Communication: Effective interaction with the AI should be straightforward. Erhan wants the AI to communicate clearly and concisely, avoiding lengthy explanations or results.
• Inquisitive Nature: The AI should ask for clarification when needed. For instance, if given a task like implementing a calculator, it should seek further details if the request is ambiguous (e.g., whether a scientific or standard calculator is needed).
• Reliability: Trustworthiness is crucial. The AI engineer must consistently deliver dependable results and perform reliably in various tasks.

Erhan’s vision highlights the importance of a personalized, efficient, and reliable AI engineer who can communicate effectively and seek clarification when necessary.

The Four Pillars of Personalizing AI Engineers

Moving on with the session, Erhan highlighted the importance of personalizing AI engineers to ensure they effectively integrate and perform within an organization.

Here’s a closer look at the four key aspects of personalization:

• Context Awareness: Personalization begins with the AI engineer’s ability to understand and leverage existing work. This includes being familiar with everything on the local machine, such as existing code and documentation. The AI engineer needs to grasp both the problem domain it is working within and the solutions it is meant to provide.
• Connection: The AI engineer should connect with various sources of information within the organization. This means having access to organizational repositories, JIRA tickets, and other relevant resources. By connecting to these sources, the AI gains better context and visibility, which helps it perform its tasks more effectively.
• Coaching: Coaching involves setting explicit guidelines and instructions for the AI. Quality leads and test engineers can provide specific directives on coding standards and compliance. This ensures that the code generated adheres to the required guidelines and maintains high quality across the system.
• Customization: Customization allows for the adjustment and fine-tuning of AI models using the organization’s entire codebase as training data. This level of personalization ensures that the AI engineer is tailored to meet the specific needs and preferences of the organization, enhancing its effectiveness and reliability.

The Crucial Role of Context in AI Agent Performance

In his talk, Erhan emphasized the fundamental role context plays in the effectiveness of AI agents.

Here’s a detailed look at how context impacts AI performance and the challenges associated with managing it:

Understanding Context

Context is everything when working with AI agents. The quality of results from an AI agent improves dramatically when it is provided with appropriate and comprehensive context. This means that the AI should be aware of a wide range of information relevant to the task at hand.

Range of Context

The level of context an AI agent can access varies. At one end, the agent might have no context beyond the immediate task. At the other end, it could be fully contextualized with access to the entire organization’s information, including non-code sources like JIRA and Confluence. This range impacts how well the AI can understand and address requests.

When given a simple request, such as displaying a table of student scores, the AI agent considers a multitude of factors:

• The current file and other open files
• The current selection and errors
• Project metadata and git history
• Imported libraries and connected repositories
• Non-code sources of information

This broad awareness helps the AI deliver more accurate and relevant results, avoiding unnecessary duplication and technical debt.

Challenges with Context Windows

Language models, which power AI agents, have a limited context window and are measured in tokens. Typical context windows range from 8,000 to 200,000 tokens, with recent advancements aiming for up to 1 million tokens. However, as more information is added to the context window, latency increases, and the model’s accuracy can decrease.

The challenge lies in balancing and prioritizing the vast amounts of information available within the context window. For instance, when generating results, the AI agent must efficiently integrate and convey this information to produce accurate and useful outputs.

Erhan’s insights highlight the intricate balance required to manage context to maximize the effectiveness of AI agents. The better the AI is at contextualizing information, the more precise and useful its outputs will be.

Implementation Techniques

To handle context effectively, techniques such as Retrieval-Augmented Generation (RAG) are used. This involves:

• Writing a prompt
• Using a retrieval algorithm to consult local and remote repositories
• Creating an augmented prompt
• Generating results with a custom or universal model

The retrieval algorithm plays a crucial role in enhancing context awareness.

Coaching and Reliability

Coaching provides a control layer for AI agents, allowing users to set specific guidelines and rules.

For instance, during code reviews, tools like Tabnine can automatically flag issues based on predefined rules, such as ensuring rate limiting for server requests. Users can also add custom rules in natural language.

This approach ensures that AI agents operate effectively within the given context, improving their reliability and alignment with organizational standards.

Best Practices for Enhancing Code Quality

Erhan discussed how to effectively use AI agents to enforce coding standards and best practices:

• Coding Rules in Natural Language: Users can input coding guidelines in natural language, such as “Don’t connect to the database without using the appropriate connector.” This enables AI agents to understand and enforce specific do’s and don’ts.
• Golden Repository of Code: Providing a golden repository of code and expert solutions allows the AI to prioritize these best practices. This repository can include code snippets and examples of high-quality solutions.
• Integration and Flagging: The AI agent integrates seamlessly with source control systems. It flags deviations in pull requests, merge requests, and command line integrations. This helps ensure that the code aligns with established rules and practices.
• Custom Rules and Enforcement: Users can create custom rules in natural language to enforce organizational standards. For instance, you can set rules to use standard or internal libraries and prevent the use of outdated APIs. This customization helps maintain consistency and prevents new legacy issues.
• Preventing New Legacy Code: By guiding developers away from old libraries and APIs, the AI helps avoid the creation of legacy code, ensuring that new developments adhere to current best practices.

In a Nutshell…

Erhan’s session highlighted key aspects of leveraging AI agents to enhance code quality. He stressed the importance of context, noting that AI performance improves with rich contextual information from code bases and organizational tools like JIRA. The onboarding process involves careful selection of relevant systems to ensure effective operation and avoid legacy code.

The discussion also touched on balancing reliability, accuracy, and test coverage, with current efforts focusing on improving coverage. Overall, Erhan’s insights offer a path to using AI agents for better coding standards and higher-quality software development.

Time for Some Q&A

1. What are the two potential risks of relying on AI agents for code generation?

Erhan: Code generation with AI agents carries certain risks. Firstly, the quality of the generated code depends significantly on human judgment. If developers accept code without careful review, it can compromise the overall code quality. Secondly, AI agents might produce duplicate or similar pieces of code, leading to technical debt.

This redundancy can create refactoring challenges and clutter the codebase. While efforts are underway to mitigate these issues, it’s essential for users to be aware of these risks and apply diligent oversight when integrating AI into their development workflows.

2. Can AI agents assist in automating complex task scenarios or edge cases that are difficult to cover with traditional testing methods?

Erhan: AI agents excel at generating test data and can significantly boost test coverage quickly. They are particularly effective in taking coverage from a low to a high level by creating diverse and interesting test cases that might not be immediately obvious to humans. For instance, an AI agent might add tests for scenarios like a future date of birth, which a human might overlook.

However, these agents may not be as effective in addressing the final, complex edge cases that require nuanced human reasoning. Their strength lies in rapidly enhancing coverage rather than fine-tuning the last details of test scenarios.

3. How can I test agents such as validate the correctness of their generated tests?

Erhan: One challenge with generating tests from existing source code is that the AI agent often has to infer specifications rather than work from clear, high-level requirements. This can lead to gaps between what the code is expected to do and what the tests actually cover.

However, by integrating with tools like JIRA to access detailed specifications and descriptions, future AI agents will be better able to align tests with defined test scenarios, bridging this gap and ensuring that tests accurately reflect the intended specifications.

4. How do you find a good balance between reliability, accuracy and state of the test?

Erhan: Currently, the focus is on generating tests to improve coverage rather than optimizing test latency. The team is aware of the importance of considering latency but has not yet integrated this factor into the process. This is a valuable point for future consideration, and they are open to exploring how to address it.

Please don’t hesitate to ask questions or seek clarification within the LambdaTest Community.