Revolutionizing API Testing and Development with AI [Testμ 2024]

In today’s tech landscape, AI is transforming API testing and development. In this session, our renowned speaker, Pooja Mistry, Developer Advocate at Postman, explores Postman’s advanced AI and automation techniques to enhance workflows.

This topic delves into how AI can improve testing strategies, automate repetitive tasks, and boost efficiency. She suggests leveraging the Postman AI toolkit for game-changing testing and documentation.

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the LambdaTest YouTube Channel.

Pooja’s Journey from Manual QA to Becoming a Postman Advocate

Pooja began her session by sharing her journey as a QA engineer, starting in 2015 with manual UI testing, which was challenging due to time-consuming cycles, frequent outages, and evolving product requirements. Her team struggled with broken tests and limited coverage despite attending workshops to transform manual testers into developers. The breakthrough came with Postman, which empowered both manual and automation teams, significantly improving their testing processes and marking a turning point in their efforts.

Pooja and her team discovered the power of Postman while enhancing their testing processes. Initially popular among manual testers for hitting APIs, Postman quickly became a favorite as it allowed them to organize tests, create scripts, and use automation tools like the collection runner. This led to a significant increase in test coverage and collaboration.

Pooja’s success with Postman sparked a deep interest, eventually leading her to become a Developer Advocate at the company in 2022. Today, Postman is used by over 35 million developers globally, supporting the entire API lifecycle. The annual “State of the API” report, with insights from 40,000 professionals, highlights AI’s potential to boost developer productivity by 25%. Pooja is excited to share how Postman is incorporating AI to enhance testing and development.

The Critical Shift from Manual to Automated API Testing

Pooja emphasized the ongoing challenge of manual testing in API development. Despite advancements, a significant portion of time is still dedicated to manual testing and debugging, especially in larger organizations. She pointed out that companies with more than ten developers spend around 37% of their time writing APIs, but this drops as manual testing demands increase.

Pooja highlighted the inverse relationship between company size and the time spent on manual tasks, underscoring the need to boost productivity by automating API testing. While manual testing offers control, at scale, automation is crucial for enhancing efficiency and coverage. Pooja urged teams to empower testers to shift towards API automation to significantly improve productivity and streamline the development process.

Exploring API Testing Methods and the Role of AI

Pooja then delved into the various API testing methods available, highlighting manual testing for exploratory development and debugging, unit testing with BDD assertions and contract validations, integration testing with end-to-end scenarios, regression testing, and performance testing like load, stress, and endurance spikes. These methods address different aspects of the software architecture landscape.

She shared insights from the State of the API report, noting that the majority of tests written are functional and integration tests, which focus on how an API functions and the data it returns. Pooja emphasized the growing role of AI in enhancing productivity in these areas, showcasing its potential to streamline and improve testing processes.

The Importance of a Testing Plan in API Testing

Pooja emphasized the critical role of starting with a solid testing plan, whether performing manual or automated API testing. She outlined three essential steps in the API testing process. First, it’s crucial to understand the API requirements—its purpose, functionality, expected results, target consumers, and how it fits into the application’s workflow.

Next, she advised specifying the inputs and outputs of the API, such as headers and body types for inputs, and status codes and data validation for outputs. Finally, she recommended organizing tests based on similar functionalities. For example, grouping happy path tests separately from negative path tests or organizing tests by different API endpoint functionalities ensures a more structured and effective testing process.

Further, Pooja discussed the essential aspects of API testing, emphasizing what should be tested to ensure robust and reliable APIs. Given the complexity and number of APIs, she highlighted several critical areas: verifying correct status code responses, testing error conditions like invalid inputs, missing parameters, and content type headers, and ensuring that authentication and authorization are properly enforced.

Additionally, she stressed the importance of validating data accuracy and consistency with expected values. These key testing areas form the foundation of a typical API testing workflow, ensuring that APIs function as intended under various conditions.

The API Testing Workflow: From Preparation to Continuous Monitoring

Pooja explained the comprehensive process of API testing, emphasizing that it is a systematic workflow. This workflow begins with preparing tests, which includes understanding API requirements, building and organizing API requests for various endpoint types like REST, GraphQL, gRPC, and SOAP APIs. With a solid test plan in place, the next step is authoring tests. Traditionally, this meant setting up an endpoint in Postman and manually sending requests. However, with advancements in AI, much of the test script creation can now be automated, making the process more efficient.

Running tests is the next critical phase, where tools like Postman’s Collection Runner, Postman CLI, or Newman can be used to execute tests through CI/CD pipelines, schedule runs, and automate monitoring for specific triggers. Finally, continuous monitoring and analyzing test results are essential, especially in handling test failures and making necessary adjustments.

Pooja highlighted that Postman is not just a testing platform but a comprehensive tool for authoring, documenting, prototyping, and creating mock servers and environments, making it indispensable for effective API testing and development.

Key Building Blocks for Effective API Testing in Postman

Pooja outlined the three fundamental building blocks of Postman that are essential for comprehensive API testing:

• Collections: Collections in Postman are versatile tools that help organize tests, define API requests, and build entire test suites. They allow users to include variables, create documentation, and manage workflows efficiently. Collections also facilitate collaboration, enabling teams to work together seamlessly on organizing and executing tests.
• Workspaces: Workspaces serve as the collaborative environment where all API-related assets, including collections and tests, reside. They enable both internal and external collaboration, allowing teams to build, organize, and manage their API assets together. Workspaces can be tailored for private or public access, making them flexible for various project needs.
• Scripts: Scripts are the backbone of automation in Postman, providing the logic that drives dynamic testing. They allow users to automate complex workflows and enhance the functionality of their tests. The recent Postman v11 release introduced a package library, enabling teams to reuse scripts across workspaces without duplicating code. This update streamlines the process, ensuring consistency and efficiency in test automation.

Common Test Scripts in Postman

Test scripts are an integral part of API testing, enabling users to validate various aspects of their API responses:

• Status Code Validation: One of the first things to check when testing an API is whether it returns the correct status code. Pooja provided examples of using the `pm.test` method to verify specific status codes like `201` or `200`, ensuring that the API is responding as expected.
• Response Time Testing: Performance is crucial in API testing. Pooja mentioned the importance of testing response times, such as ensuring that a response time is less than 200 milliseconds. This helps in setting performance thresholds and maintaining the efficiency of API calls.
• Data Accuracy: Verifying that the API returns the correct data is another key aspect of testing. Pooja emphasized the need to check that expected values are accurate and that the API’s response JSON contains the correct fields and data types.
• Contract Testing: One of the more complex tests is contract testing, which ensures that the API response matches the predefined contract. This type of testing is essential for maintaining consistency and reliability in API behavior.

These tests—status code validation, response time testing, data accuracy checks, and contract testing—are fundamental to any API testing process, ensuring that APIs function correctly and efficiently.

Introducing AI with Postbot

Pooja then shifted focus to the role of AI in API testing, introducing Postbot, Postman’s AI assistant. She highlighted how Postbot enhances developer productivity by automating the creation of test scripts, documentation, and debugging processes:

• Automated Test Case Design: Postbot assists in designing test cases for API calls, reducing the time and effort required to write extensive test suites. It can automatically generate tests for status codes, response times, data accuracy, and more.
• Visualization and Debugging: Postbot not only helps in writing tests but also in visualizing API responses, such as mapping launch locations on a globe. It can identify and fix issues, like incorrect regex in date tests, improving the accuracy of the tests.
• Contract Test Autocomplete: Postbot simplifies the creation of complex contract tests by autocompleting scripts based on the API schema, enabling users to write comprehensive tests in seconds.
• Automated Documentation: One of the most tedious aspects of working with APIs is documentation. Postbot automates this process by generating detailed documentation for requests, including input/output details, JSON body content, and more. It even allows for the inclusion of fun elements like emojis, making the documentation process both efficient and enjoyable.
• Postman AI Expert Support: Postbot is trained on Postman’s Learning Center, enabling it to provide expert support on various Postman features. Users can ask Post Bot questions about mock servers, dynamic variables, reusable scripts, or even how to write specific tests. This integrated support gears users up for the next stage in their automation workflows.

Pooja emphasized that Postbot is a powerful tool that not only accelerates the testing process but also ensures accuracy and thoroughness in API testing and documentation. This AI-driven approach is particularly valuable in the current era, where APIs are becoming increasingly central to technological advancements.

Exploring Automation Capabilities in Postman

Pooja continued the session by exploring the various ways automation can be leveraged within Postman, focusing on tools and features that streamline the testing process.

• Collection Runner: Pooja highlighted the Collection Runner as a powerful tool in Postman, allowing users to run every request within a collection. This ensures comprehensive testing of all API endpoints in a systematic and automated manner.
• Postman CLI: The Postman Command Line Interface (CLI) is another key feature discussed. After writing tests using Post Bot or other scripting methods, users can employ the Postman CLI to integrate their tests into Continuous Integration/Continuous Deployment (CI/CD) pipelines. Pooja mentioned popular CI/CD providers like Jenkins, CircleCI, and Bitbucket, which can be seamlessly integrated with Postman for automated testing.
• Scheduled Runs: Pooja also touched on the importance of scheduled runs in automation. This feature allows users to trigger specific API tests at predefined times. Scheduled runs are particularly useful for monitoring API performance and checking for failures or errors at regular intervals.

Leveraging Postman Flows and End-to-End API Lifecycle Management

Moving on, Pooja introduced another powerful feature within Postman—Postman Flows. This tool enables users to connect different API endpoints, allowing them to build applications or create visual representations of API workflows.

• Postman Flows: Pooja described Postman Flows as a feature that allows users to connect various API endpoints to build new applications or create visual representations of API processes. This capability is particularly useful for those looking to understand the relationships between different APIs and how they work together within an application.
• End-to-End API Management: To provide a holistic view of working with APIs in Postman, Pooja outlined the entire lifecycle, emphasizing the importance of collaboration. Users can start by building a workspace and working with their teammates on a collection. This includes setting up API requests, documenting them using Post Bot, and adding tests and automation to ensure the APIs function as intended.
• Collaboration and Demo: Once the API requests and tests are in place, the next step is to demo these APIs to customers, users, or internal stakeholders. This process ensures that everyone involved—from product managers to other stakeholders—can collaborate effectively on a unified platform.
• Unified Platform for API Management: Pooja highlighted that Postman serves as a single platform where both producers and consumers of APIs can work together. Whether working remotely or in-person, users can define, develop, test, secure, discover, evaluate, and deploy their APIs. This unified approach streamlines the entire API lifecycle, making it easier for teams to collaborate and ensure the quality of their APIs.

Through Postman Flows and the comprehensive API lifecycle management process, Pooja illustrated how Postman enables seamless collaboration and efficient API development, testing, and deployment.

Live Demonstration of Postbot in Action

Pooja utilized the remaining time in the session to showcase a live demonstration of Postbot’s capabilities within Postman. She walked through a practical example, demonstrating how Postbot can enhance productivity by automating various tasks related to API testing and documentation.

1. Workspace Setup: Pooja started by introducing a workspace with some APIs, including an “upcoming launches” API. This API was fairly simple, without any scripts or excessive headers. It primarily fetched data related to upcoming launches.
2. Using Postbot for Test Generation: Instead of manually writing scripts, Pooja demonstrated how Postbot could be launched from the bottom of the screen to automatically add tests to the API request. Postbot generated five essential tests, including:

a. Response code validation

b. Response time validation

c. Data validation tests

d. Launch date format validation

3. Fixing Tests with Postbot: Some of the tests initially failed, such as the response time test, which exceeded the expected 200 milliseconds. Pooja showed how Postbot could be used to adjust and fix these tests automatically. By tweaking the response time limit, Postbot successfully brought the test under 300 milliseconds, demonstrating its ability to adapt tests to meet specific requirements.
4. Handling Complex Assertions: For more complex test cases, such as matching a regex pattern for a specific value, Pooja highlighted how Postbot could assist in fixing these assertions using natural language commands. Regex, known for its complexity, was made easier with Postbot’s help, as it updated the date format to match the required pattern, passing the test successfully.
5. Documentation Generation: Pooja emphasized the importance of documentation in API development and how Postbot simplifies this task. By using Postbot, users can automatically generate documentation for each request. In the demo, Postbot quickly created comprehensive documentation for the “upcoming launches” API, detailing the GET request, JSON format, and key parameters like ID, mission name, launch date, and location.

Conclusion

Pooja wrapped up the session by highlighting the potential of Postbot as a valuable tool for simplifying API testing, especially for those interested in integrating AI into their workflow. She encouraged attendees to explore Post Bot and experiment with its capabilities to enhance their API testing process.

Additionally, Pooja demonstrated how Postman allows users to visualize response data, such as converting it into a table format for easier analysis. This feature further underscores how AI can enhance the testing and development experience by offering new ways to interact with and understand data.

Pooja also introduced the upcoming 2024 State of the API Report, inviting everyone to participate in the survey. The insights gathered from this survey will help shape the future of Postman and its AI tools, much like the data from the 2023 report that influenced the development of Postbot. She mentioned the exciting prizes available for participants, including an Apple Vision Pro, a PlayStation 5, and Amazon gift cards.

Time for Some Q&A

1. So does API testing with Postman also factor in the security aspects of the API under test?

Pooja: We recently introduced a new security feature called Postman Vaults. It’s designed to help you securely store and manage encrypted data and sensitive information for your APIs directly on your local platforms.

While Postman is great for functional testing, including the tests generated by Post Bot, Postman Vaults focuses on enhancing security, particularly when it comes to handling authorization, tokens, codes, and other security-related aspects of API testing. This feature provides additional methods for encryption and security, complementing the functional and workflow testing capabilities you’re already familiar with.

2. Could you discuss the limitations of current AI technologies in software development and testing, and how we might overcome them in the future?

Pooja: While tools like Post Bot can enhance productivity by understanding common testing patterns, they still require testers to guide and build out their specific workflows. AI doesn’t yet fully grasp individual testing workflows or anticipate the specific data needed for different testing levels.

Therefore, while AI can support and streamline the process, testers, and developers remain essential in planning, navigating, and ensuring the accuracy of these workflows. AI is a tool that augments, rather than replaces, human involvement in the development and testing process.

Got more questions? Drop them on the LambdaTest Community.

LambdaTest

LambdaTest is a continuous quality testing cloud platform that helps developers and testers ship code faster.

See author's profile