Generating Automated Test Reports After Each App Release Using Jenkins

Continuous Integration and Continuous Deployment/Delivery are two key areas that simplify and speed up software development. Automated testing is the core of the CI/CD process, which helps deliver robust and reliable applications to end users.

In this blog, I will show you how we can combine Appium, a popular mobile automation tool, TestNG, a widely used testing framework, and Jenkins, a popular CI/CD tool, to get an efficient and robust solution for automation testing.

We will also discuss creating an insightful test report using Jenkins that details the validation of the application’s functionality and provides actionable details when a new version is released. Integrating Appium with Jenkins will help ensure seamless testing and a high-quality application release.

Automated Testing in Continuous Application Release

Automated testing is recommended due to its speed, accuracy, and maximum test coverage in minimum time. It also helps teams perform other tasks that could enhance the application’s quality.

It is highly recommended that you use automated testing to perform unit, integration, and end-to-end testing of the features and continuous testing at every stage using an automated pipeline.

Manual testing is also very important to check for UI/UX, usability, app performance and data integrity. It could be done at the end once all the automated tests with the basic validations are green.

Jenkins can be used as a tool for creating an automated pipeline and running all the tests, from static code analysis to unit tests and end-to-end tests. The below image is an example of an automated pipeline that could be used to implement the test pyramid. This pipeline would help software teams continuously test their builds and receive faster feedback.

Jenkins is a leading open-source automation server that can automate multiple tasks related to building, testing, delivering and deploying software. It is built with Java and provides over 1800 plugins for automating tasks, building software, generating reports, and more. At the time of writing this blog, Jenkins had over 23.5k Stars and 8.9k Forks on GitHub, making it extremely popular with the software development and DevOps community.

It allows real-time monitoring of test builds and execution of automation tests, which can be useful for detecting, analyzing, and fixing failures easily. It also provides test report plugins that allow us to generate a comprehensive report after all the tests are executed, thus providing insightful data on test execution. This data could be used to analyze the software behavior and performance and, accordingly, fix the issues before we move to production.

In the next section, we will learn how to write tests to automate Android apps using Appium. We will then run the tests and generate test reports using Jenkins.

Writing Mobile Automation Tests Using Appium and TestNG

Let’s begin with writing some mobile automation tests using Appium and TestNG. The tests will be executed using Jenkins Jobs performing the execution on the real mobile devices available on LambdaTest cloud grid.

LambdaTest is an AI-native test execution platform that lets you perform Appium automation with TestNG on a real device cloud. With LambdaTest, you can also integrate Appium automated tests with CI/CD tools such as Jenkins.

Setting Up the Project

Let’s create a Maven project and update the dependencies for the Appium Java client and TestNG in the pom.xml file.

We would also be adding the following maven plugins in the pom.xml file.

• maven-surefire-plugin
• maven-compiler-plugin

Here is the full pom.xml after updating the dependencies and plugins.

The following test scenario will be used for demonstrating the Appium mobile automation testing using Jenkins on a Samsung Galaxy Ultra S24 Ultra real Android device on the LambdaTest cloud platform.

Test Scenario:

Open the Android Proverbial App in a Samsung Galaxy S24 Ultra real mobile device on the LambdaTest cloud platform. On the Homepage of the app verify that the text “Hello! Welcome to lambdatest Sample App called Proverbial” is displayed. Click on the “Text” button and verify that the text changes to “Proverbial”.

Test Implementation

We’ll implement the test scenario in two parts. The first part involves writing the BaseTest to handle the test setup and configurations. The second part involves writing the code to implement the test scenario.

Test Configuration

Let’s create a BaseTest class for configuring the tests. This class will set the LambdaTest capabilities and instantiate the AndroidDriver class to start a new session of AndroidDriver.

The setupTest() method instantiates the AndroidDriver class and sets the required basic capabilities to set up an AndroidDriver session on the LambdaTest cloud platform.

With the breaking changes coming in with Appium 2.0, all Appium Drivers setup classes have been separated. Initially, in Appium V1, the capabilities were set using the DesiredCapabilities() class. However, with Appium 2.0, the Android capabilities are set using the UiAutomator2Options class, and similarly, for iOS, the XCUITestOptions class is used.

The UiAutomator2Options class supplies the Android capabilities, as we will be running the tests on the Android platform.

The ltOptions() method sets all the mandatory and optional capabilities needed by the LambdaTest cloud platform to start the real mobile device.

The LambdaTest capabilities can be easily set up using the LambdaTest Capabilities Generator website, which generates the required code on the fly when the required capabilities are selected from its UI.

The tearDown() method runs after the tests are executed. It updates the test’s status on the LambdaTest cloud platform and gracefully closes the AndroidDriver session.

Writing the Test

Let’s create a new ProverbialAppTests class to implement the test scenario. This class inherits fields and methods from the BaseTest class, enabling their reuse.

The testTextOnHomePage() method implements the test scenario using the Page Object Pattern. This pattern helps focus on testing by allowing code maintainability and reuse. Separate classes are created to hold the objects of different application web pages. So, if the locator changes or any action needs to be modified, only the respective page object class can be modified, leaving the rest of the code unchanged.

The HomePage class has all the locators of the Homepage of the mobile application.

The tapOnTextBtn() method performs tapping on the “Text” button, and the getText() method fetches the text displayed on the page. The assertions are performed using the assertEquals() method of the TestNG framework to check that the correct text is returned when the Homepage is loaded and after theText button is clicked. The tests will be executed on a Samsung Galaxy S24 Ultra real mobile Android device on the LambdaTest cloud platform using Jenkins.

In the next section, we will learn about installing, setting up, creating a job, and generating reports using Jenkins.

Configuring Jenkins For Generating Test Report

There are multiple ways to install Jenkins. Some of these are listed below:

• Using the ‘Generic Java Package (war)’
• For Windows, the ‘Jenkins Installer package’ can be downloaded.
• For macOS, Jenkins can be installed using ‘Homebrew’
• Using Docker
• Using Kubernetes
• For Ubuntu/Debian Linux distribution, Jenkins can be installed using ‘apt’

We would use Docker Compose to install Jenkins, as it is simple and quick and requires less effort.

Setting Up Jenkins Using Docker Compose

As a prerequisite, it is mandatory to have Docker installed, up and running on the local machine.

Step 1: Create a new folder, “jenkins_compose,” and place this docker-compose.yaml file in it. Inside the “jenkins_compose” folder, create another folder named “jenkins_configuration” inside the “jenkins_compose” folder.

The directory structure should look as follows:

jenkins_compose/
├── jenkins_configuration/
└── docker-compose.yml

The following ‘docker-compose.yml’ can be used for installing Jenkins.

File Walkthrough:

This docker-compose file pulls the latest Jenkins image from Docker Hub and exposes the 2 ports, 8080 and 50000. The name of the container is defined as “jenkins” and will mount the local folder ‘jenkins_compose\jenkins_configuration’ with the ‘jenkins_home’ folder in the docker volume.

Step 2: Open the terminal, navigate to the “jenkins_compose” folder and run the following command:

Step 3: Navigate to http://localhost:8080. Here, Jenkins will ask for a password, which can be retrieved from docker logs by opening a new terminal window, navigating to the same folder location ‘/jenkins_compose‘ and running the below-given command:

Copy the password retrieved in the logs and use it to unlock Jenkins.

Click on Continue to proceed to the next screen, select “Install Suggested Plugins” and let Jenkins install the required plugins.

Step 4: On the next screen, Jenkins will ask for the user details. It is recommended that we update our Username and Password so that we can log in with our user details next time.

Click Save and Continue. On the next screen, an option to customize the Jenkins URL will appear. We can skip this and leave it at http://localhost:8080.

This completes the Jenkins installation and setup. We are not ready to start using Jenkins.

Setting Up Jenkins Agent Using Docker Compose

The Jenkins architecture is designed to support distributed builds. At its core, one node serves as the controller (previously known as Master) and oversees other nodes running as Jenkins Agents (previously known as Slave).

The controller holds the central Jenkins configuration, manages the agent connection plugins, and coordinates project workflows.

Step 1: We’ll need to generate the SSH Keys to allow the controller to access the Jenkins Agent via SSH. The SSH Keys can be generated by running the following command in the terminal

The “-f jenkins-agent” argument in the command is for generating the SSH keys with the filename “jenkins-agent”. After executing the command, you will be asked to enter the Passphrase to generate the file. Make sure you remember the passphrase as it will be required while setting up Jenkins Agent.

Here, two files will be generated, one file – “jenkins-agent” holding the private key and another one – “jenkins-agent.pub” holding the public key.

Step 2: The SSH credentials can be added to Jenkins using Manage Jenkins >> Credentials >> System >> Global Credentials (unrestricted) menu.

Click on the Add Credentials button and then follow the steps given below:

1. Select the “SSH Username with private key” in the “Kind” dropdown field.
2. Enter values for the “ID” and “Description” fields.
3. Enter “Username” as ‘jenkins’.
4. Select the “Enter Directly” radio button displayed under “Private Key” and click on the “Add” button.
5. Open the “jenkins-agent” file (generated in the previous step) that contains the private key and paste the values of the file in this field.
6. Enter the “Passphrase” that was used while generating the SSH Keys.
7. Click on “Create”.

Step 3: In this step, we will be updating the “docker compose” file and adding the jenkins-agent service with the public SSH key.

Add the following agent block in the ‘docker-compose.yaml‘ file.

Make sure to update the values after “JENKINS_AGENT_SSH_PUBKEY=” with the public key values of the SSH key generated at your end.

So, now, after adding the agent block, the docker-compose.yaml file should look as follows:

Step 4: Stop the already running docker-compose by using the following command:

Again, start docker-compose by running the following command:

It should pull the images for the Jenkins agent service as defined in the docker-compose file and start Jenkins.

Step 5: Navigate to Jenkins >> Manage Jenkins >> Nodes and click on the New Node button.

The following details need to be provided to set up a new node.

Field Name	Value
Name	jenkins_agent
Remote root directory	/home/jenkins/agent
Launch Method	Launch agents via SSH
Host	agent
Credentials	jenkins(Jenkins Agent SSH)
Host Key Verification Strategy	Non verifying verification strategy

Click on the “Advanced” button displayed under the Host Key Verification Strategy field and update the details for the fields as follows:

Field Name	Value
Port	22
Connection Timeout in Seconds	60
Seconds to Wait between Retries	15

Click on the Save button to save the details, leaving the rest of the fields with the default values. The newly added Node, jenkins_agent, should be up and running in a few seconds. The agent’s logs can be checked to confirm this.

With the Jenkins Agent setup, let’s prepare Jenkins for running the automated pipeline as a Job.

Installing the Maven Integration Jenkins Plugin

The “Maven Integration” plugin is required in Jenkins to set up the Jenkins job for a Maven project. It can be installed by navigating to Manage Jenkins >> Plugins >> Available Plugins.

Search for “Maven Integration”, select the plugin and click Install.

After the plugin installation is complete, Jenkins should restart with the plugin. Next, we need to install Maven using the Manage Jenkins >> Tools window.

Scroll down to Maven Installations and click on Add Maven, enter Name and tick the Install automatically checkbox.

Click Save and Apply to save the configuration. Jenkins is now fully set up to start running Jobs.

Creating a New Jenkins Job

A new Jenkins Job can be created by following the steps given below:

Step 1: On the Dashboard screen, click on the “New Item” button and select Maven Project. Enter the name of the Job and click on the OK button to proceed.

Step 2: On the Configuration page, click on “Source Code Management” and update the repository URL. Since I am working with GitHub, I have updated the GitHub repository link. However, you could update the relevant SCM links such as BitBucket, GitLab, etc.

Step 3: In the Build Triggers section, select the “GitHub hook trigger for GITScm polling” checkbox that will allow Jenkins to poll GitHub SCM and trigger the build when any changes are made to the code.

Step 4: In the Build section, the Root POM value should be “pom.xml”. Add the command “clean install” in the “Goals and options” field, and add the command “clean install.”

Click on the Save and Apply buttons to save the job details and create the Job.

I have updated the goal to “clean install -Dsuite-xml=testng-android-app-tests.xml” because I have multiple testng.xml files in the project and have set the configuration to call the respective testng.xml using the maven surefire plugin in pom.xml. Hence, by using the -Dsuite-xml argument, the project will be built using the “testing” file.

Creating and using multiple testng.xml files for execution helps keep the tests related to different platforms and features separate and allows us to execute the required test suite as applicable.

Since we will be executing the tests on the LambdaTest cloud platform, the next important part is to add the LambdaTest Username and Access Key to the Jenkins Environment variables.

Following the below-given steps, Environment Variables can be added to Jenkins.

1. Navigate to the Jenkins Dashboard >> Manage Jenkins
2. In the System Configuration section, Click on “System” to access the global configuration settings.
3. Scroll down to the “Global properties” section and click on the “Environment variables” checkbox.
4. Add two new variables one for LambdaTest Username and another one for LambdaTest Access Key.

Make sure to use the correct variable as defined in the code. I have defined the variables LT_USERNAME and LT_ACCESS_KEY in the code. Hence, I will be setting the same name for the Environment Variables in Jenkins.

As we are planning to run the test execution when a commit is pushed to the remote repository, we need to set the GitHub Webhook using the jenkins accessible link.

Generating Test Reports in Jenkins

Test Reports provide better visibility, identify bottlenecks and allow software development teams to better control testing activities. They also provide the required statistics and test execution details that could be used to analyze the cause of test failures and check the stability of the application under test.

Jenkins supports multiple test reports, such as the TestNG Report, Allure Report, and Extent Report. However, each report requires some implementation code to be written in the project in order to be generated using Jenkins. Along with the code implementation, certain plugins need to be installed in Jenkins to generate the report.

The TestNG Report plugin can be used in Jenkins to generate the TestNG report after a test is executed. Similarly, the HTML Publisher plugin can be used to generate Extent reports in Jenkins.

Let’s integrate the Allure Report into the project we used to implement the test scenario. We will need to make some minor changes to the project to configure and set up the Allure report.

Configuring Allure Report

The pom.xml file should be updated with the Allure-testng dependency.

The Allure report uses AspectJ to implement the @Step and @Attachment annotations. Therefore, it should be updated in the Plugins section with the Maven Surefire plugin.

The Allure results location should be specified by creating a ‘allure.properties‘ file in the ‘src\test\resources‘ folder and adding the following to it.

Updating the Tests

We need to modify the test method in the project in order to get more insights such as Steps, Description and Tags in the Allure report.

In the ProverbialAppTests class, let’s place the annotation @Description over the testTextOnHomePage() method to show the description of the test in the Allure report, likewise the @Tag annotation is placed to tag that this test method belongs to Android platform. This helps in easy identification of tests in case you have multi platform tests in the project. Those tags – “smoke”, “sanity”, “web”, “mobile”, etc could also be used in the report.

Let’s split the testTextOnHomePage() method into two different methods that will act as Allure Report Steps. The first method verifyHomePageTextOnAppLoad() will verify the text displayed on the Homepage of the app after it is loaded successfully.

The second method, verifyTextChangeOnTappingTextButton(), verifies the text displayed after a tap action is performed on the Text button. Both methods have the @Step annotation with the step description over it.

Next, let’s move to Jenkins and install the Allure report plugin, which will generate the Allure Report as a post-build action.

Installing the Allure Report Plugin

The Allure report plugin can be installed in Jenkins by navigating to Manage Jenkins >> Plugins >> Available Plugins and searching for “Allure”.

The plugin should be displayed in the Installed plugins section after installation is complete.

Next, we need to install Allure Commandline by navigating to Manage Jenkins >> Tools. Scroll down to Add Allure Commandline.

Provide a Name, tick the “Install automatically” checkbox and Click on Save and Apply to complete the installation process.

Generating the Allure Report in Post-Build Actions

In the Jenkins Job, edit the configuration and scroll down to Post-build Actions. Click on Add post-build action and select Allure Report from the list.

Modify the path to “target/allure-results,” as we have set it in the allure.properties file. The allure results will be generated on this path, and accordingly, the path will be used to generate the report.

Click on Save and Apply to confirm the changes. This will set us up to generate the Allure Report after the Build Job completes in Jenkins.

Running the Build Job in Jenkins

A Build Job can be started in Jenkins in the following two ways:

• Pushing the code to the remote repository.
• Clicking on the Build Now button.

Running the Build Job by pushing code to the remote repository

We have already configured the Jenkins Webhook using an accessible link. Let’s push some changes to the remote repository and check for the Build Job being triggered in Jenkins.

The logs of the build execution can be checked in the Console Output window. After the build is successful, an Allure Report will be generated as it is configured as a post-build action in Jenkins.

Running the Build Job by manually clicking the Build Job button in Jenkins

Jenkins builds can be manually triggered by navigating to the Jenkins Job and clicking on Build Now. This option instantly starts building the project and running the tests, generating the respective report configured in the post-build action.

Checking Allure Report in Jenkins After Test Execution

After the build run is complete, the Allure Report link will appear on the Job Status page. By clicking on this link, you can view the Allure report to check the details of the test execution.

The Allure Overview page shows details such as the date and time of test execution, the number of tests run, the Pass/Fail percentage, and details about the Suite and executors. Jenkins is shown in the Executors section, as this report was generated by Jenkins.

Test details can be found on the Suites page where details of test execution are displayed with steps, tags, descriptions, etc.

Checking Test Execution Details on LambdaTest

As the tests were executed on the real mobile device on the LambdaTest Cloud platform, the LambdaTest Dashboard could be checked for more details about the test execution.

The Build details page can be viewed to check the video of the test execution, operating system, version, device details, and logs, etc.

Combining the power of Appium, TestNG, Jenkins and the LambdaTest cloud platform, mobile automation tests can be easily run with a single command. The Allure Report generated in Jenkins after test execution and the LambdaTest Analytics can be combined to get better visibility into the application’s stability and identify bottlenecks. These insights can be useful for deciding whether to release the application to production.

Summary

CI/CD has become an integral part of the software development process. The automated pipelines are triggered whenever the developer pushes a commit to the remote repository. When integrated with the CI/CD jobs, test reports can help monitor the test execution status, analyze failures, and fix them quickly.

In my experience, using cloud platforms is the best way to run your automated tests on the pipeline. These platforms provide ready-to-use on-demand infrastructure, so we can easily run the tests on the required device to get faster feedback on the builds. This also helps us easily check and reproduce production-related issues on the mentioned devices, giving us an upper hand in analyzing and fixing the issues as quickly as possible.

Citations

• Install the UiAutomator2 Driver: https://appium.io/docs/en/latest/quickstart/uiauto2-driver/
• Write a Test (Java): https://appium.io/docs/en/latest/quickstart/test-java/