Why You Need To Care About Automated Functional Testing In 2020?

How many times have you come across products that have good UI but really bad functionality such as severe lagging experience and ample number of bugs or vice-versa. There could be multiple reasons for the product to go live, but it definitely gives an indication that thorough testing was not performed. There could be scenarios where a minor software update which was not tested for all the ‘corner scenarios’ could break the existing functionalities in a software product.

In today’s digital economy, testing a ‘software update’ or ‘complete software package’ before going live is primarily important else users of your product would witness a buggy product. This can harm the reputation of your organization and could also result in losses, since a ‘fix’ has to be provided in order to solve the problem.

Rather than performing test during the final stages of product development, it is highly recommended to perform continuous testing in DevOps which emphasizes that testing is conducted at every critical phase of software development. With methodologies such as Shift-Left testing, organizations have now started incorporating testing even before the development part is completed. ‘Functional testing’ is one category of testing that should be done in order to make sure that all the functionalities are working as per requirements documented in Software Requirements Specification (SRS), Business Requirements Specification (BRS), and other ‘Requirements Specification’ documents.

What is Functional Testing?

Functional Testing is a form of testing performed for estimating the quality of a software product by verifying the features provided by the application with respect to client-requirement documentation. Before the start of software development, the requirement gathering team comes up with different sets of documents like Software Requirements Specification (SRS), Business Requirements Specification (BRS), Customer Requirements Specification (CRS), and Functional Testing is one measure to make sure whether all the documented requirements have been met & implemented as a part of the final product.

While doing functional testing, the tester should do ‘black box testing’ and should not bother about the implementation details of the functionality. Depending on the nature of the product, you could use a mix of manual & automation testing methodologies to get maximum test coverage. Databases, Application Programming Interfaces (APIs and browser compatibility), API performance testing, client/server access, user interface interactions, security testing, and cross browser testing are some of the things that can be performed as a part of Functional Testing. The scenarios will definitely differ depending on the type & functionalities of the product being developed by your team.

The ‘test results’ are compared with the ‘expected results’ and based on the output/result, particular functional testing is considered as ‘Pass/Fail’. Since it is a part of black-box testing, a test engineer well-versed with black-box testing methodologies can perform functional testing as well.

Importance of Functional Testing

As discussed earlier, a tester acquainted with functional testing has to validate the software product from all angles and make sure that the requirements are well documented. Also, different requirement specification documents should be met in the product/application under test. Hence, functional testing is performed by the tester where the product is tested more from a customer/consumer point of view i.e. ‘customer requirements’, unlike Non-functional testing is performed by giving priority to ‘customer expectations’. We would look into the difference between Functional testing & Non-functional testing in subsequent sections.

Unlike other forms of testing like unit testing where tests are performed on ‘particular blocks’ or ‘pieces’, functional testing is done on the integrated product. The kind of issues that are discovered post integration testing may be quite different from other forms of testing. This is why the bug fixes pushed to solve those issues, could cause possible ‘side-effects’. This side effect is a regression defect. A regression defect is a consequence bug that is caused because of a recent bug fix in the application. Understanding regression defects are important for every release process.

As a part of Functional testing, the test team can execute scripts to verify the data-flow, security aspects, UI/UX flow, other features mentioned in the CRS. A Functional tester should have good know-how about the product and should possess good domain competency as well.

Tester can do manual testing or automated functional testing or a combination of both. More testing means better test coverage and better product testing. Along with Functional testing, it is recommended that your test team also looks into Stress testing, Non-functional testing, Regression testing, System testing, etc. depending on the timelines & the nature of software product under development.

Understanding The Difference Between Functional Testing & Non-Functional Testing

Before we look into the difference between the testing methodologies, it is important to understand the ‘main objectives of Functional testing’. Some of the core objectives of Functional testing are mentioned below:

• Usability testing: Using manual testing/automation testing/mix of both, testing is done to make sure that there are no usability issues with the product. The user should be able to seamlessly navigate between different screens.
• Error Checking: In order to increase ‘test coverage’, all the boundary & error scenarios are tested thoroughly as a part of Functional testing.
• Compare ‘output’ with ‘expected outcomes’: By supplying relevant test data, all the major functions in the product/application are tested.

Now, let’s have a look at the top level differences between Functional & Non-Functional testing

• Objectives: Functional testing is performed to check whether ‘requirements mentioned in SRS, BRS, CRS, etc.’ are translated into ‘equivalent features/functionalities’. On the other hand, Non-functional testing looks into other parameters like scalability, speed, reliability, and other non-functional aspects in the software product/application.
• Test Methodology: Functional testing can be performed using manual testing and automated functional testing approach or mix of both, but ‘manual testing’ is a must since testers have to make sure that all the necessary functionalities are implemented. On the other hand, Non-functional testing should be performed using automation tools as you need to test the ‘system/application’ under different scenarios that could equate to real-world scenario. For example – Increasing the load on all the CPU cores on which the application is running and checking the performance of the application under such a scenario.
• Which comes first: Functional testing is normally performed before Non-functional tests are executed as ‘necessary features’ should be ready before non-functional tests can be performed. Once the ‘basic feature’ is implemented, developers have to make sure that the ‘feature’ is scalable and would perform well under different scenarios (like varying CPU load, varying Thermal conditions, installation of application in non-secure environment).

Some of the common ‘Functional testing’ methodologies are – White-box testing, Black-box testing, Unit testing, Smoke testing, Cross-browser testing, etc. Some of the common ‘Non-Functional testing’ methodologies are Stress testing, Load testing, Reliability testing, Security testing, etc.

Functional Testing With Example

Imagine that you are building a ‘B2C e-commerce’ application that does the following

• User Sign-up & sign-in
• List the products (in different categories) from the back-end inventory
• Facilitates buying of products via different online payment options

Now, for instance, we will consider some test factors to keep in mind while performing functional testing with example cases mentioned above.

1. Screen testing: Allow a new user to sign-up using the sign-up page. A ‘returning’ user should be able to log-in using existing credentials and ‘Forgot Password’ feature should be in place in case the user does not remember the password. Performing the functional testing of example scenario, the team should test the ‘Sign-up & Sign-In’ page with different user/password combinations and record the output. They need to make sure that the navigation between screens is fast and features (listed in SRS) for this page are implemented.
2. Feature testing: Based on the target market, the product team & development team would have identified the ‘potential browsers & devices’ on which the application should be fully-functional with respect to browser compatibility testing. To validate this scenario we perform ‘browser compatibility testing’ where the application is tested on a combination of different devices (desktops, mobiles, tablets, etc.), Operating Systems (Windows, Linux, Android, iOS, KaiOS, etc.), and Browsers (Firefox, Chrome, Opera, etc.).
3. Testing fallback mechanisms: Let us understand this from cross browser testing point of perspective. In case your application caters to the target market where lot of people use older versions of browsers, your development team should make sure all the requirements are translated into implementation. Once they detect that the web-application is accessed using an ‘unsupported or old version of a particular browser’, there should be a fallback mechanism.

Here is a functional testing example, If the browser does not support ‘Adobe Flash’, the team should make sure that an alternate link to the video is provided so that the video can be played in ‘that version of the browser’. The test team that is doing manual/ automated functional testing for such a scenario should document the ‘browser types & versions’ and test the features in those versions of browsers (and different device combinations).

By using a combination of both manual & automated functional testing, the team should test the ‘application screens’ on different combinations or screen resolutions to deliver cross device accessibility. Based on inputs from the product team, you should prioritize testing on combinations which might be widely used by the target customer-segment. Cross-browser testing using manual approach is not highly scalable since it requires procurement of the necessary infrastructure required for testing and the test team cannot achieve good test coverage using that approach. There still are ways to speed up manual cross browser testing.

Role of Automation in Functional Testing

As Functional testing involves verification of the functionalities implemented in the product, the test team has to make sure that every ‘feature’ is tested. This also includes covering error scenarios and boundary conditions. Manual testing has its own set of limitations since you may need a bigger test team if your product has huge number of features.This will increase the overall expenses. You may not be able to achieve best ‘test coverage’ if you only use manual testing and your QA might reject your product if it does not meet the necessary ‘coverage guidelines’.

This is where ‘Automated Functional testing’ can play a significant role as tests can be automated by writing scripts. These test automation scripts can be scaled to cover different test scenarios and improve overall test coverage & efficiency.

Automated Functional testing can come in handy in scenarios where your project is under extremely tight deadlines. It improves overall productivity of the employees since they are engaged in more ‘interesting work’ and the organization can save time & cost. Manual testing is done by testers who are experts in that domain and once they leave the organization, a new person has to shoulder his responsibilities and build the same level of competency.

Points to ‘Ponder’ before venturing into ‘Automated Functional Testing’

Though automation testing has a huge number of advantages, the product team should make a decision on features that should be tested using ‘manual testing’ & ‘automation functional testing’. There are certain scenarios where manual testing will have an upper-hand; hence your test team needs to figure out the role of automation in verification of product features & functionalities. In any case, you can never replace manual testing with automated functional testing since all test scenarios can never be automated (especially those where manual intervention is required).

Team working on automated functional testing should not work in silo. In fact, the team should work in close coordination with the ‘manual testing team’ & ‘product team’ so that they can understand the scenarios where automated functional testing could be more instrumental. This is necessary to gain domain knowledge and also to build a more robust (and scalable) test suite by writing effective test cases.

In order to get the best out of automated functional testing, you should choose the right automation tools. The automation tool should be selected based on its scalability, past working experience within the team, cost, and ease of maintenance. ‘Test reports’ provide important information about pass & fail scenarios and ‘powerful bug reporting’ (along with customer support) should be a must-have while shortlisting the automation tool. Selecting a ‘wrong automation tool’ can result in loss of time which can affect the project timelines and unnecessary re-work could demotivate your test team. Hence, selecting the right & best test automation tool is vital for success of your test automation strategy.

Myths Associated With Functional testing

There is a sizeable audience that interchangeably addresses different testing mechanisms without knowing the difference between those testing methodologies. There is a lot of confusion around this topic and it is necessary that we break the myths associated with Functional testing.

‘Functional testing’ Is Same As ‘System testing’

Functional testing is one form of System testing or should we say a sub-part of system testing. Though functional testing is done to test the basic functionalities in the product/application, stress testing is more rigorous in nature. As a part of System testing, the tester does the following:

• Testing of the completely integrated product.
• Test cases focus not only on the design, but also the ‘functionalities’ of the product/application.
• Functional Requirements Specification (FRS) and Software Requirements Specification (SRS) are considered the backbone of system testing.
• Test cases are also focused on security, performance, boundary conditions, power management, etc. The coverage of test cases heavily depends on the ‘product under testing’.

If you are building a hardware product, it also involves certification testing, compliance testing, etc. These tests are not included as a part of Functional testing. System testing also includes Stress testing & Load testing where the AUT (Application under Test) is tested against different system scenarios and its performance is measured using tools. Since Functional testing is only a subset of System testing, it is always performed before System testing.
Ideally, a product/web application that does not clear Functional test scenarios should not be validated for system testing.

‘End-to-End testing’ Is A Part Of ‘Functional testing’

End-to-end testing is a form of testing where the ‘product’ or ‘application under test’ is tested from all angles from start to finish. In simple words, it is a testing approach performed from a user’s perspective. Referring to the earlier example of B2C e-commerce application, while browsing the products on the site/app, there is a possibility that the user may face throttling internet connectivity. What should the app display to the user or what happens when there is no internet connectivity?

End-to-end testing would take such kind of extreme scenarios (including third-party dependencies) also into consideration as it has a direct impact on the ‘usability factor’ of the product.

On the other hand, if the product passes Functional test; it means that the basic flow of the product is working fine and the development team can work on the ‘further improvements of the product’.

Always Aim For A 100% Automated Functional testing

Automated Functional testing definitely has a lot of advantages over manual testing, the primary advantage being better test coverage and savings in terms of time & cost. However, there are many challenges for automated website testing. In test scenarios that cannot be automated or where automation might not be cost-effective i.e. these test scenarios normally require manual intervention or require manual observation.

Referring to the earlier example of B2C e-commerce application, assume that the app is being tested on an Android Phone (that has Android P) and the device switches from ‘Portrait’ mode to ‘Landscape’ mode. There is a minor issue with the graphics where a certain part of the graphics does not appear properly. This cannot be automated since only a person who is involved in manual testing can observe such an issue.

It is important that the test team (involved with manual and automated functional testing) sit together and work on test scenarios that can be automated. The whole intention of automated functional testing is more test coverage and building a robust test suite. In ideal cases, a stable combination of manual & automated functional testing should be used to test the product against varied scenarios.

To summarize, the question is never about what test cases can be automated; but it is about how well you can fit automated functional testing in the entire test plan to ship a more stable end-product.

Cross Browser Functional Testing Using LambdaTest

Achieving the right mix of manual & automated cross browser functional testing is extremely critical and testing your product/web application against combinations of (Browsers + Operating Systems + Devices) has become more important than before. Using the cloud infrastructure of LambdaTest, you can put your site/web application under test under different scenarios.

You can select from a range of operating systems & devices to perform ‘responsive screen testing’ across devices. You can choose from a range of devices & also adjust the ‘viewport size’ & ‘screen size’. In cases, where you plan to take a screenshot to verify product from UX & UI point of view, simply invoke screenshot after a certain delay. Shown below is a screenshot from LambdaTest console, where the verification of the LambdaTest website is performed on Galaxy S8+ (in the Portrait Mode).

You also have the option of using ‘Screenshot testing’ and ‘Smart testing’ that is a part of the Visual UI testing. With Screenshot and Smart testing, you can perform visual regression testing and hasten your screenshot testing process by capturing bulk screenshots of your website in a single test session.

LambdaTest is now also up with Automated cross browser functional testing of the platform to execute (and port) automation test cases written in Python, C#, PHP, Ruby, JavaScript, etc. with Selenium as the test framework. LambdaTest offers a Selenium grid consisting 3000+ browsers for you to perform test upon.

Conclusion

Functional testing should always be performed in the product development cycle to ensure that all the necessary/basic functionalities are working fine in the product. It should be done at an early stage of development so that bugs are logged at the right time, before they snowball into ‘release blockers’.

Depending on the nature of the software product, other types of testing – stress testing, cross browser compatibility testing, Integration testing, Compliance testing, White box testing, and more need to be performed as well. With the growing number of devices, cross browser testing has become more important and it is a type of test that can no longer be ignored as it is co-related to the usability aspect of the product. Using the LamdaTest’s cross-browser testing on the cloud, testers can reduce the overall build times (by executing tests in parallel) and achieve better test coverage.

Himanshu Sheth

Himanshu Sheth is a seasoned technologist and blogger with more than 15+ years of diverse working experience. He currently works as the ‘Lead Developer Evangelist’ and ‘Senior Manager [Technical Content Marketing]’ at LambdaTest. He is very active with the startup community in Bengaluru (and down South) and loves interacting with passionate founders on his personal blog (which he has been maintaining since last 15+ years).

See author's profile