Metrics & Challenges For Testing Streaming Applications In 2019

Streaming rich media has become an integral part of our daily lives. From watching tutorials on YouTube, Udemy etc. to playing RPGs(Role-Playing Games) on the internet, a major percentage of internet traffic nowadays spends their data on browsing through audio and video contents. With the data speed increasing day by day, media streaming has become the primary way of spreading information to the crowd.

Ensuring your media content runs seamlessly for every user who is visiting your website, irrespective of the device being used to relish the media content. If only it was as easy as picking a pie! The performance of a streaming application may vary depending on the network speed and browser, device, lag, format, frames and many more factors. This article, we will be looking into the challenges related to testing streaming applications in 2019 so your website or web-app doesn’t fail to deliver a delightful user experience.

What Is A Streaming Application?

Remember the time when we used to play Mario, Contra, Pacman, watch shows and news through TV sets as that was the only source of entertainment through rich media. Now, with so many mobiles, laptops, desktop and smart TVs stepping into the picture, we have noticed the shift from offline gaming, news, sports channels to online gaming such as Fortnite, PUBG and news/sports channels such as WatchESPN, MUTV and many more.

Even when we talk about music streaming, the shift is clearly evident from audio cassettes and CDs to cloud-based, streaming applications such as Apple Music, Google Play Music and many more.

Technically speaking, instead of watching the media in a browser, streaming applications are advantageous because instead of consuming large data, they come with certain pre-downloaded features and download the required elements in the background while streaming.

When we think about the term Streaming applications then the first thing that pops to our mind is the video streaming websites or web applications. However, the scope of streaming applications also offers Gaming, Podcasts, Music etc.

Why Streaming Applications Became A Huge Hit?

Well, we all remember Spielberg’s Back to the Future. Especially that scene from the future where Marty’s home in 2015 is filled with screens everywhere and just by voice command he could access any kind of video content.

Well, at present we have made the advancement towards Voice assistance. Just a few clicks or even voice command using Amazon’s Alexa can get you whatever content you want on your Smart TV, Desktop or handheld device.

Some renowned video streaming applications:

• YouTube
• Netflix
• Hulu
• WatchESPN

Some renowned gaming streaming applications are:

• Playstation Now
• GEFORCE Now
• XBOX Game Pass
• Twitch

Fun Fact: If you are familiar with role-playing action games, you must have played or heard about Assassin’s Creed series and their latest addition to the anthology, Assassins Creed Odyssey. For people who don’t have a highly configured system required to run the game, Google is creating something that is going to be a revolution in the world of gaming. In Oct 2018, Google collaborated with Ubisoft to work on Project Stream. And after 3 months of Beta testing, they were successful to enable players from around the world to access and play Assassins Creed Odyssey streaming via the cloud.

Very soon, a day will come when we don’t need to buy an expensive graphics card for running games. Just like movies and TV series, they are also going to be accessed via a streaming application.

Streaming applications have taken rich media entertainment to a new level by providing users with the access to their favourite tv series, games, sports channel, and news channel at their fingertips. We have relied on streaming media primarily because we can access it from any time, any place as long as we have internet access of course.

Why Should We Test Streaming Applications? Aren’t They All About Internet Speed & Connectivity?

Well, the performance of any streaming applications depends on a lot more than just internet speed and connectivity. Performing testing of streaming applications has become a necessity considering the number of competitors increasing day by day. Testing your streaming application is important so that you deliver your potential customers an application that is:

• Completely breakage free
• Can handle huge internet traffic
• Flawlessly streams data under very low network condition
• Operates across all devices without any responsive issues
• When used on the desktop, no matter what browser is used does not cause any cross-browser compatibility issues.

However, all these scenarios can be checked only when you run appropriate testing scenarios that test all the required conditions. That is why all kinds of testing of streaming applications is a compulsion. But testing them can be a bit challenging since the tester has to think completely out of the box, sometimes from a user’s perspective and imagine scenarios when the application may cause breakage.

Metrics To Check While Testing A Streaming Application

After having gone through the challenges of testing streaming applications, you must have understood that load testing is the primary testing phase to execute while testing a video streaming application. Let’s discuss the metrics to consider to ensure that your app is performing properly without any issues

Buffering Time – In YouTube you may have seen whenever data speed is low, a grey bar starts filling up before the video is played. This is known as the buffering bucket. Measuring the buffering time is very important to check how your app is performing under low network connectivity.

Bit Rate – Bitrate is a very important metric that measures the quality of your video. It can be measured by checking the number of bits transferred per second. The bit rate of a video is directly proportional to the resolution of the video.

Lagging Time – Once buffering is completed, the video is supposed to play smoothly if the download speed of your network keeps up with the average bit rate. Often you may have seen that whenever the video timeline reaches the buffering level, the video comes to a halt, starting more buffering sequence. Lagging time is an important metric that measures the total waiting time while playing a video. This also includes the initial waiting time for buffering.

Data Consumed – This metric includes the data spent by the user while using the application or playing any specific video. This helps to determine the demand for streamed data.

Lagging Ratio – This is the ratio of video play time and buffering time. It keeps into account the initial buffering time as well. The ratio is usually very low but it never falls below 1.

Platform Compatibility – This is a critical metric for every website or web-app but is highly significant in case of a streaming application. As a streaming application, you target users throughout the globe and making sure that your media rendering stays intact along with fast buffer is crucial. Sometimes the UA(user agent) of the end-user may lead to abrupt streaming.

Subtitles Synchronization – Subtitles are indispensable for a streaming web application or website if you wish to spread virality of your media content. Targeting an audience throughout the globe, you need to make sure to keep them away from accent specific dilemma. You can do that with the help of subtitles, but often subtitles tend to be out of synchronization which ends up being very annoying for the end user.

In order to ensure complete customer satisfaction, run the appropriate tests before launching it or releasing any updates. However, testing streaming applications can be a little tricky as you may have to predict the real-time user interaction with your web application. There are some common roadblocks or challenges of testing streaming applications. Let me address them for you.

Challenges Of Testing Streaming Applications

Testing a streaming application requires many factors to consider. Apart from device, browser, and hardware, the tester also has to keep in mind external features like network speed, internet traffic, etc. Let’s discuss the common challenges which a tester faces while testing a streaming application.

Response Time in Different Network

To work correctly and play a video with very less buffering a time, excellent data speed is required. It is mandatory to test the impact of the network on the response time of the application. The ideal procedure is to test the application’s performance over different network speed.

This is kind of challenging because most testing teams do not have access to test an application across different networks of varying data speed. Also, there are other conditions like when a customer is travelling, his device may move from one network to another. Conditions like that hugely impact the information flow and impacts on the app’s performance.

Lack of Devices For Cross Device Testing

End users can use any devices of their choice to play a media from a streaming application, ranging from a small smartphone to a 4k television. Testing teams often lack the budget to test their applications across so many devices. However, ensuring that the resolution of the streaming media and the app’s performance is not impacted across different devices is important.

The ideal scenario, in that case, is to use a cloud-based, cross-device and cross browser testing platform like LambdaTest where you can seamlessly test your application across hundreds of devices in parallel, brought up by VM(Virtual Machines) hosted through LambdaTest servers on the cloud.

Issues Occurring Due to End User Interaction

Streaming an application is a process of downloading static assets and displaying them via the app. For example, when a movie is played, chunks of video data are downloaded, buffered and displayed. When a video is played for the first time, the app usually plays it on a default aspect ratio based on the network speed and the device’s physical memory.

In YouTube, you may have seen often that under low network whenever a video is played at 240p and you tried to change it to 720p, it just halts and goes to an infinite buffering sequence. While testing, you have to keep in mind the behaviour of the end user. They may not be so tech-savvy and without understanding the network’s performance if they see that when they try to change the resolution, the player halts, they may give a negative review for your application.

Keep scenarios like that in mind during testing and suggest the developers develop fixes accordingly. The fixes may come in form of error message or disabling resolution changing on low network speed.

Cross Browser Compatibility Testing Of A Streaming Application

Let’s suppose you are testing an application like Netflix on the browser of a desktop. In case of such browser-based applications, you have to ensure that browser compatibility testing is properly executed before concluding the testing phase. However, it can prove to be a bit challenging since your team may not have the combinations of all browsers and operating systems.

You can use virtual windows or emulators, but when the application runs in real time, an unforeseen error may arise which the emulator cannot detect. Often I have seen multiple websites causing breakages when running on older versions of Mac and Safari.

LambdaTest provides you with manual and automated cross browser testing through Selenium grid consisting more than 2000+ browsers and their different versions, all running on multiple operating systems such as Mac, Windows or Linux for desktops & iOS or Android for mobile website testing.

Is Your Streaming Application Durable Enough To Entertain Excessive Traffic?

Load testing is very important to check how the app behaves when huge traffic tries to access a specific video. No matter how perfectly you developed your app, there is a limit to the traffic it can handle. Even in popular applications, scenarios are quite common when a video suddenly became viral and huge internet traffic tried to access it and it just crashed.

With such a huge population of the world, it gets challenging for a tester to guess the right number of maximum user count that can access a specific video at any given point of time. Load and performance testing of a streaming application seriously demands some out of the box approach.

Is Your Streaming Application Capable Of Handling Streaming Techniques?

Hope u remembered the earlier days when a video on a site could not be played because you didn’t have “Flash Player Installed” on your desktop.

Thankfully those days are gone. We have MPEG DASH, Smooth by Microsoft, Adobe’s HDS and Apple’s HLS. All of these are dynamic streaming technologies allow transferring of large chunks of data using very little bitrate.

Wrapping It Up

Now that we have understood the challenges of testing streaming application, it’s better if as a tester you keep in mind the different streaming techniques, especially when a new technology is developed and it replaces the old one. And with the number of streaming technologies and mediums growing day by day, the job of a tester will eventually get difficult as more new technologies are bound to bring more challenges.

So, keep yourself updated as a tester with all the latest technologies and if you are the product owner, make sure that no matter what it relates to, whether your app is a gaming platform or a video/music streaming application, or an application for hosting any kind of services, it overcomes all the challenges and can be accessed by the user anywhere without any lag or complications.

Arnab Roy Chowdhury

Arnab Roy Chowdhury is a UI developer by profession and a blogging enthusiast. He has been writing content for about 5 years and has strong expertise in technical blogs, travelogues, and content in the latest programming languages.

See author's profile