“In God we trust. All others must bring data.” – W. Edwards Deming.
In part 1 and part 2 of this series, we introduced the key role of a powerful streaming intelligence platform and associated core tools to ensure a service operator can meet critical business challenges. This third part extends the application to automation of quality management based on context.
A fuller version of the complete three part Streaming Intelligence blog series is available in download form.
Traditional analytics often requires the linkage between insight and action to be manual. This post addresses a world beyond this convention, to outline tools that can play a more active role in optimizing both the viewer experience and the service cost base. And the foundation of this approach is an objective way to measure video quality as seen by the viewer.
Before venturing into any discussion about how best to improve the quality of video streams for viewers has to be a commentary on just how to measure this quality. As Peter Drucker famously said, “If you can’t measure it how can you improve it?”
There is a large body of scholarly research on the topic of video quality measurement, and to our team one thread stands out. The ITU study (BT.2095-1) “Subjective assessment of video quality using Expert Viewing Protocol” published in 2017 defines the EVP protocol for measuring perceived quality with groups of viewers in controlled experiments. Such an approach has been very useful to map different quality measurement techniques onto a computed video quality measurement. When our team applied this EVP approach to rate the video quality viewers actually experience when video is streamed to their device over the internet, they arrived at the now patented iMOS scoring system.
Pragmatically, but rigorously, the iMOS approach gives a quality rating, based on the ITU EVP principles, for each different bitrate video segment in an an ABR presentation prior to stream delivery. The quality rating approach factors in not just bitrate, but content scene complexity. In real-time it's then possible to track which of those segments was actually seen by the user during the course of a streaming session. An iMOS score for that session is then an objective estimate of the quality experience across the session timeline as experienced by the viewer.
To go on a deeper dive for information on video quality research and the iMOS approach download our detailed white paper.
The use of the “greedy” ABR algorithm is costing the industry more and more each year as surplus bandwidth is consumed by well connected devices for no significant gain in viewer QoE. |
Under a great deal of competitive pressure, streaming service operators need such objective insights to help optimize their service delivery - definitely to offer viewers an exceptional quality of experience, but also to contain the costs of doing this in every way possible. Content aware encoding and other advanced techniques provide some useful tools for content preparation, but streaming optimization is essentially a real-time problem that's related to the content itself and the ever-shifting characteristics of the delivery network and last mile context—in other words it needs to happen at the session level.
One significant challenge in squaring this circle is the fundamentally "bandwidth greedy" nature of most ABR client device players. Every few seconds, the basic native ABR algorithm asks the question, “For the next display period, what is the highest bitrate segment I can download?” and acts on such a basic assessment—regardless of whether the highest bitrate segment is beneficial to the viewer experience necessary or not.
In some significant edge cases when access bandwidth is significantly and consistently higher than the bitrate of the highest profile in the bitrate ladder, the ABR algorithm will just default to the highest bitrate for the entire session, resulting in a constant bitrate experience. In a high bandwidth broadband home, for example, there will be tendency for all devices to just consume at the highest bitrate possible regardless of whether this benefits the viewer experience.
The "greediness" and the "contextual blindness" of basic ABR prevents optimization in many such situations. Streaming service operators often face seeming conflicting challenges caused by last mile issues and which cannot be addressed by centrally deployed solutions. For example:
Viewers using high speed networks (eg fibre to the home) where native ABR players end up continuously defaulting to the highest bitrate profile. This incurs material incremental costs for OTT operators and discourages the offering of higher bitrate profiles, which might prove to offer a better experience for viewers with large screen smart TVs for example.
Viewers using constrained or unstable networks (eg public WiFi or mobile) where native ABR is attempting to request higher bandwidth segments than necessary, causing re-buffering and often resulting in a catastrophic user experience
A rigorous, real-time quality measurement approach like iMOS provides the basis on which to achieve significantly better results for both the viewer and the operator in the context of each and every viewing session.
The iMOS technology for stream quality optimization comes in two parts: Content preparation and run-time management.
The iMOS Content Analyzer is a simple tool that can analyze all time segments of all bitrates in an ABR package and generate a set of iMOS quality scores (on a scale from 1-5) based on segment-by-segment analysis of scene complexity. The iMOS approach is non-referential (ie does not require visibility of the original mezzanine file) and, using patented technology, closely mimics the output of ITU EVP analysis. iMOS scores also align with outputs from trusted open-source tools (such as VMAF), but the non-reference, lightweight nature and speed of the iMOS Content Analyzer makes it uniquely suited to real time video analysis. The output of the tool is a set of iMOS hintfiles that are paired with the ABR manifest file for each content asset.
Multiple hintfiles can be used to address perceptual quality requirements on screen sizes across the range from mobile to smart TV. Use of the iMOS Content Analyzer is not limited to on-demand content—since the evaluation process typically runs faster than 10x real-time, the creation of iMOS hintfiles can be integrated into a continuous real-time process that runs parallel to the generation and packaging process for live broadcast ABR streams. It's important to clarify that the existing content encoding and packaging workflow is un-modified by this integration and that the ABR manifest files are not modified in any way since they do not carry any data for the QBR process.
Behind the iMOS measurement tool is a vast Machine Learning library, developed over years of video analysis, which is used to fine tune iMOS and the QBR optimization process to each operator’s content networks and devices. This process happens during deployment and ensures SmartSight QBR delivers optimized performance to each operator environment
Also note that the iMOS Content Analyzer Machine Learning engine can continue its training during deployment and operation. Operationally, the SmartSight Platform accumulates a core of baseline iMOS measurement data which is augmented by session performance data from the QBR algorithm during service operation.
The real-time optimization of the streaming sessions is an iMOS measurement driven process called SmartSight QBR. The only prerequisite to take advantage of QBR optimization is the integration of the SmartSight SDK with the video player on client devices. In many cases this integration will have already have been undertaken to provide rich streams of QoE and Ad measurement data for the other MediaMelon SmartSight tools.
In operation, whenever the video player loads a new manifest file, the SmartSight SDK initiates a parallel load process for a corresponding iMOS hintfile. The SDK also then contacts the SmartSight Platform to register the initiation of a session and acquire any run-time business rule directives that may have been configured for content type, time of day, user profile, or class of device.
During video presentation, the core device player shares ABR track bitrate, buffer status, and other device information with the SmartSight SDK, which can then draws on the SmartSight Platform machine learning resources to drive the optimization process even in a changing viewer context. The player also communicates information about any incompatible tracks and protected tracks, such as advertisements, where optimization is not available.
The critical interactions of the core player with the SmartSight SDK occurs just before requesting a new segment download, when the initial recommendation is communicated to the SDK. The SDK can either confirm the segment selection or propose a different segment to download. The SmartSight SDK uses hintfile quality data for the current time segment, device screen size, and any platform defined business rule objectives for the current streaming session to inform the ABR switching decisions and optimize the quality and bandwidth outcomes. All of this logic happens more or less transparently to the other content streaming and ad-insertion decision making logic that may be occurring at the same time. Periodically, throughout the play session, the SmartSight SDK additionally communicates with the SmartSight Platform to deliver usage statistics for SmartSight dashboards.
Deploying SmartSight QBR offers data savings of around 30%-45% compared to standard ABR and can enhance the quality of complex scenes. |
The core player is never aware of the existence of iMOS hintfiles and its internal logic is never altered, disrupted, or slowed by the initialization process. If a relevant hintfile is not found by the SmartSight SDK, then operation is unaffected. Using carefully constructed fallback logic, it is easy for an overall deployment to operate with a mix of QBR-enabled and non-QBR enabled devices (this facilitates a proof of concept deployment for example), and the whole architecture is design to default to standard ABR operation if any part of the QBR system is unavailable. For example, if the SmartSight Platform can't be contacted during the player initialization (e.g. before the play starts), the SmartSight SDK will fall back to ABR mode. If the SmartSight Platform is available during the player initialization, but becomes unavailable at any point after the asset began playing, then the QBR process will to operate normally, though playback statistics may not be reported accurately to the platform during the outage.
This system is in active deployment at major streaming services globally - please see existing case study content and watch this space for new updates.
SmartSight QBR takes on the challenge of meeting streaming QoE goals and solves it within a new objective measurement framework. SmartSight QBR uses objective perceptual video quality information generated by our patented iMOS evaluation tools. In conjunction with per-session business rules this reference data is used to enhance the playback performance of all types of Adaptive Bitrate (ABR) streaming, resulting in improved delivery QoE, reduced buffering and lower streaming costs.
Importantly, the tools work seamlessly within existing streaming content workflows, with no need to modify the encoder, CDN or player ecosystems. They are especially effective at dynamically optimizing bandwidth management across different device and network types. SmartSight QBR can result in material savings in data costs, but a core service QoE benefit is the ability to add higher bitrate profiles to the content bitrate ladder in the confidence that SmartSight QBR will only select those profiles when content complexity and QoE targets dictate. This could result, for example, in data costs being unchanged whilst QoE for a premium content library being materially enhanced.
At MediaMelon, we focus on building powerful streaming intelligence tools that don’t just offer data, but help you gain better visibility into your content engagement and business leverage. We leverage a best-in-class ML-enriched data platforms to derive actionable insights for streaming solutions. From offering real-time data to providing content and subscriber insights, we take care of streaming intelligence end-to-end.
To learn more about MediaMelon, please reach out to us here.