Integrating SmartSight QBR Within Your Video Workflow

System Overview

The diagram here shows the two integration points for SmartSight QBR into customer’s content preparation and deployment infrastructure. At the head end, the iMOS Content Analyzer must be integrated with the customer’s CMS/MAM to trigger the production of the hintfiles which are stored with the ABR assets through the delivery pipeline.

Figure 1. SmartSight integrates with your CMS/MAM to create hintfiles,
and  support QBR player delivery.

On the end-user device side, to enable SmartPlay operation, operators must integrate their players with the SmartPlay plug-in. During installation and startup, the MediaMelon Support team works with the Operator’s team to deploy and validate the integration between the Content Analyzer and the customer’s CMS/MAM and the player and the SmartPlay plug-in.

iMOS Content Analyzer Approach

In terms of installation, MediaMelon will support a customer to integrate Content Analyzers on their cloud accounts, and provide examples templates for on-prem and other setups. In terms of player integration, the SmartSight SDK is integrated into the majority of the commercial ABR plater offerings, and can be added to additional or bespoke players with a rapid turn-around time.

It is important to note that QBR enhanced players seamlessly support partial deployments of hinted video assets. If hintfiles don’t exist for a given asset, players simply use existing ABR logic. If hintfiles do exist but the player doesn’t yet support QBR, it will ignore the hintfiles and use standard ABR logic.

Most customers start with deployment of the simplest or the most popular device platform at first, and add other players incrementally over time. They can also start with adding hintfiles for the top 10% of existing video assets or most popular channels. In this manner, customers can deploy SmartSight QBR in small incremental steps, validating costs and quality ROI at each step, and achieve the full benefits for supported assets and players over time.

Installing and Integrating the Content Analyzer

The iMOS Content Analyzer processes, on the fly, an unencrypted snapshot of all content in an ABR package during encoding, giving each segment a rating on a scale from 1-5, to produce an encrypted hintfile for streaming delivery. As described below, this hintfile is delivered to the video player at the start of the playback session and is used by the SmartSight SDK to better inform the playback segment request process.

The iMOS Content Analyzer produces at least two hintfiles; one for mobile devices and another for all other devices. If the ABR technologies deployed by the operator use different segment sizes, the Content Analyzer creates separate hintfiles for all technologies.

In operation, the Content Analyzer utilizes a Machine Learning engine that is trained on the operator’s content. During installation, the system is trained on a subset of customer content. During the deployment, SmartSight QoE provides feedback on the performance of the QBR algorithm across the entire deployment which is fed back into the system to further improve the quality-related accuracy of the Content Analyzer.

Content Analyzer Speeds and Feeds

MediaMelon can, for example, provide the iMOS Content Analyzer as a Docker container, to be installed either in a stand-alone machine on-prem or in the cloud on a virtual compute instance. Customers can deploy as many instances as needed to produce hintfiles for their existing libraries as well as for their newly acquired content.

The Content Analyzer is codec agnostic but must be tuned separately for each codec. MediaMelon has tested very extensively with all currently deployed codec variants and supported video formats, including MP4, TS, and fMP4 wrappers, either unsegmented or segmented for HLS or DASH.

Figure 2. You can run the Content Analyzer manually (shown),
via batch commands, watch folders, or a REST API.

Typically, operators integrate the Content Analyzer using watch folders or REST API calls with their CMS/MAM and MediaMelon offers a web-based interface for manual asset processing (see above). In addition to the Content Analyzer GUI and REST API, operators can generate QBR hintfiles using the QBRPrep command-line utility. This does not contain the database and backup capabilities of the Content Analyzer tool but is suitable for direct integration into a workflow via a command-line interface.

A single instance of the Content Analyzer on a modern system runs at about 10x real-time, assuming a typical ABR asset. Processing speed depends on NAS and network speed as well. Operators can deploy multiple Content Analyzers in parallel and feed off of the same queue (watch folder or other automation).

As a rough calculation, assume an operator has 10,000 hours of existing content. Depending upon their priorities (e.g. time-to-market vs. compute availability), they can choose to deploy one or more Content Analyzers. Simple math, for 10k hours of content, they will need 10k/10=1k hours = 42 days for a single Content Analyzer instance, or ~ 5 days for 10 CA instances. It’s safer to also budget a 10% overhead for delays, reruns etc.

As mentioned, most operators produce hintfiles for their content incrementally, as any content without a hint file is simply distributed via existing ABR logic.

Integrated Operation

As mentioned, most operators produce hintfiles for their content incrementally, as any content without a hintfile is simply distributed via existing ABR logic. Most operators run the Content Analyzer via integration with their CMS/MAM. These steps describe the sequence of events from the time a new media asset becomes available to the time it is played by the end-user device:

1. New media asset. A new media asset becomes available in the main media storage repository. The asset is already transcoded to the customer’s encoding ladder and optionally segmented/packaged in ABR delivery protocols (such as HLS or MPEG-DASH).

2. New Job file. The CMS/MAM creates a new Job file (Figure 3) and lists the properties of the new media asset and its variants. The Job file is placed in the shared repository that has been configured for the SmartPlay Service.

Figure 3. The new Job file tells the Content Analyzer to produce the hintfiles.​

3. New Analysis job. Once the new Job file is detected in the shared repository, the SmartPlay Service generates a new analysis job task and adds it to the processing queue. The job file is removed from the shared storage. The queued jobs are executed in FIFO fashion by one or more worker instances that analyze all segments in the ABR package and create the hintfile.

4. New Hintfile(s). Once the analysis is done, one or more hintfiles are produced, depending on the segmentation variations (e.g. 2 sec vs 6 sec segment duration) and the target playback device families (e.g. big vs small screen sizes). The hintfiles are stored in the shared storage repository and the corresponding media assets are removed.

5. Publishing of new media content. Once the hintfiles are detected in the shared storage, the CMS/MAM service publishes them to the VoD Portal where the ABR assets of the media content reside already.

6. Playback of new media content. When the end-user starts to watch the new media content, the corresponding assets and hintfiles are downloaded (from the VoD Portal or the CDN) and the SmartPlay-enabled player is able to control which segments are retrieved for playback.

Working with Just-In-Time or Dynamic Packagers

The use of just-in-time or dynamic packagers does not impact SmartPlay unless the segment duration is changed from the duration used to create the initial hintfiles, which typically isn’t the case. When the segment duration is changed for some reason, the operator must generate a new hintfile.

In these cases, the Content Analyzer doesn’t have to start from scratch. Rather it can use quality-related metadata stored during the initial content analysis to accelerate hintfile generation.

SmartSight QBR Live

During live streaming, the Content Analyzer takes RTMP or MPEG2TS streams as input and produces a metadata stream in a sliding-window fashion similar to HLS and DASH manifests as opposed to a metadata file. Players subscribe to the stream to access the hintfile information which uses a presentation time stamp to correlate between the metadata stream and the video stream.

The process of analyzing and generating hints itself takes less time than the segment length, typically around 200 ms, which can be reduced even further using certain optimizations like not decoding all the profiles, and other techniques. For this process, the Content Analyzer can use a different feed than what is being sent to the packager to eliminate any chance of disruption or delay.

In some instances, the Content Analyzer can provide the hintfile before the video reaches the player, which happens because of inherent delays in the OTT stream or delays for regulatory reasons. In these cases, the player has an advance view of video quality and if the buffer space is available, it can fully implement quality mode and increase bitrates of the video to improve quality. If the advance view is not available, SmartPlay can only reduce bitrates, providing cost savings but no quality increase.

MediaMelon has tested SmartPlay Live with a number of tier-1 operators. The results showed that SmartPlay can reduce CDN costs even when videos are encoded with content-aware encoders like Harmonic EyeQ and Elemental QVBR. The test also showed that if the hint file stream gets to the player in advance, SmartPlay can improve quality as well.

Player Integration

The SmartSight SDK is available in JavaScript, Java, C++, and Roku’s Brightscript and is compatible with any platform that provides access to buffer status and the ability to direct the player to select an alternate bitrate / ABR track. The plug-in’s memory and CPU footprints are minimal as it doesn’t decode or display video, it just downloads the hintfile, parses the data, and communicates with the media player and the SmartSight platform. In MediaMelon lab tests, the plug-in adds about 5% to CPU overhead and consumes about 200 KB of memory.

Installing the plug-in requires a minor integration with the player to add communication calls that the SmartSight SDK needs for the switching decisions, which is briefly described below. MediaMelon has existing integrations with most commercial and open-source players, which provide compatibility with any mobile device, smart TV, browser or STB.

For currently unsupported players, MediaMelon staff works with the player vendor to enable the necessary communication between the player’s software development kit and the SmartSight SDK, which typically requires less than 10 lines of code.

SmartSight QBR Operation

SmartSight QBR operation involves the ABR Player, the SmartSight SDK, and the SmartSight QBR tool itself, in conjunction with the other components of the SmartSight platform. When a player is initialized (i.e. when the player window is shown but before the user presses play), the SmartSight SDK is also initialized, and contacts the SmartSight platform to register the user (operator, device, network etc) and obtain the SmartSight QBR initialization parameters, such as operating mode (Quality vs. Bitrate), target quality levels, the location of hintfiles, etc.

Figure 4. Signal flow diagram for SmartSight QBR integration.

When the ABR player loads a new manifest file, the SmartSight SDK initiates a parallel load process for the corresponding hintfile. The critical video delivery path is never altered/disrupted, so the manifest files are not modified and do not carry any QBR signals.

The core player is never aware of the existence of hintfiles, which are loaded and used by theSmartSight SDK to inform the switching decisions.

During operation, the ABR player provides track information from the master manifest to the SmartSight SDK, along with any incompatible tracks and any protected tracks, such as advertisements, that the plug-in should not attempt to optimize. Before retrieving a segment, the ABR player sends that recommendation to theSmartSight SDK which either confirms the decision or sends the ABR Player a different segment to download.

Periodically, throughout the play session, the SmartSight SDK communicates with the SmartSight platform to deliver usage statistics and update its configuration.

Redundancy and Fallback

SmartSight QBR is designed to ensure against any service interruptions. If the customer instance of the SmartSight platform is not accessible during the player initialization (e.g. before the play starts), the SmartSight SDK will default back to standard ABR mode. If the platform is accessible during the player initialization and becomes inaccessible after the asset began playing, then the SmartSight SDK continues to operate normally, though playback statistics may not be reported accurately during the outage.