Netflix recently introduced its new Dynamic Optimizer, which employs AI techniques to analyse each individual frame of a video in order to compress it as much as possible without diminishing the image quality. This new encoding method reduces the amount of data used and is largely targeted at the growing number of viewers who watch video over mobile and tablet devices.
“We’re allergic to rebuffering,” said Todd Yellin, a vice president of innovation at Netflix at a demo at Netflix House, a mansion in the hills overlooking Barcelona, which the company is in residence at during the Mobile World Congress trade show. “No one wants to be interrupted in the middle of Bojack Horseman or Stranger Things.”
Yellin said that its customers in many regional markets, such as India, Japan and South Korea, watch Netflix on their phones and tablets more often than on televisions. Yellin added their genre of choice is the TV drama, saying, “It’s portable, it’s like bringing that book you love and carrying it with you everywhere.”
For mobile users, the lower bandwidth used for streaming video means that subscribers can still use the service and avoid exceeding their monthly bandwidth cap. For viewers watching via set-top boxes, minimizing the bandwidth means that the connection is freer for use by others in a household, and potentially allows television owners who have lower-speed bandwidth connections to view HD content.
The Dynamic Optimizer is the result of a partnership between Netflix, USC and the University of Nantes. As part of its development, hundreds of users were shown hundreds of thousands of shots who had to rate each shot’s various qualities. The results were put to use through the training of an algorithm to determine the quality of an image. The usual method for encoding online video is to compress it uniformly for a particular connection speed, which for a low-bandwidth connection is why a video may have grainy and pixelated images after it has been highly compressed.
Dynamic Optimizer, however, analyses the entire video at particular quality and resolution points to work out the optimal compression trajectory for an encode. Netflix utilizes VMAF, its subjective video quality metric, as its optimization objective. By matching the level of compression to a scene’s individual content rather than the show as a whole means the overall stream size can still be lower, but its overall quality is higher. A colourful action scene involving swordplay, extras and multiple locations, for instance, would get more bandwidth than a simple romantic comedy scene between a man and a woman on a park bench.
“We can now optimize [a stream] scene by scene with an almost infinite matrix of possibilities,” said Ioannis Katsavounidis, a senior research scientist on Netflix’s video algorithms team.
Movies and TV shows are usually encoded at ten different bitrates. Users are bumped to a lower bitrate stream if network issues arise; however, using the Dynamic Optimizer will mean that there is not a significant change in quality
In a recent blog post, Netflix opened up about the main challenges it has faced in implementing the Dynamic Optimizer framework in production. Netflix said the largest problem it faced was retrofitting its parallel encoding pipeline to process significantly more encode units. By transitioning from chunk to shot-based encoding, a single episode of Stranger Things, for instance, requires the processing of 900 shots. The new framework has increased the number of chunks “by more than two orders of magnitude per encode, per title” and this“exposed system bottlenecks related to the number of messages passed between compute instances”. In response, Netflix pioneered two engineering innovations: Collation and Checkpoints, which it discusses at length in the same post.
Netflix has not yet indicated when the Dynamic Optimizer will be implemented across its content, but says that it aiming for a process that is as “as automatic as possible” in order to quickly cover its complete library.