Why did Apple fail to stream its historic product launch recently? Why did the Oscars fail, or ESPN fail during the World Cup after it reached 1.7M simultaneous users? Why do large and small live streaming events fail at all? The reason is very simple; live streaming is a long chain of individual processes that must work in synch, in order for the live stream to work properly, because if one individual process fails, the delivery chain fails. Streaming is about implementing the right architecture, having proper configurations, and using best practices.
In Rayburn’ blog, many commentators debated about the use of JSON request and caching, as the primary culprit for Apple’s streaming failure, but in the grand scheme of things, JSON and caching is 1% of the problem in the streaming video delivery chain. The streaming video delivery chain is massive, and there are literally hundreds of different configurations possible and steps. Don’t believe me, look at the illustration below. Here are two vastly different configurations for big event live streaming; one is running on Akamai, and the other is running on Level 3 that leverages their MVNS (video fiber network). The difference, Level 3 can take control of the native video stream once it hits the encoder at the customer site. From there, Level 3 takes ownership, and ensures the stream is delivered to the end user successfully, and they will offer a strict SLA to backs it up. Akamai is just as good.
Live Streaming Video Delivery Value Chain
Personal Notes on Apple’s Streaming Failure
- Apple is fully responsible for the streaming event failure
- It was not Akamai’s fault, period. Akamai has done thousands and thousands of live streaming events, and this one is no different
- Capacity was not the issue, as Akamai successfully streamed 6Tbps+ of traffic during World Cup
- Apple is unlikely to publish what went wrong, because they are an extremely private company, and extremely sensitive company
- Apple pretty much has a muzzle on Akamai, so we might never find out what truly happened
- Broadcasters and CDNs speak two different languages. There are only a handful of folks that truly understand both worlds. Level 3 and Akamai have a few of these folks on staff
Best Practices of Live Video Streaming
- For large important streaming events, companies should use Akamai or Level 3, as they have more than a decade of experience streaming big live events
- Always use expensive and redundant hardware based encoders at the filming location, like MediaExcel or Harmonic, as they are workhorses that never break down
- At filming site, use three or more connections for delivering the video stream in its native or encoded format for added redundancy
- One of those connections must use wireless encoders / transmitters such as the Teradek, for backup, in case the Carrier Central Office goes caput for whatever reason, like a flood
- Publish live streams to multiple POPs, known as publishing points in case one server, cluster of servers, or CDN POP fails
- Use Akamai and Level 3 in an 80% / 20% active/active configuration
- If using Level 3, use digital connection such as HD-SDI/ASI instead of IP, as it is geared more toward broadcasters, and is more reliable. Level 3 offers very strict SLAs with it
