HAMILTON – When Comcast, the largest cable company in North America, needed to upgrade its network to handle the massive demands of delivering IP video at scale, there was no where to turn.
No vendor could provide the complete solution Comcast needed. So the company developed one itself, Neill Kipp, a distinguished engineer and software architect for Comcast, told last week’s Clearcable Networks Technology Summit in Hamilton, Ont.
“I think what is intriguing about the Comcast story is that they turned to open source to find a solution. A lot of small operators think open source is not viable,” said Rob McCann, president of Clearcable Networks.
Comcast built its IP video solution on a number of innovations, says Kipp.
The first is adaptive bitrate streaming that allows for positive user experiences no matter the connectivity or CPU. Kipp says the demands of the system are heavy, pointing out that each two-second segment encoded at multiple bitrates is about 3 MB.
“The text of all the works of Shakespeare is about 5 MB. We are producing the equivalent of the entire library of Shakespeare every four seconds. That is significant bandwidth consumption and a tax on our systems.”
The second key innovation is Dynamic Adaptive Streaming over HTTP or MPEG-DASH that allowed Comcast to build a new architecture with a standard back office media format that can be translated into various download formats for player clients.
For live television, the conversion into MPEG-4 using H.264 compression is happening in real time and packaged to DASH on products built by Comcast, said Kipp.
“We couldn’t find a vendor to keep up with us and what we wanted to do. So we hired internal engineers and built what we needed ourselves.”
That includes Super8, a just-in-time packager that serves IP video to Xfinity Stream and other apps, a manifest manipulation system that handles local simulcast substitutions, advertising and blackouts, and load balancers to manage internet traffic across the country. Hundreds of Super8 instances are deployed across the country to serve on demand, linear, cloud DVR, start over, and emergency alerts.
Comcast deploys a system of web caches to protect its origins from too much traffic. Comcast’s video on demand origins are held in two national sites. This redundancy is so that regional disasters such as Hurricane Sandy in 2012 will not affect the entire country.
Comcast delivers 13,445 channels of linear TV and 150,000 hours – more than 17 years’ worth – of video-on-demand content. Its networks handle 15 billion transactions and deliver 5 PB (or, 5 million gigabytes) of data per day. Comcast's content delivery network capacity is 6.2 TB per second.
“We do more transactions than Twitter has tweets in a 24-hour period. That is a lot of data downstream and upstream.” – Neill Kipp, Comcast
“On-demand is easy. Netflix is easy. Linear TV is not easy. It is hard,” said Kipp. “We do more transactions than Twitter has tweets in a 24-hour period. That is a lot of data downstream and upstream,” said Kipp. “Every time a user clicks a remote, a transaction goes to the application server.” (Ed note: Twitter says it records more than 600 million tweets per day.)
The system uses open-source Hadoop to generate big data minute-by-minute to monitor the system for hardware failures and predict when something might go down.
The final innovation in Comcast’s IP video arsenal is its Xfinity X1 cable box guide that lets users watch recordings stored in the cloud. A set-top hard drive stores a version, but to watch on a phone or tablet or computer, there must be version stored in the cloud. And because of copyright laws, each user must have a unique recording. That requires a series of open-source microservers.
“They are containers, like virtual machines, but they are smaller and more expendable. They give us great efficiency on the hardware side.”
(As readers will know, Comcast’s X1 system is currently being deployed by Shaw Cable in Canada – and will soon be launched by Rogers.)
The 640 PB cloud DVR alone took more than a year to purchase, rack and network. If they were stacked on top of each other, the racks would extend five miles off the ground.
Comcast has built its system so that security patches and upgrades can be done in real time, under load, and no one is affected, says Kipp. “We may have overprovisioned a little but we’ve done it very efficiently. There are no overnight maintenance windows anymore. People are watching all the time now.”
Comcast’s cutting-edge Rio portal for the 2016 Summer Games was a highest-level test of the company’s IP video system and integration of linear TV with VOD. The ambitious software deployment was fully automated through the open-source Kubernetes platform.
“We develop software in production, we test it in production and we deliver it in production,” added Kipp. “We test loads and trial features in production and our time to market is faster.”
