Vitaliy Kushnerov, principal IT infrastructure architect at Tesla Motors and Rajaseka Jegannathan, a Tesla Web platform architect, described the process of creating a private CDN at the recent Varnish Summit in San Francisco. The CDN project was conceived out of the need for a more scalable architecture and a CDN that could provide a caching engine, a robust Geo-DNS, and global service discovery. Moreover, such a CDN was necessary given that the Tesla website is content-heavy, and laden with space-consuming videos and images.
Global consistency was a necessary consideration in building the private CDN, as the Tesla team required reliable CDN that extended to China. To achieve this, Jegannathan analyzed traffic patterns for data across the globe and compiled a list of CDN requirements derived from his insights, including:
- The ability to use existing hardware and network resources in order to save money.
- Persistent storage-based cache for files ranging from 650 MB to 1.7 gigs each.
- The ability to pick storage based on the application.
- Visibility of the data to the team.
- The ability to control the network.
To this end, Varnish was selected to develop Jegannathan’s proof-of-concept. Varnish offers an open source HTTP accelerator for large websites along with a massive storage engine module that caches large quantities of data that can be quickly reloaded in the event of a crash.
Varnish engineers revised the MSE code to accept re-directs in compliance with Jegannathan’s wishes, such that, the CDN could resolve requests to the closest location. Moreover, VCL’s flexibility and control over the CDN proved to be ideal from Jegannathan’s standpoint.
“Put in front of application server; it is super simple. Therefore, it is also 200 – 1,000 times faster. So every time you move data from caching layer to application server, Varnish will supply the data in 30-40 microseconds as opposed to typical cache which is 10 – 20 milliseconds,” said Per Buer, founder and chief technology officer of Varnish Software, earlier that day.
Varnish uses one partitioned CPU, with priority given to services, video, and image applications in that order. It does not use much memory and the pre-fetching process constantly detects newly available software and videos and pre-populates popular caches in high-volume regions. Regional tiers can also download oft-used content before a request is even made, solving delivery issues beforehand.
Jegannathan also created a backend process in VCL to monitor hardware so that if any piece goes down, the process can redirect to a different server automatically. Video streaming engineers have global visibility to address any issues that arise.
This resulted in a four-tier CDN system, with Tiers 1 and 2 being used for content delivery, and Tiers 3 and 4 used as a console for service delivery. Currently, Tesla’s website supports 2,000-10,000 downloads per day, with the downloaded content reaching 1.7 gigs in size.