There’s no doubt that Netflix is at the top of their game, serving over 75 million global members. But the remarkable thing for the streaming industry is that they built the foundation of their site using open source code. This post will summarize the tools and techniques Netflix used to turn source code into a movie/TV streaming juggernaut and may shed light on the future of network infrastructure.

First of all, it’s important to look at Netflix’s process when creating their site from the source.

1.  They built the code using Nebula: a collection of Gradle plugins that they created
2.  Changes are applied to a central git repository
3.  A Jenkins job executes Necula–building, testing and packaging the application for deployment
4.  Builds are “baked” into Amazon Machine Images
5.  Spinnaker pipelines are used to deploy and promote the code changes

The Build

To begin the build, they created Nebula, a set of plugins for the Gradle build system that supports building, testing and packaging Java applications. They chose Gradle because it enabled them to easily  write their own testable plugins with a small file size. In their build.gradle file, they declare a few Java dependencies and applied three Gradle plugins that are part of Nebula.

1. “nebula:” internal Gradle plugin that provides configurations for integration
2. “nebula.dependency-lock:” allows the project to create a .lock file
3. “netflix.ospackage-tomcat:” this is the operating system package which is used if the repository is an application

Integration

Once the code had been built and tested using Nebula, the code was then sent to a git repository for integration. At this step, they have dedicated teams who find git workflows and apply them to each code to see what works best.

After applying all changes, Netflix issued a Jenkins job. Jenkins is an open source automation server built with Java that helps to provide plugins for building, testing, deploying and automating projects. Currently, Netflix runs 25 Jenkins servers  in their AWS data centers.

Each Jenkins job uses Nebula to build, test and package the application code all in order to create their repositories. If they’re building a library, Nebula will publish the .jar to their artifact repository, while if the repository being built is an application, the netflix.ospackage-tomcat plugin is deployed. The application will then be bundled into a Debian or RPM package and then packaged into the repository for the next step

Pan Dulce

Netflix created something that they call “The Bakery,” which manages an API that helps to create Amazon Machine Images or AMI’s from the source code. AMI’s provide the information required to launch individual virtual servers in the cloud that they call “instances.” When you launch an instance, you can specify an AMI or launch as many AMI’s as you need It consists of the following pieces:

• Template for the root volume of the instance
• Launch permissions to control what accounts use the AMI
• A block device that specifies the volumes to attach to the instance when launched

The Bakery API service functions by scheduling the bake job on worker nodes that then create images. Users must authorize what package they want installed and what foundation image they want installed with it, which will then trigger a bake.The foundation image contains all the tools and services to then integrate with the Netflix site. The chain for this process usually begins when a Jenkins job is successful, which triggers a Spinnaker pipeline that reads the Nebula package and calls the Bakery API to then trigger a bake.

Deployment

After the bake is finished, the AMI is now available for deployment to thousands of instances, used across multiple environments. The successful bake typically triggers the next stage of a Spinnaker pipeline, which authorizes deployment, allowing teams to begin testing it on custom pipelines. Netfilx uses Spinnaker though, to manage the deployments, allowing them flexibility and control with how they deploy their code.

One thing this system that Netflix has created to build their ecosystem has helped them to do, is become innovators at addressing challenges in the market. For instance, they’ve been addressing issues such as how they manage binary dependencies, working on lowering bake time, fulfilling demand for build and deploy toolsets, providing containers to help improve the build, bake and deploy experience, and so on. As they continue to expand, it’ll be interesting to see how the face of the streaming industry and the tool used to build it will develop.