Building a global infrastructure footprint is a complex process. Whether a company is building a CDN, Global Cloud Security Footprint, or WAN as-a-Service, all three should use the same ISP’s, data center facilities, and hardware. If the builder selects the wrong provider, the mistake is going to cost the builder plenty. There are many do’s and don’ts in the building process. I’ll briefly cover both.
The best builders of global infrastructures are the folks working for established CDNs, like Akamai and Limelight. For the up-and-coming CDNs, it usually takes a while for them to figure things out. The technical and business perspectives must be taken into account when building a global footprint. I’ll cover some best practices below.Â
What Providers Are Not the Right Fit
First, I would like to start off by stating that Amazon, Rackspace, SoftLayer, and Terremark are not the right fit for the global infrastructure footprint. Amazon, Rackspace and Softlayer don’t work because the builder needs total control of the routers, switches, servers, and hardware. Root access to the routers is required, so the builders can set up EBGP, ISIS, OSPF, communities, and other protocols, under the builders own Autonomous System (AS).
It doesn’t make sense for the builder to do all the work in building out a global network, than have all that work go to someone else like Amazon. Also, root access to the router is required so the builder may implement TCP/IP protocol optimizations. Squeezing out every millisecond of the middle mile for the POP-to-POP delivery, or Origin-to-POP delivery is important.
The Terremark ‘s of the world don’t work because the builder needs to deploy the hardware infrastructure in specific cities and locations. The POPs must be located at the major Internet Peering Exchanges in LAX, Miami, Amsterdam, Frankfurt, and so on.
How to Build a Global Infrastructure Footprint
Basically, a global infrastructure footprint consist of placing servers, routers, and switches at various co-location’s around the globe. Most global companies use the same routers, servers, switches, and storage. One key ingredient for building a robust infrastructure is to keep all the hardware consistent. If the builder is using SuperMicro servers, than SuperMicro should be used at all locations. Same goes for routers and switches. Mixing and matching hardware from different vendors will cause problems down the line when it comes to management, monitoring and patch maintenance.
How Many POP’s Are Needed
Most global CDNs like Limelight and Aryaka Networks have around 30 POPs. Any more than that is unnecessary. I’m still wondering why CloudFlare plans on building out 100 POPs, or why Zscaler has 100 POPs. For me personally, it’s doesn’t make sense. CloudFlare may deploy 100 POPs, or 1000 POPs, and Limelight and Fastly will still be faster. There is diminishing returns on anything above 30 POPs.
It’s better to add more hardware to the existing 30 POPs, than it is to add more POPs. The ISP networks in the US, Canada, and Europe are well developed and highly peered. Round trip times between locations in Europe, and US are usually under 50ms, whether a company has 10 POPs or 40 POPs. For the new cloud provider, I would start with 10 POPs, than grow from there. Below are the names of some of my preferred vendors.
Data Center
- Equinix is the best choice
- Coresite – for US presence
- Telecity – Europe
IP Bandwidth – Tier 1 Providers
- Level 3
- NTT
- Cogent – not a good fit. They are too oversubscribed
Routers
- Brocade – less expensive choice but still as good as Juniper
- Juniper
Server, Routers, Switches & Load Balancers
- Dell Servers
- SuperMicro Servers
Central Origin Storage (Not Edge Storage)
- Isilon – Internap Uses Isilon
- Data Direct Networks
- Network Appliance – not a good fit for high volume of read/writes
Major Internet Peering Locations
- Los Angeles – West Coast Delivery and Gateway to APAC
- San Jose – West Coast Delivery and Gateway to APAC
- Miami – Gateway to all of LatAm
- Ashburn – Serves East Coast
- Dallas – Central US delivery
- London – Delivery to Europe
- Amsterdam – Largest Peering Center Globally
- Frankfurt – Gateway to Middle East
- Singapore – Serves APAC and India
- Tokyo – Serves APAC
If anyone would like more detail, please email me.