Will Google Abandon BGP Next Year and The Four Giants Will Follow

Google doesn’t play by the same set of rules that ISP’s, Carriers, CDN’s, and others play by. Sometime next year, will Google abandon BGP once and for all within their core? To date, Google has the largest private network in the world that carries 20-25% of all Internet traffic, hundreds of Global Cache Locations, 15 data centers, 90 PoPs, and they’re just getting started. Their facilities blanket the global map, unlike anything or anyone in the past.

Google data centers are connected via a private WAN known as Espresso that uses a custom routing protocol to deliver packets to/fro within their network. Espresso leverages a custom BGP stack, packet labels, and high performance messaging system to route packets based on performance data, as opposed to forwarding it to the closest neighbor. Google is a highly secretive company, and we can bet there is a lot more under the hood than what’s being announced, which will be revealed in due time. In the world of CDN infrastructure, the war between private WAN and overlay networks continues, however, we know there is no comparison because private networks trounce overlay networks like Akamai’s Sureroute. That means once a request reaches the Google peering edge, it gets routed over a high performance private network vs a slower CDN overlay network. Google Espress0 didn’t happened overnight. The process begun years ago and Google Espresso is the representation of Silicon’s Valley distaste for all things BGP. It doesn’t end there, when Google makes a big splash, the four giants follow suit.

That would be Apple, Facebook, Amazon and Microsoft, which we refer to as GAFAM, the $3 trillion juggernaut that’s wreaking havoc on the traditional business model within the networking, server, database and storage industries. This leads to one question – will Google move off of BGP completely within their core (defined as origin to the peering edge) by the end of next year? It’s very likely. If you’re a CDN its best to plan for the worse because Google is going after that market.

Google is challenging the very concept of the global network and CDN business model, which they can do because they have unlimited funds, a strong desire to destroy legacy business models, and an incredible appetite to own the data and flow of data end-to-end. Some believe that BGP is the slowest routing protocol in the networking stack, being almost 30 years old, invented in the time of pre-Cloud, pre-iPhone, and pre-IoT, thus Google Espresso is the first attempt to break away from the concept of BGP, at least at its core. Bottom line – Google has a competitive advantage over the competition, because building a global private WAN is an expensive proposition, let alone having the business model to generate enough revenue to support it.

Therefore, the non-GAFAM crowd must find new ways to route traffic via hybrid networks comprising of BGP and custom routing, that is if they wish to remain competitive. Google is the biggest threat to the CDN Ecosystem, followed by AWS, and both are vying to become the default underlying infrastructure that powers the global technology industry in the areas of compute, storage and delivery. We applaud Google for their creativity in trying to change the rules of the game when it comes to global networking and content delivery.

Post Mortem: 12/14/17

Its been almost seven months since we published this post. A lot has happened during this time. Now we can look back and ask the question – were we right or wrong in our assessment? We were wrong because BGP is still going strong and will do so for many years. And we were right because Facebook just introduce the Open/R routing platform. In their announcement, they state the following:

While traditional routing protocols have worked well over the past 30 years, it can be challenging and time-consuming to quickly push new protocols into networking devices.

And “most protocols were initially designed based on constrained hardware and software environment assumptions from decades ago. To continue delivering rich, real-time, and highly engaging user experience over networks, it’s important to accelerate innovation in the routing domain.”

And “we are excited to announce that we have been working with several external partners to bring Open/R to production, both inside the Facebook network and externally.”

What does this exactly mean? Facebook hates protocols and standards that are decades old. BGP happens to fall in that category. And they are going to do everything in their power to force change on the protocols being used within the networking industry. So far, they have been successful in challenging the hardware and networking incumbent industries with their Open Compute Project and Telecom Infra Project. Therefore, they are likely to do the same with networking protocols.

Now this poses an interesting question. Who is leading the way in creating a new routing ideology that will force change upon the industry. Google and Facebook are tied, but other GAFAM members are likely to join the party soon. The giants have a herd mentality when it comes to challenging incumbent industries and players, so together, they might be able to pull it off. Regardless, whatever happens happens, it doesn’t bother us because we don’t sell networking hardware or services.

 

Also, make sure to read these insights by Cato Networksย ย 

18 thoughts on “Will Google Abandon BGP Next Year and The Four Giants Will Follow”

  1. Hi Ernie. Isn’t Google using a tuned version of BGP rather than dropping BGP entirely? Most CDNs have their own “secret sauce” to improve routing and performance. What is the main difference with the Google approach? Also what are the performance improvement they are seeing vs traditional BGP routing?

    • Hi Laurent. For now, Google’s does BGP at the edge but not at the core (B4). And their core is being extended to the edge. Once a request hits their edge, they route via the non-BGP WAN. I don’t know how many locations are on the WAN but my guess is next year everything runs on the WAN.

      Google’s approach to routing is based on SDN where they have agents feeding data to a controller and the controller makes real time routing decisions. Thus, they have global network visibility. CDN’s don’t do that – they have no global network visibility nor do the monitoring companies. All they can do is detect congested paths via probes/agents.

    • Ugh, this is so wrong I don’t even know where to begin. Let me take a stab at it.

      First off, private WANs are already responsible for vastly more traffic than the entire Internet combined. One Google mega-datacenter replicating to another Google mega-datacenter to make web scale applications work can EASILY do more traffic than an entire Tier 1, or even an entire Netflix, sending to “the Internet”. In some ways I believe that’s actually the point you’re trying to make when you reference the extreme network scale of these large web-scale companies, but you’re convoluting internal vs external traffic. Yes Google may be responsible for 20-25% of traffic to “the Internet” (thanks, YouTube), but its internal traffic before a packet ever left its network absolutely dwarfs this by comparison. Even without invoking any specialized industry knowledge here, I believe the large web-scale companies have publicly described the ratios between their internal and external traffic on quite a few occasions.

      But to the original point, none of this has ANYTHING to do with “BGP being dead”. That internal traffic has NEVER been based around BGP, BGP is primarily an external facing protocol, for synchronizing routing state between distinct networks (aka “the Internet”). Alas, and despite the many flaws of BGP, that is in absolutely no danger of changing any time soon.

      What you’re seeing in the recent spat (*) of posts and presentations from these folks are attempts to introduce better application-level intelligence into the content mapping decisions like “which server does a user get served from”. Here, BGP is absolutely abysmal at describing anything more than basic theoretical reachability, and CDNs have been doing everything in their power to shy away from this for many, many years. The innovation here is that folks like Google are using their own routers, with their own software stack, to help move even further away from relying on BGP as an edge site selection mechanism at deeper portions of the network layer than ever before. But like I said before, it’s in no way “dead”, it’s still the same basic first level of “theoretical reachability” that it’s always been.

      (*) When I say “spats”, keep in mind that nothing you’re seeing publicly is in any way “interesting” material to those of us who build big networks. Folks like Google and Facebook are engaged in a constant propaganda war over which one of them is the “coolest”, for recruiting purposes, and BGP is universally “uncool”. You should take all of this material with that grain of salt, and please don’t think you’re discovering any great new inventions or news in them. ๐Ÿ™‚

      (FYI, I’m a former CTO of a Tier 1, and someone who builds multi-hundred Terabit/sec networks, so I know a little of what I’m talking about :P)

      • I appreciate what you’re trying to say, but I think you’re getting some of the details seriously wrong.

        For starters, declaring BGP to be an “old protocol” isn’t entirely true. Yes the fundamentals and framework are old, and the message formats themselves can be needlessly complicated to a modern eye just because someone was trying to optimize away writing 2 extra bytes to the wire 20 years ago, but at the same time it’s also one of the most thoroughly extended and modern protocols out there. In fact, one of the well known jokes in the routing community is “don’t worry, we’ll just put it in bgp!” for how often the protocol is (mis?)used to support new technologies never imagined when it was first created. Due to its extensible format, BGP has been used to carry everything from dynamic firewall filters, to every form of VPN imaginable (including thoroughly modern ones used to enable massive-scale cloud networks, like EVPN), and everything in between.

        Think of BGP as the “lowest common denominator” necessary to start speaking routing to the Internet in any serious way. You don’t WANT this to be something which requires a massive amount of compute and storage, you’d cut off 99% of the developing world, not to mention every rural and edge network in existence. BGP exists not for its complexity, but for its relative simplicity, to be the common language that every network on the planet can speak when needed. In that role, BGP is in absolutely no danger of going away any time soon.

        A better way to state the original point (I think) you were trying to make is that more and more networks are trying to move away from simple “destination based routing” as a decision maker about where and how to serve network traffic. One router talking to another router and saying “hey, you can send 4.0.0.0/8 to me, I’m good for it!” might have been sufficient in 1995, but it’s woefully insufficient in a modern world full of tens of thousands of routers and hundreds of thousands of servers trying to talk to billions of users. It tells you nothing about the optimal way to send and balance traffic, about latency, packet loss, application capacity, etc. So it’s not that BGP is in any way being replaced, it’s that BGP by itself isn’t (and hasn’t been for a long time) sufficient to make intelligent decisions at scale. BGP is in no way being replaced, it’s just the first in a long line of tests which must be performed to determine both the source servers and destination networks.

        Now don’t get me wrong, there are in fact plenty of terrible design flaws inherent to BGP which we seem largely unable to engineer away. For example, consider the inherit lack of security in the protocol, which lets pretty much anyone who gains access to the protocol start injecting data into the global routing table with precious little validation, often causing wide-spread outages as a result. Ironically, the people most eager to make “changes” for change sake have also been the ones with the least knowledge or practical experience, leading to flawed attempt after flawed attempt, while doing little to solve actual problems (e.g. RPKI). If you want a far-out idea that still isn’t going to happen any time soon, that is at least grounded in reality, “blockchain based BGP”.

        As for the comments about IP packet size… I’m trying really hard not to say things like “put down the crack pipe and step away from the Internet”, because you seem like a nice enough guy, but see original post re: “that’s not how any of this works”. Ignoring all the stuff about machine learning in the packet, because I obviously lack the necessary pharmaceutical accessories to converse here, one fundamental concept which limits “packet size” even to practical sizes like “many kilobytes” is something called “serialization delay”. A packet is the smallest unit of discrete information that can move through the network, and while it is transmitting (or “serializing”) down a link, nothing else behind it can move. Remember that the Internet supports link sizes of ALL kinds, from 100 Gbps to 100 Kbps, and needs to work consistently across all of them. One of the reason that latency and jitter sensitive applications work today in parallel with bulk bandwidth applications is that this serialization delay of any individual packet is kept to a minimum in a modern network. This wasn’t how things were originally designed mind you, but they’ve evolved that way over time thanks to packet sizes having not changed much in 40 years, while bandwidth speeds have gotten significantly faster. Your idea is something akin to the network version of “hey man I know how to get more oxygen to your muscles, lets make HUGE 20lb blood cells!”.

        I appreciate far out ideas as much as the next guy, but I’d highly recommend you learn a little bit more about how the existing systems work first, otherwise it reads like a 13 year old writing a comic book. ๐Ÿ™‚

        • Beautiful explanation Richard. You’re from a networking company and you’re very passionate about building global networks. I wouldn’t expect you or Cisco or Akamai or AT&T or the IEEE to agree with me because your business would suffer.

          I’m not alone in this. I’m just setting up the groundwork for more in-depth research that will take place later this year or early next year. I actually have a couple of PhD’s on board. Once I reach a certain point in my research, they’re going to help me and develop some machine learning routing protocols. Based on what we know today, we’re going to use Particle Swarm, Reinforcement Learning Q-routing and AntNet algorithms as a starting point.

          Thereafter, we might get academia involved to test the new machine learning protocols. We’re going at problem as though BGP has never existed. It will be interesting to see what happens.

          IBM just announced their 16-qubit quantum processor and SDK, so that introduces another hurdle. There are lots of moving parts to consider.

          When the time comes, I will get the PhD to publish a technical paper that addresses your points, and all the other points I received, and its a lot.

          • Stefan, may I ask who you are? Not that it matters.

            I publish research on this site behind the paywall. My research focuses on radical ideas and concepts that I believe will be the future. I don’t believe in the status quo or what’s working or not working. And I’m not looking for validation from folks who’ve built stuff in past. There is no right or wrong answer, only perspective.

            The last thing anyone should do is follow the herd mentality. It’s better to challenge indoctrinated systems and institutions than go with the flow.

            Regarding machine learning routing, this was a topic I covered last month in my research. However, this particular subject was of great interest to me so I decided to start imlrp.com. I’m going to publish my ideas there in the form of short blog post. I’m still in the process of hashing out a lot of concepts. Coming up with concepts that will replace BGP in its entirety is taxing on the imagination.

            The topic for June is focused on quantum computing.

      • Thanks Stefan. This is Google Networking 2.0. Once they get to 5.0, it will look totally different.

    • Absolutely, Richard. I wish there were a way to “like” your post, so I will have to settle for just backing you up on this one.

    • Couldn’t agree more, this notion is so wrong-headed it’s difficult to even consider where to begin.

      • Ernie, I won’t try to surpass the response by Richard, whose reputation far outstrips most folks walking around commenting on such things.

        Honestly? Your statements come off a bit ridiculous. Along the lines of “I’ve published all this amazing research that nobody can read unless they cough up some cash.”

        I’m tempted to refer to you as a mere charlatan, but I’ll stop at “misguided.” It’s borderline Jim Fleming IPv8-level wacky.

        • I didn’t know Richard all to well until he posted here. Takeaway – he is someone reputable – that’s good to know. And I have no idea who Jim Flemings is nor do I want to.

          If the statements are ridiculous please explain why and back it up with a convincing story. Matthias did a great job. My hats off to him. And not only explain the bits and bytes but the big picture. Is that too much to ask.

          • You should really Google about Jim and IPv8 to read up. It’s old stuff now, but you should see what the parallels are.

            The 1TB packet thing was pretty rich. Richard summed it up with his 20lb red blood cell comment. Nothing further needed there.

            It seems to me that you really don’t have a good handle on how MPLS works. When you say folks like Google only run MPLS at the edge, and don’t run it across their cores, to me, that just says, “Duh, of course. You don’t run BGP on P-nodes inside the core.” Inside the core, you run an IGP, and then (typically) iBGP is run PE to PE, between loopbacks at edge. Or, of course, in much larger networks, PE’s to a couple of route reflectors, which would be fully meshed between the other RR’s.

          • Appreciate the feedback. I’ve sold MPLS in the past but that was a few years ago. I didn’t mention the word MPLS in the post, only Private WAN. I know that MPLS would have to connect the core to the edge and at the edge it would be BGP, where they peer with ISPs.

            Also, I’m trying to get a couple of CTOs to respond to my post with their own point of view that is different from Richard. I would put them in the same category as Richard in technical know how, but have radical view of the future of the network. Stay tuned on that.

      • FWIW, I actually have as deep a desire to replace BGP as anyone you’re likely to run across. In and of itself that’s not an unreasonable goal, BGP is indeed deeply flawed in ways that only those of us who have truly used it in every way possible can truly understand. Hell, I even wrote my own BGP protocol speaker once, many many years ago, so I can promise you it’s not even terribly good at a protocol level. ๐Ÿ™‚

        But I think the better place to start would be to ask yourself if you truly understand what BGP does today, or how you might go about replacing it. From everything you’ve said, I’d have to argue that you currently have a very limited understanding of what BGP does and how it does it. That’s making it impossible to converse further on the subject.

        As for the question “How do Facebook/Google use or not use BGP to deliver content at scale”, which I think is your real question, let me try my best to explain it.

        BGP is only the start of the equation, not the end. BGP is, quite literally, the most fundamental thing you can do to establish the most primitive notions of “can I reach this address over this network”. If that is where your intelligence stops when you decide where and how to serve your traffic, then yes you’re absolutely going to have a terrible time of it.

        The art of “delivering content” over the Internet has evolved quite a bit over the years. In the earliest days, you had simple scenarios like “I’ve got this content over here, and I’m looking at multiple possible ways to get it to the destination, using BGP”. Next we saw the evolution of CDNs, with scenarios like “I’ve distributed this content to many places, and I’m using destination based routing via BGP to pick which server the request goes to, so you’ll probably be talking to a close server”. Now we have application level intelligence, which says “based on my own interactive measurements of loss, latency, resource contention, and client experience, I’ve selected the following sources and destination networks to use”. This level of intelligence is the only way you serve interactive content reliably to billions of users.

        Now, through all the scenarios above, please understand that nothing about BGP itself actually changed. BGP is as it has always been, quietly sitting there, mostly working, and enabling the most basic establishment of connectivity between networks. What has changed are the layers that have been added on TOP of BGP, augmenting the single simple answer which it provides with dozens of additional data points to help make intelligent decisions about where and how to serve content.

        If your contention is that BGP is becoming less and less relevant to the process of selecting “where and how” to serve content, then not only do I wholeheartedly agree with you, but I’d say it has already been this way for many, many years. This is the entire foundation upon which large scale content delivery is founded. But if your contention is that BGP is becoming less and less relevant to the question of “how do I establish basic connectivity between these networks”, or that it’s going to be replaced by any other routing protocol which does better, then I’d argue you have absolutely no idea what you’re talking about. ๐Ÿ™‚

Comments are closed.