Images snipped because of size, but important to the article.
An interesting follow up to an earlier post, Chicken. It's pretty clear that incumbent ISPs have a firm grip on the industry, even if they can weasel their way around being called a monopoly.We received a lot of positive feedback, as well as a lot of questions when Mike posted his recent story, Chicken. Many of the questions asked for more specific data about the scale and size of the problem. This post is an attempt to provide some of that data.
Our Internet Services product sits on top of this global infrastructure:
Image
The orange lines are cable systems that Level 3 built and fully owns (the yellow lines are owned by multiple carriers or leased). That means thousands of miles of fiber in trenches across land and thousands of miles of fiber in cables on the seabed. In all, our network contains approximately 180,000 miles of fiber – enough to circle the equator seven times.
That fiber is then turned into usable bandwidth by installing equipment in data centers in each of those red and yellow dots, and also roughly every one hundred miles along each terrestrial cable. That bandwidth is then turned into an Internet Service by installing routers and switches in key locations. The Level 3 Internet Service consists of more than 10,000 Ethernet connections – getting bigger every month. The original invested capital in the Level 3 network was approximately $40 billion.
Level 3 uses that network to sell Internet Services to tens of thousands of customers all around the world. But, despite the huge amount of infrastructure that we built and contribute, we are only one part of the global Internet. When we sell Internet Services, we have to make available every single route on the Internet to our customers – not just the routes we ourselves own. That means we have to provide access to all of the networks owned and operated by others, which right now means about 46,000 other networks – some of which also make use of Level 3’s fiber and bandwidth services.
Level 3 builds a route map of the Internet by connecting its tens of thousands of customers together and allowing them to communicate. So a Level 3 customer in Hong Kong can communicate with a Level 3 customer in Sao Paulo. But to complete the map we also need to fill in interconnection to everyone who isn’t a direct Level 3 customer, so that our customers can also communicate with those who are not our customers. We do that through connections to other networks and their customers. This latter sort of connectivity is often called peering. Peering connections allow for exchanges of traffic between the respective customers of each peer.
While Level 3 has tens of thousands of customers, it only has 51 peers[1]. That total set of interconnections enables our customers to “see” the whole Internet. And what is important here is the “distance” our customers see between themselves and any other part of the Internet. That is often referred to as the number of “hops”; or number of other networks a packet has to traverse to reach its destination. We strive to make that number as low as possible to offer our customers the best performance; more hops can introduce more delay and more potential for quality degradations when the other networks don’t invest enough in performance, redundancy and capacity.
So how does all this compare to other networks? Renesys does a good job describing the interconnectedness of the Internet, but their reports are often misunderstood. They do not show how much traffic each network carries on any sort of relative basis. They merely show the interconnectedness of the networks on a relative basis. As you can see, Level 3’s network is the most interconnected. This list of companies along with our combined investment, innovation and competition has enabled the Internet to grow dramatically by carrying enormous flows of traffic around the globe. Removing these middlemen would leave a massive hole in the Internet.
Much has been made of peering agreements. Many peering agreements were made between engineers in the early days of the Internet and consisted of not much more than a single page of text – if there was anything written down at all. They weren’t really contracts in the way you might consider a formal legal agreement. But over the last decade or so, they have become legal contracts that have a defined term and a set of expectations that each party agrees to adhere to. The vast majority of those contracts are settlement free. For example, 48 of the 51 Level 3 peering agreements are settlement free. In one case, a peer pays us for access to a number of routes in a region where their network doesn’t go; a choice they made rather than buying Internet Services from another party. As we have explained a number of times, our policy is to refuse to pay arbitrary charges to add interconnection capacity (more detail to come in our forthcoming solutions blog post).
But there are also typically shared costs for networks to interconnect. Each party pays to augment its own network to allow for more traffic exchange (the expense to augment capacity is not significant for either party). And since we often choose to interconnect in a third party data center, the networks usually agree to share the cost of the cross connects by paying for them on an alternating basis.
The table below shows the connection locations Level 3 has with its peers, and the total interconnection capacity exceeds 13,600Gbps.
Image
Level 3 has 51 peers that are interconnected in 45 cities through over 1,360 10 Gigabit Ethernet ports (plus a few smaller ports). The distribution of that capacity with individual peers ranges from a single 10 Gigabit Ethernet port to 148 ports. The average number of interconnection cities per peer is five, but ranges from one to 20.
The average utilization across all those interconnected ports is 36 percent. So you might be asking – what is all the fuss about with peering? And why did we write the Chicken post? Well, our peers fall into two broad categories; global or regional Internet Services providers like Level 3 (those “middlemen” listed in the Renesys report), and Broadband consumer networks like AT&T. If I use that distinction as a filter to look at congested ports, the story looks very different.
A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.
That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.
Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.
As an example, this is what one of those congested interconnections looks like. It is a 100Gbps interconnect in Dallas for the week ending April 3. The graph on the left shows flat tops for most of each day – the port is congested and cannot accept all of the traffic that is trying to get through. Not only are packets being dropped (the number dropped are on the right), but all those not being dropped are also subject to delay. The effect of dropped and delayed packets is discussed in our prior post.
Image
For comparison, below is an uncongested interconnection. This is also 100Gbps but in Washington, D.C. with another peer. This shows no congestion, although there isn’t much headroom, so a capacity augment is underway. The graph on the right shows absolutely no dropped packets.
Image
One final point; the companies with the congested peering interconnects also happen to rank dead last in customer satisfaction across all industries in the U.S.[2] Not only dead last, but by a massive statistical margin of almost three standard deviations.
Image
Shouldn’t a broadband consumer network with near monopoly control over their customers be expected, if not obligated, to deliver a better experience than this?
[1] Our peers are generally other networks that complement our own and that agree with us to exchange traffic on fair and reasonable terms – see our peering policy.
[2] According to the 2013 American Customer Satisfaction Index ( http://www.theacsi.org )
