There’s something rotten in the state of video streaming
Posted: 2014-02-06 10:43am
There’s something rotten in the state of online video streaming, and the data is starting to emerge - Gigaom
If you’ve been having trouble with your Netflix streams lately, or maybe like David Rafael, director of engineering for a network security company in Texas, you’re struggling with what appears to be throttled bandwidth on Amazon Web Services, you’re not alone.
It’s an issue I’ve been reporting on for weeks to try to discover the reasons behind what appears to be an extreme drop on broadband throughput for select U.S. internet service providers during prime time. It’s an issue that is complicated and shrouded in secrecy, but as consumer complaints show, it’s becoming increasingly important to the way video is delivered over the internet.
The problem is peering, or how the networks owned and operated by your ISP connect with networks owned and operated by content providers such as Amazon or Netflix as well as transit providers and content deliver networks. Peering disputes have been occurring for years, but are getting more and more attention as the stakes in delivering online video are raised. The challenge for regulators and consumers is that the world of peering is very insular and understanding the deals that companies have worked out in the past or are trying to impose on the industry today is next to impossible.
Which is why we need more data. And it’s possible that the Federal Communications Commission has that data — or at least the beginnings of that data. The FCC collects data for a periodic Measuring Broadband Quality report that was most recently issued in Feb. 2013. In that report the FCC said it would to look at data from broadband providers during September 2013 and issue a subsequent report in that same year. That hasn’t happened, but the agency is preparing one likely for late Spring. The report measures how fast actual U.S. broadband speeds are relative to the advertised speeds. While the initial report published in 2011 showed that some ISPs were delivering sub par speeds versus their advertised speeds, ISPs have since improved their delivery and FCC rankings. As a result, the reports goals have shifted to measuring mobile broadband and even caps.
But the FCC’s next report will have what is likely to be a hidden trove of data that paints a damning picture of certain ISPs and their real-world broadband throughput. The data is provided in part by Measurement-Lab, a consortium of organizations including Internet 2, Google, Georgia Tech, Princeton and the Internet Systems Consortium. M-Lab, which gathers broadband performance data and distributes that data to the FCC, has uncovered significant slowdowns in throughput on Comcast, Time Warner Cable and AT&T. Such slowdowns could be indicative of deliberate actions taken at interconnection points by ISPs.
[chart via Google (not an image)]
When contacted prior to publishing this story, AT&T didn’t respond to my request for comment, and both Time Warner Cable and Comcast declined to comment. I had originally asked about data from Verizon and CenturyLink, but M-Labs said those companies had data that was more difficult to map.
So what are we looking at in the above chart? It’s showing the median broadband throughput speeds at the listed ISPs. As you can see, certain providers have seen a noticeable decline in throughput. Measurement Lab was created in 2008 in the wake of the discovery that Comcast was blocking BitTorrent packets. Vint Cerf, who is credited as one of the creators of the internet, and Sascha Meinrath of the Open Technology Institute decided to help develop a broadband measurement testing platform that took into account the experience that an end user of an actual web service like Google or Netflix might experience.
The idea was to capture data on traffic management practices by ISPs and test against servers that are not hosted by the ISP. The company gives its data to the FCC as part of the agency’s Measuring Broadband America Report, and provides the data under an open source license to anyone who asks for it.
The FCC also uses additional data from SamKnows, a U.K. firm that provides routers to customers around the country and tracks their broadband speed, to produce its report. SamKnows did not respond to requests for comment on this story, and the FCC did not respond to my questions about the M-Lab data. So right now it’s an open question if the upcoming Measuring Broadband Report will have M-Lab’s data incorporated into the overall results, or if, because of the terms under which the FCC gets the M-Lab data, the agency will merely release the data without validating it.
Dueling methodologies
Ben Scott, a senior advisor to the Open Technology Institute at the New America Foundation who is working on the M-Lab data, said he and researchers at M-Lab are exploring new ways to test the data to see if they can “give more clarity about the cause or causes” of the slowdown.
While it does that, it will also have to address why its data is so different from the existing FCC data (a source at the Open Technology Institute explained that the FCC says SamKnows data is not showing the same trends) or even data available from Ookla, which runs the popular Speedtest.net broadband tests. Checks with other companies that monitor broadband networks also don’t show these trends. For a contrast, here’s what Ookla shows for Comcast’s speeds over the same time period as the M-Lab data.
Scott said that the goal behind M-Lab’s tests is to replicate what an average user experiences. That means measuring results not just from a carefully tuned server designed for offering bandwidth tests, but to include some of the many and varied hops that a packet might take in getting from Point A to Point B. Thus, the M-Lab tests include data on throttling, latency and over 100 variables that influence performance.
The servers that act as the end point for the M-Lab tests are in a variety of places such as cloud providers, universities and research institutions, and may connect to the end ISP via any number for different transit or CDN providers. For example, Level 3, Cogent, XO, Voxel, Tata and others own some of the transit networks that M-Lab’s tests traverse. Some of these companies such as Cogent have had established peering disputes that have affected traffic on their networks.
It’s at those transit and CDN providers where the packets make those different hops, and that’s where Scott said he and his researchers are focusing.
Ookla, the company behind Speedtest.net, is probably the most popular speed test out there but it also tends to have a few weak points. When you run a speed test using Speedtest.net, the app sends a package of packets to the closest server, which can be hosted at a local ISP or data center where interconnection points are common. There are several areas where the owner of the testing server can “tune” the test so it delivers maximum speeds. From the Ookla wiki:
Ookla also eliminates the fastest 10 percent and slowest 30 percent of the results it receives before averaging the rest to get a sense of the reported throughput. Critics say the ability to tune servers and use ISP-hosted servers skews the results.The Speedtest is a true measurement of HTTP throughput between the web server and the client. Packet size and all other TCP parameters are controlled by the server itself, so there is a certain amount of tuning which can be done to ensure maximum throughput.
The consumer impact is growing, or at least the complaints are
What might look like an esoteric debate over the best way to measure broadband speeds is hiding a real issue for America’s broadband networks. Sources at large content providers believe the M-Lab data shows how ISPs are interfering with the flow of traffic as it reaches their last-mile networks.
So you might get something that looks like this — as I did on Saturday night while watching a show on Amazon (I had a similar experience while watching a Hulu stream the evening before).
Time Warner Cable, my ISP, is investigating why my Blu-ray player was tracking at 1.9 Mbps when an Ookla test show 28 Mbps down.
While I was seeing my episode of The Good Wife falter at what appeared to be 1.9 Mbps, I was able to measure connection speeds of 28 Mbps to my house using a Speedtest.net test from Ookla. This is exactly the dichotomy that the M-Lab data is showing, and my example is not an isolated one; Comcast users have been complaining for months.
During the summer the CEO of Cogent accused Verizon of throttling traffic Cogent was delivering onto the Verizon network because Cogent wasn’t paying for interconnection. Yesterday, an IT worker in Plano, Texas named David Raphael put up a blog post that accuses Verizon of violating network neutrality because it appears to have admitted to blocking Amazon AWS traffic.
The real fight is over a business model for the internet
While peering disputes are not a network neutrality issue because those disputes are not actually governed by the recent legal decision striking down the Open Internet Order, it is an issue of competition and whether the last-mile ISPs are behaving like a monopoly.
Wednesday’s blog post from Raphael documents a Verizon technician apparently admitting that Verizon is throttling Amazon traffic. That might be a mistaken admission by a tech (as Verizon said in a statement) but the post does a credible job of laying out exactly what many consumers are experiencing and providing traceroute documentation.
Verizon’s statement on the post emphasized that it treats all traffic equally before noting that a variety of factors could influence the actual application performance including, “that site’s servers, the way the traffic is routed over the Internet, and other considerations.” For details on the ways the application provider can fail users, Comcast’s head of network engineering provides a much more in-depth post in response to user complaints of poor quality Netflix streams.
ISPs are correct to point out where their control ends and begins. Decisions made on server capacity, whether to buy content delivery services and choosing transit providers have an impact on the ability of content companies to deliver internet content to your TV or computer. Anyone who tries to visit a smaller blog after a post or photo has gone viral has seen those limits in action; those 404 errors are because of insufficient server capacity.
But pointing to Amazon, Netflix, Hulu or other internet giants and assuming they aren’t dedicating the resources to serve their customers is a hard sell. In fact, the pressure to build out that infrastructure may actually be behind some of the escalation in user complaints.
Industry watchers who count both ISPs and content companies as customers say that the decision by Netflix to create its own CDN last summer has prompted ISPs to get more aggressive in their peering negotiations, which has led to the consumer complaints. That aggression may come from not wanting to give Netflix — which increasingly competes with many ISPs’ own pay TV services — a “free ride” into the network, or it may be a grab for incremental revenue from a company that ISPs view as making bank off their pipes. Meanwhile, just this week rumors surfaced that Apple is building its own CDN business.
Colbert can't rise above a poor connection. I pay both Hulu and Time Warner cable, so why is there a problem?
What’s happening is as the traffic on the web has consolidated into a few large players, those players are both a threat to their existing video businesses and a source of revenue for ISPs who control the access to the end consumers. As those players build out their infrastructure, the ISPs are halting them at the edge of their networks with refusals to peer or to peer for pay. The result of that “negotiation” between the two sides can be a slowdown in service as certain CDNs or transit providers are unable to peer directly with an ISP without paying up.
As frustration mounts, intervention seems far away
In conversations with sources at ISPs who are uncomfortable or prohibited from speaking on the record, the feeling is that content providers need to help pay for the upgrades to the last mile network that the rise in overall traffic is causing, as well as frustration that Netflix and others are somehow “getting around paying for transit or CDN services” by building their own systems. ISPs also say they don’t want to have to host half a dozen servers that cache content for the big internet providers with the prospect of more coming as new services grow, citing power and space constraints.
All vehemently deny throttling traffic while pointing out that certain transit providers such as Cogent (every ISP will use Cogent as a scapegoat) are known bad actors and won’t pay to peer directly with ISPs. Unfortunately ISPs gloss over the real debate, which is whether transit providers, content companies and CDNs should have to pay to peer — that is, pay for the right to deliver all of the traffic that an ISP’s users are demanding — given that the end user has paid the ISP to deliver the content the user has asked for?
That is the heart of the debate with issues such as the lack of broadband competition at the last mile, and the possibility that ISPs who have their own pay TV businesses have an interest in blocking competing TV services just adds more complexity. The challenge is proving that such slowdowns are happening, show where they are happening and then have a debate about what should be done about this. The data from M-Lab is a start, and if it can refine the data to deliver proof of ISP wrongdoing, then the FCC should take it into consideration.
So when the Measuring broadband Report eventually comes out, a lot of people will be looking for the M-Lab data. As for right now, myself and other consumers are looking for a conversation about broadband quality that so far the agency isn’t having.
