There’s something rotten in the state of video streaming

[TronPaul]

There’s something rotten in the state of online video streaming, and the data is starting to emerge - Gigaom

If you’ve been having trouble with your Netflix streams lately, or maybe like David Rafael, director of engineering for a network security company in Texas, you’re struggling with what appears to be throttled bandwidth on Amazon Web Services, you’re not alone.

It’s an issue I’ve been reporting on for weeks to try to discover the reasons behind what appears to be an extreme drop on broadband throughput for select U.S. internet service providers during prime time. It’s an issue that is complicated and shrouded in secrecy, but as consumer complaints show, it’s becoming increasingly important to the way video is delivered over the internet.

The problem is peering, or how the networks owned and operated by your ISP connect with networks owned and operated by content providers such as Amazon or Netflix as well as transit providers and content deliver networks. Peering disputes have been occurring for years, but are getting more and more attention as the stakes in delivering online video are raised. The challenge for regulators and consumers is that the world of peering is very insular and understanding the deals that companies have worked out in the past or are trying to impose on the industry today is next to impossible.

Which is why we need more data. And it’s possible that the Federal Communications Commission has that data — or at least the beginnings of that data. The FCC collects data for a periodic Measuring Broadband Quality report that was most recently issued in Feb. 2013. In that report the FCC said it would to look at data from broadband providers during September 2013 and issue a subsequent report in that same year. That hasn’t happened, but the agency is preparing one likely for late Spring. The report measures how fast actual U.S. broadband speeds are relative to the advertised speeds. While the initial report published in 2011 showed that some ISPs were delivering sub par speeds versus their advertised speeds, ISPs have since improved their delivery and FCC rankings. As a result, the reports goals have shifted to measuring mobile broadband and even caps.

But the FCC’s next report will have what is likely to be a hidden trove of data that paints a damning picture of certain ISPs and their real-world broadband throughput. The data is provided in part by Measurement-Lab, a consortium of organizations including Internet 2, Google, Georgia Tech, Princeton and the Internet Systems Consortium. M-Lab, which gathers broadband performance data and distributes that data to the FCC, has uncovered significant slowdowns in throughput on Comcast, Time Warner Cable and AT&T. Such slowdowns could be indicative of deliberate actions taken at interconnection points by ISPs.

[chart via Google (not an image)]

When contacted prior to publishing this story, AT&T didn’t respond to my request for comment, and both Time Warner Cable and Comcast declined to comment. I had originally asked about data from Verizon and CenturyLink, but M-Labs said those companies had data that was more difficult to map.

So what are we looking at in the above chart? It’s showing the median broadband throughput speeds at the listed ISPs. As you can see, certain providers have seen a noticeable decline in throughput. Measurement Lab was created in 2008 in the wake of the discovery that Comcast was blocking BitTorrent packets. Vint Cerf, who is credited as one of the creators of the internet, and Sascha Meinrath of the Open Technology Institute decided to help develop a broadband measurement testing platform that took into account the experience that an end user of an actual web service like Google or Netflix might experience.

The idea was to capture data on traffic management practices by ISPs and test against servers that are not hosted by the ISP. The company gives its data to the FCC as part of the agency’s Measuring Broadband America Report, and provides the data under an open source license to anyone who asks for it.

The FCC also uses additional data from SamKnows, a U.K. firm that provides routers to customers around the country and tracks their broadband speed, to produce its report. SamKnows did not respond to requests for comment on this story, and the FCC did not respond to my questions about the M-Lab data. So right now it’s an open question if the upcoming Measuring Broadband Report will have M-Lab’s data incorporated into the overall results, or if, because of the terms under which the FCC gets the M-Lab data, the agency will merely release the data without validating it.
Dueling methodologies

Ben Scott, a senior advisor to the Open Technology Institute at the New America Foundation who is working on the M-Lab data, said he and researchers at M-Lab are exploring new ways to test the data to see if they can “give more clarity about the cause or causes” of the slowdown.

While it does that, it will also have to address why its data is so different from the existing FCC data (a source at the Open Technology Institute explained that the FCC says SamKnows data is not showing the same trends) or even data available from Ookla, which runs the popular Speedtest.net broadband tests. Checks with other companies that monitor broadband networks also don’t show these trends. For a contrast, here’s what Ookla shows for Comcast’s speeds over the same time period as the M-Lab data.

Scott said that the goal behind M-Lab’s tests is to replicate what an average user experiences. That means measuring results not just from a carefully tuned server designed for offering bandwidth tests, but to include some of the many and varied hops that a packet might take in getting from Point A to Point B. Thus, the M-Lab tests include data on throttling, latency and over 100 variables that influence performance.

The servers that act as the end point for the M-Lab tests are in a variety of places such as cloud providers, universities and research institutions, and may connect to the end ISP via any number for different transit or CDN providers. For example, Level 3, Cogent, XO, Voxel, Tata and others own some of the transit networks that M-Lab’s tests traverse. Some of these companies such as Cogent have had established peering disputes that have affected traffic on their networks.

It’s at those transit and CDN providers where the packets make those different hops, and that’s where Scott said he and his researchers are focusing.

Ookla, the company behind Speedtest.net, is probably the most popular speed test out there but it also tends to have a few weak points. When you run a speed test using Speedtest.net, the app sends a package of packets to the closest server, which can be hosted at a local ISP or data center where interconnection points are common. There are several areas where the owner of the testing server can “tune” the test so it delivers maximum speeds. From the Ookla wiki:

The Speedtest is a true measurement of HTTP throughput between the web server and the client. Packet size and all other TCP parameters are controlled by the server itself, so there is a certain amount of tuning which can be done to ensure maximum throughput.

Ookla also eliminates the fastest 10 percent and slowest 30 percent of the results it receives before averaging the rest to get a sense of the reported throughput. Critics say the ability to tune servers and use ISP-hosted servers skews the results.
The consumer impact is growing, or at least the complaints are

What might look like an esoteric debate over the best way to measure broadband speeds is hiding a real issue for America’s broadband networks. Sources at large content providers believe the M-Lab data shows how ISPs are interfering with the flow of traffic as it reaches their last-mile networks.

So you might get something that looks like this — as I did on Saturday night while watching a show on Amazon (I had a similar experience while watching a Hulu stream the evening before).
Time Warner Cable, my ISP, is investigating why my Blu-ray player was tracking at 1.9 Mbps when an Ookla test show 28 Mbps down.

While I was seeing my episode of The Good Wife falter at what appeared to be 1.9 Mbps, I was able to measure connection speeds of 28 Mbps to my house using a Speedtest.net test from Ookla. This is exactly the dichotomy that the M-Lab data is showing, and my example is not an isolated one; Comcast users have been complaining for months.

During the summer the CEO of Cogent accused Verizon of throttling traffic Cogent was delivering onto the Verizon network because Cogent wasn’t paying for interconnection. Yesterday, an IT worker in Plano, Texas named David Raphael put up a blog post that accuses Verizon of violating network neutrality because it appears to have admitted to blocking Amazon AWS traffic.
The real fight is over a business model for the internet

While peering disputes are not a network neutrality issue because those disputes are not actually governed by the recent legal decision striking down the Open Internet Order, it is an issue of competition and whether the last-mile ISPs are behaving like a monopoly.

Wednesday’s blog post from Raphael documents a Verizon technician apparently admitting that Verizon is throttling Amazon traffic. That might be a mistaken admission by a tech (as Verizon said in a statement) but the post does a credible job of laying out exactly what many consumers are experiencing and providing traceroute documentation.

Verizon’s statement on the post emphasized that it treats all traffic equally before noting that a variety of factors could influence the actual application performance including, “that site’s servers, the way the traffic is routed over the Internet, and other considerations.” For details on the ways the application provider can fail users, Comcast’s head of network engineering provides a much more in-depth post in response to user complaints of poor quality Netflix streams.

ISPs are correct to point out where their control ends and begins. Decisions made on server capacity, whether to buy content delivery services and choosing transit providers have an impact on the ability of content companies to deliver internet content to your TV or computer. Anyone who tries to visit a smaller blog after a post or photo has gone viral has seen those limits in action; those 404 errors are because of insufficient server capacity.

But pointing to Amazon, Netflix, Hulu or other internet giants and assuming they aren’t dedicating the resources to serve their customers is a hard sell. In fact, the pressure to build out that infrastructure may actually be behind some of the escalation in user complaints.

Industry watchers who count both ISPs and content companies as customers say that the decision by Netflix to create its own CDN last summer has prompted ISPs to get more aggressive in their peering negotiations, which has led to the consumer complaints. That aggression may come from not wanting to give Netflix — which increasingly competes with many ISPs’ own pay TV services — a “free ride” into the network, or it may be a grab for incremental revenue from a company that ISPs view as making bank off their pipes. Meanwhile, just this week rumors surfaced that Apple is building its own CDN business.
Colbert can't rise above a poor connection. I pay both Hulu and Time Warner cable, so why is there a problem?

What’s happening is as the traffic on the web has consolidated into a few large players, those players are both a threat to their existing video businesses and a source of revenue for ISPs who control the access to the end consumers. As those players build out their infrastructure, the ISPs are halting them at the edge of their networks with refusals to peer or to peer for pay. The result of that “negotiation” between the two sides can be a slowdown in service as certain CDNs or transit providers are unable to peer directly with an ISP without paying up.
As frustration mounts, intervention seems far away

In conversations with sources at ISPs who are uncomfortable or prohibited from speaking on the record, the feeling is that content providers need to help pay for the upgrades to the last mile network that the rise in overall traffic is causing, as well as frustration that Netflix and others are somehow “getting around paying for transit or CDN services” by building their own systems. ISPs also say they don’t want to have to host half a dozen servers that cache content for the big internet providers with the prospect of more coming as new services grow, citing power and space constraints.

All vehemently deny throttling traffic while pointing out that certain transit providers such as Cogent (every ISP will use Cogent as a scapegoat) are known bad actors and won’t pay to peer directly with ISPs. Unfortunately ISPs gloss over the real debate, which is whether transit providers, content companies and CDNs should have to pay to peer — that is, pay for the right to deliver all of the traffic that an ISP’s users are demanding — given that the end user has paid the ISP to deliver the content the user has asked for?

That is the heart of the debate with issues such as the lack of broadband competition at the last mile, and the possibility that ISPs who have their own pay TV businesses have an interest in blocking competing TV services just adds more complexity. The challenge is proving that such slowdowns are happening, show where they are happening and then have a debate about what should be done about this. The data from M-Lab is a start, and if it can refine the data to deliver proof of ISP wrongdoing, then the FCC should take it into consideration.

So when the Measuring broadband Report eventually comes out, a lot of people will be looking for the M-Lab data. As for right now, myself and other consumers are looking for a conversation about broadband quality that so far the agency isn’t having.

[TheFeniX]

Ah, so this explains the wife's issue with Netflix. I just thought it was low bandwidth and wasn't going to both looking into it until we moved (hopefully, very soon now).

Comcast and at&t are utter garbage (at&t being mildly less rotten in this regard), but they are the only gigs generally available outside wireless providers and satellite, neither of which I can use as a gamer.

All vehemently deny throttling traffic while pointing out that certain transit providers such as Cogent (every ISP will use Cogent as a scapegoat) are known bad actors and won’t pay to peer directly with ISPs.

I'm pretty sure Comcast was caught red-handed a few years ago doing this and I know for sure they were throttling my bandwidth years back as it was the reason I left them for U-Verse. That and the years old equipment they wouldn't upgrade. Like, literally would not upgrade, I couldn't even pay them to do it. New customers got the good stuff, existing customers could get fucked.

tl;dr: this isn't a surprise to anyone. When Comcast bought out huge swaths of Time Warner real estate in Texas, the drop in quality was immediately apparent. They're basically the EA Games of service providers. at&t just fired anyone who knew what they were doing and they're so broken up into different little "businesses," you may not even be talking to the same support guys.

[Borgholio]

New customers got the good stuff, existing customers could get fucked.

That sort of thing happens all the time and always gets me in a huff. You'd think that keeping people for long-term income would be more valuable to them than constantly sending out techs to connect / disconnect hardware and having accounting drones constantly turning on and off service.

I mean it seems it's more important to always attract new customers than it is to keep existing ones. They should be equally important.

[TheFeniX]

EDIT: I got off on an off-topic rant. Suffice to say the idea that new customers deserve better treatment than existing ones is a big issue with me currently.

[Oskuro]

I mean it seems it's more important to always attract new customers than it is to keep existing ones. They should be equally important.

Depends on the metrics you live by. If your shareholders/corporate culture puts more value on growth than long-term sustainability (which seems to be the case with... Well, with pretty much everyone) then focusing on new customers is the logical course of action.

[Borgholio]

Wouldn't growth be overall higher if you keep a large base of loyal customers and add to it at a steady rate, rather than losing people who are tired of your dickery and trying to lure them back again?

[TronPaul]

The issue with those statements is that the state of the US ISP industry is of local monopolies. There are 3 providers in the Chicagland area: Comcast (cable), AT&T (dsl), and RCN (cable). Comcast provides the fastest most reliable speeds, but can cause throughput issues. AT&T has crap speed offerings (6mbps is the maximum) and far over Comcast's prices. RCN has decent speed and prices, but services can drop rapidly during peak use. Comcast is the only viable high speed choice. Even if Comcast overcharges me, or screws me with stupid techs I will continue to use them as I cannot get any speed otherwise.

The same holds true for much of the US. One main cable provider and one main phone provider who sell access to their lines to one or two competitiors who have much worse speed or throughput.

[TheFeniX]

Yea, it's atrocious here as well. at&t bought back up a lot of what they lost and Comcast was in the business of doing the same. Also the exclusivity contracts they bought with many apartment complexes and some neighborhoods. It's a fucking joke what Americans have to put up with Internet providers. They can fuck you because they know you'll come crawling back, if you can leave at all, once the other group of assholes get done with you.

Your only real option is to just take them up on all the free goodies, then swap back to the other guys for more free goodies. However, it's just damned annoying. I'm considering dealing with satellite TV and only worrying about swapping Internet providers when we get our new place. But the wife "needs" certain channels, so I might be screwed either way.

[Lord Revan]

there's still 6 mbps max connections?! I mean sure Finland is suppose to be at top countries when it comes internet connections but connection is 8 mbps (IIRC) and it would have been easily faster if it wasn't for the older lines (I couldn't get any faster connection) where I live and I pay only about 30€ per month for my connection too.

[Edi]

With regard to internet connections, consumer protection and such, the US is a third world banana republic. The big companies shamelessly use regional monopolies and anti-competitive practices and push through laws that prevent sny competitors even from entering the market. As a result, they can gouge whatever prices they want from a captive audience and generally act like goat molesting fuckwits. Since they also own the legislatures (like industry groups generally do in the US, both at the local and federal level), there is practially no regulation worth the name to protect the little guy.

If any ISP pulled even 10% of that shit in Finland, FICORA and the courts would be give it such an ass-reaming it would be remembered with horror twenty years from now.

[InsaneTD]

Yea, it's atrocious here as well. at&t bought back up a lot of what they lost and Comcast was in the business of doing the same. Also the exclusivity contracts they bought with many apartment complexes and some neighborhoods. It's a fucking joke what Americans have to put up with Internet providers. They can fuck you because they know you'll come crawling back, if you can leave at all, once the other group of assholes get done with you.

Your only real option is to just take them up on all the free goodies, then swap back to the other guys for more free goodies. However, it's just damned annoying. I'm considering dealing with satellite TV and only worrying about swapping Internet providers when we get our new place. But the wife "needs" certain channels, so I might be screwed either way.

Might be able to get that channel online. Some of the bigger channels allow you to subscribe to them that way. I'd use that to get a couple I want but they won't deal with outside the US.

[muse]

Of course they're throttling, the US infrastructure, particularly the local nodes and "last mile" aren't built to handle constant high bandwidth use. They were built with high bandwidth at low duty cycles in mind, in other words mostly web browsing and once in a while a large file is transferred, they weren't meant to be hammered on at full bandwidth for hours on end as would happen with video streaming.

Having large intermittent loads is quite different than high constant loads, with the former it's unlikely that everyone in the neighbourhood is spiking their usage at the same time so the local node can get away with using less equipment, so that's what they did. Video streaming makes it far easier to max out the local node since everyone is demanding full bandwidth for hours at a time instead of minutes, there's a lot more demand overlap and the local node gets swamped.

So they have two choices, throttle bandwidth or upgrade their infrastructure and charge the costs to their customers. The latter isn't cheap, price out a commercial/business line with the same bandwidth as your home plan and that's what your home plan would cost.

With that said, the "last mile" in the US is a monopoly with all the wonderful things that go along with it. I'm really not sure what the solution would be.

[Borgholio]

The issue with those statements is that the state of the US ISP industry is of local monopolies. There are 3 providers in the Chicagland area: Comcast (cable), AT&T (dsl), and RCN (cable). Comcast provides the fastest most reliable speeds, but can cause throughput issues. AT&T has crap speed offerings (6mbps is the maximum) and far over Comcast's prices. RCN has decent speed and prices, but services can drop rapidly during peak use. Comcast is the only viable high speed choice. Even if Comcast overcharges me, or screws me with stupid techs I will continue to use them as I cannot get any speed otherwise.

The same holds true for much of the US. One main cable provider and one main phone provider who sell access to their lines to one or two competitiors who have much worse speed or throughput.

I was referring to TV as well, which gives even more options in the way of satellite. They're always offering good deals to those who sign up but even if you had a dish on your roof from the same company for years they don't give you shit.

[Edi]

With that said, the "last mile" in the US is a monopoly with all the wonderful things that go along with it. I'm really not sure what the solution would be.

Tighten regulations, forcibly break up the local loop monopoly and impose a mandate that the local loop must be leased to competitors at a reasonable price if the customer wants competitor's product.

This will force prices down due to competition, encourage network infrastructure upgrades and generally create a healthy market. It's what was done in Finland nearly 15 years ago and as a result, we have some of the lowest prices for best performance anywhere in the world in broadband and also in mobile broadband. And given how low our population density is that is saying something (even if big chunks are clustered in the larger cities).

Places like Japan and South Korea have even lower prices, but their population densities are such that for the same investment to network infra, they have easily five hundred times the customers to help spread the costs out.

[bilateralrope]

The same thing happened in New Zealand. While I doubt our prices or performance compare well to Finland, they are fast enough that I'm not sure if I would notice faster speeds, plans without transfer limits are affordable and my ISP has a free service that lets me bypass regional restrictions on websites without me needing to anything more than use their DNS servers.