MESSAGE
| DATE | 2014-05-06 |
| FROM | Ruben Safir
|
| SUBJECT | Re: [NYLXS - HANGOUT] Fwd: Net neutrality emergency
|
Observations of an Internet Middleman
Mark Taylor / 1 day ago
We received a lot of positive feedback, as well as a lot of questions
when Mike posted his recent story, Chicken. Many of the questions asked
for more specific data about the scale and size of the problem. This
post is an attempt to provide some of that data.
Our Internet Services product sits on top of this global infrastructure:
network_mapThe orange lines are cable systems that Level 3 built and
fully owns (the yellow lines are owned by multiple carriers or leased).
That means thousands of miles of fiber in trenches across land and
thousands of miles of fiber in cables on the seabed. In all, our network
contains approximately 180,000 miles of fiber – enough to circle the
equator seven times.
That fiber is then turned into usable bandwidth by installing equipment
in data centers in each of those red and yellow dots, and also roughly
every one hundred miles along each terrestrial cable. That bandwidth is
then turned into an Internet Service by installing routers and switches
in key locations. The Level 3 Internet Service consists of more than
10,000 Ethernet connections – getting bigger every month. The
original invested capital in the Level 3 network was approximately $40
billion.
Level 3 uses that network to sell Internet Services to tens of thousands
of customers all around the world. But, despite the huge amount of
infrastructure that we built and contribute, we are only one part of the
global Internet. When we sell Internet Services, we have to make
available every single route on the Internet to our customers – not
just the routes we ourselves own. That means we have to provide access
to all of the networks owned and operated by others, which right now
means about 46,000 other networks – some of which also make use of
Level 3’s fiber and bandwidth services.
Level 3 builds a route map of the Internet by connecting its tens of
thousands of customers together and allowing them to communicate. So a
Level 3 customer in Hong Kong can communicate with a Level 3 customer in
Sao Paulo. But to complete the map we also need to fill in
interconnection to everyone who isn’t a direct Level 3 customer, so
that our customers can also communicate with those who are not our
customers. We do that through connections to other networks and their
customers. This latter sort of connectivity is often called peering.
Peering connections allow for exchanges of traffic between the
respective customers of each peer.
While Level 3 has tens of thousands of customers, it only has 51
peers[1]. That total set of interconnections enables our customers to
“see” the whole Internet. And what is important here is the
“distance” our customers see between themselves and any other
part of the Internet. That is often referred to as the number of
“hops”; or number of other networks a packet has to traverse to
reach its destination. We strive to make that number as low as possible
to offer our customers the best performance; more hops can introduce
more delay and more potential for quality degradations when the other
networks don’t invest enough in performance, redundancy and
capacity.
So how does all this compare to other networks? Renesys does a good job
describing the interconnectedness of the Internet, but their reports are
often misunderstood. They do not show how much traffic each network
carries on any sort of relative basis. They merely show the
interconnectedness of the networks on a relative basis. As you can see,
Level 3’s network is the most interconnected. This list of companies
along with our combined investment, innovation and competition has
enabled the Internet to grow dramatically by carrying enormous flows of
traffic around the globe. Removing these middlemen would leave a massive
hole in the Internet.
Much has been made of peering agreements. Many peering agreements were
made between engineers in the early days of the Internet and consisted
of not much more than a single page of text – if there was anything
written down at all. They weren’t really contracts in the way you
might consider a formal legal agreement. But over the last decade or so,
they have become legal contracts that have a defined term and a set of
expectations that each party agrees to adhere to. The vast majority of
those contracts are settlement free. For example, 48 of the 51 Level 3
peering agreements are settlement free. In one case, a peer pays us for
access to a number of routes in a region where their network doesn’t
go; a choice they made rather than buying Internet Services from another
party. As we have explained a number of times, our policy is to refuse
to pay arbitrary charges to add interconnection capacity (more detail to
come in our forthcoming solutions blog post).
But there are also typically shared costs for networks to interconnect.
Each party pays to augment its own network to allow for more traffic
exchange (the expense to augment capacity is not significant for either
party). And since we often choose to interconnect in a third party data
center, the networks usually agree to share the cost of the cross
connects by paying for them on an alternating basis.
The table below shows the connection locations Level 3 has with its
peers, and the total interconnection capacity exceeds 13,600Gbps.
peering_routesLevel 3 has 51 peers that are interconnected in 45 cities
through over 1,360 10 Gigabit Ethernet ports (plus a few smaller ports).
The distribution of that capacity with individual peers ranges from a
single 10 Gigabit Ethernet port to 148 ports. The average number of
interconnection cities per peer is five, but ranges from one to 20.
The average utilization across all those interconnected ports is 36
percent. So you might be asking – what is all the fuss about with
peering? And why did we write the Chicken post? Well, our peers fall
into two broad categories; global or regional Internet Services
providers like Level 3 (those “middlemen” listed in the Renesys
report), and Broadband consumer networks like AT&T. If I use that
distinction as a filter to look at congested ports, the story looks very
different.
A port that is on average utilised at 90 percent will be saturated,
dropping packets, for several hours a day. We have congested ports
saturated to those levels with 12 of our 51 peers. Six of those 12 have
a single congested port, and we are both (Level 3 and our peer) in the
process of making upgrades – this is business as usual and happens
occasionally as traffic swings around the Internet as customers change
providers.
That leaves the remaining six peers with congestion on almost all of the
interconnect ports between us. Congestion that is permanent, has been in
place for well over a year and where our peer refuses to augment
capacity. They are deliberately harming the service they deliver to
their paying customers. They are not allowing us to fulfil the requests
their customers make for content.
Five of those congested peers are in the United States and one is in
Europe. There are none in any other part of the world. All six are large
Broadband consumer networks with a dominant or exclusive market share in
their local market. In countries or markets where consumers have
multiple Broadband choices (like the UK) there are no congested peers.
As an example, this is what one of those congested interconnections
looks like. It is a 100Gbps interconnect in Dallas for the week ending
April 3. The graph on the left shows flat tops for most of each day
– the port is congested and cannot accept all of the traffic that is
trying to get through. Not only are packets being dropped (the number
dropped are on the right), but all those not being dropped are also
subject to delay. The effect of dropped and delayed packets is
discussed in our prior post.
route_info_1For comparison, below is an uncongested interconnection.
This is also 100Gbps but in Washington, D.C. with another peer. This
shows no congestion, although there isn’t much headroom, so a
capacity augment is underway. The graph on the right shows absolutely no
dropped packets.
route_info_2One final point; the companies with the congested peering
interconnects also happen to rank dead last in customer satisfaction
across all industries in the U.S.[2] Not only dead last, but by a
massive statistical margin of almost three standard deviations.
csatShouldn’t a broadband consumer network with near monopoly
control over their customers be expected, if not obligated, to deliver a
better experience than this?
[1] Our peers are generally other networks that complement our own and
that agree with us to exchange traffic on fair and reasonable terms
– see our peering policy.
[2] According to the 2013 American Customer Satisfaction Index (
www.theacsi.org )
The following two tabs change content below.
Bio
Latest Posts
Mark Taylor
I work as VP of Content and Media here at Level 3. English expat and
passionate new tech energy evangelist.
Related posts:
Digital Technology Interface“Chicken” | A Game Played as a Child
and by some ISPs with the Internet iStock_000020193425Small-1The
“Internet of Things” Challenge offense defensePeering Disputes
– The Passive Aggressive Approach Traffic jamQoS Doesn’t Work
May 5, 2014 in Global Connectivity. Tags: global connectivity, internet
congestion, internet service provider, packet loss, peer to peer,
peering
Post navigation
??Observations of an Internet Middleman”
Raindeer
May 5, 2014 at 9:31 am
So why does Level 3 not terminate the peering agreement with these
companies. They are clearly in violation of your peering policy. If they
want access to your customers they would be better off buying access
from a transit provider.
:-)
Reply
Mark Taylor
May 5, 2014 at 12:42 pm
Because if we terminated the connectivity we would damage the
customer experience (for our customers and for the broadband
provider’s customers) even more. We are not willing to do that.
Reply
Jeff
May 5, 2014 at 2:14 pm
Perhaps it’s what needs to happen to show how fragile
Comcast’s (and other networks) peering really is. Shove a few extra
x00 Gbps through their already congested Tata ports.
No pain, no gain?
Reply
Tristan
May 5, 2014 at 2:46 pm
Solid. As consumers, how can we deal directly with Level 3
instead of these broadband providers?
Reply
Steinar H. Gunderson
May 5, 2014 at 3:31 pm
You cannot. The closest thing you can do is to to try
pick an ISP that has good transit connectivity, but as a consumer,
it’s really hard to figure this out, and you hardly ever have a
choice of ISPs anyway.
You could of course try to complain to their helpdesk,
but the odds are that it will never be forwarded anywhere.
Reply
Jeff
May 5, 2014 at 3:43 pm
You most likely can’t since you’ll need to meet
networks like Level3 at a colocation facility or be lucky enough to have
access to transport from a last-mile network to reach a colocation
facility. Costs a residential consumer will likely not want to bear.
Reply
Chris Snell
May 6, 2014 at 10:59 am
Mark, I feel that you’re damaging your own
customers’ experience by maintaining these saturated interconnects.
The customers (both Level3?s and the cable companies’) see the
packet loss and there’s a 50% chance that they’re blaming
Level3. Consequently, by allowing this situation to continue, Level3 is
diminishing the value of it’s own service.
Level3 and the rest of the short list of global fiber
providers are the only ones with the power to do anything about this
situation. Force these guys to fulfill their end of the agreement or
give them the boot. Your customers will thank you.
-A Comcast Business Class customer
Reply
Mark Taylor
May 6, 2014 at 4:06 pm
“Giving them the boot”, as you say, would
actually make the situation worse for our customers and for the
broadband network’s customers. That is not something we are willing
to do.
Reply
Nydeiron
May 5, 2014 at 1:09 pm
How much would fixing the interconnection issue at a single location
like your Dallas example cost for an ISP? I’m thinking it’s not
in the millions of dollars.
Reply
Mark Taylor
May 5, 2014 at 2:45 pm
Each 10Gbps port would be $10-20k. Additional costs might be
triggered if there were no more card slots in the router there.
Reply
Jeff
May 5, 2014 at 3:34 pm
Marks figure is about inline with what my cost would be on a
Juniper MX.
Which really isn’t that much money to these huge
networks.
Assuming a billion dollars in net profit a year. Let’s
say the cost of the port hardware is $20k / per port, that’s 10-11
minutes of operating time to pay for that port. Surely these networks
have much larger economies of scale so you can come to your own
conclusions on the cost for ports to these networks.
Reply
Aguy
May 5, 2014 at 10:20 pm
10 Gbit for 20k? wow
how come some datacenters providers sells 10gbit connections
lines guaranteed for 500-1k usd for colo or server rent? Overselling?
Reply
Mark Taylor
May 6, 2014 at 1:04 pm
That’s the cost of the network interface card. Which
I think answers the question I was asked. There is a network behind that
which costs a lot more to build and maintain. Actually that was the
point of the first part of the post. To give some idea of the scale and
cost of building a global network.
Reply
Bill Woodcock
May 5, 2014 at 1:16 pm
Mark:
I’m very glad to see this degree of detail, backing up an
excellent post. I think it’s invaluable to shine a light on Internet
peering global norms, as well as the harm to consumers that’s done
by those who deviate from them. Thank you for a very well-written,
well-supported, and timely article. We couldn’t agree more with the
sentiments expressed, and our own data show a very similar group of
outliers, who are abusing their own customers to extract excess rent
from their effective monopolies:
https://www.pch.net/resources/papers//peering-survey/PCH-Peering-Survey-2011.pdf
See figures 6 and 7.
Thanks,
-Bill Woodcock
Reply
Jay
May 5, 2014 at 1:30 pm
“Shouldn’t a broadband consumer network with near monopoly
control over their customers be expected, if not obligated, to deliver a
better experience than this?”
I don’t think you understand how monopolies work :P
But on a serious note, thanks for writing this and the
“Chicken” article.
Reply
Eric Cope
May 6, 2014 at 9:57 am
Earned monopolies must keep up a minimum customer experience or
a start-up can swoop in and take big chunks of market share. Government
enforced monopolies don’t have this problem. Most Telecoms are
government enforced monopolies because municipal governments won’t
grant easement to lay new infrastructure, or state governments won’t
issue licenses, etc.
Reply
Hugh
May 6, 2014 at 1:17 pm
Precisely.
Reply
AdamT
May 5, 2014 at 1:58 pm
So which five are the guilty ISP’s serving which areas? Can you
name names?
Reply
Mark Taylor
May 5, 2014 at 3:42 pm
At this time, we have decided not to specifically identify the
peers with significant congestion; however we can say they are large
incumbent broadband providers in the U.S. and Europe.
Reply
Joe Decker
May 5, 2014 at 7:15 pm
That’s a pity.
Reply
Pete Mitchell
May 5, 2014 at 7:55 pm
Oh, come on. Name and shame them.
Reply
T
May 5, 2014 at 10:26 pm
I’m sure there are other considerations at play here,
but on the surface the decision not to name the offenders seems to
contradict your claim to put customer satisfaction first. Nothing will
change as long as taxpayers remain unaware of the problem providers. If
nothing changes they will have no incentive to improve their service.
Reply
T
May 5, 2014 at 10:39 pm
Just want to add that the bad peers probably have a lot
of political power (which is how they became a monopoly in the first
place, and how they manage to keep their monopoly position despite
atrocious customer satisfaction), so your caution is understandable.
Reply
John Nagro
May 6, 2014 at 6:42 am
name names! sunlight is the best disinfectant. you should
regularly publish a regional carrier congestion report, give consumers
the information they need to decide (or elect officials to decide). such
as the Netflix ISP Index http://ispspeedindex.netflix.com/usa
Reply
Bob Evans
May 5, 2014 at 2:42 pm
Mark, You explained this very well. Thank You….
As far as customer satisfaction surveys go, you will never find ISPs
in the higher ranking industries. This is due to the the fact that
people use IP traffic more hours per day than they travel on an
airplane. People today, recall the problem they had for 10 minutes and
they will remember it for months. It overshadows all the years of great
performance experience. Clearly, industry satisfaction comparison is not
relative. In this industry, ISPs are only as good as the length of time
since your last outage. ISPs even get blamed for Google’s outages. A
single Google resource un-available while someone at Google changes that
routing is the ISPs fault. The end customer feels the entire ISP network
was down because they couldn’t reach that one server.
Reply
Pingback: An internet middleman's take on net neutrality - iDoss
Morgan
May 5, 2014 at 3:23 pm
if Comcast was bold enough to get Tom Wheeler his job as FCC
chairman what else do you think they’ll do to protect their revenue
streams? you guys need to stop being so naive. Comcast owns the FCC and
most of Congress. This blog post will do nothing. If you want anything
to change then you guys need to take action. Call Milo Medin at Google
and make him an offer he cannot refuse. You guys need to help bring in
new blood. Can you tell us how it is working with Google Fiber compared
to Comcast? Which company would you say you would prefer? Everyone I
know would switch to Google Fiber in an instant if they had the chance
to. Comcast, Time Warner, Verizon, Cox, AT&T… these companies
you’re afraid to even mention by name… they will never change
their ways. It’s on YOU to help us, otherwise they will just keep
growing and becoming more powerful.
Reply
Carl Youngblood
May 6, 2014 at 12:06 am
Morgan, Mark is, by his own admission, an Internet “middle
man.” He does not walk in the same circles as the people you
mention, nor does he have control over these decisions.
Reply
Joseph
May 5, 2014 at 4:03 pm
Has Level 3 ever considered getting into the last mile game? I know
it would be expensive, but you guys literally own the backbone already.
Why not drum up some investors or partners and start expanding fiber to
people’s homes? We’re (consumers) out here ready to buy. Hook up
with some electric utilities or others. We’re begging you! Get us
out of the clutches of AT&erizon!
Help us Level 3 Kenobi, you’re our only hope!
Reply
BenC
May 5, 2014 at 4:14 pm
Mark, I understand their may be legal reasons that you won’t
disclose the names of the companies, but if consumers are to fight back
they must be informed. If it would truly cost the ISPs less than a
dollar per subscriber to add another interconnect between your
networks…well, that’s indefensible to their customers. but the
customers have to know.
Reply
Level3 CDN
May 5, 2014 at 7:41 pm
Looking at those graphs is interesting. The traffic appears to be
very unidirectional. Does Level 3 peer with any companies that have the
same traffic profile, but in the reverse? Does Level 3 offer peering to
Netflix, Akamai and other CDNs.
Reply
Mark Taylor
May 6, 2014 at 10:41 am
The answer to the first part of your question is yes. We are not
mentioning any names.
Reply
Yanwen Xia
May 5, 2014 at 8:55 pm
Thanks for the clear presentation of the internet service. Now I
have some idea why our internet speed is so slow. We used to be Time
Warner customer, then switched to AT&T when TW kept raising price and
lowing quality. Before I made the switched, I called and made sure the
performance was ok for what I normally do everyday, like browsing,
internet shopping, watching youtube. After a month or so, the speed
became intolerably slow and I wanted to switch back.
I don’t understand why. Is there anything that customers can do
about this?
I just came back from Beijing. There is a firewall against some
sites, but people can always use some software to “jump over the
wall.” The speed is super fast and the cost is half of what I pay
here in the states. Plus, there are many free internet access after you
register [free]. Honestly, the U.S. is supposed to be the richest
country in the world and yet still enjoy second-class service.
Reply
Randall
May 6, 2014 at 12:17 am
I’d be so, so happy if a future post listed the providers with
technical details, or at least if L3 announced plans to do so if nothing
changes.
Reply
Tom
May 6, 2014 at 1:03 am
How much of a dent can companies with a web presence put into this
problem if they were to compress, minify and optimize their content to
reduce overall internet bandwidth consumption?
I think bandwidth costs for many large companies are considered
negligible when compared to the rest of their expenses. Large
corporations mainly care about making money. Reducing bandwidth
consumption would be the very last thing on their mind. What can be done
to make bandwidth conservation a priority for all websites?
Reply
Mark Taylor
May 6, 2014 at 1:05 pm
Actually all content owners who care about performance already
do this to the greatest extent possible. That’s because reducing the
payload improves the delivery speed. And faster speeds are directly
correlated to higher customer experience scores and greater revenue per
customer.
Reply
Joly MacFie
May 6, 2014 at 1:06 am
If consumers are to mount a “mad as hell” campaign, what
exactly should they be demanding? NN doesn’t seem to exactly cut it
– it’s noy a question of equality but capacity, and the
“cheesing” of both customers and content, holding them to ransom
for a premium.. I have a button machine, maybe I should make some
“NO CHICKEN” badges?
Reply
Pingback: Internet congestionata (ISPs vs Level 3)
Rutger
May 6, 2014 at 5:09 am
Feel like naming and shaming? I am especially curious who the one
European peer is that shows this behaviour.
Reply
Andrew
May 6, 2014 at 6:40 am
Hey Mark, great post!
East coast US resident here. If you cannot name the guilty parties,
could you name 1-2 companies who are NOT intentionally congesting
traffic? If no to that, will this type of data be public in any form for
those who want to go digging around?
Thanks for posting this. I have a small hope now that Comcast
won’t rule the world.
-Andrew
Reply
Mark Taylor
May 6, 2014 at 9:09 am
We are not disclosing any names at this time. The data isn’t
readily available unfortunately. As I said in another reply the Sam
Knows web site and TE Netflix blog have some data that shed some light
on it.
Reply
jackroberts
May 6, 2014 at 7:48 am
Hi great article. I always assumed it was down to the ISP not paying
the tier 1 isp sufficient amounts and thus over subscribing their
available bandwidth. You would not be able to get away with such fraud
in any other industry but thanks to contention ratio non sense the ISP
can get away with over subscribing bandwidth because they rely on the
customers to not use all their available bandwidth all the time.
May be it is time for Level 3 to start selling gigabit internet to
residential users (in the us) and instead of cutting the middle man out,
cut the ISP out that are using your bandwidth to rip off the end user.
Reply
Ray
May 6, 2014 at 8:08 am
Where can we get the data on usage between peers? It would fairly
easy to make a website showing congestion on a local ISP if we can get
the data from the bandwidth providers. This tool would really help
smaller ISPs get more customers and hopefully start to get away from the
monopoly we have today.
I am currently working on my CCNP. Would I need a CCIE to work with
L3?
Reply
Mark Taylor
May 6, 2014 at 9:14 am
The Sam Knows website and the Netflix streaming data on their
blog get close to this. But it’s actually hard to do because there
are so many network connections. Figuring out all the underlying
topology is very, very difficult. A problem for this task but actually a
strength for the resilience of the Internet.
Reply
Luther
May 6, 2014 at 9:12 am
Monopolies have no incentives to make their customer’s
experiences better.
The nay-sayers would argue we have no broadband monopolies in the
U.S., but THEY LIE. There are generally two players in each region. The
cable operator and the phone company. Only Verizon has attempted to
bring a competitive Internet service to their regions to compete against
the incumbent cable operator. So in most regions this leaves the cable
operator as the single, high speed carrier.
And Comcast, Time Warner knows this is the case.
And they want to squeeze every penny they can from any point of
revenue – from the consumers to the peering network providers to
even the content creators (Netflix).
The problem with Internet traffic is that the long haul segments
have to grow capacity in lock step with the last mile providers. If
either one of them gets greedy, the whole system becomes out of
balanced.
Right now, the last mile providers have the advantage.
We, the consumers, wait patiently for a better last mile connection
like we waited for the cable operators to save us from the old plain old
telephone line providers and their dial-up and DSL.
Isn’t it ironic that our saviors 14 years ago have now become
our masters!
Reply
Scott
May 6, 2014 at 9:31 am
I would gladly switch broadband providers except, as you point out,
there is no real alternative. At some point Google Fiber or Sonic’s
Fusion/Fiber might become available to me but the ROI is not there, yet.
:-(
Thanks for an enlightening post!
Reply
Scott
May 6, 2014 at 9:48 am
Has Level 3 reported the 5 US based ISPs to the FCC or the FTC
regarding consumer protection and quality of service? If this was going
on for over a year, then it would have been a case for the FCC to
investigate. Consumer protection is their job. Why are we just now
hearing about this? What political agenda/lobbying firm is Level 3
supporting regarding this?
Reply
Pingback: Internet providers continue to intentionally slow down
speeds for additional money | Android Authority
Pingback: Internet providers continue to intentionally slow down
speeds for additional money - AndroidBoard | Latest Android News and
Reviews | AndroidBoard | Latest Android News and Reviews
Pingback: droidultimate » Internet providers continue to
intentionally slow down speeds for additional money
Rex
May 6, 2014 at 11:07 am
Even as a fellow expat I’d contribute to a US-based crowd
funding campaign that was targeting two things… 1: a structured form
of corporate representation via a consumer advocate board member working
towards circumventing bad actors; and 2: transparency into both the
monopolies and government actions that sustains them.
Obviously there’s the LVLT stock, but purchasing it doesn’t
equate into the access and advocacy that is so needed for both privacy
and neutrality at the consumer level.
Thanks for writing Mark. LVLT might be sitting on a massive customer
base that would pay not just for goal of future service, but advocacy
from an entity large enough to grab government ears..
Reply
Pingback: Level 3 Accuses Six ISPs of Not Updating Networks to Ease
Congestion »
Pingback: Six Major ISPs Accused of Deliberate Traffic Throttling |
Digital Trends
Pingback: Level 3 charges 6 unnamed ISPs of not adhering to their
peering arrangements | Solus Network Solutions
Pingback: Level 3 Accuses Six ISPs of Not Updating Networks to Ease
Congestion | Telecom Judge
Pingback: The people who run your ISP are putzes | Beeblehead
Leave a Reply
Your email address will not be published. Required fields are marked *
Name *
Email *
Website
five + 1 =
Comment
You may use these HTML tags and attributes: title=""> datetime="">
Contact Us
ContactUs Button 5
Get our RSS Feed by Email
submit button
Search
Follow Level 3
View by Category
BCDR
Cloud Computing
Collaboration
Contact Centers
Content Delivery
Convergence
Culture
Customer Experience
Data Center Networking
Digital Entertainment
Español
Ethernet
Europe
Finance
Gaming
Global Connectivity
Government
Healthcare
High Performing Websites
Information Technology
Internet Broadcast
Level 3 Network
Low Latency
Network Operators
Português
Security
SIP Trunking
Social Media
Sports
The Looking Glass
Toll Free
Uncategorized
Uncategorized -at-es
Unified Communications
|
|
