Subject: Routing at the Edges of the Internet

From: Adam Novak <[email-masked]>
Date: Thu, 25 Aug 2011 20:57:32 -0500

I trust that some of you have seen this article from a while back:

<http://moblog.wiredwings.com/archives/20110315/How-We-Killed-The-Internet-And-Nobody-Noticed.html>

An informative except:

"When I open my laptop, I see over ten different wifi access points.
Say I wanted to send data to my friend in the flat next to mine. It is
idiotic that nowadays, I would use the bottleneck subscriber line to
my upstream ISP and my crippled upload speed and push it all the way
across their infrastructure to my neighbors ISP and back to the Wifi
router in reach of mine. The Internet is not meant to be used that
way. Instead, all these wifi networks should be configured to talk to
each other."

I also trust that you are aware of what happened to the Internet in
Egypt (and elsewhere) this spring, where Internet connectivity was
disrupted by shutting down major ISP networks.

I would like to bring the attention of the IETF to what I see as a
fundamental problem with the current architecture of the Internet:

The Internet is not a network.

As part of the development of the Internet, fault-tolerant routing
protocols have been developed that allow a connecdestined fortion to
be maintained, even if the link that was carrying goes down, by
routing packets around the problem. Similarly, packets can be
load-balanced over multiple links for increased bandwidth. However,
the benefits of these technologies are not available to end users. If
I have a smartphone with both a 3G and a Wi-Fi connection, downloads
cannot currently be load-balanced across them. The two interfaces are
on two different networks, which are almost certainly part of two
different autonomous systems. Packets must be addressed to one of the
two interfaces, not the device, and packets addressed to one interface
have no way to be routed to the other. Similar problems arise when a
laptop has both a wired and a wireless connection. Wired networks also
suffer from related difficulties: If I have Verizon and my friend has
Comcast, and we string an Ethernet cable between our houses, packets
for me will still all come down my connection, and packets for my
friend will still all come down theirs.

The Internet, as it currently appears to end-users, has a logical tree
topology: computers connect to your home router, which connects to
your ISP, which connects to the rest of the Internet. Cell phones
connect to the tower, which connects through a backhaul link to the
rest of the Internet. Almost all of the devices involved have multiple
physical interfaces and full IP routing implementations, but only the
default route is ever used. This results in a brittle Internet: the
failure of one ISP router can disconnect a large number of end-users
from the Internet, as well as interrupting communication between those
users, even when those users are, physically, only a few feet from
each other.

My question is this: what IETF work would be needed to add more
routing to the edges of the Internet? If each home or mobile device
was essentially it's own autonomous system, what would this do to
routing table size? To ASN space utilization? How can individuals
interconnect home networks when RIRs do not assign address and AS
number resources to individuals? How might individuals interconnect
home networks without manual routing configuration? Under what
circumstances could an ISP trust a client's claim to have a route to
another client or to another ISP? How might packets sent to a device's
address on one network be routed to that device's address on another
network, while packets to immediately adjacent addresses take the
normal path?

---

From: Eric Burger <[email-masked]>
Date: Fri, 26 Aug 2011 09:56:14 -0400

I disagree with the fundamental premise of this concept, that it is a PROBLEM that the Internet is not a network.  Um, last I looked, the Internet is an interconnection of networks.  Not a network in that sense.

Edge devices can today, in the scenario you portray, pick the "best" network to connect to.  Last thing we need is to ossify a method of doing that.  Let the edge be the edge and do what it wants.

On Aug 25, 2011, at 9:57 PM, Adam Novak wrote:

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: "Worley, Dale R (Dale)" <[email-masked]>
Date: Fri, 26 Aug 2011 11:04:26 -0400

> From: Adam Novak [[email-masked]]
> 
> "Say I wanted to send data to my friend in the flat next to mine. It is
> router in reach of mine."

This is a valid point, but it's also rather rare that one wants to
send large amounts of data directly to a friend in a neighboring flat
but one has not manually adjusting the local routing to take that into
account.

> If each home or mobile device was essentially [its] own autonomous
> system, what would this do to routing table size? To ASN space
> utilization?

There must be at least a few hundred million mobile phones with data
capability, and a similar number of homes and small businesses with
WiFi systems.  So we can estimate that a large fraction of a billion
entries would be added to the routing tables.  How would that work?

Dale

---

From: Scott Brim <[email-masked]>
Date: Fri, 26 Aug 2011 12:03:39 -0400

> entries would be added to the routing tables.  How would that work?

You do it in the endpoint.  The original poster should look at all the
work already being done in IETF WGs (and elsewhere), e.g. 6man and
mif, intarea and tsvwg.

---

From: David Morris <[email-masked]>
Date: Fri, 26 Aug 2011 10:20:09 -0700 (PDT)

> entries would be added to the routing tables.  How would that work?

I don't see this as a routing difficulty since the updated tables would be
highly local to the edge routers which would only need to know about
the more precise route between peers.

BUT I see enormous issues in terms of providing the capability in a secure
form that can be successfully enabled by the average end user. Also,
this is more than a routing issue since most file sharing involves
an itermediary with both edge devices connecting to a remote server. Not
only do the edge routers need to be configured for secure edge routing,
but the systems need to have applications which would deliver data
directly.

I think that folks with a requirement for local sharing will figure out
a local solution, often sharing an AP and uplink. If there is a business
case here, it wouldn't be hard for an enterprising AP vendor to offer
APs which create a shared network, perhaps even providing the 'server'
component. Could also be a device which has two radios and hence can
connect to two (or more) in range networks.

Dave Morris

---

From: Keith Moore <[email-masked]>
Date: Fri, 26 Aug 2011 14:08:31 -0400

On Aug 26, 2011, at 12:03 PM, Scott Brim wrote:

> mif, intarea and tsvwg

In other words, it's not a solved problem, and though there's wide recognition that the problem exists, nobody really has found a good solution.  

(Though mtcp strikes me as having broader applicability than most.)

Keith

---

From: Doug Barton <[email-masked]>
Date: Fri, 26 Aug 2011 13:49:48 -0700

> but the systems need to have applications which would deliver data

I have a related-but-different example of how end nodes being able to
know/discover direct paths to one another could be useful. Imagine a
busy server network with some web servers over here, some sql servers
over there, etc. All of these systems are on the same network, same
switch fabric, and have the same gateway address. In an ideal world I
would like them to be able to know that they can speak directly to one
another without having to go through the gateway (and without my having
to manually inject static routes on the hosts, which of course is both
painful and un-scale'y.

Doug

-- 

	Nothin' ever doesn't change, but nothin' changes much.
			-- OK Go

	Breadth of IT experience, and depth of knowledge in the DNS.
	Yours for the right price.  :)  http://SupersetSolutions.com/

---

From: Adam Novak <[email-masked]>
Date: Fri, 26 Aug 2011 15:57:13 -0500

> to manually inject static routes on the hosts, which of course is both

Shouldn't that be covered by the subnet mask? As long as they know
they're on the same subnet (and ARP broadcasts will reach everyone)
they should just ARP for each other and not involve the router at all.

If they are on different IP subnets, but the same Ethernet, then we
can either come up with a new way to do routing, or tell people not to
number things that way. Perhaps a subnet mask or CIDR prefix is not
expressive enough?

---

From: "Worley, Dale R (Dale)" <[email-masked]>
Date: Fri, 26 Aug 2011 17:05:19 -0400

> From: Doug Barton [[email-masked]]
> 
> All of these systems are on the same network, same

I'm no expert in this, but isn't this what ICMP Redirect messages are
for?  Aren't routers required to generate them in these cases?

Dale

---

From: Doug Barton <[email-masked]>
Date: Fri, 26 Aug 2011 14:08:36 -0700

On 08/26/2011 13:57, Adam Novak wrote:
> Shouldn't that be covered by the subnet mask?

Mostly, yes of course, but I'm dramatically simplifying my example for
dramatic effect. :)

> As long as they know
> If they are on different IP subnets, but the same Ethernet,

Yes, this is more often the case that I'm dealing with. (Working on
fixing a problem I inherited for a new client, so per your comment below
"don't number that way" may be the right answer.)

Doug

> then we

--

---

From: "Michel Py" <[email-masked]>
Date: Fri, 26 Aug 2011 14:47:05 -0700

> Worley, Dale R wrote:
> I'm no expert in this, but isn't this what ICMP Redirect messages
> are for?  Aren't routers required to generate them in these cases?

Unfortunately, ICMP redirects are often broken. It is a well-known issue
that the introduction of Windows XP SP2 (a while ago) and the Windows
Firewall did that.

The typical setup was a network with multiple subnets/VLANs and a
firewall/NAT/VPN box. The default gateway for the Internet and remote
VPN tunnels was the firewall, the default gateway for other VLANs was
the L3 switch that was doing the inter-VLAN routing. 

In theory, the host would send the traffic for a given destination, if
the traffic was an inside VLAN the firewall would send the redirect to
the host, forward the traffic to the L3 switch, and further traffic
would go directly to the L3 switch as the result of the ICMP redirect.
Before XP SP2 this was straightforward, a "route print" on the host
would indeed show the new route installed by the ICMP redirect.

In practice after XP SP2, the result was that the firewall indeed sent
the redirect to the host but since the host ignored it and kept sending
traffic to the wrong gateway, a large amount of firewall-to-L3switch was
present, effectively clogging the network at times.

Maintaining a correct routing table in hosts has always been the
Achilles' heel of networks with multiple gateways, which is why many
enterprise network operators tend to design a one-gateway solution.

Michel.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 27 Aug 2011 10:46:40 +1200

> work already being done in IETF WGs (and elsewhere), e.g. 6man and

Not to mention shim6, which is already a PS with running code.

Get used to the idea of end systems running with multiple simultaneous
addresses.

   Brian

---

From: Glen Zorn <[email-masked]>
Date: Sat, 27 Aug 2011 12:49:27 +0700

On 8/27/2011 4:08 AM, Doug Barton wrote:

...

> "don't number that way" may be the right answer.)

Old joke (but apropos):

Patient: Doctor, it hurts when I do this.
Doctor:  Don't do that.

...

---

From: Joel jaeggli <[email-masked]>
Date: Sun, 28 Aug 2011 12:33:42 -0700

> "don't number that way" may be the right answer.)

overlayed subnets are pretty straight-forward with ipv6 RA.

---

From: Joel jaeggli <[email-masked]>
Date: Sun, 28 Aug 2011 13:12:41 -0700

>> router in reach of mine."

there are other ways for devices with proximity to discover each other
and establish a relationship than via existing networks.

> entries would be added to the routing tables.  How would that work?

putting device mobility into the DFZ is just dumb. it was a fairly bad
idea when boeing did it and at any kind of scale it would be still more
obvious.

---

From: Hector Santos <[email-masked]>
Date: Fri, 26 Aug 2011 13:07:30 -0400

This sounds like yet another repeated cyclic centralization to/from 
distribution viewpoint.  The more things change, the more it remains 
the same. Inevitably someone will get the bright idea to be more, to 
consolidate more and once again offer central/services for its 
surroundings and then at one point, we want to go distributed again 
once Apple is able to stick an iPAD in your brain!  He also seem to 
mix in the commercial interest has being a conflict and incorrectly 
stated Fidonet prohibited it.  Well, it was because of the commercial 
conflict with the FTSC (Fidonet Technical Standard Committee) to take 
over the nodelist (akin to DNS) and begin charging membership fees 
that got it disbanded in a vote at the next to last FidoCon Meeting 
(1994) - the last one was empty so I heard.  Very few choice in free 
software in Fidonet and clearly the commercial better supported 
Fidonet mailers were dominant. I don't even recall the original 
inventor and fidonet mailer was free, no it wasn't he wrote it under 
contract to connect two company division computers with a simple 
XMODEM-based protocol 24x7 data exchange mailer.

-- 
Sincerely

Hector Santos
http://www.santronics.com

---

From: "Worley, Dale R (Dale)" <[email-masked]>
Date: Mon, 29 Aug 2011 21:51:15 -0400

> From: Michel Py [[email-masked]]
> 

Someone says, We should have a mechanism to do X!

I reply, We already have a mechanism that does X.

Someone says, Many deployed systems don't implement that mechanism
correctly.

It seems that the answer is to fix the deployed systems, rather than
designing a new mechanism.

Dale

---

From: "Michel Py" <[email-masked]>
Date: Mon, 29 Aug 2011 19:50:45 -0700

> Worley, Dale R wrote:
> Someone says, Many deployed systems don't
> implement that mechanism correctly.

That's not what I said; the mechanism is deployed correctly, the problem
is that there is another layer on top of it (in that case, the Windows
Firewall, but it's not the only culprit) that prevents the otherwise
working code from working correctly. I forgot to mention it, but the #1
remedy to the problem was to disable the SP2 Windows Firewall.

This problem has become highly visible because of the sheer number of
Windows XP hosts out there. Even as of today, XP still ranks #1 in the
deployed host platform. That being said, I have seen many firewall
policies on the firewall side that blocked it too; the problem is not
only a Microsoft one.

The mechanism (ICMP redirects) is technically fine and socially not.
People have become paranoid and now they firewall everything. It is a
behavioral animal. I'm not saying it's a good idea; the market answer to
crossing firewalls is to encapsulate everything into HTTPS, which is
probably worse. But then again, we have to deal with market pressure
against technically sound solutions, and the market almost always wins.

> It seems that the answer is to fix the deployed
> systems, rather than designing a new mechanism.

It is not the deployed systems we have to fix.
P.I.C.N.I.C.

Michel.

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 30 Aug 2011 13:10:05 +0200

> against technically sound solutions, and the market almost always wins.

That brings us back to the problem that "free routing" is apparently
insecure.  OTOH, there are large expectations from RIRs and network
providers, about security and policy routing, especially on port 25.
On closer inspection, though, those chaps don't seem to be eager to
play net-cops.

Should we go for secure routing, now that we have secure DNS?

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Tue, 30 Aug 2011 14:26:47 +0100

You could start by looking at MANET work, both in the WG of that
name and work outside the IETF under that name and as ad hoc
networks (the mobile in MANET can be misleading, D for dynamic
might be mor to the point) and mesh networks. There are real
networks (such as the Freifunk network in Germany) that do some
of what you are talking about, and use protocols based on IETF
work. (Freifunk uses OLSR - RFC 3626 - and intends, as I
understand, to use OLSRv2, once we manage to finish it.)

Note: I'm an author of OLSRv2, but have no connection to
Freifunk.

There are more issues, some of which you touch on, such as
with regard to addressing issues (the MANET WG is about
routing). The AUTOCONF WG was intended to address these but
that has not been a great success. Nor am I claiming MANET
has produced all the answers to all the routing-related
problems. Part of what's missing is rationale and explanatory
material explaining how you can do more than might be obvious
with what does exist. (There are RFCs 2501 and 5889, but
there is more material known to people working in these areas
than those capture.)

-- 
Christopher Dearlove
Technology Leader, Communications Group
Communications and Networks Capability
BAE Systems Advanced Technology Centre
West Hanningfield Road, Great Baddow, Chelmsford, CM2 8HN, UK
Tel: +44 1245 242194  Fax: +44 1245 242124

BAE Systems (Operations) Limited
Registered Office: Warwick House, PO Box 87,
Farnborough Aerospace Centre, Farnborough, Hants, GU14 6YU, UK
Registered in England & Wales No: 1996687

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Adam Novak
Sent: 26 August 2011 02:58
To: [email-masked]
Subject: Routing at the Edges of the Internet

                    *** WARNING ***

  This message has originated outside your organisation,
  either from an external partner or the Global Internet. 
      Keep this in mind if you answer this message.

<http://moblog.wiredwings.com/archives/20110315/How-We-Killed-The-Intern
et-And-Nobody-Noticed.html>

An informative except:

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************