Subject: Multi-homed BCP38

From: "John R Levine" <[email-masked]>
Date: 9 Jan 2014 10:09:15 -0500

I was at a meeting talking to ops people from some large ISPs, who tell me 
that when they tell their large customers about BCP 38, the customers say 
forget it, because they're multihomed.  I gather the situation is 
typically that the customer has multiple address ranges, say from 
providers A and B.  Normally traffic from range A goes out through 
provider A, and vice-versa, except sometimes when it doesn't.  Sometimes 
it's failover, or it may be deliberate asymmetic routing.  The customers 
may not be running BGP, or if they do, they don't want to announce range A 
to provider B for business reasons I don't entirely understand but that 
are not going away.

The ISPs tell me that the customers are often ISPs themselves, so there 
are lots of address ranges, far more than anyone could track manually even 
if they wanted to.

I see BCP 84, which is now ten years old.  The ISPs are aware of it, but 
it doesn't seem to have done the trick.  I can think of some hacks to 
pseudo-announce ranges for filtering purposes, but surely I am not the 
only person to have noticed this problem.  What have people done to 
address this issue?*  I figure the first thing to do is to understand 
what's failed before.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

* - other than calling the customers stupid, which they are not, and is 
not helpful

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 9 Jan 2014 16:43:49 +0100

Not being an IP-routing person, but chair of SSAC in ICANN, I hear people for the reasons you bring up are more in favor of a mechanism where one talk about explicitly "filtering at the edge of the Internet" and not at every point where routes are exchanged.

Because of this, please also include SAC004 in your investigation.

<https://www.icann.org/en/groups/ssac/documents/sac-004-en.htm>

   Patrik

On 9 jan 2014, at 16:09, John R Levine <[email-masked]> wrote:

> I was at a meeting talking to ops people from some large ISPs, who tell me that when they tell their large customers about BCP 38, the customers say forget it, because they're multihomed.  I gather the situation is typically that the customer has multiple address ranges, say from providers A and B.  Normally traffic from range A goes out through provider A, and vice-versa, except sometimes when it doesn't.  Sometimes it's failover, or it may be deliberate asymmetic routing.  The customers may not be running BGP, or if they do, they don't want to announce range A to provider B for business reasons I don't entirely understand but that are not going away.
> 
> The ISPs tell me that the customers are often ISPs themselves, so there are lots of address ranges, far more than anyone could track manually even if they wanted to.
> 
> I see BCP 84, which is now ten years old.  The ISPs are aware of it, but it doesn't seem to have done the trick.  I can think of some hacks to pseudo-announce ranges for filtering purposes, but surely I am not the only person to have noticed this problem.  What have people done to address this issue?*  I figure the first thing to do is to understand what's failed before.
> 
> Regards,
> * - other than calling the customers stupid, which they are not, and is not helpful
>

---

From: "John R Levine" <[email-masked]>
Date: 9 Jan 2014 11:09:55 -0500

> <https://www.icann.org/en/groups/ssac/documents/sac-004-en.htm>

I see it mentions the exact scenario I mentioned, but then says "This is 
only a slight complication and does not invalidate the approach."  Maybe 
it was slight in 2002, but it seems to be more than slight now.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 9 Jan 2014 16:19:07 -0000

John,

Coming at this from a different perspective and not sure it is relevant to the question you asked, but this seems to be one of the problems that Homenet is trying to solve and is giving rise to "new" routing concepts within the home network in order to hit the right egress homing point.

Adrian

---

From: Tim Chown <[email-masked]>
Date: Thu, 9 Jan 2014 16:23:36 +0000

> Coming at this from a different perspective and not sure it is relevant to the question you asked, but this seems to be one of the problems that Homenet is trying to solve and is giving rise to "ne…

Indeed, though that may require extensions to support routing based on both src & dst addresses.  There have been presentations about that at the last two or three homenet WG sessions.

Tim

---

From: Scott Brim <[email-masked]>
Date: Thu, 9 Jan 2014 11:52:44 -0500

> Not being an IP-routing person, but chair of SSAC in ICANN, I hear people for the reasons you bring up are more in favor of a mechanism where one talk about explicitly "filtering at the edge of the…

Where is "the edge"? Is there a boundary outside of your control that
you truly trust? Even if there's supposedly unbreakable filtering at
most of the edge, wouldn't you want to do your own filtering as well?

It looks like the sites John talked to are multihomed, NOT doing
routing at their edge (??), and yet wanting to filter? I don't think
you get all three of those.

Scott

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 9 Jan 2014 13:33:46 -0500

> Where is "the edge"? Is there a boundary outside of your control that

ideally a customer filters at every LAN port ... that way it's pretty
simple to tell what you should be filtering :)

after that it'd be nice if their egress port to 'internet' included:
"permit ip MYSRC any"
so that in case of internal issues with config consistency/etc they
still don't leak out bad sources.

At the 'ISP' level, this sort of thing is practical really only at the
customer link, it's not super practical to filter on links between
ISPs (maybe it is for TierX but not Tier X-M)...

> most of the edge, wouldn't you want to do your own filtering as well?

sure, can you define the filter between (for pathological cases)
Level3 and ATT ? or NTT and Cogent?

it gets much harder to do this in a not-super-brittle way pretty
quickly at that level of the network hierarchy :(

> routing at their edge (??), and yet wanting to filter? I don't think

that's not clear, from john's description at least.

> you get all three of those.

For some of the cases John is talking about it's probably 'hard' to do
the filter, but for many folk as long as you are already filtering the
routes the 'customer' sends (and you ARE doing that, right? you dont'
want to be PCCW do you? :) ) it's pretty darned simple to just filter
inbound properly there. (or for the customer to properly egress filter
to you, since they know the routes because they told you the routes)

-chris

---

From: "John Levine" <[email-masked]>
Date: 9 Jan 2014 18:45:50 -0000

>that's not clear, from john's description at least.

I can ask (the meeting's not over yet) but I gather that these are
multihomed customers who don't do BGP.  The ISP knows what ranges it's
routing to the customer, but it doesn't know what ranges are routed to
the same customer by someone else.

R's,
John

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 9 Jan 2014 13:57:17 -0500

> the same customer by someone else.

if they don't do bgp .. .then you'd expect only static routing and
thus you know exactly what they ought to be sourcing.

There are pathological (aka: stupid) cases of multihomed to 2 isps,
getting static-routing of each isp's PA space on only that one ISP...
and default routing up both pipes... but really that's just broken and
dumb, so getting customers to fix this by filtering is a fine answer.

-chris

---

From: "John R Levine" <[email-masked]>
Date: 9 Jan 2014 14:00:22 -0500

> dumb, so getting customers to fix this by filtering is a fine answer.

As I said in the original message, calling the customers stupid is not 
useful.

They know they're doing asymmetrical routing, and they're paying the ISP a 
lot of money to do so.

Regards,

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 9 Jan 2014 14:04:00 -0500

> As I said in the original message, calling the customers stupid is not

right, my assumption is that some marketting/pr/customer-service
person would actually tell the customers a form of: "Hey, do you like
reliability? we do to... we also like cleaner internets, we're going
to filter outbound traffic from you for the sources we should  see
from you, currently that's: A/24 and B/24... if we should add more,
speak up by <60days from now>"

> They know they're doing asymmetrical routing, and they're paying the ISP a

sure, they are also paying for reliability and lack of problems with
the traffic they send, so making that part the star, not their
stupidty is probably the way forward.

---

From: "John Levine" <[email-masked]>
Date: 9 Jan 2014 19:17:47 -0000

>speak up by <60days from now>"

The ISP says the customer responds "if you're not willing to handle
our 20 gigabit account, we're sure we can find someone else who can."

Also, if you'll review the messages to which you're responding, you'll
note that the customer is often an ISP with customers of its own so
there are far too many ranges to handle manually.

R's,
John

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 9 Jan 2014 14:21:01 -0500

> there are far too many ranges to handle manually.

yup, as I said, for the case of ISP to ISP (at a point) it's pretty
hard to keep the filtering proper/reliable, so I'd pass on most of
that.

for static customers though, it's pretty simple and for them and their
isp to sort this out.

-chris

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 10 Jan 2014 08:30:55 +1300

> the same customer by someone else.

This is not a new problem; we certainly discussed it in the multi6 WG
ten years ago:
http://tools.ietf.org/html/draft-huitema-multi6-ingress-filtering-00

Actually it's one of the reasons that multi6 led to shim6, which mainly
avoids this problem (but is disliked by ISPs for other reasons).

    Brian

---

From: joel jaeggli <[email-masked]>
Date: Thu, 09 Jan 2014 11:35:22 -0800

On 1/9/14, 7:09 AM, John R Levine wrote:
> I was at a meeting talking to ops people from some large ISPs, who tell
> me that when they tell their large customers about BCP 38, the customers
> say forget it, because they're multihomed.  I gather the situation is
> may not be running BGP, or if they do, they don't want to announce range
> A to provider B for business reasons I don't entirely understand but
> that are not going away.
> 
> are lots of address ranges, far more than anyone could track manually
> even if they wanted to.
> 

Peers the do URPF strict, and also having regionally distinct prefix
range advertisements (e.g. we peer in multiple regions) for peers  are a
problem e.g. due to internal issues traffic that was flowing through a
port where the prefix was announced is now flowing towards me through a
port which it is not, likewise in the other direction, in a case where
the out of region prefix would due to local pref be reached via a
transit provider, if the transit route goes away  may well blackhole at
the other peer if it instead crosses your backbone to another exit. We
use loose to prevent this in the inboud direction with those peers  we
also announce covering agregates to particular URPF strict peer.

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 9 Jan 2014 19:55:36 +0000

> I see BCP 84, which is now ten years old.  The ISPs are aware of it, but it doesn't seem to have done the trick.  I can think of some hacks to pseudo-announce ranges for filtering purposes, but sur…

There are some drafts you may find interesting:

http://tools.ietf.org/html/draft-boutier-homenet-source-specific-routing
http://tools.ietf.org/html/draft-troan-homenet-sadr
http://tools.ietf.org/html/draft-baker-ipv6-isis-dst-src-routing
http://tools.ietf.org/html/draft-baker-ipv6-ospf-dst-src-routing
http://tools.ietf.org/html/draft-baker-rtgwg-src-dst-routing-use-cases
http://tools.ietf.org/html/draft-xu-homenet-traffic-class
http://tools.ietf.org/html/draft-xu-homenet-twod-ip-routing

Homenet has been looking at this from the perspective of egress routing in residential multihoming. I have been looking at it more from source/destination routing in OSPF/IS-IS, for which "egress routing" and "residential multihoming" are special cases.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 9 Jan 2014 21:04:23 +0100

> On 9 jan 2014, at 17:52, Scott Brim <[email-masked]> wrote:
> 

Oh, boy, why do I try to explain what I have in my head... ;-)

I agree that is a problem, but those people being multi homed should filter, and, that should be included in contracts, enforced etc. However you as upstream detect filtering is not happening.

I thought the main difference had to be with the difference between edge of your AS and nodes that really are only single homed, where the latter should have filter on their upstream.

Anyway, I just wanted to point out the SSAC document, and let others interpret.

   Patrik

---

From: Mark Andrews <[email-masked]>
Date: Fri, 10 Jan 2014 09:12:43 +1100

SIDR should in the end make this easier to automate.

Once your customers have certs that cover the range of addresses
they are allocated they can announce those address ranges as valid
sources to the other ISP securely similarly to the way they can
anounce they want those address ranges routed to them.

BCP 38 can be securely automated if we are willing to spend engineering
effort to achieve it.

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Måns Nilsson <[email-masked]>
Date: Fri, 10 Jan 2014 09:33:06 +0100

> there are far too many ranges to handle manually.

An "ISP" who does not understand and use BGP is probably not serious. Were
I in the business of selling Internet connectivity (and I'm not, I'm a
multihomed customer who does speak BGP with my upstreams) I'd look at such
potential customers with a mixture of suspicion and badly hidden contempt.

Which is not constructive,  quarter-of-year-cash-flow-wise, but it is
indeed better for the Internet. IMNSHO. And, in the long run, there is
a definitive cost associated with running crap setups like multihoming
without a proper dynamic routing protocol.

-- 
Måns Nilsson     primary/secondary/besserwisser/machina
MN-1334-RIPE                             +46 705 989668
I can't decide which WRONG TURN to make first!!  I wonder if BOB
GUCCIONE has these problems!

---

From: "John R Levine" <[email-masked]>
Date: 10 Jan 2014 09:12:11 -0500

> An "ISP" who does not understand and use BGP is probably not serious.

It must be nice to live in a world where you can turn away customers who 
pay a lot of money, just because they're not serious enough.

Regards,

---

From: [email-masked]
Date: Sat, 11 Jan 2014 07:49:28 +0100

Subject: Re: Multi-homed BCP38 Date: Fri, Jan 10, 2014 at 09:12:11AM -0500 Quoting John R Levine ([email-masked]):
> It must be nice to live in a world where you can turn away customers
> who pay a lot of money, just because they're not serious enough.

The ISP business is not really a high profit margin one. Some customers
are a loss. The tendency of wanting to multihome without running a routing
protocol is a good indicator of lower profit margin. If for nothing else
then for the increased support load.

There used to be threshold problems with getting AS numbers and such, but the 
increased AS number space helps a lot.  The arguments against DTRT are weaker. 

-- 
Måns Nilsson     primary/secondary/besserwisser/machina
MN-1334-RIPE                             +46 705 989668
GOOD-NIGHT, everybody ... Now I have to go administer FIRST-AID to my
pet LEISURE SUIT!!

---

From: "John R Levine" <[email-masked]>
Date: 11 Jan 2014 02:09:59 -0500

> then for the increased support load.

It must be nice to live in a world where you know the profit margins of 
American ISPs better than they do.

By the way, not all of the multi-homed customers we're talking about are 
small ISPs, although some are.

Regards,