Subject: RE: Operational Renumbering Procedure

From: Fred Baker <[email-masked]>
Date: Sun, 27 Apr 2003 20:48:41 -0700

At 05:20 PM 4/27/2003 -0700, Christian Huitema wrote:
>Pekka points out one issue with the "stable with two addresses" phase. At 
>that point, the network is in effect  multihomed, and the two-address 
>phase is only stable if we know how to handle well multi-homed sites.

I'm not quite sure I agree, given a network administration with a clue. But 
maybe this document is for those who lack a clue...

It has two prefixes, yes, but his concern is still with a single ISP and 
presumably a single router in front of the (presumably enterprise) network. 
Hence, it is single-homed using multiple prefixes. His presupposition is 
that folks would be attempting to use the new prefix to send traffic into 
the network. If they do, I'll argue that something isn't as specified; that 
shouldn't happen until the next step, when the new prefix becomes the 
preferred prefix in RFC 2462-speak (and therefore might be used as a source 
address), and DNS is advised of the new set of addresses (and therefore the 
new prefix might be used as a destination address to a server).

But apart from that, he is assuming that the ISP is filtering traffic 
coming from the network using the second prefix because it fails an RPF 
check - there is no route to that address, or the route faces into the ISP. 
Alternativeively, his words referred to RFC 2827, which suggests that 
traffic arriving from the enterprise network should be compared to a source 
address access list which permits traffic from the enterprises assigned 
prefix(es) and denies all other.

I would argue that an enterprise legitimately using any prefix with any 
ISP, whether it is one or many, is responsible to tell the ISP it is doing 
so. The ISP might not be contracted to accept the route or to route to that 
prefix, but it should not be filtering out traffic using it.

>  You don't discuss this in your draft

So I think you're suggesting that section 3 become section 3.1, enveloped 
in a section 3 entitled "how not to shoot one-self in the foot". Section 
3.2, entitled "managing routing and ingress filtering issues" says "if you 
expect routing and filtering to work through your ISP, you should tell him 
what prefixes you are using". Close? 

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Pekka Savola <[email-masked]>
Date: Mon, 28 Apr 2003 12:00:51 +0300 (EEST)

On Sun, 27 Apr 2003, Fred Baker wrote:
> Hence, it is single-homed using multiple prefixes.

You missed the point.  The example was simplified on purpose.

But this shows that different people seem to have different expectations 
wrt. renumbering -- so this discussion is very worthwhile.

My view is that people will not renumber to any interesting degree unless
they're also changing ISP's (stuff like migrating from 6BONE address space
to one obtained from RIR's is semi-exception, but it is not interesting
yet either).  It just isn't fun enough to warrant it.

You may be assuming that "within an ISP" renumbering is interesting (and
focus of the memo), and the document seems to be implicitly assuming that.  
That's ok -- if that's what you want to describe -- but the applicability
must be spelled out. Because there is a subset of problems which may be
very difficult to solve if you can't assume that.

> His presupposition is that folks would be attempting to use the new
> prefix to send traffic into the network.  If they do, I'll argue that
> something isn't as specified; that shouldn't happen until the next step,
> when the new prefix becomes the preferred prefix in RFC 2462-speak (and
> therefore might be used as a source address), 

This is a bit different ordering of events as one would suppose.  To make
that work, you'd have to advertise prefixes with preferred_lifetime=0 and
have the hosts configure them as deprecated.  I'm not sure if that'd work,
and whether there might be similar other issues.  You also can't 
configure any manual addresses as you typically can't set them deprecated.

(The ordering I've always had in mind to get around certain issues is to
try to instantaneously both deprecate the old addresses and introduce the
new ones -- to avoid dual-use.)

> and DNS is advised of the
> new set of addresses (and therefore the new prefix might be used as a
> destination address to a server).

You have an assumption that having stable routing with two prefixes is 
possible.  I argue that it's at least difficult if not impossible.
 
> prefix, 

...

> but it should not be filtering out traffic using it.

Many ISP's will filter them, you can be sure of that.  That's the right
thing to do, operationally and security-wise.  And it's the simplest thing
to do too: just a plain uRPF check without exceptions.  We certainly do
so.

You may also consider a scenario with:

 customer--.
    |@      \@
   ISP1-*---ISP2
   $|*       |
   ISPA     ISPB

Regular, "first-hop" ingress filtering is done at "@"'s.

Now, ISPA might be performing address filtering at "$".  That'd mean that 
ISP1 would have to tell about the customer's special case upstream too 
(possibly multiple hops).

ISP1 might also be filtering out all the source addresses belonging to its 
space at "*"'s -- from other ISP's and from transits.  We certainly do 
that.  This would have to special-cased here too.

Ingress filtering is just *not* a tool you may believe it is, a binary
on/off toggle at the first-hop ISP -- the Internet is not a homogeneous,
single-administration, single-security-domain place.

> what prefixes you are using". Close? 

Pretty close, but the thing is far from so trivial: what if the ISP says
"you have our prefix, we can't route traffic which doesn't have it as the
source address, please don't send other traffic here"?

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Fred Baker <[email-masked]>
Date: Mon, 28 Apr 2003 06:42:17 -0700

>new ones -- to avoid dual-use.)

So you really don't mind that the DNS (and DHCP?) records are still 
outstanding, and don't mind imposing an outage related to that?

>what if the ISP says "you have our prefix, we can't route traffic which 
>doesn't have it as the

You are saying that you would rather drive your customer away than add a 
second line to the access list?

The ISP is cutting its own nose off. You suggest that the enterprise is in 
the process of switching ISPs. In the case you present, though, I should 
think that the enterprise is adding an ISP (becoming multihomed); if he 
plans to drop the first later, he may, but he's not doing it now. Making 
life difficult for its customer would, if I were the enterprise's CIO, be a 
reason for converting that from "adding an ISP" to "switching ISPs".

So, OK, this document is also for ISPs that have no concept of customer 
service. I'll point out that intransigence on the part of the ISP is not a 
wise marketing move.

---

From: Pekka Savola <[email-masked]>
Date: Mon, 28 Apr 2003 17:25:23 +0300 (EEST)

> outstanding, and don't mind imposing an outage related to that?

I haven't followed that thought in enough detail to know whether this is a
problem or not.

In particular, it seems to depend on two issues:
 - the procedures of removing the old DNS etc. data, "outstanding 
information"
 - how a host would react to a TCP SYN to a deprecated address.  TCP RST?  
If application is done properly, it will try the other record then, with a 
delay of one RTT.

> second line to the access list?

To respond with an analogue..

Are you saying you would rather drive your customer 
away than punching a hole in some other ISP's aggregate by announcing a 
few (IPv4) /24's to the DFZ?

Yes, someone has to say NO.

Moreover: it's just not the ISP who is the "bad guy".  There is zero
guarantee that even if ISP made manual fixes in all its access lists, the
site's use of the wrong source address would still work.  Someone else
(ISP's neighbors, transits, the owner of the superblock ie. the site's
other ISP, etc.) would probably have similar filters.

---

From: Fred Baker <[email-masked]>
Date: Mon, 28 Apr 2003 08:21:51 -0700

>few (IPv4) /24's to the DFZ?

There is routing and there is ingress filtering. If you don't choose to 
route the traffic *to* the other ISP's prefix, that's fine with me. The 
other ISP can do that just fine.

We're discussing ingress filtering, and you're telling me that you will 
arbitrarily tell your customer that he may either use your service or be 
multihomed, but not both. You can say that if you want, but if the CIO of 
your customer's network has decided to be multihomed, you are deciding you 
don't need his business.

For the price of adding a line to an access list on the ingress filter, you 
can retain his business and continue receiving money from a multihomed 
customer. Is this not your business objective?

---

From: Pekka Savola <[email-masked]>
Date: Mon, 28 Apr 2003 19:58:42 +0300 (EEST)

On Mon, 28 Apr 2003, Fred Baker wrote:
> multihomed, but not both. 

No, that's not the point.  The point is saying "use a mechanism that 
works".  Sending packets with wrong source address is not that.

> You can say that if you want, but if the CIO of 

Thus, we need better multihoming mechanisms, which work without damaging 
the networks.
 
> customer. Is this not your business objective? 

It's not as simple as that.  The changes need to be made pretty much
everywhere, but even that could be manageable.  What do you do when your
multihomed customer complains that it can't reach a lot of networks when
the traffic goes by you? (as other folks are doing ingress filtering from
your direction)

Of course, you can sell a service with no guarantees and strong
disclaimers "this is probably not going to work", but IMO it's better to
say "this really won't work properly, you'd be better off using a working
multihoming technique".

---

From: Michael Thomas <[email-masked]>
Date: Mon, 28 Apr 2003 15:00:23 -0700 (PDT)

> prefix, but it should not be filtering out traffic using it.

Fred, I'm completely lost. An ISP certainly has
the right to do RPF checking, and is encouraged to
do so with BCP 38.  If I put a source address of
ISP A into a packet and it's routed to ISP B's
link for whatever reason, it will be filtered by
ISP B by default (or at least should). So it seems
that Christian has a valid point. And
operationally I wonder what "telling the ISP"
might be. They allow me to inject a route?  Does
that apply to my DSL service provider when I use
my cable service provider's prefix? Is there
technology, security and business reasons for an
ISP to do this?

		Mike

---

From: Fred Baker <[email-masked]>
Date: Mon, 28 Apr 2003 15:01:47 -0700

At 12:00 PM 4/28/2003 +0300, Pekka Savola wrote:
>and whether there might be similar other issues.  You also 
>can't  configure any manual addresses as you typically can't set them 
>deprecated.

It seems to me, on further consideration, that you may have identified an 
issue in RFC 2462. If the network administrator cannot deploy a new prefix 
without it automatically coming into use in some set of cases, he cannot 
renumber without his network experiencing some level of outage while he 
does so or until routing has converged.

Who is the proper target of a complaint with 2462?

---

From: Fred Baker <[email-masked]>
Date: Mon, 28 Apr 2003 20:00:01 -0700

At 03:00 PM 4/28/2003 -0700, Michael Thomas wrote:
>the right to do RPF checking,

My point is that the customer should be able to tell his ISP what addresses 
he is legitimately using, and therefore bias the ingress filter, regardless 
of whether he is singly homed or multi-homed. If I have address X, Y, and 
Z, and X is provided by ISP X', I should be able to send traffic from 
addresses x', y', and z' to any of my ISPs and have them forward them.

Asking otherwise requires source-destination addressing (you have a 
separate route table, or at least egress router, for each source address 
prefix), which is not a feature of IP routing, whether v4 or v6. We can 
discuss providing same, if you like, or we can discuss customers refusing 
to buy services from ISPs that require of them things that are not provided 
in the IP architecture as presently defined. Take your pick.

I think the net result of ISP intransigence is ISP bankruptcy. But that's 
just my opinion.

---

From: Pekka Savola <[email-masked]>
Date: Tue, 29 Apr 2003 08:39:20 +0300 (EEST)

> Who is the proper target of a complaint with 2462? 

RFC2461/2462 will be revised at some point (it's on charter), so the
complain should be filed with the chairs and the authors (I believe), or
maybe a specific I-D describing the particular problem (and that only).  
YMMV.

---

From: Michael Thomas <[email-masked]>
Date: Tue, 29 Apr 2003 12:23:46 -0700 (PDT)

> addresses x', y', and z' to any of my ISPs and have them forward
them.

A few questions:

1) How do I assert the prefix in a way that my ISP
   would trust me? There are obvious security issues here.
2) At what scale would you expect this to work? If not my home
   between my cable and DSL provider, then how big? And is it
   acceptible to write those small fries off?

 > I think the net result of ISP intransigence is ISP bankruptcy. But that's 

Well I'd say that something runs counter to BCP 38
is going to run into a lot of operational
resistance, to say the least. Depending on the
answers to the above, I'm not sure that this is
solely a matter of ISP intransigence.

	 Mike

---

From: Fred Baker <[email-masked]>
Date: Tue, 29 Apr 2003 17:47:40 -0400

>    would trust me? There are obvious security issues here.

I would expect that this would either be by contract or by BGP, using the 
ISP/customer security environment (hopefully BGP/TCP+MD5 or IPSEC) that is 
in use today.

>    acceptible to write those small fries off?

I don't see any reason that it would not work at home between cable and DSL 
providers, given that they were willing to play ball. Christian's point was 
that they are the folks with clout, not the SOHO network, so they are not 
motivated to do so. And I don't see any obvious reason that it wouldn't 
work on a continental scale, given an operator that was silly enough to try 
it on that scale. I doubt you'll find many, though. The would find a way to 
subdivide the problem and do it in pieces, for their own sake.

>Well I'd say that something runs counter to BCP 38 is going to run into a 
>lot of operational resistance, to say the least. Depending on the answers 
>to the above, I'm not sure that this is solely a matter of ISP intransigence.

We seem to differ on what it means to violate BCP 38.

BCP 38 says "service providers should police their customers by verifying 
that the customers are sending from legitimate addresses" and "an example 
of such a procedure would use an access list on the customer-facing 
interface that discards traffic not sourced from the customer's prefix. It 
doesn't go into the complexities of multihomed networks, or networks that 
have over time accreted multiple prefixes - Cisco being an example of both. 
It simply says "don't forward traffic that comes from addresses the 
customer is not legitimately using."

Now, let's take Cisco as a case. We are in fact multihomed, through eight 
or ten service providers and four DMZs. We have accreted a number of 
prefixes, in the 64.0.0.0/8, 128.0.0.0/8, 169.something, 170.something, 
171.something, and 172.something, and by the way the 192.something spaces. 
And yes, we use NATs internally, such as in front of labs. If you're going 
to tell me that having multiple prefixes and multiple providers 
automatically breaks BCP 38, then this bumblebee doesn't fly, and you have 
some explaining to do as it quite obviously is in fact flying.

What is required is that the enterprise keep all of its providers aware of 
all of the prefixes that it is legitimately using, and have the providers 
accept traffic from those prefixes. The provider can do it using RPF checks 
by putting static routes into the access router or by adding them to the 
ingress ACL, its choice. Whether or not it actually uses the routes itself 
or advertises them to other networks is its choice, and contracts come into 
play in making that judgement.

---

From: Michael Thomas <[email-masked]>
Date: Tue, 29 Apr 2003 17:34:46 -0700 (PDT)

> ISP/customer security environment (hopefully BGP/TCP+MD5 or IPSEC) that is 

So, it seems to me that you're pretty much
limiting yourself to either people you accept BGP
advertisements from, or to the size of a crowd you
could support using phones, faxes and web sites
with the (??) human resources needed for
verification about the prefix in question... which
sounds like a pretty tier 1 kind of customer to
me.

 > subdivide the problem and do it in pieces, for their own sake.

Well, I'm fairly certain that my DSL provider's
never going to be convinced to accept BGP
announcements from me, and I sort of doubt that
they're going to come up with the resources
necessary to sort out who legitimately should be
allowed to modify their RPF checks and/or routes,
so I really can't see how this is applicable to
small fries like me. Or for that matter, just
about anybody off the Fortune 500 list.

Unless you've got something else in mind, because
I just don't get it, and it's not obvious in your
draft...

 > customer is not legitimately using."

Sure, and RPF works great for that until you start
throwing multihoming into the equation which is
really what this renumbering proposal is in
disguise. And it seems to me that the basic
problem is the security problem of how do you deal
with customers claiming that they are multihomed
off of another provider's prefix -- a customer
that you probably trust between very little or not
at all. For starters, there isn't a way to, say,
cryptographically assert ownership of a prefix.
And it's not clear that even if you could, you'd
want that. So I have real questions about how you
could make that scale up to millions of sites,
which is what you'd need in the residential case,
and even if you s-can residential, I would assume
you'd want this to at least scale for small and
medium sized business and organizations, right?

 > some explaining to do as it quite obviously is in fact flying.

No, I'm saying that Cisco is an outlier mainly
because we are a 900lb primate. This seems like a
fairly plausible case where high touch or high
trust is feasible. It's JoeClueless.com and
3liteHax0r.net that are much more problematic.

		    Mike

---

From: Fred Baker <[email-masked]>
Date: Wed, 30 Apr 2003 00:07:15 -0400

I'm not sure where you're going with your comments. It sounds for all the 
world like you're saying that because you can think of a case where two 
ISPs might choose to not cooperate with what they consider an insignificant 
customer, or because there exist ISPs that will remove their nose to spite 
their face, the question of how a network that chooses to renumber itself 
might do so operationally is illegitimate to ask.

If that's your perspective, tell me now. I'll know to ignore any commentary 
from you as pointless and unproductive. I don't have that low an opinion of 
ISPs, or of enterprise network managers. I will admit readily to the 
existence of homo ignoramus; I will not presume that every person I meet 
belongs to that species.

---

From: Michael Thomas <[email-masked]>
Date: Wed, 30 Apr 2003 07:29:34 -0700 (PDT)

No. I'm saying that it's not at all obvious how
the renumbering scales to anything much smaller
than Cisco sized companies, if you have the
barrier of RPF checking at your provider. If you
think that it can scale larger, and indeed to even
residential multihoming/renumbering, what,
*specifically*, would I have to do with my two
providers, or between themselves that they could
hope to deploy operationally without lessening the
security that RFP, etc, give?

	  Mike, who suspects this isn't
           news to the multi-6 crowd.

---

From: Pekka Savola <[email-masked]>
Date: Wed, 30 Apr 2003 17:43:24 +0300 (EEST)

Well..

For what it's worth, IMHO, Mike shows a surprising amount (to this wg :-)  
of operational/security clue.

Source address checking is not a thing to be dusted under the carpet, and
those who refuse to put the dust under the carpet are *not* the bad guys.

Instead, we should try to figure out the extent of such an approach, and
more importantly, work on mechanisms that do not require such violations.