Subject: [dhcwg] Background on CMTSs and routing

From: "Woundy, Richard" <[email-masked]>
Date: Wed, 14 Feb 2007 23:44:20 -0500

Folks,

I think it might be worthwhile for the WG to have a common understanding
about CMTSs and routing.

>In the various discussions we've had about this, it's been stipulated
that one requirement is that the CMTS not be required to participate in
any dynamic routing protocol at all.

>So the very notion should be impossible.  We shouldn't even be able to
consider a routing protocol solution to a problem that, I thought I
understood, must span networks that do not implement any routing
protocols.

While the DOCSIS specification allows CMTS implementations to be
"routers" or "bridges", in practice most CMTSs are implemented as
routers, and support multiple routing protocols. Some CMTSs can support
all of the following protocols: RIP, IS-IS, OSPF, BGP, IGRP, PIM, etc.

A typical CMTS today would be configured to use a "good IGP" (OSPF,
IS-IS, iBGP) for routing across the regional backbone (over GigE or OC-x
interfaces), and to use a "poor man's IGP" (RIP routing, or no dynamic
routing at all) with a few customer cable modems across the DOCSIS/HFC
network (to support customer-specific IP subnets). The CMTS would
redistribute (inject) the RIP-learned routes into OSPF/IS-IS/iBGP, as
permitted by local routing policy/configuration. The regional backbone
would leverage OSPF/IS-IS/iBGP routing, and leverage eBGP
peering/transit with one or more national backbones.

I understand that in some cable deployments CMTSs are configured to
participate in the regional backbone iBGP mesh, and in some other
deployments they are not. Since this isn't a routing or cable operator
forum, I ask that we not debate the technical merits of either approach
here. Hopefully the rest of this email will show that this choice is
irrelevant to a "RAAN" comparison discussion.

Note: some CMTS implementations go as far as to support BGP/MPLS VPNs
(RFC 2547), which implies support for Multiprotocol Extensions for BGP4
(RFC 2283).

The CMTS is usually the DHCP relay agent on behalf of the DOCSIS
network(s) it supports.

>The implication being that the CMTS would enter only a local
"pseudo-static" route in its table, thanks to this option.  The "real"
router it was connected to (possibly via other similarly
non-dynamically-routing devices) would advertise the prefix's route
dynamically (by also participating as a DHCPv6 relay, and being
delivered similar options).

>Basically, to use DHCPv6 relay chaining to reach a router that does
participate in dynamic routing, which either advertises the delegated
prefix or relies upon external routing updates to direct it to send
routes to the nearest non-dynamic router.

The CMTS would be able to use the RAAN option to add a (pseudo-)static
local route, and to redistribute this route into the regional backbone
via OSPF/IS-IS/iBGP routing as appropriate. So instead of learning about
a customer-specific route via RIP from a potentially untrusted source
(i.e. a cable modem located in the customer premise), the CMTS would
learn about a customer-specific route via the DHCP RAAN option from a
trusted source (i.e. the service provider's own DHCP server).

A very similar approach is documented for multiple broadband
technologies in RFC 4779, e.g. section 5.2.2.5.4 "If DHCP-PD is used for
address assignment, a static route is automatically installed on the
CMTS/ER for each delegated /48 prefix. The static routes need to be
redistributed into the IGP at the CMTS/ER so there is no need for a
routing protocol between the CMTS/ER and the GWR. If the CMTS is also
acting as an ER, it runs an IGP such as OSPFv3 or IS-IS."

(Note: RFC 4779 requires a lot of updates to synchronize with our
projected cable IPv6 deployments, but this isn't the forum for that
discussion.)

Note that in the RAAN approach, when the CMTS discovers (using local
state) that the CM is unreachable (e.g. DOCSIS network issue) or has
gone offline, the CMTS can remove the prefix from the local forwarding
table, and pull the prefix from IGP routing.

To be fair, Fred Templin has some other use case in mind, in which the
relay agent is NOT a router,
<http://www1.ietf.org/mail-archive/web/dhcwg/current/msg06913.html>.
Maybe Fred's environment represents a relatively small cable operator --
or perhaps a non-cable operator environment within Boeing? I am pretty
sure this environment does NOT reflect a CMTS deployment in my company,
or in any large cable operator that I can think of.

>This implication is reinforced by reports from various people in the WG
and without, that CMTS operating software does not implement dynamic
routing protocols, is not provisioned with sufficient processor power or
memory (even to manage a "small" local network), or...since CMTS people
are "RF people" and not "IP people"...simply aren't well suited in their
implementation of the protocols.

CMTS operating software typically implements dynamic routing protocols.

CMTSs have sufficient memory for reasonably-sized routing/forwarding
tables (e.g. hundreds and/or thousands of entries), but probably not
enough memory for a full default-free routing/forwarding table (e.g.
hundreds of thousands of entries).

Nowadays, "CMTS people" (engineers and operators) tend to be both "RF
people" and "IP people", pretty smart in general, evem if they don't
attend every IETF meeting. Maybe it isn't a good idea to form opinions
about "CMTS people" based on "reports from various people". I won't
comment on this any further.

>So I'm pretty confused by looking at a routing protocol centric view of
solutions to this problem (which I thought was an impossible candidate),
and placing upon it criticisms for limitations that are both the same
and measurably worse in this DHCPv6-option centric view I've been
delivered previously.

>So I don't get it.  Either I'm dense (probable), the problem hasn't
been explained sufficiently (possible), or there are so many problems
and solutions that no one can keep them straight.

Let me assume that we are on the same page now, technically.

Stig raised the possibility that perhaps the DHCP server could use
dynamic routing protocols (particularly OSPF) to inject routes for
delegated prefixes. I took this idea seriously, thought about the
implications, and raised some concerns about whether "stale routes"
could/would be injected by the DHCP server. While I used an OSPF
example, the same concerns apply to iBGP route injection -- the DHCP
server would not know about the local state of the CMTS network element,
i.e. we cannot guarantee "fate-sharing" between the DHCP server and the
CMTS. In other words, if a DOCSIS network interface on a CMTS goes down,
how would the DHCP server know about it?

The other relevant consideration is that DHCP server experts tend not to
be IGP/routing experts. DHCP isn't trivial, neither is OSPF or BGP. That
is an issue for both software development as well as day-to-day
operations.

-- Rich

P.S. Dave, thanks for the mini-tutorial on iBGP.

Actually, I did a presentation on implementation issues with iBGP to the
IDR working group during the 1996 Montreal IETF,
<http://www3.ietf.org/ietf/96jun/area-routing-96jun.txt>. At the time,
the operational problem I discovered was that different routing
implementations (specifically Cisco IOS and gated) used a different set
of iBGP tie-breakers, which could cause routing loops if both
implementations co-existed within the same Autonomous System (such as in
the backbone I worked on at the time, where we were replacing
gated-based routers with Ciscos). In fact, tie-breakers for both iBGP
implementations differed from what is documented in RFC 1771. My
work-around at the time was to implement the Cisco iBGP route selection
in gated as an alternate configuration option, mostly because I didn't
have any ability to change the IOS implementation. The response from the
BGP experts at that meeting (e.g. Dennis Ferguson, Tony Li, Yakov
Rekhter) was that the Cisco IOS and gated implementations ought to
conform to the tiebreakers specified in RFC 1771 -- which I don't think
ever happened, because operators relied so heavily on the non-conforming
default tiebreaker rules, e.g. choosing the route with the shortest
AS-PATH.

I stepped away from this issue for almost 10 years, but it looks like
the issue was finally resolved by the latest BGP-4 specification, RFC
4271, where they effectively adopted Cisco IOS's iBGP route selection
algorithm as the new standard. Compare section 9.1.2.2 of RFC 4271 to
section 9.2.1.1 of RFC 1771.

How does that related to this discussion on CMTS routing? I suspect that
if one CMTS vendor implemented BGP-4 per RFC 1771, and another CMTS
vendor implemented BGP-4 per RFC 4271, and they are co-located in the
same BGP AS, there could be potential routing loops on a cable
operator's network. I wonder how many CMTS vendors have implemented RFC
4271 yet?

_______________________________________________
dhcwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dhcwg

---

From: Ted Lemon <[email-masked]>
Date: Wed, 14 Feb 2007 22:01:43 -0700

> how would the DHCP server know about it?

Why do we care?   The delegated prefix has a lifetime; when the  
lifetime expires, we nuke the route.   I guess if the lack of a route  
to the CMTS is discovered high in the network hierarchy, that's going  
to save some packets going over that wire, but on a practical level I  
don't see why it makes much difference - if the device is gone, it's  
not going to be doing things that generate traffic anyway.   So I  
don't really see this as a reason to do routing injection at the  
relay agent.

> is an issue for both software development as well as day-to-day

This is certainly true.

To be honest, the only thing I've heard in this debate that I find at  
all compelling is Ralph's mention of the idea of different  
administrative domains running DHCP servers within the same routing  
infrastructure, as is common in at some ISPs in Asia.   In that case,  
injecting routes from the DHCP server may be somewhat harder to  
arrange than in the case where a single administrative domain  
controls both the DHCP server and the routing infrastructure.

However, what I would really like to see here is some input from  
folks who *are* routing experts.   I would like to see a  
recommendation for how to proceed coming from them, not from us.    
Either way, though, whether the injection happens at the router or at  
the DHCP server, I would really like to not see this protocol  
piggybacked on top of the usual DHCP transaction with the DHCP client.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 14 Feb 2007 22:05:19 -0700

On Feb 14, 2007, at 10:01 PM, Ted Lemon wrote:
> I guess if the lack of a route to the CMTS is discovered high in  
> the network hierarchy,

This was poorly phrased.   What I meant is that if the lack of a  
route to the customer is discovered by the CMTS and quickly  
propagated into the network, there may be some minor benefits to  
that, but I don't find them particularly compelling.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 15 Feb 2007 01:06:13 -0700

On Feb 15, 2007, at 1:02 AM, Ole Troan wrote:
> route injection from the edge does require state in the L3 edge
> device. the act of doing prefix delegation does create this state and
> it is very convenient to take advantage of that so that the edge
> router can inject the route based on PD state.
>
> since a DHCP relay should (at least architecturally) be stateless,  
> then
> a DHCP relay and edge route injection are mutually exclusive.

So do you have any thoughts about how we should actually do this?    
What you're saying implies to me that the DHCP server doing the PD  
should somehow contact the correct router, which might be identified  
by the relay agent, but that the relay agent oughtn't to do anything  
more than identify which router to talk to.   But I am afraid I may  
be reading too much into what you've said.

---

From: Ole Troan <[email-masked]>
Date: Thu, 15 Feb 2007 17:13:02 +0900

> be reading too much into what you've said.

in the edge route injection case (there are many others), which if I
remember correctly is described loosely in RFC3633, the delegating
router must be the first L3 device. so in that case DHCP is kept only
on the edge of the ISP network.

a backend provisioning system can be used. the DHCP server/delegating
router on the edge device can synthesize DHCP replies based on
information from e.g a RADIUS server.

if an "end-to-end" DHCP infrastructure is wanted, I suppose one could
use a DHCP proxy on the edge device. server downstream, client
upstream. that should take care of the state maintenance issues that
came up with the RAAN proposal.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 15 Feb 2007 07:12:13 -0800

> <http://www1.ietf.org/mail-archive/web/dhcwg/current/msg06913.html>.

No - the relay agent is a router in my use case; it's just
selective about which routes it will inject into the IGP.
But I'll agree that it is a different use case - I didn't
even know what CMTS stood for before seeing some of the
messages on this list.

Fred
[email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 15 Feb 2007 07:44:00 -0800

> that, but I don't find them particularly compelling.

I don't want to muddy the waters with my limited understanding
of the CMTS use case, but it seems to me that if some/all of
the routes pertaining to a particular requesting router become
defunct then prospective correspondent nodes would probably
appreciate getting "unreachable" feedback from the CMTS.

Especially with IPv6, the unreachables could give correspondent
nodes an early indication that they might want to try again with
a different address - and, that might make all the difference
between communication failures and a successful connection.

Fred

---

From: Stig Venaas <[email-masked]>
Date: Mon, 19 Feb 2007 13:54:28 +0100

Woundy, Richard wrote:
[...]
> example, the same concerns apply to iBGP route injection -- the DHCP

Just to clarify, I never suggested that this was a way to do things,
rather the contrary. What I said was that route injection by the DHCP
server generally is not possible (as also Ole mentioned later), but
pointed out that there was a theoretical possibility with OSPF. Then
I also pointed out the issue with "stale routes".

I brought this up in response to Ralph and someone else talking about
the possibility of the DHCP server doing injection. My view is that
injection by the DHCP server is not the way to go.

Stig