Subject: Re: FW: Fwd: I-D ACTION:draft-ietf-ppvpn-cl-tunneling-vpn-00.txt

From: Eric Rosen <[email-masked]>
Date: Fri, 01 Nov 2002 11:10:47 -0500

I don't  remember the discussion  which concluded that  this should be  a WG
document.  But anyway ... 

The draft really contains two separate and unrelated ideas.  The first idea
is a  proposal for  tunneling L3VPN packets  through a backbone.  The second
idea  is a  proposal for  creating and  purging "cut-through"  routes (i.e.,
routes between spokes) in a hub-and-spoke VPN. 

Let's look at these separately. 

The tunneling proposal is the following: 

- Each VFI has its own, unique  IPv6 address, different from the address of
  its PE, and different from the address  of any other VFI in the same or in
  a different VPN.

- At an ingress PE, a VFI must  map a VPN packet's IP destination address to
  the IPv6 address of an egress VFI.

- The VPN packet is then  tunneled across the backbone by being encapsulated
  in IPv6, according to RFC 2473 ("Generic Packet Tunneling in IPv6"), which
  is basically  just an "anything  inside IPv6" encapsulation.  The  meat of
  the proposal is that the IPv6 destination address field must be set to the
  address of the egress VFI.  Thus, no other demux field is needed. 

Is this a useful technique in the context of L3VPN?  I have a few comments: 

- The proposal seems to be  restricted to backbones that support IPv6.  This
  limits  its  utility rather  severely.   (Well,  one  could use  a  6-in-4
  encapsulation around  the RFC 2473  encapsulation, but that seems  kind of
  silly.)

- The  proposal requires  that  the ingress  VFI  map a  VPN IP  destination
  address  to an  IPv6 address  which uniquely  identifies a  VFI.   It then
  assumes that somehow each PE belonging  to a particular VPN knows the IPv6
  addresses of all the remote VFIs  in that VPN.  It's not worth considering
  the  case in  which  every VFI  in every  PE  is provisioned  with the  v6
  addresses of  every remote  VFI in  the same VPN;  that would  subvert the
  manageability/scalability claims that are  being made.  So we must suppose
  that there  is some sort  of discovery scheme  which passes around  the v6
  addresses dynamically.

  Given this sort of discovery scheme, my observation is that it could just
  as easily pass, for each VFI, the IP address of a PE, along with a 32-bit
  (or 20-bit) label  value that selects a particular VFI  in that PE.  This
  could then be carried as part of  a GRE encaps (or as part of an MPLS-in-
  IP-or-GRE encaps). 

  Then we would have an encapsulation  which works equally well on v4 and v6
  backbones,  is much  more  efficient  on v4  backbones  (if somewhat  less
  efficient on  v6 backbones), and  is more scalable from  the manageability
  perspective, as  it doesn't require  the provisioning of v6  addresses for
  the VFIs.

  I think the  authors need to make a  case as to why using a  v6 address to
  identify a VFI is better than any of the existing schemes.

- The proposed encaps  doesn't seem applicable to the 2547  style of VPN, as
  that  distributes  PE  IP  addresses  and MPLS  labels,  rather  than  VRF
  addresses.  Of course, one  could imagine  extending 2547  to pass  the v6
  addresses, but it's hard to see the advantage of doing so.  

  I don't know whether the proposed encaps has any particular advantages for
  the VR style of VPN, it might be nice to hear from the proponents of VR.

So I just  don't see why the proposed encapsulation is  a good idea.  Unless
it gets integrated into  one of the L3VPN schemes, I don't  see how it could
advance to  proposed standard.  I  don't see much  point in advancing  it to
informational either, unless it already has significant deployment.  There's
an unlimited number of possible encapsulations, and I don't think we want an
RFC to describe each one.

Now let's look at the "cut-through" proposal. 

This is  apparently meant to apply only  in a "hub and  spoke" topology.  At
the spokes, the VFI has three kinds of routes: 

- a default route pointing to the hub VFI,

- routes to the locally attached site,

- "cut-through routes", i.e., routes pointing to other spokes. 

The cut-through routes are installed and removed dynamically by means of the
"Connectionless Tunneling Control Protocol" (CTCP). 

The hub VFI does not contain  cut-through routes, but it must contain routes
to all the sites attached via all the spokes. 

The CTCP procedures are the following: 

- If a hub  H receives a packet with destination address  D, from ingress PE
  S, and if H's route for D goes through another PE, R, then: 

  * H forwards the packet to R, and

  * If S  is a spoke,  H sends  a CTCP message  (redirect) to S,  telling it
    "your route to D is via R".  H then forgets all about it.

  * When S  receives such a  CTCP message, it  installs the new route  to R.
    Thereafter, S sends packets to D via R instead of via H

- If a spoke R receives a packet with destination address D, from ingress PE
  S, and if R's route for D goes through another PE, then: 

  * R drops the packet, and 

  * If S is a spoke, R sends a CTCP message (purge) to S, telling it "your
    route to D is not via R".  (This is black hole prevention.)

  * When S receives such a CTCP message, is removes the route to D via R. 

The draft  doesn't really discuss what  happens if there  are multiple hubs,
though  I assume that  the hubs  don't want  to send  CTCP messages  to each
other.  The draft seems to assume  that each PE knows (for each VFI) whether
it is a hub  or a spoke; presumably this is a  matter of configuration.  But
to avoid  situations in which  hubs send CTCP  to each other, each  hub must
know of all the other hubs.  The spokes also have to know of the hubs, so as
not to  send purge messages  to hubs.  (At  least, I assume that  the spokes
don't  want to  be  bombarding the  hubs  with useless  purge messages.)   I
imagine  that hubs are  supposed to  ignore any  received purge  or redirect
messages, though the draft doesn't seem to mention that explicitly.  

It's very  difficult to know what  topologies are supposed to  be ruled out,
how  those  rules   are  to  be  enforced,  and   what  happens  if  through
misconfiguration an  "unsupported topology" is created.  The  strict hub and
spoke  hierarchy  seems   to  be  the  only  thing   that  protects  against
long-standing  loops,  so  it's  important  to understand  whether  this  is
restriction  is acceptable  and  enforceable.  For  example, requiring  that
there only  be a single hub is  obviously not acceptable, due  to the single
point of failure issues.  

Even ignoring these issues, there are some problems. 

One problem  is that the  hub doesn't  seem to remember  that it has  sent a
redirect to a spoke.   So suppose that S1 and S2 are  spokes attached to the
same site, but S1's route is, at some time, the preferable one.  The hub may
now redirect some  third spoke S3, so  that packets from S3 for  the site in
question go  directly via S1.  Then  let us suppose that  routing changes so
that S2's  route to the site is  now the preferable one.   S3's packets will
continue going via S1, as long as  S1 has a direct route to the site.  There
doesn't seem to be any way to get S3 back on the optimal route, ever.

(I am assuming here that one  does not want to relegate multiply homed sites
to the case of an unsupported topology.) 

I  think there  needs  to be  a rule  that  cut-through routes  must not  be
distributed into the site via the site's IGP, i.e., that the CE routers must
only default-route to the spokes. Otherwise I think loops may be possible.

Another  problem: any  procedure in  which control  messages are  sent  as a
response to the  reception of data messages is  somewhat suspect; throttling
of the control messages becomes  a necessity, though this isn't mentioned in
the draft.  (Perhaps this is regarded as an implementation matter.)

Another problem is  that the redirect and purge  messages are unreliable and
unsequenced.  Purge messages  may fail to get through in  any sort of timely
manner, and the result may be black holes that last for quite awhile.  Purge
messages which  arrive late may already  be obsolete, undoing  the effect of
redirect  messages which  were  sent later  but  arrived earlier.   Redirect
messages which  arrive late may already  be obsolete, undoing  the effect of
purge messages which were sent later  but arrived earlier.  If a spoke loses
a route,  there will be  a lag time  during which the hub  hasn't recomputed
routes yet;  during this  time, there may  be a  "war" during which  the hub
constantly  emits redirects  for the  lost route,  and the  spoke constantly
emits purges.  With no sequencing,  some of these messages may get processed
after  the hub  recomputes the  routes, and  this may  greatly  increase the
effective routing convergence time. 

The use  of a connectionless control  plane is also suspect  from a security
perspective.   There  needs  to  at   least  be  some  optional  method  for
authenticating and  ensuring the integrity  of the CTCP messages.   When the
control  plane sets up  connections, one  can talk  about using  IPsec, TLS,
etc., but when each control packet  is a datagram, I'm not sure I understand
how the  security would be  done.  The "security considerations"  section is
only one sentence long; no one has  ever let me get away with writing such a
short security considerations section ;-)

It  would be  interesting to  hear from  proponents of  the VR  model  as to
whether they see any need for  cut-through routes.  If they do, then perhaps
there  might be some  merit in  having the  WG look  into these  issues more
closely. 

Right now  I think there's no question  of CTCP going forward  as a proposed
standard.  I  don't see  how that could  happen unless CTCP  were integrated
into the  VR model  in some  way.  With regard  to its  going forward  as an
informational RFC, I don't really see why it should.  Setting up cut-through
routes  is  not exactly  a  new idea  (cf.   NHRP,  especially the  infamous
router-to-router NHRP), and this particular  solution seems to have a number
of  unresolved issues.   Unless there  is already  widespread  deployment of
CTCP, I don't see what the interest is.

---

From: Takeshi KUWAHARA <[email-masked]>
Date: Mon, 11 Nov 2002 13:11:35 +0900

Dear Eric,

Thank you very much for your comments on our draft.
And sorry for delay of my reply.

>   encapsulation around  the RFC 2473  encapsulation, but that seems  kind of

We don't think the utility of our proposal is severely limited compared with
MPLS/VPN, as MPLS/VPN is restricted to backbone that supports MPLS or MPLS
over GRE/IP.
Of course, MPLS as well as IPv6 has several alternatives as a lower layer
network.
We agree with your point of the usage of a 6-in-4 encapsulation in this case.

>   addresses of all the remote VFIs  in that VPN.

Each spoke-VFI belonging to a particular VPN doesn't need to know addresses
of all the remote VFIs in that VPN. A spoke-VFI has to know the addresses
of the hub-VFIs, then other addresses of remote VFIs are learned automatically
by CTCP.

>It's not worth considering
>   manageability/scalability claims that are  being made.

The provisioning of every VFI with addresses for all the VFIs, whch makes a
full mesh topology, would obviously subvert the scalability. Therefore, we
have adopted the hub-and-spoke topology that would lighten the provisioning
of each spoke-VFI with only addressees of the hub-VFIs.

>   So we must suppose

Thus, we are proposing the CTCP to solve this issue.

>   perspective, as  it doesn't require  the provisioning of v6  addresses for

As a matter of the flexibility perspective in configuring the SP network,
our proposal can introduce the same configuration approach as an IPv6 network,
because it is based on IPv6.

>   identify a VFI is better than any of the existing schemes.

We consider that CL protocol is necessary to adopt CTCP.
As for the CL protocol, IPv6 is suitable because it has huge address space
that would make it possible to accommodate a large number of sites and VPNs.

>   addresses, but it's hard to see the advantage of doing so.

RFC2547 proposes mainly about MPLS signaling and BGP-4 extension for supporting
VPN. On the other hand, our draft proposes a method of controlling VPN tunnels
by using IP-in-IP. So our draft is not directly related to RFC2547, above all,
it is not necessary to refer to the Info. RFC.

>there only  be a single hub is  obviously not acceptable, due  to the single

Multiple hubs can be connected to form any type of topology. In this case,
routes between DF-DF, DF-PE and PE-PE are controlled by routing protocols.
A tree topology is obviously desirable as a mesh-type topology increases
the load of routing. Also, as you have mentioned above, hub(DF)s must
ignore any received purge or redirect messages in any type of topology.
As for the bombarding to the hubs, we think it only occurs in the case of
a malfunction of routing protocols, and if it's necessary, a PE may be
restricted to send duplicated massages.

>doesn't seem to be any way to get S3 back on the optimal route, ever.

                                     +<-----DF
                                     |
                                     v
   D<---(preferable route)<----S1<---+<---------S3
   |
   +---------------------------S2

In this case, when S2's route becomes the preferable route by some reasons,
the route from the DF to the site will be changed from DF->S1->D to DF->S2->D
by routing protocol, hence S1 will then sends a purge message to S3. After
some packets going via the DF, the route from S3 to the site will finally
be changed to S3->S2->D. Therefore, the problem you have mentioned will not
occur.

>only default-route to the spokes. Otherwise I think loops may be possible.

As described above, a CE router can be multiply homed to PEs, and
loops will not occur more than that may be occurred by routing protocols.

>the draft.  (Perhaps this is regarded as an implementation matter.)

Unnecessary overlaped redirect/purge messages can be resiricted in sending
PEs, but we think that is an implementation matter.

>effective routing convergence time.

Restricting unessesary overlaped redirect/purge messages would lighten this
problem. Additionally, if the topology is a hub-and-spoke or a tree type,
the time lag of setting routes would not be so long since the PE only needs
to send accommodating addresses to the upper PE/DF.

>short security considerations section ;-)

A closed IPv6 network would keep the security for VPN, just the same as
the 2547 style of VPN keeps the security by closing the MPLS core network.

Regards,
Takeshi Kuwahara.

---

From: Eric Rosen <[email-masked]>
Date: Mon, 11 Nov 2002 11:22:15 -0500

> In  this  case, when  S2's  route becomes  the  preferable  route by  some
> reasons, the route from the DF  to the site will be changed from DF->S1->D
> to DF->S2->D by routing protocol, hence S1 will then sends a purge message
> to S3.

S1 sends a purge message to S3 because the DF's route to D has changed?  How
does S1 know that the DF's route to D has changed?  

In this  example, D is  "locally attached" to  S1 (as well  as to S2),  so I
don't see that S1 has any reason to send a purge. 

> routes  between   DF-DF,  DF-PE  and  PE-PE  are   controlled  by  routing

Are you saying  that when there are multiple hubs (or  anything other than a
strict hub  and spoke  topology), you don't  use CTCP?  For  robustness, I'd
think that every VPN would have at least two hubs.

---

From: Takeshi KUWAHARA <[email-masked]>
Date: Tue, 12 Nov 2002 20:37:08 +0900

Dear Eric,

>don't see that S1 has any reason to send a purge.

In the case of link failure between S1 and D, S1's route to D will be changed
to S1->DF->S2->D by the routing protocol. Then, S1 will send a purge message
to S3 when it receives packets from S3 to D. Consequently, the route from S3
to D will then be changed to S3->DF->S2->D, and by the redirection message
from DF to S3, it will finally be changed to S3->S2->D.

It is difficult to control the packet from S3 to D dynamically by the DF so as
to load balance between two routes S3->S1->D and S3->S2->D. To solve this issue,
one can be managed by ether not letting the DF to send a redirection message
for the packets destined for D or operating a routing protocol to set S1-S3
and S2-S3 as routing peers.

Therefore, we think CTCP does not restrict the network topology.

>think that every VPN would have at least two hubs.

CTCP can be applied to the topology of multiple hubs.
When there are multiple hubs, a spoke is able to choose an appropriate hub to
send packets by the routing protocol operated among multiple hubs and spokes.
So, it is possible to change the next hop to the backup hubs in the case of
failures.

Regards,
Takeshi Kuwahara.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 12 Nov 2002 08:21:51 -0500

There are two cases recognized in your document.
In one case, you require a full mesh abong all PE devices, and do not use 
CTCP.  This is roughly similar to many other tunnel protoocols.

However, the primary focus of your document is CTCP.  The use of CTCP 
places very specific restrictions on the operator(s) VPN support topology.
All cut-through creation messages are generated by DF.  In order to 
generate them, the DF must have received the message directly from a PE 
device, and be forwarding the message directly to a PE device.  Therefore, 
in order to function, all PE devices must be exchanging routing information 
with each DF.  Yes, you can have multiple DFs for load sharing of packet 
forwarding activity.  However, it that case they ALL must have routing 
exchanges with ALL PEs in the VPN.  And all PEs in the VPN must know about 
and deliver routing to ALL the DFs.
Note that this requirement for complete coverage applies to all PEs in the 
VPN across all providers.
The necessity for such topology restrictions, including the additional 
restrictions that DF entities MUST be distinct from PE entities, is driven 
by the fact that as shown in the earlier IETF work on cut-through, without 
such restrictions it is extremely difficult to avoid inducing loops and 
black holes when using cut-through protocols.

You could try to claim that by creating separate groups, with PE devices 
forwarding from one group to another, you can get multiple hub and spoke 
clusters.  There are quite a number of interesting topological constraints 
you would have to your document in order to make this work.  Even 
determining how such a PE would know which cluster to forward packets to is 
quite complex and beyond what is described in your document.

Note also that you describe the control exchange between the DF and PE 
devices as if it were a normal routing instance.  While not a very 
complicated exchange, it is certainly different from any existing 
exchange.  For the DF receiving the advertisements from the PEs, it must 
learn the reachability and associate it with the IPv6 soure (a minor 
tweak).  However, the DF should not advertise reachability to other DFs or 
to the PEs.  PEs must ignore any reachability information they get from the 
DFs since the nature of the protocol requires that all packets from the 
custoemr site are forwarded to the DF, and packets from the provider are 
checked against customer learned routing only for forwarding.

Yours,
Joel M. Halpern

---

From: Eric Rosen <[email-masked]>
Date: Tue, 12 Nov 2002 13:02:16 -0500

> In the  case of link  failure between S1  and D, S1's  route to D  will be
> changed to S1->DF->S2->D by the routing protocol. 

I don't believe I mentioned "link failure" in my example. 

> It is difficult to  control the packet from S3 to D  dynamically by the DF
> so as to load balance between two routes S3->S1->D and S3->S2->D. To solve
> this issue,  one can  be managed  by ether not  letting the  DF to  send a
> redirection message for the packets  destined for D or operating a routing
> protocol to set S1-S3 and S2-S3 as routing peers. 

Well, sure, you can always turn CTCP off.

> Therefore, we think CTCP does not restrict the network topology. 

The fact  that you can turn CTCP  off doesn't mean that  it doesn't restrict
the network topology. 

> CTCP can  be applied  to the  topology of multiple  hubs.  When  there are
> multiple  hubs, a  spoke is  able  to choose  an appropriate  hub to  send
> packets by the  routing protocol operated among multiple  hubs and spokes.
> So, it is possible  to change the next hop to the  backup hubs in the case
> of failures. 

Now you  are suggesting to  run both CTCP  and a "routing  protocol operated
among multiple hubs  and spokes"?  I don't understand at  all how that would
work.   During a  routing transient  you  could get  two hubs  independently
redirecting the spokes, and the routing might never converge.

---

From: Takeshi KUWAHARA <[email-masked]>
Date: Thu, 14 Nov 2002 11:06:18 +0900

Joel,

Thank you for your comments on our draft.  Please see my comments inline.

At 08:21 02/11/12 -0500, you wrote:
>in order to function, all PE devices must be exchanging routing 
>information with each DF.  Yes, you can have multiple DFs for load sharing 
>of packet forwarding activity.  However, it that case they ALL must have 
>routing exchanges with ALL PEs in the VPN.  And all PEs in the VPN must 
>know about and deliver routing to ALL the DFs.
>Note that this requirement for complete coverage applies to all PEs in 
>the VPN across all providers.

Thanks again for your clarifications.
We also see loops and blackholes in the cut-though control as major problems.
However, as for scalability perspective, we find it very attractive that we
introduce the approach of keeping reachability by using default routes and
improving packet forwarding by using cut-through routes. Therefore, we have
been looking for an approach that would solve the problem even if there are
some kind of restriction if it doesn't severely higher the load of routers.

>determining how such a PE would know which cluster to forward packets to 
>is quite complex and beyond what is described in your document.

We also think multiple hub and spoke clusters are effective to scale with 
large networks.
In this case, delivering routes among multiple hubs is considerable. We 
think multiple hubs
can take any type of topology and routes are delivered by routing protocols.
However, the mesh topology is desirable for the hubs' topology, as 
cut-through controls
are needed for each hop of the hubs. Here, the destination of a 
cut-through route is changed
hub after hub until it finally reaches to an egress PE.

>to the PEs.  PEs must ignore any reachability information they get from 
>the DFs since the nature of the protocol requires that all packets from 
>the custoemr site are forwarded to the DF, and packets from the provider 
>are checked against customer learned routing only for forwarding.

It is possible to operate DF and PE devices as a normal routing instance.
However, we believe above-mentioned approach is much effective in order to
lower the load of routing by using the hub and spoke topology.

Regards, Takeshi Kuwahara.

---

From: Takeshi KUWAHARA <[email-masked]>
Date: Thu, 14 Nov 2002 11:14:29 +0900

Eric,

Thanks for your reply. Please see the comments below.

>redirecting the spokes, and the routing might never converge.

The reasons for introducing multiple hubs are to lighten the load of hub and
to improve reliability. Let me explain a procedure with the following example.

In the figure below, each spoke announces accommodating routes to both hub1
and hub2 by routing protocols. And each of them chooses an appropriate hub
to forward packets. Here, the load of each hub is lightened by spokes choosing
hubs dispersively.

       +-----+       +-------+       +-----+
       |     |-------|spoke 1|-------|     |
       |     |       +-------+       |     |
       |     |                       |     |
       |     |       +-------+       |     |
       |     |-------|spoke 2|-------|     |
       |     |       +-------+       |     |
       |hub 1|                       |hub 2|
       |     |       +-------+       |     |
       |     |-------|spoke 3|-------|     |
       |     |       +-------+       |     |
       |     |                       |     |
       |     |       +-------+       |     |
       |     |-------|spoke 4|-------|     |
       +-----+       +-------+       +-----+

When a failure occurs in a route between hub and spoke, the spoke determines
the failure by routing protocol, and then changes the forwarding route to the
backup hub. In this way, reliability of the network can be achieved.

As hub1 and hub2 have the same routing information to each spoke, routing would
converge. Even when improper redirection messages are sent to the spokes during
a routing transient, each spoke sends a purge message when it cannot forward
packets to the accommodating CEs, so that the routes are changed properly.