Subject: Re: [Softwires] SOFTWIRE working group last call on 6rd

From: Mark Townsley <[email-masked]>
Date: Mon, 01 Mar 2010 09:46:51 +0100

On 3/1/10 8:44 AM, Gabi Nakibly wrote:
> Hi all,
> I would like to comment on the loop avoidance issue addressed in the 
> Security Considerations section.
> First, I agree that the the best mitigation measure here for loops via 
> external relays is to simply drop at the border of the 6rd domain 
> packets sent from or destined to a 6rd BR.
> Second, regarding loops via internal relays the draft mentions that:
>    To avoid forwarding loops via other internal relays, the BR should
>    employ outgoing and incoming IPv4 packets filters, filtering out all
>    known relay addresses for internal 6rd BRs, ISATAP routers or 6to4
>    relays, including the well known anycast address space for 6to4.
> Does the SP necessarily knows of every ISATAP router or 6to4 
> relay deployed in the 6rd domain?
Deployed by the SP.
> After all, a customer can deploy an ISATAP router or even a 6to4 relay 
> for that matter without the knowledge of the SP, while using 
> a configured tunnel (by a tunnel broker) for its IPv6 connectivity.
As presented in Hiroshima, I don't think an attack between a 
subscriber's own equipment and the BR is a top concern. The amount of 
traffic a single subscriber can generate with a loop is no more than the 
subscriber could generate by downloading and uploading a lot of v6 
content at the same time from outside the 6rd domain. Further, with a 
tunnel loop, the download path would be rate-limited by the upload path, 
which is generally quite asymmetric (e.g., today a bit-torrent user 
within a decent swarm off-net could generate more traffic on the BR than 
a loop). Point is, there a are a lot of ways a user can fill his own 
pipe, and a loop between relays is probably way down low on the list of 
interesting things to do.

> I suggest to keep the filtering of internal 6rd BRs. But to avoid 
> loops via internal ISATAP routers or 6to4 relays, I would suggest 
> employing the destination/source address check described in Section 
> 3.1 in http://tools.ietf.org/html/draft-nakibly-v6ops-tunnel-loops-01 
> we have recently submitted. It is not a perfect solution, and indeed 
> it should be updated in case other types of automatic tunnels come up, 
> but it does the job for ISATAP and 6to4 loops without demanding 
> a knowledge of every internal ISATAP router or 6to4 relay.
If you are referring to this kind of check:

    For example, if tunnel router A (ISATAP router or 6to4 relay)
    forwards a packet into a tunnel interface with an IPv6 source address
    that matches the 6to4 prefix 2002:IPa::/48, the router discards the
    packet if IPa is one of its own IPv4 addresses.  In a second example,
    if tunnel router B (ISATAP router or 6to4 relay) receives an IPv6
    packet on a tunnel interface with an IPv6 destination address that
    matches the ISATAP address suffix ::0200:5EFE:IPb, the router
    discards the packet if IPb is one of its own IPv4 addresses.

My concern here is that checking all possible local IPv4 addresses at a 
particular location in the IPv6 address for every packet switched 
through the BR would result in very significant overhead for a 
hardware-optimized implementation operating at high speed. I have no 
objection to someone doing this kind of check in their implementation, 
but their customers had better be willing to pay for it ;-)

Currently, we have this paper referenced:

    [RoutingLoop]
               Nakibly and Arov, "Routing Loop Attacks using IPv6
               Tunnels", August 2009,<http://www.usenix.org/event/
               woot09/tech/full_papers/nakibly.pdf>.

Should we refer to your Internet Draft instead?

> Other than this I fully support this draft for advancement.
Thanks,

- Mark

> Gabi
>
>
> ------------------------------------------------------------------------
> *From:* "Durand, Alain" <[email-masked]>
> *To:* softwires <[email-masked]>; DHC WG <[email-masked]>
> *Cc:* Ole Troan <[email-masked]>
> *Sent:* Mon, February 22, 2010 11:43:00 PM
> *Subject:* [Softwires] SOFTWIRE working group last call on 6rd
>
> All -
>
> We'd like to announce the softwire WG LC on the 6rd technology. I've 
> copied both SOFTWIRE and DHC mailing lists and solicit comments from 
> both groups of experts. The draft is here:
>
> _http://www.ietf.org/internet-drafts/draft-ietf-softwire-ipv6-6rd-07.txt
> _
> Please reply with comments to this thread by 2010.03.08 at 1700 PST
>
> Thanks
>
> - Alain
> _______________________________________________
> Softwires mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/softwires_
>
>
> _______________________________________________
> Softwires mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/softwires
>

---

From: Mark Townsley <[email-masked]>
Date: Mon, 01 Mar 2010 10:08:22 +0100

Hi Martin, great news on the interoperability. Thanks for sharing this.

A good IPv4 ECMP load-balancing algorithm will take into account the 
source and destination IPv4 address pair and try to keep associated 
flows along the same path if possible. However, the fewer the IP 
addresses, the more difficult this becomes, and at some point the 
algorithm could favor balancing the load vs. keeping the detectable 
flows together. If your test was with a small number of RGs, perhaps 
this is what you were running into.

In a typical deployment with a fairly large number of subscribers, each 
with their own source IPv4 address together with the relatively large 
link speeds on the routers performing ECMP vs. the subscriber's upstream 
bandwidth, I don't think you will see the same behavior you did in the lab.

This is really good input for an operational document on 6rd though.

Thanks,

- Mark

On 2/25/10 9:05 AM, [email-masked] wrote:
>
> Hi,
>
> We have succesfully shown interoperability between the 6RD CEs of two 
> different manufacturers (integrated into VDSL routers) and the 6RD 
> border relays of yet another two manufacturers. All implementations 
> run smoothly, and we have not discovered any issues with the 6RD 
> technology. 6RD fits very well into a service provider enironment, 
> allowing for a cost efficient IPv6 deployment at a large scale. I 
> stronlgy support this draf'ts advancement to standards track.
>
> We are running a setup with redundant border relays. At first, we had 
> some issues with packet reordering when both relays had the same IGP 
> metric, and load balancing was performed on a per packet basis. This 
> adversely affected the TCP througphut of Windows XP. Other OS did not 
> suffer from this. We have now changed from load balancing to an 
> active/standby configuration.
>
> Best regards,
>
> Martin Gysi, Swisscom
>
> *From:* [email-masked] [mailto:[email-masked]] 
> *On Behalf Of *Durand, Alain
> *Sent:* Monday, February 22, 2010 10:43 PM
> *To:* softwires; DHC WG
> *Cc:* Ole Troan

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Mar 2010 13:49:29 -0800

Ole,

Just getting back to e-mail after a three-day weekend:

> -----Original Message-----
> From: Ole Troan [mailto:[email-masked]]
> Sent: Thursday, February 25, 2010 4:36 PM
> To: Templin, Fred L
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: Re: SOFTWIRE working group last call on 6rd
> 
> Fred,
> 
> > Regarding the load balancing issue, if the CE routers are exposed to
> > the list of unicast addresses of BRs they can use Equal Cost
> > MultiPath (ECMP) to direct the load balancing if any load balancing
> > is desired. This would allow multiple active BRs (as opposed to
> > active/standy) and still give a nice distribution of traffic without
> > packet reordering issues.
> 
> if you do per-packet load balancing you could get re-ordering in any place in the network, assuming
> multiple paths. regardless of what the CE does.
> it would of course be beneficial if intermediate routers manage to do flow based load sharing also
> for tunneled packets.

I'm sorry I wasn't more specific, but what I intended to
say was per-flow; not per-packet. It says this explicitly
in Section 6.3.2 of draft-templin-intarea-vet. There is
also the possibility for in-the-network ECMP if the tunneling
mechanism were to use UDP encaps. as discussed in Section
4.2 of that document.

> in any case one could adjust the IGP metrics for the anycast routes to avoid any load balancing
> between them from a single CE.

You mean so that one BR always appears more favorable than
all of the others? That may allow for fault tolerance but
would seem to erase the opportunity for per-flow load
balancing when there are multiple BRs that should really
be seen as equivalent.

Fred
[email-masked] 

> cheers,
> Ole
> 
> >
> > Fred
> > [email-masked]
> >
> > From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> > Sent: Thursday, February 25, 2010 12:05 AM
> > To: [email-masked]; [email-masked]
> > Cc: [email-masked]
> > Subject: Re: [Softwires] SOFTWIRE working group last call on 6rd
> >
> > Hi,
> >
> > We have succesfully shown interoperability between the 6RD CEs of two different manufacturers
> (integrated into VDSL routers) and the 6RD border relays of yet another two manufacturers. All
> implementations run smoothly, and we have not discovered any issues with the 6RD technology. 6RD fits
> very well into a service provider enironment, allowing for a cost efficient IPv6 deployment at a
> large scale. I stronlgy support this draf'ts advancement to standards track.
> >
> > We are running a setup with redundant border relays. At first, we had some issues with packet
> reordering when both relays had the same IGP metric, and load balancing was performed on a per packet
> basis. This adversely affected the TCP througphut of Windows XP. Other OS did not suffer from this.
> We have now changed from load balancing to an active/standby configuration.
> >
> > Best regards,
> > Martin Gysi, Swisscom
> >
> > From: [email-masked] [mailto:[email-masked]] On Behalf Of Durand, Alain
> > Sent: Monday, February 22, 2010 10:43 PM
> > To: softwires; DHC WG
> > Cc: Ole Troan
> > Subject: [Softwires] SOFTWIRE working group last call on 6rd
> >
> > All -
> >
> > We'd like to announce the softwire WG LC on the 6rd technology. I've copied both SOFTWIRE and DHC
> mailing lists and solicit comments from both groups of experts. The draft is here:
> >
> > http://www.ietf.org/internet-drafts/draft-ietf-softwire-ipv6-6rd-07.txt
> >

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Mar 2010 14:04:59 -0800

Ole,

> From: Ole Troan [mailto:[email-masked]] On Behalf Of Ole Troan
> Sent: Thursday, February 25, 2010 4:44 PM
> To: Templin, Fred L
> Cc: Brian E Carpenter; [email-masked]
> >> From: Brian E Carpenter [mailto:[email-masked]]
> >> Sent: Thursday, February 25, 2010 12:09 PM
> >> To: Templin, Fred L
> >> Cc: [email-masked]; [email-masked]
> >> Hi Fred,
> >> On 2010-02-26 06:25, Templin, Fred L wrote:
> >>>
> >>>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Alexandre
> Cassen
> >>>> Sent: Wednesday, February 24, 2010 2:05 AM
> >>>> To: [email-masked]
> >>>> On Mon, 22 Feb 2010 16:43:00 -0500, Durand, Alain wrote:
> >>>> This document truly depict what we are running into production in our
> >>>> backbone here. I strongly support it!
> >>>>
> >>>> I read some post on the ML regarding fragmentation and MTU related.
> >>>> IMHO, section 9.1 is enough explicit, anyone playing around with
> >>>> encapsulation will face at one point MTU issues and fragmentation, it is
> >>>> really common. Any operational networking guy will/have to keep in mind
> >>>> this MTU stuff while designing/setting up its encapsulation
> >>>> architecture. What is important here, 6rd is operating inside a specific
> >>>> routing domain so that if there is MTU/fragmentation issues then it is
> >>>> networking domain designer job/responsibility to fix it ! if networking
> >>>> design has been made into a crappy way then it will transitively operate
> >>>> into a crappy way too. IMHO, it should certainly not be the focus of an
> >>>> operational protocol draft, like 6rd, to provide a networking course.
> >>>>
> >>>> In our production env, MTU is set to 1480 at CPE side.
> >>>
> >>> Here's another thing to consider wrt the 1480. Once the
> >>> CPE statically sets a 1480, it will be next to impossible
> >>> to get it to set something larger in the future.
> >>
> >> So what? 6RD is definitely a transition technique running over
> >> legacy IPv4. I think we have running code experience that
> >> defaulting the MTU to 1480 will work in most normal circumstances,
> >> whereas reliance on PMTUD is known to be problematic.
> >
> > To be clear, I am not asking for reliance on PMTUD. I am
> > instead asking for reliance on something better than that
> > (SEAL) so that no artificial limits like 1480 are needed.
> >
> >> If 1480
> >> becomes an issue, it will become part of the motivation to upgrade
> >> to native IPv6.
> >
> > This kind of argument strikes me as a business-oriented
> > rather than technical viewpoint on the problem space. We
> > now have technologies by which we can deploy tunneled
> > service that is as good as or perhaps even better than
> > native (and yes, I include 6rd as part of the solution).
> > So, why not have a once-and-done deployment now as opposed
> > to putting up a temporary service which we know will need
> > replacement in the near future? So that network equipment
> > vendors can sell more products and services?
> 
> I do agree that MTU handling is a general problem, and not even just for tunnels.
> I thought we agreed while discussing earlier versions of this draft,

We had not reached an agreement by my reckoning; I was expecting
follow-up from you, and perhaps you were expecting follow-up from
me and the ball got dropped.

> not to encumber the 6rd
> specification with the general solutions to these MTU problems.

"Encumber" is IMHO not the correct word; the more accurate
word would be "improve". The SEAL spec is going to give
flexibility whereby the tunnel egress can reassemble as
much or as little as it wants, with no reassembly at all
as the limiting case. The spec will say that the egress
SHOULD reassemble IPv6 packets up to 1500 bytes and MAY
reassemble more. But, that would not prevent a tunnel
egress from performing no reassembly at all and simply
report packet too bigs back to the ingress as necessary.
Either way, unlike with classical path MTU discovery the
ingress gets path MTU feedback that it can reliably relay
back to the source.

Fred
 
> 6rd references RFC4213 for most of the tunnel mechanics. could we move that debate to rfc4213bis or
> something?
> 
> cheers,
> Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Mar 2010 17:09:13 -0800

Remi,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Rémi Després
> Sent: Thursday, February 25, 2010 11:04 PM
> To: Ole Troan
> Cc: [email-masked]; Templin, Fred L
> Although the SEAL approach has IMHO some merit of its own for some generalized tunneling scenarios,
> using it  in 6rd would defeat the objective of simplicity for "rapid deployment" of IPv6.

There is an opportunity at hand to get tunneling done right
the first time and then have a long-term quality IPv6 service.
6rd is a large piece to the puzzle, but there are other pieces
that would make for a more complete solution. IMHO, we can
have it both ways of doing it fast *and* doing it right on
the first iteration.

Fred

> The draft should therefore proceed as is on this subject.
> 
> RD
> 
> > Fred,
> >
> See above.
> > I thought we agreed while discussing earlier versions of this draft, not to encumber the 6rd
> This is IMHO the right approach.
>

---

From: Rémi Després <[email-masked]>
Date: Tue, 2 Mar 2010 08:54:53 +0100

> have it both ways of doing it fast *and* doing it right on

Different view here.
As far as 6rd is concerned, doing it right is IMO keeping it as is.

Regards,
RD

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 02 Mar 2010 21:59:38 +1300

> As far as 6rd is concerned, doing it right is IMO keeping it as is.

Yes, given that we have just been told at APRICOT that the vast majority
of Google and YouTube's IPv6 traffic comes from one particular ISP in
France. It's a real success story, with a clamped MTU.

   Brian

---

From: Gabi Nakibly <[email-masked]>
Date: Tue, 2 Mar 2010 06:37:13 -0800 (PST)

Mark,

>As presented in Hiroshima, I don't think an attack between a subscriber's own equipment and the BR is a top concern. The amount of traffic a single 
>subscriber can generate with a loop is no more than the subscriber could generate by downloading and uploading a lot of v6 content at the same time from 
>outside the 6rd domain. Further, with a tunnel loop, the download path would be rate-limited by the upload path, which is generally quite asymmetric (e.g., 
>today a bit-torrent user within a decent swarm off-net could generate more traffic on the BR than a loop). Point is, there a are a lot of ways a user can fill 
>his own pipe, and a loop between relays is probably way down low on the list of interesting things to do.

I definitely see your point here: due to the clients' limited upload bandwidth a routing loop via internal relay may cause only small damage to the BR. If this is the case, why should the draft address the internal loop case at all? Guiding the SP to track all its internal relays may incur too much administrative overhead for such a small threat. On the other hand, one should note that the internal routing loop will probably clog up  the upstream of the client and completely DoS it. So this may be a big threat for the client itself.

>My concern here is that checking all possible local IPv4 addresses at a particular location in the IPv6 address for every packet switched through the BR 
>would result in very significant overhead for a hardware-optimized implementation operating at high speed. I have no objection to someone doing this kind 
>of check in their implementation, but their customers had better be willing to pay for it ;-)

If you still decide to address the internal loop case, it should be noted that actually only one address must be checked for each packet:
- Source address check: the loop will succeed only if the IPv4 source address corresponds to the IPv6 source address, so before forwarding a packet into the tunnel interface the BR must make sure that the IPv6 source address does not correspond to the IPv4 address that is intended to be the packet's IPv4 source address, only. The other IPv4 addresses of the BR are irrelevant here.
- Destination address check: as I gather from Section 7.1.1 in the draft, currently only a single 6rdBRIPv4Address is supported by a CE for each BR. If this means that a BR advertises in the SP's IGP only one IPv4 address, then the BR is reachable to the clients only via this single address. Only this address should be verified in the destination address check. The loop will not work with any other IPv4 addresses, even if they belong to the BR.

>Should we refer to your Internet Draft instead?

I think you should refer to this draft only if you intend to reference one of the mitigation measures we propose there. If you only refer the attacks themselves the current reference is better. 
In any case, I think that our draft will be revised to refer to the 6rd case, as well. 

Gabi
________________________________
From: Mark Townsley <[email-masked]>
To: [email-masked]
Sent: Mon, March 1, 2010 10:46:51 AM
Subject: Re: [Softwires] SOFTWIRE working group last call on 6rd

On 3/1/10 8:44 AM, Gabi Nakibly wrote:
> Hi all,
> I would like to comment on the loop avoidance issue addressed in the Security Considerations section.
> First, I agree that the the best mitigation measure here for loops via external relays is to simply drop at the border of the 6rd domain packets sent from or destined to a 6rd BR.
> Does the SP necessarily knows of every ISATAP router or 6to4 relay deployed in the 6rd domain?
Deployed by the SP.
> After all, a customer can deploy an ISATAP router or even a 6to4 relay for that matter without the knowledge of the SP, while using a configured tunnel (by a tunnel broker) for its IPv6 connectivity.
As presented in Hiroshima, I don't think an attack between a subscriber's own equipment and the BR is a top concern. The amount of traffic a single subscriber can generate with a loop is no more than the subscriber could generate by downloading and uploading a lot of v6 content at the same time from outside the 6rd domain. Further, with a tunnel loop, the download path would be rate-limited by the upload path, which is generally quite asymmetric (e.g., today a bit-torrent user within a decent swarm off-net could generate more traffic on the BR than a loop). Point is, there a are a lot of ways a user can fill his own pipe, and a loop between relays is probably way down low on the list of interesting things to do.

> I suggest to keep the filtering of internal 6rd BRs. But to avoid loops via internal ISATAP routers or 6to4 relays, I would suggest employing the destination/source address check described in Section 3.1 in http://tools.ietf.org/html/draft-nakibly-v6ops-tunnel-loops-01 we have recently submitted. It is not a perfect solution, and indeed it should be updated in case other types of automatic tunnels come up, but it does the job for ISATAP and 6to4 loops without demanding a knowledge of every internal ISATAP router or 6to4 relay.

My concern here is that checking all possible local IPv4 addresses at a particular location in the IPv6 address for every packet switched through the BR would result in very significant overhead for a hardware-optimized implementation operating at high speed. I have no objection to someone doing this kind of check in their implementation, but their customers had better be willing to pay for it ;-)

  [RoutingLoop]

Thanks,

- Mark

> We'd like to announce the softwire WG LC on the 6rd technology. I've copied both SOFTWIRE and DHC mailing lists and solicit comments from both groups of experts. The draft is here:
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Mar 2010 08:51:52 -0800

Brian,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian E Carpenter
> Sent: Tuesday, March 02, 2010 1:00 AM
> To: Rémi Després

You are right that the MTU clamping seems to be the approach
being advocated under 6rd, but I disagree that that is the
best we can (or should) do.

During the NGTRANS days, the ISATAP team was strongly required
to include a "Domain of Applicability" statement, and I think
the same should be required of 6rd. I see a number of aspects
of 6rd that IMHO would limit its applicability, including:

- IPv6 prefix tied to IPv4 address (renumbering implications)
- potential black holes when ICMPs can't be translated
- provider-aggregated addressing only (no provider-independent)
- not compatible with multihoming via a single, stable IPv6 prefix
- not compatible with traffic engineering (i.e., CE can't select
  specific BRs)
- interactions with load balancing; ECMP within provider network
- MTU clamping exposes degenerate MTUs to the customer network
  and doesn't allow for automatic discovery of larger MTUs
- requires operators to exercise tight controls on link MTUs
- requires operators to exercise tight controls on ICMP
  generation and ICMP filtering

So, while I see the 6rd stateless prefix delegation as a
highly desirable contribution, I think the implications
of these limitations should somehow be captured in an
applicability statement the same as was required for the
NGTRANS mechanisms.

Thanks - Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 03 Mar 2010 13:23:10 +1300

Fred,

I agree that distinguishing between pragmatic operational practices
(such as clamping the MTU) and generic solutions is correct, but
as far as the current last call goes, I think that leaving the generic
solution FFS is also correct.

I don't think the text in section 9.1 really needs changing. It
doesn't exclude future mechanisms, it simply recommends what to do
if there is no MTU discovery or management in place.

Regards
   Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 3 Mar 2010 09:56:55 -0800

Hi Brian,

> Sent: Tuesday, March 02, 2010 4:23 PM
> To: Templin, Fred L
> Cc: Rémi Després; [email-masked]

Still, I think an applicability statement with disclosure
of some of the limitations I cited would be appropriate.
It's not that I think that aspects of 6rd would not fit
into a generic solution (I do!) - it is rather that the
current manifestation of 6rd is not in itself a generic
solution.
 
> if there is no MTU discovery or management in place.

Focusing solely on the section 9.1 text for now:

   + MTU and fragmentation issues for IPv6 in IPv4 tunneling are discussed
   + in detail in section 3.2 of RFC4213 [RFC4213]. 6rd's scope is limited
   + to a service provider network.  IPv4 Path MTU discovery MAY be used
   + to adjust the MTU of the tunnel as described in section 3.2.2 of
   + RFC4213 [RFC4213] or the 6rd Tunnel MTU may be explicitly configured.

RFC4213 cites limitations of the IPv4 Path MTU discovery
method, which I assume is why the explicit configuration
option is listed and examined more closely in the following:
   
   + If the MTU is well-managed such that the IPv4 MTU on the CE WAN side
   + interface is set so that no fragmentation occurs within the boundary
   + of the SP, then the 6rd Tunnel MTU should be set to the known IPv4
   + MTU minus the size of the encapsulating IPv4 header (20 bytes).  For
   + example, if the IPv4 MTU is known to be 1500 bytes, the 6rd Tunnel
   + MTU may be set to 1480 bytes.  

But with SEAL, there is no need for a clamped MTU limit and
the tunnel MTU can be set to infinity.

   + Absent of more specific information
   + the 6rd Tunnel MTU SHOULD default to 1280 bytes.

Setting SEAL aside and considering only the existing
Section 9.1 text, no discussion of v6/v4 tunnel MTU is
complete without also examining the setting of the DF bit
in the IPv4 header. In particular, when the 6rd BR uses
an anycast address as the IPv4 source address, it truly
cannot allow fragmentation on the outer packet since that
could easily result in fragment misassociation within a
CE's reassembly buffer.

For example, if BRs A and B both use the same IPv4 source
address X and destination address Y, then happen to also
choose the same IP_ID, any fragments from A would be
indistinguishable from B's fragments, and CE Y could be
left with a dangerous reassembly misassociation that
evades upper layer integrity checks. So, the document
should say that BRs that use an anycast source address
MUST set DF=1.

Yes, this means that BR use of an anycast source address
would be incompatible with IPv4 networks that mis-manage
their MTUs and/or include links with MTUs smaller than 1280.
This is not a show-stopper at all, but simply another bullet
for inclusion in an applicability statement.  

   + A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined
   + automatically or configured directly, on the LAN side by setting the
   + MTU option in Router Advertisements [RFC4861] messages to the 6rd
   + Tunnel MTU.

Here, hosts connected to the LAN that receive the RA MTU
option will be stuck with a degenerate MTU (1480) even for
communications that do not leave the LAN. This means that
even if the LAN supports a 9KB MTU any hosts that receive
the MTU option will only ever be able to talk to each other
using 1480. The other issue is that the customer's network
may have an arbitrarily-complex topology of additional links
and routers. So, unless the 1480 is somehow propagated deeply
into the customer network, hosts connected to those links
will not observe the MTU clamping and continue to use
1500+ packets.

Fred

---

From: WashamFan <[email-masked]>
Date: Thu, 04 Mar 2010 10:13:12 +0800

Hi,

>  should say that BRs that use an anycast source address

Or we fix this problem by some measures. How about allocating
different IP_ID range for each BR? Although it might entail
modification on BRs.

>  will not observe the MTU clamping and continue to use

Why advertise WAN side MTU to LAN side? Doesn't routers
advertise LAN MTU via RA? PMTU would work out the optimal
MTU for end-to-end communication.

washam

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 04 Mar 2010 15:23:00 +1300

> MTU for end-to-end communication.

Because PMTUD is not reliable.

   Brian

---

From: Rémi Després <[email-masked]>
Date: Thu, 4 Mar 2010 14:25:46 +0100

> Because PMTUD is not reliable.

The dilemma seems to be:
- either one relies on ICMPv6 packet-too-big error messages being properly forwarded and processed,
- or, for IPv6 to work, hosts have to be clamped to 1280.

This is IMHO a good reason to insist, again and again, that ICMPv6 PTB messages MUST be always forwarded in intermediated nodes, and always processed in their destination hosts for PMTUD to become reliable.

RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Mar 2010 08:38:55 -0800

Hi Washam,

> From: WashamFan [mailto:[email-masked]]
> Sent: Wednesday, March 03, 2010 6:13 PM
> To: Templin, Fred L

Reducing the IP_ID range on each BR would increase
the incidence of reassembly errors at high data rates
[RFC4963]. So, I'm not sure new patch-type mechanisms
like this would be all that helpful without taking the
full step of going to something like SEAL.

AFAICT, that just means that, under the current 6rd
mechanisms, SP networks that want to have their BR's
use an anycast source address had better ensure that
the tunnels won't ever fragment. 6rd BRs that use
anycast can help by setting DF=1. 

> MTU for end-to-end communication.

PMTUD within end sites *has* to work in order for there
to be any measure of robustness. Customer end systems
and middleboxes that filter ICMPs are only hurting
themselves and need to change their ways.

That said, I believe the reason for asking the CE to
advertise 1480 internally to the customer site is to
minimize the incidence of invoking PMTUD. Otherwise,
customer end systems would be constantly sending 1500
packets initially which would be thrown away by the
CE until PMTUD kicks in. As I mentioned however, the
issue with advertising the 1480 is that it would cause
an MTU reduction even for communications that never
leave the end site. Customers with 9K links might
not like that.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Mar 2010 08:45:41 -0800

Remi,

> Sent: Thursday, March 04, 2010 5:26 AM
> To: Brian E Carpenter
> This is IMHO a good reason to insist, again and again, that ICMPv6 PTB messages MUST be always
> forwarded in intermediated nodes, and always processed in their destination hosts for PMTUD to become
> reliable.

I agree that customer end systems and middleboxes within
the customer network *must* be made capable of honoring
PMTUD; otherwise, there is no hope. The difficulty lies in
reliance on PMTUD for detecting MTU restrictions that occur
*outside* of the site. That is where Brian is correct in
pointing out that PMTUD is unreliable.

That said, it would be very useful to reduce the incidence
of invoking PMTUD within the end site if possible without
the side effect of clamping MTUs that don't need to be
clamped. That is an area in which SEAL can help.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Mar 2010 09:18:35 -0800

Apologies for coming late to this thread. One thing I
wanted to point out is that the ISATAP issues identified
by 'draft-nakibly-v6ops-tunnel-loops-01' pertain to the
ISATAP on-link prefix model. The vulnerability lies in an
ISATAP router's mistaken assumption that a correspondent
is an ISATAP host when in fact it is another router that
could incorrectly reflect the packet. So, in order to have
a fine-grained rather than coarse-grained mitigation, the
router that implements the mitigation may need to perform
an IPv6 prefix check as well as the embedded IPv4 address
check. (We believe this is only true for embedded IPv4
addresses taken from RFC1918 private space, however.)

Fred

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Gabi Nakibly
> Sent: Tuesday, March 02, 2010 6:37 AM
> To: Mark Townsley; [email-masked]
> >As presented in Hiroshima, I don't think an attack between a subscriber's own equipment and the BR
> is a top concern. The amount of traffic a single
> >subscriber can generate with a loop is no more than the subscriber could generate by downloading and
> uploading a lot of v6 content at the same time from
> >outside the 6rd domain. Further, with a tunnel loop, the download path would be rate-limited by the
> upload path, which is generally quite asymmetric (e.g.,
> >today a bit-torrent user within a decent swarm off-net could generate more traffic on the BR than a
> loop). Point is, there a are a lot of ways a user can fill
> I definitely see your point here: due to the clients' limited upload bandwidth a routing loop via
> internal relay may cause only small damage to the BR. If this is the case, why should the
> draft address the internal loop case at all? Guiding the SP to track all its internal relays may
> incur too much administrative overhead for such a small threat. On the other hand, one should note
> that the internal routing loop will probably clog up  the upstream of the client and completely DoS
> it. So this may be a big threat for the client itself.
> 
> >My concern here is that checking all possible local IPv4 addresses at a particular location in the
> IPv6 address for every packet switched through the BR
> >would result in very significant overhead for a hardware-optimized implementation operating at high
> speed. I have no objection to someone doing this kind
> - Source address check: the loop will succeed only if the IPv4 source address corresponds to the IPv6
> source address, so before forwarding a packet into the tunnel interface the BR must make sure that
> the IPv6 source address does not correspond to the IPv4 address that is intended to be the packet's
> IPv4 source address, only. The other IPv4 addresses of the BR are irrelevant here.
> - Destination address check: as I gather from Section 7.1.1 in the draft, currently only a single
> 6rdBRIPv4Address is supported by a CE for each BR. If this means that a BR advertises in the SP's IGP
> only one IPv4 address, then the BR is reachable to the clients only via this single address. Only
> this address should be verified in the destination address check. The loop will not work with any
> other IPv4 addresses, even if they belong to the BR.
> 
> I think you should refer to this draft only if you intend to reference one of the mitigation measures
> we propose there. If you only refer the attacks themselves the current reference is better.
> > First, I agree that the the best mitigation measure here for loops via external relays is to simply
> drop at the border of the 6rd domain packets sent from or destined to a 6rd BR.
> > After all, a customer can deploy an ISATAP router or even a 6to4 relay for that matter without the
> knowledge of the SP, while using a configured tunnel (by a tunnel broker) for its IPv6 connectivity.
> As presented in Hiroshima, I don't think an attack between a subscriber's own equipment and the BR is
> a top concern. The amount of traffic a single subscriber can generate with a loop is no more than the
> subscriber could generate by downloading and uploading a lot of v6 content at the same time from
> outside the 6rd domain. Further, with a tunnel loop, the download path would be rate-limited by the
> upload path, which is generally quite asymmetric (e.g., today a bit-torrent user within a decent
> swarm off-net could generate more traffic on the BR than a loop). Point is, there a are a lot of ways
> a user can fill his own pipe, and a loop between relays is probably way down low on the list of
> > I suggest to keep the filtering of internal 6rd BRs. But to avoid loops via internal ISATAP routers
> or 6to4 relays, I would suggest employing the destination/source address check described in Section
> 3.1 in http://tools.ietf.org/html/draft-nakibly-v6ops-tunnel-loops-01 we have recently submitted. It
> is not a perfect solution, and indeed it should be updated in case other types of automatic tunnels
> come up, but it does the job for ISATAP and 6to4 loops without demanding a knowledge of every
> internal ISATAP router or 6to4 relay.
> My concern here is that checking all possible local IPv4 addresses at a particular location in the
> IPv6 address for every packet switched through the BR would result in very significant overhead for a
> hardware-optimized implementation operating at high speed. I have no objection to someone doing this
> kind of check in their implementation, but their customers had better be willing to pay for it ;-)
>

---

From: Ole Troan <[email-masked]>
Date: Thu, 4 Mar 2010 21:16:07 +0100

Fred,

[...]

> leave the end site. Customers with 9K links might

I can remove the sentence recommending advertising the tunnel MTU on the LAN-side interface. or change it from a SHOULD to a MAY. as you say this is just to minimize the incidence of invoking PMTUD.

it should be rare that path MTU discovery doesn't work on the first hop.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 4 Mar 2010 13:03:09 -0800

Hi Ole,

> Sent: Thursday, March 04, 2010 12:16 PM
> To: Templin, Fred L
> Cc: WashamFan; [email-masked]
> I can remove the sentence recommending advertising the tunnel MTU on the LAN-side interface. or
> change it from a SHOULD to a MAY. as you say this is just to minimize the incidence of invoking
> PMTUD.

Removing the sentence might be best. The MAY would be OK,
but then you might need to also say something like: "(Note
that advertising the tunnel MTU on the LAN-side interface
would cause all hosts attached to the link to use a reduced
MTU even for link-local communications.)"  
 
> it should be rare that path MTU discovery doesn't work on the first hop.

If by first hop you mean exiting the customer network
and entering the tunnel interface, then I agree. However,
over-reliance on PMTUD has negative implications also,
including degraded efficiency. Worst case is for short
transactions (e.g., http over VPN) where up to 2x as
many packets as necessary would be used. Also, setting
a conservative fixed MTU on the tunnel interface would
make it very difficult to go back and set a larger
value later.

This, I think, is a key question for how we want to see
the IPv6 service deployed: if we want to go out with a
robust MTU service in the first iteration, then we have
the tools at hand to do so. Otherwise, we wait for the
SP network to turn up IPv6 on all of their routers.

Turning to SEAL again for the moment, SEAL is about more
than just PMTUD. SEAL provides a way for tunnel endpoints
to prevent source address spoofing even if the SP network
itself has insufficient mitigations. That might possibly
be even better than what we can do with native IPv6.

Fred

---

From: Rémi Després <[email-masked]>
Date: Fri, 5 Mar 2010 15:33:42 +0100

Fred, Ole, Brian,

Le 4 mars 2010 à 22:03, Templin, Fred L a écrit :

>> ...I can remove the sentence recommending advertising the tunnel MTU on the LAN-side interface. or

Fred, Ole, Brian,

Until PMTUD is made reliable in IPv6 (btw an important objective), all IPv6-enabled hosts should have their MTU set to 1280 to avoid IPv6 blackholes.
This could in principle apply only to IPv6 packets that leave customer sites, but as long as hosts have only one MTU parameter, it has to apply to all packets.
Since typical hosts have longer default MTUs than 1280, advertising MTU=1280 on links that offer paths to the global Internet appears to be the only available tool at hand.  

The proposal is then to replace:
"A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined automatically or configured directly, on the LAN side by setting the MTU option in Router Advertisements [RFC4861] messages to the 6rd Tunnel MTU."
by:
"As long as the path MTU discovery has not been made reliable, a 6rd CE SHOULD advertise on the LAN side, in the MTU option of Router Advertisements [RFC4861], an  MTU of 1280 octets . This is to prevent that longer packets that could have traversed the local 6rd domain may be discarded, because of their size, further in the Internet because of their size. Note that this may cause all hosts attached to the link to use a reduced MTU even for link-local communications, IPv6 and IPv4."

Regards,
RD

---

From: Matthew Ford <[email-masked]>
Date: Fri, 05 Mar 2010 16:05:08 +0000

Hi Rémi,

> "As long as the path MTU discovery has not been made reliable, a 6rd CE SHOULD advertise on the LAN side, in the MTU option of Router Advertisements [RFC4861], an  MTU of 1280 octets . This is to p…

I'm not sure I see how adopting this approach makes meeting your 
'important objective' any more likely.

   - Mat

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Mar 2010 08:52:10 -0800

Remi,

> From: Rémi Després [mailto:[email-masked]]
> Sent: Friday, March 05, 2010 6:34 AM
> To: Templin, Fred L; Ole Troan; Brian E Carpenter
> Cc: [email-masked]
> Until PMTUD is made reliable in IPv6 (btw an important objective), all IPv6-enabled hosts should have
> their MTU set to 1280 to avoid IPv6 blackholes.
> This could in principle apply only to IPv6 packets that leave customer sites, but as long as hosts
> have only one MTU parameter, it has to apply to all packets.
> Since typical hosts have longer default MTUs than 1280, advertising MTU=1280 on links that offer
> paths to the global Internet appears to be the only available tool at hand.
> 
> "A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined automatically or configured
> directly, on the LAN side by setting the MTU option in Router Advertisements [RFC4861] messages to
> the 6rd Tunnel MTU."
> by:
> "As long as the path MTU discovery has not been made reliable, a 6rd CE SHOULD advertise on the LAN
> side, in the MTU option of Router Advertisements [RFC4861], an  MTU of 1280 octets . This is to
> prevent that longer packets that could have traversed the local 6rd domain may be discarded, because
> of their size, further in the Internet because of their size. Note that this may cause all hosts
> attached to the link to use a reduced MTU even for link-local communications, IPv6 and IPv4."

The 1280 would be disappointing; I'm sure we can do better.

Thanks - Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 06 Mar 2010 09:37:59 +1300

> The 1280 would be disappointing; I'm sure we can do better.

As has been said before, in theory there is no difference between theory
and practice, but in practice, there is.

Right now today a number of ISPs are planning to deploy 6RD, and we have
already seen clear evidence from 6to4 deployment that setting the MTU to
1280 is a necessary evil at the moment. So I think this advice should be
in the specification, as a SHOULD. Otherwise we'd be misleading those
ISPs.

I don't like this any more than you do, from a long term perspective.

     Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 5 Mar 2010 15:33:27 -0800

Hi Brian,

> Sent: Friday, March 05, 2010 12:38 PM
> To: Templin, Fred L
> Cc: Rémi Després; Ole Troan; [email-masked]

You leave us with little hope.
 
> I don't like this any more than you do, from a long term perspective.

It still nags me that I feel we can do better; in fact, I
am sure we can...

Thanks - Fred

---

From: Timothy Baldwin <[email-masked]>
Date: Sun, 07 Mar 2010 00:33:14 +0000

Durand, Alain wrote:

> copied both SOFTWIRE and DHC mailing lists and solicit comments from both
> groups of experts. The draft is here:
> 

The end of section 4 states:
  "The prefix lifetimes
   advertised in Router Advertisements or used by DHCP on the CE LAN
   side MUST be equal to or shorter than the IPv4 address lease time."

PPP doesn't have the concept of a lease time, which makes that MUST 
impossible. Instead the advertised prefixes must be revoked if the IPv4 
address changes, and if the prefix has not be written to persistent storage 
the advertised must be short (about a minute) to prevent prolong mis-
configuration of the hosts in the event of the IPv4 address changing after a 
reboot of the router.

In the case of DHCPv4, is it intended to prohibit assigning a new address to 
CE router before its address lease expires, even if the router reboots to 
the DHCPv4 INIT state?

---

From: Ole Troan <[email-masked]>
Date: Sun, 7 Mar 2010 08:27:56 +0100

Timothy,

> CE router before its address lease expires, even if the router reboots to 

the intention is to achieve some resemblance of a renumbering event. difficult when you typically have zero overlap between the old and the new prefix.

cheers,
Ole

---

From: Ole Troan <[email-masked]>
Date: Mon, 8 Mar 2010 14:33:04 +0100

all,

thank you very much for good and constructive comments!
I've tried to summarize the comments received and the changes made below.

cheers,
Ole

** Include option Parameter Request List option for the DHCPv4 option:
   New text:
	 <t>The CE MUST include a Parameter Request List Option <xref
	 target="RFC2132"></xref> for the OPTION_6RD. Because the
	 OPTION_6RD contains one IPv4MaskLen/6rdPrefixLen/6rdPrefix
	 block, and because DHCP cannot convey more than one instance
	 of an option, OPTION_6RD is limited to provision at most a
	 single 6rd domain.  Provisioning of a CE router connected to
	 multiple 6rd domains is outside the scope of this protocol
	 specification.</t>

   => Accepted

** Loop avoidance
   Suggestion was to adopt solution in
   draft-nakibly-v6ops-tunnel-loops. Issues with scalability and
   checking of "well-known" bits in ISATAP addresses.

   => Keep existing text.  

** MTU
  There are well documented issues with tunnels and MTU. There are
  issues with anycast and fragmentation. The draft references the
  existing mechanisms for tunnelling and MTU, and recommend following
  existing practice. Since the mechanism is contained within a single
  SP network, one can assume that the MTU is well managed. Just as in
  an MPLS network. 

  => Keep existing text. Should the working group start a separate
  effort for a more general solution to MTU issues?
   
** Use of anycast
   There are issues with the use of anycast. The draft
   supports the use of unicast BR addresses too. The issues around
   ECMP, fragmentation and ICMP messages ending up on the wrong BR are
   to some extent operational. It depends on the network having a well
   maintained MTU as recommended in the draft. ECMP should be done on
   a per flow basis not on a per packet.

   => Keep existing text.

** Analysis of domain of applicability
   The draft already sets the context of the solution to within an SP
   network. More detail and operational scenarios might be more
   suitable to a 6rd operational document.

   => No action.
  
** IPv4 lease time in case of PPP
   What lease-time should be used on the LAN-side in IPv6 RAs when the
   IPv4 WAN side address is aquired via IPCP?

  Proposed new text:

      <t>6rd IPv6 address assignment and hence the IPv6 service itself
      is tied to the IPv4 address lease, thus the 6rd service is also
      tied to this in terms of authorization, accounting, etc. For
      example, the 6rd delegated prefix has the same lifetime as its
      associated IPv4 address. The prefix lifetimes advertised in
      Router Advertisements or used by DHCP on the CE LAN side MUST be
      equal to or shorter than the IPv4 address lease time. If the
      IPv4 lease time is not known, the lifetime of the 6rd delegated
      prefix SHOULD follow the defaults specified in <xref
      target="RFC4861"></xref>.</t>

   => Accepted.

---

From: Rémi Després <[email-masked]>
Date: Mon, 8 Mar 2010 15:01:14 +0100

Hi Ole,

Thanks for this useful summary.
The point below, discussed on Friday is one more for your list.

Cheers,
RD

> 
> De : Brian E Carpenter <[email-masked]>
> Date : 5 mars 2010 21:37:59 HNEC
>

---

From: Mark Townsley <[email-masked]>
Date: Mon, 08 Mar 2010 15:51:29 +0100

> in the specification, as a SHOULD. Otherwise we'd be misleading those
I don't think that the evidence from 6to4 is directly applicable to 6rd 
as the administrative domain of deployment is quite different between 
the two. Absent working automatic mechanisms, MTU is very much a 
administratively controlled item today. Since a given 6rd deployment is 
limited to one (or perhaps a few) administrative domain(s), the MTU 
problem is generally more tractable than with 6to4 which typically has 
no readily assignable responsible party to address a problem when it occurs.

We do have evidence that 6rd works fine with a 1480 MTU over an 
IPv4-enabled link with MTU of 1500. Some bugs, not necessarily 
6rd-specific, have been uncovered in the process, and fixed. That's a 
good thing. Let's not throw in the towel until the game is truly over.

- Mark

---

From: Rémi Després <[email-masked]>
Date: Mon, 8 Mar 2010 16:37:26 +0100

Hi Mark,

Further comments below.

> I don't think that the evidence from 6to4 is directly applicable to 6rd as the administrative domain of deployment is quite different between the two. Absent working automatic mechanisms, MTU is very much a administratively controlled item today. Since a given 6rd deployment is limited to one (or perhaps a few) administrative domain(s), the MTU problem is generally more tractable than with 6to4 which typically has no readily assignable responsible party to address a problem when it occurs.
> 
> We do have evidence that 6rd works fine with a 1480 MTU over an IPv4-enabled link with MTU of 1500.

Hi Mark,

I certainly don't contest this where 6rd is deployed today (being as you well know a happy user of Free's implementation ;-) ).

But the point is that if one of my outgoing packets leaves Free's network and reaches another network where the MTU is 1280, which is permitted, it will be discarded.
If PMTUD cannot be relied on, this can create a blackhole.  

(Clearly paths to Google and to the IETF have MTUs at least 1480, which is fine, but the recommended solution must work for all IPv6 servers. Note that, as long as PMTU is considered unreliable, both Google and the IETF should also  better send IPv6 packets with an assumed 1280 path MTU.  

> Some bugs, not necessarily 6rd-specific, have been uncovered in the process, and fixed. That's a good thing.

> Let's not throw in the towel until the game is truly over.

Be sure I will not throw the towel and continue to fight for IPv6 to be reliably usable (first), and with more and more efficiency (next). 

Cheers,
RD

---

From: Ole Troan <[email-masked]>
Date: Mon, 8 Mar 2010 16:43:02 +0100

Remi,

> Be sure I will not throw the towel and continue to fight for IPv6 to be reliably usable (first), and with more and more efficiency (next). 

but you are making an argument that _every_ IPv6 link should be a maximum of 1280 bytes. I don't think that is something 6rd should prescribe.

cheers,
Ole

---

From: <[email-masked]>
Date: Mon, 8 Mar 2010 17:03:01 +0100

Mark

>- Mark

I have to support you. The MTU is not arbitrary within a service provider network, but well engineered. 6RD is not targeted for the open Internet, but is really intended for SP usage. Hence, an MTU of 1480 is the way to go; in fact, the MTU can even be higher provided that the SP makes sure that the network elements between CPE and Border Relays all support a higher MTU. That way it is possible to support an MTU of 1500 Bytes for IPv6 using 6RD!

I also believe that this discussion should be placed into a separate operational document.

Regards,
Martin

---

From: Rémi Després <[email-masked]>
Date: Mon, 8 Mar 2010 17:05:35 +0100

> but you are making an argument that _every_ IPv6 link should be a maximum of 1280 bytes. I don't think that is something 6rd should prescribe.

Only every IPv6-enabled link where it is not known, by some exterior means, that larger MTUs will work for all possible e2e paths.

In my understanding, we MUST prefer a 100% reliable connectivity to a 15.6% gain in MTU size (1480 instead of 1280, working in typical cases but not in all).

Now, if one could prove that there is no more blackhole possibility with 1480 than with 1280, I would be glad to withdraw my proposed amendment.

But the available proof is rather that blackholes are possible with 1480, and not with 1280. (I have not been aware of this for long, but being now well aware, I believe there is no real choice.)

Like others, I am not enthusiastic about this, and will work improve the future, but let's be consistent in the present, and make IPv6 reliable.    

Cheers,
RD

---

From: Alexandre Cassen <[email-masked]>
Date: Mon, 08 Mar 2010 17:09:52 +0100

On Mon, 2010-03-08 at 17:03 +0100, [email-masked] wrote:
> Mark
> I have to support you. The MTU is not arbitrary within a service
> provider network, but well engineered. 6RD is not targeted for the
> open Internet, but is really intended for SP usage. Hence, an MTU
> of 1480 is the way to go;

+1

AGREED !

--
Alexandre

---

From: Satoru Matsushima <[email-masked]>
Date: Tue, 9 Mar 2010 01:25:20 +0900

How about "IPv6 over IPv4 over PPP over L2TP over IPv4 over Ethernet".
It is majority of FTTH broadband environment in Japan.

Even though the IPv6 MTU with 6rd is stable, but it does not mean 1480byte MTU can be kept.

--satoru

On 2010/03/08, at 23:51, Mark Townsley wrote:

> We do have evidence that 6rd works fine with a 1480 MTU over an IPv4-enabled link with MTU of 1500. Some bugs, not necessarily 6rd-specific, have been uncovered in the process, and fixed. That's a good thing. Let's not throw in the towel until the game is truly over.
> 
> - Mark
> 
>

---

From: Rémi Després <[email-masked]>
Date: Mon, 8 Mar 2010 17:31:20 +0100

Martin, Mark, Ole,

You may be satisfied with the conclusion at the end.

Regards,
RD
 
Le 8 mars 2010 à 17:03, <[email-masked]> a écrit :

> I have to support you. The MTU is not arbitrary within a service provider network, but well engineered.

Agreed, and that's how it has been done. 
(BTW, congratulation on the 6rd deployment at Swisscom.)

> 6RD is not targeted for the open Internet, but is really intended for SP usage.

Yes, it is internal, but if it has IPv6 traffic that goes to the global Internet, then it needs a guaranteed path MTU.
(In IPv4, packets may be fragmented in the network if sent with a size beyond the path MTU, but not in IPv6.)

> Hence, an MTU of 1480 is the way to go; in fact, the MTU can even be higher provided that the SP makes sure that the network elements between CPE and Border Relays all support a higher MTU. That way it is possible to support an MTU of 1500 Bytes for IPv6 using 6RD!
> 

Note that it is only the sentence about the MTU option in RAs which is a problem, namely:
"A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined automatically or configured directly, on the LAN side by setting the MTU option in Router Advertisements [RFC4861] messages to the 6rd Tunnel MTU."

Conclusion: If this sentence is deleted, rather than improved with words like those I proposed, the inconsistency is at least avoided.
I would regret the loss of information, but would no longer have a strongly founded objection.

---

From: Ole Troan <[email-masked]>
Date: Mon, 8 Mar 2010 17:33:04 +0100

Satoru,

> Even though the IPv6 MTU with 6rd is stable, but it does not mean 1480byte MTU can be kept.

the draft recommends the WAN-side IPv4 MTU - 20. which covers the above too.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 09:53:45 -0800

Ole,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ole Troan
> Sent: Monday, March 08, 2010 5:33 AM
> To: Durand, Alain
> Cc: softwires; DHC WG; David Ward
> Subject: [dhcwg] 6rd LC - summary of comments
> 
> all,
> 

If 6rd used SEAL, the CE would be able to set a much
larger tunnel MTU (say, 64KB). Then, end systems in
customer sites would be able to automatically discover
and utilize larger MTUs as long as the SP network links
can support them. (And no; I am *not* requiring the CE
or BR devices to support segmentation and reassembly
unless they want to.)

But, SEAL is about more than just MTU - it also includes
mechanism to detect and defeat IPv4 source address spoofing.
Since the 6rd checks are useless if IPv4 source address
spoofing is possible, it is necessary to consider the
domain of applicability to determine whether anti-
spoofing mitigations are needed. SEAL provides such
a mitigation.

>    a per flow basis not on a per packet.

I have shown that there is a possibility for data corruption
for BR use of anycast source address - to the point that the
BRs MUST ensure that no fragmentation will occur when anycast
source is used. That means that the BR MUST set DF=1 if it
uses an anycast source address. The document needs to say
that somewhere.

>    suitable to a 6rd operational document.

I disagree; there are operational limitations that would
limit applicability to only tightly-managed SP network
scenarios as I have pointed out in my messages. The
document therefore requires an applicability statement,
as has been required for other "transition" mechanisms.

Fred

> dhcwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dhcwg

---

From: Ole Troan <[email-masked]>
Date: Mon, 8 Mar 2010 20:07:33 +0100

Fred,

[...]

> or BR devices to support segmentation and reassembly

not quite sure what the purpose of a 64KB MTU on the tunnel link would be, but sure a 1500 byte MTU would have been nice. I'm unsure of the consequences of reassembly/fragmentation on the tunnel end-points. so far experience has shown that to be 'harmful'. would SEAL be any different?
6rd is a patch to 6to4. one goal of 6rd was to be able to ride on an existing 6to4 implementation with only small changes. SEAL would not be that. 

I'm fine with working on more general solutions to the MTU problem which may then apply to any tunnel. I would not want to accept SEAL as that solution without much more analysis.

> spoofing mitigations are needed. SEAL provides such

I missed that part of SEAL. pointer?

> uses an anycast source address. The document needs to say

it follows the rules and setting of the DF bit as specified in RFC4213.

> as has been required for other "transition" mechanisms.

in the introduction we have:

   6rd specifies a protocol mechanism to deploy IPv6 to sites via a
   Service Provider's (SP's) IPv4 network.  It builds on 6to4 [
   RFC3056],  with the key differentiator that it utilizes an SP's own IPv6 address
   prefix rather than a well known prefix (2002::/16).  By using the
   SP's IPv6 prefix, the operational domain of 6rd is limited to the SP
   network and under its direct control.  From the perspective of
   customer sites and the IPv6 Internet at large, the IPv6 service
   provided is equivalent to native IPv6.

I'm not sure what more needs to be said. at least not in the basic protocol spec. (which I originally wanted to be a single page saying: this is 6to4 with a SP prefix instead of 2002::/16). :-)

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 12:00:19 -0800

Ole,

> Sent: Monday, March 08, 2010 11:08 AM
> To: Templin, Fred L
> Cc: Durand, Alain; softwires; David Ward; DHC WG
> Subject: Re: [dhcwg] 6rd LC - summary of comments
> 
> Fred,
> 
> [...]
> 
> >> ** MTU
> not quite sure what the purpose of a 64KB MTU on the tunnel link would be, but sure a 1500 byte MTU
> would have been nice.

The 64KB MTU is to admit packets into the tunnel without any
pre-conceived notion of an initial size restriction. Any size
restrictions will then be discovered dynamically, with a
reduced size conveyed back to the original source if necessary.

> I'm unsure of the consequences of reassembly/fragmentation on the tunnel end-
> points. so far experience has shown that to be 'harmful'. would SEAL be any different?

With SEAL, the tunnel egress can choose to reassemble as much
or as little as it likes, with the limiting case being that the
egress performs no reassembly at all. In that case, SEAL still
provides value since the tunnel will still be able to support
larger MTUs (e.g., 9KB) as long as all links in the SP network
between ingress->egress support an MTU at least that large.

In the case that the egress decides to support some reassembly
(e.g., up to 1500 bytes), SEAL avoids the issues incurred for
IP fragmentation and reassembly since it uses a (much) larger
ID to unambiguously associate fragments of the same packet.
SEAL also has a way of gracefully "backing down" if reassembly
becomes problematic due to excessive congestion so that "loss
unit smaller than retransmission unit" would be a short-lived
transient condition if it ever occurs in operational practice
(which I am told should be rare).

> 6rd is a patch to 6to4. one goal of 6rd was to be able to ride on an existing 6to4 implementation
> with only small changes.

It is still my view that 6rd "closes the gap" between 6to4
and ISATAP. Indeed, all three are implemented within the
same *.c module in the linux kernel.

> SEAL would not be that.

I have working code for RFC5320 with full segmentation and
reassembly, but tunnel endpoints that do not wish to support
reassembly can omit ~90% of that.
 
> I'm fine with working on more general solutions to the MTU problem which may then apply to any
> tunnel. I would not want to accept SEAL as that solution without much more analysis.

Then, let's get on with the analysis!

> I missed that part of SEAL. pointer?

(draft-templin-intarea-seal), Section 4.6.

> it follows the rules and setting of the DF bit as specified in RFC4213.

RFC4213 does not examine the implications of setting the DF
bit in the presence of anycast source address; 6rd must.
 
> I'm not sure what more needs to be said. at least not in the basic protocol spec. (which I originally
> wanted to be a single page saying: this is 6to4 with a SP prefix instead of 2002::/16). :-)

>From a few mails back, here are the items which I believe bear
specific mention in an applicability statement:

  "During the NGTRANS days, the ISATAP team was strongly required
  to include a "Domain of Applicability" statement, and I think
  the same should be required of 6rd. I see a number of aspects
  of 6rd that IMHO would limit its applicability, including:
  
    generation and ICMP filtering"

Thanks - Fred

---

From: Mark Townsley <[email-masked]>
Date: Mon, 08 Mar 2010 21:46:22 +0100

On 3/8/10 9:00 PM, Templin, Fred L wrote
>>> spoofing is possible,
6rd aims to be as good as the IPv4 deployment it rides upon in terms of 
anti-spoofing, no better.
>>> it is necessary to consider the
Source address anti-spoofing measures (for v4 and v6) are within the 
scope of the SAVI WG.

- Mark

---

From: Ole Troan <[email-masked]>
Date: Mon, 8 Mar 2010 21:56:41 +0100

Fred,

[...]

> same *.c module in the linux kernel.

OK, so we don't quite see that the same then. I see ISATAP and RFC2529 as siblings. and 6to4 and 6rd together. all with automatic tunnels from rfc1933 as their common ancestor. anyway, not particularly important.

> Then, let's get on with the analysis!

sure, but let's do that on a separate thread. I do not want to make 6rd dependent on SEAL at this point.

> (draft-templin-intarea-seal), Section 4.6.

OK, still didn't get it.

> bit in the presence of anycast source address; 6rd must.

OK, what about: 
"The use of an anycast source address may lead to any ICMP error message generated on the path being sent to a different BR. Therefore using dynamic tunnel MTU [section 3.2.2, RFC4213] is subject to IPv4 Path MTU blackholes."

>    generation and ICMP filtering"

as in:
http://tools.ietf.org/html/rfc5214#section-4

we have covered that reasonably well already. I think some of those issues could well be covered in a deployment document though.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 13:20:18 -0800

Mark,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Townsley
> Sent: Monday, March 08, 2010 12:46 PM
> To: [email-masked]
> Subject: Re: [Softwires] [dhcwg] 6rd LC - summary of comments
> 

I didn't know about that, but I'm not sure I see your
point. What I have done in SEAL doesn't bear any
resemblance to what I saw over in SAVI, yet it solves
a real problem for tunneling in environments where
source address spoofing is possible. If you want to
broaden the domain of applicability for 6rd, then I
think you should take a look at SEAL. On the other
hand, if you think 6rd should only be deployed in
environments where IPv4 source address spoofing is
not possible, then there should be an applicability
statement saying so.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 8 Mar 2010 13:39:00 -0800

Ole,

> Sent: Monday, March 08, 2010 12:57 PM
> To: Templin, Fred L
> OK, so we don't quite see that the same then. I see ISATAP and RFC2529 as siblings. and 6to4 and 6rd
> together. all with automatic tunnels from rfc1933 as their common ancestor. anyway, not particularly
> important.

The more accurate comparison is between 6rd and VET
rather than 6rd and ISATAP. Both 6rd and VET work within
a closed domain (which VET chooses to call an "enterprise"
or "site", but for which an ISP SP network is simply one
use case example) and both 6rd and VET deal with discovery
and utilization of exit routers that can reach networks
outside of the SP network domain. 6rd CE routers fit the
same role as VET EBRs, and 6rd BRs fit the same role as
VET EBGs. In many ways, 6rd is just VET with a number of
shortcuts taken (e.g., anycast, stateless PD etc.).

> sure, but let's do that on a separate thread. I do not want to make 6rd dependent on SEAL at this

This gets back again to the crux question. If 6rd can go
forward with robust MTU handling in the first iteration
it would greatly reduce the pressure for ISPs to upgrade
their entire infrastructure to native IPv6 - perhaps even
indefinitely. So, why not get it right the first time?

> > (draft-templin-intarea-seal), Section 4.6.

Can you be more specific? Did you not understand what
was written, or do you think there is something wrong
with it?

> "The use of an anycast source address may lead to any ICMP error message generated on the path being
> sent to a different BR. Therefore using dynamic tunnel MTU [section 3.2.2, RFC4213] is subject to
> IPv4 Path MTU blackholes."

I have told you several times what is wrong with anycast
source and DF=0. If you want to remain silent and allow
open a potential for undetected data corruption, that
seems like a serious omission.

> deployment document though.

I gave you a valid list of limitations above. These speak
to the need for truth in advertising in terms of where 6rd
is suited for operation and where it is not.

Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Mar 2010 11:06:49 +1300

Ole,

I agree with this as a summary of the common ground in the
discussions, even if a few people may disagree here and there.

>   effort for a more general solution to MTU issues?

Agreed to the existing text. Also, I'm not sure that the general
discussion is specific to SOFTWIRE. Maybe it should be raised
in INTAREA?

Regards
   Brian Carpenter

---

From: Rémi Després <[email-masked]>
Date: Mon, 8 Mar 2010 23:25:37 +0100

> between ingress->egress support an MTU at least that large.

The discussion on SEAL, despite some interesting technical issues, does not belong to this last call discussion:
- 6rd is an transition technique which, to be as useful as it can be for rapid IPv6 deployments, should be stabilized asap. 
- The DHCP option, which is specified in this document and nowhere else is urgent.
- Combining SEAL with 6rd is a disjoint subject. 

>  - potential black holes when ICMPs can't be translated

- There is no potential blackhole caused by 6rd per se.
- Potential blackholes of IPv6 packets as long as ICMPv6 error messages and PMTUD are not treated properly belong to IPv6 in general, not to 6rd. 
- Such blackholes can be completely avoided in the short term, despite buggy ICMPv6 and/or PMTUD treatments, by sending IPv6 packets that don't exceed 1280 octets, the guaranteed IPv6 path MTU of RFC 2460.
- Provided *the last sentence of the section 9.1 on MTUs is deleted* (as already proposed in my previous e-mail), the document is therefore sound in this respect.  

IMHO, with modifications Ole has listed, and with the sentence deletion above, the draft can be accepted.

Regards,
RD

---

From: Ole Troan <[email-masked]>
Date: Wed, 10 Mar 2010 12:12:02 +0100

Fred,

[...]

> indefinitely. So, why not get it right the first time?

if there was consensus on what's right...
SEAL appears to be a solution described in section 3.1, RFC4459.
the 6rd draft is leaning towards the solution described in section 3.3 of RFC4459.

> was written, or do you think there is something wrong

OK, I see, so you pass a nonce around to use for verification against source spoofing. that requires state.

[...]

> seems like a serious omission.

sorry, yes I agree we should point that out.
what about:

  "The use of an anycast source address may lead to any ICMP error message generated on the path being sent to a different BR. Therefore using dynamic tunnel MTU [section 3.2.2, RFC4213] is subject to IPv4 Path MTU blackholes.

Multiple BRs using the same anycast source address may send fragmented packets to the same IPv6 CE at the same time. If the fragmented packets from different BRs happen to use the same fragment ID, incorrect reassembly may occur. For this reason a BR using an anycast source address MUST set the IPv4 Don't Fragment flag."

[...]

cheers,
Ole

---

From: Rémi Després <[email-masked]>
Date: Wed, 10 Mar 2010 14:01:53 +0100

Le 10 mars 2010 à 12:12, Ole Troan a écrit :
...
> Multiple BRs using the same anycast source address may send fragmented packets to the same IPv6 CE at the same time. If the fragmented packets from different BRs happen to use the same fragment ID,…

Note that in "well managed" 6rd domains (ref. sec 9.1), there is no fragmentation between BRs and CEs.
The DF bit has therefore no effect.
I don't see the need for this addition.

RD

---

From: Mark Smith <[email-masked]>
Date: Thu, 11 Mar 2010 07:41:58 +1030

On Mon, 8 Mar 2010 17:05:35 +0100

> In my understanding, we MUST prefer a 100% reliable connectivity to a 15.6% gain in MTU size (1480 instead of 1280, working in typical cases but not in all).

100% reliable connectivity on the Internet doesn't exist, regardless of
whether the protocol is IPv4 or IPv6, or MTUs are large or small. Hosts
need to be able to cope with packet loss, whether it be TCP acks, PMTUD
feedback etc.

PMTUD has it's issues, which is why methods defined in 

http://tools.ietf.org/rfc/rfc4821.txt

exist, that don't rely on routers sending back next link MTU
information.

The Linux kernel has had an implementation of it for at least two years.

> Now, if one could prove that there is no more blackhole possibility with 1480 than with 1280, I would be glad to withdraw my proposed amendment.

Do you have evidence to support your view? Sure it's possible, but from
my experience of running a 6to4 tunnel for quite a number of years with
a local/initial MTU of 1472 bytes for years (1492 being because of
PPPoE's 1492 MTU) without any issues that have ended up being because
of broken PMTUD. 

> But the available proof is rather that blackholes are possible with 1480, and not with 1280. (I have not been aware of this for long, but being now well aware, I believe there is no real choice.)

Your right about 1280. However, following that logic, the IPv4 Internet
should have an MTU of no greater than 576 bytes, because blackholes are
possible with MTU's larger than that. Yet the common MTU on the IPv4
Internet is 1500. Sure there are occasional work arounds like MSS hacks
having to be put in place, however they're a reasonable trade off
verses having packet per second forwarding rates go up by around 3
times.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 11 Mar 2010 10:24:02 +1300

Mark,

> Internet is 1500.

That's true today. But my experience while travelling and staying in
random hotels towards the end of the dial-up era was that I often
had to clamp the IPv4 MTU at (say) 512 to make things actually work.
I've had experience with 6to4 of abject PMTUD failures with remote servers
trying to do 1480. So I think that Remy is correct as far as a fail safe
solution *today* goes. Since 6RD is very much a transitional technique
before an ISP is ready to run native, I don't see this as a significant
inefficiency.

     Brian

Regards
   Brian Carpenter

---

From: Rémi Després <[email-masked]>
Date: Thu, 11 Mar 2010 11:08:18 +0100

Le 10 mars 2010 à 22:11, Mark Smith a écrit :
...
> need to be able to cope with packet loss, whether it be TCP acks, PMTUD

It depends on what is meant by "connectivity".
I took it as meaning that if all packets between two hosts have x% chances to reach their destination, with x significantly > 0, there is connectivity between them (the opposite of blackhole where some packets have 0% chances to go through).

But I don't know any official definition saying this.
Your interpretation may therefore be as legitimate as this one. 

> Your right about 1280.

> However, following that logic, the IPv4 Internet
> possible with MTU's larger than that.

In IPv4:
- Packets can be fragmented in the network.
- The minimum PMTU is 68, i.e. the maximum IPv4 header length plus the minimum IPv4 fragment length (RFC 1858, RFC 2132)
- All packets having its DF bit = 0, have no risk to be discarded because of their size. 
- In particular, a 1500B packet can traverse paths whose PMTUs are as small as 576 (or even as small as 68).

In IPv6:
- Packets cannot be fragmented in the network
- A minimum PMTU is standardized, i.e. 1280B.
- 1500B packets cannot traverse any IPv6 path having as PMTU 1480 (or 1280 or any value < 1500).

This justifies a different behavior.

Regards,
RD

---

From: Mark Smith <[email-masked]>
Date: Fri, 12 Mar 2010 00:25:00 +1030

On Thu, 11 Mar 2010 11:08:18 +0100

> - In particular, a 1500B packet can traverse paths whose PMTUs are as small as 576 (or even as small as 68).

It's late here, so I certainly won't claim an authoritative big
picture view. However, I like the following quote, particularly the
"biggest smallest" part :

"I forget the exact terminology, but Brian is talking about the biggest
packet one can send and be guaranteed it will not be fragmented. The is
576 bytes for IPv4 and 1280 bytes for IPv6.

I think of it as the biggest smallest packet one can send :-)"

http://www.ops.ietf.org/lists/v6ops/v6ops.2007/msg00835.html

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 15:04:59 -0800

Hi Ole,

Catching up on the recent exchanges:

> Sent: Wednesday, March 10, 2010 3:12 AM
> To: Templin, Fred L

Good question.

> SEAL appears to be a solution described in section 3.1, RFC4459.

No, that's not right; RFC4459, Section 3.1 talks about
*outer* fragmentation and reassembly, but SEAL explicitly
seeks to tune out any outer fragmentation whenever it occurs.
SEAL is also not an example of an inner fragmentation scheme
referred to in RFC4459, Section 3.4.

Instead, SEAL is a mid-layer segmentation and reassembly
approach described in RFC2764, Section 3.1.7. RFC4459 missed
this approach in its analysis.

> the 6rd draft is leaning towards the solution described in section 3.3 of RFC4459.

That section includes text that shows a clear need for an
applicability statement. For example:

  "This approach is highly assumptive of the deployment scenario."
 
> OK, I see, so you pass a nonce around to use for verification against source spoofing. that requires

That is true. Tunnel endpoints that wish to synchronize
sequence numbers must maintain state, e.g., in neighbor
cache entries. Tunnel endpoints that do not wish to
synchronize sequence numbers can omit this state, but
will not realize the benefits of avoiding off-path
spoofing attacks.

>   "The use of an anycast source address may lead to any ICMP error message generated on the path
> being sent to a different BR. Therefore using dynamic tunnel MTU [section 3.2.2, RFC4213] is subject
> to IPv4 Path MTU blackholes.
> 
> Multiple BRs using the same anycast source address may send fragmented packets to the same IPv6 CE at
> the same time. If the fragmented packets from different BRs happen to use the same fragment ID,
> incorrect reassembly may occur. For this reason a BR using an anycast source address MUST set the
> IPv4 Don't Fragment flag."

That pretty much matches up with what I asked for, so I
don't have a problem with it per se. Still, I feel we can
do so much better of a service to end users if we can get
them a solid 1500 as well as the ability to discover and
utilize larger MTUs without any assumptions of what links
might be in the ISP network path.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 15:08:54 -0800

Remi,

> Sent: Wednesday, March 10, 2010 5:02 AM
> To: Ole Troan
> Cc: Templin, Fred L; Durand, Alain; softwires; DHC WG
> > Multiple BRs using the same anycast source address may send fragmented packets to the same IPv6 CE
> at the same time. If the fragmented packets from different BRs happen to use the same fragment ID,

Still, DF=0 is an open invitation to the network to
fragment the packet if it decides to do so, and there
is no recourse to say back to the network "you should
not have fragmented that". DF=1 gives the network no
leg to stand on whereby it could legitimately fragment
the packet.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 15:21:23 -0800

Hi Mark,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Smith
> Sent: Wednesday, March 10, 2010 1:12 PM
> To: Rémi Després
> Cc: [email-masked]
> > > but you are making an argument that _every_ IPv6 link should be a maximum of 1280 bytes. I don't
> think that is something 6rd should prescribe.
> >
> > In my understanding, we MUST prefer a 100% reliable connectivity to a 15.6% gain in MTU size (1480
> instead of 1280, working in typical cases but not in all).
> >
> 

The RFC4821 is indeed a potential tool for end systems to use
that could make life much better in the transitional period of
moving to larger MTUs in the Internet, and I highly support it.
However, the tunnel endpoints still need to do their part to
provide solid underpinnings over which the RFC4821 signaling
riders, i.e., end system use of RFC4821 does not release the
tunnel endpoints of their obligation to maintain a solid MTU.

> > Now, if one could prove that there is no more blackhole possibility with 1480 than with 1280, I
> would be glad to withdraw my proposed amendment.
> >
> 
> > But the available proof is rather that blackholes are possible with 1480, and not with 1280. (I
> have not been aware of this for long, but being now well aware, I believe there is no real choice.)
> >
> 

I'd like to touch on this MSS clamping question for a moment.
I think we can expect to see more and more non-TCP traffic,
and this seems to be substantiated by at least one recent
CAIDA study (reference needed). I think we may also be
forgetting that VPNs are becoming more and more prevalent
where there may be no visibility into the MSS option nor
ability to clamp it. Hence, I do not see MSS clamping as
being a viable recommended practice for all environments.

Thanks - Fred

> > Like others, I am not enthusiastic about this, and will work improve the future, but let's be
> consistent in the present, and make IPv6 reliable.
> >
> > Cheers,
> > RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 15:30:24 -0800

Brian,

> Since 6RD is very much a transitional technique
> before an ISP is ready to run native,

But, why limit the thinking with 6rd to only viewing
it as a "transition mechanism"? Transition mechanism
innovation was deemed complete in 2002 with the closing
of NGTRANS, the emergence of v6ops, and the declaration
of IPv6 as "operational". Yet, we still have IPv4 in
widespread deployment and use and no sign of that use
abating.

6rd already has us on a trajectory to deploy a *pretty
good* IPv6 service in coexistence with IPv4. So, why not
take the small additional step to make it *great*?

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 15:33:02 -0800

Remi,

> Sent: Thursday, March 11, 2010 2:08 AM
> To: Mark Smith
> Cc: [email-masked]
> I took it as meaning that if all packets between two hosts have x% chances to reach their
> destination, with x significantly > 0, there is connectivity between them (the opposite of blackhole
> where some packets have 0% chances to go through).
> 

All truly spoken.

Fred

---

From: Mark Townsley <[email-masked]>
Date: Fri, 12 Mar 2010 00:35:39 +0100

On 3/10/10 10:24 PM, Brian E Carpenter wrote:
> solution *today* goes.
And thank goodness we still don't fragment all IPv4 traffic >= 512 bytes.
> before an ISP is ready to run native, I don't see this as a significant
The problem in this logic is that the argument for 1280 is not about the 
MTU within the confines of the 6rd deployment, but outside across the 
Internet. As such, it would apply equally to native IPv6 as it would to 
6rd.

- Mark

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 11 Mar 2010 15:37:44 -0800

Mark,

> Sent: Thursday, March 11, 2010 5:55 AM
> To: Rémi Després; [email-masked]

I wish it were true that we could unambiguously count on
576 everywhere, but 576 is only a recommendation (e.g.,
see RFC3819) whereas 68 is the *law* as accurately cited
by Remi.

Thanks - Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Mar 2010 14:46:21 +1300

> Internet. As such, it would apply equally to native IPv6 as it would to

My experience was that while I was using 6to4 I experienced MTU and MSS
related problems, and they went away as soon as the university gave me
native connectivity. I can't explain that rationally, of course.

    Brian

---

From: Rémi Després <[email-masked]>
Date: Fri, 12 Mar 2010 11:38:28 +0100

...
> The problem in this logic is that the argument for 1280 is not about the MTU within the confines of the 6rd deployment, but outside across the Internet. As such, it would apply equally to native IPv6 as it would to 6rd.

Yes, it DOES apply also to directly native IPv6.
That's why it makes sense, in a 6rd RFC, to avoid dealing with any recommendation to advertise on a LAN link another MTU than that of the link itself.

This means deleting this sentence:
"A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined automatically or configured directly, on the LAN side by setting the MTU option in Router Advertisements [RFC4861] messages to the 6rd Tunnel MTU."

RD

---

From: "Lee, Yiu" <[email-masked]>
Date: Fri, 12 Mar 2010 13:15:48 -0500

+1. I agree with the current text to handle MTU and I support to move
forward for this draft.

On 3/8/10 5:25 PM, "Rémi Després" <[email-masked]> wrote:

> - 6rd is an transition technique which, to be as useful as it can be for rapid
> IPv6 deployments, should be stabilized asap.
> - Potential blackholes of IPv6 packets as long as ICMPv6 error messages and
> PMTUD are not treated properly belong to IPv6 in general, not to 6rd.
> - Such blackholes can be completely avoided in the short term, despite buggy
> ICMPv6 and/or PMTUD treatments, by sending IPv6 packets that don't exceed 1280
> octets, the guaranteed IPv6 path MTU of RFC 2460.
> - Provided *the last sentence of the section 9.1 on MTUs is deleted* (as
> already proposed in my previous e-mail), the document is therefore sound in
> this respect.  
>

---

From: "Lee, Yiu" <[email-masked]>
Date: Fri, 12 Mar 2010 13:15:48 -0500

---

From: Rémi Després <[email-masked]>
Date: Sat, 13 Mar 2010 12:38:22 +0100

Lee,

1.
Good to have support for the 6rd RFC to progress now quickly.

How to interpret your support is however not 100% clear:
- either you agree on ALL what I propose, as the +1 suggests (i.e. with the deletion of the last sentence of 9.1).
- or you agree ONLY if this sentence is not deleted (as your "with the current text" suggests).

This is in fact a good opportunity to summarize, after many mails exchanged on-list and off-list, the problem with the following sentence:
<< A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined automatically or configured directly, on the LAN side by setting the MTU option in Router Advertisements [RFC4861] messages to the 6rd Tunnel MTU. >>

2.
The rationale to delete the sentence is AFAICT:

- A 6rd RFC is NOT the right place to recommend that the MTU advertised to host, on a given link, should differ from the real MTU of this link: the purpose of such a recommendation concerns IPv6 in general, not specifically 6rd. 

- Not having the sentence in the 6rd RFC doesn't prevent implementations to advertise reduced MTUs. 

- In any case, if advertising reduced MTUs is recommended, the technically consistent value is fixed (not ISP dependent), and is 1280 as explained below.

3.
The reason why, for IPv6 in general, 1280 is the right default transmit MTU comes from the following facts:

- 1280 is the PMTU that is guaranteed to be supported across all IPv6 paths. (Most paths are expected to have larger local MTUs than 1280, e.g. 1480 for 6to4 or 6rd across Ethernet links, but the only guarantee is "at least 1280"). 

- Standardizing a minimum PMTU was not needed in IPv4 because fragmentation is permitted within the network, which is NOT the case in IPv6. (The real minimum in IPv4, which is header + smallest fragment, i.e. 68, is not, like 1280, a practicable value for a transmit MTU). 

- 1280 is such that, on links having the most common MTU, i.e. 1500, a small number of piled up encapsulations can be supported, with a good margin in typical cases. 

- Hosts having 1280 as their default transmit MTU are not prevented from using larger MTUs on all e2e paths known to have larger MTUs: PMTU Discovery is designed for this. (PMTUD has some reported short term problems, but this should eventually be fixed.)

4.
Coming back to the 6rd draft:

- The text says:
"The 6rd Tunnel MTU should be set to the known IPv4 MTU minus the size of the encapsulating IPv4 header (20 bytes). For example, if the IPv4 MTU is known to be 1500 bytes, the 6rd Tunnel MTU may be set to 1480 bytes".
The first sentence clearly suggests that some ISPs will have 6rd tunnel MTUs smaller than 1480. 

- Now, assume such an ISP has 1400 as its tunnel MTU. Then, if a 1480B packet is sent to it (from some ISP where 1480 is supported) it will fall into a blackhole: it will be discarded when entering the ISP network that has the 1400 MTU. 

Unless I am mistaken somewhere, the 6rd RFC MUST NOT recommended this to happen.    

On the other hand, if all hosts send 1280B packets maximum, the standard value has its intended effect: NO MTU RELATED BLACKHOLE. 
There is a cost in terms of performance of this MTU security but, thanks to a good choice of the standard value, this cost remains quit reasonable in typical cases: in the extreme case of TCP packets having all their maximum size, using 12810 instead of 1280 as transmit MTU reduces the number of packets by less than 17%).

Note also that the transmit MTU of Teredo is AFAICT 1280.

5. 
Conclusion: as more and more IPv6 will be deployed, it will be more and more important to make all IPv6 specifications right, and for this to understand how to avoid future blackholes.  

Best regards,
RD

---

From: Mark Smith <[email-masked]>
Date: Sun, 14 Mar 2010 11:16:49 +1030

On Fri, 12 Mar 2010 11:38:28 +0100

> "A 6rd CE SHOULD advertise the 6rd Tunnel MTU, whether determined automatically or configured directly, on the LAN side by setting the MTU option in Router Advertisements [RFC4861] messages to the…

So all communications from hosts on the LAN is to the Internet? 

If you shrink the LAN MTU for the tunnel's benefit (or even just the
native IPv6 Internet's benefit), you also shrink the LAN's MTU for any
communications between the devices on the LAN. Consider the performance
impact of dropping the LAN MTU to 1280 if the chosen LAN MTU is 9000
bytes for local LAN performance. I don't think the presence of a 6RD
tunnel always implies that the dominant traffic path is to and from the
Internet and the LAN - it could be there just for remote management /
reachability e.g. ssh, email, snmp, occasional software / OS upgrades
and updates. An example of such a scenario might be where somebody puts
a HPC cluster in remote colo somewhere in the world to gain the
benefits of cheaper cooling and power, and the ISP/Colo provider uses
6RD to provide interim IPv6 connectivity. Because a very high
percentage of large amounts traffic is local to the cluster, larger LAN
MTUs provide performance benefits.

That's all assuming that the directly upstream router is the 6RD tunnel
end-point. If there is a routed local IPv6 infrastructure, does the
6RD tunnel's MTU now force the whole site to have a 1280 byte MTU?

These are the sorts of scenarios which have made me wonder whether
there should be some sort of mechanism to express different off-link
and on-link MTUs or on-site and off-site MTUs (distinguished by an
on-site aggregate prefix).

Regards,
Mark.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 14 Mar 2010 14:19:22 +1300

On 2010-03-14 13:46, Mark Smith wrote:
...
> on-site aggregate prefix).

I hope they're discussing that over in MIF. A different set of settings
(not just MTU) per prefix seems entirely rational; a nice large MTU
for a ULA prefix might make sense.

    Brian

---

From: Mark Smith <[email-masked]>
Date: Sun, 14 Mar 2010 12:30:03 +1030

On Sun, 14 Mar 2010 14:19:22 +1300

> for a ULA prefix might make sense.

I've thought you could go a bit further, allowing nodes to have
different MTUs on the local link.

The current RA announced MTU is made the offlink unicast/multicast MTU,
onlink multicast MTU and default MTU, if the following options aren't
understood. E.g. for the 6RD scenario, this would be 1280 bytes.

Another RA option is created to announce the maximum MTU of the link,
being the largest that the link supports e.g. the maximum frame
size the link layer can forward. If all switches were configured to
support 9K byte frames, this value would be 9K. This would default to
the normal or common link layer MTU values e.g. 1500 bytes for ethernet.

A ND NA option is created for a node to announce it's local MRU. A
sending node that understands these RA/ND NA options would then be able
to send unicast packets up to the smallest value of it's local
MTU, the maximum link MTU and the neighbor's MRU. E.g. if the local
interface only supports 8K MTU/MRU, then in that node's NAs it would
announce that 8K MRU, and other nodes that supported 9K MTUs would only
send 8K frames to it.

Regards,
Mark.

---

From: Rémi Després <[email-masked]>
Date: Mon, 15 Mar 2010 08:10:16 +0100

Mark,

Note that the first and main purpose of this discussion is a final consensus on the 6rd RFC, not to solve now ALL IPv6 remaining problems.

In this context, the point I make is:
1) EITHER the proposed RFC continues to recommend that an MTU smaller than that of the LAN link be advertised in RAs, in which case the recommended value SHOULD BE 1280,
2) OR, the proposed RFC says nothing about RA-advertised MTUs. 

Because choice 1 deals with a problem that is not 6rd specific, I proposed to agree on choice 2.

In view of your comments, choice 2 seems to also be yours.
Would you confirm?    

Regards,
RD

> So all communications from hosts on the LAN is to the Internet? 

Of course not.
I wonder how you reached this conclusion.

> Internet and the LAN

It doesn't.

> - it could be there just for remote management /

---

From: Rémi Després <[email-masked]>
Date: Mon, 15 Mar 2010 08:22:41 +0100

Mark, Brian, 

> for a ULA prefix might make sense.

1.
On-link path MTU being a well defined concept, having two different default transmit MTUs for on-link and off-link makes a lot of sense.
+1 for this idea.
+1 also for this discussion to take place in a larger context than the 6rd RFC. (But 6man could IMHO be a better place than MIF, because the solution also concerns single-interface hosts.)

2.
Even better than TWO default transmit MTUs, hosts should in my understanding have FOUR: on-link and off-link, IPv4 and IPv6.
Agreed?
(Of course, hosts that discover larger MTUs for some e2e paths can, independently from default values, apply these MTUs to these identified paths.)

Regards,
RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 15 Mar 2010 08:28:42 -0700

> Sent: Saturday, March 13, 2010 6:00 PM
> To: Brian E Carpenter
> Cc: [email-masked]

Iljitsch van Beijnum talked about this not long ago:

http://tools.ietf.org/id/draft-van-beijnum-multi-mtu-02.txt

I talked about it years and years ago:

http://tools.ietf.org/id/draft-templin-ndiscmtu-00.txt

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 15 Mar 2010 09:15:46 -0700

Brian,

> Sent: Saturday, March 13, 2010 5:19 PM
> To: Mark Smith
> Cc: [email-masked]

If you want to consider the IPv6 6rd global for off-site
communications only, then that would be doable but I
still think we should try for 1500 or larger instead of
1280 if at all possible. If you then want to consider an
IPv6 ULA for on-site communications with larger MTU, that
would be fine too.

Within the customer site, however, both the 6rd global
and ULA can be serviced using ISATAP if native IPv6 is
not yet available.

Fred

---

From: Mark Smith <[email-masked]>
Date: Tue, 16 Mar 2010 03:12:26 +1030

On Mon, 15 Mar 2010 08:10:16 +0100

> In view of your comments, choice 2 seems to also be yours.

I don't strongly disagree with 1, as a SHOULD, however I would if it
was a MUST. Possibly a MAY would be a better choice if the text were to
remain.

However, bearing in mind the possibility of multiple downstream
subnets, I think 2 is probably the better option to reach consensus.

Regards,
Mark.

---

From: Rémi Després <[email-masked]>
Date: Mon, 15 Mar 2010 17:59:57 +0100

> was a MUST. Possibly a MAY would be a better choice if the text were to

Agreed: the MUST would be inappropriate.

> subnets, I think 2 is probably the better option to reach consensus.

We then do agree on 2.
Let's hope others can adopt this as the desired consensus.

Cheers,
RD

---

From: Rémi Després <[email-masked]>
Date: Mon, 15 Mar 2010 18:59:21 +0100

> Even better than TWO default transmit MTUs, hosts should in my understanding have FOUR: on-link and off-link, IPv4 and IPv6.

Actually, it is a little more subtle than I first thought:
- The link MTU being common to IPv6 and IPv4, one is enough.
- In IPv6, the default transmit MTU for off-link transmissions SHOULD be 1280.
- In IPv4, there is no standard value to  be recommended as default transmit MTU for off-link destinations (the network will fragment where needed). It may however be useful to have one that differs from the link MTU, especially on links that support 9000B Jumbo Frames.  

The bottom line could then be:
- Hosts SHOULD have a least 2 default transmit MTUs, one for all on-link transmissions and one for off-link IPv6 destinations.  
- They MAY advantageously have third one, modifiable by advanced users, for off-link IPv4.

Thoughts?

RD

---

From: Ole Troan <[email-masked]>
Date: Mon, 15 Mar 2010 19:11:50 +0100

all,

[...]

> subnets, I think 2 is probably the better option to reach consensus.

yes, I think so too. I'll remove that paragraph from the revision-08 draft, unless someone has strong objections.

do we have other last call comments that people feel haven't been addressed fully?
I have asked for a 5 minute slot in Anaheim to go through the LC comments/resolutions, but there will not be time for extensive discussions in the room.

may I suggest that those interested in discussing MTU more generally move to a separate thread?

Best regards,
Ole

---

From: "Lee, Yiu" <[email-masked]>
Date: Mon, 15 Mar 2010 14:59:58 -0400

I agree with Ole. MTU problem isn't unique to 6RD, it also happens to
DS-lite or any other tunneling technique. We should move forward for 6RD and
perhaps start another draft for the MTU topic.

Any thought?

On 3/15/10 2:11 PM, "Ole Troan" <[email-masked]> wrote:

>>> 1) EITHER the proposed RFC continues to recommend that an MTU smaller than
>>> that of the LAN link be advertised in RAs, in which case the recommended
>>> value SHOULD BE 1280,
> I have asked for a 5 minute slot in Anaheim to go through the LC
> comments/resolutions, but there will not be time for extensive discussions in
> the room.
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 15 Mar 2010 12:34:23 -0700

Brian,

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Templin, Fred L
> Sent: Monday, March 15, 2010 9:16 AM
> To: Brian E Carpenter; Mark Smith
> Cc: [email-masked]

Going back slightly on what I said, what do you have
in mind - a per-prefix MTU? As I understand it, routers
advertise MTUs per-*link*; not per-*prefix*. Were you
thinking we could invent a per-prefix MTU advertisement
and use it to keep site-local communications at a higher
MTU than global communications? I don't se any standard
support mechanisms for something like that at this time
unless I'm missing something.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 15 Mar 2010 12:34:47 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Lee, Yiu
> Sent: Monday, March 15, 2010 12:00 PM
> To: Ole Troan; softwires

Yes - 1500+ is within our grasp, so why let it slip
through our fingers and settle for 1280?

Fred

---

From: Ole Troan <[email-masked]>
Date: Mon, 15 Mar 2010 23:42:14 +0100

> through our fingers and settle for 1280?

so what are you proposing?

1) that we leave current work on all tunnel-based solutions off the table, while we work on a general solution for the MTU problem.

2) develop a general MTU solution in parallel with current work.

cheers,
Ole

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 15 Mar 2010 16:18:13 -0700

Hi Ole,

> Sent: Monday, March 15, 2010 3:42 PM
> To: Templin, Fred L
> Cc: Lee, Yiu; softwires

As you may or may not know, SEAL has been in the works for
a long time now. This coming Aug/Sep will mark 8 years, and
coincided with the closure of NGTRANS (search for "pool-pah"):

  ftp://ftp.ietf.org/ietf-mail-archive/ngtrans/2002-08.mail

the declaration of IPv6 as "operational":

  http://www.ops.ietf.org/lists/v6ops/v6ops.2002/msg00000.html

and the first meeting of the v6ops working group: 

  http://www.ops.ietf.org/lists/v6ops/v6ops.2002/msg00012.html

RFC5320 has an experimental implementation to match the
document's experimental category:

  http://osprey67.com/seal

and now, 'draft-templin-intarea-seal' is ready for general
consumption and development of interoperable implementations.
In fact, if anyone wants to beat me to forward-porting the
existing linux code to match the new spec, I would be glad
to challenge them to a bake-off in the near future.
Implementations for other OS's would of course also
be welcome.

As I said before, 6rd with a 1280 clamping might be pretty
good, but why not go the extra step to remove artificial
size restrictions and make it *great*.

Thanks - Fred

---

From: "Lee, Yiu" <[email-masked]>
Date: Mon, 15 Mar 2010 19:44:18 -0400

Hi Fred,

I support to have a different draft to describe operation recommendation for
MTU in deployment with tunnel. And SEAL can be one of the recommendations.
But I do not support to make SEAL a requirement to 6RD.

Thanks,
Yiu

On 3/15/10 7:18 PM, "Templin, Fred L" <[email-masked]> wrote:

>> 1) that we leave current work on all tunnel-based solutions off the table,
>> while we work on a general
>> solution for the MTU problem.
>>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Mar 2010 14:46:52 +1300

Below...

> unless I'm missing something.

That's why I said that I hope they are discussing this
in MIF. Parameters per interface is one level of granularity,
parameters per prefix is another. I haven't had time to see
what MIF is up to.

    Brian

---

From: Mark Smith <[email-masked]>
Date: Tue, 16 Mar 2010 19:14:14 +1030

On Tue, 16 Mar 2010 14:46:52 +1300

> parameters per prefix is another. I haven't had time to see

I agree. I think there are 3 scopes for different MTUs - per link, per
site (possibly distinguished by a prefix), and offsite i.e. the
Internet.

---

From: Rémi Després <[email-masked]>
Date: Tue, 16 Mar 2010 12:11:23 +0100

> size restrictions and make it *great*.

The reason why 6rd IS "great", is because it has been kept simple, and in particular stateless.
I therefore hope you don't mean that including SEAL in it would be an improvement: that would destroy its main virtue.

RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 16 Mar 2010 10:31:34 -0700

Mark and Brian,

> From: Mark Smith [mailto:[email-masked]]
> Sent: Tuesday, March 16, 2010 1:44 AM
> To: Brian E Carpenter
> Cc: Templin, Fred L; [email-masked]
> Subject: Re: MTU per * [SOFTWIRE working group last call on 6rd]
> 

In my understanding, per-link is what implementations
do today. An MTU is assigned to an interface, and an
interface is attached to a link. So, I don't see any
existing mechanism for associating an MTU with a prefix
unless for example each prefix could be associated with
a different *interface*, and multiple interfaces could
attach to the same link.

I can't see that happening without significant
modifications to IPv6 ND and the conceptual sending
algorithm - can you? For example, there would need
to be an MTU associated with each Prefix Information
Option as part of the router and prefix discovery
procedure and there would need to be a source address
based interface selection as part of the sending
algorithm.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 16 Mar 2010 10:47:41 -0700

Remi,

> Sent: Tuesday, March 16, 2010 4:11 AM
> To: Templin, Fred L
> Cc: Ole Troan; softwires; Lee, Yiu

"Fully stateless" does not in itself translate to "greatness".
I have identified a number of operational limitations, many of
which would be addressed with a modicum of soft state. 6rd to
me is not great if it would be deployed with full knowledge
that it would have to be replaced in the near future due to
limitations brought on primarily through strict adherence to
statelessness as a "virtue".

Fred

---

From: Rémi Després <[email-masked]>
Date: Wed, 17 Mar 2010 08:42:32 +0100

Fred,

Here are some quick reactions to the list of issues you see for an applicability-statement document on 6rd.  

Le 2 mars 2010 à 17:51, Templin, Fred L a écrit :
>  I see a number of aspects
A site has a valid IPv6 address as long as it has a valid IPv4 address. 

> - potential black holes when ICMPs can't be translated
I don't see which are these "ICMPs that can't be translated"?

> - provider-aggregated addressing only (no provider-independent)
I don't see the problem.

> - not compatible with multihoming via a single, stable IPv6 prefix
Same as previous point?

> - not compatible with traffic engineering (i.e., CE can't select
It doesn't need to.

> - interactions with load balancing; ECMP within provider network
No problem, this is the advantage of stateless solutions.

>  and doesn't allow for automatic discovery of larger MTUs
This is a general 6rd problem (not 6rd specific).
 
> - requires operators to exercise tight controls on link MTUs
ISPs that want to use 6rd have to know which IPv4 MTU they support, this MTU being at least sufficient to encapsulate  1280B IPv6 packets
The draft is clear, and what it says is sufficient. 

>  generation and ICMP filtering
Nothing special, ICMPv6 error messages must be generated, and forwarded as usual.
Due to the absence of fragmentation of 6rd packets in 6rd domains, ICMPv4 error messages can always be translated. 

Regards,
RD

---

From: Rémi Després <[email-masked]>
Date: Wed, 17 Mar 2010 10:13:51 +0100

Le 1 mars 2010 à 08:44, Gabi Nakibly a écrit :
> Does the SP necessarily knows of every ISATAP router or 6to4 relay deployed in the 6rd domain? After all, a customer can deploy an ISATAP router or even a 6to4 relay for that matter without the knowledge of the SP, while using a configured tunnel (by a tunnel broker) for its IPv6 connectivity.

Gabi,

I didn't succeed to create a detailed configuration where a 6rd CE, which is also an ISATAP router, can create a routing loop.

Would you have one that you can share?

Cheers,
RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 17 Mar 2010 10:44:00 -0700

Hi Remi,

Responding to your post:

> Sent: Wednesday, March 17, 2010 12:43 AM
> To: Templin, Fred L

The point is that, if a site's IPv4 address changes, its
(6rd-derived) IPv6 prefix also changes. If that were the
case, then all IPv6 routers and hosts within the site
would have to renumber. Thus, as a limitation to the domain
of applicability, 6rd is appropriate only to those sites
that either a) are guaranteed to always have the same IPv4
address assignment, or b) can tolerate an IPv6 prefix
renumbering event. This would need to be discussed in
an applicability statement.

> I don't see which are these "ICMPs that can't be translated"?

ICMPv4 messages are permitted by standards to contain only
a minimal packet-in-error consisting of the IPv4 header plus
8 bytes beyond the header. For ipproto-41 packets, that means
that only the first 8 bytes of the encapsulated IPv6 header
may be present in the ICMPv4 header. That is not enough
information for stateless translation to a corresponding
ICMPv6 message, so unless some state is kept in the 6rd
router no ICMPv6 message can be delivered to the original
IPv6 source. Thus, 6rd has a potential black hole condition
in environments where standards-compliant routers might
produce only minimal ICMPv4 messages.
 
> > - provider-aggregated addressing only (no provider-independent)

Some end systems may wish to use a single and stable IPv6
source address prefix which the routing system can represent
to multiple providers for fault tolerance and load balancing
purposes. Thus, as a limitation to the domain of applicability,
multihoming and traffic engineering would have to be directed
by the end systems instead of the 6rd router, and session
continuity cannot be preserved in the event of failure of
a provider link. 
 
> > - not compatible with multihoming via a single, stable IPv6 prefix

Discussed above.

> > - not compatible with traffic engineering (i.e., CE can't select

In some 6rd domains, the routing system may give preference
to certain BRs over others for reasons that are unrelated to
(and perhaps in conflict with) the use of 6rd. So, unless
the 6rd routers s become involved in explicit BR selection
based on load balancing, ECMP, default router preferences,
etc. any traffic engineering is at the whim of the network
and not under the control of the 6rd router.

This comment is oriented to the exclusive use of anycast for
BR selection, while later versions of the 6rd spec (i.e., the
Townsley/Troan spec) have hinted that unicast addresses may
be made available at some point. Thus, if an applicability
statement is included it should discuss traffic engineering
assumptions in relation to anycast vs unicast BR addresses.

> No problem, this is the advantage of stateless solutions.

This folds into the discussion point above, and if an
applicability statement is included it needs to discuss
interactions between 6rd and network-based mechanisms
such as load balancing and ECMP.

> This is a general 6rd problem (not 6rd specific).

Maybe you meant a general problem for tunneling? If an
applicability statement is included, however, it should
also mention the implications of MTU clamping. 
 
> ISPs that want to use 6rd have to know which IPv4 MTU they support, this MTU being at least
> sufficient to encapsulate  1280B IPv6 packets

If an applicability statement is included, it should
also mention the need for well-managed MTUs in the
provider network.

> Due to the absence of fragmentation of 6rd packets in 6rd domains, ICMPv4 error messages can always

This folds into the discussion above regarding standards-
compliant ICMPv4s that contain insufficient information
for translation.

The point about fragmentation brings up a critical issue
wrt the use of anycast as an IPv4 source address. If
multiple 6rd BRs use the same IPv4 source address when
sending tunneled packets to the same 6rd CE router, and
the BRs do not coordinate the IP IDs they use, then
collisions within the CE router reassembly cache can
more readily lead to undetected data corruption than
is the case for non-anycast. That means that when 6rd
BRs use an anycast source address, there *really* needs
to be strong assurance against fragmentation in the
network. Clamping the MTU and setting DF=1 would be
best in that case.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 17 Mar 2010 10:55:05 -0700

Remi,

I'm sorry, I forgot to add that the 6rd domain must be
made secure against IPv4 source address spoofing. Domains
that cannot ensure this may not qualify as being within
the 6rd domain of applicability.

Fred

> Sent: Wednesday, March 17, 2010 10:44 AM
> To: Rémi Després
> Cc: [email-masked]