Subject: [Int-area] Comments for draft-ietf-intarea-tunnels-02

From: Vincent Roca <[email-masked]>
Date: Wed, 8 Jun 2016 16:44:44 +0200

Hello everybody,

First of all, I need to say that I found the draft-ietf-intarea-tunnels-02 I-D extremely useful
and wish I found it before.

We exchanged a few private emails with Joe last week and I promised to send detailed
comments on the list. Here they are... Sorry if this is a bit long.

** Section 2.2. Terminology, Link definition: it is said:
        "Link: a communication device that transfers messages between network devices..."
The notion of network device is undefined. I guess you mean "network nodes" defined above.

** In Ingress and Egress definitions: it is said:
        "Ingress: a network node that..."
The Ingress (resp. Egress) is not a network node. Section 3.4 has a better definition
IMHO. So:

OLD:
   o  Ingress: a network node that receives messages, encapsulates them
      according to the tunnel protocol, and transmits them into the
      tunnel. Note that the ingress and source can be co-located.

NEW:
   o  Ingress: a tunnel entry endpoint at the network node that the tunnel
      interconnects. It it typically described as "network interface"."
      The Ingress receives messages, encapsulates them according to the tunnel
      protocol, and transmits them into the tunnel. Note that the ingress and
      source can be co-located.

(similar changes for the Egress definition)

** In Egress definition: "The egress decapsulates datagrams..."
Maybe "IP datagrams" to be coherent with other uses of this term.

** I suggest adding the min LinkMTU values as they are considered throughout this I-D.
I also noticed that "LinkMTU" is used in the I-D to denote this value, not "LMTU".

OLD:
   o  Link MTU (LMTU): the largest message that can transit a link. Note
      that this need not be the native size of messages on the link.

NEW:
   o  Link MTU (LinkMTU): the largest message that can transit a link. LinkMTU is
      at minimum equal to 68 octets with IPv4 [RFC791] or 1280 octets with IPv6 [RFC2460].
      Note that this need not be the native size of messages on the link.

** In RMTU definition:
"receiver" is ambiguous. I understand that definitions can be generic, however
throughout this I-D we only consider the Egress Reassembly MTU so it's worth
specializing.
I also clarified the case of encapsulation header size that is deducted.

OLD:
   o  Reassembly MTU (RMTU): the largest message that can be reassembled
      by a receiver, and is not directly related to the link or path
      MTU. Sometimes also referred to as "receiver MTU".

NEW:
   o  Reassembly MTU: the largest message that can be reassembled
      by a receiver, i.e., either the Egress or the Destination. It is not
      directly related to the link or path MTU. Sometimes also referred to
      as "receiver MTU".

   o  Egress Reassembly MTU (EgressRMTU): the largest message that can be reassembled
      by the Egress. The minimum EgressRTMU value is 576 octets with IPv4 [RFC791]
      and 1500 octets with IPv6 [RFC2460] minus the encapsulation header size.

** Path MTU definition: I think it's worth clarifying that we focus on the tunnel PMTU in this I-D.
I also explain that the encapsulation header size has already been subtracted from the
Tunnel PMTU.

OLD:
   o  Path MTU (PMTU): the largest message that can transit a path.
      Typically, this is the minimum of the link MTUs of the links of
      the path.

NEW:
   o  Path MTU: the largest message that can transit a path.
      Typically, this is the minimum of the link MTUs of the links of
      the path.

   o  Tunnel Path MTU (TunnelPMTU): the largest Tunnel Transit Packet (or fragment)
      that can transit inside the tunnel's path. The encapsulation header size is already
      subtracted from TunnelPMTU and this TunnelPMTU  is not visible from outside the
      tunnel, unlike the Tunnel MTU.

** In tunnel MTU definition:
Say explicitly that the encapsulation header size has already been subtracted from
the Tunnel MTU,  since encapsulation headers are considered as "headers from lower
protocols" and therefore excluded.

OLD:
   o  Tunnel MTU (TMTU): the largest message that can transit a tunnel.
      Typically, this is limited by the egress reassembly MTU.

NEW:
   o  Tunnel MTU (TunnelMTU): the largest message that can transit a tunnel.
      The encapsulation header size is already subtracted from TunnelMTU.
      Typically, the TunnelMTU is limited by the EgressRMTU.

** I think a figure like this one (70 characters width ;-) can be useful. At least it
helped me a lot!

Tunnel Transit
Packet (TTP) or                Tunnel Link Packet
"tunneled pkt"  +------------+      (TLP) or       +------------+
--------------->|Network Node|   "tunnel packet"   |Network Node| --->
                |    +-------|  ________________   |-------+    |
                |    |Ingress|-|________________|->| Egress|    |
                +----+-------+                     +-------+----+
                   Tunnel MTU   Tunnel Path MTU     Egress Reassembly
                  or Link MTU                       MTU
with:
Link MTU == Tunnel MTU == Egress RMTU
(the maximum encapsulation header size is already deducted)

** In the doc there is a misspelling, the correct acronym is PLPMTUD (two P's),
not PLMTUD (5 occurrences).

** Section 3.1: this is a detail. It is said:
        "... the tunnel serves as a link to the devices it connects (here, Ra and Rb)."
Instead of "device" that is not defined in section 2.2, I think "network nodes" would
be more appropriate.

** Section 4.1, Fragmentation:
I suggest to change the algorithms along with their introduction text as follows:

   // VR: added a reminder that LinkMTU == TunnelMTU
   These rules apply at the host/router where the tunnel is attached (remind that the
   Link MTU is equal to the Tunnel MTU):

      // VR: I've changed the test order to follow the same logic in the two algorithms
      // VR: TTPsize is the official name, so I've removed TTP and sizeof(TTP)
      if (TTPsize <= linkMTU) then
         send TTP into the tunnel "interface" (i.e., ingress)
      else
         if (TTP can be fragmented, e.g., IPv4 DF=0) then
            // VR: as we test against linkMTU, it's better to mention linkMTU below,
            //     not TunMTU even if both are the equal
            split TTP into fragments of linkMTU size
            and send each fragment into the tunnel ingress
         else
            // VR: it's important to detail how the MTU field of the ICMP PTB is initialized
            drop TTP and send ICMP "too big" to TTP source that
            advertises Next-Hop MTU = linkMTU
         endif
      endif

   // VR: added a reminder that TunnelMTU == EgressRMTU
   These rules apply at the tunnel ingress of the host/router where the
   tunnel is attached (remind that the Tunnel MTU is equal to the Egress
   Reassembly MTU):

      // VR: as said above, TTPsize, not sizeof() operator
      if (TTPsize <= TunnelPathMTU) then
         encapsulate TTP as received and emit
      else
         // VR: this test is sufficient and IMHO more lisible
         // VR: I prefer to use TunnelMTU since we are at the ingress
         if (TTPsize <= TunnelMTU) then
            // VR: there was a mistake below that mentioned TunnelMTU (!) chunks.
            fragment TTP into chunks of size TunnelPathMTU
            // VR: there was a mistake below that mentioned TTP instead of TTP chunks
            encapsulate and emit each TTP chunk
         else
            {never happens; host/router already dropped by now}
         endif
      endif

NB: I personally prefer :
        if
        else if
        else
        endif
to nested if's, but this is not a big deal.

** Same section 4.1
The 3rd algorithm should be better introduced. For instance it is not clear
that this is just an example for algorithm 2, using minimum values.
Also it is not clear reading this paragraph if:
        option 1: the tunnel Path MTU must be >= (1280 - 40 - TOptSz), or
        option 2: the tunnel Path MTU must be >= 1280, to which one needs to
                subtract (40 + TOptSz) because of encapsulation
This is option1 if we compare with previous algorithm, but it's far from obvious...
It comes from the fact we compare with the TTP size before encapsulation. If
we compare with the size of encapsulated TTP, it would be different.

Globally, when to count or ignore encapsulation headers in MTUs is extremely
subtle and error prone. This I-D must avoid any ambiguity of this kind.

Here is a proposal:

OLD:
   For IPv4 or IPv6 over IPv6, the tunnel path MTU is a minimum of 1280
   minus the encapsulation header (40 bytes) with its options (TOptSz)
   and the egress reassembly MTU is 1500 minus the same amount:

NEW:
   As an example let us consider IPv4 over IPv6, or IPv6 over IPv6 tunneling,
   where IPv6 encapsulation adds a 40 byte fixed header plus options (i.e.,
   header extensions) of total size TOptSz. From [RFC2460] it follows that the
   Tunnel MTU must be at least 1280 bytes and the Egress Reassembly MTU must
   be at least 1500 - (40 + TOptSz) bytes. The Tunnel Path MTU must be a minimum
   of 1280 - (40 + TOptSz) bytes. Considering these minimum values, the previous
   algorithm becomes:

** Still section 4.1: we cannot say that 1500 is the minimum EgressRMTU as
we need to remove the encapsulation header size before as detailed above.
I also tried to explain why... Not easy! Do you have a better explanation?

NEW:
   When using IP directly over IP, the minimum Egress Reassembly MTU
   equals (576 - encapsulation header size) bytes for IPv4 [RFC791] and (1500 -
   encapsulation header size) bytes for IPv6 [RFC2460].
   Note that the encapsulation header size must be deducted from the  576 (resp. 1280)
   value because [RFC791] (resp. [RFC2460]) requires the destination (here the egress)
   to "be able to accept a fragmented packet that, after reassembly, is as large as 576
   (resp. 1500) octets". In our case the reassembled packet corresponds to the encapsulated
   TTP (i.e., packets (a) and (c) of Fig. 10) that was fragmented by the Ingress.

** Section 5.2: I fully agree with:
   "Detect when the egress MTU drops below the required minimum and shut
   down the tunnel if that happens - configuring the tunnel down and
   issuing a hard error may be the only way to detect this anomaly, and
   it's sufficiently important that the tunnel SHOULD be disabled."

However this not only an implementation aspect. This is also required for security
reasons and therefore it should be discussed in the Security Considerations section
as well. E.g., we observed opposite behaviors from an on-the-shelf IPsec
implementation and discussed consequences in our (now expired)
draft-roca-ipsecme-ptb-pts-attack-00.

[WG: we already discussed this aspect in private emails with Joe last week.]

** Section 5.4.4: it is said, for the "consistent with this doc" part, that:
        "Shuts the tunnel down if the tunnel path MTU isn't => 1280."
This contradicts the example of section 4.1 where TunnelPMTU == 1280-40-TOptSz
is valid...
Do you mean "Shuts the tunnel down if the Tunnel MTU, seen from the outside network,
isn't >= 1280"?

** Section 6, Security considerations
As already discussed privately with Joe, I may have other items but still need
to think it over.

** Section A.1: the following sentence is ambiguous:
OLD:
        "When the encapsulated packet exceeds the MTU of the tunnel, the
        packet needs to be fragmented."
This is the Tunnel Path MTU, not MTU of the tunnel which can be understood as
synonymous to the Tunnel MTU.

NEW:
        "When the packet (iH, iD) size exceeds the Tunnel Path MTU, the encapsulated packet
         needs to be fragmented."

** Section A.1/ A.2:
Discussion in section 4.1 assumes an Outer Fragmentation scheme. The notion of
Egress Reassembly also assumes an Outer Fragmentation. In fact it seems to be
the model assumed throughout this I-D but this is not clearly stated unless I missed
something. Am I right?

It's clearly more universal as it does not make any assumption on the inner packet
(can it be fragmented on path or not, as explained in A.2).
I think the I-D should identify from the beginning the assumption made (Outer vs.
Inner fragmentation) as it will impact the discussion. Said differently move Appendix A
in Section 3 for instance.

Cheers,

  Vincent

---

From: Joe Touch <[email-masked]>
Date: Wed, 8 Jun 2016 13:48:48 -0700

Hi, Vincent,

Thanks for your detailed comments. All are queued up for the next revision.

Comments below only where needed.

Joe

On 6/8/2016 7:44 AM, Vincent Roca wrote:
> Hello everybody,
>
>
> First of all, I need to say that I found the
> draft-ietf-intarea-tunnels-02 I-D extremely useful
...
> I also clarified the case of encapsulation header size that is deducted.
It's worth noting throughout that MTU is defined as the size of the payload.

>       tunnel, unlike the Tunnel MTU.

This definition is what the tunnel MTU is intended to convey.

I think we do need another term for the path MTU between the egress and
ingress but NOT inside the tunnel. I prefer to just call that the PMTU
between the ingress and egress, though.

>       Typically, the TunnelMTU is limited by the EgressRMTU.
This should be clarified as the MTU that can transit *inside* a tunnel.

...
> I also tried to explain why... Not easy! Do you have a better explanation?

It's because IPv6 uses the term MTU in terms of the link payload, not as
the network payload.

Throughout this doc, we similarly refer to the tunnel as a link, so link
MTU is the correct notion.

...
> [WG: we already discussed this aspect in private emails with Joe last

See below - I think Vincent and I are still working on this...

> Do you mean "Shuts the tunnel down if the Tunnel MTU, seen from the

Yes. That's consistent with tunnel = link and MTU = payload.

> synonymous to the Tunnel MTU.

It's the tunnel MTU, as clarified above.

> the model assumed throughout this I-D but this is not clearly stated

You are correct.

> in Section 3 for instance.

Point taken, esp. with discussions on other lists (e.g., v6ops) and the
notion of "recursive" tunneling (X in X, for any X).

Joe

---

From: Vincent Roca <[email-masked]>
Date: Fri, 10 Jun 2016 09:45:04 +0200

Hello Joe,

> Thanks for your detailed comments. All are queued up for the next revision.

You're welcome.

A few answers below. I removed all items we agreed.

> It's worth noting throughout that MTU is defined as the size of the payload.

Yes but there are two notions of payloads depending on the point of view.
Figure 10 can be useful:
	inner packet payload => iD
	tunnel link packet's payload => iH+iD

> This definition is what the tunnel MTU is intended to convey.

I don't understand. If we re-use example of 4.1, Tunnel MTU=1460 - TOptSz
while tunnel path MTU=1240 - TOptSz. The first one is visible to source,
the second one is not, and the difference is due to reassembly at egress.

> between the ingress and egress, though.

Okay, this is the regular IP vision of PMTU. If we need it, we can define it.

> This should be clarified as the MTU that can transit *inside* a tunnel.

yes, it's simple and consistent with the tunnel=link viewpoint.

> It's because IPv6 uses the term MTU in terms of the link payload, not as

You're right, RFC2460 uses "link MTU". I never paid attention to this.

> MTU is the correct notion.

Yes.

Regards,

  Vincent

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 10 Jun 2016 18:23:19 +0000

Hi Vincent,

I would like to invite you to review Section 3.1.3 of the AERO spec:

https://datatracker.ietf.org/doc/draft-templin-aerolink/

which is titled: "AERO Interface MTU and Fragmentation" and presents an alternate
viewpoint. In particular, it allows for leveraging operational assurance of sufficient
MTU (RC4459, Section 3.3) and path MTU discovery from within the tunnel (RFC4459,
Section 3.2) in environments where it is assured that those techniques will not result
in black holes.

In other cases, the tunnel ingress makes sure that packets of minimum sizes get
through even if a little fragmentation and reassembly are needed, and allows
larger packet into the tunnel unfragmented at risk that they may be dropped
silently. It is therefore up to the original source to ensure that PLPMTUD is used
when it sends large packets.

Thanks in advance for your review and comments,

Fred
[email-masked]

> -----Original Message-----
> From: Int-area [mailto:[email-masked]] On Behalf Of Vincent Roca
> Sent: Friday, June 10, 2016 12:45 AM
> To: Joe Touch <[email-masked]>
> Cc: [email-masked]; [email-masked]
> Subject: Re: [Int-area] Comments for draft-ietf-intarea-tunnels-02
> 
> Hello Joe,
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 10 Jun 2016 11:46:14 -0700

Comments below...

On 6/10/2016 12:45 AM, Vincent Roca wrote:
...
> 	tunnel link packet's payload => iH+iD

We should never be concerned with the E2E payload, i.e., iD. All we need
to know is the size of the entire E2E message that a link (or tunnel,
which is a link) must transfer.

However, I see the confusion here.

As context:
    TCP MSS describes the payload, exclusive of IP and TCP headers and
their options (RFC6991)

    ICMPv4  frag needed and IPv6 PTB indicates "next-hop MTU" as the
largest IP packet that the next-hop link can forward without on-path
fragmentation (RFC1191, RFC4443)

So MTU generally refers to the *payload* that a link or path can transfer.

--

For this document:

path MTU should indicate the IP packet that can transit a path, and it's
the min of the hop MTUs

hop MTU should indicate the one-hop version of the path MTU, i.e., the
largest IP packet that can *transit* over a link (it does NOT include
the  link headers)
    these are the ones that are 64 or 1280

link MTU should be defined more clearly as the link-layer packet that
can transit the link
    these must transit (64 or 1280) + link headers

We should use two different terms now:

tunnel hop MTU: the largest IP packet that can transit (inside) a tunnel

tunnel link MTU: the largest (encapsulated) packet that can transit
between ingress and egress (in the current doc, we call this the tunnel MTU)

We should add a new term:

E2E MTU: the largest IP message that can transit between a source and
destination, which is 576/1500 at a minimum.

And yes, a new figure can help explain all of this more clearly.

--

With regard to your suggestions:

- I don't think it's useful to define link MTU as 64/1280 because it
isn't; it's those values + link (as noted above)

- the reassembly MTU definition needs to explain that it's at the level
being reassembled, NOT the payload of the reassembled message. at IP
receivers, this is the E2E MTU but not the path MTU. at tunnel egress,
this is the tunnel link MTU (i.e., the E2E MTU between ingress and egress).

Does that make more sense?

> Okay, this is the regular IP vision of PMTU. If we need it, we can define it.

See above; E2E MTU between ingress and egress is different than any PMTU.

Joe

---

From: Vincent Roca <[email-masked]>
Date: Fri, 17 Jun 2016 08:47:52 +0200

Hello Fred,

Sorry for the delay...

> when it sends large packets.

I don't know the exact motivations for this proposal, and therefore won't comment on
this aspect. I "just" read the current I-D, section 13, and here are some comments.

** 3.13 Intro: it is said:
  "Although IPv4 specifies a smaller minimum link MTU of 68 bytes [RFC0791], AERO
   interfaces also observe a 1280 byte minimum for IPv4 even if some fragmentation
   is needed."

I fully agree! We observed on our side that Linux/Windows hosts react badly to
ICMP PTB announcing PMTUs below 576 (this is not the exact value). So this idea
of imposing a minimum MTU that is larger is fine, and aligning it with IPv6 can make
sense.

** 3.13 Intro: it is said:
  "Original source hosts have therefore become
   conditioned to expect that IP packets up to 1500 bytes in length will
   either be delivered to the final destination or a suitable PTB
   message returned."

Really? Isn't "conditioned to expect" a bit too strong? This is true for standard
links but with the multiplication of "virtual links" (e.g., tunnels) I wouldn't make
such a claim.

** 3.13.2 vs. 3.13.3 criteria
I understand that behind:
   "When the original source, ingress and egress are all within the same well-managed
    administrative domain"
there is in fact the idea of having a guaranty that there is no ICMP blackhole, and therefore
PMTUD is effective. Is it really what you mean? In that case why don't you say that clearly?
And could a "well managed admin. domain" decide to filter or not generate ICMP(v6) PTB
for some legitimate reason (or assumed so)?
In summary, we need a definition of what is meant by "well-managed" (I think it's not the
case in previous sections).

** 3.13.2: notions of L3 versus L2 PTB message not defined.
I understand that L3 PTB refers to ICMP(v6) PTB sent to the source, while L2 PTB refers
to ICMP(v6) PTB exchanged within the tunnel, seen as a link. Is it what you mean?
I'd suggest to say that explicitly in the I-D.

** 3.13.3, first bullet:
   "... i.e., unless lost due to congestion or routing errors."
Don't try to provide an exhaustive list of reasons why a packet won't arrive, it's impossible.

** 3.13.3, 2nd bullet:
By "minimum number of non-overlapping fragments", I guess you mean two. Say that.
Also, I don't really understand why you recommend the egress to be capable of reassembling
at least 2KB. Why 2KB rather than, say, 1500+100? It looks like a magic number.

** 3.13.3, 4th bullet, 1)
This I-D is about tunneling, so why should it RECOMMEND that end-host use PLPMTUD?
It's out of scope. A lower case "recommended" would be preferable.

** 3.13.3, last two paragraphs:
Once again, the following paragraph seem to assume that ICMP(v6) PTB is operational.
Say that, especially as it can be the case even if ingress/egress are not in the same
"well-managed admin domain". Same comment for the last paragraph.

Cheers,

  Vincent

---

From: Vincent Roca <[email-masked]>
Date: Fri, 17 Jun 2016 09:23:46 +0200

Hello Joe,

Here also, sorry for the delay!

> So MTU generally refers to the *payload* that a link or path can transfer.

I'm 100% okay. My comment is just that "payload" is misleading if the context is
not perfectly defined (transport vs. network vs. link, with vs. without tunnels).

**largest** IP packet...

>    these are the ones that are 64 or 1280

rather than "it does NOT include..." I'd prefer:
"this network layer definition does not consider link-layer headers"

> And yes, a new figure can help explain all of this more clearly.

Great. Those definitions will help a lot.

That's a good reason indeed ;-)
With the updated definitions, there won't be any risk of mistake.

> Does that make more sense?

Yes, perfectly clear.

> See above; E2E MTU between ingress and egress is different than any PMTU.

Thanks for the updated definitions.
Cheers,

  Vincent

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 17 Jun 2016 17:16:38 +0000

Hi Vincent,

My sincere appreciation for the detailed review. You have contributed many
points that will greatly improve the specification. Some responses to your
questions below:

> From: Vincent Roca [mailto:[email-masked]]
> Sent: Thursday, June 16, 2016 11:48 PM
> To: Templin, Fred L <[email-masked]>
> Cc: Vincent Roca <[email-masked]>; Joe Touch <[email-masked]>; [email-masked]; [email-masked]

OK.

> links but with the multiplication of "virtual links" (e.g., tunnels) I wouldn't make

Most end systems connect to media like Ethernet and WiFi with a native 1500 MTU.
Without MSS clamping somewhere in the network, that is the size (minus headers)
that will be propagated to TCP correspondents in the MSS option. If there is a tunnel
in the path, that tunnel would then either have to also somehow configure a 1500 or
larger MTU, or return ICMP PTB messages if it configures a smaller MTU (e.g., 1480).
That is why the text says "or a suitable PTB message returned". Can you suggest
a wording change that might better address your point?

> PMTUD is effective. Is it really what you mean? In that case why don't you say that clearly?

Yes, I can say that - thanks.

> case in previous sections).

We could possibly give examples of well managed administrative domains, e.g., major
corporate enterprise networks, aviation networks, operator networks, etc. Would
adding some examples help? Or, if you think a definition is needed I would be happy
to review any text suggestions you may have.

> I'd suggest to say that explicitly in the I-D.

Yes, I can make this clear in the text. Thanks.

> Don't try to provide an exhaustive list of reasons why a packet won't arrive, it's impossible.

Good point; will fix. Thanks.

> By "minimum number of non-overlapping fragments", I guess you mean two. Say that.

It would be a minimum of two for 1500 and smaller data packets, yes, but we may
need to account for fragmentation of larger control messages. For example, if a
control message in the future needs to carry something like an X.509 certificate
it could become quite large (e.g., 3-4K or even larger). In that case, there could
be more than two fragments.

> at least 2KB. Why 2KB rather than, say, 1500+100? It looks like a magic number.

2KB is 1500 plus ample room for additional encapsulation headers. 2KB is also
a natural memory allocation boundary that should be friendly to memory
management systems in end systems and network devices (i.e., it would
match more naturally with operating system memory page sizes).

> It's out of scope. A lower case "recommended" would be preferable.

Agree. I will fix this.

> "well-managed admin domain". Same comment for the last paragraph.

Right, it sounds like some more examination of well-managed admin domains
in the document would be helpful. I would be happy to review any text 
suggestions you may have.

About expectation of operational ICMP PTB between actors that are in
different administrative domains, what I mean to say is that if the two
administrative domains can only reach each other via the public Internt
then all bets are off and there can be no reliance on ICMP PTB and/or
sufficiently large link MTUs in the path. Niether administrative domain
has the ability to control what goes on in the outside Internet.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 17 Jun 2016 17:32:56 +0000

Hi Vincent,

Although Joe and I may not agree on all points (yet), I am pretty sure that
one point we do agree on is that tunnels will ultimately need to account for
fragmentation of one form or another. Without fragmentation, nested
tunnels within tunnels can only recurse so far until an MTU underrun
is encountered. And, without fragmentation, tunnels cannot support a
minimum MTU if they traverse links with sufficiently small MTUs even
if there is no nesting.

It is true that one possibility is for the tunnel to simply shut down if it
encounters an MTU underrun meaning that one or more destinations
will simply become unreachable.  But, that sort of arrangement may not
be acceptable for safety-critical communications where destinations
should be made reachable through any means available.

Would you agree that tunnels will ultimately need to account for
fragmentation and reassembly?

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 17 Jun 2016 11:19:15 -0700

> minimum MTU if they traverse links with sufficiently small MTUs even
Yup.

> should be made reachable through any means available.
Yes - this is the only end-run that seems to work within the constraints
of existing specs. It does have a downside, though - it basically works
like "the tree in the forest". Until it actually falls, there's no point
in preventing its use. However, once it falls, it cannot get "back up"
again - if it did, it would act like a link that requires a different MTU.

So it's "optimistic" with a fairly bad failure mode. IMO, its' the only
safe way to deploy non-reassembling tunnels, even in a "controlled"
environment -- because "controlled" is only true as long as it can be
known, and the "shut-down" approach serves a a monitor to enforce that
control.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 17 Jun 2016 21:28:09 +0000

Hi Joe,

> From: Joe Touch [mailto:[email-masked]]
> Sent: Friday, June 17, 2016 11:19 AM
> To: Templin, Fred L <[email-masked]>; Vincent Roca <[email-masked]>

It has taken a long time for us to bring this realization to the community,
but I think we are coming to a consensus understanding.

> known, and the "shut-down" approach serves a a monitor to enforce that

As an eternal optimist, that kind of failure mode really bums me out. We
definitely can't use it for airplane comms where we have to ensure safety
of flight even over data links that have sub-Mbps throughput and hence
fragmentation may be the only option. But, I guess in other environments
if you hit the MTU wall there is no choice but to go belly-up...

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 17 Jun 2016 14:47:12 -0700

> if you hit the MTU wall there is no choice but to go belly-up...
My view is that it's much better than just "assume the network has been
engineered properly", largely because all networks may ultimately be
re-deployed over tunnels unknowingly. Better to detect the failure to
support the required MTU than to ignore it.

(yes, I'd be even happier if all networks periodically tested the
minimum MTU as part of operations monitoring)

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 20 Jun 2016 14:52:28 +0000

Hi Joe,

> Sent: Friday, June 17, 2016 2:47 PM

The problem is that the ingress is not the source of the original packets;
it is only the source of the tunneled packets. And, the ingress may need
to encapsulate packets coming from many original sources that may have
many different flow label and/or TOS values. Therefore, there is no way
for the ingress to ensure that any probes it sends will travel over the
same paths that ordinary data packets will travel due to multipath.

This means that the ingress cannot consider receipt of a probe reply as
conclusive evidence that all paths in the multipath can deliver packets
as large as the probe packet. Otherwise, the tunnel could be working
for some data packets but silently black-holing other data packets. This
is why the tunnel ingress cannot employ RFC4821-style MTU probing.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jun 2016 12:06:30 -0700

> same paths that ordinary data packets will travel due to multipath.

For the benefit of others, Fred and I have discussed this before.

My position is that a network path is determined by network header
reachability.

Fred - your point is correct, but it also means that there is absolutely
no packet that can be used a probe for any other packet unless they are
identical in headers, content, and length. If your network "path" is
susceptible to that, then maybe it really should behave unexpectedly.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 22 Jun 2016 20:46:44 +0000

Hi Joe,

> Sent: Wednesday, June 22, 2016 12:07 PM

I guess maybe for the past 15yrs or so? :^}

> identical in headers, content, and length.

That is right. And, if the ingress is asked to encapsulate data packets that
have thousands of distinct flow labels there is no one probe packet that
can tell the whole story. The only alternative is to treat all data packets
as probes and have the network send a PTB message if a probe is lost.
Which is exactly RFC1191/RFC1981 which we know have problems.

> If your network "path" is

It has been suggested to me that some networking gear may inspect not
only the outer packet headers but also inner packet headers when making
ECMP/LAG decisions - deep packet inspection. That means that even if the
ingress used an identical encapsulation header for all data packets the
network could still decide to forward them along various paths of the
multipath.

I don't know how many years it has been that you have been telling me
that fragmentation is inevitable - probably almost as long as I have known
you. I tried everything under the sun to come up with a way to avoid
fragmentation in all cases (see my expired drafts) but I am forced to
conclude that you were right.

Thanks - Fred