Subject: Re: [Int-area] New Version Notification for draft-bonica-intarea-gre-mtu-00.txt

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 2 Jun 2013 19:22:30 +0000

Hi Ron,

Thanks for writing this, and for sending it for review to this list.

Please find my review below, which includes sets of comments and suggestions organized as more substantive / more editorial and prefaced with "CMP:". Hope these are clear and useful, and glad to iterate on them.

More Substantive:
================

       Generic Routing Encapsulation (GRE) Fragmentation Strategy

Abstract

   This memo documents a GRE fragmentation strategy upon which many
   vendors have converged.  Specifically, it defines procedures to be
   executed by GRE ingress routers.  It is published so that those
   ...

CMP: The focus of this document is only point-to-point GRE tunnels,
CMP: and does not include any type of Multipoint GRE tunnels. As such,
CMP: I would make this focus explicit in the Abstract, Intro, and in
CMP: the document title.

1.  Problem Statement
...
   However, implementors and network operators have discovered that some
   fragmentation strategies work better than others.  A poorly chosen
   fragmentation strategy can cause operational issues, including black-
   holing, packet reassembly on GRE egress routers and unexpected
   interactions with Path MTU Discovery [RFC1191] [RFC1981].

CMP: A big part of a generic discussion of this takes place in RFC 4459.
CMP: That's an informational document, that would make of a great
CMP: Informational pointer since Section 3 of RFC 4459 contrasts 
CMP: different approaches.

1.1.  How To Use This Document

   This memo is presented in sections.  Section 2 enumerates design
   goals.  Section 3 defines procedures that all GRE ingress routers
   must execute.

CMP: The impact of this "must" is not clear to me; please see below the
CMP: comments on that section.

1.2.  Terminology

   The following terms are specific MTU discovery:

   o  link MTU (LMTU) - the maximum transmission unit, i.e., maximum
      packet size in octets, that can be conveyed over a link without
      fragmentation

   o  path MTU (PMTU) - the minimum LMTU of all the links in a path
      between a source node and a destination node

CMP: There are two specific clarifications that are important here, I
CMP: believe: i. PMTU is unidirectional, and on a per-direction. So
CMP: in this case is the LMTU from source to destination. ii. there
CMP: is not necessarily "a path", but likely there are multi-paths.
CMP: so given ECMP, this needs to clarify it is of all paths from 
CMP: source to sink (and not *the* path *between* source and sink).

3.2.  Tunnel MTU (TMTU) Discovery

   Implementations MUST maintain a local data structure that reflects
   the TMTU of each GRE tunnel that originates on the node.  The TMTU
   MUST be equal to the PMTU associated with the path between the tunnel
   ingress and the tunnel egress, minus the GRE overhead.

CMP: This "MUST" is only applicable when PMTUD is in effect, which is
CMP: not a MUST for this scenario. I would condition the MUST to "when 
CMP: PMTUD is being used for the tunnel". For completeness, an editorial:
CMP: The way the first sentence is written appears to be too restrictive
CMP: of how to implement TMTU. The TMTU can be kept as an element in the
CMP: existing data structures hosting tunnel endpoint state. 

   By default, implementations MUST discover the PMTU associated with
   the path between the tunnel ingress and the tunnel egress.  PMTU
   discovery procedures defined in [RFC1191] and [RFC1981] and will
   never permit the PMTU to exceed the LMTU associated with the first IP
   hop in the path to the tunnel egress.

CMP: This MUST seems too strict to me. For example, for cases in which
CMP: ICMPs are blocked between the source and destination of the GRE
CMP: tunnel, this requirement can lead to black-holing (instead of 
CMP: suboptimal performance at the destination GRE endpoint), and the 
CMP: medicine would be worst than the sickness. Or put a different way,
CMP: PMTUD MUST work for this to be a MUST -- however, it is impossible
CMP: to ensure PMTUD MUST work if ICMP 3/4s are blocked. And there is 
CMP: no (to my knowledge) PL-PMTUD defined for GRE -- only for TCP.
CMP:
CMP: A nit as well: This is not necessarily a LMTU, but in the case of
CMP: ECMP in the first hop, this is the smallest of the LMTUs associated
CMP: with the paths...

   However, implementations MUST include a configuration option that
   disables PMTU Discovery for GRE tunnels.  This configuration option
   may be required to mitigate certain denial of service attacks (see
   Section 7).  When PMTU discovery for GRE tunnels is disabled, the
   TMTU for a tunnel MUST default to the LMTU associated with the first
   IP hop in the path to the tunnel egress, minus the GRE overhead.
   However, implementations MAY include a configuration option through
   which the TMTU can be set to another value, which is likely to be
   lower.

CMP: This is good for the case of a single GRE tunnel, but it changes
CMP: if the GRE tunnel is protected with IPsec -- unless, the IPsec
CMP: overhead is somehow counted towards the "GRE Delivery Header".

CMP: I believe there is another key requirement for TMTUD for IPv4: the
CMP: configurability of how to set the DF-bit in the Delivery Header.
CMP: Specifically, the DF can be copied from the tunneled packet (if v4)
CMP: or it can always be set. RFC 2003 (which should also be an Info
CMP: pointer) allows for both. A discussion on why setting DF helps 
CMP: speed up the convergence of PMTUD (and consequently of TMTUD)
CMP: would also operators. And a requirement needs to exist for the
CMP: ability to configure this behavior.

CMP: Actually, on third important point as well :-) For PMTUD for GRE
CMP: and TMTUD, when the GRE Tunnel uses GRE Keys, then the Key 
CMP: included in the embedded packet of the ICMP PTB should be checked
CMP: against the configured/used Key in the GRE Tunnel (like with TCP).
CMP: This needs to be called out also in the Security Considerations.

4.1.  Tunneling GRE Over IPv4

   When the GRE ingress router tunnels an IPv4 payload over IPv4, and
   the DF Bit in the payload header is set to 0 (May Fragment), by
   default, the GRE ingress router MUST set the DF bit in the delivery
   header to 1.  However, implementations MAY include a configuration
   option that allows the DF bit to be copied from the payload header to
   the delivery header.

CMP: Wouldn't this be updating RFC 2003? It seems to me that a MUST is
CMP: too strong of a requirement. I can see providing a RECOMMENDED,
CMP: perhaps.

   The GRE ingress router MUST NOT emit a delivery header in which the
   MF bit is set to 1 (More Fragments).

CMP: Similarly, this MUST NOT seems too limiting (as per the explanation
CMP: above). It seems to me that the success of PMTUD with ICMP being 
CMP: best effort delivery is not one to underpin this hard requirement.

5.1.  IPv4 Payloads

   If the DF bit in the payload header is set to 1 (Don't Fragment), the
   GRE ingress router MUST discard the packet and sent an ICMPv4
   [RFC0792] Destination Unreachable message to the payload source, with
   type equal to 4 (fragmentation needed and DF set).  The ICMP
   Destination Unreachable message MUST contain an Next-hop MTU (as
   specified by [RFC1191]) and the next-hop MTU MUST be equal to the
   TMTU associated with the tunnel.

CMP: This is perfect, but for the case in which PMTUD and TMTUD are on.
CMP: This section should condition the requirements to that (because
CMP: it is assuming it is always on).
CMP: I do believe strongly that this is doing the right thing, BTW. I
CMP: am not sure, though, if we can always REQUIRE this.

5.3.  MPLS Payloads

   The GRE ingress router MUST discard the packet.  As it is impossible
   to reliably identify the payload source, the GRE ingress router MUST
   NOT attempt to send an ICMPv4 Destination Unreachable message or an
   ICMPv6 Packet Too Big message to the payload source.

CMP: This is a most serious concern, because mandating PMTUD/TMTUD (
CMP: the MUST in Section 3.2, 5.1, etc.) without a way to fragment,
CMP: and without a way to signal to the source (e.g., ICMP PTB for IP),
CMP: since in the MPLS case there is no source with merging, then this
CMP: "MUST" effectively creates a permanent black-hole for MPLS over GRE.
CMP: 
CMP: Also, I would change this to say something like: if it can be
CMP: reliably determined that the payload of the MPLS packet is IPv4, 
CMP: then it can be prefragged and copy the MPLS header to all fragments,
CMP: quite efficiently. 

7.  Security Considerations

CMP: see comment about about the security consideration of ICMP spoofing.
CMP: This attack is such that makes it hard to justify a "MUST have PMTUD 
CMP: enabled by default" in this document.

7.2.  Attacks Against PMTU Discovery

  PMTU Discovery is vulnerable to two denial of service attacks (see
  Section 8 of [RFC1191] for details).  Both attacks are based upon on
  a malicious party sending forged ICMPv4 Destination Unreachable or
  ICMPv6 Packet Too Big messages to a host.  In the first attack, the
  forged message indicates an inordinately small PMTU.  In the second
  attack, the forged message indicates an inordinately large MTU.  In
  both cases, throughput is adversely affected.  On order to mitigate
  such attacks, GRE implementations MUST include a configuration option
  to disable PMTU discovery on GRE tunnels.

CMP: Lastly, in addition (and better sometimes) than just disable PMTUD,
CMP: a tunnel endpoint should provide the configuration option to set 
CMP: the minimum TMTU that can be set as a result to ICMP PTB. This
CMP: is to protect against the case of an ICMP PTB with too small MTU.

More Editorial:
==============

Abstract

   This memo documents a GRE fragmentation strategy upon which many
   vendors have converged.

CMP: This is not "GRE fragmentation" (i.e., which strictly is having 
CMP: things like "B" and "E" fragmentation bits in the GRE header. 
CMP: THis is a nit, but just for clarity (given that this topic can be 
CMP: confusing), I'd use something like "fragmentation in GRE point-to-
CMP: point tunnel scenarios".

1.  Problem Statement
...
   However, implementors and network operators have discovered that some
   fragmentation strategies work better than others.  A poorly chosen
   fragmentation strategy can cause operational issues, including black-
   holing, packet reassembly on GRE egress routers and unexpected
   interactions with Path MTU Discovery [RFC1191] [RFC1981].

CMP: I would add here a pointer to the discussion in Section 4.1.4 of
CMP: RFC 3931 as part of the discussion there is applicable to other 
CMP: p2p tunnels. This is an Informational pointer.

1.1.  How To Use This Document

   Section 4 defines procedures affecting generation of the GRE delivery
   header.  It is divided into two subsections.  Section 4.1 is
   applicable when GRE is tunneled over IPv4[RFC0791] and Section 4.2 is
   applicable when GRE is tunneled over IPv6 [RFC2460].

CMP: Another nit, since the rest of the doc does such a good job at 
CMP: being precise with terminology: GRE is not "tunneled" over IP. It
CMP: is "delivered".

1.2.  Terminology

   The following terms are specific to GRE and are taken from [RFC2784]:

   o  GRE delivery header - an IPv4 or IPv6 header whose source address
      is that of the GRE tunnel ingress and whose destination address is
      that of the GRE tunnel egress.  The GRE delivery header
      encapsulates a GRE header.

CMP: This, again, describes a p2p GRE tunnel. Multipoint flavors should 
CMP: be explicitly out of scope.

   o  GRE payload - a network layer packet that is encapsulated by the
      GRE header.  The GRE payload can be IPv4, IPv6 or MPLS.

CMP: ... "or other. This document only covers IPv4, IPv6, and MPLS only."

Thanks!

-- Carlos.

On May 29, 2013, at 9:36 AM, Ronald Bonica <[email-masked]> wrote:

> Folks,
> 
> If you get a chance, please provide a sanity check on this document. It is very short, so it won't take much time to review.
> 
>                                         Ron
> 
> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]]
>> Sent: Wednesday, May 29, 2013 9:34 AM
>> To: Ronald Bonica
>> Subject: New Version Notification for draft-bonica-intarea-gre-mtu-
>> 00.txt
>> 
>> 
>> A new version of I-D, draft-bonica-intarea-gre-mtu-00.txt
>> has been successfully submitted by Ron Bonica and posted to the IETF
>> repository.
>> 
>> Filename:	 draft-bonica-intarea-gre-mtu
>> Revision:	 00
>> Title:		 Generic Routing Encapsulation (GRE) Fragmentation
>> Strategy
>> Creation date:	 2013-05-29
>> Group:		 Individual Submission
>> Number of pages: 8
>> URL:             http://www.ietf.org/internet-drafts/draft-bonica-
>> intarea-gre-mtu-00.txt
>> Status:          http://datatracker.ietf.org/doc/draft-bonica-intarea-
>> gre-mtu
>> Htmlized:        http://tools.ietf.org/html/draft-bonica-intarea-gre-
>> mtu-00
>> 
>> 
>> Abstract:
>>   This memo documents a GRE fragmentation strategy upon which many
>>   vendors have converged.  Specifically, it defines procedures to be
>>   executed by GRE ingress routers.  It is published so that those
>>   building new implementations will be aware of best common practice.
>>   It is also published so that those building applications over GRE
>>   will understand how GRE works.
>> 
>> 
>> 
>> 
>> 
>> The IETF Secretariat
>> 
> 
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 2 Jun 2013 19:22:31 +0000

Joe,

On May 29, 2013, at 4:31 PM, Joe Touch <[email-masked]<mailto:[email-masked]>> wrote:

On 5/29/2013 1:06 PM, Linda Dunbar wrote:
Ron,

I do have a few questions and suggestion about the practices documented in the draft:

- Section 4.1, second paragraph:
why DF bit "MUST" set to 1  when the payload header has "0"? I would think default should be same as the "payload" DF setting.

DF in a tunnel header should be set to match the capability of that tunnel mechanism. Either the tunnel wants to find the MTU and match it or not; in the former case, it has to set DF=1 all the time.

For this doc, whether to set or copy the DF bit depends on whether PMTUD is supported on a given tunnel. If it is (whether PMTUD or PLMTUD), then set DF; otherwise, I agree - copy DF.

- Section 5.1, last sentence: "The GRE egree router will forward the
payload fragments to their ultimate destination where there will be
reassembled"

The GRE egress router should reassemble the fragmented payloads
before sending to the destination. In general, the network outside
the Egress router should be same as the network before entering the
Ingress router. If the data frame size is less than then MTU of the
network before entering the Ingress router, the decapsulated frame
size should be less than the MTU of the network outside the Egress
router.

I agree; my general recommendation has always been "the egress should always clean up any mess created by an ingress" - which means using "outer" fragmentation rather than "inner".

Why do you characterize this as a "mess"? Thinking of the tunnel as a logical link with its own logical link MTU (LMTU), and this link is unwilling/incapable of performing fragmentation and reassembly at its "data-link", then the appropriate behavior would be:

  *   If the incoming IPv4 datagram has DF=1, drop and send a PTB back; and
  *   if the incoming IPv4 datagram has DF=0, then fragment it and send it over the link.

What's different?

Yes, this can cause repeated frag/reassembly, but the alternative is to shift work to the end host, which I think is inappropriate.

I do not understand why it is "inappropriate" -- when thinking of the tunnel as a link.

There is also a potential mischaracterization when you say "shift work to the end host", because that can lead to assumptions that there is "a single" end host (singular). In the case of a p2p tunnel, the challenge is that there is a single pair of endpoints in the tunnel, but a multitude of hosts behind and before them. A different view would be that it's more efficient to distribute reassembly to many endpoints instead of attacking the tunnel tailend and making it work on behalf of many hosts/

Thanks,

-- Carlos.

Further, this sort of clean-up is required by IPv6 and for IPv4 when when DF=1 on the inner packet anyway.

In addition, it is the Ingress node split the frame, the final
destination may not be aware of how to reassemble the data frame.

This is a spurious point; the information is there if the inner packet was fragmentable.

- There should be two options when the encapsulated data frame
exceeds MTU: a) split the data frame to two smaller frames, with each
frame being encapsulated; b) truncate the frame if the Egress node
needs to receive the data frame but doesn't have the capability to
reassemble split frames.

There are two viable options:

- fragment and reassemble (same as your option "a)" above)

- drop the packet (and presumably send a signal upstream)

Sending a truncated frame serves no useful purpose.

Joe

_______________________________________________
Int-area mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/int-area

---

From: Joe Touch <[email-masked]>
Date: Mon, 03 Jun 2013 10:43:59 -0700

Hi, Carlos,

...
>> I agree; my general recommendation has always been "the egress should
>> always clean up any mess created by an ingress" - which means using
>> "outer" fragmentation rather than "inner".
>
> Why do you characterize this as a "mess"?

The ingress is creating a situation that takes work to correct (the 
"mess" I refer to).

A. Using outer fragmentation places that work at the egress.

B. Using inner fragmentation pushes that work to the destination.

Note that fragmentation is a lot cheaper (less work) than reassembly. So 
(A) makes it easy for routers to support GRE more cost-effectively, but 
drains things like my iPhone battery as a result.

I don't like that optimization. If you make a mess, IMO you should clean 
it up.

 > Thinking of the tunnel as a
> logical link with its own logical link MTU (LMTU), and this link is
> unwilling/incapable of performing fragmentation and reassembly at its
> "data-link",

There's a *big* difference between incapable and unwilling.

Incapable is fine - drop and send the ICMP.

Unwilling isn't a reason to skip the hard work.

> then the appropriate behavior would be:
>

It's exactly the difference between incapable and unwilling.

Here's the difference for GRE:

	- GRE adds N bytes total (GRE header + IP header)

	- GRE over IP supports 65536-byte packets

So if a packet arrives that is smaller than 65536-N, IMO GRE ought to 
fragment and reassemble it.

If a packet arrives that is larger than that, then GRE *cannot* tunnel 
it, and *MUST* drop it and sent a PTB.

I.e., only 65536 is "too big". Everything else is just whining about 
"bigger than I want it to be" ;-)

>> Yes, this can cause repeated frag/reassembly, but the alternative is
>> to shift work to the end host, which I think is inappropriate.
>

See above.

> There is also a potential mischaracterization when you say "shift work
> to the end host", because that can lead to assumptions that there is "a
> single" end host (singular). In the case of a p2p tunnel, the challenge
> is that there is a single pair of endpoints in the tunnel, but a
> multitude of hosts behind and before them. A different view would be
> that it's more efficient to distribute reassembly to many endpoints
> instead of attacking the tunnel tailend and making it work on behalf of
> many hosts/

It would be an attack, except that you're talking about a tunnel. A 
tunnel is an ingress and an egress. Are you suggesting that making an 
egress do its job is an "attack"?

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 03 Jun 2013 11:35:33 -0700

> why DF bit "MUST" set to 1  when the payload header has "0"? I would think default should be same as the "payload" DF setting.

It depends on the goal:

The current mechanism always sets DF=1 so it (GRE) can discover the 
tunnel MTU.

Setting it to match would expose the endpoint to the overall path MTU, 
but comes at a risk. If a packet is too big somewhere inside the tunnel, 
that router will send an ICMP that goes back to the tunnel ingress -- 
which might not have enough information to relay an ICMP back to the source.

This was discussed here:
http://tools.ietf.org/html/draft-ietf-intarea-tunnels

(this is still a WG doc, but we didn't get much feedback on the issues 
back in 2010 - maybe it's time to revive and complete it??)

...
> frame being encapsulated;

FWIW, fragmentation rarely splits things into equal sizes; it's more 
typical to split into "max that fits" and "leftover".

Joe

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Mon, 3 Jun 2013 23:46:03 +0000

Hi, Joe,

On Jun 3, 2013, at 1:43 PM, Joe Touch <[email-masked]> wrote:

> The ingress is creating a situation that takes work to correct (the "mess" I refer to).
> 
> Note that fragmentation is a lot cheaper (less work) than reassembly. So (A) makes it easy for routers to support GRE more cost-effectively, but drains things like my iPhone battery as a result.
> 

[I think you mean (B) and not (A) in the sentence above]

A different way of looking at it is that a Host is generally more optimized to perform this function than a router. Or, if you prefer the view that tunnel endpoints act as hosts for the Tunnel (delivery), then a "host" is more "host" than a router for reassembly.

Frankly, for the case of draining smartphone battery, you could just use IPv6 or set DF.

> I don't like that optimization. If you make a mess, IMO you should clean it up.
> 

I understand you have a preference and you do not like it. On the other hand, it is still a valid case. It is documented, for example, in the second para of S4.1.4 of RFC 3931 http://tools.ietf.org/html/rfc3931#section-4.1.4

> So if a packet arrives that is smaller than 65536-N, IMO GRE ought to fragment and reassemble it.
> 
> If a packet arrives that is larger than that, then GRE *cannot* tunnel it, and *MUST* drop it and sent a PTB.
> 
> I.e., only 65536 is "too big". Everything else is just whining about "bigger than I want it to be" ;-)

[For scope, this is an IPv4 discussion; with IPv4, the mechanism for this is to set DF.]

If DF is not set, which specification defines that a packet cannot be fragmented? It really is a trade-off; and as such, IMHO, there is not one-size-fits-all but there are deployment decisions and practices.

>> I do not understand why it is "inappropriate" -- when thinking of the

I am still not sure how "inappropriate" is a technical term. I do understand your preference and I do follow the logic. My point is that it does not seem to be a mandated behavior by a spec., and seems to be a trade-off to be documented with pros and cons for this case.

Let me try a different example, and let me know which step you disagree with :-)

0. GRE Tunnel between R1 and R2, tunneling IPv4 from end hosts.
1. The delivery (encapsulating) header sets DF (as allowed by RFC791, for example because R2 does not have sufficient resources to reassemble internet fragments.)
2. The GRE Tunnel is realized as a logical interface.
3. A 2000 octet IPv4 datagram from an end host has the GRE tunnel as the next-hop; R1 encapsulates in GRE, then IPv4 with DF set, and then when trying to send the datagram sees that it is larger than the physical MTU of the out interface.
4. R1 sends an ICMPv4 PTB to itself and drops the datagram, and the tunnel learns its MTU.
5. Another datagram is received for the tunnel, with the DF bit clear, what should R1 do? -> fragment the encapsulated packet and encapsulate it.

> It would be an attack, except that you're talking about a tunnel. A tunnel is an ingress and an egress. Are you suggesting that making an egress do its job is an "attack"?

A different perspective: If the tunnel is a link in a logical topology, then this preference would be equivalent to asking every link to perform data-link-level fragmentation and reassembly and allow any size packets.

Thanks,

-- Carlos.

---

From: Joe Touch <[email-masked]>
Date: Mon, 03 Jun 2013 17:10:37 -0700

> [I think you mean (B) and not (A) in the sentence above]

Yup.

> A different way of looking at it is that a Host is generally more
> optimized to perform this function than a router.

Some host are; others (like my phone) do this at a heft cost (to me).

Some routers can do this (ones that are designed well); others cannot 
(like ones that claim they support GRE but really don't because they 
don't implement at-rate reassembly).

> Or, if you prefer the
> view that tunnel endpoints act as hosts for the Tunnel (delivery), then
> a "host" is more "host" than a router for reassembly.

I disagree; the router liked being a host when it encapsulated the 
packets. That includes generating unique IPv4 IDs (if DF=0), generating 
checksums, etc. - that part it's OK with. If it wants to play host 
there, it can play host at the egress too.

> Frankly, for the case of draining smartphone battery, you could just use IPv6 or set DF.

I can't set DF for incoming packets, nor can I force use of IPv6 unless 
I disable my IPv4 address.

> I understand you have a preference and you do not like it. On the
> other hand, it is still a valid case. It is documented, for example, in
> the second para of S4.1.4 of RFC 3931
> http://tools.ietf.org/html/rfc3931#section-4.1.4

There are lots of things documented in many IETF protocols that aren't
always the best way to do things. This may be just another good example 
of that.

...
...
> [For scope, this is an IPv4 discussion; with IPv4, the mechanism for this is to set DF.]

DF means "don't fragment this packet and deliver the fragments". It's 
always the choice of a capable link layer to fragment and then 
reassemble packets.

E.g., DF=1 can go over ATM cells just fine; ATM fragments and 
reassembles the packet within ATM. A tunnel ought to behave the same way.

The exception is when the tunnel - or link - cannot fragment and 
reassemble. That's when the source should drop and send PTB.

> If DF is not set, which specification defines that a packet cannot
> befragmented?

Nothing says you cannot; I claim that you should not unless you HAVE to. 
Tunnels like GRE don't have to.

> It really is a trade-off; and as such, IMHO, there is not
> one-size-fits-all but there are deployment decisions and practices.
>
> I am still not sure how "inappropriate" is a technical term. I do
> understand your preference and I do follow the logic. My point is that
> it does not seem to be a mandated behavior by a spec., and seems to be a
> trade-off to be documented with pros and cons for this case.

The spec for DF behavior has a purpose - when DF=1, it's to help 
endpoints discover ways to adjust their MTU so packets can traverse 
links that *cannot* otherwise be traversed by larger packets.

When DF=0, it's to help endpoints traverse paths by allowing a packet 
that *cannot* traverse a link to be broken down and reassembled at the 
destination.

Nothing about DF says "hey, fragment anytime you feel like it". RFC1812 
in specific says you're not supposed to fragment unless you have to. 
(that's the relevant RFC from the viewpoint of the traversing packet - 
the tunnel looks like a link to a router).

My complaint for GRE is that there are two different MTUs - a 'native' 
MTU that may describe efficiency but does NOT limit traversal, and an 
true MTU (payload - N, where N is the size of the GRE+IP header). Using 
the latter is the only one that complies with RFC1812.

> 0. GRE Tunnel between R1 and R2, tunneling IPv4 from end hosts.

OK.

> 1. The delivery (encapsulating) header sets DF (as allowed by
> RFC791,for example because R2 does not have sufficient resources to reassemble
> internet fragments.)

This is the incorrect step; either DF=0 for IPv4 in general, or DF=1 and 
you run PLMTUD so you can generate (outer) fragments that will traverse 
the path correctly (e.g., as would be needed for IPv6, but would also be 
useful for IPv4).

> 2. The GRE Tunnel is realized as a logical interface.

Sure.

> 3. A 2000 octet IPv4 datagram from an end host has the GRE tunnel as
> the next-hop; R1 encapsulates in GRE, then IPv4 with DF set, and then
> when trying to send the datagram sees that it is larger than the
> physical MTU of the out interface.

There are two cases:

- you are running PLMTUD in background, at which point you already knew 
this was going to happen at the GRE ingress, and you should have 
fragmented correctly at the ingress (IPv6-like situation)

or

- your ingress should have set DF=0

> 4. R1 sends an ICMPv4 PTB to itself and drops the datagram, and the tunnel learns its MTU.

Only if the ICMP error is correctly received.

> 5. Another datagram is received for the tunnel, with the DF bit
> clear,what should R1 do? -> fragment the encapsulated packet and encapsulate it.

It should have done the same thing with both packets.

>> It would be an attack, except that you're talking about a tunnel.
>> Atunnel is an ingress and an egress. Are you suggesting that making an
> A different perspective: If the tunnel is a link in a logical
> topology, then this preference would be equivalent to asking every link
> to perform data-link-level fragmentation and reassembly and allow any
> size packets.

Up to the size it *cannot* handle as payload, yes.

Joe

---

From: Ronald Bonica <[email-masked]>
Date: Wed, 5 Jun 2013 17:00:55 +0000

Carlos,

Thanks for this thoughtful review. All comments accepted. Expect an updated version of the draft next week.

                                                 Ron

> From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
> Sent: Sunday, June 02, 2013 3:23 PM
> To: Ronald Bonica
> Cc: Internet Area
> Subject: Re: [Int-area] New Version Notification for draft-bonica-
> intarea-gre-mtu-00.txt
> 
> Hi Ron,
> 
> Please find my review below, which includes sets of comments and
> suggestions organized as more substantive / more editorial and prefaced
> with "CMP:". Hope these are clear and useful, and glad to iterate on
> them.
> 
> 
> More Substantive:
> ================
> 
> > If you get a chance, please provide a sanity check on this document.
> It is very short, so it won't take much time to review.
> >
> >                                         Ron
> >
> >

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 24 Jun 2013 21:16:18 +0000

Hi Ron and Carlos,

Here are my comments on this draft:

Fred
[email-masked]

Section 1.2:
************
Under the bullet: "o Path MTU Discovery (PMTUD)", should add a
trailing citation such as:

  "[RFC2923] documents operational issues with PMTUD that apply
   equally to IPv4 and IPv6."

Section 2.1:
************
Under the heading: "In Strategy 1", this seems to be taking a view
that applies only to IPv4 (and not IPv6). Suggest rewriting as:

  "In Strategy 1, the tunnel ingress router encapsulates the entire
   payload into a single GRE-delivery packet, then fragments the packet
   if necessary and forwards each fragment in the direction of the tunnel
   egress. For IPv4, if any fragment exceeds the LMTU of any link along
   the path to the tunnel egress, the router directly upstream of that
   link fragments it (further). The tunnel egress router reassembles
   the GRE-delivery packet, de-encapsulates its payload, and processes
   the payload appropriately."

Section 2.1:
************
Under the heading: "In Strategy 2", this again seems to be taking a
view that applies only to IPv4. Suggest rewriting as:

  "In Strategy 2, the tunnel ingress router performs PMTUD procedures
   or some variant thereof (e.g., PLMTUD). When the tunnel ingress
   router receives a non-fragmentable packet that is larger than the
   minimum IP MTU and also so large that it cannot be forwarded through
   the tunnel, it discards the packet and sends an ICMPv4 "fragmentation
   needed" [RFC0792][RFC1191] or an ICMPv6 "packet too big" [RFC1981]
   message to the packet source. The ICMP message contains a Next-hop
   MTU equal to the TMTU associated with the tunnel. If the ICMP message
   reaches the packet source, and if the packet source executes PMTUD
   procedures, the packet source adjusts its PMTU for the packet
   destination and emits subsequent packets with size less than
   the TMTU."

Section 2.1:
************
Under the heading: "In Strategy 3", change: "less than the TMTU of
any tunnel" to: "greater than the TMTU of any tunnel".

Section 2.1:
************
Under the heading: "Strategy 1 is an attractive alternative to
Strategy 1...", Suggest rewriting to:

  "Strategy 1 is an attractive alternative to Strategy 2, because
   it does not rely on PMTU. However, Strategy 1 may not be
   operationally practical in all environments because it assigns
   the task of reassembly to the tunnel egress router. When the
   tunnel supports high data rates, reassembly at the tunnel egress
   can be cost-prohibitive. For IPv4, the tunnel ingress must also
   monitor the rate at which it sends in order to avoid reassembly
   misassociations [RFC4963]."

Section 2.2:
************
This section seems to forget that the tunnel MUST support the
minimum MTU for IPv4/IPv6. So, if PMTUD within the tunnel is
broken, and if the tunnel receives a non-fragmentable packet
that is no larger than minMTU but larger than the tunnel can
pass in one piece, the tunnel ingress MUST encapsulate the
packet and perform fragmentation on the encapsulated packet
before sending each fragment toward the egress. The same is
true for both IPv4 and IPv6, and the text needs to say this.

Section 3.2:
************
The statement: "When PMTUD is disabled, the TMTU MUST be set to a
value that is less than or equal to the LMTU associated with the
next-hop towards the tunnel egress, minus the GRE overhead."

In this statement, how can the tunnel ingress discover the smallest
LMTU of all links on the path to the tunnel egress if PMTUD is
disabled? Only options are to either do some sort of out-of-band
path probing, or to assume the minimum MTU (1280 for IPv6 or 576
for IPv4). The text doesn't currently say this, but needs to.

Section 4.1:
************
This section is missing the mark in that it does not talk at
all about the minimum MTU the tunnel MUST support. Packets
that are no larger than the IPv4/IPv6 minimum MTU MUST be
admitted into the tunnel even if it means some fragmentation
may be necessary.

Section 4.2:
************
This section needs re-working. For tunnels over IPv6, if the
packet is no larger than the IPv6 minMTU (1280 bytes) but
larger than TMTU, then the tunnel ingress MUST fragment it
and have the fragments reassembled by the tunnel egress; it
has no other choice if it means to be compliant with RFC2460.

Section 5.1:
************
This section does not currently honor the IPv4 minMTU of 576
bytes. In particular, for packets that are no larger than the
IPv4 minMTU the tunnel ingress MUST ensure that the packet is
delivered to the tunnel egress even if some fragmentation is
necessary.

Section 5.2:
************
This section does not currently honor the IPv6 minMTU of 1280
bytes. In particular, for packets that are no larger than the
IPv6 minMTU the tunnel ingress MUST ensure that the packet is
delivered to the tunnel egress even if some fragmentation is
necessary.

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 25 Jun 2013 14:04:38 +0000

Ron, 

Your draft recommends that the ingress node discards the frame and sends ICMP msg to the source node when the size of GRE encapsulated frame exceeds link MTU. 
We experienced that Window XP doesn't adjust frame size after receiving ICMP message. Many Linux based applications don't do anything either with the MTU ICMP message. I think your draft should point those issues out and recommend fragmentation approach under those circumstances. 

Linda

---

From: Joe Touch <[email-masked]>
Date: Tue, 25 Jun 2013 09:49:01 -0700

On 6/25/2013 7:04 AM, Linda Dunbar wrote:
> Ron,
>
> Your draft recommends that the ingress node discards the frame and
> sends ICMP msg to the source node when the size of GRE encapsulated
> frame exceeds link MTU.
> We experienced that Window XP doesn't adjust frame size after
> receiving ICMP message. Many Linux based applications don't do anything
> either with the MTU ICMP message. I think your draft should point those
> issues out and recommend fragmentation approach under those circumstances.

Do these apps set DF=1? If so, they ought to be prepared to handle ICMP 
PTB messages. If not, they shouldn't be setting PTB.

However, my view remains that PTB is for when a link cannot handle a 
packet - not when it chooses not to do so electively. The only packet 
that ought to be too big for GRE is one that exceeds the max payload of 
IP after encapsulation. Others ought to be fragmented at the GRE ingress 
and reassembled at the GRE egress at the outer IP layer, exactly because 
IP *can* do this.

Joe

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Tue, 25 Jun 2013 17:22:35 +0000

Joe,

On Jun 25, 2013, at 12:50 PM, "Joe Touch" <[email-masked]> wrote:

> Do these apps set DF=1? If so, they ought to be prepared to handle ICMP PTB messages. If not, they shouldn't be setting PTB.
> 
> However, my view remains that PTB is for when a link cannot handle a packet - not when it chooses not to do so electively. The only packet that ought to be too big for GRE is one that exceeds the max payload of IP after encapsulation. Others ought to be fragmented at the GRE ingress and reassembled at the GRE egress at the outer IP layer, exactly because IP *can* do this.

If a physical link has an MTU of 1500, can you configure it and set it to be 1300, despite the hardware being able to do 1500?

Are you suggesting that a GRE tunnel has a Tunnel MTU of 65535 - encap always?

> Joe

Thumb typed by Carlos Pignataro.
Excuze typofraphicak errows

---

From: Joe Touch <[email-masked]>
Date: Tue, 25 Jun 2013 11:13:22 -0700

> Are you suggesting that a GRE tunnel has a Tunnel MTU of 65535 - encap always?

A GRE tunnel, IMO, has the MTU of whatever its outer encapsulation is.

I.e., GRE over IP with no options has an MTU of 65535-40-{gre_size}. 
Beyond that it *cannot* reassemble.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 25 Jun 2013 19:08:19 +0000

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Joe Touch
> Sent: Tuesday, June 25, 2013 11:13 AM
> To: Carlos Pignataro (cpignata)
> Cc: Ronald Bonica; Internet Area
> >> Do these apps set DF=1? If so, they ought to be prepared to handle
> ICMP PTB messages. If not, they shouldn't be setting PTB.
> >>
> IP after encapsulation. Others ought to be fragmented at the GRE
> ingress and reassembled at the GRE egress at the outer IP layer,
> exactly because IP *can* do this.
> >
> > If a physical link has an MTU of 1500, can you configure it and set
> it to be 1300, despite the hardware being able to do 1500?
> >

I have a slightly different view, but one that still involves
fragmentation. The GRE tunnel MUST observe the minimum IPv6 MTU
of 1280 bytes. But, to uphold the de facto Internet cell size of
1500 bytes we should extend this up to 1500. So, if the MTU within
the tunnel is unknown and/or unknowable, then the ingress MUST use
fragmentation to make sure that packets no larger than 1500 bytes
get through - the egress then gets to reassemble.

Packets larger than 1500 can be admitted into the tunnel unfragmented
and allowed to sink or swim on their own. The ingress may or may not
get ICMP PTBs if one of these is dropped, but the source should be
using RFC4821 when it sends packets larger than 1500 bytes anyway,
so loss of PTBs should not affect the hosts.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Tue, 25 Jun 2013 14:38:00 -0700

Hi, Fred,

...
> of 1280 bytes.

In what it sends over IP (i.e., in the chunks it uses to transit from 
ingress to egress). There's no reason to impose that limit on what it 
receives at the ingress.

> But, to uphold the de facto Internet cell size of

This is an artificial limit that is unnecessary.

IMO, MTU is just that - "MAXIMUM". Not 'preferred'. So if GRE *can* 
transit packets arriving at the ingress that are 65535-IP-GRE, why 
should it do otherwise?

(yes, it *can* be configured lower if desired, just as with any other 
link, but there's no reason to assume 'lower than maximum' as a default)

> and allowed to sink or swim on their own.

Whether and how much GRE fragments what it receives at an ingress ought 
to be determined by its understanding of its own ingress-egress MTU 
capability. However, those fragments are not directly related to what 
the tunnel can transit *because* the ingress can fragment and the egress 
can reassemble.

Acting like the ingress and egress can't participate in frag/reassembly 
of the outer packet introduces an artificial limit to GRE tunnels. If 
that's what you want to do - or what has already been done - that's 
fine, but *that* is what is suboptimal.

 > The ingress may or may not

Agreed on that point, but not the 1500-byte transit limit.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 25 Jun 2013 22:01:23 +0000

Hi Joe,

> From: Joe Touch [mailto:[email-masked]]
> Sent: Tuesday, June 25, 2013 2:38 PM
> To: Templin, Fred L
> Cc: Carlos Pignataro (cpignata); Ronald Bonica; Internet Area

What I am suggesting is that the tunnel ingress sets an indefinite
MTU, meaning that all packets are admitted into the tunnel regardless
of their size. But, see more below... 

> This is an artificial limit that is unnecessary.

What I am after is a fragmentation threshold designed to set an
upper bound on the amount the tunnel egress will have to reassemble.
This also happens to be the size of packets that MUST make it through
the tunnel. IPv6 says that this value must be set to 1280 at a minimum.
I am saying that it should be bumped up to 1500 to mirror the minimum
MTU used on the vast majority of links in the Internet today. But, see
more below...
 
> transit packets arriving at the ingress that are 65535-IP-GRE, why

What I am saying is that all packets are admitted into the tunnel
regardless of their size, but the tunnel will use fragmentation only
for those packets that MUST make it through to the other side, i.e.,
all packets that are no larger than 1500 bytes. This means that the
tunnel egress need only configure a reassembly buffer size of 1500
bytes plus the tunnel overhead. Packets larger than 1500 bytes will
still make it through the tunnel (in one piece) if the MTU within the
tunnel can accommodate that size. But, the most the ingress will ever
ask the egress to reassemble is 1500 (plus overhead).
 
> the tunnel can transit *because* the ingress can fragment and the

Right. The egress has to reassemble whatever the ingress fragments.
What I am saying is that the ingress put a conservative upper bound
on the amount it will ever ask the egress to reassemble.

> fine, but *that* is what is suboptimal.

No, I fully agree that *some* fragmentation and reassembly is
necessary. I'm just saying that the ingress should set a limit
on the most it would ever ask the egress to reassemble. But,
that does *not* put an artificial limit on the tunnel MTU. 

> Agreed on that point, but not the 1500-byte transit limit.

Please reread the algorithm I proposed, Joe. What I am suggesting
is an "infinite" tunnel ingress MTU - not a 1500 byte MTU.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Tue, 25 Jun 2013 15:19:29 -0700

Hi, Fred,

> MTU used on the vast majority of links in the Internet today. But, see

You're introducing an artificial limit based on the assumption that GRE 
should never transit more than the minimum, but there's no reason for 
that limit; GRE is sourcing packets to the egress, and can easily 
generate and reassemble fragments. The rest of the path may impose a 
limit - such as based on transiting Ethernet, but there's no reason to 
introduce that limit here.

> all packets that are no larger than 1500 bytes.

Why? Why are you introducing that as an artificial constraint for GRE?

> This means that the

IP reassembly requires tsupport for max-size IP packets, not just ones 
whose sizes you like ;-)

> on the amount it will ever ask the egress to reassemble.

If the egress supports IPv6, the upper bound is already 66635. Why are 
you introducing a lower constraint when that's already required?

> that does *not* put an artificial limit on the tunnel MTU.

But that's what you're doing; you will reject packets larger than 1500 
that transit, even though you can reassemble them (and you have to if 
you implement IPv6 reassembly anyway).

I understand what you're trying to save, but an IP encapsulation egress 
is still required to support IPv6 host-side capabilities, so there's no 
savings.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 25 Jun 2013 22:32:44 +0000

Hi Joe,

The constraint you seem to be missing is that RFC2460 only requires
IPv6 nodes to reassemble as much as 1500 bytes. Nodes are permitted
to reassemble more than that, but 1500 is all that can be expected
unless there is specification stating that it needs to be more.

Granted, my algorithm is asking GRE egress tunnel endpoints to
configure a reassembly buffer size of 1500 *plus* the tunnel
encapsulation overhead, which is more than RFC2460 ensures.
But, it will never ask for more than that.

Also think about it this way. Why should the tunnel work hard to
pass a packet that is larger than 1500 bytes and also larger than
the minimum link MTU within the tunneled path when the source could
instead adjust the size of the packets it is sending downward?
Such "jumbos" should be dropped by the tunnel and not accommodated
by a resource-intensive process such as frag/reass. "Take care of
the smalls, and let the bigs take care of themselves."

BTW, the algorithm I have articulated is exactly the SEAL algorithm.

Thanks - Fred

> Sent: Tuesday, June 25, 2013 3:19 PM
> To: Templin, Fred L

---

From: Joe Touch <[email-masked]>
Date: Tue, 25 Jun 2013 20:10:02 -0700

Hi, Fred,

On 6/25/2013 3:32 PM, Templin, Fred L wrote:
> IPv6 nodes to reassemble as much as 1500 bytes.

Right; sorry for not noticing that further. Had IPv4 on the brain.

...
> encapsulation overhead, which is more than RFC2460 ensures.

Agreed.

...
> instead adjust the size of the packets it is sending downward?

That's not a good argument. I agree that all that need be satisfied is 
the largest reassembled IPv6 packet, but not that we should assume the 
source can or will adjust downward - if that were the case, you could 
ignore the overhead of GRE and just claim that the MTU is 1280.

Joe

---

From: Ronald Bonica <[email-masked]>
Date: Wed, 26 Jun 2013 12:59:52 +0000

Linda,

What do these platforms do when they receive an ICMP Packet too big from a router that isn't a tunnel ingress?

                                            Ron

> From: Linda Dunbar [mailto:[email-masked]]
> Sent: Tuesday, June 25, 2013 10:05 AM
> To: Joe Touch; Carlos Pignataro (cpignata)
> Subject: RE: [Int-area] New Version Notification for draft-bonica-
> intarea-gre-mtu-00.txt
> 
> Ron,
> 
> We experienced that Window XP doesn't adjust frame size after receiving
> ICMP message. Many Linux based applications don't do anything either
> with the MTU ICMP message. I think your draft should point those issues
> out and recommend fragmentation approach under those circumstances.
> 
> Linda
> 
> 
>

---

From: Bob Hinden <[email-masked]>
Date: Wed, 26 Jun 2013 16:18:10 +0300

Linda,

On Jun 25, 2013, at 5:04 PM, Linda Dunbar <[email-masked]> wrote:

> We experienced that Window XP doesn't adjust frame size after receiving ICMP message.

Given's Windows XP's minimal IPv6 implementation (even assuming it actually installed by the user), I don't think it's a good to platform to compare against.  Windows 7 or 8 make more sense.

Bob

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 26 Jun 2013 15:42:04 +0000

Hi Joe,

> Sent: Tuesday, June 25, 2013 8:10 PM
> To: Templin, Fred L

OK, but IPv4 also has a limit of minMRU=576. So, we have:

  IPv4 minMTU = 576 (*)
  IPv4 minMRU = 576
  IPv6 minMTU = 1280
  IPv6 minMRU = 1500

(*) Even though the specs say that IPv4 minMTU = 68, everyone
seems to be saying that for practical purposes it is now 576.

> > encapsulation overhead, which is more than RFC2460 ensures.

Right. So, then the fragmentation threshold for GRE over IPv6
needs to be (1500 - HLEN) instead of 1500.
 
> ignore the overhead of GRE and just claim that the MTU is 1280.

In my lab, my implementation sends a single packet for packet
sizes no larger than (1280 - HLEN). For packets that are larger
than (1280 - HLEN) but no larger than 1500, it fragments the
packet into two pieces and sends both pieces to the tunnel far
end which then needs to reassemble. But, reassembly will be
delayed by the inter-packet delay plus the time it takes to
run the reassembly routine, and the delay is measurable. So,
we really only want to use frag/reass when we absolutely have to.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 26 Jun 2013 16:09:07 +0000

Authors,

I have already worked out the details of the algorithms your draft
is seeking in SEAL:

http://datatracker.ietf.org/doc/draft-templin-intarea-seal/

There are exactly two differences between SEAL and GRE:

  - SEAL does not use IP frag/reass; it uses its own frag/reass
    mechanisms at a layer above IP

  - SEAL requires that the tunnel egress reassemble enough so
    that all packets that are no larger than 1500 bytes will
    be accommodated unconditionally. With GRE, the best the
    tunnel ingress can assume is that the egress has a 1500
    byte reassembly buffer, which is not big enough to hold
    a 1500 byte packet plus the encapsulation overhead

I will say again that for IPv6 the GRE tunnel MUST accommodate
packet sizes up to 1280 as specified in RFC2460. And, when the
path over which the tunnel is configured only has a 1280 MTU
the GRE tunnel MUST use IP frag/reass to meet the IPv6 minMTU
requirement. But, we are seeing in other discussions that both
IPv6 frag/reass and IPv6 PMTUD may not work very well in some
environments.

Thanks - Fred

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 26 Jun 2013 20:09:35 +0000

Ron, 

Those applications ignore the ICMP messages and keep sending data frames as if there is no ICMP message received. 
We ran into this problem in testing Video distribution system. After the first few frames, the receivers can't receive anything. After lengthy debugging, we found out that an intermediate node drops packets when GRE encapsulated frames exceed the link MTU. We tested that ICMP message is sent back, but the source node didn't respond. We tried several setting as Joe pointed out. Some guest OSs change behavior, some don't.  

Linda

> From: Ronald Bonica [mailto:[email-masked]]
> Sent: Wednesday, June 26, 2013 8:00 AM
> To: Linda Dunbar; Joe Touch; Carlos Pignataro (cpignata)
> Cc: Internet Area
> What do these platforms do when they receive an ICMP Packet too big
> from a router that isn't a tunnel ingress?
> 
>                                             Ron
> 
> 
> receiving