Subject: [nvo3] questions about PMTU discovery and encapsulations

From: Alia Atlas <[email-masked]>
Date: Thu, 11 Aug 2016 20:53:09 -0400

I have been reviewing the three encapsulations and have several comments on
each.
One aspect, however, seems to have not been discussed at all and I would
like to highlight it here.
This primarily applies to Geneve and GUE (though GUE does add MTU
fragmentation at the encapsulator as an option), but is also not described
for VXLAN-GPE although PMTU discovery and setting of the DF bit for IPv4
packets is required.

I found draft-saum-nvo3-pmtud-over-vxlan-03 to have some useful pieces of
the problem
and in specific to articulate the need for the VTEP to translate and relay
ICMP messages back to the VM so that the MTU can be adjusted.

In draft-ietf-nvo3-geneve-02 Sec 4.1, the use of PMTU discovery is
recommended.
Even if a router in the underlay is implementing RFC 4884, this guarantees
at most 128 bytes of the incoming packet data.  The expectation is that
this will include all necessary headers.  In this case, that needs to be
enough for the VTEP to be able to relay and translate the ICMP message back
to the VM.

However, for Geneve and GUE, there is a real risk that 128 bytes is not
sufficient to include the encapsulated IP header.     If the underlay is
IPv6, that takes up 48 bytes for the IPv6 header plus the UDP header.   The
base Geneve header is 12 bytes and an encapsulated IPv6 header is another
40.   That leaves at most 28 bytes for the Geneve options - which is to say
that at most 3 can fit.   Of course, the situation with IPv4 is better - in
that case 68 bytes are available for Geneve options.

It is likely, of course, that there are a number of implementations that
simply put in as much of the triggering packet as possible, but I'm not
convinced that is an assumption that we can make.

What solutions do you see in this space?  Is there a reason that this
wouldn't be an issue?
The normal assumption is that the VMs would just be given an MTU that
assumes the encapsulation; with variable options, I can guess that
misconfigurations might happen as the options included changed.

Regards,
Alia

---

From: Tom Herbert <[email-masked]>
Date: Fri, 12 Aug 2016 08:41:19 -0700

On Thu, Aug 11, 2016 at 5:53 PM, Alia Atlas <[email-masked]> wrote:
> at most 128 bytes of the incoming packet data.  The expectation is that this
> will include all necessary headers.  In this case, that needs to be enough
> for the VTEP to be able to relay and translate the ICMP message back to the
> VM.
>
Alia,

There has been a lot of discussion on tunnel MTU and fragmentation
(draft-ietf-intarea-tunnels, RFC4459). The conclusion seems to be that
fragmentation over tunnels is necessary and that in most deployments
the MTUs can be configured to avoid fragmentation.

For GUE, if the PMTUD were run for the tunnel, the ICMP PTB message
could only refer to the tunnel. So PTB would result in the MTU at the
tunnel ingress point to be lowered. Subsequent packets would from the
source would then get PTB from the NVE.

There is another problem though with PMTUD, with the added headers of
encapsulation the path MTU for the overlay network could fall below
1280 which is the minimum MTU for IPv6. We cannot send PTB message for
IPv6 packets that are already 1280 bytes or less.

Also, the GUE payload will often encrypted anyway so the NVE wouldn't
be able to access the inner packet in an ICMP message in that case
even if the headers were small.

Given all this we don't recommend PMTUD be run for the underlay
network of tunnels, instead we implement encapsulation level
fragmentation (draft-herbert-gue-extensions). We have real cases where
we can't control the MTUs (like at a POP) so we need to fragment. This
has not become a major issue. Such fragmentation is always exactly two
fragments, packet loss in the backend network is low, and fragments
are delivered in order-- we're not seeing the problem of inordinate
memory consumption for reassembly.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Aug 2016 14:45:59 -0700

> the MTUs can be configured to avoid fragmentation.
FWIW, some of the conclusions of intarea-tunnels will likely update some
of the recommendations in RFC 4459 (as per section 5.5.5 of
intarea-tunnels).

There appears to be consensus for all of this in INTAREA, FWIW (after
many years of debate on the topic). The key rules are in Sec 4.2, with a
pending update to add special handling for IPv6 in IPv4.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Aug 2016 14:46:59 -0700

On 8/12/2016 8:41 AM, Tom Herbert wrote:
> (draft-ietf-intarea-tunnels, RFC4459). 
This doc discusses much more than just MTU and fragmentation, FWIW.

Joe