Subject: Re: [nvo3] New draft: PMTUD Over Vxlan

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Wed, 1 Jul 2015 11:51:58 +0000

Hello All,

We have submitted version-01 of "draft-saum-nvo3-pmtud-over-vxlan².
The draft outlines existing problems and a potential implementable
solution to pmtud specific issues in vxlan network,
without tinkering with existing standards.
Version-01 also attends to comments provided by Joe.

https://datatracker.ietf.org/doc/draft-saum-nvo3-pmtud-over-vxlan/

Please provide your comments and suggestions on the same.

Thanks and Regards,
Saumya.

---

From: Joe Touch <[email-masked]>
Date: Wed, 01 Jul 2015 10:50:00 -0700

On 6/30/2015 8:29 PM, Saumya Dikshit (sadikshi) wrote:
> Hi Joe,
> 
> Thanks for your comments. I will surely remove these references from
> draft. 
> Those were more of typo misses and shouldn¹t have been there in first
> place.
> 
> The idea in this draft is to provide a solution, wherein
> Vxlan gateway can translate the icmp errors (either PTB or Fragmentation)
> generated in the underlay and pass it on to the end-point device.

If that is your goal, then it should focus on the specific issue of ICMP
translation in the title. However, it's important to understand exactly
how this is unique or different for Vxlan vs any other IP encapsulation.

If it isn't provably different, then IMO this WG should leave it alone
and cite existing specs.

Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Wed, 1 Jul 2015 18:38:38 +0000

Hi Joe,

Vxlan is a Layer-2 tunnel and carries layer-2 frame(complete packet
including l2 header) as payload.
Hence the encap/decap and information derivation differs from L3-tunneling
mechanism. 
L2 payload can contain non-IP encapsulation as well (ARP etc)

Secondly, Vxlan Gateway can potentially act as a Layer-2 gateway and
Layer-3 gateway for client devices
connected over same or different subnet respectively. For Layer-2 gateway
case, MTU derivation may not
make sense, as  the destination is one L3-hop away. Though this is
contentious, and it can still be
implemented to allow judicious use of resources (computational and
bandwidth) in core network ,
but the end point device should not be expecting an icmp error in this
case.

If we venture out in l2 gateway scenarios, then there may be cases of
Vxlan packet encap is done with Outer
Destination IP as multicast address bound to the vni. Though I can recheck
its feasibility for L3-gateway case.

Vxlan gateways in a typical datacenter deployment may act as ToR and hence
may have anycast address configuration.
That needs to be avoided while generating ICMP errors in underlay network
from these devices. Although anycast configuration should be
client facing and not part of the underlay.

Thanks
Saumya.

---

From: Joe Touch <[email-masked]>
Date: Wed, 01 Jul 2015 11:45:07 -0700

> including l2 header) as payload.

If that is correct, then it has no business interacting with ICMP.

It should be supporting the minimum required transit MTU for IPv4 or
IPv6 (respectively), and where that's not directly possible it must
support fragmentation and reassembly on its own.

...
> make sense, as  the destination is one L3-hop away. 

IP hops are measured by the number of routers, not link. I.e.,
traversing one hop is one router, not one link. See RFC1812.

Although I appreciate you're trying to optimize, ICMP PTBs are for "I
can't carry this message at all", not "I wish it were smaller". If you
try to interpret the PTBs incorrectly, you'll only create black holes.

Further, the use of PMTUD is deprecated in favor of PLMTUD (RFC4821),
precisely so the path properties to not rely on ICMPs - which are often
blocked anyway.

Again, these comments are NOT intended as issues to be "fixed" with
document edits.

Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Wed, 1 Jul 2015 19:09:41 +0000

Hi Joe

Please see inline with tag <Saumya>

Thanks
Saumya.

>If that is correct, then it has no business interacting with ICMP.
<Saumya> That¹s what we intend to solve here. For a typical data center
deployment, where
there is need/clamour to save an iota of resource units. Hence digging in
deep
and checking for L2 packet payload for an IP|ICMP or IPv6|ICMPv6 encap and
convey it to
end point may do some good to that. Imagine a web server application
trying to reach out to its storage
Node (blade server) in a cloud network over a vxlan network, in a time
critical operation.
>traversing one hop is one router, not one link. See RFC1812.
<Saumya> From a deployment perspective or rather client perspective, two
devices with same subnet are viewed (although virtually) to be directly
Connected over an L3 network, although there can be L2 switches (Vxlan
gateways) which 
can potentially bridge the packets between them, as devices will be
part of same bridge domain (vni).
>try to interpret the PTBs incorrectly, you'll only create black holes.
<Saumya> Blackholing will anyway happen without this solution.
All we are looking for a IP|ICMP or Ipv6|ICMPv6 encap in l2 payload
and working on it further, in Vxlan gateway.
>precisely so the path properties to not rely on ICMPs - which are often
<Saumya> This solution can be viewed as an enabler for PLMTUD if required
on any end-point 
device connected to vxlan network and not the Vxlan gateways themselves.

---

From: Joe Touch <[email-masked]>
Date: Wed, 01 Jul 2015 12:54:41 -0700

> device connected to vxlan network and not the Vxlan gateways themselves.

PLMTUD does not use ICMPs. It relies on positive feedback from the data
plane, rather than negative feedback from the control plane.

Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Wed, 1 Jul 2015 20:38:15 +0000

Hi Joe

As far as I know, PLMTUD provides an extension to PMTUD and does not
deprecates the legacy.
ICMP can be one of the potential upper-layer protocols which can be used
for PLMTUD as well.

Even though PLMTUD is ICMP agnostic, ICMP PTB¹s can be leveraged in
determining probe success/failures,
and also detecting ICMP black hole to further refine the probe.

Ensuring that icmp PTB generated from the underlay is percolated to end
point device
should only aid PLMTUD and reduce the scope of icmp black-holing.

Thanks
Saumya.

---

From: Joe Touch <[email-masked]>
Date: Wed, 1 Jul 2015 13:54:31 -0700

> As far as I know, PLMTUD provides an extension to PMTUD and does not

There's not much to deprecate, per se. PLMTUD is certainly preferred.

> ICMP can be one of the potential upper-layer protocols which can be used

It can be, but it's not needed at all.

> and also detecting ICMP black hole to further refine the probe.

You cannot detect an ICMP black hole. You *infer* that event from a set
of conditions, and that set can have several causes.

> should only aid PLMTUD and reduce the scope of icmp black-holing.

ICMP PTBs should be received from L2 only when L2 cannot transit a
message, i.e., when it cannot fragment and reassemble it. It should NOT
be the result of L2 preferring a native size.

I.e., the basis of Sec 3.1.1 is false. A packet that is fragmented and
reassembled at the encapsulation layer (L2, to the E2E L3) is NOT "too
big", and should never generate an ICMP error anyway.

Joe

---

From: "Black, David" <[email-masked]>
Date: Thu, 2 Jul 2015 01:16:31 +0000

Joe,

> -overall: in a tunneling system, the packetization layer is the tunnel
> encapsulator, which means that layer should be doing MTU discovery

> support fragmentation and reassembly on its own.

Well, the "running code" doesn't do either of those, sorry :-(.

The underlying situation is that VXLAN, was never standardized by the
IETF - RFC 7348 on VXLAN is an Independent Submission RFC that has
this to say on path MTU:

   VTEPs MUST NOT fragment VXLAN packets.  Intermediate routers may
   fragment encapsulated VXLAN packets due to the larger frame size.
   The destination VTEP MAY silently discard such VXLAN fragments.  To
   ensure end-to-end traffic delivery without fragmentation, it is
   RECOMMENDED that the MTUs (Maximum Transmission Units) across the
   physical network infrastructure be set to a value that accommodates
   the larger frame size due to the encapsulation.  Other techniques
   like Path MTU discovery (see [RFC1191] and [RFC1981]) MAY be used to
   address this requirement as well.

If the underlay (physical network) MTU is constant between encap and
decap nodes (the VTEPs), then the packet drop occurs at the ingress VTEP.

Otherwise, the absence of a requirement to set DF in the outer IPv4
header can result in black-holing for IPv4 in the underlay when an
intermediate router fragments and the egress VTEP drops the fragments.

Thanks,
--David

> -----Original Message-----
> From: nvo3 [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Wednesday, July 01, 2015 2:45 PM
> To: Saumya Dikshit (sadikshi); [email-masked]
> Cc: [email-masked]
> Subject: Re: [nvo3] New draft: PMTUD Over Vxlan
> 
> 
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Thu, 2 Jul 2015 03:45:24 +0000

Please see inline.

Thanks
Saumya.

>There's not much to deprecate, per se. PLMTUD is certainly preferred.
<Saumya> Certainly not much, but PLMTUD surely firms up the Classic PMTUD

>It can be, but it's not needed at all.

<Saumya>Yes, ICMP can be used as  *ONE* of the instruments, and as long
that is an option
this draft may be helpful
 
>of conditions, and that set can have several causes.

<Saumya> That’s right, black-hole is inferred, may be wrong choice of
words at my end.
Where ICMP PTB definitely plays a part.

>big", and should never generate an ICMP error anyway.

<Saumya> The encapsulation and reassembly mention in 3.1.1 is with respect
to the outer header encapsulation  (for Vxlan tunnel encap).
Frag/reassembly in underlay is performed in context of outer encap which
is L3 based. That’s one of the reason Vxlan scores over other L2-tunneling
schema.

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Jul 2015 09:02:48 -0700

> On Jul 1, 2015, at 6:16 PM, Black, David <[email-masked]> wrote:
> 
> Joe,
> 

That could easily be inconsistent with existing requirements, e.g., if running IPv6 over (eventually) IPv6.

> decap nodes (the VTEPs), then the packet drop occurs at the ingress VTEP.

Right - that would be one source of such an inconsistency.

While I appreciate this isn’t what running code does, it’s not appropriate to propose an approach that cannot support existing standards.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Jul 2015 10:30:03 -0700

> is L3 based. That’s one of the reason Vxlan scores over other L2-tunneling

That's good, but then the PTB needs to be interpreted correctly.

PTB is TB. It isn't "packet larger than I'd like, but I can deal with
it". There is no such signal for the latter.

Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Fri, 3 Jul 2015 08:12:00 +0000

On 7/2/15, 11:00 PM, "Joe Touch" <[email-masked]> wrote:

>> is L3 based. That¹s one of the reason Vxlan scores over other
>>L2-tunneling
>> schema.
>

Sure, That¹s what is being conveyed here; for underlay indeed its a PTB
(without frag). I believe that it will serve good, if this TB which is
generated by any node in the path (overlay or underlay), is not
black-holed.

---

From: Joe Touch <[email-masked]>
Date: Mon, 6 Jul 2015 10:20:42 -0700

On 7/3/2015 1:12 AM, Saumya Dikshit (sadikshi) wrote:
> (without frag). 

That's not what PTB means. It doesn't mean "I can't get there optimally"
- it means "I can't get there at all".

When an ingress sends PTB rather than doing frag and reassembly, it
makes the network more fragile. If the PTB indicates a transit payload
smaller than the required smallest size, then the link is effectively no
longer supporting that protocol (e.g., 1280 for IPv6).

> I believe that it will serve good, if this TB which is
> black-holed.=

It is only helping relieve a problem you're creating by forcing the
ingress to be more limited. Yes, it helps fix a problem you create, but
that's not a good thing. Stop creating the problem.

Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Wed, 8 Jul 2015 15:13:41 +0000

On 7/6/15, 10:50 PM, "Joe Touch" <[email-masked]> wrote:

>makes the network more fragile.
But isn’t it fair to communicate/relay errors  generated from transit
devices
and not only the gateways, to the client-end point.
As the core networks can be brownfield ones ,
where as the gateways only the new addition for supporting Vxlan.

This may require operator intervention to configure and adjust the MTUs
across all the links in both underlay and client side network.
The public cloud operator may not want to go around these issues.

>If the PTB indicates a transit payload
Lesser than 1280 for sure should be taken care of in case there IPv6
enabled end points

>that's not a good thing. Stop creating the problem.
Problem is already there on the field right now. Its just that there not
enough
IPv6 applications (except Skype :) ) on field to rake this up.

---

From: Joe Touch <[email-masked]>
Date: Wed, 08 Jul 2015 11:33:41 -0700

Resposes below...

...

It is, but again, PTB means "I cannot get there at all". If you send
that signal when you *can* get there with fragmentation, you're just
hobbling your own network.

What you want is a "packet larger than I'd prefer" or "packet larger
than is optimal for me". There is currently no such ICMP message.

> The public cloud operator may not want to go around these issues.

It's your decision when to send PTB, but when you send it, it means only
one thing. If you want it to mean "I really can't get there" but you
want to also avoid fragmentation, you need to control the network over
which the tunnels go.

...
> enough IPv6 applications (except Skype :) ) on field to rake this up.

And the Web, including Google, Twitter, and Facebook. And email. And
NetFlix:
http://www.internetsociety.org/deploy360/blog/2012/06/netflix-now-streaming-videos-over-ipv6/

FWIW, if you're not supporting IPv6 in this doc *right now*, I doubt it
will make it past the IESG.

Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Wed, 8 Jul 2015 20:12:07 +0000

>than is optimal for me". There is currently no such ICMP message.
This one seems definitely worth pursuing.
>will make it past the IESG.
Joe, Your Comments and views are worthy every ‘bit' of them.
The intention has always been to support IPv6 in this draft, right from
the word go.
If there are any missing pieces, as pointed by you, then there is a
concerted effort to
fill them up.