Subject: Re: [Int-area] Call for adoption of draft-xu-intarea-ip-in-udp-03

From: Xuxiaohu <[email-masked]>
Date: Wed, 1 Jun 2016 07:16:11 +0000

> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]]
> Sent: Wednesday, June 01, 2016 1:03 PM
> To: Xuxiaohu; Brian E Carpenter
> Cc: joel jaeggli; Fred Baker (fred); Wassim Haddad; [email-masked]
> Subject: Re: [Int-area] Call for adoption of draft-xu-intarea-ip-in-udp-03
> 
> 
> 
> On 5/31/2016 7:43 PM, Xuxiaohu wrote:
> > In the above case, there are three FEASIBLE ways:
> Let's take these in order then:
> 
> > 1) Configure the transit core to carry an MTU at least large enough to
> > accommodate the added encapsulation headers. Note that this is the
> > dominant and mature choice in most MPLS networks;
> 
> So you are deploying IPv6 over an IPv6 tunnel. Even without UDP, you still need
> 40 bytes for the encapsulation header.
> 
> So in order to deploy a compliant IPv6 1280-byte MTU tunnel, you need to use a
> core that you know can transit IPv6 packets that are 1320 bytes.

1320 is not a problem at all. Many modern routers can even support 9000-byte and even large MTU. In a words, setting the MTU of the core large enough to accommodate the added encapsulation headers is easy in Softwire networks which are well-managed SP networks.

> Unless you examine every physical link to ensure that the technology supports a
> larger MTU, you're stuck assuming that your IPv6 equipment can only guarantee
> 1280. Which means you can't know this will work.

It's no problem to ensure each physical link supports a larger MTU. It's a Best Current Practice in MPLS networks which are well-managed SP networks as well.

> Even if you do know this for a physical path, every path you *think* is physical
> will be virtualized at some point, and at some point the number of layers of
> tunneling will end up eating the whole of any "headroom"
> you engineer.

It's true that some paths in the MPLS networks are virtualized (e.g., LDP over TE) as well. However, as said before, configuring the MTU of the core large enough is still be Best Current Practice in MPLS networks. Imagine that the MTU of the core is set to 5000, it will accommodate many layers of tunneling. It's hard to understand that large headroom is eaten up in reality.  

> >  2) Fragment the packet before encapsulation if allowed;
> 
> That covers only IPv4 with DF=0. It won't cover DF=1 or IPv6.

Yes "if allowed" means IPv4 with DF=0.

> >  3) Drop it if inner fragmentation is not allowed.
> You've now black-holed your network, all because you didn't want to use the
> 1500-byte fragmentation and reassembly that every IPv6 source and sink are
> already required to support (per RFC2460).
> 
> Note that every tunnel ingress is a IPv6 source and every IPv6 egress is an IPv6
> sink, so if you claim that many vendors don't support this, you're just pointing
> out that they don't satisfy existing IPv6 requirements.

IMHO, RFCs are not academic papers. Hence it would be beneficial for both vendors and operators if the requirements specified in RFCs could take the reality, especially the best current practices more into account.

Xiaohu

> Joe

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 1 Jun 2016 10:08:48 +0000

> From: Xuxiaohu
> Sent: Wednesday, June 01, 2016 3:16 PM
> To: 'Joe Touch'; Brian E Carpenter
> Subject: RE: [Int-area] Call for adoption of draft-xu-intarea-ip-in-udp-03
> 
> 
> 
> > > 1) Configure the transit core to carry an MTU at least large enough
> > > to accommodate the added encapsulation headers. Note that this is
> > > the dominant and mature choice in most MPLS networks;
> >
> > So in order to deploy a compliant IPv6 1280-byte MTU tunnel, you need
> > to use a core that you know can transit IPv6 packets that are 1320 bytes.
> 
> 1320 is not a problem at all. Many modern routers can even support 9000-byte
> and even large MTU. In a words, setting the MTU of the core large enough to
> accommodate the added encapsulation headers is easy in Softwire networks
> which are well-managed SP networks.
> 
> > Unless you examine every physical link to ensure that the technology
> > supports a larger MTU, you're stuck assuming that your IPv6 equipment
> > can only guarantee 1280. Which means you can't know this will work.

BTW, assume SPs couldn't guarantee the tunnel path MTU is large than 1280, the network throughput would be sharply decreased by 50% to 90% (according to some vendor's document). Please ask SPs whether they could afford that cost.

Xiaohu 

> 
> It's no problem to ensure each physical link supports a larger MTU. It's a Best
> Current Practice in MPLS networks which are well-managed SP networks as
> well.
> 
> > Even if you do know this for a physical path, every path you *think*
> > is physical will be virtualized at some point, and at some point the
> > number of layers of tunneling will end up eating the whole of any "headroom"
> > you engineer.
> 
> It's true that some paths in the MPLS networks are virtualized (e.g., LDP over TE)
> as well. However, as said before, configuring the MTU of the core large enough
> is still be Best Current Practice in MPLS networks. Imagine that the MTU of the
> core is set to 5000, it will accommodate many layers of tunneling. It's hard to
> understand that large headroom is eaten up in reality.
> 
> > You've now black-holed your network, all because you didn't want to
> > use the 1500-byte fragmentation and reassembly that every IPv6 source
> > and sink are already required to support (per RFC2460).
> >
> > Note that every tunnel ingress is a IPv6 source and every IPv6 egress
> > is an IPv6 sink, so if you claim that many vendors don't support this,
> > you're just pointing out that they don't satisfy existing IPv6 requirements.
> 
> IMHO, RFCs are not academic papers. Hence it would be beneficial for both
> vendors and operators if the requirements specified in RFCs could take the
> reality, especially the best current practices more into account.
> 
> Xiaohu
> 
> > Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 1 Jun 2016 10:33:36 -0700

On 6/1/2016 12:16 AM, Xuxiaohu wrote:
> 1320 is not a problem at all. Many modern routers can even support 9000-byte and even large MTU.

Routers aren't typically the problem; links are.

Yes, you can engineer a system to use links that give enough headroom
for many layers of encapsulation. Avoiding fragmentation is always
better than doing fragmentation and reassembly -- when you can do so.
However, unless you're calling this "IP in UDP but only over jumbo
ethernet", you need to account for the possibility (probability) that a
path may end up going over links that support only the minimums.

And that assumes you believe vendors follow what you call "academic
documents" (i.e., RFC standards).

If they don't, you can't even assume the minimums.

IP isn't designed to work efficiently; it's designed to work everywhere.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 2 Jun 2016 08:23:24 +1200

Hi,

> I would like to add a dedicated Applicability Statement section

I think that would be very useful. Indeed, the whole Softwires concept
needs an Applicability Statement.

Regards
   Brian

On 01/06/2016 18:26, Xuxiaohu wrote:
>> From: Brian E Carpenter [mailto:[email-masked]]
>> Sent: Wednesday, June 01, 2016 1:14 PM
>> To: Xuxiaohu; Joe Touch
>> I think I detest fragmentation, or rather re-assembly, as much as anybody,
>> whether designing hardware or software solutions. And of course, in a closely
>> managed environment, you can ensure that the outer MTU is sufficient to
>> contain the inner MTU. That isn't the point. The point is that designing a protocol
> 
> It said in the draft that "... this specification defines an IP-in-UDP encapsulation method dedicated for Softwire service (including both mesh and hub-spoke modes). " Softwire networks are well-managed SP networks. I would like to add a dedicated Applicability Statement section to emphasize it if the current statement seems not enough.
> 
> Best regards,
> Xiaohu
> 
>> proposed as a general-purpose protocol for the Internet, that might be used for
>> a hundred years, we can't guarantee that it will always be deployed in such a
>> managed environment. In fact, we can pretty much guarantee that it will be
>> used in completely unmanaged (or badly managed) ways.
>>
>> This isn't theory. We've seen it a lot where IPv6 has been tunneled across
>> IPv4 islands, with lots of MTU and fragmentation failures. That's even simpler
>> than IP in UDP in IP.
> 
>> Regards
>>    Brian
>>
>> On 01/06/2016 16:09, Xuxiaohu wrote:
>>>
>>>
>>>> From: Brian E Carpenter [mailto:[email-masked]]
>>>> Sent: Wednesday, June 01, 2016 4:01 AM
>>>> To: Xuxiaohu; Joe Touch
>>>> On 31/05/2016 20:13, Xuxiaohu wrote:
>>>>>
>>>>>
>>>>>> From: Brian E Carpenter [mailto:[email-masked]]
>>>>>> Sent: Tuesday, May 31, 2016 4:46 AM
>>>>>> To: Joe Touch
>>>>>> Cc: joel jaeggli; Xuxiaohu; Fred Baker (fred); Wassim Haddad;
>>>>>> [email-masked]
>>>>>> And being pedantic...
>>>>>> On 31/05/2016 06:12, Joe Touch wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 5/29/2016 4:23 PM, joel jaeggli wrote:
>>>>>>>>>> I.e., you MUST support source fragmentation at the ingress at
>>>>>>>>>> the outer
>>>>>>>>>> IPv6 layer (because UDP doesn't have support for fragmentation
>>>>>>>>>> and reassembly). If you make this requirement, you can handle
>>>>>>>>>> IPv6 over the tunnel.
>>>>>>>> Yeah I don't support it for this reason. getting IP fragments
>>>>>>>> back together in the same place a reassembled is hard is in some
>>>>>>>> cases especially when you hash. (see frag drop) given
>>>>>>>> alternatives that better address such situations it seems hard to justify.
>>>>>>>
>>>>>>> If you intend to support recursive IP tunneling* and believe that
>>>>>>> IP has a minimum MTU, then you have to accept reassembly.
>>>>>>
>>>>>> If you intend to support recursive datagram tunneling and believe
>>>>>> that any path has a minimum MTU, then you have to accept reassembly.
>>>>>>
>>>>>> This is physics, and nothing to do with design details.
>>>>>>
>>>>>> (Something I discovered in about 1983, when implementing OSI/CLNP
>>>>>> at CERN over a homebrew network with 128 byte packets.)
>>>>>
>>>>> Reassembly on the tunnel egress may be acceptable at that old time.
>>>>> However,
>>>> due to the considerable improvement in network bandwidth capability,
>>>> the practice acceptable at the old time may have become outdated today.
>>>>
>>>> I don't understand what network capacity has to do with the physical
>>>> and mathematical fact that packets larger than N bytes will not fit
>>>> into a packet limited to N bytes.
>>>
>>> This article
>> (http://learning.nil.com/assets/Tips-/The-Never-Ending-Story-of-IP-Fragmentati
>> on.pdf) may be useful for you to understand why network capability is a key
>> factor to be considered for fragmentation and reassembly on routers (here
>> routers are not software routers or CPU-only routers which were dominant in
>> the old time). Of course, you could also have a look at the current fragmentation
>> and reassembly implementations of major router vendors if you believed that
>> article is a little bit old.
>>>
>>> Xiaohu
>>>
>>>> That was true in 1983 and will still be true in 2083.
>>>>
>>>>    Brian
>>>>
>>>>> See the MAP implementation experience shared by Ole recently.
>>>>>
>>>>> Xiaohu
>>>>>
>>>>>>    Brian
>>>>>>
>>>>>>>
>>>>>>> Joe
>>>>>>>
>>>>>>> * where "recursive IP tunneling" is IP in [zero or more other
>>>>>>> protocols] in IP.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Int-area mailing list
>>>>>>> [email-masked]
>>>>>>> https://www.ietf.org/mailman/listinfo/int-area
>>>>>>>

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 1 Jun 2016 22:50:33 +0000

Hi,

> From: Xuxiaohu [mailto:[email-masked]]
> Sent: Tuesday, May 31, 2016 7:32 PM
> To: Templin, Fred L <[email-masked]>; Joe Touch <[email-masked]>; Brian E Carpenter <[email-masked]>
> Cc: [email-masked]
> > From: Int-area [mailto:[email-masked]] On Behalf Of Templin, Fred L
> > Sent: Wednesday, June 01, 2016 12:30 AM
> > To: Joe Touch; Brian E Carpenter
> > Cc: [email-masked]
> > Brian and Joe are correct - tunnels within tunnels means that fragmentation and
> > reassembly are inevitable. Wish it weren't so, but that is the reality.
> 
> If so, could you give a concrete example in real network environment where outer fragmentation is widely enabled and works very
> well? It would be better if the reassembly buffer size needed in the tunnel egress could be given as well.

Any tunnel that traverses a 1280 link has to fragment, but instead of outer fragmentation
 it should use tunnel fragmentation which is something I have been advocating for a very
long time. The tunnel egress should configure at least a 2KB reassembly buffer size.

Thanks - Fred
[email-masked]

> > Thanks - Fred
> > > From: Int-area [mailto:[email-masked]] On Behalf Of Joe
> > > Touch
> > > Sent: Tuesday, May 31, 2016 8:34 AM
> > > To: Brian E Carpenter <[email-masked]>
> > > Cc: [email-masked]
> > > On 5/30/2016 1:45 PM, Brian E Carpenter wrote:
> > > Agreed- at least one of the layers between the message tunnel and when
> > > it recurses must support fragmentation and reassembly.
> > >
> > > Those who deploy or sell systems otherwise are in denial, not a
> > > counter-proof. Again, this is where compliance validation would be useful.
> > >
> > > Joe
> > >

---

From: Joe Touch <[email-masked]>
Date: Wed, 1 Jun 2016 16:36:16 -0700

> long time. The tunnel egress should configure at least a 2KB reassembly buffer size.

If the only layer you have that fragments is the outermost IP, that's
where you MUST do it.

Sure, if you can fragment at an intermediate "transport" layer inside of
IP, that's also an option. I'm not sure why it should be preferred
except for the flaws of current equipment in supporting standards, though.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Wed, 1 Jun 2016 16:57:14 -0700

> except for the flaws of current equipment in supporting standards, though.
We defined fragmentation option in GUE
(draft-herbert-gue-fragmentation-02). The listed benefits are:

      o A 32 bit identifier can be defined to avoid issues of the 16 bit
        Identification in IPv4.

      o Encapsulation mechanisms for security and identification such as
        virtual network identifiers can be applied to each segment.

      o This allows the possibility of using alternate fragmentation and
        reassembly algorithms (e.g. fragmentation with Forward Error
        Correction).

      o Fragmentation is transparent to the underlying network so it is
        unlikely that fragmented packet will be unconditionally dropped
        as might happen with IP fragmentation.

We should probably add that all fragments will have the same outer
addresses and UDP ports in the encapsulation so it is likely that all
the fragments will follow the same path. Security means that we can
cover fragmentation information with checksum or stronger integrity
checks.

This option probably only makes use of fragmentation a little more
palatable, but doesn't change the fact that it's better to avoid
having to fragment in the first place if at all possible ;-).

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 1 Jun 2016 17:02:48 -0700

Oh, certainly if the intermediate layer's variant of fragmentation is
better than IP's, sure - SEAL has similar benefits.

Your 4th bullet point is the one I hope we can start objecting to,
though - if IP can't allow fragments, it's not IP and we should simply
return that sort of mislabeled equipment (presuming it says "supports IP").

On your last point regarding UDP ports, that should matter only for
IPv4. For IPv6, the flow ID is a better place to coordinate path
fate-sharing.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Wed, 1 Jun 2016 17:34:11 -0700

> return that sort of mislabeled equipment (presuming it says "supports IP").
It's a conundrum. The reason that fragments aren't allowed is probably
that packets with IP extension headers are being dropped contrary to
RFC2460. Since there is no interesting use case of extension headers,
there's not much incentive for vendors to fix this (6man discussion).
Even if vendors were forced by the protocol police to comply with
RFC2460, packets with EH can be thrown into some ridiculously slow
software path and they can claim standards compliance (like what
happened with IP options)-- but that really does nothing to make them
usable. Segment routing might be a compelling incentive, but I worry
that might have too narrow deployment to change the overall mindset
about EH.

> IPv4. For IPv6, the flow ID is a better place to coordinate path
Agreed. I've been talking with engineers at Microsoft and they are
looking into getting flow labels set in the next version of Windows,
so non-zero IPv6 flow labels on the Internet should become the norm
fairly soon! (IOS already sends them and I believe we are waiting for
next Linux rebase in Android).

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 1 Jun 2016 17:42:36 -0700

On 6/1/2016 5:34 PM, Tom Herbert wrote:
> RFC2460.

Several reasons:
1) IPv6 extension headers are more expensive to parse, and why build
hardware to parse them when you can simply drop them instead?
2) DPI devices tend to want to drop them too, because otherwise they
would need to reassemble first
        Reassembly at the DPI device is expensive (see #1).
        Reassembly at the DPI device is mistaken as violating the spec
(i.e., intermediate devices must not reassemble - but, by analogy, they
must not peek beyond the network layer too)
        DPI devices otherwise don't know how to handle fragments for
ECMP (but they could have a default for non-initial frags).
3) Both IPv4 and IPv6 fragments can be used as an endpoint DOS attack
(overload the reassembly buffer)

>  Since there is no interesting use case of extension headers,
IPsec?
IP routing headers?

In the past, the discussion on 6man and v6ops centered around
effectively deprecating extension headers. That tide has started to
turn, and now the view is to require support for at least a reasonable
chain.

> that might have too narrow deployment to change the overall mindset
Again (and again and again), we REALLY need a compliance arm of the
ISOC. If vendors had a "truth in advertising" requirement, they wouldn't
be able to claim 40Gbps line-rate forwarding without EH support.
> next Linux rebase in Android).

Yup - the better solution in all these cases is to fix broken stuff
first - the protocol only when the protocol is the thing that's broken.

Joe

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 2 Jun 2016 02:04:53 +0000

> Sent: Thursday, June 02, 2016 4:23 AM
> To: Xuxiaohu; Joe Touch
> I think that would be very useful. Indeed, the whole Softwires concept needs an
> Applicability Statement.

Hi Brian,

Thanks for your advice.

Best regards,
Xiaohu

> >> I think I detest fragmentation, or rather re-assembly, as much as
> >> anybody, whether designing hardware or software solutions. And of
> >> course, in a closely managed environment, you can ensure that the
> >> outer MTU is sufficient to contain the inner MTU. That isn't the
> >> point. The point is that designing a protocol
> >
> > It said in the draft that "... this specification defines an IP-in-UDP encapsulation
> method dedicated for Softwire service (including both mesh and hub-spoke
> modes). " Softwire networks are well-managed SP networks. I would like to add
> a dedicated Applicability Statement section to emphasize it if the current
> statement seems not enough.
> >
> > Best regards,
> > Xiaohu
> >
> >> proposed as a general-purpose protocol for the Internet, that might
> >> be used for a hundred years, we can't guarantee that it will always
> >> be deployed in such a managed environment. In fact, we can pretty
> >> much guarantee that it will be used in completely unmanaged (or badly
> managed) ways.
> >>
> >> IPv4 islands, with lots of MTU and fragmentation failures. That's
> >> even simpler than IP in UDP in IP.
> >
> >> Regards
> >>    Brian
> >>
> >>>>>>>>>> IPv6 layer (because UDP doesn't have support for
> >>>>>>>>>> fragmentation and reassembly). If you make this requirement,
> >>>>>>>>>> you can handle
> >>>>>>>>>> IPv6 over the tunnel.
> >>>>>>>> back together in the same place a reassembled is hard is in
> >>>>>>>> some cases especially when you hash. (see frag drop) given
> >>>>>>> If you intend to support recursive IP tunneling* and believe
> >>>>>>> that IP has a minimum MTU, then you have to accept reassembly.
> >>>>>>
> >>>> due to the considerable improvement in network bandwidth
> >>>> capability, the practice acceptable at the old time may have become
> outdated today.
> >>>>
> >>>> I don't understand what network capacity has to do with the
> >>>> physical and mathematical fact that packets larger than N bytes
> >>>> will not fit into a packet limited to N bytes.
> >>>
> >>> This article
> >> on.pdf) may be useful for you to understand why network capability is
> >> a key factor to be considered for fragmentation and reassembly on
> >> routers (here routers are not software routers or CPU-only routers
> >> which were dominant in the old time). Of course, you could also have
> >> a look at the current fragmentation and reassembly implementations of
> >> major router vendors if you believed that article is a little bit old.
> >>>
> >>> Xiaohu
> >>>

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 2 Jun 2016 02:03:04 +0000

> From: Templin, Fred L [mailto:[email-masked]]
> Sent: Thursday, June 02, 2016 6:51 AM
> To: Xuxiaohu; Joe Touch; Brian E Carpenter
> Cc: [email-masked]
> > To: Templin, Fred L <[email-masked]>; Joe Touch
> > <[email-masked]>; Brian E Carpenter <[email-masked]>
> > Cc: [email-masked]
> > > Brian and Joe are correct - tunnels within tunnels means that
> > > fragmentation and reassembly are inevitable. Wish it weren't so, but that is
> the reality.
> >
> > If so, could you give a concrete example in real network environment
> > where outer fragmentation is widely enabled and works very well? It would be
> better if the reassembly buffer size needed in the tunnel egress could be given as
> well.
> 
> Any tunnel that traverses a 1280 link has to fragment, but instead of outer
> fragmentation  it should use tunnel fragmentation which is something I have
> been advocating for a very long time. The tunnel egress should configure at least
> a 2KB reassembly buffer size.

Hi Fred,

The major concern about tunnel fragmentation and reassembly is whether reassembly needs to be performed by tunnel egress routers (Note that routers here mean hardware routers rather than software routers). When performing reassembly on tunnel egress routers, the throughput would be significantly decreased (e.g., by 50% to 90%). If SPs are not willing to afford that cost, the only feasible way for them to deal with this MTU issue is to configure the MTU of the core large enough so as to accommodate the added encapsulation headers. 

With the 2K reassembly buffer size, what's the maximum throughput that tunnel egress routers are assumed to support? especially in the case where the traffic over the tunnel is IPv6 traffic and the tunnel needs to traverse a 1280 link (in other words, all traffic would have to be fragmented after encapsulation).  

BTW, did you know any router vendors who had implemented the tunnel fragmentation approach that you had been advocating for a very long time? 

Best regards,
Xiaohu

> > > > > If you intend to support recursive datagram tunneling and
> > > > > believe that any path has a minimum MTU, then you have to accept
> reassembly.
> > > >
> > > > Agreed- at least one of the layers between the message tunnel and
> > > > when it recurses must support fragmentation and reassembly.
> > > >

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Jun 2016 11:08:17 -0700

On 5/27/2016 3:50 AM, [email-masked] wrote:
> It is not possible to implement reassembly complying with IETF RFCs.

a) ATM does this at ridiculously high fragment rates. Granted IP frags
can come out of order, but the fragments are generally much larger.

b) What is the alternative, given we have minimum MTU requirements?

If you're limiting yourself to IPv4 payloads where DF=0, sure, there
there is an alternative. But you've just disabled IPv6 and IPv4 with DF=1.

I.e., it's not possible to NOT implement this and comply with IETF RFCs.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 2 Jun 2016 21:16:53 +0000

Agree with Joe, and also just to mention that RFC4459 discusses tunnel MTU
and fragmentation considerations.

Thanks - Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Thursday, June 02, 2016 11:08 AM
> To: [email-masked]; Xuxiaohu <[email-masked]>
> Cc: Softwires WG <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [Int-area] [nvo3] [Softwires] Is it feasible to perform fragmentation on UDP encapsulated packets.
> 
> 
>

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 3 Jun 2016 02:53:37 +0000

> Sent: Friday, June 03, 2016 2:08 AM
> To: [email-masked]; Xuxiaohu
> Cc: Softwires WG; [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [nvo3] [Softwires] Is it feasible to perform fragmentation on UDP
> encapsulated packets.
> 
> 
> 
> a) ATM does this at ridiculously high fragment rates. Granted IP frags can come
> out of order, but the fragments are generally much larger.

As pointed in RFC4459,

     "At the time of reassembly, all the information (i.e., all the
      fragments) is normally not available; when the first fragment
      arrives to be reassembled, a buffer of the maximum possible size
      may have to be allocated because the total length of the
      reassembled datagram is not known at that time. Furthermore, as
      fragments might get lost, or be reordered or delayed, the
      reassembly engine has to wait with the partial packet for some
      time (e.g., 60 seconds [9]).  When this would have to be done at
      the line rate, with, for example 10 Gbit/s speed, the length of
      the buffers that reassembly might require would be prohibitive. "

Have you heard the wide adoption of 622M (STM-1) beyond ATM interfaces between ATM switches in the previous ATM networks? If not, the highest non-ATM interface in the past ATM networks was the FE interface which is 100M bps (around the year of 2000), If I remembered it correctly. Furthermore, have you heard the reordering issue with ATM cells? If no, that means once an ATM cell of a given ALL PDU gets lost, all the received ATM cells of that AAL PDU would be dropped and therefore the reassembly buffer for that AAL datagram would be released. In other words, there is no need to wait for the lost or reordered fragment for a certain period of time before releasing the reassembly buffer. Such behavior is not possible for IP fragmentation and reassembly. Last but not least, there is no need to assign a reassembly buffer per AAL PDU (as opposed to per IP datagram in the IP fragmentation and reassembly case), instead, only one reassembly buffer per VC since all cells within a given VC would be received in order. Since the SAR is only needed on the edge of the ATM networks, the number of VCs is very limited. In a word, there are significant differences between IP fragmentation and ATM SAR. 

Best regards,
Xiaohu

> If you're limiting yourself to IPv4 payloads where DF=0, sure, there there is an
> alternative. But you've just disabled IPv6 and IPv4 with DF=1.
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Jun 2016 21:34:27 -0700

>       the buffers that reassembly might require would be prohibitive. "

Yes, but the alternative of declaring that you don't reassemble has a
cost in terms of dropped segments too.

Taking that drop into account, buffering for a smaller amount of
potential reordering and accounting for reasonable reassembly sizes (for
IPv6, the smallest "max" that's required is 1500) need not be prohibitive.

> Have you heard the wide adoption of 622M (STM-1) beyond ATM interfaces between ATM switches in the previous ATM networks? If not, the highest non-ATM interface in the past ATM networks was the FE interface which is 100M bps (around the year of 2000), If I remembered it correctly. 

OC12 ATM NICs were commonly available in 1998. That was back when
ethernet was pushing 100M and the only other near-gigabit tech was
Myricom (a spin-off of USC/ISI and Caltech). I.e., with the tech
available at that time,  600Mbps SAR was possible.

> Furthermore, have you heard the reordering issue with ATM cells? If no, that means once an ATM cell of a given ALL PDU gets lost, all the received ATM cells of that AAL PDU would be dropped and therefore the reassembly buffer for that AAL datagram would be released. In other words, there is no need to wait for the lost or reordered fragment for a certain period of time before releasing the reassembly buffer.
Reordered no, but just because they arrive in order does not mean they
are required to arrive *adjacent*. The same requirement applies in terms
of needing reassembly buffers for a number of interleaved segments.

>  Such behavior is not possible for IP fragmentation and reassembly. Last but not least, there is no need to assign a reassembly buffer per AAL PDU (as opposed to per IP datagram in the IP fragmentation and reassembly case), instead, only one reassembly buffer per VC since all cells within a given VC would be received in order. Since the SAR is only needed on the edge of the ATM networks, the number of VCs is very limited. 
You could easily decide to allocate a fixed number of reassembly buffers
per IP flow to conserve resources too. The amount of reordering,
together with the number of flows supported would determine your ability
to avoid packet drops -- but recall that drops at high load are not
unusual for IP.

Joe

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 3 Jun 2016 07:38:40 +0000

> Sent: Friday, June 03, 2016 12:34 PM
> To: Xuxiaohu; [email-masked]
> >> Subject: Re: [nvo3] [Softwires] Is it feasible to perform
> >> fragmentation on UDP encapsulated packets.
> >>
> >>
> >>
> >> a) ATM does this at ridiculously high fragment rates. Granted IP
> >> frags can come out of order, but the fragments are generally much larger.
> > As pointed in RFC4459,
> >
> Yes, but the alternative of declaring that you don't reassemble has a cost in
> terms of dropped segments too.

The alternative is to configure the MTU of the core large enough to accommodate the added encapsulation header. This is a feasible and proven approach in well-managed SP networks. No packet loss and no forwarding performance degradation.

> Taking that drop into account, buffering for a smaller amount of potential
> reordering and accounting for reasonable reassembly sizes (for IPv6, the
> smallest "max" that's required is 1500) need not be prohibitive.

In the IPv6-in-IPv4 Software network case, assume the MTU of the core is not large enough than 1280+20, all the IPv6 traffic across the tunnels would have to be fragmented and then reassembled. Not a small amount.

> > Have you heard the wide adoption of 622M (STM-1) beyond ATM interfaces
> between ATM switches in the previous ATM networks? If not, the highest
> non-ATM interface in the past ATM networks was the FE interface which is 100M
> bps (around the year of 2000), If I remembered it correctly.
> 

Sorry, there was a spelling mistake ( s/STM-1/STM-4).

OC12/STM-4=622M. My question is have you heard the wide adoption of 622M (STM-4) BEYOND ATM interfaces (e.g., STM-16/OC48) at that time?

> ethernet was pushing 100M and the only other near-gigabit tech was Myricom
> (a spin-off of USC/ISI and Caltech). I.e., with the tech available at that time,
> 600Mbps SAR was possible.

Was that 600Mbps SAR capability proved in real networks? And for what service?

> > Furthermore, have you heard the reordering issue with ATM cells? If no, that
> means once an ATM cell of a given ALL PDU gets lost, all the received ATM cells
> of that AAL PDU would be dropped and therefore the reassembly buffer for that
> AAL datagram would be released. In other words, there is no need to wait for
> the lost or reordered fragment for a certain period of time before releasing the
> reassembly buffer.
> Reordered no, but just because they arrive in order does not mean they are
> required to arrive *adjacent*. The same requirement applies in terms of
> needing reassembly buffers for a number of interleaved segments.

However, the reassembly buffer requirement is limited due to the fact that only one buffer is needed per VC and the total number of VCs is not large on the edge of ATM networks. In contrast, the number of IP flows is uncontrollable.

> >  Such behavior is not possible for IP fragmentation and reassembly. Last but
> not least, there is no need to assign a reassembly buffer per AAL PDU (as
> opposed to per IP datagram in the IP fragmentation and reassembly case),
> instead, only one reassembly buffer per VC since all cells within a given VC would
> be received in order. Since the SAR is only needed on the edge of the ATM
> networks, the number of VCs is very limited.
> You could easily decide to allocate a fixed number of reassembly buffers per IP
> flow to conserve resources too. The amount of reordering, together with the

What about in the DDoS attack case? Do you believe that SPs would allow the existence of that obvious DDoS attack risk in their networks and afford the significant forwarding performance degradation? especially when there has been a feasible and proven solution to the MTU issue (i.e., configure the MTU of the core large enough)?

Best regards,
Xiaohu

> number of flows supported would determine your ability to avoid packet drops
> -- but recall that drops at high load are not unusual for IP.
> 
> Joe

---

From: Rémi Després <[email-masked]>
Date: Fri, 3 Jun 2016 14:48:04 +0200

> 2016-06-03 09:38, Xuxiaohu <[email-masked]> :
> 
...
> The alternative is to configure the MTU of the core large enough to accommodate the added encapsulation header. This is a feasible and proven approach in well-managed SP networks. No packet loss an…

Just for information on this subject, an earlier analysis is a available in https://tools.ietf.org/html/rfc6751#section-6.4.
(No plan to participate again in IETF technical debates.)

Regards to all,
RD

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 3 Jun 2016 16:27:06 +0000

> The alternative is to configure the MTU of the core large enough to accommodate the added encapsulation header.

But what if you can't do that? All it takes is one link with a small and/or
misconfigured MTU...

Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Xuxiaohu
> Sent: Friday, June 03, 2016 12:39 AM
> To: Joe Touch <[email-masked]>; [email-masked]
> The alternative is to configure the MTU of the core large enough to accommodate the added encapsulation header. This is a feasible
> and proven approach in well-managed SP networks. No packet loss and no forwarding performance degradation.
> 
> In the IPv6-in-IPv4 Software network case, assume the MTU of the core is not large enough than 1280+20, all the IPv6 traffic across
> the tunnels would have to be fragmented and then reassembled. Not a small amount.
> 
> However, the reassembly buffer requirement is limited due to the fact that only one buffer is needed per VC and the total number of
> VCs is not large on the edge of ATM networks. In contrast, the number of IP flows is uncontrollable.
> 
> What about in the DDoS attack case? Do you believe that SPs would allow the existence of that obvious DDoS attack risk in their
> networks and afford the significant forwarding performance degradation? especially when there has been a feasible and proven
> solution to the MTU issue (i.e., configure the MTU of the core large enough)?
> 
> Best regards,
> Xiaohu
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 3 Jun 2016 09:44:40 -0700

> What about in the DDoS attack case? Do you believe that SPs would allow the existence of that obvious DDoS attack risk in their networks and afford the significant forwarding performance degradatio…
That is precisely solution #3 in RFC4459:

"Ensure that in the specific environment, the encapsulated packets
will fit in all the paths in the network, e.g., by using MTU bigger
than 1500 in the backbone used for encapsulation."

If you're able to implement this in you network that's great, but it
is not feasible in all situations and hence can't be the standard. We
may not control all the underlay networks (definitely not over the
Internet) and hence not be able to set all the MTUs large enough. Also
an MTU greater than 1500 require jumbo frames, not all networks
enabled those.

One other point, when a device sources or sinks IP packets like in
tunneling it is taking on the role of a host and so we expect host
requirements apply. For instance, routers may perform fragmentation in
IPv4, but they do not perform fragmentation for IPv6 or reassembly
(those are hosts functionalities). There has been at least one attempt
to carve out exceptions in applying host requirements to devices that
perform tunneling, that is an allowance to send a zero UDP checksum in
in IPv6. Documenting this took two RFCs and we still needed a whole
lot of description in at least GRE/UDP on requirements that to use the
non-zero checksum. I don't think standardizing exceptions in
fragmentation for tunnels would be any easier.

Tom

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 6 Jun 2016 20:37:58 +0000

Hi Tom,

> From: tsvwg [mailto:[email-masked]] On Behalf Of Tom Herbert
> Sent: Friday, June 03, 2016 9:45 AM
> To: Xuxiaohu <[email-masked]>
> Cc: [email-masked]; [email-masked]; Joe Touch <[email-masked]>; [email-masked]; [email-masked]; Softwires WG
> <[email-masked]>; [email-masked]
> Subject: Re: [tsvwg] [Int-area] [nvo3] [Softwires] Is it feasible to perform fragmentation on UDP encapsulated packets.
> 
> > However, the reassembly buffer requirement is limited due to the fact that only one buffer is needed per VC and the total number
> of VCs is not large on the edge of ATM networks. In contrast, the number of IP flows is uncontrollable.
> >

I agree. So then, what *should* be the standard is GUE plus fragmentation
plus direct IP encapsulation. I think that should cover all use cases, right?

Thanks - Fred

---

From: Lloyd Wood <[email-masked]>
Date: Sun, 12 Jun 2016 05:11:48 +0000 (UTC)

blockquote, div.yahoo_quoted { margin-left: 0 !important; border-left:1px #715FFA solid !important; padding-left:1ex !important; background-color:white !important; }  Fragmentation should be strongly discouraged.
if you've designed a tunnelling solution and you have fragmentation happening as a matter of course, you've designed it wrong. 

Lloyd [email-masked]

On Friday, May 27, 2016, 8:10 PM, Xuxiaohu <[email-masked]> wrote:

<Note that I have changed the subject of the email hence it has nothing to do with the WG adoption call now. It's just a discussion on a particular issue which is related to those WGs which are working on UDP tunnels. The reason for containing the old email is to use it as a background which may be useful for better understanding of this particular issue>

The possible side-effect of performing fragmentation on UDP encapsulated packets is to worsen the reassembly burden on tunnel egress since fragments of UDP encapsulated packets are more likely to be forwarded across different paths towards the tunnel egress than those of IP or GRE encapsulated packets.

It seems that most X-over-UDP proposals choose to prohibit the tunnel ingress from performing fragmentation on UDP encapsulated packets. See the following quoted text regarding fragmentation from those X-over-UDP drafts:

LISP:

When an ITR receives a packet from a site-facing interface and adds H
  octets worth of encapsulation to yield a packet size greater than L
  octets, it resolves the MTU issue by first splitting the original
  packet into 2 equal-sized fragments.  A LISP header is then prepended
  to each fragment.

VXLAN:

VTEPs MUST NOT fragment VXLAN packets.  Intermediate routers may
  fragment encapsulated VXLAN packets due to the larger frame size.
  The destination VTEP MAY silently discard such VXLAN fragments.

VXLAN-GPE:

VTEPs MUST never fragment an encapsulated VXLAN GPE packet, and when
  the outer IP header is IPv4, VTEPs MUST set the DF bit in the outer
  IPv4 header.

GEVENE:

  To prevent fragmentation and maximize performance, the best practice
  when using Geneve is to ensure that the MTU of the physical network
  is greater than or equal to the MTU of the encapsulated network plus
  tunnel headers.

GUE:

    If a packet is fragmented before encapsulation in GUE, all the
    related fragments must be encapsulated using the same source port
    (inner flow identifier). An operator may set MTU to account for
    encapsulation overhead and reduce the likelihood of fragmentation.

GRE/UDP

Regarding packet fragmentation, an encapsulator/decapsulator SHOULD
  be compliant with [RFC7588] and perform fragmentation before the
  encapsulation.

However, the above choice seems conflict with the requirements as described in https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02

I wonder whether the IETF should reach a consensus on whether or not the fragmentation on UDP encapsulated packets should be allowed.
  
Best regards,
Xiaohu

> Sent: Thursday, May 26, 2016 4:35 PM
> To: 'Joe Touch'; Fred Baker (fred); Wassim Haddad
> Cc: [email-masked]
> > Sent: Thursday, May 26, 2016 2:11 AM
> > To: Xuxiaohu; Fred Baker (fred); Wassim Haddad
> > Cc: [email-masked]
> > On 5/24/2016 7:24 PM, Xuxiaohu wrote:
> > > Hi Joe,
> > >
> > > I wonder whether you want to tell me the following truth by the
> > > example that you gave: no matter whatever improvements we had done
> > > with this draft, those persons who dislike it by the light of nature
> > > would dislike it in the end.
> >
> > The only improvements that would make this doc useful would be to add
> > capabilities already in GRE/UDP or GUE/UDP, which we already have.
> 
> Let's go over the four things you mentioned earlier in GRE/UDP and GUE/UDP:
> 
>     - stronger checksums
> 
> In GRE/UDP, in order to use UDP-zero-checksum, it gave the following
> restrictions:
> " 6. UDP Checksum Handling
> 
>    6.1. UDP Checksum with IPv4
> 
>    For UDP in IPv4, the UDP checksum MUST be processed as specified in
>    [RFC768] and [RFC1122] for both transmit and receive. The IPv4
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 12]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation            March 2016
> 
>    header includes a checksum which protects against mis-delivery of
>    the packet due to corruption of IP addresses. The UDP checksum
>    potentially provides protection against corruption of the UDP header,
>    GRE header, and GRE payload. Disabling the use of checksums is a
>    deployment consideration that should take into account the risk and
>    effects of packet corruption.
> 
>    When a decapsulator receives a packet, the UDP checksum field MUST
>    be processed. If the UDP checksum is non-zero, the decapsulator MUST
>    verify the checksum before accepting the packet. By default a
>    decapsulator SHOULD accept UDP packets with a zero checksum. A node
>    MAY be configured to disallow zero checksums per [RFC1122]; this may
>    be done selectively, for instance disallowing zero checksums from
>    certain hosts that are known to be sending over paths subject to
>    packet corruption. If verification of a non-zero checksum fails, a
>    decapsulator lacks the capability to verify a non-zero checksum, or
>    a packet with a zero-checksum was received and the decapsulator is
>    configured to disallow, the packet MUST be dropped and an event MAY
>    be logged.
> 
>    6.2. UDP Checksum with IPv6
> 
>    For UDP in IPv6, the UDP checksum MUST be processed as specified in
>    [RFC768] and [RFC2460] for both transmit and receive.
> 
>    When UDP is used over IPv6, the UDP checksum is relied upon to
>    protect both the IPv6 and UDP headers from corruption. As such, A
>    default GRE-in-UDP Tunnel MUST perform UDP checksum; A TMCE GRE-in-
>    UDP Tunnel MAY be configured with the UDP zero-checksum mode if the
>    traffic-managed controlled environment or a set of closely
>    cooperating traffic-managed controlled environments (such as by
>    network operators who have agreed to work together in order to
>    jointly provide specific services) meet at least one of following
>    conditions:
> 
>    a. It is known (perhaps through knowledge of equipment types and
>      lower layer checks) that packet corruption is exceptionally
>      unlikely and where the operator is willing to take the risk of
>      undetected packet corruption.
> 
>    b. It is judged through observational measurements (perhaps of
>      historic or current traffic flows that use a non-zero checksum)
>      that the level of packet corruption is tolerably low and where
>      the operator is willing to take the risk of undetected packet
>      corruption.
> 
> 
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 13]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation            March 2016
> 
>    c. Carrying applications that are tolerant of mis-delivered or
>      corrupted packets (perhaps through higher layer checksum,
>      validation, and retransmission or transmission redundancy) where
>      the operator is willing to rely on the applications using the
>      tunnel to survive any corrupt packets.
> 
>    The following requirements apply to a TMCE GRE-in-UDP tunnel that
>    use UDP zero-checksum mode:
> 
>      a. Use of the UDP checksum with IPv6 MUST be the default
>        configuration of all GRE-in-UDP tunnels.
> 
>      b. The GRE-in-UDP tunnel implementation MUST comply with all
>        requirements specified in Section 4 of [RFC6936] and with
>        requirement 1 specified in Section 5 of [RFC6936].
> 
>      c. The tunnel decapsulator SHOULD only allow the use of UDP zero-
>        checksum mode for IPv6 on a single received UDP Destination
>        Port regardless of the encapsulator. The motivation for this
>        requirement is possible corruption of the UDP Destination Port,
>        which may cause packet delivery to the wrong UDP port. If that
>        other UDP port requires the UDP checksum, the mis-delivered
>        packet will be discarded.
> 
>      d. It is RECOMMENDED that the UDP zero-checksum mode for IPv6 is
>        only enabled for certain selected source addresses. The tunnel
>        decapsulator MUST check that the source and destination IPv6
>        addresses are valid for the GRE-in-UDP tunnel on which the
>        packet was received if that tunnel uses UDP zero-checksum mode
>        and discard any packet for which this check fails.
> 
>      e. The tunnel encapsulator SHOULD use different IPv6 addresses for
>        each GRE-in-UDP tunnel that uses UDP zero-checksum mode
>        regardless of the decapsulator in order to strengthen the
>        decapsulator's check of the IPv6 source address (i.e., the same
>        IPv6 source address SHOULD NOT be used with more than one IPv6
>        destination address, independent of whether that destination
>        address is a unicast or multicast address). When this is not
>        possible, it is RECOMMENDED to use each source IPv6 address for
>        as few UDP zero-checksum mode GRE-in-UDP tunnels as is feasible.
> 
>      f. When any middlebox exists on the path of a GRE-in-UDP tunnel,
>        it is RECOMMENDED to use the default mode, i.e. use UDP
>        checksum, to reduce the chance that the encapsulated packets to
>        be dropped.
> 
> 
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 14]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation            March 2016
> 
>      g. Any middlebox that allows the UDP zero-checksum mode for IPv6
>        MUST comply with requirement 1 and 8-10 in Section 5 of
>        [RFC6936].
> 
>      h. Measures SHOULD be taken to prevent IPv6 traffic with zero UDP
>        checksums from "escaping" to the general Internet; see Section
>        8 for examples of such measures.
> 
>      i. IPv6 traffic with zero UDP checksums MUST be actively monitored
>        for errors by the network operator. For example, the operator
>        may monitor Ethernet layer packet error rates.
> 
>      j. If a packet with a non-zero checksum is received, the checksum
>        MUST be verified before accepting the packet. This is
>        regardless of whether the tunnel encapsulator and decapsulator
>        have been configured with UDP zero-checksum mode.
> 
>    The above requirements do not change either the requirements
>    specified in [RFC2460] as modified by [RFC6935] or the requirements
>    specified in [RFC6936].
> 
>    The requirement to check the source IPv6 address in addition to the
>    destination IPv6 address, plus the strong recommendation against
>    reuse of source IPv6 addresses among GRE-in-UDP tunnels collectively
>    provide some mitigation for the absence of UDP checksum coverage of
>    the IPv6 header. A traffic-managed controlled environment that
>    satisfies at least one of three conditions listed above in this
>    section provides additional assurance.
> 
>    A GRE-in-UDP tunnel is suitable for transmission over lower layers
>    in the traffic-managed controlled environments that are allowed by
>    the exceptions stated above and the rate of corruption of the inner
>    IP packet on such networks is not expected to increase by comparison
>    to GRE traffic that is not encapsulated in UDP.  For these reasons,
>    GRE-in-UDP does not provide an additional integrity check except
>    when GRE checksum is used when UDP zero-checksum mode is used with
>    IPv6, and this design is in accordance with requirements 2, 3 and 5
>    specified in Section 5 of [RFC6936].
> 
>    Generic Router Encapsulation (GRE) does not accumulate incorrect
>    state as a consequence of GRE header corruption. A corrupt GRE
>    packet may result in either packet discard or forwarding of the
>    packet without accumulation of GRE state. Active monitoring of GRE-
>    in-UDP traffic for errors is REQUIRED as occurrence of errors will
>    result in some accumulation of error information outside the
>    protocol for operational and management purposes. This design is in
>    accordance with requirement 4 specified in Section 5 of [RFC6936].
> 
> 
> 
> Yong, Crabber, Xu, Herbert                                    [Page 15]
> --------------------------------------------------------------------------------
> 
> Internet-Draft          GRE-in-UDP Encapsulation            March 2016
> 
>    The remaining requirements specified in Section 5 of [RFC6936] are
>    not applicable to GRE-in-UDP.  Requirements 6 and 7 do not apply
>    because GRE does not include a control feedback mechanism.
>    Requirements 8-10 are middlebox requirements that do not apply to
>    GRE-in-UDP tunnel endpoints (see Section 7.1 for further middlebox
>    discussion).
> 
>    It is worth mentioning that the use of a zero UDP checksum should
>    present the equivalent risk of undetected packet corruption when
>    sending similar packet using GRE-in-IPv6 without UDP [RFC7676] and
>    without GRE checksums.
> 
>    In summary, a TMCE GRE-in-UDP Tunnel is allowed to use UDP-zero-
>    checksum mode for IPv6 when the conditions and requirements stated
>    above are met. Otherwise the UDP checksum need to be used for IPv6
>    as specified in [RFC768] and [RFC2460]. Use of GRE checksum is
>    RECOMMENED when the UDP checksum is not used.
> "
> 
> In GUE, to support UDP-checksum-zero, it said
> 
> " Therefore, when GUE is used over
>    IPv6, either the UDP checksum must be enabled or the GUE header
>    checksum must be used.  An encapsulator MAY set a zero UDP checksum
>    for performance or implementation reasons, in which case the GUE
>    header checksum MUST be used or applicable requirements for using
>    zero UDP checksums in [GREUDP] MUST be met. If the UDP checksum is
>    enabled, then the GUE header checksum should not be used since it is
>    mostly redundant."
> 
> It's easy for me to add the similar words to the IP-in-UDP draft like "the
> applicable requirements for using zero UDP checksum in [GREUDP] MUST be
> met when zero UDP checksum is used by the tunnel ingress". However, the
> major goal for disabling the UDP checksum is to improve the performance.
> When GUE header checksum is used and/or the bunch of applicable
> requirements as described in GRE/UDP are verified, is the goal of improving
> performance still achievable? If not, why not directly enable the UDP-checksum
> instead?
> 
>     - fragmentation support
> 
> In GRE/UDP, it said
> 
> " 4.1. MTU and Fragmentation
> 
>    Regarding packet fragmentation, an encapsulator/decapsulator SHOULD
>    be compliant with [RFC7588] and perform fragmentation before the
>    encapsulation. The size of fragments SHOULD be less or equal to the
>    PMTU associated with the path between the GRE ingress and the GRE
>    egress tunnel endpoints minus the GRE and UDP overhead ..."
> 
> in GUE, it said
> 
> " 4.9. MTU and fragmentation
> 
>    Standard conventions for handling of MTU (Maximum Transmission Unit)
>    and fragmentation in conjunction with networking tunnels
>    (encapsulation of layer 2 or layer 3 packets) should be followed.
>    Details are described in MTU and Fragmentation Issues with In-the-
>    Network Tunneling [RFC4459]... "
> 
> It seems that the only missing thing in the IP-in-UDP draft is to allow the outer
> fragmentation. However, as it said in
> (https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02#page-13), " ...IPsec
> performs only Outer Fragmentation; this distinguishes it from IP-in-IP, which
> performs only Inner Fragmentation. " Note that IP-in-IP is the dominant
> encapsulation choice within Softwires networks. In other words, performing
> only inner fragmentation works very well in practice. Furthermore, the outer
> fragmentation issue (e.g., reassembly cost for the egress) would become even
> worse since the fragments of X-in-UDP packets are more likely to be forwarded
> across different paths than those of X-in-IP and X-in-GRE packets. Hence, I'm
> wondering whether it's worthwhile to support the outer fragmentation on UDP
> encapsulated packets which seems useless in practice.
> 
>     - signalling support (e.g., to test whether a tunnel is up or
>     to measure MTUs)
> 
> I haven't found any description of this in both GRE/UDP and GUE. Did you?
> 
>     - support for robust ID fields (related to fragmentation,
>     e.g., to overcome the limits of IPv4 ID as per RFC 6864)
> 
> I haven't found any description of this in both GRE/UDP and GUE. Did you?
> 
> Xiaohu
> 
> > It is not our obligation to find a way for your document to proceed -
> > that onus is on you.
> >
> > Joe

---

From: "Saumya Dikshit (sadikshi)" <[email-masked]>
Date: Sun, 12 Jun 2016 06:25:47 +0000

In case of typical datacenter brownfield deployments,
the underlay and overlay may be completely disconnected operationally,
hence there can be cases where the mtu mismatch in underlay network may not
be notified to overlay endpoints and subsequently to end hosts.

From: Lloyd Wood <[email-masked]<mailto:[email-masked]>>
Date: Sunday, June 12, 2016 at 10:41 AM
To: Softwires WG <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Xuxiaohu <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] [tsvwg] Is it feasible to perform fragmentation on UDP encapsulated packets.

Fragmentation should be strongly discouraged.

Lloyd Wood
[email-masked]<mailto:[email-masked]>

On Friday, May 27, 2016, 8:10 PM, Xuxiaohu <[email-masked]<mailto:[email-masked]>> wrote:

LISP:

VXLAN:

VXLAN-GPE:

GEVENE:

GUE:

GRE/UDP

Best regards,
Xiaohu

> Cc: [email-masked]<javascript:return>
> > From: Joe Touch [mailto:[email-masked]<javascript:return>]
> > Cc: [email-masked]<javascript:return>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 13 Jun 2016 09:29:05 -0700

On Sat, Jun 11, 2016 at 10:11 PM, Lloyd Wood <[email-masked]> wrote:
> if you've designed a tunnelling solution and you have fragmentation
> happening as a matter of course, you've designed it wrong.
>

As mentioned before we do not always have control of the underlay MTUs
to be able to set Jumbo Frames to make MTU larger than 1500 to account
for tunnel fragmentation. At Facebook for instance, we run IP tunnels
between L4 load balancers and backend servers. In certain POPs we
cannot get more than 1500 MTU in the path. The ingress MTU into the
load balancer is 1500 so it is possible to receive packets from the
Internet that are greater than the tunnel MTU. In this case we will
fragment. Since the tunnel is behind a firewall and our devices and
the path from load balancers to backends is low loss there is little
concern of being DOSed.

Tom

> [email-masked]
> <Note that I have changed the subject of the email hence it has nothing to
> do with the WG adoption call now. It's just a discussion on a particular
> issue which is related to those WGs which are working on UDP tunnels. The
> reason for containing the old email is to use it as a background which may
> be useful for better understanding of this particular issue>
>
> The possible side-effect of performing fragmentation on UDP encapsulated
> packets is to worsen the reassembly burden on tunnel egress since fragments
> of UDP encapsulated packets are more likely to be forwarded across different
> paths towards the tunnel egress than those of IP or GRE encapsulated
> packets.
>
> It seems that most X-over-UDP proposals choose to prohibit the tunnel
> ingress from performing fragmentation on UDP encapsulated packets. See the
> following quoted text regarding fragmentation from those X-over-UDP drafts:
>
> LISP:
>
> However, the above choice seems conflict with the requirements as described
> in https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02
>
>
> I wonder whether the IETF should reach a consensus on whether or not the
> fragmentation on UDP encapsulated packets should be allowed.
>
> Best regards,
> Xiaohu
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 Jun 2016 13:12:31 -0700

> Fragmentation should be strongly discouraged.

Avoided where possible for sure.
> happening as a matter of course, you've designed it wrong.

If you have a solution that recurses (X over X, with some intervening
layers or not), and X has a required minimum message size, then you
either have a solution that - in at least some non-trivial situations -
will fragment as a matter of course or a fantasy that cannot exist.

The only reason we avoid this in typical operation is by avoiding X over
X. Once you have a tunnel, that might not be avoidable (or even
detectable, depending on encryption).

Joe

>     <Note that I have changed the subject of the email hence it has
>     nothing to do with the WG adoption call now. It's just a
>     discussion on a particular issue which is related to those WGs
>     which are working on UDP tunnels. The reason for containing the
>     old email is to use it as a background which may be useful for
>     better understanding of this particular issue>
>
>     The possible side-effect of performing fragmentation on UDP
>     encapsulated packets is to worsen the reassembly burden on tunnel
>     egress since fragments of UDP encapsulated packets are more likely
>     to be forwarded across different paths towards the tunnel egress
>     than those of IP or GRE encapsulated packets.
>
>     It seems that most X-over-UDP proposals choose to prohibit the
>     tunnel ingress from performing fragmentation on UDP encapsulated
>     packets. See the following quoted text regarding fragmentation
>     from those X-over-UDP drafts:
>
>     LISP:
>
>     However, the above choice seems conflict with the requirements as
>     described in https://tools.ietf.org/html/draft-ietf-intarea-tunnels-02
>
>
>     I wonder whether the IETF should reach a consensus on whether or
>     not the fragmentation on UDP encapsulated packets should be allowed.
>      
>     Best regards,
>     Xiaohu
>
>     > Cc: [email-masked] <javascript:return>
>     > > From: Joe Touch [mailto:[email-masked] <javascript:return>]
>     > > Cc: [email-masked] <javascript:return>