Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-frag-fragile-01.txt

From: Ron Bonica <[email-masked]>
Date: Mon, 15 Oct 2018 19:11:48 +0000

Hi Tom,

Hope you are feeling better. Comments inline.....

                      Ron

> Message: 1
> Date: Wed, 10 Oct 2018 13:55:16 -0700
> From: Tom Herbert <[email-masked]>
> To: int-area <[email-masked]>
> Subject: Re: [Int-area] I-D Action:
> 	draft-ietf-intarea-frag-fragile-01.txt
> Message-ID:
> 	<CALx6S34SbOeAw7RbXLAr96fnaRrTPA7MGt6ci41x21LQpKFOKg@ma
> il.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
> 
> Thanks for the updated draft. Here are a few comments:
> 
> "While this document identifies issues associated with IP fragmentation, it
> does not recommend deprecation.  Some applications (e.g., [I-D.ietf-intarea-
> tunnels]) require IP fragmentation."
> 
> I would add that use of fragmentation is also expected to work and be
> common in limited domains where issues in security and interoperability in
> intermediate nodes can be addressed,

Fair enough..... Does the following edit work for you

OLD>
   While this document identifies issues associated with IP
   fragmentation, it does not recommend deprecation.  Some applications
   (e.g., [I-D.ietf-intarea-tunnels]) require IP fragmentation.
<OLD

NEW>
   While this document identifies issues associated with IP
   fragmentation, it does not recommend deprecation.  Some applications
   (e.g., [I-D.ietf-intarea-tunnels]) require IP fragmentation. Furthermore,
   fragmentation is expected to work in limited domains where security and interoperability 
   issues can be addressed.
<NEW

> 
> >From the draft: "This section explains how IP fragmentation reduces
> the reliability of Internet communication." In reality, for most of the examples
> in this section it's really implementations (NAT, firewall, etc.) that are breaking
> IP fragmentation (and other things as well).
> 
> Sections 4.1-4.4 could be summarized to say that some intermediate devices
> perform functions based on inspecting transport layer headers, and these fail
> when fragments are presented that don't contain the transport layer
> information. This could be further subdivided into stateful and stateless
> mechanisms. Policy routing, firewalls, NAT are just examples of functions that
> break with fragmentation.

This is insightful!

The draft can be summarized as follows:

- IP fragmentation is a stateful procedure in an otherwise stateless protocol.
- When the end-to-end principle was in force, fragmentation worked well. The stateful process was relegated to the endpoints and everything just worked.
- As years went by, vendors built stateless middle boxes. Being stateless, they offer good performance at an attractive price. However, being stateless, they are oblivious to fragmentation and behave badly in the presence of fragments.
- These stateless middle boxes are very widely deployed. Because they are economically attractive, even more of them are likely to be deployed in the future.

So, the spirit of the robustness principle, all parties should be conservative in what they do and liberal in what they accept.

- Application developers should avoid reliance on IP Fragmentation. (Don't trip on the bad behavior or middle boxes if you can avoid it).
- Middle box developers should produce devices that don't behave badly in the presence of fragmentation. This may mean that middle boxes should become more stateful to support fragmentation.

> 
> For section 4.5 I'm not sure that being insecure makes fragmentation
> unreliable. Similarly 4.6 isn't really a problem with fragmentation but a
> process to avoid fragmentation. Maybe section 4 should be "Problems related
> to the use of IP fragmentation".

This may be a semantic argument. We agree that security issues exist. We agree that many operators filter fragments because of these security issues.

The only thing that we are arguing about is whether that constitutes a reduction in reliability.

> 
> "It is difficult to determine why network operators drop fragments."
> In what sense is this "difficult"? Difficult because there are no ICMP errors for
> dropped fragments, difficult because network operators don't share their
> policy on fragmentation, or other meaning?

I can observe operator behavior. But I can only speculate about their motivation.

                                                            Ron

> Tom
*************************************

---

From: Tom Herbert <[email-masked]>
Date: Mon, 15 Oct 2018 12:30:32 -0700

>    fragmentation is expected to work in limited domains where security and interoperability

It's good.

> - As years went by, vendors built stateless middle boxes. Being stateless, they offer good performance at an attractive price. However, being stateless, they are oblivious to fragmentation and beha…

I don't follow. Isn't the problem with _stateful_ devices, or at least
intermediate devices that choose to parse transport layer.

> - These stateless middle boxes are very widely deployed. Because they are economically attractive, even more of them are likely to be deployed in the future.

A stateless middlebox that doesn't parse transport layer should work
perfectly well with fragmentation. This would describe a device
adhering to end to end model.

> - Application developers should avoid reliance on IP Fragmentation. (Don't trip on the bad behavior or middle boxes if you can avoid it).

IMO, this argument might make sense if the problem was only
fragmentation since there are inherent problems of fragmentation. But
there are a whole bunch of things with similar characteristics like
IPv6 extension headers or non-TCP protocols. We need a path where
those can be prductively used on the Internet.

> I can observe operator behavior. But I can only speculate about their motivation.

Can we poll the operators? I don't see why this should be a secret. My
suspision is that they do it because they've always done it and aren't
going to change unless customers are clamoring for.

Tom

> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Fred Baker <[email-masked]>
Date: Mon, 15 Oct 2018 12:50:02 -0700

> On Oct 15, 2018, at 12:11 PM, Ron Bonica <[email-masked]> wrote:
> 

I might add one thought to 7.3: recommend that middle boxen reassemble a datagram before operating on it. The attacks that happen with fragmentation result in issues with reassembly, and the obvious solution is to detect the situation and drop the datagram in question.

I would also comment that the first fragment created (the one at offset zero) might not be the first fragment received. The second bullet in 7.3, the one about selecting a next-hop, assumes that the first fragment is also the first one received. However, if the datagram were reassembled before operating on it, that wouldn't be an issue.

---

From: Ron Bonica <[email-masked]>
Date: Mon, 15 Oct 2018 20:35:36 +0000

Hi Tom,

The examples in Sections 4.1-4.4 all refer to stateless devices. The problem could be solved by making them all stateful. However, that may not be practical because of:

- price/performance concerns
- size of the installed base.

Years back, Tom Taylor and others speculated on why operators drop fragmented packets. This work was never complete.

                    Ron

> -----Original Message-----
> Sent: Monday, October 15, 2018 3:31 PM
> To: Ron Bonica <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-frag-fragile-01.txt
> 
> >> "While this document identifies issues associated with IP
> >> (e.g., [I-D.ietf-intarea-
> >> common in limited domains where issues in security and
> >> interoperability in intermediate nodes can be addressed,
> >
> >> the reliability of Internet communication." In reality, for most of
> >> the examples in this section it's really implementations (NAT,
> >> firewall, etc.) that are breaking IP fragmentation (and other things as well).
> >>
> >> Sections 4.1-4.4 could be summarized to say that some intermediate
> >> devices perform functions based on inspecting transport layer
> >> headers, and these fail when fragments are presented that don't
> >> contain the transport layer information. This could be further
> >> subdivided into stateful and stateless mechanisms. Policy routing,
> >> firewalls, NAT are just examples of functions that break with
> fragmentation.
> >
> > This is insightful!
> >
> > - When the end-to-end principle was in force, fragmentation worked well.
> The stateful process was relegated to the endpoints and everything just
> worked.
> > - As years went by, vendors built stateless middle boxes. Being stateless, they
> offer good performance at an attractive price. However, being stateless, they
> are oblivious to fragmentation and behave badly in the presence of fragments.
> 
> > - These stateless middle boxes are very widely deployed. Because they are
> economically attractive, even more of them are likely to be deployed in the
> future.
> 
> A stateless middlebox that doesn't parse transport layer should work perfectly
> well with fragmentation. This would describe a device adhering to end to end
> model.
> 
> >
> > So, the spirit of the robustness principle, all parties should be conservative in
> what they do and liberal in what they accept.
> >
> > - Application developers should avoid reliance on IP Fragmentation. (Don't
> trip on the bad behavior or middle boxes if you can avoid it).
> 
> IMO, this argument might make sense if the problem was only fragmentation
> since there are inherent problems of fragmentation. But there are a whole
> bunch of things with similar characteristics like
> IPv6 extension headers or non-TCP protocols. We need a path where those
> can be prductively used on the Internet.
> 
> > - Middle box developers should produce devices that don't behave badly in
> the presence of fragmentation. This may mean that middle boxes should
> become more stateful to support fragmentation.
> >
> >>
> >> unreliable. Similarly 4.6 isn't really a problem with fragmentation
> >> but a process to avoid fragmentation. Maybe section 4 should be
> >> "Problems related to the use of IP fragmentation".
> >
> > This may be a semantic argument. We agree that security issues exist. We
> agree that many operators filter fragments because of these security issues.
> >
> >> In what sense is this "difficult"? Difficult because there are no
> >> ICMP errors for dropped fragments, difficult because network
> >> operators don't share their policy on fragmentation, or other meaning?
> >
> suspision is that they do it because they've always done it and aren't going to
> change unless customers are clamoring for.
> 
> Tom
> 
> >
> >                                                             Ron
> >
> >>
> >> Tom
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> > man_listinfo_int-2Darea&d=DwIBaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-
> ndb3voD
> > TXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=TA5bXwvykXToe
> > 3EIRTv5F-u14FctzxaZwzNHU6DsHg8&s=Y-
> DP3RKKdPSm6X6TJUvPy0YrQb2TZnvFxZQDm
> > tBzNIk&e=

---

From: Ron Bonica <[email-masked]>
Date: Mon, 15 Oct 2018 20:38:40 +0000

Hi Fred,

The first iteration of Section 7.3 actually included the word "reassemble". That is one possible implementation.

I dropped the word when I realized that there may be other ways to get the desired behavior. While they don't all require reassembly, that all require the maintenance of a little more state.

                                                                                           Ron

> From: Fred Baker <[email-masked]>
> Sent: Monday, October 15, 2018 3:50 PM
> I might add one thought to 7.3: recommend that middle boxen reassemble a
> datagram before operating on it. The attacks that happen with fragmentation
> result in issues with reassembly, and the obvious solution is to detect the
> situation and drop the datagram in question.
> 
> I would also comment that the first fragment created (the one at offset zero)
> might not be the first fragment received. The second bullet in 7.3, the one
> about selecting a next-hop, assumes that the first fragment is also the first one
> received. However, if the datagram were reassembled before operating on it,
> that wouldn't be an issue.

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Mon, 15 Oct 2018 20:45:44 +0000

Hi Ron,

There is a technique known as "Virtual Fragmentation Reassembly" where
the middlebox gathers all fragments of a datagram, performs any necessary
checks and then releases the fragments without reassembling them so that
the destination host still has to reassemble. Is this one of the techniques you
are referring to?

Thanks - Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Ron Bonica
> Sent: Monday, October 15, 2018 1:39 PM
> To: Fred Baker <[email-masked]>
> Cc: [email-masked]
> I dropped the word when I realized that there may be other ways to get the desired behavior. While they don't all require
> reassembly, that all require the maintenance of a little more state.
> 
>                                                                                            Ron
> 
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 15 Oct 2018 13:53:35 -0700

> The examples in Sections 4.1-4.4 all refer to stateless devices. The problem could be solved by making them all stateful. However, that may not be practical because of:
Ron,

I think you're referring to statefull in this context as meaning
reassembly in the network. That could be done, but really doesn't
solve the problem because now the problems of being stateful are
raised (like every fragment MUST be forwarded through some
intermediate device).

I'm not sure that the distinction between stateful and stateless in
any of these cases is so relevant (state here being more generic to
include transport layer state). For instance, IP fragmentation will
break stateful firewalls and stateful NAT, which are very common, just
as easily as the stateless variants. In any case I don't think that's
really relevant, the common problem across all of these is that some
intermediate devices want to inspect transport layer headers in
packets and non-first fragments don't carry transport headers.

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 15 Oct 2018 14:09:50 -0700

FWIW - it also involves forwarding all the fragments according to info
in the first, which might include NAT translation and/or port-based
forwarding. 

That state can also be soft - first-fragments create the entry and
non-first frags are queued until a first-frag comes through. 

Joe

On 2018-10-15 13:45, Templin (US), Fred L wrote:

> - Application developers should avoid reliance on IP Fragmentation. (Don't trip on the bad behavior or middle boxes if you can avoid it). - Middle box developers should produce devices that don't behave badly in the presence of fragmentation. This may mean that middle boxes should

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Oct 2018 11:07:53 +1300

> Years back, Tom Taylor and others speculated on why operators drop fragmented packets. This work was never complete.

I seem to recall that the speculation was that they drop fragments (after the first one) because they cannot meaningfully apply firewall rules to it, since the layer 4 header is absent. So better safe than sorry.

(Obviously - that is a stateless mechanism, with the decision taken at line speed.)

However, in RFC3234 (February 2002) we wrote this:

   "Stateless" firewalls typically allow all IP fragments through since
   they do not contain enough upper-layer header information to make a
   filtering decision.  Many "stateful" firewalls therefore reassemble
   IP fragments (and re-fragment if necessary) in order to avoid leaking
   fragments, particularly fragments that may exploit bugs in the
   reassembly implementations of end receivers.

I assume we had some basis for those factual claims, but I don't see anything in my archive.

   Brian

---

From: Fred Baker <[email-masked]>
Date: Mon, 15 Oct 2018 15:10:57 -0700

> On Oct 15, 2018, at 1:50 PM, Ron Bonica <[email-masked]> wrote:
> 
> Exactly, but I didn't want to introduce and define the term "Virtual Fragment Reassembly". So, I just described the output. The middle box:
> 
> - makes a decision on that first fragment

The one at offset 0 (and therefore containing the transport header), or the first one it receives (which might be at the final offset or any other)? What if they're not the same?

> - applies that decision to the first fragment and all subsequent fragments
> 
>                                                  Ron
> 
> 
>> From: Templin (US), Fred L <[email-masked]>
>> Sent: Monday, October 15, 2018 4:46 PM
>> To: Ron Bonica <[email-masked]>; Fred Baker
>> <[email-masked]>
>> Cc: [email-masked]
>> Subject: RE: [Int-area] I-D Action: draft-ietf-intarea-frag-fragile-01.txt
>> 
>> Hi Ron,
>> 
>> There is a technique known as "Virtual Fragmentation Reassembly" where the
>> middlebox gathers all fragments of a datagram, performs any necessary
>> checks and then releases the fragments without reassembling them so that the
>> destination host still has to reassemble. Is this one of the techniques you are
>> referring to?
>> 
>> Thanks - Fred
>> 
>>> I dropped the word when I realized that there may be other ways to get
>>> the desired behavior. While they don't all require reassembly, that all require
>> the maintenance of a little more state.
>>> 
>>> 
>>> Ron
>>> 
>>> 
>>>> the presence of fragmentation. This may mean that middle boxes
>>>> should become more stateful to support fragmentation.
>>>> 
>>>> I might add one thought to 7.3: recommend that middle boxen
>>>> reassemble a datagram before operating on it. The attacks that
>>>> happen with fragmentation result in issues with reassembly, and the
>>>> obvious solution is to detect the situation and drop the datagram in
>> question.
>>>> 
>>>> I would also comment that the first fragment created (the one at
>>>> offset zero) might not be the first fragment received. The second
>>>> bullet in 7.3, the one about selecting a next-hop, assumes that the
>>>> first fragment is also the first one received. However, if the
>>>> datagram were reassembled before operating on it, that wouldn't be an
>> issue.
>>> 
>>> man_listinfo_int-2Darea&d=DwIFAg&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-
>> ndb3voD
>> AWF2EfpHcAwrDThKP8&m=9LgfnIl19MFJu
>>> 
>> utKdbF3K8gh03dGa8ysrFnSrEVkXoE&s=mLM6w6j83V7i_S2wdo_SdtsOfd6Td
>> b_XiWnb_
>>> HigsBQ&e=

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: Tom Herbert <[email-masked]>
Date: Mon, 15 Oct 2018 15:26:17 -0700

On Mon, Oct 15, 2018, 3:11 PM Fred Baker <[email-masked]> wrote:

> > Exactly, but I didn't want to introduce and define the term "Virtual
> Fragment Reassembly". So, I just described the output. The middle box:
> >
> The one at offset 0 (and therefore containing the transport header), or
> the first one it receives (which might be at the final offset or any
> other)? What if they're not the same?
>

For instance, a while back Linux always sent the first fragment last when
fragmenting. This wreaked havoc on middle boxes trying to reassemble. That
was changed to send first fragment first. But even so, I think there might
be some ECMP implementations that could route first fragment (off=0)
differently to create ooo packets relative to fragment train.

Tom

> the
> >>> The first iteration of Section 7.3 actually included the word
> "reassemble".
> >> That is one possible implementation.
> >>>

---

From: Ron Bonica <[email-masked]>
Date: Mon, 15 Oct 2018 20:50:30 +0000

Ron

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Oct 2018 13:48:34 +1300

On 2018-10-16 11:26, Tom Herbert wrote:
> 
> 
> On Mon, Oct 15, 2018, 3:11 PM Fred Baker <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> 
>     > On Oct 15, 2018, at 1:50 PM, Ron Bonica <[email-masked] <mailto:[email-masked]>> wrote:
>     >
> For instance, a while back Linux always sent the first fragment last when fragmenting. This wreaked havoc on middle boxes trying to reassemble. That was changed to send first fragment first. But even so, I think there might be some ECMP implementations that could route first fragment (off=0) differently to create ooo packets relative to fragment train.

Clearly, at some stage we had the hope that the flow label would be useful to mitigate this sort of problem. It's intentionally included in every IPv6 fragment.

   Brian

>     >> From: Templin (US), Fred L <[email-masked] <mailto:[email-masked]>>
>     >> To: Ron Bonica <[email-masked] <mailto:[email-masked]>>; Fred Baker
>     >> <[email-masked] <mailto:[email-masked]>>
>     >> Cc: [email-masked] <mailto:[email-masked]>
>     >>> From: Int-area [mailto:[email-masked] <mailto:int-area-bounces@ietf..org>] On Behalf Of Ron
>     >>> Bonica
>     >>> To: Fred Baker <[email-masked] <mailto:[email-masked]>>
>     >>> Cc: [email-masked] <mailto:[email-masked]>
>     >>>> From: Fred Baker <[email-masked] <mailto:[email-masked]>>
>     >>>> To: Ron Bonica <[email-masked] <mailto:[email-masked]>>
>     >>>> Cc: [email-masked] <mailto:[email-masked]>
>     >>>>> On Oct 15, 2018, at 12:11 PM, Ron Bonica <[email-masked] <mailto:[email-masked]>>
>     >> wrote:
>     >>>>>
>     >>> [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 15 Oct 2018 18:21:33 -0700

On Mon, Oct 15, 2018, 5:48 PM Brian E Carpenter <[email-masked]>
wrote:

> > On Mon, Oct 15, 2018, 3:11 PM Fred Baker <[email-masked]
> <mailto:[email-masked]>> wrote:
> >
> >
> >
> >     > Exactly, but I didn't want to introduce and define the term
> "Virtual Fragment Reassembly". So, I just described the output. The middle
> box:
> >     >
> >     The one at offset 0 (and therefore containing the transport header),
> or the first one it receives (which might be at the final offset or any
> > For instance, a while back Linux always sent the first fragment last
> when fragmenting. This wreaked havoc on middle boxes trying to reassemble.
> That was changed to send first fragment first. But even so, I think there
> might be some ECMP implementations that could route first fragment (off=0)
> Clearly, at some stage we had the hope that the flow label would be useful
> to mitigate this sort of problem. It's intentionally included in every IPv6
> fragment.
>

Brian,

Agreed, flow label fixes ECMP problem in fragmentation. The problems with
flow label depoyment in ECMP have been more about whether it's consistent
for every packet in connection (to satisfy ad hoc requirements that
stateful devices see every packet for a connection). That is something that
has been addressed more recently.

Tom

> >     >>> From: Int-area [mailto:[email-masked] <mailto:
> int-area-bounces@ietf..org>] On Behalf Of Ron
> >     >>> Bonica
> >     >>>> To: Ron Bonica <[email-masked] <mailto:[email-masked]
> >>
> >     >>>>> So, the spirit of the robustness principle, all parties should
> be
> >     >>>>> conservative in
> >     >>>>> - Middle box developers should produce devices that don't
> behave
> >     >>>>> badly in