Subject: [Int-area] Stateless devices and IP fragmentation

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Nov 2018 08:07:33 -0800

Hi Ron,

In the int-area meeting you mentioned that the proposed solution to
stateless middleboxes and fragmentation is to performing some
reassembly at the device thereby making it "a little" stateful. I
doubt this is a feasible solution. This would force devices that were
previously work conserving to be non-work conserving which entails
significant complexity. They will need memory to hold fragments,
timers, and additional logic. I would expect that the appeal of
stateless devices is that they can be low end and low cost, and I
believe it's going to be a hard sell to vendors to do this.

This also brings in the problems of stateful devices. Reassembly in
the network must assume that all fragments hit the same middlebox
device, there is no guarantee for that. Additionally, this introduces
the possibility of DOS attacks on reassembly at the middlebox. Since
the middlebox is likely a front end to many hosts, a DOS attack would
potentially block the service for all the hosts.

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 7 Nov 2018 08:28:44 -0800

The current “solution” of ignoring this need isn’t tenable either, though. It isn’t “work conserving” to simply drop traffic that is deemed too expensive to handle.

It’s always appealing - and profitable - for vendors to shortchange customers. 

Joe

> On Nov 7, 2018, at 8:07 AM, Tom Herbert <[email-masked]> wrote:
> 
> Hi Ron,
> 
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Nov 2018 08:45:27 -0800

> The current “solution” of ignoring this need isn’t tenable either, though. It isn’t “work conserving” to simply drop traffic that is deemed too expensive to handle.

The device can choose to drop the first fragment, but allow non-first
fragments to pass. This retains the work conserving nature of the
stateless device, and provides the required functionality of the
firewall since the packet cannot be reassembled at the destination.

> It’s always appealing - and profitable - for vendors to shortchange customers.

I don't believe customers will have much interest in paying premium
for a device just because it deals with fragmentation.

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 7 Nov 2018 09:24:41 -0800

It conserves no work to pass fragments that will be dropped later. 

Customers want devices that work. This is a tragedy of the commons issue. Nobody wants other people’s devices to drop their fragments. But nobody wants to pay to not drop fragments of others. 

We need compliance checks to fix this, or to find another way to put more pain where the problem lies. 

Joe

> On Nov 7, 2018, at 8:45 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Nov 2018 10:08:32 -0800

> Customers want devices that work. This is a tragedy of the commons issue. Nobody wants other people’s devices to drop their fragments. But nobody wants to pay to not drop fragments of others.

But reassembly in the network doesn't work. As I said, it requires
that all fragments go through the intermediate device performing
reassembly, there is no requirement in IP that that has to be true.
This is a known problem of stateful devices, they implicitly place
strict requirements on routing in the network in order to "work".
Stateless devices do not have this problem.

> We need compliance checks to fix this, or to find another way to put more pain where the problem lies.

I would like to hear from the vendors of the stateless devices whether
they think this is feasible solution.

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 07 Nov 2018 10:32:32 -0800

Sorry - previous response got cut off. Let's try again...

On 2018-11-07 10:08, Tom Herbert wrote:

>> Customers want devices that work. This is a tragedy of the commons issue. Nobody wants other people's devices to drop their fragments. But nobody wants to pay to not drop fragments of others.
> 
> But reassembly in the network doesn't work.

First, I'm talking about virtual reassembly, i.e., keeping state to let
fragments follow their leader. 

> As I said, it requires

Agreed, but that's a problem for exactly because devices like this
forward and/or modify packets based on information beyond the IP header,
which are incorrectly deployed if they don't know all fragments will be
seen. 

> strict requirements on routing in the network in order to "work".

It's a known problem due to layer violation. Just because you WANT to do
something doesn't mean it works. 

The bigger problem is the tragedy of the commons - these devices create
a mess that affects others. In the case of not seeing all the fragments,
the issue is coming back to bite them - i.e., it's devices LIKE THEM
that make problems for devices LIKE THEM. 

> Stateless devices do not have this problem.

It's not just stateless; it's layer-following. 

> they think this is feasible solution.

They will answer as to what is economically in their individual
interest. That's how we got IPv6 NATs, etc. 

Joe 

> The current "solution" of ignoring this need isn't tenable either, though. It isn't "work conserving" to simply drop traffic that is deemed too expensive to handle.
> 
> It's always appealing - and profitable - for vendors to shortchange customers.

---

From: Ole Troan <[email-masked]>
Date: Thu, 8 Nov 2018 09:50:15 +0700

> they think this is feasible solution.

Implementations tend to live within the laws of physics, so demanding something isn’t going to make it so.

In our open source virtual reassembly implementation we go to great lengths to try to make it work.
But fragments will have a significantly higher drop probability.

Virtual reassembly opens up a big attack vector, and a lot of work is required to be able to handle these attacks.
At least in our implementation we do not have enough buffers to be anywhere near RFC compliant.
IPv4 has a 15s reassembly timer, IPv6 a 60s reassembly timer. If I remember correctly we have default 100ms reassembly timer, with a maximum fragment chain of 3.
And a maximum outstanding reassemblies of 1000. Which is not very much given we can process hundreds of millions of packets per second.

I have looked at some packet traces with fragments. From the ones I saw:
 - a significant amount of fragmented packets were attackes UDP port 80 etc.
 - didn’t reassemble at all
 - were out of order

I would be very interested if anyone had more data on this. Both to tune our virtual reassembly algorithm and to see how big the problem is.

Btw: This is for IPv4 reassembly. Where we as a community have decided that we must make forwarding decisions on transport layer ports.
So please don’t bother with the “ah your middlebox shouldn’t be there in the first place”.

Code: https://git.fd.io/vpp/tree/src/plugins/map/ip4_map.c#n511

Cheers,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Thu, 8 Nov 2018 08:09:25 -0800

On Wed, Nov 7, 2018 at 10:32 AM, Joe Touch <[email-masked]> wrote:
> Customers want devices that work. This is a tragedy of the commons issue.
> Nobody wants other people's devices to drop their fragments. But nobody
> wants to pay to not drop fragments of others.
>
>

That assumes that the first fragment is received first. IP does not
guarantee that. If a non-first fragment is received before a first
fragment then it needs to be queued pending the arrival of the first
fragment. Once the first fragment is received then the decision can be
made whether to forward the queued fragments. This is a non work
conserving algorithm and creates a new memory requirement on otherwise
stateless devices to hold queued packets.

Tom

> Agreed, but that's a problem for exactly because devices like this forward
> and/or modify packets based on information beyond the IP header, which are
> incorrectly deployed if they don't know all fragments will be seen.
>
>
>
> The bigger problem is the tragedy of the commons - these devices create a
> mess that affects others. In the case of not seeing all the fragments, the
> issue is coming back to bite them - i.e., it's devices LIKE THEM that make
> problems for devices LIKE THEM.
>
>
>
> They will answer as to what is economically in their individual interest.
> That's how we got IPv6 NATs, etc.
>
> Joe
>
>
>
>
>
>
> Tom
>
>
> Joe
>
> The current "solution" of ignoring this need isn't tenable either, though.
> It isn't "work conserving" to simply drop traffic that is deemed too
> expensive to handle.
>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Nov 2018 08:15:58 -0800

Dropping those non initial fragments isn’t work conserving either.

The point is that work conservation needs to consider the impact of the forwarded or dropped messages, not merely whether info is passed on an outgoing wire.

And to be clear, the only party that puts the onus of keeping state on this device is THIS DEVICE.

It always can forward based on IP info alone. It is only the desire to alter packets in transit or forward based on transport info that gets in the way.

You make the mess, you clean it up, IMO.

Joe

> On Nov 8, 2018, at 8:09 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Ole Troan <[email-masked]>
Date: Thu, 8 Nov 2018 23:43:38 +0700

> On 8 Nov 2018, at 23:15, Joe Touch <[email-masked]> wrote:
> 
> 

Joe, I have told you many times over that this is wrong for IPv4. Repeating it does not make it less wrong. 

Ole

---

From: Tom Herbert <[email-masked]>
Date: Thu, 8 Nov 2018 08:48:41 -0800

> It always can forward based on IP info alone. It is only the desire to alter packets in transit or forward based on transport info that gets in the way.

Exactly my point. For fragmented packets a stateless devices can
simply forward non-first fragments and inspect first fragments. This
is stateless and the allows the device to provide the functionality
that requires DPI. Sure, this means the destination may try to
reassemble a packet when the first fragment is dropped, but that's not
much of an issue. Unlike intermediate devices, hosts are built to
properly reassemble packets and to deal with reassembly failures and
other edge conditions. If you're worried about this being used for
DOS, then I claim it's about as easy to DOS virtual reassembly by
sending a bunch of non-first fragments towards the device doing
virtual reassembly. Alternatively, one could send fragmented packets
that are likely to be accepted by the stateless firewall (like DNS
maybe) where the last fragment is withheld; in this case the DOS
attack is "two for one" since both the destination and the
intermediate device maintain state for reassmbly that is the target of
the attack.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Nov 2018 09:02:57 -0800

Please indicate the basis of your belief. Mine is RFC1812. 

> On Nov 8, 2018, at 8:43 AM, Ole Troan <[email-masked]> wrote:
> 
> 
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Nov 2018 09:07:18 -0800

Boxes that do DPI ***are*** edge devices, regardless of where they are(mid)placed. 

That’s why they should be required to do the equivalent work. 

Just because they ‘don’t want to’ doesn’t make it so. 

Joe

> On Nov 8, 2018, at 8:48 AM, Tom Herbert <[email-masked]> wrote:
> 
> Unlike intermediate devices, hosts are built to
> other edge conditions

---

From: Ole Troan <[email-masked]>
Date: Fri, 9 Nov 2018 00:13:16 +0700

> On 9 Nov 2018, at 00:02, Joe Touch <[email-masked]> wrote:
> 

CGNs, RFC6346 et al. 
Your belief system is unfortunately outdated. 
I don’t like it anymore than you do. 

And no, you cannot pretend these don’t exist by declaring them extensions of a host. Not even under same administrative control. 

Ole

---

From: Tom Herbert <[email-masked]>
Date: Thu, 8 Nov 2018 09:27:53 -0800

> Boxes that do DPI ***are*** edge devices, regardless of where they are(mid)placed.

I don't know what "edge devices" are in the context of IP protocols. I
do know these devices are NOT hosts-- the destination address of
packets is not their address. They cannot do the equivalent work of
hosts since they don't have all required information. I've already
pointed out the reassembly in the network doesn't work because there's
no guarantee that an intermediate node will see all the packets.

> That’s why they should be required to do the equivalent work.

They can't do the equivalent work of hosts for the reasons above. I
don't see how we can require something that is impossible to do
correctly.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Nov 2018 09:55:22 -0800

Experimental. For a reason. 

> On Nov 8, 2018, at 9:13 AM, Ole Troan <[email-masked]> wrote:
> 
> 
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Nov 2018 14:05:31 -0800

> On Nov 8, 2018, at 9:27 AM, Tom Herbert <[email-masked]> wrote:
> 
> I don't know what "edge devices" are in the context of IP protocols.

Simple. Anything that depends on information beyond the IP header. It’s what they look at, not where they are (mis)located. 

> I
> packets is not their address.

They’re reading info intended for that host and inferring its meaning the same way. 

> They cannot do the equivalent work of
> hosts since they don't have all required information.

That’s because of where they’re (mis)located. That’s the fault of how they’re deployed vs what they’re trying to do. 

> I've already

Again, this is the fault of what is desired vs where the box is placed. If you don’t have the info to do your job, you have the wrong job in the first place. 

Joe

---

From: Ole Troan <[email-masked]>
Date: Fri, 9 Nov 2018 07:53:43 +0700

Joe,

> On 9 Nov 2018, at 00:55, Joe Touch <[email-masked]> wrote:
> 

No. The protocol instances are standard documents. Rfc7596, 7597, 7599. 

We have had this discussion before. 
We are not going to agree, and there might not be any purpose disturbing the wg with this. 

But just as you know these mechanisms are rolled out in increasing numbers (of course, since there is no choice until universal deployment of IPv6) and there are millions of users behind them already. 

Ole

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Nov 2018 17:07:28 -0800

And none of these update RFC1812.

These things can be rolled out in any numbers you like - they are creating the problem that only they can clean up. Killing capability in the rest of the Internet to make these mechanisms work isn’t a viable solution and never has been.

Joe

> On Nov 8, 2018, at 4:53 PM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
>

---

From: Ole Troan <[email-masked]>
Date: Mon, 12 Nov 2018 10:36:25 +0100

> These things can be rolled out in any numbers you like - they are creating the problem that only they can clean up. Killing capability in the rest of the Internet to make these mechanisms work isn’…

And if that makes the IPv4 Internet die quicker, I’ll take that as a feature.

Cheers,
Ole

---

From: Joe Touch <[email-masked]>
Date: Mon, 12 Nov 2018 02:11:56 -0800

> And if that makes the IPv4 Internet die quicker, I’ll take that as a feature.

Sure, as long as your goal is to kill IPv6 too.

Joe

---

From: Ole Troan <[email-masked]>
Date: Mon, 12 Nov 2018 11:22:31 +0100

> On 12 Nov 2018, at 11:11, Joe Touch <[email-masked]> wrote:
> 
> 
> 

And that’s probably as good an end to this thread as any…

Ole

---

From: Ron Bonica <[email-masked]>
Date: Mon, 12 Nov 2018 18:35:56 +0000

Folks,

It warms my heart to see old friends share a spirited debate ;-)

But what does this mean for draft-ietf-intarea-frag-fragile? The draft observes that some stateless middle boxes behave poorly in the presence of IP fragments. In order to avoid this situation, the draft makes the following recommendations:

1) Application developers SHOULD NOT develop new applications that rely on IP fragmentation.
2) Application-layer protocols that depend upon IPv6 fragmentation SHOULD be updated to break that dependency.
3) Middle box developers SHOULD implement devices that support IP fragmentation. These boxes SHOULD not fail or cause failures when processing fragmented IP packets.

Note the generous application of the word "SHOULD". These are recommendations, not iron-fisted dictates. If  developers heed these recommendations, they will avoid some amount of pain.  However, it is understood that these recommendations will not be heeded by all. Hence the word "SHOULD".

                                                                                                        Ron
> -----Original Message-----
> From: Ole Troan <[email-masked]>
> Sent: Monday, November 12, 2018 5:23 AM
> To: Joe Touch <[email-masked]>
> Cc: Tom Herbert <[email-masked]>; Ron Bonica
> <[email-masked]>; int-area <[email-masked]>
> Subject: Re: [Int-area] Stateless devices and IP fragmentation
> 
> 
> 
> >>> These things can be rolled out in any numbers you like - they are creating
> the problem that only they can clean up. Killing capability in the rest of the
> Internet to make these mechanisms work isn’t a viable solution and never has
> been.
> >>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Nov 2018 10:54:23 -0800

> 3) Middle box developers SHOULD implement devices that support IP fragmentation. These boxes SHOULD not fail or cause failures when processing fragmented IP packets.
Ron,

I'm not sure what "SHOULD implement devices that support IP
fragmentation" means. By specification, intermediate devices are
supposed to ignore fragments and hence implicitly support them. But, I
think the implication of this discussion is that there are middleboxes
that do process fragments somehow leading to detrimental or at least
non-compliant behavior. If the intent is to address those
implementations, they will need guidance on how to fix things. IMO, it
wouldn't be correct or feasible to recommend they do virtual
reassembly.

Tom

---

From: Joe Touch <[email-masked]>
Date: Mon, 12 Nov 2018 11:01:34 -0800

Ron 

As I noted, SHOULDs always need to explain why they are not MUSTs and under what conditions they should be excepted... see below...

Joe 

> On Nov 12, 2018, at 10:35 AM, Ron Bonica <[email-masked]> wrote:
> 
> Folks,
> 

I’d call this protocol developers, not just application developers (to cover tunnel and security designs too), I.e., anything that goes as payload in IP...

They CAN rely on IP *source* fragmentation when needing to support direct IP-in-IP tunnels, as for existing protocols such as IPsec tunnel mode and IP-in-IP encapsulation.

> 2) Application-layer protocols that depend upon IPv6 fragmentation SHOULD be updated to break that dependency.

Same issue...

> 3) Middle box developers SHOULD implement devices that support IP fragmentation.

Here’s the problem - under what cases is an exception warranted? 

AFAICT, Middlebox developers SHOULD support processing of IP fragments with the same services they support atomic data grams.

> These boxes SHOULD not fail or cause failures when processing fragmented IP packets.

Middleboxes SHOULD forward IP fragments without processing if processing cannot be supported. The MAY drop fragments only when there are significant indications of an overload condition.

How’s that? That’s still staying in SHOULD/MAY land as much as possible.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 12 Nov 2018 11:03:32 -0800

See my suggested text - I.e., if you can’t handle the fragments they way you WANT, then you need to forward them.

IMO, if you don’t know what to do with fragments you can’t process AT ALL, then your middlebox service is either improperly designed or improperly deployed.

Joe

> On Nov 12, 2018, at 10:54 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Ron Bonica <[email-masked]>
Date: Mon, 12 Nov 2018 21:08:22 +0000

Joe, Tom,

Does the following text work for you?

                                    Ron

======================================================= 

7.1.  For Protocol Developers

   Protocol developers SHOULD NOT develop new protocols that rely
   on IP fragmentation. However, they MAY develop new protocols that
   rely on IP fragmentation when no viable alternative exists.

   Legacy protocols that depend upon IPv6 fragmentation SHOULD be updated 
   to break that dependency.  However, in some cases, there may be no viable
   alternative to IPv6 fragmentation (e.g., IPSec tunnel mode, IP-in-IP encapsulation).
   In these cases, the protocol MAY continue to rely on IPv6 fragmentation.

   A protocol can break its dependency upon IP fragmentation by
   using a sufficiently small MTU (e.g.  the protocol minimum link MTU),
   disabling IP fragmentation, and ensuring that the transport protocol in
   use adapts its segment size to the MTU.  Another approach is to deploy
   a sufficiently reliable PMTU discovery mechanism (e.g., PLMPTUD).

7.3.  For Middle Box Developers

   Middle box developers SHOULD implement devices that do not fail or cause
   failures when processing fragmented IP packets.

   For example, in order to support IP fragmentation, a load balancer
   might execute the following procedure:

   o  Receive a fragmented packet

   o  Identify a next-hop using information drawn from the first
      fragment (i.e., the fragment containing offset 0)

   o  Forward the first fragment and all subsequent fragments through
      the above-mentioned next-hop

   In some cases, given price and performance constraints, it may be impossible to design a middle box that processes IP fragments correctly. In these cases, the vendor MUST document the issue and propose work-arounds.

> From: Joe Touch <[email-masked]>
> Sent: Monday, November 12, 2018 2:02 PM
> To: Ron Bonica <[email-masked]>
> Cc: Ole Troan <[email-masked]>; Tom Herbert
> <[email-masked]>; int-area <[email-masked]>
> As I noted, SHOULDs always need to explain why they are not MUSTs and
> under what conditions they should be excepted... see below...
> 
> Joe
> 
> > But what does this mean for draft-ietf-intarea-frag-fragile? The draft
> observes that some stateless middle boxes behave poorly in the presence of IP
> fragments. In order to avoid this situation, the draft makes the following
> recommendations:
> >
> I’d call this protocol developers, not just application developers (to cover
> tunnel and security designs too), I.e., anything that goes as payload in IP...
> 
> They CAN rely on IP *source* fragmentation when needing to support direct
> IP-in-IP tunnels, as for existing protocols such as IPsec tunnel mode and IP-in-
> IP encapsulation.
> 
> > 2) Application-layer protocols that depend upon IPv6 fragmentation
> SHOULD be updated to break that dependency.
> 
> Same issue...
> 
> AFAICT, Middlebox developers SHOULD support processing of IP fragments
> with the same services they support atomic data grams.
> 
> Middleboxes SHOULD forward IP fragments without processing if processing
> cannot be supported. The MAY drop fragments only when there are significant
> indications of an overload condition.
> 
> > Note the generous application of the word "SHOULD". These are
> recommendations, not iron-fisted dictates. If  developers heed these
> recommendations, they will avoid some amount of pain.  However, it is
> understood that these recommendations will not be heeded by all. Hence the
> word "SHOULD".
> >
> >
> > Ron
> >> the problem that only they can clean up. Killing capability in the
> >> rest of the Internet to make these mechanisms work isn’t a viable
> >> solution and never has been.
> >>>>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Nov 2018 13:25:00 -0800

Ron,

I think that algorithm describes virtual reassmbly. Based on the first
packet a virtual state for the fragment train is created and used to
route the non-first fragments. Problems with this are: 1) the device
is no longer stateless 2) it assumes the first fragement is received
first 3) it requires that all fragments are forwarded through the same
intermediate node.

If the goal is to implement a stateless device, then all packets
including fragments need to be processed independently. For a
stateless firewall, just inspecting and potentially dropping the first
fragment should yield the correct behavior. For an L4 load balancer,
it is a hard problem to get consistent stateless forwarding between
fragments and non-fragments. I believe that is best solved by routing
based on flow label instead of ports. The flow label is available in
all packets and doesn't require DPI, it should be the suggested
solution for IPv6 at least.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Mon, 12 Nov 2018 23:56:52 +0000

Tom,

OK. Let's see if the following text works any better for you.

                                    Ron
======================== 

 Middle box developers SHOULD implement devices that process IP fragments in a predictable manner.
 For example,  a stateless load balancer might exhibit one of the following behaviors:

- Balance load based on L3 information only

- Balance load based upon both L3 and L4 information, but discard all IP fragments

- For non-fragmented packets, balance load upon both L3 and L4 information. For fragmented packets, balance load based upon L3 information only. Therefore, packets belonging to a single flow can be assigned to two different interfaces. (See Section 4.4 for details).

- Balance load based upon both L3 and L4 information,  but maintain enough state so that all packets and packet fragments belonging to a flow are assigned to the same interface.

Undesirable outcomes can be attributed to each of the above mentioned behaviors. Therefore, middle box vendors MUST document the behaviors that their devices exhibit.

> From: Tom Herbert <[email-masked]>
> Sent: Monday, November 12, 2018 4:25 PM
> Cc: Joe Touch <[email-masked]>; Ole Troan
> <[email-masked]>; int-area <[email-masked]>
> I think that algorithm describes virtual reassmbly. Based on the first packet a
> virtual state for the fragment train is created and used to route the non-first
> fragments. Problems with this are: 1) the device is no longer stateless 2) it
> assumes the first fragement is received first 3) it requires that all fragments are
> forwarded through the same intermediate node.
> 
> If the goal is to implement a stateless device, then all packets including
> fragments need to be processed independently. For a stateless firewall, just
> inspecting and potentially dropping the first fragment should yield the correct
> behavior. For an L4 load balancer, it is a hard problem to get consistent
> stateless forwarding between fragments and non-fragments. I believe that is
> best solved by routing based on flow label instead of ports. The flow label is
> available in all packets and doesn't require DPI, it should be the suggested
> >    In some cases, given price and performance constraints, it may be
> impossible to design a middle box that processes IP fragments correctly. In
> these cases, the vendor MUST document the issue and propose work-arounds.
> >
> >> observes that some stateless middle boxes behave poorly in the
> >> presence of IP fragments. In order to avoid this situation, the draft
> >> makes the following
> >> recommendations:
> >> >
> >> > 1) Application developers SHOULD NOT develop new applications that
> >> > rely
> >> on IP fragmentation.
> >>
> >> I’d call this protocol developers, not just application developers
> >> (to cover tunnel and security designs too), I.e., anything that goes as
> payload in IP...
> >>
> >> They CAN rely on IP *source* fragmentation when needing to support
> >> direct IP-in-IP tunnels, as for existing protocols such as IPsec
> >> tunnel mode and IP-in- IP encapsulation.
> >>
> >> AFAICT, Middlebox developers SHOULD support processing of IP
> >> fragments with the same services they support atomic data grams.
> >>
> >> > These boxes SHOULD not fail or cause failures when processing
> >> > fragmented
> >> IP packets.
> >>
> >> Middleboxes SHOULD forward IP fragments without processing if
> >> processing cannot be supported. The MAY drop fragments only when
> >> there are significant indications of an overload condition.
> >>
> >> understood that these recommendations will not be heeded by all.
> >> Hence the word "SHOULD".
> >> >
> >> >
> >> > Ron

---

From: Joe Touch <[email-masked]>
Date: Mon, 12 Nov 2018 16:06:52 -0800

Notes below...

> On Nov 12, 2018, at 3:56 PM, Ron Bonica <[email-masked]> wrote:
> 
> Tom,
> 

The above should refer to just IP - the issue exists for both IPv4 and IPv6.

>   A protocol can break its dependency upon IP fragmentation by

it *might* be able to (it might not, though)

> - Balance load based upon both L3 and L4 information, but discard all IP fragments

If you’re balancing load, the box has NO legitimate reason to discard fragments.

This behavior would be needed ONLY if the box is doing something that can’t be done on fragments, .e., NAT. But in that case, it either ought to do virtual reassembly OR be deployed elsewhere. 

The big issue is that it should never silently just discard fragments — and there’s no ICMP error that exactly matches this case. IMO, AT A MINIMUM, the device ought to issue ICMP code 11 type 1 (reassembly timer exceeded). Yes, I know this comes from a host - which is *exactly* correct in this case (even though it’s really a lie because reassembly isn’t being emulated as needed).

Silent drops are NOT operationally acceptable, though.

> Undesirable outcomes can be attributed to each of the above mentioned behaviors. Therefore, middle box vendors MUST document the behaviors that their devices exhibit.

IMO, the only one we can’t have is silent drop.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Nov 2018 16:27:11 -0800

> OK. Let's see if the following text works any better for you.

Hi Ron.

>  For example,  a stateless load balancer might exhibit one of the following behaviors:
Seems like you're offering middle box developers a choice amonst
different behaviors. May this "consistent" as opposed to "pedictable"
behavior.

> - Balance load based on L3 information only

Probably want to be more specific here, presumably this means hash
over pair of addresses.

Load balance over 3-tuple of IP address and flow label should have its
own recommendation. This is effectively operating on L4 information.

> - Balance load based upon both L3 and L4 information, but discard all IP fragments
Which pretty much makes fragmentation useless, so it's not a great
recommendation. There's another problem here in that L4 information
might not be available in other types of packets. When we say L4, what
that probably means is just plain TCP or UDP packets.

> - For non-fragmented packets, balance load upon both L3 and L4 information. For fragmented packets, balance load based upon L3 information only. Therefore, packets belonging to a single flow can be…

It depends on what the backend of the load balancer is doing. If it is
an L4 load balancer to VIPs, then fragments and non-fragments for a
flow have to go to the same node.

> - Balance load based upon both L3 and L4 information,  but maintain enough state so that all packets and packet fragments belonging to a flow are assigned to the same interface.

Sure, but then the device can no longer be called stateless. AFAICT
using the flow label in the hash is the only way that a stateless L4
load balancer could get a consistent hash between fragments and
non-fragments.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 13 Nov 2018 17:25:29 +1300

>> 7.3.  For Middle Box Developers

I agree with all of Tom's comments below, but also think that this
tries to roll too many different functions into one class. Server
load balancing is a very different thing from ISP load balancing,
and is unlikely to be stateless, so one size of recommendation
definitely doesn't fit all.

    Brian

---

From: Ole Troan <[email-masked]>
Date: Tue, 13 Nov 2018 08:18:47 +0100

> See my suggested text - I.e., if you can’t handle the fragments they way you WANT, then you need to forward them.

To where?
When the forwarding decision is based on L4 information, as it is inthe IPv4 Internet, the best we could do is to send an ICMP destination unreachable back.

> IMO, if you don’t know what to do with fragments you can’t process AT ALL, then your middlebox service is either improperly designed or improperly deployed.

We have accepted the fact that the only way to make the IPv4 Internet continue operate is to do so improperly.
The earth is also round.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Nov 2018 12:39:35 -0800

On 2018-11-12 23:18, Ole Troan wrote:

>> See my suggested text - I.e., if you can't handle the fragments they way you WANT, then you need to forward them.
> 
> To where?

There is no appropriate message for failing to route a packet based on
L4 information that comes from a router. 

>> IMO, if you don't know what to do with fragments you can't process AT ALL, then your middlebox service is either improperly designed or improperly deployed.
> 

And some operators are deploying things incorrectly, and some boxes are
incorrectly implemented. 

Just because you want it so doesn't make it so. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Nov 2018 12:45:53 -0800

On 2018-11-13 05:38, Ron Bonica wrote:

> Joe, Tom, 
> 
> It looks like consensus is going to be difficult to reach, so let's take it one section at a time. Would both of you agree to the following text for Section 7.1? 
> 
> Ron 
> 
> =========================== 
> 

That sounds fine. 

> Legacy protocols that depend upon IP fragmentation SHOULD be updated 
> 
> alternative to IP fragmentation (e.g., IPSec tunnel mode, IP-in-IP encapsulation). 
> 
> In these cases, the protocol MAY continue to rely on IP fragmentation.

WILL. Not MAY. Not this document's decision. 

And some protocols won't be updated. 

(and check capitalization of IPsec) 

> A protocol may be able to break its dependency upon IP fragmentation by 
> 

There are several MTUs here - you want to talk about the *path* MTU; a
link MTU could be interpreted as either its one-hop path MTU or its
receiver reassembly MTU. 

Again, here, it is important to note that many protocols -notably those
supporting a network layer with its own MTU requirements - may not be
able to do this at all. 

> FROM: Joe Touch <[email-masked]> 
> SENT: Monday, November 12, 2018 7:07 PM
> TO: Ron Bonica <[email-masked]>
> CC: Tom Herbert <[email-masked]>; Ole Troan <[email-masked]>; int-area <[email-masked]>
> SUBJECT: Re: [Int-area] Stateless devices and IP fragmentation 
> 
> Notes below...
> 
> If you're balancing load, the box has NO legitimate reason to discard fragments. 
> 
> This behavior would be needed ONLY if the box is doing something that can't be done on fragments, .e., NAT. But in that case, it either ought to do virtual reassembly OR be deployed elsewhere.  
> 
> The big issue is that it should never silently just discard fragments -- and there's no ICMP error that exactly matches this case. IMO, AT A MINIMUM, the device ought to issue ICMP code 11 type 1 (reassembly timer exceeded). Yes, I know this comes from a host - which is *exactly* correct in this case (even though it's really a lie because reassembly isn't being emulated as needed). 
> 
> IMO, the only one we can't have is silent drop. 
> 
> Joe 
> 
> Legacy protocols that depend upon IPv6 fragmentation SHOULD be 
> 
> updated
> 
> to break that dependency.  However, in some cases, there may be no 
> 
> viable
> 
> alternative to IPv6 fragmentation (e.g., IPSec tunnel mode, IP-in-IP 
> 
> encapsulation).
> 
> Middle box developers SHOULD implement devices that do not fail or 
> 
> cause
> 
> On Nov 12, 2018, at 10:35 AM, Ron Bonica <[email-masked]>

wrote:

> I'd call this protocol developers, not just application developers

payload in IP...

> 3) Middle box developers SHOULD implement devices that support IP 
> 
> fragmentation.
> 
> Here's the problem - under what cases is an exception warranted?
> 
> How's that? That's still staying in SHOULD/MAY land as much as possible.
> 
> Joe
> 
> On 12 Nov 2018, at 11:11, Joe Touch <[email-masked]>

wrote:

> On Nov 12, 2018, at 1:36 AM, Ole Troan <[email-masked]>

the problem that only they can clean up. Killing capability in the
rest of the Internet to make these mechanisms work isn't a viable
solution and never has been.

>> And if that makes the IPv4 Internet die quicker, I'll take that as a

feature.

> And that's probably as good an end to this thread as any...
> 
> Ole

---

From: Ole Troan <[email-masked]>
Date: Tue, 13 Nov 2018 22:01:02 +0100

> On 13 Nov 2018, at 21:39, Joe Touch <[email-masked]> wrote:
> 
> 
>  
> 
> 
> There is no appropriate message for failing to route a packet based on L4 information that comes from a router.
>  

RFC7596 specifies destination unreachable, host unreachable for this. 
> And some operators are deploying things incorrectly, and some boxes are incorrectly implemented.
>  

I want everyone to move to IPv6. As that’s not happening we have a responsibility to keep IPv4 running. You seem to think that will have no consequence for the IPv4 architecture...
Perhaps you should come up with your own proposal to extend IPv4 beyond it’s limits?

Ole

---

From: Fred Baker <[email-masked]>
Date: Tue, 13 Nov 2018 13:27:14 -0800

> On Nov 13, 2018, at 1:01 PM, Ole Troan <[email-masked]> wrote:
> 
> I want everyone to move to IPv6. As that’s not happening we have a responsibility to keep IPv4 running.

I actually disagree with that. I also want everyone to implement IPv6, and in time shut IPv4 down. If IPv4 isn't working (for some definition of same), I'm not sure we have either the responsibility or capability to fix it. The result of not fixing it is likely to be implementation of some form of CGN, which either works or doesn't according to the choice of application and views of the person commenting. If they don't like what they get and have an option, at some point they might think about the option.

In context, I found myself speaking with a group (that I explicitly choose to not specify) that is looking at a new network deployment. They are hearing a set of voices that argue for IPv4 with CGN, and I'm part of a group recommending IPv6 - and specifically IPv6-only with some form of IPv4aas overlay. The argument that I give them is that, in the open market, 5000 IPv4 /24s will cost them on the order of $20M, with ongoing costs as they add customers, and the smallest prefix an ISP can get will cost them $2100/year and is vastly larger than that need. And Oh By The Way, re CGN read RFC 2993 and the open comments of networks that have deployed it - it winds up driving the cost of the help desk.

If that kind of reasoning doesn't lead toward IPv6 deployment, I don't know what will.

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Nov 2018 14:08:59 -0800

You have to be careful; that would cause the origin to cease sending all
traffic to that IP address - which works for that RFC (which is L3
oriented), when in fact it is only the lack of the L4 info that makes it
unreachable. 

IMO, if there is a message here, it's reassembly timeout. Anything else
runs the risk of killing off non-fragmented packets to the same
destination. 

And some operators are deploying things incorrectly, and some boxes are
incorrectly implemented. 
Just because you want it so doesn't make it so. 
I want everyone to move to IPv6. As that's not happening we have a
responsibility to keep IPv4 running. You seem to think that will have no
consequence for the IPv4 architecture... 

No, I think there is a consequence - on device designers too. If this
means they can do things in IPv6 (flow-based forwarding) that they
should never have tried in IPv4, sure - go for it. 

> Perhaps you should come up with your own proposal to extend IPv4 beyond it's limits?

I'm recognizing its limits, not trying to ignre them. 

Joe

---

From: Ron Bonica <[email-masked]>
Date: Tue, 13 Nov 2018 13:38:13 +0000

Joe, Tom,

It looks like consensus is going to be difficult to reach, so let’s take it one section at a time. Would both of you agree to the following text for Section 7.1?

                                 Ron

From: Joe Touch <[email-masked]>
Sent: Monday, November 12, 2018 7:07 PM
To: Ron Bonica <[email-masked]>
Cc: Tom Herbert <[email-masked]>; Ole Troan <[email-masked]>; int-area <[email-masked]>
Subject: Re: [Int-area] Stateless devices and IP fragmentation

Notes below...

On Nov 12, 2018, at 3:56 PM, Ron Bonica <[email-masked]<mailto:[email-masked]>> wrote:

Tom,

                                   Ron
========================

Joe

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 4:25 PM
To: Ron Bonica <[email-masked]<mailto:[email-masked]>>
Cc: Joe Touch <[email-masked]<mailto:[email-masked]>>; Ole Troan
<[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Int-area] Stateless devices and IP fragmentation

On Mon, Nov 12, 2018 at 1:08 PM, Ron Bonica <[email-masked]<mailto:[email-masked]>> wrote:

Joe, Tom,

                                   Ron

Ron,

Tom

-----Original Message-----
From: Joe Touch <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 2:02 PM
To: Ron Bonica <[email-masked]<mailto:[email-masked]>>
Cc: Ole Troan <[email-masked]<mailto:[email-masked]>>; Tom Herbert
<[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Int-area] Stateless devices and IP fragmentation

Ron

Joe

On Nov 12, 2018, at 10:35 AM, Ron Bonica <[email-masked]<mailto:[email-masked]>>
wrote:

Folks,

But what does this mean for draft-ietf-intarea-frag-fragile? The
draft
observes that some stateless middle boxes behave poorly in the
presence of IP fragments. In order to avoid this situation, the draft
makes the following
recommendations:

Same issue...

Joe

Ron

-----Original Message-----
From: Ole Troan <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 5:23 AM
To: Joe Touch <[email-masked]<mailto:[email-masked]>>
Cc: Tom Herbert <[email-masked]<mailto:[email-masked]>>; Ron Bonica
<[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Int-area] Stateless devices and IP fragmentation

On 12 Nov 2018, at 11:11, Joe Touch <[email-masked]<mailto:[email-masked]>>
wrote:

On Nov 12, 2018, at 1:36 AM, Ole Troan <[email-masked]<mailto:[email-masked]>>
wrote:

These things can be rolled out in any numbers you like - they
are creating
the problem that only they can clean up. Killing capability in the
rest of the Internet to make these mechanisms work isn’t a viable
solution and never has been.

And if that makes the IPv4 Internet die quicker, I’ll take that as a
feature.

Joe

Ole

---

From: Ron Bonica <[email-masked]>
Date: Tue, 13 Nov 2018 21:11:59 +0000

Joe,

Ack. I will run with your comments unless I hear an objection from Tom Herbert.

                                                         Ron

Sent: Tuesday, November 13, 2018 3:46 PM

                                 Ron

That sounds fine.

There are several MTUs here - you want to talk about the *path* MTU; a link MTU could be interpreted as either its one-hop path MTU or its receiver reassembly MTU.

Again, here, it is important to note that many protocols -notably those supporting a network layer with its own MTU requirements - may not be able to do this at all.

Cc: Tom Herbert <[email-masked]<mailto:[email-masked]>>; Ole Troan <[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>

                                   Ron
========================

The big issue is that it should never silently just discard fragments — and there's no ICMP error that exactly matches this case. IMO, AT A MINIMUM, the device ought to issue ICMP code 11 type 1 (reassembly timer exceeded). Yes, I know this comes from a host - which is *exactly* correct in this case (even though it's really a lie because reassembly isn't being emulated as needed).

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 4:25 PM
To: Ron Bonica <[email-masked]<mailto:[email-masked]>>
Cc: Joe Touch <[email-masked]<mailto:[email-masked]>>; Ole Troan

                                   Ron

Ron,

Tom

-----Original Message-----
From: Joe Touch <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 2:02 PM
To: Ron Bonica <[email-masked]<mailto:[email-masked]>>
Cc: Ole Troan <[email-masked]<mailto:[email-masked]>>; Tom Herbert

Ron

Joe

Folks,

Same issue...

Joe

Ron
-----Original Message-----
From: Ole Troan <[email-masked]<mailto:[email-masked]>>
Sent: Monday, November 12, 2018 5:23 AM
To: Joe Touch <[email-masked]<mailto:[email-masked]>>
Cc: Tom Herbert <[email-masked]<mailto:[email-masked]>>; Ron Bonica

Joe

Ole

---

From: Ole Troan <[email-masked]>
Date: Wed, 14 Nov 2018 11:16:32 +0100

> You have to be careful; that would cause the origin to cease sending all traffic to that IP address - which works for that RFC (which is L3 oriented), when in fact it is only the lack of the L4 info that makes it unreachable.
> 
> IMO, if there is a message here, it's reassembly timeout. Anything else runs the risk of killing off non-fragmented packets to the same destination.
> 

If this signal stopped the host from sending IPv4 packets all the better. ;-)

Seriously though, I think the IETF should be a lot more agreesive in documenting the consequences to the IPv4 Internet architecture of the IPv4 run-out.
This draft is part of it, but unfortunately you are pushing to smooth things over and make the draft say how IPv4 should have worked in an optimal pre-runout world, instead of accepting how IPv4 works now.

We as in the IETF should also make very clear that the IPv4 Internet:
 - requires client-server initiation
 - only supports TCP, UDP and some ICMP
 - with limited number of ports available, and sessions will time out on you.
 - IP fragmentation has a higher drop probability than other TCP, UDP traffic

And to use the end to end Internet that’s only available on IPv6.

Ole

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 06:32:47 -0800

Again, this does nothing for middleboxes that can handle IPv6 fragments. Flow labels don’t fix DPI or NAT devices.

Joe

---

From: Ole Troan <[email-masked]>
Date: Wed, 14 Nov 2018 16:01:41 +0100

Joe,

> Again, this does nothing for middleboxes that can handle IPv6 fragments. Flow labels don’t fix DPI or NAT devices.

It wasn’t meant to. If you read my post, I clearly stated the IPv4 Internet.

There is much less justification to “encourage deprecation” of IPv6 fragments.
I think that if so has to be a much more archirectural view of the layering, and that network layer fragments problematic for the end host, because it can’t do any validation checks on them before reassembly.
From an IETF perspective middleboxes do not exist in IPv6 (sic). And if we really want to punt fragmentation to the transport layer, I think the argument should not be made using middleboxes for justification.

Do you see why I insist on making a distinction between IPv4 and IPv6?

Cheers,
Ole

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 07:36:06 -0800

> On Nov 14, 2018, at 7:01 AM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 

Overlap checks can be done.

What other kind of check do you think would be meaningful or helpful on fragments alone?

> From an IETF perspective middleboxes do not exist in IPv6 (sic).

And *I’m* the one in denial??

> And if we really want to punt fragmentation to the transport layer, I think the argument should not be made using middleboxes for justification.

You might consider that middleboxes are not IP-only devices; they depend on and modify transport layer headers and even content sometimes.

> Do you see why I insist on making a distinction between IPv4 and IPv6?

Except for the flow ID for IPv6 - which helps only when used properly AND for load distribution - no.

Joe

---

From: Ole Troan <[email-masked]>
Date: Wed, 14 Nov 2018 20:33:44 +0100

Joe,

> What other kind of check do you think would be meaningful or helpful on fragments alone?

Exactly, and that’s the problem with fragmentation at the network layer. The end host must do reassembly before deciding to throw the packet or not.

> And *I’m* the one in denial??

;-)

> Except for the flow ID for IPv6 - which helps only when used properly AND for load distribution - no.

For IPv4 part of the transport headers are now part of the network layer. Forwarding is done one address and port.
I.e it’s now part of ‘normal’ routing/forwarding. That’s absolutely not the case. In IPv6 packets are delivered to hosts based on longest match on destination addresses, while for IPv4 it’s a match IPv4 DA and DP.

So no, we can’t treat the protocols the same.

Cheers,
Ole

---

From: Ron Bonica <[email-masked]>
Date: Wed, 14 Nov 2018 19:35:21 +0000

Folks,

Since we seem to have reached consensus on Section 7.1, let's take another stab at Section 7.3. I am looking particularly to Tom Herbert and Joe Touch for feedback, since they objected to the previous formulation.

Proposed text follows....

                                       Ron

Middle boxes SHOULD process IP fragments in a manner that is compliant with RFC 791 and RFC 8200. In many cases, middle boxes must maintain state in order to achieve this goal. 

Price and performance considerations frequently motivate network operators to deploy stateless middle boxes. These stateless middle boxes may perform sub-optimally or process IP fragments in a manner that is not compliant with RFC 791 or RFC 8200. Providers of such middle boxes MUST document product's their behavior.

For example, a network operator may be motivated to deploy a stateless load balancer. Because the load balancer is stateless, it is likely to exhibit on of the following behaviors:

- For all packets, implement a load balancing algorithm whose inputs are drawn from the IP header. This algorithm facilitates RFC compliant behavior. However,  it may distribute load unevenly.

- For fragmented packets, implement a load balancing algorithm whose inputs are drawn from the IP header. For non-fragmented packets, implement a load balancing algorithm whose inputs are drawn from the IP header and the Layer 4 header.  This algorithm facilitates RFC compliant behavior. It also improves load distribution for flows that do not included fragmented packets. However, if a flow contains fragmented packets, packets belonging to that flow can be distributed between two different outgoing interfaces.

- For all packets, implement a load balancing algorithm whose inputs are drawn from the IP header and the Layer 4 header. Discard all IP fragments. This algorithm does not facilitate RFC compliant behavior and can cause black holes.

All of the above-mentioned behaviors are sub-optimal. However, one behavior may be acceptable to one network operator while another behavior is acceptable to another network operator. Middle box vendors MUST provide network operators with all of the information required to  make intelligent middle box deployment decisions.

---

From: Tom Herbert <[email-masked]>
Date: Wed, 14 Nov 2018 11:46:25 -0800

> For IPv4 part of the transport headers are now part of the network layer. Forwarding is done one address and port.

Ole,

I don't believe this can be true. Not all protocols even have port
numbers (e.g. ICMP, ESP) and yet we still expect them to be
deliverable. Maybe your referring to ECMP, which does route based on
flow (e.g. port information)? But, ECMP is not required to make IP
work either. Forwarding solely based on destination IPv4 address for
unicast packets needs to work.

> I.e it’s now part of ‘normal’ routing/forwarding. That’s absolutely not the case. In IPv6 packets are delivered to hosts based on longest match on destination addresses, while for IPv4 it’s a match…

ECMP is also done for IPv6. I don't see how routing is fundamentally
different in IPv6 except that the flow label could be a good
substitute for ports in ECMP.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Wed, 14 Nov 2018 12:02:21 -0800

Hi Ron,

Maybe s/may/sometimes/ to avoid any hint that we're advocating
non-compliant behavior.

> - For all packets, implement a load balancing algorithm whose inputs are drawn from the IP header. This algorithm facilitates RFC compliant behavior. However,  it may distribute load unevenly.

I don't think I would phrase it this way. It might be more along the
lines that if load balancer takes into account transport layer
information then it usually performs better. Also, in the case of IPv6
the flowlabel is part of IP header and could be used to achieve
similar load balancing characteristics as the case that the transport
ports are considered. So this statement might only be applicable to
IPv4.

> - For fragmented packets, implement a load balancing algorithm whose inputs are drawn from the IP header. For non-fragmented packets, implement a load balancing algorithm whose inputs are drawn fro…

Again, this would only be relevant in IPv4, for IPv6 flow label solves
the problem. For IPv4, I think this algorithm is about a best as could
be done. It might be worth mentioning that all fragments of a packet,
including the first that does contain L4 info, need to be forwarded
the in the same manner.
> - For all packets, implement a load balancing algorithm whose inputs are drawn from the IP header and the Layer 4 header. Discard all IP fragments. This algorithm does not facilitate RFC compliant…
If it's non-compliant, why include it as a valid alternative?

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 12:22:49 -0800

On 2018-11-14 11:33, Ole Troan wrote:

> Joe,
> 
> Again, this does nothing for middleboxes that can handle IPv6 fragments. Flow labels don't fix DPI or NAT devices. 
> It wasn't meant to. If you read my post, I clearly stated the IPv4 Internet.
> 
> There is much less justification to "encourage deprecation" of IPv6 fragments.
> I think that if so has to be a much more archirectural view of the layering, and that network layer fragments problematic for the end host, because it can't do any validation checks on them before reassembly.

What other kind of check do you think would be meaningful or helpful on
fragments alone? 
Exactly, and that's the problem with fragmentation at the network layer.
The end host must do reassembly before deciding to throw the packet or
not. 

Fragmentation at ANY (every) layer has the same problem. If you want to
look beyond he fragment headers, you need to reassemble (or do a virtual
reassembly to carry info across fragments), and 

>>>>  EVERY device that looks beyond a single fragment >>IS<< an end host to that frag layer <<<< 

Trying to ignore this at IP won't solve the problem - it exists at all
layers of fragmentation for some variety of middleboxes... 

However, NOBODY does validation on fragments (except for overlap) --
including the end host. I.e., the middlebox is not in a disadvantaged
position regarding such checks. 

> And *I'm* the one in denial??

;-)

> Except for the flow ID for IPv6 - which helps only when used properly AND for load distribution - no.

For IPv4 part of the transport headers are now part of the network
layer. Forwarding is done one address and port.
I.e it's now part of 'normal' routing/forwarding. That's absolutely not
the case. In IPv6 packets are delivered to hosts based on longest match
on destination addresses, while for IPv4 it's a match IPv4 DA and DP. 

For ECMP, this is never an issue. ECMP can easily use a default for
fragments and move on with life. 

For end-system load balancing, similar defaults can work, but it's also
reasonable to expect that a device that acts on behalf of an end system
that way should do the work of an end system to reassemble. 

For DPI, reassembly - and sometimes restoration of the TCP data stream
across segments - is necessary anyway. 

For NAT, virtual reassembly is necessary. 

None of these are specific to IPv4; even in IPv6, flow IDs are not a
magic solution -- they need similar fragmentation support. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 12:28:31 -0800

> Middle boxes SHOULD process IP fragments in a manner that is compliant with RFC 791 and RFC 8200. In many cases, middle boxes must maintain state in order to achieve this goal.

I'd encourage a sentence here that indicates the conditions under which
the SHOULD can be relaxed or not, i.e., they really MUST but when they
cannot they need to be less silent about failure or something...
(foreshadowing the last paragraph below). 

> Price and performance considerations frequently motivate network operators to deploy stateless middle boxes. These stateless middle boxes may perform sub-optimally or process IP fragments in a mann…

IMO, they really ought to also MUST default to at least occasionally
sending frag reassembly timeouts if fragments are consistently dropped. 

> All of the above-mentioned behaviors are sub-optimal. However, one behavior may be acceptable to one network operator while another behavior is acceptable to another network operator. Middle box ve…

As above, I hope we can also add "at least occasionally send an ICMP".
However, the discussion above focuses on load balancing, which has a
reasonable safe solution (don't balance fragments). The discussion needs
to address DPI and NAT, the former having a solution considered less
safe (pass fragments without inspection) and the latter would completely
fail without some sort of state. 

All three need to be addressed, AFAICT. 

Joe

---

From: Ole Troan <[email-masked]>
Date: Wed, 14 Nov 2018 21:31:51 +0100

Tom,

> unicast packets needs to work.

I can’t see how you can expect anything but TCP and UDP and (some) ICMP to be deliverable.
These just do not pass through stuff like A+P, CGNs.

Of course you can pass non TCP/UDP through limited domains, but you cannot expect that to work across the Internet anymore.

Cheers,
Ole

---

From: Ole Troan <[email-masked]>
Date: Wed, 14 Nov 2018 21:35:19 +0100

Joe,

>> I.e it's now part of 'normal' routing/forwarding. That's absolutely not the case. In IPv6 packets are delivered to hosts based on longest match on destination addresses, while for IPv4 it's a match IPv4 DA and DP.
> For ECMP, this is never an issue. ECMP can easily use a default for fragments and move on with life.

I am not sure why you bring in ECMP.
(It is actually a problem for ECMP, since fragmented and non-fragmented packets are prone to take different routes, but that seems to be far off topic).

> For end-system load balancing, similar defaults can work, but it's also reasonable to expect that a device that acts on behalf of an end system that way should do the work of an end system to reassemble.
> 
> For DPI, reassembly - and sometimes restoration of the TCP data stream across segments - is necessary anyway.
> 

If the packet doesn’t isn’t TCP or UDP and the ports are available any IPv4 packet will have a significantly higher drop probability.
I’m fine with you not accepting that. 

> 
> None of these are specific to IPv4; even in IPv6, flow IDs are not a magic solution -- they need similar fragmentation support.
> 

Of course they are. One is IETF mandated and required to be able to forward packets, the other is not.

Ole

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 15 Nov 2018 09:43:46 +1300

> Of course you can pass non TCP/UDP through limited domains, but you cannot expect that to work across the Internet anymore.

You mean the IPv4nternet, I hope ;-).

   Brian

---

From: Ole Troan <[email-masked]>
Date: Wed, 14 Nov 2018 21:48:25 +0100

> You mean the IPv4nternet, I hope ;-).

I hope so too. ;-)

Ole

---

From: Tom Herbert <[email-masked]>
Date: Wed, 14 Nov 2018 13:02:32 -0800

On Wed, Nov 14, 2018 at 12:50 PM, Ron Bonica <[email-masked]> wrote:
> Tom,
>
> Please look inline for a little compromise and a little pushback. I hope that we can reach consensus in this round.
>
>                                              Ron
>
>
>> Sent: Wednesday, November 14, 2018 3:02 PM
>> Cc: int-area <[email-masked]>; Joe Touch <[email-masked]>
>> > Since we seem to have reached consensus on Section 7.1, let's take another
>> stab at Section 7.3. I am looking particularly to Tom Herbert and Joe Touch for
>> feedback, since they objected to the previous formulation.
>> >
>> > Middle boxes SHOULD process IP fragments in a manner that is compliant
>> with RFC 791 and RFC 8200. In many cases, middle boxes must maintain state
>> in order to achieve this goal.
>> >
>> > Price and performance considerations frequently motivate network
>> operators to deploy stateless middle boxes. These stateless middle boxes may
>> perform sub-optimally or process IP fragments in a manner that is not
>> compliant with RFC 791 or RFC 8200. Providers of such middle boxes MUST
>> document product's their behavior.
>>
>> Hi Ron,
>>
>> Maybe s/may/sometimes/ to avoid any hint that we're advocating non-
>> compliant behavior.
>>
>
> Fair enough, change accepted.
>
>> >
>> > For example, a network operator may be motivated to deploy a stateless
>> load balancer. Because the load balancer is stateless, it is likely to exhibit on of
>> the following behaviors:
>> >
>> > - For all packets, implement a load balancing algorithm whose inputs are
>> drawn from the IP header. This algorithm facilitates RFC compliant behavior.
>> However,  it may distribute load unevenly.
>>
>> I don't think I would phrase it this way. It might be more along the lines that if
>> load balancer takes into account transport layer information then it usually
>> performs better. Also, in the case of IPv6 the flowlabel is part of IP header and
>> could be used to achieve similar load balancing characteristics as the case that
>> the transport ports are considered. So this statement might only be applicable
>> to IPv4.
>
>
> How does the following text strike you......
>
> - For all packets, implement a load balancing algorithm whose inputs are drawn from the IP header. This algorithm facilitates RFC compliant behavior. However, this algorithm may not distribute load as evenly as another algorithm whose inputs  are drawn from the IP and transport layer headers.
>
> I am going to push back on the flow label part because a) it is not available in IPv4 and b) In IPv6, we have no guarantee that it will be set. So, I weasel around the issue by saying that the algorithm *may* not distribute load as evenly. I never say that it *will* not distribute load as evenly.

I believe flow label is now being set by all major OSes. Even if it's
not set, the algorithm just falls back to hashing over IP addresses.
Flow label was a good addition to IPv6 exactly for the purposes of
things like load balancing and it solves the problem in this case.
It's use should be encouraged here.

>
>
>> >
>> > - For fragmented packets, implement a load balancing algorithm whose
>> inputs are drawn from the IP header. For non-fragmented packets, implement
>> a load balancing algorithm whose inputs are drawn from the IP header and
>> the Layer 4 header.  This algorithm facilitates RFC compliant behavior. It also
>> improves load distribution for flows that do not included fragmented packets.
>> However, if a flow contains fragmented packets, packets belonging to that
>> flow can be distributed between two different outgoing interfaces.
>>
>> Again, this would only be relevant in IPv4, for IPv6 flow label solves the
>> problem. For IPv4, I think this algorithm is about a best as could be done. It
>> might be worth mentioning that all fragments of a packet, including the first
>> that does contain L4 info, need to be forwarded the in the same manner.
>> >
>
> I would like to push back on the flow label arguments for the same reasons as stated above. In addition, this is by far the most widely deployed strategy.

Maybe so, but I don't see how use of flow label can make things worse
wrt fragmentation.

>
>> > - For all packets, implement a load balancing algorithm whose inputs are
>> drawn from the IP header and the Layer 4 header. Discard all IP fragments.
>> This algorithm does not facilitate RFC compliant behavior and can cause black
>> holes.
>> >
> It may be non-compliant, but it is perfectly reasonable in an enterprise that filters fragments anyway.

Then can we at least say that the arbitrarily discarding all fragments
is NOT RECOMMENDED.
>
>>
>> > All of the above-mentioned behaviors are sub-optimal. However, one
>> behavior may be acceptable to one network operator while another behavior
>> is acceptable to another network operator. Middle box vendors MUST provide
>> network operators with all of the information required to  make intelligent
>> middle box deployment decisions.
>> >
>> >
>> >

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 13:04:50 -0800

On 2018-11-14 12:35, Ole Troan wrote:

> I.e it's now part of 'normal' routing/forwarding. That's absolutely not the case. In IPv6 packets are delivered to hosts based on longest match on destination addresses, while for IPv4 it's a match IPv4 DA and DP. For ECMP, this is never an issue. ECMP can easily use a default for fragments and move on with life.

Most of Ron's text is based on load balancing - which is composed of two
cases: ECMP and end server. 

> (It is actually a problem for ECMP, since fragmented and non-fragmented packets are prone to take different routes, but that seems to be far off topic).

That can't be any more a problem than different ports taking different
routes. 

> If the packet doesn't isn't TCP or UDP and the ports are available any IPv4 packet will have a significantly higher drop probability.

There's a big difference between "higher" and "100%". The former is
always OK, the latter is not, at least it should not be silent. 

> I'm fine with you not accepting that. 
> 

Both IPv4 and IPv6 are IETF mandated. Both require support for source
fragmentation both at the endpoints and in forwarding. 

Joe 

Ole

---

From: Ron Bonica <[email-masked]>
Date: Wed, 14 Nov 2018 21:20:17 +0000

Joe,

I half agree with you. Stateless middle boxes really should send feedback to the source when they process fragments in funky ways. However, I don’t think that they should sent an ICMP Time Exceeded with code (1) reassembly time exceeded.

This message isn’t exactly accurate. The middle box never attempted to reassemble the packet and won’t attempt to reassemble the packet if it receives it again. We probably need a new ICMP message.

I would be glad to work with you on this project, but I think that it is out of scope for the current draft. (This draft is a BCP that comments on the current state of IP fragmentation. It has no business defining new ICMP messages.)

Can we address the new message in a separate draft?

                                                                            Ron

From: Joe Touch <[email-masked]>
Sent: Wednesday, November 14, 2018 3:29 PM
To: Ron Bonica <[email-masked]>
Cc: Tom Herbert <[email-masked]>; int-area <[email-masked]>
Subject: Re: [Int-area] Stateless devices and IP fragmentation

                                       Ron

I'd encourage a sentence here that indicates the conditions under which the SHOULD can be relaxed or not, i.e., they really MUST but when they cannot they need to be less silent about failure or something... (foreshadowing the last paragraph below).

IMO, they really ought to also MUST default to at least occasionally sending frag reassembly timeouts if fragments are consistently dropped.

As above, I hope we can also add "at least occasionally send an ICMP". However, the discussion above focuses on load balancing, which has a reasonable safe solution (don't balance fragments). The discussion needs to address DPI and NAT, the former having a solution considered less safe (pass fragments without inspection) and the latter would completely fail without some sort of state.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 15 Nov 2018 10:25:03 +1300

> It's use should be encouraged here.

To be clear, if you do ECMP or LAG based entirely on the 3-tuple
{dest_addr, source_addr, flow_label} then fragments will all follow
exactly the same path as unfragmented packets. So, perfection.
This is true even if the flow label is zero, of course.

Trying to use the traditional 5-tuple will not work well with
fragments.

It's more complicated at the server farm end.
     "Moreover, the transport-layer
      information such as the source port is not repeated in fragments,
      which generally prevents stateless load balancers from supporting
      fragmented traffic since they generally cannot reassemble
      fragments."  [RFC7098].

    Brian

> .

---

From: Ron Bonica <[email-masked]>
Date: Wed, 14 Nov 2018 21:34:34 +0000

Folks,

We thrashing over the example. Can everybody agree to the following text?

                                               Ron

The behaviors exhibited by the above-mentioned devices are not desirable. However, one behavior may be acceptable to one network operator while another behavior is acceptable to another network operator. Middle box vendors MUST provide network operators with all of the information required to  make intelligent middle box deployment decisions.

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 13:44:46 -0800

On 2018-11-14 13:25, Brian E Carpenter wrote:

> ...

That seems overstated. A stateless load balancer can always have a
default, in which case it could easily support fragments - though less
efficiently than intended. 

The problem lies with NATs, for which a default cannot exist. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2018 13:48:39 -0800

On 2018-11-14 13:20, Ron Bonica wrote:

> Joe, 
> 
> I half agree with you. Stateless middle boxes really should send feedback to the source when they process fragments in funky ways. However, I don't think that they should sent an ICMP Time Exceeded with code (1) reassembly time exceeded. 
> 
> This message isn't exactly accurate. The middle box never attempted to reassemble the packet and won't attempt to reassemble the packet if it receives it again. We probably need a new ICMP message.

 That's fraught with its own problems, but that's not driving my
reasoning.

IMO, the issue here is exactly that the timeout (of zero) is exceeded
because as you increase capability in the middlebox, the timeout delay
would increase proportionally. 

So this isn't a perfect message, but it does have a correct
interpretation and it wouldn't affect non-fragment packets. 

Creating a separate "frag not supported" would basically be saying "I
don't support existing requirements", and I don't think we should codify
that with a separate ICMP. 

> SENT: Wednesday, November 14, 2018 3:29 PM
> CC: Tom Herbert <[email-masked]>; int-area <[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 14 Nov 2018 13:54:41 -0800

>       fragmented traffic since they generally cannot reassemble
Brian,

That might be a bit misleading. Rather or not the load balancer
properly "supports" fragemented traffic depends on the requirements
load balancing. If the only point of a load balancer is too make
uniform traffic distribution, then randomly selection of output
interface per packet (Random Packet Spraying) beats all any other
algorithm and requires no input from the packet. The implicit
assumption in this discussion is that packets of a flow are expected
to be routed the same. This is either just an optimization, or could
be requirement as in the case of load balancing amoungst VIPs. I'm
doubtful that consistent forwarding to VIPs is feasible with a
stateless load balancer and that most likely requires a complex
stateful device (see Maglev for instance). So for the stateless load
balancer, maybe the strongest requirement is best effort to forward
all packets of a flow in the same manner.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Wed, 14 Nov 2018 20:50:41 +0000

Tom,

                                             Ron

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 15 Nov 2018 11:21:18 +1300

> all packets of a flow in the same manner.

Indeed. But the sentence I quoted was written by Willy Tarreau and
reflected his knowledge of products in ~2013. Whether that's still
true today, I don't know.

The underlying point is that if ECMP or LAG in a carrier network
sends fragments on a different path, the worst that should happen
is out-of-order delivery at the reassembly point. But if a server
load balancer makes the same error, the result is a failed transaction
because some of the fragments are mis-delivered.

You know this, but I believe the draft needs to make it clear,
because it makes the conseqeuences of fragmentation more serious.

    Brian

---

From: Tom Herbert <[email-masked]>
Date: Wed, 14 Nov 2018 14:28:13 -0800

> because it makes the conseqeuences of fragmentation more serious.
Agreed. There are a lot of subtleties and implicit requirements in
this discussion. And load balancing is only one use case, there could
be (and maybe should be) an equally animiated discussion on how
stateless firewalls deal with fragments.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Thu, 15 Nov 2018 21:35:57 +0000

Tom, Joe, Brian,

I haven't seen a response to this message. Can I assume that you are OK with this text?

                                     Ron

> From: Ron Bonica
> Sent: Wednesday, November 14, 2018 4:35 PM
> To: Tom Herbert <[email-masked]>
> Subject: RE: [Int-area] Stateless devices and IP fragmentation
> 
> Folks,
> 
> Middle boxes SHOULD process IP fragments in a manner that is compliant with
> RFC 791 and RFC 8200. In many cases, middle boxes must maintain state in
> order to achieve this goal.
> 
> Price and performance considerations frequently motivate network operators
> to deploy stateless middle boxes. These stateless middle boxes may perform
> sub-optimally or process IP fragments in a manner that is not compliant with
> RFC 791 or RFC 8200. Providers of such middle boxes MUST document
> product's their behavior.
> 
> The behaviors exhibited by the above-mentioned devices are not desirable.
> However, one behavior may be acceptable to one network operator while
> another behavior is acceptable to another network operator. Middle box
> vendors MUST provide network operators with all of the information required
> to  make intelligent middle box deployment decisions.

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Nov 2018 14:23:34 -0800

>> product's their behavior.

I already noted that "documenting" that behavior isn't sufficient; IMO,
they MUST also provide at least occasional network signals to indicate
that non-compliance (as I noted, using reassembly timeout seems both
reasonable and of minimal collateral impact). 

>> to  make intelligent middle box deployment decisions.

I might suggest "deployment and/or configuration", but sure. 

Joe

---

From: Ron Bonica <[email-masked]>
Date: Thu, 15 Nov 2018 22:30:32 +0000

Joe,

Respectfully, I think that this is beyond the scope of this document. This document is a BCP describing the current state of IP Fragmentation. It is not a PS document that describes new procedures and bits on the wire.

Again, if you would like to work on such a document, I would be glad to work on it with you. But it just doesn’t belong in the current document.

Chairs,

We appear to be at an impasse. Advice?

                                        Ron

From: Joe Touch <[email-masked]>
Sent: Thursday, November 15, 2018 5:24 PM
To: Ron Bonica <[email-masked]>
Cc: Tom Herbert <[email-masked]>; Brian E Carpenter <[email-masked]>; int-area <[email-masked]>
Subject: Re: [Int-area] Stateless devices and IP fragmentation

                                     Ron

-----Original Message-----
From: Ron Bonica
Sent: Wednesday, November 14, 2018 4:35 PM
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: int-area <[email-masked]<mailto:[email-masked]>>; Joe Touch <[email-masked]<mailto:[email-masked]>>
Subject: RE: [Int-area] Stateless devices and IP fragmentation

Folks,

                                               Ron

I already noted that "documenting" that behavior isn't sufficient; IMO, they MUST also provide at least occasional network signals to indicate that non-compliance (as I noted, using reassembly timeout seems both reasonable and of minimal collateral impact).

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 16 Nov 2018 16:44:20 +1300

>  These stateless middle boxes may perform
> RFC 791 or RFC 8200.

That seems to skirt round the real concern. Middleboxes don't exist in
the world assumed by RFC 791 or 8200, so those RFCs don't place any
compliance requirements on middleboxes. It's simply implicit that datagrams
get delivered whether fragmented or not. RFC 1812 doesn't seem to mention
that routers MUST forward fragments, presumably because it's blindingly
obvious. Same for draft-ietf-6man-rfc6434-bis and draft-v6ops-ipv6rtr-reqs.

So at least can we say:

These stateless middle boxes may perform
sub-optimally, process IP fragments in a manner that is not compliant with
RFC 791 or RFC 8200, or even discard IP fragments completely.

Regards
   Brian

---

From: Ron Bonica <[email-masked]>
Date: Fri, 16 Nov 2018 15:02:24 +0000

Hi Brian,

Fair enough. Does the following text work?

                                   Ron

Price and performance considerations frequently motivate network operators to deploy stateless middle boxes.  These stateless middle boxes may perform sub-optimally, process IP fragments in a manner that is not compliant with RFC 791 or RFC 8200, or even discard IP fragments completely. Providers of such middle boxes MUST document product's their behavior.

> From: Brian E Carpenter <[email-masked]>
> Sent: Thursday, November 15, 2018 10:44 PM
> To: Ron Bonica <[email-masked]>; Tom Herbert
> <[email-masked]>; Joe Touch <[email-masked]>
> Cc: int-area <[email-masked]>
> >  These stateless middle boxes may perform sub-optimally or process IP
> > fragments in a manner that is not compliant with RFC 791 or RFC 8200.
> 
> That seems to skirt round the real concern. Middleboxes don't exist in the
> world assumed by RFC 791 or 8200, so those RFCs don't place any compliance
> requirements on middleboxes. It's simply implicit that datagrams get delivered
> whether fragmented or not. RFC 1812 doesn't seem to mention that routers
> MUST forward fragments, presumably because it's blindingly obvious. Same
> for draft-ietf-6man-rfc6434-bis and draft-v6ops-ipv6rtr-reqs.
> 
> So at least can we say:
> 
> These stateless middle boxes may perform sub-optimally, process IP fragments
> in a manner that is not compliant with RFC 791 or RFC 8200, or even discard
> IP fragments completely.
> 
> Regards
>    Brian
> 
> >> Middle boxes SHOULD process IP fragments in a manner that is
> >> compliant with RFC 791 and RFC 8200. In many cases, middle boxes must
> >> maintain state in order to achieve this goal.
> >>
> >> operators to deploy stateless middle boxes. These stateless middle
> >> boxes may perform sub-optimally or process IP fragments in a manner
> >> that is not compliant with RFC 791 or RFC 8200. Providers of such
> >> middle boxes MUST document product's their behavior.
> >>
> >> another behavior is acceptable to another network operator. Middle
> >> box vendors MUST provide network operators with all of the
> >> information required to  make intelligent middle box deployment decisions.

---

From: Joe Touch <[email-masked]>
Date: Fri, 16 Nov 2018 10:55:15 -0800

I respectfully then conclude that there is nothing "best" in this BCP. 

It certainly isn't even "current" practice. 

So, if we're speaking of documenting only procedures that (a) already
exist AND that (b) we want others to follow, it seems the path is clear
to simply pass on this document entirely. 

Joe

On 2018-11-15 14:30, Ron Bonica wrote:

> Again, if you would like to work on such a document, I would be glad to work on it with you. But it just doesn't belong in the current document. 
> 
> Chairs, 
> 
> SENT: Thursday, November 15, 2018 5:24 PM
> CC: Tom Herbert <[email-masked]>; Brian E Carpenter <[email-masked]>; int-area <[email-masked]>

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 17 Nov 2018 08:16:59 +1300

> Fair enough. Does the following text work?

Yes, I guess so, in the context of this section. From a user point
of view, there's an impression that we're saying that it's OK for
some cheapskate operator to break connectivity by dropping fragments,
when some other cheapskate operator has set a low MTU. But that
discussion probably belongs in a higher level section of the
document.

So: I think the document should say much more emphatically that
there is no exception to the 1280 requirement for every IPv6 hop.
It's mentioned briefly but maybe it should be underlined three times :-).

Amd maybe, since the draft is aimed at BCP, we could even go so far as
to change this assumption into a SHOULD, if not a MUST:

>    So, for the purposes of this document, we assume that the IPv4
>    minimum link MTU is 576 bytes.
Regards
    Brian

---

From: Tom Herbert <[email-masked]>
Date: Sat, 17 Nov 2018 09:00:45 -0800

Ron,

Devices that maintain protocol state in the network aren't protocol
compliant either.

> Price and performance considerations frequently motivate network operators to deploy stateless middle boxes.  These stateless middle boxes may perform sub-optimally, process IP fragments in a manne…

This is too permissive for middle boxes to implement whatever ad hoc
behavior they want. I would say it like:

"Price and performance considerations frequently motivate network
operators to deploy stateless middle boxes.  Some deployed stateless
middle boxes process IP fragments in a manner that is not compliant
with RFC 791 or RFC 8200, or even discard IP fragments completely.
Such behaviors are NOT RECOMMENDED. If a middleboxes implements
non-standard behavior with respect to IP fragmentation, then that
behavior MUST be clearly documented."

> The behaviors exhibited by the above-mentioned devices are not desirable. However, one behavior may be acceptable to one network operator while another behavior is acceptable to another network ope…

I don't think this paragraph is needed with the suggested text above.

Tom

---

From: Joe Touch <[email-masked]>
Date: Sat, 17 Nov 2018 09:08:48 -0800

> On Nov 17, 2018, at 9:00 AM, Tom Herbert <[email-masked]> wrote:
> 

They’re called endpoints — exactly because that’s what they have to be acting like.

> behavior MUST be clearly documented.”

That sounds OK to me - at least in the spirit of consensus.

> I don't think this paragraph is needed with the suggested text above.

Agreed, FWIW.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 17 Nov 2018 09:28:11 -0800

> They’re called endpoints — exactly because that’s what they have to be acting like.
If the destination address of a packet is a local address of the
device then they are not endpoints. They are intermediate nodes.
Stateful intermediate nodes do attempt to emulate some of the
functions of an endpoint, but that requires non-standard assumptions
and cannot be protocol conformant with host requirements.

---

From: Joe Touch <[email-masked]>
Date: Sat, 17 Nov 2018 09:39:38 -0800

> On Nov 17, 2018, at 9:28 AM, Tom Herbert <[email-masked]> wrote:
> 

They’re acting as if those addresses are theirs - otherwise, transport info would have no meaning (it is meaningful only between pairs of endpoints anyway). Consider that two endpoints can run DNS on port 80 - or can run a single TCP connection that rotates ports all day long. That’s entirely an endpoint decision.

I.e., the fact that they do anything beyond L3 *makes* them endpoints at those addresses.

Joe

---

From: Ron Bonica <[email-masked]>
Date: Wed, 21 Nov 2018 13:40:55 +0000

Brian,

Fair enough. I have worked the 1280 byte requirement into Section 7.4. New text is included below.

                                                       Ron

7.4.  For Network Operators

   As per RFC 4890, network operators MUST NOT filter ICMPv6 PTB
   messages unless they are known to be forged or otherwise
   illegitimate.  As stated in Section 4.6, filtering ICMPv6 PTB packets
   causes PMTUD to fail.  Operators MUST ensure proper PMTUD operation
   in their network, including making sure the network generates PTB
   packets when dropping packets too large compared to outgoing
   interface MTU.  Many upper-layer protocols rely on PMTUD.

   As per RFC 8200, network operators MUST NOT deploy IPv6 links whose
   MTU is less than 1280 bytes.

   Network operators SHOULD NOT filter IP fragments if they originated
   at a domain name server or are destined for a domain name server.

> So: I think the document should say much more emphatically that there is no
> exception to the 1280 requirement for every IPv6 hop.
> Amd maybe, since the draft is aimed at BCP, we could even go so far as to
> change this assumption into a SHOULD, if not a MUST:
> 
decisions.

---

From: Ron Bonica <[email-masked]>
Date: Wed, 21 Nov 2018 13:53:57 +0000

Tom,

Fair enough...

Section 7.3 now reads as follows:

   Middle boxes SHOULD process IP fragments in a manner that is
   compliant with RFC 791 and RFC 8200.  In many cases, middle boxes
   must maintain state in order to achieve this goal.

   Price and performance considerations frequently motivate network
   operators to deploy stateless middle boxes.  These stateless middle
   boxes may perform sub-optimally, process IP fragments in a manner
   that is not compliant with RFC 791 or RFC 8200, or even discard IP
   fragments completely.  Such behaviors are NOT RECOMMENDED.  If a
   middleboxes implements non-standard behavior with respect to IP
   fragmentation, then that behavior MUST be clearly documented.

                            Ron

> Sent: Saturday, November 17, 2018 12:01 PM
> Cc: Brian E Carpenter <[email-masked]>; Joe Touch
> <[email-masked]>; int-area <[email-masked]>
> Devices that maintain protocol state in the network aren't protocol compliant
> either.
> 
> perform sub-optimally, process IP fragments in a manner that is not compliant
> with RFC 791 or RFC 8200, or even discard IP fragments completely. Providers
> of such middle boxes MUST document product's their behavior.
> 
> operators to deploy stateless middle boxes.  Some deployed stateless middle
> boxes process IP fragments in a manner that is not compliant with RFC 791 or
> RFC 8200, or even discard IP fragments completely.
> Such behaviors are NOT RECOMMENDED. If a middleboxes implements non-
> standard behavior with respect to IP fragmentation, then that behavior MUST
> be clearly documented."
> 
> >
> > > >  These stateless middle boxes may perform sub-optimally or process
> > > > IP fragments in a manner that is not compliant with RFC 791 or RFC
> 8200.
> > >
> > > That seems to skirt round the real concern. Middleboxes don't exist
> > > in the world assumed by RFC 791 or 8200, so those RFCs don't place
> > > any compliance requirements on middleboxes. It's simply implicit
> > > that datagrams get delivered whether fragmented or not. RFC 1812
> > > doesn't seem to mention that routers MUST forward fragments,
> > > presumably because it's blindingly obvious. Same for draft-ietf-6man-
> rfc6434-bis and draft-v6ops-ipv6rtr-reqs.
> > >
> > > So at least can we say:
> > >
> > > These stateless middle boxes may perform sub-optimally, process IP
> > > fragments in a manner that is not compliant with RFC 791 or RFC
> > > 8200, or even discard IP fragments completely.
> > >
> > > Regards
> > >    Brian
> > >
> > > > I haven't seen a response to this message. Can I assume that you
> > > > are OK with
> > > this text?
> > > >
> > > >                                      Ron
> > > >
> > > >
> > > >> We thrashing over the example. Can everybody agree to the
> > > >> following
> > > text?
> > > >>
> > > >>                                                Ron
> > > >>
> > > >> compliant with RFC 791 and RFC 8200. In many cases, middle boxes
> > > >> must maintain state in order to achieve this goal.
> > > >>
> > > >> operators to deploy stateless middle boxes. These stateless
> > > >> middle boxes may perform sub-optimally or process IP fragments in
> > > >> a manner that is not compliant with RFC 791 or RFC 8200.
> > > >> Providers of such middle boxes MUST document product's their
> behavior.
> > > >>
> > > >> However, one behavior may be acceptable to one network operator
> > > >> while another behavior is acceptable to another network operator.
> > > >> Middle box vendors MUST provide network operators with all of the

---

From: Tim Chown <[email-masked]>
Date: Wed, 21 Nov 2018 16:52:07 +0000

Hi,

> On 21 Nov 2018, at 13:40, Ron Bonica <[email-masked]> wrote:
> 
> Brian,
> 

There is some similar-ish text in the RFC6434-bis document (in AUTH48 now), see https://tools.ietf.org/html/draft-ietf-6man-rfc6434-bis-09#section-5.7.1

The flip side of your last para is what we say there:

"While an IPv6 link MTU can be set to 1280 bytes, it is recommended
   that for IPv6 UDP in particular, which includes DNS operation, the
   sender use a large MTU if they can, in order to avoid gratuitous
   fragmentation-caused packet drops."

Tim

---

From: Ron Bonica <[email-masked]>
Date: Wed, 21 Nov 2018 17:28:48 +0000

Hi Tim,

I will shamelessly plagiarize this text in the next version of the draft!

                                                                    Ron