Subject: [Int-area] Comments on draft-ietf-intarea-frag-fragile-06

From: Tom Herbert <[email-masked]>
Date: Tue, 29 Jan 2019 11:45:45 -0800

Hello,

I have suggested text for the draft to address some previous comments
made on the list.

Last paragraph in section 4.3:

"This problem does not occur in stateful firewalls or Network Address
Translation (NAT) devices. Such devices maintain state so that they
can afford identical treatment to each fragment that belongs to a
packet. Note, however, that stateful firewalls and NAT devices impose
the external requirement that all packets of a flow and fragments of a
packets for a flow must traverse the same stateful device; stateless
devices do not force this requirement."

Section 4.5:
"IP fragmentation causes problems for some routers that support Equal
Cost Multipath (ECMP). Many routers that support ECMP execute the
algorithm described in Section 4.4 in order to perform flow based
forwarding; therefore, the exhibit they same problematic behaviors
described in Section 4.4. In IPv6, the flow label may alternatively
used as input to the algorithm as opposed to parsing the transport
layer of packets to discern port numbers. The flow label should be
consistently set for a packets of flow including fragments, such that
a device does not need to parse packets beyond the IP header for the
purposes of ECMP."

Add to section 7.3:

"Routers SHOULD use IPv6 flow label for ECMP routing as described in [RFC6438]."

---

From: Fred Baker <[email-masked]>
Date: Tue, 29 Jan 2019 15:37:46 -0800

> On Jan 29, 2019, at 11:45 AM, Tom Herbert <[email-masked]> wrote:
> 
> Hello,
> 
> packet.

Do we have a list of such products? I'm unaware of one. Making a blanket statement such as this troubles me, because it seems to say that this is invariably true, and I don't know that it is or isn't.

> Note, however, that stateful firewalls and NAT devices impose

Here's an example of why blanket statements need to be qualified. If you take NPTv6 to be a form of NAT, it doesn't force route symmetry.

> forwarding;

As far as I know, routers that hash fields in the IP header to select a en ECMP next hop do so because all packets in a flow will hash the same way (modulo the issues with the transport port number), not because they are doing per-flow forwarding. The do so explicitly to avoid having to maintain per-flow state and yet make all fragments of a message follow the same path.

> therefore, the exhibit they same problematic behaviors
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: Tom Herbert <[email-masked]>
Date: Tue, 29 Jan 2019 16:55:00 -0800

> Here's an example of why blanket statements need to be qualified. If you take NPTv6 to be a form of NAT, it doesn't force route symmetry.
Fred,

Yes, but the statement is in the context of stateful NATs. I agree
that this section is murky, but I believe that's because it referring
to non-standard behaviors. It would be so much easier if we could just
reference the specification that says how stateful intermediate
devices are supposed to work.

> As far as I know, routers that hash fields in the IP header to select a en ECMP next hop do so because all packets in a flow will hash the same way (modulo the issues with the transport port number…

That's true for IPv4,the only way to do stateless ECMP and have
fragments follow the same path as non-fragments is to hash over the IP
addresses only. For IPv6 we can do better. The flow label allows finer
grained per-flow routing, but still only requires inspection of IP
header so keeping fragments in order just works.

Tom

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 30 Jan 2019 14:13:52 +0000

> As far as I know, routers that hash fields in the IP header to select a en ECMP next hop do so because all packets in a flow will hash the same way (modulo the issues with the transport port number…

I agree with Fred. ECMP is normally done to distribute the load over the 
available next hops on a best effort basis. Originally it was done per 
packet, but that gave problems with out of order packet delivery, so the 
routers moved to doing it based on the five tuple described in this 
draft. It is a stateless best effort ECMP process with no regard to 
specific flows and the path for any five tuple may move arbitrarily if 
routing changes its mind on the ECMP set.

Fragmented packets are really bad news in networks that need ECMP. There 
is not enough entropy in the SA/DA/Protocol triplet and anything else 
results in misorder. But if ECMP is not done this overloads the default 
path.

MPLS is also stateless but there are more options, although the most 
common is to look past BoS to the five tuple, however some "features" 
make mistakes and look at a non-existent five tuple despite hints in the 
packet that thus is a bad idea.

>> "Routers SHOULD use IPv6 flow label for ECMP routing as described in [RFC6438]."

If we want to migrate to the FL then we really need to state that the FL 
MUST be set by the sender. Without, that we are never going to wean 
routers off looking at the five tuple, if indeed we ever succeed in 
doing that.

It we really need to fragment a packet, it would be better to stick the 
fragments inside a common UDP/IP(no frag) shim. Then the forwarders 
could carry on just as they are. We would never get misorder and we 
would not be faced with the impossible problem of changing the Internet 
core forwarding behaviour to a single consistent model.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 30 Jan 2019 14:17:42 +0000

> addresses only.

There is not enough entropy in that.

I remember the original ECMP studies, and if the designers could have 
got away with just SA/DA/Prot they would have. In those days we were 
using s/w forwarders and the ECMP "feature" cost headline PPS which in 
those days was the key metric.

> For IPv6 we can do better. The flow label allows finer

If you can trust it.

- Stewart

---

From: Tom Herbert <[email-masked]>
Date: Wed, 30 Jan 2019 08:30:59 -0800

> > header so keeping fragments in order just works.
Stewart,

I trust it more than I trust that vendors will ever go out of their
parse any transport layer protocols other than TCP or UDP to extract
port information, or that they'll happily skip over extension headers
or parse N levels of encapsulation to find anthe embedded transport
layer, or that they'll figure out a reasonable way to deal with
protocols that don't have enough information in plain text to discern
a five tuple.

Tom

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 30 Jan 2019 16:55:25 +0000

Hi Stewart,

>> It we really need to fragment a packet, it would be better to stick the fragments inside a common UDP/IP(no frag) shim.

I agree. Two different approaches for UDP fragmentation that avoid IP fragmentation
are currently under consideration:

https://datatracker.ietf.org/doc/draft-ietf-intarea-gue-extensions/
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-udp-options/

Thanks - Fred

From: Int-area [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: Wednesday, January 30, 2019 6:14 AM
To: Fred Baker <[email-masked]>; Tom Herbert <[email-masked]>
Cc: int-area <[email-masked]>
Subject: Re: [Int-area] Comments on draft-ietf-intarea-frag-fragile-06

Section 4.5:

forwarding;

I agree with Fred. ECMP is normally done to distribute the load over the available next hops on a best effort basis. Originally it was done per packet, but that gave problems with out of order packet delivery, so the routers moved to doing it based on the five tuple described in this draft. It is a stateless best effort ECMP process with no regard to specific flows and the path for any five tuple may move arbitrarily if routing changes its mind on the ECMP set.

Fragmented packets are really bad news in networks that need ECMP. There is not enough entropy in the SA/DA/Protocol triplet and anything else results in misorder. But if ECMP is not done this overloads the default path.

MPLS is also stateless but there are more options, although the most common is to look past BoS to the five tuple, however some "features" make mistakes and look at a non-existent five tuple despite hints in the packet that thus is a bad idea.

purposes of ECMP."

Add to section 7.3:

If we want to migrate to the FL then we really need to state that the FL MUST be set by the sender. Without, that we are never going to wean routers off looking at the five tuple, if indeed we ever succeed in doing that.

It we really need to fragment a packet, it would be better to stick the fragments inside a common UDP/IP(no frag) shim. Then the forwarders could carry on just as they are. We would never get misorder and we would not be faced with the impossible problem of changing the Internet core forwarding behaviour to a single consistent model.

- Stewart

Int-area mailing list

[email-masked]<mailto:[email-masked]>

     Sun Tzu

Int-area mailing list

[email-masked]<mailto:[email-masked]>

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 30 Jan 2019 18:15:00 +0000

Hi Fred

I had something quite simple in mind:

Fragment the IP packet just as you do today and send each fragment as 
opaque data in a simple 8 byte basic UDP payload with port set to IP. 
Set the source port based on a hash of the 5 tuple. Then resemble the IP 
just like you always would.

- Stewart

On 30/01/2019 16:55, Templin (US), Fred L wrote:
>
> Hi Stewart,
>
> >> It we really need to fragment a packet, it would be better to stick 
> the fragments inside a common UDP/IP(no frag) shim.
>
> *From:*Int-area [mailto:[email-masked]] *On Behalf Of 
> *Stewart Bryant
> *Sent:* Wednesday, January 30, 2019 6:14 AM
> *To:* Fred Baker <[email-masked]>; Tom Herbert 
> <[email-masked]>
> *Cc:* int-area <[email-masked]>
> *Subject:* Re: [Int-area] Comments on draft-ietf-intarea-frag-fragile-06
>
> I agree with Fred. ECMP is normally done to distribute the load over 
> the available next hops on a best effort basis. Originally it was done 
> per packet, but that gave problems with out of order packet delivery, 
> so the routers moved to doing it based on the five tuple described in 
> this draft. It is a stateless best effort ECMP process with no regard 
> to specific flows and the path for any five tuple may move arbitrarily 
> if routing changes its mind on the ECMP set.
>
> Fragmented packets are really bad news in networks that need ECMP. 
> There is not enough entropy in the SA/DA/Protocol triplet and anything 
> else results in misorder. But if ECMP is not done this overloads the 
> default path.
>
> make mistakes and look at a non-existent five tuple despite hints in 
> the packet that thus is a bad idea.
>
> If we want to migrate to the FL then we really need to state that the 
> FL MUST be set by the sender. Without, that we are never going to wean 
> It we really need to fragment a packet, it would be better to stick 
> the fragments inside a common UDP/IP(no frag) shim. Then the 
> forwarders could carry on just as they are. We would never get 
> misorder and we would not be faced with the impossible problem of 
> changing the Internet core forwarding behaviour to a single consistent 
> model.
>
> - Stewart
>
>         [email-masked]  <mailto:[email-masked]>
>
>     [email-masked]  <mailto:[email-masked]>
>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 30 Jan 2019 18:29:20 +0000

Hi Stewart,

Sounds like that would require some sort of encapsulation protocol and
low-level code in the kernel or hardware to strip the UDP headers, right?

Fred

From: Stewart Bryant [mailto:[email-masked]]
Sent: Wednesday, January 30, 2019 10:15 AM
To: Templin (US), Fred L <[email-masked]>; Fred Baker <[email-masked]>; Tom Herbert <[email-masked]>
Cc: int-area <[email-masked]>
Subject: Re: [Int-area] Comments on draft-ietf-intarea-frag-fragile-06

Hi Fred

Fragment the IP packet just as you do today and send each fragment as opaque data in a simple 8 byte basic UDP payload with port set to IP. Set the source port based on a hash of the 5 tuple. Then resemble the IP just like you always would.

- Stewart

Thanks - Fred

From: Int-area [mailto:[email-masked]] On Behalf Of Stewart Bryant
Sent: Wednesday, January 30, 2019 6:14 AM
To: Fred Baker <[email-masked]><mailto:[email-masked]>; Tom Herbert <[email-masked]><mailto:[email-masked]>
Cc: int-area <[email-masked]><mailto:[email-masked]>
Subject: Re: [Int-area] Comments on draft-ietf-intarea-frag-fragile-06

Section 4.5:

forwarding;

purposes of ECMP."

Add to section 7.3:

- Stewart

Int-area mailing list

     Sun Tzu

Int-area mailing list

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 30 Jan 2019 18:34:50 +0000

I think pretty much anything would need that wouldn't it?

- Stewart

On 30/01/2019 18:29, Templin (US), Fred L wrote:
> *From:*Stewart Bryant [mailto:[email-masked]]
> *Sent:* Wednesday, January 30, 2019 10:15 AM
> *To:* Templin (US), Fred L <[email-masked]>; Fred Baker 
> <[email-masked]>; Tom Herbert <[email-masked]>
> Set the source port based on a hash of the 5 tuple. Then resemble the 
> IP just like you always would.
>
> - Stewart
>
>     >> It we really need to fragment a packet, it would be better to
>     stick the fragments inside a common UDP/IP(no frag) shim.
>
>     *To:* Fred Baker <[email-masked]>
>     <mailto:[email-masked]>; Tom Herbert
>     <[email-masked]> <mailto:[email-masked]>
>     *Cc:* int-area <[email-masked]> <mailto:[email-masked]>
>     I agree with Fred. ECMP is normally done to distribute the load
>     over the available next hops on a best effort basis. Originally it
>     was done per packet, but that gave problems with out of order
>     packet delivery, so the routers moved to doing it based on the
>     five tuple described in this draft. It is a stateless best effort
>     ECMP process with no regard to specific flows and the path for any
>     five tuple may move arbitrarily if routing changes its mind on the
>     ECMP set.
>
>     There is not enough entropy in the SA/DA/Protocol triplet and
>     anything else results in misorder. But if ECMP is not done this
>     overloads the default path.
>
>     MPLS is also stateless but there are more options, although the
>     most common is to look past BoS to the five tuple, however some
>     "features" make mistakes and look at a non-existent five tuple
>     despite hints in the packet that thus is a bad idea.
>
>           
>
>     If we want to migrate to the FL then we really need to state that
>     the FL MUST be set by the sender. Without, that we are never going
>     to wean routers off looking at the five tuple, if indeed we ever
>     succeed in doing that.
>
>     It we really need to fragment a packet, it would be better to
>     stick the fragments inside a common UDP/IP(no frag) shim. Then the
>     changing the Internet core forwarding behaviour to a single
>     consistent model.
>
>     - Stewart
>
>               
>
>               
>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 30 Jan 2019 18:38:59 +0000

Right, but when considering bringing an encapsulation protocol on board, why not
bring in a mature one like GUE and use either GUE fragmentation or Joe Touch's
UDP trailer idea?

Fred

Sent: Wednesday, January 30, 2019 10:35 AM

- Stewart

Fred

To: Templin (US), Fred L <[email-masked]><mailto:[email-masked]>; Fred Baker <[email-masked]><mailto:[email-masked]>; Tom Herbert <[email-masked]><mailto:[email-masked]>

Hi Fred

- Stewart

Thanks - Fred

Section 4.5:

forwarding;

purposes of ECMP."

Add to section 7.3:

- Stewart

Int-area mailing list

     Sun Tzu

Int-area mailing list

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 30 Jan 2019 18:42:32 +0000

Because everyone already knows how to receive basic UDP and everyone 
already knows how to reassemble ordinary IP packets.

Along the path this is just ordinary UDP which everyone knows how to handle.

There is almost nothing to specify.

- Stewart

On 30/01/2019 18:38, Templin (US), Fred L wrote:
> bring in a mature one like GUE and use either GUE fragmentation or Joe 
> Touch’s
>
> UDP trailer idea?
>
> Fred
>
> *Sent:* Wednesday, January 30, 2019 10:35 AM
>     *To:* Templin (US), Fred L <[email-masked]>
>     <mailto:[email-masked]>; Fred Baker
>     <[email-masked]> <mailto:[email-masked]>; Tom
>     Herbert <[email-masked]> <mailto:[email-masked]>
>     Fragment the IP packet just as you do today and send each fragment
>     as opaque data in a simple 8 byte basic UDP payload with port set
>     to IP. Set the source port based on a hash of the 5 tuple. Then
>     resemble the IP just like you always would.
>
>     - Stewart
>
>         *From:*Int-area [mailto:[email-masked]] *On Behalf
>         Of *Stewart Bryant
>         I agree with Fred. ECMP is normally done to distribute the
>         load over the available next hops on a best effort basis.
>         Originally it was done per packet, but that gave problems with
>         out of order packet delivery, so the routers moved to doing it
>         based on the five tuple described in this draft. It is a
>         stateless best effort ECMP process with no regard to specific
>         flows and the path for any five tuple may move arbitrarily if
>         Fragmented packets are really bad news in networks that need
>         ECMP. There is not enough entropy in the SA/DA/Protocol
>         triplet and anything else results in misorder. But if ECMP is
>         not done this overloads the default path.
>
>         MPLS is also stateless but there are more options, although
>         the most common is to look past BoS to the five tuple, however
>         some "features" make mistakes and look at a non-existent five
>         tuple despite hints in the packet that thus is a bad idea.
>
>               
>
>         If we want to migrate to the FL then we really need to state
>         that the FL MUST be set by the sender. Without, that we are
>         never going to wean routers off looking at the five tuple, if
>         indeed we ever succeed in doing that.
>
>         stick the fragments inside a common UDP/IP(no frag) shim. Then
>         the forwarders could carry on just as they are. We would never
>         get misorder and we would not be faced with the impossible
>         problem of changing the Internet core forwarding behaviour to
>         a single consistent model.
>
>         - Stewart
>
>                   
>
>                   
>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 30 Jan 2019 18:55:42 +0000

GUE is basic UDP, and GUE fragmentation remedies IP fragmentation shortcomings
(e.g., reassembly errors at high data rates).

So, I get your point that the low-level code to insert and strip UDP headers is lighter
weight in that it leverages the ubiquitous IP fragmentation and reassembly code.
But, it still inherits the IP fragmentation shortcomings - so, maybe say that this
is applicable for IPv6 but not as much for IPv4?

Fred

Sent: Wednesday, January 30, 2019 10:43 AM

Because everyone already knows how to receive basic UDP and everyone already knows how to reassemble ordinary IP packets.

- Stewart

Fred

- Stewart

Fred

Hi Fred

- Stewart

Thanks - Fred

Section 4.5:

forwarding;

purposes of ECMP."

Add to section 7.3:

- Stewart

Int-area mailing list

     Sun Tzu

Int-area mailing list

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 31 Jan 2019 09:00:57 +1300

On 2019-01-31 03:13, Stewart Bryant wrote:
...

> If we want to migrate to the FL then we really need to state that the FL MUST be set by the sender. Without, that we are never going to wean routers off looking at the five tuple, if indeed we ever…

I would have loved to make it a MUST in RFC6437 but that wasn't the consensus, so it's a SHOULD, confirmed this very day by RFC8504 (https://tools.ietf.org/html/rfc8504#page-7).

As I said the other day, if and only if the flow label is non-zero, the {source, destination, flow_label} tuple is a fine thing for a load balancer to use. If the flow label is zero, you need to fall back to the 5-tuple, which is broken for fragmented packets as we know. So for IPv6 packets with a zero flow label and a fragmentation header, you have no choice but the {source, destination} tuple.

However, that is a much better picture than for IPv4.

    Brian

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 31 Jan 2019 10:15:49 +0000

On 30/01/2019 18:55, Templin (US), Fred L wrote:
>
> IP fragmentation shortcomings
>
How big a problem is this in reality?

If you are going to run at speed you ought to set a conservatively low MTU?

I am always told that the core runs jumbo anyway, so where does the 
problem arise in practical networks?

- Stewart

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 31 Jan 2019 15:53:36 +0000

Hi Stewart,

Check section 4.5 and 6.4 of the latest draft version:

https://datatracker.ietf.org/doc/html/draft-ietf-intarea-frag-fragile-08

Also, RFC4963 provides a detailed study of "IPv4 Reassembly Errors at
High Data Rates".

But, there is more to it besides just the question of whether to use IP
reassembly or higher layer reassembly. There must be some way for
the sender that inserts the UDP encapsulation to know that the receiver
is capable of removing the encapsulation. So, there must be some form
of tunnel endpoint coordination because, after all, it is really just another
type of tunnel.

Thanks - Fred

Sent: Thursday, January 31, 2019 2:16 AM

I am always told that the core runs jumbo anyway, so where does the problem arise in practical networks?

- Stewart