Subject: [Int-area] draft-ietf-intarea-frag-fragile-03

From: Ron Bonica <[email-masked]>
Date: Wed, 21 Nov 2018 14:32:25 +0000

Chairs,

I have posted a new version of draft-ietf-intarea-frag-fragile, working in comments from Tom and Brian.

If you see fit, please initiate a working group last call.

                                                       Ron

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 21 Nov 2018 16:46:45 +0000

Hi Ron,

There needs to be a new subsection in Section 6 on UDP applications that
rely on IP fragmentation for greater performance. Here is proposed text:

"Some UDP applications rely on IP fragmentation to achieve acceptable levels
of performance. These applications use UDP datagram sizes that are larger than
the path MTU so that more data can be conveyed between the application and
the kernel in a single system call.

Historically, NFS version 2 [RFC1094] set a UDP datagram size of 8KB which is
greater than the path MTU of most paths, resulting in IP fragmentation.
Currently, the Licklider Transmission Protocol [RFC5326] which is in current
use on the International Space Station (ISS) uses UDP datagram sizes larger
than the path MTU to achieve acceptable levels of performance even though
this too invokes IP fragmentation. Also, the commonly-used iperf3 [IPERF3]
performance testing utility by default sets an 8KB UDP datagram size even
though IP fragmentation is invoked since the performance of smaller UDP
datagrams is much lower.

While it is natural to suggest that such applications should adjust their
application layer framing to better match the path MTU, such does
not always result in greater performance. For example, although the
"sendmmsg()" system call was designed to present the kernel with
multiple UDP datagrams in a single call, not all applications benefit
from its use."

Thanks - Fred

> -----Original Message-----
> From: Int-area [mailto:[email-masked]] On Behalf Of Ron Bonica
> Sent: Wednesday, November 21, 2018 6:32 AM
> To: int-area <[email-masked]>; [email-masked]
> Subject: [Int-area] draft-ietf-intarea-frag-fragile-03
> 
> Chairs,
> 
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 27 Nov 2018 15:50:33 +0000

Ron,

Also, this document needs to cite RFC4963 ("IPv4 Reassembly Errors at
High Data Rates") as it is a clear commentary on the fragile nature of IP
fragmentation.

Thanks - Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Templin (US), Fred L
> Sent: Wednesday, November 21, 2018 8:47 AM
> To: Ron Bonica <[email-masked]>; int-area <[email-masked]>; [email-masked]
> Subject: Re: [Int-area] draft-ietf-intarea-frag-fragile-03
> 
> Hi Ron,
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 27 Nov 2018 08:34:28 -0800

> though IP fragmentation is invoked since the performance of smaller UDP
Hi Fred,

I agree that NFS and ISS protocol are worth mentioning, but iperf shouldn't
be mentioned since it is just a test program.

> multiple UDP datagrams in a single call, not all applications benefit

The hardest part is pmtu discovery, signaling discovered pmtu to the UDP
application, and then the application needs to know how to make smaller
packets to fit into mtu. The last step is especially problematic since it's
application specific logic. The natural recourse applications have is to
just assume minimal MTU, but as you point out that can be poor performance
compared to using fragmentation

Tom

---

From: Ron Bonica <[email-masked]>
Date: Tue, 27 Nov 2018 17:53:59 +0000

Fred,

Good idea. I will post a new version with a reference to RFC 4963 later today.

                                         Ron

> From: Templin (US), Fred L <[email-masked]>
> Sent: Tuesday, November 27, 2018 10:51 AM
> Subject: RE: draft-ietf-intarea-frag-fragile-03
> 
> Ron,
> 
> Also, this document needs to cite RFC4963 ("IPv4 Reassembly Errors at High
> Data Rates") as it is a clear commentary on the fragile nature of IP
> fragmentation.
> 
> Thanks - Fred
> 
> > There needs to be a new subsection in Section 6 on UDP applications
> > that rely on IP fragmentation for greater performance. Here is proposed
> text:
> >
> > "Some UDP applications rely on IP fragmentation to achieve acceptable
> > levels of performance. These applications use UDP datagram sizes that
> > are larger than the path MTU so that more data can be conveyed between
> > the application and the kernel in a single system call.
> >
> > Historically, NFS version 2 [RFC1094] set a UDP datagram size of 8KB
> > which is greater than the path MTU of most paths, resulting in IP
> fragmentation.
> > Currently, the Licklider Transmission Protocol [RFC5326] which is in
> > current use on the International Space Station (ISS) uses UDP datagram
> > sizes larger than the path MTU to achieve acceptable levels of
> > performance even though this too invokes IP fragmentation. Also, the
> > commonly-used iperf3 [IPERF3] performance testing utility by default
> > sets an 8KB UDP datagram size even though IP fragmentation is invoked
> > since the performance of smaller UDP datagrams is much lower.
> >
> > While it is natural to suggest that such applications should adjust
> > their application layer framing to better match the path MTU, such
> > does not always result in greater performance. For example, although
> > the "sendmmsg()" system call was designed to present the kernel with
> > > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_ma
> > > ilman_listinfo_int-
> 2Darea&d=DwIFAg&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb
> > > 3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=-tXT2W9
> > > S1KeEUa8Vw0vfz5JusrI4p_7kjY2d_7xcPGM&s=vCvphBVobYH-
> feMAXuNcPc6x9Ymgl
> > > Dwy8pTZEmL8HhY&e=
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> > man_listinfo_int-2Darea&d=DwIFAg&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-
> ndb3voD
> > TXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=-
> tXT2W9S1KeEU
> > a8Vw0vfz5JusrI4p_7kjY2d_7xcPGM&s=vCvphBVobYH-
> feMAXuNcPc6x9YmglDwy8pTZE
> > mL8HhY&e=

---

From: Ron Bonica <[email-masked]>
Date: Tue, 27 Nov 2018 18:02:53 +0000

Fred,

I am happy to add a section about LTP. However, I agree with Tom that we should not be talking about iPERF because:

-         It is a test utility

-         I can’t find a stable reference

Also, given that NFSv2 was obsoleted by RFC 3010 in 2000. I don’t think that it is relevant any longer.

                                                                                       Ron

From: Tom Herbert <[email-masked]>
Sent: Tuesday, November 27, 2018 11:34 AM
To: Templin (US), Fred L <[email-masked]>
Cc: Ron Bonica <[email-masked]>; int-area <[email-masked]>; [email-masked]
Subject: Re: [Int-area] draft-ietf-intarea-frag-fragile-03

On Wed, Nov 21, 2018, 8:47 AM Templin (US), Fred L <[email-masked]<mailto:[email-masked]>> wrote:
Hi Ron,

I agree that NFS and ISS protocol are worth mentioning, but iperf shouldn't be mentioned since it is just a test program.

The hardest part is pmtu discovery, signaling discovered pmtu to the UDP application, and then the application needs to know how to make smaller packets to fit into mtu. The last step is especially problematic since it's application specific logic. The natural recourse applications have is to just assume minimal MTU, but as you point out that can be poor performance compared to using fragmentation

Tom

Thanks - Fred

> From: Int-area [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Ron Bonica
> To: int-area <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/int-area<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_int-2Darea&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=dnt1U1J4Efpx1ypoLNGr5X-jSympqrAGJodux4abHZ8&s=35whtuRpRpLDWchEp0AJSJ0DWb1sUMYabC515g2LyjU&e=>

_______________________________________________
Int-area mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/int-area<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_int-2Darea&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=dnt1U1J4Efpx1ypoLNGr5X-jSympqrAGJodux4abHZ8&s=35whtuRpRpLDWchEp0AJSJ0DWb1sUMYabC515g2LyjU&e=>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 27 Nov 2018 18:46:51 +0000

Hi Ron,

Regarding NFSv2 [RFC1094] being obsoleted, that is why I qualified it as “Historic”.
But, I believe it is useful to include because it includes explicit text regarding
fragmentation:

    “For efficient operation over a local network, 8192 bytes of data are normally
     used. This may result in lower-level fragmentation (such as at the IP level).”

This goes to show the mindset of the 1980’s that has carried forward into the
present day with LTP and iperf3 as examples.

Regarding iperf3, I think it is informative because it shows that the concept
of the use of fragmentation for improved performance is still alive within
the present-day mindset. Also, informative references need not be RFCs;
in fact, the draft already has four non-RFC informative references that have
URLs as their location. For [IPERF3], the citation is simply: https://iperf.fr/

Can these be included?

Thanks - Fred

From: Ron Bonica [mailto:[email-masked]]
Sent: Tuesday, November 27, 2018 10:03 AM
To: Tom Herbert <[email-masked]>; Templin (US), Fred L <[email-masked]>
Cc: int-area <[email-masked]>; [email-masked]
Subject: RE: [Int-area] draft-ietf-intarea-frag-fragile-03

Fred,

-         It is a test utility

                                                                                       Ron

From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, November 27, 2018 11:34 AM
To: Templin (US), Fred L <[email-masked]<mailto:[email-masked]>>
Cc: Ron Bonica <[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Int-area] draft-ietf-intarea-frag-fragile-03

Tom

Thanks - Fred

---

From: Ron Bonica <[email-masked]>
Date: Tue, 27 Nov 2018 19:16:33 +0000

Fred,

In your mind, are these blocking issues?

                                               Ron

From: Templin (US), Fred L <[email-masked]>
Sent: Tuesday, November 27, 2018 1:47 PM
To: Ron Bonica <[email-masked]>; Tom Herbert <[email-masked]>
Cc: int-area <[email-masked]>; [email-masked]
Subject: RE: [Int-area] draft-ietf-intarea-frag-fragile-03

Hi Ron,

URLs as their location. For [IPERF3], the citation is simply: https://iperf.fr/<https://urldefense.proofpoint.com/v2/url?u=https-3A__iperf.fr_&d=DwMGaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=hoRaVRF6o5_q4CRg4kbGZQXyKPU4hrnhra0kKrPwH20&s=7rL921FM5ZTBqOQXsIzaU76BxSD5fQzTbuhBPh0jxiQ&e=>

Can these be included?

Thanks - Fred

From: Ron Bonica [mailto:[email-masked]]
Sent: Tuesday, November 27, 2018 10:03 AM
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>; Templin (US), Fred L <[email-masked]<mailto:[email-masked]>>
Cc: int-area <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: [Int-area] draft-ietf-intarea-frag-fragile-03

Fred,

-         It is a test utility

                                                                                       Ron

Tom

Thanks - Fred

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 27 Nov 2018 19:57:22 +0000

Hi Ron,

> In your mind, are these blocking issues?

I don’t have an opinion about “blocking”, but IMHO the recommendations
contribute to the completeness of the document.

Thanks - Fred

---

From: Ron Bonica <[email-masked]>
Date: Tue, 27 Nov 2018 20:22:57 +0000

Fred,

If the NFSv2 and iPERF issues are not blocking, I would like to omit them. The following are rational:

- Section 6 is an admittedly incomplete list of applications that rely on IP fragmentation. In its words, "This list is not comprehensive, and other protocols that rely on IP
   fragmentation may exist.  They are not specifically considered in the context of this document."

- In draft-ietf-intarea-frag-fragile-04 we say, "Some UDP applications rely on IP fragmentation to achieve acceptable levels of performance.  These applications use UDP datagram sizes that are larger than the path MTU so that more data can be conveyed between the application and the kernel in a single system call." This captures the mindset, even if that mindset is in conflict with RFC 8085".

- Mechanically, it is difficult to reference an RFC that has been obsoleted in an internet draft. The NIT checker complains bitterly.

- You and Tom Herbert disagree on iPERF. Since the example is not essential to the draft, I would prefer not to hold up the draft on this issue.

                                                                                                 Ron

> Sent: Tuesday, November 27, 2018 2:57 PM

---

From: Joe Touch <[email-masked]>
Date: Tue, 27 Nov 2018 12:42:24 -0800

FWIW:

On 2018-11-27 12:22, Ron Bonica wrote:

> ...

Those complaints are warnings only to help those who cite such documents
inadvertently; you can simply ignore them. (I do all the time - esp. for
historical discussions that cite early versions of newer RFCs or
historical standards). 

Joe

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 29 Nov 2018 12:06:15 +0000

But, always worth including a "do be deleted" note to the reviewers to 
stop then all sending in feedback about the nits failure.

Stewart

On 27/11/2018 20:42, Joe Touch wrote:
>> - Mechanically, it is difficult to reference an RFC that has been 
>> obsoleted in an internet draft. The NIT checker complains bitterly.
> Those complaints are warnings only to help those who cite such 
> documents inadvertently; you can simply ignore them. (I do all the 
> time - esp. for historical discussions that cite early versions of 
> newer RFCs or historical standards).
> Joe
>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 29 Nov 2018 06:42:44 -0800

They don’t need to be deleted if you include them deliberately. There is no prohibition on citing such RFCs for your own documents historical background.

Joe

> On Nov 29, 2018, at 4:06 AM, Stewart Bryant <[email-masked]> wrote:
> 
> But, always worth including a "do be deleted" note to the reviewers to stop then all sending in feedback about the nits failure.
> 
> Stewart
> 
> 
>> Those complaints are warnings only to help those who cite such documents inadvertently; you can simply ignore them. (I do all the time - esp. for historical discussions that cite early versions of newer RFCs or historical standards).
>>  
>> Joe
>> 
>>

---

From: Ron Bonica <[email-masked]>
Date: Thu, 29 Nov 2018 16:29:20 +0000

Folks,

Even if citations weren’t an issue, we are really reaching into the distant past.

-         NFSv2 was specified by RFC 1094 in 1989.

o    I was 31 years old then.

-          NFS2 was obsoleted by NFSv4 (RFC 3010) in 2000.

-         RFC 3010 was obsoleted by RFC 3530 in 2003

-         RFC 3530 was obsoleted by RFC 7530 in 2015

-         We are citing NFSv2 in a draft that will be published in 2019

o   I will be 61 years old by then.

                                                                               Ron

From: Joe Touch <[email-masked]>
Sent: Thursday, November 29, 2018 9:43 AM
To: Stewart Bryant <[email-masked]>
Cc: Ron Bonica <[email-masked]>; int-area <[email-masked]>; [email-masked]
Subject: Re: [Int-area] draft-ietf-intarea-frag-fragile-03

Joe

On Nov 29, 2018, at 4:06 AM, Stewart Bryant <[email-masked]<mailto:[email-masked]>> wrote:

Stewart

FWIW:

...

Joe

Int-area mailing list

https://www.ietf.org/mailman/listinfo/int-area<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_int-2Darea&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=WZtxIWwD7rKWm0NwifH_EJaXipQUe586Y2w8wTSTVA0&s=E5ulB1h8gcizfJIlKPsEdxujR-KbKY2CVxH0yIfLitk&e=>

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 29 Nov 2018 16:37:43 +0000

Either way it is useful to give the reviewer a heads up as to nits 
giving errors and this being OK.

S

On 29/11/2018 14:42, Joe Touch wrote:
> They don’t need to be deleted if you include them deliberately. There 
> is no prohibition on citing such RFCs for your own documents 
> historical background.
>
> Joe
>
> On Nov 29, 2018, at 4:06 AM, Stewart Bryant <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> But, always worth including a "do be deleted" note to the reviewers 
>> to stop then all sending in feedback about the nits failure.
>>
>> Stewart
>>
>>

---

From: Joe Touch <[email-masked]>
Date: Thu, 29 Nov 2018 08:38:37 -0800

And yet:
https://askubuntu.com/questions/974696/how-can-i-make-the-nfs-server-support-protocol-version-2-in-ubuntu-17-10

I’m not arguing either way, but it’s not the case that old, deprecated protocols aren’t still running.

Joe

> On Nov 29, 2018, at 8:29 AM, Ron Bonica <[email-masked]> wrote:
> 
> Folks,
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 29 Nov 2018 08:40:45 -0800

I would hope it would be evident from context, but sure.

Joe

> On Nov 29, 2018, at 8:37 AM, Stewart Bryant <[email-masked]> wrote:
> 
> 
> Either way it is useful to give the reviewer a heads up as to nits giving errors and this being OK.
> 
> S
>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 29 Nov 2018 18:08:06 +0000

Hi, regardless of how old and obsolete NFSv2 is, the text in RFC1094 perfectly
addresses the point. I was working for Digital Equipment Corporation in the
late 1980’s when Sun Microsystems first came out with NFS. The two companies
tried to form an agreement where DEC would provide VAXen as NFS servers for
Sun clients, but many DEC VAX systems used an Ethernet line card called the
DELQA that was too slow to keep up with 10Mbps Ethernet line rates (!).

When the VAX received a burst of IP fragments of an 8KB NFS/UDP datagram,
it would drop one or more fragments and print “qerestart” in the system log.
The DELQA would then reset itself and the network would be out of service
until the restart completed. This rendered VAXen with DELQAs useless as an
NFS server, so DEC had to qualify which of the VAX product line could actually
support NFS. And, all because of IP fragmentation.

So, I think there is value in keeping the citation.

Thanks - Fred

From: Int-area [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Thursday, November 29, 2018 8:41 AM
To: Stewart Bryant <[email-masked]>
Cc: Ron Bonica <[email-masked]>; int-area <[email-masked]>; [email-masked]
Subject: Re: [Int-area] draft-ietf-intarea-frag-fragile-03

Joe

On Nov 29, 2018, at 8:37 AM, Stewart Bryant <[email-masked]<mailto:[email-masked]>> wrote:

S

Joe

Stewart

FWIW:

...

Joe

Int-area mailing list

---

From: Ron Bonica <[email-masked]>
Date: Thu, 29 Nov 2018 18:26:59 +0000

Fred,

Without being too abrupt, there were then , and are now,  much better ways to solve this problem.

A better approach would have been to fragment / reassemble at a higher layer. That is the whole point of this document.

                                                    Ron

From: Templin (US), Fred L <[email-masked]>
Sent: Thursday, November 29, 2018 1:08 PM
To: Joe Touch <[email-masked]>; Stewart Bryant <[email-masked]>
Cc: Ron Bonica <[email-masked]>; int-area <[email-masked]>; [email-masked]
Subject: RE: [Int-area] draft-ietf-intarea-frag-fragile-03

Thanks - Fred

To: Stewart Bryant <[email-masked]<mailto:[email-masked]>>

Joe

S

Joe

Stewart

FWIW:

...

Joe

Int-area mailing list

https://www.ietf.org/mailman/listinfo/int-area<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_int-2Darea&d=DwMGaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=FJe-ZLz5W_5P4zZNXPS2c1Uzn88Qj9c3vCcWT2wlPdw&s=j5WzcrfP668O1LWElFJ6h4LCpj4WKk7rkh-wSo4UuQg&e=>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 29 Nov 2018 18:39:05 +0000

Hi Ron,

Sun could not get the performance they wanted with smaller than 8KB
NFS/UDP datagrams – that is why they wrote it into the spec:

   “For efficient
   operation over a local network, 8192 bytes of data are normally used.
   This may result in lower-level fragmentation (such as at the IP
   level).  Since some network interfaces may not allow such packets,
   for operation over slower-speed networks or hosts, or through
   gateways, transfer sizes of 512 or 1024 bytes often provide better
   results.”

In the final sentence in the above, they were referring to the DELQA.

Fred

From: Ron Bonica [mailto:[email-masked]]
Sent: Thursday, November 29, 2018 10:27 AM
To: Templin (US), Fred L <[email-masked]>; Joe Touch <[email-masked]>; Stewart Bryant <[email-masked]>
Cc: int-area <[email-masked]>; [email-masked]
Subject: RE: [Int-area] draft-ietf-intarea-frag-fragile-03

Fred,

                                                    Ron

From: Templin (US), Fred L <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, November 29, 2018 1:08 PM
To: Joe Touch <[email-masked]<mailto:[email-masked]>>; Stewart Bryant <[email-masked]<mailto:[email-masked]>>
Cc: Ron Bonica <[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: [Int-area] draft-ietf-intarea-frag-fragile-03

Thanks - Fred

Joe

S

Joe

Stewart

FWIW:

...

Joe

Int-area mailing list

---

From: Fred Baker <[email-masked]>
Date: Thu, 29 Nov 2018 10:54:42 -0800

I agree that this was true 30 years ago. 30 years ago, I was working on what we would today call a variation on an Ethernet Switch, and DEC complained bitterly about the fact that it wouldn't run at line rate. By that, they meant that if they streamed Ethernet packets at it at 10 MBPS back to back, it would invariably lose some - it couldn't handle a 96 bit inter-frame gap as specified in the Ethernet II specification. The issue was that, at the time, there were only three or four commodity chipsets for Ethernet interfaces, and none of them could receive and store Ethernet traffic at 10 MBPS back to back.

I'm not worried about 10 MBPS Ethernet in 2018, nor would I expect anyone to worry about DELQA cards in Vaxes. Unless the comment is specifically tagged as a historical example, I would hope that we could discuss current issues if we're writing current advice.

The only current issue that comes quickly to mind is that you use a test/diagnostic tool that uses fragmentation to artificially generate a high data rate, which becomes an issue on some interfaces. I recognize that the tool exists and has the limitation, but that doesn't seem to me to rise to the level of general Internet advice.

> On Nov 29, 2018, at 10:39 AM, Templin (US), Fred L <[email-masked]> wrote:
> 
> Hi Ron,
> 

--------------------------------------------------------------------------------
The fact that there is a highway to hell and a stairway to heaven is an interesting comment on projected traffic volume...

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 29 Nov 2018 19:02:21 +0000

Hi Fred,

> From: Fred Baker [mailto:[email-masked]]
> Sent: Thursday, November 29, 2018 10:55 AM
> Cc: Ron Bonica <[email-masked]>; Joe Touch <[email-masked]>; Stewart Bryant <[email-masked]>; int-area
> <[email-masked]>; [email-masked]
> I agree that this was true 30 years ago. 30 years ago, I was working on what we would today call a variation on an Ethernet Switch, and
> DEC complained bitterly about the fact that it wouldn't run at line rate. By that, they meant that if they streamed Ethernet packets at it
> at 10 MBPS back to back, it would invariably lose some - it couldn't handle a 96 bit inter-frame gap as specified in the Ethernet II
> specification. The issue was that, at the time, there were only three or four commodity chipsets for Ethernet interfaces, and none of
> them could receive and store Ethernet traffic at 10 MBPS back to back.
> 
> I'm not worried about 10 MBPS Ethernet in 2018, nor would I expect anyone to worry about DELQA cards in Vaxes. Unless the
> comment is specifically tagged as a historical example, I would hope that we could discuss current issues if we're writing current
> advice.

The comment was specifically tagged as historical.

> The only current issue that comes quickly to mind is that you use a test/diagnostic tool that uses fragmentation to artificially generate
> a high data rate, which becomes an issue on some interfaces. I recognize that the tool exists and has the limitation, but that doesn't
> seem to me to rise to the level of general Internet advice.

If you are referring to iperf3, that is exactly what it does (uses fragmentation
to generate a high data rate), but I would not call it "artificial" - fragmentation
is a real Internet service.

Thanks - Fred

> > Those complaints are warnings only to help those who cite such documents inadvertently; you can simply ignore them. (I do all the
> time - esp. for historical discussions that cite early versions of newer RFCs or historical standards).
> >
> > Joe
> >
> >

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Nov 2018 11:25:59 -0800

Ron,

I'm not sure I agree that it's always better to fragment/reassemble at
a higher layer. The salient point about NFSv2 is that it's UDP and 8K
blocks are more efficient to send. There are still similar use cases
of UDP applications in datacenters, and there's no uniform way to deal
with frag+reassmbly at the application layer. Just the other day I was
talking to someone that works in a very large datacenter environment,
and he mentioned that were in the process of changing MTUs to be a
little above 4K (sending pages in a single packet is very low overhead
and this make zero copy trivial). A problem of a large sprawling DC is
that there are no flag days where everything is transitioned to new
configurations, so it's very possible that some portion of the DC is
not updated to support the larger MTU. In that case, it's much easier
to just rely on PMTU discovery and IP fragmentation, rather than
rewriting all the UDP applications to implement fragmentation.

Tom

---

From: Fred Baker <[email-masked]>
Date: Thu, 29 Nov 2018 12:52:49 -0800

> On Nov 29, 2018, at 11:02 AM, Templin (US), Fred L <[email-masked]> wrote:
> 

But iperf3 is a test/diagnostic tool, not a "real" internet application. The reason for bringing up NFS is that it was, at least at one time, a real service that was widely used in real networks, and breaking it would have been a real operational problem. Even for that, since fragmentation would likely occur at the originating host (as in IPv6), apart from tunnels or links that reduced the PMTU, the issue would have been between the endpoints and any middleware en route, and since NFS didn't cross network boundaries, it would be unusual to find firewalls in the path.

For NFSv4, RFC 5661 requires the use of TCP, SCTP, or "whatever is recommended at the time", which could perhaps refer to QUIC today, and discourages the use of UDP. It also refers to RDMA etc.

I frankly don't see what "observing the demonstrable and measurable fact that there are issues with Internet Fragmentation and Reassembly" hurts. The paper doesn't recommend that the service be deprecated or removed. It does, however, suggest that applications and transports not depend on Internet fragmentation and reassembly; applications above the IP layer that need packets smaller than the relevant PMTU should find a way (such as by using TCP) to avoid it. I don't see how iperf3, or for that matter any application, is harmed by that.

I think that what we really need to understand is what measurable harm this comment and recommendation causes. Maybe you could start your reply with that issue?
--------------------------------------------------------------------------------
The fact that there is a highway to hell and a stairway to heaven is an interesting comment on projected traffic volume...

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 29 Nov 2018 21:05:32 +0000

iperf3 is a real Internet application in the same way that ping, traceroute,
tcpdump, etc. are real applications. It is a well-known tool that network
engineers use on a daily basis and demonstrates that UDP performance
is highly correlated with UDP datagram size (i.e., even for sizes that
exceed the PMTU).

> Sent: Thursday, November 29, 2018 12:53 PM
> <[email-masked]>
> But iperf3 is a test/diagnostic tool, not a "real" internet application. The reason for bringing up NFS is that it was, at least at one time, a
> real service that was widely used in real networks, and breaking it would have been a real operational problem. Even for that, since
> fragmentation would likely occur at the originating host (as in IPv6), apart from tunnels or links that reduced the PMTU, the issue
> would have been between the endpoints and any middleware en route, and since NFS didn't cross network boundaries, it would be
> unusual to find firewalls in the path.
> 
> For NFSv4, RFC 5661 requires the use of TCP, SCTP, or "whatever is recommended at the time", which could perhaps refer to QUIC
> today, and discourages the use of UDP. It also refers to RDMA etc.
> 
> I frankly don't see what "observing the demonstrable and measurable fact that there are issues with Internet Fragmentation and
> Reassembly" hurts. The paper doesn't recommend that the service be deprecated or removed. It does, however, suggest that
> applications and transports not depend on Internet fragmentation and reassembly; applications above the IP layer that need packets
> smaller than the relevant PMTU should find a way (such as by using TCP) to avoid it. I don't see how iperf3, or for that matter any
> application, is harmed by that.
>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 29 Nov 2018 15:40:31 -0800

> is highly correlated with UDP datagram size (i.e., even for sizes that

AFAIK the main difference between large and small UDP packets is doing
the fragmentation/reassembly in the application instead of in the
kernel's UDP stack. For an 8K packet, that means 6 socket calls in the
application case, versus 1 in the kernel case. That is indeed some
overhead. It is not a big deal for medium speed application like video,
but i can see how it will get in the way of running QUIC at several
Gbps, for instance. But then, the overhead could be trivially eliminated
with API improvements, such as passing several packets in a single call.
We can ponder why we have not seen such improvements yet, the main
explanation being lack of demand. I fully expect that this will change
if QUIC gets widely used. In fact, it would be good if the fragmentation
draft discussed this API issue.

-- Christian Huitema

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Nov 2018 15:54:32 -0800

> with API improvements, such as passing several packets in a single call.

sendmmsg and recvmmsg do that.

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 30 Nov 2018 15:25:44 +0000

Hi Christian,

> From: Christian Huitema [mailto:[email-masked]]
> Sent: Thursday, November 29, 2018 3:41 PM
> To: Templin (US), Fred L <[email-masked]>; Fred Baker <[email-masked]>
> Cc: Ron Bonica <[email-masked]>; int-area <[email-masked]>

Right, we have talked about that. A disadvantage is that the application
framing headers need to be repeated in each UDP datagram, and when
the application headers are large the amount of room left for actual data
is diminished.

Thanks - Fred

---

From: Christian Huitema <[email-masked]>
Date: Fri, 30 Nov 2018 07:40:23 -0800

> On Nov 30, 2018, at 7:25 AM, Templin (US), Fred L <[email-masked]> wrote:
> 

Seems like a weak argument. If the application is important and if performance is an issue, the application should be updated.

-- Christian Huitema

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Nov 2018 07:49:51 -0800

If what you say were true, then we should all run everything over UDP and skip TCP. There are two reasons to the contrary:

1) the lower down the fragmentation occurs, the less overhead is needed (i.e., when performance is an issue, it’s even more important to fragment as low as possible)
2) it’s useful to not repeat effort and learning curves in each app

Add that to the fact that ti’s possible to improve lower layers without revising every app (i.e., shared benefit from updates).

Joe

> On Nov 30, 2018, at 7:40 AM, Christian Huitema <[email-masked]> wrote:
> 
> 
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Nov 2018 08:09:31 -0800

> Seems like a weak argument. If the application is important and if performance is an issue, the application should be updated.

Rather the application performs fragmentation or it's done at a lower
should show little difference in performance; either way there's
equivalent work and packet overhead. The real performance advantage
comes when the PMTU is large so that no fragmentation is done, that
can substantially reduce number of packets sent and received.
Fragmentation is needed along with PMTU discovery then because not all
PMTUs are guaranteed to be the same even in a closed network.

A material difference between application doing fragmentation and
doing at IP layer is that the former requires every application to
implement support, and the latter generically supports all
applications. I think the draft is correct in saying that IP
fragmentation should be avoided on the Internet. But in a closed
environment like a datacenter where there's no typically no
problematic devices like stateful firewalls in the path and there's
little chance of attack on the reassmebly queue, IP fragmentation is
perfectly useful and in use. However, even in that environment there
are improvements that could be made like using flow label to to get
consistent forwarding between fragments and non-fragments.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Fri, 30 Nov 2018 16:11:56 +0000

Folks,

Do we need to explore this particular rabbit hole before progressing the draft?

Please recall the following:

- Nobody has proposed deprecation of fragmentation functionality. In fact, we took that off the table on Page 1.
- Without passing any judgement, we said that some applications fragment not so much to overcome MTU issues, but to improve performance. We also offered a *modern* example (Licklider Transport Protocol).

I am wondering why were are expending so many calories arguing about whether additional examples are required. We only need one example to make the point. We don't need twelve.

                                                                                                                         Ron

> Sent: Friday, November 30, 2018 10:50 AM
> To: Christian Huitema <[email-masked]>
> Cc: Templin (US), Fred L <[email-masked]>; Ron Bonica
> <[email-masked]>; int-area <[email-masked]>
> If what you say were true, then we should all run everything over UDP and
> skip TCP. There are two reasons to the contrary:
> 
> 1) the lower down the fragmentation occurs, the less overhead is needed (i.e.,
> when performance is an issue, it’s even more important to fragment as low as
> possible)
> Add that to the fact that ti’s possible to improve lower layers without revising
> every app (i.e., shared benefit from updates).
> 
> Joe
> 
> >> Right, we have talked about that. A disadvantage is that the
> >> application framing headers need to be repeated in each UDP datagram,
> >> and when the application headers are large the amount of room left
> >> for actual data is diminished.
> >
> > Seems like a weak argument. If the application is important and if
> performance is an issue, the application should be updated.
> >
> > -- Christian Huitema
> > man_listinfo_int-2Darea&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-
> ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=KmLvgEOF-
> 3Tf1O5M01llb5ZbsHUSGdqWdt0dSafMq-s&s=3mf04eYiBWbVhxq-
> BXKuS2dqAL8A2aujgdSIVqwgbro&e=

---

From: Ole Troan <[email-masked]>
Date: Fri, 30 Nov 2018 18:22:28 +0100

> On 30 Nov 2018, at 16:49, Joe Touch <[email-masked]> wrote:
> 

That sounds like an unfounded myth. 
I would think it highly dependent on implementation.  

Ole

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Nov 2018 09:33:46 -0800

> On Nov 30, 2018, at 9:22 AM, Ole Troan <[email-masked]> wrote:
> 
> 
> 

Reality:

- every layer down you do it avoids a layer of header in-between *at every fragment*
ie., IP fragments have only ONE UDP header and ONE application header, but app-fragments have multiples of both.

Do the math.

Joe

---

From: Ole Troan <[email-masked]>
Date: Fri, 30 Nov 2018 18:46:14 +0100

> On 30 Nov 2018, at 18:33, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Every ipv6 fragment has an additional 8 byte header. But the network might not be the bottleneck here, and a few more bytes might not matter. As I said it depends. 
When it comes to performance making blanket statements is rarely a good idea. 

Ole

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Nov 2018 09:55:18 -0800

> On Nov 30, 2018, at 9:46 AM, Ole Troan <[email-masked]> wrote:
> 
> 
> 
> Every ipv6 fragment has an additional 8 byte header.

For IPv6, yes (not for IPv4). But that only counters the extra UDP header; there’s still the application fragmentation header too.

> But the network might not be the bottleneck here, and a few more bytes might not matter. As I said it depends. 

Such as "When it comes to performance making blanket statements is rarely a good idea.”, agreed.

But it’s nowhere near an unfounded myth.

Joe

---

From: Ron Bonica <[email-masked]>
Date: Fri, 30 Nov 2018 17:57:13 +0000

Guys,

Does it really matter! Nobody is arguing that applications SHOULD send packets larger than the MTU in order to improve performance. 

At best, they are arguing that a few outliers (LTP, NFSv2, iPERF) do.

                                                                                      Ron

> From: Ole Troan <[email-masked]>
> Sent: Friday, November 30, 2018 12:46 PM
> To: Joe Touch <[email-masked]>
> Cc: Christian Huitema <[email-masked]>; Ron Bonica
> >>> 1) the lower down the fragmentation occurs, the less overhead is
> >>> needed (i.e., when performance is an issue, it’s even more important
> >>> to fragment as low as possible)
> >>
> > - every layer down you do it avoids a layer of header in-between *at
> > every fragment* ie., IP fragments have only ONE UDP header and ONE
> application header, but app-fragments have multiples of both.
> >
> > Do the math.
> 
> Every ipv6 fragment has an additional 8 byte header. But the network might
> not be the bottleneck here, and a few more bytes might not matter. As I said it
> depends.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Nov 2018 10:52:16 -0800

> Such as "When it comes to performance making blanket statements is rarely a good idea.”, agreed.

That is true and it follows that more bytes in headers doesn't
necessarily mean worse performance. For instance, it is much more
expensive to compute the UDP checksum across IP fragments than if the
application fragmented the message it self and each one has its own
UDP header (checksum calculation is offloaded to the the NIC in the
latter case).

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Nov 2018 11:12:42 -0800

Ron,

I think the question is whether these really are outliers (DNS
definitely isn't). For instance, while NFSv2 may have been deprecated,
there are other block data protocols that still are in use and
presumably could benefit from IP fragmentation (ROCEv2 for example).
It's pretty obvious that fragmentation isn't currently feasible on the
open Internet, but for other environments, like the datacenter, it may
not only be relevant but desirable compared to other solutions.

While the draft doesn't deprecate IP fragmentation, it does say
"Protocol developers SHOULD NOT develop new protocols that rely on IP
fragmentation.".  That's a pretty negative and general statement
against fragmentation. Without more qualification, I'm not sure
getting consensus on this recommendation will be easy.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Fri, 30 Nov 2018 19:43:21 +0000

Inline......

> Sent: Friday, November 30, 2018 2:13 PM
> To: Ron Bonica <[email-masked]>
> Cc: Ole Troan <[email-masked]>; Joe Touch
> <[email-masked]>; int-area <[email-masked]>
> > Does it really matter! Nobody is arguing that applications SHOULD send
> packets larger than the MTU in order to improve performance.
> >
> I think the question is whether these really are outliers (DNS definitely isn't).

DNS doesn't fragment to improved performance. It fragments to overcome PMTU issues.

> For instance, while NFSv2 may have been deprecated, there are other block
> data protocols that still are in use and presumably could benefit from IP
> fragmentation (ROCEv2 for example).
> It's pretty obvious that fragmentation isn't currently feasible on the open
> Internet, but for other environments, like the datacenter, it may not only be
> relevant but desirable compared to other solutions.

So what makes the data center different from the global Internet in this regard. Are there no middle boxes in the data center? Are the security vulnerabilities associated with IP fragmentation not in play in the data center?

> 
> While the draft doesn't deprecate IP fragmentation, it does say "Protocol
> developers SHOULD NOT develop new protocols that rely on IP
> fragmentation.".  That's a pretty negative and general statement against
> fragmentation. Without more qualification, I'm not sure getting consensus on
> this recommendation will be easy.
> 

There is a big difference between SHOULD NOT and MUST NOT. If your application only needs to work in a constrained environment, you can do whatever you want. But if you want your application to work on the wide, wild Internet, I would stay away from IP fragmentation.

> > > >>> needed (i.e., when performance is an issue, it’s even more
> > > >>> important to fragment as low as possible)
> > > >>
> > > > - every layer down you do it avoids a layer of header in-between
> > > > *at every fragment* ie., IP fragments have only ONE UDP header and
> > > > ONE
> > > Every ipv6 fragment has an additional 8 byte header. But the network
> > > might not be the bottleneck here, and a few more bytes might not
> > > matter. As I said it depends.
> > TXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=p2eBgDa2mRb2N
> > 653HhD35xvAlC7u6kSPLz959fuBoxE&s=uxbkh59r-QS-rEskdqTWAGRWGLl-
> WGSTfTzUp
> > O8C2O8&e=

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Nov 2018 12:27:27 -0800

Ron,

Typically, you won't find middleboxes in the datacenter. Datacenter
networks are often CLOS fabrics built to be non-blocking and minimize
latency between hosts. Middleboxes live at the edge of the
datacenter-- firewalls, proxies, frontend servers, NAT, etc. There are
probably two primary use cases of bandwidth in the datacenter-- RPC
and remote memory/storage access. These have been implemented with
TCP, UDP, or other IP protocols including proprietary IP protocols.
There have also been attempts to use non-IP protocols like ROCE and
FCoE. Fortunately, it seems most of the use cases have converged to
use TCP/IP or UDP/IP. Also, at least in the datacenter environments I
work in, multicast is not relevant as a application visible protocol.

Datacenters are protected from the outside world from firewalls, so
security is mostly assumed internally. The model does change somewhat
if the datacenter supports third party multi-tenancy as that opens up
the possibility of internal attacks, so the traditional model of
perimeter security breaks done. In that case there may be middleboxes,
either network devices or in hypervisors of servers, that isolate
virtual networks from one another.

> There is a big difference between SHOULD NOT and MUST NOT. If your application only needs to work in a constrained environment, you can do whatever you want. But if you want your application to wor…
I agree with that sentiment, but maybe the wording of the
recommendation should be scoped correspondingly to only apply to
protocols intended for use over the Internet.

Tom

---

From: Fred Baker <[email-masked]>
Date: Fri, 30 Nov 2018 12:39:45 -0800

> getting consensus on this recommendation will be easy.

Thank you for your constructive comment. It at least gives us a direction to try to improve the draft in.

Two thoughts:

"SHOULD" means, per RFC 2119, that it is a case in which the author would like to make a blanket statement but recognizes that there may be exceptions that s/he has not considered, and requires an implementor that doesn't follow the mandate to state the exception. In this case, a new protocol that uses IP fragmentation rather than using some higher layer would be expected to say why - the default interpretation being that s/he was lazy or ignorant and couldn't be bothered. In the event that a new protocol is developed that depends on IP fragmentation, would you argue that the developer should not be asked to think about and respond to the question "why"?

The reason the draft makes a negative statement about fragmentation/reassembly is that there is a measurable issue with fragmentation/reassembly. To pick a data point, Geoff Huston has been measuring a blogging in the past year or two about MTUs observed in DNS root and resolver servers, and the reduced reliability of DNS as a result of dependence on IP fragmentation. Fernando Gont has been doing experiments measuring the impact of IPv6 extension headers in transit, one of which is the fragmentation header. Another point, although I refer to general commentary rather than data, is that PLPMTUD is not generally used for some reason, and PMTUD - and packet fragmentation in general - is frequently disrupted by filters. That is the reason that, as you state, fragmentation is not generally useful on packets that will cross the open Internet. The issue with saying it can only be used in a confined space is that the application generally doesn't know the location or path that its packets will use. I'm thinking of a certain banking configuration that has been discussed in IETF circles in the past couple of years that has multiple layers of load balancers and firewall-equivalents within a single data center. The application simply doesn't know whether IP fragmentation will be useful in a given environment. If it is forced to make a non-fragmentation solution available as well as a fragmentation-based solution, the questions arise how it distinguishes the cases and why it really needs both.

So I'm really looking for alternative text that makes the point but gives people the wiggle room they need to be comfortable. Happy to discuss by phone/video if that's useful.

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: Tom Herbert <[email-masked]>
Date: Fri, 30 Nov 2018 14:57:51 -0800

> "SHOULD" means, per RFC 2119, that it is a case in which the author would like to make a blanket statement but recognizes that there may be exceptions that s/he has not considered, and requires an…
Bear in mind that "application developers" is a very broad category of
people that include commercial developers, students, amateurs, etc.
Right now, it's pretty easy to write a meaningful UDP application in
just a couple of hours. I'm not sure it's reasonable for the college
student writing a quick UDP app in their dorm room to have to consider
all the pitfalls of fragmentation and whether they need to increase
their complexity by an order of magnitude to implement fragmentation
in their application. The promise of doing IP fragmentation at the
network layer is that the application developer doesn't need to be
concerned with such things.

> The reason the draft makes a negative statement about fragmentation/reassembly is that there is a measurable issue with fragmentation/reassembly. To pick a data point, Geoff Huston has been measuri…
My fundamental concern is that the wording in the draft generally
recommends against the use of a standard protocol feature. Not because
the protocol is broken or the feature has been obsoleted, but because
some implementations are non-conformant with the protocol standard.
I'm worried that this sort of recommendation and the rationale for it
will be extrapolated to recommending against use of other standard
features like extension headers (which you referred to above), use of
any protocols other than TCP or UDP, or even the use of IPv6.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Nov 2018 15:16:00 -0800

> On Nov 30, 2018, at 2:57 PM, Tom Herbert <[email-masked]> wrote:
> 
> ...student writing a quick UDP app in their dorm room to have to consider
> concerned with such things

Agreed. That and issues with IP frag are why we’re developing UDP frag. 

Joe