Subject: IPv6 Fragment Retransmission

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 9 Nov 2021 18:42:15 +0000

Hi, there were many questions and concerns on the chat during my talk at today's
6man session and I think most/all can be directly addressed.

First and foremost - it is a truth that some applications will see greater performance
by sending IP packets larger than the path MTU while invoking IP fragmentation and
reassembly. This has been known since 1986 when Sun Microsystems came out with
their Network File System (NFS) and delivered their code to the DEC Ultrix team where
I was working at the time. This would have been with IPv4, but iperf3 clearly shows
that the same truth holds for IPv6 today.

The Sun/Solaris workstations were well equipped to handle the performance, but
the DEC VAXstations of the time had inadequate network gear and fragment bursts
under the largest NFS/UDP/IP packet size caused the NIC to crash. This was a big
deal; people took note of it and made darned sure to design their future networking
gear to handle fragmentation and reassembly at high data rates. And, what we have
today is a very capable set of devices that would be asked to engage in fragmentation
and reassembly.

That said, fragmentation and reassembly does not belong everywhere - you might
not want it on a high-speed core router or on a low end IoT device. But, by locating
the fragmentation and reassembly functions on key nodes in the network everything
gets more efficient and sees improved performance. I will offer that AERO/OMNI
give full discussion of the role of IP fragmentation/reassembly in a truly advanced
networking solution and I would recommend folks to have a look.

Many of the comments on chat got down in the weeds about security, and they can
be addressed very simply. First and foremost, a node should only accept a fragment
if it has some way of verifying  data origin - but, even that is not 100% necessary
if the reassembly cache is instrumented to quickly flush bogus reassemblies rather
than hold onto all fragments for a full MSL. And, modern reassembly implementations
are designed to do exactly that.

So the whole purpose of data origin authentication is really just to keep garbage out
of the reassembly cache to keep things working smoothly. Again, you can read about
all this in AERO/OMNI but this draft is intended for general IPv6 operations and not
just for AERO/OMNI. So, the next draft version will have a brief and crisp security
considerations section.

I would like to now invite a discussion of any other comments/questions/concerns
anyone might have here on the list. The draft can be found here:

https://datatracker.ietf.org/doc/draft-templin-6man-fragrep/

Thanks - Fred

---

From: Bob Hinden <[email-masked]>
Date: Tue, 9 Nov 2021 11:29:05 -0800

Fred,

> On Nov 9, 2021, at 10:42 AM, Templin (US), Fred L <[email-masked]> wrote:
> 

I am familiar with how NFS used IP fragmentation to get higher performance.   However, toward the end of my stint at Sun ~1995 (now that was a while ago…), we started to run NFS over TCP.   We discovered the performance went up by doing this.

I think newer versions of the NFS standard now work over TCP.

Bob

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 9 Nov 2021 19:49:58 +0000

Bob,

> I am familiar with how NFS used IP fragmentation to get higher performance.   However, toward the end of my stint at Sun ~1995 (now that
> was a while ago…), we started to run NFS over TCP.   We discovered the performance went up by doing this.

I got involved with the Sun NFS starting in 1986 - Sun gave us at DEC their NFS code to merge
into our Ultrix distribution (a 4.2BSD variant). NFS had configurable block sizes and as I recall
there were three of interest: ~1500, ~4KB and ~8KB ( I do not recall the exact sizes). Sun ran
their NFS filesystems at the full 8KB on SparcStations and got good performance. When we
tried on our VAXstations our little DEQNA/DELQA Ethernet NICs could not absorb the fragment
burst and emitted a little "qe reset" diagnostic as a dying gasp before crashing. So, we had to
back out filesystem block size down to 4KB which produced IP fragments that were within the
operating threshold of the NICs but significantly degraded performance.

Newer network gear does not have this problem, but it would have been interesting to check
the performance by boosting the NFS block size all the way to ~64KB as I inferred in my email.

> I think newer versions of the NFS standard now work over TCP.

Yes, that is my understanding as well but we are starting to see newer transports like QUIC
and an impending wide-scale emergence of IPv6 tunnels - both of which do not use TCP.
And, the performance gain for using the largest possible IPv6 packet size that can incur
fragmentation/reassembly is very significant.

Thanks - Fred

> -----Original Message-----
> From: Bob Hinden [mailto:[email-masked]]
> Sent: Tuesday, November 09, 2021 11:29 AM
> To: Templin (US), Fred L <[email-masked]>
> Cc: Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
> Subject: Re: IPv6 Fragment Retransmission
> 
> Fred,
> 
> I am familiar with how NFS used IP fragmentation to get higher performance.   However, toward the end of my stint at Sun ~1995 (now that
> was a while ago…), we started to run NFS over TCP.   We discovered the performance went up by doing this.
>

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Wed, 10 Nov 2021 13:25:52 +0100

Hi Fred,

> https://datatracker.ietf.org/doc/draft-templin-6man-fragrep/

Unfortunately, I had no time to fully read your draft yet, but saw your 
presentation yesterday. A few observations from my side:

1) fragmentation could be a "packet loss multiplier"
    (see the "Fragmentation Considered Harmful" paper from
     Kent and Mogul, https://dl.acm.org/doi/10.1145/205447.205456)
    and that's what you'd like to address I guess.

2) there are good arguments for application level framing
    (https://dl.acm.org/doi/10.1145/99517.99553), i.e.,
    the application knows best how to split data into
    logical data units (LDU) and how to react to a
    missing LDU.

3) fragmentation may cause micro-bursts in switches and
    routers, because fragments are sent back-to-back unless paced,
    causing higher buffer utilization if not packet loss.
    So my guess is that fragment loss will be highly correlated
    for tail-drop buffers and that then multiple segments need
    to be retransmitted.

4) the added complexity at the sender side by requiring a
    buffer for sent packets/fragments is only a start. For how long
    should these be kept (i.e., suitable timeout values or
    just until overwritten in a circular buffer)?
    Usually, you also need timers for ARQ at the sender side.
    However, in your proposal it seems that the sender only
    reacts to a NACK from the receiver side, which is different
    from typical ARQ, where the sender automatically
    retransmits non-acknowledged packets after a timeout.
    It should be explicitly mentioned if the latter is not the
    case in your proposal. What happens if the FRAGREP
    is lost? I guess nothing, since you don't want a
    reliable service.

So all in all, I guess that (modern) transport protocols
probably do a much better and more complete job
(e.g., what about flow control problems and additional
latencies by queuing) for all of the implied related
problems and that a closer analysis where the supposed
speedup comes from would be helpful.

Regards,
  Roland