Subject: Non-Last Small IPv6 Fragments

From: Timothy Winters <[email-masked]>
Date: Thu, 10 Jan 2019 10:33:20 -0500

We have encountered a potential Interoperability issue at the UNH-IOL while
running some testing.  The issue is around fragments.

The Linux Kernel updated based on the following CVE:

https://nvd.nist.gov/vuln/detail/CVE-2018-5391.

The fix was to reject IPv6 fragments less than 1280 that aren't last
fragment.  Section 4.5 of RFC 8200 allows for sending any fragment for
fragments as long they add up to the original packet.  This means that an
implementation that generates a non-last fragments with a size then 1280,
will be dropped by the updated kernel.

I'm willing to write a draft about the expected behavior, but before I do
that I wanted to get the working group feedback on if we think an
implementation should drop non-last fragments less then 1280.

~Tim

---

From: Bob Hinden <[email-masked]>
Date: Thu, 10 Jan 2019 09:47:37 -0800

Tim,

> On Jan 10, 2019, at 7:33 AM, Timothy Winters <[email-masked]> wrote:
> 
> We have encountered a potential Interoperability issue at the UNH-IOL while running some testing.  The issue is around fragments.
> 

I read some of the reports on the link, but am still not clear what the underlying problem is.   Why does Linux have a problem with receving intermediate fragments less than 1280?

Bob

> 
> The fix was to reject IPv6 fragments less than 1280 that aren't last fragment.  Section 4.5 of RFC 8200 allows for sending any fragment for fragments as long they add up to the original packet.  This means that an implementation that generates a non-last fragments with a size then 1280, will be dropped by the updated kernel.
> 
> I'm willing to write a draft about the expected behavior, but before I do that I wanted to get the working group feedback on if we think an implementation should drop non-last fragments less then 1280.
> 
> ~Tim
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: 神明達哉 <[email-masked]>
Date: Thu, 10 Jan 2019 10:55:09 -0800

At Thu, 10 Jan 2019 10:33:20 -0500,

> implementation should drop non-last fragments less then 1280.

I don't see a reason why an implementation can drop such fragments.
As you pointed out, the standard protocol doesn't prohibit it.  I
would even consider it "reasonable" (not just "allowed by the
standard") if an implementation divides a 1300-octet packet into two
650-octet fragments (ignoring common header overhead for brevity)
instead of 1280 + 20, even if there's currently no implementation that
actually does that.

Looking at the description about CVE-2018-5391 in
https://lists.debian.org/debian-lts-announce/2018/08/msg00014.html
my guess is that "the attack" is to send many small fragments (perhaps
also in random order?) in an attempt of increasing the management
overhead to the level of denial of service.  If I'm correct, I see
that such a possibility of attack allows an implementation to
introduce some implementation-specific limitation, but I think
dropping non-last fragments smaller than 1280 octets goes too far for
that purpose.

--
JINMEI, Tatuya

---

From: Tom Herbert <[email-masked]>
Date: Thu, 10 Jan 2019 11:17:46 -0800

On Thu, Jan 10, 2019 at 10:55 AM 神明達哉 <[email-masked]> wrote:
> actually does that..
The patch for this in Linux was submitted Aug. 3, 2018 ("ipv6: defrag:
drop non-last frags smaller than min mtu"). The commit log or
discussiosn doesn't mention the security advisory and seems to
indicate this is more of an optimization with the assumption that
intermediate fragments must be at least 1280. It might just be a bug
that is easily reverted. I can take it up on the netdev list.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Thu, 10 Jan 2019 19:32:35 +0000

> I read some of the reports on the link, but am still not clear what the
> underlying problem is.   Why does Linux have a problem with receving
> intermediate fragments less than 1280?
> 

Hi Bob,

Might we be defending against an attack in which a packet contains:

- An IPv6 header (40 bytes)
- A Fragment Header (8 bytes)
- A TCP header (20 bytes)
- TCP Payload (1200 bytes)

This packet doesn't need to be fragmented at all because the total length is only 1268 bytes. However, a mischievous source node divides the packet into 1200 fragments. The first fragment contains an IPv6 header, a fragment header, the TCP header, and one byte of the TCP payload. Each subsequent fragment contains the IPv6 header, a fragment header, and one byte of TCP payload.

Are reassembly algorithms clever enough to protect against such attacks? If so, I don't see the problem either. But if not, we may have a problem.

                                                                                           Ron

---

From: Erik Kline <[email-masked]>
Date: Thu, 10 Jan 2019 11:42:11 -0800

On Thu, 10 Jan 2019 at 11:32, Ron Bonica <[email-masked]> wrote:

> This packet doesn't need to be fragmented at all because the total length
> is only 1268 bytes. However, a mischievous source node divides the packet
> into 1200 fragments. The first fragment contains an IPv6 header, a fragment
> header, the TCP header, and one byte of the TCP payload. Each subsequent
> fragment contains the IPv6 header, a fragment header, and one byte of TCP
> payload.
>
> Are reassembly algorithms clever enough to protect against such attacks?
> If so, I don't see the problem either. But if not, we may have a problem.
>

I'm recently familiar with an IPv6 fragment reassembly implementation, as
it turns out.  The core implementation uses/makes liberal reference to:

    https://tools.ietf.org/html/rfc815

It works generally in terms of managing a hole descriptor list.  It would
successfully reassemble the sequence of packets you describe.  Whether
that's an "attack" or not, I don't really see it.  With local policy caps
on the lifetime of unreassembled fragment bits and so on, it seems possible
to limit and manage the total resources allocated to reassembly.

---

From: Ron Bonica <[email-masked]>
Date: Thu, 10 Jan 2019 19:52:34 +0000

Erik,

Thanks for the response.

So, I understand that if I were to launch a stream of such packets at a target:

  *   The target might drop many of the attack packets (but that’s ok)
  *   The target would still process non-fragmented packets at a reasonable speed
  *   The target would still be able to reassemble fragments that are from other sources and not part of the attack

If this is the case, we have nothing to worry about.

                                                          Ron

From: Erik Kline <[email-masked]>
Sent: Thursday, January 10, 2019 2:42 PM
To: Ron Bonica <[email-masked]>
Cc: Bob Hinden <[email-masked]>; Timothy Winters <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Non-Last Small IPv6 Fragments

Hi Bob,

I'm recently familiar with an IPv6 fragment reassembly implementation, as it turns out.  The core implementation uses/makes liberal reference to:

    https://tools.ietf.org/html/rfc815<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc815&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-AWF2EfpHcAwrDThKP8&m=-dVqPKvvhh60cA1adnmR9mFsqrX0ADki0K4BlrOQqGc&s=6m7aXa5azbXR0bSACw5GJgOfJx06tbs_1LydP-h2aqs&e=>

It works generally in terms of managing a hole descriptor list.  It would successfully reassemble the sequence of packets you describe.  Whether that's an "attack" or not, I don't really see it.  With local policy caps on the lifetime of unreassembled fragment bits and so on, it seems possible to limit and manage the total resources allocated to reassembly.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 11 Jan 2019 09:01:18 +1300

> If this is the case, we have nothing to worry about.

Well, we have to worry about a broken Linux implementation unless this "fix" is reversed.

(I can see a pragmatic argument for dropping non-final fragments that are really small, which might be diagnostic of an attack. But then you have to define "really small".)

   Brian

---

From: Ron Bonica <[email-masked]>
Date: Thu, 10 Jan 2019 20:32:47 +0000

Brian,

The following are two more fundamental questions:

- Would the attack be effective?
- If the attack is effective, is there a better way to mitigate it (e.g., by limiting the number of fragments that a receiving node is willing to reassemble for a single packet)?

                                                                        Ron

> -----Original Message-----
> From: Brian E Carpenter <[email-masked]>
> Sent: Thursday, January 10, 2019 3:01 PM
> To: Ron Bonica <[email-masked]>; [email-masked]
> Cc: IPv6 List <[email-masked]>; Bob Hinden <[email-masked]>
> (I can see a pragmatic argument for dropping non-final fragments that are
> really small, which might be diagnostic of an attack. But then you have to
> define "really small".)
> 
>    Brian
> 
> >
> >                                                           Ron
> >
> >
> > Cc: Bob Hinden <[email-masked]>; Timothy Winters
> <[email-masked]>; IPv6 List <[email-masked]>
> > On Thu, 10 Jan 2019 at 11:32, Ron Bonica
> <[email-masked]<mailto:[email-masked]>> wrote:
> > This packet doesn't need to be fragmented at all because the total length is
> only 1268 bytes. However, a mischievous source node divides the packet into
> 1200 fragments. The first fragment contains an IPv6 header, a fragment
> > Are reassembly algorithms clever enough to protect against such attacks? If
> so, I don't see the problem either. But if not, we may have a problem.
> >
> > I'm recently familiar with an IPv6 fragment reassembly implementation, as it
> turns out.  The core implementation uses/makes liberal reference to:
> >
> >     https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_rfc815&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBX
> eMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=sK5K5wuiRYsxdqBoO01uXstXrB6pcOH7vIaVlqPk
> bw8&s=wRD2EDX32nGJdkVKcg_MlfkjpiweHbKU_7X3BJXHQks&e=<https://u
> rldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_rfc815&d=DwMFaQ&c=HAkYuh63rsuhr6Scbfh0UjB
> XeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=-
> dVqPKvvhh60cA1adnmR9mFsqrX0ADki0K4BlrOQqGc&s=6m7aXa5azbXR0bS
> ACw5GJgOfJx06tbs_1LydP-h2aqs&e=>
> >
> successfully reassemble the sequence of packets you describe.  Whether that's
> an "attack" or not, I don't really see it.  With local policy caps on the lifetime of
> unreassembled fragment bits and so on, it seems possible to limit and manage
> the total resources allocated to reassembly.
> >
> >
> > Administrative Requests:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_ipv6&d=DwIFaQ&c=HAkYuh63rsuhr6S
> cbfh0UjBXeMK-ndb3voDTXcWzoCI&r=Fch9FQ82sir-BoLx84hKuKwl-
> AWF2EfpHcAwrDThKP8&m=sK5K5wuiRYsxdqBoO01uXstXrB6pcOH7vIaVlqPk
> bw8&s=aU3laJhpXnj8ataCCjgCdmeHhXP6jyerRBW6vUlk-SI&e=

---

From: Tom Herbert <[email-masked]>
Date: Thu, 10 Jan 2019 13:19:03 -0800

> - Would the attack be effective?
Hi Ron,

I imagine it could be. In Linux for instance, the reassembly queue
requires an skbuf structure for each fragment. That means one packet
in reassembly could tie up thousands of such structures in memory in
your proposed attack.

> - If the attack is effective, is there a better way to mitigate it (e.g., by limiting the number of fragments that a receiving node is willing to reassemble for a single packet)?

That might be reasonable, however requiring intermediate fragments to
be at least 1280 MTU in IPv6 also solves that without needing to
define some aritrary new limit. I'd also point out that while Linux
was changed to enforce a minimum size on intermediate fragments for
IPv6, doing the same thing in IPv4 was rejected since intermediate
nodes can fragment.

Tom

---

From: Mark Andrews <[email-masked]>
Date: Fri, 11 Jan 2019 08:24:34 +1100

A resonable limit is 1280/2.  This allow a 1281 packet to be split into
2 even sizes packets.  Even sized packets have less reordering issues
than all PMTU but the last.

> On 11 Jan 2019, at 8:19 am, Tom Herbert <[email-masked]> wrote:
> 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Simon Hobson <[email-masked]>
Date: Thu, 10 Jan 2019 21:39:04 +0000

Tom Herbert <[email-masked]> wrote:

> define some aritrary new limit.

In setting a minimum payload size for a fragment, is there a risk of conflicting with some known, or future unknown, link type with a small MTU ?

---

From: Erik Kline <[email-masked]>
Date: Thu, 10 Jan 2019 13:44:39 -0800

> In setting a minimum payload size for a fragment, is there a risk of conflicting with some known, or future unknown, link type with a small MTU ?

Technically that link wouldn't meet the 1280 min mtu requirement for
IPv6, right?

Even with 1280 byte min fragment sizes, a reassembly engine still
probably needs to place some limits in practice on the resources it
will devote to reassembly.  An attacker and sent 1280-sized fragments
for an endless series of 65535-sized datagrams...

---

From: David Farmer <[email-masked]>
Date: Thu, 10 Jan 2019 15:51:08 -0600

So basically you are adding a limitation of "but not more fragments than
the number of Minimum MTU fragments necessary to send the original packet"
to the current rule allowing the "for sending any fragment for fragments as
long they add up to the original packet."

I was thinking along these lines too, and it seems prudent to limit the
number of fragments that can be created.

Or, maybe "but not more than one additional fragment than the number of
Minimum MTU fragments necessary to send the original packet"

Which would be essentially 1280/3, this would allow some flexibility
without leaving it completely open-ended.

On Thu, Jan 10, 2019 at 3:24 PM Mark Andrews <[email-masked]> wrote:

> >> - If the attack is effective, is there a better way to mitigate it
> (e.g., by limiting the number of fragments that a receiving node is willing
> to reassemble for a single packet)?
> >
> >>>>  *   The target would still process non-fragmented packets at a
> reasonable
> >>> speed
> >>>>  *   The target would still be able to reassemble fragments that are
> from
> >>> other sources and not part of the attack
> >>>>
> >>> Well, we have to worry about a broken Linux implementation unless this
> "fix"
> >>> is reversed.
> >>>
> that's

-- 
===============================================
David Farmer               Email:[email-masked]
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

---

From: Mark Andrews <[email-masked]>
Date: Fri, 11 Jan 2019 08:58:41 +1100

> On 11 Jan 2019, at 8:44 am, Erik Kline <[email-masked]> wrote:
> 
> for an endless series of 65535-sized datagrams…

approximate packet counts.

65535/1280
51

65535/1500
43

or even

65535/640
102

but a IPv6 node doesn’t have to support reassembly up to 65535

If you are worried about too many fragments, count and limit the fragments
not the fragment size.  If you get too many fragments for a packet just
discard them all.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 10 Jan 2019 14:19:43 -0800

> approximate packet counts.

Mark,

I think that might lead to some odd scenarios. For instance, in Ron's
tiny fragment attack, the receiver would start reassembling fragments
up to the limit. In some sense the attack should be obvious, no
rationale implementation should be sending tiny fragments in this
manner. But once the limit is hit, then what happens? Unless the
receiver keeps some state about the failed reassembly (which would be
new complexity in the implementation), if would just restart the
reassembly of the packet on the next received fragment when further
efforts to reassmble are obviously futile.

Tom

---

From: Erik Kline <[email-masked]>
Date: Thu, 10 Jan 2019 14:20:03 -0800

> approximate packet counts.

I don't yet see the need to limit the fragment size (but perhaps some
persuasive argument will emerge).

What I was trying to describe was: take all of those numbers you
reference per 32bit fragment ID.

An attacker could forge the source address and send 50 of the 51
packets for each of 4 billion fragment IDs.  The missing fragment
could even be the one with the L4 header too (to try to circumvent L4
ACLs that might cause the rest of the fragments to be dropped).

I think reassembly implementations may well want to manage several
resources, including: the total number of fragments, the lifetime of
the fragments, and possibly the total reassembled size (based on
expected legitimate traffic patterns).

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 11 Jan 2019 00:30:45 +0100

On Jan 10, 2019, at 19:55, 神明達哉 <[email-masked]> wrote:
> I
> instead of 1280 + 20,

Actually, something more on the side of 20 + 1280 might also be a useful choice.

https://tools.ietf.org/html/draft-bormann-lwig-6lowpan-virtual-reassembly-00#section-4

That said, that example is more useful for adaptation layer fragmentation as in 6LoWPAN, and I fully sympathize with an IP implementation that might want to reject silly fragments.

Grüße, Carsten

---

From: 神明達哉 <[email-masked]>
Date: Thu, 10 Jan 2019 16:56:27 -0800

At Fri, 11 Jan 2019 00:30:45 +0100,

> Actually, something more on the side of 20 + 1280 might also be a useful
choice.

(I don't think you misunderstood my point but just in case) that's not
the point.  My point was that it's not surprising (to me at least) if
some implementation prefers the "650+650" fragmentation and so it
would be too "illiberal" to drop such fragments unconditionally.  Of
course "1280+20" can also make sense (if nothing else, it would be
simpler to implement), and in fact I believe most implementations take
that approach.

--
JINMEI, Tatuya

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 11 Jan 2019 13:56:42 +1300

> - Would the attack be effective?

Like all resource-consumption attacks, it all depends. Yes, one-byte fragments would probably be an issue but would 256-byte fragments? I just don't know.

Once in my misspent youth I wrote code for fragmentation/reassembly of CLNP packets over a PMTU of 64 bytes. I did not enjoy writing the code, but I never saw any particular performance issues, even on a microsecond-per-instruction minicomputer. However, I think there's no generic answer to your question.

> - If the attack is effective, is there a better way to mitigate it (e.g., by limiting the number of fragments that a receiving node is willing to reassemble for a single packet)?

Yes, I think quite a few pragmatic suggestions have been made in this thread. But given that this is 6man, maybe we should be asking whether this is a gap in the IPv6 specification.

     Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 11 Jan 2019 16:02:48 +1300

If an RFC7195 translator receives an IPv4 fragment, it will translate it into an IPv6 fragment. So if the IPv4 fragment is "small" by some definition, so is the IPv6 fragment.

I think that's a legitimate route to non-last fragments shorter than 1280, regardless of what is considered normal for IPv6.

Regards
   Brian

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 01:14:48 -0300

> IPv6, doing the same thing in IPv4 was rejected since intermediate

Sorry, I don't see why this helps. You drop my less-than-1280 fragments?
Ok, I shoot you 1280-byte fragments... so?

Thanks,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 01:18:35 -0300

On 10/1/19 18:58, Mark Andrews wrote:
[...]
> not the fragment size.  If you get too many fragments for a packet just

+1

And also count the total number of frags in the queue. (An attacker
wishing to perform this sort of attack will typically spoof the Source
Addr from at least the /64 she/he has available)

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 01:22:31 -0300

> ACLs that might cause the rest of the fragments to be dropped).

Indeed.

>From the frag6(1) (htps://github.con/fgont/ipv6toolkit.git) man page:

  # frag6 -s ::/0 --flood-frags 100 -l -z 5 -d fc00:1::1 -v
    Send  100  fragments  (every  5 seconds) to the host fc00:1::1,
    using a forged IPv6 Source Address from the  prefix  ::/0.  The
    aforementioned fragments  should  have  an  offset  of  0, and
    the M bit set (i.e., be first-fragments). Be verbose.

Thanks,

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 11 Jan 2019 08:24:42 +0100 (CET)

> than all PMTU but the last.

I have seen implementations in the wild that do just this, fragments a 
packet into two evenly sized packets.

So while I am sympathetic to disallowing very small non-last fragments, I 
think requiring all non-last fragments to be 1280 is too strict. 1280/2 or 
something similar to that sounds like a reasonable tradeoff.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Fri, 11 Jan 2019 19:07:09 +1100

Yep, you do something like this.

#include <stdio.h>
#define HEADER_SIZE 40
#define FRAGMENT_SIZE 40
#define AVAILABLE (1280 - HEADER_SIZE - FRAGMENT_SIZE)

int
main() {

        for (int i = 1200; i <= 0xffff; i++) {
                int payload = i - HEADER_SIZE;
                int fragments = (payload + AVAILABLE - 1) / AVAILABLE;
                int fragpayload = (payload + fragments - 1) / fragments;
		/* Round up to chunk size */
                fragpayload = ((fragpayload + 7) / 8) * 8;
                printf("%d %d:", i, fragpayload);
                while (payload > fragpayload) {
                        printf(" %d", fragpayload);
                        payload -= fragpayload;
                }
                printf(" %d\n", payload);
        }
};

> On 11 Jan 2019, at 6:24 pm, Mikael Abrahamsson <[email-masked]> wrote:
> 
>> A resonable limit is 1280/2.  This allow a 1281 packet to be split into 2 even sizes packets.  Even sized packets have less reordering issues than all PMTU but the last.
> 
> I have seen implementations in the wild that do just this, fragments a packet into two evenly sized packets.
> 
> So while I am sympathetic to disallowing very small non-last fragments, I think requiring all non-last fragments to be 1280 is too strict. 1280/2 or something similar to that sounds like a reasonable tradeoff.
> 
> --

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Fri, 11 Jan 2019 13:31:46 +0000

On 11 Jan 2019, at 3:02, Brian E Carpenter wrote:

Hi,

> If an RFC7195 translator receives an IPv4 fragment, it will translate 
> it into an IPv6 fragment. So if the IPv4 fragment is "small" by some 
> definition, so is the IPv6 fragment.
>
> I think that's a legitimate route to non-last fragments shorter than 
> 1280, regardless of what is considered normal for IPv6.

I think this topic has been re-hashed in the past often enough.

I want us to stop thinking that there’s anything but IPv6 if we ever 
want fragments to be “clean” and sorted and usable and working.  
It’s a total waste of time to sort this out for a transition 
technology as it’d be a major change to IPv6 implementations and 
involving end nodes and translator nodes (*) and by that time that would 
have happened IPv4 is no longer relevant.

Can we please start looking ahead to the decades of IPv6-only before we 
need to extend “real-time” communications to Mars and outside of our 
galaxy? (+)

/bz

(*) If you really wanted to you’d need a way to signal the min-MTU 
violation from the translator exception case to the end not and you’d 
probably need some kind of extension header for that.  Sadly this 
wasn’t done 20 years ago and neither for 8200.

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Fri, 11 Jan 2019 13:43:03 +0000

> instead of 1280 + 20, even if there's currently no implementation that

Well that won’t work as 650 octets is not a multiple of 8 and hence 
the first fragment would be dropped.  You cannot just do size/2; that 
only works for every nth packet size (n to be determined by the reader 
;-) ).

/bz

---

From: Timothy Winters <[email-masked]>
Date: Fri, 11 Jan 2019 08:44:47 -0500

As Tom mentioned he has started a discussion on the Linux Network list
about FragmentSmack.  I'll report back if they have a more elegant solution
to the problem.

This is the best write-up I've found discussing the issue.

https://access.redhat.com/articles/3553061

Based on the general comments on list and reading about this attack I'm
going to create a draft to give some guidance to implementors for this
attack.

~Tim

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 11:20:45 -0300

> On 11 Jan 2019, at 3:02, Brian E Carpenter wrote:
[....]
> probably need some kind of extension header for that.  Sadly this wasn’t
> done 20 years ago and neither for 8200.

Then you'd need routers on the public Internet to pass packets with EHs,
such that the signaling was sort of "reliable"...

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 11:26:38 -0300

On 11/1/19 10:44, Timothy Winters wrote:
> about FragmentSmack.  I'll report back if they have a more elegant
> solution to the problem.
> 

Guidance: whenever there is some data structure that could potentially
grow without bounds, or a resource that consumed until it is exhausted,
and that would affect the system or the communications in which the
system is directly or indirectly involved, enforce limits and try to
avoid fair use of such resource. Otherwise, we will keep disvovering the
same kind of vulnerabiities we have discovered during the last 30 years
or so.

:-)

IP fragmention is no special.

---

From: Ole Troan <[email-masked]>
Date: Fri, 11 Jan 2019 15:40:11 +0100

> IP fragmention is no special.

That’s not quite true. IP reassembly is made an easy target because it forces the receiver to complete reassembly before it can determine that the complete packet was unwanted.
Isn’t there a reason we see many IP reassembly attacks and not TCP segmentation attacks?
(And that we encourage deprecation of fragmentation at the network layer).

Cheers,
Ole

---

From: Timothy Winters <[email-masked]>
Date: Fri, 11 Jan 2019 10:10:44 -0500

On Fri, Jan 11, 2019 at 9:40 AM Ole Troan <[email-masked]> wrote:

> That’s not quite true. IP reassembly is made an easy target because it
> forces the receiver to complete reassembly before it can determine that the
> complete packet was unwanted.

Ultimately, that is the issue in this case.    I think the safest thing to
help against an attack like this, is to update the size to something like
640 to allow implementations to not waste CPU time on processing super
small fragments.

~Tim

---

From: Tom Herbert <[email-masked]>
Date: Fri, 11 Jan 2019 07:37:00 -0800

> > done 20 years ago and neither for 8200.

Take a look at draft-hinden-6man-mtu-option. This is a Hop-by-Hop
option and is facilitated by the provision in RFC8200 that relaxes the
requirements that every intermediate node process Hop-by-Hop options.

> such that the signaling was sort of "reliable"...

I agree, we need the broken routers to start complying with Internet standards.

Tom

---

From: Simon Hobson <[email-masked]>
Date: Fri, 11 Jan 2019 18:38:13 +0000

Fernando Gont <[email-masked]> wrote:

> ... enforce limits ...

The guidance also needs to suggest sensible limits - otherwise what one implementer (eg the Linux stack that discards any small fragment that isn't the last) may not interoperate with another (eg make the last two fragments approximately equal in size). In a significant part, that seems to have been what this thread has been about - what is reasonable guidance for this.

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 16:31:45 -0300

On 11/1/19 11:40, Ole Troan wrote:
[....]
> Isn’t there a reason we see many IP reassembly attacks and not TCP segmentation attacks?

TCP is harder to understand, probably, and not many publicly available
tools out there (I haven't seen anything other than the tcp6 I
crafted... and it's v6-only). And for the script-kidde, his fw is
already shooting himself in his own foot.

There's lot's of things you can do with TCP, such as flooding nodes with
connections -- even ones for which there's no controlling process
anymore, etc. This warrants some testing which I never did
comprehensively, indeed.

> (And that we encourage deprecation of fragmentation at the network layer).

I do agree that network layer is bad idea, and also that it is even
harder to defend against frag-based attacks (in a sense since you cannot
even, say, limit #fragments/prefix, since spoofing is trivial).
Although I don't think this sort of mitigation is actually implemented
in TCP, either, particularly for the v6 cases, where you have a
virtually unlimitted number of addresses to spoof from.

Still, my comment above still holds: Connections will be degreded... but
at least you won't get DoS in the sense of ping o' death -- which kind
of seems to be the case here, if I understand correctly.

I've reproduced DoS-type of attacks for virtually every IPv6 data
structure we have: routes, configured addresses, neighbor cache, etc.
Things have improved a bit lately, though.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Fri, 11 Jan 2019 16:49:36 -0300

On 11/1/19 15:38, Simon Hobson wrote:
> The guidance also needs to suggest sensible limits - otherwise what one implementer (eg the Linux stack that discards any small fragment that isn't the last) 

IMO, this is bad behavior advice. As an attacker, I'd fire 1280-sized
frags, and your check hasn't achieved anything.

>may not interoperate with another (eg make the last two fragments approximately equal in size). In a significant part, that seems to 
> have been what this thread has been about - what is reasonable
guidance for this.

In general, what you do is AIMD (additive increase, multiplicative
decrease). The numbers limits are a few thousand packets. Which allow
fragments to work for some cases - few connections, rather low
throughput, but will not allow system breakage when the attacker wants
to play.

Dropping small frags as in Linux is bad, because you hurt
interoperability without any significant/real gain.

OTOH, limiting the number of frags is good, because limits kick in at a
point in which "you need to do something, because otherwise - DoS".

Obviously, you cannot expect big pipes with fragmentation to work. Given
high bandwith, multiply 60s * throughput, and you'll need a lot of
memory to hold the frags.

Adapting the Frag reassembly time out can also help -- there's not much
of a point to keep a frag in memory for 60s, particularly when under
heavly load. -- well before that TCP timers will fire, or the frag si
not useful anymore, anyway.

---

From: David Farmer <[email-masked]>
Date: Fri, 11 Jan 2019 15:13:06 -0600

On Fri, Jan 11, 2019 at 1:58 PM Fernando Gont <[email-masked]> wrote:

> > The guidance also needs to suggest sensible limits - otherwise what one
> implementer (eg the Linux stack that discards any small fragment that isn't
> the last)
>
> >may not interoperate with another (eg make the last two fragments
> approximately equal in size). In a significant part, that seems to

I just had a thought for the guidance;

1. System-wide limit the number of fragments based on the overall system
resources available.
2. As you near said limit, maybe 75% of the limit, drop outright, or at a
high probability, non-final fragments smaller than (1280/2) 640 bytes.

The reasonable fragmentation algorithms I can think of do not generate
non-final fragments that are less than 640 bytes. Nevertheless, such
fragments SHOULD NOT be dropped except for when the system is under
stress.

What do others think?

---

From: Tom Herbert <[email-masked]>
Date: Fri, 11 Jan 2019 13:26:49 -0800

On Fri, Jan 11, 2019 at 1:13 PM David Farmer <[email-masked]> wrote:
> 1. System-wide limit the number of fragments based on the overall system resources available.
> 2. As you near said limit, maybe 75% of the limit, drop outright, or at a high probability, non-final fragments smaller than (1280/2) 640 bytes.
>
> The reasonable fragmentation algorithms I can think of do not generate non-final fragments that are less than 640 bytes. Nevertheless, such fragments SHOULD NOT be dropped except for when the system is under stress.
>
> What do others think?

I don't see any practical purpose of sending tiny fragments with eight
byte payloads in non-first fragments, these can only happens under DOS
attack or synthetic testing. So I do believe it's prudent to have some
guidance on setting limits for minimal sizes of non-first fragments.
The limit should probably be configurable with some recommended
default value. I believe the 1280 limit in Linux is too high, but
there are some compelling arguments for 640.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Fri, 11 Jan 2019 22:15:28 +0000

David,

That sounds reasonable.

                    Ron

What do others think?

--

---

From: David Farmer <[email-masked]>
Date: Fri, 11 Jan 2019 16:33:00 -0600

On Fri, Jan 11, 2019 at 3:27 PM Tom Herbert <[email-masked]> wrote:

> >> > The guidance also needs to suggest sensible limits - otherwise what
> one implementer (eg the Linux stack that discards any small fragment that
> isn't the last)
> >>
> > 2. As you near said limit, maybe 75% of the limit, drop outright, or at
> a high probability, non-final fragments smaller than (1280/2) 640 bytes.
> >
> fragments SHOULD NOT be dropped except for when the system is under stress.
> >
> > What do others think?
>

Why non-first fragments? Did you mean non-final? Maybe both first and last
should be exempted?

I mostly agree with you, but on the other hand is there any compelling
reason to potentially break anything if the host isn't under stress?

Further, if you only limit tiny fragment, there is still a problem, you
just have to use packets bigger that the limit, what ever it is.  A few
tiny fragments aren't a problem, only lots of them are.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 11 Jan 2019 14:47:44 -0800

On Fri, Jan 11, 2019 at 2:33 PM David Farmer <[email-masked]> wrote:
> Why non-first fragments? Did you mean non-final? Maybe both first and last should be exempted?
>
Yes.

> I mostly agree with you, but on the other hand is there any compelling reason to potentially break anything if the host isn't under stress?

That depends on what resource is "under stress" in the host.
Reassembly is an expensive packet processing operation that requires
one lookup to find the state for the ID being reassembled, and then a
second lookup to find the position of the fragment in the queue.
Because fragmentation is considered to be rare, there hasn't been a
lot of work to optimize these operations or rely on hardware offload
(there is UDP fragmentation offload, but that is not ubiquitous). I
would be concerned that a well crafted attack with small packets could
be bad for low powered devices. A limit on the fragment size not only
caps the amount of memory burned, but also caps the packet rate and
hence CPU needed for reassmbly.

Tom

>
> Further, if you only limit tiny fragment, there is still a problem, you just have to use packets bigger that the limit, what ever it is.  A few tiny fragments aren't a problem, only lots of them are.
>
> --

---

From: Bob Hinden <[email-masked]>
Date: Fri, 11 Jan 2019 14:57:09 -0800

Tim,

> On Jan 11, 2019, at 5:44 AM, Timothy Winters <[email-masked]> wrote:
> 
> As Tom mentioned he has started a discussion on the Linux Network list about FragmentSmack.  I'll report back if they have a more elegant solution to the problem.
> 
> Based on the general comments on list and reading about this attack I'm going to create a draft to give some guidance to implementors for this attack.

Reading the article, the potential attack applies to IPv4 and IPv6.  The article doesn’t say anything about < 1280 sized IPv6 fragments, just:

“….change the default 4MB and 3MB values of net.ipv4.ipfrag_high_threshand net.ipv4.ipfrag_low_thresh (and their IPv6 counterparts net.ipv6.ipfrag_high_thresh and net.ipv6.ipfrag_low_thresh) sysctl parameters to 256 kB and 192 kB (respectively) or below.”

Still confused as to the issue.

Thanks,
Bob

> 
> ~Tim
> 
> On Fri, Jan 11, 2019 at 8:31 AM Bjoern A. Zeeb <[email-masked]> wrote:

---

From: David Farmer <[email-masked]>
Date: Fri, 11 Jan 2019 18:10:57 -0600

On Fri, Jan 11, 2019 at 4:47 PM Tom Herbert <[email-masked]> wrote:

> >> >> > The guidance also needs to suggest sensible limits - otherwise
> what one implementer (eg the Linux stack that discards any small fragment
> that isn't the last)
> >> >>
> >> > 2. As you near said limit, maybe 75% of the limit, drop outright, or
> at a high probability, non-final fragments smaller than (1280/2) 640 bytes.
> >> >
> >> > The reasonable fragmentation algorithms I can think of do not
> generate non-final fragments that are less than 640 bytes. Nevertheless,
> such fragments SHOULD NOT be dropped except for when the system is under
> stress.
> >> >
> >> > What do others think?
> >>

I'd be fine with adding something like the following;

Destination nodes SHOULD NOT drop non-final fragments shorter than 640
bytes unless they are under stress. However, highly constrained devices,
such as devices with minimumal memory or CPU, or very low-power devices,
could be considered as always under stress, and therefore it might be
prudent for such devices to always drop non-final fragments shorter than
640 bytes.

Would that cover your issues?

---

From: Tom Herbert <[email-masked]>
Date: Fri, 11 Jan 2019 16:30:44 -0800

On Fri, Jan 11, 2019 at 4:11 PM David Farmer <[email-masked]> wrote:
> Destination nodes SHOULD NOT drop non-final fragments shorter than 640 bytes unless they are under stress. However, highly constrained devices, such as devices with minimal memory or CPU, or very low-power devices, could be considered as always under stress, and therefore it might be prudent for such devices to always drop non-final fragments shorter than 640 bytes.
>

David,

It make sense from a receiver point of view, but not the transmitter
who wouldn't know that its peer is a such a device. There should be
guidance to senders to the effect that if you want to maximize the
chances your fragments are reassembled then make them at least this
size. From a pragmatic point a view, given that we've identified a
potential DOS attack, I think it's likely that we'll end up setting
the default limit, in Linux at least, probably to 640 anyway unless
there is good argument otherwise that doing so would break operability
(with real implementations and not just in synthetic conformance
tests).

Tom

---

From: Erik Kline <[email-masked]>
Date: Fri, 11 Jan 2019 17:02:37 -0800

If concrete guidance is to be given I think it would be good to see if
it could be based on some research into what's actually effective at
mitigating various attacks/scenarios.

Is anyone aware of presentations or papers about work in this area?

---

From: Fernando Gont <[email-masked]>
Date: Sat, 12 Jan 2019 00:26:13 -0300

[....]
>> Why non-first fragments? Did you mean non-final? Maybe both first and last should be exempted?

So why should I, as an attacker, send you first fragments, then? If for
some reason I just want to send you, I'd send fragments such that
Offset!=0 and M=0, with small size, WHat have you gained with the check
you propose?

---

From: Fernando Gont <[email-masked]>
Date: Sat, 12 Jan 2019 00:32:36 -0300

> Is anyone aware of presentations or papers about work in this area?

See Section 4.1.1.1., 4.1.2. of RFC6274. If anything, just s/68/1280/,
and it applies to IPv6 as well.

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Fri, 11 Jan 2019 19:47:18 -0800

> Offset!=0 and M=0, with small size, WHat have you gained with the check

That sort of attack would exhaust the limit on how many packets are
allowed to be in reassembly at which point such packets would start to
be dropped. It's also easy enough to prioritize packets for reassmbly
for which the first fragment has already been received when
considering eviction from the reassembly queue.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Sat, 12 Jan 2019 01:37:31 -0300

> considering eviction from the reassembly queue.

I agree with the above (please see the ref I posted to RFC6274, which
basically argues about this). HOowever, it is still not clear to me
what's the problem you are addressing by enforcing a minimum fragment
size. If you allow MIN_MTU/2, then you require first frags to be 640
bytes.. and teh attacker is required to send 640 as opposed to 60 bytes
or  so. Doesn't seem to be much of a difference to me.
And as noted by Bob, I still don't see what's the big deal with the
small fragments.

There are some DoS attacks where you send a last frag and possibly a
first frag, and the implementation allocates a buffer that would fit the
rest of the missing fragments -- obviously exacerbating the problem a
bit. BUt still i this cases, the size of the first frag is not a big deal.

Thanks,

---

From: Christian Huitema <[email-masked]>
Date: Fri, 11 Jan 2019 23:34:36 -1000

> I think that's a legitimate route to non-last fragments shorter than 1280, regardless of what is considered normal for IPv6.

There is that, and there is also a semi-legit other case that involves
middleboxes. Suppose the end system "knows" that the PMTU is 1400 bytes,
and splits the original UDP datagram, insert fragment headers, etc.. The
fragments are 1400 bytes. And then, somewhere along the path, some
helpful middlebox realizes that the PMTU is now 1280 bytes, so splits
each of the 1400 bytes fragments into one 1280 bytes fragment and
another 120 bytes long.

-- Christian Huitema

---

From: Ole Troan <[email-masked]>
Date: Sat, 12 Jan 2019 10:48:57 +0100

> bit. BUt still i this cases, the size of the first frag is not a big deal.

In e.g. VPP limiting fragment size would not help at all.
The scarce resource is number of available buffers. One fragment occupies at least one buffer regardless of how much smaller than 2K (default buffer size) it is.

Mitigations against attack is highly likely to be implementation specific.
As an example see:
https://github.com/FDio/vpp/blob/master/src/vnet/ip/ip6_reassembly.c

Default reassembly timeout is 100ms (constrasting that with the RFC 60s)
Maximum consequtive reassemblies in progress is limited to 1024.
And in other virtual reassembly algorithms we have also limited maximum allowed number of fragments in chain to 5.

Now, if we in the IETF think we can provide some guidance here, I will second Erik’s suggestion of getting more research.

The packet traces I have looked at in the past, the majority of fragmeted packets looked like attacks. UDP port 80, DNS queries to “thisdomainnamewillgiveaverylongresponsepurelyforthepurposeofattack.org”,
didn’t reassemble, 64K size… But it would be interesting to understand fragmentation behaviour in various parts of the network.

- how long are fragment chains (both in packets and in time span)
- ratio of out of order fragments
- which applications use fragments
- ratio of attack traffic using fragments
- average/min/maximum sizes, same for numbers of fragments in chain
- ratio of complete fragment chain following the same path in the network
- ratio of fragments to total traffic

Cheers,
Ole

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sat, 12 Jan 2019 16:21:51 +0000

> - ratio of fragments to total traffic

Ole, I do not think a study of ratios and averages will provide much value.  A general implementation like Linux kernel must (will want to) support all use cases. No regressions!  I've seen several applications, all homegrown so you are unlikely to encounter them for any study, where the use of fragments is an effect of the application's long life and evolution -- it started with reasonably-sized UDP packets but demands on the amount of data in a single datagram grew over the years.

As for the Linux implementation, I think the limit there is not a complete solution but an assertion that non-final fragments shorter than X cannot possibly be legit.  This assertion increases the costs on the attacker.

Ultimately, Fragment Smack and Segment Smack are NOT attacks on memory of the machine but on the bookkeeping Algorithm. For obvious reasons, Linux tries hard to avoid memcopy to consolidate the fragments as they arrive, so it keeps each fragment in its own buffer, which has a significant memory overhead.  Once the entire IP packet is received and processed (discarded, forwarded, or copied to userspace), all these buffers are freed.  However, Linux does careful memory accounting, and if the memory for these fragment buffers grows past a certain threshold, Linux attempts to perform compaction (the dreaded memcopy), which requires a linear scan through all the accumulated fragment buffers.  In a well-behaved case, that compaction will free up a lot of extra memory, but in the attack case, that compaction will free only a little memory (just enough for one more fragment).  All subsequent attack packets will cause the linear compaction scan.

The algorithmic improvement to the above could include a faster compaction (extra bookkeeping to know in advance which fragments can be coalesced and not spend any time examining the ones that cannot be coalesced) and/or a simple heuristic after a single compaction that determines whether "enough" progress has been made and/or a "quality" heuristic that is maintained as fragments are being received.

- Igor

> From: Ole Troan <[email-masked]>
> Sent: Saturday, January 12, 2019 4:49 AM
> To: 6man WG <[email-masked]>
> > bytes.. and teh attacker is required to send 640 as opposed to 60
> > bytes or  so. Doesn't seem to be much of a difference to me.
> > first frag, and the implementation allocates a buffer that would fit
> > the rest of the missing fragments -- obviously exacerbating the
> > problem a bit. BUt still i this cases, the size of the first frag is not a big deal.
> 
> The scarce resource is number of available buffers. One fragment occupies at
> least one buffer regardless of how much smaller than 2K (default buffer size)
> it is.
> 
> And in other virtual reassembly algorithms we have also limited maximum
> allowed number of fragments in chain to 5.
> 
> Now, if we in the IETF think we can provide some guidance here, I will second
> Erik’s suggestion of getting more research.
> 
> The packet traces I have looked at in the past, the majority of fragmeted
> packets looked like attacks. UDP port 80, DNS queries to
> “thisdomainnamewillgiveaverylongresponsepurelyforthepurposeofattack.or
> g”,
> didn’t reassemble, 64K size… But it would be interesting to understand
> fragmentation behaviour in various parts of the network.
>

---

From: Tom Herbert <[email-masked]>
Date: Sat, 12 Jan 2019 09:08:00 -0800

> And as noted by Bob, I still don't see what's the big deal with the
Hi Fernando,

Consider that the smallest packet containing an IPv6 fragment is:

IP hdr. 40
Fragment header: 8
Fragment data: 8

Sums to 56 bytes. Maximum reassemble packet size is 65,535, so a
single packet could be composed of 8,186 such fragments. A stream of
these packet creates high load on both CPU and memory. Fundamentally,
I see no other purpose for sending a stream of fragments like this
other than DOS attack. I don't think it's at all unreasonable for a
host to outright reject these packets.

But as you point out, this isn't on the only way to attack
fragmentation, and it's easy to contrive other attacks. Consider, a
fragment like this:

IP hdr:40
Dst: opt: 2
722 Destination options of two byte length, where each option type is
00 in high order bits
Fragment header: 8
Fragment data: 8

Sums to 1500, ie. typical MTU. So this fragment will pass the minimum
1280 check but still carries the minimum fragment data so it can
attack memory used in reassembly. Processing long lists of options is
also a good attack on CPU. This attack could be mitigated by applying
limits on headers like in draft-int-area-header-limits. The default
limit number of bytes in network headers in that draft is 256 bytes,
so the minimum size of fragment data that could be accepted under that
limit is 384 bytes (much better then 8 bytes).

In any case, this discussion underscores a big challenge we have with
have with IPv6. There are several very powerful mechanisms in the
specifications that were defined without any limits. To get to
practical and deployable implementations we need to set some limits.
IMO the best way do that is to publically define the limits needed,
make them configurable, and suggest default values. I believe this is
a *FAR* better approach than implementions providing their own ad hoc
solutions (such as those that just drop all fragments or packets with
extension headers).

Tom

---

From: Tom Herbert <[email-masked]>
Date: Sat, 12 Jan 2019 09:17:56 -0800

> And in other virtual reassembly algorithms we have also limited maximum allowed number of fragments in chain to 5.
Ole,

I don't know what the protocol requirements are for virtual
reassembly. Is there an I-D or RFC on this?

Thanks,
Tom

---

From: Ole Troan <[email-masked]>
Date: Sat, 12 Jan 2019 19:50:31 +0100

> On 12 Jan 2019, at 18:17, Tom Herbert <[email-masked]> wrote:
> 

Virtual reassembly is just an optimization. The protocol requirements are the same. 

Cheers 
Ole

---

From: Tom Herbert <[email-masked]>
Date: Sat, 12 Jan 2019 11:27:18 -0800

> Virtual reassembly is just an optimization. The protocol requirements are the same.
The same as what requirements? RFC8200 defines how reassembly occurs
at a destination (the addressed node), not how an intermediate node in
the network reassembles packets. It is not the same thing. For
instance, there is an additional implicit requirement in virtual
reassembly that all fragments are routed through the same node. Also,
Hop-by-Hop options and Destination options can precede the fragment
header, so is the requirement that same that nodes MUST process these
before the fragmentation header is processed?

If virtual reassembly is brought up in the context of standard
protocol discussion, it would really be nice to have a normative
desciption of that protocol mechanism.

Tom

---

From: Ole Troan <[email-masked]>
Date: Sat, 12 Jan 2019 20:45:09 +0100

Tom,

> On 12 Jan 2019, at 20:27, Tom Herbert <[email-masked]> wrote:
> 

The code example I gave was for host reassembly.
I didn’t intend with my comment any more than that this is quite implementation specific, that it’s not obvious that the IETF can provide anything useful here, apart from removing network layer fragmentation, and I was absolutely not suggesting we should have a document on in-network reassembly. 

Cheers 
Ole

---

From: Bob Hinden <[email-masked]>
Date: Sun, 13 Jan 2019 10:54:33 -0800

Unleveling a bit.

I think the w.g. should be very cautious about making protocol changes based on what appears to be a fragmentation resource management problem in an implementation.

The fix of “..reject IPv6 fragments less than 1280 that aren't last fragment” seems pretty limited as discussed on the list.   Seems to me that this implementation needs more robust ability to handles it’s reassembly resources.

Bob

> On Jan 12, 2019, at 11:45 AM, Ole Troan <[email-masked]> wrote:
> 
> Tom,
>

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 14 Jan 2019 08:25:01 +1300

> I think the w.g. should be very cautious about making protocol changes based on what appears to be a fragmentation resource management problem in an implementation.

I agree. However, I wonder if we need to clarify that there is no rule that non-last fragments must be at least 1280. It seems that the Linux patch assumed that there is
such a rule.

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Sun, 13 Jan 2019 11:48:35 -0800

Brian,

Yes, but then what are the rules? AFAICT from RFC8200 the minimum
fragment data is eight bytes. I think it's going to be a hard sell to
convince OS developers that they should accept such packets just
because the spec requires it-- the DOS attack vector in this is too
obvious.

I'd also point out that this really isn't a Linux specific problem.
We've already had proposals to recommend that fragmentation should not
be used in the Internet. Frankly, such recommendations are not being
made because of deficient host implementations, because of
intermediate nodes that don't handle fragmentation properly. Narrowing
the requirements to make them more practical may be a good step
getting these devices to properly support fragmentation.

Tom

---

From: Ole Troan <[email-masked]>
Date: Sun, 13 Jan 2019 21:03:30 +0100

> On 13 Jan 2019, at 20:48, Tom Herbert <[email-masked]> wrote:
> 
> Frankly, such recommendations are not being
> getting these devices to properly support fragmentation

That’s highly unlikely given that the problem is that the packet is split in multiple pieces, which might take different paths. 

Let’s fix path MTU discovery. There’s no fix for fragmentation.

Cheers 
Ole

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 13 Jan 2019 20:12:36 +0000

> because the spec requires it-- the DOS attack vector in this is too
No intermediate node is going to fragment anything other than the last 
frame to 8 bytes, or at least if it does, the network is so severely 
crippled that fragmentation is the least of its problems.  It would be 
fair to assume that if host B receives a stream of fragments where the 
fragment size is unfeasibly small, then the fragmentation was done by 
host A.

This is not a good DOS vector, to be honest.  There's nothing to stop 
random small packets being streamed from A to B and it doesn't matter 
much whether they are fragments of a larger packet or not.

There doesn't seem to be a compelling case to change the protocol to 
accommodate this particular corner case.  It ranks as annoyance rather 
than a serious threat.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 13 Jan 2019 20:21:01 +0000

> Let’s fix path MTU discovery. There’s no fix for fragmentation.

pmtu discovery is hard because it needs a way for an intermediate node 
to be able to signal a transmitting node to dynamically drop the MTU if 
network conditions change.  The only way to reliably work around this is 
to transmit at 1280 and claim implementation / operational breakage if 
this cannot get through.  This, however, robs the host devices of the 
ability to use higher MTUs.

Nick

---

From: Ole Troan <[email-masked]>
Date: Sun, 13 Jan 2019 21:27:19 +0100

> On 13 Jan 2019, at 21:21, Nick Hilliard <[email-masked]> wrote:
> 
> pmtu discovery is hard because it needs a way for an intermediate node to be able to signal a transmitting node to dynamically drop the MTU if network conditions change.  The only way to reliably work around this is to transmit at 1280 and claim implementation / operational breakage if this cannot get through.  This, however, robs the host devices of the ability to use higher MTUs.

“Fix” as in something different than rfc8201. 

Ole

---

From: Tom Herbert <[email-masked]>
Date: Sun, 13 Jan 2019 12:29:15 -0800

> fragment size is unfeasibly small, then the fragmentation was done by
No legitimate host implementation would send this either. If B is
receiving these packets, the logical conclusion is that it's under DOS
attack and someone is spoofing A's address.

> much whether they are fragments of a larger packet or not.
As I mentioned fragments require much work work that simply processing
an IP packet, so it's a pretty good attack since it potentially
attacks memory, CPU utlization, and has is also a tuple attack on
paritcular CPUs. The flaw from the attackers point of view is that the
signature of the attack is obvious, small non-first fragments, and
therefore easily mitigated with a limit.

> accommodate this particular corner case.  It ranks as annoyance rather

Maybe not changing the protocols, but I still don't see a compelling
case to knowingly accept packets that are highly likely to be a part
of a DOS attack.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 13 Jan 2019 20:57:28 +0000

> attack and someone is spoofing A's address.

or that host A has a broken ipv6 implementation.

> case to knowingly accept packets that are highly likely to be a part

me neither, but that's an operational decision.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Sun, 13 Jan 2019 13:08:35 -0800

> me neither, but that's an operational decision.

Sure, except that once we put a limit like this in Linux it eventually runs
on billions of host and the majority of servers. I'd rather have alignment
with standards instead making yet another ad hoc standard that someone
eventually has to reverse engineer.

Tom

.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 14 Jan 2019 10:40:41 +1300

On 2019-01-14 08:48, Tom Herbert wrote:
> Yes, but then what are the rules? 

Well, I'm an advocate of one step at a time.

Step 1: As an addendum to RFC8200, state that there is no rule that non-last fragments must be at least 1280.

Step 2: Discuss whether there *should* be any rules about non-last fragment size, given that there are legitimate cases where they will be less than 1280.

Step 3: TBD

   Brian

> AFAICT from RFC8200 the minimum

---

From: "Manfredi (US), Albert E" <[email-masked]>
Date: Sun, 13 Jan 2019 22:23:48 +0000

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Brian E Carpenter

> Well, I'm an advocate of one step at a time.

Me too, although I also think that when there's no compelling reason, best to leave things alone.

> Step 1: As an addendum to RFC8200, state that there is no rule that non-last fragments must be at least 1280.

Yes, but if nothing is said, that is understood. It seems difficult to expect that every non-rule should be explicitly stated as not being a rule.

> Step 2: Discuss whether there *should* be any rules about non-last fragment size, given that there are legitimate cases where they will be less than 1280.

The rationale for any such minimum intermediate fragment size rule has been DOS attacks. But quite frankly, it's easy enough to conduct a DOS attack via fragmentation, no matter what minimum fragment size is stipulated. Even if one has to open more than one socket to do so.

I'm thinking, no reason to do anything, at least not with RFC 8200.

Bert

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Sun, 13 Jan 2019 23:57:51 +0000

On 13 Jan 2019, at 21:40, Brian E Carpenter wrote:

>>>> I think the w.g. should be very cautious about making protocol 
>>>> changes based on what appears to be a fragmentation resource 
>>>> management problem in an implementation.
>>>
>>> I agree. However, I wonder if we need to clarify that there is no 
>>> rule that non-last fragments must be at least 1280. It seems that 
>>> the Linux patch assumed that there is
>>> such a rule.

I think if you want to clarify anything than it is “What does minimum 
MTU of 1280 mean?”  I would assume that’s where the problem comes 
from. To my best memory it’s formulated as loser text and not that 
formal (*)(**).   The misunderstanding that one needs to be able to send 
packets with up-to 1280 doesn’t mean that one has to send packets that 
big.

(*) and I seem to remember that I always have MTU vs. MRU on my mind 
when I read these paragraphs and think that’s conflated in the text.
(**) Also that section 4.5 and 5 contradict each other as section 4.5 
says “fragment when no fit into path’s MTU”, while section 5 says 
“you may fragment already for anything larger than 1280”, e.g., a 
path’s mtu of 1500 can sill fragment a 1432 octet packet.

> Step 1: As an addendum to RFC8200, state that there is no rule that 
> non-last fragments must be at least 1280.

I am kind-of against this.  Just when we find any buggy implementation 
we cannot amend an RFC or any other document to clarify.  A bug is a bug 
and a OS vendor problem.  The document nowhere states what was 
implemented so why give them special treatment?

That’s a problem that community needs to solve and they’ll have good 
reference in public mailing list archives that what they did will break 
their communications and that’s it.

> Step 2: Discuss whether there *should* be any rules about non-last 
> fragment size, given that there are legitimate cases where they will 
> be less than 1280.
>
> Step 3: TBD

Now that’s a change to the current specification and that can be 
discussed but beware that changing current behaviour (and it was 
mentioned in this thread that there seem to be implementations of 
different sending behaviour) will mean that OS developers will have to 
carry “backward compatibility” for a long time (at least allowing 
the user to make a choice).

/bz

---

From: Fernando Gont <[email-masked]>
Date: Sun, 13 Jan 2019 23:40:25 -0300

On 12/1/19 06:48, Ole Troan wrote:
[...]
> The scarce resource is number of available buffers. One fragment occupies at least one buffer regardless of how much smaller than 2K (default buffer size) it is.

Some might argue that, by sending small fragments an attacker need not
waste bandwidth for the attack. But I don't think *that* is the
bottleneck the attacker will have for attacking. -- i.e., as you note, I
don't think requiring any minimum size for first frags will help the
implementation defending against this.

> And in other virtual reassembly algorithms we have also limited maximum allowed number of fragments in chain to 5.

Seems good. 60s frag timeout is just asking for trouble. And even
attacks aside, you will not get anything useful by trying to reassemble
a packet after a few seconds of receiving some frag.

> Now, if we in the IETF think we can provide some guidance here, I will second Erik’s suggestion of getting more research.

I will do some testing, plus assessment of what happens in TCP-land, too
(the later being unrelated to 6man).

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Sun, 13 Jan 2019 23:46:40 -0300

> “Fix” as in something different than rfc8201. 

RFC4821? :-)

---

From: Fernando Gont <[email-masked]>
Date: Sun, 13 Jan 2019 23:48:52 -0300

> Step 1: As an addendum to RFC8200, state that there is no rule that non-last fragments must be at least 1280.

Isn't that implicit?

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 14 Jan 2019 16:17:34 +1300

>> Step 1: As an addendum to RFC8200, state that there is no rule that non-last fragments must be at least 1280.

On the basis that anything that is not forbidden is allowed, yes, but apparently this wasn't apparent to the Linux devs involved. We don't really have a lightweight way of handling cases like this, since cost of producing an RFC is clearly too much.

   Brian

---

From: Erik Kline <[email-masked]>
Date: Sun, 13 Jan 2019 21:19:04 -0800

> network conditions change.  The only way to reliably work around this is to
> transmit at 1280 and claim implementation / operational breakage if this
> cannot get through.  This, however, robs the host devices of the ability to
> use higher MTUs.
>

How about a version of the fragment header that:

    (a) *always* has the L4 header (or a configurable number of bytes) in
every non-first fragment

This could be documented on a per-L4+ basis how much needs to be included,
i.e. we could have one doc for UDP and (say) a separate doc for information
QUIC would need on a per-fragment basis.

This increases the overhead in a given fragment, but also helps to ensure
that (eventually) intermediate systems can examine this field and
preemptively make a drop/no-drop decision.

    (b) state clearly that every non-last packet must contain
$CONSENSUS_MINIMUM_BYTES of actual fragment payload

No opinion on the exact value.

---

From: Mark Smith <[email-masked]>
Date: Mon, 14 Jan 2019 17:10:37 +1100

> QUIC would need on a per-fragment basis.

"Transport Options for UDP"

https://tools.ietf.org/id/draft-ietf-tsvwg-udp-options-05.txt

"5.6. Maximum Segment Size (MSS)...............................13
      5.7. Fragmentation (FRAG).....................................14"

---

From: Fernando Gont <[email-masked]>
Date: Mon, 14 Jan 2019 03:22:48 -0300

Hi, Erik,

On 14/1/19 02:19, Erik Kline wrote:
>     > pmtu discovery is hard because it needs a way for an intermediate
>     node to be able to signal a transmitting node to dynamically drop
>     the MTU if network conditions change.  The only way to reliably work
>     around this is to transmit at 1280 and claim implementation /
>     operational breakage if this cannot get through.  This, however,
>     robs the host devices of the ability to use higher MTUs.
> 
>     (a) *always* has the L4 header (or a configurable number of bytes)
> in every non-first fragment
> 
> This could be documented on a per-L4+ basis how much needs to be
> included, i.e. we could have one doc for UDP and (say) a separate doc
> for information QUIC would need on a per-fragment basis. 

If you are requiring that the first frag always contains the L4 ehader,
that's already achieved by RFC7112.

If you really mean that the FH contains the L4 header, then:

* There's the issue of EHs in the fragmentable part

* It may be hard to enforce in the presence of tunnels, unless the
tunnel reassembles and refragments where necesary

Still, that doesn't solve the biggest issues with fragmention:

* a stateful operation for anotherwise stateless protocol

* i'd argue that for the IPv6 case, one of the base reasons for which
fragments are dropped is because they employ EHs

> This increases the overhead in a given fragment, but also helps to
> ensure that (eventually) intermediate systems can examine this field and

Then we're back on draft-gont-v6ops-ipv6-ehs-packet-drops

> $CONSENSUS_MINIMUM_BYTES of actual fragment payload

In which cases do you expect this to make a substantial difference?

Thanks,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 07:18:32 -0800

> * a stateful operation for anotherwise stateless protocol
Fernando,

There is no fundamental problem with fragmentation. The core problem
is that some intermediate nodes are processing headers and data beyond
the IP and Hop-by-Hop options header. That is non-conformant with
RFC8200: "Extension headers (except for the Hop-by-Hop Options header)
are not processed, inserted, or deleted by any node along a packet's
delivery path". This becomes a material interoperability problem when
intermediate nodes take invasive action, such as dropping packets or
degrading the service, when they encounter protocols they don't
understand. This problem applies not only to fragmentation, but also
other extension headers, non-TCP transport protocols, and other
protocols outside some narrow set of typically used protocols. To fix
this, either non-conformant implementations should be changed to
conform to the standard, or the standard should be changed to conform
to the implementation.

Tom

---

From: Simon Hobson <[email-masked]>
Date: Mon, 14 Jan 2019 16:00:55 +0000

Nick Hilliard <[email-masked]> wrote:

> No intermediate node is going to fragment anything other than the last frame to 8 bytes, or at least if it does, the network is so severely crippled that fragmentation is the least of its problems.  It would be fair to assume that if host B receives a stream of fragments where the fragment size is unfeasibly small, then the fragmentation was done by host A.

What if ...
An intermediate node gets a packet inbound of N bytes, but the MTU downstream only allows it to send N-8. If it doesn't have other packets* to degragment/merge with first, then it's only option is to send two smaller packets downstream - and one logical method would be to send N-8 followed by 8 bytes.

* And bear in mind that there is no rules that says the packets before or after this one have to be carried by the same route - hence (one reason) why nodes need to handle out of order fragments.

>> Step 1: As an addendum to RFC8200, state that there is no rule that non-last fragments must be at least 1280.

It may be, but (as already mentioned) we have a significant implementation where it would seem the devs didn't see that.

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 14 Jan 2019 16:10:31 +0000

Simon Hobson wrote on 14/01/2019 16:00:
>> No intermediate node is going to fragment anything other than the
>> last frame to 8 bytes, or at least if it does, the network is so
>> severely crippled that fragmentation is the least of its problems.
>> It would be fair to assume that if host B receives a stream of
>> fragments where the fragment size is unfeasibly small, then the
>> fragmentation was done by host A.
> 
> What if ... An intermediate node gets a packet inbound of N bytes,
> but the MTU downstream only allows it to send N-8. If it doesn't have
> other packets* to degragment/merge with first, then it's only option
> is to send two smaller packets downstream - and one logical method
> would be to send N-8 followed by 8 bytes.

I was thinking about a variation of this situation earlier today, e.g. 
1492 MTU DSL products with jumbo transmitter, or possibly frames 
transmitted / load balanced over a different physical path to the path 
that PMTU first figured out.

In summary: tiny fragments which aren't last in the sequence shouldn't 
be expected often, but they form a legitimate corner case.

Nick

---

From: Fernando Gont <[email-masked]>
Date: Mon, 14 Jan 2019 13:51:13 -0300

On 14/1/19 12:18, Tom Herbert wrote:
> the IP and Hop-by-Hop options header.

If you need to reassemble fragments in the network, or at an end system,
and you're receiving packets at a very high data rate, you just cannot
keep up with fragments. that's the Inherent problem. Yo need to keep
packets in memory without even knowing whether they have been spoofed.

> That is non-conformant with

Please look at draft-gont-v6ops-ipv6-ehs-packet-drops. There are
operational reasons for that. So, you need to look into the packet, but
you cannot do it in the fast path. Hence you just drop them.

Operators need to do their thing. And nobody is going to pay extra money
to do it on the past path -- even then, it would take time for all
current boxes to be phased out.

Your argument would be on the same line with my own regarding why there
shouldn't be any people under the line of poverty. -- that doesn't
change facts.

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 09:58:37 -0800

Fernando,

I don't know what it means to "reassemble fragments in the network".
Reassembly is specified as an operation at destination end points.
Unless there's a specification for in-network reassembly that
describes the requirements and procedures, I see no point in
considering it.

> packets in memory without even knowing whether they have been spoofed.
This has been know since fragmentation was first defined.  Host
implementations have implemented several mitigations including
lowering reassembly timeout, limiting number of reassembly states,
etc. This isn't a unique problem either, SYN attacks have many similar
characteristics, and we've learned over time how to deal with those as
well. As we learn more and identify attack vectors, such as the
aforementioned non-last tiny fragments, we can adapt the
implementation accordingly.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 14 Jan 2019 18:12:52 +0000

> Reassembly is specified as an operation at destination end points.

some types of stateful middleboxes do in-line reassembly.

There are many horrors on the Internet.

Nick

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Mon, 14 Jan 2019 18:39:29 +0000

> There are many horrors on the Internet.

And maybe it’s time to call them out, say screw them, and either 
people have to live with broken stuff or get end-to-end connectivity 
again.   Maybe it’s time to stop catering for the brokenness and 
control of others.  Maybe it’s time to play it hard.

/bz

---

From: Simon Hobson <[email-masked]>
Date: Mon, 14 Jan 2019 18:47:34 +0000

Bjoern A. Zeeb <[email-masked]> wrote:

> And maybe it’s time to call them out, say screw them, and either people have to live with broken stuff or get end-to-end connectivity again.   Maybe it’s time to stop catering for the brokenness and control of others.  Maybe it’s time to play it hard.

And when you hit the reality of large corporates that need to watch out for what's going in/out of their network ? Or organisations that are legally required to control things - think healthcare or military contractors ? Or ...

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 14 Jan 2019 18:47:48 +0000

Bjoern A. Zeeb wrote on 14/01/2019 18:39:
> And maybe it’s time to call them out, say screw them, and either people 
> have to live with broken stuff or get end-to-end connectivity again.   
> Maybe it’s time to stop catering for the brokenness and control of 
> others.  Maybe it’s time to play it hard.

End-to-end died the day the IETF published rfc1631.

In-line reassembly may be distasteful but in the scale of things it 
doesn't cause massive breakage for the types of clients it attempts to 
provide service to.

There are lots of other things that are far worse offenders, e.g. 
encoding transport endpoint values inside data streams (here's looking 
at you, SIP).

Nick

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 11:06:29 -0800

> control of others.  Maybe it’s time to play it hard.
Bjoern,

I tend to agree in principle, however I think before playing hardball,
we should make it easier to be protocol compliant. One of the issues
that intermediate nodes (as well as end nodes) have with processing
IPv6 is the open endedness of the protocol mechanisms. By the spec,
end hosts are supposed to be able to reassemble a single packet
composed of over 8000 fragments. Before RFC8200, all intermediate
nodes were expected to process packets that could contain over 700
individual Hop-by-Hop options. Those simply aren't practical to
support in an implementation, and as Fernando points out no one is
going to spend money on fixing that. But, we can set reasonable
expections of support, such a minimum fragment size or allow a limit
on number of options in packet (eight is suggested for instance). With
such limits and improved packet processing tehniques and device
programmability, I think it's feasible and cost-effective for new
network devices to increase their protocol compliance.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 11:31:57 -0800

> And when you hit the reality of large corporates that need to watch out for what's going in/out of their network ? Or organisations that are legally required to control things - think healthcare or…
Simon,

Private entities are free to run their networks however they want. The
problem is when mechansims are placed in plublic networks that
unilaterally prohibit standard protocols or standard protocol features
from being used in their network. Ostensibly, providers may claim that
they are providing a benefit to their users, but in reality they
aren't. This is what is limiting innovation on the Internet.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 15 Jan 2019 08:47:06 +1300

> End-to-end died the day the IETF published rfc1631.

The IETF didn't publish RFC1631. The RFC Editor published it. Although
the Independent Submissions stream hadn't been formally named at that
time, that's what this was.

   Brian

---

From: Mark Andrews <[email-masked]>
Date: Tue, 15 Jan 2019 07:19:02 +1100

> On 15 Jan 2019, at 6:06 am, Tom Herbert <[email-masked]> wrote:
> 
> composed of over 8000 fragments.

No, a node is required to support reassembly up to 1500 octets (the
limit in IPv4 was 576) which works out at 183 minimum sized fragments.
Most nodes support larger than minimum reassembly buffers.

> Before RFC8200, all intermediate

Again this was people being idiotic. Firstly there was never a requirement
to reassemble packets in routers.  So unless you actually had link packets
that were 2^16-1 bytes in size you would never see such counts on a packet.
Also given that it is impossible to process HBH options in other than the
0 offset fragment (you can’t know if the start of a non 0 offset fragment is
a HBH option) the practical count in a 1500 octet MTU network is considerably
smaller.  In practice any packet can have be fragmented down to 1280 so that
was always the practical limit into which HBH options needed to fit.  Thats
still a lot of HBH options but no hardware needed to handle 700 option.

Mark

---

From: Erik Kline <[email-masked]>
Date: Mon, 14 Jan 2019 12:40:43 -0800

> that's already achieved by RFC7112.

Not what I suggested.  I said every fragment header would include the
L4 header (or other specifiable bytes).

> * There's the issue of EHs in the fragmentable part

That would be addressed in an associated L4 doc, and it would probably
say to ignore non-transport headers.

> Then we're back on draft-gont-v6ops-ipv6-ehs-packet-drops

Understood, but any L3 solution is going to necessitate running into
this.  UDP trailers is certainly cute (and +1 from me), but every
transport will have to have some similar fix (or applications will
have to consider rfc8085#section-3.2 type guidelines).  If there's to
be an L3 solution to this L3 problem it will face some deployment
issues but if its usefulness is sufficiently attractive it could find
reasonably wide-spread support in due time (in the absence of ipng-ng,
IPv6 is it for the long haul).

[Silly Idea #1]

(/me thinks back to spud bof)

Here's another random suggestion that maybe belongs in panrg or (more
likely) the dumpster:

A new hop-by-hop header under "00 1", i.e.

    00 - skip over this option and continue processing the header

    1 - Option Data may change en route

of the form, for example, Option Type 00 1 00001, Opt Data Len 2,
Option Data == MTU.  Here, the observation is that the main thing we
have experience with "working" (for the most generous definition of
"works") in this problemspace in the internet today is TCP MSS
Clamping.  So let's have an MTU (or MSS) HbH option that can be
revised monotonically downwards but never lower than 1280.  If it
helps pad to 8 byte alignment we can have a 2nd 2-byte field that is
untouched representing the sender's original MTU/MSS.

In this way, a sender could include an HbH with either:

    [HbH | 0 | 0x0202 | 2 | (1500=0x5dc) | Pad1 | Pad1]

or

    [HbH | 0 | 0x0202 | 4 | (1500=0x5dc) | (1500=0x5dc) ]

with one of the MTU/MSS fields to be munged along the path.

If this were the only HbH option included it adds only 8 bytes, fits
(along with most L4 headers) within a 128 byte hardware classifier,
and end systems could explore its usefulness via ping.  (Separately,
there could of course be sockopt/cmsg things to set/get the values.)

[Silly Idea #2]
Separate, even sillier idea: everyone converges on using 16 bits of
the flow ID to encode the lowest MTU encountered along the path.  Here
again, TCP MSS clamping style behaviour would be applied to this
field.

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 14 Jan 2019 21:20:37 +0000

> time, that's what this was.

several people commented on this privately. The point I was trying to 
make to Bjoern was that the end-to-end principal ended a long time ago - 
certainly more than the 25 years ago that this rfc was published.  The 
large address space of IPv6 will not make it return.

Nick

---

From: Ole Troan <[email-masked]>
Date: Mon, 14 Jan 2019 22:34:22 +0100

> On 14 Jan 2019, at 21:40, Erik Kline <[email-masked]> wrote:
> 

https://tools.ietf.org/html/draft-hinden-6man-mtu-option-00

---

From: Erik Kline <[email-masked]>
Date: Mon, 14 Jan 2019 14:04:16 -0800

Clearly I'm way behind in my document reading!!

I'd just add that it seems reasonable to see about broader Internet
usage at some point.

---

From: Fernando Gont <[email-masked]>
Date: Mon, 14 Jan 2019 19:14:02 -0300

> say to ignore non-transport headers.

In that case, we should be removing EHs in the fragmentable part (not
that it would be a a big deal.. just pointing it out).

>>> preemptively make a drop/no-drop decision.

Agreed.

> IPv6 is it for the long haul).

Certainly what you describe is better than what we have -- still, I'd
say not good enough for fragments to survive (e.g. cases where they need
to be reassembled at high speed) -- and there's still the fundamental
problem of processing EHs.

>     [HbH | 0 | 0x0202 | 2 | (1500=0x5dc) | Pad1 | Pad1]

You mean as a kind of PMTUD?

> again, TCP MSS clamping style behaviour would be applied to this

I assume this is for PMTUD, too. This option would be way better than
the preovious one (node will just not process EHs), but...give the path
that we have followed for FL standrdization, it wouldn't be useful for
anything else other than random bits. One might hack some other fields
e.g... special value for Payload length, but.. that would certainly look
ugly.

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 14:24:38 -0800

Erik,

Please look at draft-herbert-fast-03 also. This protocol agnostic and
secure way for applications to signal the network. In order fo this to
work, as well as other similar proposals in MTU reporting and in-situ
OAM, we really need Hop-by-Hop extension header to at least pass
though intermediate nodes.

Tom

---

From: Christian Huitema <[email-masked]>
Date: Tue, 15 Jan 2019 11:15:02 -1000

On 1/14/2019 11:20 AM, Nick Hilliard wrote:

> make to Bjoern was that the end-to-end principal ended a long time ago
> - certainly more than the 25 years ago that this rfc was published. 
> The large address space of IPv6 will not make it return.

Actually, the pendulum went from "everything end-to-end" to "lots of
processing in the middle", and it is now swinging back. Developers have
realized that if they want to keep a particular function "end-to-end",
they should use encryption. For example, using HTTPS instead of HTTP
prevents intermediaries from messing with the content of the pages.
Using QUIC instead of TCP prevents intermediaries from messing with the
transport headers.

Of course, the IP header itself is not encrypted, so encryption does not
prevent NAT. That's why encrypted transports run over UDP. A side-effect
of that is that UDP does expose the L4 ports, which sort of meets the
requirement of intermediaries that run packet classifiers.

PMTU discovery finds itself at the border between the clear text IPv6
header and the encrypted transports. It is sort of solved by declining
to use fragmentation, and sending end-to-end encrypted probes to
reliably discover the current PMTU. If an encrypted message of length
end arrives to the other end and is acknowledged, the peers know that
the PMTU is at least L.

End to end PMTU probing is not perfect. It requires sacrificial probes,
which has some overhead. It also requires guesswork to discover when a
PMTU decreases. This is an area that would benefit from some innovation.
For example, instead of requiring fragmentation support, it might be
tempting to use truncation, so the truncated packet arrives to the other
end. Of course, supporting truncation would require changes in routers
and changes in the way transports deploy encryption. But that's probably
the best long term way to solve the issues with fragmentation and
reassembly.

-- Christian Huitema

---

From: Tom Herbert <[email-masked]>
Date: Tue, 15 Jan 2019 14:42:06 -0800

On Tue, Jan 15, 2019 at 1:15 PM Christian Huitema <[email-masked]> wrote:
> several people commented on this privately. The point I was trying to make to Bjoern was that the end-to-end principal ended a long time ago - certainly more than the 25 years ago that this rfc was published.  The large address space of IPv6 will not make it return.
>
> Actually, the pendulum went from "everything end-to-end" to "lots of processing in the middle", and it is now swinging back. Developers have realized that if they want to keep a particular function "end-to-end", they should use encryption. For example, using HTTPS instead of HTTP prevents intermediaries from messing with the content of the pages. Using QUIC instead of TCP prevents intermediaries from messing with the transport headers.
>
> Of course, the IP header itself is not encrypted, so encryption does not prevent NAT. That's why encrypted transports run over UDP. A side-effect of that is that UDP does expose the L4 ports, which sort of meets the requirement of intermediaries that run packet classifiers.
>
> PMTU discovery finds itself at the border between the clear text IPv6 header and the encrypted transports. It is sort of solved by declining to use fragmentation, and sending end-to-end encrypted probes to reliably discover the current PMTU. If an encrypted message of length end arrives to the other end and is acknowledged, the peers know that the PMTU is at least L.
>
> End to end PMTU probing is not perfect. It requires sacrificial probes, which has some overhead. It also requires guesswork to discover when a PMTU decreases. This is an area that would benefit from some innovation. For example, instead of requiring fragmentation support, it might be tempting to use truncation, so the truncated packet arrives to the other end. Of course, supporting truncation would require changes in routers and changes in the way transports deploy encryption. But that's probably the best long term way to solve the issues with fragmentation and reassembly.
>
Christian,

The only thing worse than intermediate nodes inspecting packets in
non-standard ways is when they modify packets in non-standard ways. A
better approach might be draft-hinden-6man-mtu-option where the
application can probe the path with small packets before sending big
packets. This uses a modifiable HBH options and is completely
idependent of the transport layer.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 15 Jan 2019 22:45:20 +0000

Christian Huitema wrote on 15/01/2019 21:15:
>  processing in the middle", and it is now swinging back. Developers
> have realized that if they want to keep a particular function
> "end-to-end", they should use encryption. For example, using HTTPS
> instead of HTTP prevents intermediaries from messing with the content
> of the pages. Using QUIC instead of TCP prevents intermediaries from
> messing with the transport headers.

it's more a change in what we mean by end-to-end. Historically it 
referred to transport layer end-to-end service:  device A could directly 
address device B, or vice versa.  Data integrity + confidentiality 
guaranteed by strong crypto seems like a more relevant proposition to 
today's internet though - it allows device A to talk to device B without 
having to worry about whether to trust all the devices in the middle, 
although it also generally dispensed with the symmetrical requirement 
for device B to be able to initiate communications with device A. 
Although orthogonal in approach, the functionality change is not 
necessarily a bad trade-off.

Nick

---

From: Warren Kumari <[email-masked]>
Date: Tue, 15 Jan 2019 19:31:01 -0500

> > again, TCP MSS clamping style behaviour would be applied to this
https://media.giphy.com/media/3o6ZthnDqxfAOFIhdC/giphy.gif
Because there are deployed systems already, I don't think that this can be
deployed **and fully relied upon**, but I do think that it might actually
be really useful. If the first 2 bits of the flow label are '11' or '10' or
something, then it could mean that the next N (13?) bits of the flow label
contain the minimum MTU along the path. It doesn't matter that it cannot be
relied upon, because it could simply be used as a hint to the path MTU
discovery (I tried to figure out where in RFC8201 if could be tied).
Basically, if the N bits are less than 1280, ignore them. If they are
larger than the host MTU, ignore them. If they fall between 1280 and
$host_mtu, guess that they might be valid and prime the path MTU discovery
process with this hint.
Initially devices along the path will a: not touch the flow ID or b:
scribble all over it like they currently do. Over time, new devices could
be updated to do this, and the signal would slowly become more useable.

It *does* mean that we would be removing (by convention) some of the
possible entropy in the flow ID which might make ECMP harder, and may also
discourage some of the other "lets scribble marking X into this field"
ideas, but that might be a feature, not a bug.

I cannot tell if I've just taken a sharp turn into crazy town, or if this
really is a good idea (and I haven't thought about it in depth - this is
all a knee jerk reaction to something sufficiently hacky that it entertains
me...)

> e.g... special value for Payload length, but.. that would certainly look

What if you view it as an (initially) noisy hint, becoming more believable
over time? As far as I can tell, an on-path attacker could degrade service
by artificially signalling a lower MTU than really exists, but an onpath
attacker could already do this (and much worse!).
W

-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 16 Jan 2019 14:02:02 +1300

For my comment, search for "Aaargh"

On 2019-01-16 13:31, Warren Kumari wrote:
> 
> 
> On Mon, Jan 14, 2019 at 5:19 PM Fernando Gont <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Because there are deployed systems already, I don't think that this can be deployed **and fully relied upon**, but Ido think that it might actually be really useful. If the first 2 bits of the flow label are '11' or '10' or something, then it could mean that the next N (13?) bits of the flow label contain the minimum MTU along the path. 

Aaargh. That's a gross violation of the standard for the flow label. Short form: you can't use the flow label for signaling. Long form: RFC6436, RFC6437, plus RFC6294 for a menagerie of other terrible ways to abuse the flow label.

> It doesn't matter that it cannot be relied upon, because it could simply be used as a hint to the path MTU discovery (I tried to figure out where in RFC8201 if could be tied).
> Basically, if the N bits are less than 1280, ignore them. If they are larger than the host MTU, ignore them. If they fall between 1280 and $host_mtu, guess that they might be valid and prime the path MTU discovery process with this hint.
> Initially devices along the path will a: not touch the flow ID or b: scribble all over it like they currently do. Over time, new devices could be updated to do this, and the signal would slowly become more useable. 
> 
> It *does* mean that we would be removing (by convention) some of the possible entropy in

s/some/most/

Which is serious, since the layer 4 info is decreasingly available as entropy fodder.

   Brian

> the flow ID which might make ECMP harder, and may also discourage some of the other "lets scribble marking X into this field" ideas, but that might be a feature, not a bug.
> 
> I cannot tell if I've just taken a sharp turn into crazy town, or if this really is a good idea (and I haven't thought about it in depth - this is all a knee jerk reaction to something sufficiently hacky that it entertains me...)
> 
>  
> 
> What if you view it as an (initially) noisy hint, becoming more believable over time? As far as I can tell, an on-path attacker could degrade service by artificially signalling a lower MTU than really exists, but an onpath attacker could already do this (and much worse!).
> W
> 
>  
> 
> 
>     -- 
>     Fernando Gont
>     SI6 Networks
>     e-mail: [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>
> I don't think the execution is relevant when it was obviously a bad idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants.
>    ---maf
>

---

From: Fernando Gont <[email-masked]>
Date: Wed, 16 Jan 2019 04:42:54 -0300

On 15/1/19 21:31, Warren Kumari wrote:

> Because there are deployed systems already, I don't think that this can
> be deployed **and fully relied upon**, but Ido think that it might
> actually be really useful. If the first 2 bits of the flow label are
> '11' or '10' or something, then it could mean that the next N (13?) bits

Next question is: This of course implies that, since using this would
make the FL take a few deterministic values, it cannot be used for e.g.
ECMP. Hence processing the EH chain becomes a requirement (well, in
practice it already is).

> possible entropy in the flow ID which might make ECMP harder, and may
> also discourage some of the other "lets scribble marking X into this
> field" ideas, but that might be a feature, not a bug.

Depends on whether you want the FL to work as a FL, and for it to be
useable for hashing input when doing ECMP.

> What if you view it as an (initially) noisy hint, becoming more
> believable over time? As far as I can tell, an on-path attacker could
> degrade service by artificially signalling a lower MTU than really
> exists, but an onpath attacker could already do this (and much worse!).

Somewhere I got lost in this discussion:

RFC1981 doesn't reliably work due to ICMP filtering
RFC4821 has long convergence time

[then we come up with this discussion]

It would seem to be the easier and best way to go is to do RFC4821 for
blackhole detection, while still processing ICMP messages (in fact,
that's one of the possible approaches in RFC4821)

Put another way: how would anything else makes since better, and become
more feasible?

---

From: Fernando Gont <[email-masked]>
Date: Wed, 16 Jan 2019 04:47:49 -0300

> For my comment, search for "Aaargh"

:-) -- I was catching up with email, and read them in sequence. --
thanks for the note, I *expected* it :-)

> Aaargh. That's a gross violation of the standard for the flow label. Short form: you can't use the flow label for signaling. Long form: RFC6436, RFC6437, plus RFC6294 for a menagerie of other terri…

Or, put another way: if we were to use the FL for this, it would screw
up the currently-intended use.

---

From: Fernando Gont <[email-masked]>
Date: Wed, 16 Jan 2019 04:51:32 -0300

On 15/1/19 19:42, Tom Herbert wrote:
[....]
> idependent of the transport layer.

You cannot use HBH in the public Internet. Well, you can... but there
high chances your packets will be dropped.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 16 Jan 2019 09:04:35 +0100 (CET)

> that it would be a a big deal.. just pointing it out).

https://tools.ietf.org/html/rfc6980 that you wrote talks about this. 
Having more generic wording that all headers must be in the first packet 
would make a lot of sense for the generic use-case. Do we have that 
requirement anywhere for generic IPv6 packets?

---

From: Tim Chown <[email-masked]>
Date: Wed, 16 Jan 2019 08:15:43 +0000

> On 16 Jan 2019, at 08:04, Mikael Abrahamsson <[email-masked]> wrote:
> 
> https://tools.ietf.org/html/rfc6980 that you wrote talks about this. Having more generic wording that all headers must be in the first packet would make a lot of sense for the generic use-case. Do we have that requirement anywhere for generic IPv6 packets?

Isn't that RFC 7112, for which Fernando is one author, which is now cited in RFC 8200?  See note (3) in Section 4.5 of RFC 8200 - "Extension headers, if any, and the Upper-Layer header.  These headers must be in the first fragment."

Tim

---

From: Fernando Gont <[email-masked]>
Date: Wed, 16 Jan 2019 05:11:19 -0300

> https://tools.ietf.org/html/rfc6980 that you wrote talks about this.

FWIW, my original intention was to ban all EHs from ND. The WG agreed
just on the FH.

> requirement anywhere for generic IPv6 packets?

You mean something like this: RFC7112?

---

From: Warren Kumari <[email-masked]>
Date: Wed, 16 Jan 2019 10:11:39 -0500

On Wed, Jan 16, 2019 at 2:52 AM Fernando Gont <[email-masked]> wrote:

> be deployed **and fully relied upon**, but Ido think that it might actually
> contain the minimum MTU along the path.
> >
> > Aaargh. That's a gross violation of the standard for the flow label.
> Short form: you can't use the flow label for signaling. Long form: RFC6436,
> RFC6437, plus RFC6294 for a menagerie of other terrible ways to abuse the
> flow label.
>

Yup, I had actually read those the above, but it was a while back...

I'm not really sure it does screw up the intended use -- the FL is 20 bits.
If we were to do something like Erik had proposed, and we wanted to be able
to signal up to a 9K MTU, we would need 13bits (LN(9K-1280)/LN(2) = 12.9).
20 - 13 gives 7 bits (128) for the hash entropy. "7 bits of entropy, evenly
distributed, should be enough for anyone", he said, hoping no-one points at
the obvious correlation to 640K of RAM...
If we only wanted to signal full granularity between 1500 and 1280, and
then require that any MTU between 1500 and 9K falls into one of 36 buckets,
we could reserve one byte to do so (1500-1280=220. 256-220=36).

This likely is still a horrible, no good idea, but it does at least make
the MTU available to things looking at the L3 header only, and doesn't
require routers to have to dig into extension headers.

Whatever the case, when we finally design IP-NG-NG, let's set aside 512bits
(at least) to be "reserved for future abuse" :-P Or, perhaps, every bit
should be fully cast in stone (with no room for extensions) so that they
don't become attractive nuisances for ugly hacks...

:-)
W

---

From: Tom Herbert <[email-masked]>
Date: Wed, 16 Jan 2019 07:41:51 -0800

Fernando,

By that same thinking we can't use any extension headers at all, any
transport protocol other then TCP (and maybe UDP), ICMP, the TCP
authentication option, TCP fast open, UDP options, or even IPv6 on the
Internet. None these will universally pass through the Internet. But
it's also true that not all such packets are dropped, there are a lot
of networks where these will work and they do conform to the
standards.

We have some workarounds for the problem. We can follow the IPv6 happy
eyeballs approach: if an application see that packets aren't making it
to a destination, then fallback back to using a more rudimentary
protocol with higher probability of success but less service offered
to the user. Encrypting as much of the packet as possible, including
transport layer headers like QUIC does, also helps-- what the network
can't see, it can ossify.

As for HBH, RFC8200 relaxed the requirement that intermediate nodes
need to process them. If a node just wants to get to the transport
layer it can skip over the EH very with a few simple operations. I see
no excuse for new devices being deployed to systematically drop HBH or
destination options EHs. If they are  doing that then their
implementation does not comply with the IPv6 standard and we shouldn't
have any compunction about publically calling them out on that.

Tom

---

From: Ole Troan <[email-masked]>
Date: Wed, 16 Jan 2019 17:09:07 +0100

> On 16 Jan 2019, at 16:11, Warren Kumari <[email-masked]> wrote:
> 
> 
> 
> I'm not really sure it does screw up the intended use -- the FL is 20 bits. If we were to do something like Erik had proposed, and we wanted to be able to signal up to a 9K MTU, we would need 13bits (LN(9K-1280)/LN(2) = 12.9). 
> 20 - 13 gives 7 bits (128) for the hash entropy. "7 bits of entropy, evenly distributed, should be enough for anyone", he said, hoping no-one points at the obvious correlation to 640K of RAM...
> If we only wanted to signal full granularity between 1500 and 1280, and then require that any MTU between 1500 and 9K falls into one of 36 buckets, we could reserve one byte to do so (1500-1280=220. 256-220=36).
> 
> This likely is still a horrible, no good idea, but it does at least make the MTU available to things looking at the L3 header only, and doesn't require routers to have to dig into extension headers. 

A splendid idea indeed. 
Of course you could also take the topmost bit of the payload field and encode the 8 bits required in a vector of packets. I suggest using a magic cookie first. E.g the string “MTU”, rot-13 encoded in morse. This way you should be able to signal the MTU in less than a hundred packets. 

Ole

> 
> Whatever the case, when we finally design IP-NG-NG, let's set aside 512bits (at least) to be "reserved for future abuse" :-P Or, perhaps, every bit should be fully cast in stone (with no room for extensions) so that they don't become attractive nuisances for ugly hacks...
> 
> :-)
> W
>> 
>> 
>> 
>> -- 
>> Fernando Gont
>> SI6 Networks

---

From: Fernando Gont <[email-masked]>
Date: Wed, 16 Jan 2019 13:33:36 -0300

On 16/1/19 12:41, Tom Herbert wrote:
> By that same thinking we can't use any extension headers at all,

Indeed.

> any
> Internet. 

I assume that such options focus on specific scenarios (e.g. TCP-AO for
BGP), or they have fall-back options (e.g. traditional open for TCP-FO).
Otherwise they are doomed to fail.

> have any compunction about publically calling them out on that.

See https://tools.ietf.org/html/draft-gont-v6ops-ipv6-ehs-packet-drops

The folks running the networks have work to do. And will certainly not
pay extra money for functionality that somehow is currently not
required. Vendors will not invest extra to sell devices at the same price.

We can call them out. Vendors for selling boxes that perform very badly
for packets that contain EHs, and operators for "looking into packets in
the middle of the network, breaking the sacred e2e principle (or some
interpretation of it)".

Then I guess *we* will be called out to pretend doing engineering, while
doing stuff that doesn't work in practice.

We burying our heads into sand doesn't help the situation, either...

---

From: Fernando Gont <[email-masked]>
Date: Wed, 16 Jan 2019 13:35:52 -0300

On 16/1/19 12:11, Warren Kumari wrote:
> 
> 
> On Wed, Jan 16, 2019 at 2:52 AM Fernando Gont <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
[...]
> 
> Whatever the case, when we finally design IP-NG-NG, let's set aside
> 512bits (at least) to be "reserved for future abuse" :-P Or, perhaps,
> every bit should be fully cast in stone (with no room for extensions) so
> that they don't become attractive nuisances for ugly hacks...

To be included in the TODO list for IPv11, I guess -- IPv10 has been
taken already, it seems ;-)

---

From: Fred Baker <[email-masked]>
Date: Wed, 16 Jan 2019 09:20:23 -0800

> On Jan 15, 2019, at 11:42 PM, Fernando Gont <[email-masked]> wrote:
> 

If you want to monkey with the Flow Label, I'd suggest you get Brian Carpenter into the conversation.

---

From: Tim Chown <[email-masked]>
Date: Wed, 16 Jan 2019 17:24:52 +0000

> On 16 Jan 2019, at 17:20, Fred Baker <[email-masked]> wrote:
> 

I believe he already said "Aaaargh!" :)

Tim

---

From: Tom Herbert <[email-masked]>
Date: Wed, 16 Jan 2019 09:28:11 -0800

> required. Vendors will not invest extra to sell devices at the same price.
I'm not sure what you mean by "functionality that somehow is currently
not required.". Who determines what is required?

> doing stuff that doesn't work in practice.
There is engineering happening. The requirement that every node needs
to process HBH options where there could be hundreds of options in a
single packet was, in retrospect, completely impractical to expect to
be supported. So, RFC8200 relaxed the nodes must process requirement,
and we can define some reasonable limits as to how many options can be
processed. These are examples of updating standards that make
engineering feasible.

> We burying our heads into sand doesn't help the situation, either...

Agreed. If someone were to *prove* conclusively that it is impossible
to support even the minimum requirement that allows device to skip
over HBH options, then I would agree that it may make sense to give up
and deprecate them. But until we have that proof in hand, the problem
is real and needs to be dealt with.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 Jan 2019 08:24:51 +1300

Hi,

A small comment at the end...

> is real and needs to be dealt with.

My conclusion about what we should do is a bit different, and is
expressed in draft-carpenter-limited-domains (preferred list for
discussion is [email-masked]).

   Brian

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 Jan 2019 08:47:22 +1300

> the obvious correlation to 640K of RAM...

I guess it all depends what you expect from the entropy. 20 bits gives
you a 1-in-a-million chance of a clash. 7 bits gives you a 1-in-128 chance
of a clash. This probably doesn't matter for a simple ECMP or LAG kind
of load sharing, but who's to say it doesn't matter for some more
fancy kind of sharing across a large array of servers?

> don't become attractive nuisances for ugly hacks...

I hope that by then we will have found a better approach to flexibility
and extensibility that allows for interfering middleboxes from the start.
But that discussion is definitely out of scope for the "IPv6 Maintenance" WG ;-)

    Brian

---

From: Michael Richardson <[email-masked]>
Date: Wed, 16 Jan 2019 15:54:38 -0500

> ECMP. Hence processing the EH chain becomes a requirement (well, in

Only 15 bits of the 20 are deterministic, and Warren suggests says to use
8-10 bits in another message, leaving 10 to 12 bits of entropy.

I'm not really sure that the probes need this much priming, but I sure would
like to get PLPMTU written down as MUST.

I think it will be a decade or so before MTU to the home grows beyond 1500,
but I'd love to be wrong here.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Wed, 16 Jan 2019 18:36:00 -0500

Warren writes about putting MTU info into flow Label:
    >> 20 - 13 gives 7 bits (128) for the hash entropy. "7 bits of entropy,
    >> evenly distributed, should be enough for anyone", he said, hoping
    >> no-one points at

    > fancy kind of sharing across a large array of servers?

You'd have to have more than 128 servers and/or paths.
Maybe one of our SPRING people in this WG can tell us if that's a real
problem today.  Obviously, we can't guess if it will be bad in the future,
but if it's a problem today, then we can know that immediately.

Now when I pushed for draft-ietf-6man-rfc6434-bis-09 and RFC8200 to say
that PLMTUD to be made MUST, I got various push backs that amounted to:
  1) we don't have enough evidence yet.
  2) it doesn't work for UDP and other traffic.

(1) turned out to be a real issue. I thought that some of the big players
    could easily get, or already would have, that kind of evidence.
    (Linux does not ship with PLPMTUD on by default.  If you want it, btw,
    sysctl -w net.ipv4.tcp_mtu_probing=2.  Yes. ipv4. It affects both)
    Turns out I was told that they always set their TCP segment size such that
    they likely will never fragment for v4 or v6, because due to hardware
    Transit Offload, the cost of missing a tx-op exceeds the benefit of
    making the packet slightly bigger.  I suspect that this is true in
    general to UDP and QUIC traffic too.
    I imagine a next generation 10G NICs might offer QUIC offload, including
    doing the crypto.  That's what I'd be coding if I worked in that space.

2) I will admit that I personally don't care that much about UDP traffic,
   except in that it lets me run IPsec through NAT44s.   I know that I use
   QUIC and WebRTC regularly, and that corporate enterprise users gets
   screwed by lack of UDP regularly.

So I think question is: is there really a problem that needs to be solved?
Maybe I will have to back and read the beginning of this thread again to
recall what the issue was.   Does this belong in SPUD?

I wish that SCTP had flown higher...

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 16 Jan 2019 19:12:01 -0500

Just to clarify one aspect of the way entropy in path selection, I want 
to point out a complication.

It is not anywhere near enough to have as much entropy data as the 
number of choices.  The problem is that you need enough randomness so 
that you can expect a good distribution of flows.  And that even the 
smaller number of larger flows will likely get distributed across the 
choices.    Reducing the amount of available entropy can be quite 
problematic.

Yours,
Joel

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 Jan 2019 13:30:23 +1300

> choices.    Reducing the amount of available entropy can be quite 

Right. And for the server farm case, I don't think it's science fiction
these days to think about hundreds or thousands of servers. Also, if the
load sharing algorithm attempts to ensure that a given server has only
one big job at a time, then a high collision rate in the hash can 
defeat it. A form of the birthday paradox applies: not "what is the
chance of a clash per flow" but "what is the chance that out of a
thousand servers, one of them gets two big jobs at the same time"?

I am strongly against breaking the flow label just at the time when
the major o/s are starting to set it correctly.

I'm all for fixing the fragmentation problem; draft-ietf-intarea-frag-fragile
exists for a reason. But not by breaking something else.

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Wed, 16 Jan 2019 16:39:19 -0800

> exists for a reason. But not by breaking something else.
Agreed. Turning up flow labels was one of few things we were able to
do immediately at scale and doesn't seem to have caused problems.
Unfortunately, there aren't any available reserved bits in the IP
header and no bits that could easily be purposed (even if there were
it would a major discussion about what use cases warrant allocation of
bits in the primary IP header). IPv6 is extensible by virtue of
extension headers, that is where things like this belong.

Tom

---

From: Michael Richardson <[email-masked]>
Date: Wed, 16 Jan 2019 21:39:41 -0500

> thousand servers, one of them gets two big jobs at the same time"?

Based upon my reading of the netflix blogs, they have experiemented
extensively with the load sharing, and they really don't care about
flow-labels in their decision process. (Of course, because IPv4 has
no such things)

It's about how fresh the (disk read) caches on the servers are, what content
is being streamed, and other things that have nothing to do with the
packets themselves.

Architecturally with IPv6, if you have an entire /64 (or more) to play with
and you can statelessly forward packets at wire speed,  then there are
other interesting off-path choices one can do.  (For instance, assign
new server/128 for each client connection, and then when the connection
arrives, dynamically map it to a particular server.  This pushes the state
storage from layer-4 to the neighbour cache, which might not be a win)

So I seriously question whether any of this matters to server farms.

:-)

    > exists for a reason. But not by breaking something else.

My quick read says that it looks great to me.

Again, I don't really think that using the flow label to seed PLPMTUD
is much of a win, but if it did provide something useful, I think it could be
done without too much harm.

To reiterate: I don't think the benefit is high enough to warrant the
   risk, despite the fact that I don't think the risk is as high as you
   are suggesting.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 Jan 2019 15:56:28 +1300

> flow-labels in their decision process. (Of course, because IPv4 has

Indeed, but that's exactly why we brought in a load sharing expert
to help us with RFC7098. And there are residual problems even in the
ideal world where the flow label is perfect. We played with some ideas
in https://tools.ietf.org/html/draft-tarreau-extend-flow-label-balancing
but it didn't really go anywhere. In a nutshell, what's really needed
is a bidirectional session ID, not a unidirectional flow ID. And
that's not a layer 3 concept.

   Brian

---

From: Mark Smith <[email-masked]>
Date: Thu, 17 Jan 2019 14:10:39 +1100

> > to point out a complication.

<snip>

> the major o/s are starting to set it correctly.

It would be being set quite widely now.

OS X and Linux (courtesy of Tom Herbert) have set it by default for a
number of years.

The Windows 10 Creator Update released in July of 2017 started setting it

https://blogs.technet.microsoft.com/networking/2017/07/13/core-network-stack-features-in-the-creators-update-for-windows-10/

And Microsoft have said that as of June last year Windows 10 was on
700 million PCs.

https://windowsreport.com/windows-10-market-share-2018/

So there would have to be be 100s of millions of devices now setting
the flow label per RFC 6437.

---

From: Erik Kline <[email-masked]>
Date: Wed, 16 Jan 2019 19:17:51 -0800

> that's not a layer 3 concept.

The potential utility of an operational convention that treats the
flowlabel akin to the TCP MSS field would seem to me to be much higher
than trying to help internet hyper giants with their load balancing.
All the more so if the load balancing is better solved with info from
layers above L3 (going to need to load balance on QUIC connection IDs
now, too).

(Writing a patch to the Linux kernel to stamp the MTU of the outgoing
interface into the flow label doesn't look like it's too hard.)

---

From: Mark Smith <[email-masked]>
Date: Thu, 17 Jan 2019 14:27:32 +1100

> that's not a layer 3 concept.

I think really what you want is an anycast IPv6 service address in DNS
for the load balanced service that the client uses to establish the
initial transport layer connection, and then a method to announce to
the client and then hand off that session to the unicast address of
the server actually handling the session. That would make the load
balancer with the anycast service address more of a session broker
rather than something that is inline with all the sessions' traffic.

Multipath TCP would fit the bill, and I assume the multipath
extensions for QUIC will too.

---

From: Tom Herbert <[email-masked]>
Date: Wed, 16 Jan 2019 20:35:30 -0800

Mark,

Alternatively, a new record type could be added to DNS that returns
blocks of IPv6 address (could be called BBBB records). The BBBB record
would be something like a base IPv6 address and an extent. Given the
enormous size IPv6 addresses, a single record could contain billions
of addresses for a service. The client just needs to pick address in
the blaock at random, and load balancing to backend servers is
accomplished by routing to back end servers solely based on
destination address (each backend server serves some portion of the
address block). No need for VIPs, anycast, DPI into the transport
layer, or stateful load balancing. This also has the advantage of
introducing a lot more bits of entropy for other load balancing
techniques like ECMP (using a different IPv6 source IP address for
every connection would have a similar effect).

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 Jan 2019 17:36:14 +1300

Hi Erik,

> layers above L3 (going to need to load balance on QUIC connection IDs

The flow label is a proxy for the information above layer 3. As we
know, that information is unavailable in many cases, because of
encryption, intervening extension headers, or fragmentation.

Certainly a QUIC sender must set the flow label; RFC6437 is agnostic
about the transport layer. It says so explicitly, written years
before QUIC was invented:

"  Additionally, if classifiers depend only on IP-layer headers, later
   introduction of alternative transport-layer protocols will be easier."

> interface into the flow label doesn't look like it's too hard.)

No, but it would violate the relevant IETF standards.

   Brian

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 17 Jan 2019 05:48:47 +0000

There is a lot of complexity in injecting entropy into IP addresses. As long as using a 4-tupple for TCP and UDP traffic, 2-tupple+SPI for IPSec, 2-tupple+Key for GRE, etc. "just works", there is little incentive to deploy complex solutions that involve DNS, neighbour caches, etc. The hope for new protocols on top of ipv6 is the flow label, because it is so simple to use right by everyone: the sender, the receiver, and the middle boxes.

- Igor

-----Original Message-----
From: Tom Herbert [[email-masked]]
Received: Wednesday, 16 Jan 2019, 11:36PM
To: Mark Smith [[email-masked]]
CC: Michael Richardson [[email-masked]]; IPv6 List [[email-masked]]
Subject: Re: Never fragment: getting PMTU info transmitted reliably

Mark,

Tom

---

From: Tom Herbert <[email-masked]>
Date: Thu, 17 Jan 2019 07:12:09 -0800

On Wed, Jan 16, 2019, 9:48 PM Lubashev, Igor <[email-masked] wrote:

> There is a lot of complexity in injecting entropy into IP addresses. As
> long as using a 4-tupple for TCP and UDP traffic, 2-tupple+SPI for IPSec,
> 2-tupple+Key for GRE, etc. "just works", there is little incentive to
> deploy complex solutions that involve DNS, neighbour caches, etc. The hope
> for new protocols on top of ipv6 is the flow label, because it is so simple
> to use right by everyone: the sender, the receiver, and the middle boxes.
>

Igor,

I'm not sure I'd say that it "just works". Take a look at Maglev load
balancer (
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf)
and all the complexity needed to ensure consistent routing to backends, and
even with all that it still will drop some number of connections when
backends change.

Using flow label or transport layer information is only best effort for
persistence. Intermediate devices are allowed to change devices flow
labels, NAT evictions of UDP may change the client side ports and address
mid-flow. QUIC is interesting since it doesn't use the IP addresses or
ports for identifying connections. That conceptually allows a client to
purposely randomize both the source address and the port on every packet to
make it hard for intermediate devices to correlate packets as belonging to
the same flow (for security).

Tom

- Igor
> *From:* Tom Herbert [[email-masked]]
> *Received:* Wednesday, 16 Jan 2019, 11:36PM
> *To:* Mark Smith [[email-masked]]
> *CC:* Michael Richardson [[email-masked]]; IPv6 List [[email-masked]
> ]
> *Subject:* Re: Never fragment: getting PMTU info transmitted reliably
>
> > > in
> https://tools.ietf.org/html/draft-tarreau-extend-flow-label-balancing

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 17 Jan 2019 17:58:54 +0000

On Thursday, January 17, 2019 10:12 AM Tom Herbert <[email-masked]> wrote:

> On Wed, Jan 16, 2019, 9:48 PM Lubashev, Igor <mailto:[email-masked] wrote:
> balancer
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__static.googleusercontent.com_media_research.google.com_en__pubs_archive_44824.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=Ov_Q64EPg0L4YTIJM46mUQ1XCRAUa2erO899x4BVQK8&s=HmQ5z09TFfKB0FmiHCgAgj6I_xnkpP6de0-529U28kw&e=)

I am familiar with Maglev.  In fact, I run a load balancer system whose backend pool changes every few seconds.  To preserve connections, you either needs to do some clever tricks and/or keep per-connection state (our load balancer for TCP is mostly stateless).  It would not matter what part of the packet contains your load balancing connection key: IP, ports, QUIC CID, IPSec SPI, etc.  You either find a way for the server to pass a token to the client to include in each of their packets or you keep a per-connection state on the load balancer.

Moreover, I would argue that any method that has the client select a load balancing connection key (an ephemeral src port or less significant bits of IP address) is going to require load balancer to keep per-connection state or drop some connections on a backend pool change.  This is exactly why IETF QUIC has packets from clients feature a server-supplied Connection ID (after handshake completion).

> persistence. Intermediate devices are allowed to change devices flow labels,
> NAT evictions of UDP may change the client side ports and address
> mid-flow.

You can also add IP fragmentation to the list.  NAT rebinding is a potential problem for UDP-based protocols, indeed (hence QUIC is trying to mitigate it).

I do not think intermediary devices are allowed to change flow labels, though, since RFC 6437 forbids this: "Once set to a non-zero value, the Flow Label is expected to be delivered unchanged to the destination node(s).  A forwarding node MUST either leave a non-zero flow label value unchanged or change it only for compelling operational security reasons as described in Section 6.1." (Off topic: "Section 6.1" makes me amused that it supposes that networks worried about covert exfiltration of information by the sender will allow packet transmissions by that sender sans the flow label.)

> QUIC is interesting since it doesn't use the IP addresses or ports
> for identifying connections.

QUIC allows either endpoint to provide a 0-length connection ID, effectively using IPs and ports for connection identification.  I believe Chromium does (did?) just that.  Moreover, a change in a 4-tuple during QUC handshake is likely to kill that handshake.

> That conceptually allows a client to purposely
> randomize both the source address and the port on every packet

That would not work too well for QUIC (at least the current version), since QUIC requires path validation (QUIC does not want to be a vector for reflection attacks).  The exact details of connection migration and validation are not completely settled, yet, but what is certain is that while frequent connection migration should work, it would need to be much less frequent than every packet.

- Igor

---

From: Christian Huitema <[email-masked]>
Date: Thu, 17 Jan 2019 11:02:38 -1000

On 1/17/2019 5:12 AM, Tom Herbert wrote:

> Using flow label or transport layer information is only best effort
> for persistence. Intermediate devices are allowed to change devices
> flow labels, NAT evictions of UDP may change the client side ports and
> address mid-flow. QUIC is interesting since it doesn't use the IP
> addresses or ports for identifying connections. That conceptually
> allows a client to purposely randomize both the source address and the
> port on every packet to make it hard for intermediate devices to
> correlate packets as belonging to the same flow (for security).

That's not what the current QUIC spec says. The nodes monitor the
addresses and ports from which their peer is sending data. A change of
source address is identified as a potential NAT rebinding, or migration.
It triggers a series of address validation probes, and then migration of
the session to that new address if it is validated. Changing address on
each packet would be some kind of attack against this mechanism. (See
section 8 of the QUIc transport spec,
https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#address-validation).

-- Christian Huitema