Subject: [v6ops] IPv6 fragmentation experience

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Mar 2018 11:22:22 +0000

I am listening to the fragmentation discussion in the V6OPS meeting in
London. Here is a small related experience:
https://huitema.wordpress.com/2018/03/03/having-fun-and-surprises-with-ipv6/.

The symptom was, "why are you sending QUIC packets in the wrong order".
The root cause was, packet was fragmented, fragments were received in
different order than non fragmented packets.

-- Christian Huitema

---

From: Tom Herbert <[email-masked]>
Date: Mon, 19 Mar 2018 04:44:07 -0700

> different order than non fragmented packets.
Christian,

That will be common in the presence of ECMP or RSS. There's no way to
include ports when a fragment is being forwarded so these fallback to
doing ECMP based on 3-tuple hash only for fragments which means
fragmented packets produce a different hash then 5-tuple would and
hence take a different path. In order to keep fragements in order, if
MF is set ECMP should not use the ports for the first fragment even
though they are present.

The fix for this is IPv6 is to use the flow label in the ECMP/RSS hash
and ignore the ports (hash over addrs and flow label). For a flow the
same flow label should be used for fragments and non-fragments so they
should be routed the same way.

Tom

> -- Christian Huitema
>
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Fernando Gont <[email-masked]>
Date: Mon, 19 Mar 2018 12:46:21 +0000

> https://huitema.wordpress.com/2018/03/03/having-fun-and-surprises-with-ipv6/.

Fragmentation experience with IPv6: RFC7872.

Short version of it: it fails on the IPv6 Internet. And the same is true
for other EHs (yes...that unfortunately includes IPsec-related).

Cheers,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Joe Touch <[email-masked]>
Date: Mon, 19 Mar 2018 07:14:47 -0700

> On Mar 19, 2018, at 4:44 AM, Tom Herbert <[email-masked]> wrote:
> 
> include ports when a fragment is being forwarded

UDP options provide a workaround - allowing the app to transmit in UDP-sized chunks while allowing UDP to source fragment at the transport layer, which includes the conventional UDP header in each fragment.

That said, I agree with the conclusion that this is what IPv6 flow labels were intended to also support.

Joe

---

From: Fred Baker <[email-masked]>
Date: Mon, 19 Mar 2018 16:41:31 +0000

> On Mar 19, 2018, at 11:22 AM, Christian Huitema <[email-masked]> wrote:
> 

The question I would ask is "why would QUIC care? IP is a best effort network, and will therefore reorder some percentage of traffic".

---

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Mar 2018 21:33:16 +0000

> The question I would ask is "why would QUIC care? IP is a best effort network, and will therefore reorder some percentage of traffic".
The Quic trial actually succeeded. The packets were resent, etc. But in
that specific case the recovery used a strange code path, because the
out of order packet was encrypted with a key negotiated in the
"previous" packet. In any case, out of order transmission is not helpful.

-- Christian Huitema

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 19 Mar 2018 22:56:08 +0000

Christian Huitema wrote:
> In any case, out of order transmission is not helpful.

this depends on the use case: there are plenty of situations on
bandwidth-constrained edge networks where it is useful, e.g. where
transmission of a single large mtu packet takes an appreciable amount of
time.  In general, o-o-o transmission is not useful in network core.

Nick

---

From: Fred Baker <[email-masked]>
Date: Tue, 20 Mar 2018 08:30:13 +0000

> On Mar 19, 2018, at 9:33 PM, Christian Huitema <[email-masked]> wrote:
> 

No doubt. But the Internet is a Best Effort Network; by definition, it can drop packets, it can duplicate packets, and it can reorder packets. It is nice when they arrive in order, because that is the usual case and the simplest code path in the receiver. But complaining about reordering is a fool's errand. It happens.

---

From: Gert Doering <[email-masked]>
Date: Tue, 20 Mar 2018 09:37:21 +0100

Hi,

On Tue, Mar 20, 2018 at 08:30:13AM +0000, Fred Baker wrote:
> No doubt. But the Internet is a Best Effort Network; by definition,
> it can drop packets, it can duplicate packets, and it can reorder
> packets. It is nice when they arrive in order, because that is the
> usual case and the simplest code path in the receiver. But complaining
> about reordering is a fool's errand. It happens.

It does.  But the point I found interesting here is that it did not
happen "every now and then" but for every single connection setup, which
makes this peculiar enough to look more closer.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Mar 2018 03:51:17 -0700

> No doubt. But the Internet is a Best Effort Network; by definition, it can drop packets, it can duplicate packets, and it can reorder packets. It is nice when they arrive in order, because that is…
Fred,

That's true, on end hosts the biggest effect of OOO delivery should be
that some protocol optimizations might be lost for those packets-- but
that should never have a major impact on communications. For QUIC
trial, it seems like that probably just uncovered a nice bug.

One cause of OOO delivery is when packets within a flow take a
different path which can lead to problems in itself. This happens in
the case of fragmentation because ECMP might choose different paths
between fragmented and non-fragmented packets. Also happens when ECMP
is based on flow label and the label changes during the flow lifetime.
Per definition of IP, packets taking a different path is not supposed
to be a problem (i.e. the Internet is not circuit switched). But it
does seem to cause problem for stateful firewalls, load balancers, and
other intermediate devices that somehow assume flow semantics in the
path.

Tom

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 20 Mar 2018 11:58:43 +0100 (CET)

On Tue, 20 Mar 2018, Tom Herbert wrote:

> One cause of OOO delivery is when packets within a flow take a different 
> path which can lead to problems in itself. This happens in the case of 
> fragmentation because ECMP might choose different paths between 
> fragmented and non-fragmented packets. Also happens when ECMP is based 
> on flow label and the label changes during the flow lifetime. Per 
> definition of IP, packets taking a different path is not supposed to be 
> a problem (i.e. the Internet is not circuit switched). But it does seem 
> to cause problem for stateful firewalls, load balancers, and other 
> intermediate devices that somehow assume flow semantics in the path.

This is another case of "please don't do that, but others need to 
understand that it sometimes happens, and try to handle it". Same as the 
IP fragments advisory, or PMTUD advisory etc.

Network operators should try to assure that re-ordering doesn't happen, 
and applications need to gracefully handle when this happens anyway.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Lee Howard <[email-masked]>
Date: Tue, 20 Mar 2018 11:06:35 +0000

On 03/20/2018 10:58 AM, Mikael Abrahamsson wrote:
>> is based on flow label and the label changes during the flow 
>> lifetime. Per definition of IP, packets taking a different path is 
>> not supposed to be a problem (i.e. the Internet is not circuit 
>> switched). But it does seem to cause problem for stateful firewalls, 
>> load balancers, and other intermediate devices that somehow assume 
>> flow semantics in the path.
>
> understand that it sometimes happens, and try to handle it". Same as 
> the IP fragments advisory, or PMTUD advisory etc.
>

How would an operator do that?

Lee

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 20 Mar 2018 12:10:59 +0100 (CET)

> How would an operator do that?

Well, one way would to not configure their ECMP to be per-packet, and make 
sure it's per-flow. Yes, there is equipment where this is a thing, and 
people who configure it that way.

So while there are corner cases where re-ordering happens sometimes for 
some traffic, but there are configuration knobs where re-ordering is going 
to happen a lot more frequently. Don't do that.

---

From: Jen Linkova <[email-masked]>
Date: Tue, 20 Mar 2018 22:13:18 +1100

> How would an operator do that?

The trivial example: do not use per-packet load-balancing but use
per-flow instead.
In general - tune their hashing configuration to minimize the chances
of out-of-order packets.

-- 
SY, Jen Linkova aka Furry

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Mar 2018 04:21:09 -0700

> In general - tune their hashing configuration to minimize the chances
Jen,

The problem with that is that not all packets for a flow are
guaranteed to contain the same information or even the necessary
information for computing a flow specific hash. Fragments beyond the
first one don't include transport layer ports for instance.

Tom

---

From: Ole Troan <[email-masked]>
Date: Tue, 20 Mar 2018 11:23:33 +0000

> Well, one way would to not configure their ECMP to be per-packet, and make sure it's per-flow. Yes, there is equipment where this is a thing, and people who configure it that way.
> 
> So while there are corner cases where re-ordering happens sometimes for some traffic, but there are configuration knobs where re-ordering is going to happen a lot more frequently. Don't do that.

I guess this is more dependent on implementation rather than something the operator can control.
A good ECMP implementation should fall back to load-balancing on the 3-tuple instead of the 5-tuple if MF=1 or fragment offset > 0.
Or use only 4-tuple SA, DA, NH and Flow, if flow>0.

Ole

---

From: Sander Steffann <[email-masked]>
Date: Tue, 20 Mar 2018 11:28:06 +0000

Hi,

> A good ECMP implementation should fall back to load-balancing on the 3-tuple instead of the 5-tuple if MF=1 or fragment offset > 0.

Which will cause fragmented and non-fragmented packets to take different paths... It's only stable if you use the 3-tuple (or 4-tuple) all the time.

Cheers,
Sander

---

From: Jen Linkova <[email-masked]>
Date: Tue, 20 Mar 2018 22:32:31 +1100

> first one don't include transport layer ports for instance.

Indeed. There are other cases (besides fragmentations) when packets
might arrive out of order, at least for short period of time.
I do not think we can 100% prevent out-of-order packets.
As I've said, operators should do their best to prevent it in most
cases (== have sane config on the devices).

---

From: Ole Troan <[email-masked]>
Date: Tue, 20 Mar 2018 11:34:25 +0000

Sander,

> Which will cause fragmented and non-fragmented packets to take different paths... It's only stable if you use the 3-tuple (or 4-tuple) all the time.

eh, no?
It will use 3-tuple for all fragments, including the first.

Cheers,
Ole

---

From: Sander Steffann <[email-masked]>
Date: Tue, 20 Mar 2018 11:36:29 +0000

Hi,

> Op 20 mrt. 2018, om 11:34 heeft Ole Troan <[email-masked]> het volgende geschreven:
> 

What about non-fragmented packets in the flow? Not every packet will be fragmented.

Cheers,
Sander

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 20 Mar 2018 12:37:41 +0100 (CET)

> Or use only 4-tuple SA, DA, NH and Flow, if flow>0.

In my example, this isn't even about fragments. There is equipment out 
there that can be configured to do ECMP per-packet, and not even try to do 
flows. Much less try to match fragments to a flow.

---

From: Ole Troan <[email-masked]>
Date: Tue, 20 Mar 2018 11:42:26 +0000

> What about non-fragmented packets in the flow? Not every packet will be fragmented.

Yes, that's true. But the fragments should be kept on the same path, but no guarantee that the set of fragments and non-fragmented would follow same path.

Cheers,
Ole

---

From: Sander Steffann <[email-masked]>
Date: Tue, 20 Mar 2018 11:43:36 +0000

Hi,

> Yes, that's true. But the fragments should be kept on the same path, but no guarantee that the set of fragments and non-fragmented would follow same path.

Indeed, and that can cause out-of-order delivery from the application's point of view.

Cheers,
Sander

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Mar 2018 04:45:08 -0700

> Yes, that's true. But the fragments should be kept on the same path, but no guarantee that the set of fragments and non-fragmented would follow same path.
RIght, and that is going to be a problem for network devices that
expect packets in a flow to always take the same path.

Tom

---

From: Jen Linkova <[email-masked]>
Date: Tue, 20 Mar 2018 22:51:20 +1100

> expect packets in a flow to always take the same path.

What I've failed to understand for far is why do those devices have
such expectations in the first place?
Out-of-order packets will happen, have expectations like 'all packets
on the flow are on the same path and arrive in order' is
unrealistically optimistic.

BTW another thing operators could do to minimize the impact is to
ensure that their 'equal' paths have similar RTTs.

---

From: joel jaeggli <[email-masked]>
Date: Tue, 20 Mar 2018 08:01:37 -0700

> cases (== have sane config on the devices).
The point is not to avoid entirely reordering which is implausible, it's
in general to avoid pathological reordering. A case where you reorder
all initial and non-initial fragments would certainly fall into the case
of being pathological. If you were to apply qos policy based on
transport header policies the non-initial fragments could well end up in
a different queue.

---

From: Bob Hinden <[email-masked]>
Date: Tue, 20 Mar 2018 18:19:28 +0000

> On Mar 20, 2018, at 11:28 AM, Sander Steffann <[email-masked]> wrote:
> 
> Hi,
> 

As well as the first fragment and subsequent fragments taking different paths.  Hence why we wrote the draft.

Bob

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 Mar 2018 18:33:39 -0700

Hi, all,

At some level, we need to understand that "fragmentation is harmful" is
fundamentally incorrect and misleading.

The reality is that "network devices that rely on transport info without
reconstituting it are harmful".

I.e., any device that forwards, filters, translates, folds, spindles, or
otherwise mutilates IP packets based on transport information needs to
either:

    a) act like the transport endpoint it becomes once it relies on
transport info, and reassemble the fragments into a full packet

or

    b) emulate the net effect of (a) by caching initial fragment state
and processing subsequent fragments in that context (perhaps including
holding non-initial fragments until that initial frag creates that state).

Devices that claim to be line-rate forwarders but can't do the above at
line rate are broken and/or lying about their performance, not
"optimized". Optimization works better than the default; the default
case still has to work.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 21 Mar 2018 09:20:51 +0100 (CET)

> "optimized". Optimization works better than the default; the default 

Unfortunately your view of things seem to be shared by only a small 
minority of equipment vendors and customers. It'd also increase cost a lot 
to create devices that do what you say, so that's not going to happen 
anytime soon.

The network isn't perfect, and applications need to handle this 
gracefully. The network should strive to be perfect, but we all know it 
won't always. The pain to the users when it isn't, should be mitigated. 
I'm fine with customer performance degradation when the network is doing 
things wrong, but it shouldn't stop completely.

---

From: Ole Troan <[email-masked]>
Date: Wed, 21 Mar 2018 09:41:45 +0000

Hi Joe,

> "optimized". Optimization works better than the default; the default

For IPv4 (and only IPv4) I do think we have a strong case against fragmentation.
Routing on the transport layer ports is now a fundamental part of the IPv4 Internet architecture.
If the layer 4 information isn't there, then the packet cannot be routed.

See RFC6346, RFC7596, 7597, 7599, et al.

Note that in this architecture any protocol that does not carry ports, i.e. anything which isn't UDP or TCP will not be forwarded.
(So from that perspective you can argue that IPv4 fragments would fare somewhat better than non TCP/UDP traffic. :-))

Cheers,
Ole

---

From: Joe Touch <[email-masked]>
Date: Wed, 21 Mar 2018 06:04:29 -0700

> On Mar 21, 2018, at 1:20 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
>> Devices that claim to be line-rate forwarders but can't do the above at line rate are broken and/or lying about their performance, not "optimized". Optimization works better than the default; the default case still has to work.
> 
> Unfortunately your view of things seem to be shared by only a small minority of equipment vendors and customers.

Vendors are in the business of making money, which sometimes includes misrepresenting their products.

Customers can’t always tell which device is causing the problem or may have no control over that device. I don’t know of many customers who are big fans of NATs, e.g.

The real issue is that the IETF lacks a compliance arm.

> It'd also increase cost a lot to create devices that do what you say, so that's not going to happen anytime soon.

Yes, but it also increases cost to run at line rate for min size packets vs just saying you do and hoping the packets don’t get so small you can’t keep up.

And some grocery stores put sawdust in their ground beef.

Again, the lack of compliance assurance is the issue.

> 
> The network isn't perfect, and applications need to handle this gracefully. The network should strive to be perfect, but we all know it won't always. 

There’s a big difference between tolerating bugs and faulty equipment vs. tolerating behavior that violates our specs. If we ever really try to do the latter, we should pack the IETF and go home.

> The pain to the users when it isn't, should be mitigated. I'm fine with customer performance degradation when the network is doing things wrong, but it shouldn't stop completely.

So how well does your router work when the IP protocol isn’t 4 or 6, or when the EtherID is incorrect? At some point we have to stand behind our specs.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 21 Mar 2018 06:10:54 -0700

> On Mar 21, 2018, at 2:41 AM, Ole Troan <[email-masked]> wrote:
> 

DPI into the packet body is also a fundamental part too.

However, if the lack of layer 4 info creates a situation where a packet can’t be routed, then the device isn’t a router.

Yes, the Internet has many many broken parts. That includes RFCs that are inconsistent with our core architecture requirements.

If anyone in the IAB is listening, you’d earn your middle name better by dealing with those than trying to seek new “fun” stuff on which to pontificate.

Joe

---

From: Alexandre Petrescu <[email-masked]>
Date: Wed, 21 Mar 2018 14:45:48 +0100

Le 21/03/2018 à 14:10, Joe Touch a écrit :
> 
> 
>> On Mar 21, 2018, at 2:41 AM, Ole Troan <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
> However, if the lack of layer 4 info creates a situation where a packet 
> can’t be routed, then the device isn’t a router.
> 
> Yes, the Internet has many many broken parts. That includes RFCs that 
> are inconsistent with our core architecture requirements.

Worse, there are implementations that dont have specifications.

E.g. IPv6-over-802.11 spec does not exist, implementations of it frag at
both IP and MAC layer (e.g. ping -s 2000 issues IP Fragments carried
with MAC frag headers), and people seem to agree it's the wrong thing to do.

Alex

> 
> If anyone in the IAB is listening, you’d earn your middle name better by 
> dealing with those than trying to seek new “fun” stuff on which to 
> pontificate.
> 
> Joe
> 
>

---

From: Lee Howard <[email-masked]>
Date: Wed, 21 Mar 2018 13:49:25 +0000

> DPI into the packet body is also a fundamental part too.

Not generally at scale, and not with many of the new and upcoming 
protocols.
>
> However, if the lack of layer 4 info creates a situation where a 
> packet can’t be routed, then the device isn’t a router.

I don't understand this. I understand you mean the subsequent fragments, 
except that routers as I understand them perform "network layer 
forwarding" [RFC1812, Requirements for IP Version 4 Routers]
Layer 4 is not the network layer; are devices that need L4 info 
considered routers?
> If anyone in the IAB is listening, you’d earn your middle name better 
> by dealing with those than trying to seek new “fun” stuff on which to 
> pontificate.

As I take off several hats, I'm not sure what to ask of the IAB here.

Is it to be the "compliance arm" you mentioned elsewhere in the thread?

Lee

---

From: Gert Doering <[email-masked]>
Date: Wed, 21 Mar 2018 21:04:45 +0100

Hi,

On Wed, Mar 21, 2018 at 06:04:29AM -0700, Joe Touch wrote:
> There???s a big difference between tolerating bugs and faulty equipment vs. tolerating behavior that violates our specs. 

In which way, exactly, does a device "vioulate our specs" (which ones,
exactly?) by routing one packet left and one packet right, based on 
some local decision process how to distribute packets across multiple
links?

If "left" and "right" happen to have very different RTTs, this will
lead to (unintended) reordering, but which standard exactly forbids 
this for IP packets?

Gert Doering
        -- NetMaster

---

From: Joe Touch <[email-masked]>
Date: Wed, 21 Mar 2018 16:09:40 -0700

> Not generally at scale, and not with many of the new and upcoming
Yes, but that's how NAT started too...
> I don't understand this. I understand you mean the subsequent
> fragments, except that routers as I understand them perform "network
> layer forwarding" [RFC1812, Requirements for IP Version 4 Routers]
No, but that's the point. If a router can't forward a fragment *because*
it lacks transport info, then it's not acting like a router.

> Is it to be the "compliance arm" you mentioned elsewhere in the thread?
It is the inconsistency part mentioned above.

E.g., take a look at "TRILL over IP transport" and explain to me what
"slicing" a packet is in the context of TCP as a tunnel.

Or, for that matter, what "IP transport" is.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 21 Mar 2018 16:10:27 -0700

> some local decision process how to distribute packets across multiple
It doesn't when it does that.

It does when it drops a fragment *because* it lacks a transport header.

Joe

---

From: Gert Doering <[email-masked]>
Date: Thu, 22 Mar 2018 11:20:10 +0100

Hi,

> It does when it drops a fragment *because* it lacks a transport header.

Where exactly in this thread was "dropping fragments" mentioned?

Fragmented packets arrived *earlier* than non-fragmented, which caused
surprises.  "Arrive earlier" is sort of the opposite thing from "drop".

Gert Doering
        -- NetMaster

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Mar 2018 08:32:24 -0700

> On Mar 22, 2018, at 3:20 AM, Gert Doering <[email-masked]> wrote:
> 

Sorry - I was assuming that was the problem because — (see next comment)
> surprises.  "Arrive earlier" is sort of the opposite thing from "drop”.

If reordering is a “surprise” then you are not running IP.

Joe

---

From: Jen Linkova <[email-masked]>
Date: Fri, 23 Mar 2018 03:45:15 +1100

> Where exactly in this thread was "dropping fragments" mentioned?

Actually,,, There is one really nasty scenario when hashing is causing
not just reordering but packet loss.
Scenario 1.1: a router R1 is doing ECMP, sends 1st fragment to R2
while other fragments went to R3. If the paths to the destination
(anycast!) on R2 and R3 do not converge, the first and subsequent
fragments might never meet.
Scenario 1.2: a load-balancer using 5-tupes and a number of end
systems behind it. If the first fragment is forwarded by using L4
info, then the chances for the end system to re-assemble will be ~1/N,
where N is number of end systems.

> surprises.  "Arrive earlier" is sort of the opposite thing from "drop".

In some cases they arrive eariler and to the wrong host.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 22 Mar 2018 09:58:09 -0700

Jen,

That's one of the problems with intermediate nodes that attempt to
infer or participate in transport layer semantics. Similar behavior
can happen if a device is using flow label for load balancing and the
flow label changes mid flow, or if there is a routing change an
packets of a flow hit a different load balancer that implements a
different hash. In these cases it's not IP protocol that is broken,
but technically it's incorrect and convenient assumptions being made
within the network that fail under certain circumstances.

Tom

---

From: Jen Linkova <[email-masked]>
Date: Fri, 23 Mar 2018 04:21:29 +1100

On Fri, Mar 23, 2018 at 3:58 AM, Tom Herbert <[email-masked]> wrote:
> infer or participate in transport layer semantics.

I agree it's most unfortunate that almost every intermediate network
device is interested in transport layer semantics
of transit packets. However I'm not sure it can be changed.
Obviously the situations when such behavior is causing the packets
being dropped need to be fixed, and I totally agree
that it's not end points but intermediate devices fault.
On the other hand it's unrealistic to expect that  any modern network
of decent size would be able to operate w/o taking transport layer
into account
(well, there was an awesome idea of deprecating L4 ports and using
IPv6 addresses instead but...;)))

So from operational perspective we might need to think about how to
limit the damage. As an example, network operators might want to look
into
how ECMP and L2 balancing work in their networks and maybe talk to
vendors about adding special treatment for EHs - as well as
reconsidering filtering policies.

The expectations need to be set and documented. I mean, it's
reasonable to expect that an ISP might use L4 for balancing and/or
applying QoS policies (which in MPLS TE networks, for example, might
mean 'your RTT will vary'). Filtering fragments to/from your DNS
servers? Not such a brilliant idea..

>Similar behavior

Exactly why it's a good idea to document most common scenarios and
discuss who we can ...well, maybe not to solve them completely but
make hurt less.

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Mar 2018 17:26:06 -0700

On 3/22/2018 9:45 AM, Jen Linkova wrote:
> (anycast!) on R2 and R3 do not converge, 
(to finish that sentence correctly) then your routing is broken anyway
and all bets are off. Don't blame either fragmentation or ECMP for that
problem.

...
> info,
(again, to finish the sentence correctly) then the load balancer should
either reassemble the fragments or keep enough state to make sure they
fate-share to the same recipient. Otherwise, what you call a "load
balancer" is basically doing random forwarding, which is broken routing,
and again -- all bets are off.

Fragmentation isn't the cause of these problems; broken routing is.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Mar 2018 17:28:49 -0700

On 3/22/2018 10:21 AM, Jen Linkova wrote:
> The expectations need to be set and documented.

On that, I agree.

The expectation is that a device that inspects L4 will preserve L4
semantics - which means that if it starts forwarding on L4, it had
better keep doing so for the rest of the fragments for that L4.

Just saying "that's what everyone does" isn't an answer. Everyone (in
general) disobeys speed limits; that doesn't mean we stop putting them
up or stop enforcing them.

Joe

---

From: Ole Troan <[email-masked]>
Date: Fri, 23 Mar 2018 01:35:58 +0100

Joe,

It seems you are in denial about the consequences for IPv4 with ever increasing address sharing. 

Ole

> On 23 Mar 2018, at 01:28, Joe Touch <[email-masked]> wrote:
> 
> 
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Mar 2018 17:50:27 -0700

> On Mar 22, 2018, at 5:35 PM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 

You are willing to pay money to be lied to. I am not

Joe

---

From: Ole Troan <[email-masked]>
Date: Fri, 23 Mar 2018 10:20:48 +0000

Joe,

> You are willing to pay money to be lied to. I am not

I am the one taking your money and implementing it, and trying not to lie to you in the process.

I have also written some of the specs in this space. I can guarantee you that A+P has a higher fragment drop probability than an 1:1 IPv4 Internet. Given that we do not have that luxury, the resulting decaying IPv4 Internet is what we have to live with. We've told you to move to IPv6 for quite a while now.

And if you are upset about fragments having an increasing drop probability, then please don't try to put anything but 6 or 17 in that protocol field.

The relevant path of the IPv4 forwarding path I've implemented is here:
https://git.fd.io/vpp/tree/src/vnet/map/ip4_map.c#n195

Ole

---

From: Joe Touch <[email-masked]>
Date: Fri, 23 Mar 2018 07:04:32 -0700

> On Mar 23, 2018, at 3:20 AM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 

Here’s what you need to do to stop lying that you’re selling me an IP router (or A+P device):

1) if you look at L4, then you need to either reassemble the packet or cache the front fragment and use it as context for later fragment forwarding.

2) if you can’t keep up with #1, then you need to ignore L4 context for that packet and forward it using only L3

3) remember that ECMP still requires a valid P (path); if a “path” doesn’t go to a destination for a fragment, it’s not a path and should not be in your forwarding table

I’m doing what I can to help reduce the amount of #2 traffic here in the IETF (UDP options) and certainly everyone should be doing some sort of PLPMTUD too. Sure, we all should also try to move to IPv6.

But that doesn’t mean those who sell IPv4 devices get to lie.

We all need to be aware of implementation *errors* and fix them.

Joe