Subject: Re: AD review of draft-ietf-6man-flow-3697bis

From: RJ Atkinson <[email-masked]>
Date: Tue, 21 Jun 2011 08:51:11 -0400

Earlier, Brian Carpenter wrote:
> As far as routers go, I think we have to say that an implementor has
> to choose between a reassembly-based solution using the 5-tuple and
> simply using the 2-tuple (maybe also the fragmentation ID - there is
> some scope for ingenuity here).

OBSERVATION:
A deployed IPv6 router will encounter many more non-fragmented 
packets than fragmented packets.  

RECOMMENDED APPROACH:

For the case of non-fragmented packets, use of the full 5 input
parameters ought to be mandated.

For the case of fragmented packets, use of reduced inputs that
are available in the IPv6 header alone should be permitted
as an option for implementers.

Yours,

Ran

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 22 Jun 2011 01:44:44 +1200

> parameters ought to be mandated.

I'd go for RECOMMENDED; we really should leave some scope for
implementors I think.

    Brian

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Jari Arkko <[email-masked]>
Date: Tue, 21 Jun 2011 17:13:51 +0300

Ran and others,

> as an option for implementers.

I think you are making it easier than it really is.

Here's the problem.

The first packet arrives and its small. The router can read the 5-tuple. 
All fine so far. Picking

    flowid = f(5-tuple)

The next packet arrives, only this time it is fragmented. The router 
cannot read the 5-tuple from all fragments. Now, what is the router 
supposed to do? Here are its options:

1. Reassemble to find the 5-tuple information. All is fine, except that 
its a heavy implementation burden (and a DoS vulnerability).

2. Revert to using 2-tuple for the fragmented packet:

     flowid' = f(2-tuple)

    Now flowid != flowid' This is really bad, as it means that a normal 
packet and a fragmented packet for the same flow would have different 
flowids, and might take different routes, be re-ordered, etc.

3. Do some magic matching of fragment identifiers. Again, we can find 
the 5-tuple, but now the router has to remember per-session state 
(another DoS vulnerability).

Or am I missing something? I think the first-hop router has to set the 
flowid in every packet of the flow, doesn't it?

Jari

---

From: RJ Atkinson <[email-masked]>
Date: Tue, 21 Jun 2011 11:00:53 -0400

On 21  Jun 2011, at 10:13 , Jari Arkko wrote:
> The next packet arrives, only this time it is fragmented.
> The router cannot read the 5-tuple from all fragments. Now,
> what is the router supposed to do? Here are its options:
> 
> 1. Reassemble to find the 5-tuple information. All is fine,
> except that its a heavy implementation burden (and a DoS vulnerability).

A router that I built would not try to reassemble transit traffic.

> Now flowid != flowid' This is really bad, as it means that
> a normal packet and a fragmented packet for the same flow
> would have different flowids, and might take different routes,
> be re-ordered, etc.

We agree it is undesirable, but I would not write "really bad".

I think we differ on whether it is practical to avoid having 
different Flow Label values for fragmented vs non-fragmented 
packets of a given flow *in the particular case* where the 
originating node sent the packet with zeros in the Flow Label 
field.
  
In the approach I outlined, for some given flow where the 
originating node did not set a Flow Label value, the 
non-fragmented packets all will have some Flow Label value A, 
while fragmented packets all will have some Flow Label value B.  
With high probability, (A != B).   Still, this outcome is not 
obviously worse than having all of those packets with a Flow Label 
value of zero.

We might also disagree on how common fragmented IPv6 packets are 
in the deployed world, I'm not sure.  In my experience they are 
uncommon in the deployed world.  I gather they are very common 
in the small niche of supercomputing, but in that special case 
the number of fragmented packets will dwarf the number of 
non-fragmented packets.  This results in the vast majority of
packets within the flow having the same Flow Label value.

Separately, packet re-ordering can (and routinely does) happen 
in the deployed world already, regardless of contents of the
Flow Label field.  So receiving nodes already have to be able 
to cope with reordered packets.

Also, modern TCP implementations cope MUCH better with re-ordered 
packets than older TCP implementations.  So the performance hit of 
a re-ordered packet is not nearly as drastic as it was (for example) 
15 years ago.

> (another DoS vulnerability).

(A) A router I built would not be trying to keep that per-session state.

(B) Literally every step in packet handling is a potential DOS 
    vulnerability.  It is not possible to avoid DOS vulnerabilities 
    in useful packet networks.  So one ought to be claiming that 
    the DOS vulnerability is lower-cost for the attacker than 
    existing DoS vulnerabilities and also expensive for the victim.  

    [While I do believe those 2 properties hold in this case, it is 
    crucial to examine/discuss those 2 properties for any claim of 
    any DoS vulnerability in a packet network.  If we try to evaluate 
    "DOS vulnerability" on a binary basis, we are unlikely to make 
    scientifically sound decisions.]

There is a 4th implementation option, which is to use only fields 
in the base IPv6 header for all cases where a router adds a Flow Label.  
That implementation option adds little or no value for load-balancing 
as compared with a zero Flow Label, because existing load-balancing 
algorithms in the deployed world already tend to use the variable 
fields of the IPv6 header (e.g. source IPv6 address, destination IPv6 
address, maybe ToS byte) for deployed load-balancing situations.  
For non-fragmented packets, the deployed world already tends to use 
the 5 input values that these I-Ds discuss, by the way.

Part of my point in mentioning that hardware forwarding engines 
(i.e. ASIC, FPGA) are already commonplace, even in very low cost devices, 
was to point out that such forwarding engines can look at any combination 
of any fields in any IP header (and often also in the TCP/UDP/SCTP/ICMP 
headers), with no incremental performance cost, at wire-speed.  In 
at least one IPv6 capable forwarding ASIC implementation that I'm 
very familiar with, this flexibility did not significantly increase 
the gate count and did not increase the die size.  Folks who worked 
on a different widely deployed IPv6 capable forwarding ASIC have told 
me that these properties were also true in their silicon.

IMHO, the vast majority of the benefit to using the IPv6 Flow Label 
for load-balancing accrues to those IPv6 packets that have been 
fragmented where the originating node inserts the non-zero Flow Label 
value based on the documented 5 input parameters.

In the other cases, the 5 values either already can be read 
spontaneously at wire speed by commonly used merchant silicon 
forwarding engines (e.g. for non-fragmented packets) OR the 
after-the-fact inserted Flow Label performs no better than using 
base IPv6 header fields that can be read spontaneously at wire speed 
(for fragmented packets).

By having routers use the usual 5 inputs to generate a non-zero 
Flow Label for all non-fragmented packets that arrive with zero value 
Flow Labels, the common case of a non-fragmented IPv6 packet is
handled a bit better and the less common case of a fragmented IPv6
packet is not really worse off.  After all, a router can't know
from examination of packet N of flow X whether some future packet
N + Delta in the same flow X will or won't contain a Fragment
Header.  

IMHO, Yours,

Ran

---

From: Shane Amante <[email-masked]>
Date: Tue, 21 Jun 2011 22:07:11 -0600

Ran, Jari, All,

To emphasize some of the excellent points that Ran made ...

> to cope with reordered packets.

An example which routinely happens in today's networks would be link restoration.  In that case, the network is restoring traffic from a much longer path to a shorter, more optimal path in the network.  Depending on the transmission rate of the transmitter, this can/will lead to temporary reordering of microflows at the receivers.  Do note that in well-operated networks this reordering is, hopefully, transient and of an extremely short duration.  

> the 5 input values that these I-Ds discuss, by the way.

I think it's important to re-emphasize and expound upon the above.  Namely, deployed IPv6 routers *already* [should] identify fragmented vs. non-fragmented packets, presumably by inspecting the "Next Header field of the last header of the Unfragmentable Part" for value 44 [RFC 2460, Section 4.5] for 'Fragment Header' for the purposes of deciding whether /or not/ they should attempt to identify Next Headers containing the Upper Layer protocol and, subsequently, the {protocol, src_port & dst_port} that will be fed, (along with {src_ip, dst_ip}), into LAG and/or ECMP hash algorithms for fine-grained load-balancing.

"Conservative" router/switch implementations strive to reduce the risk of _persistent_ reordering of an individual microflow.  IOW, since non-first fragments will not contain Upper Layer protocol information, (specifically: {src_port, dst_port}), that can be fed as input-keys to LAG and/or ECMP hash algorithms, the "safe" thing they should do is to only use the 2-tuple of {src_ip, dst_ip} as input-keys for _all_ fragments within a microflow.  Obviously, this leads to 'coarse-grained' load-balancing for microflows containing fragmented packets.

As with many things, Engineering is about properly managing a series of trade-offs.  Currently, the advantage of avoiding persistent reordering of fragmented microflows out-weighs the disadvantage of only being able to perform coarse-grained load-balancing of the assumed very small amount of fragmented microflows.  If this draft is widely implemented & deployed and originating hosts are encoding a "uniformly distributed", non-zero flow-label in all packets (fragmented or not), then it would seem logical that routers would be adapted so that:
a)  If they encounter a Fragment Header they use: {src_ip, dst_ip + flow_label} as input-keys to the LAG + ECMP hash algorithms; and/or,
b)  If they encounter a Next Header with, for example, an Upper Layer Protocol that they have *not* (yet?) implemented a parsing routine to extract appropriate input-keys (or, can't, because it's too deep in the packet's headers), then they revert back to using {src_ip, dst_ip + flow_label} as input-keys to the LAG + ECMP hash algorithms[1]; and/or,
c)  [Assuming widespread use of the flow-label], they no longer even bother looking at any Next Headers in all packets and _always_ use {src_ip, dst_ip + flow_label} for input-keys to LAG + ECMP hash algorithms.

Personally, I see (a) & (b) as being a short- to medium-term "wins" that could be safely implemented, by default, in the next-spin of NP, FPGA SW and ASIC HW, given the existence of this, hopefully soon, RFC.  Obviously, (c) is going to be a little further out.  I assume that, similar to today's router implementations, router vendors will likely provide the flow-label as yet another input-key that may be used as input-keys for LAG + ECMP hash algorithms.  It will then be up to individual operators to determine the appropriate time to configure their routers/switches to, for example, only use: {src_ip, dst_ip + flow_label} when they are comfortable doing so for all traffic.

> value based on the documented 5 input parameters.

+1 in the short- to medium (?) term.  I would also point out a substantial additional advantage is [long-term] architectural flexibility in that the end-points (hosts) may freely use *new* transport protocols (SCTP, DCCP, UDP-lite, etc.) so long as they continue to label all packets with a "uniformly distributed", non-zero flow-label so that [Core] routers/switches have something they can safely use as input-keys for LAG and/or ECMP hash algorithms.  At least, that's one part of the network that we don't need to worry about upgrading to support new transport-layer protocols.  Unfortunately, middleboxes (FW's or, more generally, "security GW's") might still have to be adapted depending on the applicability of the new transport-layer protocol to various network types, (e.g.: SOHO vs. Large-ish Enterprise).

Thanks,

-shane

[1] One example I can think of here is UDP-lite.  Silly as though it may seem, (since the format of UDP and UDP-lite headers are nearly identical), parsing routines to extract {src_port, dst_port} from UDP-lite headers are not [widely] implemented in deployed equipment, today, because it is assumed this isn't a widely used transport-layer protocol.  Depending on the implementation (ASIC, NP or FPGA), they might be adapted to recognize UDP-lite, but that's a lot of cost & work ... *just* for one additional transport-layer protocol!

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 22 Jun 2011 06:57:53 -0400

On 22  Jun 2011, at 00:07 , Shane Amante wrote:
> An example which routinely happens in today's networks would be link restoration.  
> In that case, the network is restoring traffic from a much longer path to a
> shorter, more optimal path in the network.  Depending on the transmission rate
> of the transmitter, this can/will lead to temporary reordering of microflows
> at the receivers.  Do note that in well-operated networks this reordering is,
> hopefully, transient and of an extremely short duration.  

Note that similar to the above, having first packet in a flow
travel a slightly different path than subsequent fragments
due to ECMP/LAG differences is also predictably "transient" and
"of an extremely short duration" (to use Shane's excellent phrasing).

> Namely, deployed IPv6 routers *already* [should] identify fragmented
> vs. non-fragmented packets, presumably by inspecting the "Next Header
> field of the last header of the Unfragmentable Part" for value 44
> [RFC 2460, Section 4.5] for 'Fragment Header' for the purposes of
> deciding whether /or not/ they should attempt to identify Next Headers
> containing the Upper Layer protocol and, subsequently, the
> {protocol, src_port & dst_port} that will be fed, (along with
> {src_ip, dst_ip}), into LAG and/or ECMP hash algorithms for
> fine-grained load-balancing.

Exactly so.

Side comment:	Because many users/operators require that routers implement
		wire-speed Access Control Lists, and most of those ACLs 
		include variables such as transport-layer protocol, 
		and transport-layer port numbers, ASIC/FPGA-based IPv6
		routers will continue to parse into the IPv6 packet
		beyond the IPv6 base header *regardless* of whether
		that information is needed for ECMP/LAG/load-balancing
		purposes.

	I have heard credible reports of an ISP that apparently deploys 
	hundreds of ACL rules into the exterior border routers of their AS, 
	and apparently that ISP deploys a smaller but still non-trivial 
	number into interior routers.  

	Of course, this also means they select their largest routers in part 
	upon the routers' ability to support large numbers of ACLs at full 
	wire-speed and also the ability to look past the occasional IPv6 
	extension header to examine the ICMP/TCP/UDP/etc headers behind it.

> "Conservative" router/switch implementations strive to reduce the risk of
> _persistent_ reordering of an individual microflow.  IOW, since non-first
> fragments will not contain Upper Layer protocol information, (specifically:
> {src_port, dst_port}), that can be fed as input-keys to LAG and/or ECMP hash
> algorithms, the "safe" thing they should do is to only use the 2-tuple of
> {src_ip, dst_ip} as input-keys for _all_ fragments within a microflow.  
> Obviously, this leads to 'coarse-grained' load-balancing for microflows
> containing fragmented packets.

Yes, and an important word above is "persistent", to echo Shane's emphasis.

As Shane observes, this implementation approach is common today.  I believe 
some implementations today already include the Flow Label, simply because 
it is another available differentiator (albeit most commonly zero just now), 
not because of this current set of specifications.

> If this draft is widely implemented & deployed and originating hosts are
> encoding a "uniformly distributed", non-zero flow-label in all packets
> (fragmented or not), then it would seem logical that routers would be
> adapted so that:
> 
> a)  If they encounter a Fragment Header they use: {src_ip, dst_ip + flow_label}
>     as input-keys to the LAG + ECMP hash algorithms; and/or,
> 
> b)  If they encounter a Next Header with, for example, an Upper Layer Protocol
>     that they have *not* (yet?) implemented a parsing routine to extract
>     appropriate input-keys (or, can't, because it's too deep in the packet's
>     headers), then they revert back to using {src_ip, dst_ip + flow_label}
>     as input-keys to the LAG + ECMP hash algorithms[1]; and/or,
> 
> c)  [Assuming widespread use of the flow-label], they no longer even bother
>     looking at any Next Headers in all packets and _always_ use {src_ip,
>      dst_ip + flow_label} for input-keys to LAG + ECMP hash algorithms.
> 
> Personally, I see (a) & (b) as being a short- to medium-term "wins"
> that could be safely implemented, by default, in the next-spin of NP,
> FPGA SW and ASIC HW, given the existence of this, hopefully soon, RFC.  

Agreed.  In some cases, I think (a) and (b) are already deployed.

> Obviously, (c) is going to be a little further out. 

I would apply the caveat from earlier to (c).  

Routers will still be looking beyond the IPv6 header *for ACL purposes*, 
even if those same routers use non-zero Flow Labels for ECMP/LAG, 
rather than using transport-layer information for ECMP/LAG.

> I would also point out a substantial additional advantage is [long-term]

> architectural flexibility in that the end-points (hosts) may freely use

> *new* transport protocols (SCTP, DCCP, UDP-lite, etc.) so long as they

> continue to label all packets with a "uniformly distributed",

> non-zero flow-label so that [Core] routers/switches have something

> they can safely use as input-keys for LAG and/or ECMP hash algorithms.  

With respect to ECMP/LAG narrowly, I agree with the above.

> At least, that's one part of the network that we don't need to worry
> about upgrading to support new transport-layer protocols.  Unfortunately,
> middleboxes (FW's or, more generally, "security GW's") might still have
> to be adapted depending on the applicability of the new transport-layer
> protocol to various network types, (e.g.: SOHO vs. Large-ish Enterprise).

Exactly so.   Router-based ACLs (which are widespread -- even in 
some deployed transit/backbone routers) will still need to support those 
new transport-layer protocols BEFORE those new transport-layer protocols 
will be practical to widely deploy.  

Yours,

Ran Atkinson

PS: For implementations prior to this current set of documents, and for 
    routers creating Flow Labels on the fly, a useful additional input 
    to an ECMP/LAG function would be the "SPI" value of an ESP or AH header.

---

From: Thomas Narten <[email-masked]>
Date: Mon, 27 Jun 2011 15:49:42 -0400

> obviously worse than having all of those packets with a Flow Label 

IMO, it is worse. It means that packets with the different Flow Labels
may take different paths, and thus arrive at the destination out of
order relative to each other. This is known to cause problems with
TCP. Sure, implementations may (generally) be able to handle this
better today than in the past, it is not the case that reordering
packets has no impact. It will depend on various factors...

Setting the Flow Label using only the 2-typle for all packets would
prevent this reording. (It would have other undesirable effects of
having the load balancing not work ideally, but that is a different
issue.)

That said, I agree that there is no easy way to end up with the same
flow label value for both fragmented and unfragmented packets (unless
set by the source). I also agree that having routers reassemble
packets in order to generate proper Flow Ids would be a silly
requirement (or suggestion) and no router would implement that.

I think we are stuck with having different flow ids being generated
for fragmented vs. unfragmented packets.

Thomas

---

From: RJ Atkinson <[email-masked]>
Date: Mon, 27 Jun 2011 16:44:50 -0400

On 27  Jun 2011, at 15:49 , Thomas Narten wrote:
> order relative to each other.

This is ALSO true for packets containing an all-zero Flow Label.

Existing deployed routers typically go read the transport-protocol
and transport-layer ports, and load-balance based on all 5 values 
(IP addresses, transport protocol, plus transport-layer port numbers).  
If not all 5 values are accesssible, for example due to presence of 
a Fragmentation Header, then existing deployed routers typically use 
the 2 values available (i.e. IP addresses).

For the case where the Flow Label is zero, I fully expect that routers
will use the currently deployed behaviour.  So having 2 different
Flow Label values in the case I describe at top yields **identical**
behaviour as today -- therefore "not obviously worse" in the quote
at top.

> This is known to cause problems with TCP. Sure, implementations
> may (generally) be able to handle this better today than in the past,
> it is not the case that reordering packets has no impact. 

Historically, back in the days of 4.3 BSD TCP performance was seriously 
harmed by packet reordering.  Modern TCP implementations, however,
are more robust.  So the performance impact of small amounts of packet 
reordering (which is the case here) ranges between minor and ignorable.

> Setting the Flow Label using only the 2-typle for all packets
> would prevent this reordering.

No.  It would not.  Packet reordering will still occur from time to time
in the deployed Internet -- no matter what the Flow Label values might be.

> for fragmented vs. unfragmented packets.

I'm glad we ended up the same place, although we clearly followed
different reasoning to get there. :-)

Cheers,

Ran

---

From: Thomas Narten <[email-masked]>
Date: Mon, 27 Jun 2011 17:08:56 -0400

> behaviour as today -- therefore "not obviously worse" in the quote

OK. We are in agreement on the technical points. But I disagree that
just because this is what happens today (with the limited IPv6 usage
at hand), it is fine. We don't have enough experience with IPv6 usage
today to really say that this is fine.

Your use of the term "not obviously worse" is a bit misleading, IMO,
since we have almost zero experience with the IPv6 Flow Label today.

Anyway, we are on the same page in terms of what we can do about it.

Thomas

---

From: Jari Arkko <[email-masked]>
Date: Tue, 28 Jun 2011 01:40:38 +0200

I still have an uneasy feeling about the changing flow IDs across the 
same TCP session. It feels wrong.

That being said, Ran's argument that different classifications for 
fragmented/non-fragmented packets already happening for load-balancing 
reasons (and presumably even for IPv4) when a fragmented packet hits a 
backbone router and the port numbers are not in the packet.

I'm fine with Brian's suggested text. I would personally open the text 
up a bit more, however. I would explain, for instance, why RFC 4311 is 
relevant. I would explain the tradeoffs between packet reassembly, 
different flow IDs, and using just 2-tuples.

Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jun 2011 08:20:05 +1200

> different flow IDs, and using just 2-tuples.

OK, I have to provide more words, clearly... it may take another day or two.

   Brian