Subject: DRAFT: Request for guidance about the flow label

From: Brian E Carpenter <[email-masked]>
Date: Thu, 22 Apr 2010 14:56:26 +1200

Hi,

Sheng and I would like to continue our attempt to make the flow label
useful. The discussions in Anaheim and on this list have been very
stimulating. I think we need to simplify the change proposed in
draft-carpenter-6man-flow-update-02 even more after the recent
discussions, while maintaining the proposed duality (RFC 3697-like
use still possible, but locally-defined use also possible for those who
want it).

Independently, I expect to continue with draft-hu-flow-label-cases
as background material and with draft-carpenter-flow-ecmp as a
specific RFC 3697 use case, with my co-authors on those two drafts.

The guidance we need from the 6MAN WG is: should we start to draft
rfc3697bis, fixing the issues that have been raised and incorporating
the (simplified) proposal from draft-carpenter-6man-flow-update?

That would also need a new milestone added to the 6man charter.

Regards
   Brian Carpenter

---

From: Rémi Després <[email-masked]>
Date: Thu, 22 Apr 2010 14:50:52 +0200

Le 22 avr. 2010 à 04:56, Brian E Carpenter a écrit :

> I think we need to simplify the change proposed in

In my understanding, restoring domain-entrance FL-values at domain exit should be a "MUST" for any domain-specific use:
- Efficiency of FL-based load balancing requires that FLs are in general different for different 5-tuples.
- For fragmented datagrams, only source hosts are in a position to base FL values on 5-tuples (intra-domain nodes aren't).  

Provided this obligation is complied with, any domain-specific use of FLs is clearly permitted (it remains a purely local matter).

> the (simplified) proposal from draft-carpenter-6man-flow-update?

IMHO, yes.

> That would also need a new milestone added to the 6man charter.

Support for this.

Regards,
RD

---

From: Shane Amante <[email-masked]>
Date: Thu, 22 Apr 2010 09:31:06 -0600

Brian, Remi,

> In my understanding, restoring domain-entrance FL-values at domain exit should be a "MUST" for any domain-specific use:

I'm concerned with Remi's above proposal about restoring FL-values at domain exit.

IMHO, instead of restoring FL-values at domain exit, it is /much/ more simple to declare that Flow-Label values are _mutable_ when a packet crosses over from one administratively defined domain to the next.  I doubt any existing, already deployed high-speed network equipment (routers) would support any scheme to "restore" domain-specific FL-values from entrance to exit of a domain -- although, I haven't conducted a thorough investigation with them; rather, I'm basing this on knowledge of the forwarding architecture of various, **already deployed** (and, will keep getting deployed) platforms whose ASIC's are extremely limited, to say the least.

Of course, declaring the FL as "mutable" would still allow two (or more), "consenting" administrative domains to decide that they want to preserve a special "administrative-domain" flow-label treatment between them and that is fine.  In fact, this would be similar to how the IP DSCP/TC field is dealt with today, particularly in the context of VPN's that span two, or more, administrative domains.

> - Efficiency of FL-based load balancing requires that FLs are in general different for different 5-tuples.

Agreed.

> - For fragmented datagrams, only source hosts are in a position to base FL values on 5-tuples (intra-domain nodes aren't).  

I would instead rephrase this as "sources are in the _best_ position" and we should certainly encourage them to encode a FL value on all packets, (incl. fragmented packets).  However, intermediate nodes/routers can detect a fragmented datagram (by looking for a fragment Ext. header) and once they do so, revert to just using the 2-tuple of IPv6 src + dst address, which is not great, but better than nothing.

Related to the above, I was thinking that a 3967bis may want to recommend the following:
- For "plain" IPv6 packets, (i.e.: non-tunneled, non-encrypted packets), just encode, say, the 3-tuple of {protocol, src port, dst port}, since the source and destination addresses are already available to intermediate nodes.
- OTOH, if it is a tunneled and/or encrypted packet (i.e.: IP/IPv6 or IPSec/IPv6), then the host should encode the entire 5-tuple of the inner IP packet's header in the flow-label, in order that we have the most granularity in the flow-label to distribute those packets as widely as possible over LAG and ECMP paths.

>> the (simplified) proposal from draft-carpenter-6man-flow-update?

Yes, I support that.

>> That would also need a new milestone added to the 6man charter.

Yes, I support that.

-shane

> Regards,
> RD
> 
> 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Rémi Després <[email-masked]>
Date: Thu, 22 Apr 2010 18:50:00 +0200

> IMHO, instead of restoring FL-values at domain exit, it is /much/ more simple to declare that Flow-Label values are _mutable_ when a packet crosses over from one administratively defined domain to…

Would these deployment use domain-specific computations of flow labels?
If yes, have you more information on what they do?
(Any computation of FLs based on 5-tuples in intermediate routers would be well beyond what an asic can do.)

The main reason I see to restore FLs is that, if a domain replaces a "good" FL coming from upstream (host generated and 5-tuple based ), by another that isn't 5-tuple based, routers that are further downstream won't be able to use the "good" FL for their load sharing.

Regards,
RD

---

From: Shane Amante <[email-masked]>
Date: Thu, 22 Apr 2010 13:30:42 -0600

Remi,

I think we may be in agreement that domain-specific uses of the flow-label are problematic for the reason you illustrate below?  Namely, that once one domain sets it to something that is not known to be a 3- or 5-tuple of the IPv6 header, then you lose the ability to use that flow-label for, say, calculating a load-balancing hash for LAG and ECMP paths.  See below for more.

> Would these deployment use domain-specific computations of flow labels?

Since I'm not a proponent of domain-specific, "special-use" flow-labels, that's a question I can't answer, particularly given each domain-specific proposal likely has various amounts of computational complexity.  (I'll save my comments on 'draft-hu-flow-label-cases' for a separate e-mail). 

> (Any computation of FLs based on 5-tuples in intermediate routers would be well beyond what an asic can do.)

How would we expect to implement restoral of domain-specific flow-labels upon entrance or exit of a administratively-defined domain, (hopefully in a _stateless_ manner)?  I'm not sure if this has already been discussed, but the only thing I've come up with is that upon entrance to a domain, the ingress PE would have to "push" the existing IPv6 flow-label into a new, to-be-defined IPv6 Extension Header for transport across the domain.  Then, at exit from the administrative domain, the egress PE would have to be able to "pop" the Extension Header containing the old IPv6 flow-label and put it back into the outermost IPv6 flow-label field.  That's: a) likely difficult/impossible to do in existing HW; b) creates more challenges for routers in dealing with, possibly, unknown Extension Headers; and, c) challenging operationally, particularly if we were to require operators to turn this behavior on/off at each border interface to their network.

> The main reason I see to restore FLs is that, if a domain replaces a "good" FL coming from upstream (host generated and 5-tuple based ), by another that isn't 5-tuple based, routers that are furthe…

I completely agree that this is a challenging problem; however, I see a few ways of potentially dealing with it:
a)  Restoral of domain-specific flow-label values, (your proposal), at egress from an administratively defined domain -- which I find might be impractical to implement; or,
b)  Declare that the flow-label is _mutable_ by each administrative domain.  Therefore, I _MAY_ not trust an incoming IPv6 flow-label from another administrative domain.  However, in thinking about this a bit more, this has problems of it's own, namely: a) the dependency of intermediate routers to (re)calculate a "good" flow-label based off the 3- or 5-tuple in the IPv6 header (possible, but not guaranteed); and, b) it could unintentionally overwrite "good" flow-labels, particularly those of inter-domain IP-in-IPv6 tunnels (cf: draft-carpenter-flow-ecmp), where the outer IPv6 header may have a "good" flow-label based on the inner IP header's 5-tuple -- which is bad.
c)  Declare the flow-label is _immutable_ and must be set by all hosts and must only contain a 3- or 5-tuple hash of the appropriate IPv6 headers.  Further use cases for the flow-label would be restricted.  IMHO, this would be by far the easiest from an implementation and operational point-of-view, however this is unlikely to appeal to proponents of domain-specific special-uses of flow-labels, unfortunately.  

FWIW, related to the last point, (and as I'm sure you're well aware of), I'd also add that we're well beyond the alpha and beta stages of IPv6 deployment and well into mad deployment phase of IPv6 to mitigate IPv4 address exhaustion.  Therefore, if a legitimate use of flow-labels has not been found in the last 15 years of IPv6 development, it's time to move on and use the flow-label for something useful and practical in production networks, (i.e.: a hash of the 3- or 5-tuple of L3 + L4 headers for LAG + ECMP load-balancing).

-shane

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Thu, 22 Apr 2010 15:23:27 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Shane Amante

> c)  Declare the flow-label is _immutable_ and must be set by 
> all hosts and must only contain a 3- or 5-tuple hash of the 
> appropriate IPv6 headers.  Further use cases for the 
> flow-label would be restricted.  IMHO, this would be by far 
> the easiest from an implementation and operational 
> point-of-view, however this is unlikely to appeal to 
> proponents of domain-specific special-uses of flow-labels, 
> unfortunately.  

Count me in. In following this thread, I've come to believe that this is the best course of action. And it keeps the flow label immutable, as originally prescribed.

> Therefore, if a legitimate use of flow-labels has not been 
> found in the last 15 years of IPv6 development, it's time to 
> move on and use the flow-label for something useful and 
> practical in production networks, (i.e.: a hash of the 3- or 
> 5-tuple of L3 + L4 headers for LAG + ECMP load-balancing).

Indeed. This becomes doable easily in IPv6, no matter the extension headers. What better purpose could there be for this field? And it's certainly NOT a stretch to think that the traditional 5-tuple was, in fact, used to define what some might call a "flow."

Bert

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 23 Apr 2010 09:40:43 +1200

(Sorry, I messed up the subject of this thread by including "DRAFT")

On 2010-04-23 07:30, Shane Amante wrote:

<snip>

> I completely agree that this is a challenging problem;
> however, I see a few ways of potentially dealing with it: 

a)
> Restoral of domain-specific flow-label values, (your
> proposal), at egress from an administratively defined domain
> -- which I find might be impractical to implement; or, 

There are ways to do this, but they are complex and
IMHO undesirable.

b)
> Declare that the flow-label is _mutable_ by each
> administrative domain.  Therefore, I _MAY_ not trust an
> incoming IPv6 flow-label from another administrative domain.
> However, in thinking about this a bit more, this has problems
> of it's own, namely: a) the dependency of intermediate
> routers to (re)calculate a "good" flow-label based off the 3-
> or 5-tuple in the IPv6 header (possible, but not guaranteed);
> and, b) it could unintentionally overwrite "good"
> flow-labels, particularly those of inter-domain IP-in-IPv6
> tunnels (cf: draft-carpenter-flow-ecmp), where the outer IPv6
> header may have a "good" flow-label based on the inner IP
> header's 5-tuple -- which is bad. 

However, consider that the flow label is a forgeable field, not
protected by any checksum (including IPSEC). So you can't trust
it and an attacker could always attack your ECMP or LAG by
giving all packets the same label. It may be that the only safe
way is to recalculate the label at a trusted border. (Good luck
doing that at 100 Gb/s.)

So maybe mutability is in fact the only way to make it safe to
use across domain boundaries?

c)  Declare the flow-label
> is _immutable_ and must be set by all hosts and must only
> contain a 3- or 5-tuple hash of the appropriate IPv6 headers.
> Further use cases for the flow-label would be restricted.
> IMHO, this would be by far the easiest from an implementation
> and operational point-of-view, however this is unlikely to
> appeal to proponents of domain-specific special-uses of
> flow-labels, unfortunately.

True, and it's also the current standard, which as far as I
know is not being used.

> 
> FWIW, related to the last point, (and as I'm sure you're well
> aware of), I'd also add that we're well beyond the alpha and
> beta stages of IPv6 deployment and well into mad deployment
> phase of IPv6 to mitigate IPv4 address exhaustion.

Oh yes, we certainly need to enable that, but it doesn't need
any change in the current standards. It's a product issue.

     Brian

---

From: Fred Baker <[email-masked]>
Date: Thu, 22 Apr 2010 15:10:53 -0700

personally, I would prefer to see it *mutable*, and undefined. Given that there are many host implementations out there, I could imagine difficulties getting any specific specification (such as a hash) widely deployed in a finite amount of time. However, if a network could use it to identify an egress or the path to an egress (or, using multiple values per egress, sets of traffic headed to an egress) in a scalable fashion, it might allow for load sharing among what would today be non-sharable MPLS LSPs.

> Indeed. This becomes doable easily in IPv6, no matter the extension headers. What better purpose could there be for this field? And it's certainly NOT a stretch to think that the traditional 5-tupl…

http://www.ipinc.net/IPv4.GIF

---

From: Fred Baker <[email-masked]>
Date: Thu, 22 Apr 2010 15:58:51 -0700

On Apr 22, 2010, at 2:40 PM, Brian E Carpenter wrote:

> protected by any checksum (including IPSEC). 

> use across domain boundaries?

Interesting. I had it in my head that AH protected it.

Mutability, to me, is a good thing. It means that an ISP can use it, as suggested, to carry a load balancing hash, or an egress identifier. Or for that matter an ingress identifier for tracing purposes. I'll have to think about the implications of that.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 22 Apr 2010 21:57:34 -0400

The problem for me is that if it is arbitrarily mutable, then we can not 
use the flow label in a reliable and useful fashion in the ECMP / LAG. 
After all, if it is arbitrarily mutable some ISP might set it to 0xFACE 
because that was useful to them without regard to specific flows.
I would really like to be able to move towards a regime in which the 
flow label is useful for ECMP/LAG, and is actually used, as that 
resolves a number of awkward cases.

yours,
Joel

---

From: Fred Baker <[email-masked]>
Date: Thu, 22 Apr 2010 22:21:48 -0700

It is already mutable, as it turns out. It always was.

On Apr 22, 2010, at 6:57 PM, Joel M. Halpern wrote:

> The problem for me is that if it is arbitrarily mutable, then we can not use the flow label in a reliable and useful fashion in the ECMP / LAG. After all, if it is arbitrarily mutable some ISP might set it to 0xFACE because that was useful to them without regard to specific flows.
> I would really like to be able to move towards a regime in which the flow label is useful for ECMP/LAG, and is actually used, as that resolves a number of awkward cases.
> 
> yours,
> Joel
> 
> Fred Baker wrote:

---

From: Mark Smith <[email-masked]>
Date: Sat, 24 Apr 2010 00:57:52 +0930

On Thu, 22 Apr 2010 15:23:27 -0500

> Count me in. In following this thread, I've come to believe that this is the best course of action. And it keeps the flow label immutable, as originally prescribed.

There's the interesting question. If it was supposed to be end-to-end
immutable, then it should have been covered by an end-to-end checksum,
which in the case of IPv6 means a transport layer checksum, via a
transport layer IPv6 pseudo-header. Even further, as the IPsec AH header
doesn't cover it, then that means that not only isn't it protected
against network faults, it also means it's integrity isn't ever assured
from a 'proper' security perspective. 

Since it wasn't end-to-end protected, then I think that fundamentally
indicates that it is mutable as it traverses the network. I think that
therefore means that you can only trust it's value across a domain upon
which you can trust the network's integrity, and I think that domain
only corresponds to a network which you control and operate, or one
that you are paying to use, with error-free service level agreements.

And there's the interesting quandary. The attractive value of the flow
label is that the only checksum that protects it is the layer 2 checksum
(should there be one). Conversely, the fundamental problem with the flow
label is that it isn't protected by a checksum, other than a layer 2
one. So it's not protected end-to-end, it isn't protected against layer
2 errors when layer 2 doesn't implement checksums, and it isn't
protected against memory errors in routers, even when layer 2 has a
checksum that protects it. 

In other words, similar to IPv6 packets themselves, the flow label
fields is only a 'best effort' field, and can only be trusted within a
domain that can be controlled, or one that the flow label serves the
purpose of a hint, not a value to be absolutely relied upon.

I think there is a fair amount value in the field (verses new extension
headers that could be used for the same purpose), and I think there are
a number of uses for it. We just need to be careful about uses which
place absolute trust in it, outside domains where the trust in it's
integrity can be assumed and assured.

Regards,
Mark.

---

From: Shane Amante <[email-masked]>
Date: Sun, 25 Apr 2010 16:38:31 -0600

Hi Brian,

Apologies for the delay in responding.  Snipping parts that I agree with.

On Apr 22, 2010, at 15:40 MDT, Brian E Carpenter wrote:
<snip>

> use across domain boundaries?

That seems like a reasonable conclusion.  Given the packet inspection capabilities on most newer PE devices have to be pretty good, (in order to perform the usual classification on, say, the 5-tuple to either forward/drop the packet for security reasons or to set/reset the IP Traffic Class field), it doesn't seem too unreasonable to expect them to (re-)calculate a flow-label as a packet enters a domain.  

Although, in the one case of Inter-Domain IP-in-IPv6 encapsulation it /might/ be challenging in the near term for intermediate routers, at a border entrance, to grab enough of the inner L4 + L3 headers + outer packet headers to form a 'decent' outer header flow-label.  Regardless, this doesn't [currently] comprise a substantial portion of the Inter-Domain traffic to be of short-term concern.  In addition, the end-result of this work should bring awareness of the issue of tunneled packets to SP's and equipment vendors so that, eventually, equipment can support it properly, which is the best we can do.

Finally, to ameliorate some of the concerns of having PE's (a.k.a. intermediate routers), with high-speed interfaces, identify "flows" as packets enter the domain, it would be beneficial to have [much] simpler IPv6 Extension Headers, as James Woodyatt mentioned a couple of weeks ago in a different thread on this list:
http://www.ietf.org/mail-archive/web/ipv6/current/msg11566.html
... thus, I would support that effort, as well, given the benefits to not only application writers, but also intermediate routers, middleboxes, etc. (eventually).  Of course, work on draft-krishnan can proceed in parallel and there shouldn't be any contingency on moving the flow-label work forward.  (If anything, the flow-label work can help make a better case for draft-krishnan).

-shane