Subject: AD review of draft-ietf-6man-flow-3697bis

From: Jari Arkko <[email-masked]>
Date: Mon, 20 Jun 2011 19:09:08 +0300

I have reviewed this draft.

I think it is in good shape and can move forward once we resolve one 
issue. Here's the issue:

> A node that forwards a flow whose flow label value in arriving
> packets is zero MAY change the flow label value.  In that case, it is
> RECOMMENDED that the forwarding node sets the flow label field for a
> flow to a uniformly distributed value as just described for source
> nodes.
> o  The same considerations apply as to source hosts setting the flow
>    label; in particular, the normal case is that a flow is defined by
>    the 5-tuple.
> o  This option, if implemented, would presumably be used by first-hop
>    or ingress routers.  It might place a considerable per-packet
>    processing load on them, even if they adopted a stateless method
>    of flow identification and label assignment.  This is why the
>    principal recommendation is that the source host should set the
>    label.
>   
I think this recommendation is problematic. I agree that the first hop 
router should insert the flow label, but requiring it to do fragment 
reassembly in order to find the 5-tuple is a big burden, and I'm not 
sure its even called for.

The RFC 2119 language above is fine. But I'd like to change the part 
about normal case being the 5-tuple. I think the normal case should be 
the 2-tuple under these circumstances. The source has access to the 
5-tuple; a router is not guaranteed to have access to it.

In addition, I'm not sure I understand how a router knows that it is a 
first hop router. Are there cases where a device might mistakenly 
believe it is a first hop router at a point where the traffic has 
already been load-balanced to multiple routers? Are there situations 
where the multiple first hop routers are used from the same host? The 
document should provide some guidance about operational conditions where 
the recommendations for the first hop router can be applied. The 
document should state how such functionality is turned on (per 
configuration? automatically?) and provide assurances that problematic 
conditions can be avoided.

Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 21 Jun 2011 10:20:44 +1200

Jari,

Thanks for the review.

> sure its even called for.

Yes, this is an embarassingly good catch. I think we need to add some language
to the recommendation that source hosts SHOULD set their own flow labels,
pointing out that in a fragmented case this is the only way to do things
properly. Reverting to the 2-tuple is a serious penalty since a lot of
the entropy in headers is in one of the port numbers.

> 5-tuple; a router is not guaranteed to have access to it.

As far as routers go, I think we have to say that an implementor has
to choose between a reassembly-based solution using the 5-tuple and
simply using the 2-tuple (maybe also the fragmentation ID - there is
some scope for ingenuity here).

> already been load-balanced to multiple routers? 

It's surely a configuration issue - enable flow labelling should
be configurable. Otherwise we have to be far too clever, as your
following points show.

> Are there situations

I'd like to hear from my co-authors and the WG before producing a new version.

    Brian

---

From: Bob Hinden <[email-masked]>
Date: Mon, 20 Jun 2011 16:31:32 -0700

Jari,

On Jun 20, 2011, at 9:09 AM, Jari Arkko wrote:

> I think it is in good shape and can move forward once we resolve one issue. Here's the issue:
> 
> I think this recommendation is problematic. I agree that the first hop router should insert the flow label, but requiring it to do fragment reassembly in order to find the 5-tuple is a big burden, and I'm not sure its even called for.

I agree that requiring a router to reassembly isn't worthwhile.  Better to just leave the flow label as zero.

> 
> The RFC 2119 language above is fine. But I'd like to change the part about normal case being the 5-tuple. I think the normal case should be the 2-tuple under these circumstances. The source has access to the 5-tuple; a router is not guaranteed to have access to it.
> 
> In addition, I'm not sure I understand how a router knows that it is a first hop router. Are there cases where a device might mistakenly believe it is a first hop router at a point where the traffic has already been load-balanced to multiple routers? Are there situations where the multiple first hop routers are used from the same host? The document should provide some guidance about operational conditions where the recommendations for the first hop router can be applied. The document should state how such functionality is turned on (per configuration? automatically?) and provide assurances that problematic conditions can be avoided.

I suppose the simple test is if the source address is from a prefix on the incoming interface.  

RFC 4311 "IPv6 Host-to-Router Load Sharing" requires (SHOULD) that hosts send packets for a particular destination to the same router (when it has multiple routers to choose from).  I think the flow label draft behavior for setting zero flow labels is compatible with this.  

Does this help answer your questions?

Bob

> 
> Jari
> 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: "George, Wesley" <[email-masked]>
Date: Tue, 21 Jun 2011 09:57:29 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Monday, June 20, 2011 6:21 PM
To: Jari Arkko
Cc: IETF IPv6 Mailing List; [email-masked]
Subject: Re: AD review of draft-ietf-6man-flow-3697bis

WEG> Given the fact that only end hosts can actually fragment IPv6 packets, and that's only when MTU less than 1280 is supported, I would expect fragmentation to be more the exception than the rule. I agree that for completeness it is worthwhile to discuss as Brian recommends, but I don't think it's a major concern, and it should be phrased that way in the draft.

WEG> I agree that this is a detail best left to implementers and operators, but I do not support an explicit recommendation to only do 2-tuple on fragmented packets. If the draft wants to put some language that it may need to be a consideration, I'm ok with that, but I do not want this to be a situation where the implementation that we get is only 2-tuple because of a perceived (and potentially overstated) scale risk for fragmented packet reassembly. I'd much rather router vendors implement 5-tuple, and then let operators decide how they want to manage this potential scale issue via configuration knobs. This also leaves it open for ingenuity as Brian says. Smart vendors could probably even use some of their existing rate-limiting machinery to do some amount of reassembly and then revert to 2-tuple if it exceeds a certain threshold in order to preserve box performance.
If I were to put this in normative language, I would say SHOULD use 5 tuple for fragments, MAY revert to 2-tuple if reassembly at higher traffic levels becomes a concern in a given implementation.
However, to Brian's comment about fragmentation ID, is there any reason why the draft can't recommend (or even require) this as a possible improvement over straight 2-tuple in the revert case?

> Are there situations
> where the multiple first hop routers are used from the same host?

WEG> I think that first-hop router in the context of this draft is more accurately defined as the first router in the path which is configured to re-mark zero-value flow labels, which probably makes "first-hop router" a poor choice of words here. Ingress router might be a better choice by itself, since that could more generically cover something like an ASBR as well.
Alternatively, you could make a distinction between first-hop router and ingress router in cases where that device is not the host, but is going to be encapsulating the data (tunnels, for example) in such a way so that subsequent routers cannot see the true 5-tuple data without looking well past the packet headers. In this case, we may want to strongly recommend that it sets the flow label on the outer packet based on data from the inner packet, since like the host, it has the best visibility into the underlying flows and this will be gone on subsequent hops.

Because this is strictly optional and will be incrementally deployed, there is no way to assume that the first-hop router is anywhere near the originating host, and I think it's overreaching to discuss such specific cases as above. We've already said that the host should be the one doing this because it has the best knowledge of its flows. If it doesn't, it's best effort based on what the network can infer from the available data. I'm not sure how the case of one host connected to multiple first-hop routers is different - it sends packets to all of them, either due to ECMP or other bundling or some other algorithm, but unless it's doing something relatively strange like per-packet round robin, each of the first-hop routers should be able to use the information it has to determine flows on the traffic it sees and mark accordingly. At most, this might be a good example for why per-packet distribution without explicitly setting the flow label is a bad idea.

> The document should provide some guidance about operational conditions where

WEG> I think I need some additional clarity on what you mean by the above before I can give much in the way of useful feedback. I'm not sure that much guidance on operational conditions is necessary, since the introduction already explains the types of networks that would see benefit to enhanced load-balancing, as well as the reasons why FL is a good addition to the existing data used for LB decisions. If your network meets those tests, and your router supports the feature at scale, enable it. Unless folks can come up with specific cases where enabling FL marking is explicitly NOT recommended...
Regarding avoiding problematic conditions, I can think of the following:
Inconsistent balancing due to incorrect identification of flows or collision of flow label values
Out of order packet delivery due to the above
Scaling considerations with managing reassembly of fragments for 5-tuple flow marking
Most of these are either going to be discussed based on this review, or are side effects of the fact that this is a best-effort, incrementally deployed and optional implementation. What other problematic conditions are there that we need to avoid?

I think defaults are probably an implementation detail, but if the draft has to make a recommendation, I'd say host-level defaults to on, router-level defaults to off, and have enabling rewriting zero-value flow labels be a separate configuration item from paying attention to existing nonzero flow label values for load-balancing. Whether it's a global knob or a per-interface one is probably getting too far into the weeds, but I'd say that at least for the rewriting, it might make sense to have the granularity to enable/disable on a per-interface basis.

Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 26 Jun 2011 13:06:56 +1200

Hi,

The discussion Jari's issue has died down, so I'd like to propose some revised text:

A node that forwards a flow whose flow label value in arriving
   packets is zero MAY change the flow label value.  In that case, it is
   RECOMMENDED that the forwarding node sets the flow label field for a
   flow to a uniformly distributed value as just described for source
   nodes.
   o  The same considerations apply as to source hosts setting the flow
      label; in particular, the preferred case is that a flow is defined
      by the 5-tuple.  However, there are cases in which the complete
      5-tuple for all packets is not readily available to a forwarding
      node, in particular for fragmented packets.  In such cases a flow
      can be defined by fewer IPv6 header fields, typically using only
      the 2-tuple {dest addr, source addr}.  A forwarding node
      implementation MAY use this 2-tuple in all cases.

[BC: this version indicates the problem that Jari discovered, but is
agnostic on the question of whether a router could or should solve it
by reassembling the fragmented packet.]

   o  The deployment of this option MUST be consistent with [RFC4311].

[BC: This last sentence is to cover Jari's point about a router knowing it's
appropriate for it to set the label.]

Comments? If not, I will post the draft in a day or two.]

Regards
   Brian Carpenter

---

From: Brian Haberman <[email-masked]>
Date: Mon, 27 Jun 2011 09:48:10 -0400

Brian,

> Comments? If not, I will post the draft in a day or two.]

These changes look fine to me.

Regards,
Brian

---

From: Shane Amante <[email-masked]>
Date: Mon, 27 Jun 2011 08:29:33 -0600

Brian,

>      implementation MAY use this 2-tuple in all cases.

I'm not comfortable with the words "in all cases" in the last sentence.  Perhaps change "in all cases" to "in these cases"?  (IOW, I don't believe you're trying to relax the rule that use of the 5-tuple is generally preferred to construct a hash for the flow-label where _practical_).

---

From: Jari Arkko <[email-masked]>
Date: Tue, 28 Jun 2011 00:57:19 +0200

Bob,

> Does this help answer your questions?

Yes, at least for the part where there several first hop routers. Thank 
you. Perhaps something like what you said above could be noted in the 
document, too.

Jari

---

From: Thomas Narten <[email-masked]>
Date: Tue, 28 Jun 2011 09:06:06 -0400

> appropriate for it to set the label.]

Could you please explain what the above is intended to do? I don't see
right off what the point is.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 28 Jun 2011 09:12:49 -0400

> >> In addition, I'm not sure I understand how a router knows that it
> >> is a first hop router.

IMO, this is not necessarily intended to be something routers just
know automatically. The point is that routers closer to the source
tend to have better knowledge about how to set the Flow Label
properly. For example, access routers, which may already be screening
the port for invalid MAC src addrs, improper src IP addresses,
etc. Routers that do this would be well placed to do Flow Label
setting as well.

That all said, any router that sees a Flow Label of zero, and wants to
change it to something better presumably should/can. When would that
NOT be the case?

> >> Are there cases where a device might
> >> mistakenly believe it is a first hop router at a point where the
> >> traffic has already been load-balanced to multiple routers? Are
> >> there situations where the multiple first hop routers are used
> >> from the same host?

Sure. But why should that impact how the Flow Label is rewritten from
zero to something else?

> >> The document should provide some guidance about operational
> >> conditions where the recommendations for the first hop router can
> >> be applied. The document should state how such functionality is
> >> turned on (per configuration?  automatically?) and provide
> >> assurances that problematic conditions can be avoided.

What problematic conditions do we need to worry about here? 

Thomas

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jun 2011 08:13:01 +1200

> right off what the point is.

Jari's question was how a router knows whether it should or shouldn't
set the flow label. That is hard to answer succinctly because it depends
on local configuration, but I agree that more explanation is needed,
beyond consistency with 4311. I'll try to come up with something.

  Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jun 2011 08:18:22 +1200

> change it to something better presumably should/can. When would that

If the network manager of the site decides not to do it there? I think
it needs to be configurable.

> Sure. But why should that impact how the Flow Label is rewritten from

Because different routers might pick different labels for packets
that belong to the same flow.

   Brian

---

From: Thomas Narten <[email-masked]>
Date: Tue, 28 Jun 2011 16:23:35 -0400

> it needs to be configurable.

Absolutely, and by default should probably be off.

> that belong to the same flow.

But, this implies packets from the same flow are being sent through
different routers. If that is happening, something is broken, and this
document can't (and shouldn't) fix it ...

And it won't happen at the first hop. I.e., ND addresses this as does
RFC 4311. All packets from the same flow go through the same router at
the first hop. Load balancing at the ND level means different flows
can go through different default routers.

Thomas

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jun 2011 09:44:56 +1200

By way of introduction, I have a preference for standards text to be as short
as possible, because the more we say, the more likely we are to be wrong
(especially when speculating about future router design). So here is my
next proposal for the text about routers setting the flow label. As always,
comments welcome (suggested text especially welcome...):

   o  The same considerations apply as to source hosts setting the flow
      label; in particular, the preferred case is that a flow is defined
      by the 5-tuple.  However, there are cases in which the complete
      5-tuple for all packets is not readily available to a forwarding
      node, in particular for fragmented packets.  In such cases a flow
      can be defined by fewer IPv6 header fields, typically using only
      the 2-tuple {dest addr, source addr}.  There are alternative
      approaches that implementers could choose, such as:
      *  A forwarding node might use the 5-tuple to define a flow
         whenever possible, but use the 2-tuple when the complete
         5-tuple is not available.  In this case, unfragmented and
         fragmented packets belonging to the same transport session
         would receive different flow label values, altering the effect
         of subsequent load distribution based on the flow label.
      *  A forwarding node might use the 2-tuple to define a flow in all
         cases.  In this case, subsequent load distribution would be
         based only on IP addresses.
   o  This option, if implemented, would presumably be of value in
      first-hop or ingress routers.  It might place a considerable per-
      packet processing load on them, even if they adopted a stateless
      method of flow identification and label assignment.  Also, it
      should not interfere with host-to-router load sharing [RFC4311].
      Therefore, it needs to be under the control of network managers,
      to avoid unwanted processing load and any other undesirable
      effects.  For this reason it MUST be a configurable option,
      disabled by default.

   The preceding rules taken together allow a given network to include
   routers that set flow labels on behalf of hosts that do not do so.
   The complications described explain why the principal recommendation
   is that the source hosts should set the label.

---

From: Jari Arkko <[email-masked]>
Date: Tue, 28 Jun 2011 23:51:21 +0200

I'm fine with this text.

Jari

---

From: Bob Hinden <[email-masked]>
Date: Wed, 29 Jun 2011 09:16:57 -0700

> I'm fine with this text.

As am I.

Bob

---

From: Thomas Narten <[email-masked]>
Date: Wed, 29 Jun 2011 12:42:41 -0400

I'm generally OK with this text.

>       should not interfere with host-to-router load sharing [RFC4311].

This sentence is ambiguous. is the "should not" a statement of fact,
or is it a directive, i.e., is there some implication for implementors
that they need to be aware of so as not to intefere with 4311?

Thomas

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 30 Jun 2011 07:44:22 +1200

> that they need to be aware of so as not to intefere with 4311?

There's a fair amount of wiggle room in 4311, but I think it is true that
setting the flow label in a router *will not* interfere. So I think
that s/should not/will not/ is correct.

   Brian

---

From: Jari Arkko <[email-masked]>
Date: Wed, 29 Jun 2011 23:59:33 +0200

Thomas, Brian,

> that belong to the same flow.

Yes, that was my concern.

> can go through different default routers.

Ok. I have a bit better feeling about this in that case.

Jari