Subject: Flow Labels: what problem are we solving?

From: Thomas Narten <[email-masked]>
Date: Tue, 11 Jan 2011 08:41:10 -0500

Sorry to get back to basics, but I have not followed all the Flow
Label discussions or read all the drafts. I have read

      draft-ietf-6man-flow-ecmp-00.txt
      draft-ietf-6man-flow-update-01.txt

pretty carefully and I still don't quite understand what real problem
we are trying to solve - and thus, whether the proposed changes
actually help or are a no op.

Is there a document that speaks to this?

Question:

I understand the value  of ECMP type load balancing. But how much of a
problem is it today (with IPv6) if the Flow Label is not used?

If you hash on just the 5 tuple (excluding the flow label),  you get
(I assume) the equivalent of what you have in IPv4 today. Why is that
not good enough?

Also, splitting flows across different links would seem to have value
primarily if you hvae a single source (or rather single src/dest pair)
generating a *lot* of traffic/flows, i.e., so that if you split
traffic from that source/dest pair, you see measurable load-splitting.

Is this happening in practice today? Can operators please speak to
this? And if it is a problem, is it primarily with tunneled traffic
(where the tunnel aggregates many flows), or is it really between
individual pairs of nodes that are sending a *lot* of traffic to each
other? Are there examples of this?

(I'm not necessarily opposed to this work going forward, but I'm not
entirely convinced we are solving a real problem. Help me please.)

Thomas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 Jan 2011 09:13:20 -0500

The goal is not to split single flows across multiple links.  In fact, 
based on the experience that this causes excessive disordering, we 
generally require that load splitting techniques must avoid splitting 
flows across links.

Which gets us to the problem.  In using multiple links (ECMP / LAG) the 
routers generally want the finest granularity possible for that 
operation.  That leads, as you say, to using a hash of the 5-tuple. 
However, the 5-tuple is not always available.  The simplest case is 
tunnels.  Unless the tunnel devices add new UDP headers (which is 
largely gratuitous and has other problems relative to the IPv6 
specifications), the 5-tuple is not visible to the devices forwarding 
teh resulting packets.
This also leads to IPSec needing a UDP header.  It is also one of the 
contributors to why using new extension headers is such a bad idea.

Thus, in working at the problem of how to get various kinds of traffic 
to be handled for ECMP / LAG, it seemed to many of us that capgturing 
the randomness of the port numbers in the flow label would be very helpful.
(Yes, this assumes that the current hash approach to ECMP / LAG is 
reasonable.  There is reportedly some question on that, but I take it as 
the best practice we have, and therefore worth continuing.)

Yours,
Joel

On 1/11/2011 8:41 AM, Thomas Narten wrote:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 11 Jan 2011 15:32:36 +0100

Hi Thomas:

I've seen different requirements depending on where the utilization
would be.

a) Close to the source of the source or destination, the flow label
could be used in an application-aware fashion, for instance to influence
the routing of the packet in VRFs. We'll note that we do not have a .1Q
equivalent in IPv6 and that though too small, the flow label is the only
place before the IPv6 header to indicate a context in which the
destination address is to be used.

b) Farther in the network, it could be used for blind NECM.

>From the discussions I've seen, people are mostly focused in b), with an
additional assumptions that a flow is something that has a given
source/destination pair. Sadly (for me) the case I'm most interested in
is a), and the flows I look at can often be useful only if you can group
multiple source / destinations in a single flow. 

What I'd really like to see is that whatever is decided to help b) does
not prevent a). I think it can be done easily if the border routers are
allowed to overwrite a FL, which is not the case today.

Cheers,

Pascal
http://www.xtranormal.com/watch/7011357/

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Thomas Narten
> Sent: Tuesday, January 11, 2011 2:41 PM
> To: [email-masked]
> Subject: Flow Labels: what problem are we solving?
> 
Label
> discussions or read all the drafts. I have read
> 
we are
> trying to solve - and thus, whether the proposed changes actually help
or are
> a no op.
(I assume)
> the equivalent of what you have in IPv4 today. Why is that not good
enough?
> Also, splitting flows across different links would seem to have value
primarily
> if you hvae a single source (or rather single src/dest pair)
generating a *lot*
> of traffic/flows, i.e., so that if you split traffic from that
source/dest pair, you
> see measurable load-splitting.
> 
this? And
> if it is a problem, is it primarily with tunneled traffic (where the
tunnel
> aggregates many flows), or is it really between individual pairs of
nodes that
> are sending a *lot* of traffic to each other? Are there examples of
this?
> (I'm not necessarily opposed to this work going forward, but I'm not
entirely
> convinced we are solving a real problem. Help me please.)
> 
> Thomas

---

From: Thomas Narten <[email-masked]>
Date: Tue, 11 Jan 2011 09:33:52 -0500

> The goal is not to split single flows across multiple links.

Sorry, I know that. Was I that unclear? :-(

But isn't it then a goal (or rather a *requirement*) to split traffic
between a given src/dst pair across multiple links? I.e., there is
enough traffic between a given src/dest pair to justify splitting it
into separate flows for load sharing purposes?

That would seem to be what we are after. I'm trying to understand
exactly which operational scenarios would actually see benefit from
this, and get a sense of how common they are. Are we trying to solve a
common problem, or is are we really solving an edge case?

> operation.

I understand that they may *want* this. But how bady do they *need*
this?

Is there real data here? Is the current status quo 80% good enough?
95%? or only 30%?

> That leads, as you say, to using a hash of the 5-tuple. 
> tunnels.

Again, how common is this? Is this a 5% problem, or a 50% problem? Or
is this an attempt to prepare for the future, since we don't have
extension headers defined that are a problem today, and too little
traffic is currently encrypted via IPsec...

> Unless the tunnel devices add new UDP headers (which is largely
> gratuitous and has other problems relative to the IPv6
> specifications), the 5-tuple is not visible to the devices
> forwarding teh resulting packets.  This also leads to IPSec needing
> a UDP header.  It is also one of the contributors to why using new
> extension headers is such a bad idea.

I can see how tunneling would be a problem.

But not for all kinds of tunneling. What about GRE with its key field?
Is this not widely enough used?

Do we have data about what types of tunneling are most common? 

> the randomness of the port numbers in the flow label would be very

In other words, the actual goal here then is to take the data
(entropy) that is currently in the port fields, and put it into the
flow label, and make this happen often enough in practice that routers
can use it *instead of* looking at the port numbers? Is that *really*
what the main goal is here?

> the best practice we have, and therefore worth continuing.)

Right. The question of what makes a good hash is a separate topic.

Thomas

---

From: "Rosomakho, Yaroslav" <[email-masked]>
Date: Tue, 11 Jan 2011 15:01:43 +0000

Hello, Thomas!

>common problem, or is are we really solving an edge case?

IPv4 5-tuples load-balancing is not always working perfectly for everyone.
I have encountered lot of occasions where it's far from perfect. Simplest
example would be a VPN concentrator on a LAG.

IPv6 is not going to be better then IPv4 when it comes to 5-tuples
computation. In fact due to many tunnel transition techniques, it will be
worse in general. Also, don't forget that extension header structure is
more complex and in many cases routers (especially hardware-based) will
not be able to look into layer 4 information.

>I understand that they may *want* this. But how bady do they *need*

Yes, they do. In a perfect world everybody is happily running 100GE
everywhere, but it's not happening in real life. People want to keep their
current transport for as long as they can. It's not an uncommon request to
have more then ten STM-16s bundled and it's a common expectation to have
effective throughput of aggregated link to be an exact sum of components.
And if you put any "smart" MPLS TE UCMP on top of that, you can get in a
real trouble with 5-tuples.

>Is there real data here? Is the current status quo 80% good enough?

I'm not sure what are you proposing to measure...

>traffic is currently encrypted via IPsec...

Again, 5% of what? Packets traveling in the internet?

These drafts try to deal with fundamental load-balancing issue. There is a
chance that traffic patterns in 10 years from now will be completely
different. Maybe everything will finally collapse inside TCP/80. Maybe
everything will be inside ESP. Maybe end-to-end will be reborn. Nobody can
tell. But obviously we don't want IPv6 to fail just because of that.

Load-balancing is too important to leave it to layer 4.

>Is this not widely enough used?

No, it is not.

>Do we have data about what types of tunneling are most common?

ESP is the most common non-UDP and non-TCP tunneling mechanism these days.
Nobody can reliably predict how will it change in the future.

>what the main goal is here?

Don't forget that load-balancing is often done by layer-2 devices. You
can't expect cheap ASIC-based switches to dig through extension headers
and find port numbers or GRE keys. But they definitely can look into
statically-placed fixed-length flow label at very low processing cost.

>Right. The question of what makes a good hash is a separate topic.

Load-balancing hashes have traditionally been implementation-specific.
Maybe we should leave them this way.

>Thomas

Best Regards,
Yaroslav

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 Jan 2011 10:10:10 -0500

Yes, the usage of ECMP / LAG is very common.  Yes, there is plenty of 
data that shows that using just the src and dst IP address in the hash 
is NOT good enough.  (This does start to get into the quesiton of 
whether blindly using the ports is good enough,l but that is the current 
practice.)

It is the case that much of the current hardware is built to work around 
the common packet cases so as to permit ECMP and LAG.  (All?  How much? 
  I do not have the data.)

However, we are seeing a LOT more uses of tunneling.  IPSec VPNs are one 
use, but they are pretty rare.  And they generally have to do enough 
processing that the UDP can be done in those cases.  So it is.
It should be noted that since tunnels aggregate a lot of sources / 
destinations into the tunnel end-point addresses.  So if we lose the 
inner addresses, and lose the port numbers, we lose a lot of entropy. 
In this case, if we could use the flow label, the tunnel device could 
creat one that restores much of that entropy.  (Not all, by any means.)

But no, GRE does not work.  The problem is that the hardware / firmware 
is built to look for wpeific things.  The more different cases you ahve 
to handle in different ways, the harder it gets.  So generally, as I 
understand it, the ECMP / LAG logic in most devices which do this in 
formware / hardware can not use the entropy in the GRE labels.
Also, even if we could use them, GRE labels are not per-flow.  they are 
typically per underlying user, which at least restores the granularity 
of the senders, but not the ports.

Admittedly, part of my original motivation was to find a way to allow 
tunnels to be handled without UDP headers so as to not fight about the 
UDP header checksum.  However, I can not get there in a useful time, so 
no, this proposal can not fix any of the immediate issues.
Short of giving up and saying we can never change any of the fields in 
the IPv6 header and never fix any of the problems, I am trying to make 
this useful for the problems we do have.

Yours,
Joel

---

From: Shane Amante <[email-masked]>
Date: Tue, 11 Jan 2011 12:15:18 -0700

Hi Thomas,

Token operator here ... :-)  See below.

> problem is it today (with IPv6) if the Flow Label is not used?

Today, it's a significant one, especially so for tunneled traffic, e.g.: IPvX in IPv6, GRE, IPSec, LISP, etc. and WAN acceleration products for fast file xfer's.  In the future, it could significantly inhibit (even, prohibit) the IETF from developing new protocols that either use new IPv6 Extension Headers or Destination Options, since Core/Edge router/switch HW (primarily built around ASIC's) cannot (easily) be adapted to recognize the granularity of 'individual flows' in those new protocols -- worst-case, existing HW cannot be adapted and would requiring swapping it out, which is essentially a non-starter.  Even when existing core HW can be adapted, through SW changes, to recognize the granularity of new protocols it could be upwards of a decade before the cycle of coaxing/prodding/cajoling vendors to develop the capability in SW through operators testing and eventually deploying the capability in their networks.

I would also raise the "architectural purity" argument of: do you really want millions of routers (and, L2 switches) that are really just supposed to forward based solely on IP source address, IP destination address (and, IP Traffic Class) *attempting* to go deeper into the packet (headers) to discern useful, granular information that could be used as input-keys for load-balancing across LAG and ECMP paths?  Unfortunately, every device manufacturer is going to make their own decisions on what they can (or, will) be able to look at in the IP and Extension Headers as input-keys for a load-balancing hash, but the end-result will be (and, is currently) a mess in terms of deployment and operation.  (As an example, some vendors can read entire IPv6 addresses from a IPv6 header in new(er) HW, others can only read parts of the v6 address, etc.)  Finally, there is this jewel in RFC 2460:
---cut here---
   With one exception, extension headers are not examined or processed
   by any node along a packet's delivery path, until the packet reaches
   the node (or each of the set of nodes, in the case of multicast)
   identified in the Destination Address field of the IPv6 header.
[...]
   The exception referred to in the preceding paragraph is the Hop-by-
   Hop Options header, which carries information that must be examined
   and processed by every node along a packet's delivery path, including
   the source and destination nodes.
---cut here---
While I wasn't around during during the creation of RFC 2460 (so, please correct or inform me if I'm wrong), but it seems to at least imply (if not, mandate?) that intermediate nodes (such as routers and L2 switches) shouldn't be trying to interpret the characteristics of the upper-layer protocols being transported.  This would make sense when viewed in light of the end-to-end principle, but perhaps I'm taking too strict an interpretation.

> (I assume) the equivalent of what you have in IPv4 today. Why is that

For one, HW can't glean a 5-tuple when it encounters tunneled packets.  Second, it inhibits/prohibits the IETF from developing new upper-layer protocols (if/when the need should arise) and getting them deployed in a reasonable timeframe.  Lastly, I would like to get to a point where I can tell my router/switch vendors to build more simple (and, thus, more cost-effective) HW, because they don't have to keep piling complexity into their ASIC's to recognize the various legacy and new permutations of transport-layer protocols for input-keys for LAG and ECMP load-balancing hashes. Instead, I would ideally be able to tell them: just use {IP src, IP dst and IPv6 flow-label} as input-keys -- oh, and look at that ... they're all at fixed offsets in the IPv6 header and at the very beginning of the packet so the time (and, amount of memory) for you to copy that region of the packet to extremely expensive SRAM (packet buffer memory) has just been reduced, (reducing cycle times to process the packet, etc.).  Now, I will grant you this latter point is a bit of wishful thinking for now, given that we'll need to be successful in getting these drafts agreed upon and published.  Then, hosts (and, 1st-hop) routers will need to start writing useful flow-labels (and, hopefully, firewalls or their administrators do not screw around and write zero back over the flow-label).  However, I'm of the opinion that if you don't start to make a small change now, you will never see an improvement down the road.

> entirely convinced we are solving a real problem. Help me please.)

I think I've answered these questions above; however, if you still aren't satisfied with those answers please let me know.  

Lastly, I would add that LAG and ECMP have been around for several [dozen] years and will remain with us indefinitely.  IOW, even if 100 GbE were cost-effective and deployed today, I know of several operators who will still be using LAG or ECMP over Nx 100 GbE trunks in order to continue to carry the traffic demand on their networks.  So, in summary, this is not a problem that is going away with 100 GbE, or beyond.

-shane

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 12 Jan 2011 10:21:43 +1300

Thomas,

> Is there a document that speaks to this?

I think others have tried to tackle your question by explaining
the perceived need to do some things that we can't do with IPv4.

There are also 3 meta-problems, which is why I keep mentioning
draft-hu-flow-label-cases (in the Independent stream queue).

1. The existence of 20 unused bits in the header is in itself a
problem in some people's eyes - we are wasting bits and energy
storing and transmitting (or compressing) those bits.

2. There are intrinsic ambiguities in RFC 3697 that have prevented
serious use of the flow label, even in the simple use case
defined in draft-carpenter-flow-ecmp.

3. Those ambiguities (and probably other factors) have led to
a variety of quite baroque proposals for using the flow label,
both posted as I-Ds and published in academic format. For gory
details, see draft-hu-flow-label-cases.

    Brian