Subject: RE: Flow label draft issues & new text

From: [email-masked]
Date: Wed, 11 Sep 2002 10:10:12 +0300

Margaret,

The flow label specification should define the default usage of the flow label, and allow for future specification of more specific usages. IPv6 WG needs to reach consensus on the default usage, document that in the flow label specification, and not set any unnecessary or ideological restrictions on the usages to be defined later.

Enabling usage for "load balancing" is within default usage of the current draft. I think we need now have a discussion on what exactly this means for the spec.

You wrote:
> Load balancing routers want to make balancing decisions based on the 
> value of the flow label field, with no other signaling required. A
> very simple mechanism (such as hashing the flow label field) will 

Does "load balancing" here mean selection of one of the parallel paths (with the same metric), or selection of one of parallel links to the next hop?

In any case the source and destination addresses should be included in the hash.

IMO the address fields alone would be enough for "core" routers, where it is unlikely that a single host pair can dominate the traffic.

> allow load balancing routers to consistently send packets from a 
> single flow over the same path.  This offers a couple of big 
> advantages:
> 
>         - It reduces packet reordering, potentially resulting in
>                 significant performance increases.

We need to understand how significant issue this is.

First, this may be an performance issue for the final destination only, as routers do not reassemble flows.

If the selection of the path can have a significant effect on the overall packet delivery time, the load balancing decision should not be based on a blind flow identity only.

IP has never promised to not reorder packets. I do not think we need to build in any guarantees now either.

So, exactly when could we expect to get significant performance increases?

>         - It makes the PMTU mechanism work better, since packets
>                 for a single connection will take a consistent
>                 path.
> 

In practice, is or will path load balancing be made over paths where the PMTU is different among the parallel paths?

Is it forbidden for e.g. TCP to share PMTU information between different connections, if those connections are between same addresses?

> But, for this simple mechanism to work, load balancing routers have to
> be able to rely on the fact that the flow label will label a _flow_
> (one direction of a TCP, UDP or SCTP connection).
> 

Also this is an issue to be discussed. For the load balancing to be effective, the bandwidth of the identified (and "balanced") unit should be of small enough portion of link/path capacity. There is no guarantee that a single TCP connection is "low bandwidth" compared to some other aggregate of packets between a pair of addresses. On the contrary, TCP tries to adapt to the path and get the maximum throughput (e.g. for a file transfer).

It seems that if a given TCP connection is consistently mapped to the same flow identity, the simple path load balancing will work the same regardless the granularity of the aggregate from the source host. But note that a Flow Label alone does NOT identify the flow, you need to consider the source and destination addresses also (which is also very clearly spelled out in the current draft :-)

	Jarno

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Sep 2002 15:22:22 +0200

> Enabling usage for "load balancing" is within default usage of the current draft. I think we need now have a discussion on what exactly this means for the spec.

[BC] IMHO, nothing. The spec as written allows load balancing.

> IMO the address fields alone would be enough for "core" routers, where it is unlikely that a single host pair can dominate the traffic.

[BC] Probably, but near the source or destination this is not necessarily
true. In a non-diffserv, non-intserv scenario, using the triplet
{source, dest, flow label} to split the traffic could be interesting.
 
> We need to understand how significant issue this is.

[BC] In terms of TCP performance, it is highly significant; anecdotally 
at least a 50% throughput hit for systematic reordering.

> First, this may be an performance issue for the final destination only, as routers do not reassemble flows.

[BC] Yes, but it's the e2e throughput that matters.

> IP has never promised to not reorder packets. I do not think we need to build in any guarantees now either.

[BC] No guarantees, but we need to do our best. However, it's irrelevant to
the flow label draft.

> In practice, is or will path load balancing be made over paths where the PMTU is different among the parallel paths?

[BC] Who knows?

> Is it forbidden for e.g. TCP to share PMTU information between different connections, if those connections are between same addresses?

[BC] It may not be forbidden, but it sounds very dangerous to me.

> It seems that if a given TCP connection is consistently mapped to the same flow identity, the simple path load balancing will work the same regardless the granularity of the aggregate from the sour…

[BC] This argument is beside the point, which is getting the flow label 
draft finished. Since the draft allows load balancing as Margaret
described it to co-exist with other usage, this WG simply shouldn't care.

    Brian

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 11 Sep 2002 09:38:45 -0400

Hi Brian,

>{source, dest, flow label} to split the traffic could be interesting.

And, how do load balancing routers determine that this is a non-diffserv,
non-intserv scenario?

>described it to co-exist with other usage, this WG simply shouldn't care.

Of course this WG should care.  The document is a work item of the WG,
so the WG is responsible for the contents of the document, which should
reflect working group consensus in every regard.

Margaret

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Sep 2002 14:47:06 +0200

> And, how do load balancing routers determine that this is a non-diffserv,

Jarno answered this one I think, but my point is that *they don't need to
know*. They just behave the same way in all cases, and the traffic that
doesn't carry fine-grain flow labels will just not get load balanced.

> reflect working group consensus in every regard.

The WG should care that its output is technically consistent and necessary
and sufficient, but this isn't a QOS working group or the NSIS working
group or the load balancing WG and it shouldn't attempt to second-guess
those topics. That's what I mean by "shouldn't care." 

Philosophically, I think the WG standardising the datagram layer should
be as minimalist as possible in its assumptions about how people will
use datagrams. In this case, that means making as few assumptions as
possible about how people will use the label, beyond the assumption
of immutability.

    Brian

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 12 Sep 2002 08:57:42 -0400

>doesn't carry fine-grain flow labels will just not get load balanced.

The problem is that the traffic that "doesn't carry fine-grain flow labels"
will still get sent through the load balancing mechanics, and packets with
the same flow label will still get forwarded the same way.

This is probably fine if a few related TCP connections are labelled with
the same flow label, or something like that.

It would be very bad, though, if the flow label were used, for example,
to tag packets that contain a TCP SYN, or packets that contain IP options,
or all HTTP packets, or packets that want a certain class of services, etc.

My largest problem with this draft is that I think it is needlessly
complex...  What is wrong with specifying a simple default flow label
mechanism, and requiring an API switch to override the values for
each flow?

Margaret

---

From: Keith Knightson <[email-masked]>
Date: Thu, 12 Sep 2002 09:44:12 -0400

All,

What happened to the proposal to partition the label space to clearly 
and formally delineate the different possible uses?

Surely this would elminate any possible ambiguous and/or conflicting use.

Regards
Keith Knightson

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Sep 2002 17:22:01 +0200

> Surely this would elminate any possible ambiguous and/or conflicting use.

There was no consensus to go that way. So we looked for a more
general solution.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Sep 2002 17:33:20 +0200

> to tag packets that contain a TCP SYN, or packets that contain IP options,

Agreed. That doesn't meet anyone's definition of a flow, I think.

> or all HTTP packets, or packets that want a certain class of services, etc.

Why on earth is that bad? It seems perfectly fine to me. That is
exactly the kind of scenario I am interested in.

> mechanism, and requiring an API switch to override the values for

Because we don't agree on what the default should be (apart from label=0).
And a per-flow override is not a general solution; the scenarios
I have in mind would certainly not be satisfied by that, because
they are not microflow based scenarios.

   Brian

---

From: Michael Thomas <[email-masked]>
Date: Thu, 12 Sep 2002 09:20:05 -0700 (PDT)

Margaret,

Maybe I'm in left field here, but I thought that a
transmitter who didn't mark packets' flow label
was supposed to set it to zero. In that case, the
router could conceivably resort to classifying packets
the old fashioned way -- eg transport headers.

 > or all HTTP packets, or packets that want a certain class of services, etc.

Er, why would this be bad? Are you thinking of the
possibility of a host gaming a router's fair
queuing by changing the flow label on packet by
packet basis?

		    Mike

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 12 Sep 2002 13:02:40 -0400

>the old fashioned way -- eg transport headers.

The problem is that the current specification allows information other
than a flow label, such as a packet classifier to be placed in the
flow label field.  So, a router cannot know that a non-zero value labels
a flow (i.e. one direction of a TCP connection).

>Er, why would this be bad?

There are a couple of ways in which you would like to be able to use
the flow label field to simplify load balancing.

One is a router with support for multi-path routing.  There is more than
one path out of this box that will reach the same destination, perhaps
with some sort of weighting value to indicate what portion of the traffic
should be sent over each path.

Today, these routers need to do packet classification on the IP addresses
and the upper-layer port numbers if they want to avoid sending data from
the same connection over different paths, which can result in packet
reordering, etc.

If the flow label actually labeled a flow, these routers could use the
IP addresses and the flow label to keep packets from the same flow together,
instead of having to parse the variable-length IP headers to find the
upper-layer port numbers.

Another application is a load-spreading router that spreads incoming TCP
connections across a group of servers.  This box sets up state for each
incoming connection, and needs to ensure that all packets for the same
connection are sent to the same server -- otherwise the connection will
be reset.

It would be useful if these boxes could set-up state based on the IP
address and the flow label, instead of having to parse headers down to
the TCP header to find the port numbers.

Both of these mechanisms rely on the flow label labeling one direction of
an upper-layer connection (the definition I usually associate with the
word "flow").

If some nodes set their flow label to zero (indicating that they don't
participate in flow labeling), these mechanisms will still work.  The
lookup for those packets will be less efficient, but it would still work.

What doesn't work is if there may be non-zero values in the flow label
that actually don't label flows.  How is a load-balancing or load-spreading
router supposed to know that this isn't a flow label?

I gave two examples which Brian Carpenter said he specifically had in
mind for possible uses of the flow label:

         - All HTTP packets get the same flow label.

                 How would this work with a load spreader that is
                         trying to spread connections between a group
                         of web servers?

         - The flow label contains a packet classifier.

                 This *might* be okay, if values aren't reused too
                         often, and if a single connection always uses
                         the same value.

But, Brian specifically stated that he wants finer-grained control over
flow label values than a per-connection value.  How would that work with
load-spreaders?

If the WG really wants to define the flow label so that it can be used
for signalling-based mechanisms like RSVP, NSIS and diffserv, with the
clear understanding that this makes the value _USELESS_ for the types
of applications I've described above, that is fine with me.

But, it isn't true that the defined mechanism lets us do both.

Margaret

---

From: Michael Thomas <[email-masked]>
Date: Thu, 12 Sep 2002 10:38:43 -0700 (PDT)

> a flow (i.e. one direction of a TCP connection).

   I guess I don't understand why that's a bad thing.
   Routers currently classify flows with transport headers,
   etc, but that's because it's the only thing they _can_
   classify with. The flow label pushes classification 
   back to the host which, IMO, is where it really 
   belongs since it's the only that that _really_
   knows what constitutes a "flow".
   
 > >Er, why would this be bad?

   [much elided] 

 > router supposed to know that this isn't a flow label?

   Er, well, it _doesn't_. I guess I just don't
   attach any special meaning to the word "flow";
   that is, a flow is whatever the host says is a
   flow. If it's bizarre, then well, why should
   the router care? Again, other than maybe gaming
   fair queuing algorithms[*], why do routers
   actually care about what constitutes the
   sematics of the flow? It's really in the host's
   best interest to be network friendly, right? To 
   give routers information which increases the
   probability of forwarding its packets in the
   manner it hopes for, right?

 >                          trying to spread connections between a group

   Well assumedly the server farm wouldn't all
   have the same IP address, so they'd be different
   flows.

 >          - The flow label contains a packet classifier.

   Er, but the flow label *is* an element of a
   packet classifier, or rather perhaps a
   replacement for the current method of classifying
   packets. I thought as far as routers are
   concerned, flow label were opaque and that no
   internal semantics were visible.

 > flow label values than a per-connection value.  How would that work with

   This seems to me to be a self-healing problem:
   if it screws up load balancers by using strange
   definitions of flow, hosts are probably what
   will notice via reduced throughput and... stop
   doing that ("Doc it hurts when I hit myself in
   the head..."). Though I must say that I still
   don't see that it would hurt them.

 > of applications I've described above, that is fine with me.

   I'm still completely lost. How does it make it
   useless?

		Mike

[*] this could conceivably be mitigated by first
    classifying by IP address and then draw down 
    each different flow from its own token bucket,
    thus a host would only be screwing itself.
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: "Tony Hain" <[email-masked]>
Date: Thu, 12 Sep 2002 12:03:45 -0700

Michael Thomas wrote:
> ... snip
>  > What doesn't work is if there may be non-zero values in 
> the flow label  > that actually don't label flows.  How is a 
> load-balancing or load-spreading  > router supposed to know 
> that this isn't a flow label?
> 
>  ... snip 

Mike is right, as far as middle-boxes are concerned the FL is opaque
unless they have been given specific state to believe otherwise. There
is no reason a load-spreader would have a problem with frequent reuse,
or even all nodes deciding to use the same value for HTTP as long as it
is basing its decisions on the Src/Dst/FL rather than just the FL. 

I do have comments on the text.  I would like to see the following
changed from:
4.  Flow Labeling Requirements
(4)  The source SHOULD assign each new transport connection (e.g.
        TCP, SCTP) to a new flow.

to:
(4)  The source SHOULD assign each unrelated transport connection (e.g.
        TCP, SCTP) to a new flow.

This would keep it from conflicting with (3).

5.  Flow State Establishment Requirements
...
 To enable co-existence of different methods in IPv6 nodes, the
   methods MUST meet the following basic requirements:
...
(3)  The IPv6 node facility keeping track of the Flow Label and the
        associated Source and Destination Addresses MUST be utilized
        when assigning Flow Label values to new flows (see section 4
        above).

That wording seems awkward, how about:
(3)  The IPv6 source node MUST provide a facility for keeping track 
of the Flow Label values associated with particular Source and 
Destination Addresses for use when assigning Flow Label to new flows 
(see section 4 above).

Tony

---

From: Brian Haberman <[email-masked]>
Date: Thu, 12 Sep 2002 16:31:40 -0400

>    I'm still completely lost. How does it make it

I must be lost as well.  In my reading of the flow label spec, I see
the capability to do load balancing quite easily.  The host sending
the data assigns unique flow labels however it wants (just like Erik
had mentioned).  The routers can then either forward based solely on
the DA or it can hash the <DA,SA,FlowLabel> for load balancing.  If
you want to talk DiffServ, IntServ, or something of that flavor, the
flow label would be signaled, the router would recognize it during
packet classification and deal with it how it sees fit.

So, my opinion is that the spec does what it set out to accomplish.

Brian

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 12 Sep 2002 16:51:29 -0400

>If

So, all of the routers would have to participate in signalling to
know whether the flow label was used for DiffServ, IntServ, etc.?

Or, it would just work to use the flow label for load balancing,
even it it contained a DiffServ or IntServ value, instead of
labeling a microflow?

Margaret

---

From: Brian Haberman <[email-masked]>
Date: Thu, 12 Sep 2002 17:05:49 -0400

> even it it contained a DiffServ or IntServ value, instead of

How DiffServ and IntServ work with the flow label is yet to be
determined.  All I am saying is that the current spec does not
preclude any of the functions mentioned above.  Isn't that the
point of the draft?

As Matt said, the host knows best what a flow is.  Whether that
flow gets signaled to routers for special handling or not is
outside the scope of this doc.

Brian

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 12 Sep 2002 18:05:13 -0400

I think that the problem is that we have different ideas about
what the purpose of this document is.

I, personally, would like to see a document that defines how hosts
should set the flow label, in the absence of knowledge to the
contrary.  Something like:

         - Start with a random number
         - Label each microflow (i.e. one half of a TCP, UDP or
                 SCTP communication) with a new flow number
         - Supply an API for flow label override on a per-microflow
                 basis

Based on that document, it would be safe for routers to assume
that a non-zero flow-label labels a microflow.

If other WGs want to define signalling mechanisms that run
above IP, those mechanisms can use the aforementioned API to set
the appropriate signalled values.  But, anything that wants to do
this would have to be consistent with the fact that some routers
will assume that the src/dest/flow label triplet labels a microflow.

This seems a lot simpler than the current doc, and would allow
routers to make some immediate use of the flow label field to
limit situations where they need to parse into the packet to
find upper-layer ports (parse once, and cache the value based
on src/dst/flow label).

Granted, this might make it harder to use the flow label field for
mechanisms that want to do things other than label microflows, but
it we already have a traffic class field for things that want to
label packets with traffic classes.

Margaret

---

From: "Tony Hain" <[email-masked]>
Date: Thu, 12 Sep 2002 16:03:41 -0700

Margaret Wasserman wrote:
> I think that the problem is that we have different ideas 
> about what the purpose of this document is.
> 
> I, personally, would like to see a document that defines how 
> hosts should set the flow label, in the absence of knowledge 
> to the contrary.  Something like:
> 

This is what it specifically can't do if the host is expecting multiple
connections to be treated as a bundle!

> Based on that document, it would be safe for routers to 
> assume that a non-zero flow-label labels a microflow.
> 
> above IP, those mechanisms can use the aforementioned API to 
> set the appropriate signalled values.  But, anything that 
> wants to do this would have to be consistent with the fact 
> that some routers will assume that the src/dest/flow label 
> triplet labels a microflow.
> 
> This seems a lot simpler than the current doc, and would 
> allow routers to make some immediate use of the flow label 
> field to limit situations where they need to parse into the 
> packet to find upper-layer ports (parse once, and cache the 
> value based on src/dst/flow label).

It can be argued that a router never *needs* to parse into the packet
beyond the addresses. It can also be argued that if the FL is non-zero
that the router has absolutely no business parsing into the packet,
because the host has identified anything with this label as needing the
same treatement by the router. As far as I can tell the document is
about as simple as it can be, and trying to become more specific about
how the FL gets used will only complicate it once all the possible uses
are added.

Tony

> 
> Granted, this might make it harder to use the flow label 
> field for mechanisms that want to do things other than label 
> microflows, but it we already have a traffic class field for 
> things that want to label packets with traffic classes.
> 
> Margaret
> 
> > > >you want to talk DiffServ, IntServ, or something of that flavor, 
> > > >the flow label would be signaled, the router would recognize it 
> > > >during packet classification and deal with it how it sees fit.
> > >
> >determined.  All I am saying is that the current spec does 
> not preclude 
> >any of the functions mentioned above.  Isn't that the point of the 
> >draft?
> >
> >As Matt said, the host knows best what a flow is.  Whether that flow 
> >gets signaled to routers for special handling or not is outside the 
> >scope of this doc.
> >
> >Brian
>

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 13 Sep 2002 08:58:15 +0200

Margaret Wasserman wrote:
> contrary.  

That is not the intended purpose of the current draft, which is
to set boundary conditions that allow hosts, routers, and network
processors to stop treating the label as an MBZ field. Once we 
have set these boundary conditions, we can then work out specific 
scenarios. It is not a given that the default scenario is load 
balancing; I doubt if there is a default scenario, apart from label=0.

Michael Thomas, Tony Hain and Brian Haberman have responded to your 
other comments much as I would. But basically, a flow is whatever
the source host decides is a flow.

    Brian

> ...Something like:

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter 
Distinguished Engineer, Internet Standards & Technology, IBM 
On assignment at the IBM Zurich Laboratory, Switzerland
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 13 Sep 2002 09:01:35 +0200

Tony's comments on the text seem valid to me.

   Brian

Tony Hain wrote:
...