Subject: Re: Flow Label: 12 bits mutable and 8 bits immutable

From: Rémi Després <[email-masked]>
Date: Tue, 3 Aug 2010 10:53:20 +0200

Le 30 juil. 2010 à 14:42, Pascal Thubert (pthubert) a écrit :

> Hi Mohacsi:
> 
> 
>> - applications relying on 20 bits flow bits should be sought and
> fixed. I believe
>> they assumed the flow field is immutable
> 
> That was the second point I made at the mike. We'll have to have a
> deprecation path if we change anything. We faced that with RH0, we faced
> that with site local. The sooner that happens the less it will hurt.
> 
> Let's face it, the current status is that we have 20 bits in every
> header that are mostly wasted.

> Providing rules restricts the freedom of
> what we could have done with those bits, but at least they become
> useful.
Full agreement on this point.

What about this combination (not documented yet, it seems)
- Hosts that send non 0 FLs, MUST do it with a value that:
 . is common to all packets of their flow
 . generally is different from one flow to another
- non 0 FLs MUST be preserved e2e.
- 0 FLs may be changed anywhere, but with the same constraints as in hosts.

In addition,
- A flow is specified by its 5-tuple, if it exists, or its 3-tuple otherwise.
- The FL value assigned to a flow MAY be EITHER:
 . a hash of the flow identification (simple because stateless), OR
 . a pseudo random number (more complex because stateful, but providing an utmost privacy protection
The choice between hash or randomness is made where the FL is set. Other nodes don't need to know what has been chosen.

Regards,
RD

PS, I added to the list Tony Hain with whom we discussed FL issues in Maastricht.

> I'll try to post a small draft soon.
> 
> Pascal
> 
> 
>> -----Original Message-----
>> From: Mohacsi Janos [mailto:[email-masked]]
>> Sent: Thursday, July 29, 2010 3:28 PM
>> To: Pascal Thubert (pthubert)
>> Cc: Aleksi Suhonen; [email-masked]; [email-masked]
>> Subject: RE: Flow Label: 12 bits mutable and 8 bits immutable
>> 
>> Dear All,
>> 	splitting this field not so obvious:
>> - API should be changed - or at least truncate the existing flow field
> to
>> 8 bit.
>> 
>> - applications relying on 20 bits flow bits should be sought and
> fixed. I believe
>> they assumed the flow field is immutable
>> 
>> 
>> Although I dont see the motivation for the flow field to be mutable.
>> 
>> 
>> 
>> Janos Mohacsi
>> Head of HBONE+ project
>> Network Engineer, Deputy Director of Network Planning and Projects
>> NIIF/HUNGARNET, HUNGARY
>> Key 70EF9882: DEC2 C685 1ED4 C95A 145F  4300 6F64 7B00 70EF 9882
>> 
>> On Thu, 29 Jul 2010, Pascal Thubert (pthubert) wrote:
>> 
>>> Hi Aleksi:
>>> 
>>> I think that the premise in the discussion was that the network
> egress
>>> is incapable of restoring zero.
>>> 
>>> This taken into account we observe that:
>>> 
>>> - there is a clear and documented need for the network to tag the
>>> packet. This happens for load balancing reasons, but also for
> routing
>>> reasons, when multiple topologies can be used (eg multihoming, RPL
>>> etc...)
>>> - 12 mutable bits is seen as enough. We had a proposed usage in RPL
> that
>>> needed just that as well. RPL ended up with the Hop by Hop mainly
>>> because the use of flow label was too largely undefined to feel safe
>>> using it, with the side effects that you know of, like IP in IP.
>>> 
>>> - flow label is the only field that an application can use to place
> an
>>> abstract tag in the packet in order to dialog with the network.
> Removing
>>> that capability seems really short sighted.
>>> - 8 bits have been enough for a considerable amount of functionality
> in
>>> the past. For the one I have in mind - application aware tagging, it
> is
>>> enough.
>>> 
>>> What can I say? We are asked to choose between 2 evils when a simple
>>> trade off exists...
>>> 
>>> Pascal
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: [email-masked] [mailto:[email-masked]] On
> Behalf
>>> Of
>>>> Aleksi Suhonen
>>>> Sent: Wednesday, July 28, 2010 5:01 PM
>>>> To: [email-masked]
>>>> Subject: Re: Flow Label: 12 bits mutable and 8 bits immutable
>>>> 
>>>> Hi,
>>>> 
>>>> On 07/28/10 13:24, Tina TSOU wrote:
>>>>> I like the proposal from Pascal Thurbert in today's meeting.
>>>>> I believe that It's more acceptable for the majority of the
>>> different
>>>>> camps.
>>>> 
>>>> I hated it. :-(
>>>> 
>>>> I feel that changing the structure of the IPv6 header like that at
>>> this stage is too
>>>> late. And I feel the change is bigger than people think it is.
>>>> 
>>>> Some earlier discussion has suggested that if the label is "0",
> then
>>> it's mutable
>>>> and otherwise it's immutable. And if an operator does change it,
> they
>>> should
>>>> reset it on egress, and maybe hijack some other bit in the header
>>> (e.g. from
>>>> DSCP?) to signal that the flow label has been fiddled with. This
> bit
>>> should of
>>>> course also be reset at the same time as the label is reset.
>>>> 
>>>> --
>>>> 	Aleksi Suhonen
>>>> 
> --------------------------------------------------------------------
>>>> IETF IPv6 working group mailing list
>>>> [email-masked]
>>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>>> 
> --------------------------------------------------------------------
>>> --------------------------------------------------------------------
>>> IETF IPv6 working group mailing list
>>> [email-masked]
>>> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
>>> --------------------------------------------------------------------
>>> 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Sander Steffann <[email-masked]>
Date: Tue, 3 Aug 2010 11:04:28 +0200

Hi Rémi,

> - 0 FLs may be changed anywhere, but with the same constraints as in hosts.

Do you mean that once a FL is set it shouldn't be changed further along the path, or that it should be reset to 0 at some point?

- Sander

---

From: Rémi Després <[email-masked]>
Date: Tue, 3 Aug 2010 14:49:07 +0200

> Do you mean that once a FL is set it shouldn't be changed further along the path, or that it should be reset to 0 at some point?

That it shouldn't be modified again.
This is to facilitate its use for load sharing (the reported original intention): once a hash or pseudo random number is assigned to a flow, there is no point in doing the job again. 

In my understanding, a reason why it is usually set to 0 is that a stateful operation, which is complex, is so far mandatory (because a pseudo-random number has to be assigned to each flow, not a stateless hash).
If a 5-tuple hash is permitted in the future, stateless operation will become possible, and more and more hosts can be expected to set the field.

RD

---

From: Shane Amante <[email-masked]>
Date: Tue, 3 Aug 2010 11:20:25 -0600

Remi,

> The choice between hash or randomness is made where the FL is set. Other nodes don't need to know what has been chosen.

Because of your last two bullets I have to ask the following.  How would a receiving host deterministically distinguish (1) flow-labels that were created by network devices (just a 5-tuple was put into a flow-label) vs. (2) flow-labels that were created by a source-host w/ a pseudo-random number + 5-tuple[1]?  (Please read on before answering :-)

If flow-labels were only created by source hosts (scenario #2), then a recv'ing host wouldn't necessarily care how the flow-label was constructed, it would just have to compare recv'd flow-labels over some period of time and/or number of packets to ensure an off-path attack is (most likely) not occurring.

OTOH, if you try to combine scenario #1 and #2, then would a recv'ing host have to first check to see if the flow-label was a 5-tuple hash?  Assuming it is just based off a 5-tuple, then it ignores checking for off-path attacks (since there is no pseudo-random number in it known only by a source host).  Of course, if the recv'd flow-label is determined not to be a 5-tuple hash, then the recv'ing host has to make a leap of faith it's a pseudo-random number + 5-tuple hash and can proceed with using it to verify that no off-path attack is occurring.  This is certainly more work on the part of the recv'ing host for every recv'd packet, but perhaps not impossible.  (Of course, I may be under-thinking this and would welcome any thoughts you have on if the aforementioned checks I described are necessary or better ways to perform them).

Some other potential things to consider by combining scenarios #1 and #2 are as follows:
a)  We'd need to be very precise in terms of the precise hash algorithm used, so that a recv'ing host could consistently check the recv'd flow-label is a hash of the 5-tuple.  It may be the case that different classes of devices (e.g.: mobile phones, desktop/server-class PC's, etc.) may have different reqmt's for an optimal hash algorithm they could use.
b)  <asbestos-suit-on>If NAT66 were to be adopted and deployed on middleboxes, then the IP addresses in packets would change in-flight in the network.  If the middlebox receives a packet with a non-zero flow-label (perhaps because a source host set it to a 5-tuple + pseudo-random number), what happens then?<asbestos-suit-off>

So, in summary, it may be that we have to consider EITHER scenario #1 OR #2 as a path forward for the flow-label while acknowledging the associated limitations with both scenarios when doing so.

-shane

[1] Perhaps along the lines of the following draft?
http://tools.ietf.org/html/draft-blake-ipv6-flow-label-nonce-02

---

From: Aleksi Suhonen <[email-masked]>
Date: Tue, 03 Aug 2010 22:25:01 +0000

Hi,

>> - 0 FLs may be changed anywhere, but with the same constraints as in hosts.

Earlier I suggested that a modified "0 FL" should be reset back to zero 
at the egress of the network that modified it. Someone said this is 
impossible, which I don't believe at all. However, all I really care 
about is that "0 FL" should be mutable and others shouldn't. Whether it 
is or should be reset on egress doesn't matter to me.

> Because of your last two bullets I have to ask the following.  How would a receiving host deterministically distinguish (1) flow-labels that were created by network devices (just a 5-tuple was put…

Why does a receiving host care about the flow label at all? It exists to 
make sure that all intermediate nodes give correct treatment to the 
flow, but once it reaches its destination it's "safe", right?

What have I missed?

-- 
	Aleksi Suhonen
	Department of Communications Engineering
	Tampere University of Technology

---

From: Shane Amante <[email-masked]>
Date: Tue, 3 Aug 2010 17:49:17 -0600

Aleksi,

On Aug 3, 2010, at 16:25 MDT, Aleksi Suhonen wrote:
> Earlier I suggested that a modified "0 FL" should be reset back to zero at the egress of the network that modified it. Someone said this is impossible, which I don't believe at all. However, all I really care about is that "0 FL" should be mutable and others shouldn't. Whether it is or should be reset on egress doesn't matter to me.

I believe I made the point during the 6MAN meeting that while it's technically possible to change the flow-label back to 0 upon exit of a "flow-label domain", it's likely to be operationally challenging to do so.  IOW, SP's would have to ensure systems/processes are in place to properly configure each of their interfaces appropriately (i.e.: reset the FL to 0 at exit).  Likewise, if my network is downstream from yours ... I have to trust that you've properly configured your interfaces and that your router SW/HW is working properly so that I will always receive a 0 in the FL.  That's asking a lot, IMHO.  :-/

So, while I wouldn't call that "impossible", I would say it's highly unlikely, particularly in [very] large networks.  Of course, that's just my opinion and YMMV.

> Why does a receiving host care about the flow label at all? It exists to make sure that all intermediate nodes give correct treatment to the flow, but once it reaches its destination it's "safe", right?

It depends on your worldview.  I think Brian Carpenter (?) may have said it best in a private e-mail -- let me paraphrase (and, Brian, please correct me if I'm wrong).  The flow-label can belong either to the network -or- to the host(s): pick one[1].  

If you believe the flow-label belongs to the network, then you're statement above is correct ... hosts must ignore (or, not care about) the received flow-label.

OTOH, if you believe the flow-label belongs to hosts and you potentially want to enable applications like draft-blake-ipv6-flow-label-nonce-02, which could prevent off-path attacks, then you (likely) can't have routers messing around with the flow-label.  Routers may read a flow-label, but they can't attempt to change it.

-shane

[1] If you put aside the debate about whether or not locally-defined flow-labels are a good idea, this is likely what the choice about what to do with the flow-label field boils down to: it's either a field that belongs to the network or a field that belongs to the hosts.  Once that is decided, legitimate uses of the flow-label (including, potentially, locally-defined flow-labels) likely fall out from that.

---

From: Aleksi Suhonen <[email-masked]>
Date: Wed, 04 Aug 2010 02:12:50 +0000

Changed subject to better reflect content.

Today I wrote:
>> Why does a receiving host care about the flow label at all? It
>> exists  to make sure that all intermediate nodes give correct
 >> treatment to theflow, but once it reaches its destination it's
 >> "safe", right?

On 08/03/10 23:49, Shane Amante wrote:
> It depends on your worldview. I think Brian Carpenter (?) may have
> said it best in a private e-mail -- let me paraphrase (and, Brian,
> please correct me if I'm wrong). The flow-label can belong either to the
> network -or- to the host(s): pick one[1].

{snip}

> OTOH, if you believe the flow-label belongs to hosts and you
> potentially want to enable applications like
> draft-blake-ipv6-flow-label-nonce-02, which could prevent off-path
> attacks, then you (likely) can't have routers messing around with the
> flow-label. Routers may read a flow-label, but they can't attempt to
> change it.

Hmm ... OK, that makes sense. I don't believe in it tho.

"0 FL mutable" would work reasonably well with both world views,
as there's one chance in a million that rnd() would yield zero.
One connection in a million would fail, and cause a retry?

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Aug 2010 15:21:26 +1200

On 2010-08-04 00:49, Rémi Després wrote:
...
> If a 5-tuple hash is permitted in the future, stateless operation will become possible, and more and more hosts can be expected to set the field.

No, that can't be the reason, because RFC 3697 sets no strong rules -
it doesn't even forbid just counting 1,2,3... as successive labels.
At the time, nobody wanted to set any rules.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Aug 2010 15:48:47 +1200

Aleksi,

>> network -or- to the host(s): pick one[1].

In terms of who sets it, that is the simplest way to state our main
choice, I believe.

> Hmm ... OK, that makes sense. I don't believe in it tho.

You don't believe in the draft-blake proposal?

Another usage is when the destination host is actually a multithreaded
server, which chooses to do internal load balancing based on the flow
label. That raises some very interesting MITM attacks, by swapping
labels between two flows directed to the same server.

> One connection in a million would fail, and cause a retry?

Our draft already says:

       The RECOMMENDED scheme is that, whether set by the source host
       according to RFC 3697, or by an FL router according to the rules
       below, the label contains a pseudo-random value between 1 and
       0xFFFFF.

We are not very far from your proposal or that of Rémi, but the real
question is whether there is consensus for this sort of change,
or simply for a tidied up and clarified RFC3697bis.

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 03 Aug 2010 22:05:15 -0400

{there is a thread which started on [email-masked], and [email-masked],
and then seemed to have dropped [email-masked]. I'm not on [email-masked],
so there likely are message there I've missed}

okay, so I've read Carsen's opinion.
I read RFC 3697 (which I didn't know about).

draft-hu-flow-label-cases-00 is a very good read.  I pull out something
from section 4 (Discussion);

   The other choice, for designers who wish to use the flow label to
   control switching or QoS directly, is to bypass the rules within a
   given domain (a set of cooperating nodes) in a way that nodes outside
   the domain cannot detect.  In this case, any deviation from [RFC3697]
   has no possible effect outside the domain in question.

I don't know where this subject line is from: 12 bits/8bits.
Is there a draft that explains that idea that I've missed?

My claim is that ROLL's RPL is a set of cooperating nodes.

But, it's better than that --- it's a set of routers which are tuned to
support specific applications, and the applications want in this case to
be given information like, "what flow label" to use.  RPL's
RPLinstanceID has all the properties required of a flow label (or,
rather, it has no requirements presently, and therefore can have the
flow label requirements imposed upon it, specifically:
   2.  "Nodes MUST NOT assume any mathematical or other properties of
       the Flow Label"
)

The non-mutability of the label isn't a problem either --- the
applications *AT EACH END*, even if one end of the application is
several AS's away (a very unusual case for 95% of RPL's target use),
that application still needs to know something about what label to use.

There are three cases to consider:
      a) traffic between two RPL nodes
      b) traffic exiting an RPL
      c) traffic entering an RPL

case (a) -- is the "set of cooperating nodes", and therefore is no problem.

case (b) -- the flow label is set to get through the RPL/LLN, and out to
            the network, and the flow label has done it's job, and
            the RPL/LLN network could care less what happens to the flow
            label at that point.   The rest of the network might have a
            problem (i.e. a bug) when RPL networks start sending
            non-zero  flow labels, but that's the rest of the network's
            problem.

case (c) - flow labels of zero are not a problem.  There is either a
         default RPLinstanceID to use (and traffic flows, perhaps not
         optimally), or there isn't (and ICMP Host unreachable occurs).

         - non-zero flow labels which do not map to an RPLinstanceID,
           are simply considered zero, see above.
         - non-zero flow labels which map to RPLinstanceIDs are used.
           *If* it is a problem for outsiders to invoke that LLN's
           DODAG, then there are bigger issues, which the flow label can neither
           help nor hinder --- the flowlabel is not a magic security cookie.
           A firewall may still be required.

The only real problem I can see is when a packet needs to do (b) and
(c).   e.g. use label A to exit LLN alpha, and label B to enter LLN beta.

I don't have a solution to this.  Some have suggested IPIP tunnels,
which sound nice in theory, but in practice do not work in the
wilderness found behind the walls of walled gardens.

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition.

---

From: Rémi Després <[email-masked]>
Date: Wed, 4 Aug 2010 08:41:48 +0200

Hi Aleksi,
Is it right to understand from your comments that you accept the proposed combination (mutable if and only if 0; stateless hash or stateful random number where it is set)?
RD

---

From: Fred Baker <[email-masked]>
Date: Wed, 4 Aug 2010 08:33:29 +0200

On Aug 4, 2010, at 5:48 AM, Brian E Carpenter wrote:

>>> The flow-label can belong either to the network -or- to the host(s): pick one[1].

++

Frankly, much of this discussion has been about dividing the flow label into two parts, one for the host to use to instruct the network on what it wants done with its datagrams, and one for the network to use to manage the flow of datagrams. IMHO, which is of course always right (TM), we already have a field for the host to state what service it wants - the DSCP. Networks overwrite it in IPv4 because they have no other option and they need to be in control of the flow of datagrams. If you want them to not do so, give them a place to tag datagrams.

Or plan on them using MPLS to accomplish their goals, making the entire discussion irrelevant because they aren't looking at the IP (v4 or v6) header.

---

From: Rémi Després <[email-masked]>
Date: Wed, 4 Aug 2010 09:17:48 +0200

Le 4 août 2010 à 05:21, Brian E Carpenter a écrit :

> No, that can't be the reason,

Not "the" reason, just "a" reason ;-).

> because RFC 3697 sets no strong rules -

The quote is: "To avoid accidental Flow Label value reuse, the source node SHOULD select new Flow Label values in a well-defined sequence (e.g., sequential or pseudo-random) and use an initial value that avoids reuse of recently used Flow Label values each time the system restarts".

Right, well-defined sequences aren't all pseudo random, and there is only a SHOULD.
But it does implicitly advise to avoid 5-tuple hashes (they don't produce well-defined sequences). 
Right? 

Tony Hain told me that, to his recollection, the reason why hashes were not clearly permitted was privacy protection.
This seems to me an overkill in the name of privacy.

Explicitly leaving the choice between stateless and stateful would make the rule less strong in favor of statefulness.
It would make it more practicable because of the inherent simplicity of statelessness.

RD

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Aug 2010 10:00:56 +0200

>>>> The flow-label can belong either to the network -or- to the

<aol>

we have spent over a decade trying to find a home/use for flow-labels.
e2e signaling makes simple sense.  let's try to declare victory and move
on.

randy

---

From: Fred Baker <[email-masked]>
Date: Wed, 4 Aug 2010 10:13:43 +0200

intellectually, end to end signaling might make sense. If so, it belongs in the end-to-end headers. More importantly, who's using it?

If using it end to end or end-to-network isn't being useful, either find a use, or deprecate it.

> e2e signaling makes simple sense.  let's try to declare victory and move

http://www.ipinc.net/IPv4.GIF

---

From: Rémi Després <[email-masked]>
Date: Wed, 4 Aug 2010 12:23:05 +0200

Hi Fred,

Le 4 août 2010 à 10:13, Fred Baker a écrit :

> intellectually, end to end signaling might make sense. If so, it belongs in the end-to-end headers.
+1

> More importantly, who's using it?
> 

RFC 3697, the current standard-track RFC on FLs, doesn't need IMHO to be deprecated.
It only needs to be improved by relaxing two unnecessary and harmful constraints (see below).

Reasons to keep RFC 3697 are the following:
- The official purpose of FLs is to facilitate flow-dependent processing in networks.
- At least draft-carpenter-flow-ecmp-02 describes a pragmatic use of such FLs for load sharing among equal-cost paths.
- More generally, any load sharing application could take advantage of these FLs, because they facilitate flow separation:
 . With the help of FLs, two packets having different FLs are known to belong to different flows.
 . Without FLs, scanning complete extension-headers is necessary to find out ports that identify individual flows (a complex task, and one that becomes ineffective in case of datagram fragmentation).

It is a fact that FLs are today generally set to 0 by sources.
But RFC 3697 requires in its section 3 a stateful and rather complex construction of non 0 FLs.
This is bound to discourage host implementors as long as networks don't use FLs.
Now, networks aren't permitted to set FLs themselves even if set to 0 by hosts.
There is then a vicious circle where hosts wait for networks that wait for hosts. 

The two modifications of RFC 3679 that may break the vicious circle are:
- Permit intermediate routers to set FLs in place of hosts if received = 0. 
- Permit hosts, and routers when they set FLs, to use simple hashes of the 2- 3- or- 5-tuples that identify flows. (This stateless behavior could even be recommended so that things are kept simple where there are no good reasons to make them more complex.)

I hope this isn't too long, but it is the best I can see on this subject.
Having a consensus in this direction would permit to leverage what exists (rather than restarting from scratch), and to move on.
If this this approach is retained, I could contribute on detailed changes to RFC 3679, with whoever is interested. 

Regards,
RD

With FLs, rather with a heuristic flow ID

---

From: Steven Blake <[email-masked]>
Date: Wed, 04 Aug 2010 12:42:05 -0400

On Wed, 4 Aug 2010 12:23:05 +0200, Rémi Després <[email-masked]>
wrote:

>> intellectually, end to end signaling might make sense. If so, it
belongs
>> in the end-to-end headers.
> +1
> 
>> If using it end to end or end-to-network isn't being useful, either
find
>> a use, or deprecate it.
> - At least draft-carpenter-flow-ecmp-02 describes a pragmatic use of
such
> FLs for load sharing among equal-cost paths.
> - More generally, any load sharing application could take advantage of
> these FLs, because they facilitate flow separation:
>  . Without FLs, scanning complete extension-headers is necessary to find
>  out ports that identify individual flows (a complex task, and one that
>  becomes ineffective in case of datagram fragmentation).
> 
> There is then a vicious circle where hosts wait for networks that wait
for
> hosts. 
> - Permit hosts, and routers when they set FLs, to use simple hashes of
the
> 2- 3- or- 5-tuples that identify flows. (This stateless behavior could
even
> be recommended so that things are kept simple where there are no good
> reasons to make them more complex.)
> 
> Having a consensus in this direction would permit to leverage what
exists
> (rather than restarting from scratch), and to move on.
> If this this approach is retained, I could contribute on detailed
changes
> to RFC 3679, with whoever is interested. 

I agree with this in principle, but there are still a few issues:

- If the sending host sets FL=0, and an intermediate router resets it
non-zero, the receiving host cannot determine whether the sending host or
an intermediate router set the FL.  This may break some e2e applications of
the FL.

- Permitting sending hosts to use a 2-5 tuple hash to set FL is fine, but
it would better for my pet e2e use of the FL if the hash function was
strongly recommended to be keyed, so that the hash is unpredictable.

It will be awhile before hosts start setting FL non-zero, no matter what
is decided.

Addressing the first issue: for ECMP/LAG load balancing, where a router
receiving a FL=0 packet would like to write a 5-tuple hash into FL, I don't
think that the router needs to write more than 8 bits, because the
downstream routers should still compute a <srcaddr, dstaddr, FL> hash to
determine the load split.  In this case, the router should write an 8-bit
3-tuple hash <protocol, srcport, dstport> into the FL. 

Given this assumption, *one* way to address the first issue is to set
aside a range of FL label values that hosts never set and that routers can
use, such as the following:

- Hosts SHOULD set the FL pseudo-randomly in the range 0x00001 - 0xFFEFF
(in line with RFC 3679, but permitting a keyed 5-tuple hash).
- Routers receiving a packet with FL=0 are allowed to rewrite it
pseudo-randomly in the range 0xFFF00 - 0xFFFFF.
- Routers MUST NOT rewrite a non-zero FL in the range 0x00001 - 0xFFEFF.
- Routers are not required to rewrite a non-zero FL in the range 0xFFF00 -
0xFFFFF to 0.
- E2E applications of the FL should exclude received FL values in the
ranges 0x00000, 0xFFF00 - 0xFFFFF.

This is more complicated than I would prefer, but vastly preferable to
dividing the FL into sub-fields IMHO.

Regards,

// Steve

---

From: Aleksi Suhonen <[email-masked]>
Date: Thu, 05 Aug 2010 02:34:09 +0000

Hi,

Remi Despres a ecrit:
>> changes to RFC 3679, with whoever is interested.

> an intermediate router set the FL.  This may break some e2e applications of

Actually, having had a couple of nights more to sleep thinking
about this, I have a couple more questions regarding this argument:

If the sending host sets FL=0, then the receiving host is clearly
not even expecting anything special in that field?

If some e2e application is using the FL, it will surely be using some
signalling method associated with the FL values?

So when a host receives an FL that doesn't match earlier signalling,
it can determine that it was set by an intermediate node, and that
the original FL was zero?

Is there an internet-draft out there that would answer anything
other than "yes" to the above questions?

Can it be altered with reasonable effort so that it too would
yield "yes"es?

---

From: Aleksi Suhonen <[email-masked]>
Date: Thu, 05 Aug 2010 03:10:48 +0000

Hi,

> You don't believe in the draft-blake proposal?

What I really meant was that I'm not a firm believer that the FL
belongs to end hosts only.

> labels between two flows directed to the same server.

My personal philosophy is that the primary function of the FL
is to tell individual flows apart, so that network behaviour
characteristics can be assigned to and guaranteed for them.
Most of all this could be achieved with DSCPs, but packet
ordering cannot be guaranteed by DSCPs alone.

I'm not against tacking other meanings on the FL, like the few
drafts just mentioned. However, I'm deeply suspicious about trying
to force too much meaning on the FL, especially if the whole
communication can fail due to a change in the FL. There are too
many security implications, both for the host and for the network.

To explain where I'm coming from with my philosophy: Internet
Exchange Points are desperate for any easy sources of entropy
for their switch matrix link aggregation. Most of the traffic
is between a handful of MAC addresses, so they need to look
deeper into the stack, but going beyond the first header just
won't be feasible in IPv6.

If IXPs can't trust the FL as an input to LAG hashes, they might
end up being forced to just forget about packet ordering and do
the best they can with round robin. And then nobody wins.

(And here's to hoping that some day we will have enough IPv6
traffic on an IXP to fill even a single LAG member circuit. O:-)

Having said all of the above, I don't want to stand in the way
of progress. I just wanted to voice my concerns.

Yours,

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Aug 2010 15:34:41 +1200

> the original FL was zero?

That doesn't prevent a case where the intermediate node, for whatever
reason (including MITM), sets a label in the format that does match
the signalling. We don't know what the signalling is like...

> other than "yes" to the above questions?

I think this only works if we assign at least one bit in the label
to mean 'e2e or mutable'. And even that is not immune to MITM forgery.

> Can it be altered with reasonable effort so that it too would

I am pessimistic at the moment, which is why I suggested we have
a binary choice.

    Brian

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 9 Aug 2010 11:53:56 +0200

Hi Brian:

I think this went unanswered, sorry about that. 

RPL voted not to use the flow label because we were afraid we could "shoot ourselves in the foot" by using a method that could become invalid. Instead, the group elected to go to the RPL option in a Hop by Hop header as was presented to 6MAN. This resolution causes RPL border routers to insert additional bytes in every packet for 1) the RPL header, and 2) the IP in IP encaps that comes with it. That really hurts in our constrained environment.

The initial approach (up to draft 7) was to use the flow label as specified in http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 .

As you see, that approach would also leave 12 bits to the network to network communication, and 8 for the app to network communication, in our case to tag a routing topology with the so-called instanceID. If we had pursued that path, we'd probably have had to further specify the 8 bits, in particular to allow for the case where a source leaves the FL it to 0 and the BR actually sets it, and to indicate the FL scope, that is which section of the network the FL setting is supposed to affect (eg source subnet or destination subnet or global).

Cheers,

Pascal

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: Thursday, July 29, 2010 11:35 PM
> To: Michael Richardson
> Cc: Pascal Thubert (pthubert); [email-masked]
> On 2010-07-30 01:11, Michael Richardson wrote:
> ...
> > If the ROLL WG does not feel conformtable with the above, then I think
> > we need to go ask some grandfathers of IPv6... What's the FLOW LABEL
> > FOR then?
> 
> That's the problem. It was never determined. That's why RFC 3697 had such
> limited goals.
> 
> The introduction to draft-hu-flow-label-cases-00 gives a short summary of the
> history.
> 
> >
> > I think our usage of flow label is completely within the pervue of
> > what was intended.
> 
> Can you summarise that use for those of us who don't follow ROLL, and indicate
> how it plays with the rules in RFC 3697?
> 
> Thanks
>      Brian

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 9 Aug 2010 12:17:55 +0200

Hi Michael:

With http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 I
tried to stay within the lines of RFC 3697 as you also defend in your
mail. 

I think the question we have now is not whether that proposal is lawful
but whether the new law being defined at 6MAN would prevent it in the
future.
If the updated rules allow, then I'll be glad to work on an FL-based
alternate to the IPinIP/HbH. 

It appeared at the 6MAN WG meeting that 12 bits mutable was exactly what
the core network was asking for to do its load balancing.
A first question to the group could be whether 12 mutable bits are
enough for the sensible usages envisioned so far?

Pascal

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Michael Richardson
> Sent: Wednesday, August 04, 2010 4:05 AM
> To: [email-masked]; Carsten Bormann
and then
> seemed to have dropped [email-masked]. I'm not on [email-masked], so
there likely
> are message there I've missed}
> 
> draft-hu-flow-label-cases-00 is a very good read.  I pull out
something from
> section 4 (Discussion);
> But, it's better than that --- it's a set of routers which are tuned
to support
> specific applications, and the applications want in this case to be
given
> information like, "what flow label" to use.  RPL's RPLinstanceID has
all the
> properties required of a flow label (or, rather, it has no
requirements presently,
> and therefore can have the flow label requirements imposed upon it,
> specifically:
applications *AT
> EACH END*, even if one end of the application is several AS's away (a
very
> unusual case for 95% of RPL's target use), that application still
needs to know
> something about what label to use.
> 
which sound
> nice in theory, but in practice do not work in the wilderness found
behind the
> walls of walled gardens.
> ] [email-masked] http://www.sandelman.ottawa.on.ca/
|device
> driver[
>    Kyoto Plus: watch the video
> <http://www.youtube.com/watch?v=kzx1ycLXQSE>
> 	               then sign the petition.
> 
> 
> 
>

---

From: Rémi Després <[email-masked]>
Date: Mon, 9 Aug 2010 13:49:58 +0200

> the core network was asking for to do its load balancing.

Would you have more details on WHY it would be "exactly" what was needed (???) ? 

> enough for the sensible usages envisioned so far?

I'd rather see the first question as being "Are there some improvements of RFC 3697 that would make it practicable?".

If yes, improving this RFC rather than deprecating it after 6+ years of standard-track existence would IMHO be better for IPv6 credibility (and IMHO for that of IETF in general).

The two improvements that seem to make sense (see various exchanged mails) are:
- permit stateless support (with flow-ID hashes in place of flow-based stateful processing)
- permit intermediate nodes to set FLs when source hosts haven't done it (still as shorthands for flow ISs made available in IPv6 first headers).

Regards,
RD

---

From: Philip Levis <[email-masked]>
Date: Mon, 9 Aug 2010 08:59:22 -0700

> enough for the sensible usages envisioned so far?

My 2c:

In the case of the hop-by-hop header, 12 bits is clearly enough for an 8-bit RPLInstanceId, but clearly not enough for a 16-bit Rank. I think that trying to compress them (especially Rank) would be a mistake, whose complexity would far outweigh the benefits. 

Phil

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 Aug 2010 11:41:39 +1200

Pascal,

Thanks for this and your previous message. I would need to think
very carefully whether this proposal would really have been strictly
compatible with RFC 3697 - it actually depends on how the agreement
to use these semantics imposed on the flow label is made (what is
called 'flow state establishment' in the RFC). I see more weaknesses
in 3697 each time I look at it :-(

More below...

> alternate to the IPinIP/HbH. 

So I understand that ROLL would definitely need the ability to define
explicit semantics on the label. Would it also need the label to be mutable?

> enough for the sensible usages envisioned so far?

To me it seems that 12 bits could contain enough randomness to drive
a load balancing mechanism, but there's no need for those bits to be
mutable that I can see. On the other hand, 12 bits seems very small for
any kind of nonce or for a flow identification scheme, so it seems
that draft-blake and draft-donley would be knocked out.

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 08:57:00 -0400

>>>>> "Brian" == Brian E Carpenter <Brian> writes:
    >> With
    >> http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 I
    >> tried to stay within the lines of RFC 3697 as you also defend in
    >> your mail.
    >> 
    >> I think the question we have now is not whether that proposal is
    >> lawful but whether the new law being defined at 6MAN would
    >> prevent it in the future.  If the updated rules allow, then I'll
    >> be glad to work on an FL-based alternate to the IPinIP/HbH.

    Brian> So I understand that ROLL would definitely need the ability
    Brian> to define explicit semantics on the label. Would it also need
    Brian> the label to be mutable?

The only case where there is a problem is when there is a packet that
arrives from the outside, to a ROLL "border router" (we don't have such
a term in ROLL. The ROLL term would be grounded DODAG root).

The problem would be if:
    a) the flow label was non-zero.
    b) the flow label named a RPLinstanceID which did exist.
    c) the originating IP was not permitted to use that instanceID
    d) yet we wanted traffic to continue.

In that case, we have a problem and we might want to set the flow-label
to zero.  I think this situation is simply illegal, and I think the
culprit is part (d).

As I indicated in a previous message, the above could happen when there
were two ROLL networks connected together by the "internet" (lower-case
I intentional, this doesn't have to be the public Internet, but it does
have to cross administrative boundaries of some kind), and one needed
flow-label A to exit network 1, and flow-label B to enter network 2.

("needed" --- only that flow label would name the right RPLinstance,
which had the correct set of routing constraints)

    >> It appeared at the 6MAN WG meeting that 12 bits mutable was
    >> exactly what the core network was asking for to do its load
    >> balancing.  A first question to the group could be whether 12
    >> mutable bits are enough for the sensible usages envisioned so
    >> far?

    Brian> To me it seems that 12 bits could contain enough randomness
    Brian> to drive a load balancing mechanism, but there's no need for
    Brian> those bits to be mutable that I can see. On the other hand,
    Brian> 12 bits seems very small for any kind of nonce or for a flow
    Brian> identification scheme, so it seems that draft-blake and
    Brian> draft-donley would be knocked out.

ROLL is sees the label/RPLinstanceID as being an opaque number with no
specific properties.  We actually punt to the administrator to even
define/configure it.

I doubt that many ROLL networks will have as many as 2^8 instances, let
alone 2^12 instances.  The rational for lots of bits would be to
encourage random allocation such that in the event that two
uncoordinated ROLL networks happened to merge (even for a few
minutes!!!) that the likelyhood of cross talk would be reduced.

A merge like I describe here is not directly envisaged by the current
set of requirements in my opinion, but I think that requirement will
emerge in the future.

- --

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 10 Aug 2010 15:34:07 +0200

Hi Brian:
 
> > lawful but whether the new law being defined at 6MAN would prevent it
> > in the future.
> So I understand that ROLL would definitely need the ability to define explicit
> semantics on the label. Would it also need the label to be mutable?
> 

[Pascal] The FL based proposal for RPL uses 12 mutable bits. 

They are used as an in-band control plane that checks the consistency of routing states along a path. Those states can easily get out of sync due to the nature of the links, but the maintenance can be costly. We want to accelerate the repairs on those links that are actually used when an inconsistency is detected that will impact the delivery.

RPL also builds multiple topologies (virtual networks identified by an instanceID) to address various constraints, in terms that can be such as metrics but also administrative. E.g. a power utility Command & Control network at home will have a specific ID that's different from the default general purpose one, so the power related traffic ends up at the power utility GW along a specific graph. 

An opaque instanceID for the topology can be known by the end points. The instanceID is not necessarily related to QoS and does not map to a per hop behavior, but in our case, to acts more like L3 .1Q tag. The Flow Label is the only field available for an application to pass such information to the network. Thus the non-mutable 8 bits in the proposal. 

> >
> > It appeared at the 6MAN WG meeting that 12 bits mutable was exactly
> > what the core network was asking for to do its load balancing.
> To me it seems that 12 bits could contain enough randomness to drive a load
> balancing mechanism, but there's no need for those bits to be mutable that I
> can see. On the other hand, 12 bits seems very small for any kind of nonce or
> for a flow identification scheme, so it seems that draft-blake and draft-donley
> would be knocked out.

I'm unsure why draft_donley does not copy the IPv4 TOS Byte in the IPv6 Traffic Class Octet, if differentiated service is what the draft is after. 
In any case, it appears that the need is not really for a flow but for a flow type, in which case 8 non mutable bits are enough.

I'm also unsure why draft blake can be preferred over syn cookies, but if it is, then, obviously the more bits the better. Still, this nonce is not used by the network, so the network header should not be encumbered by it. If you look at it, a similar problem appears when a node receives a packet back in an ICMP error, in which case it would be safe to make sure that the node really issued that packet in the first place, otherwise actions taken upon the ICMP could be misused by an attacker from anywhere in the Internet.. If we take the path of a nonce, I'd favor a new "source option" with semantics by which the network is not concerned with the header and  that would enable the source to place its own information such as this nonce.

Thanks a bunch for your help,

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 10 Aug 2010 16:24:42 +0200

Salut Rémi,

Please see below:

Pascal

> From: Rémi Després [mailto:[email-masked]]
> Sent: Monday, August 09, 2010 1:50 PM
> Cc: 6man 6man; Michael Richardson; [email-masked]; Carsten Bormann

[Pascal] That does not come from me. I asked at the mike during the WG meeting, and the answer that was given was that for the purpose of a load balancing hash in the SP network, 11 or 12 bits would be enough.

> If yes, improving this RFC rather than deprecating it after 6+ years of standard-
> track existence would IMHO be better for IPv6 credibility (and IMHO for that of
> IETF in general).
> 
> - permit intermediate nodes to set FLs when source hosts haven't done it (still
> as shorthands for flow ISs made available in IPv6 first headers).
> 
[Pascal] We agree here. The point if that there might be enough room for both, depending on what we are doing.
Note that for the latter, the discussion was also that it is not obvious at the egress of the network to reset the FL to zero in order to enable the next AS to reuse it. So the conclusion was rather to use the bits is a DSCP fashion.

Pascal

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 10:27:39 -0400

>>>>> "Pascal" == Pascal Thubert <(pthubert)" <[email-masked]>> writes:
 
    >> On 2010-08-09 22:17, Pascal Thubert (pthubert) wrote: > Hi
    >> Michael:
    >> >
    >> > With
    >> http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 I >
    >> your > mail.
    >> > I think the question we have now is not whether that proposal
    >> is > lawful but whether the new law being defined at 6MAN would
    >> prevent it > in the future.  > If the updated rules allow, then
    >> I'll be glad to work on an FL-based > alternate to the
    >> IPinIP/HbH.
    >> 
    >> So I understand that ROLL would definitely need the ability to
    >> define explicit semantics on the label. Would it also need the
    >> label to be mutable?
    >> 

    Pascal> [Pascal] The FL based proposal for RPL uses 12 mutable bits.

    Pascal> They are used as an in-band control plane that checks the
    Pascal> consistency of routing states along a path. Those states can
    Pascal> easily get out of sync due to the nature of the links, but
    Pascal> the maintenance can be costly. We want to accelerate the
    Pascal> repairs on those links that are actually used when an
    Pascal> inconsistency is detected that will impact the delivery.

I don't understand how the FL helps us on this!  Probably I missed
something important.  What you write below was my entire understanding
of our need for an in-band tag.

    Pascal> RPL also builds multiple topologies (virtual networks
    Pascal> identified by an instanceID) to address various constraints,
    Pascal> in terms that can be such as metrics but also
    Pascal> administrative. E.g. a power utility Command & Control
    Pascal> network at home will have a specific ID that's different
    Pascal> from the default general purpose one, so the power related
    Pascal> traffic ends up at the power utility GW along a specific
    Pascal> graph.

    Pascal> An opaque instanceID for the topology can be known by the
    Pascal> end points. The instanceID is not necessarily related to QoS
    Pascal> and does not map to a per hop behavior, but in our case, to
    Pascal> acts more like L3 .1Q tag. The Flow Label is the only field
    Pascal> available for an application to pass such information to the
    Pascal> network. Thus the non-mutable 8 bits in the proposal.

+1

---

From: Rémi Després <[email-masked]>
Date: Tue, 10 Aug 2010 17:11:16 +0200

Hi Pascal,
More comments below.

> [Pascal] That does not come from me. I asked at the mike during the WG meeting, and the answer that was given was that for the purpose of a load balancing hash in the SP network, 11 or 12 bits woul…

What is clear, though, is that a hash of 11-12 bits has a higher probability of collisions than a hash on 20 bits.
If hashes become permitted (which simplifies support of FLs for load balancing), it is then clear that 20 is better than 12.

> Note that for the latter, the discussion was also that it is not obvious at the egress of the network to reset the FL to zero in order to enable the next AS to reuse it.

RFC 3697 isn't concerned with ASes, and doesn't need to be.
The proposal is only that, where load balancing is performed, 0 FLs MAY be replaced by meaningful values for this purpose.
A FL, once set to a non 0 value, never needs to be reset.

> So the conclusion was rather to use the bits is a DSCP fashion.

I am not sure what is meant by used in a DSCP fashion (some DSCP actual uses seem to me rather complex).

In any case, using FLs ONLY as FLs (essentially RFC 3697, but completed to be more practicable), is IMHO the most reasonable approach to improve the situation in a reasonable timeframe.

I understand that 20 unused bits in IP headers is bound to arouse envy, but adding a new role to FLs, with the original role becoming less efficient rather than more efficient, is IMHO a wrong approach. 

Cheers,
RD

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 11:21:13 -0400

>>>>> "Joel" == Joel M Halpern <[email-masked]> writes:
    Joel> Off-list, although you may decide that my confusion is wider, and take it
    Joel> back to the list:

okay, I think it's worthwhile.

    Joel> When a ROLL network is talking to the rest of the world, using IPv6, it would
    Joel> seem that packets can arrive with
    Joel> a) a non-zero flow label
    Joel> b) which happened to be an extant RPLinstanceID
    Joel> c) and apparently the sender, since it is outside of ROLL, is not permitted
    Joel> to use that instanceID
    Joel> d) but we clearly want the traffic to continue

    Joel> WHhy is this at all unlikely?  Are you counting on not having
    Joel> collisions between values in a 20 bit field?  It simply isn't
    Joel> big enough for that to be a safe assumption.

I think that (d) is unlikely.
That is, random nodes on the network, not only will not have a reason to
talk to the light bulb in my living room, but also won't be permitted to
do so.
(If it's actually me at the random IP, then likely I'll come through an
IPsec tunnel...) 

I'm not saying that this will apply to *ALL* ROLL deployments, but to
the majority of them that are envisaged today.  (In fact, the things I
want to deploy likely will suffer from this more than the average will)

In particular, the commodity situation where some node on my network
decides to do an HTTP request to commodity site FOO, and the packets in
question here are replies to my request.  

I do not know how the reply flow labels will be set, but if it will be
simply echoed by TCP passive opens, then in fact, (c) might not apply.
That is, the node will have permission to use that instanceID.  

It's not because the node is outside of the ROLL  that it can not use
the instanceID. It's just that nodes outside the ROLL won't know much
about the instanceIDs, if the applications running on them have not been
configured properly.

That is, my network device used instanceID XYZ to send the request out,
and it is appropriate for replies to also use XYZ. There might even be a
stateful firewall opening a pinhole on the ACL for instanceID XYZ.

If it should come that it is accepable for an intermediate node to reset
the FL to zero, that would solve the problem for incoming packets.

I guess, I'd rather we allocate well known FL=1 (or 0b1111...) to
indicate, "has been reset" rather than "was never set", I'd be happier.

If it is permissible for a "firewall" to drop packets with non-zero FLs,
then that might also solve the problem.  I'd hate to see that in any
kind of commodity system: I've had it with "secure" banks that can not
speak ECN, or that have broken PMTU.

- --

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 12:09:26 -0400

>>>>> "Rémi" == Rémi Després <[email-masked]> writes:
    Rémi> RFC 3697 isn't concerned with ASes, and doesn't need to be.
    Rémi> The proposal is only that, where load balancing is performed, 
    Rémi> 0 FLs MAY be replaced by meaningful values for this purpose.   
    Rémi> A FL, once set to a non 0 value, never needs to be reset.

okay, so what you are saying is that load balancing uses of the FL are
only upset when they see zero.  So for instance, if layer-4s (i.e. end
points) were mandated that they must now always set a FL consistently on
a flow, and set it to a non-zero value, that this would satisfy the
requirements of load balancers.

In this context, permission would be given to set zero FL to a non-zero
value.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 12:28:07 -0400

>>>>> "Carsten" == Carsten Bormann <[email-masked]> writes:

    >> a term in ROLL. The ROLL term would be grounded DODAG root).

    Carsten> I haven't yet quite found out whether RPL tries to be
    Carsten> useful for networks with hosts, but if it does, the same
    Carsten> kind if problem would occur when a host sets the flow
    Carsten> label. 

RPL networks consists of leafs and routers.  Both typically act as hosts.
Routers are just hosts that happen to be between other nodes.
(Although, some hosts are too weak to be routers)

Little to no traffic on the RPL is unaware of which RPLinstanceID to
use, because the applications there needs to send using a specific set
of constraints, which is encapsulated into the RPLinstanceID.

For ignorant applications, the OS may well be able to set the
instanceID.  In other cases, an instanceID of zero may be most correct.

There may be some cases where there are ordinary hosts which are on a
network which is not RPL, but is connected to an RPL network. 
(The group does not have any consensus if these will be layer-2 bridged,
or layer-3 routed yet).  In these cases the application either will be
aware, or won't care.

- --

---

From: Philip Levis <[email-masked]>
Date: Tue, 10 Aug 2010 10:29:10 -0700

On Aug 10, 2010, at 9:28 AM, Michael Richardson wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 

...

> or layer-3 routed yet).  In these cases the application either will be

I feel like we're running in circles, in part due to different expectations of how RPL will be used.

It's clear that in proprietary or vertically integrated networks running RPL, it's possible to state how controllers/other nodes set the flow label in order to play nice with RPL. E.g., if my home automation network receives commands from a special device (which itself might have a web/SOAP/etc. interface), then that special device can set the flow label. This isn't the problem case.

The problem case for the use of the flow label is when arbitrary Internet hosts (e.g., my laptop) want to communicate with devices running RPL. E.g., I have a home RPL network and want to directly talk to a RPL device from my laptop. Requiring the packet source to be aware of flow labels and how to set them breaks the whole idea of Internet connectivity: suddenly an end host needs to be aware of what routing protocol is used to reach the other host, as well as configuration parameters of that host. I need to be able to SSH to the RPL-connected device without requiring a new ssh program that uses ioctls to set flow labels. We can't say that only OSes which set the flow label to 0 (or some other value) can be used. Nor can we expect to change how OSes set the flow label.

So the problem is that a packet originating from outside the RPL network, destined to a RPL node, may have an arbitrary flow label. The RPL border router needs to handle the packet in an effective way.  Whatever solution we come up with needs to handle this effectively if the Internet is going to support connectivity to LLN devices. It's critical that the solution not make assumptions about how the flow labels are set or what the behavior of the non-RPL end host is. We want Windows, Linux, OS X, Windows Mobile, Android, Symbian, and other current Internet devices to be able to talk with LLNs.

Phil

---

From: Fred Baker <[email-masked]>
Date: Tue, 10 Aug 2010 12:06:54 -0700

I would find that surprising. There are ample cases where the originator of a high data rate flow (sensor data from a radio telescope to a number cruncher, to pick one example) might want to use the flow label to send data from one session down multiple paths. Multi-path TCP would be another example. I suppose you could argue that consistently alternating among N flow labels can be thought of as N separate flows, but if it's TCP there are other ramifications.

In any event, the requirement of the load balancer is not that the originator gives things a tag; we can develop such tags from a hash of the source and destination address quite well, thanks. The important thing is that things are tagged in a manner that helps the load balancer. There are two major uses of load balancers. In data centers, we balance sessions to/from sets of servers, and the question when a new session comes up is which server is most likely to be able to effectively serve it (has the necessary resources including access to the right disks, CPU cycles, etc). In bandwidth allocation uses (spreading load across multiple otherwise-equivalent paths), the question is how to use each of the paths most effectively - we want some traffic on each of them and we want none of the paths to be overloaded or introducing more delay or loss than others. Everyone's instinctive knee-jerk response to the question seems to be "hash something and depend on the law of large numbers to serve the need", and the observation of numerous operators is that the law of large numbers is a good start but is not an adequate solution. You really want the load balancer to be able to be smarter than that.

Which is why people that build load balancers frequently do so using NAT technology or some other way of tagging individual sessions. And why they are looking for mutable flow labels in this thread.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Aug 2010 18:04:33 +0200

> a term in ROLL. The ROLL term would be grounded DODAG root).

I haven't yet quite found out whether RPL tries to be useful for networks with hosts, but if it does, the same kind if problem would occur when a host sets the flow label.

Gruesse, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Aug 2010 18:16:13 +0200

On Aug 10, 2010, at 14:57, Michael Richardson wrote:

> The rational for lots of bits would be to

Hmm.  If this is intended as a routine way to handle the situation, the likelihood is still way to high.
(If this is just intended as a mitigation to increase robustness in a case that already is an error situation that is being actively avoided by other means, your argument might be more interesting.)

> set of requirements in my opinion, but I think that requirement will

DANGER.  Adding complexity to a protocol to cater for requirements that don't exist yet leads to convoluted solutions.  Never do that.
(Note that this situation is different from the selection of one out of a set of same-complexity alternatives, where it is quite useful to conjecture about potential future requirements to generate a tie breaker.  Note also that extensibility by itself is a known current requirement for any protocol, which is quite different from designing in all conceivable future extensions.)

Gruesse, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Aug 2010 20:21:56 +0200

> (Although, some hosts are too weak to be routers)

OK, I'm not talking of "host" as in originates or terminates traffic, but "host" in the sense of "does not participate in routing".
It appears there is no such thing inside a RPL world then.
(Note that the traditional usage I'm using here is slightly different from the definition used in RFC2460, which says "does not forward" -- the old 1980's "hosts with RIP routing" would be hosts in RFC2460, but we have learned not to like these monsters.  That's why IPv6 has a separate ES-IS protocol called ND.)

> of constraints, which is encapsulated into the RPLinstanceID.

Hosts (as in "don't participate in routing") would have little idea of such concepts.

What you are describing is somehow surprisingly remote from IP.
In IP, hosts send packets to an IP address.
Sometimes there are additional considerations, which we so far have been addressing [sic] with the TOS byte, i.e., the DSCP today.
I don't understand yet why RPL needs a different concept.

(Note that Phil's point is also valid, but mostly orthogonal to mine.)

Gruesse, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 16:09:07 -0400

>>>>> "Philip" == Philip Levis <[email-masked]> writes:
    Philip> I feel like we're running in circles, in part due to
    Philip> different expectations of how RPL will be used. 

    Philip> It's clear that in proprietary or vertically integrated
    Philip> networks running RPL, it's possible to state how
    Philip> controllers/other nodes set the flow label in order to play
    Philip> nice with RPL. E.g., if my home automation network receives
    Philip> commands from a special device (which itself might have a
    Philip> web/SOAP/etc. interface), then that special device can set
    Philip> the flow label. This isn't the problem case. 

    Philip> The problem case for the use of the flow label is when
    Philip> arbitrary Internet hosts (e.g., my laptop) want to
    Philip> communicate with devices running RPL. E.g., I have a home
    Philip> RPL network and want to directly talk to a RPL device from
    Philip> my laptop. Requiring the packet source to be aware of flow
    Philip> labels and how to set them breaks the whole idea of Internet
    Philip> connectivity: suddenly an end host needs to be aware of what
    Philip> routing protocol is used to reach the other host, as well as
    Philip> configuration parameters of that host. I need to be able to
    Philip> SSH to the RPL-connected device without requiring a new ssh
    Philip> program that uses ioctls to set flow labels. We can't say
    Philip> that only OSes which set the flow label to 0 (or some other
    Philip> value) can be used. Nor can we expect to change how OSes set
    Philip> the flow label. 

I guess I don't see the problem to be as big a deal as you suggest.
I'm happy if flow label 0 gets some default RPLinstanceID.  

It would be convenient if the rules were relaxed such that on ingress,
the RPL edge router could *set* the FL if it were zero, and it appears
that this matches up well with what the load balancing people want to
do.

Today, if you ssh or HTTP or what-not, without doing anything special,
you will get a flow label of zero, which with the above rule, means you
get connectivity.  Only if some intermediate node sets the FL=0 would it
be a problem today, and that's illegal so far.

BTW: the flowlabel is not in an ioctl, it's right in the sockaddr_in6.
Too bad that inet_ntop(3)/RFC2373/4291 was not specified to include the flow
label, as then it would be just a matter of user education...  Maybe
there is some way to use the %ifname notation used for ifindex mapping.
(okay, it falls apart once you want to use DNS)

It seems to me that within RPL, if we use RPLinstanceID, we need to
decide what to do when we get a flow label from an untrusted source that
is non-zero.  Inserting an end-host option, with the unstrusted FL
seems like the right thing to do to me.  This facility might be
generally useful to entire Internet.

- --

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 16:16:03 -0400

>> (Although, some hosts are too weak to be routers)

    Carsten> OK, I'm not talking of "host" as in originates or
    Carsten> terminates traffic, but "host" in the sense of "does not
    Carsten> participate in routing". 

    Carsten> It appears there is no such thing inside a RPL world then.

Yes, by that definition that's perhaps a fair statement!
I prefer to think of each device as being a router, which an attached host.
(the ip_output()/ip_input() parts... are the host, the ip_forward() is
the router...) 

    >> of constraints, which is encapsulated into the RPLinstanceID.

    Carsten> Hosts (as in "don't participate in routing") would have
    Carsten> little idea of such concepts. 

Yes, but the *applications* do know about such things.

    Carsten> In IP, hosts send packets to an IP address.

You can do that.  Your packet might not get there in time.  It might
take a much longer route than necessary.  It might not get there at all,
if whatever default RPLinstance that exists does not include your destination.

    Carsten> Sometimes there are additional considerations, which we so
    Carsten> far have been addressing [sic] with the TOS byte, i.e., the
    Carsten> DSCP today. 

If you prefer, you may think of the RPLinstance as naming something akin
to an MPLS or ATM "circuit", except that it's NBMA-ish tree rather than P2P.

- --

---

From: Fred Baker <[email-masked]>
Date: Tue, 10 Aug 2010 13:53:29 -0700

> It appears there is no such thing inside a RPL world then.

A RPL or Manet world doesn't have the 1970's-mentality limitation that hosts have to be stupid. Of all networking architectures, the Internet is the only one with that delusion.

---

From: Mark Smith <[email-masked]>
Date: Wed, 11 Aug 2010 07:09:34 +0930

Hi Pascal,

On Tue, 10 Aug 2010 16:24:42 +0200

> > Le 9 août 2010 à 12:17, Pascal Thubert (pthubert) a écrit :
<snip>
> Note that for the latter, the discussion was also that it is not obvious at the egress of the network to reset the FL to zero in order to enable the next AS to reuse it. So the conclusion was rathe…

The DSCP domain model use was my inspiration when I originally suggested
the idea of having flow domains. If the flow label stays mutable in
the network, the issue with the mandatory resetting to zero
idea is that downstream domains probably aren't going to trust upstream
domains to always do it - so they'll probably choose to do it
themselves anyway. Which is similar to how upstream DSCP values are or
rather aren't trusted.

One use of IPv4 DSCP I've seen in the past was to label or classify
packets for billing purposes. I disagreed with using DSCP for that,
because it is specifically for QoS use. It seems to me that the IPv6
flow label field is or could be a more general purpose field, and if
that DSCP labelling idea had been carried forward to IPv6, would be the
field to use for it.

Regards,
Mark.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Aug 2010 09:53:01 +1200

Thanks for all the enlightment about ROLL.

My personal conclusion is that the ROLL considerations
are too complex and too subtle to be compatible with using
a general-purpose IPv6 header field (i.e. the flow label)
for ROLL purposes. They seem to be an extreme case of the
challenges of defining a local-use regime for the flow label,
which have already been a stumbling block for the various
versions of draft-carpenter-6man-flow-update.

So IMHO, ROLL/RPL was quite wise to drop the IPv6 header flow label
proposal.

     Brian

---

From: Philip Levis <[email-masked]>
Date: Tue, 10 Aug 2010 17:18:09 -0700

> be a problem today, and that's illegal so far.

Can you promise me that every device in the Internet does this, i.e. that there are no IP stacks which set the flow label?

Phil

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 21:05:56 -0400

>>>>> "Fred" == Fred Baker <[email-masked]> writes:
    Fred> I would find that surprising. There are ample cases where the
    Fred> originator of a high data rate flow (sensor data from a radio
    Fred> telescope to a number cruncher, to pick one example) might
    Fred> want to use the flow label to send data from one session down
    Fred> multiple paths. Multi-path TCP would be another example. I
    Fred> suppose you could argue that consistently alternating among N
    Fred> flow labels can be thought of as N separate flows, but if it's
    Fred> TCP there are other ramifications. 

    >> requirements of load balancers.

I don't mean to say that the FL has to be a single value: I guess it
could be from a set of values if one wants to do multi-path.

    Fred> In any event, the requirement of the load balancer is not that
    Fred> the originator gives things a tag; we can develop such tags
    Fred> from a hash of the source and destination address quite well,
    Fred> thanks. The important thing is that things are tagged in a

Well... if FL is unique per sender's concept of flow, that's 128+20 bits
for the tcam vs 256+8+32 bits if you use the 5-tuple.  (why use a tcam
for this... I don't know... there are no wildcard bits.  A cam or
patricia-tree would be as good...)

    Fred> manner that helps the load balancer. There are two major uses
    Fred> of load balancers. In data centers, we balance sessions
    Fred> to/from sets of servers, and the question when a new session
    Fred> comes up is which server is most likely to be able to
    Fred> effectively serve it (has the necessary resources including
    Fred> access to the right disks, CPU cycles, etc). In bandwidth
    Fred> allocation uses (spreading load across multiple
    Fred> otherwise-equivalent paths), the question is how to use each
    Fred> of the paths most effectively - we want some traffic on each
    Fred> of them and we want none of the paths to be overloaded or
    Fred> introducing more delay or loss than others. Everyone's
    Fred> instinctive knee-jerk response to the question seems to be
    Fred> "hash something and depend on the law of large numbers to
    Fred> serve the need", and the observation of numerous operators is
    Fred> that the law of large numbers is a good start but is not an
    Fred> adequate solution. You really want the load balancer to be
    Fred> able to be smarter than that. 

    Fred> Which is why people that build load balancers frequently do so
    Fred> using NAT technology or some other way of tagging individual
    Fred> sessions. And why they are looking for mutable flow labels in
    Fred> this thread. 

So, they want to mutate all flow labels, or just non-zero ones?

    > Rémi> A FL, once set to a non 0 value, never needs to be reset.

    mcr> okay, so what you are saying is that load balancing uses of the FL are
    mcr> only upset when they see zero.  So for instance, if layer-4s (i.e. end
    mcr> points) were mandated that they must now always set a FL consistently on
    mcr> a flow, and set it to a non-zero value, that this would satisfy the
    mcr> requirements of load balancers.
    mcr> 
    mcr> In this context, permission would be given to set zero FL to a non-zero
    mcr> value.

---

From: Rémi Després <[email-masked]>
Date: Wed, 11 Aug 2010 08:12:36 +0200

> requirements of load balancers.

Right.

To be even more precise: 
- Flow endpoints (sometimes layer 4 and sometimes layer 3) should from now on be mandated to set FLs with non-0 values that statistically differ from a flow to another.
- However, we have to face that, so far, they are generally mandated to set FLs to 0.
- To mitigate the negative effect of this fact on uses of FLs as heuristic flow discriminants, intermediate nodes may replace 0 values by values that statistically differ from a flow to another.

Note also that intermediate nodes typically don't reassemble multi-packet datagrams. 
In this common case, they can only identify the IPv6 flow of a packet based on its two addresses and its next header. 
Setting the FL with a hash of this ID is still useful for downstream nodes to be able to use it (rather than treating the packet as if in the carryall set of 0-FL packets, or having to recompute its flow ID). 

Regards,
RD

---

From: Rémi Després <[email-masked]>
Date: Wed, 11 Aug 2010 09:06:54 +0200

Le 10 août 2010 à 21:06, Fred Baker a écrit :

> I would find that surprising. There are ample cases where the originator of a high data rate flow (sensor data from a radio telescope to a number cruncher, to pick one example) might want to use the flow label to send data from one session down multiple paths. Multi-path TCP would be another example.

> I suppose you could argue that consistently alternating among N flow labels can be thought of as N separate flows, but if it's TCP there are other ramifications.

Packets that are expected to go down multiple paths should be identified as belonging to different flows as far as FLs are concerned.
In the case of multi-path TCP, where "one or both peers are expected to have multiple addresses", this is already the case: different paths are taken by packets of different MPTCP "subflows", i.e. packets having different 5-tuples.

> 
> In any event, the requirement of the load balancer is not that the originator gives things a tag; we can develop such tags from a hash of the source and destination address quite well, thanks.

Originators of TCP and UDP connections should clearly hash 5-tuples when they exist (not 2-tuples in this case as you seem to assume).

> The important thing is that things are tagged in a manner that helps the load balancer.

Agreed.

> There are two major uses of load balancers.

> In data centers, we balance sessions to/from sets of servers, and the question when a new session comes up is which server is most likely to be able to effectively serve it (has the necessary resources including access to the right disks, CPU cycles, etc). In bandwidth allocation uses (spreading load across multiple otherwise-equivalent paths), the question is how to use each of the paths most effectively - we want some traffic on each of them and we want none of the paths to be overloaded or introducing more delay or loss than others. Everyone's instinctive knee-jerk response to the question seems to be "hash something and depend on the law of large numbers to serve the need", and the observation of numerous operators is that the law of large numbers is a good start but is not an adequate solution.

"Hash something" (and possibly a 2-tuple even when a 5-tuple is available) is clearly not as good as "hash 5 tuples if available", which I propose to become a SHOULD.

Taking a "good start" as starting point, and trying to improve from there, is what I try to propose.  

> You really want the load balancer to be able to be smarter than that.
> 
> Which is why people that build load balancers frequently do so using NAT technology or some other way of tagging individual sessions.

Are you arguing in favor of NATs in IPv6, adding load balancing as a reason to use them :-(?

As a convinced promoter of IPv6, and of its restored e2e transparency, I find that IPv6 load balancing solutions that don't need NATs are worth specifying.

> And why they are looking for mutable flow labels in this thread.

I don't see why 0-FL mutability wouldn't be sufficient for them (provided new values must statistically different for different flows, including subflows, micro-flows, whatever can be identified).

But more data on what would be their problem are most welcome. 

Regards,
RD

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 11 Aug 2010 12:00:38 +0200

> I don't understand how the FL helps us on this!  Probably I missed
something
> important.  What you write below was my entire understanding of our
need for
> an in-band tag.

[Pascal] This point is not about the flow label specifically but about
the RPL strategy, and it works whether we use the flow label or the hop
by hop techniques to transport the info. In short, the mere fact that
there is a packet on the broken path causes us to 1) detect and 2) fix
the issue. If there's no packet on a broken path then it can stay broken
for a while. There will probably be many broken paths at any point of
time in the network, due to transient link interference, movement,  and
all sorts of routing desyncs. Considering the cost of proactive rapid
resolution, we have chosen not to fix the routing issues as soon as they
appear but 1) lazily and 2) on-demand.
The lazy piece is a global reconstruction of the DAG that happens
periodically. The on-demand piece is a data path validation that allows
us to fix a path as it is being used, either using a mutable flow label
field or the Hop by Hop header.

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 11 Aug 2010 12:00:36 +0200

Hi Brian:

The Hop by Hop is certainly the clean solution. 

The trouble is that it requires additional bytes in every packet for the
header and for the IP-in-IP encapsulation that goes with it; yet RPL
operates in a domain where devices can be strictly constrained in energy
and frames can be very small, so every bit counts quite dramatically.
Thus we're looking for more affordable alternates. We have started a
6MAN document on the IP in IP avoidance for instance.

Here, the point is not to get the flow label definition impacted by RPL
but to see if the flow label can be used by RPL within the definition
that 6MAN will refine. If we have enough mutable bits, then we can allow
an alternate mode in RPL whereby those they are used as opposed to the
HbH. 

For the instance ID, it's certainly a lot more complex since we are
defining an interaction between the app layer and the network. I do not
know if the result of a 6MAN work on such interaction can be used by RPL
in the end, but I think it is a good subject to address before we make
all bits mutable.

My initial take would be that there should at least be one bit somewhere
that indicates that the source does not want (a piece of) the FL to be
modified. 

Pascal

Of
> Brian E Carpenter
> Sent: Tuesday, August 10, 2010 11:53 PM
> To: Fred Baker (fred)
> Cc: Carsten Bormann; [email-masked]; [email-masked]; Michael Richardson
> Subject: ROLL choice to not use the Flow Label
> 
> My personal conclusion is that the ROLL considerations are too complex
and
> too subtle to be compatible with using a general-purpose IPv6 header
field (i.e.
> the flow label) for ROLL purposes. They seem to be an extreme case of
the
> challenges of defining a local-use regime for the flow label, which
have already
> been a stumbling block for the various versions of
draft-carpenter-6man-flow-
> update.
proposal.
> >> OK, I'm not talking of "host" as in originates or terminates
traffic, but "host"
> in the sense of "does not participate in routing".
> > A RPL or Manet world doesn't have the 1970's-mentality limitation
that hosts
> have to be stupid. Of all networking architectures, the Internet is
the only one
> with that delusion.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 11 Aug 2010 10:48:43 -0400

>>>>> "Pascal" == Pascal Thubert <(pthubert)" <[email-masked]>> writes:
    Pascal> Hi Brian:

    Pascal> The Hop by Hop is certainly the clean solution.

    Pascal> The trouble is that it requires additional bytes in every
    Pascal> packet for the header and for the IP-in-IP encapsulation
    Pascal> that goes with it; yet RPL operates in a domain where
    Pascal> devices can be strictly constrained in energy and frames can
    Pascal> be very small, so every bit counts quite dramatically.  Thus
    Pascal> we're looking for more affordable alternates. We have
    Pascal> started a 6MAN document on the IP in IP avoidance for
    Pascal> instance.

    Pascal> Here, the point is not to get the flow label definition
    Pascal> impacted by RPL but to see if the flow label can be used by
    Pascal> RPL within the definition that 6MAN will refine. If we have
    Pascal> enough mutable bits, then we can allow an alternate mode in
    Pascal> RPL whereby those they are used as opposed to the HbH.

So one solution is for RPL to just do what it wants to inside the LLN,
and use IP-in-IP for packets that arrive at the LLN border with a
non-zero FL.  (ditto for packets that have to leave)

This makes the extra bytes the exception rather than the rule.

It also can be turned off in closed networks where the FL has been
coordinated across the AS.

This would be facilitated by the rule that we can set a zero FL to a
non-zero value.

---

From: Philip Levis <[email-masked]>
Date: Wed, 11 Aug 2010 16:34:32 -0700

> - Flow endpoints (sometimes layer 4 and sometimes layer 3) should from now on be mandated to set FLs with non-0 values that statistically differ from a flow to another.

The intention is to have a BCP for network stack implementers to follow?

> - However, we have to face that, so far, they are generally mandated to set FLs to 0.

I apologize for the lack of context (I'm coming from ROLL): your sentence seems to suggest that flow labels today are mandated to be 0. This doesn't seem to be right: among other things, ping6 supports setting the flow label, and by default allocates a random flow label.[1] Basically, I'm confused if you're talking in the present tense of what's done with flow labels today, or the future tense of how flow labels should be used in the future.

Phil

[1] http://linux.die.net/man/8/ping6

---

From: Philip Levis <[email-masked]>
Date: Wed, 11 Aug 2010 16:41:32 -0700

> field or the Hop by Hop header.

Maybe a rewording might make this clearer.

Basically, RPL puts up to two pieces information in packets that it routes. The first is which routing topology this packet should be routed on: RPL supports multiple parallel topologies, e.g., one optimized for low latency and a second optimized for low energy. This information is called the RPLInstanceId. 

The second is information about the "cost" of the forwarder's route to the destination. This hop-by-hop "cost" information is called Rank. Rank is needed to detect routing loops: if a router receives a packet whose prior router's Rank is less than or equal to its own, Rank is not strictly decreasing and there may be a routing loop.

Being able to embed this information in the flow label, rather than requiring additional headers and IP-in-IP, would greatly reduce header overhead. In RPL, we're often concerned about very short, such as 10 byte, packets, so this could be a big savings.

The challenge is that the RPLInstanceId is 8 bits and Rank is 16 bits.

Does that make sense?

Phil

---

From: Philip Levis <[email-masked]>
Date: Wed, 11 Aug 2010 16:45:56 -0700

> So IMHO, ROLL/RPL was quite wise to drop the IPv6 header flow label

Brian,

I agree that the technical challenges seems significant. It was clear that, as we were writing RPL, we didn't yet know how to do it, and so we dropped the idea from the base draft. 

That being said, the possible benefits are pretty enormous to RPL networks. I'm therefore of the opinion that we're doing the right thing, trying to think this through and reach consensus on if there's a way to do use the flow label cleanly and effectively. The answer may be no, but the answer may also be yes: only discussion and good engineering design will let us figure it out.

Phil

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Aug 2010 14:02:31 +1200

> The intention is to have a BCP for network stack implementers to follow?

I don't there is a clear intention just yet, but my personal view is coming
round to a 3697bis document, which would presumably be Proposed Standard,
not BCP. But certainly we need more precise normative guidance.

> I apologize for the lack of context (I'm coming from ROLL): your sentence seems to suggest that flow labels today are mandated to be 0. This doesn't seem to be right: among other things, ping6 supp…

What 3697 says is
"A Flow Label of zero is used to indicate packets not part
of any flow.
...
A source node which does not assign traffic to flows MUST set
the Flow Label to zero."

It's a little self-referential, since flow *by definition* is
a set of packets with the same flow label (and address pair).
The rule is there to prevent accidental flow labels because
the programmer simply forgot to zero the field.

That being so, it isn't clear to me that we need a mandatory
flow label, although I can make some arguments for a SHOULD.

    Brian

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 12 Aug 2010 10:47:34 +0200

> > the RPL strategy, and it works whether we use the flow label or the
> > hop by hop techniques to transport the info. In short, the mere fact
> > that there is a packet on the broken path causes us to 1) detect and
> > 2) fix the issue. If there's no packet on a broken path then it can
> > stay broken for a while. There will probably be many broken paths at
> > any point of time in the network, due to transient link
interference,
> > movement,  and all sorts of routing desyncs. Considering the cost of
> > proactive rapid resolution, we have chosen not to fix the routing
> > issues as soon as they appear but 1) lazily and 2) on-demand.
> > periodically. The on-demand piece is a data path validation that
> > allows us to fix a path as it is being used, either using a mutable
> > flow label field or the Hop by Hop header.
> 
> Basically, RPL puts up to two pieces information in packets that it
routes. The
> first is which routing topology this packet should be routed on: RPL
supports
> multiple parallel topologies, e.g., one optimized for low latency and
a second
> optimized for low energy. This information is called the
RPLInstanceId.
> 
> The second is information about the "cost" of the forwarder's route to
the
> destination. This hop-by-hop "cost" information is called Rank. Rank
is needed
> to detect routing loops: if a router receives a packet whose prior
router's Rank
> is less than or equal to its own, Rank is not strictly decreasing and
there may be
> a routing loop.
> 
> Being able to embed this information in the flow label, rather than
requiring
> additional headers and IP-in-IP, would greatly reduce header overhead.
In RPL,
> we're often concerned about very short, such as 10 byte, packets, so
this could
> be a big savings.

[Pascal] Agreed. We'll note that the Hop by Hop + IP in IP is costly but
solves the generic problem *within* the RPL network. The use of the Flow
label *within* the RPL network would be an alternate so it could have a
more limited applicability, like a more limited number of instances or a
Rank floor that can be expressed in 1 octet or less, which is probably
more than enough considering that infinity is 16 in RIP. This could
cover a large number of foreseeable deployments.

There is certainly a strong benefit in Low Power Lossy Networks to make
the flow label mutable. Still, making it all mutable will definitely
block a communication path between the app and the network, which I feel
is a mistake.

Pascal

---

From: Rémi Després <[email-masked]>
Date: Thu, 12 Aug 2010 14:54:46 +0200

Le 12 août 2010 à 10:47, Pascal Thubert (pthubert) a écrit :
> We'll note that the Hop by Hop + IP in IP is costly but
> the flow label mutable.

How strong the benefit isn't clear to me.
Since RPL networks are recognized as such by hosts, it seems easy to add a short field, just before the IP header, to contain whatever is useful for this kind of network.
Mixing functions with those of FLs would thus be avoided.

Did I miss something?
RD

---

From: Rémi Després <[email-masked]>
Date: Thu, 12 Aug 2010 15:01:12 +0200

Le 12 août 2010 à 04:02, Brian E Carpenter a écrit :

> not BCP.

+1

> But certainly we need more precise normative guidance.

Yes.
RD

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 12 Aug 2010 16:00:06 +0200

Yes, Rémi, you are.

We have explained that inserting data requires a new header and thus IP in IP encapsulation. This happens for instance when a packet comes from the Internet into a RPL network. Which we can hardly accept in constrained networks an devices. In most cases, we could squeeze the information we need in the Flow Label should it be mutable.
 
Pascal

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Rémi Després
> Sent: Thursday, August 12, 2010 2:55 PM
> Cc: [email-masked]; Carsten Bormann; [email-masked]; Brian E Carpenter;
> [email-masked]
> Subject: Re: [Roll] Flow Label: 12 bits mutable and 8 bits immutable
> 
> 
> > We'll note that the Hop by Hop + IP in IP is costly but solves the
> > generic problem *within* the RPL network. The use of the Flow label
> > *within* the RPL network would be an alternate so it could have a more
> > limited applicability, like a more limited number of instances or a
> > There is certainly a strong benefit in Low Power Lossy Networks to
> > make the flow label mutable.
> 
> Since RPL networks are recognized as such by hosts, it seems easy to add a
> short field, just before the IP header, to contain whatever is useful for this kind
> of network.

---

From: "George, Wes E [NTK]" <[email-masked]>
Date: Tue, 17 Aug 2010 10:53:39 -0500

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Tuesday, August 03, 2010 1:20 PM
Subject: Re: Flow Label: 12 bits mutable and 8 bits immutable

Because of your last two bullets I have to ask the following.  How would a receiving host deterministically distinguish (1) flow-labels that were created by network devices (just a 5-tuple was put into a flow-label) vs. (2) flow-labels that were created by a source-host w/ a pseudo-random number + 5-tuple[1]?  (Please read on before answering :-)
[[WEG]] Perhaps I'm oversimplifying, but wouldn't it be possible to use one bit to flag who set it? 0 for end-host 1 for network or vice versa? Yes, that requires us to define who falls into what category, but that's manageable.

This e-mail may contain Sprint Nextel proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.

---

From: Shane Amante <[email-masked]>
Date: Tue, 17 Aug 2010 11:59:37 -0600

Wes,

> [[WEG]] Perhaps I'm oversimplifying, but wouldn't it be possible to use one bit to flag who set it? 0 for end-host 1 for network or vice versa? Yes, that requires us to define who falls into what c…

IMHO, this would likely be the start of a slippery slope.  Specifically, a few points to bear in mind:
1)  Once you "steal" one bit to note who set the flow-label, who's the say that requests to steal more bits don't surface?  For example, perhaps someone wants another bit to indicate if the remaining bits are mutable (i.e.: changeable by other downstream network elements) or immutable?  Or, perhaps, a third (or, fourth) bit to indicate if the flow-label is: a) 'locally-defined'; b) a "normal" hash of the IPv6 header's 3- or 5-tuple; or, c) some other well-defined application (e.g.: ROLL's request for a "InstanceID" to be inserted into the FL field).  Granted, some applications could get compressed together and one might be able to overload one-bit position to mean multiple things ... but, in general, this is just asking for trouble and, inevitably, people would want to leave 1 or 2 bits unused/reserved so that if/when you needed to define yet more interpretations of the FL you had bits to do so ...
2)  We have to get router vendors to build the logic to pay attention to and/or ignore the appropriate combination of "flags" bits in order that, for example, intermediate routers would be able to identify *when* to use the left-over FL bits as an input key for LAG and/or ECMP.  (Likewise, host vendors, etc. would have to implement similar logic to know when they should pay attention and parse incoming FL's).

Instead of all that complexity, I'd really prefer a very simple approach that just clarified the semantics of the flow-label in RFC 3697, preferably along the the lines that FL's must be constructed in a way that uniquely identifies a "flow" to intermediate routers (or, L2/L2.5 switches) in the network ... in order that a FL could always "safely" be used as an input-key to LAG or ECMP load-balancing.  Assuming that general design philosophy is agreed upon, then it's likely a variety of "application"-type drafts can safely be built upon a RFC 3967bis, e.g.: draft-blake, draft-donley, etc., assuming they don't conflict with the underlying semantics of the FL.

-shane

---

From: "George, Wes E [NTK]" <[email-masked]>
Date: Tue, 17 Aug 2010 14:10:51 -0500

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Tuesday, August 17, 2010 2:00 PM
To: George, Wes E [NTK]
Cc: Pascal Thubert (pthubert); 6man 6man; [email-masked]; Mohacsi Janos; Tony Hain
Subject: Re: Flow Label: 12 bits mutable and 8 bits immutable

Wes,

[[WEG]] I get that, and I share your concern. I'm not necessarily advocating this as a solution, more thinking out loud. However, since we're already mulling the idea of a 12/8 split (as per the subject line) maybe stealing a few bits for administrative flags is not as big of a deal given the value that might (?) be derived from having the info. That's all I'm saying.

[[WEG]] Agree, this is non-trivial, although it's not like router vendors aren't already going to have to do some implementation work in order to be able to look for/act on/(re)write the value in the FL header that they're presently ignoring. I think the LOE is going to be about the same as long as we define it up front and don't change it midstream.

[[WEG]] I generally agree (I always prefer simpler), but I think it'll end up coming back to your original assertion about hosts vs network, and if we have to try to compromise so that both can use it, you might end up needing some of that complexity.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Aug 2010 09:57:24 +1200

On 2010-08-18 07:10, George, Wes E [NTK] wrote:

> but I think it'll end up coming back to your original assertion about hosts vs network, and if we have to try to compromise so that both can use it, you might end up needing some of that complexity.

That is completely obvious to me from this very complex thread, and
from the complexity of some of the proposals I have seen for encoding
meaning in the bits of the label (including an earlier version of
draft-carpenter-6man-flow-update).

The only way I can envisage even a 51% rough consensus is on something
very simple.

Also remember that we already have 6 mutable bits to define per-hop
behaviour, and they are in practice used only for very simple deployments
with about three or four different behaviours at most. I'd like to hear
from some core router development engineers about their plans for using
a lot more mutable bits...

    Brian