Subject: Flow label (im)mutability

From: Brian E Carpenter <[email-masked]>
Date: Wed, 08 Sep 2010 13:18:41 +1200

Hi,

The authors of draft-carpenter-6man-flow-update (now also
including Shane Amante) are working on a new version. One
fundamental issue that has come up is about the (lack of)
security properties of the flow label. The most brutal
expression of this is:

The flow label field is always unprotected (no IP header
checksum, not included in transport checksums, not included in
IPsec checksum). It cannot be verified and can be used as a
covert channel, so it will never pass a security analysis. Thus
some firewalls *will* decide to clear it, whatever the IETF
wants. This is inevitable, for exactly the same reason that the
diffserv code point is rewriteable at domain boundaries.

If this is correct, it is futile to assert that the flow label
MUST be delivered unchanged to the destination, because we
cannot rely on this in the real world.

Are we ready to accept this analysis?

-- 
Regards
   Brian Carpenter

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 07 Sep 2010 21:32:00 -0400

While there may be a few firewalls that will do whatever they think they 
need to in order to shut down covert channels, I do not see that as a 
significant factor.  I imagine most devices will not do so, since it 
does represent a meaningful threat to the site being protected.
(There are other covert channels available, that I have never heard of 
conventional commercial firewalls attempting to close.  Heck, I could 
have a pattern of connections to two remote sites for a covert channel. 
  I don't buy this as a driver.)

Yours,
joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 07 Sep 2010 21:50:06 -0400

That was supposed to read "since it does NOT represent a meaningful threat."
Joel

On 9/7/2010 9:32 PM, Joel M. Halpern wrote:
> significant factor. I imagine most devices will not do so, since it does
> represent a meaningful threat to the site being protected.
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 7 Sep 2010 22:44:22 -0400

> Are we ready to accept this analysis?

what's the threat if it changes in flight? multiple times even?

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 08 Sep 2010 15:38:16 +1200

Below...

> what's the threat if it changes in flight? multiple times even?

That presumably depends on the use case. The idea is that someone
figures out what flow label values will screw you, and sets such
values. Let's assume you're using it for ECMP or LAG. You're hashing
the flow label as part of the ECMP/LAG hash. Someone figures out
how to compute a flow label for each packet arriving on your 10GB
line that will bias your hash, and therefore defeat the load sharing.

Note, I'm not saying it will happen, just that it might, and that
seems to be how some security people think.

We can choose to not worry about this, but that's why I want to
discuss it.

   Brian

---

From: Fred Baker <[email-masked]>
Date: Wed, 8 Sep 2010 12:48:06 +0900

>> cannot rely on this in the real world.

Anything that cannot be proven to have the same original value at the end of the ride has this characteristic.

>> Are we ready to accept this analysis?

I, personally, am very ready to accept this analysis. It might be key to making the flow label useful.

> what's the threat if it changes in flight? multiple times even?

Speaking for myself, I see no threat. On the other hand, the question to me is what might be the benefit. What has been proposed is some variation on a load-sharing hash, one that might, for example, let the ingress node in a network direct specified traffic streams to one of several egresses - essentially what is done using MPLS but without the fuss and bother.

One could imagine this being something akin to an ECMP route, but without any real discussion of "routing cost". Imagine that we could assign egress numbers to routers - akin to an IP address, but fits in 20 bits. An ingress node could look at some tuple (source/destination address, DSCP, whatever else) and decide that the correct egress router is <something>, and put the <something>'s number in the flow label to direct traffic towards the egress. One could imagine several policies - traffic toward a given egress is up to some limit rate, and arriving traffic that exceeds that rate goes to a different egress for example, or 1/3 of sessions this way and 2/3 that, or whatever served the ISP.

---

From: Fred Baker <[email-masked]>
Date: Wed, 8 Sep 2010 12:53:07 +0900

On Sep 8, 2010, at 12:38 PM, Brian E Carpenter wrote:

> The idea is that someone
> figures out what flow label values will screw you

In the model I proposed, the network the packet is in, as with the DSCP, is in control of the flow label value.

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 8 Sep 2010 00:00:45 -0400

> line that will bias your hash, and therefore defeat the load sharing.

If the flow label is used as the only input to the hash then changing
it willy nilly will likely change the LAG member (or ECMP path), this
MAY cause pain for the ISP, it MAY additionally cause pain for the end
users of the flow. If the flow label is the only input, there may be
less reliable traffic spreading/sharing than is desired anyway.

It seems that either of the above isn't really good, but whether or
not the flow label changes in flight (even multiple times) it doesn't
look like it'd matter in a significant way, to me. I'd defer to Shane
though, since he seemed to have some strong feelings about this.

> We can choose to not worry about this, but that's why I want to

so far my vote is to not worry...

-chris

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 8 Sep 2010 00:02:51 -0400

> Anything that cannot be proven to have the same original value at the end of the ride has this characteristic.

seems like a fine idea.

> One could imagine this being something akin to an ECMP route, but without any real discussion of "routing cost". Imagine that we could assign egress numbers to routers - akin to an IP address, but…

this all gets 'crazy', I suppose if we wanted to route on flow-label
not destination-ip-address this might happen, but ... that seems
'crazy' as I said before :) since not everyone would be using the
flow-label (maybe) and inconsistent routing would suddenly make the
internet look a lot like a PSTN network (to me).

'crazy' I say!!

-chris

---

From: Fred Baker <[email-masked]>
Date: Wed, 8 Sep 2010 14:37:17 +0900

> internet look a lot like a PSTN network (to me).

well, we could talk about MPLS... that has always seemed to me a might PSTN-ish.

My presumption here is that flow label routing would be within a single administration or a set of consenting administrations, and as such would not be affected by whether neighboring administrations chose to use it r chose how to use it.

One example of a use case for it is in the IAB-TE.pdf deck (google that, it's under iab.org somewhere) that Jason Schiller used in the IAB Routing Workshop a few years back. One of the questions was how to simplify this case: We have N+1 ISPs, one that is in a bad way and N others trying to get data to it. They are interconnected by some number of undersea cables (and therefore lambdas), which are in turn operated by multiple administrations. 

        ,-------.           ,-------.
     ,-'         `-.     ,-'         `-.
    /     ISP #1    \   /     ISP #2    \
   (                 ) (                 )
    \               /   \               /
     `-.         ,-'     `-.         ,-'
        `-------' ||     || `-------'
                  ||     ||
          Undersea||     ||Undersea
             Cable||     ||Cable
           Network||     ||Network
                  ||     ||
                  ,-------.
               ,-'         `-.
              /               \
             (     ISP #3      )
              \               /
               `-.         ,-'
                  `-------'

ISP #3 has a problem; due to the issues of laying undersea cable, "just add bandwidth" is a lot more easily said than done. The total amount of traffic destined to it approximates the capacity of the cables/lambdas in use, but of course varies in composition over time. As a result, ISP #3 is forever gerrymandering /24s to bias traffic towards one link or another. ISPs #1 and #2 go slightly mad with route flaps as a result.

What I am about to describe doesn't solve load spreading across ISPs #1 and #2. But, for the traffic that arrives at let's say ISP #1, if its ingress nodes were each given permission to route traffic for ISP #3 to one of the relevant cables/lambdas up to some rate, and then to a second cable/lambda up to some rate, and so on. The sum of the rates granted to the ingress points is less than or equal to the rate of the cable/lambda in question. What this means is that ISP #1 is in control of its own routing without being micromanaged by the ISP downstream, and can sell a service to ISP #3 of delivering the indicated data in a manner that is reasonably load balanced across the cables/lambdas connecting them.

One could just as easily imagine doing that with MPLS; the point is to have an appropriate set of tags. I just happen to not be fond of circuit or virtual circuit networks, and I think this doesn't require the explicit N^2 problem implied by a mesh of LSPs.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 8 Sep 2010 10:44:49 +0200

> diffserv code point is rewriteable at domain boundaries.

Two related remarks:

There is currently no writeup of how to use the flow label for ECMP. And as far as I can tell there are no implementations of this either. Which is a real shame.

There is work going on on creating "multipath TCP" where a TCP flow is split into subflows which take different paths. (See the MPTCP wg.) Currently, it is assumed that the paths are defined by the source/destination address pairs, but there are many paths that can't be selected this way. A different way to do this would be to have a path selector value in packets which the MPTCP (or other multipath transport) can use to tell routers to use different paths for different subflows. The flow label would be a very good choice for this, it would then bascially be a "subflow label".

Considering the above, in my opinion:

- we shouldn't lock down the flow label such that only one flow label per flow is allowed because this would impede future innovation

- zero flow labels are still created by many systems, but these would hamper a flow label based ECMP. Rewriting zero flow labels into a real flow label somewhere in the network would therefore be a useful function

- arbitrarily changing flow labels could break stuff like flow label based multipath and flow label based ECMP

In other words: the flow label wouldn't be immutable, but non-zero values SHOULD NOT be rewritten.

---

From: Rémi Després <[email-masked]>
Date: Wed, 8 Sep 2010 11:23:22 +0200

> Are we ready to accept this analysis?

IMHO,yes.

The consequence could be that a FL:
- SHOULD be set by the packet source to a value that generally differs from a flow to another (e.g. a 5-tuple hash)
- MAY be reset to zero in intermediate nodes, but only for security reasons
- MAY be reset to a non-zero value in intermediate nodes, provided this value generally differs from a flow to another.
An intermediate node that can identify 3-tuples but not 5-tuples SHOUL NOT reset FLs to non-zero values.

RD

---

From: Rémi Després <[email-masked]>
Date: Wed, 8 Sep 2010 11:33:44 +0200

Le 8 sept. 2010 à 05:38, Brian E Carpenter a écrit :
> ... Let's assume you're using it for ECMP or LAG. You're hashing

Depending on whether the ECMP/LAG operator is more or less paranoid about this threat, it could:
- ignore received FL values, and base its ECMP/LAG hash on its own 3- or -5-tuple hashes
- OR take the risk, and use received FL values.

This could IMHO remain an implementation/operation option.

RD

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 8 Sep 2010 08:52:59 -0400

> - MAY be reset to zero in intermediate nodes, but only for security reasons

clarifying question: "only for security reasons"
which are? (some examples at least here, perhaps not in the end-text)

-chris

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 8 Sep 2010 08:57:57 -0400

> well, we could talk about MPLS... that has always seemed to me a might PSTN-ish.

yea... sad, eh? (but handy at times, sadly)

> My presumption here is that flow label routing would be within a single administration or a set of consenting administrations, and as such would not be affected by whether neighboring administratio…

ah! I missed the 'single admin domain' part in the original message.
Inside a single admin-domain sure, this looks like mpls without the
mpls-label, maybe that's an advantage, maybe not.

> One example of a use case for it is in the IAB-TE.pdf deck (google that, it's under iab.org somewhere) that Jason Schiller used in the IAB Routing Workshop a few years back. One of the questions wa…

yup, the latam example.

(nice art!)

> What I am about to describe doesn't solve load spreading across ISPs #1 and #2. But, for the traffic that arrives at let's say ISP #1, if its ingress nodes were each given permission to route traff…

seems crazy still (still managing state somewhere), but I grant the
example works out in principle. As you say below, today this is done
with mpls or other encap technologies (or could be), using something
NOT encap'd sounds like a win.

> One could just as easily imagine doing that with MPLS; the point is to have an appropriate set of tags. I just happen to not be fond of circuit or virtual circuit networks, and I think this doesn't…

thanks! (that clarifies it for me, and I hope gets us out of a rathole)
-Chris

---

From: Rémi Després <[email-masked]>
Date: Wed, 8 Sep 2010 14:59:27 +0200

Le 8 sept. 2010 à 14:52, Christopher Morrow a écrit :

>>> covert channel, so it will never pass a security analysis.

<<<
>>> Thus
>>> wants.

>>> This is inevitable, for exactly the same reason that the

This was simply in reference to the isolated sentence above (between <<< and >>>).
Regards,
RD

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 8 Sep 2010 09:09:41 -0400

> This was simply in reference to the isolated sentence above (between <<< and >>>).

ok, thanks.
-Chris

---

From: Steven Blake <[email-masked]>
Date: Wed, 08 Sep 2010 10:40:50 -0400

> Are we ready to accept this analysis?

FWIW, covering the FL in a header/transport checksum would not guarantee
immutability, since a firewall could always re-calculate either of these.

There are already a variety of covert channels available (e.g., packet
size, packet timing, DSCP, hop count), so I wouldn't lose sleep about the
FL adding an additional one.

Regards,

// Steve

---

From: Shane Amante <[email-masked]>
Date: Wed, 8 Sep 2010 10:53:37 -0600

Brian, Steve,

> immutability, since a firewall could always re-calculate either of these.

Good point.  However, a firewall *could* calculate a (pseudo-header) checksum (inclusive of the recv'd flow-label, which is not specified today) to verify that the received flow-label is "intact".

Of course, this doesn't guarantee that a middlebox/MiTM somewhere in the path didn't alter the flow-label and recalculate + replace the pseudo-header checksum.  Further, your point above is still completely valid that a legitimate firewall (or, more generally, middlebox) can rewrite the FL and recalculate the checksum and forward it on to an internal host and the internal host would be none the wiser.  IMHO, that would be unfortunate, however the extra effort in changing the FL and recalculating the pseudo-header checksum doesn't seem worth it for mainstream firewall vendors or, more importantly, FW administrators.

> FL adding an additional one.

+1.

I would add two points:
1)  If FW administrators are that paranoid about covert channels, they are (or, should be) using IPSec encryption to avoid potential tampering with [inner/tunneled] packets in the first place -- or, disconnect from the Internet.  :)
2)  Assuming something like draft-gont-6man-flowlabel-security-00 and/or draft-blake-ipv6-flow-label-nonce-02 were eventually to be adopted, the benefits of those proposals in preventing/reducing off-path attacks would substantially outweigh (ridiculous) beliefs that the FL could be used as a covert channel.

-shane

---

From: "George, Wes E [NTK]" <[email-masked]>
Date: Wed, 8 Sep 2010 12:13:12 -0500

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Tuesday, September 07, 2010 9:19 PM
To: 6man
Subject: Flow label (im)mutability

Hi,

The flow label field is always unprotected (no IP header
checksum, not included in transport checksums, not included in
IPsec checksum).
[[WEG]] I think that this alone should be reason to update/invalidate the appropriate language in 3697, perhaps with some discussion of alternate means of carrying immutable packet header data that would otherwise end up in the flow label. It's a lot harder to dispute the technical basis behind this assertion plus the "it is futile to assert..." than it is your continuing thought about security below.

It cannot be verified and can be used as a

[[WEG]] Yes, but as I implied above, I think the security angle is just one example of several illustrating why expecting flow label to be immutable end to end is not realistic, rather than the primary argument, and for that reason I'd like to see a little more separation between the two. Ultimately, firewall policy is a correctable (or at least containable) problem, especially if one or more end applications need flow label and are important enough to the folks on the other side of said firewall, and therefore I'm not sure you'd get consensus that it alone constitutes a deal-killer for immutable flow label. I'd rather see some additional examples - one that I'd add is general bit-error corruption by the lower-layer transport media. Since it's not covered by any checksum, it is quite possible that this field can be corrupted by a bit error and not cause the packet/frame to be discarded. Since most networks are not 100% error-free, this is a reasonable risk. Yes, it's unlikely that FL bits would be corrupted without any other impact very often, but it's worth covering when being complete about why we want to change this.

This e-mail may contain Sprint Nextel proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 14:29:02 -0300

Hi, Brian,

> line that will bias your hash, and therefore defeat the load sharing.

I think that the lesson learned from similar fields in other protocols
is that the Flow Label needs to be unpredictable by an off-path attacker.

This was the motivation for writing the stuff in
draft-gont-6man-flowlabel-security (which was written quite some time
ago, but only recently published as an IETF I-D).

Thanks!

Kind regards,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 14:43:24 -0300

Hi, Iljitsch,

> There is currently no writeup of how to use the flow label for ECMP.
> And as far as I can tell there are no implementations of this either.
> Which is a real shame.

There are a few proposals on how to set it (e.g., Blake's and mine). --
This could be a good way to start.

> - we shouldn't lock down the flow label such that only one flow label
> per flow is allowed because this would impede future innovation

The only problem with this is that if you allow a given flow to use
multiple flows, the flow-label reuse frequency might increase, and
collisions might occur. -- then you violate the requirement on
uniqueness fo each pair (src addr, dst addr).

> - zero flow labels are still created by many systems, but these would
> hamper a flow label based ECMP. Rewriting zero flow labels into a
> real flow label somewhere in the network would therefore be a useful
> function

Some of these boxes set the flow label inconsistently. See the paper
published by Malone et al (referenced in Blakes flow-label I-D, and
probably in mine, too).

If you want consistency, you might have to rewrite both, unfortunately.

Thanks!

---

From: Thomas Narten <[email-masked]>
Date: Wed, 08 Sep 2010 14:08:34 -0400

> values.

And exactly how is this is different that what we have today, where
the same can be done by setting  values for the src/dest addresses and
ports?

> Let's assume you're using it for ECMP or LAG. You're hashing

And is easier (and thus more of a threat) than, say, any variety DDOS
attack that targets a particular bottleneck link?

> seems to be how some security people think.

Let's discuss the threat (if there is one), but let's put it in
context. How much worse a threat (or more exploitable of one) is this
than other existing ones?

Thomas

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 15:21:57 -0300

> the same can be done by setting  values for the src/dest addresses and

No worse *if* you randomize the Flow Label -- because we do randomize
the source ports.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 15:34:29 -0300

> which are? (some examples at least here, perhaps not in the end-text)

See draft-gont-6man-flowlabel-security-00.txt

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 15:42:13 -0300

Hi, Steven,

> FL adding an additional one.

Eliminate all cover channels is virtually impossible. However, of the
ones you mentioned, the Flow Label is the one that is more reliable and
that provides more bandwidth.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 15:43:19 -0300

Hi, Steven,

> FL adding an additional one.

Eliminating all cover channels is virtually impossible. However, of the
ones you mentioned, the Flow Label is the one that is more reliable and
that provides more bandwidth.

Thanks,

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 8 Sep 2010 22:51:13 +0200

> collisions might occur.

So what?

Obviously if you have a couple of gigabit streams then it would be nice if they don't share a flow label. But if you have 5000 streams it really doesn't matter if a few have the same flow label.

In the multipath case this is even less of a problem because if a certain flow label value doesn't work well, it will just use another one.

> Some of these boxes set the flow label inconsistently.

Not sure what you mean with "inconsistent" but I'll read it as "undesired".

That should not be taken as a reason to overwrite the flow label, though. If that becomes common practice then a whole class of possible uses goes out the door.

Remember, the flow label is an optimization, use it if it's helpful, ignore it otherwise. Don't try to be smarter than everyone else and impose your own logic, nobody stays smarter than everyone else for long.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 09 Sep 2010 10:29:31 +1200

Thanks for all the discussion. Just commenting on a few points:

On 2010-09-09 08:51, Iljitsch van Beijnum wrote:
...
> Remember, the flow label is an optimization, use it if it's
> helpful, ignore it otherwise.

I wonder if we all agree on this? I do, but people who want to
encode specific semantics in the label clearly don't.

If we do agree on this, it's very helpful, because it guides all
further decisions. For example, it allows us to see that the
label is immutable on a best effort basis, rather than
mathematically immutable. So we ccould say both that forwarding
nodes MUST NOT change the label, *and* that downstream nodes
MUST NOT rely on the value being unchanged.

On 2010-09-08 20:44, Iljitsch van Beijnum wrote:
...
> There is currently no writeup of how to use the flow label
> for ECMP. And as far as I can tell there are no
> implementations of this either. Which is a real shame.

There is work in progress, but IMHO we need to resolve the
issues in RFC 3697 before this can be completed.

> 
> There is work going on on creating "multipath TCP" where a
> TCP flow is split into subflows which take different paths.
> (See the MPTCP wg.) Currently, it is assumed that the paths
> are defined by the source/destination address pairs, but
> there are many paths that can't be selected this way. A
> different way to do this would be to have a path selector
> value in packets which the MPTCP (or other multipath
> transport) can use to tell routers to use different paths for
> different subflows. The flow label would be a very good
> choice for this, it would then bascially be a "subflow
> label".

Actually, that is not forbidden by RFC 3697 (therefore it's
allowed). Read carefully the definition of a flow in Section 1:

"A flow is a sequence of packets ... that the source
 desires to label as a flow."  (etc.)

> - we shouldn't lock down the flow label such that only one
> flow label per flow is allowed because this would impede
> future innovation

Since it's the source that decides what a flow is, this has
been the case since 2004.

> - zero flow labels are still created by many systems, but
> these would hamper a flow label based ECMP. Rewriting zero
> flow labels into a real flow label somewhere in the network
> would therefore be a useful function
> 
> - arbitrarily changing flow labels could break stuff like
> flow label based multipath and flow label based ECMP
> 

I agree. Except for the firewall problem, that is.

> the same can be done by setting  values for the src/dest addresses and

Well, rewriting the addresses or ports will misdirect a packet.
We don't actually know what rewriting the flow label would do,
since we don't know what it's being used for. So it is a bit
different, but I agree that it's the same class of risk.

As Fernando and Steve have pointed out, the argument is the
other way round for off-path attacks - a randomised flow
label will be even harder to guess than a randomised port.

   Brian

---

From: Fred Baker <[email-masked]>
Date: Thu, 9 Sep 2010 07:57:41 +0900

Brian:

Joel reminds me, in private email, that nobody seems to be commenting on you original question, which has to do with covert channels and whether that should affect the discussion of flow labels.

Here's a covert channel for you. I was at an agency last year that uses specialized mail services, the kind that (for the right email under the right circumstances) might decide to deliver a certain email to a set of persons within a stated period of time. They basically really wanted to have the service, but didn't want to have to upgrade their very special version of a mail system to mark DSCPs - at most, they wanted to reconfigure the mail server.

I asked them if the equipment they used could support a second NIC; yes. I asked them if they could configure their email rules to send using one interface in the "routine" case and the other in the "flash" case; yes.

I pointed out that if they were to do so, on the router in front of the server, I could, with an ACL, mark the DSCP for them. My sales guy commented that I wasn't helping him sell a fancy bit of gear.

In this case, the IP Source Address contains a covert channel; certain kinds of email transactions can be identified by their source address.

Hmm. Should we allow source addresses through security gear? What crazy implications might there be regarding such covert channels?

I think the covert channel question is a red herring. Yes, creative people can do creative things. And the point is...?

Fred

---

From: Fernando Gont <[email-masked]>
Date: Wed, 08 Sep 2010 20:05:33 -0300

Hi, Brian,

> Are we ready to accept this analysis?

Yes.

Note (in response to comments about the covert channelthing): the reason
for which firewalls/packet scrubbers do/will modify the flow label is
irrelevant. But assuming that it gets delivere unchanged, when it
doesn't doesn't make sense.

Also, see the paper published by Malone (referenced in Steven Blake's
flowlabel I-D and possibly in draft-gont-6man-flowlabel-security). Some
systems don't even set *themselves* the flow-label consistently. In
practice, this is the same thing as if a middlebox was modifying it.

Thanks!

Kind regards,

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 9 Sep 2010 07:49:08 +0200 (CEST)

On Thu, 9 Sep 2010, Brian E Carpenter wrote:

> If we do agree on this, it's very helpful, because it guides all further 
> decisions. For example, it allows us to see that the label is immutable 
> on a best effort basis, rather than mathematically immutable. So we 
> ccould say both that forwarding nodes MUST NOT change the label, *and* 
> that downstream nodes MUST NOT rely on the value being unchanged.

Last I checked, the standards said that if precedence/dscp changed, the 
host should reset the session (correct me if I'm wrong, I don't really 
have time to check it right now).

In real life, ISPs consider DSCP as one thing they have the right to 
change (along with TTL) in transit. I can imagine the flow label being 
considered the same thing regardless of what the standard says.

For me, the flow label would be most useful in tunnel scenarios. We have 
mobile base station controllers which outputs hundreds of megabits/s of 
tunneled packets and this is all between the same SRC/DST IP. The way 3GPP 
does things, this is only going to increase. Being able to use flow labels 
to identify an individual user tunnel and thus being able to hash more 
efficiently when doing load sharing would be most beneficial.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 9 Sep 2010 07:51:12 +0200 (CEST)

On Thu, 9 Sep 2010, Fred Baker wrote:

> I think the covert channel question is a red herring. Yes, creative 
> people can do creative things. And the point is...?

I think the lesson is that security minded people (the same people who 
wants to filter all ICMP because ICMP is always evil to them), will want 
to reset the flow label. Some people might think this is wrong but it's 
going to happen anyway. Make the standard permissive of this scenario is 
the only way forward that I can see.

---

From: Tim Chown <[email-masked]>
Date: Thu, 9 Sep 2010 12:05:01 +0100

On 9 Sep 2010, at 06:49, Mikael Abrahamsson wrote:
> 
> In real life, ISPs consider DSCP as one thing they have the right to change (along with TTL) in transit. I can imagine the flow label being considered the same thing regardless of what the standard says.

The interesting thing in the academic networks through Europe and I believe Internet2 as well is that - the last time I checked - there is agreement between the NRENs (national academic ISPs) to pass DSCP values unaltered between networks.

In general in those academic networks the only rewriting that may occur is at site exit routers where site policy may either determine the DSCP value a packet is marked with, or trump the DSCP value set by a user/application.    I believe where policing detects excess traffic of a certain DSCP value, that traffic is dropped rather than being remarked.

There are agreed semantics for DSCP values between multiple NRENs, e.g. for Premium, BE and LBE, who also agree to not alter the DSCP in transit, even though there's nothing stopping them doing so per se.

I don't see why the Flow Label should be treated differently.   If cooperating ISPs agree to pass it unaltered, that's fine.   As the spec stands, it's hard to see how we could say otherwise.

Tim

---

From: Rémi Després <[email-masked]>
Date: Thu, 9 Sep 2010 15:04:42 +0200

Hi Iljitsch,

Le 8 sept. 2010 à 10:44, Iljitsch van Beijnum a écrit :
> ...
> There is work going on on creating "multipath TCP" where a TCP flow is split into subflows which take different paths. (See the MPTCP wg.) Currently, it is assumed that the paths are defined by the source/destination address pairs, but there are many paths that can't be selected this way.

Subflows of MPTCP have different 5-tuples.
They should be treated as different flows as far as load balancing is concerned.

> A different way to do this would be to have a path selector value in packets which the MPTCP (or other multipath transport) can use to tell routers to use different paths for different subflows. The flow label would be a very good choice for this, it would then bascially be a "subflow label".
> 

This shows the need to define what is meant by "flow" independently in EACH context.

For load balancing, what is key is that:
- packets of a FL flow should be forwarded without changing their original order (subject to best effort)
- packets of different FL flows may have their order freely modified within the network.

If MPTCP people would have considered one "meta-flow" comprising several "flows", there would have been little risk of confusion between MPTCP flows and FL flows.
But the fact is that than they talk about one "flow" comprising several "subflows".
Then an MPTCP subflow is a FL flow.

RD

---

From: Shane Amante <[email-masked]>
Date: Thu, 9 Sep 2010 08:20:00 -0600

Mikael,

On Sep 8, 2010, at 23:49 MDT, Mikael Abrahamsson wrote:
>> If we do agree on this, it's very helpful, because it guides all further decisions. For example, it allows us to see that the label is immutable on a best effort basis, rather than mathematically immutable. So we ccould say both that forwarding nodes MUST NOT change the label, *and* that downstream nodes MUST NOT rely on the value being unchanged.
> 
> Last I checked, the standards said that if precedence/dscp changed, the host should reset the session (correct me if I'm wrong, I don't really have time to check it right now).
> 
> For me, the flow label would be most useful in tunnel scenarios. We have mobile base station controllers which outputs hundreds of megabits/s of tunneled packets and this is all between the same SRC/DST IP. The way 3GPP does things, this is only going to increase. Being able to use flow labels to identify an individual user tunnel and thus being able to hash more efficiently when doing load sharing would be most beneficial.

With respect to your last two paragraphs, we're trying to do just that for packets tunneled over IPv6.  Take a look at the following:
http://tools.ietf.org/html/draft-carpenter-flow-ecmp-02

IMHO, there seems to be agreement that since a Tunnel End-Point is the "source node" of the tunnel packets that having it write a flow-label into the outermost IPv6 header is alignment with RFC 3697.  The larger debate seems centered around plain (non-tunneled) IPv6 packets and what, how, when, etc. the flow-label should be written into them.

-shane

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 9 Sep 2010 16:38:44 +0200

> They should be treated as different flows as far as load balancing is concerned.

That depends on the flavor of MPTCP. Certainly with the type of MPTCP that the wg of the same name is working on right now this is the case. But setting up a new subflow with new port numbers etc is rather involved as it involves both ends and needs security etc etc, while simply adopting a new flow label for a subflow can be done unilaterally by the sender. It would even be possible to create an MPTCP that doesn't change TCP semantics so only the sender is aware of the multiple paths and the receiver is an unmodified TCP receiver.

Also note that for instance SCTP, which is also multipath, doesn't have port numbers.

> This shows the need to define what is meant by "flow" independently in EACH context.

That seems superfluous. Not everythig in life needs to be defined.

> - packets of different FL flows may have their order freely modified within the network.

Note that there is no requirement that the network delivers packets in order. In fact, reordering used to be quite common.

Of course we know that reordering is bad for performance, so indeed it is better that packets belonging to the same flow aren't reordered with respect to each other, while reordering of packets belonging to different flows is inconsequential.

But that's common knowledge, I'm not sure how our discussion about the flow label impacts this (or the other way around).

> Then an MPTCP subflow is a FL flow.

You are splitting semantic hairs.

---

From: Rémi Després <[email-masked]>
Date: Thu, 9 Sep 2010 17:21:38 +0200

> That depends on the flavor of MPTCP.
> Certainly with the type of MPTCP that the wg of the same name is working on right now this is the case.

The WG MPTCP was the only MPTCP I was considering ;-).

> Also note that for instance SCTP, which is also multipath, doesn't have port numbers.

Well, SCTP *has* port numbers.
It is "per-flow" port numbers that it doesn't have.

Since each SCTCP flow (in this case SCTP "association") has its own address pair, 5-tuples can still be used to identify load-balancing flows.

RD

---

From: Fernando Gont <[email-masked]>
Date: Thu, 09 Sep 2010 14:12:00 -0300

> have time to check it right now).

You're right. And it doesn't make sense. See Section 11.1.4 of
http://www.gont.com.ar/papers/tn-03-09-security-assessment-TCP.pdf

Thanks,

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 9 Sep 2010 19:51:52 +0200 (CEST)

> http://www.gont.com.ar/papers/tn-03-09-security-assessment-TCP.pdf

Yes, it doesn't make sense, and luckily nobody (as far as I've been able 
to discern) actually follows this part. So even though it might still be 
in the standard, it's not a real-world problem.

So whatever happens, please make the flow label something that can be 
changed without any adverse reaction from the end system.

---

From: Shane Amante <[email-masked]>
Date: Thu, 9 Sep 2010 12:58:10 -0600

Tim,

> I don't see why the Flow Label should be treated differently.   If cooperating ISPs agree to pass it unaltered, that's fine.   As the spec stands, it's hard to see how we could say otherwise.

One counter-point worth considering.  If there is a desire "raise the bar" higher than BCP 38, in terms of preventing/minimizing off-path attacks in IPv6[1], then IMHO it would be best to discourage ISP's from changing a non-zero flow-label[2] of plain/vanilla (non-tunneled) IPv6 packets mid-flight (i.e.: leave it unchanged).  This would allow end-hosts/receivers, perhaps even middleboxes (e.g.: firewalls if they wanted to keep track of that much state), some ability to "inexpensively" sort out the good, from bad, packets as described in [1].

Assuming that folks generally agree the above is a good (primary?) goal, the question may then become can (or, should?) we make the flow-label compatible as an input-key for flow-based load-balancing across LAG & ECMP paths?  At least as I understand them now, the two drafts [1] are compatible with using the input-key for flow-based load-balancing decisions (more or less), which IMO is a great thing: two uses for the price of one!  ;-)

-shane

[1] draft-gont-6man-flowlabel-security-00 or draft-blake-ipv6-flow-label-nonce-02
[2] I am slightly concerned about, for example, first-hop routers over-writing a non-zero flow-label in plain/vanilla IPv6 (non-tunneled) packets, because the sending host hasn't set one.  More specifically:
    a)  Would routers be capable of implementing something like draft-gont or draft-blake (in HW) or is there too much state (i.e.: ensuring that previous flow-labels aren't re-used within a certain [time] window)?
    b)  Assuming that, e.g., first-hop routers couldn't implement either draft, then they would like revert to implementing a stateless hash of various IPv6 header fields with no (?) crypto/authentication properties.  If that is the case, then it seems the "security value" of a received flow-label at a remote host is virtually nil and we're not much better off than BCP 38 ...

-shane

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 10 Sep 2010 09:59:02 +1200

> in the standard, it's not a real-world problem.

I suspect it was intended to prevent "misuse" of high IP Precedence during
TCP SYN/ACK exchanges, or something like that. But it really does make
no sense.

> changed without any adverse reaction from the end system.

I agree, and that's behind my suggestion earlier in this thread that
"downstream nodes MUST NOT rely on the value being unchanged".

   Brian