Subject: Deploying IPv6 flow labels on the Internet

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2015 17:17:34 -0700

Hello 6man list,

We are currently planning start deployment of IPv6 flow labels for the
purposes of ECMP load balancing as described in RFC6438. In an attempt
to avoid conflicts with stateful flow label mechanisms, we have
adopted the convention to set the high order bit in the flow labels.
We plan to test this internally in our network at Facebook first, but
assuming we don't see any unforeseen issues, we would like to enable
automatic flow labels in Linux for all connection by default (which
means a lot of packets with non-zero flow labels should start to be
seen on the Internet).

My question is are there any known problems that we might hit when
setting non-zero flow labels are the Internet? For instance, are there
any known devices that feel compelled to drop or otherwise provide
degraded service when the flow label is set?

Thanks,
Tom

---

From: Mark Smith <[email-masked]>
Date: Wed, 29 Jul 2015 11:24:26 +1000

Hi Tom,

> adopted the convention to set the high order bit in the flow labels.

I'd be curious to know more about these stateful flow label
mechanisms, and why it is useful or necessary to set a bit in the flow
label field to avoid impacting them.

If I recall correctly when the Flow Label spec update was being
discussed, there were some proposals to encode or allow encoding of
some host or network meaning in specific places within the flow label
field, effectively creating subfields, but ultimately that was
abandoned.

> degraded service when the flow label is set?

So I noticed your patch to enable automatic flow labels for the Linux
kernel a while ago, and have enabled it on commonly used Linux hosts
were I can set it for a number of months i.e, a couple of commonly and
uncommonly used desktops and a couple of commonly and uncommonly used
laptops. I haven't encountered any issues as far as I know from doing
so, although happy-eyeballs might be hiding IPv6 problems from me.

No very scientific, but I troubleshoot problems I have with IPv6
rather than switching it off ;-)

Regards,
Mark.

> Thanks,
> Tom
>
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Sheng Jiang <[email-masked]>
Date: Wed, 29 Jul 2015 01:30:24 +0000

Speaking as one of the author of RFC 6437 - Flow Label Specification, I don't think there should be any foreseeable issues for the non-zero flow labels. If there is any devices that drop or provide degraded service when the flow label is set, they are certainly violate the IETF standards.

Regards,

Sheng

>-----Original Message-----
>From: ipv6 [mailto:[email-masked]] On Behalf Of Tom Herbert
>Sent: Wednesday, July 29, 2015 8:18 AM
>To: [email-masked]
>Subject: Deploying IPv6 flow labels on the Internet
>
>Hello 6man list,
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2015 20:05:01 -0700

On Tue, Jul 28, 2015 at 7:19 PM, Erik Kline <[email-masked]> wrote:
> On 29 July 2015 at 04:16, Erik Kline <[email-masked]> wrote:
>> tcpdump'ing on my Mac laptop I see what appears to be flowlabel
>> randomization going on:
>>
>> 11:11:18.444350 IP6 (flowlabel 0xea8f0, hlim 64, next-header UDP (17)
>> payload length: 48) 2401:fa00:hell0:t0m.56941 >
>> 2404:6800:4008:c05::bd.443
>> 11:11:18.519303 IP6 (flowlabel 0x6a4db, hlim 64, next-header TCP (6)
>> payload length: 44) 2401:fa00:hell0:t0m.63660 >
>> 2404:6800:4008:c05::81.443
>>
>> I recall noticing this some time ago, so I'd assume you'll be fine.
>
> Perhaps this is the effect of IN6P_AUTOFLOWLABEL?
>
Yes, looks like MacOS may have enabled them (ip6_auto_flowlabel
defaults to 1 in
https://github.com/aosm/xnu/blob/master/bsd/netinet6/in6_proto.c).
That is certainly an encouraging data point.

Tom

> From http://opensource.apple.com/source/xnu/xnu-2782.1.97/bsd/netinet6/raw_ip6.c
> :
>
>     if (in6p->inp_flowhash == 0)
>         in6p->inp_flowhash = inp_calc_flowhash(in6p);
>     /* update flowinfo - RFC 6437 */
>     if (in6p->inp_flow == 0 && in6p->in6p_flags & IN6P_AUTOFLOWLABEL) {
>         in6p->inp_flow &= ~IPV6_FLOWLABEL_MASK;
>         in6p->inp_flow |=
>             (htonl(in6p->inp_flowhash) & IPV6_FLOWLABEL_MASK);
>     }

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2015 20:15:26 -0700

> label field to avoid impacting them.
It's Linux flow label manager, a fairly complex framework to allow
applications to explicitly set flow_labels for connections.
http://xpg.dk/projects/linux-ipv6-flow-label-mangement-api/ seems to
be the best documentation. When we first proposed the autoflow labels
we wanted to turn them on by default, but the flow manager guys pushed
backs because of the potential conflicts. By setting the high order
bit we're hoping to avoid most conflicts under the assumption that
most user of flow label manager are assigning small values.

> field, effectively creating subfields, but ultimately that was
Stateful flow labels probably only make sense in a closed network so
users should be able to adapt to the convention pretty easily. I doubt
this is something that needs to be standardized.

Thanks,
Tom

---

From: Mark Smith <[email-masked]>
Date: Wed, 29 Jul 2015 14:15:58 +1000

Hi Tom,

On 29 July 2015 at 13:15, Tom Herbert <[email-masked]> wrote:
> be the best documentation.

A bit more searching managed to turn up the following, which looks to
cover design rational:

"IPv6 Flow Labels in Linux-2.2"
www.fifi.org/doc/iproute/api-ip6-flowlabels.ps.gz

> When we first proposed the autoflow labels

That API seems to support requesting generating pseudo-random flow
labels and also looks to generate errors if a flow label is requested
and already in use. So theoretically, apps should be trying to use
another flow label value if the one they attempted to use is already
in use.

Regards,
Mark.

---

From: Erik Kline <[email-masked]>
Date: Wed, 29 Jul 2015 04:16:48 +0200

---

From: Erik Kline <[email-masked]>
Date: Wed, 29 Jul 2015 04:19:29 +0200

:

---

From: "Bjoern A. Zeeb" <[email-masked]>
Date: Wed, 29 Jul 2015 07:13:51 +0000

> On 29 Jul 2015, at 00:17 , Tom Herbert <[email-masked]> wrote:
> 
> Hello 6man list,
> 

David Malone had a survey on Flow Labels a few years ago [1][2]  (aged by now).

But also given, e.g., Netflix is streaming from FreeBSD and for IPv6 these boxes should have flow labels on by default (at least FreeBSD does and had for(;;) ) I would have expected to have heard about issues by now.

Bjoern

[1] http://www.maths.tcd.ie/~dwmalone/p/ec2nd05slides.pdf
    Slides from EC2ND conference about using the IPv6 flow label for
    firewalling, written by Orla McGann and David Malone.

[2] http://www.maths.tcd.ie/~dwmalone/p/ec2nd05.pdf
    Paper from EC2ND conference about using the IPv6 flow label for
    firewalling, written by Orla McGann and David Malone.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jul 2015 19:43:14 +1200

> another flow label value if the one they attempted to use is already

Regardless of that, there is no ambiguity in the standards: there
are no encoded bits in the flow label; there are 20 pseudo-random
bits. There were a whole bunch of failed proposals for somewhat
stateful uses of the flow label, documented in RFC 6294. Apparently
we missed Alexey Kuznetsov's and Paul Fleischer's work (although
my co-author Qinwen Hu did use Linux 2.6.27 in his work).

In the RFC 6438 context why would you care about this anyway? The flow
label you attach to tunnelled packets is independent of anything the
original host might have done.

BTW Tom, despite the Appendix to RFC 6437, I suggest looking at FNV
as a hash algorithm for creating flow labels. It worked well in some
tests we did a few years ago and is lightweight and unencumbered.
http://www.cs.auckland.ac.nz/~brian/flowhashRep.pdf

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Wed, 29 Jul 2015 09:41:17 -0700

> my co-author Qinwen Hu did use Linux 2.6.27 in his work).
True, however RFC6437 does describe stateful flow labels and their
potential to be used as a signaling mechanism to intermediate nodes
which I would assume is the reason flow label manager was conceived in
the first place. It's not practical to consult flow label manager for
potential collisions every time we create a stateless label, so hence
the bit setting. This mode is controlled by sysctl anyway so can
easily be turned off if necessary.

> http://www.cs.auckland.ac.nz/~brian/flowhashRep.pdf
Linux uses Jenkins hash for most packet hashing. I did look at FNV (as
well as Toeplitz, CRC hash using Intel crc32, simple xor).  While FNV
can be efficiently implemented, it's Avalanche effect did not seem to
be as good as Jenkins hash. Toeplitz has by far the best Avalanche
effect but is very expensive in software.

Tom