Subject: [v6ops] Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 27 Jul 2020 09:08:23 +0000

Hi Fernando,
Hence again, following the logic of this draft (the level of detalization that you have given to 5.1) - may be you need additional section 5.1.x: Load Balancer have to look into TCP/UDP ports. Moreover, it could not trust "Flow label" - it is not reliable practice for LB.
Or alternatively you could say something about LB in section 5.1.2, but because it is a little special case - may be better to have separate 5.1.x

Eduard
-----Original Message-----
From: v6ops [mailto:[email-masked]] On Behalf Of Fernando Gont
Sent: 26 июля 2020 г. 8:46
To: IPv6 Operations <[email-masked]>
Cc: [email-masked]
Subject: [v6ops] Operational Implications of IPv6 Packets with Extension Headers (Fwd: New Version Notification for draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt)

Folks,

We have posted a rev of our IETF I-D "Operational Implications of IPv6 Packets with Extension Headers".

The I-D is available at: 
https://www.ietf.org/internet-drafts/draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt

Your feedback will be appreciated.

Thanks!

Cheers,
Fernando

-------- Forwarded Message --------
Subject: New Version Notification for
draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt
Date: Sat, 25 Jul 2020 22:28:50 -0700
From: [email-masked]
To: Fernando Gont <[email-masked]>, Gert Doering <[email-masked]>, Geoff Huston <[email-masked]>, Warren Kumari <[email-masked]>, Nick Hilliard <[email-masked]>

A new version of I-D, draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt
has been successfully submitted by Fernando Gont and posted to the IETF repository.

Name:		draft-gont-v6ops-ipv6-ehs-packet-drops
Revision:	04
Title:		Operational Implications of IPv6 Packets with Extension Headers
Document date:	2020-07-25
Group:		Individual Submission
Pages:		15
URL: 
https://www.ietf.org/internet-drafts/draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt
Status: 
https://datatracker.ietf.org/doc/draft-gont-v6ops-ipv6-ehs-packet-drops/
Htmlized: 
https://tools.ietf.org/html/draft-gont-v6ops-ipv6-ehs-packet-drops-04
Htmlized: 
https://datatracker.ietf.org/doc/html/draft-gont-v6ops-ipv6-ehs-packet-drops
Diff: 
https://www.ietf.org/rfcdiff?url2=draft-gont-v6ops-ipv6-ehs-packet-drops-04

Abstract:
    This document summarizes the security and operational implications of
    IPv6 extension headers, and attempts to analyze reasons why packets
    with IPv6 extension headers may be dropped in the public Internet.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

_______________________________________________
v6ops mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/v6ops

_______________________________________________
v6ops mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Jul 2020 06:31:47 -0300

Hello, Eduard,

On 27/7/20 06:08, Vasilenko Eduard wrote:
> Hi Fernando,
> Hence again, following the logic of this draft (the level of detalization that you have given to 5.1) - may be you need additional section 5.1.x: Load Balancer have to look into TCP/UDP ports. 

You mean anything else of the coverage we already have for ECMP?

> Moreover, it could not trust "Flow label" - it is not reliable practice for LB.

Could you please elaborate?  Are you referring to the fact that some 
host implementations might not set the Flow Label? Something else?

Thanks!

Cheers,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Mark Smith <[email-masked]>
Date: Mon, 27 Jul 2020 19:47:01 +1000

Hi,

If load balancers are going to be discussed, it also needs to be pointed
out that they don't follow the definition of unicast addresses.

A unicast address is to uniquely identify a single node.

Load balancers share a single unicast address between multiple nodes,
contradictory to the unique and single node identification purpose of
unicast addresses.

That is why they need to do non-RFC compliant things like digging into
transport layer headers to work with unicast protocols like UDP, TCP and
ICMP, trying to make them work across a fleet of nodes sharing a single
unicast address.

An anycast address, combined with a multi-path transport layer protocol is
the way to do service load balancing in an RFC compliant way.

See Section 5.7.7 of this draft for how.

https://tools.ietf.org/html/draft-smith-6man-form-func-anycast-addresses-01#section-5.7.7

Regards,
Mark.

On Mon, 27 Jul 2020, 19:08 Vasilenko Eduard, <[email-masked]>
wrote:

> Hi Fernando,
> Hence again, following the logic of this draft (the level of detalization
> that you have given to 5.1) - may be you need additional section 5.1.x:
> Load Balancer have to look into TCP/UDP ports. Moreover, it could not trust
> "Flow label" - it is not reliable practice for LB.
> Or alternatively you could say something about LB in section 5.1.2, but
> because it is a little special case - may be better to have separate 5.1.x
>
> Eduard
> Subject: [v6ops] Operational Implications of IPv6 Packets with Extension
> Headers (Fwd: New Version Notification for
> draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt)
>
> Folks,
>
> To: Fernando Gont <[email-masked]>, Gert Doering <[email-masked]>,
> Geoff Huston <[email-masked]>, Warren Kumari <[email-masked]>, Nick
> Hilliard <[email-masked]>
>
>
> Please note that it may take a couple of minutes from the time of
> submission until the htmlized version and diff are available at
> tools.ietf.org.
>
> The IETF Secretariat
>
>
>

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 27 Jul 2020 09:48:31 +0000

Hi Fernando,
Sorry, I do not believe that LB is doing ECMP - it is much more clever.
And yes, you could rewrite section 5.1.2 that it would be not about "ECMP-only". It is potential option to proceed.
But looking to the detalization that you have given already to other small use cases in section 5.1 - it is more logical to have LB separate. IMHO.
Eduard
-----Original Message-----
From: Fernando Gont [mailto:[email-masked]] 
Sent: 27 июля 2020 г. 12:32
To: Vasilenko Eduard <[email-masked]>; IPv6 Operations <[email-masked]>
Cc: [email-masked]
Subject: Re: Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

Hello, Eduard,

> Moreover, it could not trust "Flow label" - it is not reliable practice for LB.

Could you please elaborate?  Are you referring to the fact that some host implementations might not set the Flow Label? Something else?

Thanks!

Cheers,

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 27 Jul 2020 09:50:05 +0000

Then it is definitely separate 5.1.x looking to other small uses cases inside 5.1 that already got their separate chapter.

From: Mark Smith [mailto:[email-masked]]
Sent: 27 июля 2020 г. 12:47
To: Vasilenko Eduard <[email-masked]>
Cc: Fernando Gont <[email-masked]>; IPv6 Operations <[email-masked]>; [email-masked]
Subject: Re: [v6ops] Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

Hi,

If load balancers are going to be discussed, it also needs to be pointed out that they don't follow the definition of unicast addresses.

Load balancers share a single unicast address between multiple nodes, contradictory to the unique and single node identification purpose of unicast addresses.

That is why they need to do non-RFC compliant things like digging into transport layer headers to work with unicast protocols like UDP, TCP and ICMP, trying to make them work across a fleet of nodes sharing a single unicast address.

An anycast address, combined with a multi-path transport layer protocol is the way to do service load balancing in an RFC compliant way.

Regards,
Mark.

On Mon, 27 Jul 2020, 19:08 Vasilenko Eduard, <[email-masked]<mailto:[email-masked]>> wrote:
Hi Fernando,
Hence again, following the logic of this draft (the level of detalization that you have given to 5.1) - may be you need additional section 5.1.x: Load Balancer have to look into TCP/UDP ports. Moreover, it could not trust "Flow label" - it is not reliable practice for LB.
Or alternatively you could say something about LB in section 5.1.2, but because it is a little special case - may be better to have separate 5.1.x

Eduard
-----Original Message-----
From: v6ops [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Fernando Gont
Sent: 26 июля 2020 г. 8:46
To: IPv6 Operations <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [v6ops] Operational Implications of IPv6 Packets with Extension Headers (Fwd: New Version Notification for draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt)

Folks,

The I-D is available at:

Thanks!

Cheers,
Fernando

-------- Forwarded Message --------
Subject: New Version Notification for
draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt
Date: Sat, 25 Jul 2020 22:28:50 -0700
From: [email-masked]<mailto:[email-masked]>
To: Fernando Gont <[email-masked]<mailto:[email-masked]>>, Gert Doering <[email-masked]<mailto:[email-masked]>>, Geoff Huston <[email-masked]<mailto:[email-masked]>>, Warren Kumari <[email-masked]<mailto:[email-masked]>>, Nick Hilliard <[email-masked]<mailto:[email-masked]>>

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

_______________________________________________
v6ops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/v6ops

_______________________________________________
v6ops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/v6ops

_______________________________________________
v6ops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Jul 2020 07:09:02 -0300

Hello, Mark,

> out that they don't follow the definition of unicast addresses.

Don't such addresses become anyast addresses, automatically, as per 
Section 2.6 of RFC4291?

> contradictory to the unique and single node identification purpose of 

As per RFC4291, such addresses would be anycast addresses, rather than 
unicast.

Thanks!

Cheers,

---

From: Brian Carpenter <[email-masked]>
Date: Mon, 27 Jul 2020 22:21:26 +1200

Why isn't the flow label reliable, now that it's set by the major client
o/s? But I agree, server load balancing is severely broken if the balancer
is incapable of parsing the header chain. That's a design error in the
balancer. See RFC7098 for discussion.

Regards
    Brian
    (via tiny screen & keyboard)

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 27 Jul 2020 10:31:11 +0000

Hi Brian,
It is not technology problem - No additional standard is needed.
Just if somebody would use “flow label” not properly – LB should still deliver expected results. Hence, it should parse L4 anyway.
Eduard
From: Brian Carpenter [mailto:[email-masked]]
Sent: 27 июля 2020 г. 13:21
To: Vasilenko Eduard <[email-masked]>
Cc: Fernando Gont <[email-masked]>; IPv6 Operations <[email-masked]>; [email-masked]
Subject: Re: [v6ops] Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

Why isn't the flow label reliable, now that it's set by the major client o/s? But I agree, server load balancing is severely broken if the balancer is incapable of parsing the header chain. That's a design error in the balancer. See RFC7098 for discussion.
Regards
    Brian
    (via tiny screen & keyboard)

On Mon, 27 Jul 2020, 21:08 Vasilenko Eduard, <[email-masked]<mailto:[email-masked]>> wrote:
Hi Fernando,
Hence again, following the logic of this draft (the level of detalization that you have given to 5.1) - may be you need additional section 5.1.x: Load Balancer have to look into TCP/UDP ports. Moreover, it could not trust "Flow label" - it is not reliable practice for LB.
Or alternatively you could say something about LB in section 5.1.2, but because it is a little special case - may be better to have separate 5.1.x

Eduard
-----Original Message-----
From: v6ops [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Fernando Gont
Sent: 26 июля 2020 г. 8:46
To: IPv6 Operations <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [v6ops] Operational Implications of IPv6 Packets with Extension Headers (Fwd: New Version Notification for draft-gont-v6ops-ipv6-ehs-packet-drops-04.txt)

Folks,

The I-D is available at:

Thanks!

Cheers,
Fernando

-------- Forwarded Message --------

Abstract:

The IETF Secretariat

---

From: Tom Herbert <[email-masked]>
Date: Mon, 27 Jul 2020 07:14:55 -0700

> Hence again, following the logic of this draft (the level of detalization that you have given to 5.1) - may be you need additional section 5.1.x: Load Balancer have to look into TCP/UDP ports. More…

Eduard,

What exactly does "not reliable practice for LB" mean.? If this means
that the flow label might change during the flow's lifetime then I
will point out that UDP is equally unreliable for that reason (i.e.
the UDP ports for a QUIC connection can change during the connection's
lifetime). For that matter, a TCP header might be encrypted or in an
encapsulation that the LB doesn't understand so the LB can't access
the TCP ports at all-- in the this case the flow label is actually
more reliable than TCP since it's never encrypted and is purposely
designed to be consumed by intermediate nodes without the need for
expensive DPI.

Tom

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 27 Jul 2020 18:05:16 +0000

Hi Tom,
I believe that risk still exist that some application would no use flow label properly.
Hence, it is better on LB to deep dive into UDP/TCP. More predictable/reliable result.
Eduard
-----Original Message-----
From: Tom Herbert [mailto:[email-masked]] 
Sent: 27 июля 2020 г. 17:15
To: Vasilenko Eduard <[email-masked]>
Cc: Fernando Gont <[email-masked]>; IPv6 Operations <[email-masked]>; [email-masked]
Subject: Re: [v6ops] Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

Eduard,

What exactly does "not reliable practice for LB" mean.? If this means that the flow label might change during the flow's lifetime then I will point out that UDP is equally unreliable for that reason (i.e.
the UDP ports for a QUIC connection can change during the connection's lifetime). For that matter, a TCP header might be encrypted or in an encapsulation that the LB doesn't understand so the LB can't access the TCP ports at all-- in the this case the flow label is actually more reliable than TCP since it's never encrypted and is purposely designed to be consumed by intermediate nodes without the need for expensive DPI.

Tom
> Or alternatively you could say something about LB in section 5.1.2, 
> but because it is a little special case - may be better to have 
> separate 5.1.x
>
> Eduard
> Subject: [v6ops] Operational Implications of IPv6 Packets with 
> Extension Headers (Fwd: New Version Notification for 
> To: Fernando Gont <[email-masked]>, Gert Doering 
> <[email-masked]>, Geoff Huston <[email-masked]>, Warren Kumari 
> <[email-masked]>, Nick Hilliard <[email-masked]>
>
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 27 Jul 2020 12:23:39 -0700

> I believe that risk still exist that some application would no use flow label properly.

Exactly which applications are you concerned about?

> Hence, it is better on LB to deep dive into UDP/TCP. More predictable/reliable result.

You seem to be assuming a very limited traffic profile that consists
only of plain UDP/TCP packets. Any use case that deviates from that
then breaks your LB (i.e. encapsulation, tunneled encryption, SCTP,
IPSec, etc.). Flow label is in all IPv6 packets and is mostly set
properly for the vast majority of packets on the Internet and
processing it is completely independent of the protocols, so when
taking the full scope of the problem into account it is actually more
generic and more predictable/reliable.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 28 Jul 2020 09:01:18 +1200

> Exactly which applications are you concerned about?

Given that the most widespread socket APIs (POSIX and WinSock) do not allow
an application to set the flow label, I too have that question.

And again: please read RFC7098 where these issues are discussed in some
depth. If that RFC is now out of date due to changing reality (such
as TLS 1.3), please let us know.

    Brian

---

From: Geoff Huston <[email-masked]>
Date: Tue, 28 Jul 2020 07:57:57 +1000

It might be useful to appreciate how load balancers are actually used in the public Internet before quoting RFC and drafts at each other. A good starting point is https://www.cmand.org/workshops/202006-v6/slides/cunha.pdf which describes recent work by UFMG’s Italo Cunha.

> On 27 Jul 2020, at 7:47 pm, Mark Smith <[email-masked]> wrote:
> 
> Hi,
> 
> On Mon, 27 Jul 2020, 19:08 Vasilenko Eduard, <[email-masked]> wrote:
> Hi Fernando,

---

From: Tom Herbert <[email-masked]>
Date: Mon, 27 Jul 2020 15:24:55 -0700

> an application to set the flow label, I too have that question.
Brian,

Flow label is automatically set per flow. The easiest way to do this
is to save a hash value in the connection context, in Linux that's
simply intialized to a random number and when packets are sent the low
order twenty bits of the value are simply copied into the flow label.
In cases where we don't have a connection context, like when we're
forwarding and encapsulating in IPv6, we hash over the packet four
tuples and use bits from that hash value. In most cases we can either
use the saved context hash or get a suitable 4-tuple hash from the
receiving device for "free", but if those are not available then we
will do the work to compute the hash in the CPU (expensive operations
for CPU so we try to avoid that).

Generally, we really don't want individual applications setting the
flow label, it's just as easy to let the kernel do it as it can
guarantee a proper uniform distribution, good waterfall properties in
the hash, and consistent use of the same hash for the lifetime of the
connection. The one exception is if the flow label has more semantics
that just being an entropy identifier, in that case there are APIs
that can be used.

Tom

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 28 Jul 2020 08:36:53 +0000

Hi Tom,
Is APNIC a reliable source for you?
https://blog.apnic.net/2018/01/11/ipv6-flow-label-misuse-hashing/
RIPE has replicated it here: https://labs.ripe.net/Members/joel_jaeggli/ipv6-flow-label-misuse-in-hashing
"Cases have been observed where devices that connect to hashed endpoints do not in fact honour this behaviour. By in large, this flow label changing behaviour has been traced to IPv6 supporting CPE/firewalls, which change the flow label between the initial syn and the ack."
They have disputable conclusion: "Avoid using the flow label as a hash component".

Additional corner case: https://github.com/pypa/warehouse/issues/5454

Eduard
-----Original Message-----
From: Tom Herbert [mailto:[email-masked]] 
Sent: 27 июля 2020 г. 22:24
To: Vasilenko Eduard <[email-masked]>
Cc: Fernando Gont <[email-masked]>; IPv6 Operations <[email-masked]>; [email-masked]
Subject: Re: [v6ops] Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

> Hence, it is better on LB to deep dive into UDP/TCP. More predictable/reliable result.

You seem to be assuming a very limited traffic profile that consists only of plain UDP/TCP packets. Any use case that deviates from that then breaks your LB (i.e. encapsulation, tunneled encryption, SCTP, IPSec, etc.). Flow label is in all IPv6 packets and is mostly set properly for the vast majority of packets on the Internet and processing it is completely independent of the protocols, so when taking the full scope of the problem into account it is actually more generic and more predictable/reliable.

Tom

> Cc: Fernando Gont <[email-masked]>; IPv6 Operations 
> <[email-masked]>; [email-masked]
> > https://www.ietf.org/internet-drafts/draft-gont-v6ops-ipv6-ehs-packe
> > t-
> > drops-04.txt
> >
> > https://www.ietf.org/internet-drafts/draft-gont-v6ops-ipv6-ehs-packe
> > t-
> > drops-04.txt
> > Status:
> > https://datatracker.ietf.org/doc/draft-gont-v6ops-ipv6-ehs-packet-dr
> > op
> > s/
> > Htmlized:
> > https://datatracker.ietf.org/doc/html/draft-gont-v6ops-ipv6-ehs-pack
> > et
> > -drops
> > Diff:
> > https://www.ietf.org/rfcdiff?url2=draft-gont-v6ops-ipv6-ehs-packet-d
> > ro
> > ps-04
> >
> > Abstract:

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 28 Jul 2020 08:42:51 +0000

Hi Geoff, thanks. Striking statistics. I did not expect that IPv6 flow label is used only for 3% of traffic.
Ed/

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2020 12:18:12 -0700

> "Cases have been observed where devices that connect to hashed endpoints do not in fact honour this behaviour. By in large, this flow label changing behaviour has been traced to IPv6 supporting CPE…

The rationale for this is: "The use of the flow label as a hash
element for any stateful application requires that the flow label
remain unchanged over the life of the flow.". This is really directed
towards stateful intermediate nodes like stateful firewalls. Yes, it's
true that they will break if the flow label changes in flight. On the
host, we've tried to ensure that the flow label is consistent for the
lifetime of the flows, I don't know what firewalls are doing in
changing them.

> They have disputable conclusion: "Avoid using the flow label as a hash component".

This guidance could just as easily be "Avoid fragmentation", "Avoid
using any protocols other than UDP or TCP", "Avoid using extension
headers", "Avoid ICMP", "Avoid using encapsulation", "Avoid using
encryptions" as any of these can break some meddling device somewhere
such that packet delivery becomes unreliable on the Internet. More
generally, the guidance could simply be "Avoid doing anything that
disrupts the status quo"!

---

From: Geoff Huston <[email-masked]>
Date: Wed, 29 Jul 2020 06:03:21 +1000

> On 29 Jul 2020, at 5:18 am, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> disrupts the status quo”!

It all depends on the motivation Tom. If the motivation is “maximise the
likelihood of successful packet delivery on transit paths across the public 
Internet", then yes, all these are relevant considerations. And your point
is?

Geoff

---

From: Fernando Gont <[email-masked]>
Date: Tue, 28 Jul 2020 16:47:19 -0300

On 28/7/20 16:18, Tom Herbert wrote:
> towards stateful intermediate nodes like stateful firewalls. 

IIRC, FreeBSD used to change the Flow Label during the TCP 3WHS vs data 
transfer phase...

> disrupts the status quo"!

Unfortunately, if one is in the "business" of getting data through, one 
would rather stick to whatever works.

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2020 14:38:38 -0700

> IIRC, FreeBSD used to change the Flow Label during the TCP 3WHS vs data
Fernando,

We had an issue in Linux where the Flow Label can change when closing.
This was causing bad interaction with FW. We treated this as a bug and
fixed it promptly and now flow labels should be consistent for the
lifetime of the flow. Software implementation can have much faster
turnaround time to fix issues and I am still hoping, perhaps naively,
that with the emergence of SDN and programmable datapaths in devices
we can start to get that ability in other network devices and start to
address some long standing issues of protocol non-conformance.

> would rather stick to whatever works.
Yep, this has driven host implementations to support the least common
denominator in protocols. The only general workaround of protocol
ossification to date is encrypting as much of the packet as possible
like in QUIC, but when we do that networks visibility is reduced as is
the opportunities for hosts and network to work in concert to solve
users problems.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jul 2020 11:11:30 +1200

Hi Geoff,

We can do two things in the IETF: document reality, and try to make the
best standards and guidelines we can. Unfortunately these two goals are
sometimes contradictory.

On the "document reality" side, thanks for your pointer. And that sort
of factual input should probably be used in v6ops work starting with
"Operational Implications...". But I believe that at the same time we
need to do as much as we can on the standards & guidelines side of the
house to make reality better in the long term. That was rather the point
of RFC7098.

Middleboxes that gratuitously change the flow label are faulty. That
needs to be spelled out again, apparently.

Regards
   Brian

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2020 16:50:02 -0700

> needs to be spelled out again, apparently.

I agree. My feedback on the APNIC guidance would be to ask if before
recommending to disable a core protocol feature in an Internet
standard, did they try to identify the malfunctioning devices and ask
the responsible parties to fix them.

Tom

---

From: Geoff Huston <[email-masked]>
Date: Wed, 29 Jul 2020 10:45:19 +1000

> On 29 Jul 2020, at 9:50 am, Tom Herbert <[email-masked]> wrote:
> 
…

> 
> I agree. My feedback on the APNIC guidance  ….

Oh please - APNIC publishes a blog that solicits contributions, and
says clearly at the end of each of these articles: "The views expressed
by the authors of this blog are their own and do not necessarily reflect 
the views of APNIC.”

APNIC as a regional address registry has no views one way or the other
as to what folk do with Extension Headers in IPv6 packets.

Geoff

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2020 18:20:10 -0700

> by the authors of this blog are their own and do not necessarily reflect

Thanks for the clarification!

---

From: Fernando Gont <[email-masked]>
Date: Tue, 28 Jul 2020 22:45:49 -0300

> <[email-masked]> wrote:
[...]
> the responsible parties to fix them.

A problem with the assumption of "malfunctioning devices" is the 
assumption itself.

It seems evident that the IPv6 packet structure makes a cost-effective 
implementation rather difficult. Since engineering does not only have to 
do with science, but also with economics, it follows that:

* Folks (group #1) will not be willing to pay for features that their 
clients do not explicitly require.

* Vendors (group #2) will not spend cycles on something which will not 
get them money in return.

* Other set of folks (group #3) will not actively try to leverage 
features with a high failure rate, since they would also need to 
implement workarounds -- and if the workarounds already solve the 
problem, with a much higher success rate, there's not much of a point in 
spending time (and ultimately $$) in the feature that usually fails.

* Users (group #4) use whatever solves their problem.

We may be unhappy with the above fact. But unless we're willing to pay 
the bill :-), I'm not sure to what extent embarking into a Crusade would 
solve the "problem".

Thanks,

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 29 Jul 2020 16:03:29 +1200

below...

> the bill :-), I'm not sure to what extent embarking into a Crusade would 

Not what I meant, really. All we can do here is write documents. So if
the consensus is still what the standards track documents say, including
Node Requirements, we have actually done our duty. And we have, in RFC8504:
"All nodes SHOULD support the setting and use of the IPv6 Flow Label
field as defined in the IPv6 Flow Label specification [RFC6437].
Forwarding nodes such as routers and load distributors MUST NOT
depend only on Flow Label values being uniformly distributed.  It is
RECOMMENDED that source hosts support the flow label by setting the
Flow Label field for all packets of a given flow to the same value
chosen from an approximation to a discrete uniform distribution."

After that it really is up to vendors and operators to do what is best
for them. If it takes ten years for all products to meet the Node
Requirements, so be it.

   Brian

---

From: Fernando Gont <[email-masked]>
Date: Wed, 29 Jul 2020 01:59:35 -0300

Hi, Brian,

On 29/7/20 01:03, Brian E Carpenter wrote:
[....]
> Not what I meant, really. 

FWIW, I do agree with you. My comment was a response to Tom where he was 
objecting to folks that, for one reason or another, happen to drop 
packets. (in this particular case the advice given by JOel in his 
presentation)

And what I meant is that rather that being mad about them, one should 
try to find the reasons for which they are resorting to that (which is 
what this document is meant to), and then figure out and do what we can 
from a protocol engineering perspective.

> All we can do here is write documents. So if
> Node Requirements, we have actually done our duty.

That's in part correct. However, for cases where standards track 
documents implementation become unfeasible or cost-ineffective, one may 
need to consider being more flexible and reconsider the spec.

A good example was the IPsec requirement in an early revision of Node 
Reqs: it was words on paper that would never be complied with - so the 
spec was changed to reflect that.

I'm not necessarily saying that this is the case here, but rather that 
that's an option that should be on the table.

> And we have, in RFC8504:
> depend only on Flow Label values being uniformly distributed.

I retrospective, if it cannot depend on the FL (which I agree), is there 
much of a point in using it?

Thanks,

---

From: Tim Chown <[email-masked]>
Date: Wed, 29 Jul 2020 08:31:44 +0100

> On 29 Jul 2020, at 05:03, Brian E Carpenter <[email-masked]> wrote:
> 
> <snip>
> 

Worth noting that RFC8504 is the third iteration of the IPv6 node requirements document, and what it says may and does change, partly as new RFCs are published, partly with more operational experience.

That said, the text above about flow label usage is the same in RFC8504 as it was in RFC6434, so the guidance has been the same for at least ten years or so.  The flow label wasn’t explicitly mentioned in the first node requirements document, RFC4294.

Tim

---

From: Vasilenko Eduard <[email-masked]>
Date: Wed, 29 Jul 2020 08:06:57 +0000

Thanks to Brian.
After Brian has extracted " Forwarding nodes such as routers and load distributors MUST NOT depend only on Flow Label values being uniformly distributed." from RFC 8504 (Node Requirement).
I believe that original dispute about "should LB relay only on Flow Label" is finished. LB should look deeper into headers.
If somebody is going to dispute this - the only way is to push 8504bis where Flow label "MAY be used as the only value" or "MUST be used".

I was very happy about Fernando's explanation for business logic - I was long time in the business too - it is exactly like his said.
Do not ignore the economy, because economy is very capable to ignore you.

Ed/
-----Original Message-----
From: v6ops [mailto:[email-masked]] On Behalf Of Fernando Gont
Sent: 29 июля 2020 г. 8:00
To: Brian E Carpenter <[email-masked]>; Tom Herbert <[email-masked]>
Cc: IPv6 Operations <[email-masked]>; [email-masked]
Subject: Re: [v6ops] Operational Implications of IPv6 Packets with Extension Headers - Load Balancer

Hi, Brian,

>> It seems evident that the IPv6 packet structure makes a 
>> cost-effective implementation rather difficult. Since engineering 
>> does not only have to do with science, but also with economics, it follows that:
>>
>> * Vendors (group #2) will not spend cycles on something which will 
>> not get them money in return.
>>
>> problem, with a much higher success rate, there's not much of a point 
>> in spending time (and ultimately $$) in the feature that usually fails.
>>
>> We may be unhappy with the above fact. But unless we're willing to 
>> pay the bill :-), I'm not sure to what extent embarking into a 
>> Crusade would solve the "problem".
> 

FWIW, I do agree with you. My comment was a response to Tom where he was objecting to folks that, for one reason or another, happen to drop packets. (in this particular case the advice given by JOel in his
presentation)

And what I meant is that rather that being mad about them, one should try to find the reasons for which they are resorting to that (which is what this document is meant to), and then figure out and do what we can from a protocol engineering perspective.

> All we can do here is write documents. So if the consensus is still 
> what the standards track documents say, including Node Requirements, 
> we have actually done our duty.

That's in part correct. However, for cases where standards track documents implementation become unfeasible or cost-ineffective, one may need to consider being more flexible and reconsider the spec.

A good example was the IPsec requirement in an early revision of Node
Reqs: it was words on paper that would never be complied with - so the spec was changed to reflect that.

I'm not necessarily saying that this is the case here, but rather that that's an option that should be on the table.

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Wed, 29 Jul 2020 09:37:42 -0700

Fernando,

I agree with that and I do appreciate that section 3 of the draft
describes several of the mitigations to make EH processing more
palatable like the provision of RFC8200 that allows nodes to ignore
HBH and the limits in RFC8504. It might be helpful in the draft to
mention the relevant mitigations when discussing specific reasons for
drops. For instance, if DOS is being discussed then RFC8504 should be
mentioned in reference since the one purpose of limits on number of
options is to mitigate DOS attacks (i.e. without limits an attacker
could fill an MTU with 100s of unknown HBH options that an
implementation, SW or HW, will choke on it).

Another area that should be elaborated here, or maybe necessitates a
separate document, is the parsing buffer problem described in section
5.1.1 by "If an IPv6 header chain is sufficiently long that its header
exceeds the packet look-up capacity of the router, then it may be
dropped due to hardware inability to determine how it should be
handled.". I believe this is a real problem, but it's never been
quantified. For instance, when a host creates a packet how can it
possibly guess than some router in the path will drop the packet
because the header chain is too long (not just IPv6 header chain by
the way, but really all headers a node needs to process which can
include transport or even transport payload). The draft for sending
ICMP errors when packets are dropped is in the RFC editor queue, but I
believe we can provide some sort of guidance that intermediate nodes
are expected to be able to process at least X bytes of headers. This
probably needs to broken down by what headers nodes process: a
forwarding node might just look at IPv6 header and HBH options, an
intermediate destination needs to look at routing header, a stateful
firewall needs to get transport header, and application layer firewall
needs to go even farther, etc.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 29 Jul 2020 21:46:32 +0100

> implementation, SW or HW, will choke on it).

there are often no mitigations as limits are often related to hardware.

> quantified. For instance, when a host creates a packet how can it > possibly guess than some router in the path will drop the packet
> include transport or even transport payload).

First, only SRv6 proposes to add EHs and I think we're all familiar with 
the difficulties that were outlined in that discussion.

So in the general case, it's up to the host to decide how many EHs to 
attach to a packet.  Obviously as there's no EH negotiation mechanism, 
there's no way for a host to know what will work and what won't, and 
even if there were, intermediate paths change so what might work on 
setup might not work later on.  We have the same problem with MTUs.

There are some limits already defined in IETF standards track documents 
(e.g. that you can have no more than a single packet with EHs attached), 
but real life limits are much lower.

It may be useful for a future document to quantify the problem.  This 
aim of this document is to state that there's a problem.

Nick

---

From: Geoff Huston <[email-masked]>
Date: Thu, 30 Jul 2020 06:54:57 +1000

> 
> It may be useful for a future document to quantify the problem.  This aim of this document is to state that there's a problem.

RFC7872?

also some of my measurements:

https://www.potaroo.net/ispcol/2017-08/xtn-hdrs-2.pdf

https://www.potaroo.net/ispcol/2020-07/dns6.html

Yes, there’s a problem. And yes, its a widespread problem and there is documentation on measurements that substantiate this. Next?

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 29 Jul 2020 22:06:10 +0100

Geoff Huston wrote on 29/07/2020 21:54:
> Next?

next would be quantifying in cases where EHs are passed, how many EHs 
might, or mightn't get through.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Wed, 29 Jul 2020 14:41:37 -0700

> setup might not work later on.  We have the same problem with MTUs.

It's not really the same thing. There has been a lot of work in PMTU
discovery mechanisms, but not a lot of work in
Path-Parsing-Buffer-Size discovery.

> but real life limits are much lower.
Okay, so then we recognize a gap between the protocol specification
and reality. How do we close that gap? Seems like a first step is to
quantify the problem. For instance, if you say real life limits are
much lower then can you quantify exactly how much lower they are?

> aim of this document is to state that there's a problem.

Sure, but our ability to analyze the problem is largely dependent on
how well the problem is described. For instance, if we say some nodes
drop packets because header chains are too long without defining what
"too long" means, that isn't an actionable engineering description of
a problem.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Fri, 31 Jul 2020 08:55:18 -0300

Hi, Tom,

On 29/7/20 18:41, Tom Herbert wrote:
[....]
> Path-Parsing-Buffer-Size discovery.

FWIW, some rough measurements I've performed indicate that the drop rate 
is proportional to the EH length. At the time, I did  measurements with 
lengths such as 8, 256, and 512.

If the group thinks more finer-grained data would be of help, then I 
could run these measurements again, in more detail, and report back to 
the working group.

Thanks!

Cheers,