Subject: Re: I-D Action: draft-han-6man-in-band-signaling-for-transport-qos-00.txt

From: Brian E Carpenter <[email-masked]>
Date: Sat, 14 Oct 2017 15:16:16 +1300

Hi,

RSVP in IPv6 [RFC2205] uses packets with a Router Alert hop-by-hop
option. So although it is not embedded in application packets, it is
otherwise very similar: in band, following the same path as application
packets, and triggering hop-by-hop processing in the same routers.
RSVP for Integrated Services was a complete failure. So my first
questions are:

1: What is the authors' full analysis of why RSVP failed?
2: How does the proposed solution avoid the same failure?

In particular, I do not understand your argument about
scalability. While that was an obvious issue for RSVP, it
could be fixed in the way you describe: by limiting the number
of using applications. The fundamental scaling issue for
RSVP was the number of session state items per router, and your
solution needs exactly the same number of session state items.
So why will you succeed where RSVP failed?

> End user application cannot directly use DiffServ.

Not true. The advanced socket API allows the transport user to set
the DSCP via IPV6_TCLASS [RFC3542]. It's been done for many years,
for example in IP telephones.

What is lacking is a host software package to make this API feature
useful, but any QoS solution needs that. In your terminology, that
is the application interface to the closed-loop control system.

Note that RTCWEB has chosen to support Diffserv
[https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rtcweb-qos/]

> 3.2.  IP In-band signaling
> 
>    There is no definition for IP in-band signaling.

Not true. That's exactly what DiffServ is. You could argue that
it's too limited, but after 19 years we have not exhausted the
DSCP code space defined in [RFC2474]. (As far as I know, nobody
has yet combined use of the DSCP with the Flow Label. There is
scope for creativity there.)

>    Flow level In-band Signaling
>       The control message and data packet share the same flow
>       identification.  The flow identification could be 5 tuples ...

I agree with the comment that the IPv6 flow label is a much better
choice. It is already well defined [RFC6437] and widely supported
by operating systems, it works for any transport protcol, works
for encrypted packets, and works for packets with multiple extension
headers. If you want per-flow granularity, the flow label is perfect.

>  The HbH-EH may be examined and processed by the nodes that are
>    explicitly configured to do so [RFC8200]

which also says in section 4.8:

"  New hop-by-hop options are not recommended because nodes may be
   configured to ignore the Hop-by-Hop Options header, drop packets
   containing a Hop-by-Hop Options header, or assign packets containing
   a Hop-by-Hop Options header to a slow processing path.  Designers
   considering defining new hop-by-hop options need to be aware of this
   likely behavior.  There has to be a very clear justification why any
   new hop-by-hop option is needed before it is standardized."

I believe you need much more explanation of why we should ignore
this recommendation.

> 8.  Security Considerations
> 
>    There is no security issue introduced by this document

I think this section needs more work.

Regards
   Brian Carpenter

On 12/10/2017 07:05, [email-masked] wrote:
> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> 
> 
>         Title           : IPv6 in-band signaling for the support of transport with QoS
>         Authors         : Lin Han
>                           Guoping Li
>                           Boyan Tu
>                           Xuefei Tan
>                           Frank Li
>                           Richard Li
>                           Jeff Tantsura
>                           Kevin Smith
> 	Filename        : draft-han-6man-in-band-signaling-for-transport-qos-00.txt
> 	Pages           : 40
> 	Date            : 2017-10-11
> 
> Abstract:
>    This document proposes a method to support the IP transport service
>    that could guarantee a certain level of service quality in bandwidth
>    and latency.  The new transport service is fine-grained and could
>    apply to individual or aggregated TCP/UDP flow(s).
> 
> 
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-han-6man-in-band-signaling-for-transport-qos/
> 
> There are also htmlized versions available at:
> https://tools.ietf.org/html/draft-han-6man-in-band-signaling-for-transport-qos-00
> https://datatracker.ietf.org/doc/html/draft-han-6man-in-band-signaling-for-transport-qos-00
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
> 
> _______________________________________________
> I-D-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i-d-announce
> Internet-Draft directories: http://www.ietf.org/shadow.html
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>

---

From: Lin Han <[email-masked]>
Date: Mon, 16 Oct 2017 23:52:40 +0000

Hi, Brain

Thanks for the comments, see my reply in line with [LH]

Lin

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Friday, October 13, 2017 7:16 PM
To: 6man <[email-masked]>
Subject: Re: I-D Action: draft-han-6man-in-band-signaling-for-transport-qos-00.txt

Hi,

RSVP in IPv6 [RFC2205] uses packets with a Router Alert hop-by-hop option. So although it is not embedded in application packets, it is otherwise very similar: in band, following the same path as application packets, and triggering hop-by-hop processing in the same routers.
[LH] The difference of this idea with RSVP is that RSVP needs a separate protocol to be running, but this in-band signaling is always carried in a packet of upper layer protocols (TCP, UDP, etc) and does not need to run a new protocol.
RSVP for Integrated Services was a complete failure. So my first questions are:

1: What is the authors' full analysis of why RSVP failed?
[LH] The complexity of the RSVP makes its scalability an issue. Specifically, RSVP needs every router run the protocol and maintain the state for every RSVP session on the router; To maintain the RSVP soft-state, each source of RSVP session has to periodically (~30sec and configurable) multicast the PATH message with Tspec to whole network and the interested receiver has to return the RESV with Rspec to root of the mcast tree; All these control plane work make the RSVP session supported on a router limited by the controller CPU processing power and memory size. 
2: How does the proposed solution avoid the same failure?
[LH] The in-band signaling proposal is based on the following ideas and assumptions
1) Network processor widely used in latest network device are capable of not only doing table lookup and switching but also doing general packet processing and other works which was only available by controller CPU (in old generation network device).
2) If we simplify the QoS protocol work in request, reserve and  state keeping to be fitting into the capability of a NPU, we can greatly reduce the complexity of what RSVP or IntServ tries to do to provision the flow-level QoS and maintain the soft-sate for each flow with QoS.
3) Following are some details, 
3.1) We simplify the Tsepc and Rspec to explicit bandwidth and latency request, and rely on the NPU to program the hardware to satisfy the QoS requirement, All QoS related traffic shaping, scheduling and queuing are hardware dependent and should be irrelevant to the control plane; 
3.2) Instead of using two directional message (in RSVP) to finish the reservation for a session, this propose only uses one directional in-band signal messaging to program the hardware for the QoS for the flow on same direction. The reason we can do such simplification is because we don't have to have the complicated feature like the merging of request in RSVP. The proposed in-band signaling only supports unicast, but RSVP was designed to support both unicast and multicast, so, RSVP needs merging feature, and needs two directional messages to finish the resource reservation. The new idea will also be fitting to the IP nature that IP path is not symmetric in many scenarios. As comparison, to use RSVP for asymmetric path is a problem. 
3.3) The soft-state of each flow's QoS channel is maintained by the data packet with the same flow identification. The QoS state programmed in the hardware will be erased and the resource will be released if there is no data packet for a programmed QoS state for a period of time. A host can simply send zero-size data packet to refresh the QoS state for the flow if there is no application data. This could completely remove the state refreshing mechanism by a separate control protocol like RSVP which was a big overhead for CPU to be scalable.
3.4) The NPU will use the pre-defined flow identification to forward a packet with QoS guarantee, and the forwarding state will be updated at each hop-by-hop aware node and feed back to the source host. Source host can easily decide if it needs to re-program the QoS for a flow if there is some failure (link, node, topology change, etc)
3.5) In summary, this approach will use a simplified mechanism (compared with RSVP) to provision QoS, and it also moves the QoS control complexity and processing burden from centralized controller CPU to distributed NPU of line card. Normally there are more NPU than CPU in a router, so, the redistribution of the complexity will greatly improve the scalability. 
3.6) Here are some hypothetical and experimental data in our POC: A typical Huawei router can only support about 5k RSVP sessions (No data from other vendors, but should be in same scale hopefully -:)). In POC, we apply the new approach to a low/middle class access router (ATN9xxx, the NPU was designed more than 5 year ago), each NPU can handle about 1k QoS session (The limit is majorly caused by the memory for queue and scheduler allocation), then whole system can support (num. of NPU) x 1k QoS session. Depending on port configuration, a router can easily have more than 10 NPU. 
4) Since we only did the experiment with our own proprietary NPU, and not sure if other company's NPUs have the similar capability as above, we put the document as "informational" (not sure if "experiment" is better, but can be changed). 

In particular, I do not understand your argument about scalability. While that was an obvious issue for RSVP, it could be fixed in the way you describe: by limiting the number of using applications. The fundamental scaling issue for RSVP was the number of session state items per router, and your solution needs exactly the same number of session state items.
So why will you succeed where RSVP failed?

> End user application cannot directly use DiffServ.

Not true. The advanced socket API allows the transport user to set the DSCP via IPV6_TCLASS [RFC3542]. It's been done for many years, for example in IP telephones.
[LH] agreed, will correct the document.

What is lacking is a host software package to make this API feature useful, but any QoS solution needs that. In your terminology, that is the application interface to the closed-loop control system.

Note that RTCWEB has chosen to support Diffserv [https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rtcweb-qos/]

>    There is no definition for IP in-band signaling.

Not true. That's exactly what DiffServ is. You could argue that it's too limited, but after 19 years we have not exhausted the DSCP code space defined in [RFC2474]. (As far as I know, nobody has yet combined use of the DSCP with the Flow Label. There is scope for creativity there.)

[LH] The difference of this approach with DiffServ is the granularity of QoS. DiffServ was very successful. As I stated in the document, some application needs much more bandwidth (than others) and we need a way to specify  the QoS requirement explicitly for those flows. 

>       identification.  The flow identification could be 5 tuples ...

I agree with the comment that the IPv6 flow label is a much better choice. It is already well defined [RFC6437] and widely supported by operating systems, it works for any transport protcol, works for encrypted packets, and works for packets with multiple extension headers. If you want per-flow granularity, the flow label is perfect.

[LH] Agreed, 

I believe you need much more explanation of why we should ignore this recommendation.

[LH] The propose does not introduce new hop-by-hop option header, we only introduce new option carried in the existing hop-by-hop option header, it is the same as other defined options carried by hop-by-hop option header, such as "Jumbo Payload", "RPL Option", "Quick-Start", etc. I think the IPv6 document sometime uses the general "hop-by-hop option" word as the " hop-by-hop option extension header" really confuse people. I believe the above document does not recommend to introduce new hop-by-hop option extension header, but not stop introducing new options carried in the current hop-by-hop option header.  
I will check with IPv6 author for clarification.
Below are existing IPv6 hop-by-hop and destination options defined (HbH: hop-by-hop option header; Dst: Destination extension header)

0x00	00	0	00000	Pad1	[[IPV6]]	- HbH
0x01	00	0	00001	PadN	[[IPV6]]	- HbH
0xC2	11	0	00010	Jumbo Payload	[RFC2675]	- HbH
0x63	01	1	00011	RPL Option	[RFC6553]	- HbH
0x04	00	0	00100	Tunnel Encapsulation Limit	[RFC2473]	- Dst
0x05	00	0	00101	Router Alert	[RFC2711]	- HbH
0x26	00	1	00110	Quick-Start	[RFC4782][RFC Errata 2034]	- HbH
0x07	00	0	00111	CALIPSO	[RFC5570]	- HbH
0x08	00	0	01000	SMF_DPD	[RFC6621]	- HbH
0xC9	11	0	01001	Home Address	[RFC6275]	- Home Addr Dst
0x8A	10	0	01010	Endpoint Identification (DEPRECATED)	[[CHARLES LYNN]]
0x8B	10	0	01011	ILNP Nonce	[RFC6744]	- Dst
0x8C	10	0	01100	Line-Identification Option	[RFC6788]	- Dst
0x4D	01	0	01101	Deprecated	[RFC7731]
0x6D	01	1	01101	MPL Option	[RFC7731]	- HbH
0xEE	11	1	01110	IP_DFF	[RFC6971]	-HbH
0x0F	00	0	01111	Performance and Diagnostic Metrics (PDM)	[RFC8250]	- Dst

[LH] Yes, will add more

Regards
   Brian Carpenter

On 12/10/2017 07:05, [email-masked] wrote:
> Please note that it may take a couple of minutes from the time of 
> submission until the htmlized version and diff are available at tools.ietf.org.
> 
> Internet-Draft directories: http://www.ietf.org/shadow.html or 
> ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> 

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 17 Oct 2017 17:04:33 +1300

Just replying on one point, since most of this discussion clearly
needs to happen in TSVWG:

On 17/10/2017 12:52, Lin Han wrote:
...
> [LH] The propose does not introduce new hop-by-hop option header, we only introduce new option carried in the existing hop-by-hop option header, 

Yes, that is exactly what the paragraph from RFC8200 means: it
says "New hop-by-hop options are not recommended".

    Brian

---

From: Ole Troan <[email-masked]>
Date: Tue, 17 Oct 2017 09:11:02 +0200

> says "New hop-by-hop options are not recommended".

Blindly reciting rules without also understanding and explaining the intent leads to unintended consequences.

In this particular case, where every hop along the path would need to support it, then that's the exact use case for the hop by hop header.

Ole

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 17 Oct 2017 16:34:58 +0200

> 2: How does the proposed solution avoid the same failure?

Not an author, but some thoughts:

Given the existance of RSVP-TE and GMPLS, i wouldn't say RSVP failed.

IMHO there are lots and lots of issues with RSVP-IPs history that we now know how to
avoid. I think inband signaling is the one core architecture option we have explored
way too little and has only practically become feasible within the last maybe 10
years.

IMHO one key reason RSVP(-IP) was not adopted more is because of non-ubiquitous
support of the host stack/APIs. And OSs where it once was supported did withdrew
support for it. 

It was also killed by the initial focus on admission control - permit/deny flows.
Most customers thought thats what i was good for. And they needed that but came up
with faster to deply workarounds (eg: in IPTV).

As opposed to unequal bandwidth allocation. And nobody ever wrote a draft showing how
to implement the various options how to avoid eg: per-flow queues. By the time we wanted
to get back to these topics, there was already the generation of IETF'ler burned by prior
bad experience that where pushing back on any innovation in this space
("been there, done that, failed, go away").

Router alert also killed RSVP. Lots of router implementations punt packets with
router alert independent of the indicated protocol. So even if you don't run RSVP,
your router would punt RSVP router alert packets. So SPs try to get rid of router-alerts
on their network edge. I personally think we can't resurrect router alert option
like we resurrect ECN but have to choose other options (like what Lin propooses).

Onpath signalling from RSVP is also fundamentally different from inband signalling.
Just try to get RSVP through a NAT or firewall. Or try to make it stay onpath
when you have ECMP. I once had to work through all the path selection rules of one
back then popular router OS to propose an architecture how to expose all the options so
RSVP could stay onpath. Short summary: @$^%*$(@*&(*&@^$*&.

There was absoutely no consideration for RSVP to be designed in a way that would
allow for it to be easily processable in the forwarding plane (called NPU in the draft).
Or "inband" as we like to call it now.

The industry has a lot of experience with high performance per-flow state building
from control-plane (CPU) signaling. IP multicast. So we know how much of a pain it
is. And the industry hated it. Unlike RSVP they just didn't find any workarounds
for IP multicast and had intra-domain end-to-end use-cases (IPTV, finance) so individual
vendors could push vendors to make it work - over 20 year. Just check out the interest
in BIER to see how the industry wants to get rid of per-flow state maintenance and
instead prefers a more lightweight "in-band-singnaling" of replication.

In fact, both RSVP/IP-multicast state building was a lot more scaleable on
CPU-only forwarding plane type devices 20 years ago then it is in todays CPU/NPU
separated systems. Its getting through the bottleneck of IPC between CPU and NPU for
example.

I could go on...

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Tue, 17 Oct 2017 08:53:53 -0700

> Blindly reciting rules without also understanding and explaining the intent leads to unintended consequences.
RFC8200 also relaxes the requirement that HbH has to be examined by
every host which I think mitigates some of the concerns that prompted
the recommendation against new hop-by-hop options.

> In this particular case, where every hop along the path would need to support it, then that's the exact use case for the hop by hop header.
I don't think that every hop in the path would necessarily need to
support this. It may be the case that traffic engineering is only
being done on one or two links in the path (backbone links for
instance) so support is needed there, but nodes at other hops could
ignore the HbH option.

Tom

---

From: Ole Troan <[email-masked]>
Date: Tue, 17 Oct 2017 18:28:18 +0200

Tom,

> instance) so support is needed there, but nodes at other hops could

Right. the HBH is the most efficient way we have to inform an intermediate node, please take a closer look at this packet if you are configured to do so.
There are of course significant deployment issues with HBH and a higher drop probability if you wanted to carry that header outside of the AS.

While the 6man working group can certainly help the authors with those considerations, I believe the work would have to be owned somewhere else.
And it might be a little too early to go into the details of packet formats etc.

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Tue, 17 Oct 2017 09:54:52 -0700

> There are of course significant deployment issues with HBH and a higher drop probability if you wanted to carry that header outside of the AS.
Ole,

On one hand, IETF defined extension headers as the extensibility
mechanism of IPv6, but yet on the other hand the message seems to be
to not use them because deployed devices don't correctly support the
protocol. The upshot is that we see proposals in other WGs of stuffing
network layer information into transport protocols (options or
payloads) with the full expectation that intermediate nodes will parse
into the transport layer, process the embedded information, and
possibly even overwrite transport layer information. IMO that is an
architectural abomination and further abandonment of the end to end
model!

> And it might be a little too early to go into the details of packet formats etc.
Agreed, but I do think the 6man working group has a vested interest to
make sure that network layer information stays in the network layer.
At some point the information needed is going to be so complex that it
won't be able to be conveyed in any fields of the IP header.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 17 Oct 2017 20:16:46 +0200

As mentioned in my prior mail: RSVP-IP deployments got killed by SPs that filtered
packets with router-alert because common network devices punted packets because of
the presence of router-alert. Instead of punting them because of presence of
RSVP-router-alert option. And of course you can blame IP multicast: every router
started to punt router-alert because of MLD (and of course us multicast folks
missed the chance to fix this in MLDv2 because it only tried to duplicate IGMPv3),
but no RFC told developers to punt because of router-alert + value in the router alert.
AFAIK, the same applies to hop-by-hop option in general. Alas, even rfc7045 did not discuss
this issue.

IMHO we need a mechanism that specifies: devices MUST NOT punt/slow down packets
because of the presence of this mechanism, but only because of presence of
(mechanism,value) where value is intended to be supported by device. If the device
can not do this, then it MUST IGNORE this mechanism and forward packets with the mechanism
as if the option was not present.

One simple way to do this is to define hop-by-hop-fixed that does say exactly this
and then hopefully allow all existing hop-by-hop TLV to live underneath that
fixed hop-by-hop-fixed option. But i would certainly suggest to review processing
complexity and extensibility before such a conclusion.

Personally i prefer inband UDP layer signaling, not because i do not like IPv6
option headers but because there is just no ubiquitous API to allow apps
independent of OS enhancements to set arbitrary options headers, especially ones
that are not yet defined (allow apps to fully defined the whole header bit by bit).
Lets say from a javascript app in a browser. And i am more interested in ease of
adoption than in architectural cleanlyness. But of course its perfectly valid to
do the architectural clean appraoch and then prove me wrong and get that option
adopted more ;-))

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Tue, 17 Oct 2017 12:38:52 -0700

> independent of OS enhancements to set arbitrary options headers, especially ones

The API does exist. OSes like Linux have implemented APIs described in
RFC2292 and other common APIs.

> do the architectural clean appraoch and then prove me wrong and get that option

It's not just about architectural cleanliness, it's about correctness.
In the case that an intermediate node parses a UDP payload it is
likely doing this based on the some common destination port number.
Port numbers don't have global and persistent meaning (e.g. TCP port
80 does not always have to be HTTP) so this leads to the possibility
of misinterpretation as described in RFC7605. If the node improperly
writes data into the UDP payload and updates the checksum, then this
becomes silent data corruption. Extension headers do not have this
possibility of misinterpretation in the network.

This potential misinterpretation of UDP ports and many other issues
with trying to turn UDP into a network layer protocol were brought up
in the SPUD/PLUS discussions.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Oct 2017 08:40:34 +1300

> In this particular case, where every hop along the path would need to support it, then that's the exact use case for the hop by hop header.

Of course. But that in itself isn't the strong justification that
RFC8200 calls for, IMHO.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Oct 2017 08:57:35 +1300

On 18/10/2017 07:16, Toerless Eckert wrote:
> the presence of router-alert. 

Router alert in IPv6 *is* a hop-by-hop option [RFC2711], so the
code path for this proposal starts exactly the same as the code
path for processing an RSVP message.

> Instead of punting them because of presence of

That's pretty much what RFC7045/RC8200 say, except that they
say it as a warning.

As Ole said - within a domain where hop-by-hop option X is
supported, you can reasonably expect that all routers process X.
But (excuse me repeating myself), whether X is worth doing is not
really a question for 6man. In this case, it belongs where QoS is
discussed, which is TSVWG.

    Brian

---

From: Tom Herbert <[email-masked]>
Date: Tue, 17 Oct 2017 13:16:41 -0700

> Of course. But that in itself isn't the strong justification that
I believe an experimental HbH option number could be use to develop
the protocol and help provide the justification if an assignment is
needed.

Tom

---

From: Lin Han <[email-masked]>
Date: Thu, 19 Oct 2017 16:53:51 +0000

Hi, Brain

I have forwarded the draft to TSVWG for further discussion and review.

Thanks

Lin

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Tuesday, October 17, 2017 12:58 PM
To: Toerless Eckert <[email-masked]>; Tom Herbert <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: Re: I-D Action: draft-han-6man-in-band-signaling-for-transport-qos-00.txt

On 18/10/2017 07:16, Toerless Eckert wrote:
> As mentioned in my prior mail: RSVP-IP deployments got killed by SPs 
> that filtered packets with router-alert because common network devices 
> punted packets because of the presence of router-alert.

Router alert in IPv6 *is* a hop-by-hop option [RFC2711], so the code path for this proposal starts exactly the same as the code path for processing an RSVP message.

> Instead of punting them because of presence of RSVP-router-alert 
> option. And of course you can blame IP multicast: every router started 
> to punt router-alert because of MLD (and of course us multicast folks 
> missed the chance to fix this in MLDv2 because it only tried to 
> duplicate IGMPv3), but no RFC told developers to punt because of router-alert + value in the router alert.
> AFAIK, the same applies to hop-by-hop option in general. Alas, even 
> rfc7045 did not discuss this issue.
> 
> IMHO we need a mechanism that specifies: devices MUST NOT punt/slow 
> down packets because of the presence of this mechanism, but only 
> because of presence of
> (mechanism,value) where value is intended to be supported by device. 
> If the device can not do this, then it MUST IGNORE this mechanism and 
> forward packets with the mechanism as if the option was not present.

That's pretty much what RFC7045/RC8200 say, except that they say it as a warning.

As Ole said - within a domain where hop-by-hop option X is supported, you can reasonably expect that all routers process X.
But (excuse me repeating myself), whether X is worth doing is not really a question for 6man. In this case, it belongs where QoS is discussed, which is TSVWG.

    Brian

> One simple way to do this is to define hop-by-hop-fixed that does say 
> exactly this and then hopefully allow all existing hop-by-hop TLV to 
> live underneath that fixed hop-by-hop-fixed option. But i would 
> certainly suggest to review processing complexity and extensibility before such a conclusion.
> 
> Personally i prefer inband UDP layer signaling, not because i do not 
> like IPv6 option headers but because there is just no ubiquitous API 
> to allow apps independent of OS enhancements to set arbitrary options 
> headers, especially ones that are not yet defined (allow apps to fully defined the whole header bit by bit).
> Lets say from a javascript app in a browser. And i am more interested 
> in ease of adoption than in architectural cleanlyness. But of course 
> its perfectly valid to do the architectural clean appraoch and then 
> prove me wrong and get that option adopted more ;-))
> 
> Cheers
>     Toerless
> 
>> protocol. The upshot is that we see proposals in other WGs of 
>> stuffing network layer information into transport protocols (options 
>> or
>> payloads) with the full expectation that intermediate nodes will 
>> parse into the transport layer, process the embedded information, and 
>> Agreed, but I do think the 6man working group has a vested interest 
>> to make sure that network layer information stays in the network layer.
>> At some point the information needed is going to be so complex that 
>> it won't be able to be conveyed in any fields of the IP header.
>>
>> Tom
>>
>>> Best regards,
>>> Ole
>>

---

From: Lin Han <[email-masked]>
Date: Thu, 19 Oct 2017 17:28:41 +0000

Hi, Tom

Agreed with you.
Actually we come out the solution by using IPv6 extension header after we failed in using upper layer information, such as TCP or UDP options.
>From the NPU processing point of view, retrieving the signaling message from either part of the data packet have almost no difference in terms of complexity and performance; 
We selected using transport layer information at the beginning was because it is independent of IPv4 and IPv6. But later on, we found that this solution will be limited in the case of un-encrypted TCP and UDP, and it cannot apply to other newer protocol such as QUIC. Considering the encrypted transport service is almost a default configuration in the future, this limit is absolutely un-acceptable. We got the same feedback after we chat with some ICCRG folks in previous IETF meeting. To use upper layer information for the network layer purpose is strongly opposed by TCP community.
One more thing to inspire us to use the IPv6 HbH is that the very original in-band singling proposed by John Harper (quoted in the draft) was finally resulting the TCP Quick-Start (RFC4782). Our draft could be seen as a further step to provide a more complete QoS support for all transport protocols.

Regards

Lin

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Tuesday, October 17, 2017 12:39 PM
To: Toerless Eckert <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: Re: I-D Action: draft-han-6man-in-band-signaling-for-transport-qos-00.txt

On Tue, Oct 17, 2017 at 11:16 AM, Toerless Eckert <[email-masked]> wrote:
> punted packets because of the presence of router-alert. Instead of 
> punting them because of presence of RSVP-router-alert option. And of 
> course you can blame IP multicast: every router started to punt 
> router-alert because of MLD (and of course us multicast folks missed 
> the chance to fix this in MLDv2 because it only tried to duplicate IGMPv3), but no RFC told developers to punt because of router-alert + value in the router alert.
> headers, especially ones

It's not just about architectural cleanliness, it's about correctness.
In the case that an intermediate node parses a UDP payload it is likely doing this based on the some common destination port number.
Port numbers don't have global and persistent meaning (e.g. TCP port
80 does not always have to be HTTP) so this leads to the possibility of misinterpretation as described in RFC7605. If the node improperly writes data into the UDP payload and updates the checksum, then this becomes silent data corruption. Extension headers do not have this possibility of misinterpretation in the network.

This potential misinterpretation of UDP ports and many other issues with trying to turn UDP into a network layer protocol were brought up in the SPUD/PLUS discussions.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Oct 2017 08:09:51 +1300

Thanks. I think we should get some useful discussion over there.

Regards
   Brian

On 20/10/2017 05:53, Lin Han wrote:
> .

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 19 Oct 2017 23:16:37 +0200

>From limited experience in the past, support is fragmented and frustrating.
But we only tried to get a few of these things work across various OSs 
(including android, iOS) and ran into variety of issues. ANd that was all
from C. Once you try to go into python, Java or (yikes) javascript, the
likelyhood of getting access to any API not needed by 99% of stupid apps
becomes lower and lower.

> becomes silent data corruption.

Preaching to the choir. Reminds me when i got from IANA one of those sparse < 512 port
number for router interception with the explicit intent to minimize use by
random other applications - only to run into some obnoxious folks a few years
later who said they had used (but of course never registered) that port forever
in their app and do not intend to change that.

> Extension headers do not have this

> in the SPUD/PLUS discussions.

So....

I wish we would have ever gotten to the point where we could really discuss
how to best do inband signaling. Alas, SPUD/PLUS was killed way in before
by what i would characterize as a perpass crowd attack.

IMHO, any inband solution needs to well define an efficient way to do per-flow
signaling, aka: not only per-packet. We're talking about per-flow behavior
in many cases anyhow, and we well understand what can and can-not be done with
per-flow state in middleboxes.

Check out 2013 MALICE/DISCUSS which proposed to use STUN packets to
signal inband (not every data packet like SPUD but just some STUN packets), and rely on
STUN cookie as signature to recognize those inband signaling packets. 

If we agreed that we wanted per-flow inband signaling, then the STUN signature
would be great option to get going. And a redefined router-alert-NG IP extension
header might be the architectural correct "look-at-me" option long term.

Finally wrt. architectural cleanlyness: Per-flow service negotition IMHO is
clearly a transport layer function, there is no concept of 5-tuple flows
at IP layer. So there should be no reason to constrain it only to the network layer.
We just need to revisit over transport-layer is end-to-end-only dogma.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 19 Oct 2017 23:23:53 +0200

> That's pretty much what RFC7045/RC8200 say, except that they

It does not analyze the fact that the existing hop-by-hop option (and options
carried via it like router alert) are burned because existing router imple entations
punt packets with hop-by-hop options even though they do ultimately not
need to process the packets (hop by hop option they don't do or router-alert
protocol they don't do). 

And IMHO the existing hop-by-hop option is burned because the architecture
RFCs did not well enough express in MUST statements that you must never
speed down packets because of hop-by-hop/router-alert unless you really know
you will support/do-something with that option. And that you achieve this eg.:
by filtering/punting based on option/protocol. And that you can NOT do anything else.

This is also the root cause why in our analysis a hacky extraction by
eg: STUN signature inside UDP is a lot safer for existing networks than relying
on any hop-by-hop option. Badly standardized, badly implemented.

> supported, you can reasonably expect that all routers process X.

Ask an enterprise operator with 10 different router models from 3 vendors ;-)

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Thu, 19 Oct 2017 21:41:42 +0000

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Toerless Eckert

> Finally wrt. architectural cleanlyness: Per-flow service negotiation
> IMHO is clearly a transport layer function, there is no concept of
> 5-tuple flows at IP layer.

I would agree I  principle, although IPv6 does have that flow label that makes the 5-tuple less essential to identify an individual "flow," and it sits at layer 3. The problem remains, though: security mechanisms, which render the layer 4 QoS knobs unworkable. Everything outside the "security enclaves" at either end, and these security enclaves might just be inside the two end hosts themselves, will have to be delivered best effort.

Bert

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 Oct 2017 15:00:35 -0700

On Thu, Oct 19, 2017 at 2:41 PM, Manfredi, Albert E <
[email-masked]> wrote:

> I would agree I  principle, although IPv6 does have that flow label that
> makes the 5-tuple less essential to identify an individual "flow," and it
> sits at layer 3.

Right.

> The problem remains, though: security mechanisms, which render the layer 4
> QoS knobs unworkable. Everything outside the "security enclaves" at either
> end, and these security enclaves might just be inside the two end hosts
> themselves, will have to be delivered best effort.
>
> Bert, I don't follow. One of the big wins using the 3-tuple is that it can
still indicate a flow even if all the transport transport headers and
payload are encrypted. So this should allow specifying QoS without
revealing anything about the content except that the sender wants to group
certain packets together as a flow and apply QoS on them.

Tom

Bert

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 00:09:35 +0200

> certain packets together as a flow and apply QoS on them.

Problem with flow label is same as with hop-by-hop option: burned because
nobody should dare use it for something useful with all the inconsistent 
code out in the field.  Nothing against redefining it (with new encoding)
with a precise set of MUST/MUST-NOTs once we agree on some way to use it.

http://snaggletooth.akam.ai/Prague-videos/ -> 03-joel-jaeggli

Cheers
    Toerless

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Thu, 19 Oct 2017 22:13:16 +0000

From: Tom Herbert [mailto:[email-masked]] 

>> The problem remains, though: security mechanisms, which render the layer
>> 4 QoS knobs unworkable. Everything outside the "security enclaves" at
>> either end, and these security enclaves might just be inside the two end
>> hosts themselves, will have to be delivered best effort.
>

True, Tom. I was responding to Toeless, when he says:

"Finally wrt. architectural cleanlyness: Per-flow service negotiation IMHO is clearly a transport layer function, there is no concept of 5-tuple flows at IP layer."

I saw that Toerless just responded, so he might be saying the same thing. But constraining the idea of in-band signaling to just IPv6 ought to be more workable, although you do need to get all those intermediate routers involved. I dunno. This always seems like the Achilles heel.

Bert

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Thu, 19 Oct 2017 22:16:23 +0000

Entschuldigen Sie bitte! Toerless!

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Manfredi, Albert E
Sent: Thursday, October 19, 2017 18:13
To: Tom Herbert <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: RE: I-D Action: draft-han-6man-in-band-signaling-for-transport-qos-00.txt

Bert

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 00:52:56 +0200

> Entschuldigen Sie bitte! Toerless!

[ Yeah, last i counted, all my toes are still there.
  I thought the correct abbreviation was self explanatory: "less" ]

All my lamenting here was exactly to avoid running into the achilles heel
of existing onpath router code that would mess up good new ideas. 

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 Oct 2017 16:38:11 -0700

On Thu, Oct 19, 2017 at 3:09 PM, Toerless Eckert <[email-masked]> wrote:

> Toerless,

I know that EH is an issue because some implementations drop packets having
them. But what is the problem with flow labels? Non-zero flow labels are
being used everyday on the Internet without problem, and I know of at least
one big DC deployment that is productively using flow labels for ECMP in
their network.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Oct 2017 13:05:49 +1300

> code out in the field.  Nothing against redefining it (with new encoding)

There is no encoding. It's 20 opaque bits (and always has been). Please
read https://tools.ietf.org/html/rfc6437.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Oct 2017 13:20:40 +1300

> need to process the packets (hop by hop option they don't do or router-alert

We discussed that before agreeing on the words in 7045, which were +- adopted
in 8200.

> by filtering/punting based on option/protocol. And that you can NOT do anything else.

The protocol design really shouldn't talk about implementation choices
in routers and the fact that FPGA people hate the IPv6 extension header
design like the plague, which has led to broken implementations. In 1994
there weren't any 'fast path' router designs, iirc, and the design simply
didn't consider this problem to be a problem. So the assumption was that
the forwarder CPU would pass on the whole header, and would only branch
off the main code path if it saw a HbH header. I know we're 20 years
later, but logically that hasn't changed. The problem is that many vendors
didn't implement it; they sacrificed correctness for speed.

> on any hop-by-hop option. Badly standardized, badly implemented.

We agree on badly implemented ;-). IPv4 Options are badly implemented, too.

> Ask an enterprise operator with 10 different router models from 3 vendors ;-)

If they want to use X they need to buy routers that support X.

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 Oct 2017 17:40:29 -0700

On Thu, Oct 19, 2017 at 5:20 PM, Brian E Carpenter <
[email-masked]> wrote:

> duplicate IGMPv3),
> down packets
> If the device
> Or at least buy routers that ignore X instead of unilaterally dropping
packets that contain X (be liberal in what you receive!).

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Fri, 20 Oct 2017 00:51:23 +0000

-----Original Message-----

> If they want to use X they need to buy routers that support X.

Absolutely, and that's the rub, isn't it? In-band QoS, no matter really whether it's done using an IPv6 3-tuple, or IP-agnostic 5-tuple (without encryption), is ultimately practical only inside walled gardens.

These discussions bring me back to ... ATM. Trying to re-create ATM with IP. But ATM died, in large part, because of this.

Bert

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 16:40:15 +0200

> > code out in the field.  Nothing against redefining it (with new encoding)
                                                          ^^^^^^^^^^^^^^^^^^^
> read https://tools.ietf.org/html/rfc6437.

Sorry, lame use of words. Meant to say "stronger definition of DO and DONTs
which may include new semantics". I never tried to analyze the state of
affairs on flow label as i did for router alert. Just taking it from what
i heard, last from Joel at the pechakucha.

Btw: IPv6 router alert did already fix one broken piece of IPv4 router
alert, the registry is more flexible ;-))

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Ole Troan <[email-masked]>
Date: Fri, 20 Oct 2017 16:58:16 +0200

Toerless,

>>> code out in the field.  Nothing against redefining it (with new encoding)

Right. If we take Joel's findings, then it appears the conclusion is that right now using a 4 tuple SA, DA, Prot, Flow for ECMP has a too high probability for failure. It would be very nice to have that fixed. Until then, we're forcing routers to parse transport headers.

Best regards,
Ole

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 17:34:43 +0200

On Fri, Oct 20, 2017 at 01:20:40PM +1300, Brian E Carpenter wrote:
> didn't consider this problem to be a problem.
>
> So the assumption was that

If you failed twice betting your architecture on correct implementations,
maybe its time to try stronger DO and DONTs and explore other options
to achieve a better result. 

Lets also remember that TLS is also seen as a tool to avoid broken
middleboxes to mess up your traffic flow. However good intentioned
the middlebox may be. So its not a bad idea to consider using encryption
to allow only those onpath devices that you trust (to at least work
correctly) to be able to do anything. Not easy to design, but possible.
But something that would make me prefer doing it in a common "flow-layer"
above IP instead of IP extension headers. Like what SPUD was suggesting
as well. I also hate the fact that we're redefining the transport prototcol
every time we need to improve something. Like we do now with QUIC.  Should
really start to think more about reusable transport layers.

> We agree on badly implemented ;-). IPv4 Options are badly implemented, too.

"Judge, its in my genes, my mother already exposed the same bad traits"

Yeah, that defense is known to reduce sentencing time but should
also make you want to design the next generation differently.

> If they want to use X they need to buy routers that support X.

http://dilbert.com/strip/2001-04-14

Enough comic relief ;-)

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 17:40:04 +0200

> packets that contain X (be liberal in what you receive!).

Just ask the SIP crowd about their experiences with SIP proxies
dropping every new option by default even the ones meant to be
passed on transparently. Aka: the issue of middleboxes isn't
limited to IP land.

Ultimately its human nature that operators who can act as middle men
have a tendency to misbehave, however good intentioned it may be.
See my last mail on encryption.

Cheers
    Toerless

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 Oct 2017 09:33:10 -0700

On Fri, Oct 20, 2017 at 7:58 AM, Ole Troan <[email-masked]> wrote:

> Right. If we take Joel's findings, then it appears the conclusion is that
> right now using a 4 tuple SA, DA, Prot, Flow for ECMP has a too high
> probability for failure. It would be very nice to have that fixed. Until
> then, we're forcing routers to parse transport headers.
>
> Ole,

I cannot find this presentation. I did find a presentation to nanog about
problems with flow labels, but that had more to do with the fact that the
flow label does not have to be persistent for the life of a connection
which breaks stateful firewalls that require consistent routing for a flow.
I didn't see how this is a problem for ECMP itself. Is there a draft on
this that details what the problems are?

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 18:41:31 +0200

> Absolutely, and that's the rub, isn't it? In-band QoS, no matter really whether it's done using an IPv6 3-tuple, or IP-agnostic 5-tuple (without encryption), is ultimately practical only inside wal…

IMHO only true for DiffServ QoS unmodified, not for other forms
of QoS. DiffServ QoS even has fundamental challenges inside walled
gardens:

- Way to complex to dynamically adjust queue share of classes
  in enterprise deployments (only useful in mission specific network
  with well defineable persistent traffic profiles).
- Unnecessary differentiation now that a lot more media traffic is elastic
  and a lot more best effort traffic not buffer-bloaty

> These discussions bring me back to ... ATM. Trying to re-create ATM with IP. But ATM died, in large part, because of this.

I don't know. There is this cool flying car from the 50th in the Seattle
Museum of Flight. I don't think its historic state proves that planes
or cars are bad ideas.  1280 is the the 53 and flying cars are also coming
back and so will QoS.

Anyhow. Even though DSCP is in IPv6 header and this WG claims that
it must be the ones deciding on onpath packet inspection, i fear
the mayority here is not interested in QoS, so lets move the discussion
to TSVWG.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Ole Troan <[email-masked]>
Date: Fri, 20 Oct 2017 20:20:03 +0200

Hi Tom,

> I cannot find this presentation. I did find a presentation to nanog about problems with flow labels, but that had more to do with the fact that the flow label does not have to be persistent for the life of a connection which breaks stateful firewalls that require consistent routing for a flow. I didn't see how this is a problem for ECMP itself. Is there a draft on this that details what the problems are?

I believe Joel's presentation was for a load balancer application.

Purely for ECMP it only has a consequence for reordering of packets. Which may be bad enough.
But sending packets of the same flow along different paths has problems with any network function that requires state. NAT64, virtual reassembly of fragments, ACLs and what not. Not that the network should do any of those of course.

Best regards,
Ole

---

From: Lin Han <[email-masked]>
Date: Fri, 20 Oct 2017 18:36:27 +0000

Hi, Albert

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Manfredi, Albert E
Sent: Thursday, October 19, 2017 5:51 PM
To: Brian E Carpenter <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: RE: Hop-by-hop [not draft-han-6man-in-band-signaling-for-transport-qos-00.txt]

-----Original Message-----

Absolutely, and that's the rub, isn't it? In-band QoS, no matter really whether it's done using an IPv6 3-tuple, or IP-agnostic 5-tuple (without encryption), is ultimately practical only inside walled gardens.
[LH] At this moment, yes. As I stated in the scope, the current solution is for a single admin domain. The inter-domain needs more study.

These discussions bring me back to ... ATM. Trying to re-create ATM with IP. But ATM died, in large part, because of this.
[LH] ATM is failed due to many reasons, scalability is one of it.
This solution is not intended to replace completely the best-effort transport service. The solution is only for those application which really needs a flow level QoS, such as AR/VR, tactile internet, etc. From this sense, it does not sacrifice the scalability of IP.

Bert

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 Oct 2017 11:43:51 -0700

On Fri, Oct 20, 2017 at 11:20 AM, Ole Troan <[email-masked]> wrote:

> > Right. If we take Joel's findings, then it appears the conclusion is
> that right now using a 4 tuple SA, DA, Prot, Flow for ECMP has a too high
> > I cannot find this presentation. I did find a presentation to nanog
> about problems with flow labels, but that had more to do with the fact that
> the flow label does not have to be persistent for the life of a connection
> But sending packets of the same flow along different paths has problems
> with any network function that requires state. NAT64, virtual reassembly of
> fragments, ACLs and what not. Not that the network should do any of those
> of course.
>
> Ole,

Right, but of course there's never been any requirement in IP that packets
in a flow always follow the same path or are never received out of order.
Even outside of using the flow label there are other ways that that for
packets of a flow to take different paths or be OOO (fragmentation, UDP
encapsulation, routing change, etc.). I don't see that there is a protocol
problem with flow labels that would justify a general recommendation
against their use. Maybe there should be a discussion in v6ops about this.

Tom

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Fri, 20 Oct 2017 19:00:13 +0000

-----Original Message-----
From: Lin Han [mailto:[email-masked]] 

> ATM is failed due to many reasons, scalability is one of it.
> This solution is not intended to replace completely the best-effort transport
> service. The solution is only for those application which really needs a flow
> level QoS, such as AR/VR, tactile internet, etc. From this sense, it does not
> sacrifice the scalability of IP.

Or at least, does not sacrifice it as much? ATM also had a UBR service.

But my point was only that out in the wild, you have no guarantee that the routers in the path will know what to do with your in-band QoS demands. Only in situations where all the routers are in your control will you have that guarantee. But sure, in walled gardens, these cool ideas can be made to work.

On the other hand, also in these controlled environments, another option, frequently, is to "throw bandwidth at the problem." Give yourself plenty of headroom. It's less elegant and less fun, but it works great.

Bert

---

From: Ole Troan <[email-masked]>
Date: Fri, 20 Oct 2017 21:04:57 +0200

Tom,

> Right, but of course there's never been any requirement in IP that packets in a flow always follow the same path or are never received out of order. Even outside of using the flow label there are other ways that that for packets of a flow to take different paths or be OOO (fragmentation, UDP encapsulation, routing change, etc.). I don't see that there is a protocol problem with flow labels that would justify a general recommendation against their use. Maybe there should be a discussion in v6ops about this.

Absolutely. There is no protocol problem with flow labels.

Cheers,
Ole

---

From: "Leddy, John" <[email-masked]>
Date: Fri, 20 Oct 2017 19:06:38 +0000

In order packet delivery requirements considered harmful to the Internet…

I’m assuming random per packet flow label assignments are within spec as long as they are set by the source.

John

From: ipv6 <[email-masked]> on behalf of Tom Herbert <[email-masked]>
Date: Friday, October 20, 2017 at 2:44 PM
To: Ole Troan <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: Re: Flow label [not draft-han-6man-in-band-signaling-for-transport-qos-00.txt]

On Fri, Oct 20, 2017 at 11:20 AM, Ole Troan <[email-masked]<mailto:[email-masked]>> wrote:
Hi Tom,

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 Oct 2017 13:53:54 -0700

> long as they are set by the source.

Yes, there is no requirement that the flow label be persistent at all in a
flow. That gives some nice properties. For instance, at FB we implemented
flowbender using the flow label so that when a host detects a poor quality
path it can randomly try another flow label for the connection to hopefully
find a better path (a type of source routing). Random Packet Spraying could
use the flow label and randomly set it per packet to achieve perfect load
balancing in a multi-path network.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 Oct 2017 23:34:56 +0200

> balancing in a multi-path network.

Seems like that spraying example is a good reason why one would need to have
maybe at least two flow labels: One thats permitted to create state, and one thats
not (eg: used for load balancing). 

Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Oct 2017 10:41:54 +1300

On 21/10/2017 09:53, Tom Herbert wrote:
> flow. 

http://mailman.postel.org/mailman/listinfo/internet-history says:

   To enable Flow-Label-based classification, source nodes SHOULD assign
   each unrelated transport connection and application data stream to a
   new flow.  A typical definition of a flow for this purpose is any set
   of packets carrying the same 5-tuple {dest addr, source addr,
   protocol, dest port, source port}.  It should be noted that a source
   node always has convenient and efficient access to this 5-tuple,
   which is not always the case for nodes that subsequently forward the
   packet.

So, technically it's a recommendation, not a requirement. But for load
balancing or ECMP to actually work, it's a requirement.

> That gives some nice properties. For instance, at FB we implemented

I think the MPTCP people might have a word or two to say about that.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Oct 2017 10:45:48 +1300

> Absolutely. There is no protocol problem with flow labels.

When we did RFC6437, 6438 and 7098 we were perfectly aware of the deployment
issues. Fixing them is a long term play. There's background discussion in
https://tools.ietf.org/html/rfc6436

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 Oct 2017 15:13:41 -0700

> balancing or ECMP to actually work, it's a requirement.

Brian,

I don't think that says anything that the flow label has to be persistent
for the life of the connection. It is incorrect, though, if any nodes
assume that the flow label is persistent. The soft state approach like in
this QoS draft doesn't require the flow label to be persistent.

Load balancing only fails when the destination address is being treated as
an anycast and the address is being used as an endpoint of connections that
are on different hosts. But such stateful load balancing becomes obsolete
as the externally visible flow information is decoupled from the end-to-end
identification of the flow like happens in QUIC, ESP, etc.

ECMP works regardless of whether the flow label is persistent as long as
flow labels have a fairly nice uniform distribution. But, even if a host
were to flip out and the flow label to be the same value in every packet
for every flow that still really doesn't break things, it just means that
ECMP effectively falls back to a 2-tuple hash.

> MPTCP might be better for TCP, but flow label works with _any_ transport
protocol. However, now that you mention it, I wonder how MPTCP works
through a load balancer. All the connections have different hashes but have
to wind up on the same backend host. Maybe it's another instance of the
externally visible flow information be decoupled from the end-to-end flow
identification.

Tom

---

From: Lin Han <[email-masked]>
Date: Fri, 20 Oct 2017 23:27:41 +0000

-----Original Message-----
From: Manfredi, Albert E [mailto:[email-masked]] 
Sent: Friday, October 20, 2017 12:00 PM
To: Lin Han <[email-masked]>; Brian E Carpenter <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: RE: Hop-by-hop [not draft-han-6man-in-band-signaling-for-transport-qos-00.txt]

-----Original Message-----

> This solution is not intended to replace completely the best-effort 
> transport service. The solution is only for those application which 
> really needs a flow level QoS, such as AR/VR, tactile internet, etc. 
> From this sense, it does not sacrifice the scalability of IP.

[LH] Yeah, everything has cost. But it is still different, ATM UBR is connection oriented, it needs setup and involves the control protocol such as PNNI. In our case, the IP architecture is not touched at all. IP best effort is as simple as before. The new solution does not impact the IP routing or any other protocols.

But my point was only that out in the wild, you have no guarantee that the routers in the path will know what to do with your in-band QoS demands. Only in situations where all the routers are in your control will you have that guarantee. But sure, in walled gardens, these cool ideas can be made to work.
[LH] Agreed. But not that strict. As I said in the document, for optimization, the HbH router is only needed for the point of the congestion or device introducing a lot latency. It does not need all routers on the path to be HbH aware. And if a router cannot guarantee the bandwidth, the application can fall back to normal TCP, it does not completely disable the transport.
The basic idea still works for multiple domain, I don't want to talk about it since it may introduce more debating. Let see if there is any flaw for a single domain. If it works, I'm sure technically we don't have problem to apply to multiple domains. 
One more infor for this domain issue. The network environment that an application may need this QoS (ultra-high bandwidth and ultra-low latency) may be flat, and very likely has a few domains (one or two). This is because of the limit of propagation delay. For example, in the architecture of 5G that is supposed to support the latency as low as single digit ms, the content provider is supposed to connect to the SP's access network directly and in the same metro. Only this deployment can provide the physical distance as short as possible.

On the other hand, also in these controlled environments, another option, frequently, is to "throw bandwidth at the problem." Give yourself plenty of headroom. It's less elegant and less fun, but it works great.
[LH] not completely get what you mean 

Bert

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Oct 2017 13:42:39 +1300

> identification of the flow like happens in QUIC, ESP, etc.

That's not really true for server load balancing. Delivering all packets
in the flow to the same *instance* of the host is essential to be
able to complete transactions. Whoever unwraps the flow identification
has to be able to select the required server instance, at line speed.
https://tools.ietf.org/html/rfc7098 sort of hints at this and we
tried to go further in
https://tools.ietf.org/html/draft-tarreau-extend-flow-label-balancing
but that faded away. And there's
https://tools.ietf.org/html/draft-wang-v6ops-flow-label-refelction

Nothing seems to quite work.

    Brian

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Oct 2017 13:50:30 +1300

On 21/10/2017 12:27, Lin Han wrote:
...
> [LH] not completely get what you mean 

Keep your average network utilisation well below 10% and your QoS problems are solved.
It's just queueing theory.

   Brian

---

From: Lin Han <[email-masked]>
Date: Sat, 21 Oct 2017 01:14:47 +0000

>It's just queueing theory.

Kind of correct. If all links are never congested, there is no problem and requirement for QoS especially for bandwidth. But this is not practical for SP. In reality, there are always oversubscription at network aggregation device, such as TOR, PGW, BRASS, etc. Thus the congestion will never go away. Most of applications don’t feel pain if it is not bandwidth/latency sensitive, but for some applications, this may not be true anymore.
For latency requirement, it is not gone even the network load is below link capacity, so, the QoS is still required. 

Lin

-----Original Message-----
From: Brian E Carpenter [mailto:[email-masked]] 
Sent: Friday, October 20, 2017 5:51 PM
To: Lin Han <[email-masked]>; Manfredi, Albert E <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: Re: Hop-by-hop [not draft-han-6man-in-band-signaling-for-transport-qos-00.txt]

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 Oct 2017 18:30:09 -0700

Brian,

The 5-tuple hashing for forwarding to VIPs only works to the extent
that the 5-tuple hash identifies the transport flow and is readily
visible. That may be true for plain TCP, but not necessarily other
protocols. For instance, in QUIC the end points identify a connection
by a connection ID that is separate from the 5-tuple. One of the goals
is to be resilient to NAT so that even if the 5-tuple changes, like
when the source address or port changes because NAT evicted a UDP
binding, the QUIC connection is still viable
(draft-ietf-quic-transport-07). So a load balancer just looking at UDP
5-tuples won't work for QUIC. There's a similar issue with MPTCP in
that all of the sub-flows have different hashes but need to be
delivered to the same server instance
(draft-duchene-mptcp-load-balancing-00).

> Nothing seems to quite work.
For IPv6, I am wondering if each server instance could just be given a
few million unique addresses and then replace load balancers with
simple routing...

In any case, I still don't see a protocol issue with flow labels.
Maybe further discussion on this topic should be on tsvwg list?

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Oct 2017 15:26:21 +1300

> (draft-duchene-mptcp-load-balancing-00).

Yes. Exactly the point of using the {source, destination, flow_label}
3-tuple in IPv6, but you know that.

> few million unique addresses and then replace load balancers with

Sure, but that means the client end would need to learn the address
from the server end, as part of the first handshake.
 
> In any case, I still don't see a protocol issue with flow labels.

I agree.

> Maybe further discussion on this topic should be on tsvwg list?

I'm not sure where server load balancing really belongs, but it's
a pretty important topic.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 20 Oct 2017 11:38:58 +1300

Switching to tsvwg. 6man is Bcc'ed. Previous discussions followed on from
https://mailarchive.ietf.org/arch/msg/ipv6/wb339ewBLP7Ijj2eCotCt-W_g8Y

> certain packets together as a flow and apply QoS on them.

Exactly. That's also why the flow label and traffic class are not covered
by IPSec authentication, in case some mechanism changes them en route.

   Brian

---

From: Mark Smith <[email-masked]>
Date: Mon, 23 Oct 2017 06:48:07 +1100

On 21 Oct. 2017 10:28, "Lin Han" <[email-masked]> wrote:

-----Original Message-----
From: Manfredi, Albert E [mailto:[email-masked]]
Sent: Friday, October 20, 2017 12:00 PM
To: Lin Han <[email-masked]>; Brian E Carpenter <
[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: RE: Hop-by-hop [not draft-han-6man-in-band-
signaling-for-transport-qos-00.txt]

-----Original Message-----

[LH] Yeah, everything has cost. But it is still different, ATM UBR is
connection oriented, it needs setup and involves the control protocol such
as PNNI. In our case, the IP architecture is not touched at all. IP best
effort is as simple as before.

Not at all.

You're creating state in the forwarding plane. State created based on
traffic characteristics  is vulnerable to state exhaustion attacks.

Even in closed domains this can be an issue, because "closed" domains
attached to the Internet aren't guarateed to stay closed. Back in 2000
Nimbda caused a DoS on a router that I looked after in a closed domain that
was creating dynamic ACL entries for basic stateful firewalling.

RFC791 and RFC8200 forwarding is stateless.

I'd even say traffic created state is worse than connection oriented state
such as switched virtual circuits in ATM. In that model, hosts are
requesting permission to send traffic on the network before they're able to
- if the network can't set up a connection for a host, the host cannot
contribute traffic load to the network.

Stateless and stateful forwarding is the fundamental difference between
packet and circuit switched networks.

See 2.3 in RFC1958.

Regards,
Mark.

The new solution does not impact the IP routing or any other protocols.

But my point was only that out in the wild, you have no guarantee that the
routers in the path will know what to do with your in-band QoS demands.
Only in situations where all the routers are in your control will you have
that guarantee. But sure, in walled gardens, these cool ideas can be made
to work.
[LH] Agreed. But not that strict. As I said in the document, for
optimization, the HbH router is only needed for the point of the congestion
or device introducing a lot latency. It does not need all routers on the
path to be HbH aware. And if a router cannot guarantee the bandwidth, the
application can fall back to normal TCP, it does not completely disable the
transport.
The basic idea still works for multiple domain, I don't want to talk about
it since it may introduce more debating. Let see if there is any flaw for a
single domain. If it works, I'm sure technically we don't have problem to
apply to multiple domains.
One more infor for this domain issue. The network environment that an
application may need this QoS (ultra-high bandwidth and ultra-low latency)
may be flat, and very likely has a few domains (one or two). This is
because of the limit of propagation delay. For example, in the architecture
of 5G that is supposed to support the latency as low as single digit ms,
the content provider is supposed to connect to the SP's access network
directly and in the same metro. Only this deployment can provide the
physical distance as short as possible.

On the other hand, also in these controlled environments, another option,
frequently, is to "throw bandwidth at the problem." Give yourself plenty of
headroom. It's less elegant and less fun, but it works great.
[LH] not completely get what you mean

Bert

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Sun, 22 Oct 2017 23:07:28 +0000

From: Mark Smith [mailto:[email-masked]] 

>> connection oriented, it needs setup and involves the control protocol
>> such as PNNI. In our case, the IP architecture is not touched at all.
>> IP best effort is as simple as before.
>
> Not at all.
>

I think that all Lin was saying was, "if you use best effort, then you haven’t sacrificed scalability." Which is true enough, I think. Whereas, with ATM, you are still having to set up that VC ahead of time, even if, with UBR service, you aren't specifying any QoS requirements along that path.

> attached to the Internet aren't guarateed to stay closed.

Yes, although I think Lin was saying, if you encounter routers that don’t know diddly about his in-band signaling, AND these routers aren't particularly congested, then all should still work fine. This traffic would hopefully sail through these lightly loaded inter-carrier routers, and then the QoS knobs would come into play in his own ISP network.

> requesting permission to send traffic on the network before they're able
> to - if the network can't set up a connection for a host, the host cannot

That's been my reaction here too. I think, at a most fundamental level, IP is a technique that allows for plenty of *cheap bandwidth*, which has made QoS guarantees mostly unnecessary. Cheap bandwidth allows, even begs for over-provisioning in the network, which makes complicated QoS guarantees unnecessary. And furthermore, apps that do require certain quality levels take it upon themselves to adjust to what's available, rather than giving that responsibility to the intervening network. I think that RTP/RTSP was quite revolutionary, in the telecom world.

On the other hand, schemes like ATM assume *expensive bandwidth*, so they make a big deal about doling out bandwidth with fine granularity. Or, more realistically, they optimistically created the knobs for doing so, which proved way too complicated, in the face of the "cheap bandwidth" connectionless IP competition of the day. But there has been a succession of efforts, from early on in IP, to add QoS knobs.

Bert

---

From: Tuboyan <[email-masked]>
Date: Mon, 23 Oct 2017 03:17:08 +0000

Hi Bert,

-----邮件原件-----
发件人: ipv6 [mailto:[email-masked]] 代表 Manfredi, Albert E
发送时间: 2017年10月23日 7:07
收件人: Mark Smith
抄送: 6man WG
主题: RE: Hop-by-hop [not draft-han-6man-in-band-signaling-for-transport-qos-00.txt]

> I'd even say traffic created state is worse than connection oriented 
> state such as switched virtual circuits in ATM. In that model, hosts 
> are requesting permission to send traffic on the network before 
> they're able to - if the network can't set up a connection for a host, 
> the host cannot contribute traffic load to the network.

[BoyanTu] Without help from network, even app in host can adjust to what's available, but app still cannot get bandwidth it need ,such as VR steam service. 

Bert

---

From: Lin Han <[email-masked]>
Date: Mon, 23 Oct 2017 23:52:49 +0000

[snip]

[snip]

Hi, Bert

This solution tries to use a simplest way to provide a decent QoS, and keep the IP network architecture and benefits of best effort service. As I emphasize in the document, it aimed to be a supplementary for the current best-effort based transport service, and only used for the service which really needs it.

"cheap bandwidth" solution can certainly solve a lot QoS problem, as we experienced in last decades. But can it solve all problems in all use case? Obviously not. For example, our current access speed does not mean E2E bandwidth. Why we don’t complain? since most of applications can tolerate the insufficient bandwidth. But some application has already show a lot challenge to network. If you want a 8K video or AR/VR service with decent quality now, no SP can provide it in a short time. 
On the other hand, "cheap bandwidth" sometime is not cheap, it actually pushes the SP to upgrade network devices to obtain more room for bandwidth. If this is practical, it will be easier to ask SP to design a network with no oversubscription at the beginning. All come to the cost of SP.

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Manfredi, Albert E
Sent: Sunday, October 22, 2017 4:07 PM
To: Mark Smith <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: RE: Hop-by-hop [not draft-han-6man-in-band-signaling-for-transport-qos-00.txt]

Bert