Subject: Re: [tsvwg] FW: New Version Notification for draft-yu-tsvwg-l3qcn-00.txt

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Mon, 21 Nov 2016 23:19:59 +0100

HI Yolanda,

I have not seen the minutes from the tsvwg session last week (and failed to
attend from remote) so please excuse me
if what I am asking was discussed there.

I personally have rejoiced over the fact that DCQCN (Rocev2) and
DCTCP/l4s/TCP-Prague
seems to be solutions of  the same common approach of relying on ECN aware
AQM mechanisms and the interaction
of such with end-node scalable ECN CC algorithms (even if there are subtle
differences)
for congestion control in the converged ethernet fabric.

I have been looking forward (had the naïve hope) to see DCQCN
 be considered in the context of the ECN/l4s activities in tsvwg.
(fairness, starvation, flow separation)

In the draft you argue that the control loop delay of DCQCN is "too big"
 and use that as a motivation for introducing L3QCN (or as a partly
motivation).

Could you provide some more information or considerations on this aspect  ?

I understand that Rocev2 have special demands for loss less but I would like
to understand to which
extend the stated limitation of DCQCN also would not carry over to DCTCP as
well.

Many thanks in advance.

BR, Karen

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of Yuxiang
> (Yolanda)
> Sent: 1. november 2016 03:42
> To: [email-masked]
> Subject: [tsvwg] FW: New Version Notification for draft-yu-tsvwg-l3qcn-
> 00.txt
>
> Dear all,
>
>     This draft introduces a congestion control method in the converged
> network (including QCN and RoCEv2).
>     Your comments, suggestions and questions are really appreciated.
>
> B.R.
> Yolanda Yu
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, October 31, 2016 5:31 PM
> To: Yuxiang (Yolanda)
> Subject: New Version Notification for draft-yu-tsvwg-l3qcn-00.txt
>
>
> A new version of I-D, draft-yu-tsvwg-l3qcn-00.txt has been successfully
> submitted by Yolanda Yu and posted to the IETF repository.
>
> Name:		draft-yu-tsvwg-l3qcn
> Revision:	00
> Title:		Layer 3 Quantized Congestion
> Notification(L3QCN)in the Converged Network
> Document date:	2016-10-31
> Group:		Individual Submission
> Pages:		12
> URL:
> https://www.ietf.org/internet-drafts/draft-yu-tsvwg-l3qcn-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-yu-tsvwg-l3qcn/
> Htmlized:       https://tools.ietf.org/html/draft-yu-tsvwg-l3qcn-00
>
>
> Abstract:
>    The more demands for the lossless and low latency network in the
>    modern datacenter appear because the proliferation of demanding
>    applications. Some congestion control schemes such as CN, PFC, ETS
>    which is introduced by IEEE 802.1 focus on the L2 network domain.
>    While current TCP/IP stacks can't meet these requirement on L3 or
>    above networks. This draft introduces the L3QCN(Layer 3 Quantized
>    Congestion Notification), an end to end congestion control scheme
>    which adopt QCN and DCQCN on L2 network. It specifies protocols,
>    procedures, and managed objects to support congestion control on the
>    datacenter network.
>
>
>
>
> Please note that it may take a couple of minutes from the time of
> submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat

---

From: "Yuxiang (Yolanda)" <[email-masked]>
Date: Tue, 29 Nov 2016 08:42:18 +0000

Hi Karen,

     Sorry for the late response. I was on the business trip in the last few days.
     For your questions I will explain as follows. Hope it will be useful.

     With the network growing bigger and the port capacity going faster, the control loop problem will become severer.
     For example, in a 3-tier CLOS network, when the congestion occurs on the first hop, the ECN solution will take another more 9 hops delay than direct-feedback solutions, nearly a RTT.
     In high speed network, such as 100G links, the situation will be worse. It is likely that when the CNP arrives at the source node, the flow has completed.
     Besides that, ECN has only two states, "non-congestion" and "congestion", which can not inform the extent of congestion clearly. 
     While the direct-feedback solutions such as L3QCN can. The congestion extent such as queue delta information could be used for a more precise rate limit algorithm.

     I will update the draft with more test results and consider more generic DC network scenarios in the following versions.
 
     Thank you for your attention and suggestions. We could have further discussion.

B.R.
Yolanda Yu

-----Original Message-----
From: Karen Elisabeth Egede Nielsen [mailto:[email-masked]] 
Sent: Tuesday, November 22, 2016 6:20 AM
To: Yuxiang (Yolanda); [email-masked]
Subject: RE: [tsvwg] FW: New Version Notification for draft-yu-tsvwg-l3qcn-00.txt

HI Yolanda,

I have not seen the minutes from the tsvwg session last week (and failed to attend from remote) so please excuse me if what I am asking was discussed there.

I personally have rejoiced over the fact that DCQCN (Rocev2) and DCTCP/l4s/TCP-Prague seems to be solutions of  the same common approach of relying on ECN aware AQM mechanisms and the interaction of such with end-node scalable ECN CC algorithms (even if there are subtle
differences)
for congestion control in the converged ethernet fabric.

I have been looking forward (had the naïve hope) to see DCQCN  be considered in the context of the ECN/l4s activities in tsvwg.
(fairness, starvation, flow separation)

In the draft you argue that the control loop delay of DCQCN is "too big"
 and use that as a motivation for introducing L3QCN (or as a partly motivation).

I understand that Rocev2 have special demands for loss less but I would like to understand to which extend the stated limitation of DCQCN also would not carry over to DCTCP as well.

Many thanks in advance.

BR, Karen

> Subject: [tsvwg] FW: New Version Notification for 
> draft-yu-tsvwg-l3qcn- 00.txt
>
> Dear all,
>
> A new version of I-D, draft-yu-tsvwg-l3qcn-00.txt has been 
> successfully submitted by Yolanda Yu and posted to the IETF repository.
>
> submission until the htmlized version and diff are available at 
> tools.ietf.org.
>
> The IETF Secretariat

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Tue, 29 Nov 2016 11:47:57 +0100

HI Yolanda,

Thank you so much for your feedback.

Please see inline below.

BR, Karen

> From: Yuxiang (Yolanda) [mailto:[email-masked]]
> Sent: 29. november 2016 09:42
> To: Karen Elisabeth Egede Nielsen <[email-masked]>
> Cc: [email-masked]; Shenli (kevin) <[email-masked]>; Zhouhong
> (Stephen) <[email-masked]>
> Subject: RE: [tsvwg] FW: New Version Notification for
> draft-yu-tsvwg-l3qcn-
> 00.txt
>
> Hi Karen,
>
>      With the network growing bigger and the port capacity going faster,
> the
> control loop problem will become severer.
>      For example, in a 3-tier CLOS network, when the congestion occurs on
> the
> first hop, the ECN solution will take another more 9 hops delay than
> direct-
> feedback solutions, nearly a RTT.
>      In high speed network, such as 100G links, the situation will be
> worse. It is
> likely that when the CNP arrives at the source node, the flow has
> completed.

[Karen Elisabeth Egede Nielsen]
I understand this as the initial window problem. Potentially divided into
the following two:
* Initial window (and slow start) CC operation not supported by the
transport
* Incast problems associated with that x*initial window problem causes
congestion (even when "proper" CC IW operation is attempted).

Is this correctly understood. Would you agree with this ?

In the draft you refer to PFC. Rocev2 uses PFC in combination with ECN-DCQCN
to address the "initial window problem" and generally as
last resort protection when ECN-DCQCN  fails to regulate the traffic
adequately.

In this respect I think that it is slightly unfair to say that you need 3QCN
to handle limitations of DCQCN, as
 PFC is presently attempted to be used to mitigate these limitation. Perhaps
it would be fair'er to state that L3QCN has advantages to
the combination of PFC and DCQCN - ? (assuming it has ?).

Something else which I am curious  to understanding better is why one does
not go in the direction of enforcing usage of
initial window and slow start alike in the transports, e.g. in Rocev2. Is it
because that avoiding extra RTT's is unacceptable in happy day congestion
free scenario.
(It think that I understand by now that using initial window and slow start
and ECN does not fully solve incast.)

Not saying that L3QCN does not look useful. Actually the opposite.
I am just trying to understand the benefits vis-à-vis the existing
mechanisms
as well as the inapplicability of the general approaches.

>      Besides that, ECN has only two states, "non-congestion" and
> "congestion",
> which can not inform the extent of congestion clearly.
>      While the direct-feedback solutions such as L3QCN can. The congestion
> extent such as queue delta information could be used for a more precise
> rate
> limit algorithm.
>
[Karen Elisabeth Egede Nielsen] OK.
 I suppose that direct-feedback solution and more granular congestion
information are two different aspects and need not be interrelated.
Good to distinguish the two.

>      I will update the draft with more test results and consider more
> generic DC
> network scenarios in the following versions.
>
[Karen Elisabeth Egede Nielsen] Attempted - if the tsvwg will allow :-)

Many Thanks.

> Subject: RE: [tsvwg] FW: New Version Notification for
> draft-yu-tsvwg-l3qcn-
> 00.txt
>
> HI Yolanda,
>
> attend from remote) so please excuse me if what I am asking was discussed
> there.
>
> DCTCP/l4s/TCP-Prague seems to be solutions of  the same common
> approach of relying on ECN aware AQM mechanisms and the interaction of
> such with end-node scalable ECN CC algorithms (even if there are subtle
> differences)
> I have been looking forward (had the naïve hope) to see DCQCN  be
> considered in the context of the ECN/l4s activities in tsvwg.
> to understand to which extend the stated limitation of DCQCN also would
> not carry over to DCTCP as well.
>
> Many thanks in advance.
>
> BR, Karen
>
>

---

From: "Yuxiang (Yolanda)" <[email-masked]>
Date: Wed, 30 Nov 2016 11:09:11 +0000

Hi Karen,

     Hope the following answers help.

Is this correctly understood. Would you agree with this ?
A (Yolanda) : Perhaps we considered with different aspects. I think it is not about initial window. It is about response time. ECN-DCQCN needs nearly a RTT to take effective. L3QCN can shorten the response time to half on average. So when L3QCN is deployed with PFC, the PFC pause frame will be triggered less than DCQCN in combination with PFC. So L3QCN can ease the congestion propagation problem brought by PFC.

Something else which I am curious to understanding better is why one does not go in the direction of enforcing usage of initial window and slow start alike in the transports, e.g. in Rocev2. Is it because that avoiding extra RTT's is unacceptable in happy day congestion free scenario.
(It think that I understand by now that using initial window and slow start and ECN does not fully solve incast.)
A (Yolanda) : Currently someone are researching on deploying RoCEv2 on a lossy network. So it is possible that RoCEv2 protocol stack will add some TCP-like algorithms such as slow start and loss recovery. Maybe this will be the direction.

B.R.
Yolanda Yu

-----Original Message-----
From: Karen Elisabeth Egede Nielsen [mailto:[email-masked]] 
Sent: Tuesday, November 29, 2016 6:48 PM
To: Yuxiang (Yolanda)
Cc: [email-masked]; Shenli (kevin); Zhouhong (Stephen)
Subject: RE: [tsvwg] FW: New Version Notification for draft-yu-tsvwg-l3qcn-00.txt

HI Yolanda,

Please see inline below.

BR, Karen

>      With the network growing bigger and the port capacity going 
> faster, the control loop problem will become severer.
>      For example, in a 3-tier CLOS network, when the congestion occurs 
> on the first hop, the ECN solution will take another more 9 hops delay 
> than
> direct-
> worse. It is likely that when the CNP arrives at the source node, the 
> flow has completed.

[Karen Elisabeth Egede Nielsen]
I understand this as the initial window problem. Potentially divided into the following two:
* Initial window (and slow start) CC operation not supported by the transport
* Incast problems associated with that x*initial window problem causes congestion (even when "proper" CC IW operation is attempted).

In the draft you refer to PFC. Rocev2 uses PFC in combination with ECN-DCQCN to address the "initial window problem" and generally as last resort protection when ECN-DCQCN  fails to regulate the traffic adequately.

In this respect I think that it is slightly unfair to say that you need 3QCN to handle limitations of DCQCN, as  PFC is presently attempted to be used to mitigate these limitation. Perhaps it would be fair'er to state that L3QCN has advantages to the combination of PFC and DCQCN - ? (assuming it has ?).

Something else which I am curious  to understanding better is why one does not go in the direction of enforcing usage of initial window and slow start alike in the transports, e.g. in Rocev2. Is it because that avoiding extra RTT's is unacceptable in happy day congestion free scenario.
(It think that I understand by now that using initial window and slow start and ECN does not fully solve incast.)

Not saying that L3QCN does not look useful. Actually the opposite.
I am just trying to understand the benefits vis-à-vis the existing mechanisms as well as the inapplicability of the general approaches.

> "congestion", which can not inform the extent of congestion clearly.
>      While the direct-feedback solutions such as L3QCN can. The 
> congestion extent such as queue delta information could be used for a 
> more precise rate limit algorithm.
>
[Karen Elisabeth Egede Nielsen] OK.
 I suppose that direct-feedback solution and more granular congestion information are two different aspects and need not be interrelated.
Good to distinguish the two.

> generic DC network scenarios in the following versions.
>

Many Thanks.

> I have not seen the minutes from the tsvwg session last week (and 
> failed to attend from remote) so please excuse me if what I am asking 
> was discussed there.
>
> I understand that Rocev2 have special demands for loss less but I 
> would like to understand to which extend the stated limitation of 
> DCQCN also would not carry over to DCTCP as well.
>
> Many thanks in advance.
>
> BR, Karen
>
>
> >     This draft introduces a congestion control method in the 
> > converged network (including QCN and RoCEv2).

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Wed, 30 Nov 2016 12:52:29 +0100

HI Yolanda,

Thanks a lot for the feedback. I think we do have a common understanding
then after all.

With slow start and loss recovery additions to Rocev2 with DC-QCN,
it will become very similar to DC-TCP/TCP_Prague.

BR, Karen

> Sent: 30. november 2016 12:09
> Cc: [email-masked]
> A (Yolanda) : Perhaps we considered with different aspects. I think it is
> not
> about initial window. It is about response time. ECN-DCQCN needs nearly a
> RTT to take effective. L3QCN can shorten the response time to half on
> average. So when L3QCN is deployed with PFC, the PFC pause frame will be
> triggered less than DCQCN in combination with PFC. So L3QCN can ease the
> congestion propagation problem brought by PFC.
>
> not go in the direction of enforcing usage of initial window and slow
> start
> alike in the transports, e.g. in Rocev2. Is it because that avoiding extra
> RTT's is
> unacceptable in happy day congestion free scenario.
> (It think that I understand by now that using initial window and slow
> start and
> ECN does not fully solve incast.) A (Yolanda) : Currently someone are
> researching on deploying RoCEv2 on a lossy network. So it is possible that
> RoCEv2 protocol stack will add some TCP-like algorithms such as slow start
> and loss recovery. Maybe this will be the direction.
>
> B.R.
> Yolanda Yu
>
> following two:
> In the draft you refer to PFC. Rocev2 uses PFC in combination with ECN-
> DCQCN to address the "initial window problem" and generally as last resort
> protection when ECN-DCQCN  fails to regulate the traffic adequately.
>
> In this respect I think that it is slightly unfair to say that you need
> 3QCN to
> handle limitations of DCQCN, as  PFC is presently attempted to be used to
> mitigate these limitation. Perhaps it would be fair'er to state that L3QCN
> has
> advantages to the combination of PFC and DCQCN - ? (assuming it has ?).
>
> not go in the direction of enforcing usage of initial window and slow
> start
> alike in the transports, e.g. in Rocev2. Is it because that avoiding extra
> RTT's is
> unacceptable in happy day congestion free scenario.
> (It think that I understand by now that using initial window and slow
> start and
> ECN does not fully solve incast.)
>