Subject: [tsvwg] WGLC Conclusion: draft-ietf-tsvwg-tunnel-congestion-feedback-03

From: "Black, David" <[email-masked]>
Date: Wed, 23 Nov 2016 23:53:29 +0000

This WGLC has concluded, but more review is needed.

There are two email threads of comments that need attention in a revised draft:
	- Tunnel Congestion Feedback: (non-)update to RFC6040
	- WGLC on Tunnel Congestion Feedback draft - David Black's comments

As noted in the meeting in Seoul, one serious review is not enough for a WGLC - Bob Briscoe and Jake Holland have signed up to do additional reviews.   As previously noted, thank you to both - it would be good to see those reviews posted to the list by the end of next week, please.

Thanks, --David

> -----Original Message-----
> From: Black, David
> Sent: Wednesday, October 26, 2016 6:52 PM
> To: [email-masked]
> Cc: Black, David
> Subject: WGLC: draft-ietf-tsvwg-tunnel-congestion-feedback-03
> 
> This email announces a TSVWG Working Group Last Call (WGLC) on:
> 
>                        Tunnel Congestion Feedback
>              draft-ietf-tsvwg-tunnel-congestion-feedback-03
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-tunnel-congestion-feedback/
> 
> Due to the upcoming Seoul meetings, this WGLC will last for about 2.5 weeks
> weeks, ending at midnight Korea Time on Sunday, November 13, in order to
> enable discussion of any issues at TSVWG sessions in Seoul later that week.
> 
> Comments should be sent to the [email-masked] list, although purely
> editorial comments may be sent directly to the authors at
> [email-masked].  Please cc: the
> WG chairs at [email-masked]  if you would like the chairs to track
> such editorial comments as part of the WGLC process.
> 
> There are two IPR disclosures that relate to this draft:
> https://datatracker.ietf.org/ipr/search/?submit=draft&id=draft-ietf-tsvwg-
> tunnel-congestion-feedback
> 
> I will also kick off WGLC by announcing a proposed resolution of the ECN
> codepoint issue for this draft - that issue involves which ECT codepoint(s)
> to use for the "faked" ECN in the outer header of a packet whose inner
> header does not enable ECN.
> 
> With some gentle guidance from the ADs, the proposed resolution is to use
> only ECT(0), and state that ECT(1) MUST NOT be used for a couple of reasons:
> 
> 1) Tunnel Congestion Feedback (TCF) needs only one ECT codepoint, hence
> RFC 3168 indicates that ECT(0) SHOULD be used for TCF.  This "SHOULD" is
> a general recommendation for all protocols that use only one ECT codepoint.
> 
> 2) There's an IETF intent to use ECT(1) with L4S, and TCF interacts badly with
> L4S to the point of being destructive.   This interaction arises within a TCF tunnel
> that uses "faked" ECN, because L4S marks ECT(1) packets as having experienced
> congestion (CE mark) with much higher probability than drop-equivalence, but
> the TCF tunnel egress treats all CE marks as drop equivalent and drops the
> packets.  The resulting sender congestion response (backoff) to the drops
> will be far more severe than it should be and avoiding that response is why
> ECT(1) MUST NOT be used for TCF.  This "MUST NOT" is specific to TCF due
> to the interaction with L4S.
> 
> WGLC comments are welcome on this proposed resolution.  I'll work out
> actual text with the TCF draft authors with input from interested parties
> (e.g., Bob Briscoe), as the above rationale does need to be captured in the
> draft.
> 
> Thanks, --David
> --------------------------------------------------------
> David L. Black, Distinguished Engineer
> Dell EMC, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953     Cell: +1 (978) 394-7754
> [email-masked]  <=== NEW ===
> --------------------------------------------------------

---

From: "Holland, Jake" <[email-masked]>
Date: Sun, 27 Nov 2016 02:17:10 +0000

Here’s the review I promised in the meeting at IETF 97.

Overall, I like the idea in this draft a lot and I hope the ideas in it see deployment. However, I think there are some problems with the document in its current state.

I’m separately sending some editorial comments to the authors and chairs, but this email has the list of content issues I’d like to see addressed before sending it out of the working group.

--------

Technical Objections:

1. "Faked" ECT, described in section 4, seems like it conflicts with guidance from RFC 6040, and probably should have a more in-depth explanation directly addressing that point and justifying the reasons.

relevant quotes from section 7 of RFC 6040:
- "The overriding guidance is: Avoid designing alternate ECN tunnelling semantics, if at all possible"
- 1. The ECN field of the outer header ought to be cleared to Not-ECT unless it is guaranteed that the corresponding tunnel egress will correctly propagate congestion markings

In order to not conflict here, this draft should probably recommend some mechanisms or advice for ensuring that the egress will correctly handle faked ECT, or should have an applicability section that restricts its deployment to environments where there’s some independent guarantee that ingress and egress will be upgraded together. I wasn't clear on which solution was intended.

2. This draft seems to implicitly require there to be a security association wherever there is an IPFIX connection. If the egress is a different administrative domain than the ingress and/or the controller, it seems this opens some unmentioned security concerns. Like issue #1, this might be a question answerable with an applicability section.

3. This draft is much lighter on implementation details than most RFCs I’ve seen that add new global registry entries. I would much prefer to see a section that outlines at least one (preferably more) example algorithms that a controller might use to perform traffic management in response to congestion, with the corresponding example template record that lists the information necessary to support the algorithm. The examples should be sufficient to demonstrate the need for the new IPFIX registry entries.

4. I would likewise much prefer to see an explanation of the expected handling of the IPFIX connections, especially the expected behavior by the egress and the controller if the template records exchanged are different from what's expected. (Or otherwise how to ensure non-harmful fallback behavior if the tunnel endpoints or controller might be incompatible.)

5. I think for this draft, a better (but still brief) overview of tunnels is called for than what’s provided in paragraph 3 of the introduction, with references to tunneling protocols that are applicable. Maybe borrow language from RFC 2983 sections 1 and/or 2?

6. I'd like to see a walkthrough of an example of the handling sequence for the IPFIX messaging, even if a full example algorithm is deemed out of scope.

----------

This section is not exactly objections, but questions about the doc that I'd like to see explained or reconsidered:

1. Why is the direct model a different case from the centralized model with the controller running on the ingress system?  It seems like the descriptions and diagrams could be simplified by defining only one model (the centralized one), but stating that the controller is allowed to be embedded in the ingress system, and if so, that it’s allowed to ensure compatible versioning and secured communication with the ingress by techniques that make use of the shared system.

2. Why does the IPFIX connection go from ingress to egress to controller, rather than from ingress to controller and separately from egress to controller?

3. It might be nice to explain much earlier in the document that this draft applies to tunnels implementing the rules from section 4 of RFC 6040 for congestion signaling.

4. In the introduction, the ConEx reference in the introduction was confusing to me. If I'm understanding correctly, it says this is NOT a ConEx protocol, but ConEx does something similar?

I would say there are at least 3 options to improve this:
a. if this is similar to ConEx but different, add a section that explains why this should NOT be considered a ConEx protocol (what's incompatible between this and ConEx?), or
b. if this should be a ConEx protocol, follow the requirements from the list in section 3 of RFC 7713 for what should be detailed when specifying a ConEx protocol, or
c. if ConEx this proposal should be considered independently of ConEx and ConEx is irrelevant, just remove the ConEx reference. I think if the intent was to use it as an example, it raises more questions than it answers.

I read the specs for a few hours, but I couldn't easily tell which was the right option, though I did notice that section 5.4 of RFC7713 says the mechanisms for ConEx shouldn't be flow-specific, but IPFIX (RFC 7011) seems to be a flow-specific carrier of information.

(I’m not sure I got this right, but the described behavior seems to be a ConEx protocol, but treating the tunnel as its own end-to-end connection?)

Or if the whole purpose of this draft is to sketch an example justification for adding the IPFIX entries, that point should be more explicitly clear, because sections of this seem to be halfway defining a protocol, but not well enough to implement it.

5. I think there are some timing issues with trying to infer loss by comparing packet counts at ingress and egress, so I think a discussion of the complications (or at least a reference for further reading) seems like an important point to mention in this draft.

-------

Again, the basic idea expressed in this draft seems worthwhile to me and I hope it sees deployment in some form, but it also seems to me this draft still needs some work, and is missing some important parts that are probably necessary.

Regards,
Jake

On 11/23/16, 3:53 PM, "Black, David" <[email-masked]> wrote:

    
    
    Thanks, --David

---

From: "Weixinpeng (Jackie)" <[email-masked]>
Date: Tue, 29 Nov 2016 11:50:21 +0000

Hi Jake,
	Thanks for your insightful comments, and please see my replies inline.

Regards,
-Xinpeng (Jackie)

>From: Holland, Jake [mailto:[email-masked]]
>Sent: Sunday, November 27, 2016 10:17 AM
>To: [email-masked]
>Cc: [email-masked]
>Subject: Re: [tsvwg] WGLC Conclusion:
>Overall, I like the idea in this draft a lot and I hope the ideas in it see deployment.
>However, I think there are some problems with the document in its current
>state.
>
>I’m separately sending some editorial comments to the authors and chairs, but
>this email has the list of content issues I’d like to see addressed before sending
>it out of the working group.
>
>--------
>
>Technical Objections:
>
>1. "Faked" ECT, described in section 4, seems like it conflicts with guidance from
>RFC 6040, and probably should have a more in-depth explanation directly
>addressing that point and justifying the reasons.
>
>- 1. The ECN field of the outer header ought to be cleared to Not-ECT unless it is
>guaranteed that the corresponding tunnel egress will correctly propagate
>congestion markings
>
>In order to not conflict here, this draft should probably recommend some
>mechanisms or advice for ensuring that the egress will correctly handle faked
>ECT, or should have an applicability section that restricts its deployment to
>environments where there’s some independent guarantee that ingress and
>egress will be upgraded together. I wasn't clear on which solution was intended.
[Wei] Yes, the "Faked" ECT seems conflicting with RFC6040. The reason we think the "Faked" ECT will work is that we assume the ingress and egress
are normally controlled under the same network operator, so the upgrade on both ingress and egress can be guaranteed. I agree with you that there should 
be some clarification on this solution's applicability in the document.
>
>
>2. This draft seems to implicitly require there to be a security association
>wherever there is an IPFIX connection. If the egress is a different administrative
>domain than the ingress and/or the controller, it seems this opens some
>unmentioned security concerns. Like issue #1, this might be a question
>answerable with an applicability section.
[Wei] We assume the ingress and egress are in the same administrative domain. For the case that egress and ingress belong to 
different domain, there would be a bunch of security issues.
>
>
>3. This draft is much lighter on implementation details than most RFCs I’ve seen
>that add new global registry entries. I would much prefer to see a section that
>outlines at least one (preferably more) example algorithms that a controller
>might use to perform traffic management in response to congestion, with the
>corresponding example template record that lists the information necessary to
>support the algorithm. The examples should be sufficient to demonstrate the
>need for the new IPFIX registry entries.
[Wei] A good suggestion. I will consider it in the upcoming version.
>
>
>4. I would likewise much prefer to see an explanation of the expected handling of
>the IPFIX connections, especially the expected behavior by the egress and the
>controller if the template records exchanged are different from what's expected.
>(Or otherwise how to ensure non-harmful fallback behavior if the tunnel
>endpoints or controller might be incompatible.)
[Wei] I will add description on this in the upcoming version.
>
>
>5. I think for this draft, a better (but still brief) overview of tunnels is called for
>than what’s provided in paragraph 3 of the introduction, with references to
>tunneling protocols that are applicable. Maybe borrow language from RFC 2983
>sections 1 and/or 2?
[Wei] Thanks for this advice, I will revise it in the upcoming version.
>
>
>6. I'd like to see a walkthrough of an example of the handling sequence for the
>IPFIX messaging, even if a full example algorithm is deemed out of scope.
[Wei] ok.
>1. Why is the direct model a different case from the centralized model with the
>controller running on the ingress system?  It seems like the descriptions and
>diagrams could be simplified by defining only one model (the centralized one),
>but stating that the controller is allowed to be embedded in the ingress system,
>and if so, that it’s allowed to ensure compatible versioning and secured
>communication with the ingress by techniques that make use of the shared
>system.
[Wei] Ok, good suggestion.
>
>
>2. Why does the IPFIX connection go from ingress to egress to controller, rather
>than from ingress to controller and separately from egress to controller?
[Wei] The IPFIX message from ingress to egress is used as an "in-band" mechanism to 
convey flow information at the ingress to egress, when egress get the IPFIX message it will append  flow information collected
at egress in the message and send it to the controller/ingress.

>
>
>3. It might be nice to explain much earlier in the document that this draft applies
>to tunnels implementing the rules from section 4 of RFC 6040 for congestion
>signaling.
[Wei] ok.
>
>
>4. In the introduction, the ConEx reference in the introduction was confusing to
>me. If I'm understanding correctly, it says this is NOT a ConEx protocol, but
>ConEx does something similar?
[Wei] Your understanding is right.
>a. if this is similar to ConEx but different, add a section that explains why this
>should NOT be considered a ConEx protocol (what's incompatible between this
>and ConEx?), or b. if this should be a ConEx protocol, follow the requirements
>from the list in section 3 of RFC 7713 for what should be detailed when
>specifying a ConEx protocol, or c. if ConEx this proposal should be considered
>independently of ConEx and ConEx is irrelevant, just remove the ConEx
>reference. I think if the intent was to use it as an example, it raises more
>questions than it answers.
[Wei] If the ConEx mentioned here seems confusing, I think I should remove it.
>
>I read the specs for a few hours, but I couldn't easily tell which was the right
>option, though I did notice that section 5.4 of RFC7713 says the mechanisms for
>ConEx shouldn't be flow-specific, but IPFIX (RFC 7011) seems to be a
>flow-specific carrier of information.
>
>(I’m not sure I got this right, but the described behavior seems to be a ConEx
>protocol, but treating the tunnel as its own end-to-end connection?)
>
>Or if the whole purpose of this draft is to sketch an example justification for
>adding the IPFIX entries, that point should be more explicitly clear, because
>sections of this seem to be halfway defining a protocol, but not well enough to
>implement it.
>
>
>5. I think there are some timing issues with trying to infer loss by comparing
>packet counts at ingress and egress, so I think a discussion of the complications
>(or at least a reference for further reading) seems like an important point to
>mention in this draft.
[Wei] Actually the document has taken the timing issue into consideration, but it might 
need more clear description on this.
>
>-------
>
>Again, the basic idea expressed in this draft seems worthwhile to me and I hope
>it sees deployment in some form, but it also seems to me this draft still needs
>some work, and is missing some important parts that are probably necessary.
>
>Regards,
>Jake
>
>
>    As noted in the meeting in Seoul, one serious review is not enough for a
>WGLC - Bob Briscoe and Jake Holland have signed up to do additional reviews.
>As previously noted, thank you to both - it would be good to see those reviews
>posted to the list by the end of next week, please.
>
>    Thanks, --David
>

---

From: Vincent Roca <[email-masked]>
Date: Tue, 29 Nov 2016 17:14:02 +0100

Dear authors,

Here is another detailed review for you I-D, version 03.
Please note that I haven't followed any of the previous discussions on the list,
so this is a totally independent review (which is not necessarily bad). As a side
effect, some topics may have already been discussed. Sorry if this is the case.

GENERAL COMMENTS

1- In Section 4, it is assumed that "all routers in the tunnel support ECN."
   Isn't it too restrictive? What happens if this is not the case?
   And 2 paragraphs later, I read: "In case all interior routers support ECN".
   This is contradictory as no other possibility is considered.

2- In Section 4, it is required that measurements be made on "the same span of
   packets" between ingress and egress nodes. A potential way to do that is
   suggested in section 5, with cumulative packet counts. However it cannot
   totally solve the problem IMHO.

   Let's imagine that a large burst of packets is sent to the tunnel, leading
   to severe congestion, and that the ingress-to-egress signalling messages are
   all lost. Later on, only a few packets are sent sporadically and this time
   the signalling message reaches the egress node. This latter concludes that
   a certain number of packets have been lost over the **full** period, but it
   cannot determine if congestion happened recently or not.
   The period over which the observation is taking place must be small enough
   and recent enough for the mechanism to work correctly.
   This is a key point as it means that congestion evaluation (and mitigation)
   can only take place if signalling messages can be exchanged in a (sufficiently)
   reliable way. Is there a solution to guaranty this property?

3- Section 6 requires that "congestion management action must be delayed by
   more than a worst-case global RTT".
   How is this worst-case global RTT evaluated by the DP (i.e., ingress node or
   controller)? Since this global RTT is end-to-end, only end hosts can evaluate
   it. Is it suggested to do some kind of DPI in the ingress node to try to identify
   request/answer packet pairs in the traffic and therefore measure the current
   RTT? In that case, the host-to-ingress time is ignored and must be assessed...
   How to do that? And what about situations where the DP is not on the data 
   path? Do you have in mind another method?

4- Does the proposed mechanism work in case of recursive uses of tunnels (tunnel
   in a tunnel)? It's not mentioned IMHO.

5- Congestion management (actions taken by an AP) is a key aspect and Section 6
   provides a few hints. I have a few additional comments:
   - saying that "congestion management needs to avoid the impacts on e2e
     congestion control" is perhaps inappropriate. I'd rather say that it must
     not **compromise** e2e congestion control.
   - It is said that "Congestion Policing is used in data center to limit...".
     This sentence suggests that CP is only meaningful in a data center. Is it
     the case, which would also mean that Circuit Breakers are the solution to
     use in Internet? Could you clarify in the I-D?

SECURITY/PRIVACY CONSIDERATIONS

Section 7 is not appropriate IMHO.

6- First of all, a direct link to [RFC 7011] is needed instead of refering to
   IPFIX. And instead of limiting this sentence to "feeding back congestion",
   it should apply to all the signaling traffic (to/from ingress, egress, and
   controller).

7- Then the global threat should be mentionned. For instance (it's only a
   suggestion):
   "Without any security measure, an attacker located on a compromised router
    in the tunnel may create false congestion notifications by selectively
    dropping well chosen packets or by corrupting IPFIX feedback packets.
    Therefore it is required that IPFIX integrity, authentication, and
    confidentiality mechanisms be used (<be more directive here, I don't know
    what is available>)..."

8- From a privacy point of view, especially with centralized feedback,
   the controller is aware of the amount of traffic exchanged in each tunnel
   it is in charge, which is meaningful. Worse, in case of fine grained
   congestion management, this controller is aware of the amount of traffic
   for specific application flows inside the tunnel which is extremmely
   intrusive. The privacy implications must be discussed.

TYPOS AND MINOR REMARKS

** Section 2, DP definition: "an logical" => "a logical" and later (same
   sentence) "that make" => "that makes".
   Same remark for AP definition.

** Section 3.1 Direct Model: Please cut the long sentence "The egress consists
   of ... Collector" into 2 or more sentences (e.g.: "Thanks to this feedback
   information, the congestion management Decision Point ...").
   Also: "convey" => "conveys".

** Section 4: first use of ECT ("or drop not-ECT packets") does not expanse
   the acronym. Please do so and also add ECT definition in Section 2.

** Section 4: I didn't understand: "... and then remarks outer header of
   Not-ECT packet as ECT...". I think it's "remark" that confuses me as it
   suggests it has already been marked.

** Section 5: "IPFIX is preferred to use SCTP as transport." => "IPFIX uses
   preferably SCTP as transport protocol." if I understand correcty.

** Section 5: "the the egress acts as IPFIX exporter" => "then the... "

** Section 6. Congestion Management: the "e2e" acronym is not defined anywhere
   in this doc. Since this is the only place where it appears, I suggest to
   use "end-to-end" instead.

Cheers,

  Vincent

> Le 24 nov. 2016 à 00:53, Black, David <[email-masked]> a écrit :
>

---

From: "Weixinpeng (Jackie)" <[email-masked]>
Date: Thu, 1 Dec 2016 06:14:07 +0000

Dear Vincent,
         Thanks for your comments, please see my reply inline.

Best Regards,
-Xinpeng (Jackie)

From: Vincent Roca [mailto:[email-masked]]
Sent: Wednesday, November 30, 2016 12:14 AM
To: Weixinpeng (Jackie)
Cc: Vincent Roca; [email-masked]; Black, David
Subject: Re: [tsvwg] WGLC Conclusion: draft-ietf-tsvwg-tunnel-congestion-feedback-03

Dear authors,

GENERAL COMMENTS

1- In Section 4, it is assumed that "all routers in the tunnel support ECN."
   Isn't it too restrictive? What happens if this is not the case?
   And 2 paragraphs later, I read: "In case all interior routers support ECN".
   This is contradictory as no other possibility is considered.
[Wei] The reason why we assume “all routers in the tunnel support ECN” is that the tunnel congestion level only could be measure based
on the ratio of CE-marked packet when all routers support ECN. For the contradictory description, I will revise it later.

[Wei] Good question. The possible solution I can imagine now is providing the signaling message with higher transmission priority.

[Wei] I think the global RTT mentioned here would be something like an  empirical value, not the value measured through  the traffic by DP.

4- Does the proposed mechanism work in case of recursive uses of tunnels (tunnel
   in a tunnel)? It's not mentioned IMHO.
[Wei] The description on this will be added later.

5- Congestion management (actions taken by an AP) is a key aspect and Section 6
   provides a few hints. I have a few additional comments:
   - saying that "congestion management needs to avoid the impacts on e2e
     congestion control" is perhaps inappropriate. I'd rather say that it must
     not **compromise** e2e congestion control.
[Wei] This is a question need to be discussed further. Maybe one option is: when  congestion level is severe in tunnel, the network should not compromise e2e congestion control,
when the congestion level is below a certain threshold, the network  allow the end host to do congestion control before the network take action.
Besides, for different kind of flow (e.g. TCP flow, UDP flow), based on whether the flow itself will act on network congestion  the network can take different action.
   - It is said that "Congestion Policing is used in data center to limit...".
     This sentence suggests that CP is only meaningful in a data center. Is it
     the case, which would also mean that Circuit Breakers are the solution to
     use in Internet? Could you clarify in the I-D?
[Wei] It’s not like what you understanding here. I will clarify it later.

6- First of all, a direct link to [RFC 7011] is needed instead of refering to
   IPFIX. And instead of limiting this sentence to "feeding back congestion",
   it should apply to all the signaling traffic (to/from ingress, egress, and
   controller).
[Wei] Ok, thanks for this suggestion.

[Wei]  Ok, this will provided later.

[Wei] Ok, this will provided later.

TYPOS AND MINOR REMARKS

Cheers,

  Vincent

Le 24 nov. 2016 à 00:53, Black, David <[email-masked]<mailto:[email-masked]>> a écrit :

Thanks, --David

-----Original Message-----
From: Black, David
Sent: Wednesday, October 26, 2016 6:52 PM
To: [email-masked]<mailto:[email-masked]>
Cc: Black, David
Subject: WGLC: draft-ietf-tsvwg-tunnel-congestion-feedback-03

Comments should be sent to the [email-masked]<mailto:[email-masked]> list, although purely
editorial comments may be sent directly to the authors at
[email-masked]<mailto:[email-masked]>.  Please cc: the
WG chairs at [email-masked]<mailto:[email-masked]>  if you would like the chairs to track
such editorial comments as part of the WGLC process.

Thanks, --David
[email-masked]<mailto:[email-masked]>  <=== NEW ===