Subject: Re: [tsvwg] Tunnel Congestion Feedback: (non-)update to RFC6040

From: Bob Briscoe <[email-masked]>
Date: Sun, 2 Apr 2017 22:33:32 +0100

Weixinpeng, co-authors,  and David,

I had promised a review of the latest TCF draft, but failed to provide 
it so far.
I half read it through on Friday, and I will finish and write up my 
review this week.

Here' a quick summary of the main points so far:

*S.1. Intro: **Motivation
*The motivation should also include a heavy load from a much larger than 
expected number of responsive flows

*S.1 Intro: Scope**
*I think the draft should say that
* it gives no advice on how to select which tunnel endpoints should be 
used in order to manage traffic over a network criss-crossed by multiple 
tunnels.
* if a congested node is part of multiple tunnels, and it causes 
congestion feedback to multiple traffic management functions at the 
ingresses of all the tunnels, the draft gives no advice on how all the 
traffic management functions should respond.
* etc. (in my full review, I will provide a full list of subjects that 
are out of scope).

*S.3. Feedback Models: **
*The architecture diagrams and text need to include components that 
apply a consistent measurement policy (template records in IPFIX-speak) 
at both ingress and egress. This is a key part of interoperability, 
which is what the IETF should be all about. Template records are 
described later when the mechanism is specified, but they are not shown 
in the layout of the components in S.3.

*S.4. Congestion Level Measurement:**
*Measurement of ECN has been changed without thinking clearly whether it 
needed to change.

     a) Earlier drafts exploited the fact that the inner holds a 
"frozen" record of the ECN field at the ingress. So ECN measurements 
could be made solely at the egress (comparing the outer with the inner). 
In draft-04, ECN codepoints are measured at the ingress and at the 
egress, with the ingress sending regular measurement reports to the 
egress for comparison. However, it is always safer to measure ECN solely 
at the egress, even though a different mechanism has to measure drops 
within the tunnel (due to non-ECN nodes, or nodes that have switched off 
ECN due to overload, or due to causes of loss other than congestion, 
e.g. transmission errors).
      - It is always safer to solely measure ECN at the egress, because 
it is not vulnerable to ingress-egress measurement packets being dropped.
      - Solely measuring ECN at egress is no less precise, because only 
the proportion of the different ECN fields is needed, and that stays 
approximately the same even if there is drop, assuming drop is not 
biased towards certain ECN codepoints (no known mechanism does that).

     b) In draft-04 the faked ECT approach has been left unchanged at 
the ingress, but it is no longer used at the egress for detecting 
deferred drops (by looking for a Not-ECN inner and a CE outer). Again, 
there is nothing wrong with measuring ECN and detecting deferred drops 
solely at the egress, even if there are drops within the tunnel. To 
measure drops within the tunnel, it is necessary to compare ingress and 
egress bytes, but that does not make it wrong to still measure and 
compare ECN solely at the egress.

*S.4. Congestion Level Measurement: Units**
*Throughout measurement involves counting packets. However, it should 
count the bytes in packets with each type of marking, not just the 
number of packets. See RFC7141 section 2.3 
<https://tools.ietf.org/html/rfc7141#section-2.3>. Although the 
recommendation in that RFC is about what a /transport/ should measure 
when responding to congestion, logically it should apply in this case 
when a traffic management function is responding in place of a 
transport. RFC7141 also gives recommendations on which headers to count.

I shall add to this review later in the week.

One question for Weixinpeng: Have you implemented this mechanism?

Bob

On 01/11/16 01:49, Weixinpeng (Jackie) wrote:
>
> Hi Bob and David,
>
> Because the deadline has passed, I will update it as part of post-WGLC 
> update. Thanks.
>
> Regards,
>
> -Xinpeng (Jackie)
>
> *From:*Black, David [mailto:[email-masked]]
> *Sent:* Tuesday, November 01, 2016 12:14 AM
> *To:* Bob Briscoe; Weixinpeng (Jackie); 
> [email-masked]
> *Cc:* tsvwg IETF list; Black, David
> *Subject:* RE: Tunnel Congestion Feedback: (non-)update to RFC6040
>
> > I assume you will update the TCF draft today before the deadline.
>
> (As draft shepherd) That’s not really necessary - the TCF draft is in 
> WGLC, so this change can be done as part of a post-WGLC update.
>
> Thanks, --David
>
> *From:*Bob Briscoe [mailto:[email-masked]]
> *Sent:* Monday, October 31, 2016 11:22 AM
> *To:* Weixinpeng (Jackie); Black, David; 
> [email-masked] 
> <mailto:[email-masked]>
> *Cc:* tsvwg IETF list
> *Subject:* Re: Tunnel Congestion Feedback: (non-)update to RFC6040
>
> Xinpeng,
>
> I agree that TCF does not update RFC6040 (as I said).
>
> But David is right that the TCF draft itself needs to be updated to 
> say that it modifies the behaviour of a RFC6040 egress so as to issue 
> or log any warnings when it receives an ECT(0) or CE outer with a 
> Not-ECT inner.
>
> I assume you will update the TCF draft today before the deadline.
>
> Cheers
>
> Bob
>
> On 31/10/16 02:38, Weixinpeng (Jackie) wrote:
>
>     Hi David, Bob,
>
>     I agree that the behavior defined in RFC6040 doesn’t to be updated
>     due to the TCF draft, because what’s defined in RFC 6040 has a
>     broader usage and not just for TCF.
>
>     I think David has provided a good suggestion below. Thanks.
>
>     Regards,
>
>     -Xinpeng (Jackie)
>
>     *From:*tsvwg [mailto:[email-masked]] *On Behalf Of *Black,
>     David
>     *Sent:* Saturday, October 29, 2016 10:28 AM
>     *To:* Bob Briscoe;
>     [email-masked]
>     <mailto:[email-masked]>
>     *Cc:* tsvwg IETF list
>     *Subject:* Re: [tsvwg] Tunnel Congestion Feedback: (non-)update to
>     RFC6040
>
>     Bob,
>
>     I think you’ve found something.  The TCF draft currently uses RFC
>     6040 behavior by reference:
>
>        when decapsulating packets at egress,
>
>        RFC6040 defined decapsulation behavior is used, and according to
>
>        RFC6040, the packets marked as CE|N-ECT will be dropped by egress.
>
>     I think this needs to be slightly revised to say that CE|N-ECT is
>     dropped without any other
>
>     action, and all other behavior is as per RFC 6040, as CE|N-ECT is
>     expected at a TCF egress and
>
>     hence the logging and alarm behavior for that combination in RFC
>     6040 is inappropriate.
>
>     Once that’s done, I think your reasoning in the following sentence
>     becomes correct:
>
>     If the network is not valid, these combination could become
>     visible to a 6040-decap,
>
>     which would then (correctly) log warnings that it was seeing
>     invalid codepoint combinations.
>
>     Thanks, --David (as draft shepherd)
>
>     *From:*Bob Briscoe [mailto:[email-masked]]
>     *Sent:* Thursday, October 27, 2016 4:39 PM
>     *To:* [email-masked]
>     <mailto:[email-masked]>;
>     Black, David
>     *Cc:* tsvwg IETF list
>     *Subject:* Tunnel Congestion Feedback: (non-)update to RFC6040
>
>     Tunnel congestion feedback authors, David as shepherd,
>
>     FYI. No action needed.
>
>     This email documents an argument with myself. After a few seconds,
>     I agreed with myself that one of me was wrong. I thought I had
>     better raise this on the list, to check that others agree with
>     both of me. If you wish to argue with just one of me, please bear
>     in mind that we both have the same email address (and anyway, we
>     both agree so an argument with one is an argument with all).
>     _________________________
>
>     Figure 4 of RFC6040 specifies which ECN codepoint a decap must
>     forward, for each combination of arriving inner (row) and outer
>     (column).
>
>                
>     +---------+------------------------------------------------+
>
>                 |Arriving |            Arriving Outer Header        
>           |
>
>                 |   Inner
>     +---------+------------+------------+------------+
>
>                 |  Header | Not-ECT | ECT(0)     | ECT(1)     |    
>     CE     |
>
>                
>     +---------+---------+------------+------------+------------+
>
>                 | Not-ECT | Not-ECT |Not-ECT(!!!)|Not-ECT(!!!)|
>     <drop>(!!!)|
>
>                 |  ECT(0) |  ECT(0) | ECT(0)     | ECT(1)     |    
>     CE     |
>
>                 |  ECT(1) |  ECT(1) | ECT(1) (!) | ECT(1)     |    
>     CE     |
>
>                 |    CE   |      CE |     CE     |     CE(!!!)|    
>     CE     |
>
>                
>     +---------+---------+------------+------------+------------+
>
>     RFC6040 goes on to say:
>
>                           [...] These currently unused (CU)
>     combinations are
>
>           indicated in Figure 4 by '(!!!)' or '(!)', where '(!!!)'
>     means the
>
>           combination is CU and always potentially dangerous, while '(!)'
>
>           means it is CU and possibly dangerous.  In these cases,
>
>           particularly the more dangerous ones, the decapsulator
>     SHOULD log
>
>           the event and MAY also raise an alarm.
>
>     TCF uses the following combinations of inner : outer
>         Not-ECT : ECT(0)
>         Not-ECT : CE
>
>     For a moment, I thought we would have to update RFC6040 to say
>     that TCF makes these combinations no longer CU, so they are now
>     valid, not dangerous, and they no longer need to be logged. This
>     would have meant that Tunnel Congestion Feedback (TCF)
>     (informational) would have to update RFC6040 (proposed standard).
>
>     Here's why I was wrong. These two combinations are certainly valid
>     at a TCF decap. But there is still no way that, in a valid
>     network, they would arrive at any RFC6040 (non-TCF) decap.
>
>     One might think that this combination could arrive at a 6040-decap
>     if there was a tunnel inside a tunnel, as follows
>
>       TCF-encap                 ---> 6040-decap ---> TCF-decap
>
>     However, that's not a valid configuration. The following
>     tunnel-in-tunnel would be valid:
>
>       TCF-encap ---> 6040-encap ---> 6040-decap ---> TCF-decap
>
>     ...then, the above two "CU" combinations would be buried within a
>     double inner, so only the TCF decap would see them, not the
>     6040-decap.
>
>     So, TCF does not need to update RFC6040. Because, in a valid
>     network, TCF will be hidden from any 6040-decap. If the network is
>     not valid, these combination could become visible to a 6040-decap,
>     which would then (correctly) log warnings that it was seeing
>     invalid codepoint combinations.
>
>     I just thought you'd like to know... no action required.
>
>
>
>     Bob
>
>     -- 
>
>     -- 
>
>     ________________________________________________________________
>
>     Bob Briscoe http://bobbriscoe.net/
>
> -- 
> ________________________________________________________________
> Bob Briscoe http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Weixinpeng (Jackie)" <[email-masked]>
Date: Wed, 5 Apr 2017 02:45:52 +0000

Hi Bob,
Thanks for your comments, and please see inline.
Regards,
-Xinpeng

From: Bob Briscoe [mailto:[email-masked]]
Sent: Monday, April 03, 2017 5:34 AM
To: Weixinpeng (Jackie); Black, David; [email-masked]
Cc: tsvwg IETF list
Subject: Re: Tunnel Congestion Feedback: (non-)update to RFC6040

I had promised a review of the latest TCF draft, but failed to provide it so far.
I half read it through on Friday, and I will finish and write up my review this week.

S.1. Intro: Motivation
The motivation should also include a heavy load from a much larger than expected number of responsive flows
[Wei] Ok, I will add it.

S.1 Intro: Scope
I think the draft should say that
* it gives no advice on how to select which tunnel endpoints should be used in order to manage traffic over a network criss-crossed by multiple tunnels.
* if a congested node is part of multiple tunnels, and it causes congestion feedback to multiple traffic management functions at the ingresses of all the tunnels, the draft gives no advice on how all the traffic management functions should respond.
* etc. (in my full review, I will provide a full list of subjects that are out of scope).
[Wei] Ok.

S.3. Feedback Models:
The architecture diagrams and text need to include components that apply a consistent measurement policy (template records in IPFIX-speak) at both ingress and egress. This is a key part of interoperability, which is what the IETF should be all about. Template records are described later when the mechanism is specified, but they are not shown in the layout of the components in S.3.
[Wei] I will provide a description of Template record in this section. But I am wondering whether a detailed description of Template Record as section 6.1 is needed here or not?

S.4. Congestion Level Measurement:
Measurement of ECN has been changed without thinking clearly whether it needed to change.

    a) Earlier drafts exploited the fact that the inner holds a "frozen" record of the ECN field at the ingress. So ECN measurements could be made solely at the egress (comparing the outer with the inner). In draft-04, ECN codepoints are measured at the ingress and at the egress, with the ingress sending regular measurement reports to the egress for comparison. However, it is always safer to measure ECN solely at the egress, even though a different mechanism has to measure drops within the tunnel (due to non-ECN nodes, or nodes that have switched off ECN due to overload, or due to causes of loss other than congestion, e.g. transmission errors).
     - It is always safer to solely measure ECN at the egress, because it is not vulnerable to ingress-egress measurement packets being dropped.
     - Solely measuring ECN at egress is no less precise, because only the proportion of the different ECN fields is needed, and that stays approximately the same even if there is drop, assuming drop is not biased towards certain ECN codepoints (no known mechanism does that).

    b) In draft-04 the faked ECT approach has been left unchanged at the ingress, but it is no longer used at the egress for detecting deferred drops (by looking for a Not-ECN inner and a CE outer). Again, there is nothing wrong with measuring ECN and detecting deferred drops solely at the egress, even if there are drops within the tunnel. To measure drops within the tunnel, it is necessary to compare ingress and egress bytes, but that does not make it wrong to still measure and compare ECN solely at the egress.
[Wei] I agree that solely measuring ECN at egress is enough to get the proportion of different ECN fields, but the reason why ECN codepoints are measured both at ingress and egress is to measure the packets dropped within the tunnel by comparing counts at ingress and egress.

S.4. Congestion Level Measurement: Units
Throughout measurement involves counting packets. However, it should count the bytes in packets with each type of marking, not just the number of packets. See RFC7141 section 2.3<https://tools.ietf.org/html/rfc7141#section-2.3>. Although the recommendation in that RFC is about what a /transport/ should measure when responding to congestion, logically it should apply in this case when a traffic management function is responding in place of a transport. RFC7141 also gives recommendations on which headers to count.
[Wei] Ok, it’s a good suggestion.

One question for Weixinpeng: Have you implemented this mechanism?
[Wei] Currently, we only implement this mechanism in a simulation environment.

Bob

On 01/11/16 01:49, Weixinpeng (Jackie) wrote:
Hi Bob and David,
Because the deadline has passed, I will update it as part of post-WGLC update. Thanks.

Regards,
-Xinpeng (Jackie)

From: Black, David [mailto:[email-masked]]
Sent: Tuesday, November 01, 2016 12:14 AM
To: Bob Briscoe; Weixinpeng (Jackie); [email-masked]<mailto:[email-masked]>
Cc: tsvwg IETF list; Black, David
Subject: RE: Tunnel Congestion Feedback: (non-)update to RFC6040

> I assume you will update the TCF draft today before the deadline.

(As draft shepherd) That’s not really necessary - the TCF draft is in WGLC, so this change can be done as part of a post-WGLC update.

Thanks, --David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Monday, October 31, 2016 11:22 AM
To: Weixinpeng (Jackie); Black, David; [email-masked]<mailto:[email-masked]>
Cc: tsvwg IETF list
Subject: Re: Tunnel Congestion Feedback: (non-)update to RFC6040

Xinpeng,

But David is right that the TCF draft itself needs to be updated to say that it modifies the behaviour of a RFC6040 egress so as to issue or log any warnings when it receives an ECT(0) or CE outer with a Not-ECT inner.

Cheers

Bob
On 31/10/16 02:38, Weixinpeng (Jackie) wrote:
Hi David, Bob,
I agree that the behavior defined in RFC6040 doesn’t to be updated due to the TCF draft, because what’s defined in RFC 6040 has a broader usage and not just for TCF.
I think David has provided a good suggestion below. Thanks.

Regards,
-Xinpeng (Jackie)

From: tsvwg [mailto:[email-masked]] On Behalf Of Black, David
Sent: Saturday, October 29, 2016 10:28 AM
To: Bob Briscoe; [email-masked]<mailto:[email-masked]>
Cc: tsvwg IETF list
Subject: Re: [tsvwg] Tunnel Congestion Feedback: (non-)update to RFC6040

Bob,

I think you’ve found something.  The TCF draft currently uses RFC 6040 behavior by reference:

I think this needs to be slightly revised to say that CE|N-ECT is dropped without any other
action, and all other behavior is as per RFC 6040, as CE|N-ECT is expected at a TCF egress and
hence the logging and alarm behavior for that combination in RFC 6040 is inappropriate.

Once that’s done, I think your reasoning in the following sentence becomes correct:

If the network is not valid, these combination could become visible to a 6040-decap,
which would then (correctly) log warnings that it was seeing invalid codepoint combinations.

From: Bob Briscoe [mailto:[email-masked]]
Sent: Thursday, October 27, 2016 4:39 PM
To: [email-masked]<mailto:[email-masked]>; Black, David
Cc: tsvwg IETF list
Subject: Tunnel Congestion Feedback: (non-)update to RFC6040

FYI. No action needed.

This email documents an argument with myself. After a few seconds, I agreed with myself that one of me was wrong. I thought I had better raise this on the list, to check that others agree with both of me. If you wish to argue with just one of me, please bear in mind that we both have the same email address (and anyway, we both agree so an argument with one is an argument with all).
_________________________

Figure 4 of RFC6040 specifies which ECN codepoint a decap must forward, for each combination of arriving inner (row) and outer (column).

            |Arriving |            Arriving Outer Header               |

            |   Inner +---------+------------+------------+------------+

            |  Header | Not-ECT | ECT(0)     | ECT(1)     |     CE     |

            | Not-ECT | Not-ECT |Not-ECT(!!!)|Not-ECT(!!!)| <drop>(!!!)|

            |  ECT(0) |  ECT(0) | ECT(0)     | ECT(1)     |     CE     |

            |  ECT(1) |  ECT(1) | ECT(1) (!) | ECT(1)     |     CE     |

            |    CE   |      CE |     CE     |     CE(!!!)|     CE     |

                      [...] These currently unused (CU) combinations are

      indicated in Figure 4 by '(!!!)' or '(!)', where '(!!!)' means the

      particularly the more dangerous ones, the decapsulator SHOULD log

For a moment, I thought we would have to update RFC6040 to say that TCF makes these combinations no longer CU, so they are now valid, not dangerous, and they no longer need to be logged. This would have meant that Tunnel Congestion Feedback (TCF) (informational) would have to update RFC6040 (proposed standard).

Here's why I was wrong. These two combinations are certainly valid at a TCF decap. But there is still no way that, in a valid network, they would arrive at any RFC6040 (non-TCF) decap.

One might think that this combination could arrive at a 6040-decap if there was a tunnel inside a tunnel, as follows

However, that's not a valid configuration. The following tunnel-in-tunnel would be valid:

...then, the above two "CU" combinations would be buried within a double inner, so only the TCF decap would see them, not the 6040-decap.

So, TCF does not need to update RFC6040. Because, in a valid network, TCF will be hidden from any 6040-decap. If the network is not valid, these combination could become visible to a 6040-decap, which would then (correctly) log warnings that it was seeing invalid codepoint combinations.

Bob

--

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 Apr 2017 15:47:38 +0100

Weixinpeng,

On 05/04/17 03:45, Weixinpeng (Jackie) wrote:
> *Sent:* Monday, April 03, 2017 5:34 AM
> *S.1. Intro: Motivation
> *The motivation should also include a heavy load from a much larger 
> than expected number of responsive flows
>
> */[Wei] Ok, I will add it./*
>
> *S.1 Intro: Scope
> used in order to manage traffic over a network criss-crossed by 
> multiple tunnels.
> */[Wei] Ok./*
> *S.3. Feedback Models:
> apply a consistent measurement policy (template records in 
> IPFIX-speak) at both ingress and egress. This is a key part of 
> interoperability, which is what the IETF should be all about. Template 
> records are described later when the mechanism is specified, but they 
> are not shown in the layout of the components in S.3.
>
> */[Wei] I will provide a description of Template record in this 
> section. But I am wondering whether a detailed description of Template 
> Record as section 6.1 is needed here or not?/*
>
I suggest that, at this stage in the doc, it would be best not to be 
specific to one technology (IPFIX) but instead to talk more generally 
about setting and storing measurement policy (at the decision point and 
the egress meter).

BTW, did you invent the terminology decision point (DP) and action point 
(AP)? Because there is already IETF terminology for these:
* Policy Decision Point (PDP)
* Policy Enforcement Point (PEP).

See COPS (Common Open Policy Service) Protocol [RFC2748] or terminology 
for policy-based management [RFC3198]
>
> *//*
>
>
>
> *S.4. Congestion Level Measurement:
> *Measurement of ECN has been changed without thinking clearly whether 
> it needed to change.
>
> could be made solely at the egress (comparing the outer with the 
> inner). In draft-04, ECN codepoints are measured at the ingress and at 
> the egress, with the ingress sending regular measurement reports to 
> the egress for comparison. However, it is always safer to measure ECN 
> solely at the egress, even though a different mechanism has to measure 
> drops within the tunnel (due to non-ECN nodes, or nodes that have 
> switched off ECN due to overload, or due to causes of loss other than 
> congestion, e.g. transmission errors).
> measure drops within the tunnel, it is necessary to compare ingress 
> and egress bytes, but that does not make it wrong to still measure and 
> */[Wei] I agree that solely measuring ECN at egress is enough to get 
> the proportion of different ECN fields, but the reason why ECN 
> codepoints are measured both at ingress and egress is to measure the 
> packets dropped within the tunnel by comparing counts at ingress and 
> egress. /*
>
I am not saying you should not also measure drop by measuring total 
bytes at ingress and egress.

However, I am saying it is less robust to have to measure at both 
ingress and egress, because it involves sending synch / measurement 
messages from ingress to egress, which can be lost/delayed.
So, I would recommend the more robust egress-only method for ECN, and 
the ingress-egress method only for total drop.

The only extra thing you learn by the currently proposed approach is how 
many of each ECN codepoint were dropped. If you think that's important, 
argue for it. I don't see any need for it tho.

If you insist on keeping the ingress-egress approach, why do you need 
the faked ECN text?

> *//*
>
>
>
> *S.4. Congestion Level Measurement: Units
> */[Wei] Ok, it’s a good suggestion./*
>
>
>
>
> */[Wei] Currently, we only implement this mechanism in a simulation 
> environment./*
>
I suspected as much, because the lack of detail in some places implies 
it is a bit theoretical.
I would encourage you to implement.

Bob
>     (As draft shepherd) That’s not really necessary - the TCF draft is
>     in WGLC, so this change can be done as part of a post-WGLC update.
>
>     Thanks, --David
>
>     But David is right that the TCF draft itself needs to be updated
>     to say that it modifies the behaviour of a RFC6040 egress so as to
>     issue or log any warnings when it receives an ECT(0) or CE outer
>     with a Not-ECT inner.
>
>         I agree that the behavior defined in RFC6040 doesn’t to be
>         updated due to the TCF draft, because what’s defined in RFC
>         6040 has a broader usage and not just for TCF.
>
>         *From:*tsvwg [mailto:[email-masked]] *On Behalf Of
>         *Black, David
>         *Subject:* Re: [tsvwg] Tunnel Congestion Feedback:
>         (non-)update to RFC6040
>
>         Bob,
>
>         *To:*
>         This email documents an argument with myself. After a few
>         seconds, I agreed with myself that one of me was wrong. I
>         thought I had better raise this on the list, to check that
>         others agree with both of me. If you wish to argue with just
>         one of me, please bear in mind that we both have the same
>         email address (and anyway, we both agree so an argument with
>         one is an argument with all).
>         forward, for each combination of arriving inner (row) and
>         outer (column).
>
>                    
>         that TCF makes these combinations no longer CU, so they are
>         now valid, not dangerous, and they no longer need to be
>         logged. This would have meant that Tunnel Congestion Feedback
>         (TCF) (informational) would have to update RFC6040 (proposed
>         standard).
>
>         Here's why I was wrong. These two combinations are certainly
>         valid at a TCF decap. But there is still no way that, in a
>         valid network, they would arrive at any RFC6040 (non-TCF) decap.
>
>         One might think that this combination could arrive at a
>         6040-decap if there was a tunnel inside a tunnel, as follows
>
>         ...then, the above two "CU" combinations would be buried
>         within a double inner, so only the TCF decap would see them,
>         not the 6040-decap.
>
>         network, TCF will be hidden from any 6040-decap. If the
>         network is not valid, these combination could become visible
>         to a 6040-decap, which would then (correctly) log warnings
>         that it was seeing invalid codepoint combinations.
>

---

From: "Black, David" <[email-masked]>
Date: Sat, 8 Apr 2017 01:20:02 +0000

I’m among those responsible for the ingress-egress measurement addition, so weighing in as draft shepherd:

> However, I am saying it is less robust to have to measure at both ingress and egress, because it involves sending synch / measurement
> messages from ingress to egress, which can be lost/delayed.  So, I would recommend the more robust egress-only method for ECN,
> and the ingress-egress method only for total drop.

That seems fine, however the total drop info does need to be reported in addition to ECN info.

> The only extra thing you learn by the currently proposed approach is how many of each ECN codepoint were dropped. If you think that's important, argue for it. I don't see any need for it tho.

I would agree.

> If you insist on keeping the ingress-egress approach, why do you need the faked ECN text?

If one is trying to measure congestion in the tunnel, drops have to be considered – even if all nodes in the tunnel support ECN, an overloaded ECN node will drop packets.
The reason for faked-ECN is buried in Bob’s “more robust” comment about egress-only ECN measurement above – use of ECN should result in better measurements by comparison to non-use.

Thanks, --David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, April 7, 2017 10:48 AM
To: Weixinpeng (Jackie) <[email-masked]>; Black, David <[email-masked]>; [email-masked]
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: Tunnel Congestion Feedback: (non-)update to RFC6040

Weixinpeng,

S.1. Intro: Motivation

S.1 Intro: Scope

S.3. Feedback Models:
The architecture diagrams and text need to include components that apply a consistent measurement policy (template records in IPFIX-speak) at both ingress and egress. This is a key part of interoperability, which is what the IETF should be all about. Template records are described later when the mechanism is specified, but they are not shown in the layout of the components in S.3.
[Wei] I will provide a description of Template record in this section. But I am wondering whether a detailed description of Template Record as section 6.1 is needed here or not?
I suggest that, at this stage in the doc, it would be best not to be specific to one technology (IPFIX) but instead to talk more generally about setting and storing measurement policy (at the decision point and the egress meter).

BTW, did you invent the terminology decision point (DP) and action point (AP)? Because there is already IETF terminology for these:
* Policy Decision Point (PDP)
* Policy Enforcement Point (PEP).

See COPS (Common Open Policy Service) Protocol [RFC2748] or terminology for policy-based management [RFC3198]

    b) In draft-04 the faked ECT approach has been left unchanged at the ingress, but it is no longer used at the egress for detecting deferred drops (by looking for a Not-ECN inner and a CE outer). Again, there is nothing wrong with measuring ECN and detecting deferred drops solely at the egress, even if there are drops within the tunnel. To measure drops within the tunnel, it is necessary to compare ingress and egress bytes, but that does not make it wrong to still measure and compare ECN solely at the egress.
[Wei] I agree that solely measuring ECN at egress is enough to get the proportion of different ECN fields, but the reason why ECN codepoints are measured both at ingress and egress is to measure the packets dropped within the tunnel by comparing counts at ingress and egress.
I am not saying you should not also measure drop by measuring total bytes at ingress and egress.

However, I am saying it is less robust to have to measure at both ingress and egress, because it involves sending synch / measurement messages from ingress to egress, which can be lost/delayed.
So, I would recommend the more robust egress-only method for ECN, and the ingress-egress method only for total drop.

The only extra thing you learn by the currently proposed approach is how many of each ECN codepoint were dropped. If you think that's important, argue for it. I don't see any need for it tho.

If you insist on keeping the ingress-egress approach, why do you need the faked ECN text?

One question for Weixinpeng: Have you implemented this mechanism?
[Wei] Currently, we only implement this mechanism in a simulation environment.
I suspected as much, because the lack of detail in some places implies it is a bit theoretical.
I would encourage you to implement.

Bob

Bob

Regards,
-Xinpeng (Jackie)

Thanks, --David

Xinpeng,

Cheers

Bob

Regards,
-Xinpeng (Jackie)

Bob,

FYI. No action needed.

Bob

--

--

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 8 Apr 2017 10:15:42 +0100

David,

Thx for all your points - all understood and I agreed with them all.

I think now that Weixinpeng, you and I all agree that the draft should 
specify ingress-egress comparison for drop and the egress-only approach 
for ECN. So then the faked ECN approach can stay, because it shifts 
some/most drops to the more robust egress-only measurement location. But 
I suggest the faked ECN text is moved up a paragraph, because the 
paragraph before it doesn't make sense unless the readers already knows 
about faked ECN.

I will still send a full review, possibly this weekend, but early next 
week is looking more likely now.

Bob

On 08/04/17 02:20, Black, David wrote:
>
> I’m among those responsible for the ingress-egress measurement 
> addition, so weighing in as draft shepherd:
>
> > messages from ingress to egress, which can be lost/delayed.  So, I 
> would recommend the more robust egress-only method for ECN,
>
> > The only extra thing you learn by the currently proposed approach is 
> how many of each ECN codepoint were dropped. If you think that's 
> important, argue for it. I don't see any need for it tho.
>
> I would agree.
>
> If one is trying to measure congestion in the tunnel, drops have to be 
> considered – even if all nodes in the tunnel support ECN, an 
> overloaded ECN node will drop packets.
>
> The reason for faked-ECN is buried in Bob’s “more robust” comment 
> about egress-only ECN measurement above – use of ECN should result in 
> better measurements by comparison to non-use.
>
> Thanks, --David
>
> *Sent:* Friday, April 7, 2017 10:48 AM
> *To:* Weixinpeng (Jackie) <[email-masked]>; Black, David 
> <[email-masked]>; 
> *Cc:* tsvwg IETF list <[email-masked]>
>     *The motivation should also include a heavy load from a much
>     larger than expected number of responsive flows
>
>     * it gives no advice on how to select which tunnel endpoints
>     should be used in order to manage traffic over a network
>     criss-crossed by multiple tunnels.
>     congestion feedback to multiple traffic management functions at
>     the ingresses of all the tunnels, the draft gives no advice on how
>     all the traffic management functions should respond.
>     *The architecture diagrams and text need to include components
>     that apply a consistent measurement policy (template records in
>     interoperability, which is what the IETF should be all about.
>     Template records are described later when the mechanism is
>     specified, but they are not shown in the layout of the components
>     in S.3.
>
>     section. But I am wondering whether a detailed description of
>     Template Record as section 6.1 is needed here or not?/*
>
> about setting and storing measurement policy (at the decision point 
> and the egress meter).
>
> BTW, did you invent the terminology decision point (DP) and action 
> point (AP)? Because there is already IETF terminology for these:
> See COPS (Common Open Policy Service) Protocol [RFC2748] or 
> terminology for policy-based management [RFC3198]
>
>
>
>     *Measurement of ECN has been changed without thinking clearly
>     whether it needed to change.
>
>     "frozen" record of the ECN field at the ingress. So ECN
>     measurements could be made solely at the egress (comparing the
>     outer with the inner). In draft-04, ECN codepoints are measured at
>     the ingress and at the egress, with the ingress sending regular
>     measurement reports to the egress for comparison. However, it is
>     always safer to measure ECN solely at the egress, even though a
>     different mechanism has to measure drops within the tunnel (due to
>     non-ECN nodes, or nodes that have switched off ECN due to
>     overload, or due to causes of loss other than congestion, e.g.
>     transmission errors).
>          - It is always safer to solely measure ECN at the egress,
>     because it is not vulnerable to ingress-egress measurement packets
>     being dropped.
>          - Solely measuring ECN at egress is no less precise, because
>     only the proportion of the different ECN fields is needed, and
>     that stays approximately the same even if there is drop, assuming
>     drop is not biased towards certain ECN codepoints (no known
>     mechanism does that).
>
>         b) In draft-04 the faked ECT approach has been left unchanged
>     at the ingress, but it is no longer used at the egress for
>     detecting deferred drops (by looking for a Not-ECN inner and a CE
>     outer). Again, there is nothing wrong with measuring ECN and
>     detecting deferred drops solely at the egress, even if there are
>     drops within the tunnel. To measure drops within the tunnel, it is
>     necessary to compare ingress and egress bytes, but that does not
>     make it wrong to still measure and compare ECN solely at the egress.
>
>     */[Wei] I agree that solely measuring ECN at egress is enough to
>     get the proportion of different ECN fields, but the reason why ECN
>     codepoints are measured both at ingress and egress is to measure
>     the packets dropped within the tunnel by comparing counts at
>     ingress and egress. /*
>
> The only extra thing you learn by the currently proposed approach is 
> how many of each ECN codepoint were dropped. If you think that's 
> important, argue for it. I don't see any need for it tho.
>
>     *Throughout measurement involves counting packets. However, it
>     should count the bytes in packets with each type of marking, not
>     just the number of packets. See RFC7141 section 2.3
>     recommendation in that RFC is about what a /transport/ should
>     measure when responding to congestion, logically it should apply
>     in this case when a traffic management function is responding in
>     place of a transport. RFC7141 also gives recommendations on which
>     headers to count.
>
>     */[Wei] Currently, we only implement this mechanism in a
>     simulation environment./*
>
>         (As draft shepherd) That’s not really necessary - the TCF
>         draft is in WGLC, so this change can be done as part of a
>         post-WGLC update.
>
>         Thanks, --David
>
>         But David is right that the TCF draft itself needs to be
>         updated to say that it modifies the behaviour of a RFC6040
>         egress so as to issue or log any warnings when it receives an
>         ECT(0) or CE outer with a Not-ECT inner.
>
>             updated due to the TCF draft, because what’s defined in
>             RFC 6040 has a broader usage and not just for TCF.
>
>             others agree with both of me. If you wish to argue with
>             just one of me, please bear in mind that we both have the
>             same email address (and anyway, we both agree so an
>             argument with one is an argument with all).
>             Figure 4 of RFC6040 specifies which ECN codepoint a decap
>             must forward, for each combination of arriving inner (row)
>             and outer (column).
>
>                        
>             For a moment, I thought we would have to update RFC6040 to
>             say that TCF makes these combinations no longer CU, so
>             they are now valid, not dangerous, and they no longer need
>             to be logged. This would have meant that Tunnel Congestion
>             Feedback (TCF) (informational) would have to update
>             RFC6040 (proposed standard).
>
>             Here's why I was wrong. These two combinations are
>             certainly valid at a TCF decap. But there is still no way
>             that, in a valid network, they would arrive at any RFC6040
>             (non-TCF) decap.
>
>             within a double inner, so only the TCF decap would see
>             them, not the 6040-decap.
>
>             So, TCF does not need to update RFC6040. Because, in a
>             valid network, TCF will be hidden from any 6040-decap. If
>             the network is not valid, these combination could become
>             visible to a 6040-decap, which would then (correctly) log
>             warnings that it was seeing invalid codepoint combinations.
>
>     Bob Briscoehttp://bobbriscoe.net/
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/

---

From: "Weixinpeng (Jackie)" <[email-masked]>
Date: Mon, 10 Apr 2017 02:03:28 +0000

Thanks David and Bob,
I think we have reach a consensus that both faked ECN + egress–only approach and ingress-egress comparison approach should be use, the first approach is for  ECN measurement and the second approach is for packet loss count.

-Xinpeng

From: Bob Briscoe [mailto:[email-masked]]
Sent: Saturday, April 08, 2017 5:16 PM
To: Black, David; Weixinpeng (Jackie); [email-masked]
Cc: tsvwg IETF list
Subject: Re: Tunnel Congestion Feedback: (non-)update to RFC6040

David,

I think now that Weixinpeng, you and I all agree that the draft should specify ingress-egress comparison for drop and the egress-only approach for ECN. So then the faked ECN approach can stay, because it shifts some/most drops to the more robust egress-only measurement location. But I suggest the faked ECN text is moved up a paragraph, because the paragraph before it doesn't make sense unless the readers already knows about faked ECN.

I will still send a full review, possibly this weekend, but early next week is looking more likely now.

Bob

I would agree.

Thanks, --David

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, April 7, 2017 10:48 AM
To: Weixinpeng (Jackie) <[email-masked]><mailto:[email-masked]>; Black, David <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: tsvwg IETF list <[email-masked]><mailto:[email-masked]>
Subject: Re: Tunnel Congestion Feedback: (non-)update to RFC6040

Weixinpeng,

S.1. Intro: Motivation

S.1 Intro: Scope

S.3. Feedback Models:

Bob

Bob

Regards,
-Xinpeng (Jackie)

Thanks, --David

Xinpeng,

Cheers

Bob

Regards,
-Xinpeng (Jackie)

Bob,

FYI. No action needed.

Bob

--

--

--

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 11 Apr 2017 10:38:43 +0100

Yup

On 10/04/17 03:03, Weixinpeng (Jackie) wrote:
>
> Thanks David and Bob,
>
> I think we have reach a consensus that both faked ECN + egress–only 
> approach and ingress-egress comparison approach should be use, the 
> first approach is for  ECN measurement and the second approach is for 
> packet loss count.
>
> -Xinpeng
>
> *Sent:* Saturday, April 08, 2017 5:16 PM
> *To:* Black, David; Weixinpeng (Jackie); 
> specify ingress-egress comparison for drop and the egress-only 
> approach for ECN. So then the faked ECN approach can stay, because it 
> shifts some/most drops to the more robust egress-only measurement 
> location. But I suggest the faked ECN text is moved up a paragraph, 
> because the paragraph before it doesn't make sense unless the readers 
> already knows about faked ECN.
>
>     > However, I am saying it is less robust to have to measure at both ingress and
>     egress, because it involves sending synch / measurement
>
>     > messages from ingress to egress, which can be lost/delayed.  So, I would
>     recommend the more robust egress-only method for ECN,
>
>     > The only extra thing you learn by the currently proposed
>     approach is how many of each ECN codepoint were dropped. If you
>     think that's important, argue for it. I don't see any need for it tho.
>
>     I would agree.
>
>     If one is trying to measure congestion in the tunnel, drops have
>     to be considered – even if all nodes in the tunnel support ECN, an
>     about egress-only ECN measurement above – use of ECN should result
>     in better measurements by comparison to non-use.
>
>     Thanks, --David
>
>     *To:* Weixinpeng (Jackie) <[email-masked]>
>     <mailto:[email-masked]>; Black, David <[email-masked]>
>     <mailto:[email-masked]>;
>     *Cc:* tsvwg IETF list <[email-masked]> <mailto:[email-masked]>
>         * if a congested node is part of multiple tunnels, and it
>         causes congestion feedback to multiple traffic management
>         functions at the ingresses of all the tunnels, the draft gives
>         no advice on how all the traffic management functions should
>         respond.
>         that apply a consistent measurement policy (template records
>         in IPFIX-speak) at both ingress and egress. This is a key part
>         of interoperability, which is what the IETF should be all
>         about. Template records are described later when the mechanism
>         is specified, but they are not shown in the layout of the
>         components in S.3.
>
>         */[Wei] I will provide a description of Template record in
>         this section. But I am wondering whether a detailed
>         description of Template Record as section 6.1 is needed here
>         or not?/*
>
>     I suggest that, at this stage in the doc, it would be best not to
>     be specific to one technology (IPFIX) but instead to talk more
>     generally about setting and storing measurement policy (at the
>     decision point and the egress meter).
>
>             a) Earlier drafts exploited the fact that the inner holds
>         a "frozen" record of the ECN field at the ingress. So ECN
>         outer with the inner). In draft-04, ECN codepoints are
>         measured at the ingress and at the egress, with the ingress
>         sending regular measurement reports to the egress for
>         comparison. However, it is always safer to measure ECN solely
>         at the egress, even though a different mechanism has to
>         measure drops within the tunnel (due to non-ECN nodes, or
>         nodes that have switched off ECN due to overload, or due to
>         causes of loss other than congestion, e.g. transmission errors).
>         because it is not vulnerable to ingress-egress measurement
>         packets being dropped.
>              - Solely measuring ECN at egress is no less precise,
>         because only the proportion of the different ECN fields is
>         needed, and that stays approximately the same even if there is
>         drop, assuming drop is not biased towards certain ECN
>         codepoints (no known mechanism does that).
>
>             b) In draft-04 the faked ECT approach has been left
>         unchanged at the ingress, but it is no longer used at the
>         egress for detecting deferred drops (by looking for a Not-ECN
>         inner and a CE outer). Again, there is nothing wrong with
>         measuring ECN and detecting deferred drops solely at the
>         egress, even if there are drops within the tunnel. To measure
>         drops within the tunnel, it is necessary to compare ingress
>         and egress bytes, but that does not make it wrong to still
>         measure and compare ECN solely at the egress.
>
>         */[Wei] I agree that solely measuring ECN at egress is enough
>         to get the proportion of different ECN fields, but the reason
>         why ECN codepoints are measured both at ingress and egress is
>         to measure the packets dropped within the tunnel by comparing
>         counts at ingress and egress. /*
>
>     ingress and egress, because it involves sending synch /
>     measurement messages from ingress to egress, which can be
>     lost/delayed.
>     So, I would recommend the more robust egress-only method for ECN,
>     The only extra thing you learn by the currently proposed approach
>     is how many of each ECN codepoint were dropped. If you think
>     that's important, argue for it. I don't see any need for it tho.
>
>         should count the bytes in packets with each type of marking,
>         not just the number of packets. See RFC7141 section 2.3
>         <https://tools.ietf.org/html/rfc7141#section-2.3>. Although
>         the recommendation in that RFC is about what a /transport/
>         should measure when responding to congestion, logically it
>         should apply in this case when a traffic management function
>         is responding in place of a transport. RFC7141 also gives
>         recommendations on which headers to count.
>
>             egress so as to issue or log any warnings when it receives
>             an ECT(0) or CE outer with a Not-ECT inner.
>
>                 I agree that the behavior defined in RFC6040 doesn’t
>                 to be updated due to the TCF draft, because what’s
>                 defined in RFC 6040 has a broader usage and not just
>                 for TCF.
>
>                 *From:*tsvwg [mailto:[email-masked]] *On
>                 Behalf Of *Black, David
>                 I think you’ve found something.  The TCF draft
>                 currently uses RFC 6040 behavior by reference:
>
>                 hence the logging and alarm behavior for that
>                 combination in RFC 6040 is inappropriate.
>
>                 This email documents an argument with myself. After a
>                 few seconds, I agreed with myself that one of me was
>                 wrong. I thought I had better raise this on the list,
>                 to check that others agree with both of me. If you
>                 wish to argue with just one of me, please bear in mind
>                 that we both have the same email address (and anyway,
>                 we both agree so an argument with one is an argument
>                 with all).
>                 Figure 4 of RFC6040 specifies which ECN codepoint a
>                 decap must forward, for each combination of arriving
>                 inner (row) and outer (column).
>
>                            
>                 For a moment, I thought we would have to update
>                 RFC6040 to say that TCF makes these combinations no
>                 longer CU, so they are now valid, not dangerous, and
>                 they no longer need to be logged. This would have
>                 meant that Tunnel Congestion Feedback (TCF)
>                 certainly valid at a TCF decap. But there is still no
>                 way that, in a valid network, they would arrive at any
>                 RFC6040 (non-TCF) decap.
>
>                 One might think that this combination could arrive at
>                 a 6040-decap if there was a tunnel inside a tunnel, as
>                 follows
>
>                 However, that's not a valid configuration. The
>                 following tunnel-in-tunnel would be valid:
>
>                 ...then, the above two "CU" combinations would be
>                 buried within a double inner, so only the TCF decap
>                 would see them, not the 6040-decap.
>
>                 valid network, TCF will be hidden from any 6040-decap.
>                 If the network is not valid, these combination could
>                 become visible to a 6040-decap, which would then
>                 (correctly) log warnings that it was seeing invalid
>                 codepoint combinations.
>