Subject: Re: [Int-area] [tsvwg] 2nd TSVWG WGLC on ecn-encap-guidelines and rfc6040-update-shim drafts, closes 6 May 2019

From: Bob Briscoe <[email-masked]>
Date: Mon, 8 Jul 2019 13:27:57 +0100

Joe,

Following up my email to you in May quoted further down, you made me 
realize that RFC6040 did not address what to do with ECN during 
fragmentation and reassembly. So I've added the following to my local 
copy of draft-ietf-tsvwg-rfc6040-update-shim (to be posted later today), 
which recently went through TSVWG last call, and will imminently be last 
called on various int-area lists, I believe.

These are quite significant updates to outer fragment processing at the 
tunnel egress. But, given something has to be said, I can't think of a 
better way (see the original quoted email about why the logical OR of 
the ECN codepoints as defined in RFC3168 is no longer sufficient - and 
it's no simpler anyway).

5.  ECN Propagation and Fragmentation/Reassembly

    The following requirements update RFC6040, which omitted handling of
    the ECN field during fragmentation or reassembly.  These changes
    might alter how many ECN-marked packets are propagated by a tunnel
    that fragments packets, but this would not raise any backward
    compatibility issues:

    If a tunnel ingress fragments a packet, it MUST set the outer ECN
    field of all the fragments to the same value as it would have set if
    it had not fragmented the packet.

    As a tunnel egress reassembles sets of outer fragments
    [I-D.ietf-intarea-tunnels] into packets, it SHOULD propagate CE
    markings on the basis that a congestion indication on a packet
    applies to all the octets in the packet.  On average, a tunnel egress
    SHOULD approximately preserve the number of CE-marked and ECT(1)-
    marked octets arriving and leaving (counting the size of inner
    headers, but not encapsulating headers that are being stripped).
    This process proceeds irrespective of the addresses on the inner
    headers.

    Even if only enough incoming CE-marked octets have arrived for part
    of the departing packet, the next departing packet SHOULD be
    immediately CE-marked.  This ensures that CE-markings are propagated
    immediately, rather than held back waiting for more incoming CE-
    marked octets.  Once there are no outstanding CE-marked octets, if
    only enough incoming ECT(1)-marked octets have arrived for part of
    the departing packet, the next departing packet SHOULD be immediately
    marked ECT(1).

    For instance, an algorithm for marking departing packets could
    maintain a pair of counters, the first representing the balance of
    arriving CE-marked octets minus departing CE-marked octets and the
    second representing a similar balance of ECT(1)-marked octets.  The
    algorithm:

    o  adds the size of every CE-marked or ECT(1)-marked packet that
       arrives to the appropriate counter;

    o  if the CE counter is positive, it CE-marks the next packet to
       depart and subtracts its size from the CE counter;

    o  if the CE counter is negative but the ECT(1) counter is positive,
       it marks the next packet to depart as ECT(1) and subtracts its
       size from the ECT((1) counter;

    o  (the previous two steps will often leave a negative remainder in
       the counters, which is deliberate);

    o  if neither counter is positive, it marks the next packet to depart
       as ECT(0);

    o  until all the fragments of a packet have arrived, it does not
       commit any updates to the counters so that, if reassembly fails
       and the partly reassembled packet has to be discarded, none of the
       discarded fragments will have updated any of the counters.

    During reassembly of outer fragments [I-D.ietf-intarea-tunnels], if
    the ECN fields of the outer headers being reassembled into a single
    packet consist of a mixture of Not-ECT and other ECN codepoints, the
    packet MUST be discarded.

    A tunnel end-point that claims to support the present specification
    MUST NOT use an approach that results in a significantly different
    ECN-marking outcome to that defined by the "SHOULD" statements
    throughout this section.  "SHOULD" is only used to allow similar
    perhaps more efficient approaches that result in approximately the
    same outcome.

Bob

On 16/05/2019 22:14, Bob Briscoe wrote:
> Joe,
>
> Sorry I missed this posting at the time (my mail filters moved both 
> cross-postings into my int-area box which I check only rarely).
>
>
> On 27/04/2019 18:13, Joe Touch wrote:
>> Cross-posting to let both communities know:
>>
>> - it would be useful for these documents to address how fragmentation 
>> and reassembly affects these signals
>> (esp. if reassembling fragments with different ECN values)
>>
> [BB] This is addressed by the re-framing section 
> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-encap-guidelines-12#section-4.6> 
> in ecn-encap-guidelines, altho it doesn't give examples of what might 
> have caused frame boundary misalignment, so fragmentation is not 
> specifically mentioned. I think I will add an explicit mention of 
> fragmentation (if only so a search finds that section).
>
> Actually I've realized that this highlights an inconsistency between 
> the advice on ECN and fragment reassembly in RFC3168 and in 
> ecn-encap-guidelines.:
>
>   * RFC3168 requires that the ECN marking of a reassembled packet is
>     the logical OR of the ECN marks on the fragments,
>   * whereas ecn-encap-guidelines recommends marking the same number of
>     outgoing as incoming octets when reassembling L2 frames or
>     tunnelled packets with different boundaries - using a simple
>     counter to track the balance.
>
> In fact, it was the review of RFC3168 by me and Jon Crowcroft back in 
> 2001 that originally raised the question of how to handle reassembly 
> of ECN-marked fragments 
> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-ip-00#section-11>. 
> I'll quote a passage from the review, which I think justifies the 
> recommendation in ecn-encap-guidelines to count marked bytes, rather 
> than use the logical OR of RFC3168:
>
>     To use the logical OR of the marking of all fragments might be a pragmatic
>     solution, particularly for congestion control protocols like TCP where one
>     loss per round trip is treated identically to many. However, it is becoming
>     more common to see large numbers of packets per round trip time as data
>     rates increase while packet sizes and the speed of light haven't increased
>     for many years. Therefore it is to be expected that newer congestion
>     control protocols might take more accurate account of the number of packets
>     marked in a round trip. Hence, the inaccuracy of a logical OR during
>     re-assembly at the IP layer is best avoided.
>
> I'm not too worried about the inaccuracy of using a logical OR, but I 
> think it best to recommend more accurate and no less costly counting. 
> The only justification for the logical OR was that TCP only reacted to 
> one ECN mark per RTT. But that is changing now, and the behaviour of 
> one transport should not be embedded in lower layers anyway.
>
>> - it would be useful for these documents to consider 
>> draft-ietf-intarea-tunnels (which relates to the above) and its 
>> discussion on many of the protocols cited
> I can't find anything in draft-ietf-intarea-tunnels that ought to be 
> cited from ecn-encap-guidelines or rfc6040-update-shim. Did you have 
> something specific in mind?
>
> I do want to raise a question about the following sentence, which 
> precedes the mention of ECN:
>     There are exceptions to this rule that are explicitly intended to
>     relay signals from inside the tunnel to the network outside the
>     tunnel, typically relevant only when the tunnel network N and the
>     outer network M use the same network.
> Was that last word meant to say "network protocol"?
>
> Then, if that is what you meant, I would disagree. Many different 
> network protocols include concepts similar to Diffserv and/or ECN 
> (e.g. IEEE802.1p, MPLS and TRILL support both, etc), and there's 
> rarely a reason /not/ to propagate the concept between different 
> network protocols when they encapsulate each other, even tho it's not 
> always straightforward to do so.
>
>
>
> Bob
>
>
> Bob
>>
>> Joe
>>
>>> On Apr 26, 2019, at 1:50 PM, Black, David <[email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>> This may be of interest to INT folks who are interested in tunnels 
>>> and encapsulations.
>>> Comments by the WGLC deadline are encouraged, but comments after the 
>>> deadline are ok, as they’d have to be dealt with anyway at IETF Last 
>>> Call.
>>> Thanks, --David
>>> *From:*tsvwg <[email-masked] 
>>> <mailto:[email-masked]>>*On Behalf Of*Black, David
>>> *Sent:*Wednesday, April 17, 2019 2:51 PM
>>> *To:[email-masked] <mailto:[email-masked]>
>>> *Subject:*[tsvwg] 2nd WGLC on ecn-encap-guidelines and 
>>> rfc6040-update-shim drafts, closes 6 May 2019
>>>
>>> [EXTERNAL EMAIL]
>>>
>>> This email announces a 2nd TSVWG Working Group Last Call (WGLC) on 
>>> two drafts:
>>> [1] Guidelines for Adding Congestion Notification to Protocols that
>>> Encapsulate IP
>>> draft-ietf-tsvwg-ecn-encap-guidelines-12
>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-encap-guidelines/
>>> This draft is intended to become a Best Current Practice RFC
>>> [2] Propagating Explicit Congestion Notification Across IP Tunnel 
>>> Headers
>>> Separated by a Shim
>>> draft-ietf-tsvwg-rfc6040update-shim-08
>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rfc6040update-shim/
>>> This draft is intended to become a Proposed Standard RFC.
>>> This WGLC will run through the end of the day on Monday, May 6, 2019.
>>> Comments should be sent to [email-masked] 
>>> <mailto:[email-masked]>list, although purely
>>> editorial comments may be sent directly to the author. Please cc: the
>>> WG chairs [email-masked] <mailto:[email-masked]> if 
>>> you would like the chairs to
>>> track such editorial comments as part of the WGLC process.
>>> No IPR disclosures have been submitted directly on either draft
>>> Thanks,
>>> David, Gorry and Wes
>>> (TSVWG Co-Chairs)
>>> _______________________________________________
>>> Int-area mailing list
>>> [email-masked] <mailto:[email-masked]>
>>> https://www.ietf.org/mailman/listinfo/int-area
>>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Black, David" <[email-masked]>
Date: Tue, 9 Jul 2019 23:50:16 +0000

Bob,

In this text:

   On average, a tunnel egress

   SHOULD approximately preserve the number of CE-marked and ECT(1)-marked octets arriving and leaving ...

I understand the rationale for CE-marked octets, but what’s the rationale for ECT(1)-marked octets?

Thanks, --David

From: Bob Briscoe <[email-masked]>
Sent: Monday, July 8, 2019 8:28 AM
To: Joe Touch; Black, David
Cc: [email-masked]; tsvwg
Subject: Re: [tsvwg] [Int-area] 2nd TSVWG WGLC on ecn-encap-guidelines and rfc6040-update-shim drafts, closes 6 May 2019

[EXTERNAL EMAIL]
Joe,

Following up my email to you in May quoted further down, you made me realize that RFC6040 did not address what to do with ECN during fragmentation and reassembly. So I've added the following to my local copy of draft-ietf-tsvwg-rfc6040-update-shim (to be posted later today), which recently went through TSVWG last call, and will imminently be last called on various int-area lists, I believe.

These are quite significant updates to outer fragment processing at the tunnel egress. But, given something has to be said, I can't think of a better way (see the original quoted email about why the logical OR of the ECN codepoints as defined in RFC3168 is no longer sufficient - and it's no simpler anyway).

   compatibility issues:

   headers.

   marked ECT(1).

   algorithm:

      as ECT(0);

   same outcome.

Bob

Sorry I missed this posting at the time (my mail filters moved both cross-postings into my int-area box which I check only rarely).

- it would be useful for these documents to address how fragmentation and reassembly affects these signals
              (esp. if reassembling fragments with different ECN values)

[BB] This is addressed by the re-framing section<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-encap-guidelines-12#section-4.6> in ecn-encap-guidelines, altho it doesn't give examples of what might have caused frame boundary misalignment, so fragmentation is not specifically mentioned. I think I will add an explicit mention of fragmentation (if only so a search finds that section).

Actually I've realized that this highlights an inconsistency between the advice on ECN and fragment reassembly in RFC3168 and in ecn-encap-guidelines.:

  *   RFC3168 requires that the ECN marking of a reassembled packet is the logical OR of the ECN marks on the fragments,
  *   whereas ecn-encap-guidelines recommends marking the same number of outgoing as incoming octets when reassembling L2 frames or tunnelled packets with different boundaries - using a simple counter to track the balance.
In fact, it was the review of RFC3168 by me and Jon Crowcroft back in 2001 that originally raised the question of how to handle reassembly of ECN-marked fragments<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-ip-00#section-11>. I'll quote a passage from the review, which I think justifies the recommendation in ecn-encap-guidelines to count marked bytes, rather than use the logical OR of RFC3168:

re-assembly at the IP layer is best avoided.
I'm not too worried about the inaccuracy of using a logical OR, but I think it best to recommend more accurate and no less costly counting. The only justification for the logical OR was that TCP only reacted to one ECN mark per RTT. But that is changing now, and the behaviour of one transport should not be embedded in lower layers anyway.

- it would be useful for these documents to consider draft-ietf-intarea-tunnels (which relates to the above) and its discussion on many of the protocols cited
I can't find anything in draft-ietf-intarea-tunnels that ought to be cited from ecn-encap-guidelines or rfc6040-update-shim. Did you have something specific in mind?

I do want to raise a question about the following sentence, which precedes the mention of ECN:

Then, if that is what you meant, I would disagree. Many different network protocols include concepts similar to Diffserv and/or ECN (e.g. IEEE802.1p, MPLS and TRILL support both, etc), and there's rarely a reason /not/ to propagate the concept between different network protocols when they encapsulate each other, even tho it's not always straightforward to do so.

Bob

Bob

Joe

On Apr 26, 2019, at 1:50 PM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:

This may be of interest to INT folks who are interested in tunnels and encapsulations.

Comments by the WGLC deadline are encouraged, but comments after the deadline are ok, as they’d have to be dealt with anyway at IETF Last Call.

Thanks, --David

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of Black, David
Sent: Wednesday, April 17, 2019 2:51 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [tsvwg] 2nd WGLC on ecn-encap-guidelines and rfc6040-update-shim drafts, closes 6 May 2019

[EXTERNAL EMAIL]
This email announces a 2nd TSVWG Working Group Last Call (WGLC) on two drafts:

[2] Propagating Explicit Congestion Notification Across IP Tunnel Headers
                          Separated by a Shim
                 draft-ietf-tsvwg-rfc6040update-shim-08
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rfc6040update-shim/
This draft is intended to become a Proposed Standard RFC.

Comments should be sent to the [email-masked]<mailto:[email-masked]> list, although purely
editorial comments may be sent directly to the author. Please cc: the
WG chairs at [email-masked]<mailto:[email-masked]>  if you would like the chairs to
track such editorial comments as part of the WGLC process.

Thanks,
David, Gorry and Wes
(TSVWG Co-Chairs)

_______________________________________________
Int-area mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/int-area

--

--

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Jul 2019 07:53:38 -0700

Hi, Bob,

This seems like a useful approach (though I’ll leave it to you and others to track the exact details).

Joe

> On Jul 8, 2019, at 5:27 AM, Bob Briscoe <[email-masked]> wrote:
> 
> Joe,
> 
>> [BB] This is addressed by the re-framing section <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-encap-guidelines-12#section-4.6> in ecn-encap-guidelines, altho it doesn't give examples of what might have caused frame boundary misalignment, so fragmentation is not specifically mentioned. I think I will add an explicit mention of fragmentation (if only so a search finds that section).
>> 
>> RFC3168 requires that the ECN marking of a reassembled packet is the logical OR of the ECN marks on the fragments, 
>> whereas ecn-encap-guidelines recommends marking the same number of outgoing as incoming octets when reassembling L2 frames or tunnelled packets with different boundaries - using a simple counter to track the balance.
>> In fact, it was the review of RFC3168 by me and Jon Crowcroft back in 2001 that originally raised the question of how to handle reassembly of ECN-marked fragments <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-ip-00#section-11>. I'll quote a passage from the review, which I think justifies the recommendation in ecn-encap-guidelines to count marked bytes, rather than use the logical OR of RFC3168:
>> 
>>>> On Apr 26, 2019, at 1:50 PM, Black, David <[email-masked] <mailto:[email-masked]>> wrote:
>>>> 
>>>> From: tsvwg <[email-masked] <mailto:[email-masked]>> On Behalf Of Black, David
>>>> To: [email-masked] <mailto:[email-masked]>
>>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-encap-guidelines/ <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-encap-guidelines/>
>>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rfc6040update-shim/ <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rfc6040update-shim/>
>>>> Comments should be sent to the [email-masked] <mailto:[email-masked]> list, although purely
>>>> WG chairs at [email-masked] <mailto:[email-masked]>  if you would like the chairs to
>>>> https://www.ietf.org/mailman/listinfo/int-area <https://www.ietf.org/mailman/listinfo/int-area>
>> 
>> -- 
>> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>
> -- 
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 10 Jul 2019 18:38:23 +0300

> On 8 Jul, 2019, at 3:27 pm, Bob Briscoe <[email-masked]> wrote:
> 

If I may offer such an alternative approach, which avoids the need to keep persistent state at the reassembly point whilst still properly handling RFC3168, L4S and SCE expected semantics:

- If all incoming fragments are Not-ECT marked, the outgoing packet must also be so marked.

- If any fragment has CE set, the reassembled packet must have CE set.

(This guarantees correct RFC3168 and SCE behaviour for each conventional AQM marking action.  Your proposal doesn't, as it will generally result in fewer CE marks downstream especially if the smaller fragments end up being marked; subsequent upstream CE marks have to relieve a counter deficit before they will be honoured.  The tradeoff is that L4S may see some technical "over marking" but this should be tolerable.)

- Notwithstanding the above rules, the ECT(0) vs ECT(1) choice should be made according to the majority of fragmented payload bytes so marked, on the individual packet being reassembled.  In the case of a tie, break in favour of ECT(0).

(We may expect that L4S packets will be entirely ECT(1) marked except for fragments or whole packets carrying CE; this also applies to obsolete Nonce Sum semantics.  SCE and RFC-3168 flows will be ECT(0) marked by default, with perhaps some ECT(1) marking applied by SCE middleboxes.  SCE is reasonably tolerant of disruption to its markings, because the control loop is fundamentally stable.)

- A mixture of Not-ECT and other ECN codepoints is unexpected and may imply upstream shenanigans.

 - Jonathan Morton

---

From: Markku Kojo <[email-masked]>
Date: Thu, 25 Jul 2019 15:50:56 +0300 (EEST)

Hi Bob, all,

catching up ...

The justification for the logical OR in RFC 3168 is not because TCP only 
reacts to one ECN mark per RTT. Instead, it ensures that no congestion 
signal is lost (that is MUST in RFC 3168), i.e., CE is delivered to end 
hosts for ECN-capable traffic equivalent to drop for non-ECN-capable 
traffic (if one fragment for a non-ECN-capable flow gets dropped by an 
AQM router, all fragments of the packet get dropped at reassembly).

I'm very concerned if the reassembly behavior is changed as proposed 
(e.g., for tunnels), because it makes it impossible for RFC3168-based 
("classic") ECN traffic to follow the leading guidelines of RFC 3168 for 
fair co-existence of ECN-capable and non-ECN-capable traffic in the 
presence of such tunnels.

Moreover, it begs for justification why two ECN-capable flows (A and 
B) that share the same ECN-enebled bottleneck within a tunnel should get 
different ECN-marking behavior, when flow A gets its packets fragmented 
before the tunnel and flow B within the tunnel but before the 
common bottleneck. And fragment sizes for A and B are (roughly) 
equivalent.

Cheers,

/Markku

On Mon, 8 Jul 2019, Bob Briscoe wrote:

> Joe,
> 
> Following up my email to you in May quoted further down, you made me realize that RFC6040 did not
> address what to do with ECN during fragmentation and reassembly. So I've added the following to
> my local copy of draft-ietf-tsvwg-rfc6040-update-shim (to be posted later today), which recently
> went through TSVWG last call, and will imminently be last called on various int-area lists, I
> believe.
> 
> These are quite significant updates to outer fragment processing at the tunnel egress. But, given
> something has to be said, I can't think of a better way (see the original quoted email about why
> the logical OR of the ECN codepoints as defined in RFC3168 is no longer sufficient - and it's no
> simpler anyway).
> 
>       Sorry I missed this posting at the time (my mail filters moved both cross-postings
>       into my int-area box which I check only rarely).
> 
>
> [BB] This is addressed by the re-framing section in ecn-encap-guidelines, altho it doesn't
> give examples of what might have caused frame boundary misalignment, so fragmentation is
> not specifically mentioned. I think I will add an explicit mention of fragmentation (if
> only so a search finds that section).
> 
> Actually I've realized that this highlights an inconsistency between the advice on ECN and
> fragment reassembly in RFC3168 and in ecn-encap-guidelines.:
>  *  whereas ecn-encap-guidelines recommends marking the same number of outgoing as incoming
>     octets when reassembling L2 frames or tunnelled packets with different boundaries -
>     using a simple counter to track the balance.
> In fact, it was the review of RFC3168 by me and Jon Crowcroft back in 2001 that originally
> raised the question of how to handle reassembly of ECN-marked fragments. I'll quote a
> passage from the review, which I think justifies the recommendation in ecn-encap-guidelines
> to count marked bytes, rather than use the logical OR of RFC3168:
> 
> I'm not too worried about the inaccuracy of using a logical OR, but I think it best to
> recommend more accurate and no less costly counting. The only justification for the logical
> OR was that TCP only reacted to one ECN mark per RTT. But that is changing now, and the
> behaviour of one transport should not be embedded in lower layers anyway.
>
>       - it would be useful for these documents to consider draft-ietf-intarea-tunnels
>       (which relates to the above) and its discussion on many of the protocols cited
> 
> I can't find anything in draft-ietf-intarea-tunnels that ought to be cited from
> ecn-encap-guidelines or rfc6040-update-shim. Did you have something specific in mind?
> 
> Then, if that is what you meant, I would disagree. Many different network protocols include
> concepts similar to Diffserv and/or ECN (e.g. IEEE802.1p, MPLS and TRILL support both,
> etc), and there's rarely a reason /not/ to propagate the concept between different network
> protocols when they encapsulate each other, even tho it's not always straightforward to do
> so.
> 
> 
> 
> Bob
> 
> 
> Bob
> 
> Joe
>
>       On Apr 26, 2019, at 1:50 PM, Black, David <[email-masked]> wrote:
> 
> Comments by the WGLC deadline are encouraged, but comments after the deadline
> are ok, as they’d have to be dealt with anyway at IETF Last Call.
>  
> Thanks, --David
>  
> From: tsvwg <[email-masked]> On Behalf Of Black, David
> To: [email-masked]
> Comments should be sent to the [email-masked] list, although purely
> WG chairs at [email-masked]  if you would like the chairs to
> [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 25 Jul 2019 09:24:09 -0400

Can you help: which lines of text were you looking at?

Gorry

On 25/07/2019, 08:50, Markku Kojo wrote:
> Hi Bob, all,
>
> catching up ...
>
> The justification for the logical OR in RFC 3168 is not because TCP 
> only reacts to one ECN mark per RTT. Instead, it ensures that no 
> congestion signal is lost (that is MUST in RFC 3168), i.e., CE is 
> delivered to end hosts for ECN-capable traffic equivalent to drop for 
> non-ECN-capable traffic (if one fragment for a non-ECN-capable flow 
> gets dropped by an AQM router, all fragments of the packet get dropped 
> at reassembly).
>
> ("classic") ECN traffic to follow the leading guidelines of RFC 3168 
> for fair co-existence of ECN-capable and non-ECN-capable traffic in 
> the presence of such tunnels.
>
> B) that share the same ECN-enebled bottleneck within a tunnel should 
> get different ECN-marking behavior, when flow A gets its packets 
> fragmented before the tunnel and flow B within the tunnel but before 
> the common bottleneck. And fragment sizes for A and B are (roughly) 
> equivalent.
>
> Cheers,
>
> /Markku
>
>> realize that RFC6040 did not
>> the advice on ECN and
>> 2001 that originally
>> raised the question of how to handle reassembly of ECN-marked 
>> fragments.. I'll quote a
>> think it best to
>> draft-ietf-intarea-tunnels
>> cited from
>> network protocols include
>> deadline
>> Subject: [tsvwg] 2nd WGLC on ecn-encap-guidelines and 
>> rfc6040-update-shim

---

From: Markku Kojo <[email-masked]>
Date: Thu, 25 Jul 2019 16:48:35 +0300 (EEST)

Hi Gorry,

not sure if you are referring text in drafts/RFC 3168 or trext in Bob's 
emails below?

This is about the conflict between RFC 3168 and the two drafts in 2nd 
TSVWG WGLC. As Bob put it, the two drafts recommend CE marking the same 
number of outgoing as incoming CE-marked octets when reassembling, while 
RFC 3168 mandates using logical OR of CE-marked fragments when 
reassembling. The latter is in Section 5.3 of RFC 3168:

"Reassembly  of a fragmented packet MUST NOT lose indications of 
congestion. In other words, if any fragment of an IP packet to be 
reassembled has the CE codepoint set, then one of two actions MUST be
taken:
- Set the CE codepoint on the reassembled packet."

and continues with other details and an alternative.

/Markku

On Thu, 25 Jul 2019, Gorry Fairhurst wrote:

>> traffic (if one fragment for a non-ECN-capable flow gets dropped by an AQM 
>> router, all fragments of the packet get dropped at reassembly).
>> 
>> I'm very concerned if the reassembly behavior is changed as proposed (e.g., 
>> for tunnels), because it makes it impossible for RFC3168-based ("classic") 
>> ECN traffic to follow the leading guidelines of RFC 3168 for fair 
>> co-existence of ECN-capable and non-ECN-capable traffic in the presence of 
>> such tunnels.
>> 
>> Moreover, it begs for justification why two ECN-capable flows (A and B) 
>> that share the same ECN-enebled bottleneck within a tunnel should get 
>> before the tunnel and flow B within the tunnel but before the common 
>> bottleneck. And fragment sizes for A and B are (roughly) equivalent.
>> 
>> Cheers,
>> 
>> /Markku
>> 
>>> tunnel egress. But, given
>>> cross-postings
>>> - it would be useful for these documents to address how fragmentation and 
>>> reassembly
>>> affects these signals
>>>  *  RFC3168 requires that the ECN marking of a reassembled packet is the 
>>> logical OR of the
>>>     ECN marks on the fragments,
>>> outgoing as incoming
>>> raised the question of how to handle reassembly of ECN-marked fragments.. 
>>> I'll quote a
>>> I do want to raise a question about the following sentence, which precedes 
>>> the mention of
>>> ECN:
>>>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 26 Jul 2019 21:53:25 -0400

Markku,

> the presence of such tunnels.
When packets are reassembled, there are less unmarked packets as well as 
less marked packets.
This is not intuitive for most people, so I recommend people do not only 
think about it intuitively.

Yes, I know RFC3168 requires the logical OR approach. It was me that 
suggested it (see draft-ietf-tsvwg-ecn-ip-00 from 2001). But I was wrong 
(in my defence, at least I said at the time that a more accurate 
approach would be preferred).

I am not proposing that we should use rfc6040-update-shim to correct 
RFC3168. The WG chairs might chose to, but I doubt it - I suspect that's 
mission creep. But at least we can ensure tunnels and lower layers are 
more correct than RFC3168 reassembly (I've only said "SHOULD" to allow 
less precise approaches as well).

Example: Imagine every packet is fragmented into two fragments. And the 
outers of 1% of fragments were marked within the tunnel. By the logical 
OR approach, that would lead to nearly 2% of reassembled packets marked 
(actually 1.99%, because some packets would be reassembled from 2 marked 
fragments).

Certainly, TCP responds to 1 or more marks within a RTT, so marks within 
the same RTT will not make any difference to TCP, but at low marking 
levels, roughly doubling the marking probability will nearly double the 
number of TCP round trips with a mark in them (again the exact 
proportion depends on the window and the marking probability). This will 
cause utilization to reduce to about 1/sqrt(2) = 71% (assuming it was 
100% before).

Not only is the logical OR approach incorrect and harmful to TCP 
performance, it is also just as harmful to other congestion controllers 
that use the proportion of marked or dropped packets, such as rmcat, etc.

> the common bottleneck. And fragment sizes for A and B are (roughly) 
By the above arguments, the byte-preserving approach is not different - 
quite the opposite - it results in the same marking probability. The 
logical OR approach would result in greater ECN marking for B.

See also
https://tools.ietf.org/html/rfc7141

Bob

PS. Of course, fragmentation adds more bytes, because it adds more 
headers. So if a link within a tunnel is congested, the same rate of 
data sent in fragments will cause more congestion at this link than 
full-sized packets would have. This increase in congestion is 
appropriate. The byte-preserving recommendation that I suggest ensures 
that it is not accompanied by further inflation because of the logical OR.
>> fragments. I'll quote a

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 27 Jul 2019 06:01:01 -0400

Jonathan,

> (This guarantees correct RFC3168 and SCE behaviour for each conventional AQM marking action.  Your proposal doesn't, as it will generally result in fewer CE marks downstream especially if the small…
Yes, with byte-preserving, as packets are re-assembled the number of 
marked packets reduces. Counter-intuitively, that's correct, even for 
compatibility with TCP's single congestion response per RTT.

I originally suggested the requirement in RFC3168 to preserve the number 
of marked packets, but it's incorrect. It's not compatible with TCP's 
single response per RTT (or the response to the proportion of marks of 
other TCP-Friendly real-time congestion controls).

This is not a matter of compatibility with just one of SCE or L4S. The 
logical OR approach is wrong for both, and the byte-preserving approach 
is correct for both - see previous response to Markku.

Reasoning: the paramount requirement when reassembling fragments is to 
reconstruct the marking probability that would have occurred had the 
packets not been fragmented when the AQM in the tunnel marked them. The 
logical OR approach increases the marking probability as if congestion 
was higher, while byte-preserving keeps it constant.

If it helps, consider the reductio ad absurdam proof that, as fragments 
get smaller, the logical OR approach would ultimately result in every 
packet being marked.

Regarding state, the problem isn't amount of state because reassembly 
requires per-packet state and the the approach I suggested adds only 2 
int's per tunnel decap.

The problem seems to be common read-write access to these variables. 
However, it's not important that they are updated before forwarding 
continues. So updates can be queued in parallel to forwarding.

Also the state can include occasional errors, so it doesn't have to be 
strictly persistent - meaning it's unnecessary to preserve during a 
re-boot or if a tunnel endpoint is moved.

> - Notwithstanding the above rules, the ECT(0) vs ECT(1) choice should be made according to the majority of fragmented payload bytes so marked, on the individual packet being reassembled.  In the ca…
My original proposed wording deliberately allows such an algorithm. 
Byte-preserving was stated as a goal. The specific mechanism was only an 
example.

(I'm not convinced majority voting would be simpler than byte counting, 
which never leads to an exception case. But that's irrelevant anyway).

> (We may expect that L4S packets will be entirely ECT(1) marked except for fragments or whole packets carrying CE; this also applies to obsolete Nonce Sum semantics.  SCE and RFC-3168 flows will be…
As above, my proposed wording is nothing to do with support for one 
semantic or another. It is for all semantics.
>
> - A mixture of Not-ECT and other ECN codepoints is \unexpected and may imply upstream shenanigans.
Yup, the wording in 3168, and my wording both agree with you here.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 27 Jul 2019 08:50:15 -0400

> On 27 Jul, 2019, at 6:01 am, Bob Briscoe <[email-masked]> wrote:
> 
> Yes, with byte-preserving, as packets are re-assembled the number of marked packets reduces. Counter-intuitively, that's correct, even for compatibility with TCP's single congestion response per RTT.
> 
> I originally suggested the requirement in RFC3168 to preserve the number of marked packets, but it's incorrect. It's not compatible with TCP's single response per RTT (or the response to the proportion of marks of other TCP-Friendly real-time congestion controls).
> 
> This is not a matter of compatibility with just one of SCE or L4S. The logical OR approach is wrong for both, and the byte-preserving approach is correct for both - see previous response to Markku.
> 
> Reasoning: the paramount requirement when reassembling fragments is to reconstruct the marking probability that would have occurred had the packets not been fragmented when the AQM in the tunnel marked them. The logical OR approach increases the marking probability as if congestion was higher, while byte-preserving keeps it constant.

I think you are still thinking in terms of marking *probability*, which is not correct in at least conventional RFC-3168 semantics.  Conventional TCPs are sensitive to the number of RTTs between marks; the DCTCP response function is sensitive to the number of marks per RTT.  Both are *time based* relationships.

Codel, the most deployed AQM, is a *time based* marking function.  Preserving the marking probability while the number of packets decreases would reduce the number of RTTs containing a mark.  The original logical-OR function correctly preserves this property, and only fails when there's a mixture of ECT(0) and ECT(1) marks on fragments for a single packet (which can legitimately happen with SCE but not L4S, ironically enough).  Preserving the number of marked bytes does not preserve the number of marks over time, and therefore does not preserve the intent of the marks applied by the AQM.

That's the logic behind my suggested series of rules.  I carefully held both L4S and SCE requirements in mind, as well as those of RFC-3168, while drafting them.

 - Jonathan Morton

---

From: "Black, David" <[email-masked]>
Date: Wed, 31 Jul 2019 13:55:40 +0000

[Writing as draft shepherd and responsible WG chair, still interested in additional responses & comments]

Back to Joe's original comment that started this.

I'm not clear on how that's either incorrect or incompatible with TCP's single response per RTT - as long as TCP sees a CE mark within an RTT interval, the right thing happens, and extra CE marks don't matter.  Picking up a line from Jonathan's response:

> Preserving the marking probability while the number of packets decreases would reduce the number of RTTs containing a mark.  

I'm definitely concerned about that.  Suppose we have a stream of packets in which each packet is fragmented into 3 fragments, and exactly one fragment is marked in each RTT.   My understanding of the requirement in the new text to preserve the number of marked octets is that it would result in a CE marked packet once every 3 RTTs after reassembly, which is wrong.  If DCTCP needs a different marking behavior, then these drafts need a warning about what goes wrong if that's not done - that's reasonable as deployment considerations for DCTCP, but not TCP, as DCTCP is currently a "controlled environment" protocol. 

Further, these two drafts currently do not modify RFC 3168, which (IMHO) is a "feature" ... not a "bug" ...

Additional thoughts/comments?

Thanks, --David

> -----Original Message-----
> Sent: Saturday, July 27, 2019 6:01 AM
> To: Jonathan Morton
> Cc: Joe Touch; Black, David; [email-masked]; tsvwg
> Subject: Re: [tsvwg] [Int-area] 2nd TSVWG WGLC on ecn-encap-guidelines
> and rfc6040-update-shim drafts, closes 6 May 2019
> 
> 
> [EXTERNAL EMAIL]
> 
> Jonathan,
> 
> tunnel egress. But, given something has to be said, I can't think of a better
> way (see the original quoted email about why the logical OR of the ECN
> codepoints as defined in RFC3168 is no longer sufficient - and it's no simpler
> anyway).
> > If I may offer such an alternative approach, which avoids the need to keep
> persistent state at the reassembly point whilst still properly handling
> RFC3168, L4S and SCE expected semantics:
> >
> > (This guarantees correct RFC3168 and SCE behaviour for each conventional
> AQM marking action.  Your proposal doesn't, as it will generally result in
> fewer CE marks downstream especially if the smaller fragments end up being
> marked; subsequent upstream CE marks have to relieve a counter deficit
> before they will be honoured.  The tradeoff is that L4S may see some
> technical "over marking" but this should be tolerable.)
> > - Notwithstanding the above rules, the ECT(0) vs ECT(1) choice should be
> made according to the majority of fragmented payload bytes so marked, on
> the individual packet being reassembled.  In the case of a tie, break in favour
> of ECT(0).
> > (We may expect that L4S packets will be entirely ECT(1) marked except for
> fragments or whole packets carrying CE; this also applies to obsolete Nonce
> Sum semantics.  SCE and RFC-3168 flows will be ECT(0) marked by default,
> with perhaps some ECT(1) marking applied by SCE middleboxes.  SCE is
> reasonably tolerant of disruption to its markings, because the control loop is
> fundamentally stable.)