Subject: Re: [tcpm] Re: draft-ietf-tcpm-ecnsyn-03.txt backwards compatibility

From: Mark Allman <[email-masked]>
Date: Fri, 04 Jan 2008 09:00:11 -0500

Folks-

Bob raised a point about a case where a CE mark can be masked in the
ECN-SYN scheme that is under WGLC.  Sally's summary is appended below.
It seems from the discussion in Vancouver and the silence on the list
that there is not much consensus to change the document's current state
(which is approach (1) below---i.e., do nothing).  If folks have a
different opinion on this question they need to speak up quite fast as
it seems this document is ready to forward.

Thanks!

allman
(tcpm co-chair)

> To Bob and the working group:
> 
> The discussion with Bob Briscoe concerns the following paragraph in
> the ECN/SYN draft:
> 
> The discussion with Bob Briscoe concerns the following paragraph in
> Section 4 of the ECN/SYN draft:
> 
>     Backwards compatibility:
>     In order for TCP node B to send a SYN/ACK packet as ECN-Capable, node
>     B must have received an ECN-setup SYN packet from node A.  However,
>     it is possible that node A supports ECN, but either ignores the CE
>     codepoint on received SYN/ACK packets, or ignores SYN/ACK packets
>     with the ECT or CE codepoint set.  If the TCP initiator ignores the
>     CE codepoint on received SYN/ACK packets, this would mean that the
>     TCP responder would not respond to this congestion indication.
>     However, this seems to us an acceptable cost to pay in the
>     incremental deployment of ECN-Capability for TCP's SYN/ACK packets.
>     It would mean that the responder would not reduce the initial
>     congestion window from two, three, or four segments down to one
>     segment, as it should.  However, the TCP end nodes would still
>     respond correctly to any subsequent CE indications on data packets
>     later on in the connection.  Thus, to be explicit, when a TCP
>     connection includes an initiator that supports ECN but *does not*
>     support ECN-Capability for SYN/ACK packets, in combination with a
>     responder that *does* support ECN-Capabililty for SYN/ACK packets, it
>     is quite possible that the ECN-Capable SYN/ACK packets will be marked
>     rather than dropped in the network, and that the responder will not
>     learn about the ECN mark on the SYN/ACK packet.
> 
> There are three choices:
> 
> (1) Leave it as it is, and accept it that sometimes, as a cost of
> incemental deployment, a SYN/ACK packet would be ECN-marked, and the
> originator, with an older TCP implementation (ECN-capable, but not
> understanding ECN-marked SYN/ACK packets), would ignore the ECN mark
> on the SYN/ACK packet.
> 
> (I don't know what the Microsoft, Max OS X, and Linux ECN
> implementations
> do about ECN-marked SYN/ACK packets, but I would assume that they
> ignore the ECN marks on SYN/ACK packets...)
> 
> (2) Use one of the four remaining TCP header flags.  On a SYN packet,
> the flag would mean ECN-SYN ("I want to use ECN, and I understand
> ECN-marked SYN/ACK packets").  On a packet other than a SYN packet,
> the TCP flag could be used for something else.  The responder already
> has to look at the TCP header flags to learn that the originator
> is ECN-capable, so looking at one more flag should not be a problem.
> 
> (3) Use a two-byte ECN-SYN TCP option on SYN packets, meaning "I
> understand ECN-marked SYN/ACK packets."  There is at most 40 bytes
> for TCP options, and other options that might be carried by SYN
> packets include Sack Permitted (2 bytes), MSS (4 bytes), Window
> Scale (3 bytes), Timestamp (10 bytes).
> 
> 
> The downside of (1) is that during congestion, SYN-ACK packets
> could be ECN-marked rather than dropped, and the TCP response
> might never find out about the ECN-mark on the SYN-ACK packet.
> This could happen if the TCP originator uses an old ECN-Capable
> TCP implementation that doesn't response to ECN marks on
> incoming SYN/ACK packets, and the congested link has ECN enabled.
> 
> The downside of (2) and (3) is that the TCP flag or TCP option
> would have to be used in perpetuity, long after all ECN-capable
> TCP implementations were updated to respond to incoming ECN-marked
> SYN/ACK packets.  This would be a drag for (3), but less of a drag
> for (2) - the cost of using one of the four remaining header flags
> might be an acceptable cost.
> 
> 
> So would the working group like us to use a TCP header flag for this,
> or not?
> 
> This is scheduled for a 5-minute slot at TCMP on Tuesday, so we
> can discuss it then.
> 
> - Sally
> http://www.icir.org/floyd/
> 
> 
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 04 Jan 2008 18:04:15 +0000

Mark,

'Do nothing' would be a relatively safe option if we assume most 
legacy ECN in TCPs used as clients will be updated to ECN+ before:
a) they interact with those legacy implementations of ECN already out 
there on servers and
b) before ECN is widely deployed on routers.

The largest implementation-base of legacy ECN is currently in Windows 
Vista which has ECN turned off by default. I assume the population of 
other implementations in TCPs used as clients is small.

I've noted a general lack of concern/interest here about 
standardising a congestion control that doesn't respond to congestion 
in a set of circumstances that might happen or might not. I assume 
this is because of the glacial deployment of ECN so far.

If the world turns out differently to the above assumptions, we will 
have to think of a plan B pretty snappish.

But I'm not going to hold this up further, as ECN+ is generally a 
good idea, and the other alternatives for incremental deployment are 
cumbersome.

Bob

> > http://www.icir.org/floyd/

____________________________________________________________________________
Notice: This contribution is the personal view of the author and does 
not necessarily reflect the technical nor commercial direction of BT plc.
____________________________________________________________________________
Bob Briscoe,                           Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Sally Floyd <[email-masked]>
Date: Fri, 4 Jan 2008 11:50:45 -0800

Bob -

...
> I've noted a general lack of concern/interest here about standardising 
> a congestion control that doesn't respond to congestion in a set of 
> circumstances that might happen or might not. I assume this is because 
> of the glacial deployment of ECN so far.
>
> But I'm not going to hold this up further, as ECN+ is generally a good 
> idea, and the other alternatives for incremental deployment are 
> cumbersome.

I think it is safe not to use a flag for ECN-Capable SYN/ACK packets,
but I wouldn't have any problems with using a flag either.

The revised text for draft-ietf-tcpm-ecnsyn is attached below.

- Sally

    Backwards compatibility:
    later on in the connection.

    Figure 3 shows an interchange with the SYN/ACK packet ECN-marked, but
    with the ECN mark ignored by the TCP originator.

         ---------------------------------------------------------------
            TCP Node A             Router                  TCP Node B
            ----------             ------                  ----------

            ECN-setup SYN packet --->
                                            ECN-setup SYN packet --->

                                          <--- ECN-setup SYN/ACK, ECT
                               <--- Sets CE on SYN/ACK
            <--- ECN-setup SYN/ACK, CE

            Data/ACK, No ECN-Echo --->
                                                       Data/ACK --->
                                 <--- Data (up to four packets), CWR
         ---------------------------------------------------------------

            Figure 2: SYN exchange with the SYN/ACK packet marked,
              but with the ECN mark ignored by the TCP initiator.

    Thus, to be explicit, when a TCP connection includes an initiator
    that supports ECN but *does not* support ECN-Capability for SYN/ACK
    packets, in combination with a responder that *does* support ECN-
    Capabililty for SYN/ACK packets, it is possible that the ECN-Capable
    SYN/ACK packets will be marked rather than dropped in the network,
    and that the responder will not learn about the ECN mark on the
    SYN/ACK packet.  This would not be a problem in a scenario with
    mostly long-lived TCP connections, but it would be more problematic
    in a scenario where all TCP connections consisted of four data
    packets, and the TCP responder was ready to send its data packets
    immediately after the SYN/ACK exchange.  Of course, with *severe*
    congestion, the SYN/ACK packets would likely be dropped rather than
    ECN-marked at the congested router, preventing the TCP responder from
    adding to the congestion by sending its initial window of four data
    packets.

    It is also possible that in some older TCP implementation, the
    initiator would ignore arriving SYN/ACK packets that had the ECT or
    CE codepoint set.  This would result in a delay in connection set-up
    for that TCP connection, with the initiator re-sending the SYN packet
    after a retransmit timeout.  We are not aware of any TCP
    implementations with this behavior.

    One possibility for coping with problems of backwards compatibility
    would be for TCP initiators to use a TCP flag that means "I
    understand ECN-Capable SYN/ACK packets".  If this document were to
    standardize the use of such an "ECN-SYN" flag, then the TCP responder
    would only send a SYN/ACK packet as ECN-capable if the incoming SYN
    packet had the "ECN-SYN" flag set.  An ECN-SYN flag would prevent the
    backwards compatibility problems described in the paragraphs above.

    One drawback to the use of an ECN-SYN flag is that it would use one
    of the four remaining reserved bits in the TCP header, for a
    transient backwards compatibility problem.  This drawback is limited
    by the fact that the "ECN-SYN" flag would be defined only for use
    with ECN-setup SYN packets;  that bit in the TCP header could be
    defined to have other uses for other kinds of TCP packets.

    Factors in deciding not to use an ECN-SYN flag include the following:

    (1) The limited installed base: At the time that this document was
    written, the TCP implementations in Microsoft Vista and Mac OS X
    included ECN, but ECN was not enabled by default [SBT07].  Thus,
    there was not a large deployed base of ECN-Capable TCP
    implementations.  This limits the scope of any backwards
    compatibility problems.

    (2) Limits to the scope of the problem: The backwards compatibility
    problem would not be serious enough to cause congestion collapse;
    with severe congestion, the buffer at the congested router will
    overflow, and the congested router will drop rather than ECN-mark
    arriving SYN packets.  Some active queue management mechanisms might
    switch from packet-marking to packet-dropping in times of high
    congestion before buffer overflow, as recommended in Section 19.1 of
    RFC 3168.  This helps to prevent congestion collapse problems with
    the use of ECN.

    (3) Detection of and response to backwards-compatibility problems: A
    TCP responder such as a web server can't differentiate between a
    SYN/ACK packet that is not ECN-marked in the network, and a SYN/ACK
    packet that is ECN-marked, but where the ECN mark is ignored by the
    TCP initiator.  However, a TCP responder *can* detect if a SYN/ACK
    packet is sent as ECN-capable and not reported as ECN-marked, but
    data packets are dropped or marked from the initial window of data.
    We will call this scenario "initial-window-congestion".  If a web
    server frequently experienced initial-window congestion (without
    SYN/ACK congestion), then the web server *might* be experiencing
    backwards compatibility problems with ECN-Capable SYN/ACK packets,
    and could respond by not sending SYN/ACK packets as ECN-Capable.

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 06 Jan 2008 00:11:51 +0000

Sally,

The new text outlines the issues well (I've suggested some minor 
changes inline - all in the para below Fig 2).

But I'm not a great fan of writing long complicated text in standards 
track documents that describes a problem, when it's sort of hinting 
that implementers don't actually need to do anything about it. One 
way round this would be to move this text about incremental 
deployment to an appendix.

Point (3) is a good one that I hadn't thought of.

Here's strawman text for a solution that could be put in the body. 
I've used the terms 'TCP clients' and 'TCP servers' for machines used 
mostly as initiators or responders respectively:

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
"There is a potential incremental deployment problem in circumstances 
we believe will be unlikely, described in (Appendix X). In monitoring 
the deployment of ECN and ECN+, if significant numbers of queues have 
ECN turned on and if we find that large numbers of TCP clients still 
have legacy ECN enabled while significant numbers of TCP servers have 
upgraded to ECN+, the management application described below could be 
deployed on ECN+ servers, particularly if most of the flows served 
fit within an initial window.

The management application could monitor the average fraction of ACKs 
with ECE set in response to:
         1. a SYN/ACK
         2. the segments in the initial window.
If the former fraction was significantly lower than the latter, the 
management application could switch ECN+ back to ECN, perhaps 
requiring manual intervention rather than automatic switching.

Once the community deemed that the deployment situation had become 
safer for ECN+, server administrators could switch ECN+ back on, 
either leaving the managment application running, or eventually 
removing the management application.

Therefore, all ECN+ TCP implementations SHOULD provide the following 
management interfaces, in order to ease installation of such a 
management application after the TCP has been upgraded to ECN+:
         o An interface to turn ECN+ on or off;
         o An interface to read a count of only those ACKs with ECE 
set that were responses to SYN/ACKs."
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

I said this is strawman text, because I'm not sure I even support it 
myself. I'm just trying to work through the minimum implementers 
would need to do about incremental deployment other than 'nothing'.

The alternative of doing absolutely nothing about incremental 
deployment right now is still possible. The above required management 
interfaces can be added by upgrading the kernel any time. I just 
figured the above might be worthwhile to ease installation of a 
little monitoring application, so at least sysadmins can simply test 
whether there is a problem if they choose to.

[BTW, there are cases where the approach in (3) wouldn't work (e.g. 
if SYN/ACKs had a significantly lower probability of ECN marking than 
data packets). But I really don't think we should start down that 
rat-hole in this I-D - it would be like describing a pimple on a spot 
on a flea.]

A little more inline...

At 19:50 04/01/2008, Sally Floyd wrote:
>>standardising a congestion control that doesn't respond to 
>>congestion in a set of circumstances that might happen or might 
>>not. I assume this is because of the glacial deployment of ECN so far.
>>
>>If the world turns out differently to the above assumptions, we 
>>will have to think of a plan B pretty snappish.
>>
>>good idea, and the other alternatives for incremental deployment 
>>are cumbersome.
>

\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

>    SYN/ACK packet.  This would not be a problem

s/ in a scenario with/ with machines serving /

>    mostly long-lived TCP connections, but it would be more problematic

s/in a scenario where all /with an ECN+ TCP responder where most /

>TCP connections consisted of

at most

>four data
>    immediately after the SYN/ACK exchange.

If correctly responding ECN-capable traffic shared a bottleneck with 
traffic from such an ECN+ machine, its rate would be continually 
beaten down by the ECN+ machine's reduced congestion response.

>Of course, with *severe*

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

Cheers

Bob

---

From: Sally Floyd <[email-masked]>
Date: Mon, 7 Jan 2008 21:21:40 -0800

Bob -

> that implementers don't actually need to do anything about it. One way  
> round this would be to move this text about incremental deployment to  
> an appendix.

Yep, I will do that.
> deployed on ECN+ servers, particularly if most of the flows served fit  
> within an initial window.
>
>         o An interface to read a count of only those ACKs with ECE set  
> that were responses to SYN/ACKs."
> myself. I'm just trying to work through the minimum implementers would  
> need to do about incremental deployment other than 'nothing'.

I am inclined to leave that out, for now.
Unless others have other opinions.

But I did add a moderated version as follows:

"The TCP implementation using ECN-Capable SYN/ACK packets SHOULD
include a management interface to allow the use of ECN to be turned
off for SYN/ACK packets.  This is to deal with possible backwards
compatibility problems such as those discussed in Appendix B."

> figured the above might be worthwhile to ease installation of a little  
> monitoring application, so at least sysadmins can simply test whether  
> there is a problem if they choose to.
>
> [BTW, there are cases where the approach in (3) wouldn't work (e.g. if  
> SYN/ACKs had a significantly lower probability of ECN marking than  
>>> congestion in a set of circumstances that might happen or might not.  
>>> I assume this is because of the glacial deployment of ECN so far.
>>>

I changed "scenario" to "environment"

> s/in a scenario where all /with an ECN+ TCP responder where most /

I changed "scenario" to "environment" here also, and changed
"all" to "much".

>> TCP connections consisted of

I left it as it.  If all TCP connections used an initial window of
one data packet, or had only one data packet to send, for example,
then ECN+ wouldn't have as large backwards compatibility problems.
(That is, the congestion control situation would be pretty bad with
or without backwards compatibility problems with ECN+.)

> beaten down by the ECN+ machine's reduced congestion response.

Yep.

- Sally

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 09 Jan 2008 00:06:47 +0000

Sally,

I'll accept all your responses...

Except, I probably didn't explain well why I didn't like:
"This would not be a problem in an environment with mostly long-lived 
TCP connections..."

This is ambiguous as to whether it means the competing flows are long 
lived or the ECN+ server's flows are long lived. The problem is short 
flows from the ECN+server. The problem hits both short and long 
competing flows.

Bob

At 05:21 08/01/2008, Sally Floyd wrote:
>>/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ \/\
>>/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ \/\
>>
>>\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ \/
>>
>>/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ \/\
>>
>>/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ \/\
>
>- Sally

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Sally Floyd <[email-masked]>
Date: Tue, 8 Jan 2008 16:50:07 -0800

Bob -

> flows from the ECN+server. The problem hits both short and long  

Great, I will add a clarification.

Mark, is it time for me to submit the revised internet-draft?

- Sally

>>> way
>>> would
>>> little
>>> if
>>>>> not.
>>>>> will
>>>>> are
- Sally

---

From: Mark Allman <[email-masked]>
Date: Tue, 08 Jan 2008 20:34:36 -0500

> Mark, is it time for me to submit the revised internet-draft?

Yep---that sounds like the right thing to do.

Thanks!

allman

---

From: Sally Floyd <[email-masked]>
Date: Tue, 8 Jan 2008 20:40:28 -0800

> Yep---that sounds like the right thing to do.

Done.  It is at:
http://tools.ietf.org/html/draft-ietf-tcpm-ecnsyn-04

- Sally

The diffs:

    NOTE TO RFC EDITOR: PLEASE DELETE THIS NOTE UPON PUBLICATION.

    Changes from draft-ietf-tcpm-ecnsyn-03:

    * General editing.  This includes using the terms "initiator"
      and "responder" for the two ends of the TCP connection.
      Feedback from Alfred Hoenes.

    * Added some text to the backwards compatibility discussion,
      now in Appendix B, about the pros and cons of using a TCP
      flag for the TCP initiator to signal that it understands
      ECN-Capable SYN/ACK packets.  The consensus at this time is
      not to use such a flag.  Also added a recommendation that
      TCP implementations include a management interface to turn
      off the use of ECN for SYN/ACK packets.  From email from
      Bob Briscoe.