Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 2 Jul 2018 17:54:28 +0200

Hi Micheal,

I addressed a couple of your comments below. 

For the other, bigger comments regarding extensibility, that I did not yet address below, we plan to add a new section to the appendix to explain extensibility options as previously discussed by mail. We will probably send a separate email on that part.

Mirja

> Am 12.03.2018 um 01:59 schrieb Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>:
> 
> Hi Mirja,
> 
> Thanks a lot for the explanation. I won't follow-up on some of the editorial suggestions.
> 
> Yet, I continue to believe that some formal wording in the document needs to change, as explained below. 
> 
> Thanks
> 
> Michael (with no hat on)
> 
> 
>> -----Original Message-----
>> From: Mirja Kühlewind [mailto:[email-masked]]
>> Sent: Monday, March 05, 2018 1:54 PM
>> To: Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>
>> Cc: [email-masked]; [email-masked]
>> Subject: Re: Comments on draft-ietf-tcpm-accurate-ecn
>> 
>> Hi Micheal,
>> 
>> thanks for your feedback and sorry for my late reply.
>> 
>> Please see inline.
>> 
>>> Am 03.12.2017 um 20:17 schrieb Scharf, Michael (Nokia - DE/Stuttgart)
>> <[email-masked]>:
>>> 
>>> Hi all,
>>> 
>>> I have read draft-ietf-tcpm-accurate-ecn-05 (without the appendix). I
>> believe this document needs further work before moving forward.
>>> 
>>> Please find below my comments marked as [ms]. I have read the
>> document independent of the review from Gorry. I apologize if there is
>> duplication.
>>> 
>>> Thanks
>>> 
>>> Michael (with no hat on)
>>> 
>>> 
>>> ******************************
>>> 
>>> * Abstract:
>>> 
>>>   Recently, new TCP mechanisms like Congestion Exposure (ConEx) or Data
>> Center TCP
>>>   (DCTCP) need more accurate ECN feedback information whenever more
>>>   than one marking is received in one RTT.
>>> 
>>> [ms] I don't think this statement is fully backed by RFC 8257. I suggest to
>> remove this, or replace it by a more generic statement that more accurate
>> information can be useful for several TCP extensions.
>> 
>> I disagree. Both ConEx and DCTCP need more accurate information. They do
>> not need the mechanism that is specified in this draft, however, this is not
>> what the sentences is saying.
> 
> In my understanding (as a non-native speaker), the use of the word "need" is not correct here. DCTCP as specified in RFC 8257 can be implemented without any such mechanism. 
> 
> What would work for me is something of the form "... Data Center TCP cannot get precise ECN feedback whenever more than one marking is received in one RTT“.

This is not correct. DCTP need more than one feedback signal per RTT and therefore cannot use RFC3168; instead it implement it’s own feedback mechanism. However, to avoid confusion such that people could assume DCTP would not work without the accECN scheme as specified in this doc, I rephrased to:

"Recently, proposed
      mechanisms like Congestion Exposure (ConEx <xref target="RFC7713"/>),
      DCTCP <xref target="RFC8257"/> or L4S <xref
      target="I-D.ietf-tsvwg-l4s-arch"/> need to know when more than one
      marking is received in one RTT which is
      information that cannot be provided by the feedback scheme as specified in
      <xref target="RFC3168"/>."

> 
>>>   This document specifies an
>>>   experimental scheme to provide more than one feedback signal per RTT
>>>   in the TCP header.  Given TCP header space is scarce, it overloads
>>>   the three existing ECN-related flags in the TCP header and provides
>>>   additional information in a new TCP option.
>>> 
>>> [ms] This statement needs to be rewritten to correctly reflect what is
>> requested from IANA. My understanding is that this experimental document
>> asks for allocation of a reserved TCP header flag. This needs to be called out
>> prominently, IMHO. In addition, since this is not a standard, the suggested
>> experimentation with the main TCP header must IMHO be explicitly
>> mentioned. I also suggest to have later in a document a section that explicitly
>> explains why it is appropriate to modify the main TCP header in an
>> experiment.
>> 
>> I don’t know if any requirement that IANA assignment need to be called out
>> in the abstract but we can do that. However, I believe the question if this
>> document should or should not assign the bit is still not completely solved, or
>> is it?
> 
> I believe this question will have to be reviewed during WGLC and, more importantly, IETF last call. For the moment, my concern is that the document correctly describes the IANA allocation.
> 
> I would like to see here a statement such as : "Given TCP header space is scarce, this specification allocates a reserved header bit and overloads the two ECN flags in the TCP header ...“. 

A bit lengthy but now:

"Given TCP header space is
      scarce, it allocates a reserved header bit, that was previously used for
      ECN-Nonce which was recently declared historic, and overloads the
      two existing ECN flags in the TCP header. Further, additional
      information can be provided in a new TCP option that however is not used
      on the TCP SYN."

> 
>>> * 1.  Introduction
>>> 
>>>   Recently, proposed mechanisms like Congestion Exposure (ConEx
>>>   [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need
>>>   more accurate ECN feedback information whenever more than one
>> marking
>>>   is received in one RTT.
>>> 
>>> [ms] At least for RFC 8257 seems to be implementable withoit this. Instead
>> of stating a "need", it would IMHO make more sense to discuss the benefits
>> of the suggested mechanism in this document of its own, independent of
>> other proposals. To me, this document should be independent of other
>> documents and specifically other experiments. We have to think about cases
>> where not all experiments are successful. Then independent documents will
>> be more future-proof in future.
>> 
>> This is a naming collision… The sentence was meant to say that these
>> mechanisms new more accurate ECN feedback than provided today by
>> RFC3168 but it was not meant to say that these mechanism have to use the
>> scheme as specified in this document.
>> 
>> I added the following part sentence:
>> 
>> „Recently, proposed mechanisms like Congestion Exposure (ConEx
>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need more
>> accurate ECN feedback information than provided by the feedback scheme
>> as specified in [RFC3168] whenever more than one marking is received in one
>> RTT. This document specifies an alternative feedback scheme that provides
>> more accurate information and could be used by these new TCP extensions.“
>> 
>> Does this help?
> 
> See my proposal for the abstract. I continue to disagree with the term "need" but I think this can be sorted out by another term.
> 
>>>   If AccECN progresses from experimental to the standards
>>>   track, it is intended to be a complete replacement for classic TCP/
>>>   ECN feedback, not a fork in the design of TCP.
>>> 
>>> [ms] This sentence should be removed, as this is speculation.
>> 
>> Why? It states an intent… and that’s the intent that we have.
>> 
>>> 
>>>   Until the AccECN experiment succeeds, [RFC3168] will remain as the
>>>   standards track specification for adding ECN to TCP.
>>> 
>>> [ms] This sentence should be removed (or reworded)
>> 
>> Why? Does it help to add an only here:
>> 
>> "Until the AccECN experiment succeeds, [RFC3168] will remain as the only
>> standards track specification for adding ECN to TCP.“
> 
> This wording is better.
> 
>>>   AccECN feedback overloads flags and fields in the main TCP header
>>>   with new definitions, so both ends have to support the new wire
>>>   protocol before it can be used.
>>> 
>>> [ms] In my reading this experimental document asks for *new* allocation
>> of a reserved TCP header flag.
>> 
>> Is this better?
>> 
>> "AccECN feedback overloads the two existing ECN flags as well as the
>>      currently reserved and previously called NS flag in the main TCP header
>>      with new definitions, so both ends have to support the new wire protocol
>>      before it can be used.“
>> 
>> I understand that you are not happy with the word „overload“ here but the
>> point of this sentence really is that the flags can/could be used differently
>> and therefore we need a new negotiation before we can use them.
> 
> For me the following would work: "AccECN feedback overloads the two existing ECN flags and 
> allocates the currently reserved and previously called NS flag in the main TCP header.
> Given the new definitions, both ends have to support the new wire protocol
> before it can be used."
> 
> I believe the wording has to be crystal clear on the reservation of bit 7 when it is discussed the first time in the text. In follow-up sections, maybe shorter terms could be used.

Okay, now:

"AccECN feedback overloads the two existing ECN flags and
    allocates the currently reserved and previously called NS flag in the
    TCP header, to be used as one field indicating the number of congestion
    experienced marked packets. Given the new definitions of these three bits,
    both ends     have to support the new wire protocol before it can be used.
    Therefore during the TCP handshake the two ends use these three bit in
    the TCP header to negotiate the most advanced feedback protocol
    that they can both support in a backward compatible way to 
    <xref target="RFC3168"/>."
> 
>> If you prefer, we can also remove the NS flag in this list, as ECN Nonce was
>> anyway never deployed.
>> 
>>> 
>>>   For that we refer to [RFC3168] or any RFC that
>>>   specifies a different response to TCP ECN feedback, for example:
>>>   [RFC8257]; or the ECN experiments referred to in
>>>   [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low Latency
>>>   Low Loss Scalable (L4S) congestion control [I-D.ietf-tsvwg-l4s-arch];
>>>   ECN-capable TCP control packets [I-D.ietf-tcpm-generalized-ecn], or
>>>   Alternative Backoff with ECN (ABE)
>>>   [I-D.ietf-tcpm-alternativebackoff-ecn].
>>> 
>>> [ms] At least ABE seems orthogonal. Anyway, I think this paragraph can just
>> be deleted. If other experiments need more accurate feedback, it is up to
>> them to explain how they would use this mechanism. This document should
>> focus on how to signal the feedback, not how to use that.
>> 
>> Yes, that is what the paragraph says. Isn’t it better to be explicit about this?
>> 
>>> 
>>>   It is likely (but not required) that the AccECN protocol will be
>>>   implemented along with the following experimental additions to the
>>>   TCP-ECN protocol: ECN-capable TCP control packets and retransmissions
>>>   [I-D.ietf-tcpm-generalized-ecn], which includes the ECN-capable SYN/
>>>   ACK experiment [RFC5562]; and testing receiver non-compliance
>>>   [I-D.moncaster-tcpm-rcv-cheat].
>>> 
>>> [ms] I am a big fan of simple, standalone documents. In my view, the TCPM
>> working group should publish draft-ietf-tcpm-accurate-ecn and draft-ietf-
>> tcpm-generalized-ecn independent documents, which probably implies that
>> draft-ietf-tcpm-generalized-ecn does not use AccECN. If experimentation
>> with ECT in SYN requires a combination, this could be done in a new, third
>> document. Apart from having simpler focused documents, this could
>> significantly help later with moving forward documents to standards track.
>> 
>> I disagree, however, this is a discussion to have on draft-ietf-tcpm-
>> generalized-ecn. I don’t see a problem in  providing a reference here that
>> says „it is likely…“ and nothing more.
>>> 
>>> 
>>> 
>>> * 1.1.  Document Roadmap
>>> 
>>> [ms] A macroscopic comment is that this document has a lot of introduction
>> and tutorial text with lot's of redundancy towards other documents. I think
>> the document can be made much easier to read by shorten it. In many cases
>> this is just an editorial change as there is redundancy. As one such example,
>> just remove this section.
>> 
>> I guess this a matter of taste. As an AD, I’m a big fan of short and concise
>> documents, however, some redundancy can also help understanding,
>> especially if you explain things multiple times but with a different level of
>> detail. I personally would not need the roadmap but I know many people
>> who find these things helpful and to be honest I don’t see how removing this
>> part makes the doc any better. If you don’t want it, don’t read it.
>> 
>>> 
>>> 
>>> * 1.2.  Goals
>>> 
>>> [ms] I think this section can also just be removed.
>> 
>> I have to say I also don’t see the point of removing this part. Given we’ve
>> done the work on requirements, I think we should also link to this doc
>> somewhere.
>>> 
>>> 
>>> * 1.3.  Experiment Goals
>>> 
>>>   TCP is critical to the robust functioning of the Internet, therefore
>>>   any proposed modifications to TCP need to be thoroughly tested. The
>>>   present specification describes an experimental protocol that adds
>>>   more accurate ECN feedback to the TCP protocol.  The intention is to
>>>   specify the protocol sufficiently so that more than one
>>>   implementation can be built in order to test its function, robustness
>>>   and interoperability (with itself and with previous version of ECN
>>>   and TCP).
>>> 
>>> [ms] I think all what is written in this paragraph is obvious, no? Can't we just
>> delete this?
>> 
>> Sure, however, I don’t think it hurts to spell it out. For me both is fine, keep it
>> or remove it.
>> 
>>> 
>>>   The experimental protocol will be considered successful if it is
>>>   deployed and if it satisfies the requirements of [RFC7560] in the
>>>   consensus opinion of the IETF tcpm working group.  In short, this
>>>   requires that it improves the accuracy and timeliness of TCP's ECN
>>>   feedback, as claimed in Section 5, while striking a balance between
>>>   the conflicting requirements of resilience, integrity and
>>>   minimisation of overhead.  It also requires that it is not unduly
>>>   complex, and that it is compatible with prevalent equipment
>>>   behaviours in the current Internet (e.g. hardware offloading and
>>>   middleboxes), whether or not they comply with standards.
>>> 
>>>   Testing will mostly focus on fall-back strategies in case of
>>>   middlebox interference.  Current recommended strategies are specified
>>>   in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>   these strategies depends on the actual deployment situation of
>>>   middleboxes.  Therefore experimental verification to confirm large-
>>>   scale path traversal in the Internet is needed before finalizing this
>>>   specification on the Standards Track.
>>> 
>>> [ms] These two paragraphs must be entirely rewritten. As I have
>> mentioned before, I don't think an RFC should speculate about TCPM and its
>> consensus opinion. I would suggest a wording along the lines of:
>>> 
>>> <ms>
>>>   The experimental protocol will be considered successful if
>>>   testing confirms that the proposed mechanism can be deployed at large
>> scale.
>>>   Testing will mostly focus on fall-back strategies in case of
>>>   middlebox interference.  Current recommended strategies are specified
>>>   in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>   these strategies depends on the actual deployment situation of
>>>   middleboxes.  Therefore experimental verification to confirm large-
>>>   scale path traversal in the Internet is needed, e.g., by support in
>>>   major TCP stacks.
>>> </ms>
>>> 
>> I don’t understand your point here. I don’t think that the paraphrase
>> speculates about the consensus of tcpm, in contrast it say tcpm has to
>> decided if the requirements previously specified by tcpm are sufficiently
>> fulfilled. I don’t see a reason to not mention the requirement draft as this
>> draft as tcpm consensus and was written for this purpose.
> 
> My suggested wording uses the expression "can be deployed at large scale" and I believe this is relevant.
> 
> The document already describes in Section 5 how the protocol satisfies the agreed requirements for a more accurate ECN feedback protocol [RFC7560]. So, if the TCPM working group publishes this document with the content of Section 5, I believe the TCPM working group already has reached consensus that the protocol meets requirements. In addition, it is possible that new requirements would be identified in future, e.g., as an outcome of the experiment, and that would obviously have to be considered by TCPM. In that case, for the success of the experiment not only RFC 7560 would matter, but also further requirements. My proposed wording does not have all these problems.
> 
> In a nutshell, I continue to believe that this section has to change.

Okay, used your proposed wording. You have a point about the requirement and I mis-read you proposal earlier as „has to be deployed large-scale“.

> 
>>> * 1.5.  Recap of Existing ECN feedback in IP/TCP
>>> 
>>> [ms] This section could probably be shortened as well.
>>> 
>>>   The last bit in byte 13 of the TCP header was defined as the Nonce
>>>   Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never deployed so
>>>   it is being reclassified as historic, making this TCP flag available
>>>   for use by the AccECN experiment instead.
>>> 
>>> [ms] This wording, as well as Figure 1, needs to take into account the IANA
>> status when draft-ietf-tsvwg-ecn-experimentation is published.
>> 
>> Is does. However, I can explicitly say that is has be re-clssified as reserved.
>> 
>> "RFC 3540 was never deployed so it is being reclassified as historic [I-D.ietf-
>> tsvwg-ecn-experimentation] and the respective flag has been marked as
>> „reserved“ in the IANA TCP Header Flags registry, making this TCP flag
>> available for use by the AccECN experiment instead.“
>> 
>> Better?
>> 
>>> In my understanding, this experimental document asks for new assignment
>> of a reserved TCP header flag.
>> 
>> As I said I’m not sure if we have fully concluded this discussion yet. However,
>> what we really would want to is mention somewhere that this experiment
>> with this flags is running. I guess there are three options:
>> 1) keep it in the registry as reserved and conserve the knowledge in tcpm
>> that this experiment is running and no other experimental RFC such use this
>> flags as long as this experiment is running.
>> 2) Keep is marked as reserved but add a note about this experiment in the
>> IANA registry
>> 3) Or assign it right away with IESG approval. I guess in this case tcpm could
>> also consider to change the registration policy to „IETF Review“.
> 
> The current registration policy for the TCP header flags is "standards action". I understand that the IESG could approve exceptions. But given the policy, I believe the document has to be very precise on the request regarding bit 7.

Okay, it now says:

"[TO BE REMOVED: IANA is requested to update the existing entry in the Transmission Control Protocol (TCP) Header Flags registration (https://www.iana.org/assignments/tcp-header-flags/tcp-header-flags.xhtml#tcp-header-flags-1) for Bit 7 to "AE (Accurate ECN), previously used by Historic as NS (Nonce Sum) [RFC3540, RFC8311]" and change the reference to this RFC-to-be instead of RFC8311.]“

I guess we could also ask IANA to add an additional comment column instead (but not sure if we then have to update RFC3168, which I think we really don’t want. Should be fine now.

> 
>>> * 2.  AccECN Protocol Overview and Rationale
>>> 
>>>   o  an essential part that re-uses ECN TCP header bits to feed back
>>>      the number of arriving CE marked packets.  This provides more
>>>      accuracy than classic ECN feedback, but limited resilience against
>>>      ACK loss;
>>> 
>>> [ms] The word "re-use" is IMHO not correct.
>> 
>> I think this is nit picking. Using a different phrasing here makes the sentence
>> unnecessary complicated. We don’t try to some how get a round the fact
>> that we need to handle the flag registration correctly. However, here the
>> point really is to explain how the protocol word. The main point of using the
>> work „re-use“ here is really that we say that these flags are or have been
>> used different by other TCP extension (and we therefore need a proper
>> negotiation scheme).
> 
> If the allocation of a reserved flag is correctly explained in the abstract and introduction, I think these sentences can use a bit relaxed terminology.
> 
>>>   The two part design was necessary, given limitations on the space
>>>   available for TCP options and given the possibility that certain
>>>   incorrectly designed middleboxes prevent TCP using any new options.
>>> 
>>> [ms] IMHO it would make sense to more explicitly mention the downsides
>> of only specifying an option and not allocating a TCP header flag, in this
>> experimental document.
>> 
>> We need to use the flags (all three of them) for the negotiation.
> 
> I think that an explanation why negotiation by a TCP option would not solve all use cases (or requirements) would help here.
> 
>>> The obvious  alternative would be to postpone the header flag allocation to
>> a follow-up standards track document and just keep it reserved.
>> 
>> We can still do that. We should discuss and make a final decision.
>> 
>>> 
>>>   The essential part overloads the previous definition of the three
>>>   flags in the TCP header that had been assigned for use by ECN.  This
>>>   design choice deliberately replaces the classic ECN feedback
>>>   protocol, rather than leaving classic ECN feedback intact and adding
>>>   more accurate feedback separately because:
>>> 
>>> [ms] Similar like previous comments, in my reading there are only _two_
>> ECN header flags.
>> 
>> I think there are three flags that "had been assigned for use by ECN“ as ECN
>> Nonce is also an ECN mechanism. The fact that one of the flags is now
>> marked as reserved instead, it not that important for me here.
>> 
>>> And, in addition, I think care is needed with wording such "replaces the
>> classic ECN feedback". I don't think this experiment replaces the ECN
>> standards. That would be up to a follow-up PS.
>> 
>> This sentence is not meant to say that RFC3168 is replaced. Actually we don’t.
>> You can still use RFC3168 even if AccECN is implemented and deploy
>> (however, we do intent that AccECN will be used as the default scheme in
>> future and RFC3168 is hopefully simply not needed anymore at some point,
>> even though you probably still need to have it implemented as the
>> negotiation specified in this draft covers that as well, anyway...). The
>> sentence says that if AccECN is negotiation, the header flags as used by
>> RFC3168 and previously ECN Nonce are used differently (aka re-used). That’s
>> all.
>>> 
>>> 
>>> 2.1.  Capability Negotiation
>>> 
>>>   AccECN is a change to the wire protocol of the main TCP header,
>>>   therefore it can only be used if both endpoints have been upgraded to
>>>   understand it.  The TCP client signals support for AccECN on the
>>>   initial SYN of a connection and the TCP server signals whether it
>>>   supports AccECN on the SYN/ACK.  The TCP flags on the SYN that the
>>>   client uses to signal AccECN support have been carefully chosen so
>>>   that a TCP server will interpret them as a request to support the
>>>   most recent variant of ECN feedback that it supports.  Then the
>>>   client falls back to the same variant of ECN feedback.
>>> 
>>> [ms] As this is an experimental specification, I would really like to see a
>> discussion how a future standards track version of more accurate ECN could
>> be negotiated.
>> 
>> As described in this draft. There will be no different. AccECN IS and will
>> always be backward compatible with RFC3168.
> 
> That is not the problem I think about. I wonder about a PS version of accurate ECN feedback that would possibly include changes as compared to this experiment, e.g., because the experiment may have some lessons learnt. What options would we have to negotiate the PS version, and how could a stack implementing the PS version figure out whether the remote end uses the experimental or the PS version of the protocol?
> 
> The reason why I ask is because I don't see any easy solution to this but I may miss something. Maybe this would not be a concern if there were some codepoints left.
> 
>>> How could both endpoints detect whether the other one implements the
>> future standards track version?
>> 
>> If the initiator implements AccECN it will request it’s use. If the receiver also
>> implement it, it will/can negotiate it, if not it will look like an RFC3168 request
>> for the receiver (as the NS flags will be ignored in the SYN) and it will
>> negotiate RFC3168 ECN feedback if implemented. There is no additional
>> detection needed.
> 
> My concern is the migration strategy for a future version, given that we only experiment right now. For instance, if the initiator implements the experimental version but the recipient implements the PS version of AccECN, how would that work? Under the assumption that there are changes, would there be a way to know? Maybe the answer to this question is no, given the small number of bits we have. But not having any room for extensions is not necessarily good protocol design.
> 
>>> For instance, would the only safe variant be that we allocate yet another
>> reserved TCP header flag in a proposed standard to negotiate the standards
>> track version, thus investing another reserved bit in the TCP header?
>> 
>> No, that’s exactly what we use the NS flags for in the handshake.
> 
> If the PS version of AccECN was different to the experimental version of AccECN, and if there was a deployed base of both, I still believe we could end up in a situation in which we had to allocate yet another TCP header flag to distinguish the different versions of AccECN. The NS flag would not work for that if it is used by the experimental version. The risk of having to spend further TCP header flags somehow concerns me. Of course, that problem would only matter if the AccECN experiment succeeds somehow, but lessons learnt would require protocol changes, which is speculation.
> 
>>> I may be wrong, but to me it is too early to speculate how the PS version
>> would look like, and whether it would have to be different to the
>> experimental version, due to lessons learnt.
>> 
>> Of course you can always be wrong. However, the handshake negotiation is
>> not the part we need experimentation for. That part is straight forward and
>> works. If we really happen to detect a problem in that part, we would need
>> to end the experiment declare failure and start over new.
> 
> My concern is ending the experiment when the experiment got (partly) deployed in the Internet. In that case neither a new RFC nor a change of the IANA registry will solve the migration issue.
> 
>>> I believe in the IETF we typically design protocols that allow future
>> extension, and it is not exactly clear to be how AccECN could be extended
>> later.
>> 
>> This is an TCP extension. If we want future extension we use the usually TCP
>> mechanism (by defining a new TCP option I guess).
> 
> The root cause of my concern is that this proposal does *not* use the usual way to experiment with TCP by options. It experiments with a header flag, including in the SYN, and it seems to consume all codepoints. So, I see the risk of a protocol design "not ready for future improvements".
> 
> I cannot easily propose text. Maybe "lack of extensibility" is just one of the short-comings of the protocol design that cannot be avoided but that short-coming would have to be noted. 
> 
>>>   An AccECN TCP client does not send the new AccECN Option on the SYN
>>>   as SYN option space is limited and successful negotiation using the
>>>   flags in the main header is taken as sufficient evidence that both
>>>   ends also support the AccECN Option.  The TCP server sends the AccECN
>>>   Option on the SYN/ACK and the client sends it on the first ACK to
>>>   test whether the network path forwards the option correctly.
>>> 
>>> [ms] For what it is worth, I would personally be quite fine with allowing (or
>> even mandating) an option in the SYN in the experimental version of this
>> protocol. For instance, saving the SYN option space would then an excellent
>> reason for moving towards the PS specification. I am also fine with being in
>> the rough part of the consensus here.
>> 
>> The point is that we really don’t need the option in the SYN as we don’t use it
>> for negotiation purposes as we use the header bits instead. So why should
>> we waste the space?
> 
> For instance, mandating the option in the SYN would be away for the receiver to distinguish the experiment from a follow-up PS version of the spec, as the PS version may not mandate the option, to save header space.
> 
> Maybe that proposal does not make any sense, and it may only have downsides. But the document already speculates about a PS-follow-up. So it seems a valid question to ask if the EXP and the PS version of the spec have to be identical. This all comes down to the SYN negotiation. 
> 
>>> * 2.3.  Delayed ACKs and Resilience Against ACK Loss
>>> 
>>>   If the AccECN Option is not available, e.g. it is being stripped by a
>>>   middlebox, the AccECN protocol will only feed back information on CE
>>>   markings (using the ACE field).  Although not ideal, this will be
>>>   sufficient, because it is envisaged that neither ECT(0) nor ECT(1)
>>>   will ever indicate more severe congestion than CE, even though future
>>>   uses for ECT(0) or ECT(1) are still unclear
>>>   [I-D.ietf-tsvwg-ecn-experimentation].
>>> 
>>> [ms] This needs to be reworded
>> 
>> Why?
>> 
>>> 
>>> 
>>> 
>>> * 2.4.  Feedback Metrics
>>> 
>>>   The CE packet counter in the ACE field and the CE byte counter in the
>>>   AccECN Option both provide feedback on received CE-marks.  The CE
>>>   packet counter includes control packets that do not have payload
>>>   data, while the CE byte counter solely includes marked payload bytes.
>>>   If both are present, the byte counter in the option will provide the
>>>   more accurate information needed for modern congestion control and
>>>   policing schemes, such as DCTCP or ConEx.
>>> 
>>> [ms] I suggest to write in the last sentence only "... the option will provide
>> the more accurate information needed for congestion control". In general, I
>> would prefer to have references to other mechanisms at only few (ideally a
>> *single*) places in the document, instead of mixing them together.
>> 
>> Sorry, I don’t see your point here. ConEx has been mentioned previously, so
>> why not also mention it here.
> 
> As written earlier, in my understanding DCTCP does not "need" this. I would suggest to have one place where to define exactly how this mechanism can be used by other TCP extensions. Having said this, in this specific paragraph I am less concerned about that than elsewhere.
> 
>>>   Feedback in bytes is recommended in order to protect against the
>>>   receiver using attacks similar to 'ACK-Division' to artificially
>>>   inflate the congestion window, which is why [RFC5681] now recommends
>>>   that TCP counts acknowledged bytes not packets.
>>> 
>>> [ms] At least the last part and the reference to RFC 5681 is IMHO not
>> needed here.
>> 
>> Why? RFC5681 explains/refers the ACK division attack, so I think it is a very
>> good reference to have here.
>>> 
>>> 
>>> 
>>> * 2.5.  Generic (Dumb) Reflector
>>> 
>>>   The ACE field provides information about CE markings on both data and
>>>   control packets.  According to [RFC3168] the Data Sender is meant to
>>>   set control packets to Not-ECT.  However, mechanisms in certain
>>>   private networks (e.g. data centres) set control packets to be ECN
>>>   capable because they are precisely the packets that performance
>>>   depends on most.
>>> 
>>>   For this reason, AccECN is designed to be a generic reflector of
>>>   whatever ECN markings it sees, whether or not they are compliant with
>>>   a current standard.  Then as standards evolve, Data Senders can
>>>   upgrade unilaterally without any need for receivers to upgrade too.
>>>   It is also useful to be able to rely on generic reflection behaviour
>>>   when senders need to test for unexpected interference with markings
>>>   (for instance [I-D.kuehlewind-tcpm-ecn-fallback] and
>>>   [I-D.moncaster-tcpm-rcv-cheat]).
>>> 
>>>   The initial SYN is the most critical control packet, so AccECN
>>>   provides feedback on whether it is CE marked.  Although RFC 3168
>>>   prohibits an ECN-capable SYN, providing feedback of CE marking on the
>>>   SYN supports future scenarios in which SYNs might be ECN-enabled
>>>   (without prejudging whether they ought to be).  For instance,
>>>   [I-D.ietf-tsvwg-ecn-experimentation] updates this aspect of RFC 3168
>>>   to allow experimentation with ECN-capable TCP control packets.
>>> 
>>> [ms] To me, the only thing that matters in this document that AccECN can
>> provide feedback on whether the SYN is CE marked. The discussion on how
>> to experiment with ECT e.g. in SYNs IMHO does not belong into this
>> document. So it seems sufficient here to note that one of the benefits of
>> AccECN is that CE marks in SYNs can be fed back.
>> 
>> I disagree. Explicitly saying that AccECN is only an feedback scheme and DOES
>> NOT define how the information is used is VERY important because people
>> come back to me over and over again and mix these things up.
>>> 
>>> 
>>> * 3.1.1.  Negotiation during the TCP handshake
>>> 
>>>   Given the ECN Nonce [RFC3540] is being reclassified as historic, the
>>>   present specification renames the TCP flag at bit 7 of the TCP header
>>>   flags from NS (Nonce Sum) to AE (Accurate ECN) (see IANA
>>>   Considerations in Section 6).
>>> 
>>> [ms] As mentioned before, this needs to be rewritten to ask for new IANA
>> allocation of bit 7 in the TCP header flags.
>> 
>> I really don’t understand this comment. That is what the IANA section does
>> as referred here correctly.
> 
> In my reading of https://www.iana.org/assignments/tcp-header-flags/tcp-header-flags.xhtml, this flag currently has no name, i.e., it does not have the name NS. So this statement on "renaming" is formally incorrect IMHO.
> 
> I'd suggest something like: "This specification assigns the name AE (Accurate ECN) to the TCP flag at bit 7; this flag has previously been known as NS (Nonce Sum) …".

Now:

"Given the ECN Nonce <xref target="RFC3540"/> has been
          reclassified as historic <xref target="RFC8311"/>, the present 		  specification re-allocates the TCP
          flag at bit 7of the TCP header flags, which was previously called NS
          (Nonce Sum), as the AE
          (Accurate ECN) flag (see IANA Considerations in <xref
          target="accecn_IANA_Considerations"/>)."
> 
>> Again, yes, we can discuss if this document should do this, but if published as
>> it is, section 6 says everything that is needs to say. (I does not put any
>> request on IANA, instead it is written as it would show up post-publication,
>> implicitly providing a request to IANA at approval time. That is fine.)
>> 
>>> 
>>>   Table 2: ECN capability negotiation between Client (A) and Server
>>> (B)
>>> 
>>> [ms] As far as I can see, in -05 this table allocates all existing codepoints,
>> while -03 had two currently unused codepoints. Not having any codepoints
>> left seems to me not really future proof, e.g., regarding future proposed
>> standards in this space (and I personally believe that TCP header flags must
>> be allocated in a PS). And I don't fully see a need of feeding back ECT0 and
>> specifically ECT1 in the TCP header flags as part of the experiment. Do we
>> know for sure that this is the only possible use case of these two unallocated
>> header bits? And why can't e.g. this be done in a TCP option instead? Or do I
>> miss something?
>> 
>> The point is really, if we don’t assign them now and start deployment we
>> effetely we not be able to every assign them again because don’t have a
>> different negotiation mechanisms. Realizing this, it is just the right think to
>> define the space completely that is negotiated as use for AccECN in the
>> handshake.
> 
> I disagree that consuming all codepoints and not having room for future extensions is the "right thing" to do. It is in fact the wrong thing. But possibly there is no alternative with the few bits, so maybe we end up doing it.
> 
> Yet, at minimum, using all codepoints needs to be reasoned in the document. Specifically since in -03 a different protocol design seemed possible, which looked to me more future-proof.
> 
>>> * 3.1.2.  Retransmission of the SYN
>>> 
>>>   However,
>>>   current measurements imply that a drop is less likely to be due to
>>>   middlebox interference than other intermittent causes of loss, e.g.
>>>   congestion, wireless interference, etc.
>>> 
>>> [ms] Such wording IMHO doesn't belong into normative text. This may
>> actually also apply to other heuristics discussed in this section, which are not
>> really important for interoperability.
>> 
>> I don’t really understand your point. This sentence is solely meant to reason
>> the design decision to not say that a sender SHOULD attempt to re-
>> negotiation after a loss.
> 
> One could split the reasoning from the normative design. The normative specification may still be relevant in 20 years from now. Middleboxes will almost likely be different in 20 years. Having said this, this is more of an editorial remark.
> 
>>> 3.2.7.  Path Traversal of the AccECN Option
>>> 
>>> 3.2.7.1.  Testing the AccECN Option during the Handshake
>>> 
>>>   The TCP client MUST NOT include the AccECN TCP Option on the SYN.
>>> 
>>> [ms] I am not sure if I really understand the motivation for not allowing a
>> option in the SYN. If the sender has space in the SYN left, what is the harm in
>> an experimental version of the protocol? And I may miss something, but
>> what would prevent the use of 2-byte option to negotiate the use of
>> AccECN, e.g., to avoid experimental allocation of bit 7 in the initial SYN?
>> 
>> I did have a draft on that as a proposal for an alternative design. However,
>> the gourd was more supportive of this design as it is proof of middlebox SYN
>> option mangling which is a know problem.
>> 
>> Therefore we simply don’t need option in the SYN and there is no reason to
>> waste the space.
>> 
>>> While I think many tutorial text in this document could be shortened, I
>> believe the use of a reserved TCP header flag should be reasoned.
>> 
>> I’m actually uncertain what you expect here.
> 
> For instance, this reply with the explanation of SYN option mangling seems useful explanation.
> 
>>> * 3.2.8.  Usage of the AccECN TCP Option
>>> 
>>>   The following rules determine when a Data Receiver in AccECN mode
>>>   sends the AccECN TCP Option, and which fields to include:
>>> 
>>>   Change-Triggered ACKs:  If an arriving packet increments a different
>>>      byte counter to that incremented by the previous packet, the Data
>>>      Receiver MUST immediately send an ACK with an AccECN Option,
>>>      without waiting for the next delayed ACK (this is in addition to
>>>      the safety recommendation in Section 3.2.5 against ambiguity of
>>>      the ACE field).
>>> 
>>>      This is stated as a "MUST" so that the data sender can rely on
>>>      change-triggered ACKs to detect transitions right from the very
>>>      start of a flow, without first having to detect whether the
>>>      receiver complies.  A concern has been raised that certain offload
>>>      hardware needed for high performance might not be able to support
>>>      change-triggered ACKs, although high performance protocols such as
>>>      DCTCP successfully use change-triggered ACKs.
>>> 
>>> [ms] To me this sounds like a perfect example for a SHOULD with additional
>> guidance why implementing this SHOULD is really important.
>> 
>> This is one of the most discussed point from the author and we really tried to
>> get additional guidance here of what to do also from outside the IETF but did
>> no clear feedback.
>> 
>> As explained this MUST enables additional functionality. However, this is an
>> experiment document. If we detect that this MUST does actually hinder
>> implementation or has just never been implemented, we should reconsider
>> this in the final PS RFC.
>> 
>> I was on the SHOULD side of the discussion, but can say that the
>> implementation in Linux was way more simple then expected. Offloading
>> might be a different topic but that is where I could not get a clear feedback
>> and offloading could probably be anyway optimized for use with AccECN (if it
>> gets deploy widely).
> 
> If a MUST requires changes in hardware, I think there must be a clear reason.
> 
> As individual contributor, with the current explanation in the text, I believe this has to be a SHOULD.
> 
>> I though we added this as something to mention in the exp goals section, but
>> obviously we didn’t. I added the following text now:
>> 
>> "Another experimentation focus is the implementation feasibiliy of change-
>> triggered ACKs as described in section 3.2.8. While on average this should not
>> lead to a higher ACK rate, it changes the ACK patter which especially can have
>> an impact on hardware offload. Further experimentation is needed to advise
>> if this should a hard requirement or just prefer behavior.“
> 
> If it is unclear if a MUST can actually be implemented, having a MUST is in my opinion the wrong approach.
> 
> One could equally state here that further experimentation is needed to determine whether the SHOULD can be upgraded to a MUST.

I’m still open for everything here, but I believe this was discussed and agreed in the working group…? 

> 
>>>   For the avoidance of doubt, the change-triggered ACK mechanism is
>>>   deliberately worded to ignore the arrival of a control packet with no
>>>   payload, which therefore does not alter any byte counters, because it
>>>   is important that TCP does not acknowledge pure ACKs.  The change-
>>>   triggered ACK approach will lead to some additional ACKs but it feeds
>>>   back the timing and the order in which ECN marks are received with
>>>   minimal additional complexity.
>>> 
>>> [ms] The additional acks create network load. I think some wording is
>> needed on the tradeoff between information accuracy and network load.
>> There are network environments in which any additional packet is very
>> expensive (e.g., energy) and it is not clear to me how the protocol design
>> takes into account the potential overhead of additional ACKs. Maybe this
>> could be another reason for a SHOULD.
>> 
>> The above. However, this is not really an additional ACK because you do
>> delay the next one. Further experimentation needed.
> 
> The document states "lead to some additional ACKs". If that does not increase network load, I think it has to be explicitly explained why the ACK load is at most equal to a current TCP stack, in all potential cases. If it can increases network load, it has to be reasoned why increasing load (and risk of reverse congestion and the like) is worth the effort.
> 
> I agree that this may be an area of experimentation, but I believe then it has to be explained to implementers what the tradeoffs are.

Added:
"Especially, if only few
          CE marks occurred or multiple marks in a row, the additional load will
          be low. Other, unexpected marking pattern could increase the load
          significantly, however, investigating the additional load it part
          of the proposed experimentation."
> 
>>> * 4.2.  Compatibility with Other TCP Options and Experiments
>>> 
>>>   AccECN is compatible (at least on paper) with the most commonly used
>>>   TCP options: MSS, time-stamp, window scaling, SACK and TCP-AO.  It is
>>>   also compatible with the recent promising experimental TCP options
>>>   TCP Fast Open (TFO [RFC7413]) and Multipath TCP (MPTCP [RFC6824]).
>>> 
>>> [ms] I would suggest the wording "... compatible with the experimental
>> TCP options ..." or even "... compatible with the TCP options …".
>> 
>> These option are to experimental..?
> 
> "It is also compatible with the experimental TCP options TCP Fast Open (TFO [RFC7413]) and Multipath TCP (MPTCP [RFC6824])." would have same technical meaning. Having said this, this is editorial only.
> 
>> The point of using „commonly used“ was to say that we checked on those as
>> they seem important. Just because your favorite experiment option is not
>> listed here it doesn’t means it incompatible, we just didn’t check. I’m okay to
>> removed „commonly used“ but I don’t think it makes anything better.
>> 
>>> 
>>> * 4.3.  Compatibility with Feedback Integrity Mechanisms
>>> 
>>> [ms] Quite a bit in this section is experimental work, which IMHO
>>> should be clearly emphasized. The one exception is…
>> 
>> I would really like to keep this section because integrity is usually the fist
>> question that come up when I present AccECN. Effectively these are two
>> independent topics, however, I really think it help people to understand the
>> whole picture if this is also discussed in this document.
>> 
>>> 
>>>      However, TCP-AO is often too brittle to use on many end-to-end
>>>      paths, where middleboxes can make verification fail in their
>>>      attempts to improve performance or security, e.g. by
>>>      resegmentation or shifting the sequence space.
>>> 
>>> [ms] I am not sure if deployment challenges of other options need to be
>> discussed in this document.
>> 
>> If we keep the discussion, I guess we should mention this as well. As the doc
>> clearly stated section 4 is not meant to be normative.
> 
> As far as I can see, there are use cases for TCP-AO where middleboxes are simply not a problem, but it is exactly this sort of discussion that may not be needed in this document. But I won't rat-hole on this comment here.
> 
>>>   Originally the ECN Nonce [RFC3540] was proposed to ensure integrity
>>>   of congestion feedback.  With minor changes AccECN could be optimised
>>>   for the possibility that the ECT(1) codepoint might be used as an ECN
>>>   Nonce . However, given RFC 3540 is being reclassified as historic,
>>>   the AccECN design has been generalised so that it ought to be able to
>>>   support other possible uses of the ECT(1) codepoint, such as a lower
>>>   severity or a more instant congestion signal than CE.
>>> 
>>> [ms] The discussion of RFC 3540 can probably be removed to a large extent.
>> 
>> Unfortunately, please still think that ECN Nonce, even though is was never
>> deployed and doesn’t really work, is the only was to provide integrity
>> protection and we need it as a prerequisite to deploy ECN at all… i would
>> really prefer to keep this in this non-normative part of the doc.
>> 
>>> 
>>> 
>>> * 6.  IANA Considerations
>>> 
>>> [ms] I think this section needs to be rewritten to request a new allocation
>> of bit 7 of the TCP header flags. At least for the process I think it would make
>> sense to have somewhere in the document a comprehensive explanation of
>> why an experimental document requests a change of the main TCP header,
>> and why this cannot be avoided (most notably in the initial SYN) by an
>> alternative protocol design.
>> 
>> As I said previously this section is written as it would look like after the
>> allocation has happened with publication approval of the IESG. This is fine.
>> 
>> Having a discussion about an experiment doc assigning a flag (or not) is a
>> question for tcpm as a whole and not specifically this document. How do we
>> envision to every use any further flags? We go to PS right away? Or should
>> we change the registration policy? For me the latter makes actually more
>> sense. However, if we don’t want/can to decide this now, we also could go
>> forward as it is with IESG approval. However, is this case it is also not needed
>> to explain this in the document. The responsible AD has to explain this to the
>> other IESG probably in the ballot or even better the shepherd could provide
>> these information in the write-up.
>> 
>>> 
>>> 
>>> * 9.  Comments Solicited
>>> 
>>>   Comments and questions are encouraged and very welcome.  They can
>> be
>>>   addressed to the IETF TCP maintenance and minor modifications working
>>>   group mailing list <[email-masked]>, and/or to the authors.
>>> 
>>> [ms] This section is not needed IMHO
>> 
>> Yes, it will be removed before publication.
>>> 
>>> 
>>> 10.  References
>>> 
>>>   [I-D.ietf-tsvwg-ecn-experimentation]
>>>              Black, D., "Relaxing Restrictions on Explicit Congestion
>>>              Notification (ECN) Experimentation", draft-ietf-tsvwg-ecn-
>>>              experimentation-07 (work in progress), October 2017.
>>> 
>>> [ms] Normative reference?
>> 
>> Don’t see why. No need to read ietf-tsvwg-ecn-experimentation to
>> understand the spec in this doc.
>> 
>> Thanks!
>> Mirja
>> 
>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Jul 2018 19:00:32 +0100

Michael, tcpm list,

As well as addressing your points, as Mirja has already mentioned below, 
we added a whole new appendix giving the rationale for the bits and 
codepoints that AccECN has proposed to use on 1) the SYN and 2) SYN/ACK. 
A 3rd subsection also identifies space for future evolution. It also 
points to where rationale was already given in the body of the draft.

The appendix is in the draft submitted last week, available here:
https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07#appendix-B

We'd be interested to hear whether this allays your concerns.

We have asked to present this in Montreal as well.

Cheers

Bob

On 02/07/18 16:54, Mirja Kühlewind wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 11 Jul 2018 11:35:58 -0700

Hi Bob,

Neal and I evaluated the earlier draft. It is well-thought out but
we're concerned about the options. Option is not mandatory but the
lack of it also reduces accuracy. Option runs into space issues w/
SACK and offload issues w/ TSO/GRO. They can be addressed for sure but
aren't easy.

We're curious how much more "accuracy" it buys over current
DCTCP-style ECN. Is there any study to show trade-offs of
full-ACE-w-options vs ACE-wo-options vs current DCTCP-ECN?

On Wed, Jul 11, 2018 at 11:00 AM, Bob Briscoe <[email-masked]> wrote:
> Michael, tcpm list,
>
> As well as addressing your points, as Mirja has already mentioned below, we
> added a whole new appendix giving the rationale for the bits and codepoints
> that AccECN has proposed to use on 1) the SYN and 2) SYN/ACK. A 3rd
> subsection also identifies space for future evolution. It also points to
> where rationale was already given in the body of the draft.
>
>> For the other, bigger comments regarding extensibility, that I did not yet
>> address below, we plan to add a new section to the appendix to explain
>> extensibility options as previously discussed by mail. We will probably send
>> a separate email on that part.
>>
>> Mirja
>>
>>
>>> In my understanding (as a non-native speaker), the use of the word "need"
>>> is not correct here. DCTCP as specified in RFC 8257 can be implemented
>>> without any such mechanism.
>>>
>>> What would work for me is something of the form "... Data Center TCP
>>> cannot get precise ECN feedback whenever more than one marking is received
>>> in one RTT“.
>>
>> This is not correct. DCTP need more than one feedback signal per RTT and
>> therefore cannot use RFC3168; instead it implement it’s own feedback
>> mechanism. However, to avoid confusion such that people could assume DCTP
>> would not work without the accECN scheme as specified in this doc, I
>> rephrased to:
>>
>> "Recently, proposed
>>> I believe this question will have to be reviewed during WGLC and, more
>>> importantly, IETF last call. For the moment, my concern is that the document
>>> correctly describes the IANA allocation.
>>>
>>> I would like to see here a statement such as : "Given TCP header space is
>>> scarce, this specification allocates a reserved header bit and overloads the
>>> two ECN flags in the TCP header ...“.
>>
>> A bit lengthy but now:
>>
>>> See my proposal for the abstract. I continue to disagree with the term
>>> "need" but I think this can be sorted out by another term.
>>>
>>>> protocol
>>> main TCP header.
>>> I believe the wording has to be crystal clear on the reservation of bit 7
>>> when it is discussed the first time in the text. In follow-up sections,
>>> maybe shorter terms could be used.
>>
>> Okay, now:
>>
>>> The document already describes in Section 5 how the protocol satisfies
>>> the agreed requirements for a more accurate ECN feedback protocol [RFC7560].
>>> So, if the TCPM working group publishes this document with the content of
>>> Section 5, I believe the TCPM working group already has reached consensus
>>> that the protocol meets requirements. In addition, it is possible that new
>>> requirements would be identified in future, e.g., as an outcome of the
>>> experiment, and that would obviously have to be considered by TCPM. In that
>>> case, for the success of the experiment not only RFC 7560 would matter, but
>>> also further requirements. My proposed wording does not have all these
>>> problems.
>>>
>> Okay, used your proposed wording. You have a point about the requirement
>> and I mis-read you proposal earlier as „has to be deployed large-scale“.
>>
>>> The current registration policy for the TCP header flags is "standards
>>> action". I understand that the IESG could approve exceptions. But given the
>>> policy, I believe the document has to be very precise on the request
>>> regarding bit 7.
>>
>> Okay, it now says:
>>
>> "[TO BE REMOVED: IANA is requested to update the existing entry in the
>> Transmission Control Protocol (TCP) Header Flags registration
>> (https://www.iana.org/assignments/tcp-header-flags/tcp-header-flags.xhtml#tcp-header-flags-1)
>> for Bit 7 to "AE (Accurate ECN), previously used by Historic as NS (Nonce
>> Sum) [RFC3540, RFC8311]" and change the reference to this RFC-to-be instead
>> of RFC8311.]“
>>
>> I guess we could also ask IANA to add an additional comment column instead
>> (but not sure if we then have to update RFC3168, which I think we really
>> don’t want. Should be fine now.
>>
>>> If the allocation of a reserved flag is correctly explained in the
>>> abstract and introduction, I think these sentences can use a bit relaxed
>>> terminology.
>>>
>>> I think that an explanation why negotiation by a TCP option would not
>>> solve all use cases (or requirements) would help here.
>>>
>>> That is not the problem I think about. I wonder about a PS version of
>>> accurate ECN feedback that would possibly include changes as compared to
>>> this experiment, e.g., because the experiment may have some lessons learnt.
>>> What options would we have to negotiate the PS version, and how could a
>>> stack implementing the PS version figure out whether the remote end uses the
>>> experimental or the PS version of the protocol?
>>>
>>> The reason why I ask is because I don't see any easy solution to this but
>>> I may miss something. Maybe this would not be a concern if there were some
>>> codepoints left.
>>>
>>> My concern is the migration strategy for a future version, given that we
>>> only experiment right now. For instance, if the initiator implements the
>>> experimental version but the recipient implements the PS version of AccECN,
>>> how would that work? Under the assumption that there are changes, would
>>> there be a way to know? Maybe the answer to this question is no, given the
>>> small number of bits we have. But not having any room for extensions is not
>>> necessarily good protocol design.
>>>
>>> If the PS version of AccECN was different to the experimental version of
>>> AccECN, and if there was a deployed base of both, I still believe we could
>>> end up in a situation in which we had to allocate yet another TCP header
>>> flag to distinguish the different versions of AccECN. The NS flag would not
>>> work for that if it is used by the experimental version. The risk of having
>>> to spend further TCP header flags somehow concerns me. Of course, that
>>> problem would only matter if the AccECN experiment succeeds somehow, but
>>> lessons learnt would require protocol changes, which is speculation.
>>>
>>> My concern is ending the experiment when the experiment got (partly)
>>> deployed in the Internet. In that case neither a new RFC nor a change of the
>>> IANA registry will solve the migration issue.
>>>
>>> The root cause of my concern is that this proposal does *not* use the
>>> usual way to experiment with TCP by options. It experiments with a header
>>> flag, including in the SYN, and it seems to consume all codepoints. So, I
>>> see the risk of a protocol design "not ready for future improvements".
>>>
>>> I cannot easily propose text. Maybe "lack of extensibility" is just one
>>> of the short-comings of the protocol design that cannot be avoided but that
>>> short-coming would have to be noted.
>>>
>>> For instance, mandating the option in the SYN would be away for the
>>> receiver to distinguish the experiment from a follow-up PS version of the
>>> spec, as the PS version may not mandate the option, to save header space.
>>>
>>> Maybe that proposal does not make any sense, and it may only have
>>> downsides. But the document already speculates about a PS-follow-up. So it
>>> seems a valid question to ask if the EXP and the PS version of the spec have
>>> to be identical. This all comes down to the SYN negotiation.
>>>
>>> As written earlier, in my understanding DCTCP does not "need" this. I
>>> would suggest to have one place where to define exactly how this mechanism
>>> can be used by other TCP extensions. Having said this, in this specific
>>> paragraph I am less concerned about that than elsewhere.
>>>
>>> In my reading of
>>> https://www.iana.org/assignments/tcp-header-flags/tcp-header-flags.xhtml,
>>> this flag currently has no name, i.e., it does not have the name NS. So this
>>> statement on "renaming" is formally incorrect IMHO.
>>>
>>> I'd suggest something like: "This specification assigns the name AE
>>> (Accurate ECN) to the TCP flag at bit 7; this flag has previously been known
>>> as NS (Nonce Sum) …".
>>
>> Now:
>>
>>> I disagree that consuming all codepoints and not having room for future
>>> extensions is the "right thing" to do. It is in fact the wrong thing. But
>>> possibly there is no alternative with the few bits, so maybe we end up doing
>>> it.
>>>
>>> Yet, at minimum, using all codepoints needs to be reasoned in the
>>> document. Specifically since in -03 a different protocol design seemed
>>> possible, which looked to me more future-proof.
>>>
>>> One could split the reasoning from the normative design. The normative
>>> specification may still be relevant in 20 years from now. Middleboxes will
>>> almost likely be different in 20 years. Having said this, this is more of an
>>> editorial remark.
>>>
>>> One could equally state here that further experimentation is needed to
>>> determine whether the SHOULD can be upgraded to a MUST.
>>
>>> The document states "lead to some additional ACKs". If that does not
>>> increase network load, I think it has to be explicitly explained why the ACK
>>> load is at most equal to a current TCP stack, in all potential cases. If it
>>> can increases network load, it has to be reasoned why increasing load (and
>>> risk of reverse congestion and the like) is worth the effort.
>>>
>>> I agree that this may be an area of experimentation, but I believe then
>>> it has to be explained to implementers what the tradeoffs are.
>>
>> Added:
>> "Especially, if only few
>>> "It is also compatible with the experimental TCP options TCP Fast Open
>>> (TFO [RFC7413]) and Multipath TCP (MPTCP [RFC6824])." would have same
>>> technical meaning. Having said this, this is editorial only.
>>>
>>> As far as I can see, there are use cases for TCP-AO where middleboxes are
>>> simply not a problem, but it is exactly this sort of discussion that may not
>>> be needed in this document. But I won't rat-hole on this comment here.
>>>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Jul 2018 23:52:46 +0100

Yuchung,

> SACK and offload issues w/ TSO/GRO. They can be addressed for sure but
We had various ideas for smart schemes to reduce the size of the option 
(the fields for the counters don't always need to be so wide and you 
often only need one field). However, we ended up opting for simplicity 
at the expense of space. If you're interested, we could dig out the 
ideas we had (between the co-authors) and see if you would prefer fewer 
option bytes but less simplicity.

I assume you saw the rules around when an option can be suppressed. 
Search for the text around "Receiver MUST give precedence to SACK 
information about loss."

Is there a specific issue with the option and TSO/GRO, or just that the 
h/w wouldn't understand what to do with this option?

> full-ACE-w-options vs ACE-wo-options vs current DCTCP-ECN?
There's a brief summary of the problem with DCTCP-style feedback in the 
appendix of the AccECN Requirements RFC here:
https://tools.ietf.org/html/rfc7560#appendix-A

Essentially, the DCTCP scheme relies on the sender using the feedback to 
keep its copy of the receiver's state machine in sync, which can get out 
of sync in the presence of pure ACK losses and therefore become highly 
ambiguous. Even if there are no ACK losses for a while, it can look like 
a gap could have been caused by a loss - the sender cannot guess what 
might have happened.

The state machine gets back into sync once losses stop. However, DCTCP 
feedback is change-triggered so the sender cannot predict which ACKs it 
should receive and which not. That combines with 2 potential reasons for 
a gap in the ACK stream: delayed ACK; or a lost pure ACK. All three 
factors combined cause considerable ambiguity.

Bob

>>>>>>     incorrectly designed middleboxes prevent TCP using any new options

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 12 Jul 2018 06:23:14 -0400

Hi Yuchung,

the question if you need this „more“ on accuracy really depends on the use case. If you use DCTCP as today, the ACE counter in the TCP is probably sufficient and you might not want to pay the additional overhead in your data center (that why the option is actually optional).

If you however, e.g., have very differently sized packets, then the byte counter in the option could give you a more accurate signal. Or if you are also interested in the ECT(1) counter, you need the option. Further the ACE counter also give your feedback on control packet and the option enables you to distinguish between CE-amrked payload and control packets, which can also become important when experimenting with making all packets ECN-capable. 

Given accECN is a general feedback mechanism that in fact is designed to enable new future uses of the ECN signal, we wanted to keep all these option available while making is still as simple as possible and as flexible as possible.

Mirja

> Am 11.07.2018 um 14:35 schrieb Yuchung Cheng <[email-masked]>:
> 
> Hi Bob,
>

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 12 Jul 2018 08:41:27 -0700

Yes packets with ACE options and (different) ACE-counter header would
break GRO. There're many legacy h/w and s/w.

Even without option, ACE header only allows reflecting up to 8 packets
for receiver segmentation offload and ACK suppression?

The appendix on ack loss causes ambiguity is good: but the pattern
drops specifically the state-switch (CE<->noCE) ACK which only happens
on delayed ACK. Is that pattern common? - or can we address most of
that by delaying less ACKs?

I am all for making ECN more accurate for wide area beyond DCTCP, but
am evaluating the pros/cons. What if we just
1. negotiate 'better' ECN via new SYN option
2. mark all packets like ECN++

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 12 Jul 2018 12:03:52 -0400

Hi Yuchung,

please see below.

> Am 12.07.2018 um 11:41 schrieb Yuchung Cheng <[email-masked]>:
> 

I’m not the expert here but my understanding is that is would not break GRO but could not make actual use or it if the ACE counter changed or the option is present. However, usually your will only have a small number of CE marks every couple of RTTs, and the counter only changes if you CE marks/the option is only present a few times per RTT. So the impact should be rather low. But this clear something to evaluate for the experiment.

> for receiver segmentation offload and ACK suppression?

Same as above, it can only up to 8 CE marked packets per ACK, however, CE marking rater are expected to be rather low. ACK suppression should probably not be used if the ACE counter has changes, however, usually the ACE counter stays stable for multiple RTTs and then ACK suppression is not a problem. However, not sure I understood you question here correctly…?

> that by delaying less ACKs?

I guess you have a 50% chance today to hit a delayed ACK. I guess delaying less (where you delay every second ACK today) would mean ACK very packet, and thus double ACK load on the network. I guess on today networks that actually in most cases not a problem, however, there might be specially cases where it is. However, that’s problem an independent question to evaluate.

> 1. negotiate 'better' ECN via new SYN option

Not sure I understand your proposal correctly, but I assume you mean negotiate via SYN option and then just use the ACE counter in the header? If so, I don’t understand why you see the negotiation part in the TCP header as the problem?

> 2. mark all packets like ECN++

That would be nice but here you actually need AccECN because you want to have feedback for control packets as well. AccECN is providing this feedback. AccECN does not change the „use of ECN“; that’s what we have ECN++ for; both thing ideally would be deployed together however. Was that your questions?

Mirja

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 12 Jul 2018 16:25:41 +0000

I had similar concerns with the new TCP option at the last tcpm meeting. I was requesting that we make the option truly optional but I got some feedback that implementations should honor and process the option on receive even if they don’t generate it. I'd prefer if we make the accurate measurements part optional. The biggest advantage of Accurate ECN is the negotiation mechanism that DCTCP lacks and the same negotiation can be used on the Internet for any scalable TCP that wants to rely on accurate ECN.

Re. GRO/LRO/RSC: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/exception-conditions-that-terminate-coalescing see condition 4 for an exception that terminates coalescing: "The segment contains one or more TCP options other than the TCP timestamp option". So while software implementations of coalescing can be updated, existing hardware will cause poor performance with new TCP options that are present in a lot of data packets. 

>> because you want to have feedback for control packets as well
I don’t fully understand this. ECN++ allows marking of control packets and defines a response. Do we really need the TCP option for processing the feedback?

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Thursday, July 12, 2018 9:04 AM
To: Yuchung Cheng <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

Hi Yuchung,

please see below.

> am evaluating the pros/cons. What if we just 1. negotiate 'better' ECN 
> via new SYN option

Mirja

>>> SACK and offload issues w/ TSO/GRO. They can be addressed for sure 
>>> but aren't easy.
>>> 
>>>> As well as addressing your points, as Mirja has already mentioned 
>>>> below, we added a whole new appendix giving the rationale for the 
>>>> bits and codepoints that AccECN has proposed to use on 1) the SYN 
>>>> and 2) SYN/ACK. A 3rd subsection also identifies space for future 
>>>> evolution. It also points to where rationale was already given in the body of the draft.
>>>> 
>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>>> ols.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-accurate-ecn-07%23appendix-B&
>>>> amp;data=02%7C01%7Cpravb%40microsoft.com%7C2e4980f91d89410fa58708d5
>>>> e8111fbb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6366700826104
>>>> 53620&amp;sdata=J5qwrZ8H%2BbfriTF1jxJXZldX6%2BA0Uy%2FTV0lGdH2UbWY%3
>>>> D&amp;reserved=0
>>>> 
>>>>> For the other, bigger comments regarding extensibility, that I did 
>>>>> not yet address below, we plan to add a new section to the 
>>>>> appendix to explain extensibility options as previously discussed 
>>>>> by mail. We will probably send a separate email on that part.
>>>>> 
>>>>> Mirja
>>>>> 
>>>>> 
>>>>>> Am 12.03.2018 um 01:59 schrieb Scharf, Michael (Nokia - 
>>>>>> DE/Stuttgart)
>>>>>> Yet, I continue to believe that some formal wording in the 
>>>>>> document needs to change, as explained below.
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Michael (with no hat on)
>>>>>> 
>>>>>> 
>>>>>>> document independent of the review from Gorry. I apologize if 
>>>>>>> there is duplication.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> Michael (with no hat on)
>>>>>>>> 
>>>>>>>> 
>>>>>>>>  (DCTCP) need more accurate ECN feedback information whenever 
>>>>>>>> more  than one marking is received in one RTT.
>>>>>>>> 
>>>>>>> remove this, or replace it by a more generic statement that more 
>>>>>>> accurate information can be useful for several TCP extensions.
>>>>>>> 
>>>>>>> I disagree. Both ConEx and DCTCP need more accurate information. 
>>>>>>> They do not need the mechanism that is specified in this draft, 
>>>>>>> however, this is not what the sentences is saying.
>>>>>> 
>>>>>> is not correct here. DCTCP as specified in RFC 8257 can be 
>>>>>> implemented without any such mechanism.
>>>>>> 
>>>>>> What would work for me is something of the form "... Data Center 
>>>>>> TCP cannot get precise ECN feedback whenever more than one 
>>>>>> marking is received in one RTT“.
>>>>> 
>>>>> This is not correct. DCTP need more than one feedback signal per 
>>>>> RTT and therefore cannot use RFC3168; instead it implement it’s 
>>>>> own feedback mechanism. However, to avoid confusion such that 
>>>>> people could assume DCTP would not work without the accECN scheme 
>>>>> as specified in this doc, I rephrased to:
>>>>> 
>>>>> "Recently, proposed
>>>>>>>>  experimental scheme to provide more than one feedback signal 
>>>>>>>> per RTT  in the TCP header.  Given TCP header space is scarce, 
>>>>>>>> it overloads  the three existing ECN-related flags in the TCP 
>>>>>>>> header and provides  additional information in a new TCP option.
>>>>>>>> 
>>>>>>> requested from IANA. My understanding is that this experimental 
>>>>>>> document asks for allocation of a reserved TCP header flag. This 
>>>>>>> needs to be called out prominently, IMHO. In addition, since 
>>>>>>> this is not a standard, the suggested experimentation with the 
>>>>>>> main TCP header must IMHO be explicitly mentioned. I also 
>>>>>>> suggest to have later in a document a section that explicitly 
>>>>>>> explains why it is appropriate to modify the main TCP header in 
>>>>>>> an experiment.
>>>>>>> 
>>>>>>> I don’t know if any requirement that IANA assignment need to be 
>>>>>>> called out in the abstract but we can do that. However, I 
>>>>>>> believe the question if this document should or should not 
>>>>>>> assign the bit is still not completely solved, or is it?
>>>>>> 
>>>>>> I believe this question will have to be reviewed during WGLC and, 
>>>>>> more importantly, IETF last call. For the moment, my concern is 
>>>>>> that the document correctly describes the IANA allocation.
>>>>>> 
>>>>>> I would like to see here a statement such as : "Given TCP header 
>>>>>> space is scarce, this specification allocates a reserved header 
>>>>>> bit and overloads the two ECN flags in the TCP header ...“.
>>>>> 
>>>>> A bit lengthy but now:
>>>>> 
>>>>>>>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] 
>>>>>>>> need  more accurate ECN feedback information whenever more than 
>>>>>>>> one
>>>>>>> 
>>>>>>> marking
>>>>>>>> 
>>>>>>>>  is received in one RTT.
>>>>>>>> 
>>>>>>> of stating a "need", it would IMHO make more sense to discuss 
>>>>>>> the benefits of the suggested mechanism in this document of its 
>>>>>>> own, independent of other proposals. To me, this document should 
>>>>>>> be independent of other documents and specifically other 
>>>>>>> experiments. We have to think about cases where not all 
>>>>>>> experiments are successful. Then independent documents will be 
>>>>>>> more future-proof in future.
>>>>>>> 
>>>>>>> This is a naming collision… The sentence was meant to say that 
>>>>>>> these mechanisms new more accurate ECN feedback than provided 
>>>>>>> today by
>>>>>>> RFC3168 but it was not meant to say that these mechanism have to 
>>>>>>> use the scheme as specified in this document.
>>>>>>> 
>>>>>>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] 
>>>>>>> need more accurate ECN feedback information than provided by the 
>>>>>>> feedback scheme as specified in [RFC3168] whenever more than one 
>>>>>>> marking is received in one RTT. This document specifies an 
>>>>>>> alternative feedback scheme that provides more accurate 
>>>>>>> information and could be used by these new TCP extensions.“
>>>>>>> 
>>>>>>> Does this help?
>>>>>> 
>>>>>> See my proposal for the abstract. I continue to disagree with the 
>>>>>> term "need" but I think this can be sorted out by another term.
>>>>>> 
>>>>>>>> track, it is intended to be a complete replacement for classic 
>>>>>>>> TCP/  ECN feedback, not a fork in the design of TCP.
>>>>>>>> 
>>>>>>>>  Until the AccECN experiment succeeds, [RFC3168] will remain as 
>>>>>>>> the  standards track specification for adding ECN to TCP.
>>>>>>>> 
>>>>>>> "Until the AccECN experiment succeeds, [RFC3168] will remain as 
>>>>>>> the only standards track specification for adding ECN to TCP.“
>>>>>> 
>>>>>> This wording is better.
>>>>>> 
>>>>>>>>  AccECN feedback overloads flags and fields in the main TCP 
>>>>>>>> header  with new definitions, so both ends have to support the 
>>>>>>>> new wire  protocol before it can be used.
>>>>>>>> 
>>>>>>> I understand that you are not happy with the word „overload“ 
>>>>>>> here but the point of this sentence really is that the flags 
>>>>>>> can/could be used differently and therefore we need a new 
>>>>>>> negotiation before we can use them.
>>>>>> 
>>>>>> For me the following would work: "AccECN feedback overloads the 
>>>>>> two existing ECN flags and allocates the currently reserved and 
>>>>>> previously called NS flag in the main TCP header.
>>>>>> Given the new definitions, both ends have to support the new wire 
>>>>>> protocol before it can be used."
>>>>>> 
>>>>>> I believe the wording has to be crystal clear on the reservation 
>>>>>> of bit 7 when it is discussed the first time in the text. In 
>>>>>> follow-up sections, maybe shorter terms could be used.
>>>>> 
>>>>> Okay, now:
>>>>> 
>>>>>>> If you prefer, we can also remove the NS flag in this list, as 
>>>>>>> ECN Nonce was anyway never deployed.
>>>>>>> 
>>>>>>>>  For that we refer to [RFC3168] or any RFC that  specifies a 
>>>>>>>> different response to TCP ECN feedback, for example:
>>>>>>>> [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low 
>>>>>>>> Latency  Low Loss Scalable (L4S) congestion control 
>>>>>>>> [I-D.ietf-tsvwg-l4s-arch];  ECN-capable TCP control packets 
>>>>>>>> [I-D.ietf-tcpm-generalized-ecn], or  Alternative Backoff with 
>>>>>>>> ECN (ABE)  [I-D.ietf-tcpm-alternativebackoff-ecn].
>>>>>>>> 
>>>>>>> be deleted. If other experiments need more accurate feedback, it 
>>>>>>> is up to them to explain how they would use this mechanism. This 
>>>>>>> document should focus on how to signal the feedback, not how to 
>>>>>>> use that.
>>>>>>> 
>>>>>>>>  It is likely (but not required) that the AccECN protocol will 
>>>>>>>> be  implemented along with the following experimental additions 
>>>>>>>> to the  TCP-ECN protocol: ECN-capable TCP control packets and 
>>>>>>>> retransmissions  [I-D.ietf-tcpm-generalized-ecn], which 
>>>>>>>> includes the ECN-capable SYN/  ACK experiment [RFC5562]; and 
>>>>>>>> testing receiver non-compliance  
>>>>>>> tcpm-generalized-ecn independent documents, which probably 
>>>>>>> implies that draft-ietf-tcpm-generalized-ecn does not use 
>>>>>>> AccECN. If experimentation with ECT in SYN requires a 
>>>>>>> combination, this could be done in a new, third document. Apart 
>>>>>>> from having simpler focused documents, this could significantly 
>>>>>>> help later with moving forward documents to standards track.
>>>>>>> 
>>>>>>> I disagree, however, this is a discussion to have on 
>>>>>>> draft-ietf-tcpm- generalized-ecn. I don’t see a problem in  
>>>>>>> providing a reference here that says „it is likely…“ and nothing 
>>>>>>> more.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.1.  Document Roadmap
>>>>>>>> 
>>>>>>> and tutorial text with lot's of redundancy towards other 
>>>>>>> documents. I think the document can be made much easier to read 
>>>>>>> by shorten it. In many cases this is just an editorial change as 
>>>>>>> there is redundancy. As one such example, just remove this 
>>>>>>> section.
>>>>>>> 
>>>>>>> I guess this a matter of taste. As an AD, I’m a big fan of short 
>>>>>>> and concise documents, however, some redundancy can also help 
>>>>>>> understanding, especially if you explain things multiple times 
>>>>>>> but with a different level of detail. I personally would not 
>>>>>>> need the roadmap but I know many people who find these things 
>>>>>>> helpful and to be honest I don’t see how removing this part 
>>>>>>> makes the doc any better. If you don’t want it, don’t read it.
>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.2.  Goals
>>>>>>>> 
>>>>> (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.iana.org%2Fassignments%2Ftcp-header-flags%2Ftcp-header-flags.xhtml%23tcp-header-flags-1&amp;data=02%7C01%7Cpravb%40microsoft.com%7C2e4980f91d89410fa58708d5e8111fbb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636670082610453620&amp;sdata=KXOFPQ%2FfWW0dekv5%2BdYM8stbLl80hLtlJVnM9s5G9lQ%3D&amp;reserved=0)

_______________________________________________
tcpm mailing list
[email-masked]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microsoft.com%7C2e4980f91d89410fa58708d5e8111fbb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636670082610453620&amp;sdata=BzLdW3YTISACG5pakJhqTGE%2BWg5mRX3T514Z7woG2oI%3D&amp;reserved=0

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 12 Jul 2018 14:16:34 -0400

Hi Praveen,

see below.

> Am 12.07.2018 um 12:25 schrieb Praveen Balasubramanian <[email-masked]>:
> 

Right, I thought that the outcome of that discussion was that this actually is covered by the current spec, but to be honest I forgot to double-check.

However, having talked to Yuchung in the mean time (greetings from NetDev), I now also understand his use case/concerns for DCTCP where you can actually have a large number of CE marking in a row and with AccECN you basically need to send an ACK at least every 8 packets in a large burst of successive EC-marked packet (e.g. if the whole window is CE marked with can happen with DCTCP). If you want to coalesce more than 8 ACKs in this case, this would warp the ACE counter in the TCP header, however, I guess you could then still send an option instead because I think the spec says that the information in the option should take precedence over the ACE counter in the TCP header. Also something I would need to double-check in the spec.

I guess it’s a fair point that in that use case AccECN would actually have a slightly higher overhead than DCTCP today. However, AccECN was designed to cover a hopefully broad set of future use cases and is more optimized to Internet-wide use case (with the assumption that the CE marking rate is usually rather low) and not specifically for DCTCP only.

Still I’m open for concrete proposals to change something but I guess it would also be nice to wrap this up at some point and actually start the experimentation (to e.g. find an concrete answer to the overhead question).

> Re. GRO/LRO/RSC: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/exception-conditions-that-terminate-coalescing see condition 4 for an exception that terminates coalescing: "The s…

I think this is still fine because you don’t have to send the TCP option very often. More problematic is actually rule 8 though:
"	• The segment contains ECN flags, as defined in RFC 3168, that meet one or both of the following criteria:
		• The segment contains a different value for the ECN field (ECT, CE) in the IP header than the previous segment.
		• The segment has a different value for the ECN flags (ECE and CWR) in the TCP header than the previous segment.“
I guess that would need to be changed to "if the flags different from the  flags in the previous packet“..

> I don’t fully understand this. ECN++ allows marking of control packets and defines a response. Do we really need the TCP option for processing the feedback?

I didn’t meant to say you need the option, actually the AccECN option does not help you here. But you need the ACE counter in the TCP header because with classic RFC3168 feedback you don’t get any feedback about marked control packets.

Mirja

---

From: Joe Touch <[email-masked]>
Date: Thu, 12 Jul 2018 22:23:23 -0700

Let’s please be clear:

> On Jul 12, 2018, at 9:25 AM, Praveen Balasubramanian <pravb=[email-masked]> wrote:
> 
> Re. GRO/LRO/RSC: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/exception-conditions-that-terminate-coalescing <https://docs.microsoft.com/en-us/windows-hardware/drivers/network/exception-conditions-that-terminate-coalescing> see condition 4 for an exception that terminates coalescing: "The segment contains one or more TCP options other than the TCP timestamp option". So while software implementations of coalescing can be updated, existing hardware will cause poor performance with new TCP options that are present in a lot of data packets. 

While that may be true, it is also true that such performance optimizations always violated TCP. If we avoid updating the protocol as defined because of such existing violations, we might as well engrain TCP coalescing as legitimate in the protocol and deprecate all options other than timestamp.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Jul 2018 09:18:12 +0100

Yuchung,

On 12/07/18 17:03, Mirja Kühlewind wrote:
> I’m not the expert here but my understanding is that is would not break GRO but could not make actual use or it if the ACE counter changed or the option is present.
[BB] Yuchung's criticism applies to any scheme that feeds back a 
continually changing signal like ECN. For instance, like AccECN 
feedback, DCTCP ECN feedback continually changes the TCP header flags, 
and I believe DCTCP is widely used in DCs.

I believe, to make GRO support ECN feedback (whether DCTCP or AccECN), 
it would be necessary for GRO to know which bits to mask when 
determining whether a packet is mergeable with others. I believe GRO 
already collects header fields separately for the TCP logic to be able 
to work on them, but I'm also not an expert.
> However, usually your will only have a small number of CE marks every couple of RTTs, and the counter only changes if you CE marks/the option is only present a few times per RTT. So the impact should be rather low. But this clear something to evaluate for the experiment.
[BB] With DCTCP as currently designed, you tend to get runs of 100% CE 
if the load of short flows is very small. In that case, DCTCP feedback 
will change the header flags less often than AccECN. However, with more 
short flows, the feedback is more on-off. Then the flags change more 
often with DCTCP than with AccECN feedback.

In general, hardware optimization is not going to optimize a protocol 
that didn't exist when the hardware was designed. It would be ideal to 
design a new protocol that takes advantage of existing hardware 
optimization. However, as long as an experimental protocol still works 
with existing hardware, I think it's reasonable to assume that hardware 
optimization will only arrive once a protocol has become well-established.

> I guess you have a 50% chance today to hit a delayed ACK. I guess delaying less (where you delay every second ACK today) would mean ACK every packet, and thus double ACK load on the network. I guess on today networks that actually in most cases not a problem, however, there might be specially cases where it is. However, that’s problem an independent question to evaluate.
[BB] If there were no delayed ACKs, the problem with DCTCP feedback 
would largely disappear. But delayed ACKs are not going away. Which is 
why we proposed replacing DCTCP feedback with AccECN feedback.
> Not sure I understand your proposal correctly, but I assume you mean negotiate via SYN option and then just use the ACE counter in the header? If so, I don’t understand why you see the negotiation…
[BB] @Yuchung, as Mirja says, your point #1 seems to be proposing a 
solution without a problem. In fact an option on the SYN would create a 
new problem 'cos of the severe shortage of space for SYN options (for 
this reason, the AccECN TCP option was designed not to be needed on the 
SYN).
Cheers

Bob
>>>>>>>> accurate
>>>>>>>>> Latency

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 13 Jul 2018 11:38:05 -0700

hi --

 I agree:
1. delayed/streched ACK aren't going away (in fact will be more common)
2. GRO isn't and should not be a show-stopper
3. SYN option is running tight
4. HW opt comes after SW

I worry:
1. GRO is a unavoidable issue in deployment (let's not produce an
undeployable RFC). the ACE counter won't work as GRO can pack up to
64KB/MTU =~ 45 pkts under heavy congestion.

2. ACE's benefit over DCTCP is when delayed ACKs and ack losses
happen. Now delayed ACKs happen frequently when the messages are
small, which means they don't solicit too many ACKs to be dropped in
the reverse path. Also that under heavy downstream loss or reordering,

3 SACK options become dominant to have ACE option.

4. If SYN option space is a concern, ACE header exchange is ok. But I
suspect the latter has more middlebox issue based on my TFO deployment
experience.

So with all that said, is it possible to have a "minimal"-ACE that
1. Does ACE handshake as proposed
2. Mark ECT on all (or at least SYN & data & rtx) packets like ECN++
3. Leave ACE-count and ACE option optional (i.e. MAY)

as a "fairly safe" to deployment compromise. And I am happy to draft a
kernel patch for that :-)

On Fri, Jul 13, 2018 at 1:18 AM, Bob Briscoe <[email-masked]> wrote:
>> I’m not the expert here but my understanding is that is would not break
>> GRO but could not make actual use or it if the ACE counter changed or the
>> option is present.
>
> [BB] Yuchung's criticism applies to any scheme that feeds back a continually
> changing signal like ECN. For instance, like AccECN feedback, DCTCP ECN
> feedback continually changes the TCP header flags, and I believe DCTCP is
> widely used in DCs.
>
> I believe, to make GRO support ECN feedback (whether DCTCP or AccECN), it
> would be necessary for GRO to know which bits to mask when determining
> whether a packet is mergeable with others. I believe GRO already collects
> header fields separately for the TCP logic to be able to work on them, but
> I'm also not an expert.
>>
>> However, usually your will only have a small number of CE marks every
>> couple of RTTs, and the counter only changes if you CE marks/the option is
>> only present a few times per RTT. So the impact should be rather low. But
>> this clear something to evaluate for the experiment.
>
> [BB] With DCTCP as currently designed, you tend to get runs of 100% CE if
> the load of short flows is very small. In that case, DCTCP feedback will
> change the header flags less often than AccECN. However, with more short
> flows, the feedback is more on-off. Then the flags change more often with
> DCTCP than with AccECN feedback.
>
> In general, hardware optimization is not going to optimize a protocol that
> didn't exist when the hardware was designed. It would be ideal to design a
> new protocol that takes advantage of existing hardware optimization.
> However, as long as an experimental protocol still works with existing
> hardware, I think it's reasonable to assume that hardware optimization will
> only arrive once a protocol has become well-established.
>
>>
>> Same as above, it can only up to 8 CE marked packets per ACK, however, CE
>> marking rater are expected to be rather low. ACK suppression should probably
>> not be used if the ACE counter has changes, however, usually the ACE counter
>> stays stable for multiple RTTs and then ACK suppression is not a problem.
>> However, not sure I understood you question here correctly…?
>>
>> I guess you have a 50% chance today to hit a delayed ACK. I guess delaying
>> less (where you delay every second ACK today) would mean ACK every packet,
>> and thus double ACK load on the network. I guess on today networks that
>> actually in most cases not a problem, however, there might be specially
>> cases where it is. However, that’s problem an independent question to
>> evaluate.
>
> [BB] If there were no delayed ACKs, the problem with DCTCP feedback would
> largely disappear. But delayed ACKs are not going away. Which is why we
> proposed replacing DCTCP feedback with AccECN feedback.
>>
>>
>> Not sure I understand your proposal correctly, but I assume you mean
>> negotiate via SYN option and then just use the ACE counter in the header? If
>> so, I don’t understand why you see the negotiation part in the TCP header as
>> the problem?
>
> [BB] @Yuchung, as Mirja says, your point #1 seems to be proposing a solution
> without a problem. In fact an option on the SYN would create a new problem
> 'cos of the severe shortage of space for SYN options (for this reason, the
> AccECN TCP option was designed not to be needed on the SYN).
>>
>>
>> That would be nice but here you actually need AccECN because you want to
>> have feedback for control packets as well. AccECN is providing this
>> feedback. AccECN does not change the „use of ECN“; that’s what we have ECN++
>> for; both thing ideally would be deployed together however. Was that your
>> questions?
>
> Cheers
>
>
> Bob
>
>> Mirja
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>> the question if you need this „more“ on accuracy really depends on the
>>>> use case. If you use DCTCP as today, the ACE counter in the TCP is probably
>>>> sufficient and you might not want to pay the additional overhead in your
>>>> data center (that why the option is actually optional).
>>>>
>>>> If you however, e.g., have very differently sized packets, then the byte
>>>> counter in the option could give you a more accurate signal. Or if you are
>>>> also interested in the ECT(1) counter, you need the option. Further the ACE
>>>> counter also give your feedback on control packet and the option enables you
>>>> to distinguish between CE-amrked payload and control packets, which can also
>>>> become important when experimenting with making all packets ECN-capable.
>>>>
>>>> Given accECN is a general feedback mechanism that in fact is designed to
>>>> enable new future uses of the ECN signal, we wanted to keep all these option
>>>> available while making is still as simple as possible and as flexible as
>>>> possible.
>>>>
>>>> Mirja
>>>>
>>>>
>>>>
>>>>>> below, we
>>>>>>> not yet
>>>>>>>> Yet, I continue to believe that some formal wording in the document
>>>>>>>> needs
>>>>>>>> to change, as explained below.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Michael (with no hat on)
>>>>>>>>
>>>>>>>>
>>>>>>>>> They do
>>>>>>>>> not need the mechanism that is specified in this draft, however,
>>>>>>>>> this is
>>>>>>>>> not
>>>>>>>>> document
>>>>>>>>> mentioned. I also suggest to have later in a document a section
>>>>>>>>> that
>>>>>>>>> explicitly
>>>>>>>>> called
>>>>>>>>> out
>>>>>>>>> in the abstract but we can do that. However, I believe the question
>>>>>>>>> if
>>>>>>>>> this
>>>>>>>>> document should or should not assign the bit is still not
>>>>>>>>> completely
>>>>>>>>> solved, or
>>>>>>>>> is it?
>>>>>>>>
>>>>>>>> more
>>>>>>>> space is
>>>>>>>       information can be provided in a new TCP option that however is
>>>>>>> not
>>>>>>> used
>>>>>>>       on the TCP SYN."
>>>>>>>
>>>>>>>>>> * 1.  Introduction
>>>>>>>>>>
>>>>>>>>>> need
>>>>>>>>> documents and specifically other experiments. We have to think
>>>>>>>>> about
>>>>>>>>> cases
>>>>>>>>> where not all experiments are successful. Then independent
>>>>>>>>> documents
>>>>>>>>> will
>>>>>>>>> these
>>>>>>>>> more
>>>>>>>>> as specified in [RFC3168] whenever more than one marking is
>>>>>>>>> received in
>>>>>>>>> one
>>>>>>>> term
>>>>>>>>>> TCP/
>>>>>>>>>> the
>>>>>>>>>> header
>>>>>>>>>      currently reserved and previously called NS flag in the main
>>>>>>>>> TCP
>>>>>>>>> header
>>>>>>>>> I understand that you are not happy with the word „overload“ here
>>>>>>>>> but
>>>>>>>>> the
>>>>>>>     experienced marked packets. Given the new definitions of these
>>>>>>> three
>>>>>>> bits,
>>>>>>>     both ends     have to support the new wire protocol before it can
>>>>>>> be
>>>>>>> used.
>>>>>>>     Therefore during the TCP handshake the two ends use these three
>>>>>>> bit
>>>>>>> in
>>>>>>>>> If you prefer, we can also remove the NS flag in this list, as ECN
>>>>>>>>> Nonce
>>>>>>>>> was
>>>>>>>>> anyway never deployed.
>>>>>>>>>
>>>>>>>>>> [ms] At least ABE seems orthogonal. Anyway, I think this paragraph
>>>>>>>>>> can
>>>>>>>>>> just
>>>>>>>>>
>>>>>>>>> be deleted. If other experiments need more accurate feedback, it is
>>>>>>>>> up
>>>>>>>>> to
>>>>>>>>> Yes, that is what the paragraph says. Isn’t it better to be
>>>>>>>>> explicit
>>>>>>>>> about this?
>>>>>>>>>
>>>>>>>>>> [ms] I am a big fan of simple, standalone documents. In my view,
>>>>>>>>>> the
>>>>>>>>>> TCPM
>>>>>>>>>
>>>>>>>>> with ECT in SYN requires a combination, this could be done in a
>>>>>>>>> new,
>>>>>>>>> third
>>>>>>>>> draft-ietf-tcpm-
>>>>>>>>> generalized-ecn. I don’t see a problem in  providing a reference
>>>>>>>>> here
>>>>>>>>> that
>>>>>>>>> and tutorial text with lot's of redundancy towards other documents.
>>>>>>>>> I
>>>>>>>>> think
>>>>>>>>> this is just an editorial change as there is redundancy. As one
>>>>>>>>> such
>>>>>>>>> example,
>>>>>>>>> and
>>>>>>>>> concise
>>>>>>>>> especially if you explain things multiple times but with a
>>>>>>>>> different
>>>>>>>>> level of
>>>>>>>>> who find these things helpful and to be honest I don’t see how
>>>>>>>>> removing
>>>>>>>>> this
>>>>>>>>> I have to say I also don’t see the point of removing this part.
>>>>>>>>> Given
>>>>>>>>> we’ve
>>>>>>>>> Sure, however, I don’t think it hurts to spell it out. For me both
>>>>>>>>> is
>>>>>>>>> fine, keep it
>>>>>>>>> or remove it.
>>>>>>>>>
>>>>>>>>> mentioned before, I don't think an RFC should speculate about TCPM
>>>>>>>>> and
>>>>>>>>> its
>>>>>>>>> fulfilled. I don’t see a reason to not mention the requirement
>>>>>>>>> draft as
>>>>>>>>> this
>>>>>>>> My suggested wording uses the expression "can be deployed at large
>>>>>>>> scale"
>>>>>>>> and I believe this is relevant.
>>>>>>>>
>>>>>>>>>>   Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never
>>>>>>>>>> deployed
>>>>>>>>>> so
>>>>>>>>>> [ms] This wording, as well as Figure 1, needs to take into account
>>>>>>>>>> the
>>>>>>>>>> IANA
>>>>>>>>>
>>>>>>>>> Is does. However, I can explicitly say that is has be re-clssified
>>>>>>>>> as
>>>>>>>>> reserved.
>>>>>>>>>
>>>>>>>>> "RFC 3540 was never deployed so it is being reclassified as
>>>>>>>>> historic
>>>>>>>>> [I-D.ietf-
>>>>>>>>> As I said I’m not sure if we have fully concluded this discussion
>>>>>>>>> yet.
>>>>>>>>> However,
>>>>>>>>> 1) keep it in the registry as reserved and conserve the knowledge
>>>>>>>>> in
>>>>>>>>> tcpm
>>>>>>>>> that this experiment is running and no other experimental RFC such
>>>>>>>>> use
>>>>>>>>> this
>>>>>>>>> 2) Keep is marked as reserved but add a note about this experiment
>>>>>>>>> in
>>>>>>>>> the
>>>>>>>>> IANA registry
>>>>>>>>> 3) Or assign it right away with IESG approval. I guess in this case
>>>>>>>>> tcpm
>>>>>>>>> could
>>>>>>>>> I think this is nit picking. Using a different phrasing here makes
>>>>>>>>> the
>>>>>>>>> sentence
>>>>>>>>> that we need to handle the flag registration correctly. However,
>>>>>>>>> here
>>>>>>>>> the
>>>>>>>>> work „re-use“ here is really that we say that these flags are or
>>>>>>>>> have
>>>>>>>>> been

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 13 Jul 2018 14:53:48 -0400

Hi Yucheng,

please see below.

> Am 13.07.2018 um 14:38 schrieb Yuchung Cheng <[email-masked]>:
> 
> hi --
> 
> I agree:

This is ECN++ and independent of AccECN. We on purposed have split the feedback and usage of ECN (which is not the case in RFC3168) to be able to change things in future independently

> 3. Leave ACE-count and ACE option optional (i.e. MAY)

I don’t understand this. If both is optional, you don’t have any feedback. Or what do you mean by „leave ACE-count optional“?

Mirja

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 13 Jul 2018 12:30:17 -0700

> This is ECN++ and independent of AccECN. We on purposed have split the feedback and usage of ECN (which is not the case in RFC3168) to be able to change things in future independently
sure - IMO marking all packets just improve the accuracy significantly
vs the counters and options. For example, ECN on SYN is very useful on
incast / loaded link.

> I don’t understand this. If both is optional, you don’t have any feedback. Or what do you mean by „leave ACE-count optional“?
use-case: We can negotiate DCTCP-style ECN for the internet.

Then interested parties can progressively experiment on more accurate
"options" (!= TCP-option)

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 13 Jul 2018 15:42:05 -0400

Hi Yuchung,

> Am 13.07.2018 um 15:30 schrieb Yuchung Cheng <[email-masked]>:
> 

Yes, but to be able to mark control packets as ECN-enables you also need a way to feedback congestion experienced information if they appear on these packets. Therefore you can use ECN++ only safely with e.g. AccECN feedback (but not with classic RFC3168 ECN feedback). Still these two things should be specified in separate documents to be able to change them separately in future.
> "options" (!= TCP-option)

As Appendix A of RFC7560 says I don’t think it is a safe option for the Internet where packet loss more likely then in a full ECN-enabled data center.

Mirja

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 13 Jul 2018 13:45:27 -0700

> As Appendix A of RFC7560 says I don’t think it is a safe option for the Internet where packet loss more likely then in a full ECN-enabled data center.

Again I disagree RFC7560 Appendix A is a big problem based on my
experience with at times loss-heavy ECN-enabled data-center (Google
data-center runs very hot and uses a DCTCP-variant).

We can quabble forever w/o data. That's why I asked for some
(non-simulation) data.

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 13 Jul 2018 13:54:00 -0700

> We can quabble forever w/o data. That's why I asked for some
I recall you mentioned you have a ACE Linux patch. Could you share w/ me?

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 13 Jul 2018 16:59:11 -0400

> Am 13.07.2018 um 16:45 schrieb Yuchung Cheng <[email-masked]>:
> 

I’m not sure if that is a question about data. With the feedback scheme that is used by DCTCP a single ACK loss can already cause feedback to get lost and I don’t think that a safe option to be deployed on the Internet. Actually in high loss situation the chance to at least get some feedback is pretty high, however, if you only have a single CE mark, chances to misses that are much higher.

The other problem with the feedback scheme used by DCTCP is that is does not provide feedback for control packets (as it relies on the ACK/SEQ number with does not increase for control packets) and therefore cannot be used with ECN++.

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 13 Jul 2018 17:02:02 -0400

> Am 13.07.2018 um 16:54 schrieb Yuchung Cheng <[email-masked]>:
> 

the implementation is here but it’s not complete regarding the latest spec and not well tested:

https://github.com/mirjak/linux-accecn

Let me see if I can look into this in August sometime!

Mirja

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 14 Jul 2018 01:52:03 +0100

Yuchung,

>>>>> 64KB/MTU =~ 45 pkts under heavy congestion.
[BB] I'm really not expert on offload, but I thought GRO collects (or 
can be made to collect) a set of fields from the headers it strips off?

If I'm wrong (quite likely), bear in mind the following:
* you only need the ACE counter (main header) when the AccECN Option is 
being stripped by a middlebox
* on those connections that need ACE (cos of middlebox meddling) 
couldn't GRO be limited to 8 packets, at least while experimenting with 
AccECN to see if it is useful and gather data?

* Appendix A.2. Example Algorithm for Safety Against Long Sequences of 
ACK Loss 
<https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07#appendix-A.2> 
is not just for ACK losses. It would be just as useful if GRO 'lost' the 
headers. It uses the TCP acknowledgement number and any SACK options to 
check whether the ACE field could have wrapped, and if so by how much.

>>>>> suspect the latter has more middlebox issue based on my TFO deployment
You may remember earlier in TCPM, Marcelo presented the measurement 
study we did. Altho it was titled as primarily about ECN++ traversal (IP 
header), it also did a lot more limited measurements of traversal of the 
ECN flags (main TCP header).

The paper summarizing the results is here, as well as the dataset:
     http://www.it.uc3m.es/amandala/ecn++/

It covered 6.5 million different paths over 20 mobile networks in 8 
European countries as well as over 50 fixed networks globally. I know 
that's diddly-squat compared to what Google can do, but the data is 
still worth a lot.

However, the data we got on traversal of the TCP header was much more 
limited. We controlled all our client vantage points, but only a few 
servers, with the bulk of the servers being the Alexa 500k. Where we 
didn't control the server, we could only rely on using tracebox which 
checks ICMP TTL exceeded messages against what was sent. But few routers 
implement the updated ICMP [RFC1812] that includes more of the transport 
header than the first 8B.

FWIW, we didn't find any mangling of the ECN TCP header flags.

>>>>> 1. Does ACE handshake as proposed
[snip #2 about ECN++]
>>> "options" (!= TCP-option)
I must point out that there is very limited space and flexibility for 
versioning experiments with multiple variants of how the 3 'ECN' flags 
in the main header are used. You will see from the latest appendix B 
(added in response to Michael Scharf's point on this) that there is no 
more space for further negotiation on the SYN or SYN/ACK, but there are 
5 unused codepoints on the final ACK of the 3WHS and 7 on the first data 
packet. Ideally the negotiation would be on the SYN and SYN/ACK though, 
especially if used in combination with TFO.

This means you will need to be much more certain about whether there 
really are GRO problems, or just maybe. And if so, we're going to have 
to try to work out a solution on paper first (or at least on private 
networks), rather than burning bits by trial and error.

This also means going into the depths of the draft in full (if you 
haven't already done so), cos the co-authors did go thru a huge number 
of possible designs ourselves over the years, and we fixed all sorts of 
potential problems in the process.

Put another way, if you start from scratch, you will be finding new 
problems to solve for a year or so after you start.

BTW, there is a lot more scope for versioning multiple variants of 
AccECN TCP options.

> data-center runs very hot and uses a DCTCP-variant).
To be clear, you're only disagreeing that packet loss is not a 
non-problem in DCs (excuse the double negative). You're not disagreeing 
that packet loss is a problem for DCTCP feedback.

Cheers

Bob
>>>>>>>>>>>>>>> [ms] This sentence should be removed, as this is speculation

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 14 Jul 2018 08:32:40 +0200

Yuchung, Bob,

I really like this discussion!

Two points:

Am 14.07.2018 um 02:52 schrieb Bob Briscoe:
>>>> Am 13.07.2018 um 15:30 schrieb Yuchung Cheng<[email-masked]>:
>>>>
>>>>>> Am 13.07.2018 um 14:38 schrieb Yuchung Cheng<[email-masked]>:
>>>>>>
>>>>>> hi --
>>>>>>
>>>>>> I agree:

[RS @ Group] To be pedantic, GRO can coalesce packets, until the ACE 
counter has increased by 7 counts (one less than the previously 
delivered "superpacket"). Under heavy CE marking, this may be as few as 
7 or 8 packets, correct. However, this is when the network expiriences 
congestion. Thus "slowing things down" by reducing the processing speed 
on the client is not an acute problem IMHO - as a slightly delayed 
ACKing would implicitily also reduce the sending speed somewhat.

With many flows, higher RTTs etc, I would expect GRO to be able to 
coalesce many more than just 7 packets.

[...]

 >>> I don’t understand this. If both is optional, you don’t have any 
feedback. Or what do you mean by „leave ACE-count optional“?

 >>> As Appendix A of RFC7560 says I don’t think it is a safe option for 
 >>> the Internet where packet loss more likely then in a full
 >>> ECN-enabled data center.
 >>>

[RS @ Yuchung] I'm afraid I'm with Mirja's comment (other fork of this 
discussing added here), I don't quite follow what you proposing. An 
AccECN-style negotiation, which is used in conjunction with ECN++ (and 
possibly L4S style ECT-1 marking of all packets instead of ECT-0?). But 
how do you envision the feedback scheme? Like the DCTCP-like state 
machine to send change-triggered ACKs, and stretches of ECE and non-ECE 
marked TCP header bits? (Which does not cater well for ACK thinning, ACK 
loss)

However, I think I see where you are coming from. Let me try to 
summarize what I assume here:

With DCTCP-style feedback in a datacenter, and some level of ACK losses, 
the sender-side estimation of the CE levels with with equal probability 
over- and under-estimated the marking levels during one specific RTT. If 
the sender underestimated congestion, the network will provide a higher 
marking level during the next RTT, and thereby correct the false 
estimate. On overestimation, the flow would only have reduced its 
sending rate more than necessary, and regular CA across all flows of 
that bottleneck link will readily make use of that free bandwidth. All 
of that happening in datacenter-typical RTTs of a few dozend nanoseconds 
to a few microseconds... So any deviation will only persist for a very 
short period of time, while most flows have very similar RTTs.

Now, the problem space AccECN tries to address includes the public 
internet, with RTTs over a bottleneck, that vary many orders of 
magnitude potentially. Thus the goal should be to avoid the over- and 
more so, the under-estimation of congestion as much as possible already 
within the first RTT.

Or summarized differently, in a datacenter, small errors in congestion 
estimation have a predictable, very short life-time until they will be 
corrected. On the public internet, it is a better approach to try and 
avoid an estimator error on congestion as good as possible, as these 
errors would result in more unfairness among flows.

I hope that is a fair summary, but please expand on how/what your 
feedback scheme would encompass.

Best regards,
   Richard

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Sun, 15 Jul 2018 20:19:22 +0000

Mirja, Bob,

Thanks a lot for the replies. I have read the latest version -07. I agree that -07 is in a better shape. To me, Appendix B contains useful information.

I'll open a new thread for some further remarks on -07.

Best regards

Michael (chair hat off)

> From: Bob Briscoe [mailto:[email-masked]]
> Sent: Wednesday, July 11, 2018 8:01 PM
> Cc: Mirja Kühlewind <[email-masked]>; draft-ietf-tcpm-
> [email-masked]; [email-masked]
> extensibility options as previously discussed by mail. We will probably send a
> separate email on that part.
> >
> > Mirja
> >
> >
> needs to change, as explained below.
> >>
> >> Thanks
> >>
> >> Michael (with no hat on)
> >>
> >>
> >> In my understanding (as a non-native speaker), the use of the word
> "need" is not correct here. DCTCP as specified in RFC 8257 can be
> cannot get precise ECN feedback whenever more than one marking is
> received in one RTT“.
> mechanism. However, to avoid confusion such that people could assume
> DCTP would not work without the accECN scheme as specified in this doc, I
> rephrased to:
> >
> > "Recently, proposed
> when it is discussed the first time in the text. In follow-up sections, maybe
> shorter terms could be used.
> > Okay, now:
> >
> removing this
> and its
> scale" and I believe this is relevant.
> >>
> the agreed requirements for a more accurate ECN feedback protocol
> [RFC7560]. So, if the TCPM working group publishes this document with the
> content of Section 5, I believe the TCPM working group already has reached
> consensus that the protocol meets requirements. In addition, it is possible
> that new requirements would be identified in future, e.g., as an outcome of
> the experiment, and that would obviously have to be considered by TCPM.
> In that case, for the success of the experiment not only RFC 7560 would
> matter, but also further requirements. My proposed wording does not have
> all these problems.
> >>
> > Okay, used your proposed wording. You have a point about the
> requirement and I mis-read you proposal earlier as „has to be deployed
> large-scale“.
> >
> deployed so
> (https://www.iana.org/assignments/tcp-header-flags/tcp-header-
> flags.xhtml#tcp-header-flags-1) for Bit 7 to "AE (Accurate ECN), previously
> used by Historic as NS (Nonce Sum) [RFC3540, RFC8311]" and change the
> reference to this RFC-to-be instead of RFC8311.]“
> >
> > I guess we could also ask IANA to add an additional comment column
> instead (but not sure if we then have to update RFC3168, which I think we
> really don’t want. Should be fine now.
> >
> >> If the allocation of a reserved flag is correctly explained in the abstract and
> introduction, I think these sentences can use a bit relaxed terminology.
> >>
> this experiment, e.g., because the experiment may have some lessons
> learnt. What options would we have to negotiate the PS version, and how
> could a stack implementing the PS version figure out whether the remote
> end uses the experimental or the PS version of the protocol?
> >>
> >> The reason why I ask is because I don't see any easy solution to this but I
> may miss something. Maybe this would not be a concern if there were some
> codepoints left.
> >>
> AccECN, and if there was a deployed base of both, I still believe we could end
> up in a situation in which we had to allocate yet another TCP header flag to
> distinguish the different versions of AccECN. The NS flag would not work for
> that if it is used by the experimental version. The risk of having to spend
> further TCP header flags somehow concerns me. Of course, that problem
> would only matter if the AccECN experiment succeeds somehow, but lessons
> learnt would require protocol changes, which is speculation.
> >>
> flag, including in the SYN, and it seems to consume all codepoints. So, I see
> the risk of a protocol design "not ready for future improvements".
> >>
> >> I cannot easily propose text. Maybe "lack of extensibility" is just one of
> the short-comings of the protocol design that cannot be avoided but that
> would suggest to have one place where to define exactly how this
> mechanism can be used by other TCP extensions. Having said this, in this
> specific paragraph I am less concerned about that than elsewhere.
> >>
> >> In my reading of https://www.iana.org/assignments/tcp-header-
> flags/tcp-header-flags.xhtml, this flag currently has no name, i.e., it does not
> have the name NS. So this statement on "renaming" is formally incorrect
> IMHO.
> >>
> (Accurate ECN) to the TCP flag at bit 7; this flag has previously been known as
> NS (Nonce Sum) …".
> > Now:
> >
> extensions is the "right thing" to do. It is in fact the wrong thing. But possibly
> there is no alternative with the few bits, so maybe we end up doing it.
> >>
> load is at most equal to a current TCP stack, in all potential cases. If it can
> increases network load, it has to be reasoned why increasing load (and risk of
> reverse congestion and the like) is worth the effort.
> >>
> >> I agree that this may be an area of experimentation, but I believe then it
> has to be explained to implementers what the tradeoffs are.
> > Added:
> > "Especially, if only few
> simply not a problem, but it is exactly this sort of discussion that may not be
> needed in this document. But I won't rat-hole on this comment here.
> >>