Subject: Re: [aqm] I-D Action: draft-ietf-aqm-ecn-benefits-04.txt

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 10 Jun 2015 17:18:52 +0200

Hi Gorry, hi Michael,

to catch up with this: I think most of my comments were addressed. Especially the
restructuring and removing of all normative language is good. Thanks!

The one point I'm still not super happy with is section 3.6 and 3.6.1. I don't
think it reflects the problems and opportunities of using a different AQM and
different congestion control for ECN accordingly. If I see this correctly, you 
didn't
change the text very much and I think that the current text is just not fully 
correct.
The main issue is that the text sounds like if you have accECN, you can just go 
ahead
and implement a different congestion control. However, if you use something like
DCTCP on the Internet, you'd be very aggressive and push away other traffic.
Therefore we should be more careful what we say here. I'm not even sure if this
should be discussed in the 'Benefits' section, but I do think it's important to
mention in this doc. Maybe we can also ask Bob Briscoe what he think should be in
this section because he has been very actively working on this recently.

The other smaller point is that I'd like to see more explicitly mentioned
somewhere that ECN does not always help with tail loss because often the queue
simply is too short if a large burst is sent. (Therefore avoiding burst might
still be useful).

These are the two points I'd like to see better addressed from my review
comments. However, based on the other comments on the list, I think there are
more people who agree with me that this document sounds extremely positive and
it should more explicitly also address potentially negative consequences. Those
things are mentioned in the document but from my point of view are rather hidden as
a reasoning for a recommendation. Therefore I would like to see some further
restructuring of the document.  This is not the same than the original 'pitfalls'
section was like because that section actually also tried to give recommendation
instead of just documenting things. However, that's nothing that is for me to 
decide;
maybe we should discuss this again in the next meeting (because I think the 
mailing list
discussion did not really led to a conclusion...?)...?

Further, I have to say, I'm personally not super happy with sections 3-5.
Even though the normative language was removed, it still tries to give
recommendations instead of just summarizing what other docs hopefully already
recommend. Further there seems to be still quite a bit of redundancy which, if
removed, would probably improve readability. At the same time on other points
the "recommendations" are still too high level to actually be really useful,
from my point of view; e.g. see section 5 'Prerequisites for network
endpoints...": This basically says (in 3 bullets) that the used transport must
implement ECN to use ECN. That's rather obvious for me. However, it does not say
more useful things like: it should still implement the ECN SYN fallback as 
specified
in RFC3168 because this cases actually still happens and that e.g. Apple has a
smaller RTO for SYNs with ECN which seems actual useful. Another things we've
just discovered while implementing this fallback for Linux was that it is actually
not clear which state TCP should be in if a second SYN without ECN was send but a
SYNACK with ECN is received. I think those things would be interesting if 
recommendations
are given. However, this might need additional expertise from different people 
that can bring
in more experience in implementing ECN and therefore I'm not sure if this doc 
should
give any recommendations. That's another point that might need to be discuss at the
next meeting (or a separate discussion on this should be started at the mailing 
list).

All in all, if there is/would be consensus to move the document forward as it is,
I'm in principle also fine with that expect that I'd would still like to see a 
few changes in
section 2.6/2.6.1. However, I personally think there is more discussion needed 
to what the scope
and purpose of this document is.

Mirja

On 05.05.2015 20:49, [email-masked] wrote:
>
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>   This draft is a work item of the Active Queue Management and Packet Scheduling Working Group of the IETF.
>
>          Title           : The Benefits of using Explicit Congestion Notification (ECN)
>          Authors         : Godred Fairhurst
>                            Michael Welzl
> 	Filename        : draft-ietf-aqm-ecn-benefits-04.txt
> 	Pages           : 17
> 	Date            : 2015-05-05
>
> Abstract:
>     The goal of this document is to describe the potential benefits when
>     applications use a transport that enables Explicit Congestion
>     Notification (ECN).  The document outlines the principal gains in
>     terms of increased throughput, reduced delay and other benefits when
>     ECN is used over network paths that include equipment that supports
>     ECN-marking.  It also describes methods that can help successful
>     deployment of ECN.  It does not propose new algorithms to use ECN,
>     nor does it describe the details of implementation of ECN in endpoint
>     devices (Internet hosts), routers or other network devices.
>
>
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-aqm-ecn-benefits/
>
> There's also a htmlized version available at:
> https://tools.ietf.org/html/draft-ietf-aqm-ecn-benefits-04
>
> A diff from the previous version is available at:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-aqm-ecn-benefits-04
>
>
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

-- 
------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Communication Systems Group
Institute TIK, ETH Zürich
Gloriastrasse 35, 8092 Zürich, Switzerland

Room ETZ G93
phone: +41 44 63 26932
email: [email-masked]
------------------------------------------

---

From: [email-masked]
Date: Thu, 11 Jun 2015 15:00:18 +0100

I wonder if some of the comments could be addressed by some small additional
text, see below for some suggested text. If the new text helps, Michael
and I are happy roll another revision if you think this or similar text
can resolve some issues. Let me know what people think.

I see you also raise some scoping questions (near the end).

Comments in-line.

Gorry

> this section because he has been very actively working on this recently.
 GF: I think points are relevant to the TCPM requirements for
 Accurate ECN. To me, they seem to be mainly of concern to transport protocol
 designers, rather than end-to-end users of the service - I’m sure work in
 TCPM will consider this, but indeed we could state something here.

Fo you think this text could cover the point as regard usage of ECN by
upper layer protocols?:

 OLD:
 This could be used by the control mechanisms in the transport to make a
 more appropriate decision on how to react to congestion, allowing it to
 react faster to changes in congestion state.

 NEW:
 This could be used by the control mechanisms in the transport to make a
 more appropriate decision on how to react to congestion, allowing it to
 react faster to changes in congestion state. Any update to the transport
 protocol requires careful consideration of the robustness of the behaviour
 when working with endpoints or network devices that were not designed for
 the new congestion reaction.

> simply is too short if a large burst is sent. (Therefore avoiding burst

GF: We suggest adding text on this topic to penultimate para of section
 2.3. This text says again that loss can not be avoided, which is a point
 made in other ECN-related documents, but for clarity this could be
restated here:

 OLD:
 When incipient congestion occurs at the tail of a burst, an ECN-capable
 network device can CE-mark the packet(s), rather than triggering drop.
 This allows the transport to avoid the retransmission timeout, ...

 NEW:
 An ECN-capable network device cannot eliminate the possibility of packet
 drop. Drop may occur due to a traffic burst exceeding the instantaneous
 available capacity of a network buffer, or a a result of an AQM algorithm
 (overload protection mechanisms, etc). However, ECN-capable network
 devices that observes incipient congestion may be expected to buffer an
 ECN-capable flow and set a CE-mark in one or more packet(s), rather than
 triggering packet drop.

 Setting a CE-mark allows the transport to avoid the retransmission
 timeout, ...

> comments. However, based on the other comments on the list, I think there
> are more people who agree with me that this document sounds extremely
positive
> and it should more explicitly also address potentially negative
consequences.
GF: I'm not sure what you ask here, since the abstract states:
"The goal of this document is to describe the potential benefits when
applications use a transport that enables Explicit Congestion Notification
(ECN). The document outlines the principal gains in terms of increased
throughput, reduced delay and other benefits when ECN is used over network
paths that include equipment that supports ECN-marking. It also describes
methods that can help successful deployment of ECN. It does not propose
new algorithms to use ECN, nor does it describe the details of
implementation of ECN in endpoint devices (Internet hosts), routers or
other network devices."

> Those things are mentioned in the document but from my point of view are
rather
> hidden as a reasoning for a recommendation. Therefore I would like to
see some
> further restructuring of the document.  This is not the same than the
original
> 'pitfalls' section was like because that section actually also tried to
give
> recommendation instead of just documenting things. However, that's
nothing that is for me
> to decide; maybe we should discuss this again in the next meeting
(because I think
> the mailing list discussion did not really led to a conclusion...?)...?
>
> recommendations instead of just summarizing what other docs hopefully
> already recommend. Further there seems to be still quite a bit of
redundancy
> which, if removed, would probably improve readability. At the same time
on other
> points the "recommendations" are still too high level to actually be really
> useful, from my point of view; e.g. see section 5 'Prerequisites for
network
> endpoints...": This basically says (in 3 bullets) that the used transport
> must implement ECN to use ECN. That's rather obvious for me. However, it
does
> not say more useful things like: it should still implement the ECN SYN
fallback as
> specified in RFC3168 because this cases actually still happens and that
e.g. Apple
> has a smaller RTO for SYNs with ECN which seems actual useful. Another
things
> we've just discovered while implementing this fallback for Linux was
that it is
> actually not clear which state TCP should be in if a second SYN without
ECN was
> send but a SYNACK with ECN is received. I think those things would be
interesting if
> recommendations are given. However, this might need additional expertise
from different
> people that can bring in more experience in implementing ECN and therefore
> I'm not sure if this doc should  give any recommendations.
>
GF: Interesting stuff, and happy to discuss this and I see we do need to
design things correctly (e.g., draft-kuehlewind-tcpm-ecn-fallbac), but are
you sure that operators or end users considering a roadmap to using ECN
needs to know how to design?

i.e., What do you hope for?: It could be that you would like you would
also like to make transport protocol design/deployment recommendations -
but if that is the case, we really didn't propose this when we brought the
document to the WG,

> That's another point that might need to be discuss at the next meeting
(or a
> separate discussion on this should be started at the mailing  list).
>
> All in all, if there is/would be consensus to move the document forward as
> it is, I'm in principle also fine with that expect that I'd would still
like to
> see a few changes in section 2.6/2.6.1. However, I personally think
there is more discussion
> needed to what the scope and purpose of this document is.
>
> Mirja
>
GF: This seems like a scope/milestone question - where we will need advice
from the Chairs.

Although, I agree design principles are useful to capture (if the IETF is
ready to do this), I still suggest this should be a separate document -
because I see the audience for good transport design are transport people
and network equipment/software manufacturers, rather than the present
focus encouraging other areas to look towards ECN deployment. Anyway,
depending on what is said we'll try and take this forward in an
appropriate direction.

((As someone from TSV,  I'm not at all sure that writing about how to
design ECN transports, should come from the IETF AQM WG, but that's
another detail to think about if people wish to do this.))

Gorry

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 12 Jun 2015 14:14:02 +0200

Hi Gorry,

I think my main issues can be resolves with some smaller changes or additions/removals. However, I still think that this document should not give any recommendation what a network node or the transport must or should do. The group has to decided on this, but I guess this would be a large change in the sections 3-5.

See below. 

> designers, rather than end-to-end users of the service - Im sure work in

Not sure what you mean by ‚end-to-end users of the service‘. For me the transport protocol or the congestion control of the transport is the user of the ECN signal. And here we should be careful what we say.

> the new congestion reaction.

That helps a little buts still doesn’t make me completely happy. The main problem is co-existence with current traffic. And this probably requires both the congestion control as well as the AQM to be updated together and makes the problem more complex. While the current text more reads like, please go ahead and design a new congestion control OR AQM but be careful… don’t think that’s enough.

Further it is not only this one sentence; there are more sentences in these paragraph which say very similar things.

What about basically simply removing these sentences:

OLD:
"2.6.  Opportunities for new Transport Mechanisms

   Loss is regarded as the standard signal from a network device
   experiencing congestion.  In contrast, CE-marked packets carry an
   indication that network queues are filling, without incurring loss.
   This introduces the possibility to provide richer feedback (more
   frequent and fine-grained indications) to transports.  This could
   utilise new thresholds and algorithms for ECN-marking.  ECN therefore
   provides a mechanism that can benefit evolution of transport
   protocols.

2.6.1.  Benefits from other forms of ECN-Marking/Reactions

   ECN requires a definition of both how network devices CE-mark packets
   and how applications/transports react to reception of these CE-marked
   packets.  ECN-capable receiving endpoints therefore need to provide
   feedback indicating that CE-marks were received.[RFC3168]provides a
   method that signals once each round trip time that CE-marked packets
   have been received.  An endpoint may provide more detailed feedback
   describing the set of received ECN codepoints using Accurate ECN
   Feedback [ID.Acc.ECN].  This can provide more information to a
   congestion control mechanism at the sending endpoint.

   Loss and CE-marking are both used as an indication for congestion.
   However, while the amount of feedback that is provided by loss ought
   naturally to be minimized, this is not the case for ECN.  With ECN, a
   network device could provide richer and more frequent feedback on its
   congestion state.  This could be used by the control mechanisms in
   the transport to make a more appropriate decision on how to react to
   congestion, allowing it to react faster to changes in congestion
   state.

   Precise feedback about the number of packet marks encountered is
   supported by the Real Time Protocol (RTP) when used over UDP
   [RFC6679] and proposed for SCTP [ST14] and TCP [ID.Acc.ECN].

   Benefit has been noted when packets are CE-marked earlier using an
   instantaneous queue, and if the receiver provides feedback about the
   number of packet marks encountered, an improved sender behavior has
   been shown to be possible (e.g, Datacenter TCP (DCTCP) [AL1]).
   DCTCP is targeted at confined environments such as a datacenter.  It
   is currently unknown whether or how such behaviour could be safely
   introduced into the Internet.“

NEW:
"2.6.  Opportunities for new Transport Mechanisms

   Loss and CE-marking are both used as an indication for congestion.
   However, while the amount of feedback that is provided by loss ought
   naturally to be minimized, this is not the case for ECN. In contrast, 
   CE-marked packets carry an indication that network queues are filling, 
   without incurring loss. This introduces the possibility to provide 
   richer feedback (more frequent and fine-grained indications) on the 
   congestion state to transports. 

   Unfortunately, the current specification of the feedback mechanism 
   in ECN from the receiver to the sender [RFC3168] provides signals 
   only once each round trip time that CE-marked packets have been 
   received. The receiving endpoint may provide more detailed feedback
   describing the set of received ECN codepoints using Accurate ECN
   Feedback [ID.Acc.ECN].  This can provide more information to a
   congestion control mechanism at the sending endpoint.

   Precise feedback about the number of packet marks encountered is
   supported by the Real Time Protocol (RTP) when used over UDP
   [RFC6679] and proposed for SCTP [ST14] and TCP [ID.Acc.ECN].

   Note that even if richer feedback is provided, ECN requires a 
   definition of both how network devices CE-mark packets
   and how the control loop in the application/transport reacts to 
   reception of these CE-marked packets. 

   An control mechanism, where CE marks are sent early based the 
   instantaneous queue length used a more fine-graind sender behavior,
   has been shown by Datacenter TCP (DCTCP) [AL1] to be possible and 
   beneficial to provide improved latency. As DCTCP require changes of 
   all end hosts and network nodes, DCTCP is targeted at confined 
   environments such as a datacenter.  It is currently unknown 
   whether or how such behavior could be safely introduced into 
   the Internet.“

And maybe also remove the following sentence in section 3:

"There is opportunity to design an AQM method for ECN that differs
   from one designed to drop packets (e.g., Random Early Detection uses
   a smoothed queue length because it was designed for loss and a
   congestion control that halves its sending rate on congestion)
   [ID.RFC2309.bis]. "

> timeout, …

Okay.

> other network devices.“

I’m questioning if what the abstract states is the right scope of this document.

Further this document still gives recommendation which is not reflected in the abstract.
And I don’t really think that the document „describes methods that can help successful deployment of ECN“ because I don’t really see any methods in this doc...

> but if that is the case, we really didn't propose this when we brought the

No, these are no points on protocol design. These are things that are good to know if you want to implement ECN as specified in RFC3168. Maybe something like lessons learnt from implementation and measurement. 

As I said, I personally have the feeling that the recommendations given in this document are not super helpful, while other recommendation that are missing could actually be more helpful. But that would rather be a different document, therefore I’d proposed to remove all recommendation from this document which would mean that a) a to of things would be removed from sections 3-5  and b) only those things remain which list problem (without giving recommendations).

And I say this again, from point of view the problem with the previous ‚pitfalls‘ section was that this section gave recommendation. However you (only) removed the word ‚pitfalls‘ but left all the recommendation in the document. I’d say the other way around would have been better… 

> depending on what is said we'll try and take this forward in an

Yes, I agree, this should be a different document. I also agree that we might be not fully ready yet to provide these design principles (not sure this is the right term here). Therefore I still recommend to remove any recommendation from this doc. However this would mean to remove quite a lot stuff in sections 3-5 (and potentially merge them into only one or two sections). 

> another detail to think about if people wish to do this.))

As the document states this correctly, it’s not enough to only change the transport/congestion control. We probably need to change both congestion control and AQM. Therefore this is probably a cross-wg thing between AQM and TCPM/TSVWG.

Mirja

---

From: [email-masked]
Date: Fri, 12 Jun 2015 13:46:43 +0100

Thanks - I see some points we can act upon now and see if we can do a
revision, but I can't promise to resolve all these issues in that revised
draft.

Since we are already in WGLC, the WG Chairs probably will need to decide
the scope - if this is changed, I expect will anyway require a new WGLC. 
Hopefully the new ID will help.

Gorry

> Hi Gorry,
>
> I think my main issues can be resolves with some smaller changes or
> additions/removals. However, I still think that this document should not
> give any recommendation what a network node or the transport must or
> should do. The group has to decided on this, but I guess this would be a
> large change in the sections 3-5.
>
> See below.
>
>>>
>>> Therefore we should be more careful what we say here. I'm not even sure
>>> if
>>> this
> Not sure what you mean by ‚end-to-end users of the service‘. For me
> the transport protocol or the congestion control of the transport is the
> user of the ECN signal. And here we should be careful what we say.
>
>
> That helps a little buts still doesn’t make me completely happy. The
> main problem is co-existence with current traffic. And this probably
> requires both the congestion control as well as the AQM to be updated
> together and makes the problem more complex. While the current text more
> reads like, please go ahead and design a new congestion control OR AQM but
> be careful… don’t think that’s enough.
>
> Further it is not only this one sentence; there are more sentences in
> these paragraph which say very similar things.
>
>>> somewhere that ECN does not always help with tail loss because often
>>> the
>>> queue
>> Notification
> And I don’t really think that the document „describes methods that can
> help successful deployment of ECN“ because I don’t really see any
> methods in this doc...
>
>
>>
> No, these are no points on protocol design. These are things that are good
> to know if you want to implement ECN as specified in RFC3168. Maybe
> something like lessons learnt from implementation and measurement.
>
> As I said, I personally have the feeling that the recommendations given in
> this document are not super helpful, while other recommendation that are
> missing could actually be more helpful. But that would rather be a
> different document, therefore I’d proposed to remove all recommendation
> from this document which would mean that a) a to of things would be
> removed from sections 3-5  and b) only those things remain which list
> problem (without giving recommendations).
>
> And I say this again, from point of view the problem with the previous
> ‚pitfalls‘ section was that this section gave recommendation. However
> you (only) removed the word ‚pitfalls‘ but left all the recommendation
> in the document. I’d say the other way around would have been better…
>
>>
> Yes, I agree, this should be a different document. I also agree that we
> might be not fully ready yet to provide these design principles (not sure
> this is the right term here). Therefore I still recommend to remove any
> recommendation from this doc. However this would mean to remove quite a
> lot stuff in sections 3-5 (and potentially merge them into only one or two
> sections).
>
>>
> As the document states this correctly, it’s not enough to only change
> the transport/congestion control. We probably need to change both
> congestion control and AQM. Therefore this is probably a cross-wg thing
> between AQM and TCPM/TSVWG.
>
> Mirja
>
>
>>
>> Gorry
>>
>>>

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 12 Jun 2015 11:29:02 -0400

> Hopefully the new ID will help.

Here are my thoughts, with chair hat on.

It's an Informational document (i.e. not Standards Track or BCP).  It
does have some advice about how not to break ECN, but it's not
altering or changing any previous standards or BCPs about how devices,
hosts, or applications behave.

I think it correctly avoids using the 2119 capitalized words (SHOULD,
MUST, etc.).  There are some non-capitalized "must" and "should" words
in section 5 when going through the high-level list of prerequisites
for successful use of ECN, and in my opinion, this is one of the more
useful parts of the document to summarize and bring the advice together.

There's definitely a valid criticism that it isn't particularly
specific about some details in this guidance, but I think that's
probably desirable, as some are still being worked out, and would
ultimately go into Standards Track and BCP documents from TSVWG or
some other working group.

I think as the AQM working group, the level of detail and strength
of recommendations made in -04 are pretty much on the mark for what
we should say.

Certainly people should let us know during this Last Call if they
feel otherwise.  It can be something we explicitly ask the AD to
confirm during their review too.

-- 
Wes Eddy
MTI Systems

---

From: Dave Taht <[email-masked]>
Date: Wed, 17 Jun 2015 11:33:39 -0700

for many positive bullet points in the present document I can think of
a negative counter-example in the real world that needs to be defeated
in detail. Just off the top of my head:

3.2 tinc, when carrying tos, encapsulates the ecn markings also and
does not apply them properly according to the various rfcs.
3.3 recently: one large providers equal cost multipath implementation
used the full 8 bit tos field as part of the tuple. This worked fine,
until CE started getting exerted by new aqms in the path, which led to
massive packet re-ordering. Fixing it required fixing a ton of pretty
modern vendor gear.
3.4: thus far, even with multiple queues, on the aqms I have, ECN
marked traffic causes extra loss and delay in non ecn marked traffic.
I agree that we should ecn mark sooner than drop, work is progressing.

I would like it if non-traditional (ab)uses of ecn were covered - 1)
attacks using ecn marked packets on dns servers, for example - and 2)
future protocols that could use it (say, Quic).
3) as an example of something I've been fiddling with for a long time,
coupling a routing protocol's metrics to something other than packet
loss, and getting better "signal strength" by using ecn marked packets
for more reliable communications under congestion.

the draft touches upon voip uses (where I kind of think ecn is not the
best idea), but does not touch upon videoconferencing well, where I
think ecn protection of iframes would be a very good idea. So the
guidance in sec 2.4 is a bit vague.

aggregating transports with retries (e.g. wifi) could use ecn
basically "for free" when experiencing trouble at the lowest layers of
the stack.

I know I have a tendency to accumulate the negatives (I do LIKE ecn),
but would certainly like to have a fora, or a living document or wiki
for potential sysadmins, vendors, and deployers to have a clear grip
on what can go wrong when attempting to roll out ecn stuff.

So I am mostly in favor of this document getting published, so long as
someone steps up to also be an "ecn news central", chock full of user
generated content on the pitfalls, tips, and tricks - and benefits! -,
to guiding ecn deployment further along.

ecn is inevitable. finally.

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 18 Jun 2015 15:50:41 +0200

Hi Wes,

the concern I have is that, even if no normative language is used (which is correct), people might still take this as a IETF recommendation and therefore I think we should review it carefully. However, as no normative language is used, people might not review this document sufficiently… Definitely good to mention this to the IESG explicitly!

Mirja

> Am 12.06.2015 um 17:29 schrieb Wesley Eddy <[email-masked]>:
>

---

From: [email-masked]
Date: Fri, 19 Jun 2015 15:05:31 +0100

Thanks Dave,

To keep people informed, we have taken the Chairs' advice and kept the
current format in the new upcoming revision (will be -05), but we have
also tried hard to address concerns over the what is recommended, fix
inconsistencies where noted, and above all tried to make the document more
accessible to others.

Specifically on David's comments below.

This will clearly state that ToS field usage is deprecated, and explain
why this is bad for ECN.

We try to "describe" things that need to be done to support ECN, rather
than develop a requirements list, similarly we tried to avoid describing
pitfalls in this document. While neither should be forgotten, they did not
seem to us to be the things to record in this particular ID.

We did add text in the new version to a section "3.4 Co-existance of ECN
and non-ECN flows" that will not say how to handle multiple queues or how
to do things better, but it does say that AQM designers need to think
about the latency of queueing CE-marked packets, and that better
algorithms could be made.

We'll send this shortly to the list.

Gorry