Subject: [aqm] adoption call: draft-welzl-ecn-benefits

From: Wesley Eddy <[email-masked]>
Date: Mon, 11 Aug 2014 10:48:16 -0400

This draft has been discussed a bit here and in TSVWG:

http://tools.ietf.org/html/draft-welzl-ecn-benefits-01

As I understand, the IAB has also discussed it a bit, and would
be happy if this was something that an IETF working group
published.  I believe the TSVWG chairs also discussed this and
would be fine if the AQM working group adopted it.

That said, there have only been a small number of mail list
comments here about it, and we (unfortunately) didn't devote
much of any meeting time to it in Toronto.

So, *please* let us know your thoughts on this in the next few
weeks.  I believe we would try to complete it relatively quickly
(e.g. 6 months) as an Informational RFC, assuming there is
consensus and AD approval to add the milestone.

I personally would like to see some stronger support for it from
this working group in order to feel good about adopting it, though
I think it is correct and may be "motherhood and apple pie" to
many of us.

-- 
Wes Eddy
MTI Systems

---

From: Dave Taht <[email-masked]>
Date: Mon, 11 Aug 2014 11:19:36 -0700

> I think it is correct and may be "motherhood and apple pie" to

I don't share the relentless optimism of this document, and would
like it - or a competing document - to go into the potential negatives.

examples of this include the TOS washing problem bob alluded to
in one of the tsvwg meetings (the monday one), the impact on
competing flows, the problem of unresponsive agents or other
misuse,  the deprecation (?) of the nonce mechanism, and how to
properly switch between marking and dropping in an aqm.

There are also the possibilities in new uses for ecn (for example, in
the original rmcat nada proposal), in usages on local links in routing
protocols, and in new protocols such as quic, etc.

> --
> Wes Eddy
> MTI Systems
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

---

From: David Collier-Brown <[email-masked]>
Date: Mon, 11 Aug 2014 14:31:17 -0400

+1, and even if some of the branches  of the discussion are "twigs", we
should indicate they're present and we're aware of them.

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain

---

From: Michael Welzl <[email-masked]>
Date: Mon, 11 Aug 2014 23:38:09 +0200

Hi,

Thanks for your feedback! More below:

On 11. aug. 2014, at 20:19, Dave Taht <[email-masked]> wrote:

> in one of the tsvwg meetings (the monday one),

This is an interesting point. I was somewhat torn about this document myself, in that it recommends to "turn on ECN", but doesn't say how - personally, in fact, I'd not want to recommend using it like it's been used on up to now, but I also have no clear recommendation yet.

Meanwhile, we (authors) have agreed that we'd rather stay away from recommending to "turn on ECN", but rather use the document to recommend to "be compatible" with it - i.e. *not* wash the TOS bits, and if you're a server, correctly reflect the bits.

This makes the TOS washing problem not a negative point of ECN, but the behavior that the document recommends against.

> the impact on
> competing flows,

This gets into the specific congestion control response to ECN, and the rules for ECN-marking. We try to stay away from such specifics in this document because we don't want to recommend one particular behavior here. I'd agree that not going into such specifics is wrong for a document that recommends to "turn on ECN", but we intend to no longer do that.

> the problem of unresponsive agents or other
> misuse,  the deprecation (?) of the nonce mechanism,

Not deprecated (yet?) AFAIK.

> and how to

Again, specific congestion control reactions.

Cheers,
Michael

---

From: John Leslie <[email-masked]>
Date: Mon, 11 Aug 2014 19:38:57 -0400

(I have read Michael's reply to this, but I'll respond here.)

>> http://tools.ietf.org/html/draft-welzl-ecn-benefits-01

   I do think this is the right place to discuss it.

>> would be fine if the AQM working group adopted it.

   Thus, I am in favor of adopting it, with the understanding that
it will see significant changes during our discussion.

> like it - or a competing document - to go into the potential negatives.

   I think it should concentrate on what its name says: the benefits
of ECN, both now and in an expected future; but that it should also
at least mention downsides this WG sees: and that it should avoid any
recommendation stronger than "make ECN available to consenting
applications".

> in one of the tsvwg meetings (the monday one),

   This definitely deserves mention.

> the impact on competing flows,

   That might have to go into a companion document. I think this
document could try to describe the bounds of such issues, but not
the details.

> the problem of unresponsive agents or other misuse,

   I'm not sure what Dave is alluding to here...

> the deprecation (?) of the nonce mechanism,

   I don't accept it as a given that the nonce should be deprecated;
though I do think that discussion will come up.

> and how to properly switch between marking and dropping in an aqm.

   I doubt this document will go into much detail there. Basically,
IMHO, an AQM should mark well before dropping becomes necessary; and
when dropping becomes necessary, ECN-capable packets should be dropped
essentially as often as non-ECN-capable packets.

   I'm not sure this document should say much about that, though...

> protocols, and in new protocols such as quic, etc.

   Sounds like interesting reading, Dave -- do you want to send pointers?

--
John Leslie <[email-masked]>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 12 Aug 2014 10:43:26 +0100

Wes, and responders so far,

A doc on the benefits and pitfalls of ECN is needed. Personally I 
wouldn't assign much of my own time as a priority for such work; I'd 
rather work on finding the best road through the protocol 
engineering. But I'm glad others are doing this.

We need to be clear that this doc (at the moment) is about the 
benefits of 'something like RFC3168 ECN'. I think that is the right 
direction. I would not be interested in a doc solely about the 
benefits of 'classic' ECN (we had RFC2884 for that).

However, if it is about the benefits of some other ECN-like thing, it 
will not be worth writing unless it is more concrete on what that 
other ECN-like thing is. At present different and sometimes 
conflicting ideas are floating around (I'm to blame for a few).

In order to write about benefits, surely experiments are needed to 
quantify the benefits? Alternatively, this could be a manifesto to 
identify /potential/ benefits of ECN that the current classic ECN is 
failing to give. I think at the moment it's the latter (and that's OK 
given that's where we have reached today).

How about the title "Explicit Congestion Notification (ECN): 
Benefits, Opportunities and Pitfalls" ?

We (in the RITE project) have agreed to start work on an 'ECN 
Roadmap' in order to identify all the potential ideas for using ECN 
coming out of research, and write down whether new standards will be 
needed for some, whether they can evolve without changing standards, 
which are complementary, which conflict, etc.

I don't know whether this ECN benefits doc ought to include this 
detailed ECN roadmap work, but if it's going to talk about "something 
like ECN" I believe it will have to include a summary of the main 
items on such a roadmap to be concrete.

more inline...

>it will see significant changes during our discussion.

I think we can and should agree the direction of those changes in 
this thread. I'd rather not agree to start on a doc and plan to meander.

I don't think this will be a quick (6 months) job, because of the 
problem of being clear about the "things like ECN" that it needs to talk about.

>recommendation stronger than "make ECN available to consenting

I agree it should be informative, rather than making too many 
detailed recommendations.

>    This definitely deserves mention.

I agree it would be useful to write about deployment problems. 
Specifically, a clear explanation of what the deployment problems 
have been, which ones are no longer problems (if any), and which ones 
are still problems, and how prevalent.

Alternatively, folks doing ECN black hole experiments might want to 
write them up in a separate draft, that this one can refer to (there 
are already a few research papers documenting ECN deployment 
experiments). I would be willing to help collect together the history 
that I have on this.

>document could try to describe the bounds of such issues, but not

+1

>    I'm not sure what Dave is alluding to here...

Dave is alluding to suppression of ECN in the feedback loop (e.g. by 
the receiver) or the sender not responding (the problem space that 
ConEx and the ECN Nonce target).

>though I do think that discussion will come up.

It would also be useful to write about cheating problems.

However, an informational doc would not be the right place to 
deprecate the ECN nonce. That needs to be in a doc that offers an 
alternative mechanism (which is where draft-moncaster-tcpm-rcv-cheat 
is heading - we have ideas on how to extend it to ECN cheating).

>    I'm not sure this document should say much about that, though...

This might be relevant if talking about how there could be benefits 
if it were done differently from RFC3168. But otherwise, I agree this 
would not be on-topic for this doc.

>    Sounds like interesting reading, Dave -- do you want to send pointers?

<http://tools.ietf.org/html/draft-zhu-rmcat-nada-03>

I assume using ECN on local links in routing protocols is just 
something Dave is saying that operators could do unilaterally.

Given QUIC includes FEC to hide losses, I guess it is a good example 
to consider whether ECN still offers sufficient benefits over and 
above just removing losses.

Cheers

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 12 Aug 2014 11:24:09 +0100

OK, so I have many comments, see below.

Gorry

On 12/08/2014 10:43, Bob Briscoe wrote:
> rather work on finding the best road through the protocol engineering.
> But I'm glad others are doing this.
>
> We need to be clear that this doc (at the moment) is about the benefits
> of 'something like RFC3168 ECN'. I think that is the right direction. I
> would not be interested in a doc solely about the benefits of 'classic'
> ECN (we had RFC2884 for that).
>
> will not be worth writing unless it is more concrete on what that other
> ECN-like thing is. At present different and sometimes conflicting ideas
> are floating around (I'm to blame for a few).
>
GF: If someone wishes to write this research paper, I'd be happy to join 
them, but it was not what I had in mind for this ID.

> How about the title "Explicit Congestion Notification (ECN): Benefits,
> Opportunities and Pitfalls" ?
>
GF: I could live with that, if the group wished this!

> We (in the RITE project) have agreed to start work on an 'ECN Roadmap'
> in order to identify all the potential ideas for using ECN coming out of
> research, and write down whether new standards will be needed for some,
> whether they can evolve without changing standards, which are
> complementary, which conflict, etc.
>
> like ECN" I believe it will have to include a summary of the main items
> on such a roadmap to be concrete.
>
>
> more inline...
>
> I think we can and should agree the direction of those changes in this
> thread. I'd rather not agree to start on a doc and plan to meander.
>
GF: +1, we can add comments to the ID to align to this, personally I've 
already said that I'd like to see text on:
- bleaching and middlebox requirements to deploy.
- Need to verify the paths actually really *do support* ECN (sorry, but 
may be needed).

> problem of being clear about the "things like ECN" that it needs to talk
GF: That depends also in part on whether these new mechanisms: will 
actually change the message to potential users of transports and people 
considering deployment. In my mind the definition of the protocol 
techniques does not HAVE to be the same document that tells people *HOW* 
to implement this in stacks or network devices. (My own choice would be 
to keep these to research papers and RFCs targeted at their respective 
communities).

> I agree it should be informative, rather than making too many detailed
> recommendations.
>
GF: Any other bullets listing additional topics are most welcome!

> Specifically, a clear explanation of what the deployment problems have
> been, which ones are no longer problems (if any), and which ones are
> still problems, and how prevalent.
>
GF: +1

> write them up in a separate draft, that this one can refer to (there are
> already a few research papers documenting ECN deployment experiments). I
> would be willing to help collect together the history that I have on this.
>
GF: I'm doing these experiments too, happy to work with others.

>> document could try to describe the bounds of such issues, but not
GF: +1, I see this as informing the Engineering, not the goals of this 
document.

> Dave is alluding to suppression of ECN in the feedback loop (e.g. by the
> receiver) or the sender not responding (the problem space that ConEx and
> the ECN Nonce target).
>
GF: I'd be glad to include something on this (not detailing a mechanism, 
but perhaps linked "how do we know the path supports ECN", and perhaps 
if the group is smart - how can the transport actually find this stuff 
out for itself? (sorry to keep returning to this)
> It would also be useful to write about cheating problems.
GF: +1

> However, an informational doc would not be the right place to deprecate
> the ECN nonce. That needs to be in a doc that offers an alternative
> mechanism (which is where draft-moncaster-tcpm-rcv-cheat is heading - we
> have ideas on how to extend it to ECN cheating).
>
GF: - I'd be surprised if this document deprecates an EXP RFC (I've said 
this before), normally I'd expect this in the draft that defines the 
replacement technique.
> This might be relevant if talking about how there could be benefits if
> it were done differently from RFC3168. But otherwise, I agree this would
> not be on-topic for this doc.
>
GF: This could be complicated, but probably depends on how far the WG 
has proceeded in other documents when this document is ready to publish?
> I assume using ECN on local links in routing protocols is just something
> Dave is saying that operators could do unilaterally.
>
GF: The IETF has acknowledged alternate ECN semantics via setting the 
DSCP field (RFC4774, I recall some historical discussion of why and 
where), at least something could be said about this. I'd like to add at 
least a pointer.

> Given QUIC includes FEC to hide losses, I guess it is a good example to
> consider whether ECN still offers sufficient benefits over and above
> just removing losses.
>
GF: And then, isn't the implication of AQM to significantly increase the 
number of "losses" unless we use ECN?

Indeed, I have the impression we are confusing many on these points - 
ECN could change the reaction to congestion signal, and FEC (even 
opportunistic CC-friendly FEC) can also change the way things react to 
congestion signals.

> Bob

Gorry

P.S. My proposal is that if this thread converges I intend to parse a 
later version and see if my co-author and I can then add some comments 
to our draft to catalogue a set of issues that we expect to see added in 
future, and an appendix of those we think may be out of scope.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 29 Aug 2014 13:16:58 +0000

Hi Gorry,

> opportunistic CC-friendly FEC) can also change the way things react to

I don't think that an AQM's implication is automatically to increase the number of losses; that may happen to specific flows (in particular, unresponsive ones), but for responsive (non-ECN) ones, the expectation would be to de-correlate the losses, and for TCP, to only have around 1 loss per window when necessary - instead of a burst loss of one window and the expensive recovery...

Perhaps it's that perception that also poses an obstacle to AQM deployment, because of the believe that a dynamic but lower mark/drop threshold will cause more losses?

Regards,
  Richard

---

From: David Collier-Brown <[email-masked]>
Date: Fri, 29 Aug 2014 09:52:30 -0400

> Perhaps it's that perception that also poses an obstacle to AQM deployment, because of the believe that a dynamic but lower mark/drop threshold will cause more losses?

Goodness gracious, from the point of  view of a queuing network, AQM
reduces losses overall, in the process of minimizing delay and keeping
bandwidth use just below the theoretical maximum.

Oversimplifying, we try to keep the buffers empty, so that if we get a
burst we can handle it without losses and without affecting other
communications. We signal via a loss or other indicator if the
non-bursty flow is enough to cause congestion, which keeps the buffers
near-empty and the system uncongested.

Failing to do so fills the buffers without signalling there is
congestion, and induces delay on everyone who's dependant on the buffer.
Not to mention allowing the congestion to go unreported!

It's not a tradeoff discussion: it's arguably one about correctness.

--dave
[Somebody like Neil Gunther could explain the math of this better, but
the behaviour is well-known in the trade, and cordially hated.
Congestion control is superior to admission control, which is what I
often use to prevent the server equivalent of congestive collapse (:-))]

---

From: Dave Taht <[email-masked]>
Date: Fri, 29 Aug 2014 08:26:42 -0700

On Tue, Aug 12, 2014 at 3:24 AM, Gorry Fairhurst <[email-masked]> wrote:
>> quantify the benefits?

+10

>> Alternatively, this could be a manifesto to

+1

>> complementary, which conflict, etc.

I'd like to see experiments done through the free.fr network as it's the
only one I know of with ecn enabled along the edge in their revolution
v6 product.

Presently cerowrt ships with ecn enabled on the inbound rate
limiter and disabled on the outbound, I have considered enabling
it by default on the outbound for connections > 4mbits.

(users can override these settings, of course)

> - Need to verify the paths actually really *do support* ECN (sorry, but may
> be needed).

I agree that verifying that a path can take a congestion notification e2e
is important.

> GF: That depends also in part on whether these new mechanisms: will actually
> change the message to potential users of transports and people considering
> deployment. In my mind the definition of the protocol techniques does not
> HAVE to be the same document that tells people *HOW* to implement this in
> stacks or network devices. (My own choice would be to keep these to research
> papers and RFCs targeted at their respective communities).
>
>
>>

+1

> GF: I'm doing these experiments too, happy to work with others.

I have a couple tools for testing bursty loss and QoS and ecn e2e under
development, they are a little buggy, and all dependent on more
recent linux kernels (amusingingly IPv6 works better than IPv4 until
very recently),  and could be improved.

are there other tools available (scamper?)

> GF: I'd be glad to include something on this (not detailing a mechanism, but
> perhaps linked "how do we know the path supports ECN", and perhaps if the
> group is smart - how can the transport actually find this stuff out for
> itself? (sorry to keep returning to this)

For TCP it seems rather hard to fix in the field. I could see supplying the
syn/ack with a CE as one example. Newer protocols (quic/nada?) could
do something more robust and periodically probe...

> GF: This could be complicated, but probably depends on how far the WG has
> proceeded in other documents when this document is ready to publish?
>
>>

I like the idea of protecting the main frame (P-frame? can never keep these
straight) with ecn, and allowing for loss on the deltas lacking ecn.

Also don't mind the idea of protecting all of a videoconferencing session
with ecn, and reacting to those congestion notifications appropriately.
(assuming that the other problems involving washing, etc, are licked)

>> Dave is saying that operators could do unilaterally.

A long running research project of mine (over 15 years running) has been to
find a routing protocol that worked well for wireless, and one more holy grail
has been to find one that reacted well to congestion and differences in
available bandwidth. Most routing protocols
stop at "reachability", and some conflate packet loss with that.

and: ecn marking bgp tcp sessions would be a good way of making sure ecn
worked right in the core. ;/

Please note that although I think I've got tantalizing hints towards
coming up with better routing metrics for wireless or congested
networks as an outgrowth of the bufferbloat work (examples - the new
babel-rtt metric, some improvements on minstrel algorithm's in-kernel
connectivity and bandwidth reporting, and the idea of throwing a
dropped or  marked packet up to user space where a routing daemon
might take a look at it and  it's path and do something smarter about
it) - I've been at it for a long time without much success and don't
want to hold up more productive research elsewhere.

> GF: The IETF has acknowledged alternate ECN semantics via setting the DSCP
> field (RFC4774, I recall some historical discussion of why and where), at
> least something could be said about this. I'd like to add at least a
> pointer.
>
>

I routinely run two wired networks with fq_codel with nearly zero loss
using ecn marking. I am curious as to the mark/drop statistics from
other networks that have deployed ecn.

> Indeed, I have the impression we are confusing many on these points - ECN
> could change the reaction to congestion signal, and FEC (even opportunistic
> CC-friendly FEC) can also change the way things react to congestion signals.

I'd like to see more hard results from FEC from quic. I'd also love to
see someone trying ECN with bittorrent traffic,
adding it to an existing uTP client like utorrent or transmission
would be straightforward, and an easy way to experiment with ecn from
userspace, and explore the issues with deploying it outside of TCP.

> P.S. My proposal is that if this thread converges I intend to parse a later
> version and see if my co-author and I can then add some comments to our
> draft to catalogue a set of issues that we expect to see added in future,
> and an appendix of those we think may be out of scope.
>
>

-- 
Dave Täht

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 29 Aug 2014 16:43:37 +0100

> near-empty and the system uncongested.
Yep that's what I think is the goal.

> often use to prevent the server equivalent of congestive collapse (:-))]

I agree that AQM often reduces losses (what I said above didn't say 
this) but it sometimes it can increase loss (e.g.  a large RTT or less 
responsive CC), but I don't recall saying something on this in the 
draft. If you see something in the draft that isn't correct then let us 
know.

Gorry

---

From: [email-masked]
Date: Sat, 30 Aug 2014 08:13:58 +0100

Please see some thoughts below.

>>> In order to write about benefits, surely experiments are needed to
GF: Interesting suggestions, thanks. We'd be happy to see research to
quantify the benefits of specific methods in a range of scenarios. 
However, this draft is intended to be a long-lived document that speaks of
the potential benefits that ECN can offer across the Internet in general.

> (users can override these settings, of course)
GF: Understood.

>>> problem of being clear about the "things like ECN" that it needs to
>>> talk
>>> about.
>>>
>> GF: That depends also in part on whether these new mechanisms:
>> will actually change the message
>> to potential users of transports and people considering deployment.
>>
>> In my mind the definition of the protocol techniques does not
>> HAVE to be the same document that tells people *HOW* to implement
>> this in stacks or network devices.
>> (My own choice would be to keep these to research
>>> Alternatively, folks doing ECN black hole experiments might
>>> want to write them up in a separate draft, that this one can
>>> refer to (there are already a few research
>>> papers documenting ECN deployment experiments).  I would be
>>> willing to help collect together the history that I have on
>>> this.
>>>
>>> Dave is alluding to suppression of ECN in the feedback loop
>>> (e.g. by the receiver) or the
>>> sender not responding (the problem space that ConEx and
>>> the ECN Nonce target).
>>>
>> GF: I'd be glad to include something on this (not detailing a,
>> mechanism but perhaps linked to
>> "how do we know the path supports ECN", and perhaps if
>> the
> For TCP it seems rather hard to fix in the field. I could
> see supplying the syn/ack with a CE as
> one example. Newer protocols (quic/nada?) could
>>> mechanism (which is where draft-moncaster-tcpm-rcv-cheat is heading -
>>> we have ideas on how to extend it to ECN cheating).
>>>
>> GF: This could be complicated, but probably depends on how
>> far the WG has proceeded in other
>> documents when this document is ready to publish?
>>
>>>> > There are also the possibilities in new uses for ecn
>>>>  (for example, in the original rmcat
>>>> > nada proposal), in usages on local links in routing
>>> something
> A long running research project of mine (over 15 years running)
> has been to find a routing protocol
> that worked well for wireless, and one more holy grail
>> DSCP  field (RFC4774, I recall some historical discussion of why
>> and where), at least something
>> could be said about this. I'd like to add at least a pointer.
>>
GF: I replied in another thread - what I wrote wasn't what I intended to say.

>> could change the reaction to congestion signal, and FEC
>> (even opportunistic CC-friendly FEC)
>> can also change the way things react to congestion signals.
>
>> P.S. My proposal is that if this thread converges I intend to
>> parse a later version and see if my co-author
>> and I can then add some comments to our draft
>> to catalogue a set of issues that we expect to see added in
> Dave TÃ¤ht
> NSFW:
> https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>

So, back to this specific draft.

We (the authors) have already stated that we think this particular draft
should not recommend a specific way of using ECN  and we therefore do not
expect the next version will conclude that "some specific version should
be turned on", but we do expect to more clearly state that ECN marking
should not be violated, i.e. routers / middleboxes shouldn't delete the
codepoint and servers should respond.

Gorry