Subject: [Tsvwg] feedback on draft-ietf-tsvwg-ecn-mpls-00.txt

From: Sally Floyd <[email-masked]>
Date: Sun, 11 Mar 2007 19:36:19 -0700

Bruce -

I read this draft, and it seems fine to me.

I have some feedback on presentation (not on the technical content)
below.

Section 9.1:
"The
situation where traffic is carried beyond a congested LSR only to be
dropped later should become less prevalent as more transports use
ECN."

Perhaps it will become *somewhat* less prevalent, but ECN is not
likely to make it go away as a concern.  Even if all 
congestion-controlled
transports used ECN, this doesn't eliminate the possibility of very
heavy congestion, with buffer overflow and the necessity of dropping
packets.  Unfortunately, I don't know any reason to think that ECN
is a magic bullet against the danger of congestion collapse.  (I
am not arguing against the design decision to use per-domain ECT
checking, just against this sentence as justification.

"Although that scheme did not carry droppable traffic to
the edge of the MPLS domain, we felt this was a small price to pay,
and it was anyway only of concern until ECN had become more widely
deployed."

I don't believe the second half of this sentence for even a second.
Do you have anything to back it up?  That carrying droppable traffic
to the edge of the MPLS domain is "only of concern until ECN had
become more widely deployed"?  You could see my paragraph above.
(Again, I am not arguing against the design decision to use per-domain
ECT checking, but I am arguing against using this sentence as
justification.)

Section 11:

"An alternative proposal currently in progress in the IETF
[I-D.briscoe-tsvwg-re-ecn-tcp] allows the network to prevent
misbehavior by senders or receivers or other routers."

The draft should clarify that at best, this alternate proposal
allows the network to prevent cheating concerning ECN marks (and
not other forms of misbehavior, such as cheating regarding dropped
packets, or the many other forms of possible misbehavior), for packets
marked on links between ingress and egress re-ECN routers.

Nits:
Section 8.1:
"like AF and best efforts" -> like AF and best effort PHBs"?

Section 8.2:
"for single PHB" -> "for a single PHB"
"simply allocated" -> "simply allocating"

- Sally
http://www.icir.org/floyd/

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 17 Mar 2007 18:32:45 +0000

Sally,

Here's my 2c (not discussed with my co-authors):

At 02:36 12/03/2007, Sally Floyd wrote:
>likely to make it go away as a concern.  Even if all congestion-controlled

I'm trying to work out what your concern is here. Obviously, if an interior 
LSR queue grows to overflow, it will drop packets. We have no magic to 
prevent that.

Perhaps the confusion is just our poor wording. In the context of this 
section, "droppable" means packets that are Not-ECN-capable-transport 
(Not-ECT) in the inner IP header, but have been congestion marked (CM) in 
the outer MPLS header. By definition, if a higher proportion of transports 
become ECN-capable, a smaller proportion of traffic will be Not-ECT. So the 
sentence seems to simply be stating a truism.

Or perhaps it's the phrase "only to be dropped later". We didn't mean 
dropped due to congestion. We meant "only to be dropped later when 
decapsulation discovers the inner IP header is Not-ECT.

Below I've suggested tightened up wording.

>ECT checking, but I am arguing against using this sentence as

Here's my suggested improvement for Section 9.1:

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
What are the consequences of marking a packet that is not ECN-capable? Even 
if it will be dropped before leaving the domain, doesn't this consume 
resources unnecessarily?

The problem only arises if there is congestion downstream of an earlier 
congested node. It might be that this second congested LSR forwards packets 
already marked, even though they will be dropped later when the inner IP 
header is found to be Not-ECT on decapsulation. Such packets might cause 
the second LSR to mark (or drop) other packets that it would not otherwise 
have had to.

The situation where traffic is carried beyond a congested LSR only to be 
dropped when decapsulated later would become less prevalent if more 
transports use ECN. This is why we decided to use the per-domain ECT 
checking approach---because if a rare inefficiency is minor even when it 
happens, we figured it is better to absorb it during early deployment 
knowing it could disappear completely later. This is why we chose not to 
use the [Floyd] alternative which introduced a low but persistent level of 
unnecessary packet drop for all time. Although that scheme did not carry 
traffic to the edge of the MPLS domain only to be dropped on decapsulation, 
we felt our minor inefficiency was a small price to pay, and it was anyway 
only of concern until ECN had become more widely deployed.

A partial solution...[para continues unchanged]
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

>marked on links between ingress and egress re-ECN routers.

How about:

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
"An alternative proposal currently in progress in the IETF
[I-D.briscoe-tsvwg-re-ecn-tcp] claims to allow the network to prevent
most misbehavior by senders or receivers or other routers."
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

The only reason for including this para was to explain that, even if re-ECN 
was adopted, we wouldn't have to come back and change MPLS again. I didn't 
intend to get some sort of official sanction into an RFC. The whole para 
could be removed as it refers to a non-WG item, but I thought it was useful 
info.

>"simply allocated" -> "simply allocating"

Thanks. We'll do an update.

Bob

>- Sally
>ttp://www.icir.org/floyd/

____________________________________________________________________________
Notice: This contribution is the personal view of the author and does not 
necessarily reflect the technical nor commercial direction of BT plc.
____________________________________________________________________________
Bob Briscoe,                           Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Sally Floyd <[email-masked]>
Date: Sat, 17 Mar 2007 21:53:44 -0700

> The only reason for including this para was to explain that, even if 
> re-ECN was adopted, we wouldn't have to come back and change MPLS 
> again. I didn't intend to get some sort of official sanction into an 
> RFC. The whole para could be removed as it refers to a non-WG item, 
> but I thought it was useful info.

Well, better, but I still don't really believe the phrase "most
misbehavior by senders or receivers or other routers.".  I would
be happy with "some misbehavior by senders or receivers or other
routers", or by a list of some of the specific misbehaviors that
re-ecn claims to prevent.

However, I assume that there are *many* forms of misbehavior by
senders, receivers, or routers that have nothing to do with cheating
concerning ECN-marking, and that even re-ecn doesn't claim to
prevent.  E.g., senders that ramp up faster than they should, in
the absence of Quick-Start or some other standardized mechanism,
even in there is no congestion at all on the end-to-end path?
Routers that misbehave by advertising false routes, or by otherwise
not following routing protocols?  Receivers that misbehave by not
reporting dropped packets?  Senders that misbehave by initiating a
distributed attack against some remote host?  The world of possible
misbehavior by senders, receivers, and routers is quite vast.  Does
re-ecn really claim to prevent *most* of it?

- Sally

---

From: Sally Floyd <[email-masked]>
Date: Sun, 18 Mar 2007 17:25:33 -0700

Bob -

> I'm trying to work out what your concern is here. Obviously, if an 
> interior LSR queue grows to overflow, it will drop packets. We have no 
> magic to prevent that.
>
> (Not-ECT) in the inner IP header, but have been congestion marked (CM) 
> in the outer MPLS header.

If it is rewritten so that context is explicit *for that sentence*, 
that would
seem fine to me.

...
> Below I've suggested tightened up wording.

Thanks.  That seems good to me.

...
> What are the consequences of marking a packet that is not ECN-capable? 
> Even if it will be dropped before leaving the domain, doesn't this 
> consume resources unnecessarily?
>
> The problem only arises if there is congestion downstream of an 
> earlier congested node. It might be that this second congested LSR 
> forwards packets already marked, even though they will be dropped 
> later when the inner IP header is found to be Not-ECT on 
> decapsulation. Such packets might cause the second LSR to mark (or 
> drop) other packets that it would not otherwise have had to.
>
> The situation where traffic is carried beyond a congested LSR only to 
> be dropped when decapsulated later would become less prevalent if more 
> checking approach---because if a rare inefficiency is minor even when 
> it happens, we figured it is better to absorb it during early 
> deployment knowing it could disappear completely later.  This is why 
> we chose not to use the [Floyd] alternative which introduced a low but 
> persistent level of unnecessary packet drop for all time. Although 
> that scheme did not carry traffic to the edge of the MPLS domain only 
> to be dropped on decapsulation, we felt our minor inefficiency was a 
> small price to pay, and it was anyway only of concern until ECN had 
> become more widely deployed.
>

Well, that seems fine to me, except for the new sentence about "rare"
and "minor" inefficiencies that "could disappear completely later".   I 
think
the sentence could just be deleted.

The new sentence:
" This is why we decided to use the per-domain ECT checking 
approach---because if a rare inefficiency is minor even when it 
happens, we figured it is better to absorb it during early deployment 
knowing it could disappear completely later."

As I said before, I am not arguing with the decision to use the 
per-domain
ECT checking approach, but I don't think it is helpful (or correct) to 
minimize
the potential costs - I think that in general, it is important to 
examine costs as well
as benefits quite clearly.

First:  "if a rare inefficiency is minor even when it happens".
Are you claiming that non-ECN-capable packets marked in an MPLS
tunnel and dropped upon decapsulation would be a "rare inefficiency"
that is "minor"?  Why do you think it would be rare?  (My assumption
would be that in the beginning a large fraction of the traffic in
MPLS tunnels won't be ECN-capable.  Traffic certainly isn't
ECN-capable now.)  So if the MPLS EXP Field for ECN is used, it
might not be rare at all for non-ECN-capable packets to be marked
in the network and dropped upon decapsulation.  It might be
perfectly acceptable, but I don't think it would be rare.

I would be willing to believe that MPLS tunnels with multiple
congested links might be rare, I don't know, but if that is
what you are assuming, then I think you should make that assumption
explicit.  Or delete the word "rare".  Or maybe I don't
understand the sentence...

Second,  "it could disappear completely later"?
Well, perhaps anything *could* happen, but in this case I don't think
it will.  That is, I don't think that non-ECN-capable traffic in MPLS 
tunnels
is going to disappear completely, and I don't know any reason to think
that MPLS tunnels with multiple congested links are going to disappear
completely either.  So that would mean that non-ECN-capable packets
marked in an MPLS tunnel, going through another congested link,
and dropped upon decapsulation would not "disapper completely".
(And even if it does "disappear completely", it is not the kind of
thing that one should seem to be counting on, in an internet-draft.)

Sorry to get stuck on another sentence that needs rephrasing...

- Sally

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 19 Mar 2007 16:06:23 +0000

Sally,

At 04:53 18/03/2007, Sally Floyd wrote:
>>re-ECN was adopted, we wouldn't have to come back and change MPLS again. 
>>I didn't intend to get some sort of official sanction into an RFC. The 
>>whole para could be removed as it refers to a non-WG item, but I thought 
>>it was useful info.
>

I'd be happy with
         "...claims to prevent certain congestion-related misbehavior...".
How about that? It side-steps whether it's some or most, which isn't 
relevant in this draft anyway.

>even in there is no congestion at all on the end-to-end path?

OK, I guess we were assuming a congestion-related context was implicit in a 
draft about congestion. But, yes, I meant congestion related misbehaviour, 
not misbehaviour generally (I have a protocol for that too, it's called UAP 
- the universally altruistic protocol :)

Quick-Start is about two separate things:
i) partly about avoiding the _risk_ of congestion an RTT later and
ii) partly about resource allocation in the absence of congestion.
Personally I think we'd be on a lot safer ground if we made sure a protocol 
didn't try to cover both, as the latter is a completely arbitrary field 
where we should allow operators and users as much freedom to do whatever 
they want AS LONG AS the former (the _risk_ of causing congestion) is under 
control.

I'm focused on enforcing congestion control because that's what matters (to 
me). Re-ECN tries to ensure the sender must repeat any congestion feedback 
it has seen so its network ingress can see it too. Then they both have the 
same congestion information at nearly the same time. Clearly this won't be 
useful to prevent cheating about things that aren't to do with congestion.

I was happy for Quick-Start to go to experimental RFC because I had no 
problem with you experimenting in your way as it didn't seem to stop others 
experimenting in other ways. But that doesn't mean a requirement has been 
created for us to have to enforce the part of Quick-Start behaviour that 
has little to do with congestion control.

Here's text I suggested (offline) could go in your alt-cc draft at that 
time when I missed that Mark & you had already agreed it would be better to 
say nothing more about spare capacity than had already been said about 
fairness. I still believe congestion control is about controlling the 
_risk_ of congestion as much as actual congestion, so I'd be happy to see 
this sort of text in any guidance on congestion control.

===================================================================
If alternate congestion controllers send at high rates during times when 
they believe there is no congestion, they always run the risk that traffic 
in flight will cause congestion. Proposed congestion control mechanisms 
that aim to use spare capacity should be evaluated on how they minimise the 
risk of causing severe congestion that will not be seen until at least a 
round trip later.

Similar to point (1), alternate congestion controllers using spare capacity 
that risk a significantly negative impact on traffic using standard 
congestion control may be suspect and this aspect should be part of the 
community's decision making with regards to the suitability of the 
alternate congestion control mechanism.
===================================================================

>not following routing protocols?

See new text above - of course I meant "congestion-related misbehaviour".

>Receivers that misbehave by not
>reporting dropped packets?

I'm the first to admit that re-ECN can only effectively police transports 
that say they are ECN-capable (We say in the I-D that as re-ECN becomes 
more widely deployed, non-ECN-capable traffic could increasingly be 
rate-limited at the network ingress).

I promised at the last IETF that we would put out a draft to complement 
re-ECN at the transport layer for this case. We've done that, at least for 
TCP (in the TCPM w-g), see draft-moncaster-tcpm-rcv-cheat-00.txt

There's a 3x3 matrix of possible cases:

      ECN-capable flow?|       Yes       |  No   |
                       +--------+--------+       |
                       |   no   |        |       |
untrusted             |  drop  |   drop |       |
----------------------+--------+--------+-------+
   sender              | - R -  |   ~R - | - - - |
   receiver            | N R -  | N ~R T | - - T |
   both                | - R -  |   ~R - | - - - |

The letters say in which scenario a scheme is effective, with:
  N = ECN nonce
  R = Re-ECN
~R = Re-ECN has some (limited) beneficial effect
  T = rcv-cheat test (TCP-only)

You might ask what "~R" really means? If an ECN flow passes through a 
resource that has become so congested it is dropping packets, re-ECN still 
has a role because the receiver is uncertain whether the drops are at a 
congested router, or the re-ECN dropper, which can tell if the receiver 
fails to feed-back such congestion events.

Also, during persistent losses (e.g. due to a DoS attack) re-ECN was 
designed to protect non-DoS traffic from DoS traffic (see paper on re-ECN 
for protecting against DDoS below).

>Senders that misbehave by initiating a
>distributed attack against some remote host?

Re-ECN does claim to greatly raise the bar against DDoS. We also claim 
re-ECN would strongly encourage universal deployment of other DDoS 
defences. These are perhaps rather optimistic claims, but we are working on 
formalising the logic given in the following initial paper:

"Using Self-interest to Prevent Malice;
Fixing the Denial of Service Flaw of the Internet"
<http://www.cs.ucl.ac.uk/staff/B.Briscoe/pubs.html#refb_dplinc>

>The world of possible

You're right, we can't stop the whole of the criminal underworld, sorry if 
I implied that. But certainly most of the Internet congestion-related 
underworld. Even possibly DDoS, which is a congestion response problem.

Cheers

Bob

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Sally Floyd <[email-masked]>
Date: Mon, 19 Mar 2007 09:33:02 -0700

> relevant in this draft anyway.

Thanks, that sounds good to me.

- Sally

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 19 Mar 2007 23:19:37 +0000

Sally,

At 00:25 19/03/2007, Sally Floyd wrote:
>>congested node. It might be that this second congested LSR forwards 
>>packets already marked, even though they will be dropped later when the 
>>inner IP header is found to be Not-ECT on decapsulation. Such packets 
>>might cause the second LSR to mark (or drop) other packets that it would 
>>not otherwise have had to.
>>
>>use the [Floyd] alternative which introduced a low but persistent level 
>>of unnecessary packet drop for all time. Although that scheme did not 
>>carry traffic to the edge of the MPLS domain only to be dropped on 
>>decapsulation, we felt our minor inefficiency was a small price to pay, 
>>and it was anyway only of concern until ECN had become more widely deployed.
>>
>and "minor" inefficiencies that "could disappear completely later".   I think
>approach---because if a rare inefficiency is minor even when it happens, 
>we figured it is better to absorb it during early deployment knowing it 
>could disappear completely later."
>
>As I said before, I am not arguing with the decision to use the per-domain
>the potential costs - I think that in general, it is important to examine 
>costs as well

You're right in this last point - the assumption was that MPLS tunnels with 
multiple congested links would be rare.

This effectively follows from a more basic assumption that congestion 
levels on LSRs are likely to be v low nearly all the time (even if PCN 
marking on virtual queues), but true, this might not be a valid assumption 
for the future.

Yes, we should express these assumptions explicitly, or not put in the 
sentence at all. I prefer to give design rationale in a draft, rather than 
just give the final result of the thought process.

>it will.  That is, I don't think that non-ECN-capable traffic in MPLS tunnels

You're right. This "disappearing completely" should go.

Here's another attempt:

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
The problem only arises if there is congestion downstream of an earlier 
congested queue in the same MPLS domain. Downstream congested LSRs might 
forward packets already marked, even though they will be dropped later when 
the inner IP header is found to be Not-ECT on decapsulation. Such packets 
might cause the downstream LSRs to mark (or drop) other packets that they 
would otherwise not have had to.

We expect congestion will typically be rare in MPLS networks, but it might 
not be. The extra unnecessary load at downstream LSRs will not be more than 
the fraction of marked packets from upstream LSRs, even in the worst case 
where no transports are ECN capable. Therefore the amount of unnecessary 
marking (or drop) on an LSR will not be more than the product of its local 
marking rate and the marking rate due to upstream LSRs within the same 
domain---typically the product of two small (often zero) probabilities.

This is why we decided to use the per-domain ECT checking 
approach---because the most likely effect would be a very slightly 
increased marking rate, which would result in very slightly higher drop 
only for non-ECN-capable transports. We chose not to use the [Floyd] 
alternative which introduced a low but persistent level of unnecessary 
packet drop for all time, even for ECN capable transports. Although that 
scheme did not carry traffic to the edge of the MPLS domain only to be 
dropped on decapsulation, we felt our minor inefficiency was a small price 
to pay. And it would get smaller still if ECN deployment widened.
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

>Sorry to get stuck on another sentence that needs rephrasing...

That's fine. That's what we want reviews for.

Cheers

Bob

---

From: Sally Floyd <[email-masked]>
Date: Mon, 19 Mar 2007 17:20:27 -0700

> You're right in this last point - the assumption was that MPLS tunnels 
> with multiple congested links would be rare.
>
> marking on virtual queues), but true, this might not be a valid 
> assumption for the future.
>
> sentence at all. I prefer to give design rationale in a draft, rather 
> than just give the final result of the thought process.
...
> earlier congested queue in the same MPLS domain. Downstream congested 
> LSRs might forward packets already marked, even though they will be 
> dropped later when the inner IP header is found to be Not-ECT on 
> decapsulation. Such packets might cause the downstream LSRs to mark 
> (or drop) other packets that they would otherwise not have had to.
>
> We expect congestion will typically be rare in MPLS networks, but it 
> might not be. The extra unnecessary load at downstream LSRs will not 
> be more than the fraction of marked packets from upstream LSRs, even 
> in the worst case where no transports are ECN capable. Therefore the 
> amount of unnecessary marking (or drop) on an LSR will not be more 
> than the product of its local marking rate and the marking rate due to 
> upstream LSRs within the same domain---typically the product of two 
> small (often zero) probabilities.
>
> increased marking rate, which would result in very slightly higher 
> drop only for non-ECN-capable transports. We chose not to use the 
> [Floyd] alternative which introduced a low but persistent level of 
> unnecessary packet drop for all time, even for ECN capable transports. 
> Although that scheme did not carry traffic to the edge of the MPLS 
> domain only to be dropped on decapsulation, we felt our minor 
> inefficiency was a small price to pay. And it would get smaller still 
> if ECN deployment widened.

Bob, many thanks.  That seems good to me.  And quite clear.

- Sally

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 20 Mar 2007 10:30:33 +0100

> At 04:53 18/03/2007, Sally Floyd wrote:

<snip> (of claim about re-ecn)
<snip>
Seems right to me too.

> TCP (in the TCPM w-g), see draft-moncaster-tcpm-rcv-cheat-00.txt

I'll send comments on this later. I'm also not sure that doing this as a TCP
method is the right place to do this - one drawback is that this then
requires each transport to have it's own method of detecting the effects of
detecting these effects.

>       ECN-capable flow?|       Yes       |  No   |

So ECN-N also protects from cheating down-stream routers that happen to lie
(or routers reset the ECN-N bits back to ECT, for whatever reason).

My understanding was that this behaviour is checked, no matter what egress
point you may have from the network (mobile and alternate routing via
several egress points). Important to me when I have to pass through
untrusted "router clouds". So, if I understand right, "N" also applies to
untrusted routers, whereas T does not?

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 20 Mar 2007 13:54:31 +0000

Gorry,

At 09:30 20/03/2007, Gorry Fairhurst wrote:
> >---snip---

I guess I'll see that discussion on TCPM. We said in the draft that we 
believe the technique is portable to other TCP-like transports (e.g. SCRP), 
but I doubt it's directly portable to unreliable transports like DCCP and 
definitely not UDP. That's why it's in TCPM, not TSVWG.

We should also think about a general transport capability to test for 
cheating receivers, but it was more pressing to produce something that 
cheating receivers couldn't just respond to with "I don't support this new 
option (like the right not to incriminate yourself in some legal systems!)".

> >       ECN-capable flow?|       Yes       |  No   |
       router              | N R -  | n ~r t | - - t |

>untrusted routers, whereas T does not?

True, assuming you're talking about ECN-capable flows, which T isn't 
designed to be applicable to.

I thought about adding an untrusted router row, but was in a hurry. I've 
added it above.

It's essentially the same as the untrusted receiver row, except I've put 
the letters in lower case wherever there's cheating about drop not ECN 
marking. That's not because the techniques are any less capable of 
detecting cheating, it's just that it would be quite hard for a downstream 
router to conceal an upstream loss by magically recreating a packet that 
had been dropped upstream (or by sending an ACK for a gap in the sequence 
space).

Cheers

Bob