Subject: [Tsvwg] Does anyone have a defence in favour of flow rate fairness?

From: Bob Briscoe <[email-masked]>
Date: Fri, 15 Jun 2007 20:32:01 +0100

TSVWG,

At the Prague IETF, I asked for the ideas discussed in the I-D below to be 
considered as a WG item (but written more diplomatically than I did).
"Flow Rate Fairness: Dismantling a Religion"
<draft-briscoe-tsvarea-fair-01.pdf> or via:
<http://www.cs.ucl.ac.uk/staff/B.Briscoe/pubs.html#rateFairDis>

Gorry Fairhurst voiced concern that the consensus was unlikely to have 
shifted so quickly away from TCP fairness for this to be a feasible WG 
item. But I'd like to hear someone argue against cost-fairness or for flow 
rate fairness (or specifically for TCP-fairness), so we can actually debate 
this issue.
<http://www.cs.ucl.ac.uk/staff/B.Briscoe/pubs.html#rateFairDis>

 From the show of hands in Prague, more than half of a big room had read 
the I-D. Given the I-D provocatively challenges anyone who wants to 
continue to use flow rate fairness to either defend it against the 
arguments or stop promoting it, I'm assuming that a lack of defence implies 
surrender. :)

It would also be good to hear from anyone who believes what I said is the 
right direction. Particularly anyone who is willing to say it changed their 
mind.

More specifically, does the TSVWG want to take on as a WG item the 
production of an informational RFC that would give a plan for the evolution 
of fairness control from where we are now, to a future where fairness 
policy can be controlled and enforced (or not) by more than just endpoints?
- stating that we have a problem and what the problem is
- stating what a good solution might look like (who/what might take control 
etc).
- stating which fairness metrics are valid and which aren't
- stating what implications the approach will have for evolution from where 
we are
- etc

I'm about to go off list for a week, so I hope there's a debate on this in 
my absence, and I'll join in on my return, or I'll provoke the discussion 
again if it hasn't happened :)

Bob

____________________________________________________________________________
Notice: This contribution is the personal view of the author and does not 
necessarily reflect the technical nor commercial direction of BT plc.
____________________________________________________________________________
Bob Briscoe,                           Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Jun 2007 11:48:34 -0700

Bob Briscoe wrote:
> TSVWG,
> 
> At the Prague IETF, I asked for the ideas discussed in the I-D below to
> be considered as a WG item (but written more diplomatically than I did).
...
> arguments or stop promoting it, I'm assuming that a lack of defence
> implies surrender. :)

There's a third option not presented that might fit more closely:

"TCP's behavior defines a kind of fairness; staying inside the bounds of
that fairness is what we currently defend, largely because it's deployed
and appears to currently be reasonably sufficient"

That does NOT mean we all consider TCP's behavior "flow rate fair", but
rather that TCPs sharing a bottleneck will get some proportion of
resources and both make progress, and that new CC mechanisms should not
upend that 'balance' - whatever it currently is.

The next steps proposed below imply that our IETF WG would engage in
IRTF-style investigation of what fairness even means. Whether we have
consensus on going down that path, I don't agree that the IETF is the
place where that work would occur. It seems like IRTF work at best.

> production of an informational RFC that would give a plan for the
> evolution of fairness control from where we are now, to a future where
> fairness policy can be controlled and enforced (or not) by more than
> just endpoints?
> - stating what a good solution might look like (who/what might take
> control etc).
> - stating what implications the approach will have for evolution from
> where we are
> - etc

Joe

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 Jun 2007 14:13:12 +0100

> shifted so quickly away from TCP fairness for this to be a feasible
> WG item. But I'd like to hear someone argue against cost-fairness or
> for flow rate fairness (or specifically for TCP-fairness), so we
> can actually debate this issue.

> the I-D. 

True. You have seem to have attention - new ideas are fine. I looking
forward to seeing how this develops.

> Given the I-D provocatively challenges anyone who wants to

That is not how I see this.

> - stating what implications the approach will have for evolution from

To me, the current revision -01, judges history, and although fine in a
research environment, taking this topic as a WG item would need an
interpretation that is endorsed by the WG, and relating this to the
architecture. There are reasons why the current methods have evolved. I do
not share the view this is just "myopia".

Are you planning a new revision of the I-D prior to the next meeting?

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Jun 2007 20:59:52 +0100

Joe,

just returned from a week off...

At 19:48 21/06/2007, Joe Touch wrote:

...

> implies surrender. :)

There's a third option not presented that might fit more
closely:

"TCP's behavior defines a kind of fairness; staying inside the
bounds of

that fairness is what we currently defend, largely because it's
deployed

That does NOT mean we all consider TCP's behavior "flow rate
fair", but

resources and both make progress, and that new CC mechanisms should
not

Hmmm. TCPs each 'make progress' is rather clutching at straws, isn't
it?

If "upend" means "disturb", I'm afraid any change to
allocations and not upending are incompatible. If a new allocation of a
bottleneck resource is chosen in favour of an old one, the balance of the
old one will be upended. You can't give to some without taking from
others, when you're at the limit.

If "upend" means something more drastic than just disturb, then
we have problems, because we need to define some degree of 'reasonable
gradualness' for a change from an old regime to a new one. That's not
only hard, but it would require some objective goal for what the
gradualness is trying to achieve in order to know whether we were
achieving it or not.

My proposed solution to this (in
draft-briscoe-tsvwg-re-ecn-tcp-03.txt)
was to allow operators to gradually tighten the controls they put in if
they choose to. Then gradualness becomes their problem, and the 'best'
gradualness becomes an outcome of market selection.

The next steps proposed below imply
that our IETF WG would engage in

IRTF-style investigation of what fairness even means. Whether we
have

consensus on going down that path, I don't agree that the IETF is
the

I agree entirely, but I much prefer the IRTF to be doing work that the
IETF has bounced to it, rather than work the IETF says it doesn't need
when it pops out of the IRTF some years hence. That's why I brought this
to the IETF first (we did also have a long discussion about it at the
last ICCRG).

I see Sally is planning a draft on this subject too, so we should decide
on a venue for this discussion fairly sharpish if it's taking too much
TSVWG time from day-to-day protocol business.

Bob

Notice: This contribution is the personal view of the author and does not
necessarily reflect the technical nor commercial direction of BT
plc.

Bob
Briscoe,
Networks Research Centre, BT Research

B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5
3RE,UK.    +44 1473 645196

---

From: Joe Touch <[email-masked]>
Date: Mon, 25 Jun 2007 21:58:55 -0700

...
> Hmmm. TCPs each 'make progress' is rather clutching at straws, isn't it?

It's a straw that is already firmly in our grasp. You're current
proposal asks us to take a few things on faith:

1) that information about the remaining path is more important than
information about the investment thus far
	i.e., using "set TTL so it's 16 at the dest" means
	we know a lot about how many hops are left, but less
	about how many hops have been made from the origin

2) that information about how resources are used along a path is useful
in a multiparty congestion control system
	vs., e.g., the fact that all component connections react
	the same way, and that congestion relief occurs more rapidly
	than congestion exploration

3) that your particular system is a good way to do either of the above

Just asserting that "leave it alone" isn't good enough (which would
likely have strong support), doesn't mean that this particular proposal
is (or is not) a better way forward.

Joe

---

From: Mark Allman <[email-masked]>
Date: Tue, 26 Jun 2007 12:19:39 -0400

Bob-

I agree with a few others who have chimed in on this thread.  What do we
need to defend the status quo against?  If there was an actual choice
here then perhaps one could talk in terms of mounting a defense or
agreeing to go some different direction.  However, we have what we have.
I cannot simply switch to something else if I wanted to.

I read this all as you want to (a) get rid of best-effort traffic in
favor of a pay-as-you-congest model and (b) you want us to obsolete
end-point congestion control as is currently specified by the IETF.  I
share Gorry's doubt that there is consensus (or even close) on either of
these points.

I would be perfectly happy to see new techniques to do different sorts
of control pursued.  I.e., I would be perfectly happy to see a framework
developed to do cost-based congestion control.  But, that does not mean
I think it is useful to---in the absence of this something else---think
that we should abandon what is currently working, however imperfectly.

Sally and I put together a quick draft.  In the intro it says:

    This document was motivated by [B07], an internet-draft on "Flow
    Rate Fairness: Dismantling a Religion" that asserts in the abstract
    that "Comparing flow rates should never again be used for claims of
    fairness in production networks."  This document does not attempt to
    be a rebuttal to [B07], or to answer any or all of the issues raised
    in [B07], or to give the "intellectual heritage" for flow-based
    fairness in philosophy or social science, or to commit the authors
    of this document to an extended dialogue with the author of [B07].
    This document is simply a separate viewpoint on some related topics.

The draft is rough and we have not actually submitted it yet, but will
before the deadline.  It is available from the URLs below and we'd
certainly be happy to hear comments on it.

    http://www.icir.org/floyd/papers/draft-floyd-tsvwg-besteffort-00b.txt
    http://www.icir.org/floyd/papers/draft-floyd-tsvwg-besteffort-00b.ps

allman

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 26 Jun 2007 18:19:24 +0100

Gorry,

>That is not how I see this.

OK.

I only said this because the IETF's hallmark is technical discussion of the 
issues in order to develop protocols based on technical merit.

>To me, the current revision -01, judges history,

I'm v happy to re-work the whole I-D (with co-author(s)) to be more forward 
looking.

>and although fine in a

Certainly 20yrs ago TCP congestion ctrl solved a serious and immediate 
problem and it has served us well since. But the world has changed a lot 
since then and is now starting to extend its behaviour (ie many more flows, 
much longer flows) way outside the pragmatic assumptions behind TCP 
fairness. What was done 20 years ago was good, but the time has come to 
move on. Sticking with TCP-fairness from now on and into the future would 
indeed be myopic.

>Are you planning a new revision of the I-D prior to the next meeting?

Before the Chicago meeting I will simply add clearer answers to a few 
questions I keep getting.

After Chicago, I would like to see a big re-work to make it more forward 
looking and less damning of the past - perhaps along the lines I suggest 
above. Louise Burness and Toby Moncaster here at BT are keen to help, but 
co-author(s) who have worked closely on TCP-fairness over the years would 
be more useful to add a different perspective.

Are there any other volunteers?

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 26 Jun 2007 19:50:10 +0100

Joe,

>information about the investment thus far

I think there's a confusion here. The re-feedback protocol is designed to 
reveal (D) downstream and (E) e2e path info to network elements, as well as 
(U) upstream info already visible. They can then use whichever is 
appropriate for different purposes...

<---------E-------------->
<--U--><------D---------->

If a network wanted to police each flow's rate response to congestion 
(which is possible but not necessary), it would use (E) e2e-path congestion

If a network wanted to limit the bulk volume of congestion a user was 
dumping into it, it would use (D) downstream congestion, simply to avoid 
accounting for congestion that the user was causing in its own network.

Neither need to be taken on faith:
(E) is what senders use for congestion response today, so the network 
ingress seeing the same info a fraction of a RTT later allows it to mirror 
the algo of the sender.
(D) is the same as (E) at the ingress to a public internetwork if we assume 
that congestion of the user's own local network is their concern. (D) 
allows a flat priced emulation of the congestion pricing that Kelly proved 
optimises global utility under reasonable assumptions using tried and 
tested methodology. Others have relaxed the assumptions and found 
congestion volume is still the useful metric in all cases.

Kelly showed that, once sources are accountable for the congestion they 
cause they will want to prioritise (weight) their individual flows to fit 
within the overall bulk envelope they choose. So we won't need per-flow 
policing.

(D) also enables inter-domain accountability to propagate recursively, by 
simple linear additivity.

When we are talking about policing at the ingress to an internetwork, the 
jump from (E) to (D) isn't a leap of faith, it's just saying let's assume 
the user's machines all sit at her interface with her public provider. She 
can 'unfairly' share her own resources as much as she wants.

 From then on the jump from policing single flows to only policing bulk 
response to congestion is theory, but certainly not faith. And the theory 
and its assumptions have been extended and checked ad infinitum by other 
researchers over the past decade without particular cause for concern. And 
we/others have built prototypes, built simulations etc.

Anyway, you also have (E) to police individual flows if you don't believe 
the theory-based jump to bulk policing. Spoilt for choice.

>         about how many hops have been made from the origin

Re-feedback of TTL is NOT something I would recommend. That's why it's in 
an appendix of draft-briscoe-tsvwg-re-ecn-tcp-03.txt. The TTL re-feedback 
idea was only necessary to show we could police the RTT dependence of 
TCP-fairness (you may have noticed I'm not a great fan of TCP-fairness any 
more).

Policing bulk congestion volume is sufficient to ensure that sources on 
paths with longer RTTs are more careful to avoid causing congestion that 
they will not be able to correct for until an RTT later. This ensures 
there's an incentive for the dynamics to be more cautious for longer RTTs. 
But flows with different RTTs but the same weight could still eventually 
converge on the same rate - the longer RTT one would just want to take 
longer to get there.

>         than congestion exploration

"Info about how resources are used along a path" is exactly what Internet 
senders use today to do multiparty congestion control. All the proposal 
adds is making this info visible to the entity (the network) that has the 
incentive to make sure the endpoints respond appropriately to it.

I'm not proposing we change anything about the dynamics of the algorithms 
(relief/exploration). All that is necessary is to allow different transfers 
between apps to behave as if they are w flows, where w is a weight that may 
be <1 or >1.

But, more freedom should be possible to allow experimentation (as with the 
slower dynamics of TFRC).

However, if in the long distant future we find that everyone must "react 
the same way" or more likely with some bound on their dynamics to keep the 
network stable, ingress networks will have an incentive to enforce that, 
and the necessary info and the control.

So revealing info to the network provides evolutionary pressure towards 
cost-fairness /and/ away from instability.

>is (or is not) a better way forward.

I totally agree that I'm saying four things:
1) flow rate fairness is pants (solid argument)
2) cost-fairness is ultility-optimal in theory (solid argument)
3) we have a reasonable implementation for cost fairness that wouldn't 
require changes to ISP flat pricing (to be worked over by the community)
4) once you have a cost-fairness mechanism, sub-communities can arrange 
their own different fairness regimes between themselves (arm-wavy but 
self-evident)

If I had just blown apart flow rate fairness, without an alternative I 
would (rightly) have been given a few cycles of people's attention then 
ignored.

Methinks you do expect too much.

Bob

---

From: Joe Touch <[email-masked]>
Date: Tue, 26 Jun 2007 19:02:31 -0700

Bob,

These are all good arguments for IRTF study. See below.

Bob Briscoe wrote:
> I think there's a confusion here. The re-feedback protocol is designed
> to reveal (D) downstream and (E) e2e path info to network elements, as
> well as (U) upstream info already visible. They can then use whichever
> is appropriate for different purposes...

How do you reveal all that information in a single existing field? You
only get one of upstream or downstream; either you start with a known or
you end with a known. You can't have both.

> ingress seeing the same info a fraction of a RTT later allows it to
> mirror the algo of the sender.

Agreed.

> (D) is the same as (E) at the ingress to a public internetwork if we
> assume that congestion of the user's own local network is their concern.

That would be a specific protocol for the ingress to public
internetworks with that assumption. That would be one proposed protocol.

> (D) allows a flat priced emulation of the congestion pricing that Kelly
> proved optimises global utility under reasonable assumptions using tried
> and tested methodology. Others have relaxed the assumptions and found

That would be a case to be made to the IETF as a whole, to get everyone
on board with your assumption of a goal for this work. I don't think it
can be assumed, regardless of what has been studied to date.

> cause they will want to prioritise (weight) their individual flows to
> fit within the overall bulk envelope they choose. So we won't need
> per-flow policing.

The above statement appears to assume that per-source policing is in
effect - i.e., that sources are accountable. That's still fairly
fine-grained, isn't it?

> (D) also enables inter-domain accountability to propagate recursively,
> by simple linear additivity.
> 
> When we are talking about policing at the ingress to an internetwork,
> the jump from (E) to (D) isn't a leap of faith, it's just saying let's
> assume the user's machines all sit at her interface with her public
> provider. She can 'unfairly' share her own resources as much as she wants.

Right, but under current nets she can basically emulate as many "she"s
as she wants; that's how we get HTTP opening multiple connections. Won't
this just encourage multiple IP addresses to get around per-source policing?

> response to congestion is theory, but certainly not faith. And the
> theory and its assumptions have been extended and checked ad infinitum
> by other researchers over the past decade without particular cause for
> concern. And we/others have built prototypes, built simulations etc.
> 
> Anyway, you also have (E) to police individual flows if you don't
> believe the theory-based jump to bulk policing. Spoilt for choice.

(agreed that RTT is a red herring)

...
> "Info about how resources are used along a path" is exactly what
> Internet senders use today to do multiparty congestion control

Endpoints know what resources are available between those endpoints;
that could include multipath. Nothing about today's (ubiquitous)
congestion control is dependent on a particular path per se. The
exception would be RSVP and/or XCP which touch routers on a path, but
they're not exactly widespread (and that's part of the problem we're
grappling with).

> All the
> proposal adds is making this info visible to the entity (the network)
> that has the incentive to make sure the endpoints respond appropriately
> to it.

Why is it you think that endpoints aren't already responding
appropriately to the information they get? How will this new system
handle virtual paths and multipath?

> I'm not proposing we change anything about the dynamics of the
> algorithms (relief/exploration). All that is necessary is to allow
> different transfers between apps to behave as if they are w flows, where
> w is a weight that may be <1 or >1.

That seems nice in principle, but in practice what's to stop an app from
faking being multiple apps to get more capacity? or multiple endpoints?
Basically, what's to stop more 'gaming the system'?

I agree that the current system can be gamed - we don't really have
per-connection (it's more like per-connection with RTT weighting, but
not exactly), and we can open multiple connections to game it. However,
replacing one game-able system with another doesn't solve that problem.

> But, more freedom should be possible to allow experimentation (as with
> the slower dynamics of TFRC).

That is a ringing endorsement for an IRTF effort.

> the same way" or more likely with some bound on their dynamics to keep
> the network stable, ingress networks will have an incentive to enforce
> that, and the necessary info and the control.
> 

That, IMO, would be a conclusion from an experiment we have not yet
conducted. I'd like to see the result before coming to that conclusion.

> Methinks you do expect too much.

For the IETF, no. For the IRTF, yes. ;-)

Joe

---

From: "Mark Handley" <[email-masked]>
Date: Wed, 27 Jun 2007 11:18:25 +0100

> For the IETF, no. For the IRTF, yes. ;-)

I don't quite share Bob's zeal on this precise issue, but I do think
we need to have a proper debate on these issues in the IETF at this
point, not the IRTF.

 - The basic theory here is something like ten years old now, and
still looks good.

  - The basic re-feedback idea is simple and effective and has been
through a ton of debate in the appropriate research venues including
the IRTF.  It has matured.

  - The incarnation of re-feedback in Re-ECN is (in my opinion) well
thought through, and probably deployable.  Even if it were used in
ways Bob hasn't foreseen, it could form a very useful building block
for all sorts of congestion monitoring/enforcement mechanisms that are
hard to do today. What remains here is mostly engineering.

 - The real remaining question seems to be about the interaction
between protocols and economics.  This is the question Bob has rather
unsubtly posed.

So, it seems to me that bouncing something like Re-ECN to the IRTF
would serve no useful purpose.  The question is _not_ "would it work?"
(in a mechanistic sense).  I'm pretty confident it would work if it
were widely deployed - at least confident enough for the IETF to take
it on and beat it up with the club of reality.  The question is _not_
would it work (in an ecomonic sense) - there's no way for the IRTF to
test the theoretic basis.  The real question is "do we want it?".

This is a question that the IRTF _cannot_ answer.

This _might_ be a question that the IETF can answer.  I don't honestly
know if the IETF can answer this question - certainly the IETF is very
very poor at answering architectural and economic questions, and this
is both.  It really ought to be the job of the IAB to figure out how
to answer such questions.

If what people are saying is that the IETF cannot or should not answer
this question, then please say so.  If that is the case, then I
suggest that Bob should publish his documents as Experimental, and
take the debate to the people who need to implement them (router
vendors, OS vendors, and the ISPs).  However, many of these people are
present at the IETF, and we also have a fair number of experienced
people with good perspective on the issues.  So it seems to me that
the IETF should not punt on these questions, but should debate them
properly.  They are after all fundamentally very important questions.

 - Mark

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 Jun 2007 14:36:38 +0100

Mark,

[I thought I sent this last night, but apparently not...]

At 17:19 26/06/2007, Mark Allman wrote:

>need to defend the status quo against?

All currently standardised IETF congestion control protocols aim to 
equalise per-flow rates through a bottleneck. For future protocol work, 
this goal should be recognised as misguided as it is likely to lead to 
extreme unfairness in economic terms.

>If there was an actual choice

I'm offering a choice at both the conceptual level (cost-fairness vs flow 
rate fairness) and at the protocol mechanism level (re-ECN which reveals 
path congestion info to network nodes so they can police either form of 
fairness and others).

But in this thread, let's keep it to the choice between fairness goals. 
Once we have consensus on the goal, we can start discussing appropriate 
mechanism.

>I read this all as you want to ...

Both the intentions a) & b) below ascribed to me are the exact opposite of 
the stated intentions in the drafts. The issues are serious, so below I've 
tried to make it clear what we're not saying.

>(a) you want to get rid of best-effort traffic in
>favor of a pay-as-you-congest model and

Most emphatically no, on both counts.

a1) Best effort would stay as it is, except it should give operators reason 
to deploy ECN much more widely so we should see much less drop.
[Also, BTW, re-ECN is applicable to all Diffserv PHBs, not just BE.]

a2) The reason we brought this to the IETF was we had worked out a way for 
operators to ensure users behave fairly within typical flat rate ISP 
contracts WITHOUT pay-as-you-congest. Quoting S.5.2:

"  Note that we use the phrase "you get what you pay for" not just "you
    pay for what you get". In Kelly's original formulation, users had to
    pay for the congestion they caused, which was unlikely to be taken up
    commercially. But the reason we are revitalising Kelly's work is
    that recent advances (Section 5.3.2) should allow ISPs to keep their
    popular flat fee pricing packages by aiming to ensure that users
    cannot cause more congestion costs than their flat fee pays for.
"

a3) In the intro to draft-briscoe-tsvwg-re-ecn-tcp-0* and every time I have 
presented re-ECN and cost-fairness in tsvwg, I have started by explaining 
that the idea is to introduce a protocol field in IP. Full stop. All the 
rest is up to operators and app/OS developers to build policy controls 
around this field if they choose, but the incentives are there to make all 
this happen:
- Enable network B to penalise network A for the congestion A's users cause 
in B and beyond. The IETF would only provide the field in IP to enable this.
- It's up to each network pair whether they would choose this contractual 
model.
- And it's up to network A whether it limits congestion from its users. The 
IETF would provide the field in IP to enable A to do this if it wanted.
- Net A has the incentive to limit congestion from its users because, if it 
doesn't, it could find it pays net B more in penalties than it gets in flat 
fees from all its users.

a4) You can't set BE against pay-as-you-congest anyway - they are in 
different sets:
- Best effort is a service model.
- Pay-as-you-congest is a payment model.

>(b) you want us to obsolete
>end-point congestion control as is currently specified by the IETF.

b) Most emphatically no again.

I do want to enable evolution of many new forms of endpoint congestion 
control within an overall envelope policed by networks - to ensure more 
responsible sharing of resources on short and long timescales.

I do want to enable evolution of currently specified end-point controls so 
that they can be weighted under policy control.

But those that don't evolve will still work just fine - they will just 
either eat up more congestion allowance (or less) than the user might have 
wished. They won't be obsolete, just unloved.

>I

If I had made these points, I agree consensus would be unlikely. But 
happily my motivations are the exact opposite in all cases.

>developed to do cost-based congestion control.

Cool

>But, that does not mean

Totally agree. You may want to take a look at the end of section 9 of the 
-01 rev of the fairness religion draft <draft-briscoe-tsvarea-fair-01.pdf>, 
which says what is likley to happen to currently spec'd congestion controls 
if cost-fairness were to become the norm.

>     This document is simply a separate viewpoint on some related topics.

Sounds like useful stuff.

>     http://www.icir.org/floyd/papers/draft-floyd-tsvwg-besteffort-00b.ps

I'm trying to meet the initial draft deadline myself for a couple of drafts 
(fitting in 2-day car journeys to my son's graduation as well), so I may 
have to defer reading. But if I get a chance before the deadline, I will 
send comments. It's possible the clarifications in this email may have 
allayed some concerns.

BTW, I assume the .ps is redundant?

Cheers

Bob

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 Jun 2007 15:37:13 +0100

Mark, Sally,

I haven't done more than scanned your useful doc, but just wanted to say 
you're pushing at an open door.

I want to ensure that what you call 'simple BE traffic' can co-exist in a 
principled way on an internetwork with what one might call 'accountable BE 
traffic'. In such a way that the balance between the two is determined by 
natural selection.

As I've outlined in every talk I've given on re-ECN at the IETF, I expect 
there to be networks that will NOT deploy any enforcement of fairness. 
Re-ECN was designed for permanent partial deployment (only some hosts and 
only some networks).

But at inter-domain borders, those liberal networks that don't limit their 
users in any way would still be accountable for the congestion their users 
cause in other networks that do care. That needn't kill such liberal 
networks, which I expect will always exist. They will simply spread the 
cost of their excessive users over all their users - as they have to do today.

But we might still find that liberal networks draw a line somewhere. E.g. a 
liberal network might still block sources of excessive congestion (e.g. 
DDoS) which would otherwise cause a liberal network to have to pay other 
networks excessive penalties (assuming the border contracts I predict come 
about).

---
Perhaps you wrongly picked up on my phrase "Comparing flow rates should 
never again be used for claims of fairness in production networks.". This 
was carefully worded to mean only what it says: No-one should /claim/ this 
traffic is fair.

It most emphatically does NOT mean I don't want this sort of traffic on the 
Internet. My personal preference is for simple, but I'm not setting myself 
up to judge what is good for the Internet or not, I'm offering a natural 
selection mechanism to decide between the two.

---
I do slightly resent the general implication that what I'm proposing will 
make the Internet more complicated. My main motivation isn't optimality. 
It's simplicity. But not just to meet your goal of "One part of the 
Internet should remain simple". Rather, my goal is "What's the simplest way 
for all the different cultures and parts of the internetwork (traditional 
public ISP, cellular network, ad hoc wifi, NGN, corporate intranet, campus 
etc) to all interconnect but all keep their own resource control cultures 
along a spectrum from laissez faire to anally retentive".

I said optimality isn't my goal. The odd 10% sub-optimality is fine, but 
straying too far from it causes pathological knock-on effects that lead to 
excess complexity, as I shall now explain. Flow rate fairness gives a user 
who has been running n flows for 24 hours, n times more bandwidth than 
another flow running for a few seconds. This has led to an Internet that is 
extremely non-optimal. Operators then have an incentive to intervene (e.g. 
throttling p2p filesharing), because it greatly improves their competitive 
position. They satisfy many more customers for the same cost. Effectively, 
a major cause of the rise of the middlebox has been the broken flow rate 
fairness goal.

In the face of all the conflicting demands on the Internet, complex boxes 
are being deployed at borders and at user access interfaces to try to 
'manage' selfish or malicious traffic. It's blocking evolution of new apps 
and increasingly causing bizarre feature interactions, with apps silently 
not working intermittently. That motivated me to propose truly simple 
mechanisms that manage the thing that most of these middleboxes are trying 
to deal with: control over sharing network resources. Hence my slight 
resentment at implications that I'm against the simple Internet.

Bob

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Jun 2007 12:26:47 -0400

> >If there was an actual choice here then perhaps one could talk in
> >terms of mounting a defense or agreeing to go some different
> >direction.  However, we have what we have.  I cannot simply switch to
> >something else if I wanted to.
> 
> I'm offering a choice at both the conceptual level (cost-fairness vs
> flow rate fairness) and at the protocol mechanism level (re-ECN which
> reveals path congestion info to network nodes so they can police
> either form of fairness and others).

My point is that conceptual mechanisms and schemes from research papers
are great but they are not real choices.  I can't twiddle the "something
else" knob here and get something different on this machine than
(roughly) IETF-standardized congestion control.  So, I don't understand
why someone needs to defend the status quo against a concept.

(That does not mean we can't judge this new concept to be worthy of
design/development/further exploration.  See below.)

> Most emphatically no, on both counts.

I think we have a difference in terminology when we say "best effort"
(this is actually why in our draft draft we define "simple best effort
traffic" to avoid such terminology issues).  

That said, I read your note as essentially saying best effort is the
same as it is today, except different.  I.e., it seems that the end game
is that there is no place for simple best effort traffic.

> b) Most emphatically no again.

And, then later you note that "They won't be obsolete, just unloved."
Obsolete / unloved / whatever.  I understand you don't want to
necessarily say "thou shell not use RFC 2581-style CC", but you're
designing a world where that seems to be the intent.

It seems to me that this is all backwards.  I am perfectly happy to
support IETF work on designing mechanisms to support cost-based
congestion control.  That seems fine to me.  That does not mean that I
have to agree that we should get rid of all simple best-effort traffic.
That does not mean I have to sign on to a particular fairness criteria.
It seems to me that this question need not gate work on cost-based
schemes.  IMHO.

allman

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 Jun 2007 18:47:40 +0100

Mark,

>why someone needs to defend the status quo against a concept.

That's perfectly fair. Except, it's a fully spec'd protocol in an I-D, not 
just a research paper. But I accept the criticism - it's not running code 
(yet). Writing that is what we're discussing right now on the 
[email-masked] list.

>traffic" to avoid such terminology issues).

I had issues with your terminology, but I'll deal with them when I've read 
it properly.

>is that there is no place for simple best effort traffic.

Not at all. On a network (or cluster of networks) that doesn't enforce any 
controls, any flows that start and end on that network would be 'simple BE 
traffic'.

- If a flow from this liberal net crossed into an 'enforcing' net, they 
would still be sort-of 'simple BE', but the liberal net (not the user) 
would be accountable for the congestion they caused elsewhere.
- If a flow entered this liberal net from an 'enforcing' net it would not 
be 'simple BE'.

But let's be clear, if it wasn't 'simple BE', it would most likely still 
behave exactly like 'simple BE' unless the sender was indulging in causing 
excessive congestion across all traffic it was generating.

>It seems to me that this question need not gate work on cost-based

[For the list, I've pasted below what I just replied to your off-list mail 
to answer this:]

I thought about this long and hard. You can't have more than one top level 
fairness goal on an internetwork because any flow may traverse any 
resource. However, you can have different fairness regimes between subsets 
of users (see my section on fairness between fairnesses).

Note by fairness "goal" I mean the criteria you use to judge fairness (not 
the relative allocations themselves): ie whether you measure rates or 
congestion volume, whether you compare flows or users.

Bob

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jun 2007 11:29:37 -0700

...
> That's perfectly fair. Except, it's a fully spec'd protocol in an I-D,
> not just a research paper. 
...

Re-ECN is a specific proposal (albeit expired); the Dismantling ID is a
research paper. Which one are we discussing here (I thought dismantling)?

Joe

-- 
----------------------------------------
Joe Touch
Sr. Network Engineer, USAF TSAT Space Segment

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 Jun 2007 23:25:37 +0100

Joe,

>research paper. Which one are we discussing here (I thought dismantling)?

I'd like to keep the discussion to 'dismantling' initially (which I had 
said in an earlier response to Mark just after this bit of text).

But given Mark was wanting code, I was just pointing out (as an aside) that 
the re-ECN protocol was /one/ possible mechanism that could provide 
cost-fairness and that it is a little more than just a research paper.

Bob

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jun 2007 18:19:16 -0700

> said in an earlier response to Mark just after this bit of text).

In that case, I'm with Mark in saying this isn't fully spec'd.
Dismantling-ID isn't a spec at all - there's no 2119 language, no
headers, etc. It's about what *can* be done, but not a specific enough
proposal to evaluate.

> But given Mark was wanting code, I was just pointing out (as an aside)
> that the re-ECN protocol was /one/ possible mechanism that could provide

I don't understand how the IETF would evaluate the Dismantling-ID if
what it does is argue that there could be possible mechanisms (which is
what, IMO, it does). That's definitely IRTF - or even just research
paper - territory. IETF ought to be specific enough to indicate a way
forward.

Joe

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 28 Jun 2007 17:49:27 +0100

Bob,

There are many ways to disagree...

I think writing an IETF document could be useful, now you're offering to 
focus this on the way forward. I suggest to keep active interest, we may 
like to establish the goals/benefits/drawbacks of TCP-fairness, 
cost-fairness, others; an architecture (how will this deal with dynamic 
networks, legacy networks, networks that wish to not do this, etc), and 
transition possibilities.

If you made a document where we could agree the basis, I'd be one of 
those who would volunteer to read and comment on the detail (and if I 
have appropriate expertise I would offer to contribute).

Best wishes,

Gorry

Bob Briscoe wrote:

> I only said this because the IETF's hallmark is technical discussion of 
> the issues in order to develop protocols based on technical merit.
> 
> I'm v happy to re-work the whole I-D (with co-author(s)) to be more 
> forward looking.
> 
>> and although fine in a
> since then and is now starting to extend its behaviour (ie many more 
> flows, much longer flows) way outside the pragmatic assumptions behind 
> TCP fairness. What was done 20 years ago was good, but the time has come 
> to move on. Sticking with TCP-fairness from now on and into the future 
> would indeed be myopic.
> 
> above. Louise Burness and Toby Moncaster here at BT are keen to help, 
> but co-author(s) who have worked closely on TCP-fairness over the years 
> would be more useful to add a different perspective.
> 
> Notice: This contribution is the personal view of the author and does 
> not necessarily reflect the technical nor commercial direction of BT plc.