Subject: Re: Last Call: draft-ietf-tsvwg-cc-alt (Specifying New Congestion Control Algorithms) to BCP

From: Pekka Savola <[email-masked]>
Date: Wed, 23 May 2007 09:24:54 +0300 (EEST)

On Fri, 4 May 2007, The IESG wrote:
> The IESG has received a request from the Transport Area Working Group WG
> (tsvwg) to consider the following document:
>
> - 'Specifying New Congestion Control Algorithms '
>   <draft-ietf-tsvwg-cc-alt-02.txt> as a BCP
>
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action.  Please send substantive comments to the
> [email-masked] mailing lists by 2007-05-18.

Sorry for missing the comment DL by a couple of days.  Too slow to

I believe this document is in a serious need of more work regarding 
its applicability and evaluation criteria.  Some of the TCP community 
may already be aware of (some of) these criteria through oral and/or 
scientific tradition.  However if IETF is requested to perform 
evaluation of these congestion control mechanisms by writing a BCP on 
it, it seems conducting a fair evaluation is impossible if it isn't 
clear enough what is evaluated.

More below.

substantial
-----------

1) the document appears to be slightly inconsistent wrt. what it's
actually specifying, or there are nuances that may not be sufficiently well
articulated.  The introduction speaks of "..evaluating suggested CC
algorithms that _significantly differ_ from the general CC principles outlined
in [RFC2914]" (emphasis mine).  The abstract does not mention 'significantly
differ', and there is Rule (0) which seems to imply that this BCP could
be applied both to the mechanisms that significantly differ from the RFC2914
guidelines and other congestion control mechanisms that still honor the RFC
2914 guidelines.  Which is it?  Does it have implications on the different
'tracks'?

Section 2 also says "Each alternate CC algorithm published is required 
to include a statement in the abstract indicating whether or not the 
proposal is considered safe for use on the Internet".  Which documents 
this applies to is not clear enough: all IETF documents (through WG or 
through an AD)? IAB documents?  IRTF documents?  Individual RFC-editor 
submissions? It is not clear to me whether this document would have 
the authority (without explicit discussion within the RFC-editor 
publications constituencies) to impose further requirements on 
non-IETF document streams especially if it doesn't include 'Updates: 
3932' in its header.  I suspect the document only means to affect the 
IETF RFC publication stream but is not clear enough about it.

Section 4 only talks of 'minimum requirements for a document to be 
approved as Experimental with approval for widespread deployment in 
the global Internet.' and further down "..approval for widespread 
deployment".  What about the rest?  Also, is omitting "in the global 
Internet" intentional? The flow of text doesn't go well as it is if 
that's the case, and if it's intentional, I'd rather use two entirely 
different wordings to make 'encouraged for widespread deployment' and 
'encouraged for widespread deployment in the global Internet' more 
clearly distinct from each other. (Also, it isn't clear if the text is 
out of sync regarding guideline (2) compared to what it says in 
section 3 and what it says here.)

All things considered, the document needs to be much clearer what its scope
is intended to be, and if the document applies to a number of different
kinds of CC algorithms, more clearly define which rules apply to which
categories of documents.

2) the document requires that 'serious scientific study .. needs to have
been done'.  Yet there is no metrics an outsider could use to evaluate
whether this test is met or not.  It may be that TCP congestion control
community has de-facto oral tradition what needs to be evaluated before an
algorithm can be looked at seriously, but based on this proposed BCP, such
metrics are not clear enough.

Unless we can define clearer requirements and metrics for evaluation,
perhaps the IETF should not attempt to make such an evaluation but leave it
to the scientific community.  I'm not sure how the IETF can liaise with the
scientific community on that, e.g., whether it's possible to get a consensus
statement from IRSG or an IRTF RG or something that could meet this
requirement (if we don't want specify these criteria within the IETF).

Also, checklist (2) has "A minimum goal for experimental mechanisms
proposed for widespread deployment in the Internet should
be that they do not perform significantly worse than TCP
in these environments."

However, 'these environments' refers to a non-exhaustive list of scenarios.
This checklist does not provide adequate information
to evaluate whether sufficient testing in the non-exhaustively defined
"difficult environments" has taken place or not.  I do not believe we can
require assessments in various environments unless it's better specified
what which environmets that various refers to.

3) The first evaluation criteria includes ".. should be evaluated when
competing with standard IETF congestion control."

What is that standard IETF congestion control referred to?  RFC 793 plus RFC
1122?  These are the only two IETF _Standard_ specifications I can think of. 
Or does this include proposed standards as well?  What constitutes "IETF
congestion control" or "standard congestion control" (in another place) that
a CC algorithm should be evaluated against?

Please do not cite the TCP roadmap RFC on this.  It's Informational and
not an IETF consensus statement, and as such has no authority to define
what constitutes "standard congestion control".

4) The first evaluation criteria also includes "We also note that this
guideline does not constrain the fairness offered for non-best-effort
traffic."

How does a TCP congestion control algorithm know whether a particular TCP
session is best-effort or non-best-effort?  You likely have an assumption
here that may need to be spelled out.

However, it is not clear why this distinction is even relevant.  The users
may not be able to control whether their traffic is labeled best-effort or
not (to a higher or lower class), and as a result if the user wants to test
a congestion control mechanism, its classification may change without the
TCP implementation knowing it, and as such causing havoc in the network.

Further, in many implementations, I believe it is not possible to choose
which TCP sessions should use a specific congestion control mechanism (I
think on recent Linux there is a setsockopt option for this though).  As
such even if in theory or in RFC a CC algorithm is only meant to be used for
non-BE traffic, in practise it may end up being used for all traffic on a
host due to implementation limitations.

5) Normative references is empty, yet this is a BCP document which, among
other things, referred to "standard congestion control" without providing a
reference (as raised above).  Please move some informational references over
here and/or add applicable references.

editorial
---------

     networks).  Recent research has yielded many alternate congestion
     control schemes ([RFC3649], [HTCP], [FAST], [BIC], [CompoundTCP],
     [XCP], and many more).  Using these new congestion control

==> please remove the references from the abstract per ID-nits.

2.  Status

==> this title seems too short, and a somewhat longer and a more descriptive
one would be useful.  Actually the section seems to be a mix and match of
different topics and a better organization might help the document
considerably.  E.g., if there was a numbered list or subsection of different
"publication paths" and descriptions of each the scope of the document
and the intent of this section would likely be much clearer.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Mark Allman <[email-masked]>
Date: Wed, 23 May 2007 09:54:26 -0400

Pekka-

> actually specifying, or there are nuances that may not be sufficiently
> well articulated.  The introduction speaks of "..evaluating suggested
> CC algorithms that _significantly differ_ from the general CC
> principles outlined in [RFC2914]" (emphasis mine).  The abstract does
> not mention 'significantly differ', and there is Rule (0) which seems
> to imply that this BCP could be applied both to the mechanisms that
> significantly differ from the RFC2914 guidelines and other congestion
> control mechanisms that still honor the RFC 2914 guidelines.  Which is
> it?  Does it have implications on the different 'tracks'?

I think you are reading a little much into this.  I have just tweaked
the language in the document to include 'significantly different' in the
abstract and have changed (0) to this:

    (0) Differences with Congestion Control Principles [RFC2914]

        Proposed congestion control mechanisms that do should include a
        clear explanation of the deviations from [RFC2914].
    
Is that more clear?

> Section 2 also says "Each alternate CC algorithm published is
> required to include a statement in the abstract indicating whether
> or not the proposal is considered safe for use on the Internet".
> Which documents this applies to is not clear enough: all IETF
> documents (through WG or through an AD)? IAB documents?  IRTF
> documents?  Individual RFC-editor submissions? It is not clear to me
> whether this document would have the authority (without explicit
> discussion within the RFC-editor publications constituencies) to
> impose further requirements on non-IETF document streams especially
> if it doesn't include 'Updates: 3932' in its header.  I suspect the
> document only means to affect the IETF RFC publication stream but is
> not clear enough about it.

The document is intended to apply to IETF-produced documents.  I added
the following to the 'Status' section.  Does this help?

    Note: we are not changing RFC publication process for non-IETF
    produced documents (e.g., those from the IRTF or independent
    RFC-Editor submissions).  However, we would hope the guidelines in
    this document inform the IESG as they consider whether to add a note
    to such documents.

> that's the case, and if it's intentional, I'd rather use two
> entirely different wordings to make 'encouraged for widespread
> deployment' and 'encouraged for widespread deployment in the global
> Internet' more clearly distinct from each other. (Also, it isn't
> clear if the text is out of sync regarding guideline (2) compared to
> what it says in section 3 and what it says here.)

I agree the text is a little weird.  I beat it a little.  Does this:

    The minimum requirements for approval for widespread deployment in
    the global Internet include guidelines (1) on assessing the impact
    on standard congestion control, (3) on investigation of the proposed
    mechanism in a range of environments, guideline (4) on protection
    against congestion collapse and guideline (8), discussing whether
    the mechanism allows for incremental deployment.

    For other guidelines, i.e., (2), (5), (6), and (7), evidence
    that the proposed mechanism has significantly more problems
    than those of TCP should be a cause for concern in approval for
    widespread deployment in the global Internet.

work better?

> whether this test is met or not.  It may be that TCP congestion
> control community has de-facto oral tradition what needs to be
> evaluated before an algorithm can be looked at seriously, but based on
> this proposed BCP, such metrics are not clear enough.
> 
> perhaps the IETF should not attempt to make such an evaluation but
> leave it to the scientific community.  I'm not sure how the IETF can
> liaise with the scientific community on that, e.g., whether it's
> possible to get a consensus statement from IRSG or an IRTF RG or
> something that could meet this requirement (if we don't want specify
> these criteria within the IETF).

The IETF and the IRTF have worked out a procedure for working on these
sorts of things (and Lars has documented it in an ION).  

At the end of the day, IMO, the IETF needs to be able to make decisions
about new CC schemes.  The document we wrote lays out a set of areas in
which authors of such proposals need to provide information to the IETF
community such that the IETF community can make sound engineering
judgments about the proposals.  For a document that intends to live for
a long time it is pretty hard to go beyond "sound scientific evaluation"
and a big-picture list of *areas* where we know we want information.

> However, 'these environments' refers to a non-exhaustive list of
> scenarios.  This checklist does not provide adequate information to
> evaluate whether sufficient testing in the non-exhaustively defined
> "difficult environments" has taken place or not.  I do not believe we
> can require assessments in various environments unless it's better
> specified what which environmets that various refers to.

You are right that we may want to better enumerate things.  But, we
cannot necessarily illuminate all such environments that one might
encounter.  I hacked out the following text (to be added, not replacing
anything that is there now).

        While it is impossible to enumerate all possible "difficult
        environments", we note that the IETF has previously grappled
        with paths with long delays [RFC2488], high delay bandwidth
        products [RFC3649], high packet corruption rates [RFC3155],
        packet reordering [RFC4653] and significantly slow links
        [RFC3150].  Aspects of alternate congestion control that impact
        networks with these characteristics should be detailed.

Is that better?

> What is that standard IETF congestion control referred to?  RFC 793
> plus RFC 1122?  These are the only two IETF _Standard_ specifications
> I can think of. Or does this include proposed standards as well?  What
> constitutes "IETF congestion control" or "standard congestion control"
> (in another place) that a CC algorithm should be evaluated against?
> 
> Please do not cite the TCP roadmap RFC on this.  It's Informational
> and not an IETF consensus statement, and as such has no authority to
> define what constitutes "standard congestion control".

Added refs:

        Proposed congestion control mechanisms should be evaluated when
        competing with standard IETF congestion control
        [RFC2581,RFC2960,RFC4340].

> guideline does not constrain the fairness offered for non-best-effort

I don't understand your point here.  All we are saying is that if---by
some means---we know that some flow is not best effort then it can be
subjected to some other criteria then it need not be constrained by the
guideline. 

> other things, referred to "standard congestion control" without
> providing a reference (as raised above).  Please move some
> informational references over here and/or add applicable references.

I have now put the references to RFCs 2581, 2914, 2960 and 4340 under
normative references.

(And, splitting the references was the dumbest thing we ever did---it
causes no end to the haggling.)

> ==> please remove the references from the abstract per ID-nits.

Done.

> 2.  Status
> 
> ==> this title seems too short, and a somewhat longer and a more
> descriptive one would be useful.  Actually the section seems to be a
> mix and match of different topics and a better organization might help
> the document considerably.  E.g., if there was a numbered list or
> subsection of different "publication paths" and descriptions of each
> the scope of the document and the intent of this section would likely
> be much clearer.

I changed the title to "Document Status", but I am not inclined to
re-arrange it further because lots of people have read this now and
nobody has complained.

The major caveat here is that *I* hacked out the changes described
above.  Sally may want to wordsmith or just plain disagree.

allman

---

From: Pekka Savola <[email-masked]>
Date: Sat, 26 May 2007 14:24:39 +0300 (EEST)

Hi Mark,

Thanks for the prompt response, and sorry for a delay in responding.

>        clear explanation of the deviations from [RFC2914].

Yes (though '..that do should..' seemed weird on the first read).

>    this document inform the IESG as they consider whether to add a note

Yes.

>    widespread deployment in the global Internet.

(Maybe you meant to add 'following' before 'guidelines' in the 2nd 
line?)

If 'following' or some such is added, the first paragraph is clearer 
on what the authors and reviewers should do: they should read through 
the listed guidelines and do what's required there.  It might not be 
optimally clear whether the author _must_ do as 'shoulds' say in those 
guidelines or are those just suggestions.  I'd guess you mean either 
"must do" or "must do, or justify why you haven't" but not "may do".

What isn't clear enough is what the authors and reviewers should do 
for the case of 2nd paragraph's guidelines.  Can the authors skip them 
completely, and the burden of proof that the proposal causes 
significantly more problems on the reviewer?

Or do you mean that the proposers should do everything guidelines (2), 
(5), (6) and (7) say, but shortcomings in the results of these 
guidelines (e.g., proposal being only slightly more troublesome than 
TCP) should not block the approval for widespread deployment in the 
global Internet.  Also the same point as above wrt 'shoulds'.

I suspect you mean the latter, but this should be clearer.

> sorts of things (and Lars has documented it in an ION).

(Note: this is still a draft version if you're referring to
http://www.ietf.org/IESG/content/ions/drafts/ion-tsv-alt-cc.txt)

This model raises one fundamental question issue of the scope of this 
document.

Who should be evaluating section 3 and 4 of this document?  Is this 
solely meant for ICCRG, the IETF or both?  If both, would both parties 
do everything described in those documents?

It would be clearer if the reviewer was ICCRG, and the IETF would not 
attempt to perform the same review, and the IETF wouldn't be allowed 
to second-guess the proposal, e.g., that it's research has not been 
done well enough if it was already positively evaluated by ICCRG.

> and a big-picture list of *areas* where we know we want information.

Maybe including a reference to this (even changing) list of evaluation 
criteria would help to mitigate my concern of enabling a 'rock fetch'. 
That way the authors and reviewers can look at it in advance to 
strengthen their case and afterwards also make an easier evaluation if 
the review was fair and according to the documented criteria.

>        networks with these characteristics should be detailed.

Yes, thanks.

What I want to avoid that when someone comes up with a proposal, folks 
that are for some reason opposed to it will try to seek specific (but 
globally not necessarily compelling) scenarios where the proposal 
doesn't behave very well, and argue that that's a major shortcoming. 
I think above should help on making the process more deterministic 
though with some experience it could maybe go even further.

>        [RFC2581,RFC2960,RFC4340].

OK.

> subjected to some other criteria then it need not be constrained by the

What I try to say is that I can't figure out how operationally and 
practically this could be achieved.  Do you have examples in mind how 
how this could apply in some specific scenario?

If not -- and it isn't a practical concern right now -- maybe we 
should not overengineer the BCP to address best-effort vs 
non-best-effort at all.

If yes, there are some approaches to this, I'd be interested in 
hearing how they'd work, but more even more than that, I'd require the 
proposers of such a mechanism to provide an evaluation how reliable 
that knowledge of best vs non-best-effort is.

What I'm afraid of is that the specification would say that it's 
applied for traffic with certain DSCP values but typically that alone 
is not IMHO a very reliable indicator of BE vs non-BE because DSCPs 
can be rewritten and used for other purposes than those envisaged by 
the proposer of the mechanism.

> causes no end to the haggling.)

(Well.. Some will argue that the current IESG's policy requires 
such a split.  Some have interpreted that in certain classes of 
Informational documents, a split may not be necessary, but I 
believe Standards Track and BCPs do need it.)

> above.  Sally may want to wordsmith or just plain disagree.

OK.  I'd have liked to change it more but I realize it's a bit late in 
the process and the other clarifications made above should help 
already.

---

From: Lars Eggert <[email-masked]>
Date: Mon, 28 May 2007 10:37:47 +0300

On 2007-5-26, at 14:24, ext Pekka Savola wrote:
> This model raises one fundamental question issue of the scope of  
> this document.
>
> solely meant for ICCRG, the IETF or both?  If both, would both  
> parties do everything described in those documents?

The basic idea behind this ION is that a researcher would bring his  
or her proposal, papers and experimental data to the ICCRG, which  
would then evaluate whether a proposal is safe for experimentation in  
the Internet or in more restricted network environments. (According  
to draft-ietf-tsvwg-cc-alt, and probably also draft-irtf-tmrg- 
metrics.) That review will accompany the resulting -00 technical  
specification when it is brought to the IETF.

> It would be clearer if the reviewer was ICCRG, and the IETF would  
> not attempt to perform the same review, and the IETF wouldn't be  
> allowed to second-guess the proposal, e.g., that it's research has  
> not been done well enough if it was already positively evaluated by  
> ICCRG.

There is a strong expectation that the IETF would usually agree with  
the review recommendation that the ICCRG made, and take on the  
document. (Sufficient people overlap, the ICCRG has stronger  
congestion control expertise, etc.) But process-wise it *is* up the  
to the WG (likely TCPM) to come to consensus on whether they want to  
adopt any given work item, and that consensus call can't be  
outsourced to the ICCRG. But I would personally see it as a sign that  
something went very wrong if TCPM would not publish something as  
Experimental that came with a strong ICCRG review.

Lars

PS: I'm editing the ION in response to IESG comments. The latest  
working version is under http://www3.tools.ietf.org/group/iesg/trac/ 
browser/ions/drafts - I'd be interested in hearing suggestions on how  
the text could be improved.
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Mark Allman <[email-masked]>
Date: Tue, 29 May 2007 09:04:46 -0400

> Yes (though '..that do should..' seemed weird on the first read).

I have fixed this (Sally caught it, too).

> Or do you mean that the proposers should do everything guidelines
> (2), (5), (6) and (7) say, but shortcomings in the results of these
> global Internet.

Yes.  And, in re-reading the text I think it is clear.

I really can't untangle your comments in this area.  If you have
specific suggestions for the text, please send them along.

About this 'rock fetch' stuff (in terms of metrics and environments): We
do not solve the problem you outline ('go fetch another rock').  But, we
did not aim to solve that problem and we cannot solve that problem in
this document.  We cannot enumerate all environments or all metrics that
might be of interest in the future.  I sympathize with your desire for a
simple checklist that one can go through and be "ready", but this is all
much too complicated for a simple checklist.  (Consider for a moment the
additional metrics that something like XCP brought to the table that
were not germane previously.)  It is not clear to me that we should
pretend we can list metrics.  Note that we have not made this 'rock
fetch' problem any worse, either.  I.e., it exists for any new CC scheme
that would be run through without this document the same as with this
document.

I am not planning to further change the document with respect to adding
metrics or more about environments unless lots of people start yelling.
This document has been widely read at this point and as far as I know
nobody else has had this concern.  (We can call the consensus 'rough' if
you prefer.)

> practically this could be achieved.  Do you have examples in mind
> how how this could apply in some specific scenario?
> 

We didn't overengineer anything.  We just said that the guideline
doesn't apply to non-BE cases.  I can't understand your point here at
all.  It is a simple statement of document scope.

allman

---

From: Pekka Savola <[email-masked]>
Date: Wed, 30 May 2007 14:27:41 +0300 (EEST)

> specific suggestions for the text, please send them along.

-03 has:

     For other guidelines, i.e., (2), (5), (6), and (7), evidence that
     the proposed mechanism has significantly more problems than those of
     TCP should be a cause for concern in approval for widespread
     deployment in the global Internet.

if the above is what you mean (and a proposer must really go through 
everything you list, e.g., all the difficult environments as well), it 
should be explicit, e.g.:

     For other guidelines, i.e., (2), (5), (6), and (7), the author
     must perform the suggested evaluations and provide recommended
     analysis.  Evidence that
     the proposed mechanism has significantly more problems than those of
     TCP should be a cause for concern in approval for widespread
     deployment in the global Internet.

My problem with existing text is that the referred guidelines use 
wording "should do ...".  Is it a "must do" or "may do" or something 
in between?  It should be more explicit.  Text proposal above 
interprets the shoulds as musts.

> all.  It is a simple statement of document scope.

Let's say I propose an informational RFC on congestion control and say 
that it only applies to non-BE traffic.  I don't have to make any of 
the evaluations or analysis required by this draft.  What's the 
procedure for non-BE congestion control agorithms?  I note that Lars's 
ION does not mention that case.

In case there is no process to define non-BE congestion control 
mechanisms at all (and the response would be "sorry, we don't support 
that"), I have no problem.

In case there is some process with a lower bar, I'd have a problem, 
because it would be possible to avoid the loops you have jump through 
defined in this document by saying the CC algorithm applies to non-BE 
traffic only, but in practice it would be end up deployed for BE 
traffic as well.

---

From: Mark Allman <[email-masked]>
Date: Wed, 30 May 2007 08:25:58 -0400

>      deployment in the global Internet.

Looks OK to me.  I have incorporated it, modulo comments from Sally.

As for the non-BE stuff: This document is a no-op.  But, why is that an
issue?  The IETF would have to grapple with the non-BE case just as it
does today (i.e., without a set of guidelines).  This one document does
not need to solve all the world's problems.  If you want to write a
document about how the IETF should handle non-BE congestion control
proposals, I think that'd be fine.  And, again, I am not hearing outcry
on this point so I think the document is fine (even if the consensus on
this one point is not completely 'smooth').

Thanks,
allman