Subject: [Iccrg] review of high-speed proposals?
From: [email-masked] (Lars Eggert)
Date: Thu Jan 27 08:35:18 2011

Hi,

On 2011-1-26, at 15:32, Eggert Lars (Nokia-NRC/Espoo) wrote:
> The IETF cannot start the process of publishing these IDs on high-speed congestion control as Experimental until they have actually been submitted to the TCPM WG for adoption.

I got some off-list feedback about how this sentence might be misunderstood. Let me attempt to clarify. The TCPM WG asked the ICCRG to review these three documents to determine whether they are safe enough for broader experimentation. This was a step in the adoption process of these documents in TCPM, which the authors requested (some time ago.)

Obviously, the ICCRG can publish Experimental RFCs directly through the "IRTF Stream" [RFC 5743]. How is this different from publishing Experimental RFCs through the "IETF Stream" [RFC 4844], which is what I think both the authors and me personally prefer for the three documents in question? In a nutshell: IETF-Stream RFCs have IETF consensus behind them, whereas IRTF-Stream RFCs do not. This may seem like a technicality, both for vendors who are trying to evaluate whether they should take a closer look at these algorithms, that difference may well matter. Plus, IETF-Stream RFCs typically undergo and IETF last call and IESG approval, and so are arguably somewhat more closely reviewed.

This difference is expressed through the boilerplate difference in the RFCs, which for IRTF-Stream documents reads:

   This document is a product of the Internet Research Task Force
   (IRTF).  The IRTF publishes the results of Internet-related research
   and development activities.  These results might not be suitable for
   deployment.  Documents approved for publication by the IRSG are not a
   candidate for any level of Internet Standard; see Section 2 of RFC
   5741.

whereas for IETF-Stream RFCs, it reads:

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Not all documents
   approved by the IESG are a candidate for any level of Internet
   Standard; see Section 2 of RFC 5741.

Lars
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
Url : http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110127/052c326d/smime.bin

From: [email-masked] (Eggert Lars (Nokia-NRC/Espoo))
Date: Wed Jan 26 13:29:14 2011
Subject: [Iccrg] review of high-speed proposals?
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]> <AANLkTi=SpFhoE9-DK1+hk=[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Hi,

to reiterate one more time:

The IETF cannot start the process of publishing these IDs on high-speed congestion control as Experimental until they have actually been submitted to the TCPM WG for adoption. Author action is required for this to happen.

Specifically, I'm talking about:

	* draft-rhee-tcpm-cubic, last updated Aug 2008
	* draft-leith-tcp-htcp, last updated Apr 2008
	* draft-sridharan-tcpm-ctcp, last updated Nov 2008

Lars
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
Url : http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110126/266d5cc4/smime.bin
>From [email-masked]  Wed Jan 26 23:25:20 2011
From: [email-masked] (=?iso-8859-1?Q?Michael_T=FCxen?=)
Date: Wed Jan 26 23:21:33 2011
Subject: [Iccrg] Re: [multipathtcp] Please review "Coupled Congestion
 Control for Multipath Transport Protocols"
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Hi Philip,

I have some questions regarding section 3:

(1) It is stated:
    "Since we require that the total throughput is no worse
     than the throughput a single TCP would get on the best path..."
    and later:
    "The formula is derived by equalizing the rate of the multipath flow
     with the rate of a TCP running on the best path, and solving for
     alpha."
    So what is considered here:
    * throughput in bytes/sec
    * rate in packets/sec
    * rate in bytes/sec

(2) I don't understand the formula for alpha completely. I understand how
    the corresponding formula in "Practical Congestion Control for Multipath
    Transport Protocol" is derived. However, there the congestion windows
    are measured in MTUs (or MSS or packets). So it is more packet rate
    based. I could understand the formula in the ID if cwnd_i / mss_i would
    be used instead of cwnd_i * mss_i. So is it a typo or could someone provide
    a hint why cwnd_i * mss_i is used.

Best regards
Michael

On Jan 19, 2011, at 3:25 PM, <[email-masked]> <[email-masked]> wrote:

> The Multipath TCP WG has developed a congestion control algorithm that enables a TCP connection to use multiple paths.  
> http://tools.ietf.org/html/draft-ietf-mptcp-congestion
>  
> We?re hoping to WG Last Call this document in February, so it would be great if anyone from ICCRG could review it
>  
> Thanks,
> Phil & Yoshifumi
> MPTCP WG Chairs
>  
> Abstract
>  
>    Often endpoints are connected by multiple paths, but communications
>    are usually restricted to a single path per connection.  Resource
>    usage within the network would be more efficient were it possible for
>    these multiple paths to be used concurrently.  Multipath TCP is a
>    proposal to achieve multipath transport in TCP.
>  
>    New congestion control algorithms are needed for multipath transport
>    protocols such as Multipath TCP, as single path algorithms have a
>    series of issues in the multipath context.  One of the prominent
>    problems is that running existing algorithms such as TCP New Reno
>    independently on each path would give the multipath flow more than
>    its fair share at a bottleneck link traversed by more than one of its
>    subflows.  Further, it is desirable that a source with multiple paths
>    available will transfer more traffic using the least congested of the
>    paths, hence achieving resource pooling.  This would increase the
>    overall utilization of the network and also its robustness to
>    failure.
>  
>    This document presents a congestion control algorithm which couples
>    the congestion control algorithms running on different subflows by
>    linking their increase functions, and dynamically controls the
>    overall aggresiveness of the multipath flow.  The result is a
>    practical algorithm that is fair to TCP at bottlenecks while moving
>    traffic away from congested links.
>  
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

From: [email-masked] (Michael Welzl)
Date: Mon Jan 24 10:02:13 2011
Subject: [Iccrg] Re: [multipathtcp] Please review "Coupled Congestion Control for Multipath Transport Protocols"
In-Reply-To: <AANLkTi=uct5-cCsjt=[email-masked]>
References: <[email-masked]> <[email-masked]> <AANLkTi=uct5-cCsjt=[email-masked]>
Message-ID: <[email-masked]>

>> - On page 6, when you first introduce alpha, you call it "a  
>> parameter ..
>> that describes the aggressiveness of the multipath flow"
>> (missing "s" in "aggressive" in that sentence BTW).
>> ... so this gives the impression that this is a per-aggregate  
>> parameter.
>> Yet, later, on page 7, you state that "alpha must be computed for  
>> each
>> multipath flow". Indeed I have a hard time imagining how alpha can  
>> ensure
>> that you send at least as much as TCP would send on the best path  
>> while just
>> having a single constant across all sub-flows. So I assume it's a
>> per-subflow-variable - but then it should get an index i.
>> I would strongly suggest giving a simple example where MPTCP is  
>> used across
>> two very different paths, just to show how alpha indeed ensures the
>> throughput of TCP on the best path because that seems to be hard to  
>> grasp
>> with intuition.
>
> Alpha is computed for each connection - the same alpha applies to all
> subflows, so as to control the overall aggressiveness.  It is not a
> per-subflow variable.
>
> Obviously our text is unclear on this - is there somewhere in
> particular that made you think it was per-subflow?  Or is it just the
> wording on page 7 ("alpha must be computed for each multipath flow")
> which could be rephased "for each multipath connection" if that helps.

It's really only this wording. and yes, this simple change does the  
trick - i guess it was just me, the word "flow" made me think of sub- 
flows.

About another sentence I wrote above: "Indeed I have a hard time  
imagining..."  - now I wonder what gave me this hard time? It actually  
seems clear enough.
Still, an example would be nice to have, but I take back the "strongly  
suggest". Sorry, I must have been tired.

Cheers,
Michael

From: [email-masked] (Mark Handley)
Date: Mon Jan 24 09:47:09 2011
Subject: [Iccrg] Re: [multipathtcp] Please review "Coupled Congestion Control for Multipath Transport Protocols"
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]>
Message-ID: <AANLkTi=uct5-cCsjt=[email-masked]>

On 24 January 2011 05:23, Michael Welzl <[email-masked]> wrote:
> Hi,
>
> Here's my review:
>
> This is, on the whole, a very well written document - short and concise, and
> clear and to the point. I noticed just a few issues:
>
>
> - On page 6, when you first introduce alpha, you call it "a parameter ..
> that describes the aggressiveness of the multipath flow"
> (missing "s" in "aggressive" in that sentence BTW).
> ... so this gives the impression that this is a per-aggregate parameter.
> Yet, later, on page 7, you state that "alpha must be computed for each
> multipath flow". Indeed I have a hard time imagining how alpha can ensure
> that you send at least as much as TCP would send on the best path while just
> having a single constant across all sub-flows. So I assume it's a
> per-subflow-variable - but then it should get an index i.
> I would strongly suggest giving a simple example where MPTCP is used across
> two very different paths, just to show how alpha indeed ensures the
> throughput of TCP on the best path because that seems to be hard to grasp
> with intuition.

Alpha is computed for each connection - the same alpha applies to all
subflows, so as to control the overall aggressiveness.  It is not a
per-subflow variable.

Obviously our text is unclear on this - is there somewhere in
particular that made you think it was per-subflow?  Or is it just the
wording on page 7 ("alpha must be computed for each multipath flow")
which could be rephased "for each multipath connection" if that helps.

 - Mark

From: [email-masked] (Michael Welzl)
Date: Mon Jan 24 05:20:30 2011
Subject: [Iccrg] Re: [multipathtcp] Please review "Coupled Congestion Control for Multipath Transport Protocols"
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Hi,

Here's my review:

This is, on the whole, a very well written document - short and  
concise, and clear and to the point. I noticed just a few issues:

- On page 6, when you first introduce alpha, you call it "a  
parameter .. that describes the aggressiveness of the multipath flow"
(missing "s" in "aggressive" in that sentence BTW).
... so this gives the impression that this is a per-aggregate  
parameter. Yet, later, on page 7, you state that "alpha must be  
computed for each multipath flow". Indeed I have a hard time imagining  
how alpha can ensure that you send at least as much as TCP would send  
on the best path while just having a single constant across all sub- 
flows. So I assume it's a per-subflow-variable - but then it should  
get an index i.
I would strongly suggest giving a simple example where MPTCP is used  
across two very different paths, just to show how alpha indeed ensures  
the throughput of TCP on the best path because that seems to be hard  
to grasp with intuition.

The formula presented on page 7 is hard to read. I have no problem  
with the formula that's similar in style on page 9 - but in this  
particular one, things just seem to get into the way of each other,  
and presenting it in a different style would be better. e.g.: what is  
max_i? Is this a function, traversing all i's, like sum_i below? But  
then it looks as if that's a function covering the whole fraction on  
the right, but the denominator (rtt_i^2) at  the same time seems to be  
connected the larger fraction... I don't get it. Or does max_i just  
refer to the numerator perhaps?

Also, upon reading this bit, I immediately began to wonder when and  
how often this is calculated. You answer this later, but the reader  
really expects an answer right here, where the formula appears - so  
you could, for instance, provide a forward reference, saying that this  
will be discussed in the next section.

Which brings me to the "when is it calculated" text on page 8. The  
paragraph starting with "It is possible to implement":
here you say that it's costly to update tot_cwnd on each ack, but  
later you say that "...implementation will update (missing 'the')  
tot_cwnd value whenever the individual subflows' windows are  
updated."  Please explain how this is not the same as updating on each  
ack? I would expect that these windows ARE updated on each ack... so  
what's the difference?

Next par.: "We propose alpha _to_ be a per-connection variable, ..."

section 5 par 1:
"...allocate _a_ zero window..."
"resulting rate _will_ fluctuate unpredictably...."
par 2:
"algorithm will allocate _a_ window to the subflows..."
"When the loss rates differ, _a_ progressively _larger_ window will be  
allocated..."

section 6: "_A_ detailed security analysis..."

That's it! I hope it's helpful.

Cheers,
Michael

On Jan 20, 2011, at 1:25 AM, <[email-masked]> <[email-masked] 
 > wrote:

> The Multipath TCP WG has developed a congestion control algorithm  
> that enables a TCP connection to use multiple paths.
> http://tools.ietf.org/html/draft-ietf-mptcp-congestion
>
> We?re hoping to WG Last Call this document in February, so it would  
> be great if anyone from ICCRG could review it
>
> Thanks,
> Phil & Yoshifumi
> MPTCP WG Chairs
>
> Abstract
>
>    Often endpoints are connected by multiple paths, but communications
>    are usually restricted to a single path per connection.  Resource
>    usage within the network would be more efficient were it possible  
> for
>    these multiple paths to be used concurrently.  Multipath TCP is a
>    proposal to achieve multipath transport in TCP.
>
>    New congestion control algorithms are needed for multipath  
> transport
>    protocols such as Multipath TCP, as single path algorithms have a
>    series of issues in the multipath context.  One of the prominent
>    problems is that running existing algorithms such as TCP New Reno
>    independently on each path would give the multipath flow more than
>    its fair share at a bottleneck link traversed by more than one of  
> its
>    subflows.  Further, it is desirable that a source with multiple  
> paths
>    available will transfer more traffic using the least congested of  
> the
>    paths, hence achieving resource pooling.  This would increase the
>    overall utilization of the network and also its robustness to
>    failure.
>
>    This document presents a congestion control algorithm which couples
>    the congestion control algorithms running on different subflows by
>    linking their increase functions, and dynamically controls the
>    overall aggresiveness of the multipath flow.  The result is a
>    practical algorithm that is fair to TCP at bottlenecks while moving
>    traffic away from congested links.
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

From: [email-masked] (Alexander Zimmermann)
Date: Tue Jan 18 07:51:12 2011
Subject: [Iccrg] Re: [tcpm] RTTM + timestamps
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Skipped content of type multipart/alternative-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 195 bytes
Desc: Signierter Teil der Nachricht
Url : http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110118/12ed959e/PGP.bin
>From [email-masked]  Wed Jan 19 14:25:54 2011
From: [email-masked] (philip.eardley)
Date: Wed Jan 19 14:23:07 2011
Subject: [Iccrg] Please review "Coupled Congestion Control for Multipath
 Transport Protocols"
Message-ID: <[email-masked]>

The Multipath TCP WG has developed a congestion control algorithm that enables a TCP connection to use multiple paths.
http://tools.ietf.org/html/draft-ietf-mptcp-congestion

We're hoping to WG Last Call this document in February, so it would be great if anyone from ICCRG could review it

Thanks,
Phil & Yoshifumi
MPTCP WG Chairs

Abstract

   Often endpoints are connected by multiple paths, but communications
   are usually restricted to a single path per connection.  Resource
   usage within the network would be more efficient were it possible for
   these multiple paths to be used concurrently.  Multipath TCP is a
   proposal to achieve multipath transport in TCP.

   New congestion control algorithms are needed for multipath transport
   protocols such as Multipath TCP, as single path algorithms have a
   series of issues in the multipath context.  One of the prominent
   problems is that running existing algorithms such as TCP New Reno
   independently on each path would give the multipath flow more than
   its fair share at a bottleneck link traversed by more than one of its
   subflows.  Further, it is desirable that a source with multiple paths
   available will transfer more traffic using the least congested of the
   paths, hence achieving resource pooling.  This would increase the
   overall utilization of the network and also its robustness to
   failure.

   This document presents a congestion control algorithm which couples
   the congestion control algorithms running on different subflows by
   linking their increase functions, and dynamically controls the
   overall aggresiveness of the multipath flow.  The result is a
   practical algorithm that is fair to TCP at bottlenecks while moving
   traffic away from congested links.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110119/f22a91c7/attachment.html
>From [email-masked]  Wed Jan 19 20:13:06 2011
From: [email-masked] (Michael Welzl)
Date: Wed Jan 19 20:13:48 2011
Subject: [Iccrg] Re: [multipathtcp] Please review "Coupled Congestion
 Control for Multipath Transport Protocols"
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

I'll do it!

Cheers,
Michael

On Jan 20, 2011, at 1:25 AM, <[email-masked]> <[email-masked] 
 > wrote:

> The Multipath TCP WG has developed a congestion control algorithm  
> that enables a TCP connection to use multiple paths.
> http://tools.ietf.org/html/draft-ietf-mptcp-congestion
>
> We?re hoping to WG Last Call this document in February, so it would  
> be great if anyone from ICCRG could review it
>
> Thanks,
> Phil & Yoshifumi
> MPTCP WG Chairs
>
> Abstract
>
>    Often endpoints are connected by multiple paths, but communications
>    are usually restricted to a single path per connection.  Resource
>    usage within the network would be more efficient were it possible  
> for
>    these multiple paths to be used concurrently.  Multipath TCP is a
>    proposal to achieve multipath transport in TCP.
>
>    New congestion control algorithms are needed for multipath  
> transport
>    protocols such as Multipath TCP, as single path algorithms have a
>    series of issues in the multipath context.  One of the prominent
>    problems is that running existing algorithms such as TCP New Reno
>    independently on each path would give the multipath flow more than
>    its fair share at a bottleneck link traversed by more than one of  
> its
>    subflows.  Further, it is desirable that a source with multiple  
> paths
>    available will transfer more traffic using the least congested of  
> the
>    paths, hence achieving resource pooling.  This would increase the
>    overall utilization of the network and also its robustness to
>    failure.
>
>    This document presents a congestion control algorithm which couples
>    the congestion control algorithms running on different subflows by
>    linking their increase functions, and dynamically controls the
>    overall aggresiveness of the multipath flow.  The result is a
>    practical algorithm that is fair to TCP at bottlenecks while moving
>    traffic away from congested links.
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

From: [email-masked] (SCHARF, Michael)
Date: Mon Jan 17 20:49:28 2011
Subject: [Iccrg] Re: [tcpm] RTTM + timestamps
References: <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Richard,

I happened to look at the RTO calculation and RTO spike issue a long time ago. For whatever it is worth, I scanned that old work and extracted some references that may or may not be well-known. A number of algorithms have indeed been developed to address these issues (e. g., Linux stack), and several papers tried to further optimize the timer calculation in particular for mobile environments, including amongst others:

R. Ludwig und K. Sklower. The Eifel Retransmission Timer. ACM SIGCOMM Computer Communications Review, 30(3), 2000, pp. 17?27 [this is not the actual Eifel algorithm]

H. Ekstroem and R. Ludwig, ?The Peak-Hopper: A New End-to-End Retransmission Timer for Reliable Unicast Transport,? Proc. IEEE INFOCOM, 2004

K. Jacobsson, H. Hjalmarsson, N. Moeller and K. H. Johansson, "Round-Trip time estimation in communication networks using adaptive Kalman filtering"

A. Kesselman, Y. Mansour, "Optimizing TCP Retransmission Timeout", Springer LNCS 3421, 2005

I. Psaras, V. Tsaoussidis, "Why TCP timers (still) don't work well", Computer Networks, Volume 51, Issue 8, 6 June 2007

etc.

And, BTW, a young wannabe-TCP researcher once tried to systematically understand the RTO spikes resulting from RFC 2988:

Michael Scharf, Marc C. Necker, Bernd Gloss: "The Sensitivity of TCP to Sudden Delay Variations in Mobile Networks", Springer LNCS 3042, 2004, pp. 76-87

If a better RTT estimation or RTO calculation was indeed needed, these papers might contain some interesting starting points.

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110117/b9ee60aa/attachment-0001.html
>From [email-masked]  Mon Jan 17 22:57:34 2011
From: [email-masked] (Scheffenegger, Richard)
Date: Mon Jan 17 22:55:23 2011
Subject: [Iccrg] RE: [tcpm] RTTM + timestamps
In-Reply-To: <[email-masked]>
References: <[email-masked]>
	<[email-masked]>
	<[email-masked]>
Message-ID: <[email-masked]>

Hi Michael,

again thank you for these references!

I wanted to bring this particular issue up, to discuss if an improved RTO calculation should be brought into a hypothetical I-D talking about improved signaling using existing options.

The one currently used by Linux has the merit of extremely wide deployment, and being conservative (only improves one particular aspect over the current standard calculation).

For non-technical issues, "Eifel" algorithms offer less good choice for any such future I-D work... (restrictions on IPR prevent the use in commercial implementations unfortunately).

However, I think most of these scenarios would also benefit from obtaining good feedback about the RTT samples (with the additional side-info, if the RTT was obtained off a reordered or retransmitted segment). But current signaling doesn't yield that feedback back from the client.

 Best regards,
   Richard

From: SCHARF, Michael [mailto:[email-masked]] 
Sent: Mon Jan 17 21:52:00 2011
To: Scheffenegger, Richard; [email-masked]; [email-masked]; [email-masked]
Cc: Matt Mathis; Mark Allman
Subject: Re: [tcpm] RTTM + timestamps

Richard,

I happened to look at the RTO calculation and RTO spike issue a long time ago. For whatever it is worth, I scanned that old work and extracted some references that may or may not be well-known. A number of algorithms have indeed been developed to address these issues (e. g., Linux stack), and several papers tried to further optimize the timer calculation in particular for mobile environments, including amongst others:

R. Ludwig und K. Sklower. The Eifel Retransmission Timer. ACM SIGCOMM Computer Communications Review, 30(3), 2000, pp. 17-27 [this is not the actual Eifel algorithm]

H. Ekstroem and R. Ludwig, "The Peak-Hopper: A New End-to-End Retransmission Timer for Reliable Unicast Transport," Proc. IEEE INFOCOM, 2004

K. Jacobsson, H. Hjalmarsson, N. Moeller and K. H. Johansson, "Round-Trip time estimation in communication networks using adaptive Kalman filtering"

A. Kesselman, Y. Mansour, "Optimizing TCP Retransmission Timeout", Springer LNCS 3421, 2005

I. Psaras, V. Tsaoussidis, "Why TCP timers (still) don't work well", Computer Networks, Volume 51, Issue 8, 6 June 2007

etc.

And, BTW, a young wannabe-TCP researcher once tried to systematically understand the RTO spikes resulting from RFC 2988:

Michael Scharf, Marc C. Necker, Bernd Gloss: "The Sensitivity of TCP to Sudden Delay Variations in Mobile Networks", Springer LNCS 3042, 2004, pp. 76-87

If a better RTT estimation or RTO calculation was indeed needed, these papers might contain some interesting starting points.

Michael

From: [email-masked] (Alexander Zimmermann)
Date: Mon Jan 17 14:32:43 2011
Subject: [Iccrg] Re: [tcpm] RTTM + timestamps
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Hi Richard,

some rash thoughts...

Am 17.01.2011 um 14:48 schrieb Scheffenegger, Richard:

> Hi group,
> 
> After having received some feedback off-list so far, I would like to summarize what I learned so far. Also, I invite everyone to discuss these points. There are benefits for the research community both empirical as well as theoretical, as well as mobile and high-speed network operators and implementers.
> 
> 
> o) one-way delay based (and delay variation based) congestion controls would benefit from knowing the clock resolution on both sides. Some research in that area is done by Mirja Kuehlewind and Bob Briscoe (http://bobbriscoe.net/pubs.html#chirp_impl), as well as David Hayes (http://caia.swin.edu.au/reports/100219A/CAIA-TR-100219A.pdf)
> 
> o) RTT variance during loss episodes is not deeply researched. Current heuristics (RFC1122, RFC1323, Karn's algorithm, RFC2988) explicitly exclude (and prevent) the use of RTT samples when loss occurs.

Eh... With RFC1323, you take RTT samples when loss occurs.

>  However, solving the retransmission ambiguity problem -

The problem is solved, isn't it? Eifel?

> and the related reliable ACK delivery problem - may allow the refinement of these algorithms further, as well as enabling new research to distinguish between corruption loss (without RTT / one-way delay impact) and congestion loss (with RTT / one-way delay impact).

This is not new. Ask Michael Welzl ;-)

And a citation form Sally Floyd to that topic:

"Inferring congestion vs. corruption at the transport level. There are also papers that investigate
algorithms for the transport end-nodes to infer that certain drops are from corruption rather than
congestion, without explicit feedback from routers. My own view would be that the burden is on such
approaches to show that they are not ignoring legitimate congestion indications from the network."

eg: S.Biaz and N. Vaidya. Discriminating Congestion Losses from Wireless Losses Using Interarrival
Times at the Receiver. IEEE Symposium on Application-Specific Systems and Software Engineering and
Technology, Mar 1999.

> This appears to be a rather neglected field however, especially when it comes to large scale, public internet investigations. Due to the very nature of this, passive investigations without signals contained within the headers are only of limited use in empirical research here.
> 
> o) A side-effect of Van Jacobson's algorithm is that RTO spikes when the path RTT suddenly drops. With the decrease of the path RTT, the variance grows. As the variance has a large effect on the calculated RTO, this leads to potentially very lengthy timeouts even though the RTT is much shorter. This particular problem has been addressed in some stacks, and the lessons learned from the deployment there could be used to update the RTO calculation specs.
> 
> o) Enabling of spurious RTO detection (Eifel, D-SACK, F-RTO) in the last years also make it possible to dynamically identify instances when the RTT estimation or RTO calculation where mislead, allowing to use a more conservative algorithm for certain paths / times.
> 
> o) Retransmission ambiguity detection during loss recovery

(what we already have)

> would allow an additional level of loss recovery control without reverting to timer-based methods.

I don't get this point.

> As with the deployment of SACK, separating "what" to send from "when" to send it could be driven one step further. In particular, less conservative loss recovery schemes which do not trade principles of packet conservation against timeliness, require a reliable way of prompt and best possible feedback from the receiver about any delivered segment and their ordering. SACK alone goes quite a long way, but using Timestamp information in addition could remove any ambiguity.

Which ambiguity do you mean here?

> However, the current specs in RFC1323 make that use impossible, thus a modified signaling (receiver behavior) is a necessity.
> 
> 
> 
> 
> 
> The first central aspect of the above mentioned points is to resolve the retransmission ambiguity

Again, Eifel?

> , and the second to give the end host a better understanding of their respective partners behavior.
> 
> I strongly believe that much about the retransmission ambiguity can be solved by exploiting synergistic signaling between the TCP SACK option and Timestamp option. In particular, the receiver-side state required by RFC1323 to choose which Timestamp to reflect when a non-contiguous segment is received could be alleviated, when the TCP session is also using SACK. (The presence of a SACK field indicates some out of sequence delivery. The current stipulations were made, afaik, to ensure that no unduly small RTT sample is entered into the RTO calculation, when ACK loss occurs. But again, SACK disambiguates between a in-sequence ACK, and a duplicate ACK).
> 

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: [email-masked]
// web: http://www.umic-mesh.net
//

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 195 bytes
Desc: Signierter Teil der Nachricht
Url : http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110117/b1825a84/PGP.bin
>From [email-masked]  Mon Jan 17 15:48:34 2011
From: [email-masked] (Scheffenegger, Richard)
Date: Mon Jan 17 15:46:14 2011
Subject: [Iccrg] RE: [tcpm] RTTM + timestamps
In-Reply-To: <[email-masked]>
References: <[email-masked]>
	<[email-masked]>
	<[email-masked]>
Message-ID: <[email-masked]>

HI Alex,

> > o) RTT variance during loss episodes is not deeply researched.
> > Current heuristics (RFC1122, RFC1323, Karn's algorithm, RFC2988)
> > explicitly exclude (and prevent) the use of RTT samples when loss
> > occurs.
> 
> Eh... With RFC1323, you take RTT samples when loss occurs.

True, but these samples are a worst-case upper bound, since the last
in-sequence delivered segment - if you are running "stateless" on the
sender. With a stateful sender, that keeps sending timestamps
independent of the TS option, and matches the (S)ACKed packets to this
(potentially very lengthy) list, you can - to some extent - circumvent
this issue.

However, it does not help you with re-retransmissions still.

Probably my wording was poor, what I intended to say was to get the best
possible RTT measurement with each delivered (and (S)ACKed) segment
possible, without incurring undue overhead (state) in the sender.

The deliberate withholding of TS info according to RFC1323 - in order to
address the problem of unreliably delivered ACKs, and - at the time -
non-availability of SACK, was what triggered this point.

> >  However, solving the retransmission ambiguity problem -
> 
> The problem is solved, isn't it? Eifel?

Only for cumulative/partial ACKs, and only then in theory, because IPR
restrictions prevent Eifel from being deployed in any commercial stack.
(There is only a single OS compatible with the IPR requirements for all
the Eifel algorithms).

For segments that are delivered out-of-sequence, it still exists.

> 
> > and the related reliable ACK delivery problem - may allow the
> refinement of these algorithms further, as well as enabling new
> research to distinguish between corruption loss (without RTT / one-way
> delay impact) and congestion loss (with RTT / one-way delay impact).
> 
> This is not new. Ask Michael Welzl ;-)
> 
> And a citation form Sally Floyd to that topic:
> 
> "Inferring congestion vs. corruption at the transport level. There are
> also papers that investigate algorithms for the transport end-nodes to
> infer that certain drops are from corruption rather than congestion,
> without explicit feedback from routers. My own view would be that the
> burden is on such approaches to show that they are not ignoring
> legitimate congestion indications from the network."
> 
> eg: S.Biaz and N. Vaidya. Discriminating Congestion Losses from
> Wireless Losses Using Interarrival Times at the Receiver. IEEE
> Symposium on Application-Specific Systems and Software Engineering and
> Technology, Mar 1999.

Thanks for the reference! Still you need some way of getting most
accurate RTT samples back to the sender, right?

> > o) Retransmission ambiguity detection during loss recovery
> 
> (what we already have)

Not really, you'll not get a lower TS back for a long delayed segment,
which was already retransmitted; also, if the retransmitted segment is
beyond some (still existing hole), Eifel + RFC1323 still fail to
disambiguate between original and retransmitted segment, when it's
delivered.

> 
> > would allow an additional level of loss recovery control without
> reverting to timer-based methods.
> 
> I don't get this point.

Ok, this revolves around lost retransmission detection at the
end-of-stream / below RecoveryPoint. First, LRD is again not widely
available (only one particular stack has that feature), nor officially
specified. Second, it depends on the ack of a segment beyond recovery
point to find that a retransmission was lost.

However, if the sender can infer the exact ordering in which all the
segements (original, retransmitted, reordered,...) were delivered to the
client, it can make a more informed choice if one particular segement
was lost (again), before going beyond RecoveryPoint (ie. Retransmit as
soon as the segment re-transmitted (!) dupthresh segments later than the
one in question, is received; For emphasis, I really mean the
re-transmitted segment, not the the delivery of a delayed original
segment (because that would be in indication, that there is a good
chance of the still-lost segement to be delivered via a slower path;
re-sending then would violate the packet conservation principle).

> 
> > As with the deployment of SACK, separating "what" to send from
"when"
> to send it could be driven one step further. In particular, less
> conservative loss recovery schemes which do not trade principles of
> packet conservation against timeliness, require a reliable way of
> prompt and best possible feedback from the receiver about any
delivered
> segment and their ordering. SACK alone goes quite a long way, but
using
> Timestamp information in addition could remove any ambiguity.
> 
> Which ambiguity do you mean here?

Hope I made this clear already: For out-of-sequence delivered,
retransmitted segments. Under certain circumstances, the sender may
re-transmit one segment more than twice, and it should be possible to
discriminate between each individual instance of that segment by using
SACK+TS information.

I can send around a sketch of such a less-conservative SACK loss
recovery scheme, using SACK+TS information, for detailed discussions
about it...?

Best regards,
   Richard

From: [email-masked] (Scheffenegger, Richard)
Date: Mon, 10 Jan 2011 18:23:49 +0000
Subject: [Iccrg] RTTM + timestamps
Message-ID: <[email-masked]>

Hi group,

in order not to spam the whole group, I would like to learn who is
interested in the heuristics and signaling around round-trip
measurement, timestamps and possible synergies between TS and other
option, as well as some empirical measurements around currently deployed
tcp stacks that diverge from the IETF specs in these aspects.

In the light of some recent developments (ie. LEDBAT, path-chirp/chirp
CC), and more efficient (end-of-flow / non-congestion) loss recovery,
improvements in that space may be of interest to a larger group of
implementers too...

With your feedback, I would like to learn the extent of possible
interest, before going further with my work in that area!

Best regards,

	Richard Scheffenegger

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oakham.cs.ucl.ac.uk/pipermail/iccrg/attachments/20110110/46630ba3/attachment.html
>From [email-masked]  Mon Jan 17 13:48:22 2011
From: [email-masked] (Scheffenegger, Richard)
Date: Mon Jan 17 13:46:06 2011
Subject: [Iccrg] RE: [tcpm] RTTM + timestamps
In-Reply-To: <[email-masked]>
References: <[email-masked]>
Message-ID: <[email-masked]>

Hi group,

After having received some feedback off-list so far, I would like to summarize what I learned so far. Also, I invite everyone to discuss these points. There are benefits for the research community both empirical as well as theoretical, as well as mobile and high-speed network operators and implementers.

 

o) one-way delay based (and delay variation based) congestion controls would benefit from knowing the clock resolution on both sides. Some research in that area is done by Mirja Kuehlewind and Bob Briscoe (http://bobbriscoe.net/pubs.html#chirp_impl), as well as David Hayes (http://caia.swin.edu.au/reports/100219A/CAIA-TR-100219A.pdf)

o) RTT variance during loss episodes is not deeply researched. Current heuristics (RFC1122, RFC1323, Karn's algorithm, RFC2988) explicitly exclude (and prevent) the use of RTT samples when loss occurs. However, solving the retransmission ambiguity problem - and the related reliable ACK delivery problem - may allow the refinement of these algorithms further, as well as enabling new research to distinguish between corruption loss (without RTT / one-way delay impact) and congestion loss (with RTT / one-way delay impact). This appears to be a rather neglected field however, especially when it comes to large scale, public internet investigations. Due to the very nature of this, passive investigations without signals contained within the headers are only of limited use in empirical research here.

o) A side-effect of Van Jacobson's algorithm is that RTO spikes when the path RTT suddenly drops. With the decrease of the path RTT, the variance grows. As the variance has a large effect on the calculated RTO, this leads to potentially very lengthy timeouts even though the RTT is much shorter. This particular problem has been addressed in some stacks, and the lessons learned from the deployment there could be used to update the RTO calculation specs. 

o) Enabling of spurious RTO detection (Eifel, D-SACK, F-RTO) in the last years also make it possible to dynamically identify instances when the RTT estimation or RTO calculation where mislead, allowing to use a more conservative algorithm for certain paths / times.

o) Retransmission ambiguity detection during loss recovery would allow an additional level of loss recovery control without reverting to timer-based methods. As with the deployment of SACK, separating "what" to send from "when" to send it could be driven one step further. In particular, less conservative loss recovery schemes which do not trade principles of packet conservation against timeliness, require a reliable way of prompt and best possible feedback from the receiver about any delivered segment and their ordering. SACK alone goes quite a long way, but using Timestamp information in addition could remove any ambiguity. However, the current specs in RFC1323 make that use impossible, thus a modified signaling (receiver behavior) is a necessity.

The first central aspect of the above mentioned points is to resolve the retransmission ambiguity, and the second to give the end host a better understanding of their respective partners behavior.

I strongly believe that much about the retransmission ambiguity can be solved by exploiting synergistic signaling between the TCP SACK option and Timestamp option. In particular, the receiver-side state required by RFC1323 to choose which Timestamp to reflect when a non-contiguous segment is received could be alleviated, when the TCP session is also using SACK. (The presence of a SACK field indicates some out of sequence delivery. The current stipulations were made, afaik, to ensure that no unduly small RTT sample is entered into the RTO calculation, when ACK loss occurs. But again, SACK disambiguates between a in-sequence ACK, and a duplicate ACK).

For the second aspect, the single most important aspect to convey would be the tcp clock resolution. For this, no signaling exists yet. However, there appears to be a downward-compatible method to extend the Timestamp SYN option, to include this signal.

According to RFC1323, TSecr is unused in the SYN segment, and most stacks appear to ignore that value. An enhanced signaling could set this initial TSecr field. To convey some feedback from the receiver to the sender in the SYN|ACK, while maintaining compatibility with stacks that do not evaluate the contents of this SYN TSecr, a simple XOR between the SYN|TSopt field and the clients respective settings could be employed.

The sender would compare the TSecr value in the SYN|ACK with it's initial sent TSval, and if they are identical, conclude to deal with an unenhanced client (no SACK+TS synergy, no one-way delay/variance CC). If the TSecr in the SYN|ACK would be different, the negotiated heuristics can be enabled for that TCP session - decoded by an XOR from the original TSval. 

SYN:
+-------+-------+---------------------+---------------------+
|Kind=8 |  10   |   TS Value (TSval)  |TS Echo Reply (TSecr)|
+-------+-------+---------------------+---------------------+
    1       1              4                     4
With TSecr containing, ie. A 16-bit Flag field plus a 16-bit IEEE float indicating the TCP clock resolution (duration or frequency), or a 8-bit Flag field plus two 12-bit floats for negotiation of an acceptable clock range. Also, perhaps a CIDR-like prefix bitstring may be of interest for future extensions, ie:

1 1 x x x - future
1 0 1 x x - use 2x12 bit floats
1 0 0 1 x - use 1x16 bit float

A few potentially meaningful flag bits discussed so far could be

TS-extension flag (always set; this limits the senders initial TS selection...)
  Reserved
    TS-negotiate range (w/ 2x 12bit floats)
    TS-fixed rate (w/ 16bit float)
      Modified exponent offset (-21 instead of -15, for very high speed networks)
        SACK+TS synergy (always reflect TS of last received segment)
          High-Precision (use HW late in data path for TS insertion, exclude much jitter)

Note that binary16 allows a dynamic range (with some lost precision) from 2^15 down to 2^-24, while a 12-bit representation, omitting sign and most significant exponent bit, plus least two significant fraction bits, would still allow a range between 2^0 to 2^-22. 

Binary16:
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|Si|   Exponent   |          Fraction           |
|gn|offset shifted|                             |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

12-bit subsample:
      +--+--+--+--+--+--+--+--+--+--+--+--+
      |  exponent |        fraction       |
      +--+--+--+--+--+--+--+--+--+--+--+--+

However even this dynamic range may not be enough to allow a tcp clock to run at the sending rate of minimal IP frames at very high (100 Gbit/s, 1Tbit/s) speeds. Using a different exponent offset shift, and omitting one additional fraction bit instead of one exponent bit in the 12 bit value should be discussed.
   +--+--+--+--+--+--+--+--+--+--+--+--+
   |   exponent   |        fraction    |
   +--+--+--+--+--+--+--+--+--+--+--+--+

From: Scheffenegger, Richard 
Date: Mon, 10 Jan 2011 19:25:00 +0000
To: [email-masked]; [email-masked]
Subject: [tcpm] RTTM + timestamps

Hi group,
in order not to spam the whole group, I would like to learn who is interested in the heuristics and signaling around round-trip measurement, timestamps and possible synergies between TS and other option, as well as some empirical measurements around currently deployed tcp stacks that diverge from the IETF specs in these aspects.
In the light of some recent developments (ie. LEDBAT, path-chirp/chirp CC), and more efficient (end-of-flow / non-congestion) loss recovery, improvements in that space may be of interest to a larger group of implementers too...
With your feedback, I would like to learn the extent of possible interest, before going further with my work in that area!
Best regards,
Richard Scheffenegger

From: [email-masked] (michawe)
Date: Thu Jan  6 12:28:13 2011
Subject: [Iccrg] Re: Request to publish  draft-irtf-iccrg-welzl-congestion-control-open-research-08
In-Reply-To: <[email-masked]>
References: <[email-masked]> <[email-masked]> <[email-masked]>
Message-ID: <[email-masked]>

Hi,

The typical notice is fine with all of us authors, the "no derivatives" clause is not needed.

Thanks,

best,
Michael Welzl

On Jan 3, 2011, at 11:46 PM, RFC Editor wrote:

> Greetings All,
> 
> We are preparing to move
> draft-irtf-iccrg-welzl-congestion-control-open-research-08 to AUTH48.
> However, please clarify whether the "no derivatives" clause is
> required for this document.  We note that the Internet-Draft included
> the 6.c.iii paragraph from the Trust License Provisions, which (to our
> understanding) is not applicable to alternate stream publications.
> Please let us know which copyright statement should be applied to this
> document:  
> 
> *Copyright Notice (typical notice for alternate streams)
> 
> Copyright (c) 2010 IETF Trust and the persons identified as
> the document authors.  All rights reserved.
> 
> This document is subject to BCP 78 and the IETF Trust's Legal
> Provisions Relating to IETF Documents
> (http://trustee.ietf.org/license-info) in effect on the date of
> publication of this document.  Please review these documents
> carefully, as they describe your rights and restrictions with respect
> to this document. 
> 
> 
> *Copyright Notice (no derivative works)
> 
> Copyright (c) 2010 IETF Trust and the persons identified as
> the document authors.  All rights reserved.
> 
> This document is subject to BCP 78 and the IETF Trust's Legal
> Provisions Relating to IETF Documents
> (http://trustee.ietf.org/license-info) in effect on the date of
> publication of this document.  Please review these documents
> carefully, as they describe your rights and restrictions with respect
> to this document. 
> 
> This document may not be modified, and derivative works of it may not
> be created, except to format it for publication as an RFC or to
> translate it into languages other than English.
> 
> Please see http://trustee.ietf.org/license-info/ for more
> information.  
> 
> We will wait to hear from you before moving forward.
> 
> Thanks,
> RFC Editor/sg
> 
> 
> 
> 
> On Fri, Oct 01, 2010 at 03:29:44PM +0200, Aaron Falk wrote:
>> RFC Editor-
>> 
>> Please publish draft-irtf-iccrg-welzl-congestion-control-open-research-08 as an Informational IRTF RFC. The document has been approved for publication by the IRSG and also reviewed for IETF conflict by the IESG. See http://trac.tools.ietf.org/group/irtf/trac/ticket/29 for details on the IESG and prior reviews. Please copy all correspondence to the document shepherd, Wesley Eddy,  [email-masked].
>> 
>> The authors would like to have a few edits made as described by the message below.
>> 
>> --aaron
>> 
>> ---
>> Aaron Falk
>> Chair
>> Internet Research Task Force
>> http://www.irtf.org
>> 
>> 
>> 
>> 
>> On 9/29/10 5:43 PM, Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP] wrote:
>>> Hi Aaron, the ICCRG document on open research issues in
>>> Internet congestion control has been through the IESG's
>>> review and is now ready for submission to the RFC Editor.
>>> 
>>> The authors and IESG agreed on the following request to
>>> the RFC Editor for modification of the draft prior to
>>> publication as an RFC (which they would like the RFC
>>> Editor to handle, rather than submit another revision):
>>> 
>>> RFC Editor Note
>>> 
>>> (1) Replace beginning of Section 3.5.3 with:
>>> 
>>>  3.5.3 Inelastic Multi-domain Pseudowires
>>> 
>>>  Extending pseudo-wires across multiple domains poses specific issues.
>>>  Pseudowires (PW) [RFC3985] may carry non-TCP data flows (e.g. TDM
>>>  traffic or Constant Bit Rate (CBR) ATM traffic) over a multi-domain 
>>>  IP network. Structure Agnostic TDM over Packet (SATOP) [RFC4553], 
>>>  Circuit Emulation over Packet Switched Networks (CESoPSN), TDM over 
>>>  IP, are not responsive to congestion control as discussed by
>>>  [RFC2914] (see also [RFC5033]). The same observation applies to ATM 
>>>  circuit emulating services (CES) interconnecting CBR equipment (e.g. 
>>>  PBX) across a Packet Switched Network (PSN).
>>> 
>>>  Moreover, it is not possible to simply reduce the flow rate of a TDM
>>>  PW or an ATM PW when facing packet loss. Providers can rate control 
>>>  corresponding incoming traffic but they may not be able to detect 
>>>  that PWs carry TDM or CBR ATM traffic (mechanisms for characterizing 
>>>  the traffic temporal properties may not necessarily be supported). 
>>> 
>>>  This can be illustrated with the following example. 
>>> 
>>> 
>>> (2) Add at the end of Section 3.8.4 Congestion Control in Multi-layered Networks
>>> 
>>>  Section 3.5.3 deals with Inelastic Multi-domain Pseudowires (PW), 
>>>  where the characteristics of the Pseudowire itself determines the 
>>>  characteristics of the traffic crossing the multi-domain PSN 
>>>  (and this independently of the characteristics of the traffic 
>>>  carried in the PW). A more complex situation arises when inelastic 
>>>  traffic is carried as part of a Pseudowire (e.g. inelastic traffic 
>>>  over Ethernet PW over PSN) whose edges do not have the means to 
>>>  characterize the properties of the traffic encapsulated into the
>>>  Ethernet frames. In this case, the problem explained in 
>>>  Section 3.5.3 is not limited to multi-domain Pseudowires but more 
>>>  generally induced by "Pseudowire carrying inelastic traffic" (over 
>>>  a single- or multi-domain PSN). The problem becomes even more
>>>  intricate when the Ethernet PW carries both inelastic and 
>>>  elastic traffic. Addressing this issue further comforts our 
>>>  observation that a general framework to efficiently deal with 
>>>  congestion control problems in multi-layer networks is absolutely
>>>  necessary but without harming its evolvability.
>>> 
>>> --
>>> Wes Eddy
>>> MTI Systems
>>> 
>>> 
>>>