Subject: Re: [rmcat] Comments on draft-johansson-rmcat-scream-cc-03

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 3 Nov 2014 10:56:02 +0000

Hi Mirja, see comments/answers inline

/Ingemar

> -----Original Message-----
> From: Mirja Kühlewind [mailto:[email-masked]]
> Sent: den 29 oktober 2014 19:31
> To: Ingemar Johansson S; Zaheduzzaman Sarker; [email-masked]
> Subject: Re: Comments on draft-johansson-rmcat-scream-cc-03
> 
> Hi Ingemar,
> 
> see inline..
> 
> On 29.10.2014 10:31, Ingemar Johansson S wrote:
> > Hi
> >
> > Many thanks for the comments/questions.
> > My, hopefully comprehensible, answers inline below, marked with [IJ].
> >
> > /Ingemar
> >
> >> -----Original Message-----
> >> From: Mirja Kühlewind [mailto:[email-masked]]
> >> Sent: den 28 oktober 2014 16:16
> >> To: Zaheduzzaman Sarker; Ingemar Johansson S; [email-masked]
> >> Subject: Comments on draft-johansson-rmcat-scream-cc-03
> >>
> >> Hi Ingemar, hi Zahed,
> >>
> >> I have a couple of questions/comments (as an individual contributor)
> >> on your draft. This is not a full review as I'm just addressing a
> >> couple of aspects which came to my mind immediately when reading (in
> random order).
> >>
> >> 1) RTCP feedback
> >> - You might want to think about which calculations should be done at
> >> the receiver and which at the sender. For LEBDAT it is proposed to
> >> calculate the the OWD at the receiver but you could probably even
> >> find a way to calculate the average (queuing) delay at the receiver
> >> and e.g. send the feedback only once an RTT. In fact if you implement
> >> a maximum increase rate, as in LEDBAT, of one packet per RTT, it's
> >> fine if you only update once per RTT as smaller updates will not open a
> packet window.
> > [IJ] So far I have focused on getting SCReAM robust. For various
> > reasons I ended up with all processing  in the sender and timely
> > feedback. But it is possible that one can do as you suggest above. The
> > ACK-clocking may need to be solved in some other way then as it
> > becomes necessary to predict how many new bytes the next ACK will
> acknowledge in order to keep a steady stream of packets.
> > But that can maybe be solved, also in cases where OWD increases
> > suddenly it is possible to send an early feedback.
> > Question is how much one can save with this in terms of RTCP overhead?.
> > Digged in the LTE  simulation logs. I ran the simulations with a video
> > source that had a max bitrate of 1.5Mbps.
> > I use reduce sized RTCP for the feedback and a full compound RTCP is
> > transmitted every 2 seconds, AVP regular mode with 70kbps RTCP BW is
> used.
> > I get an RTCP bitrate of maximum 14kbps, in practise one feedback is
> > transmitted roughly once per video frame. 14kbps is not too much to
> > worry about and it is definitely much less than 70kbps.
> > But it can perhaps become an issue if the bandwidth in the two
> > directions are highly assymetric or in certain conferencing scenarios.
> My point just was if it is possible to reduce overhead without (strong)
> performance degradation, we should do that. However, if there are good
> reasons to do it this way, you should (simply) state these reasons.
[IJ] OK, will see if I (or somebody else) has time too look closer at this, likely it is 
needed to modify how the transmission scheduler works to get it to work well.

> 
> >
> >>
> >> - If you don't need a reliable transport, you don't actually need a real ack-
> >> number. All you need is the number of newly received bytes no matter if
> in
> >> order or not. However, later on (at the end of section 4.1.1.1) you say loss
> >> could be detected by holes in the sequence number space. For actually
> >> detecting 'holes' you would even need something like SACK. I guess you
> >> instead assume dup ACK as in TCP...? Again I don't think it is needed to
> copy
> >> TCP completely..
> > [IJ] I would say that one get something like TCP SACK, instead of
> acknowledgement
> > of byte sequences. The loss detection is given by the ACK list.
> > I don't use TCP dupACK counting, instead I use a margin to cover possible
> reordering.
> > For instance, if sequence number SN is the highest acknowledged
> sequence number
> > the the algorithm looks for loss indications for SN-5 or earlier. The benefit
> with an ACK
> > list is that already there get support for retransmission and similar
> functionality as
> > RFC4585 Generic NACK.
> Ah now that I look at Figure 1 that's clear. The text is less clear...
[IJ] OK, this then needs to be explained better.

> 
> 
> >
> >>
> >> - RFC6679 specifies ECN feedback for RTP
> > [IJ] Yes, thing is that I am looking more at the accurate ECN for TCP in TCP as
> one
> > anyway has access to timely and frequent feedback in SCReAM. This may
> however
> > change if one can get less frequent fedback to work.
> >>
> >> 2) self-clocking
> >> In the intro you say scream is self-clocked but the transmission scheduling
> >> (section 4.1.2) says '[packets] are scheduled at intervals given by t_pace'.
> In
> >> this case you just need sent-out timers and there is no self-clocking
> >> anymore...?
> > [IJ] The pacing is set such that new RTP packets are clocked at intervals
> given by the
> > estimated bandwidth, given by the CWND and the RTT, and the size of the
> most
> > recently transmitted RTP packet. The pacing is there to break up coalescing
> tendencies
> > that I have seen in simulations. It does not do away with the self-clocking as
> the basic
> > rule that bytes in flight must still be satisfied.
> I'd interpret the terminology here differently. If you don't use the ack arrival
> as trigger to send out new packets but use timers instead, that's not
> self-clocked anymore. As you have a congestion window it's still the packet
> conservation principle but you don't use the ack clock.
[IJ] Yes, it can be a matter of interpretation, to me it is a matter of an additional constraint
to the ACK clock. 

> 
> >
> >>
> >> 3) Variables
> >> It's hard to follow all the variables you introduce. I would propose to have
> a
> >> section where you introduce all variables at once. Further, just based on
> the
> >> fact that there are quite a lot variables, would it be possible to reduce the
> >> number of variables (and thereby even simplify the algorithm without a
> large
> >> performance degradation)?
> >> Due to my thesis a recently re-read a lot of the congestion control
> evaluation
> >> and design literature and there is actually (e.g. in Jain's 'Congestion
> >> Avoidance in Computer Networks...') also some kind of requirement for
> >> simplicity because this allows better assessment of question on
> robustness.
> >> So for me personally my next step is to review my own algorithm (in my
> >> thesis) and try to make to make it simpler and evaluate how much
> >> performance this would cost...
> > [IJ] You are right, I will try to do this better next time, a section with all the
> > variables in one place is probably a good start.
> > Some parts of the algorithm can actually be stripped away, for instance the
> > OWD trend computation is there to reduce the jitter, also there are parts in
> > the video rate control parts that can be removed. The end result may be
> > more jitter but it may be easier to understand the algorithm description
> > May reintroduce the pseudo code later on as it may make things more
> > comprehensible.
> >
> >>
> >> 4) loss decrease by 0.8
> >> Why 0.8? Also, and i know this is often not correctly done in most existing
> >> algorithms (in TCP Cubic), you should decrease by at 0.5 after Slow Start,
> as
> >> you have doubled the window before. However as you don't slow start
> that
> >> often, that's probably not a real problem.
> > [IJ] This is set based on experimentation against TCP (NewReno) based file
> > transfers over the same CoDel or PIE AQM equipped bottleneck.
> > If I set the CWND scale in SCReAM to 0.5 i.e reduce by 50% istf 20% I see
> that the file
> > transfer outcompetes SCReAM. I believe this happens because SCReAM
> has a video
> > source with limited bitrate, while the file transfer has "unlimited" source
> bitrate.
> Okay, I'm okay to have a higher value that 0.5 but you should reason it and
> I'm
> not sure if saying you've tested it in a couple of scenarios is enough of
> reasoning. Is there an explanation why Scream has to be more aggressive (in
> the
> decrease behavior)...?
[IJ] The reason to why I need to settle with a smaller decrease in CWND (0.8 means only 20% decrease) is, as I see it the impact of the limited source bitrate. If one inspect the relation between bytes in flight and CWND one will see that the congestion window is most often underitilized, this does not (normally) happen when one use TCP with a large file transfer. Things would be slighly better if the video coder would produce frame sizes that were strictly porportional to the target bitrate. I can see if I can produce some material that explains the rationale better to the interrim.

> 
> 
> >
> >>
> >> 5) Conditions for owd_target adjustments
> >> - I not sure I really understand the conditions:
> >> "
> >>      o  owd_mem > owd_target
> >>
> >>      o  owd_target > owd_target_lo
> >> "
> >>
> >> The first condition, is actually the condition where LEDBAT decreases the
> >> window. At the same time you would want to increase the target. That
> >> doesn't sound very stable...
> >> The second condition I don't understand because you start with
> >> owd_target=owd_target_lo but then you never increase it if it is not
> larger
> >> than owd_target_lo... so you never increase...?
> > owd_mem is an average OWD. Normally this should not exceed
> owd_target
> > too much or too often but in case this happens it may be an indication of
> competing
> > flows and that the owd_target may need to be increased, depending on
> outcome of the
> > subsequent algorithm steps.
> > The second condition indicates that the owd_target is raised to make
> SCReAM cope with
> > competing flows. In that case it is necessary to check if it is possible to
> reduce owd_target
> > if for instance the competing flow has completed.
> Ah, so that's one or the other condition have to be true not both at the same
> time...? I would split this up because the first one is the condition to raise
> owd_target while the second might reduce owd_target, right?
[IJ] Yes, good suggestion. Will rearrange if this algorithm is still there in the 
next update. I am a bit curious whether SBD can be helpful here.

> 
> >
> >>
> >> - One minor comment on the same section (4.1.1.7): you say the average
> >> OWD should be zero (after restart) but you mean the queuing delay
> should
> >> be zero (because the OWD includes the end-to-end delay), right?
> > [IJ] OWD in this draft is the one way extra delay. i.e only the queuing delay,
> >   guess I need to clarify this.
> >>
> >> - In general I'm not too sure about the idea of interrupting video (also
> >> audio?)...
> > [IJ] No audio is not interrupted as it is not transmission scheduled by
> SCRaAM.
> > Only video is interrupted, one can argue how good it is but I have yet to
> find
> > another method that can make SCReAM work well against competing TCP
> over
> > tail drop links and that at the same time does not create self-inflicted
> congetsion
> > in wireless access such as LTE (and possibly even WiFi).
> > Possibly the shared bottleneck detection (SBD) can be useful here ?. There
> was
> > some questions around the behavior/performance in WiFi access, and I am
> too
> > among the curious.
> I know audio has a much lower rate but should also be congestion controlled.
> And
> interrupting the video can also be very annoying especially if you do that very
> often. Maybe you have to put a limit how often this probing is done.
[IJ] Yes. Especially if the audio codec is of Opus or EVS type and can support higher 
bitrates then it may need to be congestion controlled. The simplest thing would then be to route the audio RTP packets through the same transmission scheduler as the video RTP packets, possibly the audio RTP packets are given a higher priority. Another alternative is to let the audio RTP packets pass through untouched by the transmission scheduler and instead infer an appropriate audio bitrate from the video bitrate.   

> 
> >
> >>
> >> 6) transmission decision
> >> Could you explain the the second condition (in section 4.1.2.1) a little
> more.
> >> The idea is that you increase faster (than one packet per RTT) if the delay
> is
> >> low...? LEDBAT already provides variables for this kind of usage, by GAIN
> and
> >> ALLOWED_INCREASE. Would it be an option to use these variables
> instead?
> > [IJ] This section does not deal with the update of the CWND, instead it
> provides
> > with the ability to transmit RTP packets even though bytes in flight exceeds
> CWND
> > under certain conditions.
> > About CWND updates : Initially I tried to use the LEDBAT algorithm as is.
> > In the end I had to set ALLOWED_INCREASE to a very high value to avoid
> that
> > it limited CWND growth to much so I ended up removing the line
> > cwnd = min(cwnd, max_allowed_cwnd) completely. The problem is the
> source
> > (bitrate) limitation properties of the video coder,
> > because of this the congetsion window is not always used to its full extent.
> > The CWND is however limited by means of congestion
> > window validation but then on a longer timescale.
> Why don't you just increase cwnd instead on sending packets even though
> cwnd
> would not allow to do so. Where is the difference?
> ALLOWED_INCREASE was meant to limit the burstiness. As you use pacing
> this is
> not a problem for you anyway. However I would prefer to still have
> ALLOWED_INCREASE set to high value instead of removing it completely.
> Did you also try to use GAIN because this would be the right parameter, I
> believe.
[IJ] Yes, if it simplifies things then I can adapt this section to match RFC6817 better.
Gain is there (gain_up or gain_down). Both are set to 1.0. I belive forgot to list
the values
> 
> 
> >
> >>
> >> 7) One very general comment: Maybe it would be an option to separate
> the
> >> framework and the congestion control even more (in different
> documents)
> >> so that one could even use different congestion control schemes...? Or
> are
> >> there any reasons why explicitly LEDBAT is needed?
> > [IJ] I based the algorithm on LEDBAT, there could be other concepts like
> > delay gradient algorithms that can do equally well. It can be a good idea to
> split
> > up in several drafts.
> Also re-inserting the code in the appendix is probably still helpful...
[IJ] OK

> Mirja
> >> So far...
> >> Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 4 Nov 2014 13:43:27 +0100

Hi,

only one quick comment on the decrease factor:

Your explanation sounds like you could actually derive the ‚right‘ decrease factor from the fraction of time that the transmission was application limited… you could even try to do thus dynamically…?

Mirja

> Am 03.11.2014 um 11:56 schrieb Ingemar Johansson S <[email-masked]>:
>

---

From: Dave Taht <[email-masked]>
Date: Tue, 4 Nov 2014 08:30:49 -0800

> [IJ] Yes, thing is that I am looking more at the accurate ECN for TCP in TCP as one
> anyway has access to timely and frequent feedback in SCReAM. This may however

I have to admit that at lower rates (sub 10mbit) ecn is giving me headaches.

What I have been fiddling with lately is doing overload protection, dropping
one or more packets, then marking the delivered packet, at really low rates,
under overload. I have no idea what will happen on many tcps if a drop
is followed by a mark...

>> (section 4.1.2) says '[packets] are scheduled at intervals given by t_pace'. In
> [IJ] The pacing is set such that new RTP packets are clocked at intervals given by the
> estimated bandwidth, given by the CWND and the RTT, and the size of the most
> recently transmitted RTP packet. The pacing is there to break up coalescing tendencies
> that I have seen in simulations. It does not do away with the self-clocking as the basic
>> It's hard to follow all the variables you introduce. I would propose to have a
>> section where you introduce all variables at once. Further, just based on the
>> number of variables (and thereby even simplify the algorithm without a large

It might help to twist your brain into thinking about tcp as per laminar.

http://www.ietf.org/archive/id/draft-mathis-tcpm-tcp-laminar-01.txt

When matt first wrote that, I thought he was crazy. Now I am not so sure.

>> Due to my thesis a recently re-read a lot of the congestion control evaluation
>> simplicity because this allows better assessment of question on robustness.
>> algorithms (in TCP Cubic), you should decrease by at 0.5 after Slow Start, as
>> you have doubled the window before. However as you don't slow start that
> If I set the CWND scale in SCReAM to 0.5 i.e reduce by 50% istf 20% I see that the file
> transfer outcompetes SCReAM. I believe this happens because SCReAM has a video
> source with limited bitrate, while the file transfer has "unlimited" source bitrate.

Delighted to see you fiddling with pie and codel. These are the ns2 models?

FQ solves this fairly well. ¨DRR++¨ had some insight as well for more
bursty flows.

First hit off of this

https://www.google.com/search?client=ubuntu&channel=fs&q=DRR%2B%2B+thesis&ie=utf-8&oe=utf-8

>> owd_target=owd_target_lo but then you never increase it if it is not larger
> owd_mem is an average OWD. Normally this should not exceed owd_target
> too much or too often but in case this happens it may be an indication of competing
> flows and that the owd_target may need to be increased, depending on outcome of the
> The second condition indicates that the owd_target is raised to make SCReAM cope with
> competing flows. In that case it is necessary to check if it is possible to reduce owd_target
>> OWD should be zero (after restart) but you mean the queuing delay should
> [IJ] No audio is not interrupted as it is not transmission scheduled by SCRaAM.
> Only video is interrupted, one can argue how good it is but I have yet to find
> another method that can make SCReAM work well against competing TCP over
> tail drop links and that at the same time does not create self-inflicted congetsion
> Possibly the shared bottleneck detection (SBD) can be useful here ?. There was
> some questions around the behavior/performance in WiFi access, and I am too
> among the curious.
>
>>
>> 6) transmission decision
>> Could you explain the the second condition (in section 4.1.2.1) a little more.
>> The idea is that you increase faster (than one packet per RTT) if the delay is
>> low...? LEDBAT already provides variables for this kind of usage, by GAIN and
>> ALLOWED_INCREASE. Would it be an option to use these variables instead?
> [IJ] This section does not deal with the update of the CWND, instead it provides
> with the ability to transmit RTP packets even though bytes in flight exceeds CWND
> In the end I had to set ALLOWED_INCREASE to a very high value to avoid that
> cwnd = min(cwnd, max_allowed_cwnd) completely. The problem is the source
>> 7) One very general comment: Maybe it would be an option to separate the
>> framework and the congestion control even more (in different documents)
>> so that one could even use different congestion control schemes...? Or are
> delay gradient algorithms that can do equally well. It can be a good idea to split
> up in several drafts.
>>
>> So far...

I do personally find actual code and patches easier to understand.

-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 5 Nov 2014 11:55:40 +0000

Yes, that may actually work, good point.

/Ingemar

> Sent: den 4 november 2014 13:43
> To: Ingemar Johansson S
> Cc: Zaheduzzaman Sarker; [email-masked]
> Your explanation sounds like you could actually derive the ‚right‘ decrease
> factor from the fraction of time that the transmission was application
> limited… you could even try to do thus dynamically…?
> 
> Mirja
> 
> 
> 
> >>>> I have a couple of questions/comments (as an individual
> >>>> contributor) on your draft. This is not a full review as I'm just
> >>>> addressing a couple of aspects which came to my mind immediately
> >>>> when reading (in
> >> random order).
> >>>>
> >>>> 1) RTCP feedback
> >>>> - You might want to think about which calculations should be done
> >>>> at the receiver and which at the sender. For LEBDAT it is proposed
> >>>> to calculate the the OWD at the receiver but you could probably
> >>>> even find a way to calculate the average (queuing) delay at the
> >>>> receiver and e.g. send the feedback only once an RTT. In fact if
> >>>> you implement a maximum increase rate, as in LEDBAT, of one packet
> >>>> per RTT, it's fine if you only update once per RTT as smaller
> >>>> updates will not open a
> >> packet window.
> >>> feedback. But it is possible that one can do as you suggest above.
> >>> The ACK-clocking may need to be solved in some other way then as it
> >>> Digged in the LTE  simulation logs. I ran the simulations with a
> >>> video source that had a max bitrate of 1.5Mbps.
> >> My point just was if it is possible to reduce overhead without
> >> (strong) performance degradation, we should do that. However, if
> >> there are good reasons to do it this way, you should (simply) state these
> reasons.
> > [IJ] OK, will see if I (or somebody else) has time too look closer at
> > this, likely it is needed to modify how the transmission scheduler works to
> get it to work well.
> >
> >>
> >>>
> >>>>
> >>>> - If you don't need a reliable transport, you don't actually need a
> >>>> real ack- number. All you need is the number of newly received
> >>>> bytes no matter if
> >> in
> >>>> order or not. However, later on (at the end of section 4.1.1.1) you
> >>>> say loss could be detected by holes in the sequence number space.
> >>>> For actually detecting 'holes' you would even need something like
> >>>> SACK. I guess you instead assume dup ACK as in TCP...? Again I
> >>>> don't think it is needed to
> >> copy
> >>>> TCP completely..
> >>>> In the intro you say scream is self-clocked but the transmission
> >>>> scheduling (section 4.1.2) says '[packets] are scheduled at intervals
> given by t_pace'.
> >> In
> >>>> this case you just need sent-out timers and there is no
> >>>> self-clocking anymore...?
> >> I'd interpret the terminology here differently. If you don't use the
> >> ack arrival as trigger to send out new packets but use timers
> >> instead, that's not self-clocked anymore. As you have a congestion
> >> window it's still the packet conservation principle but you don't use the
> ack clock.
> > [IJ] Yes, it can be a matter of interpretation, to me it is a matter
> > of an additional constraint to the ACK clock.
> >
> >>
> >>>
> >>>>
> >>>> 3) Variables
> >>>> fact that there are quite a lot variables, would it be possible to
> >>>> reduce the number of variables (and thereby even simplify the
> >>>> algorithm without a
> >> large
> >>>> and design literature and there is actually (e.g. in Jain's
> >>>> 'Congestion Avoidance in Computer Networks...') also some kind of
> >>>> requirement for simplicity because this allows better assessment of
> >>>> question on
> >> robustness.
> >>> [IJ] You are right, I will try to do this better next time, a
> >>> section with all the variables in one place is probably a good start.
> >>> Some parts of the algorithm can actually be stripped away, for
> >>> instance the OWD trend computation is there to reduce the jitter,
> >>> also there are parts in the video rate control parts that can be
> >>> removed. The end result may be more jitter but it may be easier to
> >>> understand the algorithm description May reintroduce the pseudo code
> >>> later on as it may make things more comprehensible.
> >>>
> >>>>
> >>>> 4) loss decrease by 0.8
> >>>> Why 0.8? Also, and i know this is often not correctly done in most
> >>>> existing algorithms (in TCP Cubic), you should decrease by at 0.5
> >>>> after Slow Start,
> >> as
> >>> [IJ] This is set based on experimentation against TCP (NewReno)
> >>> based file transfers over the same CoDel or PIE AQM equipped
> bottleneck.
> >> Okay, I'm okay to have a higher value that 0.5 but you should reason
> >> it and I'm not sure if saying you've tested it in a couple of
> >> scenarios is enough of reasoning. Is there an explanation why Scream
> >> has to be more aggressive (in the decrease behavior)...?
> > [IJ] The reason to why I need to settle with a smaller decrease in CWND
> (0.8 means only 20% decrease) is, as I see it the impact of the limited source
> bitrate. If one inspect the relation between bytes in flight and CWND one will
> see that the congestion window is most often underitilized, this does not
> (normally) happen when one use TCP with a large file transfer. Things would
> be slighly better if the video coder would produce frame sizes that were
> strictly porportional to the target bitrate. I can see if I can produce some
> material that explains the rationale better to the interrim.
> >
> >>
> >>
> >>>
> >>>>
> > bitrates then it may need to be congestion controlled. The simplest thing
> would then be to route the audio RTP packets through the same
> transmission scheduler as the video RTP packets, possibly the audio RTP
> packets are given a higher priority. Another alternative is to let the audio RTP
> packets pass through untouched by the transmission scheduler and instead
> infer an appropriate audio bitrate from the video bitrate.
> >
> >>
> >>>
> >>>>
> >>>> 6) transmission decision

---

From: John Leslie <[email-masked]>
Date: Wed, 5 Nov 2014 10:53:54 -0500

Dave Taht <[email-masked]> wrote:
> 
> What I have been fiddling with lately is doing overload protection,
> dropping one or more packets, then marking the delivered packet, at
> really low rates, under overload.

   That sounds like valuable research. Clearly some packets will need
to be dropped under overload; but ECN-marking is a fundamentally better
signal when buffers are not yet full.

> I have no idea what will happen on many tcps if a drop is followed
> by a mark...

   There are so many cases to consider! :^(

   But I'd guess that the receiver will see the CE mark long before it
could infer the loss from timeout; and quickly notify the sender to
reduce the sending rate. I believe most TCP implementations treat
multiple losses and/or CE marks as a single event if they occur within
an RTT of each other.

   Accurate-ECN would, of course, notify the sender of multiple CE
marks during an RTT; but I wouldn't expect multiple halvings in sending
rate.

   I'm not sure exactly what Dave's implied question is here... But
I am sure there are folks here better prepared to answer it.

--
John Leslie <[email-masked]>

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 05 Nov 2014 17:12:49 +0100

Hi Dave, hi John,

why would you drop packets before marking? Isn't the idea of ECN to avoid drop 
by sending a congestion signal more early on?

Regarding the implementation at least for Linux, the network stack goes into 
either CWR or Recovery state after the first marking or, respectively, loss 
detection (3 dup ACKs) and stays there for about one RTT where it does not 
perform any further decreases (if not implemented differently using a recent 
update to the congestion control interface as DCTCP does).

Mirja

-- 
------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Communication Systems Group
Institute TIK, ETH Zürich
Gloriastrasse 35, 8092 Zürich, Switzerland

Room ETZ G93
phone: +41 44 63 26932
email: [email-masked]
------------------------------------------

---

From: John Leslie <[email-masked]>
Date: Wed, 5 Nov 2014 11:30:27 -0500

Mirja K?hlewind <[email-masked]> wrote:
> 
> Hi Dave, hi John,
> 
> why would you drop packets before marking?

   Dave will have to answer that. Perhaps he means that without AQM
the first action comes when the buffer fills. If so, I agree it's
entirely reasonable to drop ECN-enabled packet at that point.

   Dave seemed to imply that once tail-drop became necessary, he'd
add ECN-marking at a low rate in addition to tail-drop. I agree that
is reasonable as well.

   I don't think Dave was implying that this should become standard.

> Isn't the idea of ECN to avoid drop by sending a congestion signal
> more early on?

   Well...

   I'd like to get away from that "avoid drop" language. It suggests
that ECN would be a way to defer drops on ECN-capable flows (whether
or not the actual implementation reduces its rate). That would be bad.
Instead, I'd like to get folks thinking in terms of ECN as an AQM
signal sent _before_ any drops are needed.

   And, alas, the current ECN standard seems to discourage any early
action by requiring CE to result in the _same_ reduction as packet loss.
This inevitably would disadvantage ECN-capable flows if the sum of CE
and drop exceeded the drops of non-ECN flows. That is a paradigm we
have to break out of.

> Regarding the implementation at least for Linux, the network stack goes 
> into either CWR or Recovery state after the first marking or, respectively, 
> loss detection (3 dup ACKs) and stays there for about one RTT where it does 
> not perform any further decreases (if not implemented differently using a 
> recent update to the congestion control interface as DCTCP does).

   I am no Linux guru; but that is my understanding as well.

---

From: Dave Taht <[email-masked]>
Date: Wed, 5 Nov 2014 10:32:36 -0800

> entirely reasonable to drop ECN-enabled packet at that point.

I don’t necessarily want to fork this conversation on the scream draft.
In particular I wanted to know what was exactly meant by the pacing
language, also.

But as I am not attending ietf this time...

after shipping a stable release of cerowrt back in june, I have been
working with mine and  toke´s enormous data set to tear apart why
certain problems exist. For a long time now I have been very concerned
about IW10 and slow start in particular, at lower rates typically seen
in home uplinks.

Plug: Toke is giving a preso at iccrg (I believe) on his results,
the title of his talk is

¨How to reliably measure the performance of modern AQMs and what comes
of doing so?¨

The netperf-wrapper data is wonderful, as are the captures.

He did not look into ecn behavior as part of that, but that is kind
of next.... and I still keep hoping to have a viable real-world test
of webrtc and rmcat-like behaviors to fiddle with one day soon,
from somebody.

> add ECN-marking at a low rate in addition to tail-drop. I agree that

Well, I primarily work with codel, which is head drop, and fq_codel
which usually dedicates a codel queue to a given flow, which is
usually exclusively CE capable or not.

ECN was not part of the original codel research. I do happen to like
ECN and hope that someone is fiddling with it for primary frames in
rmcat and also for attempting more agressive slow start-like behavior.

The behaviors of an ECN marking queue vs a dropping queue with
that first linux codel implementation were *different*, but we got good results
under ¨nice¨ conditions for either kind of flow, if run exclusively.

After adding ECN
to the linux codel and fq_codel versions, it seemed safe to
declare that fq + aqm w/ecn was ok to deploy globally, (which is
why it is on on fq_codel by default)
but a single queue aqm with mixed ecn capable and incapable
traffic was not. (which is why it is off on pie and codel by default)

Later on we added probabilistic ECN  to the pie algorithm,
where it will revert to dropping on overload via a brick wall filter,
basically. That works but is far from satisfying in terms of
overall latency seen by all flows in the case of multiple flows
in slow start, other overload, or a misbehaving sender.

What I have been working on of late is deterministic - if codel´s
current drop schedule determines that without ecn it would drop
two or more packets in a row, it will drop all but the last packet,
mark that and deliver it
(and it is gentler still when entering drop mode, dropping
one then always ecn marking the next, to provide the earliest possible
signal)

This seems to have the desired properties of dropping more at
low rates, high loads, or misbehaving sender, while dropping less
and marking more when things are more in a steady state, while
marking first when a flow first starts ramping up.

Overall drop vs mark ratios at various rates are comparable to the
pie stuff, at the limited speeds and rtts I have tested, but there
are still problems. I think in particular we ran smack into a
recently fixed bug in linux´s reno implementation...

Note: I had originally bought into the idea of treating ECN as a
multi-bit signal as per DCTCP - but the newly landed DCTCP
implementation and tests also use a purely dedicated single queue
for it, and no other TCPs do, so doing meaner things to ECN on serious overload
seems necessary when outside the datacenter.

Anyway, I am always willing to share patches in progress, raw data
etc, if you want some stuff that applies against linux
net-next as of a few days ago, the relevant patch
set is in this directory.

http://snapon.lab.bufferbloat.net/~d/new_codel_models/everythingcompared.png

This also includes some new work on a better-than-htb rate limiter, with
some (totally failed) attempts at wedging dart-like diffserv ideas in
it, a better
version of codel and various other experimental ideas that work or do not
work to varying degrees. Note: the above data set is quite polluted as I
iterated over various options to the algorithm, dont take any of the data or
other graphs than the above at face value! And I have a couple more patches
under test....

A lot of people have noted that linux codel´s performance drops off at
higher rates,
and at high levels of traffic.

This is because the linux codel does not match the advanced ns2 model we had,
and I think this finally fixes that, now that Linux handles TSO/GSO sizing more
sanely. It is however hard to observe any difference
between fq_codel and nfq_codel as the fq part takes care of 98% of the problem
and high rates are not observed by the codel sub queues...

and far more work is needed before a comprehensive set of improvements
could be pushed upstream. ENOFUNDING here. If anyone is interested
in collaborating on this work let me know.

>    I don't think Dave was implying that this should become standard.

Overload protection is obviously needed for ECN. How to do that is
not standardized, this just happens to be one approach.

> signal sent _before_ any drops are needed.

Yes. In what I am trying to do, ECN marking happens first, then it
starts to revert to dropping if that fails to get things under control
soon enough.

Hysteresis is an ongoing problem here.

> and drop exceeded the drops of non-ECN flows. That is a paradigm we

At one level, I would like it if we kept tcps definition of ecn and morphed it
for newer protocols and applications like rmcat. Mosh treats ecn
interestingly, for example, I am using it in the babel routing
protocol interestingly
etc, etc.

>> recent update to the congestion control interface as DCTCP does).

Yea, that´s one stack.... but we dont have insight into what other stacks
actually do, as what we mostly have data on is ecn capability being available,

not being fully tested as actually working as expected, when used. Do any
test suites exist for measuring flows as to their expected CE ack window
marking behavior, and actual rate reduction, under a variety of conditions,
including reordering?

-- 
Dave Täht

http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks