Subject: [rmcat] Comments on draft-holmer-rmcat-transport-wide-cc-extensions-00

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 16 Mar 2015 07:48:26 +0000

Hi

And thanks for your submission ( http://tools.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-00.txt )
Your statements in section 1 makes much sense to me and I have too advocated to have the intelligence only on one side, mainly to make interop and software version synchronization easier.
In addition it is likely a good idea to try to make the feedback generic in some sense as it can allow for algorithms to evolve and improve over time. It is not for certain that all feedback elements and the RTP header extension is needed for all algorithms so some kind of negotiation can be useful here if one wish to minimize signaling overhead. The feedback rate is probably something that deserves some further consideration as different algorithms may have different requirements.

Related to the SCReAM algorithm :

The ACK vector was removed quite recently from the SCReAM feedback. I elaborated a bit on the reasoning behind this decision (after an email discussion with Mirja and Magnus) in http://www.ietf.org/mail-archive/web/rmcat/current/msg01100.html

The loss counter of course means that the receiver must implement some limited intelligence in the form of a loss detection algorithm on the receiver side. This is however limited to some 15 lines of code (see https://github.com/EricssonResearch/scream/blob/master/code/ScreamRx.cpp ) I don't however have any strong opinion against an ACK vector so if that is deemed a good to have for one reason or another..

One difference between your draft and SCReAM is that SCReAM ACKs the highest received sequence number similar to the highest TCP SACKed byte, this makes it possible for SCReAM to move forward the send window even though there are packets lost. Your draft´s "base transport sequence number" is similar to TCP SND.UNA and the ACK vector indicates received packets beyond that point (similar to SACK). My understanding is that "base transport sequence number" need to stay put until it is recovered ?, this should then mean that the send window cannot move more than 16 packets beyond the "base transport sequence number" and this can have negative consequences if RTT is high. Could be that I misunderstand things here though.

As regards to the RTP header extension. Currently the conclusion with SCReAM at least is that it is sufficient to keep track of the SSRC and RTP sequence number on the sender side in order to get a good bytes in flight metric.

/Ingemar

=================================
Ingemar Johansson  M.Sc.
Senior Researcher

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com

"The Earth is a very small stage in a
 vast cosmic arena"
Carl Sagan "Pale Blue Dot"
=================================

---

From: Stefan Holmer <[email-masked]>
Date: Tue, 17 Mar 2015 09:15:45 +0000

Hi everyone,

Thanks for the feedback. I was planning on bringing this up for discussion
on the list, but I was out the last week so it was a bit delayed.

This is an early draft of what we think would be a flexible protocol for
rmcat congestion control. When improving on gcc we have become aware of the
difficulty of making changes to the algorithm when the logic is split
between sender and receiver, requiring a synchronized roll-out. Also, in
terms of interop between different types of algorithms, a more flexible
feedback format would be helpful.

More replies inline.

Thanks,
/Stefan

On Mon, Mar 16, 2015 at 8:48 AM Ingemar Johansson S <
[email-masked]> wrote:

>  Hi
>
>
>
> And thanks for your submission (
> http://tools.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-00.txt
> )
>
> Your statements in section 1 makes much sense to me and I have too
> advocated to have the intelligence only on one side, mainly to make interop
> and software version synchronization easier.
>
> In addition it is likely a good idea to try to make the feedback generic
> in some sense as it can allow for algorithms to evolve and improve over
> time. It is not for certain that all feedback elements and the RTP header
> extension is needed for all algorithms so some kind of negotiation can be
> useful here if one wish to minimize signaling overhead. The feedback rate
> is probably something that deserves some further consideration as different
> algorithms may have different requirements.
>
Yes, I agree. It could be a good idea to negotiate the feedback rate as
part of negotiating the RTCP message.

> The ACK vector was removed quite recently from the SCReAM feedback. I
> elaborated a bit on the reasoning behind this decision (after an email
> discussion with Mirja and Magnus) in
> http://www.ietf.org/mail-archive/web/rmcat/current/msg01100.html
>
> The loss counter of course means that the receiver must implement some
> limited intelligence in the form of a loss detection algorithm on the
> receiver side. This is however limited to some 15 lines of code (see
> https://github.com/EricssonResearch/scream/blob/master/code/ScreamRx.cpp
> ) I don't however have any strong opinion against an ACK vector so if that
> is deemed a good to have for one reason or another..
>

There are a couple of reasons why we think it would be good to have an ack
vector:

- Per packet arrival time is needed for the way gcc is currently designed.
Reducing the number of delay samples can be possible, but I expect that it
would have some negative effect on performance.

- It can be beneficial to feed back the arrival time of every packet
received, not only the latest received packet. This can be used for
bandwidth probing, where you send packets at some predetermined bitrate (or
increasingly higher bitrate), higher than the bitrate of your current
bandwidth estimate / congestion window, to more quickly get an estimate of
the link characteristics. We have implemented a first version of this,
although we haven't updated gcc with it at this point.

- It gives you a better understanding of loss patterns, and how losses
correlate with increasing delay at shorter time scales. We haven't
experimented with this, so I can't say how useful it is.

- As I think was pointed out in the scream draft, it could be used for
stream robustness as well as congestion control. This is not something we
talk much about in the draft at the moment, but it's something we hope to
explore.

>
>
> One difference between your draft and SCReAM is that SCReAM ACKs the
> highest received sequence number similar to the highest TCP SACKed byte,
> this makes it possible for SCReAM to move forward the send window even
> though there are packets lost. Your draft´s “base transport sequence
> number” is similar to TCP SND.UNA and the ACK vector indicates received
> packets beyond that point (similar to SACK). My understanding is that “base
> transport sequence number” need to stay put until it is recovered ?, this
> should then mean that the send window cannot move more than 16 packets
> beyond the “base transport sequence number” and this can have negative
> consequences if RTT is high. Could be that I misunderstand things here
> though.
>
Our current thinking is that packets are only acked once. If a feedback
message is lost, the packets acked in that message will be assumed to have
arrived as expected. The reason for this is to emulate the behavior of a
lost rtcp report block, where if you don't receive the report block
containing a positive fraction lost, you likely assume it was zero.
Instead, action may be taken if a number of feedback messages are lost. We
would be happy to hear what you think about this.

For stream robustness (which we don't currently discuss in detail in this
draft), our thinking is that the receiver may feed back a "currently
decodable timestamp" in addition to the ack vector. That way, a sender can
stop attempting to recover packets as soon as the receiver states that it
can move past the point of the packets which haven't been acked.

>
>
> As regards to the RTP header extension. Currently the conclusion with
> SCReAM at least is that it is sufficient to keep track of the SSRC and RTP
> sequence number on the sender side in order to get a good bytes in flight
> metric.
>
Yes, it would be possible to use ssrc and rtp sequence numbers instead. One
benefit of having a transport-wide sequence number is that it's possible to
earlier detect lost packets. For instance, if you are receiving two
streams, one audio (10 ms/packet) and one video (30 fps), and the last
packet of a video frame is lost, you would be able to detect that already
when the next audio packet is received. This becomes increasingly useful
the more streams you are receiving.

In my opinion, congestion control is also something that should act on a
layer above rtp streams (i.e., per connection or transport). I don't think
rtp provides such a layer as is, so the suggested rtp extension aims to add
a "virtual" layer which can be used for this purpose. It enables us to
slightly reduce the feedback overhead since we don't have to specify ssrcs
and base sequence numbers for every stream separately.

> [email-masked]
>
> www.ericsson.com
>
>
>
> “The Earth is a very small stage in a
>
>  vast cosmic arena”
> Carl Sagan “Pale Blue Dot”

---

From: Michael Welzl <[email-masked]>
Date: Tue, 17 Mar 2015 14:20:55 +0000

Great to see this!  - and I also said that we should have such a framework before:  http://www.ietf.org/proceedings/85/slides/slides-85-rmcat-0.pdf
I felt that my statements were ignored back then, and was disappointed to watch the development of this rather messy situation that we're now facing...

This being said, I wonder: why not propose a generic simple sender-side behavior that ensures that the sender times out and backs off when it doesn't get feedback for long, and otherwise just sends at the rate that is dictated by the receiver + move all the logic to the receiver?  What made you suggest putting it in the sender?

AFAIK the rationale for doing stuff in the receiver was always the reduction of feedback, which I also never personally understood to be a major reason. So is this now a sign of people finally accepting that one will have to have a decent amount of feedback anyway, and hence there's no point in having a receiver-side CC. logic?  :-)

Cheers,
Michael

> On 17 Mar 2015, at 10:15, Stefan Holmer <[email-masked]> wrote:
> 
> Hi everyone,
> 
> Thanks for the feedback. I was planning on bringing this up for discussion on the list, but I was out the last week so it was a bit delayed.
> 
> This is an early draft of what we think would be a flexible protocol for rmcat congestion control. When improving on gcc we have become aware of the difficulty of making changes to the algorithm when the logic is split between sender and receiver, requiring a synchronized roll-out. Also, in terms of interop between different types of algorithms, a more flexible feedback format would be helpful.
> 
> More replies inline.
> 
> Thanks,
> /Stefan
> 
> On Mon, Mar 16, 2015 at 8:48 AM Ingemar Johansson S <[email-masked]> wrote:
> Hi
> 
>  
> 
> Yes, I agree. It could be a good idea to negotiate the feedback rate as part of negotiating the RTCP message. 
>  
> 
>  
> 
> There are a couple of reasons why we think it would be good to have an ack vector:
> 
> - Per packet arrival time is needed for the way gcc is currently designed. Reducing the number of delay samples can be possible, but I expect that it would have some negative effect on performance.
> 
> - It can be beneficial to feed back the arrival time of every packet received, not only the latest received packet. This can be used for bandwidth probing, where you send packets at some predetermined bitrate (or increasingly higher bitrate), higher than the bitrate of your current bandwidth estimate / congestion window, to more quickly get an estimate of the link characteristics. We have implemented a first version of this, although we haven't updated gcc with it at this point.
> 
> - It gives you a better understanding of loss patterns, and how losses correlate with increasing delay at shorter time scales. We haven't experimented with this, so I can't say how useful it is.
> 
> - As I think was pointed out in the scream draft, it could be used for stream robustness as well as congestion control. This is not something we talk much about in the draft at the moment, but it's something we hope to explore.
>  
> 
>  
> 
> One difference between your draft and SCReAM is that SCReAM ACKs the highest received sequence number similar to the highest TCP SACKed byte, this makes it possible for SCReAM to move forward the send window even though there are packets lost. Your draft´s “base transport sequence number” is similar to TCP SND.UNA and the ACK vector indicates received packets beyond that point (similar to SACK). My understanding is that “base transport sequence number” need to stay put until it is recovered ?, this should then mean that the send window cannot move more than 16 packets beyond the “base transport sequence number” and this can have negative consequences if RTT is high. Could be that I misunderstand things here though.
> 
> Our current thinking is that packets are only acked once. If a feedback message is lost, the packets acked in that message will be assumed to have arrived as expected. The reason for this is to emulate the behavior of a lost rtcp report block, where if you don't receive the report block containing a positive fraction lost, you likely assume it was zero. Instead, action may be taken if a number of feedback messages are lost. We would be happy to hear what you think about this.
> 
> For stream robustness (which we don't currently discuss in detail in this draft), our thinking is that the receiver may feed back a "currently decodable timestamp" in addition to the ack vector. That way, a sender can stop attempting to recover packets as soon as the receiver states that it can move past the point of the packets which haven't been acked.
> 
>  
> 
> Yes, it would be possible to use ssrc and rtp sequence numbers instead. One benefit of having a transport-wide sequence number is that it's possible to earlier detect lost packets. For instance, if you are receiving two streams, one audio (10 ms/packet) and one video (30 fps), and the last packet of a video frame is lost, you would be able to detect that already when the next audio packet is received. This becomes increasingly useful the more streams you are receiving.
> 
> In my opinion, congestion control is also something that should act on a layer above rtp streams (i.e., per connection or transport). I don't think rtp provides such a layer as is, so the suggested rtp extension aims to add a "virtual" layer which can be used for this purpose. It enables us to slightly reduce the feedback overhead since we don't have to specify ssrcs and base sequence numbers for every stream separately.
>  
> 
>  
> 
> /Ingemar
> 
>  
>

---

From: Stefan Holmer <[email-masked]>
Date: Tue, 17 Mar 2015 14:40:24 +0000

On Tue, Mar 17, 2015 at 3:20 PM Michael Welzl <[email-masked]> wrote:

> Great to see this!  - and I also said that we should have such a framework
> before:  http://www.ietf.org/proceedings/85/slides/slides-85-rmcat-0.pdf
> I felt that my statements were ignored back then, and was disappointed to
> watch the development of this rather messy situation that we're now
> facing...
>

Yes, you were definitely an early proponent for this, while I argued for
having most logic on the receive-side (for the reason you mention below).
Not trying to steal your thunder. ;)

I think introducing a transport-wide sequence number would also get us
closer to a truly shared congestion control in the future (which can be
shared between audio/video/data). But let's leave that to the future.

>
> This being said, I wonder: why not propose a generic simple sender-side
> behavior that ensures that the sender times out and backs off when it
> doesn't get feedback for long, and otherwise just sends at the rate that is
> dictated by the receiver + move all the logic to the receiver?  What made
> you suggest putting it in the sender?
>

I think there are a couple of reasons why we switched our views on this,
which I tried to mention in the draft:

- Feeding back a rate or a congestion window may not be enough for all
applications. For instance if the sender bitrate is application limited,
this will be unknown for the receiver. Also, for some applications (e.g.,
screencasts) an average bitrate doesn't always make a lot of sense, as it
most of the time won't try to use that bitrate. It would be more interested
in whether sending a burst every now and then is ok.

- Having things at the sender makes it easier to implement different types
of bandwidth measurement techniques (e.g., chirps), which otherwise would
require the sender to notifying the receiver about the probes, or that the
receiver infers that the sender attempted to probe the link. I think I
mentioned this as a possible drawback back then as well...

- We believe more feedback can be helpful also for media robustness, such
as FEC.

>
> AFAIK the rationale for doing stuff in the receiver was always the
> reduction of feedback, which I also never personally understood to be a
> major reason. So is this now a sign of people finally accepting that one
> will have to have a decent amount of feedback anyway, and hence there's no
> point in having a receiver-side CC. logic?  :-)
>

Yes, that has been my view point, and also the fact that you can keep the
feedback format very simple, independent of what signals you get from the
network. But as I mention above, I have come to realize that it may also be
somewhat limiting.

The most important part is still to keep one side as simple as possible
though.

> > Thanks for the feedback. I was planning on bringing this up for
> discussion on the list, but I was out the last week so it was a bit delayed.
> >
> > And thanks for your submission ( http://tools.ietf.org/id/
> draft-holmer-rmcat-transport-wide-cc-extensions-00.txt )
> >
> discussion with Mirja and Magnus) in http://www.ietf.org/mail-
> archive/web/rmcat/current/msg01100.html
> >
> > - Per packet arrival time is needed for the way gcc is currently
> designed. Reducing the number of delay samples can be possible, but I
> expect that it would have some negative effect on performance.
> >
> > For stream robustness (which we don't currently discuss in detail in
> this draft), our thinking is that the receiver may feed back a "currently
> > Yes, it would be possible to use ssrc and rtp sequence numbers instead.
> One benefit of having a transport-wide sequence number is that it's
> possible to earlier detect lost packets. For instance, if you are receiving
> two streams, one audio (10 ms/packet) and one video (30 fps), and the last

---

From: Michael Welzl <[email-masked]>
Date: Tue, 17 Mar 2015 15:02:02 +0000

> On 17 Mar 2015, at 15:40, Stefan Holmer <[email-masked]> wrote:
> 
> 
> 
> Yes, you were definitely an early proponent for this, while I argued for having most logic on the receive-side (for the reason you mention below). Not trying to steal your thunder. ;)

hehe  :-)    and sorry for my somewhat childish reaction to this....

> I think introducing a transport-wide sequence number would also get us closer to a truly shared congestion control in the future (which can be shared between audio/video/data). But let's leave that to the future.
>  
> 
> I think there are a couple of reasons why we switched our views on this, which I tried to mention in the draft:

Apologies, might have missed it - I have admittedly only skimmed through it...

> - Feeding back a rate or a congestion window may not be enough for all applications. For instance if the sender bitrate is application limited, this will be unknown for the receiver. Also, for some applications (e.g., screencasts) an average bitrate doesn't always make a lot of sense, as it most of the time won't try to use that bitrate. It would be more interested in whether sending a burst every now and then is ok.
> 
> - Having things at the sender makes it easier to implement different types of bandwidth measurement techniques (e.g., chirps), which otherwise would require the sender to notifying the receiver about the probes, or that the receiver infers that the sender attempted to probe the link. I think I mentioned this as a possible drawback back then as well...
> 
> - We believe more feedback can be helpful also for media robustness, such as FEC.

I agree!

> Yes, that has been my view point, and also the fact that you can keep the feedback format very simple, independent of what signals you get from the network. But as I mention above, I have come to realize that it may also be somewhat limiting.
> 
> The most important part is still to keep one side as simple as possible though.

and I agree about this too. And if we must decide one way, having a simple receiver is probably a good way to go.

Cheers,
Michael

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Tue, 17 Mar 2015 16:14:16 +0100

> I felt that my statements were ignored back then, and was disappointed to watch the development of this rather messy situation that we're now facing...

could you please explain the reason of your disappointment and why do 
you think the situation is messy? Because I am failing to connect your 
comment to any current situation.

BR

-- 
Zahed

---

From: Michael Welzl <[email-masked]>
Date: Tue, 17 Mar 2015 16:18:37 +0100

a nada sender can't talk to a scream receiver. a gcc receiver can't talk to a nada sender. etc. 

Sent from my iPhone

> On 17. mars 2015, at 16:14, Zaheduzzaman Sarker <[email-masked]> wrote:
> 
> 
> 
> could you please explain the reason of your disappointment and why do you think the situation is messy? Because I am failing to connect your comment to any current situation.
> 
> BR
> 
> 
> -- 
> Zahed
>

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Tue, 17 Mar 2015 18:14:20 +0100

> talk to a nada sender. etc.

Thanks for spelling out this.

Perhaps one can say this situation is messy but I think this situation 
was obvious from very beginning of RMCAT, when it was charted. RMCAT is 
chartered to produce number of experimental algorithms and one or more 
standard algorithms. If there will be more than one standard algorithms 
then the WG has to reason that with applicability and relative merits. 
To my understanding "interoperability" was not considered.

Now if one wants interop between different types of algorithms then, as 
Stefan mentioned, one can solve it with generic feedback format and 
feedback rate ( for feedback rate I dont see that much alternative than 
per packet feedback). However, with that in place I fear the actual 
algorithm will become less relevant.

>
> Sent from my iPhone
>
>> On 17. mars 2015, at 16:14, Zaheduzzaman Sarker
>> <[email-masked]> wrote:
>>
>>
>>
>>> On 2015-03-17 15:20, Michael Welzl wrote: Great to see this!  -
>>> and I also said that we should have such a framework before:
>>> http://www.ietf.org/proceedings/85/slides/slides-85-rmcat-0.pdf I
>>> felt that my statements were ignored back then, and was
>>> disappointed to watch the development of this rather messy
>>> situation that we're now facing...
>>
>> could you please explain the reason of your disappointment and why
>> do you think the situation is messy? Because I am failing to
>> connect your comment to any current situation.
>>
>> BR
>>
>>
>> -- Zahed
>>

-- 
Zahed

==================================================
ANM ZAHEDUZZAMAN SARKER

Ericsson AB
Services, Media and Network Features
Laboratoriegränd 11
97128 Luleå, Sweden
Phone +46 10 717 37 43
Fax +46 920 996 21
SMS/MMS +46 76 115 37 43
[email-masked]
www.ericsson.com

==================================================

---

From: Stefan Holmer <[email-masked]>
Date: Thu, 19 Mar 2015 15:29:38 +0000

> algorithm will become less relevant.

I'm seeing this in a different way. I don't believe rmcat will come up with
the final solution to the RTC congestion control problem, even though it
may be the best algorithm available to date. Therefore it is important to
be able to easily experiment with improvements and new algorithms, that
hopefully would still be able to interop with the currently recommended
method.

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 19 Mar 2015 12:14:48 -0400

Hi all,

I just quickly would like to remember you that we are only chartered to specify one or more algorithms while any RTP/RTCP extensions we might need, must be standardized in a different working group.

My take is, that we should try to get some exp RFCs to actually experiment with different algos, without worrying to much about how the feedback format looks like (while still talking about the feedback information and frequency) and then before we standardize one or more standard-track RFCs we can actually look at what information we need and how to put this information in the right format (as recommendation for a different wg) such that this information can be used by all standardized algos and potentially might even be useful for future additional schemes.

Does this make sense?

Mirja

> Am 19.03.2015 um 11:29 schrieb Stefan Holmer <[email-masked]>:
> 
> 
> 
> On Tue, Mar 17, 2015 at 6:14 PM Zaheduzzaman Sarker <[email-masked]> wrote:
> 
> 
> I'm seeing this in a different way. I don't believe rmcat will come up with the final solution to the RTC congestion control problem, even though it may be the best algorithm available to date. Therefore it is important to be able to easily experiment with improvements and new algorithms, that hopefully would still be able to interop with the currently recommended method.
> 
> 
> >
> > Sent from my iPhone
> >

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Sat, 21 Mar 2015 14:57:13 +0100

Hi Ingemar,

I am a little concerned with the removal of the ACK vector in the sense
that you have lost

the ability to accurately count the outstanding bytes.

You still refer to a partial_bytes_acked counter in the draft, but  as far
as I can see, you don't have the intelligence

to maintain this value accurately given that you took out the ACK vector. I
understand that you for loss detection purposes and also for OWD purposes
may not need the ACK vector.

However the fact the you now classify all data up to the highest
acknowledged packet as being received from a congestion control and

bytes in flight perspective worries me.

With the previous method (ACK vector) one might spuriously classify bytes
as no longer in flight (due to them spurious classified as  being lost -
dupthres  threshold)  and this has known robustness deficiencies towards
packet re-ordering. Your new approach however seems to have no robustness
"at all" towards packet re-ordering from a congestion control perspective.
Nor will packets lost be counted accurately.

Is the assumption that  this inaccuracy in the CWND calculation is
acceptable or that it somehow gets balanced out by some other adjustments
done ?  Perhaps the assumption is that the OWD guard would acts as a
stabilizer if the CWND overshoots due to re-ordering ?

Below you mention that the SCREAM usage of the highest  acknowledged
sequence number gives more room than the 16-bit sequence number of Stefan’s
draft which is on a “per socket” basis.  Or perhaps your argumentation more
goes to that you in SCREAM operates with the RTP sequence number space with
is per SSRC basis ?.  I am not sure. Should one not be concerned with an
approach where the sequence number space moved forward based on highest
acknowledged sequence number ?

In the context of the 16-bit sequence space limitation on throughput, then
I think that it might be interesting to observe that we at the moment, due
to SCTP header re-factorization, are discussing potential sequence space
limitations in SCTP. Here there is presently a 16-bit ordered sequence
number space per stream and a 32-bit (effectively a 31 bit space, RFC1982)
on a connection basis (all streams).  We _*think*_ that we may stick with
the 16-bit sequence space, at least for ordered messages as the receiver
window size anyway puts limitation on how many unrecovered ordered messages
one can have outstanding. For un-ordered messages we _*may*_ also do the
same (even if unordered would not be subject to receiver window blocking).
To emphasize, then here we are truly able to exploit the full 16-bit (not
15-bit only) space on a per stream basis. In either case we stick with only
releasing sequence numbers based on the full  recovered (cumack).
Interesting to see some of the same discussion come up here.

BR, Karen

*From:* rmcat [mailto:[email-masked]] *On Behalf Of *Ingemar
Johansson S
*Sent:* Monday, March 16, 2015 8:48 AM
*To:* rmcat WG ([email-masked]); [email-masked]; [email-masked];
Stefan Holmer
*Cc:* Ingemar Johansson S
*Subject:* [rmcat] Comments on
draft-holmer-rmcat-transport-wide-cc-extensions-00

Hi

In addition it is likely a good idea to try to make the feedback generic in
some sense as it can allow for algorithms to evolve and improve over time.
It is not for certain that all feedback elements and the RTP header
extension is needed for all algorithms so some kind of negotiation can be
useful here if one wish to minimize signaling overhead. The feedback rate
is probably something that deserves some further consideration as different
algorithms may have different requirements.

The loss counter of course means that the receiver must implement some
limited intelligence in the form of a loss detection algorithm on the
receiver side. This is however limited to some 15 lines of code (see
https://github.com/EricssonResearch/scream/blob/master/code/ScreamRx.cpp )
I don't however have any strong opinion against an ACK vector so if that is
deemed a good to have for one reason or another..

/Ingemar

Ingemar Johansson  M.Sc.

Senior Researcher

Ericsson AB

Wireless Access Networks

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

www.ericsson.com

 vast cosmic arena”

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 22 Mar 2015 20:59:12 +0000

Hi Karen

Comments inline below, marked with [IJ]

/Ingemar

>-----Original Message-----
>From: Karen Elisabeth Egede Nielsen [mailto:[email-masked]] 
>Sent: den 21 mars 2015 14:57
>To: Ingemar Johansson S; Zaheduzzaman Sarker
>Cc: rmcat WG
>Subject: RE: [rmcat] Comments on draft-holmer-rmcat-transport-wide-cc-extensions-00

>I am a little concerned with the removal of the ACK vector in the sense that you have lost
>You still refer to a partial_bytes_acked counter in the draft, but  as far as I can see, you don't have the intelligence
>to maintain this value accurately given that you took out the ACK vector. I understand that you for loss detection purposes and >also for OWD purposes may not need the ACK vector.
>However the fact the you now classify all data up to the highest acknowledged packet as being received from a congestion >control and 
>With the previous method (ACK vector) one might spuriously classify bytes as no longer in flight (due to them spurious classified >as  being lost - dupthres  threshold)  and this has known robustness deficiencies towards packet re-ordering. Your new approach >however seems to have no robustness "at all" towards packet re-ordering from a congestion control perspective. 
[IJ] I don’t rule out that an ACK vector can be useful, if it is after all considered useful then it can be considered to add it to the feedback again. The main reason why the ACK vector was removed is that we found out that in some cases a 32 bit ACK vector was not sufficient at high bitrates (~10Mbps), a slight CPU overload could then stall RTCP transmission with the effect that the span 
Regarding the Dupthresh handling, this is moved to the receiver where a local 16 bit ACK vector is computed. A 5 packet dupThresh is implemented there (see lines 57-63 in https://github.com/EricssonResearch/scream/blob/master/code/ScreamRx.cpp ). It could here be a better idea to consider a TCP RACK approach rather than a dupThresh here…
>Nor will packets lost be counted accurately.
[IJ] Hmm, don't believe so, the way it is implemented on the receiver side it will be possible to count correctly how many RTP packets are lost, it is not however possible to know exactly which packets are lost although this info can be extracted from the RFC4585 NACK.

>Is the assumption that  this inaccuracy in the CWND calculation is acceptable or that it somehow gets balanced out by some other >adjustments done ?  Perhaps the assumption is that the OWD guard would acts as a stabilizer if the CWND overshoots due to re->ordering ?
[IJ] There may be a possibility that there are inaccuracies, not sure though that there are any. As SCReAM does not implement any retransmission (done higher up in the application layer) it deviates a bit from TCPs and SCTPs more fine grained handling of retransmission, cannot rule out that there are a few more tricks that can be taken from these areas to improve performance, improvement suggestions are more than welcome. Yes, the OWD guard adds some extra stability, that is however of little help with queues that use e.g. CoDel AQMs. 
 
>Below you mention that the SCREAM usage of the highest  acknowledged sequence number gives more room than the 16-bit >sequence number of Stefan’s draft which is on a “per socket” basis.  Or perhaps your argumentation more goes to that you in >SCREAM operates with the RTP sequence number space with is per SSRC basis ?.  I am not sure. Should one not be concerned >with an approach where the sequence number space moved forward based on highest acknowledged sequence number ?
[IJ] Note sure, as long as one try to track loss events, as retransmissions (generated higher up in the stack) are recommended queued up in the RTP queue(s) and congestion controlled by SCReAM, the whole whole thing should be stable, I have at least not yet seen that it would break. 
 
>In the context of the 16-bit sequence space limitation on throughput, then I think that it might be interesting to observe that we >at the moment, due to SCTP header re-factorization, are discussing potential sequence space limitations in SCTP. Here there is >presently a 16-bit ordered sequence number space per stream and a 32-bit (effectively a 31 bit space, RFC1982) on a connection >basis (all streams).  We _think_ that we may stick with the 16-bit sequence space, at least for ordered messages as the receiver >window size anyway puts limitation on how many unrecovered ordered messages one can have outstanding. For un-ordered >messages we _may_ also do the same (even if unordered would not be subject to receiver window blocking). To emphasize, >then here we are truly able to exploit the full 16-bit (not 15-bit only) space on a per stream basis. In either case we stick with only >releasing sequence numbers based on the full  recovered (cumack).  Interesting to see some of the same discussion come up >here.
[IJ] Thanks for the heads up. 
 
>BR, Karen 
 
From: rmcat [mailto:[email-masked]] On Behalf Of Ingemar Johansson S
Sent: Monday, March 16, 2015 8:48 AM
To: rmcat WG ([email-masked]); [email-masked]; [email-masked]; Stefan Holmer
Cc: Ingemar Johansson S
Subject: [rmcat] Comments on draft-holmer-rmcat-transport-wide-cc-extensions-00
 
Hi

 

/Ingemar

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Sun, 22 Mar 2015 16:25:26 -0500

Hi Ingemar,

Please find follow up below.

BR, Karen

>From: Ingemar Johansson S [mailto:[email-masked]]
>Sent: Sunday, March 22, 2015 3:59 PM
>To: Karen E. E. Nielsen; Zaheduzzaman Sarker
>Cc: rmcat WG; Ingemar Johansson S
>>Subject: RE: [rmcat] Comments on
>>that you have lost the ability to accurately count the outstanding bytes.
>>You still refer to a partial_bytes_acked counter in the draft, but  as
>>far as I can see, you don't have the intelligence to maintain this value
>accurately given that you took out the ACK vector. I understand that you
>for
>loss detection purposes and >also for OWD purposes may not need the ACK
>vector.
>>acknowledged packet as being received from a congestion >control and
>as no longer in flight (due to them spurious classified >as  being lost -
>packet re-ordering. Your new approach >however seems to have no
>robustness "at all" towards packet re-ordering from a congestion control
>perspective.
>[IJ] I don’t rule out that an ACK vector can be useful, if it is after all
>considered
>useful then it can be considered to add it to the feedback again. The main
>reason why the ACK vector was removed is that we found out that in some
>cases a 32 bit ACK vector was not sufficient at high bitrates (~10Mbps), a
>slight
>CPU overload could then stall RTCP transmission with the effect that the
>span
>Regarding the Dupthresh handling, this is moved to the receiver where a
>local
>16 bit ACK vector is computed. A 5 packet dupThresh is implemented there
>(see lines 57-63 in
>https://github.com/EricssonResearch/scream/blob/master/code/ScreamRx.c
>pp ).
[Karen ]  I think this aspect of the protocol is important to stress in the
document.
 It could here be a better idea to consider a TCP RACK approach rather
>than a dupThresh here…
[Karen ] I don't disagree with that :-). Not sure exactly how it would  be
on a receiver side, but
perhaps it works just as well there.
>[IJ] Hmm, don't believe so, the way it is implemented on the receiver side
>it
>will be possible to count correctly how many RTP packets are lost, it is
>not
>however possible to know exactly which packets are lost although this info
>can be extracted from the RFC4585 NACK.
>
[Karen ] This was my point but I phrased it incorrectly. It is the accurate
byte counting that is lost and
that was want I intended to refer to.

>>acceptable
>or that it somehow gets balanced out by some other >adjustments done
>?  Perhaps the assumption is that the OWD guard would acts as a stabilizer
>if
>the CWND overshoots due to re->ordering ?
>[IJ] There may be a possibility that there are inaccuracies, not sure
>though that
>there are any.

[Karen ] The partial_bytes_ack counter generally is implemented as an
accurate counter,
 where the calculations are based on SACKs.
It seems SCReAM calculates this based on highest received sequence number.
This is the inaccuracy I am referring to.
The same issue exist in respect to SCReAMs definition and calculation of
bytes_in_flights (as far as I understand).

 As SCReAM does not implement any retransmission (done
>higher up in the application layer) it deviates a bit from TCPs and SCTPs
>more
>fine grained handling of retransmission,

[Karen ] I am not speaking about accurate count of bytes for retransmission
purpose but for congestion control purposes.

 cannot rule out that there are a few
>more tricks that can be taken from these areas to improve performance,
>improvement suggestions are more than welcome. Yes, the OWD guard adds
>some extra stability, that is however of little help with queues that use
>e.g.
>CoDel AQMs.
>
>sequence number gives more room than the 16-bit >sequence number of
>Stefan’s draft which is on a “per socket” basis.  Or perhaps your
>argumentation more goes to that you in >SCREAM operates with the RTP
>sequence number space with is per SSRC basis ?.  I am not sure. Should one
>not be concerned >with an approach where the sequence number space
>moved forward based on highest acknowledged sequence number ?
>[IJ] Note sure, as long as one try to track loss events, as retransmissions
>(generated higher up in the stack) are recommended queued up in the RTP
>queue(s) and congestion controlled by SCReAM, the whole whole thing
>should be stable, I have at least not yet seen that it would break.
[Karen ] Ok. I think this may depend on how RTP retransmission work and its
dependency to sequence number.
 >>In the context of the 16-bit sequence space limitation on throughput,
then I
>think that it might be interesting to observe that we >at the moment, due
>to
>SCTP header re-factorization, are discussing potential sequence space
>limitations in SCTP. Here there is >presently a 16-bit ordered sequence
>on
>a connection >basis (all streams).  We _think_ that we may stick with the
>16-
>bit sequence space, at least for ordered messages as the receiver >window
>size anyway puts limitation on how many unrecovered ordered messages one
>can have outstanding. For un-ordered >messages we _may_ also do the same
>(even if unordered would not be subject to receiver window blocking). To
>emphasize, >then here we are truly able to exploit the full 16-bit (not
>15-bit
>only) space on a per stream basis. In either case we stick with only
> >releasing
>sequence numbers based on the full  recovered (cumack).  Interesting to see
>some of the same discussion come up >here.
>http://tools.ietf.org/id/draft-holmer-rmcat-
>transport-wide-cc-extensions-00.txt ) Your statements in section 1 makes
>much sense to me and I have too advocated to have the intelligence only on
>one side, mainly to make interop and software version synchronization
>easier.
>some
>sense as it can allow for algorithms to evolve and improve over time. It is
>not
>for certain that all feedback elements and the RTP header extension is
>needed for all algorithms so some kind of negotiation can be useful here if
>one wish to minimize signaling overhead. The feedback rate is probably
>something that deserves some further consideration as different algorithms
>may have different requirements.
>
>receiver
>side. This is however limited to some 15 lines of code (see
>https://github.com/EricssonResearch/scream/blob/master/code/ScreamRx.c
>pp ) I don't however have any strong opinion against an ACK vector so if
>that is
> number”
>is similar to TCP SND.UNA and the ACK vector indicates received packets
>beyond that point (similar to SACK). My understanding is that “base
>transport
>sequence number” need to stay put until it is recovered ?, this should then
>mean that the send window cannot move more than 16 packets beyond the
>“base transport sequence number” and this can have negative consequences
>if RTT is high. Could be that I misunderstand things here though.
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 23 Mar 2015 09:39:03 +0000

Hi
Answers/comments/questions inline below, marked with [IJ]

/Ingemar

> Sent: den 22 mars 2015 22:25
> >>I am a little concerned with the removal of the ACK vector in the
> >>sense that you have lost the ability to accurately count the outstanding
> bytes.
> >accurately given that you took out the ACK vector. I understand that
> >you for loss detection purposes and >also for OWD purposes may not need
> >the ACK vector.
> >as no longer in flight (due to them spurious classified >as  being lost
> >- dupthres  threshold)  and this has known robustness deficiencies
> >towards packet re-ordering. Your new approach >however seems to have
> no
> >robustness "at all" towards packet re-ordering from a congestion
> >control perspective.
> >[IJ] I don’t rule out that an ACK vector can be useful, if it is after
> >all considered useful then it can be considered to add it to the
> >feedback again. The main reason why the ACK vector was removed is that
> >we found out that in some cases a 32 bit ACK vector was not sufficient
> >at high bitrates (~10Mbps), a slight CPU overload could then stall RTCP
> >transmission with the effect that the span Regarding the Dupthresh
> >handling, this is moved to the receiver where a local
> >16 bit ACK vector is computed. A 5 packet dupThresh is implemented
> >there (see lines 57-63 in
> [Karen ] I don't disagree with that :-). Not sure exactly how it would  be on a
> receiver side, but perhaps it works just as well there.
[IJ] Should be doable I believe, a list of received RTP packets and a wall clock timestamp when they are received should do the work, of course this means that even more intelligence is added to the receiver.

> >[IJ] Hmm, don't believe so, the way it is implemented on the receiver
> >side it will be possible to count correctly how many RTP packets are
> >lost, it is not however possible to know exactly which packets are lost
> >although this info can be extracted from the RFC4585 NACK.
> >
> [Karen ] This was my point but I phrased it incorrectly. It is the accurate byte
> counting that is lost and that was want I intended to refer to.
[IJ] Yes you're right SCReAMs way of computing bytes in flight is (to use the TCP analogy) equal to the highest SACKed byte minus SND.NXT. I understand that the term bytes_in_flight is perhaps a bit misleading here?, If you have a different suggestion for the terminology to minimize confusion, that would be more than welcome.

> >or that it somehow gets balanced out by some other >adjustments done ?
> >Perhaps the assumption is that the OWD guard would acts as a stabilizer
> >if the CWND overshoots due to re->ordering ?
> >though that there are any.
> 
> accurate counter,  where the calculations are based on SACKs.
[IJ] bytes_newly_acked in SCReAM is given by how much the highest acked RTP sequence number has advanced. If for instance, the last ACK indicated SN as highest received and the new ACK indicates SN+2, then bytes_newly_acked is computed as the sum of the sizes of RTP packets with sequence number SN+1 and SN+2. This means that even if SN+1 did not arrive at the receiver, it is still treated as received by the  bytes_newly_acked counter, the congestion window will however be reduced .
Again if this causes confusion when one map it to how TCP and SCTP works then it is perhaps better to use another terminology

> >higher up in the application layer) it deviates a bit from TCPs and
> >SCTPs more fine grained handling of retransmission,
> 
> >some extra stability, that is however of little help with queues that
> >use e.g.
> >CoDel AQMs.
> >
> >sequence number space with is per SSRC basis ?.  I am not sure. Should
> >one not be concerned >with an approach where the sequence number
> space
> >[IJ] Note sure, as long as one try to track loss events, as
> >retransmissions (generated higher up in the stack) are recommended
> >queued up in the RTP
> >think that it might be interesting to observe that we >at the moment,
> >due to SCTP header re-factorization, are discussing potential sequence
> >space limitations in SCTP. Here there is >presently a 16-bit ordered
> >sequence number space per stream and a 32-bit (effectively a 31 bit
> >space, RFC1982) on a connection >basis (all streams).  We _think_ that
> >we may stick with the
> >16-
> >bit sequence space, at least for ordered messages as the receiver
> >>window size anyway puts limitation on how many unrecovered ordered
> >messages one can have outstanding. For un-ordered >messages we _may_
> >also do the same (even if unordered would not be subject to receiver
> >window blocking). To emphasize, >then here we are truly able to exploit
> >the full 16-bit (not 15-bit
> >sequence numbers based on the full  recovered (cumack).  Interesting to
> >see some of the same discussion come up >here.
> >To: rmcat WG ([email-masked]); [email-masked];
> [email-masked];
> >Stefan Holmer
> >Cc: Ingemar Johansson S
> >transport-wide-cc-extensions-00.txt ) Your statements in section 1
> >makes much sense to me and I have too advocated to have the
> >intelligence only on one side, mainly to make interop and software
> >version synchronization easier.
> >In addition it is likely a good idea to try to make the feedback
> >generic in some sense as it can allow for algorithms to evolve and
> >improve over time. It is not for certain that all feedback elements and
> >the RTP header extension is needed for all algorithms so some kind of
> >negotiation can be useful here if one wish to minimize signaling
> >overhead. The feedback rate is probably something that deserves some
> >further consideration as different algorithms may have different
> >requirements.
> >
> >pp ) I don't however have any strong opinion against an ACK vector so
> >if that is deemed a good to have for one reason or another..
> >
> >highest received sequence number similar to the highest TCP SACKed
> >byte, this makes it possible for SCReAM to move forward the send window
> >even though there are packets lost. Your draft´s “base transport
> >sequence  number”
> >transport sequence number” need to stay put until it is recovered ?,
> >this should then mean that the send window cannot move more than 16
> >packets beyond the “base transport sequence number” and this can have
> >negative consequences if RTT is high. Could be that I misunderstand
> >things here though.
> >
> >SCReAM at least is that it is sufficient to keep track of the SSRC and
> >RTP sequence number on the sender side in order to get a good bytes in
> >flight metric.
> >
> >/Ingemar
> >

---

From: Stefan Holmer <[email-masked]>
Date: Mon, 23 Mar 2015 12:46:19 +0000

Hi,

I agree that it's important to focus on the algorithms, and that the
feedback format is orthogonal to some degree. At the same time, we are
currently experiencing the limitations of having logic split between sender
and receiver, and therefore it's important for us to improve our situation.

The reason we submitted this draft was to get input from the wg early on to
make sure we're building something that makes sense and won't be throw-away
work.

/Stefan

On Thu, Mar 19, 2015 at 5:14 PM Mirja Kühlewind <
[email-masked]> wrote:

> Hi all,
>
> I just quickly would like to remember you that we are only chartered to
> specify one or more algorithms while any RTP/RTCP extensions we might need,
> must be standardized in a different working group.
>
> My take is, that we should try to get some exp RFCs to actually experiment
> with different algos, without worrying to much about how the feedback
> format looks like (while still talking about the feedback information and
> frequency) and then before we standardize one or more standard-track RFCs
> we can actually look at what information we need and how to put this
> information in the right format (as recommendation for a different wg) such
> that this information can be used by all standardized algos and potentially
> might even be useful for future additional schemes.
>
> Does this make sense?
>
> Mirja
>
>
> > I'm seeing this in a different way. I don't believe rmcat will come up
> with the final solution to the RTC congestion control problem, even though
> it may be the best algorithm available to date. Therefore it is important
> to be able to easily experiment with improvements and new algorithms, that

---

From: Michael Welzl <[email-masked]>
Date: Mon, 23 Mar 2015 07:55:22 -0500

Hi,

I guess the specific wording in the charter that Mirja refers to is this:

- Determine if extensions to RTP/RTCP are needed for carrying congestion
control feedback, using DCCP as a model. If so, provide the requirements
for such extensions to the AVTCORE working group for standardization there.

=> it would probably make sense to refine and discuss this draft until there is consensus that it describes the requirements for such extensions well enough for this work to be handed over to AVTCORE?

Cheers,
Michael

> On 23. mar. 2015, at 07.46, Stefan Holmer <[email-masked]> wrote:
> 
> Hi,
> 
> I agree that it's important to focus on the algorithms, and that the feedback format is orthogonal to some degree. At the same time, we are currently experiencing the limitations of having logic split between sender and receiver, and therefore it's important for us to improve our situation. 
> 
> The reason we submitted this draft was to get input from the wg early on to make sure we're building something that makes sense and won't be throw-away work.
> 
> /Stefan
> 
> On Thu, Mar 19, 2015 at 5:14 PM Mirja Kühlewind <[email-masked]> wrote:
> Hi all,
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 23 Mar 2015 09:33:12 -0500

Hi all,

see below.

> Am 23.03.2015 um 07:55 schrieb Michael Welzl <[email-masked]>:
> 
> Hi,
> 

I didn’t meant to say to hand over the draft now. I think is useful to have and discuss the draft in rmcat. Just saying that we cannot adopt this draft as a wg document.