Subject: Re: [rmcat] WG last call: draft-ietf-rmcat-eval-criteria-07.txt

From: Joerg Ott <[email-masked]>
Date: Fri, 2 Nov 2018 18:01:14 +0100

Hi Colin,

thanks much for the comments.  Finally, a first (not yet complete) stab
at this:

>   * Section 3 begins “Each experiment is expected to…” but the draft
>     hasn’t yet introduced the idea that there are some experiments that
>     need to be completed.

Added a some context intro.

>   * Rather than talk about “RMCAT proposals”, “RMCAT flows”, etc.,
>     discuss “proposed RTP congestion control algorithms”, “congestion
>     controlled RTP flows”, and so on. The RFC will live longer than the
>     RMCAT working group.

Fixed throughout the draft.

>   * Section 3: bullet 10 should be rephrased for clarity.

Bullet 10 is the editor's note?

>   * Does Section 3.1 need to describe the precise format, or just
>     the information to be logged? Does it matter that the file is CSV
>     format?

I'd say it simplifies running comparisons in the end and avoids parsing
errors, but I don't feel strongly here.  Any format would do.

>   * Section 4.1: The introductory text only talks about high latency
>     links, then gives examples covering a range of latencies. Expands
>     remarks “, as well as low-latency links”

Done.

>   * Section 4.2: might be appropriate to add some motivating remarks.

Done.

>   * Section 4.3: add some remarks to motivate drop-tail?

This begs a broader discussion now that some time has passed since the
initial version of this draft.  Should we include AQM models (such as
CoDel or variants thereof) as well?

>   * Section 4.5.3: clarify if this recommended distribution applies to
>     both sections 4.5.1 and 4.5.2.

Only 4.5.2 is mandatory.  The same distribution could be applied to
both jitter models.  Not sure how much we need to say explicitly.

>   * Section 6.1: first paragraph ends mid-sentence.

Done.

An interesting question is if DASH-style flows should be considered, too.

Should we specify TCP congestion control?  E.g., CUBIC?

>   * Section 6.3: is it worth mentioning that QUIC flows use UDP, but
>     are expected to have dynamics that look a lot like TCP, so don’t
>     need to be explicitly included here?

Added a note here.  Should we anticipate future divergence from TCP
behavior?

>   * Section 7: security needs to be discussed. Maybe point out
>     denial-of-service issues due to lack of congestion control, or
>     denial-of-service on the congestion controlled flow due to spoofing
>     of control signals, and state that the candidate algorithms should
>     consider. The only specific security issue I can think of the these
>     criteria is checking that the algorithm works as expected in these
>     cases.

This remains tbd.  Maybe we can have a quick chat next week.

>   * Please spell-check the draft.

Done.

Will submit as soon as the I-D tool allows this again (read: on the
weekend, even if this means a short-term expiry of the draft.

Jörg

>> On 20 Jun 2018, at 10:46, Colin Perkins <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> This is to announce a working group last call on “Evaluating 
>> Congestion Control for Interactive Real-time Media” 
>> (draft-ietf-rmcat-eval-criteria-07).
>>
>> Please send any final comments to the working group mailing list and 
>> the authors by 20 July 2018 (the date of the RMCAT session at IETF 
>> 102). If no substantive comments are received by that time, we intend 
>> to submit this draft to the IESG for publication as an Informational RFC.
>>
>> Colin
>>
>>
>>
>>> On 2 May 2018, at 11:52, Colin Perkins <[email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>> Jörg – thanks for updating this draft!
>>>
>>> Any comments from the group before we progress this?
>>>
>>> Colin
>>>
>>>
>>>
>>>> On 1 May 2018, at 19:57, [email-masked] 
>>>> <mailto:[email-masked]> wrote:
>>>>
>>>>
>>>> A New Internet-Draft is available from the on-line Internet-Drafts 
>>>> directories.
>>>> This draft is a work item of the RTP Media Congestion Avoidance 
>>>> Techniques WG of the IETF.
>>>>
>>>>      Title           : Evaluating Congestion Control for Interactive 
>>>> Real-time Media
>>>>      Authors         : Varun Singh
>>>>                        Joerg Ott
>>>>                        Stefan Holmer
>>>> Filename        : draft-ietf-rmcat-eval-criteria-07.txt
>>>> Pages           : 16
>>>> Date            : 2018-05-01
>>>>
>>>> Abstract:
>>>> The Real-time Transport Protocol (RTP) is used to transmit media in
>>>> telephony and video conferencing applications.  This document
>>>> describes the guidelines to evaluate new congestion control
>>>> algorithms for interactive point-to-point real-time media.
>>>>
>>>>
>>>> The IETF datatracker status page for this draft is:
>>>> https://datatracker.ietf.org/doc/draft-ietf-rmcat-eval-criteria/
>>>>
>>>> There are also htmlized versions available at:
>>>> https://tools.ietf.org/html/draft-ietf-rmcat-eval-criteria-07
>>>> https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-eval-criteria-07
>>>>
>>>> A diff from the previous version is available at:
>>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-rmcat-eval-criteria-07
>>>>
>>>>
>>>> Please note that it may take a couple of minutes from the time of 
>>>> submission
>>>> until the htmlized version and diff are available at tools.ietf.org.
>>>>
>>>> Internet-Drafts are also available by anonymous FTP at:
>>>> ftp://ftp.ietf.org/internet-drafts/
>>>>
>>>
>>>
>>>
>>> -- 
>>> Colin Perkins
>>> https://csperkins.org/
>>
> 
> 
> 
> -- 
> Colin Perkins
> https://csperkins.org/
> 
> 
> 
>

---

From: Colin Perkins <[email-masked]>
Date: Sun, 4 Nov 2018 12:58:54 +0700

Hi Jörg,

Thanks – some replies inline.
Colin

> On 3 Nov 2018, at 00:01, Joerg Ott <[email-masked]> wrote:
> 
> Hi Colin,
> 
> Bullet 10 is the editor’s note?

Right – it looks like a to-do note, so needs to be rephrased as something that reads like a complete point, or removed.

> errors, but I don't feel strongly here.  Any format would do.

Maybe just add that as explanation?

> CoDel or variants thereof) as well?

The eval-test draft mentions CoDel and PIE as things you’re encouraged to test with, but doesn’t look to focus on them. I’d say that’s likely the right way to approach this: encourage, but not require, experimentation with AQM.

> both jitter models.  Not sure how much we need to say explicitly.

I was envisaging “Whether Random Bounded PDV or Approximately Random Subject to No-Reordering Bounded PDV, it is recommended that z(n) is…”

> An interesting question is if DASH-style flows should be considered, too.

Testing against DASH-style traffic is certainly worthwhile. Is it included in eval-test too? Should it be?

> Should we specify TCP congestion control?  E.g., CUBIC?

It’s widely deployed, so seems like a reasonable default. Ideally, we’d encourage testing against a range of congestion control algorithms.

> Added a note here.  Should we anticipate future divergence from TCP

As in future versions of QUIC might do something different, and if they do it’s important to test against them? Sure, doesn’t hurt to add a note.

> This remains tbd.  Maybe we can have a quick chat next week.

Ok.

>>> On 20 Jun 2018, at 10:46, Colin Perkins <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> This is to announce a working group last call on “Evaluating Congestion Control for Interactive Real-time Media” (draft-ietf-rmcat-eval-criteria-07).
>>> 
>>> Please send any final comments to the working group mailing list and the authors by 20 July 2018 (the date of the RMCAT session at IETF 102). If no substantive comments are received by that time, we intend to submit this draft to the IESG for publication as an Informational RFC.
>>> 
>>> Colin
>>> 
>>> 
>>> 
>>>> On 2 May 2018, at 11:52, Colin Perkins <[email-masked] <mailto:[email-masked]>> wrote:
>>>> 
>>>>> On 1 May 2018, at 19:57, [email-masked] <mailto:[email-masked]> wrote:
>>>>> 
>>>>> 
>>>>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>>>>> This draft is a work item of the RTP Media Congestion Avoidance Techniques WG of the IETF.
>>>>> 
>>>>>      Title           : Evaluating Congestion Control for Interactive Real-time Media
>>>>>      Authors         : Varun Singh
>>>>>                        Joerg Ott
>>>>>                        Stefan Holmer
>>>>> Please note that it may take a couple of minutes from the time of submission

-- 
Colin Perkins
https://csperkins.org/

---

From: Joerg Ott <[email-masked]>
Date: Mon, 5 Nov 2018 22:29:52 +0700

Hi Colin,

thanks, a few replies inline.  Just submitted an update.

> Right – it looks like a to-do note, so needs to be rephrased as something that reads like a complete point, or removed.

I moved this past the list and made this a regular pointer (no longer
an editor's note) to the cited draft.

> Maybe just add that as explanation?

I made such a note.  Hope we won't get a fight over the format, but so
far nobody really seemed to care.

> The eval-test draft mentions CoDel and PIE as things you’re encouraged to test with, but doesn’t look to focus on them. I’d say that’s likely the right way to approach this: encourage, but not requ…

We had a longer discussion in the draft earlier (I can still see the 
text in comments) but I think this was moved out for simplicity.  I
added a short note.

Done.

> Testing against DASH-style traffic is certainly worthwhile. Is it included in eval-test too? Should it be?

I will leave this open for now to get the draft shipped today.

> It’s widely deployed, so seems like a reasonable default. Ideally, we’d encourage testing against a range of congestion control algorithms.

Added a sentence to this end.

> As in future versions of QUIC might do something different, and if they do it’s important to test against them? Sure, doesn’t hurt to add a note.

Added some short and rather vague note.

>> weekend, even if this means a short-term expiry of the draft.

Just submitted the update.

Cheers,
Jörg

---

From: Colin Perkins <[email-masked]>
Date: Thu, 8 Nov 2018 10:06:27 +0700

Thanks, Jörg – looks good to me.
Colin

> On 5 Nov 2018, at 22:29, Joerg Ott <[email-masked]> wrote:
> 
> Hi Colin,
> 
> We had a longer discussion in the draft earlier (I can still see the text in comments) but I think this was moved out for simplicity.  I
> added a short note.
> 

-- 
Colin Perkins
https://csperkins.org/