Subject: [rmcat] Roman Danyliw's Discuss on draft-ietf-rmcat-eval-criteria-12: (with DISCUSS and COMMENT)

From: Roman Danyliw via Datatracker <[email-masked]>
Date: Tue, 03 Mar 2020 06:58:42 -0800

Roman Danyliw has entered the following ballot position for
draft-ietf-rmcat-eval-criteria-12: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-rmcat-eval-criteria/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Section 5.2.  Per “Sample video test sequences are available at: [xiph-seq] and
[HEVC-seq].  The following two video streams are the recommended minimum for
testing: Foreman and FourPeople.”, these test sequences seems underspecified.

** Is the “recommended” here intended to be normative?  There is no RFC2119
boiler plate in this document to guide the parsing of the text.

** From the text, there wasn’t much precision in where to find these
recommended videos (Foreman and FourPeople).  At the url pointed to by
[HEVC-seq], I found the filenames “FourPeople_1280x720_60.yuv” and
“foreman15_4000.yuv”, is that them?

** Is it expected for
http://www.netlab.tkk.fi/~varun/test_sequences/foreman15_4000.yuv to be 0
bytes?  I tried on 03/03/2020 at ~0950 EST

** Give that that one of the recommended urls doesn’t work even before this
draft is published, I have great reservation with keeping a normative
“recommended” to such external repositories.  However, providing pointers to
repositories of “sample video test sequences” makes sense to me and is helpful.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

** Section 3.1.  The purpose of this log isn’t clear.  What is the relationship
of it to the metrics described in the previous section?  Where does it fit into
the measurement workflow?  Is this constructed on a per packet capture file
basis?

** References:
-- Section 3.  Consider adding a citation for tcpdump and wireshark

-- Section 4.4.  Consider adding a citation for the “Bilbert-Elliot” model

** Editorial:
-- Section 3.  Recommend explicitly spelling out PCAP as packet capture.

-- Section 4.5. s/is is/is/

---

From: Joerg Ott <[email-masked]>
Date: Mon, 9 Mar 2020 20:56:00 +0100

Dear Roman,

thanks much for your comments.  I just posted an updated draft (-13) to
address those.  See below.

Recommended is recommended as per English language and it not used as a
verb but as an adjective, indicating a suggestion.

> “foreman15_4000.yuv”, is that them?

This is fixed in the rev now, also the link that disappeared is removed.

> bytes?  I tried on 03/03/2020 at ~0950 EST

The broken link is gone now.  Both videos can be found in a single place
now.  It is clearer specified where to find them.

> repositories of “sample video test sequences” makes sense to me and is helpful.

Should be addressed by now.

Added a clarifying sentence:

These data should suffice to compute the media-encoding independent
metrics described above.  Use of a common log will allow simplified
post-processing and analysis across different implementations.

> -- Section 3.  Consider adding a citation for tcpdump and wireshark

done

> -- Section 4.4.  Consider adding a citation for the “Bilbert-Elliot” model

We got this comment also from Mirja, to which I replied:

I could include a reference here for Gilbert-Elliot such as:
http://www.ohohlfeld.com/paper/hasslinger_hohlfeld-mmb_2008.pdf

However, this not mandatory text and the second part of the sentence
is even less specific and does (deliberately) not give a specific
example.  We don't want to create any prejudice here beyond random
losses.  The Internet changes, after all.

In order to avoid this being a showstopper here (we don't have enough
hour tonight for another rev round, I am adding this in now.  Please
provide feedback if this is really needed.  I could still remove it
in two weeks when I-D submission reopens.

> -- Section 3.  Recommend explicitly spelling out PCAP as packet capture.

This is spelled out in brackets; since this also refers to a commonly
used file format, I would keep it as is.

> -- Section 4.5. s/is is/is/

Fixed.

Thanks much,
Jörg

---

From: Roman Danyliw <[email-masked]>
Date: Wed, 18 Mar 2020 23:18:39 +0000

Hi Joerg!

Thanks for the update.  -13 addresses my DISCUSS points.  One non-blocking comment on a COMMENT inline ...

Thanks,
Roman

-----Original Message-----
From: iesg <[email-masked]> On Behalf Of Joerg Ott
Sent: Monday, March 9, 2020 3:56 PM
To: Roman Danyliw <[email-masked]>; The IESG <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Martin Stiemerling <[email-masked]>; [email-masked]; Colin Perkins <[email-masked]>
Subject: Re: Roman Danyliw's Discuss on draft-ietf-rmcat-eval-criteria-12: (with DISCUSS and COMMENT)

Dear Roman,

thanks much for your comments.  I just posted an updated draft (-13) to address those.  See below.

> Section 5.2.  Per “Sample video test sequences are available at: 
> [xiph-seq] and [HEVC-seq].  The following two video streams are the 
> recommended minimum for
> ** Is the “recommended” here intended to be normative?  There is no 
> RFC2119 boiler plate in this document to guide the parsing of the text.

Recommended is recommended as per English language and it not used as a verb but as an adjective, indicating a suggestion.

> ** Is it expected for
> http://www.netlab.tkk.fi/~varun/test_sequences/foreman15_4000.yuv to 
> be 0 bytes?  I tried on 03/03/2020 at ~0950 EST

The broken link is gone now.  Both videos can be found in a single place now.  It is clearer specified where to find them.

> ** Give that that one of the recommended urls doesn’t work even before 
> this draft is published, I have great reservation with keeping a 
> normative “recommended” to such external repositories.  However, 
> providing pointers to repositories of “sample video test sequences” makes sense to me and is helpful.

> ** Section 3.1.  The purpose of this log isn’t clear.  What is the 
> relationship of it to the metrics described in the previous section?  
> Where does it fit into the measurement workflow?  Is this constructed 
> on a per packet capture file basis?

These data should suffice to compute the media-encoding independent metrics described above.  Use of a common log will allow simplified post-processing and analysis across different implementations.

done

[Roman]  Since it would be informative, I don't think it would hurt to add this citation.

However, this not mandatory text and the second part of the sentence is even less specific and does (deliberately) not give a specific example.  We don't want to create any prejudice here beyond random losses.  The Internet changes, after all.

In order to avoid this being a showstopper here (we don't have enough hour tonight for another rev round, I am adding this in now.  Please provide feedback if this is really needed.  I could still remove it in two weeks when I-D submission reopens.

> -- Section 3.  Recommend explicitly spelling out PCAP as packet capture.

This is spelled out in brackets; since this also refers to a commonly used file format, I would keep it as is.

Fixed.

Thanks much,
Jörg

---

From: Joerg Ott <[email-masked]>
Date: Thu, 19 Mar 2020 14:21:56 +0100

Hi Roman,

thanks much, since the ref is already included in -13 we should be
happily done.

Best,
Jörg