Subject: [AVTCORE] Zaheduzzaman Sarker's Discuss on draft-ietf-payload-rtp-jpegxs-16: (with DISCUSS)

From: Zaheduzzaman Sarker via Datatracker <[email-masked]>
Date: Thu, 17 Jun 2021 07:37:21 -0700

Zaheduzzaman Sarker has entered the following ballot position for
draft-ietf-payload-rtp-jpegxs-16: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-jpegxs/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

This memo is defining a RTP payload for JPEG XS that is not publicly available.
This hampers the review of the memo, specially when it is defining
terminologies which ask me to look at the specifications.

This concerns has also been raised by other ADs.

Was this document made available during the work in the working group?

---

From: Tim Bruylants <[email-masked]>
Date: Thu, 17 Jun 2021 19:10:31 +0000

> This hampers the review of the memo, specially when it is defining terminologies which ask me to look at the specifications.
>

This is a valid comment. We are unsure how to resolve this. ISO is very strict about distribution of documents outside of the working groups (i.e. it is not allowed at all).
The ISO 21122 standards were written by some of the same authors as this RTP Payload spec (like myself and Thomas Richter), but the ISO documents were (to my knowledge) not shared.

In order to resolve this issue, we need to know how IESG handled this with other ISO standards that are not open? Or is 21122 the first? Please advise.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 17 Jun 2021 14:46:59 -0500

Just chiming in here ...

On Thu, Jun 17, 2021 at 2:10 PM Tim Bruylants <[email-masked]> wrote:

> This is a valid comment. We are unsure how to resolve this. ISO is very
> strict about distribution of documents outside of the working groups (i.e.
> it is not allowed at all).
> The ISO 21122 standards were written by some of the same authors as this
> RTP Payload spec (like myself and Thomas Richter), but the ISO documents
> were (to my knowledge) not shared.
>
> In order to resolve this issue, we need to know how IESG handled this with
> other ISO standards that are not open? Or is 21122 the first? Please advise.

I note that Stephan Wenger is listed as liaison manager for SG29 in ISO, at
https://www.ietf.org/about/liaisons/ - isn't that the ISO group that
produced that specification? I'd be somewhat surprised
if draft-ietf-payload-rtp-jpegxs is the first draft the IETF has published
with a normative reference to an SC29 specification, although that's not my
area of expertise.

Perhaps Stephan is a good person to ask what's possible, and what past
practice has been?

Best,

Spencer

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 18 Jun 2021 08:50:05 -0700

> other ISO standards that are not open? Or is 21122 the first? Please advise.

This is the first one like it in my time on the IESG, and I must admit I
didn't even think to check that this external reference was usable.  I'm
glad Zaheduzzaman caught it.

It might alleviate the problem to reduce this to an informative reference,
but it doesn't look to me like you could make the argument that you don't
need the ISO standards to implement this.  You're not simply wrapping their
protocol as-is; you need some header details, for example.

I've asked the RFC Editor if they know of any documents that proceeded with
closed references to ISO documents.  If there are any, I'll review their
processing history and see if I can find a path forward here.

-MSK

---

From: Tim Bruylants <[email-masked]>
Date: Fri, 18 Jun 2021 15:59:07 +0000

So, Stephan Wenger (as IETF liaison to SC29) is in contact with ISO SC29 secretariat and the SC29 convenor to be able to provide the said ISO/IEC 21122 documents to the AVT WG. Please see his message on this. I expect some feedback by next week (hopefully).

Tim.

From: Murray S. Kucherawy <[email-masked]>
Sent: Friday 18 June 2021 17:50
To: Tim Bruylants <[email-masked]>
Cc: Zaheduzzaman Sarker <[email-masked]>; The IESG <[email-masked]>; [email-masked]; [email-masked]; Ali Begen <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: Zaheduzzaman Sarker's Discuss on draft-ietf-payload-rtp-jpegxs-16: (with DISCUSS)

This is the first one like it in my time on the IESG, and I must admit I didn't even think to check that this external reference was usable.  I'm glad Zaheduzzaman caught it.

It might alleviate the problem to reduce this to an informative reference, but it doesn't look to me like you could make the argument that you don't need the ISO standards to implement this.  You're not simply wrapping their protocol as-is; you need some header details, for example.

I've asked the RFC Editor if they know of any documents that proceeded with closed references to ISO documents.  If there are any, I'll review their processing history and see if I can find a path forward here.

-MSK

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 18 Jun 2021 16:01:20 +0000

Murray,
First, it’s not that the 21122 is not publicly available.  It is, please see here: https://www.iso.org/standard/74535.html  I think the issue at hand is that the document is not available *for free*.  AFAIK, the IETF does not have a written requirement for normative reference to be free, but requiring every interested WG/IESG member to pay a CHF 178 fee for a license to enable a review is also not easily palatable.
Second, there are many examples where the IETF normatively references ISO documents that are equally not available for free.  You can go back to RFC 2250, or, for a more recent one, you can look at RFC 6416.
Stephan

From: avt <[email-masked]> on behalf of "Murray S. Kucherawy" <[email-masked]>
Date: Friday, June 18, 2021 at 08:50
To: Tim Bruylants <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, The IESG <[email-masked]>
Subject: Re: [AVTCORE] Zaheduzzaman Sarker's Discuss on draft-ietf-payload-rtp-jpegxs-16: (with DISCUSS)

-MSK

---

From: Bernard Aboba <[email-masked]>
Date: Fri, 18 Jun 2021 09:13:20 -0700

Thank you, Stephan.

This situation is by no means unique.

>From RFC 7241:

"3.2 <https://datatracker.ietf.org/doc/html/rfc7241/#section-3.2>. Access
to IEEE 802 Standards

   IEEE 802 standards, once approved, are published and made available
   for sale.  They can be purchased from the IEEE Standards Store, at
   <http://www.techstreet.com/IEEEgate.html>.  They are also available
   from other outlets, including the IEEE Xplore digital library, at
   <http://ieeexplore.ieee.org>."

On Fri, Jun 18, 2021 at 9:01 AM Stephan Wenger <[email-masked]> wrote:

> Murray,
>
> First, it’s not that the 21122 is not publicly available.  It is, please
> see here: https://www.iso.org/standard/74535.html  I think the issue at
> hand is that the document is not available **for free**.  AFAIK, the IETF
> does not have a written requirement for normative reference to be free, but
> requiring every interested WG/IESG member to pay a CHF 178 fee for a
> license to enable a review is also not easily palatable.
>
> Second, there are many examples where the IETF normatively references ISO
> documents that are equally not available for free.  You can go back to RFC
> 2250, or, for a more recent one, you can look at RFC 6416.
>
> Stephan
>
>
>
> *From: *avt <[email-masked]> on behalf of "Murray S. Kucherawy" <
> [email-masked]>
> *Date: *Friday, June 18, 2021 at 08:50
> *To: *Tim Bruylants <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>, "
> [email-masked]" <
> [email-masked]>, "[email-masked]" <[email-masked]>,
> The IESG <[email-masked]>
> *Subject: *Re: [AVTCORE] Zaheduzzaman Sarker's Discuss on
> draft-ietf-payload-rtp-jpegxs-16: (with DISCUSS)
>
>
>
> I've asked the RFC Editor if they know of any documents that proceeded
> with closed references to ISO documents.  If there are any, I'll review
> their processing history and see if I can find a path forward here.
>
>
>
> -MSK
>

---

From: Keith Drage <[email-masked]>
Date: Fri, 18 Jun 2021 17:21:00 +0100

I suspect that two issues are getting confused here.

1)    Is it valid to normatively reference an ISO standard from an IETF 
RFC. Yes. It is publicly available even if one has to pay for it. All 
conditions are met.

2)    Do IETF members need to be able to view the ISO standard in order 
to review and complete the IETF RFC. Possibly yes for WG review and WG 
last call. In prior cases, documents (not ISO ones) have been put up in 
an area of the IETF, possibly with some limited access, or for IETF 
participants to be given limited access to the other SDOs website (or an 
area of it), or circulated on a need to have basis, so they can be used 
in such a review. I do not remember this to be an issue for IESG last call.

There were a set of cooperation agreements with other SDOs that set out 
some of these things. I know 3GPP/ETSI, and ITU-T existed. I don't think 
there was ever an ISO one, but I may be wrong.

regards

Keith Drage

On 18/06/2021 17:01, Stephan Wenger wrote:
>
> Murray,
>
> First, it’s not that the 21122 is not publicly available.  It is, 
> please see here: https://www.iso.org/standard/74535.html 
> <https://www.iso.org/standard/74535.html> I think the issue at hand is 
> that the document is not available **for free**.  AFAIK, the IETF does 
> not have a written requirement for normative reference to be free, but 
> Second, there are many examples where the IETF normatively references 
> ISO documents that are equally not available for free.  You can go 
> back to RFC 2250, or, for a more recent one, you can look at RFC 6416.
>
> Stephan
>
> *From: *avt <[email-masked]> on behalf of "Murray S. Kucherawy" 
> <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>, 
> "[email-masked]" 
> <[email-masked]>, "[email-masked]" 
> <[email-masked]>, The IESG <[email-masked]>
>     > This hampers the review of the memo, specially when it is
>     defining terminologies which ask me to look at the specifications.
>     >
>     This is a valid comment. We are unsure how to resolve this. ISO is
>     very strict about distribution of documents outside of the working
>     groups (i.e. it is not allowed at all).
>     The ISO 21122 standards were written by some of the same authors
>     as this RTP Payload spec (like myself and Thomas Richter), but the
>     ISO documents were (to my knowledge) not shared.
>
>     In order to resolve this issue, we need to know how IESG handled
>     this with other ISO standards that are not open? Or is 21122 the
>     first? Please advise.
>
> This is the first one like it in my time on the IESG, and I must admit 
> I didn't even think to check that this external reference was usable.  
> I'm glad Zaheduzzaman caught it.
>
> It might alleviate the problem to reduce this to an informative 
> reference, but it doesn't look to me like you could make the argument 
> that you don't need the ISO standards to implement this.  You're not 
> simply wrapping their protocol as-is; you need some header details, 
> for example.
>
> with closed references to ISO documents.  If there are any, I'll 
> review their processing history and see if I can find a path forward here.
>
> -MSK
>
>
> _______________________________________________
> Audio/Video Transport Core Maintenance
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sat, 19 Jun 2021 13:59:34 -0700

Stephan,

Thanks for your comments.  My language ("usable") may have been a bit
broad; I agree with your assessment.  The problem is the basic notion of
how a reviewer, particularly an IESG member, is supposed to evaluate a
document when some of its normative references are not freely available, as
you described.

I imagine until we have some kind of automatic process in place where the
referenced ISO specification is made available to the IESG or other
reviewers during the normal processing cycle of the document, we're going
to continue to hit snags like this.  Another problem is consistency.  I
checked with the RFC Editor and found that there have been other recent
I-Ds passed through the IESG that had an external normative reference
behind a paywall and it seems on cursory review of the final ballot
positions that nobody noticed or thought to raise it; the fact that an IESG
member would actually like to see the content of the referenced document is
the reason it matters this time.

At any rate, I think the outcome of Tim's inquiry to ISO will inform our
next step hre.

-MSK

---

From: Tim Bruylants <[email-masked]>
Date: Wed, 23 Jun 2021 07:33:55 +0000

Hi all,

It was actually Stephan who contacted ISO (as he is the IETF’s liaison to ISO). I’m not sure if he heard back already (other than from ISO SC29 secretariat).

The main question is: Who needs access to ISO 21122-1, -2 and -3 for making this XS RTP Payload spec pass? It seems that this is a bigger issue than only the XS RTP spec and more between ISO and IETF at organizational level. My concern is that the XS RTP spec is held “hostage” in this discussion, while it is not something that any of the authors can resolve. (And to be clear, I also understand the DISCUSS position, which is certainly warranted). A lot of companies are actually waiting on the official RFC publication of this spec ☹

I’m absolutely willing to help in this particular case to get the XS RTP Payload spec moving again. So, if anyone can tell me what is needed exactly, then Stephan, myself and maybe other people can step in?

@Stephan Wenger<mailto:[email-masked]> did you receive any other feedback, other than the fact that you as liaison can download the ISO standards (but watermarked on your name, which you are not allowed to further distribute?).

Kind regards,
Tim.

PS: I’m not on all the mailing lists (as member), so my reply will probably not get to all the destinations. Please forward where needed.

From: Murray S. Kucherawy <[email-masked]>
Sent: Saturday 19 June 2021 23:00
To: Stephan Wenger <[email-masked]>
Cc: Tim Bruylants <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; The IESG <[email-masked]>
Subject: Re: [AVTCORE] Zaheduzzaman Sarker's Discuss on draft-ietf-payload-rtp-jpegxs-16: (with DISCUSS)

Stephan,

Thanks for your comments.  My language ("usable") may have been a bit broad; I agree with your assessment.  The problem is the basic notion of how a reviewer, particularly an IESG member, is supposed to evaluate a document when some of its normative references are not freely available, as you described.

I imagine until we have some kind of automatic process in place where the referenced ISO specification is made available to the IESG or other reviewers during the normal processing cycle of the document, we're going to continue to hit snags like this.  Another problem is consistency.  I checked with the RFC Editor and found that there have been other recent I-Ds passed through the IESG that had an external normative reference behind a paywall and it seems on cursory review of the final ballot positions that nobody noticed or thought to raise it; the fact that an IESG member would actually like to see the content of the referenced document is the reason it matters this time.

At any rate, I think the outcome of Tim's inquiry to ISO will inform our next step hre.

-MSK

On Fri, Jun 18, 2021 at 9:01 AM Stephan Wenger <[email-masked]<mailto:[email-masked]>> wrote:
Murray,
First, it’s not that the 21122 is not publicly available.  It is, please see here: https://www.iso.org/standard/74535.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.iso.org_standard_74535.html&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=LTxUGukLCEfEUdo_bq048Q&m=LLKUhkxwY4uh6PdYu-DqqB-NSjRNYUnncL-s1h2s70Q&s=1oSRA3Ja1SAj29xGSlhdQwi87bOIQWzD84-ejxICWZM&e=>  I think the issue at hand is that the document is not available *for free*.  AFAIK, the IETF does not have a written requirement for normative reference to be free, but requiring every interested WG/IESG member to pay a CHF 178 fee for a license to enable a review is also not easily palatable.
Second, there are many examples where the IETF normatively references ISO documents that are equally not available for free.  You can go back to RFC 2250, or, for a more recent one, you can look at RFC 6416.
Stephan

From: avt <[email-masked]<mailto:[email-masked]>> on behalf of "Murray S. Kucherawy" <[email-masked]<mailto:[email-masked]>>
Date: Friday, June 18, 2021 at 08:50
To: Tim Bruylants <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, The IESG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [AVTCORE] Zaheduzzaman Sarker's Discuss on draft-ietf-payload-rtp-jpegxs-16: (with DISCUSS)

-MSK

---

From: Thomas Richter <[email-masked]>
Date: Wed, 23 Jun 2021 09:47:21 +0200

Dear colleagues, Tim,

> It was actually Stephan who contacted ISO (as he is the IETF’s liaison 
> to ISO). I’m not sure if he heard back already (other than from ISO SC29 
> secretariat).

As far as I understand, Mayumi (SC29 secretariat) provided the documents 
with Stephan's name as watermark. If not so, Stephan, please ping Mayumi 
again.

Otherwise, I agree with Tim. It's really time to get this spec moving. 
There are at least two working implementations of it, so it's certainly 
functional, and clients and customers are waiting.

Greetings,
	Thomas

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 23 Jun 2021 15:10:53 +0000

Word from Mayumi: they are still checking.
S.

On 6/23/21, 00:47, "Thomas Richter" <[email-masked]> wrote:

    Dear colleagues, Tim,

    Greetings,
    	Thomas