Subject: Re: [tsvwg] Datatracker State Update Notice: <draft-ietf-tsvwg-rlc-fec-scheme-09.txt>

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 11 Oct 2018 13:25:19 -0500

Dear Wes the Document Shepherd,

So, here's the deal on this one. (the companion draft is approved, pending
my review of revisions to address comments, so don't worry about that one)

On Thu, Oct 11, 2018 at 9:51 AM IETF Secretariat <
[email-masked]> wrote:

> IESG state changed:
>
> New State: IESG Evaluation::Revised I-D Needed
>
> (The previous state was IESG Evaluation)
>
>
> Datatracker URL:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-rlc-fec-scheme/

The most significant concerns that I was catching on today's telechat for
this draft were basically related. Everything else seems to be on track to
resolve (even the question about the copyright is off as a question to the
lawyer).

Multiple ADs expressed concerns based on using code as a normative
specification, based on their experience in RAI/ART with the OPUS codec,
specified in https://tools.ietf.org/html/rfc6716#appendix-A as a reference
implementation.

I am told that everyone working on OPUS decided that was the right thing to
do then, and has now decided it was the wrong thing to do now.

There are two problems they ran into:

   - Ensuring that their reference implementation is consistent across
   generations of processors and compilers, with people using different
   versions of the C language, turns out to be really hard. Whether C89, C99,
   C11, C18, or some future version of the language will suddenly return
   unexpected results on some processor that hasn't been tested, really can't
   be known in practice. So this isn't as stable a specification as we would
   have thought it would be.
   - Worse(!), after the publication of RFC 6716, everyone immediately
   extracted the code from the RFC and started changing it for their product
   implementations. So when the working group started to update the codec
   specification, which involved changing the reference implementation, all
   the implementers had to figure out what to change in their modified code to
   match the changes in the reference implementation, and that turned out to
   be a nightmare.

So, the question to me, and now to you, is, "do you REALLY want to do this?
Because we wanted to do this with OPUS, and we all thought it was a great
plan, and we were wrong, and we don't understand why this will turn out
better than OPUS".

The theory is that this mechanism would be pretty simple to express in
text.

We didn't discuss this, but I suspect that if the implementation was an
example, and not a reference implementation, that would help a lot.

The ADs who have the most experience with this issue for OPUS are Ben
Campbell, Adam Roach, and Eric Rescorla (yeah, he's a SEC AD, but also
worked on OPUS).

If you would like for me to put this on an agenda for the next informal
telechat (would be a week from today), I'd be happy to do that.

If you would like for me to arrange a meeting in Bangkok with one or more
of these ADs, I'd be happy to do that (I'd start with Ben, who actually
closed CODEC).

Please let me know how you/the working group would like to proceed.

Spencer

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 11 Oct 2018 15:46:01 -0400

Hi Spencer, I think the working group should discuss this, because 
during the course of producing the document, I don't recall discussions 
explicitly around the topic.  It seems like it would be ideal if we 
could reference a high-quality directly implementable description of the 
algorithm that the code implements, rather than the code itself (so that 
the code could become informative rather than normative).

On 10/11/2018 2:25 PM, Spencer Dawkins at IETF wrote:
> So, here's the deal on this one. (the companion draft is approved, 
> pending my review of revisions to address comments, so don't worry 
> about that one)
>
> On Thu, Oct 11, 2018 at 9:51 AM IETF Secretariat 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     IESG state changed:
>
> The most significant concerns that I was catching on today's telechat 
> for this draft were basically related. Everything else seems to be on 
> track to resolve (even the question about the copyright is off as a 
> question to the lawyer).
>
> specification, based on their experience in RAI/ART with the OPUS 
> codec, specified in https://tools.ietf.org/html/rfc6716#appendix-A as 
> a reference implementation.
>
> I am told that everyone working on OPUS decided that was the right 
> thing to do then, and has now decided it was the wrong thing to do now.
>
>   * Ensuring that their reference implementation is consistent across
>     generations of processors and compilers, with people using
>     different versions of the C language, turns out to be really hard.
>     Whether C89, C99, C11, C18, or some future version of the language
>     will suddenly return unexpected results on some processor that
>     hasn't been tested, really can't be known in practice. So this
>     isn't as stable a specification as we would have thought it would be.
>   * Worse(!), after the publication of RFC 6716, everyone immediately
>     extracted the code from the RFC and started changing it for their
>     product implementations. So when the working group started to
>     update the codec specification, which involved changing the
>     reference implementation, all the implementers had to figure out
>     what to change in their modified code to match the changes in the
>     reference implementation, and that turned out to be a nightmare.
>
> So, the question to me, and now to you, is, "do you REALLY want to do 
> this? Because we wanted to do this with OPUS, and we all thought it 
> was a great plan, and we were wrong, and we don't understand why this 
> will turn out better than OPUS".
>
> We didn't discuss this, but I suspect that if the implementation was 
> an example, and not a reference implementation, that would help a lot.
>
> If you would like for me to put this on an agenda for the next 
> informal telechat (would be a week from today), I'd be happy to do that.
>
> If you would like for me to arrange a meeting in Bangkok with one or 
> more of these ADs, I'd be happy to do that (I'd start with Ben, who 
> actually closed CODEC).
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Sat, 13 Oct 2018 22:16:14 -0500

Hi, Wesley,

On Thu, Oct 11, 2018 at 2:46 PM Wesley Eddy <[email-masked]> wrote:

> Hi Spencer, I think the working group should discuss this, because during
> the course of producing the document, I don't recall discussions explicitly
> around the topic.  It seems like it would be ideal if we could reference a
> high-quality directly implementable description of the algorithm that the
> code implements, rather than the code itself (so that the code could become
> informative rather than normative).
>

Excellent. I understand that an IETF meeting may be happening soon, if that
helps you discuss with the working group ;-) ...

I'll keep this draft in IESG Evaluation while you discuss this with the
working group. If I should send it back to the working group in the
datatracker, just let me know.

draft-ietf-tsvwg-fecframe-ext-06 is already Approved, Announcement to be
Sent, but the secretariat will hold it ("point raised, write-up needed")
until I say it's good to go (all IESG Evaluation comments have been
addressed appropriately). I suspect you'd want me to hold onto it
until draft-ietf-tsvwg-rlc-fec-scheme is approved - does that sound right
to you?

And thanks to all of you for your work on this. I know it matters.

Spencer

On 10/11/2018 2:25 PM, Spencer Dawkins at IETF wrote:
> specified in https://tools.ietf.org/html/rfc6716#appendix-A as a
> reference implementation.
>
> I am told that everyone working on OPUS decided that was the right thing
> to do then, and has now decided it was the wrong thing to do now.
>
> this? Because we wanted to do this with OPUS, and we all thought it was a
> great plan, and we were wrong, and we don't understand why this will turn
> out better than OPUS".
>