Subject: Re: [precis] draft-whited-precis-test-vectors-00

From: Sam Whited <[email-masked]>
Date: Thu, 06 Dec 2018 09:15:51 -0600

On Wed, Dec 5, 2018, at 12:33, Marc Blanchet wrote:
> haven’t seen any trafic on this, while the doc is (to me) useful. 
> Might consider sending it to the Independent Series Editor. See more 
> information here: https://www.rfc-editor.org/about/independent/

Hi Marc, thanks for the followup.

I've been meaning to go back and double check that there aren't any duplicate vectors covering the same edge cases and prepare an update, but I haven't found a good way to format the test data still so I'd be hesitant to submit it (and there's practically no information about how to format things or write this awful XML format). I've also tried looking into the process, and the various IETF sites are inscrutable and full of contradictory information, so thanks for the link, I'll look into it and see if that would be an easier path, but no promises.

—Sam

---

From: "Tom Worster" <[email-masked]>
Date: Thu, 06 Dec 2018 11:12:49 -0500

Hi Sam,

Two points.

First, I also think this work is valuable but I don't understand how the 
ID is supposed to proceed. The point of bringing it to IETF is to get 
its authority behind the statement that these specific vectors test 
conformance to the PRECIS RFCs. How can the IETF give this without 
restarting the WG to push the ID forwards as a Standards Track RFC? I 
don't think the independent submission review process can.

Second, my experience is that formatting of test vectors is kinda all 
over the place in RFCs. I would not look there for guidance. Idk what 
XML format you refer to but I wouldn't want to go down any XML road as 
either producer or consumer. I suggest you devise the simplest 
machine-readable file format that you can imagine and publish it 
yourself, e.g. as a Github gist. Anyone can then easily transform that 
into any other format, for documentation, for a unit test data provider, 
or whatever. I think this serves your and our needs best even in the 
case that IETF would adopt the ID, since you can use the file to 
generate IETF-format docs.

For example, I was recently using this wonderful text file 
https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt which is 
both documentation and machine readable. Your ID is machine readable 
(what isn't these days?) but it isn't so simple to either read or write. 
In your position I'd look at UCD files for format ideas.

Tom

-- 
Tom Worster 
skype: spinitron.tom  -  857 210 3243  - [email-masked]

On 6 Dec 2018, at 10:15, Sam Whited wrote:

> I've been meaning to go back and double check that there aren't any 
> duplicate vectors covering the same edge cases and prepare an update, 
> but I haven't found a good way to format the test data still so I'd be 
> hesitant to submit it (and there's practically no information about 
> how to format things or write this awful XML format). I've also tried 
> looking into the process, and the various IETF sites are inscrutable 
> and full of contradictory information, so thanks for the link, I'll 
> look into it and see if that would be an easier path, but no promises.
>
> —Sam
>
> _______________________________________________
> precis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/precis

---

From: Sam Whited <[email-masked]>
Date: Thu, 06 Dec 2018 10:27:12 -0600

> don't think the independent submission review process can.

I don't really understand the process, but this makes sense to me and I think you're probably right. I'd be curious to hear if other IETF people who actually understand how any of this works agree or not though.

> either producer or consumer.

Sorry, I meant the rfc2xml format; I'm not really sure if there's a name for that. I was originally writing nroff format directly, but people kept telling me I should use the XML format for best results. Either way, I couldn't find a decent way to support the vectors without them wrapping awkwardly or being hard to read in the fixed-width formats that get generated. 

> I suggest you devise the simplest 

I'm actually already doing this (more or less): they're being generated from the unit tests in my Go implementation of PRECIS with a few minor changes. I should rework the tool that does the conversion at some point and publish it somewhere. However, this doesn't solve the formatting problems.

> In your position I'd look at UCD files for format ideas.

I would love it if the ID could be easily machine readable too, but I couldn't find a good way to do this. I can easily spit out a different format from the tools that convert the Go unit tests to xml2rfc format though, so I could probably publish something else like the UCD files if it would be helpful to others.

Thanks for the feedback and suggestions!

—Sam

-- 
Sam Whited
[email-masked]

---

From: "Tom Worster" <[email-masked]>
Date: Thu, 06 Dec 2018 11:56:04 -0500

On 6 Dec 2018, at 11:27, Sam Whited wrote:

> I would love it if the ID could be easily machine readable too, but I 
> couldn't find a good way to do this. I can easily spit out a different 
> format from the tools that convert the Go unit tests to xml2rfc format 
> though, so I could probably publish something else like the UCD files 
> if it would be helpful to others.

My suggestion is that until we understand how the IETF can be useful in 
supporting this work, which was my first point, don't bother with 
RFC-style formatting.

Even if the ID would in the future progress on standards track, there's 
no clear formatting requirement for test vectors and nobody can argue 
with a straight face that torturing the data into the bizarre RFC-type 
paginated ascii text format does anyone any good whatsoever. So don't 
cross that bridge before someone lets you near it?

Tom

---

From: Sam Whited <[email-masked]>
Date: Thu, 06 Dec 2018 11:06:09 -0600

> cross that bridge before someone lets you near it?

I see; temporarily here's a copy of the tests in a JSON representation: https://blog.samwhited.com/rfcs/machinetests.json

I'll throw this up somewhere more accessible when I'm next at my desk.

—Sam

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 11 Dec 2018 08:56:57 +0000

> I don't really understand the process, but this makes sense to me and I think you're probably right. I'd be curious to hear if other IETF people who actually understand how any of this works agree…

Based on my past own experience, I understand some of this, but not all 
of it. Besides Standards Track RFCs created by a WG, there's also the 
concept of a draft being sponsored by an Area Director for Standards 
Track. But this isn't used very much these days.

> Sorry, I meant the rfc2xml format; I'm not really sure if there's a name for that.

I'd think there is, it's 'rfc2xml format' or some such. There are 
actually several versions, but that's besides the point by now.

The fact that there's no consistent formatting for test vectors in RFCs 
is probably related to the fact that the IETF *as an organization* is 
not much into testing.

That doesn't mean that most of the people involved wouldn't understand 
the benefit of testing and use testing when developing or deploying 
protocols and similar stuff.

Regards,   Martin.

---

From: "Tom Worster" <[email-masked]>
Date: Tue, 11 Dec 2018 14:58:18 -0500

On 11 Dec 2018, at 3:56, Martin J. Dürst wrote:

>> I don't really understand the process, but this makes sense to me and 
>> I think you're probably right. I'd be curious to hear if other IETF 
>> people who actually understand how any of this works agree or not 
>> though.
>

That's a possibility. It's not a lot of effort to try. Appears it's a 
matter of asking an Applications and Real-Time Area Director to sponsor 
it and advance it to IETF-wide Last Call. In the worst case the answer 
is that such a draft requires WG review.

Section 6.2 https://datatracker.ietf.org/wg/precis/about/

Tom