Subject: [Edm] Implementation status listings and conformance tests

From: Paul Hoffman <[email-masked]>
Date: Thu, 3 Feb 2022 16:49:31 +0000

Greetings again. At the meeting the other day, we had a long discussion of ways that WGs might show the implementation status of standards that the WG has created. A related subject is conformance: someone decides that $ProductX conforms with $RfcY, or maybe with these parts of $RfcY, or maybe with $ProtocolZ that consists of cherry-picked parts of $RfcYZandA. There is probably general agreement that WGs should not be in the business of conformance testing.

Where this might get tricky is when an external organization creates a conformance test that folks in the WG likes. I bring this up because it is timely for the REGEXT WG: ICANN today announced a tool it created to test conformance to six RFCs from the REGEXT WG:
   https://www.icann.org/en/blogs/details/icann-launches-new-tool-to-verify-conformance-with-rdap-requirements-3-2-2022-en

In a previous life, I created interoperability tests for IPsec (as part of the VPN Consortium) that were the only lists that showed which vendors supported particular features in ESP, IKE, and IKEv2 at the time. In an earlier previous life, I organized events for the Internet Mail Consortium for the Internet mail industry where folks tested interoperability and conformance (and thus implementation) of parts of the mail specs, and those events had reports at the end showing which vendors there supported which features. (I am not affiliated with the REGEXT conformance testing listed above, and in fact only found out about it by reading today's announcement...)

Implementation status lists somewhat inherently imply that the members of the list actually conform to the RFC(s). As we discovered in the VPNC and IMC, conformance definitely did not mean interoperability; the same has been discovered throughout the IETF on nearly every protocol for which there has been concerted testing.

This is not a proposal for any solution to this tricky problem, just a pointer that it exists and that we have decades of worked-out examples. If the EDM program delves into this topic, each of us will have to choose whether we reach for the popcorn or the ibuprofen.

--Paul Hoffman

---

From: Tommy Pauly <[email-masked]>
Date: Fri, 04 Feb 2022 09:57:45 -0800

Hi Paul,

Thanks for sharing this. I think this is a useful caution against delving into managing conformance, or even trying to strictly define what guarantees come with implementation status / interoperability.

Perhaps there should be some standard written disclaimer text we can recommend for any pages that discuss implementation status to provide this warning?

—Tommy

> On Feb 3, 2022, at 8:49 AM, Paul Hoffman <[email-masked]> wrote:
> 
> --Paul Hoffman-- 
> Edm mailing list
> [email-masked]
> https://www.iab.org/mailman/listinfo/edm

---

From: "John Levine" <[email-masked]>
Date: 4 Feb 2022 22:43:22 -0500

>Perhaps there should be some standard written disclaimer text we can recommend for any pages that discuss implementation status to provide this warning?

I think the advice is to do what we did with the mail server EAI tests,
say these packages passed these tests without trying to say therefore
they conform to X.

In case there were places where formal conformance had little relationship to usability.
In particular, there were POP and IMAP servers that passed very few of the formal EAI
tests but even so were quite usable with EAI mail.

R's,
John