Subject: RFC conformance:

From: "Liao Wei" <[email-masked]>
Date: Fri, 19 Apr 2002 16:00:32 -0400

Each IEEE standards give out  PICS proforma for Protocol  Implementation
Conformance
Statement. The supplier of an implementation that is claimed to conform to
the standard should
complet the PICS document and provide the information necessary to identify
the supplier and
implementaion.

Is there any this kind of document for RFCs. If an internet device supplier
claim their
implementaion conform to RFCXXXX. What should subscriber do to verify the
conformance.

---

From: [email-masked]
Date: Fri, 19 Apr 2002 17:11:01 -0400

> implementaion conform to RFCXXXX. What should subscriber do to verify the

First off, decide whether "RFC conformance" or "interoperability" is what
you're actually looking for.

Second, think about whether a conformance test will actually find the broken
corner cases that cause problems in real life.

Third, ask yourself which you'd rather have - a piece of paper saying it
follows the RFC, or a piece of paper saying that they'll fix problems if
you find that it doesn't work.

As part of a project I'm involved in, I recently did a bit of basic design
work for something that would be fairly close to conformance testing (you
old-timers that were around for the UL discussion a while ago remember that ;)
and convinced myself that although for most protocols it was quite easy
to verify that a given implementation follows the protocol when everything
works well, that really doesn't tell you much - there are very few vendors who
can get away with shipping total manure-ware that's totally standard 
non-compliant in the trivial cases.

The problem is that it's *very* *hard* to come up with good testing for the error
cases - which is where all the real-world uglyness comes from.  I started
thinking about doing it for SMTP (which I think I know something about), and
realized that I didn't know how to write tests for the sort of things that I've
seen "in the wild".

For instance:  I've seen hosts that you can communicate with just fine
when SMTP pipelining is off, but which wedge up when you turn it on.
Reason?  Some router in between is tossing ICMP Frag Needed packets with
an RFC1918 address, which our border router then tosses, breaking Path
MTU Discovery - and pipelining allows enough RCPT TO: in one packet to
tickle the problem..  (Figuring out why this didn't bite during the DATA
phase is left as an exercise for the reader ;)  The worst part is that
both ends did exactly what they were supposed to in the face of
what looked like network outages - abort, requeue, and retry.

I won't get into the theory-versus-practice issues of %-hacks, !-paths,
@host: source routes, and what happens when mixing them other than to note
that I once saw the inventor of %-hacking describe it as "maybe the evil that
men do does live on..." ;)

Bottom line - ask the vendor for sites that have installed it before, and
see if they've gotten it to work with other vendor's stuff.  That's the way
we've always done conformance testing.  If the vendor's stuff doesn't play
nice, ask the vendor to fix it.  If they won't, and they're a big enough
vendor that they care about their public image, there's any number of
journalists of varying ilk all looking for "Vendor XYZ problems with..."
stories. ;)

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 19 Apr 2002 20:09:26 -0500

> what looked like network outages - abort, requeue, and retry.

Actually, that's a TCP bug, not an SMTP bug. The transmitting side should
be doing blackhole detection to catch this. Lots of stacks don't do this,
even though it isn't exactly rocket-science. Small segments get ACKs, big
segments don't. Duh, try smaller segments?

Anyway, it still proves the point. You can spend hundreds of man-hours
devising a series of tests, and then find out that some seemingly
unrelated thing is screwing the results.

Not to say that it can't be done, but that it's very expensive, and it
requires a holistic approach that is beyond the norm.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: RJ Atkinson <[email-masked]>
Date: Sat, 20 Apr 2002 09:56:28 -0400

On Friday, April 19, 2002, at 04:00 , Liao Wei wrote:
> Conformance Statement. The supplier of an implementation that is claimed
> to conform to the standard should complete the PICS document and provide
> the information necessary to identify the supplier and implementaion.
>
> Is there any this kind of document for RFCs.

No.  And my experience has been that PICS documents (from anyplace, not
just IEEE) have not been a good indicator of whether the implementation
really conformed/complied with the spec I was referencing.

> If an internet device supplier claim their implementaion conform to 
> RFCXXXX.
> What should subscriber do to verify the conformance.

	That is left entirely up to the customer.  IETF does not specify 
any such
conformance tests to run.  When I've been a customer, I have invented my 
own tests
for things I cared about and tested the equipment myself.

Ran
[email-masked]