Subject: [newtrk] Session chair summary - implementation reports

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 10 Aug 2005 13:29:30 +0200

This is a really short summary....

- Implementation reports are very useful to the community

- The IETF has legal reasons not to be seen as "approving" implementation 
reports

- Larry's suggestion is to separate the report format from the report 
content, and to have the IETF approve the report format (what we report 
interoperability on). Interoperability reports would be filed as "filer's 
responsibility", as today.

- The issue of how to "advance" specifications is related to this.

- The room encouraged Larry to flesh out his proposal into a real proposal

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Larry Masinter <[email-masked]>
Date: Sun, 14 Aug 2005 18:04:34 -0700

> content, and to have the IETF approve the report format (what 
> we report interoperability on). Interoperability reports would be filed 
> as "filer's responsibility", as today.
> 

I have a partly filled-in outline for the "interoperability report"
proposal. 

http://larry.masinter.net/draft-masinter-ior.txt 
http://larry.masinter.net/draft-masinter-ior.html
http://larry.masinter.net/draft-masinter-ior.xml

> - The room encouraged Larry to flesh out his proposal into a 

Should this be a working-group draft, then?

Larry
-- 
http://larry.masinter.net

.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 15 Aug 2005 14:54:17 +0200

Larry, just to check your intentions, you say:

For each feature:

     * Was feature implemented?
     * If so, has feature been successfully tested as interoperable with at least one independent implementation?

This seems to allow for Feature 1 to be shown to interoperate between
Implementations A and B, and for Feature 2 to be shown to interoperate
between Implementations C and D.

In other words it does not require both features to interoperate
between the same two implementations.

Is that your intention? If so, I believe it's compatible with
RFC 2026, but I know that some people think it's too weak
a requirement.

     Brian

.

---

From: Larry Masinter <[email-masked]>
Date: Mon, 15 Aug 2005 09:11:41 -0700

>      * If so, has feature been successfully tested as 
> interoperable with at least one independent implementation?
> 

For protocols with options, it is rare to find protocols that
implement all options. I think the judgment about adequacy may
hinge on whether the "features" are required or optional.

I think the way to address this, with PIRs, is to allow the PIRs to
contain enough information such that you could decide for yourself;
the PFS should identify whether features are mandatory or optional;
each implementation would have a unique ID, and a feature report
in a PIR can identify which other implementations it interoperated
with.

I might allow some flexibility between implementation and
version. The "unique ID" would likely be one URI
to identify the implementation, and another URI to identify the
exact version.

I've attempted to update the drafts on my web site to
implement these ideas, let me know what you think.

Larry

.

---

From: Douglas Otis <[email-masked]>
Date: Mon, 15 Aug 2005 09:37:15 -0700

> http://larry.masinter.net/draft-masinter-ior.txt

12.  Comparison with ISD and SRD...

Perhaps something that should be considered is that unlike the ISD  
concept, the SRD would not replicate interoperability information.   
However, the SRD approach should facilitate maintenance of  
interoperability information created asynchronously and linked to the  
SRD document.  The SRD draft would reduce efforts to list related  
RFCs, as that would be a task already completed by the SRD.  The SRD  
provides a stable reference for interoperability information.  A  
divide and conquer approach so to speak.

-Doug
.

---

From: Larry Masinter <[email-masked]>
Date: Mon, 15 Aug 2005 10:53:28 -0700

Looking at http://www.imc.org/rfcs.html, I think the biggest
hole in all of the current proposals (SRDs, ISDs, PFSs, etc.)
is dealing with nesting, references, and related items.

At some level, MIME and message format are orthogonal
to message transport, but they do interrelate. I wouldn't
expect one metadocument (SRD, ISD, PFS) to cover everything
in http://www.imc.org/rfcs.html. How to break this up?

Perhaps:

- host-to-host mail transfer
- client-to-host communication
- message format and encoding

as the basic categories; include security mechanisms
and internationalization as part of those (multiple
references to the same RFCs or feature sets?)

But where would Calendaring and Scheduling fit?
Or gateways to Non-Internet Mail? (I'm a little puzzled
about how to test independent interoperability of
such gateways).

Would you imagine having one SRD/ISD/PFS reference
another, or a part of another?

Larry

.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Aug 2005 11:12:54 +0200

> another, or a part of another?

I think this is where the nub of the normative vs informative
argument lies. If the documents are informative, the answer
to this question simply doesn't matter - we do whatever is
pragmatic and convenient. If they are normative, we will have
exactly the same problem for normative reference dependencies
that we have today in standards track RFCs. I'm pessimistic
that we can avoid circular dependencies in general.

     Brian

.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Aug 2005 11:25:21 +0200

Larry,

Two points, a generic one first and a specific one below.

I think we are in quite urgent need of de facto criteria
for implementation reports, even under the existing RFC 2026
practice. I haven't spent much time in the past at
http://www.ietf.org/IESG/implementation.html, but recently
I've been looking at it quite a bit and I am disturbed
by what I find - enormous variation in quality and quantity
and even some reports with no date!

So I have something of an immediate request here - can
we have a draft that defines minimum criteria for an
implementation report, which the IESG could consider for
use in the short term as a guideline? It's very hard to
reject an implementation report for inadequacy when we have
no visible guidelines beynd 2026.

Below...

> hinge on whether the "features" are required or optional.

Why? We don't certify implementations in the IETF. The idea
is to validate the implementability of the spec, and I don't see
why that makes MUST features any different from SHOULD or MAY
features. And if an implementation breaks a MUST NOT rule, that
says nothing bad about the spec.

> contain enough information such that you could decide for yourself;

Not if they are being used (as today) in the IESG's judgement call
about the PS->DS transition. The only question is whether the spec
allows interoperability; defects in implementations are not our
problem.

    Brian

.

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 16 Aug 2005 07:18:51 -0400

Hi All,

>no visible guidelines beynd 2026.

I agree with Brian that this would be very valuable.  Today, 
implementation reports vary from very detailed reports on the 
implementation of individual features to much more generic 
assertions.  Many of them do not contain enough information to verify 
that the specifications have been fully implemented and/or that the 
resulting implementations have been fully tested for interoperability.

It would also be good if this report could explain how 
interoperability should be determined for one-sided specifications, 
such as client-side address selection algorithms or MIB 
instrumentation.

Margaret

.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 16 Aug 2005 12:05:00 +0200

> that we can avoid circular dependencies in general.

I think that there's very little if any formal difference between three 
documents with a circular dependency between them and one big document.

But in many cases the three documents will be easier to read and easier to 
maintain.

I think being pragmatic is the right answer here, too.

               Harald

.

---

From: Larry Masinter <[email-masked]>
Date: Tue, 16 Aug 2005 08:50:39 -0700

> no visible guidelines beynd 2026.

I think this is an important point. If we're concerned
about "standards track", and, in particular, that not
many documents make the transition from PS -> DS, then
being clear about the elements of that trek seems important.

However, I think there's some disagreement about what
the minimum criteria might be.   Which of the following
are required?

* date(s) and author(s) of report (parts)
  (assertions of validity, if author disclaims knowing)
* identity of implementations (to what detail)
* assertions about independence of implementations (and
  licensing process for IPR)
* list of features tested
* success of interoperability for each feature
  identifying the implementations, or counting them
   or asserting that there are many

What else should be required?

How about if I update my draft to separate out the description
of "content of an implementation report" from the "format for
representing that content". 

Larry

.

---

From: John C Klensin <[email-masked]>
Date: Tue, 16 Aug 2005 13:50:02 -0400

--On Tuesday, 16 August, 2005 11:12 +0200 Brian E Carpenter
<[email-masked]> wrote:

>...

Brian, I either don't understand what you are saying or don't
believe it.   As RFC 3967 more or less proves, the
"downreferencing"  and normative reference dependencies rules
are whatever we decide to make them.  They are not dictated by
some Higher Authority that cannot be challenged.

The intent behind the ISD model was to have normative document
that could say things like

"   RFC 821, RFC 974, the SMTP part of 1123, and RFC 1869 are
required to support contemporary electronic mail transport
"   RFC 2821 incorporates the material in them and is expect to
evolve into a standard that replaces them.  In the interim, it
may be easier to follow although the combined earlier group is
formally authoritative.
   [...]
"   If you are trying to do a complete implementation of
Internet mail, you also need to look at ISD XXXX, which covers
header and message body formats, and RFCs NNNN, MMMM, ..."

Now, the current "enough details to confuse people into reading
far too much into the examples and text" draft doesn't say that
explicitly but rather implies a list of RFCs and descriptive
paragraphs whose format (less the descriptive paragraphs) is a
bit closer to what I think Doug is proposing.  But either or
both approaches are feasible.  

There has never been an assumption that a given RFC could not be
referenced from/ required by more than one ISD, so the need to
argue about grouping boundaries is a good deal "softer" than,
e.g., with the STD numbers.

The paragraphs above could be labeled "normative/standard,
normative/guidance, and informative/advisory" if such labels
were judged helpful.  The presence of the latter one would not
compromise the normative nature of the ISD, although it might be
useful to be a bit more precise about the relationship between
the first two paragraphs and associated expectations if people
were expected to do procurements based on "systems conforming to
ISD XXXX" statements.

The point, IMO, is that the details of what must be in an ISD,
or might be in an ISD, or would be prohibited from being in an
ISD, have never been established or specified as firmly as seems
to be generally believed.  Personally, I've never felt that the
same rules need apply to all ISDs.  There is a large role for
case-by-case discretion about anything that goes beyond what can
be automagically generated.  I'd expect the decisions to be
based on a combination of perceived need (some standards or
collections of them are more complicated than others and need
more explanation) and available resources (there are probably
only a dozen or so people active in the IETF today who could
write a comprehensive overview of the email standards and, if
none of us could be recruited to do it,... less would turn out
to be more, i.e., being less comprehensive and specific might be
better than a high risk of getting it wrong).

       john

.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 17 Aug 2005 15:27:38 +0200

John C Klensin wrote:
> believe it.

There was discussion of ISDs citing other ISDs normatively.
All I mean is that this can introduce dependency cycles - whereas if
ISDs can only cite RFCs, the problem is no worse than today.

>  As RFC 3967 more or less proves, the

I'd hope that's true, but whenever someone points out that the
3 stage model with a "no downref" rule causes dependency cycles,
the value of the three stages is invoked a few messages later
in the thread. The only way out is to make RFC 3967 variance
routine - which defeats the purpose of having 3 stages at all.

     Brian

.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 18 Aug 2005 11:20:29 +0200

> routine - which defeats the purpose of having 3 stages at all.

Ways out (in the sense of "do something that makes sense") include:

- Routine application of 3967 (which I think may dilute, but not defeat, 
the purpose of multistage)
- Forcibly making documents with normative dependency cycles part of the 
same ISD
- Making a meta-ISD that refers to all of the ISDs in the cycle, for the 
perusal of anyone who wants to "see the whole picture"

Cycles are a sign of closely linked specifications. If we make that close 
linkage visible, without shooting ourselves in the procedural foot while 
doing so, I think we benefit the community.

.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 18 Aug 2005 18:27:07 -0700

> ISDs can only cite RFCs, the problem is no worse than today.

This problem of dependencies was considered within the SRD draft, but  
not explained.  I think I talked about it on the list.  The example  
shows the referenced SRD document includes a serial number.  This  
represents a snapshot of RFCs in effect.  When an SRD is updated,  
there should be an effort to advance the serial number to current  
when possible.  It would also mean prior SRD documents will remain  
significant for situations where the prior "snapshot" remains needed.

The place where this will likely be most useful would be with general  
definitions.  srd-ietf-def.12 could be the latest set of  
definitions.  As any underlying RFC is changes, it would require a  
new serial number generated to describe such a change.  As long as  
these serial numbers are included within the "set", then essentially  
this captures the related RFCs without danger of this set changing  
from future updates.  A lazy process may simply exclude the serial  
number within the WIP document, but when the document is accepted,  
the current serial number should be captured.

-Doug 
  
.

---

From: Sam Hartman <[email-masked]>
Date: Tue, 23 Aug 2005 19:13:13 -0400

>>>>> "Brian" == Brian E Carpenter <[email-masked]> writes:

    Brian> Larry, just to check your intentions, you say: For each
    Brian> feature:

    Brian>      * Was feature implemented?  * If so, has feature been
    Brian> successfully tested as interoperable with at least one
    Brian> independent implementation?

    Brian> This seems to allow for Feature 1 to be shown to
    Brian> interoperate between Implementations A and B, and for
    Brian> Feature 2 to be shown to interoperate between
    Brian> Implementations C and D.

    Brian> In other words it does not require both features to
    Brian> interoperate between the same two implementations.

I thought Bruce was complaining about this with regard to the submit
ds ballot and we decided that it was not a requirement?

.