Subject: I'm struggling with 2219 language again

From: Dean Willis <[email-masked]>
Date: Thu, 3 Jan 2013 23:15:20 -0600

I've always held to the idea that RFC 2119 language is for defining levels of compliance to requirements, and is best used very sparingly (as recommended in RFC 2119 itself). To me, RFC 2119 language doesn't make behavior normative -- rather, it describes the implications of doing something different than the defined behavior, from "will break the protocol if you change it" to "we have reason to think that there might be a reason we don't want to describe here that might influence you not to do this" to "here are some reasons that would cause you to do something different" and on to "doing something different might offend the sensibilities of the protocol author, but probably won't hurt anything else."

But I'm ghost-editing a document right now whose gen-art review suggested replacing the vast majority of "is" "does" and "are" prose with MUST. The comments seem to indicate that protocol-defining text not using RFC 2119 language (specifically MUST) is "not normative".

This makes me cringe. But my co-editor likes it a lot. And I see smart people like Ole also echoing the though that RFC 2119 language is what makes text normative.

For example, the protocol under discussion uses TLS or DTLS for a plethora of security reasons. So, every time the draft discusses sending a response to a request, we would say "the node MUST send a response, and this response MUST be constructed by (insert some concatenation procedure here) and MUST be transmitted using TLS or DTLS.

Or, a more specific example:

For the text:

In order to originate a message to a given Node-ID or
Resource-ID, a node constructs an appropriate destination list.

The Gen-ART comment here is:
-First sentence: "a node constructs" -> "a node MUST construct"

We'll literally end up with hundreds of RFC 2119 invocations (mostly MUST) in a protocol specification.

Is this a good or bad thing? My co-editor and I disagree -- he likes formalization of the description language, and I like the English prose. But it raises process questions for the IETF as a whole: 

Are we deliberately evolving our language to use RFC 2119 terms as the principle verbs  of a formal specification language?

Either way, I'd like to see some consensus. Because my head is throbbing and I want to know if it MUST hurt, SHOULD hurst, or just hurts. But I MUST proceed in accordance with consensus, because to do otherwise would undermine the clarity of our entire specification family.

--
Dean

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 04 Jan 2013 08:03:01 +0000

On 04/01/2013 05:15, Dean Willis wrote:
...
> Either way, I'd like to see some consensus. Because my head is throbbing and I want to know if it MUST hurt, SHOULD hurst, or just hurts. But I MUST proceed in accordance with consensus, because to…

This Gen-ART reviewer believes that words like "must" have well defined meanings
in the English language, so shouting is not needed at every use. There are
standards track documents that don't use RFC 2119 at all, and I am not only
referring to RFC 791.

I think the upper case keywords should be used only when necessary to clarify
points of potential non-interoperability or insecurity. I'm quite sure that
I've broken that recommendation quite often, and it will always remain
a judgment call. However, inserting a MUST in every sentence that describes
behaviour is surely going too far. I guess the test is whether a reasonably
careful reader might interpret a sentence incorrectly while writing code;
and if so, would a normative keyword help?

     Brian

---

From: Dave Cridland <[email-masked]>
Date: Fri, 4 Jan 2013 09:57:50 +0000

> standards track documents that don't use RFC 2119 at all, and I am not only

IMAP's CONDSTORE is a relatively recent example (authored by Pete
Resnick). It got all the way to WGLC before anyone noticed, as I
recall. (Possibly even through, I can't recall).

> and if so, would a normative keyword help?

RFC 2119 language is not a stick to beat an implementor with, but a
signpost to highlight or clarify important cases. If the signpost is
used too often, the temptation is to mentally downgrade the language
when reading. It's often not needed, and only serves to make the
language look "more RFC", or just saves an author from considering how
to best describe the protocol. Neither is good.

But in any case, a Gen Art reviewers comments hold no more weight than
any others, and if your personal style, like mine, leans toward
minimizing the use of RFC 2119 language and putting the effort into
general clarity and readability, then I think you should ignore the
proposed solution and treat the comments as a suggestion that the
areas might benefit from more clarity of intent.

Dave.

---

From: Lou Berger <[email-masked]>
Date: Fri, 04 Jan 2013 08:23:23 -0500

On 1/4/2013 12:15 AM, Dean Willis wrote:
...
> Are we deliberately evolving our language to use RFC 2119 terms as
> the principle verbs  of a formal specification language?
...

My view on this has evolved over time.  I used to follow the practice of
using 2219 language only for emphasis.  Over time, primarily motivated
by reviewers comments and reader questions, I've migrated to the
position that 2119 language should be used whenever and wherever a point
of conformance is being made.  While this may be a bit of an extreme
position, it ensures that authors, reviewers, readers, implementors,
etc. are in sync as to what is expected from an interoperable
implementation that conforms to a standard.  I think the importance of
such unambiguity has increased over time as the number of implementors
and non-native English speakers in our community have increased.

I also think it's important to follow section 6 of 2119, i.e., if it's
not a point of interoperability or harmful behavior, there's no need to
use 2119 conformance language.

So, my view is now:

a) lower case usage of 2119 key words *in RFCs* means the normal English
meaning of such words, but does not place any requirement on
implementations, i.e., is purely informative text.

b) upper case usage of 2119 key words *in RFCs*, as stated in [RFC2119],
places "requirements in the specification", i.e., is conformance
language with which an implementation must follow to ensure
interoperability (or harm).  (And does not = shouting as would be the
case in other contexts).

I take this view when writing and reviewing PS drafts...

Lou

---

From: Bob Braden <[email-masked]>
Date: Fri, 04 Jan 2013 09:56:40 -0800

I believe that Brian's interpretation is exactly right. At least, it 
conforms to the Original Intent of the
applicability terms MUST, MAY, and SHOULD as defined in RFC 1122. And I 
sympathize with
Dean Willis whose head hurts; as one-time RFC Editor I was often 
confronted with wildly
inconsistent use of the applicability words. It often seemed that 
authors sprinkled them in
randomly, just enough seasoning to make a normative stew.

Bob Braden

---

From: Scott Brim <[email-masked]>
Date: Fri, 4 Jan 2013 13:05:14 -0500

It's a communication problem.  If you want your audience to understand
exactly what you're saying, and implement along very specific lines, you
need to tell them in a way they understand.  Personally I prefer a
quieter approach, but I've been told that these days one MUST use MUST
or implementors just won't get it.  "Huh, that's a requirement?  But you
didn't say MUST."  I suggest turning this thread into a survey, and
finding out how people who actually write code look for in order to know
what's required.

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 04 Jan 2013 11:15:00 -0700

Wonderful perennial topic. :)

As I always say when this comes up, when writing drafts I've settled
on using the 2119 keywords only in their uppercase form, and otherwise
using "need to", "ought to", "might" (etc.) to avoid all possible
confusion. Sure, it's a bit stilted, but we're not writing gorgeous
prose here, we're writing technical specifications that need to be
completely clear.

Peter

- --
Peter Saint-Andre
https://stpeter.im/

---

From: Hector Santos <[email-masked]>
Date: Fri, 04 Jan 2013 13:37:21 -0500

+1.

I think it is important that we have communications tools for 
documenting strong minimum protocol requirements and we only have 
RFC2119 to make that possible.

Yet, we need to be careful where the lack of RFC2119 upper case 
wordings can be used to leverage an argument for relaxation of 
existing protocols where perhaps only lower case semantics were done.

IMO, where there are multiple protocol state choices or paths and 
where common sense engineering and security functionality is clear, 
upper vs lower should not matter.

I will like to use an example protocol logic I am currently concern 
with that could be altered due to 2119 related debates:

   For protocol state condition A, you may do action B or C.
   If action B is taken, you MUST do D.

So action B has 2119 language to do D.  Part of the 2219 debate is 
that doing action B or C are both optional.

C is not defined, although it could be said it was implied via other 
indirect protocol features:

   You SHOULD do E.

and in a protocol change proposal, it is now:

   You SHOULD do E1 instead of E.

Well, to do E or E1, you have to do C, not B.

Now the history of this is that original drafts/specs (ones some folks 
want us to ignore) did have 2119 language on whether you MAY do B or 
C, and in fact in an extended augmented RFC protocol, it only allowed 
for a MUST action B for protocol condition A.  C was not an option (or 
at least it wasn't stated) in the extended RFC technology.

The problem was the lack of an upper case MAY has provided the notion 
that none of this is normative thus making the action C more possible 
and action B less attractive.

By making C more possible, this makes E more plausible and now that E1 
is a new recommendation and preferred feature than E,  the idea of 
doing action B is now watered down.

The issue is sensitive and honestly, it got to a point where one says 
  "Who cares!?" If they want to watered down B, so be it.  You don't 
need to support it protocol changes nor the newer push to the E1 
recommendation.

The problem for me?

Is why is the security aspects of this do not make it obvious of what 
needs to be done?  We got into anal debate of 2119 language and I even 
recall a few years ago where another debate of the definition of 
SHOULD got a number of people labeled as 2219 illiterates by an AD.

In the example I cited, politics comes into play because you have two 
industry mindsets under protocol condition A:

   1) Those that believe in action B as the more secured action, and
   2) Those that DO NOT believe in action B and prefer C.

There is where a person like myself is left with the idea of waiting 
for WGLC to appeal changes to the protocol that will watered down 
action B.

This is all due because of the lack of 2119 language that many believe 
were naturally in place with just using lower case wordings.

No upper case, therefore action B is weak and that was unfair to 
current implementations that only do action B as its the most secured 
implementation (and less costly) action to protect users.   By 
allowing RFC2119 or stated arguments the protocol lacked 2119 
language, weakens the protocol and makes it more complex for 
implementations to consider action C which can be more harmful if 
certain forms of C making it functionality equivalent (security wise) 
to action B is not done (this is the added complexity).

I'm sure this is a dime a dozen story, but to me it is common to see 
WG chairs, AD and other key cogs throw the proverbial 2119 book at 
developers to help push specs.   IMO, we need it but we also need some 
common sense engineering and consideration to be done when we review 
current docs.  Ignoring existing implementations SHOULD not be one of 
them.

-- 
HLS

---

From: Richard Barnes <[email-masked]>
Date: Fri, 4 Jan 2013 14:04:51 -0500

Anecdotal data point number N+1...

As an occasional implementor of IETF specs, I have to say it's much easier to check my conformance if I can just grep for "MUST" and "SHOULD".  It's also easy for developers to get in the bad habit of ONLY doing those things that are clearly marked in that way.  So ISTM that if you're not tagging things you want done with RFC 2119 language, then you're risking people not implementing them.

On Jan 4, 2013, at 1:15 PM, Peter Saint-Andre <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iEYEARECAAYFAlDnHCQACgkQNL8k5A2w/vxKmwCfXKjDtMqQiPp4a0udOB8Q9IbA
> q9QAoNiXj2r/q4yRLp0B/13m6Xxg5YN4
> =3PER
> -----END PGP SIGNATURE-----

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 4 Jan 2013 19:18:30 -0000

Lou's view matches how I write and review documents.
I would add that there is sometimes value in using 2119-style language in
requirements documents ("The protocol solution MUST enable transmission of
data...") although, in my opinion, this requires a tweak to the normal2119
boilerplate.

Adrian

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Lou
> Berger
> Sent: 04 January 2013 13:23
> To: Dean Willis
> Cc: [email-masked]
> Subject: Re: I'm struggling with 2219 language again
> 
> 
>

---

From: Ben Campbell <[email-masked]>
Date: Fri, 4 Jan 2013 15:58:11 -0600

I generally take (what I infer to be) Richard's view on the matter.  If not doing something will break interoperability or security, then make it normative. (I realize that's a gross oversimplification).

But that still doesn't mean you have to have a MUST for every step an implementation has to take. To take Dean's original example, I think it makes sense to say the implementation "... MUST send a response according to the following procedures..." then describe the procedure without peppering it with 2119 language,  except for special cases when needed for emphasis or clarity. This seems to cover the risk of people not implementing stuff, but still avoids an explosion of MUSTs (and the resulting requirements matrix).

Sticking with Dean's example, the use of TLS might qualify as one of the special cases needing additional normative emphasis, but you can always say something like "... MUST send all message over TLS" once, rather than restate it for every message.

---

From: Thomas Narten <[email-masked]>
Date: Fri, 04 Jan 2013 17:23:42 -0500

+1 to Brian and others saying upper case should be used sparingly, and
only where it really matters. If even then.

The notion (that some have) that MUST means you have to do something
to be compliant and that a "must" (lower case) is optional is just
nuts.

If the ARP spec were to say, "upon receipt of an ARP request, the
recipient sends back an ARP response," does the lack of a MUST there
mean the response is optional? Surely not. And if we make it only a
SHOULD (e.g., to allow rate limiting of responses - a very reasonable
thing to do), does lack of MUST now make the feature optional from a
compliance/interoperability perspective?

The idea that upper case language can be used to identify all the
required parts of a specificition from a
compliance/conformance/interoperability perspective is just
wrong. This has never been the case (and would be exceeding painful to
do), though (again) some people seem to think this would be useful and
thus like lots of upper case language.

Where you want to use MUST is where an implementation might be tempted
to take a short cut -- to the detriment of the Internet -- but could
do so without actually breaking interoperability. A good example is
with retransmissions and exponential backoff. You can implement those
incorrectly (or not at all), and still get "interoperability". I.e.,
two machines can talk to each other. Maybe you don't get "good"
intereoperability and maybe not great performance under some
conditions, but you can still build an interoperabile implementation.

IMO, too many specs seriously overuse/misuse 2119 language, to the
detriment of readability, common sense, and reserving the terms to
bring attention to those cases where it really is important to
highlight an important point that may not be obvious to a casual
reader/implementor.

Thomas

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 5 Jan 2013 09:48:59 +1100

+1; this is what we're doing in HTTPbis.

--
Mark Nottingham   http://www.mnot.net/

---

From: [email-masked]
Date: Fri, 04 Jan 2013 17:05:16 -0800 (PST)

> only where it really matters. If even then.

That's the entire point: The terms provide additional information as to 
what the authors consider the important points of compliance to be.

> to be compliant and that a "must" (lower case) is optional is just

In some ways I find the use of SHOULD and SHOULD NOT be to be more useful
than MUST and MUST NOT. MUST and MUST NOT are usually obvious. SHOULD and
SHOULD NOT are things on the boundary, and how boundary cases are handled
is often what separated a good implementation from a mediocre or even poor
one.

> thus like lots of upper case language.

At most it provides the basis for a compliance checklist. But such checklists
never cover all the points involved in compliance. Heck, most specifications in
toto don't do that. Some amount of common sense is always required.

> highlight an important point that may not be obvious to a casual

Sadly true.

				Ned

---

From: Hector Santos <[email-masked]>
Date: Fri, 04 Jan 2013 22:24:50 -0500

Scott Brim wrote:
> need to tell them in a way they understand.  

+1

> Personally I prefer a quieter approach, but I've been told that 
> these days one MUST use MUST or implementors just won't get it.  
> "Huh, that's a requirement?  But you didn't say MUST."  

I believe in the technical writing style of "Being specific is Terrific!"

>I suggest
> turning this thread into a survey, and

+1

We have implemented numerous protocols since the 80s. I have a 
specific method of approaching a new protocol implementation which 
allows for fastest implementation, testing proof of concept and above 
all minimum cost.  Why bother with the costly complexities of 
implementing SHOULDs and MAYs, if the minimum is not something you 
want in the end anyway?

A good data point is that for IP/Legal reasons, we do not use other 
people's code if we can help it and in the early days, open source was 
not as wide spread or even acceptable at the corporate level. In other 
words, it was all done in-house, purchased or nothing.  I also believe 
using other people's code has a high cost as well since you don't have 
an in-house expert understanding the inner workings of the externally 
developed software.

o Step 1 for Protocol Implementation:

Look for all the MUST protocol features.  This includes the explicit 
ones and watchful of semantics where its obviously required or things 
will break, perhaps it fell thru the crack.

An important consideration for a MUST is that operators are not given 
the opportunity to disable these protocol required features. So from a 
coding standpoint, this is one area you don't have to worry about 
designing configuration tools, the UI, nor including operation 
guidelines and documentation for these inherent protocol required 
features.

This is the minimum coding framework to allow for all inteop testing 
with other software and systems.

The better RFC spec is the one that has documented a checklist, a 
minimum requirement summary table, etc. Good example is RFC 1113 for 
the various internet hosting protocols. I considered RFC 1123 the "bible!"

Technical writing tip: Please stay away from verbosity especially of 
subjective concepts and please stop writing as if everyone is stupid.

     I always viewed the IETF RFC format as a blend of two steps
     of the SE process - functional and technical specifications.
     Functional specs tell us what we want and technical specs
     tell us how we do it.  So unless a specific functional requirements
     RFC was written, maybe some verbosity is needed but it should
     be minimized.

Generally, depending on the protocol, we can release code just on 
using MUST requirements - the bottom line framework for client/server 
communications.  Only when this is completely successfully, can your 
implementation consider moving on at extending the protocol 
implementation with additional SHOULD, MAY features and its optional 
complexities.

o Step 2

Look for the SHOULDs.  This is the candies of the protocol.  If the 
SHOULD is really simple to implement, it can be lumped in with step 1.

I know many believe a SHOULD are really a MUST as an alternative 
method perhaps - different version of MUST to be done nonetheless.

However, I believe these folks play down an important consideration 
for implementing SHOULD based protocol features:

    Developers need to offer these as options to deployment operators.

In other words, if the operator can not turn it off then a SHOULD was 
incorrectly used for a MUST which is required with no operator option 
to disable.

o Step 3

Look for the MAYs.   Very similar to SHOULD, a good way to consider a 
SHOULD is as a default enabled (ON out of the box) option and a MAY as 
a default disabled (OFF out of the box) option.

Summary:

   MUST   - required, no operator option to disabled. Of course,
            its possible to have a hidden, undocumented switch
            for questionable stuff.

   SHOULD - good idea, recommended. if implemented, enabled it
            out of the box.

   MAY    - similar to SHOULD, does not have to be enabled out
            of box.

In both cases for SHOULD and MAY, the operator can turn these protocol 
features off/on. For a MUST, the operator can not turn the MUST 
feature. These SHOULD/MAY features are documented for operators and 
support.

One last thing, I believe in a concept I call CoComp - Cooperative 
Competition, where all competitive implementators, including the 
protocol technology leader all share a common framework for a minimum 
protocol generic to all parties and the internet community. It is 
least required to solve the problem or provide a communication avenue. 
  All else, the SHOULDs, the MAYs, is added value for competing 
implementators.  It generally is what differentiate the various 
implementators software.

I personally believe it is doable to write a new RFC that describe a 
guideline for protocol development that will minimize conflicts at 
many levels.  Of course, a major part of that is good technical 
writing skills in principle and the ability to extract and describe 
what the protocol framework is, which brings it all back to the 
original issue using the proper communications verbiage to describe a 
protocol.

-- 
HLS

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Sat, 05 Jan 2013 11:14:03 -0800

I read the responses so far, and what can be said today is that there is 2
philosophies, with supporters in both camps.  The goal of the IETF is to make
the Internet work better, and I do believe that RFC 2119 is one of the
fundamental tool to reach this goal, but having two ways to use it does not
help this goal.

One way to find out would be to measure which philosophy results in the best
implementations.  Let's say that we can associate with each Standard Track RFC
one of these two philosophy.  If we had statistics on implementations then it
would be a simple matter of counting which one produce the less
interoperability problems, security issues and congestion problems (is there
other criteria?).  But as far as I know, there is no such data available -
maybe we should start collecting these, but that does not help for our current
problem.

Another way to look at it would be to run the following experiment:

1. Someone design a new protocol, something simple but not obvious, and write
in a formal language and keep it secret.

2. The same protocol is rewritten in RFC language but in two different
variants according to the two philosophies.  These also are kept secret.

3. The two variants are distributed randomly to a set of volunteer
implementers, who all implement the spec they received the best they can and
submit the result back, keeping their implementation secret.

4.  A test harness is written from the formal description, and all
implementations are run against each other, collecting stats related to the
criteria listed above (some criterion may be tricky to automatically assess,
we'll see).

5. Results are published, together with the protocol in formal form, the
specs, the results and the recommendation for one or the other philosophy.

That could be an interesting research project, and could even find some
funding from interested parties.

On 01/03/2013 09:15 PM, Dean Willis wrote:
> 
> I've always held to the idea that RFC 2119 language is for defining levels
> of compliance to requirements, and is best used very sparingly (as
> recommended in RFC 2119 itself). To me, RFC 2119 language doesn't make
> behavior normative -- rather, it describes the implications of doing
> something different than the defined behavior, from "will break the
> protocol if you change it" to "we have reason to think that there might be
> a reason we don't want to describe here that might influence you not to do
> this" to "here are some reasons that would cause you to do something
> different" and on to "doing something different might offend the
> sensibilities of the protocol author, but probably won't hurt anything
> else."
> 
> But I'm ghost-editing a document right now whose gen-art review suggested 
> replacing the vast majority of "is" "does" and "are" prose with MUST. The 
> comments seem to indicate that protocol-defining text not using RFC 2119 
> language (specifically MUST) is "not normative".
> 
> This makes me cringe. But my co-editor likes it a lot. And I see smart
> people like Ole also echoing the though that RFC 2119 language is what
> makes text normative.
> 
> For example, the protocol under discussion uses TLS or DTLS for a plethora
> of security reasons. So, every time the draft discusses sending a response
> to a request, we would say "the node MUST send a response, and this
> response MUST be constructed by (insert some concatenation procedure here)
> and MUST be transmitted using TLS or DTLS.
> 
> In order to originate a message to a given Node-ID or Resource-ID, a node 
> constructs an appropriate destination list.
> 
> 
> The Gen-ART comment here is: -First sentence: "a node constructs" -> "a
> node MUST construct"
> 
> 
> Is this a good or bad thing? My co-editor and I disagree -- he likes 
> formalization of the description language, and I like the English prose.
> But it raises process questions for the IETF as a whole:
> 
> Are we deliberately evolving our language to use RFC 2119 terms as the 
> principle verbs  of a formal specification language?
> 
> Either way, I'd like to see some consensus. Because my head is throbbing
> and I want to know if it MUST hurt, SHOULD hurst, or just hurts. But I
> MUST proceed in accordance with consensus, because to do otherwise would
> undermine the clarity of our entire specification family.
> 

- -- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: http://blog.marc.petit-huguenin.org
Profile: http://www.linkedin.com/in/petithug

---

From: Pete Resnick <[email-masked]>
Date: Mon, 7 Jan 2013 09:34:20 -0600

Dean, I am struggling constantly with 2119 as an AD, because if I take 
the letter (and the spirit) of 2119 at face value, a lot of people are 
doing this wrong. And 2119 is a BCP; it's one of our process documents. 
So I'd like this to be cleared up as much as you. I think there is 
active harm in the misuse we are seeing.

To Ned's points:

> what the authors consider the important points of compliance to be.

We will like end up in violent agreement, but I think the above 
statement is incorrect. Nowhere in 2119 will you find the words 
"conform" or conformance" or "comply" or "compliance", and I think 
there's a reason for that: We long ago found that we did not really care 
about conformance or compliance in the IETF. What we cared about was 
interoperability of independently developed implementations, because 
independently developing implementations that interoperate with other 
folks is what makes the Internet robust. Importantly, we specifically 
did not want to dictate how you write your code or tell you specific 
algorithms to follow; that makes for everyone implementing the same 
brittle code.

The useful function of 2119 is that it allows us to document the 
important *behavioral* requirements that I have to be aware of when I am 
implementing (e.g., even though it's not obvious, my implementation MUST 
send such-and-so or the other side is going to crash and burn; e.g., 
even though it's not obvious, the other side MAY send this-and-that, and 
therefore my implementation needs to be able to handle it). And those 
"even though it's not obvious" statements are important. It wastes my 
time as an implementer to try to figure out what interoperability 
requirement is meant by, "You MUST implement a variable to keep track of 
such-and-so-state" (and yes, we see these in specs lately), and it makes 
for everyone potentially implementing the same broken code.

>> to be compliant and that a "must" (lower case) is optional is just

You bet, Thomas!

> is often what separated a good implementation from a mediocre or even poor

Agreed. Indeed, if you have a MUST or MUST NOT, I'd almost always be 
inclined to have a "because" clause. If you can't given an explanation 
of why I need this warning, there's a good chance the MUST is 
inappropriate. (One that I've seen of late is "If the implementation 
wants to send, it MUST set the <send> field to 'true'. If it wants to 
receive, it MUST set the <send> field to 'false'." I have no idea what 
those MUSTs are telling me. Under what circumstances could I possibly 
want to send but set the <send> field to "false"?)

> toto don't do that. Some amount of common sense is always required.

And again, it's worse than incomplete. It also makes for brittle code. I 
don't want you checking to see if you coded things the same way that I 
did, which is what a compliance list gets you. I want you checking that 
your *behavior* from the net interoperates with me. Insofar as you want 
to call *that* compliance, well, OK, but I don't think that's what 
people mean.

>> do so without actually breaking interoperability.

Exactly!

>> highlight an important point that may not be obvious to a casual

And to the detriment of good code..

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: [email-masked]
Date: Mon, 07 Jan 2013 08:04:03 -0800 (PST)

> algorithms to follow; that makes for everyone implementing the same

Meh. I know the IETF has a thing about these terms, and insofar as they  can
lead to the use of and/or overreliance on compliance testing rather than
interoperability testing, I agree with that sentiment.

OTOH, when it comes to actually, you know, writing code, this entire attitude
is IMNSHO more than a little precious. Maybe I've missed them, but in my
experience our avoidance of these terms has not resulted in the magical
creation of a widely available perfect reference implementation that allows me
to check interoperability. In fact in a lot of cases when I write code I have
absolutely nothing to test against - and this is often true even when I'm
implementing a standard that's been around for many years.

In such cases the use of compliance language - and yes, it is compliance
language, the avoidance of that term in RFC 2119 notwithstanding - is
essential. And for that matter it's still compliance language even if RFC 2119
terms are not used.

I'll also note that RFC 1123 most certainly does use the term "compliant" in
regards to capitalized terms it defines, and if nitpicking on this point
becames an issue I have zero problem replacing references to RFC 2119 with
references to RFC 1123 in the future.

All that said, I'll again point out that these terms are a double-edged sword,
and can be used to put the emphasis in the wrong place or even to specify
downright silly requirements. But that's a argument for better review of our
specifications, because saying "MUST do this stupid and couterproductive thing"
isn't fixed in any real sense by removing the capitalization.

> for everyone potentially implementing the same broken code.

Good point. Pointing out the nonobvious bits where things have to be done in a
certain way is probably the most important use-case for these terms.

				Ned

---

From: Dean Willis <[email-masked]>
Date: Mon, 7 Jan 2013 14:19:09 -0600

Well, I've learned some things here, and shall attempt to summarize:

1) First. the "1" key is really close to the "2" key, and my spell-checker doesn't care. Apparently,  I'm not alone in this problem.

2) We're all over the map in our use of 2119 language, and it is creating many headaches beyond my own.

3) The majority of respondents feel that 2119 language should be used as stated in 2119 -- sparingly, and free that MUST is not a substitute for "does". But some people feel we need a more formal specification language that goes beyond "key point compliance" or "requirements definition", and some are using 2119 words in that role and like it.

I'm torn as to what to do with the draft in question. I picked up an editorial role after the authors fatigued in response to some 100+ AD comments (with several DISCUSSes) and a gen-art review that proposed adding several hundred 2119 invocations (and that was backed up with a DISCUSS demeaning that the gen-art comments be dealt with). My co-editor, who is doing most of the key-stroking, favors lots of 2119 language. And I think it turns the draft into unreadable felrgercarb.

But there's nothing hard we can point to and say "This is the guideline", because usage has softened the guidelines in 2119 itself. It's rather like those rules in one's Home Owner's Association handbooks that can no longer be enforced because widespread violations have already been approved.

There appears to be interest in clarification, but nobody really wants to revise the immortal words of RFC 2119, although there is a proposal to add a few more words, like IF and THEN to the vocabulary (I'm hoping for GOTO, myself; perhaps we can make 2119 a Turing-complete language.)

--
Dean

---

From: Riccardo Bernardini <[email-masked]>
Date: Mon, 7 Jan 2013 21:40:08 +0100

> There appears to be interest in clarification, but nobody really wants to revise the immortal words of RFC 2119, although there is a proposal to add a few more words, like IF and THEN to the vocabu…

Hmm... GOTO is bad style... Why not the COME FROM from
Intercal?http://en.wikipedia.org/wiki/INTERCAL   :-) (sorry, could not
resist...)

---

From: Scott Brim <[email-masked]>
Date: Mon, 7 Jan 2013 15:49:27 -0500

> Intercal?http://en.wikipedia.org/wiki/INTERCAL   :-) (sorry, could not

unwind-protect shall rule them all

but I digress

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Mon, 07 Jan 2013 13:06:42 -0800

On 01/07/2013 12:19 PM, Dean Willis wrote:
> 1) First. the "1" key is really close to the "2" key, and my spell-checker 
> doesn't care. Apparently,  I'm not alone in this problem.
> 
> 3) The majority of respondents feel that 2119 language should be used as 
> stated in 2119 -- sparingly, and free that MUST is not a substitute for 
> "does". But some people feel we need a more formal specification language 
> that goes beyond "key point compliance" or "requirements definition", and 
> some are using 2119 words in that role and like it.
> 
> I'm torn as to what to do with the draft in question. I picked up an 
> editorial role after the authors fatigued in response to some 100+ AD 
> comments (with several DISCUSSes) and a gen-art review that proposed
> adding several hundred 2119 invocations (and that was backed up with a
> DISCUSS demeaning that the gen-art comments be dealt with). My co-editor,
> who is doing most of the key-stroking, favors lots of 2119 language. And I
> think it turns the draft into unreadable felrgercarb.

My proposal for the aforementioned draft is to put on hold for now the edits
related to this discussion, then let the WG, IETF (during last call) and IESG
decide what to do.  The edits are done and ready to be merged, so the painful
part is already done: https://github.com/petithug/p2psip-base-master/branches.

- -- 
Marc Petit-Huguenin

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 7 Jan 2013 17:01:33 -0800 (PST)

> references to RFC 1123 in the future.

+1.  There is similar language in RFC 1122 and RFC 1812.  From the standpoint 
of making the requirements clear for an implementor, I think that these three 
specifications were among the best the IETF ever produced.

//cmh

---

From: "Robin Uyeshiro" <[email-masked]>
Date: Mon, 7 Jan 2013 15:41:28 -1000

Maybe part of the job of a working group should be to both/either produce or
approve a reference implementation and/or a test for "interoperability"?  I
always thought a spec should include an acceptance test.  Contracts often
do.

If a company submits code that becomes reference code for interoperability
tests, that code is automatically interoperable and "certified".  That might
mean more companies would spend money to produce working code.  It might
mean that more working code gets submitted earlier, as the earliest approved
code would tend to become the reference.  By code, I don't mean source,
necessarily.

Then there would be a more objective test for compliance and less dependence
on capitalization and the description.

++++++++++++++++++++++++++++++++

  Meh. I know the IETF has a thing about these terms, and insofar as they
can
  lead to the use of and/or overreliance on compliance testing rather than
  interoperability testing, I agree with that sentiment.

  OTOH, when it comes to actually, you know, writing code, this entire
attitude
  is IMNSHO more than a little precious. Maybe I've missed them, but in my
  experience our avoidance of these terms has not resulted in the magical
  creation of a widely available perfect reference implementation that
allows me
  to check interoperability. In fact in a lot of cases when I write code I
have
  absolutely nothing to test against - and this is often true even when I'm
  implementing a standard that's been around for many years.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 8 Jan 2013 02:29:22 -0000

> But some people feel we need a more formal specification language
> that goes beyond "key point compliance" or "requirements definition",
> and some are using 2119 words in that role and like it.

Having read specs like the Algol 68 report and ANSI X3.53-1976, the
PL/I standard that's largely written in VDL, I have an extremely low
opinion of specs that attempt to be very formal.  

The problem is not unlike the one with the fad for proofs of program
correctness back in the 1970s and 1980s.  Your formal thing ends up
being in effect a large chunk of software, which will have just as
many bugs as any other large chunk of software.  The PL/I standard is
famous for that; to implement it you both need to be able to decode
the VDL and to know PL/I well enough to recognize the mistakes.

What we really need to strive for is clear writing, which is not the
same thing as formal writing.  When you're writing clearly, the places
where you'd need 2119 stuff would be where you want to emphasize that
something that might seem optional or not a big deal is in fact
important and mandatory or important and forbidden.

R's,
John

---

From: John Day <[email-masked]>
Date: Mon, 7 Jan 2013 21:52:51 -0500

I have spent more than a little time on this problem and have 
probably looked at more approaches to specification than most, 
probably well over a 100.  I would have to agree.  Most of the very 
formal methods such as VDL or those based on writing predicates in 
the equivalent of first-order logic end up with very complex 
predicates.  Which of course means there is a higher probability of 
errors in the predicates than in the code.  (Something I pointed out 
in a review for a course of the first PhD thesis (King's Program 
Verifier) that attempted it.  Much to the chagrin of the professor.)

Of course protocols are a much simpler problem that a specification 
of a general program (finite state machine vs Turing machine), but 
even so from what I have seen the same problems exist.  As you say, 
the best answer is good clean code for the parts that are part of the 
protocol and only write requirements the parts that aren't.  The hard 
part is drawing that boundary.  There is much that is specific to the 
implementation that we often don't recognize.  The more approaches 
one can get, the better.  Triangulation works well!  ;-)

---

From: Dick Franks <[email-masked]>
Date: Tue, 8 Jan 2013 13:41:32 +0000

On 5 January 2013 19:14, Marc Petit-Huguenin <[email-masked]> wrote:
[snip]
> in a formal language and keep it secret.
Which raises the obvious question:  Why do we not write protocol specs
in a formal specification language instead of struggling with the
ambiguities of natural language?

Theorem provers and automated verification tools could then be brought
to bear on both specifiations and implementations.

Dick
--

---

From: John Day <[email-masked]>
Date: Tue, 8 Jan 2013 09:45:23 -0500

The reasons have been discussed or at least alluded to previously in 
this thread.  The short answer is we have been there and done that: 
30 years ago.

All those tools were developed and used successfully in the 80s.   I 
know of cases where doing the formal specification alongside the 
design phase caught lots of problems.   However, there are two 
central problems:  First, in general, programmers are lazy and just 
want to code. ;-)  Using a formal method is a lot more work.  Second, 
the complexity of the formal statements that must be written down is 
greater than the code.  So there is a higher probability of mistakes 
in the formal description than in the code.  Admittedly, if those 
statements are made, one has a far greater understanding of what you 
are doing.

Once you have both, there is still the problem that if a bug or 
ambiguity shows up, neither the code nor the formal spec nor a prose 
spec can be taken be the right one.  What is right is still in the 
heads of the authors.  All of these are merely approximations.  One 
has to go back and look at all of them and determine what the right 
answer is.  Of course, the more things one has to look at the better. 
(for a bit more, see Chapter 1 of PNA).

John

---

From: Donald Eastlake <[email-masked]>
Date: Tue, 8 Jan 2013 10:12:52 -0500

Another problem is maintenance. Protocols change. Having to maintain a
formal specification is commonly at least an order of magnitude more
effort than maintaining a prose description. So it doesn't happen and
they very rapidly get out of synch in any living protocol. As an
example, the IEEE 802.11 (Wi-Fi) standard used to have a normative
formal description of the MAC operation (see Annex C of 802.11-1999).
By 802.11-2007 this was out of synch but was still included as
informational material on the theory it might be of some use and the
section began with the words "This clause is no longer maintained...".
Although still present as informational in 802.11-2012, the first
words of that section are now "This annex is obsolete."

And, as has been mentioned before, I'd like to emphasize that the IETF
experience and principal is that, if you want interoperation,
compliance testing is useless. The way to interoperation is
interoperation testing between implementations and, to a first
approximation, the more and the earlier you do interoperation testing,
the better.

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

On Tue, Jan 8, 2013 at 9:45 AM, John Day <[email-masked]> wrote:
> The reasons have been discussed or at least alluded to previously in this
> thread.  The short answer is we have been there and done that: 30 years ago.
>
> All those tools were developed and used successfully in the 80s.   I know of
> cases where doing the formal specification alongside the design phase caught
> lots of problems.   However, there are two central problems:  First, in
> general, programmers are lazy and just want to code. ;-)  Using a formal
> method is a lot more work.  Second, the complexity of the formal statements
> that must be written down is greater than the code.  So there is a higher
> probability of mistakes in the formal description than in the code.
> Admittedly, if those statements are made, one has a far greater
> understanding of what you are doing.
>
> Once you have both, there is still the problem that if a bug or ambiguity
> shows up, neither the code nor the formal spec nor a prose spec can be taken
> be the right one.  What is right is still in the heads of the authors.  All
> of these are merely approximations.  One has to go back and look at all of
> them and determine what the right answer is.  Of course, the more things one
> has to look at the better. (for a bit more, see Chapter 1 of PNA).
>
> John
>
>

---

From: John Day <[email-masked]>
Date: Tue, 8 Jan 2013 10:51:46 -0500

Hear. Hear.  Ditto!  Absolutely.  etc.

---

From: [email-masked] (Martin Rex)
Date: Wed, 9 Jan 2013 01:36:09 +0100 (CET)

>John Day <[email-masked]> wrote:
>>
>> All those tools were developed and used successfully in the 80s.
>> I know of cases where doing the formal specification alongside the
>> want to code. ;-)  Using a formal method is a lot more work.
>> Second, the complexity of the formal statements that must be written
>> down is greater than the code.  So there is a higher probability
>> of mistakes in the formal description than in the code.

I believe that the problem with the formal logic is that it is difficult
to both write as well as read/understand, and to verify that the chosen
"axioms" actually reflect (and lead to) the desired behaviour/outcome,
the relative of scarcity of suitable tools, the complexity of the
underlying theory and tools, and the tools' resulting lack of
"intuitive usability".

>> shows up,

For any formal proofing system worth its dime, this can be 100% ruled out,
since the proofing system will emit a 100% bugfree implementation of the
spec in the programming language of your choice as a result/byproduct of the
formal proofing process.

>>
>>  neither the code nor the formal spec nor a prose spec can be taken
>> be the right one.  What is right is still in the heads of the authors.

But maybe you're not really thinking of a defect in the implementation
(commonly called "bug") but rather a "gap" in the specification that
leads to unintended or undesired behaviour of a 100% bug-free
implementation of your spec.

(see Section 4 of this paper for an example:
  http://digbib.ubka.uni-karlsruhe.de/volltexte/documents/1827
)

Donald Eastlake wrote:
> effort than maintaining a prose description.

Definitely.  Discussing protocols at the level of formal specification
language would be quite challenging (typically impossible in an open
forum, I believe it is even be difficult in a small and well-trained
design team).

> compliance testing is useless.

Ouch.  I believe this message is misleading or wrong.

Compliance testing is VERY important, rather than useless,

Compliance testing would actually be perfectly sufficient iff the spec
was formally proven to be free of conflicts and ambiguities among
specified protocol elements -- plus a significant effort spent on
ensuring there were no "gaps" in the specification.

As it turns out, however, a significant amount of implementations will
be created by humans interpreting natural language specifications,
rather than implemenations created as 100% bug-free by-products of a
formal proof tool, and often, interop with such buggy, and often
spec-incompliant implementations is desirable and necessary since
they make up a significant part of the installed base.

>
> The way to interoperation is interoperation testing between
> implementations and, to a first approximation, the more and the
> earlier you do interoperation testing, the better.

The #1 reason for interop problems and road-block to evolution of
protocols is the wide-spread lack of compliance testing (on the flawed
assumption that it is useless) and focus on black-box interop testing
of the intersection of two subsets of protocol features.

The problem with interop testing is that it really doesn't provide
much of a clue, and the results can _not_ be extrapolated to features
and areas that were not interop tested (typically the other 90% of the
specification, many optional features and most of the extensibility).

The near complete breakage of TLS protocol version negotiation is a
result of the narrow-minded interop testing in companion with a complete
lack of compliance testing.

If done right, pure compliance testing can go a long way to providing
interop.  The only area where real interop testing is necessary is
those protocol areas where the spec contains conflicts/inconsistencies
that implementors didn't notice or ignored, or where there is a
widespread inconsistencies of implementations with the specification,
of created by careless premature shipping of a defective implementation
and factual creation of an installed base that is too large to ignore.

-Martin

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 08 Jan 2013 17:02:02 -0800

On 01/08/2013 05:41 AM, Dick Franks wrote:
>> 1. Someone design a new protocol, something simple but not obvious, and
>> write in a formal language and keep it secret.
>> 
> Which raises the obvious question:  Why do we not write protocol specs in a
> formal specification language instead of struggling with the ambiguities of
> natural language?
> 
> Theorem provers and automated verification tools could then be brought to
> bear on both specifiations and implementations.
> 

IMO, the main problem with using a formal specification language is that it
would made RFCs a little bit too close to mathematics for some people, as math
is not patentable.  So do not expect any change in this direction.

- -- 
Marc Petit-Huguenin

---

From: John Day <[email-masked]>
Date: Tue, 8 Jan 2013 20:19:29 -0500

>underlying theory and tools, and the tools' resulting lack of

The tools have been available for quite some time.  It is still very difficult.

>spec in the programming language of your choice as a result/byproduct of the

C'mon. You don't really believe that do you? The statement either is 
a tautology or naive.  Ever see a compiler with bugs?  Who verified 
the verifier?  How do you know the verifier is bug free?

As I indicated before, I have been working on this problem since we 
discovered Telnet wasn't as good as we thought it was.  For data 
transfer protocols, it is relatively straightforward and can be 
considered solved for anyone who wants to bother.  The problem is 
most don't.

The real problem is the so-called application protocols where dealing 
the different semantics of objects in different systems makes the 
problem very hard and very subtle.  The representation of the 
semantics is always of what you think its seminal properties are. 
This is not always obvious.

>implementation of your spec.

One person's gap is another person's bug.  What may be obvious to one 
as something that must occur may not be so to the other.  Then there 
is that fine line between what part of the specification is required 
for the specification and what part is the environment of the 
implementation.

The human mind has an amazing ability to convince us that how we see 
the world is the way others do.  Having been on the Net when there 
were 15-20 very different machine architectures, I can assure you 
that implementation strategies can differ wildly.

I had had great hopes for the Temporal ordering approaches to 
specification since they said the least about the implementation. 
(the specification is entirely in terms of ordering events) However, 
I never met anyone who could design in them.

>forum, I believe it is even be difficult in a small and well-trained

Which means there will always be a danger of "lost in translation."

>ensuring there were no "gaps" in the specification.

Do you realize the cost of this amount of testing? (Not saying it 
isn't a good idea, just the probability of convincing a development 
manager to do it is pretty slim.)  ;-)

>they make up a significant part of the installed base.

My impression is that few implementations these days are done from 
scratch.  Available software is commandeered and adapted.  So it 
would seem getting the core aspects of the protocol well implemented 
and let laziness takes its course!  ;-)

>and factual creation of an installed base that is too large to ignore.

That last sentence is a doozy.  You are saying that interop testing 
will be required with bad implementations with sufficiently large 
installed base that you have to live with it?

John

---

From: Dick Franks <[email-masked]>
Date: Wed, 9 Jan 2013 02:46:40 +0000

On 9 January 2013 01:19, John Day <[email-masked]> wrote:
[snip]
>
> One person's gap is another person's bug.  What may be obvious to one as
> something that must occur may not be so to the other.  Then there is that
> fine line between what part of the specification is required for the
> specification and what part is the environment of the implementation.

Disagree

A gap in the specification will result in all implementations having
the same unintended behaviour, because the developers understood and
followed the spec 100%.

Bugs are distinguishable from gaps because they occur in some
implementations but not others and arise from misinterpretation of
some aspect of the specification.  In this context, over-engineering
is a bug, as distinct from competitive advantage.

Dick
--

---

From: John Day <[email-masked]>
Date: Tue, 8 Jan 2013 23:38:25 -0500

It would be interesting to see you apply that.

This is what I have been talking about.  The human mind's ability to 
believe that the whole world sees everything the same way they do. 
It really is quite amazing.

These so-called gaps often arise because they were unstated 
assumptions or things that the author believed were patently obvious 
and didn't need to be stated. Actually didn't know needed to be 
stated.  From his point of view, no one would do it differently. 
Nothing had been left out and he didn't make the "mistake."   What 
the other guys did was a bug.

There is a much greater range of interpreting these things than it 
appears that you imagine.

---

From: [email-masked] (Martin Rex)
Date: Wed, 9 Jan 2013 09:46:00 +0100 (CET)

John Day wrote:
> C'mon. You don't really believe that do you?

Which one in particular.

The KIV-tool, which I was shown and explained in 1996 by a student doing
his master thesis with it on a module of our software, can output
source code at the end of its processing.  Code that is a guaranteed
bug-free implementation of the spec.  (which MUST NOT be confused
with the given spec providing the desired behaviour and being free
of unintended (and unthough-of) behaviour).

(While the tool's name appears to originate from the University of
 Karlsruhe (->Karlsruhe Interactive Verifier), and that student back
 then was from Karlsruhe, the current homepage of the tool appears
 to be at the University of Augsburg:

  http://www.informatik.uni-augsburg.de/lehrstuehle/swt/se/kiv/
)

>
> Ever see a compiler with bugs?

Compiler bugs are comparatively rare, many of them are problems
of the optimizer (therefore avoidable) and many happen under
obscure situations that can be avoided by defensive programming
style.  Code generators often produce "defensive style code".
Using a robust and mature programming language, such as C89,
may also help to avoid compiler problems of fancy or complex
languagues or fancy features.

Look at the long list of defects of modern CPUs.  Apps programmers
that use a compiler rarely have to know (let alone take into account)
those CPU defects because most compiler code generators use only
a subset of CPU features anyway.

>
> Who verified the verifier?  How do you know the verifier is bug free?

Now you must be kidding.

Are you suggesting that anyone using the programming language Prolog
would be gambling and could never expect deterministic results?

With a decent amount of expertise and the right mindset,
even mere mortal humans can produce code with a very low error rate.
IIRC, Donald E. Knuth didn't need a formal specification and a
formal proofing tool to achieve a low error rate for TeX and Metafont.

Having started assembler programming at the age of 12 myself, I could
easily understand why DEK used Assembler for his algorithm code examples
in TAOCP when I first encountered these books at age 21.

> considered solved for anyone who wants to bother.  The problem is 

There is a perverted conditioning in our education system and
business life by rewarding mediocrity (cheating, improvising, being
superficial) which is IMO one of the underlying causes for
  http://en.wikipedia.org/wiki/Dunning_Kruger
and impairs many implementors' motivation to watch out and compensate for
defects in the spec and to recognize and respect design limits of a spec,
i.e. avoid underspecified protocol features or require official
clarification and adjust the code accordingly before shipping
implementations of underspecified protocol features.

> This is not always obvious.

I believe that the real difficulty is about designing and discussing
at the abstract level and then performing formally correct transitions
from abstract spec level to concrete specification proposla level
and comparing implementation options at the spec level ... and getting
everyone that wants to participate to understand the abstraction
and how to perform transitions from and to concrete spec proposals.

I'm experiencing that difficulty myself every once in a while.
In 1995/1996 I had a adopted notion/bias for the uselessness of
traditional GSS-API channel bindings (as defined in GSS-APIv1 rfc1508/1509
and GSS-APIv2 rfc2743/2744) and it took me a few weeks of discussion
to really understand what Nico was trying to accomplish with
cryptographic channel bindings, such as tls-unique, its prerequisites
and real properties and its benefits.

> as something that must occur may not be so to the other.

There is a significant difference between a bug in an implementation
and an unintended consequence due to a "gap" in the specification.

The SSLv3 protocol and the IETF successor TLS contains a somewhat
underspecified protocol option called renegotiation, and some apps folks
had the not very bright idea to abuse that protocol option and
made flawed assumptions about protocol properties that did not
actually exist.

Had those apps folks, rather than shortcutting on flawed assumptions,
simply collected the description of the renegotiation properties that
are sprinkeled throughout the TLS spec and put them into a single list,
and then thought about it for an hour, then the problem (the absence of
the property that they wanted to rely on) should have become obvious to them.

It became obvious for me as soon as I had put together that list and
thought about the consequences during discussion of TLS channel bindings.

> the world is the way others do.

The Dunnin-Kruger effect, "Routine", "habits", are all variations
of trainig a neural network and improving the pattern recognition
over time.  The strength of neural networks, to recognize also
variations of known patterns, is also its weakness, a bias/preference
to recognize stuff as something that is already known.
Fixing training errors and trying to add similar distinct patterns
very late usually result in a significant (re-)trainig effort --
and this is where the constant conditioning for cheating, improvising,
superficialness frequently kicks in.

"If all you have is a hammer, everything looks like a nail."

Technically, humans are capable of overcoming the Dunning-Kruger effect
and becoming aware of the limits/boundaries of their knowledge, but
many enjoy a happy life without ever trying to.

>
> Having been on the Net when there were 15-20 very different machine
> architectures, I can assure you that implementation strategies can
> differ wildly.

I'm well aware that the amount of bad code out there outnumbers
the amount of good code.  While I don't think that differing implementation
approaches imply causality for interop problems, there certainly is
the problem of ad-hoc/poor/dangerous/error-prone implementation design
that causes plenty of bugs and interop problems and unexpected
behaviours, neither of which a result of a conscious decisions,
getting shipped into the installed base.

Good specifications are sufficiently clear and precise to enable
smooth interop, yet sufficiently abstract to provide a high level
of implementation freedom as well as future extensibility.

What is much more of a problem is the common lack in recognizing
and respecting limits of a design/specification, and a lack of
compensating for relatively easy to recognize spec defects.
Relatively easy to recognize for a careful implementer, that is,
not necessarily easy to recognize for a mere reviewer.

> Which means there will always be a danger of "lost in translation."

The danger is real.  If there is a lack of decent leadership,
the resulting spec (defects) may sometimes pose a significant
challenge to implementations&interop.

For consumers of the spec outside of the SDO, the option to
ask for clarification/fix might be non-obvious -- and regularly
is non-trivial for someone who has not previously participated
in the particular SDO (or any SDO at all) -- which is actually
quite common.  There are easily two magnitudes more implementors
of specs distributed by the IETF than there are individuals who have
participated in the IETF over the past 30 years.

> manager to do it is pretty slim.)  ;-)

I fail to follow.  Compliance testing does not necessarily require
code or a test tool.  Often, compliance will be as simple as actually
_reading_and_following_ the spec, how to process some PDU element.
And compliance testing may be as simple as a code review of less than
an hour and cross-checking a handful MUST and MUST NOTs.

Or what to do with a protocol option/feature one's own implementation
does not recognize, and then _do_what_the_spec_says_
rather than doing what a number of implementations are doing: regularly
aborting with a fatal error rather than doing what the spec says.

Example: the optional TLS protocol extension "Server Name Indication",
first specified here:  http://tools.ietf.org/html/rfc3546#section-3.1

   "HostName" contains the fully qualified DNS hostname of the server,
   as understood by the client. 

Doing what the spec says with respect to this requirement:

                                                                     If
   the server only needs to match the HostName against names containing
   exclusively ASCII characters, it MUST compare ASCII names case-
   insensitively.

If that is not sufficiently clear for an average implemetor to realize
that a case-sensitive comparison (strcmp,strncmp) will be incompliant
with the spec and a case-insensitive comparison (strcasecmp,strncasecmp,
_stricmp,_strnicmp) will have to be used instead, then we might have to
give up and close down the IETF SDO.

According to http://en.wikipedia.org/wiki/Server_Name_Indication#Support
support for this protocol option was originally designed in 2004,
and added into the main codeline of OpenSSL 0.9.8 in 2007.

However, OpenSSL-1.0.1c today still does a case-sensitive comparison...

>>lack of compliance testing.

Being aware of the breakage of the TLS protocol version negotiation
within the installed base in 2009, an explicit reminder was added
into rfc5746 that processing TLS protocol version numbers correctly
in the TLS handshake was vitally important and has always been a
requirement in the TLS protocol spec.

   TLS servers implementing this specification MUST ignore any unknown
   extensions offered by the client and they MUST accept version numbers
   higher than their highest version number and negotiate the highest
   common version.  These two requirements reiterate preexisting
   requirements in RFC 5246 and are merely stated here in the interest
   of forward compatibility.

The necessity for the TLS renegotiation info extension had become
painfully obvious after it had been realized that the two TLS connection
states involved in a TLS renegotiation had been completely independent,
enabling an MitM attacker to inject/prefix data of his choice at the
beginning of a TLS session, and the document is very explicit which
precise state information of the previous/enclosing session is carried
over into the successor TLS session and how.

Curiously, one vendor happened to actually break the TLS protocol
version negotiation much worse than it already was, by performing
a check on the client_version in the premaster secret of the
renegotiation ClientKeyExchange message not against the client_version
from the ClientHello from the very same TLS handshake, but instead
check against the client_version of the initial handshake, that had
been found to have painfully independent state and created the
necessity for the rfc5746 fix in the first place...

Simply following what the spec says during implementation would have
avoided the problem.  Performing a code review of the reminded
TLS protocol version negotiation would have noticed that breakage.
Interop testing the protocol version negotiation in a meaningful
fashion, even with earlier TLS implementations from the very same
vendor would have clearly shown the breakage prior to shipment.
As it turns out, the vendor failed on all three accounts,
and this _new_ breakage got shipped...

> installed base that you have to live with it?

It seems to be a recurring event that one or more vendors create
an installed base that is clearly non-compliant with the original
spec, but too large to ignore, and as a result, everyelse adapts
to that behaviour for the sake of interoperability with that
installed base.

e.g.  http://tools.ietf.org/html/rfc4178#appendix-C
      http://tools.ietf.org/html/rfc5929#section-8

There may be others, but I'm actively tracking only very few
topics in the IETF, so these two are ones I know from the top
of my head.

-Martin

---

From: [email-masked] (Martin Rex)
Date: Wed, 9 Jan 2013 10:15:20 +0100 (CET)

> appears that you imagine.

With "bug", I mean behaviour that is non-compliant with the spec,
where the difference is not to accomodate for a defect of the spec,
and where this behaviour is detrimental to the operation of that
implementation (vulnerability or robustness issue) or detrimental
to the interop with other implementations.

With "conflict" in a spec, I refer to distinct statements in a
spec that contradict each other (such as the same symbolic protocol
element is assigned two different values in two seperate locations
of the spec).

With "ambiguity" in a spec, I refer to an omission that precludes
that a certain feature can be implemented at all.  Such as a symbolic
protocol element that represents an integer value in a PDU, but
the spec lacks the definition of the numeric value for the 
on-the-wire representation.

With "gap" in a spec, I refer to omissions that do not preclude
the implementation, but can lead to unexpected behaviour of an
implementation.  This result is vaguely similar to what happens when
a switch() statement lacks a default: case and is unexpectedly faced
with a value that is not covered by case statements.  The result
can be "undefined" / "unpredictable", yet remain formally provable
correct with respect to the given specification.

Finally, there are "omissions" in a spec, i.e. properties that a
protocol does and can not provide, yet consumers of the protocol may
believe in "magic security pixie dust" and develop a flawed assumption
about a non-existing protocol property, as it happened in with the
TLS renegotiation.  TLS renegotiation being secure/protected against
MitM was a property that was retrofitted into SSLv3/TLS with rfc5746,
and could not possibly have existed "by accident" in any of the existing
implementations.  (Although servers that did not support renegotiation
at all never were vunerable to the "interesting" attacks.)

-Martin

---

From: Hector Santos <[email-masked]>
Date: Wed, 09 Jan 2013 08:05:14 -0500

Maybe the survey to be done is a review of all the RFC, STD and see 
which ones

- had a great abstract and introduction,
- had the better writing styles
- had the least endorsement resistance
- progress faster than most,
- had the most implementators,
- with least support/questions need to be asked,

etc, etc.

Are the RFCs that followed 2119 the more successful ones, the ones 
written with clarity and knew how to separate functional and technical 
descriptions?  Had the right target audience(s) in mind?

Maybe we can also review which I-D and RFCs have high failure or have 
trouble getting industry endorsement, and even if it did, why has it 
failed in the market place?

What is the overall problem anyway?   Is this part of the reason why 
there are efforts to fast track documents? Because there are so much 
resistance by such a broad range of disciplines with different level 
mindsets and philosophies?

-- 
HLS