Subject: Use of LWSP in ABNF -- consensus call

From: Lisa Dusseault <[email-masked]>
Date: Mon, 14 May 2007 12:10:03 -0700

The IESG reviewed <http://www.ietf.org/internet-drafts/draft-crocker- 
rfc4234bis-00.txt> for publication as Internet Standard and would  
like to know if there is consensus to recommend against the use of  
LWSP in future specifications, as it has caused problems recently in  
DKIM and could cause problems in other places.

Some discussion on this point already:
  - http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html
  - http://www1.ietf.org/mail-archive/web/discuss/current/msg00463.html
  - http://mipassoc.org/pipermail/ietf-dkim/2007q1/007295.html
  - https://datatracker.ietf.org/public/pidtracker.cgi? 
command=view_comment&id=66440  (in this tracker comment, Chris Newman  
recommended to remove LWSP, but for backward-compatibility it's  
probably better to keep it and recommend against use)

Thanks for your input,
Lisa Dusseault

---

From: Julian Reschke <[email-masked]>
Date: Mon, 14 May 2007 22:55:54 +0200

Lisa Dusseault wrote:
> 
> The IESG reviewed 
> <http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt> 
> for publication as Internet Standard and would like to know if there is 
> consensus to recommend against the use of LWSP in future specifications, 
> as it has caused problems recently in DKIM and could cause problems in 
> other places.
> 
>  - 
> https://datatracker.ietf.org/public/pidtracker.cgi?command=view_comment&id=66440  
> (in this tracker comment, Chris Newman recommended to remove LWSP, but 
> for backward-compatibility it's probably better to keep it and recommend 
> against use)

I agree that LWSP can be problematic. As the LWSP rule only appears in 
appendix B, the best approach IMHO would be to either leave it in, and 
have a warning explaining the potential problems close to it, or remove it.

The latter sounds simpler, but could cause spec writers that use ABNF to 
just copy the LWSP rule from RFC4234, ignoring the potential issues with it.

The proposed solution to warn about it in the front matter doesn't seem 
to be a good idea due to locality reasons. If there are reasons to avoud 
LWSP, they should be stated close to the definition. If this means we 
need a new revision of the spec, and last-call it again, so be it. No 
reason to hurry.

Best regards, Julian

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 14 May 2007 23:43:02 +0200

Lisa Dusseault wrote:

 [LWSP]
> could cause problems in other places.

Chris saw it in the 4646bis-05 draft four days ago,
Addison fixed it in the 4646bis-06 draft published
Friday.

It also showed up in an 2821bis draft in one valid
ABNF rule, assuming that an apparently empty line
contains trailing white space, or an invalid ABNF
rule, assuming that the same line is really empty.

Frank

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 14 May 2007 17:35:46 -0700

On May 14, 2007, at 3:55 PM, Dave Crocker wrote:

>
> Lisa Dusseault wrote:
>> The IESG reviewed <http://www.ietf.org/internet-drafts/draft- 
>> crocker-rfc4234bis-00.txt> for publication as Internet Standard  
>> and would like to know if there is consensus to recommend against  
>> the use of LWSP in future specifications,
>
> 1   This issue was initially raised in the IESG by Chris Newman,  
> who changed
> his Discuss, with a statement that he recommended inserting a  
> comment, along
> the lines that others are also recommending.  Unless I've misread  
> the record,
> all other votes on advancing ABNF from Draft to Full are positive  
> or neutral,.
> except for your own Discuss.  Is this correct?

The issue was initially raised by Frank Ellerman or by various in the  
DKIM WG depending on how you look at it -- Frank explicitly suggested  
possible changes to the draft, in his posting to the IETF list.

You're right about the voting situation but here's the background: I  
took on the DISCUSS myself as a placeholder for an issue that the  
IESG had consensus to investigate further (consensus to investigate  
what the consensus is).   I could have asked somebody else to hold  
the DISCUSS but this seemed most convenient as long as the rest of  
the IESG trusted me to investigate.

>
> 2.  The ABNF is a candidate for moving from Draft to Full.  Will  
> removing a
> rule (that is already in use?) or otherwise changing the semantics  
> of the
> specification, at this point, still permit the document to  
> advance?  I had the
> impression that moving to Full was based on some serious beliefs  
> about a
> specification's being quite stable.  Making this kind of change,  
> this late in
> the game, would seem to run counter to that.

Moving to Internet Standard is indeed something we do carefully, and  
of course that means investigating proposed changes to make sure  
they're appropriate, and setting a high bar for accepting them.  I  
believe that's what we're doing here, investigating carefully.

I share your concerns about removing rules that are already in use --  
that would generally be a bad thing.  However I'm interested in the  
consensus around whether a warning or a deprecation statement would  
be a good thing.

Thanks,
Lisa

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 14 May 2007 19:21:17 -0700

At 5:35 PM -0700 5/14/07, Lisa Dusseault wrote:
>I share your concerns about removing rules that are already in use 
>-- that would generally be a bad thing.  However I'm interested in 
>the consensus around whether a warning or a deprecation statement 
>would be a good thing.

A warning would be very useful, and a deprecation or removal would be 
very bad because either would cause people to go to the earlier 
documents to see what LWSP meant earlier.

---

From: Tony Hansen <[email-masked]>
Date: Mon, 14 May 2007 23:20:28 -0400

Lisa Dusseault wrote:
> consensus around whether a warning or a deprecation statement would be a
> good thing.

LWSP has a valid meaning and use, and its being misapplied somewhere
doesn't make that meaning and usage invalid. I could see a note being
added. However, anything more than that is totally inappropriate.

	Tony Hansen
	[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Tue, 15 May 2007 09:10:41 +0100

On Mon, 14 May 2007, Dave Crocker wrote:
>
>      "Could cause problems in other places"...  The DKIM hiccup was the first
> one I'd heard about.
>
>      By contrast, "linear-white-space" was defined in RFC733, in 1977, with
> RFC822 retaining that definition.  It is defined in those places as
> essentially the same as LWSP in the current ABNF Draft Standard specification.

The LWSP defined in ABNF is more like the one in HTTP than the message
format one, in that 4234 allows space-only lines (it allows multiple CRLFs
in LWSP) whereas 2822 does not (at most one CRLF in FWS).

There is some documentation of the interoperability problems arising from
the implied-LWS rule in HTTP here:
http://www.and.org/texts/server-http#implicit-lws

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER THAMES: NORTHWEST BACKING SOUTHWEST 4 OR 5, OCCASIONALLY 6
OR 7 AT FIRST. MODERATE, OCCASIONALLY ROUGH IN GERMAN BIGHT. RAIN OR SHOWERS.
MODERATE OR GOOD, OCCASIONALLY POOR.

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 15 May 2007 09:10:53 +0100

> added. However, anything more than that is totally inappropriate.

+1

Frank's text in
<http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html>
would be fine:

  Authors intending to use the LWSP (linear white space) construct
  should note that it allows apparently empty lines consisting only
  of trailing white space, semantically different from really empty
  lines.  Some text editors and other tools are known to remove any
  trailing white space silently, and therefore the use of LWSP in
  syntax is not recommended.

However, it doesn't belong in "security considerations".

What about moving LSWP, and this text, to a separate section of Annex B:
"B.3 Deprecated constructs"?

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Keith Moore <[email-masked]>
Date: Tue, 15 May 2007 11:53:43 -0400

I think it's appropriate to include a note to
- explain the problem caused by LWSP
- suggest that it's included for backward compatibility and it's
probably not something that should be used in new specifications

I'd stop short of removing LWSP (for backward compatibility reasons) or
forbidding its use in new specifications.  ABNF is just a notation for
describing a syntax, and the spec shouldn't try to artificially
constrain what kind of syntax can be defined.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 15 May 2007 20:21:06 +0200

Lisa Dusseault wrote:
>> rule (that is already in use?) or otherwise changing the semantics of 
>> the
>> specification, at this point, still permit the document to advance? I 
>> had the
>> impression that moving to Full was based on some serious beliefs about a
>> specification's being quite stable. Making this kind of change, this 
>> late in
> consensus around whether a warning or a deprecation statement would be 
> a good thing.
Removing features that have proved to be a Bad Idea has always been 
listed as one of the possible changes from Proposed to Draft - Draft to 
Full happens so rarely that I would be hesitant to claim that there's 
tradition for such changes there.

Despite this, I agree with the people who think that a warning comment, 
rather than removal of the rule, is the Right Way.

Harald

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 15 May 2007 20:46:09 +0200

Lisa Dusseault wrote:

> The issue was initially raised by Frank

Hi, a short explanation, initially I hoped that 4234 can be
promoted to STD "as is".  I missed the (now listed) errata
in the "pending errata mbox".

Some months later 4234bis-00 was posted, and if 4234 can't
be promoted as is, then that's an opportunity to address
this (known) LWSP issue.

Just removing it is an idea, but for the reasons stated by
Dave I felt that "just deprecating it" is good enough with
less undesirable side-effects.

After all it's simple to implement LWSP as specified.  But
unfortunately it's also simple to destroy critical white
space in an apparently empty line.  

Sorry for the confusion, I should have checked the pending
errata mbox before the proposal to promote RFC 4234 to STD.

Frank

---

From: Ned Freed <[email-masked]>
Date: Tue, 15 May 2007 12:03:04 -0700 (PDT)

> added. However, anything more than that is totally inappropriate.

Full agreement with Tony here.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 May 2007 15:19:57 -0400

--On Tuesday, 15 May, 2007 12:03 -0700 Ned Freed
<[email-masked]> wrote:

>> Lisa Dusseault wrote:
>> > I share your concerns about removing rules that are already
>> > in use -- that would generally be a bad thing.  However I'm
>> > interested in the consensus around whether a warning or a
>> > deprecation statement would be a good thing.
> 
>> LWSP has a valid meaning and use, and its being misapplied
>> somewhere doesn't make that meaning and usage invalid. I
>> could see a note being added. However, anything more than
>> that is totally inappropriate.
> 

+1

ABNF is simply a tool, a grammar, and a set of definitions. I'd
almost favor a separate applicability statement that encourages
the use of some features and discourages others as appropriate
if that is really needed.  But a particular element should be
removed from the standard only if there is a case to be made
that the definition is inadequate or consistent with other parts
of the model or grammar.   If such inconsistencies actually
existed, ABNF should, IMO, be bounced back to Proposed and
fixed.  Fortunately, that doesn't seem to be the case here, but
I would really question what we are doing to ourselves if our
grammatical definitions start needing profiles

     john

---

From: Tony Finch <[email-masked]>
Date: Tue, 15 May 2007 22:54:18 +0100

On Tue, 15 May 2007, Dave Crocker wrote:
>
> So that is a total of at most 2 documented cases in 10-30 years.
> And keep in mind that the issue is not that the rule "does not work" but that
> it is very rarely mis-used.

Did you miss my post linking to a description of LWSP-related interop
problems in HTTP? http://www.and.org/texts/server-http#implicit-lws

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTH TYNE DOGGER: WEST 4 OR 5, BECOMING VARIABLE 2, THEN SOUTHERLY 3 OR 4.
SLIGHT OR MODERATE. SHOWERS THEN MAINLY FAIR. GOOD OCCASIONALLY MODERATE AT
FIRST.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 14 May 2007 15:55:07 -0700

Lisa Dusseault wrote:
> The IESG reviewed 
> <http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt> for 
> publication as Internet Standard and would like to know if there is 

Lisa,

Process:

1   This issue was initially raised in the IESG by Chris Newman, who changed
his Discuss, with a statement that he recommended inserting a comment, along
the lines that others are also recommending.  Unless I've misread the record,
all other votes on advancing ABNF from Draft to Full are positive or neutral,.
except for your own Discuss.  Is this correct?

2.  The ABNF is a candidate for moving from Draft to Full.  Will removing a
rule (that is already in use?) or otherwise changing the semantics of the
specification, at this point, still permit the document to advance?  I had the
impression that moving to Full was based on some serious beliefs about a
specification's being quite stable.  Making this kind of change, this late in
the game, would seem to run counter to that.

> as it has caused problems recently in DKIM and could cause problems in 

Semantics:

      This seems to be one LWSP problem in 30 years. Is this really a
sufficient basis for changing the semantics of something that has been stable
and widely used for 10-30 years (depending upon how you count)?

      Are we reasonably comfortable that making the change will not create new
and different problems, such as for other uses of ABNF?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Eric Allman <[email-masked]>
Date: Mon, 14 May 2007 12:46:54 -0700

I'm inclined to agree that there is a problem, and your proposed 
solution of keeping LWSP but depreciating it is probably correct. 
However, I recommended also adding a new, correct definition (for 
DKIM we used FWS from RFC 2822) and a discussion of the differences.

eric

---

From: Philip Guenther <[email-masked]>
Date: Mon, 14 May 2007 19:59:29 -0600

On Mon, 14 May 2007, Lisa Dusseault wrote:
> publication as Internet Standard and would like to know if there is consensus 
> to recommend against the use of LWSP in future specifications, as it has 
> caused problems recently in DKIM and could cause problems in other places.

LWSP was one of the factors in the SASL WG's abandoning of the DIGEST-MD5 
revision.  To be precise, the complexity inherent in the 822-style n#rule 
syntax (the emulation of which in modern ABNF uses LWSP) was found to be a 
source of implementation errors and interoperability issues.  Part of the 
discussion can be seen at:
    http://www.imc.org/ietf-sasl/mail-archive/msg02752.html

While I don't think removing LWSP from the ABNF RFC is good idea, adding a 
comment to its definition in section B.1 that warns protocol designers to 
consider carefully whether they really need or want "LWSP" instead of 
"*WSP" does seem appropriate.  Calling out that LWSP permits lines 
containing just whitespace may help drive home the issue.

Philip Guenther
Sendmail, Inc

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 11:27:38 -0700

> tradition for such changes there.

The question is the "proved to be" criterion.

The consensus call was triggered by one documented problem in 10 years.  We've 
had a posting claiming one additional problem (although my own recollection of 
that bit of history was the the list construct was the issue, rather than LWSP.)

Were we to deprecate every feature in IETF specifications that get 
mis-implemented a couple of times over 10 years, I suspect much of our 
technology would be deprecated...

d/
-- 

   Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Mon, 14 May 2007 15:55:07 -0700

Lisa,

Process:

Semantics:

d/
-- 

   Dave Crocker

---

From: Eric Allman <[email-masked]>
Date: Mon, 14 May 2007 12:46:54 -0700

eric

---

From: Eric Allman <[email-masked]>
Date: Mon, 14 May 2007 12:46:54 -0700

eric

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 11:27:38 -0700

d/
-- 

   Dave Crocker

specification.

d/
-- 

   Dave Crocker

---

From: Eric Allman <[email-masked]>
Date: Mon, 14 May 2007 12:46:54 -0700

eric

---

From: Philip Guenther <[email-masked]>
Date: Mon, 14 May 2007 19:59:29 -0600

Philip Guenther
Sendmail, Inc

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 11:27:38 -0700

d/
-- 

   Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 11:27:38 -0700

d/
-- 

   Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 15:52:40 -0700

> problems in HTTP? http://www.and.org/texts/server-http#implicit-lws

No.

d/
-- 

   Dave Crocker

---

From: Douglas Otis <[email-masked]>
Date: Wed, 16 May 2007 09:25:56 -0700

> However, it doesn't belong in "security considerations".

Discarding of lines is likely in response to some type of exploit.   
The consideration for not using LWSP would be in regard with how  
security requirements may create incompatibilities.  This is the  
correct section.

> "B.3 Deprecated constructs"?

Agreed. That would also be appropriate.

Another problem regarding LWSP is in regard to _many_ differing  
definitions.  A profusion of differing definitions alone becomes a  
valid reason to deprecate the mnemonic.  This definition represents a  
poor practice as related to security which should not be facilitated  
through standardization.  By removing this problematic construct,  
better solutions are more likely to be found.  At least (ab)use of  
the mnemonic will have been discouraged.  Any continued use of this  
mnemonic should be discouraged and the note added should clarify one  
of the reasons for this mnemonic being deprecated is specifically due  
to its varied and checkered meanings in other drafts.

-Doug

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 14 May 2007 23:49:44 +0100

> probably better to keep it and recommend against use)

I think it would be better to keep LWSP and recommending against its 
use. Running grep on various RFCs shows that this production is 
referenced in several RFCs, even in some recent ones.

I don't object to Chris' idea to add FWS to the ABNF document.

---

From: <[email-masked]>
Date: Tue, 15 May 2007 15:32:08 -0400

Add a warning note

Bill Oxley
Messaging Engineer
Cox Communications
404-847-6397
-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Tuesday, May 15, 2007 3:03 PM
To: Tony Hansen
Cc: Apps Discuss; IETF General Discussion Mailing List;
[email-masked]
Subject: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call

--
> > consensus around whether a warning or a deprecation statement would
be a

				Ned
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 12:17:35 -0400

--On Tuesday, 15 May, 2007 11:27 -0700 Dave Crocker
<[email-masked]> wrote:

> Were we to deprecate every feature in IETF specifications that
> get mis-implemented a couple of times over 10 years, I suspect
> much of our technology would be deprecated...

IMO, and at the risk of again agreeing with Dave, this is the
issue for me.  

If we have inconsistent uses of terminology across documents
that are supposed to be using the same, standardized, term, then
that is a problem with our review process.  If the term is
explicitly standardized in one of the documents, that is the
definition; things that use the term incorrectly should be
candidates for fixing.

By contrast, if we consider "misused sometimes" or "ambiguous
with other uses" as a sufficient condition for deprecating the
term itself, then we have a long list of terms to deprecate in
front of us, almost certainly starting with "IP", which refers
to several different protocols, a protocol layer, and something
that often involves lawyers.

I think some warning language about safe and unsafe contexts may
be in order for this construction but I expect that everyone who
is arguing for deprecating it entirely (or inserting a strong
"don't use this" statement) will be making a case for similar
language the next time an IPv6 document comes up for review.

       john

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 13:35:31 -0400

--On Thursday, 17 May, 2007 12:42 -0400 Sam Hartman
<[email-masked]> wrote:

> I don't see why the standardized definition is the obvious
> right place to fix things.  I thought we were committed to
> running code.  To me, one implication of that commitment is
> that sometimes the right fix is for the spec to change rather
> than the implementations.
> 
> In a terminology conflict, this often involves moving away
> from the term that has two conflicting uses to terms that are
> more clear.
> 
> Ultimately cases like this should be evaluated based on
> whether the final result is more clear overall.

Sam,

Two observations; I hope you don't think they are contradictory.

(1) We regularly get ourselves into intellectual and procedural
difficulties by treating specifications about how protocol
specifications are written as if they were protocol
specifications.  When we try to avoid that, we get ourselves
into worse problems.  Using rules that are more or less
arbitrary, we make some of these documents Proposed Standards
and then try to progress them, we make others into BCPs and,
now, we make still others into IONs.

If we are going to standardize a definitional requirement or
method -- whether it is ABNF or IPR boilerplate or something --
we need to get it right as a self-contained definition and then
live with it.  We should certainly revise and replace it if it
turns out to be unworkable (as has happened with IPR work) or if
the definition turns out to be inadequate to permit an
unambiguous interpretation (that issue spills over into my
second observation, below).  But, once other specifications
start to depend on the definitions that are there, and show
those definitions to be adequate, we should not be talking about
deprecating definitions unless we are prepared to "that was
wrong, we need to start over (even though some of the older
material may still be useful)".    Again, please note the
similarity to the IPR work.

(2) If we pretend that the ABNF metalanguage and definitions are
actually a protocol specification, then we need to evaluate it
as one.  Then, we have the following criteria (which we usually
don't state quite this precisely):

	(i) Is the definition good enough that interoperable
	implementations are possible?
	
	(ii) Do people care enough about the construct to
	actually use it in ways that show it is useful?

Now, neither of those rules prevents non-conforming
"implementations".  We may notice that those exist, but our
concern is only about implementations that appear to conform to
the spec and are (or are not) interoperable.  If non-conforming
implementations happen by accident because the text isn't clear
enough, we try to clarify the text.   But we don't say "well,
there are non-conforming implementations, so the spec is
broken".   That would make no sense at all, at least to me.

The answer to (i) appears to be "yes".  There are lots of
conforming cases.  And the answer to (ii) is, as Dave as pointed
out repeatedly, "about 30 years worth".

Is this construction dangerous if used in inappropriate
contexts?  Sure.  Does that justify a warning note to the
unwary?  Probably.  Is it possible to implement other things and
call them by the same name (i.e., create a non-conforming
implementation)?  Of course.  Should that invalidate the
definition?  Not if we want to have anything left if the
principle were applied broadly.

     john

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 13:35:31 -0400

: <[email-masked]>
X-Spam-Score: 0.0 (/)
X-Scan-Signature: 73734d43604d52d23b3eba644a169745

Sam,

	

     john

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 13:35:31 -0400

: <[email-masked]>
X-Spam-Score: 0.1 (/)

Sam,

	

     john

---

From: Tony Finch <[email-masked]>
Date: Thu, 17 May 2007 18:48:07 +0100

+1

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL: SOUTHWEST 6 TO GALE 8, INCREASING SEVERE GALE 9, PERHAPS STORM 10
LATER. VERY ROUGH OR HIGH. SHOWERS. GOOD.

---

From: Tony Finch <[email-masked]>
Date: Thu, 17 May 2007 18:48:07 +0100

: <[email-masked]>
X-Spam-Score: 0.0 (/)
X-Scan-Signature: 798b2e660f1819ae38035ac1d8d5e3ab

+1

Tony.

---

From: Tony Finch <[email-masked]>
Date: Thu, 17 May 2007 18:48:07 +0100

: <[email-masked]>
X-Spam-Score: 0.1 (/)

+1

Tony.

---

From: Tony Finch <[email-masked]>
Date: Thu, 17 May 2007 18:48:07 +0100

: <[email-masked]>
X-Spam-Score: 0.1 (/)

+1

Tony.

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 15:50:07 -0400

Sam, with one small exception, I think we are in complete
agreement.  The exception is noted below...

--On Thursday, 17 May, 2007 15:32 -0400 Sam Hartman
<[email-masked]> wrote:

> Right.  Here, I don't think the definition is wrong, I just
> think the term being defined is wrong.  We proposed a
> definition for a useful concept.  The word we chose (LWSP)
> stuck in some places but not in others.  IN fact other people
> used the same word for a different although related concept.
> sufficiently so that the definition we proposed in ABNF is not
> the most common definition in our standards.

Counting uses in this way so as to determine what is or is not
the "most common definition" is as tricky in our environment as
counting votes.  For the same reason, I'd rather that we avoid
it when we can.   However, if we do need to count, I think we
have to use some sort of system that attributes more weight to
documents that use the terminology that have been, effectively,
at full standard before some of the specifications that use
different definitions were even dreamt of.  I don't know how to
assign those weights, which, recursively, is why I don't like
counting.   But I don't think you can count up a handful of
recent misuses and then make an inference that the base
definition needs to be retired (which you haven't done, but
several others have), or even that it is not appropriately
"common".

> I think that in this instance, the value of future clarity
> justifies  coming up with a new term that will unambiguously
> mean what LWSP  means in ABNF today.  That term will have to
> start at proposed  standard.

Pragmatically, perhaps yes.  But what I keep coming back to is
that it appears to me to be perfectly clear and unambiguous
"what LWSP means in ABNF today".  That is what is written into
the spec.  As far as I can tell, we are having a discussion
about two other things:

	(1) Other specifications that use the term "LWSP" to
	refer to something different from what is unambiguously
	defined in the ABNF spec.
	
	(2) Places where people might be tempted to use LWSP but
	where they have discovered that LWSP is either
	inappropriate or risky.

The second group clearly needs new and appropriate terminology
for what they do need and use.   The first group is, IMO, just
broken.

> LWSP will need to continue in ABNF.
> 
> I see a desire to document our operational experience with the
> word: many people took this word and used it to mean something
> else. Perhaps to avoid confusion you should consider whether
> your use of the word is a good idea.  I think there is
> significant harm in choosing not to document this operational
> experience when advancing standards. After all, we both agree
> that it is this experience with running code that gives the
> IETF value.

Again, I have no problems with making carefully-considered
comments about use and misuse.  I think it is a fine idea as
long as those comments do not imply that the misuses are a
failure in the base specification.  And I think everyone should
consider a stopping rule, lest, as I indicated in my
deliberately extreme example, we discover that we want to
retire, replace, or update "IP" because of some use in the legal
community that precedes the use in network protocols by many
years.

       john

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 15:50:07 -0400

: <[email-masked]>
X-Spam-Score: 0.1 (/)
X-Scan-Signature: bdc523f9a54890b8a30dd6fd53d5d024

	

       john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 17 May 2007 21:52:45 +0200

--On 17. mai 2007 15:32 -0400 Sam Hartman <[email-masked]> wrote:

>
> Right.  Here, I don't think the definition is wrong, I just think the
> term being defined is wrong.  We proposed a definition for a useful
> concept.

Actually we defined a concept (LWSP) in a way that turned out to be much 
more
troublesome in several contexts than we thought it would be (because it 
allowed
two different versions of "blank" lines to have different semantics).

>  The word we chose (LWSP) stuck in some places but not in
> others.  IN fact other people used the same word for a different
> although related concept.  sufficiently so that the definition we
> proposed in ABNF is not the most common definition in our standards.
>
> Clearly we should not invalidate existing uses of that term.  Clearly
> we do need a definition for the term: it is being used usefully.

I don't agree with the meaning I get from this statement. The problem is 
that the construct that ABNF calls "LWSP" causes problems in protocols that 
use it.
This problem is independent of the name of the construct; the problem is in 
defining a grammar where the sequence <CRLF><CRLF> has a different meaning 
than <CRLF><SPACE><CRLF>.

Some protocols have addressed this problem by defining their own construct, 
and have used the term "LWSP" to refer to it - something that is legal by 
ABNF rules, but can cause people who read multiple specs to become confused.

There seems to be some reason to think that it's useful to warn people that 
there's reasons not to use the construct that is defined as "LWSP" in the 
ABNF spec. The revised version of the ABNF spec is one possible place to 
put that warning.

> I think that in this instance, the value of future clarity justifies
>  coming up with a new term that will unambiguously mean what LWSP
>  means in ABNF today.  That term will have to start at proposed
>  standard.
>

I agree with the last statement.

> I see a desire to document our operational experience with the word:
> many people took this word and used it to mean something else.
> Perhaps to avoid confusion you should consider whether your use of the
> word is a good idea.  I think there is significant harm in choosing
> not to document this operational experience when advancing standards.
> After all, we both agree that it is this experience with running code
> that gives the IETF value.

Fully agree with this statement too, but since I don't agree with your 
interpretation (as given above) of what the problem is, I can't be sure 
that my agreement with this statement means anything....

                Harald

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 15:50:07 -0400

: <[email-masked]>
X-Spam-Score: 0.1 (/)

	

       john

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 May 2007 09:51:46 -0700

Sam,

> Ultimately cases like this should be evaluated based on whether the
> final result is more clear overall.

What about protecting the installed base for the existing spec?

In other words, your "based on" contains a single criterion, for an 
environment that typically requires multiple.  And the current situation is 
certainly one of those.

Changing the specification makes sense when it does not yet have much 
momentum, or when the problem is basic and severe.

For technology that has been in widespread use for 10-30 years, and an 
extremely small number of problem reports, changing the specification looks 
like exactly the wrong decision.

-- 

   Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 May 2007 09:51:46 -0700

: <[email-masked]>
X-Spam-Score: 0.0 (/)
X-Scan-Signature: ea4ac80f790299f943f0a53be7e1a21a

Sam,

-- 

   Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 May 2007 09:51:46 -0700

: <[email-masked]>
X-Spam-Score: 0.1 (/)

Sam,

-- 

   Dave Crocker

---

From: "todd glassey" <[email-masked]>
Date: Thu, 17 May 2007 10:41:37 -0700

John - I would call this post of yours "Nothing but net" ... more commentary 
inline below.

T.

> specifications.

WOW - talk about being 'right on the money'!

> When we try to avoid that, we get ourselves
> arbitrary,

And this is the operable term here, that being 'arbitrary'

> we make some of these documents Proposed Standards

AMEN!

> live with it.

RIGHT!

> We should certainly revise and replace it if it
> second observation, below)

Or its also brought to our attention that law and process is being violated 
by our methods per those works

> .  But, once other specifications

AMEN!

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 15:32:11 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:

    John> If we are going to standardize a definitional requirement or
    John> method -- whether it is ABNF or IPR boilerplate or something
    John> -- we need to get it right as a self-contained definition
    John> and then live with it.  We should certainly revise and
    John> replace it if it turns out to be unworkable (as has happened
    John> with IPR work) or if the definition turns out to be
    John> inadequate to permit an unambiguous interpretation (that
    John> issue spills over into my second observation, below).  But,
    John> once other specifications start to depend on the definitions
    John> that are there, and show those definitions to be adequate,
    John> we should not be talking about deprecating definitions
    John> unless we are prepared to "that was wrong, we need to start
    John> over (even though some of the older material may still be
    John> useful)".  Again, please note the similarity to the IPR
    John> work.

concept.  The word we chose (LWSP) stuck in some places but not in

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 15:32:11 -0400

: <[email-masked]>
X-Spam-Score: 0.1 (/)
X-Scan-Signature: 769a46790fb42fbb0b0cc700c82f7081

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 15:32:11 -0400

: <[email-masked]>
X-Spam-Score: 0.1 (/)

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 16:18:00 -0400

Harald, I'm happy to accept your interpretation of the problem.
However it also leads me to the conclusion that documenting possible
reasons not to use ABNF's LWSP concept, or documenting implications of
that rule would be a good idea.  I also believe that documenting
experience with a spec in future versions of that spec as it advances
is reasonable.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 16:18:00 -0400

: <[email-masked]>
X-Spam-Score: 0.1 (/)
X-Scan-Signature: 68c8cc8a64a9d0402e43b8eee9fc4199

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 May 2007 14:27:47 -0700

Sam Hartman wrote:
>     >> Ultimately cases like this should be evaluated based on whether
>     >> the final result is more clear overall.
> 
>     Dave> What about protecting the installed base for the existing
>     Dave> spec?
> 
> I think that is not a useful criteria when we are talking about an
> informative note.  I think that criteria matters somewhat more when we
> are talking about depricating a feature but retaining it, although
> even then I think the bar would be reasonably low.  The installed base
> will continue to work.

I think you are assuming a more constrained discussion than what I've been 
seeing on this thread.  The thread has discussed everything from removing the 
rule, to redefining it, to declaring it "deprecated", to adding some 
commentary text.

It appears you are only talking about the last, although I for one missed 
that. (For reference, when I said "change the specification" I mean normative 
change.)

Although I've seen some postings against even having a comment added to the 
text, my own reading of the postings is that there is a reasonable consensus 
that it would be ok.

My own view is that comments can be helpful and are, at worst, typically 
rather benign. Indeed, the IETF approach towards specification writing is 
rather friendly towards including whatever comments folk feel might be 
helpful, modulo the obvious danger that too many comments can wind up 
obscuring a document.  (And, no, I do not think that that is a danger here.)

d/

-- 

   Dave Crocker

---

From: Tony Finch <[email-masked]>
Date: Fri, 18 May 2007 00:38:15 +0100

On Thu, 17 May 2007, John C Klensin wrote:
> [This] group is, IMO, just broken.

I agree with your sentiment but sadly there's a lot of old stuff that's
broken by this criterion. RFC 733 defined <LWSP-char> to mean what RFC
2234 calls <WSP>, and <linear-white-space> to mean what is now <LSWP>.
Fortunately these old definitions have fallen out of use. There's also
HTTP and SIP which call the problematic production <LWS> instead of
<LWSP>. MEGACO uses ABNF with its own terminal definitions instead
of referring to ABNF appendix B. CGI uses its own ABNF variant.

It would be nice if the progression of ABNF to a full standard reduces
this Babel. This implies that (a) ABNF should be used to describe syntax
in preference to any other BNF-alike, to avoid anomalies like CGI; (b)
specifications should not define a production with the same name as one in
appendix B but with a different expansion, to avoid anomalies like MEGACO;
(c) specifications should not define a production with the same expansion
as one in appendix B but with a different name, to avoid anomalies like
HTTP and SIP; (d) any production like LWSP should be discouraged because
of problems with lossage related to trailing whitespace.

Tony.

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 21:14:21 -0400

--On Thursday, 17 May, 2007 21:52 +0200 Harald Tveit Alvestrand
<[email-masked]> wrote:

> I don't agree with the meaning I get from this statement. The
> problem is that the construct that ABNF calls "LWSP" causes
> problems in protocols that use it.
> This problem is independent of the name of the construct; the
> problem is in defining a grammar where the sequence
> <CRLF><CRLF> has a different meaning than <CRLF><SPACE><CRLF>.
>...

Interesting.  I don't think that is a problem with the grammar,
and think it would be rather hard to define a grammar that would
not permit that situation.   After all
    <CRLF> Thing <SPACE><CRLF> could case similar problems if
some construction permitted it and defining a grammar that would
prohibit any <SPACE><CRLF> construction isn't easy in ABNF for
reasons that have nothing to do with LWSP.

Instead, I see the problem as using the grammar to define
situations equivalent to 
    LWSP [ optional-stuff ] CRLF
as compared to 
    LWSP AtLeastOneRequiredThing CRLF
or
    [ LWSP optional-stuff ] CRLF

I don't see either of the latter as problematic.

      john

---

From: Keith Moore <[email-masked]>
Date: Thu, 17 May 2007 21:35:43 -0400

it could be argued that the best thing to do is to remove ALL of the
rules from the ABNF spec, leaving only the language definition and
examples.  (actually I think I did argue this sometime around 1996, but
I'm too lazy to search through old email to find it.  I'm actually
surprised that a problem with one of those definitions has taken this
long to crop up.)

the rules could then be moved to a separate document, probably with
status informational.   I don't personally see any  problem with an
informational document being used to define terms that are referenced in
standards track publications - so long as the document being referenced
is not subject to change.    the referencing document should be
evaluated as if the definitions were incorporated into that document.

IMHO, sometimes we try too hard to make things fit into the standards track.

application of the proposed/draft/full criteria to ABNF has always been
a bit odd, because ABNF defines a notation rather than a protocol.  at
one time we thought about evaluating interoperability of ABNF by seeing
if there were multiple implementations (i.e. parser generators) that
when given the same grammar, recognized the same language.  but this
would be kind of meaningless given that ABNF is (in my experience)
almost never used as input to a parser generator for the purpose of
generating a protocol implementation.  which might be quite unfortunate.
> I think redefining the rule would require recycling at proposed.  I
> think it would be confusing and harmful to do so.
>
> I think removing the rule would is allowed by the process (and would
> require updates in referencing specs as they advance but would not
> break anything).  I think doing so would be harmful and is not
> supported by consensus.
>
> I do not object to deprecating the rule although I'm not completely
> convinced doing so is a great idea.I think it is clearly allowed if
> there is consensus.
>
>

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 18 May 2007 04:35:59 +0200

> I don't see either of the latter as problematic.

Depends on the protocol.  Your constructs match
      SP CRLF SP CRLF SP AtLeastOneRequiredThing CRLF
      SP CRLF SP CRLF SP optional-stuff CRLF
.........^^^^^^^^^^^^
If you do this e.g. in a mail or HHTP header, where
CRLF CRLF means "end of header", then any tool like
a text editor or other UA trying to reuse the header
field "as is" while silently removing trailing white
space would end up with
      CRLF CRLF SP AtLeastOneRequiredThing CRLF
      CRLF CRLF SP optional-stuff CRLF
......^^^^^^^^^

Frank

---

From: Tony Finch <[email-masked]>
Date: Fri, 18 May 2007 09:00:43 +0100

On Thu, 17 May 2007, John C Klensin wrote:
>
> After all <CRLF> Thing <SPACE><CRLF> could case similar problems if
> some construction permitted it ...

This is not news. There have for a long time been problems with
significant trailing space, which is why CRLF 1*WSP CRLF in a header
is part of the obs- syntax of 2822, and why quoted-printable encodes
WSP at the end of a line.

> ... and defining a grammar that would prohibit any <SPACE><CRLF>
> construction isn't easy in ABNF for reasons that have nothing to
> do with LWSP.

This is simply incorrect. It's trivial to define a whitespace
construction that only allows CRLF at the beginning of a sequence:

	NTWSP = [CRLF] 1*WSP ; non-trailing white space

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SHANNON: SOUTHWEST VEERING WEST 7 TO SEVERE GALE 9, PERHAPS STORM 10 LATER.
VERY ROUGH OR HIGH. RAIN OR SQUALLY SHOWERS. MODERATE OR GOOD.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 17:36:08 -0400

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 17:16:12 -0400

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:

    Dave> Sam,

I think that criterian is very important if we were talking about
removing a rule.  For that reason I do not favor removing the LWSP
rule.

---

From: John C Klensin <[email-masked]>
Date: Fri, 18 May 2007 10:03:31 -0400

--On Friday, 18 May, 2007 09:00 +0100 Tony Finch <[email-masked]>
wrote:

>> After all <CRLF> Thing <SPACE><CRLF> could case similar
>> problems if some construction permitted it ...
> 
> significant trailing space, which is why CRLF 1*WSP CRLF in a
> header is part of the obs- syntax of 2822, and why
> quoted-printable encodes WSP at the end of a line.
> 
>> ... and defining a grammar that would prohibit any
>> <SPACE><CRLF> construction isn't easy in ABNF for reasons
>> that have nothing to do with LWSP.
> 

Sure.  Except that much, if not most, of our textual
descriptions of these protocols describes lines, and line-like,
constructions as _ending_ in CRLF.  Moving to  "starting in
CRLF" creates a conceptual difference between prose definition
and formal syntax, which strikes me as a bad idea.   Of course,
in the above, since [CRLF] is optional, "1*WSP" alone satisfies
the production as written and still does not prevent
    <CRLF> space space space 
    <CRLF>
unless _all_ productions are written CRLF first... or one relies
on the comment as a restriction.   And a comment as the
restriction --in the prose-- is exactly what has been suggested,
IMO.

      john

---

From: Tony Finch <[email-masked]>
Date: Fri, 18 May 2007 15:23:12 +0100

On Fri, 18 May 2007, John C Klensin wrote:
> On Friday, 18 May, 2007 09:00 +0100 Tony Finch <[email-masked]>
> wrote:
> >
> and formal syntax, which strikes me as a bad idea.

Don't do that then. The suggestion I made above was for a no-trailing-
whitespace variant of 2822's FWS, and as such it would be used in a
grammar inside a logical line, not at either end. You seem to be inventing
problems that might occur in badly-designed grammars, and as such these
are bugs in the grammars and not due to limitations of ABNF (which, after
all, is equivalent to any other notation for describing context-free
grammars).

> Of course, in the above, since [CRLF] is optional, "1*WSP" alone
> satisfies the production as written and still does not prevent
>    <CRLF> space space space
>    <CRLF>

Um, no, because WSP = SP / HTAB

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: CYCLONIC, BECOMING NORTHERLY GALE 8 OR SEVERE GALE 9,
DECREASING 6 OR 7. ROUGH OR VERY ROUGH, OCCASIONALLY HIGH. SHOWERS. MODERATE
OR GOOD.

---

From: Sam Hartman <[email-masked]>
Date: Fri, 18 May 2007 10:59:51 -0400

>>>>> "Keith" == Keith Moore <[email-masked]> writes:

    Keith> it could be argued that the best thing to do is to remove
    Keith> ALL of the rules from the ABNF spec, leaving only the
    Keith> language definition and examples.  (actually I think I did
    Keith> argue this sometime around 1996, but I'm too lazy to search
    Keith> through old email to find it.  

I think this would be too big of a change going from draft to full
given our experience.  If we had huge tracts of problems with the
rules, it might be a different situation.

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 22 May 2007 16:29:55 -0700

Thanks for everybody's input on this.  I interpret the discussion as  
showing consensus for a comment with a warning near the definition of  
LWSP.

Details:  I counted 18 opinions.  I couldn't see anybody arguing for  
"no comment or text whatsoever".  I saw arguments against treating  
this as a Security Consideration.  I saw opinions in favour of  
"deprecating" the construct, but I am not sure if that's an opinion  
for or against the health warning (since the definition of  
deprecation is loose here).  In any case, even if you count those as  
"votes against" , I still see rough consensus.

Lisa

> crocker-rfc4234bis-00.txt> for publication as Internet Standard and  
> would like to know if there is consensus to recommend against the  
> use of LWSP in future specifications, as it has caused problems  
> recently in DKIM and could cause problems in other places.
>
> command=view_comment&id=66440  (in this tracker comment, Chris  
> Newman recommended to remove LWSP, but for backward-compatibility  
> it's probably better to keep it and recommend against use)
>
> Thanks for your input,
> Lisa Dusseault
>
>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 23 May 2007 09:24:26 +0100

Keith Moore said:
> examples.

While I don't support this, it does remind me of a problem. I've had
various people tell me in the past that "ABNF" includes Appendix B and,
therefore, it is neither necessary to cite the appendix or to define basic
concepts yourself.

I know that section 1 says that appendix B is "separate from its formal
status", but I suggest that the introduction to the appendix should make it
clear that citing ABNF does *not* include these rules by reference; such
inclusion by reference needs to be explicit.