Subject: Use of LWSP in ABNF -- consensus call

From: Lisa Dusseault <[email-masked]>
Date: Mon, 14 May 2007 12:10:03 -0700

The IESG reviewed <http://www.ietf.org/internet-drafts/draft-crocker- 
rfc4234bis-00.txt> for publication as Internet Standard and would  
like to know if there is consensus to recommend against the use of  
LWSP in future specifications, as it has caused problems recently in  
DKIM and could cause problems in other places.

Some discussion on this point already:
  - http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html
  - http://www1.ietf.org/mail-archive/web/discuss/current/msg00463.html
  - http://mipassoc.org/pipermail/ietf-dkim/2007q1/007295.html
  - https://datatracker.ietf.org/public/pidtracker.cgi? 
command=view_comment&id=66440  (in this tracker comment, Chris Newman  
recommended to remove LWSP, but for backward-compatibility it's  
probably better to keep it and recommend against use)

Thanks for your input,
Lisa Dusseault

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Julian Reschke <[email-masked]>
Date: Mon, 14 May 2007 22:55:54 +0200

Lisa Dusseault wrote:
> 
> The IESG reviewed 
> <http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt> 
> for publication as Internet Standard and would like to know if there is 
> consensus to recommend against the use of LWSP in future specifications, 
> as it has caused problems recently in DKIM and could cause problems in 
> other places.
> 
>  - 
> https://datatracker.ietf.org/public/pidtracker.cgi?command=view_comment&id=66440  
> (in this tracker comment, Chris Newman recommended to remove LWSP, but 
> for backward-compatibility it's probably better to keep it and recommend 
> against use)

I agree that LWSP can be problematic. As the LWSP rule only appears in 
appendix B, the best approach IMHO would be to either leave it in, and 
have a warning explaining the potential problems close to it, or remove it.

The latter sounds simpler, but could cause spec writers that use ABNF to 
just copy the LWSP rule from RFC4234, ignoring the potential issues with it.

The proposed solution to warn about it in the front matter doesn't seem 
to be a good idea due to locality reasons. If there are reasons to avoud 
LWSP, they should be stated close to the definition. If this means we 
need a new revision of the spec, and last-call it again, so be it. No 
reason to hurry.

Best regards, Julian

---

From: Dave Crocker <[email-masked]>
Date: Mon, 14 May 2007 15:55:07 -0700

Lisa Dusseault wrote:
> The IESG reviewed 
> <http://www.ietf.org/internet-drafts/draft-crocker-rfc4234bis-00.txt> for 
> publication as Internet Standard and would like to know if there is 

Lisa,

Process:

1   This issue was initially raised in the IESG by Chris Newman, who changed
his Discuss, with a statement that he recommended inserting a comment, along
the lines that others are also recommending.  Unless I've misread the record,
all other votes on advancing ABNF from Draft to Full are positive or neutral,.
except for your own Discuss.  Is this correct?

2.  The ABNF is a candidate for moving from Draft to Full.  Will removing a
rule (that is already in use?) or otherwise changing the semantics of the
specification, at this point, still permit the document to advance?  I had the
impression that moving to Full was based on some serious beliefs about a
specification's being quite stable.  Making this kind of change, this late in
the game, would seem to run counter to that.

> as it has caused problems recently in DKIM and could cause problems in 

Semantics:

      "Could cause problems in other places"...  The DKIM hiccup was the first
one I'd heard about.

      By contrast, "linear-white-space" was defined in RFC733, in 1977, with
RFC822 retaining that definition.  It is defined in those places as
essentially the same as LWSP in the current ABNF Draft Standard specification.

      This seems to be one LWSP problem in 30 years. Is this really a
sufficient basis for changing the semantics of something that has been stable
and widely used for 10-30 years (depending upon how you count)?

      Are we reasonably comfortable that making the change will not create new
and different problems, such as for other uses of ABNF?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 14 May 2007 17:35:46 -0700

On May 14, 2007, at 3:55 PM, Dave Crocker wrote:

>
> Lisa Dusseault wrote:
>> The IESG reviewed <http://www.ietf.org/internet-drafts/draft- 
>> crocker-rfc4234bis-00.txt> for publication as Internet Standard  
>> and would like to know if there is consensus to recommend against  
>> the use of LWSP in future specifications,
>

The issue was initially raised by Frank Ellerman or by various in the  
DKIM WG depending on how you look at it -- Frank explicitly suggested  
possible changes to the draft, in his posting to the IETF list.

You're right about the voting situation but here's the background: I  
took on the DISCUSS myself as a placeholder for an issue that the  
IESG had consensus to investigate further (consensus to investigate  
what the consensus is).   I could have asked somebody else to hold  
the DISCUSS but this seemed most convenient as long as the rest of  
the IESG trusted me to investigate.

> the game, would seem to run counter to that.

Moving to Internet Standard is indeed something we do carefully, and  
of course that means investigating proposed changes to make sure  
they're appropriate, and setting a high bar for accepting them.  I  
believe that's what we're doing here, investigating carefully.

I share your concerns about removing rules that are already in use --  
that would generally be a bad thing.  However I'm interested in the  
consensus around whether a warning or a deprecation statement would  
be a good thing.

Thanks,
Lisa

---

From: Tony Hansen <[email-masked]>
Date: Mon, 14 May 2007 23:20:28 -0400

Lisa Dusseault wrote:
> consensus around whether a warning or a deprecation statement would be a
> good thing.

LWSP has a valid meaning and use, and its being misapplied somewhere
doesn't make that meaning and usage invalid. I could see a note being
added. However, anything more than that is totally inappropriate.

	Tony Hansen
	[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Tue, 15 May 2007 09:10:41 +0100

> essentially the same as LWSP in the current ABNF Draft Standard specification.

The LWSP defined in ABNF is more like the one in HTTP than the message
format one, in that 4234 allows space-only lines (it allows multiple CRLFs
in LWSP) whereas 2822 does not (at most one CRLF in FWS).

There is some documentation of the interoperability problems arising from
the implied-LWS rule in HTTP here:
http://www.and.org/texts/server-http#implicit-lws

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER THAMES: NORTHWEST BACKING SOUTHWEST 4 OR 5, OCCASIONALLY 6
OR 7 AT FIRST. MODERATE, OCCASIONALLY ROUGH IN GERMAN BIGHT. RAIN OR SHOWERS.
MODERATE OR GOOD, OCCASIONALLY POOR.

---

From: Paul Overell <[email-masked]>
Date: Tue, 15 May 2007 10:05:21 +0100

In message <[email-masked]>, Tony Hansen <[email-masked]> writes
>doesn't make that meaning and usage invalid.

Agreed - well put.

> I could see a note being

I would vote against even adding a note.  It seems disproportionate to 
change a 10 year specification at this late stage on the basis of a 
single case of a misapplied, but valid, rule in another specification.

Regards
-- 
Paul Overell         Internet Platform Development Manager, Thus plc

---

From: Spencer Dawkins <[email-masked]>
Date: Tue, 15 May 2007 07:13:49 -0500

I had followed up to Tony's note privately with Tony/Lisa/Dave yesterday, 
but perhaps I should have posted it here. No time like the present.

I agree that technical changes to a specification as it moves from Draft to 
Full does not seem helpful. Although we have darned little experience 
actually DOING this, RFC 2026 says

4.1.3  Internet Standard

   A specification for which significant implementation and successful
   operational experience has been obtained may be elevated to the
   Internet Standard level.  An Internet Standard (which may simply be
   referred to as a Standard) is characterized by a high degree of
   technical maturity and by a generally held belief that the specified
   protocol or service provides significant benefit to the Internet
   community.

So, it seems to me that the bar for changing technical aspects of the 
specification needs to be very high, since this specification has been 
significantly implemented and successfully operated for a while now.

Specifically, a deprecation statement would make sense if LWSP is ALWAYS a 
bad idea, but we don't seem to be saying that. So I would speak against 
deprecating.

I believe that the IETF has a responsibility to implementers. Anytime we 
know there is a dragon, we should say "there be dragons".

So I would speak in favor of adding a warning note that clearly points to a 
dragon when this specification is used in a new specification.

Thanks,

Spencer

p.s. I have to smile a little that we're talking about ABNF being used in 
new protocol specifications after 10-30 years, depending on who is counting. 
I wish ALL of our draft standards, and full standards, were as helpful to 
the Internet community.

From: "Paul Overell" <[email-masked]>

> change a 10 year specification at this late stage on the basis of a single 
> case of a misapplied, but valid, rule in another specification.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 14 May 2007 23:49:44 +0100

> probably better to keep it and recommend against use)

I think it would be better to keep LWSP and recommending against its 
use. Running grep on various RFCs shows that this production is 
referenced in several RFCs, even in some recent ones.

I don't object to Chris' idea to add FWS to the ABNF document.

---

From: John Leslie <[email-masked]>
Date: Tue, 15 May 2007 13:16:32 -0400

Paul Overell <[email-masked]> wrote:
>>> consensus around whether a warning or a deprecation statement would be
>>> a good thing.
>>

   I did some research, and found the following mentions of LWSP:

rfc0733 obs-by rfc0822
rfc0822 defs LWSP-char = SPACE / HTAB   obs-by rfc2822
rfc0987 refs rfc0822   
rfc1138 refs rfc0822
rfc1148 refs rfc0822
rfc1327 refs rfc0822
rfc1486 refs rfc0822
rfc1505 refs rfc0822
rfc1528 refs rfc0822
rfc1848 defs <LWSP-char> ::= SPACE / HTAB
rfc2017 refs rfc0822
rfc2045 refs rfc0822
rfc2046 refs rfc0822   
rfc2110 refs rfc0822
rfc2156 refs rfc0822
rfc2184 refs rfc0822
rfc2231 refs rfc0822
rfc2234 defs LWSP = *(WSP / CRLF WSP)   obs-by rfc4234    
rfc2243 refs rfc0822
rfc2378 defs LWSP-char = SP | TAB
rfc2530 refs rfc2234
rfc2885 defs LWSP = *(WSP / COMMENT / EOL)
rfc3015 defs LWSP = *(WSP / COMMENT / EOL)
rfc3259 defs LWSP = *(WSP / CRLF WSP)
rfc3501 refs rfc2234
rfc3525 defs LWSP = *(WSP / COMMENT / EOL)
rfc3875 defs LWSP = SP | HT | NL
rfc4234 defs LWSP = *(WSP / CRLF WSP)
rfc4646 refs rfc2434

   Based on this, I recommend outright deprecation. The RFC4234
definition is wildly different from the RFC822 usage (which is
substanitally more often referenced): thus any use of it will tend
to confuse. It's also a bit dubious, quite specifically allowing
lines which appear to be blank, but aren't. :^(

   The RFC4234 definition, in fact, is referenced by only 3 RFCs:

RFC2530 Indicating Supported Media Features Using Extensions to DSN and MDN 
RFC3501 INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1
RFC4646 Tags for Identifying Languages

   The use under RFC2530 is a bit vague ("with LWSP wrapping"); likewise
under RFC3501 ("otherwise treat SP as being equivalent to LWSP"). The
use under RFC4646 has caused known problems.

   This would seem to justify deprecation, IMHO.

   YMMV, of course...

--
John Leslie <[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 15 May 2007 20:21:06 +0200

> consensus around whether a warning or a deprecation statement would be 
Removing features that have proved to be a Bad Idea has always been 
listed as one of the possible changes from Proposed to Draft - Draft to 
Full happens so rarely that I would be hesitant to claim that there's 
tradition for such changes there.

Despite this, I agree with the people who think that a warning comment, 
rather than removal of the rule, is the Right Way.

Harald

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 11:27:38 -0700

> tradition for such changes there.

The question is the "proved to be" criterion.

The consensus call was triggered by one documented problem in 10 years.  We've 
had a posting claiming one additional problem (although my own recollection of 
that bit of history was the the list construct was the issue, rather than LWSP.)

So that is a total of at most 2 documented cases in 10-30 years.

And keep in mind that the issue is not that the rule "does not work" but that 
it is very rarely mis-used.

Were we to deprecate every feature in IETF specifications that get 
mis-implemented a couple of times over 10 years, I suspect much of our 
technology would be deprecated...

d/
-- 

   Dave Crocker

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 15 May 2007 20:46:09 +0200

Lisa Dusseault wrote:

> The issue was initially raised by Frank

Hi, a short explanation, initially I hoped that 4234 can be
promoted to STD "as is".  I missed the (now listed) errata
in the "pending errata mbox".

Some months later 4234bis-00 was posted, and if 4234 can't
be promoted as is, then that's an opportunity to address
this (known) LWSP issue.

Just removing it is an idea, but for the reasons stated by
Dave I felt that "just deprecating it" is good enough with
less undesirable side-effects.

After all it's simple to implement LWSP as specified.  But
unfortunately it's also simple to destroy critical white
space in an apparently empty line.  

Sorry for the confusion, I should have checked the pending
errata mbox before the proposal to promote RFC 4234 to STD.

Frank

---

From: Ned Freed <[email-masked]>
Date: Tue, 15 May 2007 12:03:04 -0700 (PDT)

> added. However, anything more than that is totally inappropriate.

Full agreement with Tony here.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 May 2007 15:19:57 -0400

--On Tuesday, 15 May, 2007 12:03 -0700 Ned Freed
<[email-masked]> wrote:

>> Lisa Dusseault wrote:
>> > I share your concerns about removing rules that are already
>> > in use -- that would generally be a bad thing.  However I'm
>> > interested in the consensus around whether a warning or a
>> > deprecation statement would be a good thing.
> 
>> LWSP has a valid meaning and use, and its being misapplied
>> somewhere doesn't make that meaning and usage invalid. I
>> could see a note being added. However, anything more than
>> that is totally inappropriate.
> 

+1

ABNF is simply a tool, a grammar, and a set of definitions. I'd
almost favor a separate applicability statement that encourages
the use of some features and discourages others as appropriate
if that is really needed.  But a particular element should be
removed from the standard only if there is a case to be made
that the definition is inadequate or consistent with other parts
of the model or grammar.   If such inconsistencies actually
existed, ABNF should, IMO, be bounced back to Proposed and
fixed.  Fortunately, that doesn't seem to be the case here, but
I would really question what we are doing to ourselves if our
grammatical definitions start needing profiles

     john

---

From: Tony Finch <[email-masked]>
Date: Tue, 15 May 2007 22:54:18 +0100

> it is very rarely mis-used.

Did you miss my post linking to a description of LWSP-related interop
problems in HTTP? http://www.and.org/texts/server-http#implicit-lws

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTH TYNE DOGGER: WEST 4 OR 5, BECOMING VARIABLE 2, THEN SOUTHERLY 3 OR 4.
SLIGHT OR MODERATE. SHOWERS THEN MAINLY FAIR. GOOD OCCASIONALLY MODERATE AT
FIRST.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 15 May 2007 15:52:40 -0700

> problems in HTTP? http://www.and.org/texts/server-http#implicit-lws

No.

d/
-- 

   Dave Crocker

---

From: Douglas Otis <[email-masked]>
Date: Tue, 15 May 2007 16:00:12 -0700

>    This would seem to justify deprecation, IMHO.

Agreed.

 From a standards standpoint, half a dozen definitions for an ABNF  
mnemonic is absurd.

Perhaps something like:

The LWSP mnemonic has been deprecated and should not be used in  
future drafts.  Explicit definitions based upon different mnemonics  
should be used instead.  If possible, syntax should guard against  
possible security concerns related to visual deceptions.

-Doug

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2007 08:19:55 -0400

--On Tuesday, 15 May, 2007 16:00 -0700 Douglas Otis
<[email-masked]> wrote:

>> under RFC3501 ("otherwise treat SP as being equivalent to
>> LWSP"). The use under RFC4646 has caused known problems.
>> 
>  From a standards standpoint, half a dozen definitions for an
> ABNF mnemonic is absurd.
> 
> Perhaps something like:
> 
> The LWSP mnemonic has been deprecated and should not be used
> in future drafts.  Explicit definitions based upon different
> mnemonics should be used instead.  If possible, syntax should
> guard against possible security concerns related to visual
> deceptions.

Doug, John,

It seems to me that we have two separate issues here (I'm not
even going to go so far as "problems"):

(1) Some documents have used the term LWSP in a way that is not
strictly conformant with the definition in the ABNF document.
Unless the definition is unclear, that is a problem for those
documents -- and the review and approval process that permitted
them to slip through with non-conforming definitions -- and not
for the ABNF spec.   It seems to me we fix that by changing
those documents to use different production names with local
definitions, not by deprecating features of ABNF.

(2) What I learned back when I was doing definitional work for
programming languages was that one wants one's basic syntax and
syntactic metalanguages to be as simple as possible, with a
minimum  of macro constructions and with a minimum of defined
terminals.  
>From that point of view, it is easy to argue that ABNF has just
gotten too complex, both as the result of trying to formalize
some characteristics of 822 while maintaining a single-pass
syntax evaluator and possibly as a second-system effect.   

That, however, is an argument that ABNF itself is broken and
should not be on the standards track.  It is not an argument for
trying to fine-tune it by deprecating a production or two.   The
"broken" argument itself was made by a few of us back when RFC
2234 was being evaluated.  We lost and, at this point, it would
take a lot more than one sometimes-misused construction to
justify tampering with it, even to the extent of selectively
deprecating features.

      john

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 15 May 2007 09:10:53 +0100

+1

Frank's text in
<http://www1.ietf.org/mail-archive/web/ietf/current/msg46048.html>
would be fine:

  Authors intending to use the LWSP (linear white space) construct
  should note that it allows apparently empty lines consisting only
  of trailing white space, semantically different from really empty
  lines.  Some text editors and other tools are known to remove any
  trailing white space silently, and therefore the use of LWSP in
  syntax is not recommended.

However, it doesn't belong in "security considerations".

What about moving LSWP, and this text, to a separate section of Annex B:
"B.3 Deprecated constructs"?

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
THUS plc            |                            |

---

From: Philip Guenther <[email-masked]>
Date: Mon, 14 May 2007 19:59:29 -0600

On Mon, 14 May 2007, Lisa Dusseault wrote:
> publication as Internet Standard and would like to know if there is consensus 
> to recommend against the use of LWSP in future specifications, as it has 
> caused problems recently in DKIM and could cause problems in other places.

LWSP was one of the factors in the SASL WG's abandoning of the DIGEST-MD5 
revision.  To be precise, the complexity inherent in the 822-style n#rule 
syntax (the emulation of which in modern ABNF uses LWSP) was found to be a 
source of implementation errors and interoperability issues.  Part of the 
discussion can be seen at:
    http://www.imc.org/ietf-sasl/mail-archive/msg02752.html

While I don't think removing LWSP from the ABNF RFC is good idea, adding a 
comment to its definition in section B.1 that warns protocol designers to 
consider carefully whether they really need or want "LWSP" instead of 
"*WSP" does seem appropriate.  Calling out that LWSP permits lines 
containing just whitespace may help drive home the issue.

Philip Guenther
Sendmail, Inc

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 15 May 2007 09:10:53 +0100

+1

Frank's text in

---

From: Philip Guenther <[email-masked]>
Date: Mon, 14 May 2007 19:59:29 -0600

Philip Guenther
Sendmail, Inc

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 14 May 2007 19:21:17 -0700

At 5:35 PM -0700 5/14/07, Lisa Dusseault wrote:
>I share your concerns about removing rules that are already in use 
>-- that would generally be a bad thing.  However I'm interested in 
>the consensus around whether a warning or a deprecation statement 
>would be a good thing.

A warning would be very useful, and a deprecation or removal would be 
very bad because either would cause people to go to the earlier 
documents to see what LWSP meant earlier.

---

From: <[email-masked]>
Date: Tue, 15 May 2007 15:32:08 -0400

Add a warning note

Bill Oxley
Messaging Engineer
Cox Communications
404-847-6397
-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Tuesday, May 15, 2007 3:03 PM
To: Tony Hansen
Cc: Apps Discuss; IETF General Discussion Mailing List;
[email-masked]
Subject: [ietf-dkim] Re: Use of LWSP in ABNF -- consensus call

--
> > consensus around whether a warning or a deprecation statement would
be a

				Ned
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Tony Finch <[email-masked]>
Date: Wed, 16 May 2007 13:58:27 +0100

On Wed, 16 May 2007, John C Klensin wrote:
> (2) From that point of view, it is easy to argue that ABNF has just
> gotten too complex, both as the result of trying to formalize some
> characteristics of 822 while maintaining a single-pass syntax evaluator
> and possibly as a second-system effect.

I thought the problem is that protocols that have used LWSP correctly have
had too many interop problems, so they have replaced it with a simpler
rule such as FWS.

I'm surprised you say ABNF has become too complex. It's hardly changed
apart from removal of the # rule, and if you took anything else out it
would lead to rather less readable grammars.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORTH FITZROY SOLE LUNDY FASTNET: WEST OR NORTHWEST 4 OR 5, OCCASIONALLY 6 OR
7. MODERATE, OCCASIONALLY ROUGH. OCCASIONAL RAIN OR DRIZZLE. GOOD BECOMING
MODERATE OR POOR.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 16 May 2007 15:11:18 +0200

I suggest that a version of this sentence goes into 4234bis:
> I thought the problem is that protocols that have used LWSP correctly 
> have had too many interop problems, so they have replaced it with a 
> simpler rule such as FWS.

---

From: Douglas Otis <[email-masked]>
Date: Wed, 16 May 2007 09:25:56 -0700

> However, it doesn't belong in "security considerations".

Discarding of lines is likely in response to some type of exploit.   
The consideration for not using LWSP would be in regard with how  
security requirements may create incompatibilities.  This is the  
correct section.

> "B.3 Deprecated constructs"?

Agreed. That would also be appropriate.

Another problem regarding LWSP is in regard to _many_ differing  
definitions.  A profusion of differing definitions alone becomes a  
valid reason to deprecate the mnemonic.  This definition represents a  
poor practice as related to security which should not be facilitated  
through standardization.  By removing this problematic construct,  
better solutions are more likely to be found.  At least (ab)use of  
the mnemonic will have been discouraged.  Any continued use of this  
mnemonic should be discouraged and the note added should clarify one  
of the reasons for this mnemonic being deprecated is specifically due  
to its varied and checkered meanings in other drafts.

-Doug

---

From: Douglas Otis <[email-masked]>
Date: Wed, 16 May 2007 10:28:15 -0700

On May 16, 2007, at 5:19 AM, John C Klensin wrote:

>
> Doug, John,
>
> It seems to me that we have two separate issues here (I'm not even  
> going to go so far as "problems"):
>
> them to slip through with non-conforming definitions -- and not for  
> the ABNF spec.   It seems to me we fix that by changing those  
> documents to use different production names with local definitions,  
> not by deprecating features of ABNF.

This is not deprecating a "feature" of ABNF other than deprecating a  
specific, confusing, and ill-advised mnemonic found in a pre-defined  
macro library.  As this construct is often ill-advised, there is  
little justification for it to be included in the ABNF pre-defined  
macro library.  This change does not preclude any construct, it only  
affects ease of use.  This should invoke greater care and forethought.

> characteristics of 822 while maintaining a single-pass syntax  
> evaluator and possibly as a second-system effect.

While it is a poor craftsman who blames his tools, it is also  
difficult to justify standardizing a macro construct proven often to  
be problematic.  When an author is looking for a macro, this  
construct and mnemonic should not be within a pre-defined macro  
library.  Exclusion of LWSP does not represent any hardship.

> That, however, is an argument that ABNF itself is broken and should  
> not be on the standards track.  It is not an argument for trying to  
> fine-tune it by deprecating a production or two.   The "broken"  
> argument itself was made by a few of us back when RFC 2234 was  
> being evaluated.  We lost and, at this point, it would take a lot  
> more than one sometimes-misused construction to justify tampering  
> with it, even to the extent of selectively deprecating features.

Although ABNF limitations might lead to sub-optimal syntax, how does  
it prevent optimal syntax from being defined?  This change does not  
tamper with the ABNF language.  It only deprecates a pre-defined  
macro proven problematic and encumbered with differing definitions.   
Any future draft will not find the lack of this macro definition to  
represent any type of hardship.  Those reading IETF drafts however  
will find whatever replaces this macro will be specifically defined  
where security concerns are more likely to have been given greater  
consideration.  Whatever macro then commonly replaces LWSP could be  
given a different mnemonic and added to a subsequent version of this  
draft.

-Doug

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2007 20:20:59 -0400

--On Wednesday, 16 May, 2007 13:58 +0100 Tony Finch
<[email-masked]> wrote:

>...
> I'm surprised you say ABNF has become too complex. It's hardly
> changed apart from removal of the # rule, and if you took
> anything else out it would lead to rather less readable
> grammars.

Tony,

First, my apologies in advance to everyone who doesn't care
about this subject (the change of subject line is a deliberate
clue).  From my point of view, the IETF made its decision around
ten years ago and it is questionable whether the decision then
could sensibly have departed significantly from the definition
is RFC 822.  While I didn't like the decision then and don't
like it now (perhaps the comments below will explain that), I
think it would be a serious mistake to open the question up
again.

This is a matter of taste, and perhaps generational.  I find
grammatical metalanguages more understandable (and readable)
when there is exactly one way to write a given construction (for
example, no implied terms in ranges), when the constructions are
very precisely defined, and when the number of constructions is
kept to a clear minimum.     All things being equal, I like the
ability to construct a two-stage grammar (W-grammar) when that
seems to be justified by circumstances: ABNF does not permit
that, and the omission is one reason we keep having little
messes about comments in protocols whose syntax is defined in
ABNF.  Similarly, I prefer grammars that can be written as a
data stream without depending on line-ending (and indented
continuation) as important indicators.  It is worth noting that
ABNF's line-oriented structure makes ABNF much harder to
construct in xml2rfc than is really reasonable.

I've also got a second problem, which is that I'm more
comfortable with either informal descriptions that use formal
grammars for clarification or explanation for with fully-formal
grammatical descriptions (including semantics) than I am with
the in-between points.  Viewed that way, ABNF, at least as the
IETF tries to use it, tends to encourage those in-between
definitions: harder to understand than an informal description
that relies a little bit less on syntax and more on prose, but
not nearly as precise as a formal semantic definition that
largely eliminates the need for the prose.   This is definitely
a personal idiosyncrasy: YMMD and probably does.

For those of you who are interested in what a syntax-only
metalanguage would look like that would better meet my personal
criteria, I'd encourage taking a look at ISO/IEC ISO/IEC
14977:1996 "Information technology -- Syntactic metalanguage --
Extended BNF".  It is a bit earlier than, but roughly
contemporary with, RFC 2234 and similar in enough ways that one
of them should probably be referencing the other (if one can
guess from timing, the ISO/BSI document should be acknowledging
RFC 822).   Unlike the ISO (and ISO/IEC JTC1) norms, this one is
actually freely available, see the link from
http://isotc.iso.org/livelink/livelink/fetch/2000/2489/Ittf_Home/ITTF.htm
For comparison purposes, while the introductory material is
interesting and helpful, the specification itself is only ten
pages long (although one needs to adjust for smaller type and
more page layout), including examples, and defines the
metalanguage three different ways.

But, again, this is all a matter of theorizing: I think it would
be a terrible idea to try to redesign or replace ABNF at this
point.

     john

---

From: Douglas Otis <[email-masked]>
Date: Wed, 16 May 2007 17:21:35 -0700

In response to off-line comments,

Although LWSP has been placed within "core rules", LWSP is _not_ a  
rule core to the ABNF definition of ABNF.   LWSP is _not_ essential.   
Deprecating this macro does _not_ impact the definition of ABNF.   
This macro can be deprecated to ensure it will not promote use of  
this construct, nor should this macro be used to supplant other  
definitions.  The LWSP jersey can be retired without damaging the  
definition of ABNF or otherwise limiting the future use of ABNF.

-Doug

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2007 20:47:06 -0400

--On Wednesday, 16 May, 2007 17:21 -0700 Douglas Otis
<[email-masked]> wrote:

> Although LWSP has been placed within "core rules", LWSP is
> _not_ a rule core to the ABNF definition of ABNF.   LWSP is
> _not_ essential.  Deprecating this macro does _not_ impact the
> definition of ABNF.  This macro can be deprecated to ensure it
> will not promote use of this construct, nor should this macro
> be used to supplant other definitions.  The LWSP jersey can be
> retired without damaging the definition of ABNF or otherwise
> limiting the future use of ABNF.

Doug, if people want to do it, I would have no problems adding a
comment to any construction (including built-in productions) in
ABNF that has proven dangerous warning people that they should
understand it and its consequences before they use it.  I would
have no problems if that note made it clear that use of LWSP in
a context in which it could end up on a line by itself (in a
context in which "lines" are significant) can be particularly
problematic.

I see those options as very different from deprecating something
that is used successfully and correctly in a number of standards
and incorporated into them by reference.   Since it is in use,
and the definition is actually quite clear, deprecating it seems
completely inappropriate.

     john

---

From: Douglas Otis <[email-masked]>
Date: Wed, 16 May 2007 18:38:59 -0700

On May 16, 2007, at 5:47 PM, John C Klensin wrote:

> I would
> have no problems if that note made it clear that use of LWSP in a  
> context in which it could end up on a line by itself (in a context  
> in which "lines" are significant) can be particularly problematic.
>
> and incorporated into them by reference.   Since it is in use, and  
> the definition is actually quite clear, deprecating it seems  

There is no single definition for LWSP across many documents.  A  
statement that both warns about possible adverse effects and that a  
different label should be considered for new drafts solves both the  
problem of conflicting definitions AND inappropriate promotion of  
this construct.  A statement that LWSP is deprecated does not imply  
it is now obsolete.   Other documents which have referenced this  
document will simply find the definition deprecated.  New drafts  
should consider either creating a more appropriate definition, or use  
a different mnemonic with their desired definition.

-Doug

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 12:17:35 -0400

--On Tuesday, 15 May, 2007 11:27 -0700 Dave Crocker
<[email-masked]> wrote:

> Were we to deprecate every feature in IETF specifications that
> get mis-implemented a couple of times over 10 years, I suspect
> much of our technology would be deprecated...

IMO, and at the risk of again agreeing with Dave, this is the
issue for me.  

If we have inconsistent uses of terminology across documents
that are supposed to be using the same, standardized, term, then
that is a problem with our review process.  If the term is
explicitly standardized in one of the documents, that is the
definition; things that use the term incorrectly should be
candidates for fixing.

By contrast, if we consider "misused sometimes" or "ambiguous
with other uses" as a sufficient condition for deprecating the
term itself, then we have a long list of terms to deprecate in
front of us, almost certainly starting with "IP", which refers
to several different protocols, a protocol layer, and something
that often involves lawyers.

I think some warning language about safe and unsafe contexts may
be in order for this construction but I expect that everyone who
is arguing for deprecating it entirely (or inserting a strong
"don't use this" statement) will be making a case for similar
language the next time an IPv6 document comes up for review.

       john

---

From: Tony Finch <[email-masked]>
Date: Thu, 17 May 2007 17:29:36 +0100

On Wed, 16 May 2007, John C Klensin wrote:
>
> I find grammatical metalanguages more understandable (and readable) when
> there is exactly one way to write a given construction (for example, no
> implied terms in ranges), when the constructions are very precisely
> defined, and when the number of constructions is kept to a clear
> minimum.

However you then go on to say that you like W-grammars, which tend to be
highly arcane, and ISO 14977, which includes lots of alternate puctuation
symbols. In fact, ISO 14977 has other disadvantages compared to RFC 4234:
it has a more cluttered syntax and weaker repetition operators. It also
has an exception operator which is extremely tricky: it has to be limited
in a clever language-theoretical way that is necessary to keep grammars
context-free, and it's very difficult to implement in a parser (almost no
parser-generators have a similar operator). By contrast, all the operators
of RFC 4234 are easy to implement in a parser or to desugar into what your
favourite parser generator supports.

> It is worth noting that ABNF's line-oriented structure makes ABNF much
> harder to construct in xml2rfc than is really reasonable.

That's a strange thing to say. The only place ABNF has strong opinions
about whitespace is at the end of a rule or a comment, where it requires
a newline that any sensible person would include anyway.

> Viewed that way, ABNF, at least as the IETF tries to use it, tends to
> encourage those in-between definitions: harder to understand than an
> informal description that relies a little bit less on syntax and more on
> prose, but not nearly as precise as a formal semantic definition that
> largely eliminates the need for the prose.

I tend to agree that the semi-formal use of ABNF is unhelpful - e.g. the
treatment of LWS in HTTP, where some grammar productions have implied *LWS
between tokens and some do not, making the grammar useless as a complete
formal description of the syntax. However this does not imply that "ABNF
is broken and should not be on the standards track", and switching to a
different formal syntax would not fix this problem. (Especially one like
ISO 14977 which is essentially equivalent to ABNF.)

The problem that started this thread is that the LWSP rule inherited from
the old message header format is too liberal. This is a problem for
protocols that choose to use this syntax, independenly of whether they
describe the syntax in ABNF. It would help future users of ABNF if the
specification did not implicitly endorse syntax that we now know to be
unwise.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
TYNE DOGGER: VARIABLE 3 OR 4 BECOMING SOUTHWEST 5 OR 6. SLIGHT OR MODERATE.
OCCASIONAL RAIN. MODERATE OR GOOD.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 12:42:00 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:

    John> --On Tuesday, 15 May, 2007 11:27 -0700 Dave Crocker

    >> much of our technology would be deprecated...

    John> IMO, and at the risk of again agreeing with Dave, this is
    John> the issue for me.

    John> If we have inconsistent uses of terminology across documents
    John> that are supposed to be using the same, standardized, term,
    John> then that is a problem with our review process.  If the term
    John> is explicitly standardized in one of the documents, that is
    John> the definition; things that use the term incorrectly should
    John> be candidates for fixing.

I don't see why the standardized definition is the obvious right place
to fix things.  I thought we were committed to running code.  To me,
one implication of that commitment is that sometimes the right fix is
for the spec to change rather than the implementations.

In a terminology conflict, this often involves moving away from the
term that has two conflicting uses to terms that are more clear.

Ultimately cases like this should be evaluated based on whether the
final result is more clear overall.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 May 2007 09:51:46 -0700

Sam,

> final result is more clear overall.

What about protecting the installed base for the existing spec?

In other words, your "based on" contains a single criterion, for an 
environment that typically requires multiple.  And the current situation is 
certainly one of those.

Changing the specification makes sense when it does not yet have much 
momentum, or when the problem is basic and severe.

For technology that has been in widespread use for 10-30 years, and an 
extremely small number of problem reports, changing the specification looks 
like exactly the wrong decision.

-- 

   Dave Crocker

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 17 May 2007 18:54:30 +0200

John C Klensin wrote:

> From my point of view, the IETF made its decision around ten
> years ago and it is questionable whether the decision then

Sorry, I don't understand this statement.  If something's
wrong it has to be fixed, no matter how long it has been wrong.

>From my POV it's obvious that standards based on 4234 or its
predecessor avoiding LWSP like 2822 are good examples, while
something like 2616 or 2831 not using ABNF but the #-construct
with its implicit LWSP are an interoperability nightmare.

The 2616 example is horrible.  The brave folks trying to write
a 2616bis will be forced to reconstruct a syntax from scratch,
using a proper notation like 4234.

> I prefer grammars that can be written as a data stream without
> depending on line-ending (and indented continuation) as
> important indicators.

You're free to use something like XML if you prefer that.  It's
not the fault of ABNF that there's a line length limit of about
69 characters in RFCs, and it offers the usual workaround of
"folded" lines line-oriented constructs like mail headers, or
(no surprise) ABNF.

BTW, please stay away from those pseudo-IRIs in XML 1.0 3rd ed.
or worse, I'm not going to implement this anytime soon.  What
I mean is "other grammars => other warts", not only LWSP can
have subtle and ugly side-effects.

> It is worth noting that ABNF's line-oriented structure makes
> ABNF much harder to construct in xml2rfc than is really
> reasonable.

IMO that's not the case, using something like...

</t><figure><artwork type="abnf">
    your-text       = "here"
    group-char      = ALPHA / DIGIT / "-" / "+" / "_" / "."
</artwork></figure><t>

...is a no-brainer.  After you found that xml2rfc forces you
to start the closing </artwork> in column one to get it right.

Otherwise simply omit the 'type="abnf"' blurb and use Bill's
validator instead of the built-in validator for 'type="abnf"'.

> I'm more comfortable with either informal descriptions that
> use formal grammars for clarification or explanation for
> with fully-formal grammatical descriptions (including
> semantics) than I am with the in-between points.

Matter of taste, I look at the syntax, if that's not clear I
look at examples, and if it's still unclear I'd never bother
to read the prose.  Especially prose using MUSTard in almost
all statements is a royal pain:  Try to derive a conformance
test suite or an implementation and interoperability report,
what you end up with could be writing a "requirements" memo.

> This is definitely a personal idiosyncrasy: YMMD and
> probably does.

Yes,  If you'd prefer to write 2821bis in SDL let's try it,
nobody said that ABNF is a holy grail or the only game in
town.  It's merely better than no syntax at all.

> I'd encourage taking a look at ISO/IEC ISO/IEC 14977:1996
> "Information technology -- Syntactic metalanguage --
> Extended BNF".

Why is it that I fear that there's no URL for a plain text
or simple HTML version of this text ?  If that's the EBNF
used by the W3C then it's not really "better" than ABNF, it
is only slightly different.  Isn't EBNF also line-oriented ?

At the end of the day you'd want to put it into an US ASCII
Internet-Draft, and discuss your syntax on a mailing list.

> this one is actually freely available, see the link from

I get some frames with my browser, and none of these frames
mentions EBNF or 14977.  OTOH if I type "rfc 4234" anywhere
on a command line I get what want.  I can also list short
URLs for 4234, resulting in no nonsense documents with links
to the plain text RFC, errata, obsoleted RFCs, etc.:

http://purl.net/net/rfc/4234
http://tools.ietf.org/html/rfc4234

> the specification itself is only ten pages long (although
> one needs to adjust for smaller type and more page layout),
> including examples, and defines the metalanguage three
> different ways.

If you strip boilerplates, page headers, and runs of empty
lines from 4234 you'd end up with (guessing) 600 lines.
It's a short and sweet spec. ignoring the LWSP issue.

Frank

P:S.:  What was the outcome of the "cosmogol" BOF ?

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 13:35:31 -0400

--On Thursday, 17 May, 2007 12:42 -0400 Sam Hartman
<[email-masked]> wrote:

> I don't see why the standardized definition is the obvious
> right place to fix things.  I thought we were committed to
> running code.  To me, one implication of that commitment is
> that sometimes the right fix is for the spec to change rather
> than the implementations.
> 
> In a terminology conflict, this often involves moving away
> from the term that has two conflicting uses to terms that are
> more clear.
> 
> Ultimately cases like this should be evaluated based on
> whether the final result is more clear overall.

Sam,

Two observations; I hope you don't think they are contradictory.

(1) We regularly get ourselves into intellectual and procedural
difficulties by treating specifications about how protocol
specifications are written as if they were protocol
specifications.  When we try to avoid that, we get ourselves
into worse problems.  Using rules that are more or less
arbitrary, we make some of these documents Proposed Standards
and then try to progress them, we make others into BCPs and,
now, we make still others into IONs.

If we are going to standardize a definitional requirement or
method -- whether it is ABNF or IPR boilerplate or something --
we need to get it right as a self-contained definition and then
live with it.  We should certainly revise and replace it if it
turns out to be unworkable (as has happened with IPR work) or if
the definition turns out to be inadequate to permit an
unambiguous interpretation (that issue spills over into my
second observation, below).  But, once other specifications
start to depend on the definitions that are there, and show
those definitions to be adequate, we should not be talking about
deprecating definitions unless we are prepared to "that was
wrong, we need to start over (even though some of the older
material may still be useful)".    Again, please note the
similarity to the IPR work.

(2) If we pretend that the ABNF metalanguage and definitions are
actually a protocol specification, then we need to evaluate it
as one.  Then, we have the following criteria (which we usually
don't state quite this precisely):

	(i) Is the definition good enough that interoperable
	implementations are possible?
	
	(ii) Do people care enough about the construct to
	actually use it in ways that show it is useful?

Now, neither of those rules prevents non-conforming
"implementations".  We may notice that those exist, but our
concern is only about implementations that appear to conform to
the spec and are (or are not) interoperable.  If non-conforming
implementations happen by accident because the text isn't clear
enough, we try to clarify the text.   But we don't say "well,
there are non-conforming implementations, so the spec is
broken".   That would make no sense at all, at least to me.

The answer to (i) appears to be "yes".  There are lots of
conforming cases.  And the answer to (ii) is, as Dave as pointed
out repeatedly, "about 30 years worth".

Is this construction dangerous if used in inappropriate
contexts?  Sure.  Does that justify a warning note to the
unwary?  Probably.  Is it possible to implement other things and
call them by the same name (i.e., create a non-conforming
implementation)?  Of course.  Should that invalidate the
definition?  Not if we want to have anything left if the
principle were applied broadly.

     john

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 17 May 2007 19:19:54 +0200

John C Klensin wrote:

> I would have no problems adding a comment to any construction
> (including built-in productions) in ABNF that has proven
> dangerous warning people that they should understand it and
> its consequences before they use it.

That's the intention of the proposed "security considerations".
It could be placed elsewhere, or reduced to "there be dragons"
if that's appropriate language for an STD (I'd like it), these
proposals have the same effect of deprecating LWSP from my POV.

> and incorporated into them by reference.

Then our terminology of "deprecating" is different, but neither
"there be dragons" nor the longer security blurb used the verb
"deprecate" explicitly.

> Since it is in use

Do you know a single case of a x234-LWSP intentionally allowing
"apparently empty lines" ?  I'd guess that authors just copied
it without considering this effect, or because their protocol
doesn't need any "really empty lines" with a different meaning.

While I'm at it, the 4234 definition of LWSP doesn't use the
grouping notation recommended in 4234:  *(WSP / CRLF WSP) is
the same as the <q>strongly advised</q> *(WSP / (CRLF WSP))

Frank

---

From: Tony Finch <[email-masked]>
Date: Thu, 17 May 2007 18:48:07 +0100

+1

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL: SOUTHWEST 6 TO GALE 8, INCREASING SEVERE GALE 9, PERHAPS STORM 10
LATER. VERY ROUGH OR HIGH. SHOWERS. GOOD.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 17 May 2007 11:12:22 -0700

On May 17, 2007, at 9:29 AM, Tony Finch wrote:

> It would help future users of ABNF if the specification did not  
> implicitly endorse syntax that we now know to be unwise.

+1

Especially when not germane to ABNF definitions.  The construct  
should stand on its own when used.  Perhaps labeled as  
ThereBeDragons. :)

-Doug

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 15:32:11 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:

    John> If we are going to standardize a definitional requirement or
    John> method -- whether it is ABNF or IPR boilerplate or something
    John> -- we need to get it right as a self-contained definition
    John> and then live with it.  We should certainly revise and
    John> replace it if it turns out to be unworkable (as has happened
    John> with IPR work) or if the definition turns out to be
    John> inadequate to permit an unambiguous interpretation (that
    John> issue spills over into my second observation, below).  But,
    John> once other specifications start to depend on the definitions
    John> that are there, and show those definitions to be adequate,
    John> we should not be talking about deprecating definitions
    John> unless we are prepared to "that was wrong, we need to start
    John> over (even though some of the older material may still be
    John> useful)".  Again, please note the similarity to the IPR
    John> work.

Right.  Here, I don't think the definition is wrong, I just think the
term being defined is wrong.  We proposed a definition for a useful
concept.  The word we chose (LWSP) stuck in some places but not in
others.  IN fact other people used the same word for a different
although related concept.  sufficiently so that the definition we
proposed in ABNF is not the most common definition in our standards.

Clearly we should not invalidate existing uses of that term.  Clearly
we do need a definition for the term: it is being used usefully.

I think that in this instance, the value of future clarity justifies
 coming up with a new term that will unambiguously mean what LWSP
 means in ABNF today.  That term will have to start at proposed
 standard.

LWSP will need to continue in ABNF.

I see a desire to document our operational experience with the word:
many people took this word and used it to mean something else.
Perhaps to avoid confusion you should consider whether your use of the
word is a good idea.  I think there is significant harm in choosing
not to document this operational experience when advancing standards.
After all, we both agree that it is this experience with running code
that gives the IETF value.

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 15:50:07 -0400

Sam, with one small exception, I think we are in complete
agreement.  The exception is noted below...

--On Thursday, 17 May, 2007 15:32 -0400 Sam Hartman
<[email-masked]> wrote:

> Right.  Here, I don't think the definition is wrong, I just
> think the term being defined is wrong.  We proposed a
> definition for a useful concept.  The word we chose (LWSP)
> stuck in some places but not in others.  IN fact other people
> used the same word for a different although related concept.
> sufficiently so that the definition we proposed in ABNF is not
> the most common definition in our standards.

Counting uses in this way so as to determine what is or is not
the "most common definition" is as tricky in our environment as
counting votes.  For the same reason, I'd rather that we avoid
it when we can.   However, if we do need to count, I think we
have to use some sort of system that attributes more weight to
documents that use the terminology that have been, effectively,
at full standard before some of the specifications that use
different definitions were even dreamt of.  I don't know how to
assign those weights, which, recursively, is why I don't like
counting.   But I don't think you can count up a handful of
recent misuses and then make an inference that the base
definition needs to be retired (which you haven't done, but
several others have), or even that it is not appropriately
"common".

> I think that in this instance, the value of future clarity
> justifies  coming up with a new term that will unambiguously
> mean what LWSP  means in ABNF today.  That term will have to
> start at proposed  standard.

Pragmatically, perhaps yes.  But what I keep coming back to is
that it appears to me to be perfectly clear and unambiguous
"what LWSP means in ABNF today".  That is what is written into
the spec.  As far as I can tell, we are having a discussion
about two other things:

	(1) Other specifications that use the term "LWSP" to
	refer to something different from what is unambiguously
	defined in the ABNF spec.
	
	(2) Places where people might be tempted to use LWSP but
	where they have discovered that LWSP is either
	inappropriate or risky.

The second group clearly needs new and appropriate terminology
for what they do need and use.   The first group is, IMO, just
broken.

> I see a desire to document our operational experience with the
> word: many people took this word and used it to mean something
> else. Perhaps to avoid confusion you should consider whether
> your use of the word is a good idea.  I think there is
> significant harm in choosing not to document this operational
> experience when advancing standards. After all, we both agree
> that it is this experience with running code that gives the
> IETF value.

Again, I have no problems with making carefully-considered
comments about use and misuse.  I think it is a fine idea as
long as those comments do not imply that the misuses are a
failure in the base specification.  And I think everyone should
consider a stopping rule, lest, as I indicated in my
deliberately extreme example, we discover that we want to
retire, replace, or update "IP" because of some use in the legal
community that precedes the use in network protocols by many
years.

       john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 17 May 2007 21:52:45 +0200

--On 17. mai 2007 15:32 -0400 Sam Hartman <[email-masked]> wrote:

> concept.

Actually we defined a concept (LWSP) in a way that turned out to be much 
more
troublesome in several contexts than we thought it would be (because it 
allowed
two different versions of "blank" lines to have different semantics).

>  The word we chose (LWSP) stuck in some places but not in

I don't agree with the meaning I get from this statement. The problem is 
that the construct that ABNF calls "LWSP" causes problems in protocols that 
use it.
This problem is independent of the name of the construct; the problem is in 
defining a grammar where the sequence <CRLF><CRLF> has a different meaning 
than <CRLF><SPACE><CRLF>.

Some protocols have addressed this problem by defining their own construct, 
and have used the term "LWSP" to refer to it - something that is legal by 
ABNF rules, but can cause people who read multiple specs to become confused.

There seems to be some reason to think that it's useful to warn people that 
there's reasons not to use the construct that is defined as "LWSP" in the 
ABNF spec. The revised version of the ABNF spec is one possible place to 
put that warning.

> LWSP will need to continue in ABNF.

I agree with the last statement.

> that gives the IETF value.

Fully agree with this statement too, but since I don't agree with your 
interpretation (as given above) of what the problem is, I can't be sure 
that my agreement with this statement means anything....

                Harald

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 16:18:00 -0400

Harald, I'm happy to accept your interpretation of the problem.
However it also leads me to the conclusion that documenting possible
reasons not to use ABNF's LWSP concept, or documenting implications of
that rule would be a good idea.  I also believe that documenting
experience with a spec in future versions of that spec as it advances
is reasonable.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 17:16:12 -0400

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:

    Dave> Sam,
    >> Ultimately cases like this should be evaluated based on whether
    >> the final result is more clear overall.

    Dave> What about protecting the installed base for the existing
    Dave> spec?

I think that is not a useful criteria when we are talking about an
informative note.  I think that criteria matters somewhat more when we
are talking about depricating a feature but retaining it, although
even then I think the bar would be reasonably low.  The installed base
will continue to work.

I think that criterian is very important if we were talking about
removing a rule.  For that reason I do not favor removing the LWSP
rule.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 17 May 2007 14:27:47 -0700

> even then I think the bar would be reasonably low.  The installed base

I think you are assuming a more constrained discussion than what I've been 
seeing on this thread.  The thread has discussed everything from removing the 
rule, to redefining it, to declaring it "deprecated", to adding some 
commentary text.

It appears you are only talking about the last, although I for one missed 
that. (For reference, when I said "change the specification" I mean normative 
change.)

Although I've seen some postings against even having a comment added to the 
text, my own reading of the postings is that there is a reasonable consensus 
that it would be ok.

My own view is that comments can be helpful and are, at worst, typically 
rather benign. Indeed, the IETF approach towards specification writing is 
rather friendly towards including whatever comments folk feel might be 
helpful, modulo the obvious danger that too many comments can wind up 
obscuring a document.  (And, no, I do not think that that is a danger here.)

d/

-- 

   Dave Crocker

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 May 2007 17:36:08 -0400

I think redefining the rule would require recycling at proposed.  I
think it would be confusing and harmful to do so.

I think removing the rule would is allowed by the process (and would
require updates in referencing specs as they advance but would not
break anything).  I think doing so would be harmful and is not
supported by consensus.

I do not object to deprecating the rule although I'm not completely
convinced doing so is a great idea.I think it is clearly allowed if
there is consensus.

---

From: Tony Finch <[email-masked]>
Date: Fri, 18 May 2007 00:38:15 +0100

On Thu, 17 May 2007, John C Klensin wrote:
> [This] group is, IMO, just broken.

I agree with your sentiment but sadly there's a lot of old stuff that's
broken by this criterion. RFC 733 defined <LWSP-char> to mean what RFC
2234 calls <WSP>, and <linear-white-space> to mean what is now <LSWP>.
Fortunately these old definitions have fallen out of use. There's also
HTTP and SIP which call the problematic production <LWS> instead of
<LWSP>. MEGACO uses ABNF with its own terminal definitions instead
of referring to ABNF appendix B. CGI uses its own ABNF variant.

It would be nice if the progression of ABNF to a full standard reduces
this Babel. This implies that (a) ABNF should be used to describe syntax
in preference to any other BNF-alike, to avoid anomalies like CGI; (b)
specifications should not define a production with the same name as one in
appendix B but with a different expansion, to avoid anomalies like MEGACO;
(c) specifications should not define a production with the same expansion
as one in appendix B but with a different name, to avoid anomalies like
HTTP and SIP; (d) any production like LWSP should be discouraged because
of problems with lossage related to trailing whitespace.

Tony.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 17 May 2007 17:05:59 -0700

On May 17, 2007, at 2:27 PM, Dave Crocker wrote:

> I think you are assuming a more constrained discussion than what  
> I've been seeing on this thread.  The thread has discussed  
> everything from removing the rule, to redefining it, to declaring  
> it "deprecated", to adding some commentary text.

I think that only the later was suggested.  Declaring the macro  
"deprecated" would not remove the definition or redefine it.  Any  
additional comments should simply indicate this construct should be  
used with caution and defined locally with a different mnemonic.  On  
the bright side, this does not impact the ABNF definition.

-Doug

---

From: John C Klensin <[email-masked]>
Date: Thu, 17 May 2007 21:14:21 -0400

--On Thursday, 17 May, 2007 21:52 +0200 Harald Tveit Alvestrand
<[email-masked]> wrote:

> I don't agree with the meaning I get from this statement. The
> problem is that the construct that ABNF calls "LWSP" causes
> problems in protocols that use it.
> This problem is independent of the name of the construct; the
> problem is in defining a grammar where the sequence
> <CRLF><CRLF> has a different meaning than <CRLF><SPACE><CRLF>.
>...

Interesting.  I don't think that is a problem with the grammar,
and think it would be rather hard to define a grammar that would
not permit that situation.   After all
    <CRLF> Thing <SPACE><CRLF> could case similar problems if
some construction permitted it and defining a grammar that would
prohibit any <SPACE><CRLF> construction isn't easy in ABNF for
reasons that have nothing to do with LWSP.

Instead, I see the problem as using the grammar to define
situations equivalent to 
    LWSP [ optional-stuff ] CRLF
as compared to 
    LWSP AtLeastOneRequiredThing CRLF
or
    [ LWSP optional-stuff ] CRLF

I don't see either of the latter as problematic.

      john

---

From: Keith Moore <[email-masked]>
Date: Thu, 17 May 2007 21:35:43 -0400

it could be argued that the best thing to do is to remove ALL of the
rules from the ABNF spec, leaving only the language definition and
examples.  (actually I think I did argue this sometime around 1996, but
I'm too lazy to search through old email to find it.  I'm actually
surprised that a problem with one of those definitions has taken this
long to crop up.)

the rules could then be moved to a separate document, probably with
status informational.   I don't personally see any  problem with an
informational document being used to define terms that are referenced in
standards track publications - so long as the document being referenced
is not subject to change.    the referencing document should be
evaluated as if the definitions were incorporated into that document.

IMHO, sometimes we try too hard to make things fit into the standards track.

application of the proposed/draft/full criteria to ABNF has always been
a bit odd, because ABNF defines a notation rather than a protocol.  at
one time we thought about evaluating interoperability of ABNF by seeing
if there were multiple implementations (i.e. parser generators) that
when given the same grammar, recognized the same language.  but this
would be kind of meaningless given that ABNF is (in my experience)
almost never used as input to a parser generator for the purpose of
generating a protocol implementation.  which might be quite unfortunate.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 18 May 2007 04:35:59 +0200

> I don't see either of the latter as problematic.

Depends on the protocol.  Your constructs match
      SP CRLF SP CRLF SP AtLeastOneRequiredThing CRLF
      SP CRLF SP CRLF SP optional-stuff CRLF
.........^^^^^^^^^^^^
If you do this e.g. in a mail or HHTP header, where
CRLF CRLF means "end of header", then any tool like
a text editor or other UA trying to reuse the header
field "as is" while silently removing trailing white
space would end up with
      CRLF CRLF SP AtLeastOneRequiredThing CRLF
      CRLF CRLF SP optional-stuff CRLF
......^^^^^^^^^

Frank

---

From: Tony Finch <[email-masked]>
Date: Fri, 18 May 2007 09:00:43 +0100

On Thu, 17 May 2007, John C Klensin wrote:
>
> After all <CRLF> Thing <SPACE><CRLF> could case similar problems if
> some construction permitted it ...

This is not news. There have for a long time been problems with
significant trailing space, which is why CRLF 1*WSP CRLF in a header
is part of the obs- syntax of 2822, and why quoted-printable encodes
WSP at the end of a line.

> ... and defining a grammar that would prohibit any <SPACE><CRLF>
> construction isn't easy in ABNF for reasons that have nothing to
> do with LWSP.

This is simply incorrect. It's trivial to define a whitespace
construction that only allows CRLF at the beginning of a sequence:

	NTWSP = [CRLF] 1*WSP ; non-trailing white space

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SHANNON: SOUTHWEST VEERING WEST 7 TO SEVERE GALE 9, PERHAPS STORM 10 LATER.
VERY ROUGH OR HIGH. RAIN OR SQUALLY SHOWERS. MODERATE OR GOOD.

---

From: John C Klensin <[email-masked]>
Date: Fri, 18 May 2007 10:03:31 -0400

--On Friday, 18 May, 2007 09:00 +0100 Tony Finch <[email-masked]>
wrote:

>> After all <CRLF> Thing <SPACE><CRLF> could case similar
>> problems if some construction permitted it ...
> 
> significant trailing space, which is why CRLF 1*WSP CRLF in a
> header is part of the obs- syntax of 2822, and why
> quoted-printable encodes WSP at the end of a line.
> 
>> ... and defining a grammar that would prohibit any
>> <SPACE><CRLF> construction isn't easy in ABNF for reasons
>> that have nothing to do with LWSP.
> 

Sure.  Except that much, if not most, of our textual
descriptions of these protocols describes lines, and line-like,
constructions as _ending_ in CRLF.  Moving to  "starting in
CRLF" creates a conceptual difference between prose definition
and formal syntax, which strikes me as a bad idea.   Of course,
in the above, since [CRLF] is optional, "1*WSP" alone satisfies
the production as written and still does not prevent
    <CRLF> space space space 
    <CRLF>
unless _all_ productions are written CRLF first... or one relies
on the comment as a restriction.   And a comment as the
restriction --in the prose-- is exactly what has been suggested,
IMO.

      john

---

From: Tony Finch <[email-masked]>
Date: Fri, 18 May 2007 15:23:12 +0100

On Fri, 18 May 2007, John C Klensin wrote:
> On Friday, 18 May, 2007 09:00 +0100 Tony Finch <[email-masked]>
> wrote:
> >
> and formal syntax, which strikes me as a bad idea.

Don't do that then. The suggestion I made above was for a no-trailing-
whitespace variant of 2822's FWS, and as such it would be used in a
grammar inside a logical line, not at either end. You seem to be inventing
problems that might occur in badly-designed grammars, and as such these
are bugs in the grammars and not due to limitations of ABNF (which, after
all, is equivalent to any other notation for describing context-free
grammars).

> Of course, in the above, since [CRLF] is optional, "1*WSP" alone
> satisfies the production as written and still does not prevent
>    <CRLF> space space space
>    <CRLF>

Um, no, because WSP = SP / HTAB

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: CYCLONIC, BECOMING NORTHERLY GALE 8 OR SEVERE GALE 9,
DECREASING 6 OR 7. ROUGH OR VERY ROUGH, OCCASIONALLY HIGH. SHOWERS. MODERATE
OR GOOD.

---

From: Sam Hartman <[email-masked]>
Date: Fri, 18 May 2007 10:59:51 -0400

>>>>> "Keith" == Keith Moore <[email-masked]> writes:

    Keith> it could be argued that the best thing to do is to remove
    Keith> ALL of the rules from the ABNF spec, leaving only the
    Keith> language definition and examples.  (actually I think I did
    Keith> argue this sometime around 1996, but I'm too lazy to search
    Keith> through old email to find it.  

I think this would be too big of a change going from draft to full
given our experience.  If we had huge tracts of problems with the
rules, it might be a different situation.

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 22 May 2007 16:29:55 -0700

Thanks for everybody's input on this.  I interpret the discussion as  
showing consensus for a comment with a warning near the definition of  
LWSP.

Details:  I counted 18 opinions.  I couldn't see anybody arguing for  
"no comment or text whatsoever".  I saw arguments against treating  
this as a Security Consideration.  I saw opinions in favour of  
"deprecating" the construct, but I am not sure if that's an opinion  
for or against the health warning (since the definition of  
deprecation is loose here).  In any case, even if you count those as  
"votes against" , I still see rough consensus.

Lisa

> crocker-rfc4234bis-00.txt> for publication as Internet Standard and  
> would like to know if there is consensus to recommend against the  
> use of LWSP in future specifications, as it has caused problems  
> recently in DKIM and could cause problems in other places.
>
> command=view_comment&id=66440  (in this tracker comment, Chris  
> Newman recommended to remove LWSP, but for backward-compatibility  
> it's probably better to keep it and recommend against use)
>
> Thanks for your input,
> Lisa Dusseault
>
>

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Wed, 23 May 2007 09:24:26 +0100

Keith Moore said:
> examples.

While I don't support this, it does remind me of a problem. I've had
various people tell me in the past that "ABNF" includes Appendix B and,
therefore, it is neither necessary to cite the appendix or to define basic
concepts yourself.

I know that section 1 says that appendix B is "separate from its formal
status", but I suggest that the introduction to the appendix should make it
clear that citing ABNF does *not* include these rules by reference; such
inclusion by reference needs to be explicit.