Subject: Re: header parsing, trailing OWS

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Oct 2009 17:15:48 +0200

David Morris wrote:
> 
> 
> On Thu, 24 Sep 2009, Julian Reschke wrote:
> 
>> Hi,
>>
>> In the current edits, the last 'MAY' is a 'SHOULD', which makes it read
>>
>> "A field value MAY be preceded by optional whitespace (OWS); a single 
>> SP is preferred. The field value does not include any leading or 
>> trailing white space: OWS occurring before the first non-whitespace 
>> character of the field value or after the last non-whitespace 
>> character of the field value is ignored and SHOULD be removed without 
>> changing the meaning of the header field."
> 
> 
> Doesn't read smoothly .... and infact turns into a directive rather than 
> a permission.
> 
> One alternate to illustrate my point ...
>    "SHOULD be removed" --> "SHOULD be able to"
> another ... replace the remainder after "the field value is ignored and"
> with:
>     "removing OWS before or after the field value SHOULD NOT change the
>      meaning of the header field."

I think we should just say:

"OWS occurring before the first non-whitespace character of the field 
value or after the last non-whitespace character of the field value is 
ignored and can be removed without changing the meaning of the header 
field."

...replacing the MAY in draft 07, and the SHOULD in the current edits, 
by "can". RFC2119 terminology is not needed here.

BR, Julian

---

From: David Morris <[email-masked]>
Date: Wed, 7 Oct 2009 12:35:09 -0700 (PDT)

On Wed, 7 Oct 2009, Julian Reschke wrote:
>>> "A field value MAY be preceded by optional whitespace (OWS); a single SP 
>>> is preferred. The field value does not include any leading or trailing 
>>> white space: OWS occurring before the first non-whitespace character of 
>>> the field value or after the last non-whitespace character of the field 
>>> value is ignored and SHOULD be removed without changing the meaning of the 
>>> header field."
>> 
>> 
>> Doesn't read smoothly .... and infact turns into a directive rather than a 
>> permission.
>> 
> "OWS occurring before the first non-whitespace character of the field value 
> or after the last non-whitespace character of the field value is ignored and 
> can be removed without changing the meaning of the header field."
>
> ...replacing the MAY in draft 07, and the SHOULD in the current edits, by 
> "can". RFC2119 terminology is not needed here.

It seems to me that by not being more explicit, we harm interoperability.

It seems obvious to me with your last proposal, that removing 
excess whitespace before interpreting a value is the only sensible 
interpretation, but I've seen enough sloppy coding to believe, that the 
proposed wording would be interpreted as one can ignore the existance of 
excess white space and use the value, white space and all in some 
comparison. Thus I'm in favor of as strong a wording as we can use at this 
stage of the progression to standard to say that excess white space MUST 
be removed for interpreting the value.

Perhaps introduce the notion of canonical form for headers and values and 
require conversion to canonical form before processing the values in any 
context where the outcome would be different based on the existance of 
extraneous white space. Including (but not limited to) computation of a 
digest of header values, interptreting values, etc.

Dave Morris

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 08 Oct 2009 10:40:44 +1300

I think it must always be possible for an agent to remove the OWS 
without breaking the meaning of the header.  We can't really allow a 
situation where the OWS has some meaning, or is relied on, since that 
then breaks all manner of things (e.g. such as comparison of selecting 
headers for caches).

If we take that to its logical conclusion there should be

a) a MUST level requirement for implementations to not generate 
extraneous OWS
b) a SHOULD level requirement for intermediaries to remove extraneous OWS
c) a requirement for caches and clients to remove OWS when using headers 
- hard to tell whether should be SHOULD or MUST.

regards

Adrien

David Morris wrote:
>>>> "A field value MAY be preceded by optional whitespace (OWS); a 
>>>> single SP is preferred. The field value does not include any 
>>>> leading or trailing white space: OWS occurring before the first 
>>>> non-whitespace character of the field value or after the last 
>>>> non-whitespace character of the field value is ignored and SHOULD 
>>>> be removed without changing the meaning of the header field."
>>>
>>>
>>> Doesn't read smoothly .... and infact turns into a directive rather 
>>> than a permission.
>>>
>> value or after the last non-whitespace character of the field value 
>> is ignored and can be removed without changing the meaning of the 
>> header field."
>>
>> ...replacing the MAY in draft 07, and the SHOULD in the current 
>> edits, by "can". RFC2119 terminology is not needed here.
>
> It seems obvious to me with your last proposal, that removing excess 
> whitespace before interpreting a value is the only sensible 
> interpretation, but I've seen enough sloppy coding to believe, that 
> the proposed wording would be interpreted as one can ignore the 
> existance of excess white space and use the value, white space and all 
> in some comparison. Thus I'm in favor of as strong a wording as we can 
> use at this stage of the progression to standard to say that excess 
> white space MUST be removed for interpreting the value.
>
> Perhaps introduce the notion of canonical form for headers and values 
> and require conversion to canonical form before processing the values 
> in any context where the outcome would be different based on the 
> existance of extraneous white space. Including (but not limited to) 
> computation of a digest of header values, interptreting values, etc.
>
> Dave Morris
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 08 Oct 2009 11:47:38 +1300

Actually point b needs to be a MUST as well, since if it didn't remove 
the OWS when forwarding the message, it would be breaching a.

Adrien de Croy wrote:
> c) a requirement for caches and clients to remove OWS when using 
> headers - hard to tell whether should be SHOULD or MUST.
>
> regards
>
> Adrien
>
>
> David Morris wrote:
>>
>>
>>> "OWS occurring before the first non-whitespace character of the 
>>> field value or after the last non-whitespace character of the field 
>>> value is ignored and can be removed without changing the meaning of 
>>> the header field."
>>>
>> existance of excess white space and use the value, white space and 
>> all in some comparison. Thus I'm in favor of as strong a wording as 
>> we can use at this stage of the progression to standard to say that 
>> excess white space MUST be removed for interpreting the value.
>>

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Oct 2009 10:23:43 +1100

That's taking it way too far, and would make many (most?)  
intermediaries non-conformant. An intermediary can't and shouldn't  
have to mess with headers that it doesn't care about. It's enough to  
say that such OWS is to be ignored when a header is interpreted.

On 08/10/2009, at 9:47 AM, Adrien de Croy wrote:

>
> Actually point b needs to be a MUST as well, since if it didn't  
> remove the OWS when forwarding the message, it would be breaching a.
>
>
> Adrien de Croy wrote:
>>
>> without breaking the meaning of the header.  We can't really allow  
>> a situation where the OWS has some meaning, or is relied on, since  
>> that then breaks all manner of things (e.g. such as comparison of  
>> selecting headers for caches).
>>
>>>>>> non-whitespace character of the field value is ignored and  
>>>>>> SHOULD be removed without changing the meaning of the header  
>>>>>> field."
>>>>>
>>>>>
>>>>> Doesn't read smoothly .... and infact turns into a directive  
>>>>> rather than a permission.
>>>>>
>>>> field value or after the last non-whitespace character of the  
>>>> field value is ignored and can be removed without changing the  
>>> interpretation, but I've seen enough sloppy coding to believe,  
>>> that the proposed wording would be interpreted as one can ignore  
>>> the existance of excess white space and use the value, white space  
>>> and all in some comparison. Thus I'm in favor of as strong a  
>>> wording as we can use at this stage of the progression to standard  
>>> to say that excess white space MUST be removed for interpreting  
>>> the value.
>>>
>>> Perhaps introduce the notion of canonical form for headers and  
>>> values and require conversion to canonical form before processing  
>>> the values in any context where the outcome would be different  
>>> based on the existance of extraneous white space. Including (but  
>>> not limited to) computation of a digest of header values,  
>>> interptreting values, etc.
>>>
>>> Dave Morris
>>>
>>
>
> -- 

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 08 Oct 2009 16:39:48 +0200

Mark Nottingham wrote:
> That's taking it way too far, and would make many (most?) intermediaries 
> non-conformant. An intermediary can't and shouldn't have to mess with 
> headers that it doesn't care about. It's enough to say that such OWS is 
> to be ignored when a header is interpreted.

Right.

What's important is to clarify that leading and trailing whitespace is 
not part of the field value. (note that we're planning to change the 
ABNF fragments for header fields, so those will no longer contain the 
header name, colon, and surrounding whitespace; essentially what the *-v 
productions already show)

What follows from that, but doesn't necessarily need to spelled, is:

1) intermediaries may or may not throw them away

2) APIs may or may not throw them away

3) the semantics of header field definitions must not rely on the 
presence or absence of leading/trailing whitespace

WRT 3 we may want to add a specific section giving guidelines for 
defining new headers; these would also point out the importance of the 
#list rule, character encoding considerations and so on. I think most of 
this currently lives in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-07.html#rfc.section.4.2>, 
but it could be separated out into a new subsection.

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 09 Oct 2009 10:56:45 +1300

OK, I agree I took it a bit far.  I guess the key point is it must be 
clear that since it is valid to ignore OWS, therefore implementors must 
not rely on OWS being preserved through the request / response chain.  I 
guess that's what you mean by 3 below?  I think it's a broader 
requirement than just the semantics of the field definition.

Perhaps some notes to implementors then, just so the issue is brought to 
their attention.  For instance if we don't require OWS to be stripped by 
servers say parsing script output, and don't require it to be stripped 
by intermediaries, but we do require it to be ignored by anything 
interpreting the value, then I can see problems where people will deploy 
systems that effectively rely on OWS, but problems aren't found in 
testing because the OWS is by default preserved by intermediaries and 
UAs through to API clients, and not checked-for in client-side or 
server-side scripts (say).

Regards
Adrien

Julian Reschke wrote:
> header name, colon, and surrounding whitespace; essentially what the 
> *-v productions already show)
>
> #list rule, character encoding considerations and so on. I think most 
> of this currently lives in