Subject: p1: BWS

From: Mark Nottingham <[email-masked]>
Date: Thu, 18 Apr 2013 11:18:22 +1000

p1 3.2.3 says:

>    BWS is used where the grammar allows optional whitespace, for
>    historical reasons, but senders SHOULD NOT generate it in messages;
>    recipients MUST accept such bad optional whitespace and remove it
>    before interpreting the field value or forwarding the message
>    downstream.

  http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-22#section-3.2.3

Throughout our specs, BWS is used at the end of header fields:
     header-field   = field-name ":" OWS field-value BWS

and in transfer-codings:
     transfer-parameter = attribute BWS "=" BWS value

and in Expect headers:
  expectation  = expect-name [ BWS "=" BWS expect-value]
                             *( OWS ";" [ OWS expect-param ] )
  expect-param = expect-name [ BWS "=" BWS expect-value ]

and, finally, in auth-params on challenges and credentials:
  auth-param     = token BWS "=" BWS ( token / quoted-string )

Is this whitespace really "bad" enough to MUST-require that intermediaries (including load balancers and other hardware!) remove it before forwarding the message?

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 18 Apr 2013 16:30:03 +1200

> Is this whitespace really "bad" enough to MUST-require that intermediaries (including load balancers and other hardware!) remove it before forwarding the message?

For interoperability yes the whitespace is a bit problem. Its presence 
subtly breaks any implementations looking for tokens with the strict 
termination delimiter and also opens opportunities for problems related 
to WS padding headers on maliciously crafted messages.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 18 Apr 2013 08:02:11 +0200

On Thu, Apr 18, 2013 at 04:30:03PM +1200, Amos Jeffries wrote:
> >Is this whitespace really "bad" enough to MUST-require that intermediaries 
> >(including load balancers and other hardware!) remove it before forwarding 
> >the message?
> 

Agreed, but on the other hand, requiring that some intermediaries that do
not even use these fields to fix them can increase the risk of breaking
something between the client and the server. And since many of them will
not do it anyway, we'll end up with another MUST that is not respected,
so probably a SHOULD would be more appropriate ?

Willy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 18 Apr 2013 08:24:12 +0200

On 2013-04-18 08:02, Willy Tarreau wrote:
> ...
> ...

That's my concern as well.

My gut feeling is that we want BWS to be fixed during the message 
parsing, but not necessarily during the field value parsing.

We probably can fix that in prose (preferred). Alternatively, we could 
add yet another ABNF rule (this might negatively affect a few specs in 
the RFC Editor Queue, though).

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 18 Apr 2013 22:25:12 +1000

> so probably a SHOULD would be more appropriate ?

A SHOULD is not a MUST that we sort-of mean. At least, that's not what we say, even if it is how we tend to use it sometimes (more feedback forthcoming).

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Apr 2013 08:27:21 +1000

We can either:

- make it a MUST
- document what the exceptional circumstances are that cause the SHOULD (but I don't think it's that kind of SHOULD)
- downgrade it to an "ought to"

I do note that we say "generate" there (which I missed before, sorry); in our terminology, that means you DON'T need to fix it up when forwarding it; it's only when you're actually creating the element that this applies.

So, I'd suggest we make it a MUST, and change the language slightly to clarify:

"...but it MUST NOT be generated in messages..."

Also, in p2, I'd note that we do NOT allow BWS inside of media type parameters:
  https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#media.type

AIUI (thanks, Julian), this is because many implementations don't accept whitespace there at all. It might be worth noting in the text that this parameter construct is different in that aspect.

Cheers,

On 19/04/2013, at 8:17 AM, Roberto Peon <[email-masked]> wrote:

> As an implementer having written code to handle this, I don't see a lot of value in making it a MUST, when I'll need to have my servers handle it anyway, in the off-chance that the load-balancer is changed or is not there.
> It is really a nice-to-have kind of thing.
> -=R
> 
> 
> On Thu, Apr 18, 2013 at 5:25 AM, Mark Nottingham <[email-masked]> wrote:
>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 18 Apr 2013 15:17:23 -0700

As an implementer having written code to handle this, I don't see a lot of
value in making it a MUST, when I'll need to have my servers handle it
anyway, in the off-chance that the load-balancer is changed or is not there.
It is really a nice-to-have kind of thing.
-=R

On Thu, Apr 18, 2013 at 5:25 AM, Mark Nottingham <[email-masked]> wrote:

> A SHOULD is not a MUST that we sort-of mean. At least, that's not what we
> say, even if it is how we tend to use it sometimes (more feedback
> forthcoming).
>
> Cheers,
>
> --

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 19 Apr 2013 07:38:45 +0200

On Fri, Apr 19, 2013 at 08:27:21AM +1000, Mark Nottingham wrote:
> I do note that we say "generate" there (which I missed before, sorry); in our
> terminology, that means you DON'T need to fix it up when forwarding it; it's
> only when you're actually creating the element that this applies.

Ah OK then I agree with a MUST as well.

+1 then :-)

Willy

---

From: Roberto Peon <[email-masked]>
Date: Thu, 18 Apr 2013 23:56:26 -0700

+1

On Thu, Apr 18, 2013 at 10:38 PM, Willy Tarreau <[email-masked]> wrote:

> > - document what the exceptional circumstances are that cause the SHOULD
> (but I don't think it's that kind of SHOULD)

---

From: Julian Reschke <[email-masked]>
Date: Fri, 19 Apr 2013 11:34:59 +0200

> "...but it MUST NOT be generated in messages..."

I have no problem with *that* change.

What concerns we much more is the "MUST accept such bad optional 
whitespace and remove it before interpreting the field value or 
forwarding the message downstream" -- that part is fine for parsing the 
message itself (everybody needs to be able to do that), but *not* ok for 
individual field values. We need a distinction here.

> AIUI (thanks, Julian), this is because many implementations don't accept whitespace there at all. It might be worth noting in the text that this parameter construct is different in that aspect.

I don't know whether it's because of implementations following the spec, 
or the spec following implementations.

Somewhat outdated tests are over here: 
<http://greenbytes.de/tech/tc/httpcontenttype/>

I'm more than happy to require recipients to handle BWS here for 
consistency with other header fields, as long as we tell generators to 
never ever use it.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Apr 2013 21:39:53 +1000

On 19/04/2013, at 7:34 PM, Julian Reschke <[email-masked]> wrote:

> What concerns we much more is the "MUST accept such bad optional whitespace and remove it before interpreting the field value or forwarding the message downstream" -- that part is fine for parsing the message itself (everybody needs to be able to do that), but *not* ok for individual field values. We need a distinction here.

Ah, yes, that was the part that sparked my original concern.

The text in the latest draft is:

"""
BWS is used where the grammar allows optional whitespace, for historical reasons, but senders SHOULD NOT generate it in messages; recipients MUST accept such bad optional whitespace and remove it before interpreting the field value or forwarding the message downstream.
"""

I'd suggest:

"""
BWS is used where the grammar allows optional whitespace, for historical reasons, but it MUST NOT be generated in messages; recipients MUST accept such bad optional whitespace and remove it before interpreting the field value.
"""

> I don't know whether it's because of implementations following the spec, or the spec following implementations.
> 
> Somewhat outdated tests are over here: <http://greenbytes.de/tech/tc/httpcontenttype/>
> 
> I'm more than happy to require recipients to handle BWS here for consistency with other header fields, as long as we tell generators to never ever use it.

That would make current implementations non-conformant, where we already have interop. While consistency is good, that may be going too far for it.

I was thinking of something along the lines of

"""
Note that unlike some similar constructs in other headers, media type parameters do not allow whitespace (even "bad" whitespace) around the "=" character.
"""

Would that work for you?

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Apr 2013 21:46:57 +1000

Tracking this in:
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/442

---

From: Julian Reschke <[email-masked]>
Date: Fri, 19 Apr 2013 13:47:48 +0200

> Note that unlike some similar constructs in other headers, media type parameters do not allow whitespace (even "bad" whitespace) around the "=" character.

+1 on both.

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 20 Apr 2013 00:11:27 +1200

+1.

Amos

> On Thu, Apr 18, 2013 at 10:38 PM, Willy Tarreau <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     > - document what the exceptional circumstances are that cause the
>     SHOULD (but I don't think it's that kind of SHOULD)

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 23 Apr 2013 17:32:01 +1000

OK, marking both of the below for incorporation in -23.