Subject: Re: TE / transfer-coding issue with qvalue and transfer-parameters — HTTP/1.1, HTTP/2.0

From: Simon Yarde <[email-masked]>
Date: Mon, 03 Feb 2014 13:26:26 +0000

Thanks Julian — prompted by this,

>> (c) that coding extension has a "q" parameter,

I can see a way through for a sensible parser architecture;  in that rather than trying to uniformly parse parameters, it becomes better to have specific transfer-coding parsers that only recognise registered/known attributes and skip/exclude all other parameters — from a functional-language perspective, it is preferable to have a data-type with a fixed number of parameters rather than an arbitrary-length mapping that *may* contain junk attributes.

> I agree that the ABNF is not optimal here, but it seems it is extremely unlikely that we'll ever see this happening.

I've no wish to nit-pick for the sake of it and I agree it's unlikely, but someone still has to write parsers, and I was aiming to relay my own experience, in that there was still an element of 'divining' that had not improved on RFC2616.

Has this been addressed in HTTP/2.0? My apologies, but I've only had time to take an active interest in the HTTP/1.1 project since it happened to interface with my current project.

Simon Yarde

On 3 Feb 2014, at 13:05, Julian Reschke <[email-masked]> wrote:

> On 2014-02-03 13:44, Simon Yarde wrote:
>> 
>>> Why exactly is the ABNF a problem?
>> 
>> 
>> Because the rule for transfer-parameter is in conflict with t-ranking when the transfer-coding rule is imported to t-codings (copied below).
>> 
>> Is the protocol trying to say that the transfer-extension parser must halt if is sees anything matching a "q" attribute?  I.e.
>> 
>> -- > attribute          = token   ; except "q"
>> 
>> 
>> -- > transfer-coding    = "chunked" ; Section 4.1
>> -- >                    / "compress" ; Section 4.2.1
>> -- >                    / "deflate" ; Section 4.2.2
>> -- >                    / "gzip" ; Section 4.2.3
>> -- >                    / transfer-extension
>> -- > transfer-extension = token *( OWS ";" OWS transfer-parameter )
>> -- >
>> *- > transfer-parameter = attribute BWS "=" BWS value
>> -- > attribute          = token
>> -- > value              = token / quoted-string
>> 
>> -- > TE        = #t-codings
>> *- > t-codings = "trailers" / ( transfer-coding [ t-ranking ] )
>> *- > t-ranking = OWS ";" OWS "q=" rank
>> -- > rank      = ( "0" [ "." 0*3DIGIT ] )
>> -- >           / ( "1" [ "." 0*3("0") ] )
>> 
>> 
>> As a side-note on tolerance, I was also aiming to highlight that client libs are likely to construct dictionaries with arbitrary ordering of parameters (in languages where dict ordering is not guaranteed), and thus the t-coding value will likely occur anywhere in a list of parameter-like values — but yes, I do note that the rule states that t-ranking *follows* a transfer-coding (and any attendant transfer-parameters), and it would be an error to include any more text after a t-ranking.
>> ...
> 
> Yes.
> 
> In practice, all of this only becomes relevant if people (a) use qvalues on TE, (b) use a coding extension that has parameters, (c) that coding extension has a "q" parameter, and (d) that parameter does not appear last.
> 
> I agree that the ABNF is not optimal here, but it seems it is extremely unlikely that we'll ever see this happening.
> 
> Best regards, Julian

---

From: Simon Yarde <[email-masked]>
Date: Mon, 03 Feb 2014 11:41:53 +0000

Thanks Julian — I noticed this in my docs and felt it would be worth throwing out there in view of the push to wrap-up soon.

> For TE and Accept-Encoding the simplest possible fix is to add that to the sections about considerations for new encodings. Note that registration of both transfer codings and content codings require "IETF Review", so that bar is relatively high anyway.

This seems eminently sensible and unlikely to disrupt the review findings, since it simply replicates an accepted approach to add clarity.

That said, it still isn't possible to directly implement parsers for media-type and transfer-coding ABNF rules, and by association the TE and Accept rules.

Simon Yarde

On 3 Feb 2014, at 11:12, Julian Reschke <[email-masked]> wrote:

> On 2014-02-03 11:54, Simon Yarde wrote:
>> Whilst implementing parsers for HTTP/1.1 I encountered issues affecting transfer-parameters in the context of the TE rule that may have broader implications to HTTP/2.0;  namely that weight-value (aka "q" value / param) cannot be distinguished from transfer-parameters.  More generally this issue affects any situation where a weight-value is combined with parameters.
>> 
>> # HTTP/1.1 Proposal
>> 
>> For HTTP/1.1 I think the issue can be dealt with using a MAY NOT rule — here's the relevant notes from my docs as they occurred to me at the time:
>> 
>> -- /Note/ — there is an issue with the protocol whereby @weight@ (aka
>> -- @qvalue@) elements cannot be distinguished from
>> -- @transfer-parameters@.  Custom @transfer-extensions@ MUST NOT use an
>> -- attribute of @q@ (case-insensitive) in the context of a
>> -- @weighted-transfer-coding@, and should generally avoid such use to
>> -- avoid conflicts.  The ABNF rule is probably misleading because
>> -- the @weight@ element in reality may occur anywhere amongst other
>> -- parameters-like-elements, although technically the protocol says it
>> -- must follow any parameters;  nonetheless it cannot be distinguished
>> -- from parameter-like-elements.
>> 
>> I felt I needed to introduce a 'weighted-transfer-coding' parser since I could not cleanly implement transfer-coding for use within a TE parser — i.e. the transfer-coding rule/parser would already have consumed a "q" parameter as a transfer-parameter before the TE rule had a chance to operate.
>> 
>> There were also some points that may be worth highlighting for tolerance in TE:
>> 
>> -- Only a @transfer-extension@ value permits parameters.  Illegal
>> -- parameters following any other @transfer-coding@ are skipped.
>> 
>> -- If occurances of the @trailers@ element are found then the
>> -- 'AcceptTrailers' value in the result will be 'True'.  Illegal
>> -- parameters following a @trailers@ element will cause the parser to
>> -- fail, on the basis that the message is probably garbled.
>> --
>> -- Occurances of @transfer-coding@ value @chunked@ are skipped as per
>> -- protocol guidance.
>> ...
> 
> A similar issue exists with "Accept:", where we already say:
> 
> "Note: Use of the "q" parameter name to separate media type parameters from Accept extension parameters is due to historical practice. Although this prevents any media type parameter named "q" from being used with a media range, such an event is believed to be unlikely given the lack of any "q" parameters in the IANA media type registry and the rare usage of any media type parameters in Accept. Future media types are discouraged from registering any parameter named "q"." -- <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-25.html#rfc.section.5.3.2>
> 
> For TE and Accept-Encoding the simplest possible fix is to add that to the sections about considerations for new encodings. Note that registration of both transfer codings and content codings require "IETF Review", so that bar is relatively high anyway.
> 
> That being said: we are past WG LC, IETF LC, and IESG review, so I'm reluctant to make more changes at this point.
> 
> Best regards, Julian
> 
> .

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 13:11:54 +0100

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 12:12:11 +0100

"Note: Use of the "q" parameter name to separate media type parameters 
from Accept extension parameters is due to historical practice. Although 
this prevents any media type parameter named "q" from being used with a 
media range, such an event is believed to be unlikely given the lack of 
any "q" parameters in the IANA media type registry and the rare usage of 
any media type parameters in Accept. Future media types are discouraged 
from registering any parameter named "q"." -- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-25.html#rfc.section.5.3.2>

For TE and Accept-Encoding the simplest possible fix is to add that to 
the sections about considerations for new encodings. Note that 
registration of both transfer codings and content codings require "IETF 
Review", so that bar is relatively high anyway.

That being said: we are past WG LC, IETF LC, and IESG review, so I'm 
reluctant to make more changes at this point.

Best regards, Julian

.

---

From: Simon Yarde <[email-masked]>
Date: Mon, 03 Feb 2014 13:46:17 +0000

> I think I would be pragmatic and just parse everything that looks like a parameter into a name/value map (and just trust that there'll be no "q" parameter ever).

Yep — at the moment I do this and extract a "q" value as I go, resulting in a list of tuple values containing a weight and a transfer-coding.

-- > type WeightedTransferCoding .. = (Weight, TransferCoding .. )

which benefits for being very easy to sort.

But as I say, I ended up with a parser architecture/implementation that didn't *look* like the ABNFs and I had to introduce this concept of 'weighted-transfer-coding' :)

Hope this may be of some help.

Simon Yarde

On 3 Feb 2014, at 13:36, Julian Reschke <[email-masked]> wrote:

> On 2014-02-03 14:26, Simon Yarde wrote:
> I think I would be pragmatic and just parse everything that looks like a parameter into a name/value map (and just trust that there'll be no "q" parameter ever).
> 
> Understood.
> HTTP/2.0 only defines a new message format; so what HTTP/1.1 says still applies.
> 
> Best regards, Julian
> 
>

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 14:36:13 +0100

> I can see a way through for a sensible parser architecture;  in that rather than trying to uniformly parse parameters, it becomes better to have specific transfer-coding parsers that only recognise…

I think I would be pragmatic and just parse everything that looks like a 
parameter into a name/value map (and just trust that there'll be no "q" 
parameter ever).

Understood.

> Has this been addressed in HTTP/2.0? My apologies, but I've only had time to take an active interest in the HTTP/1.1 project since it happened to interface with my current project.

HTTP/2.0 only defines a new message format; so what HTTP/1.1 says still 
applies.

Best regards, Julian

---

From: Simon Yarde <[email-masked]>
Date: Mon, 03 Feb 2014 10:54:15 +0000

# HTTP/1.1 Proposal

# HTTP/2.0 Proposal / Sketch

I'm not sure if this has been dealt with, but I concluded in a note that HTTP/2.0 might want to consider some other way to deal with "weight" given that it is such an important aspect of content-negotiation and clashes badly with custom-parameters.

If something like the rule below could be achieved then it would be very easy to logically /extend/ any rules that may be used in a /weighted context/:

-- > weighted-transfer-coding = [ weight-value "~" ] transfer-coding

E.g.  1~custom-compress;foo=bar;q=no-clash, 0.9~gzip, 0.5~deflate

-- > weighted-media-type = [ weight "~" ] media-type

E.g.  1~text/plain;q=no-clash, 0.5~text/html;foo=bar

-- > weight-value = ( "0" [ "." 0*3DIGIT ] )
-- >              / ( "1" [ "." 0*3("0") ] )

Best Regards

Simon Yarde

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 5 Feb 2014 01:11:01 -0800

> -- from parameter-like-elements.

That isn't misleading.  The ABNF requires that the t-ranking (weight)
be the last parameter per transfer-coding when it is used in TE.

> I felt I needed to introduce a 'weighted-transfer-coding' parser since I could not cleanly implement transfer-coding for use within a TE parser — i.e. the transfer-coding rule/parser would already…

Or the ordering algorithm could extract them after parsing, or it
could just ignore the weights.  Those are implementation-specific
concerns.

In practice, none of this matters, since nobody sends parameters
within TE, and if they ever do they won't be so foolish as to use
'q' as a parameter name.

....Roy