Subject: Re: WGLC p1: MUST fix Content-Length?

From: Willy Tarreau <[email-masked]>
Date: Wed, 1 May 2013 09:22:11 +0200

On Tue, Apr 30, 2013 at 11:53:28PM -0600, Alex Rousskov wrote:
> Hello,
> 
>     When talking about a Content-Length header field with multiple
> identical values, Part 1 Section 3.3.2 of HTTPbis says:
> 
> > the recipient MUST either reject the message as invalid or
> > replace the duplicated field-values with a single valid
> > Content-Length field containing that decimal value prior to
> > determining the message body length.
> 
> It is not clear whether "recipient MUST replace" (a requirement on the
> recipient) also implies that "a sender MUST replace [...] when
> forwarding the message" (a requirement on the sender). This issue has
> been raised on 2011/11/28, but the discussion diverged, and I could not
> tell whether there was a consensus on what the correct interpretation is.
> 
> Please decide whether a proxy MUST "fix" such Content-Length headers
> when forwarding the message and adjust the above text to clarify one way
> or another.

That's what the discussion converged to. I even modified haproxy in order
to do so. The idea is simple : if you receive a message with multiple
content lengths, either you can't deal with them and must reject the
message, or you can deal with them and then you know how to fix the
message before interpreting it or forwarding it, so you must do so.

Do you think the text needs to be adjusted ?

It can be seen as a constraint to fix a message before forwarding it,
but in fact it's a relaxation of a case where you were supposed to
reject the message.

Willy

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 01 May 2013 01:34:12 -0600

On 05/01/2013 12:52 AM, Willy Tarreau wrote:
> On Tue, Apr 30, 2013 at 05:22:39PM -0600, Alex Rousskov wrote:

>>>>> A client [...] MUST NOT pipeline on a retry connection until it
>>>>> knows the connection is persistent.

> I think that it was the reason why it's suggested somewhere that pipelining
> should only be done on connections that we know are persistent (ie after a
> first response)

That "somewhere" may be the above MUST NOT requirement we are discussing
on this thread :-).

>> Let's start with (a): To know that a connection "is persistent", the
>> client has to receive a response on that connection. This implies that
>> the client MUST NOT retry a failed pipelined request on a connection it
>> just opened.

I know what I was missing: The client may send a retry request on a new
connection. The above MUST NOT prohibits _pipelining_ on a new
connection, but does not prohibit sending a single request on that
connection. That resolves my concern about stuck clients.

I am still not sure why we are prohibiting retry pipelining on new
connections though. Why do we have to reuse an old connection if we want
to retry a failed pipeline?

Thank you,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 01 May 2013 01:37:29 -0600

> Do you think the text needs to be adjusted ?

Yes, of course. The current text is not clear IMO, as I tried to explain
in the beginning of this message.

Cheers,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 1 May 2013 09:40:19 +0200

> to retry a failed pipeline?

I don't remember, I believe it was just that if pipeline failed on a
connection, you don't want to pipeline again on the new one, otherwise
you can do that infinitely.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 1 May 2013 09:43:41 +0200

> in the beginning of this message.

Then what about :

   recipient MUST either reject the message as invalid or replace the
   duplicated field-values with a single valid Content-Length field
   containing that decimal value prior to determining the message body
-  length.
+  length or forwarding it.

Willy

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 01 May 2013 01:51:01 -0600

> you can do that infinitely.

And if I use an old connection, the situation is guaranteed to be better
somehow?

The reasons a new connection may fail differ from the reasons an old
connection may fail, but both may fail, so I do not understand why we
are prohibiting one and requiring the other.

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 01 May 2013 01:54:26 -0600

> +  length or forwarding it.

That would work, IMO, although I would replace "it" with "the message".

Thank you,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 1 May 2013 10:16:53 +0200

> are prohibiting one and requiring the other.

No it's not that, it's that if you detect a failure on a connection where
you pipelined, you should not attempt to pipeline again on the new connection
since it will very likely end the same way. It's unrelated to the old
connection being better at all.

Willy

---

From: Ken Murchison <[email-masked]>
Date: Wed, 01 May 2013 09:33:37 -0400

On Tue, 30 Apr 2013 19:18:49 -0600,
Alex Rousskov wrote:
> > Clients MUST NOT use an entity-tag marked as weak in an If-Range
> > field value and MUST NOT use a Last-Modified date ...
>
> Please replace "use" with "generate" to explicitly exclude proxies from
> policing these headers (i.e., to allow proxies to forward these headers
> "as is"). This was already done for other If-Range header rules, but
> these two MUST NOTs have slipped through the cracks.
>   
Clients don't generate ETags, they just use what they have seen in 
responses from servers.  I think the sentence would have to be reworded 
to something like:

"Clients MUST NOT generate an If-Range field value containing an 
entity-tag marked as weak and MUST NOT generate an If-Range field value 
containing a Last-Modified date..."

> > 4.1 206 Partial Content
>
> Since HTTPbis no longer allows multipart/byteranges media type to
> determine the message body length, perhaps it would be a good idea to
> explicitly mention that a server MAY generate a 206 Partial Content
> response (with single or multiple ranges) without a Content-Length
> header and may use chunked encoding? I bet many clients will break when
> this starts happening, and there are currently no examples or warnings
> that would prepare developers for that possibility.
>   
A HTTP/1.1 server can always use chunked encoding for a response and any 
HTTP/1.1 client that can't handle chunked or doesn't expect it for a 
certain response code is already broken.  I don't see the need to 
specifically call this out for a 206 response.

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 01 May 2013 08:32:29 -0600

> connection being better at all.

If you are describing the true intent, the requirement would have been
something like "A client MUST NOT pipeline when retrying a failed
pipelined request" but the requirement is rather different:

> A client that assumes a persistent connection and pipelines
> immediately after connection establishment MUST NOT pipeline on a
> retry connection until it knows the connection is persistent.

It feels like the intent here is to say that a persistent connection has
more chances to remain persistent than a new connection has chances to
become persistent.

Cheers,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 01 May 2013 08:59:20 -0600

> containing a Last-Modified date..."

Sounds good. The term "generate" is indeed overloaded in this context
and your wording is better.

> specifically call this out for a 206 response.

Agreed. I did not realize RFC 2616 already placed chunked above
multipart/byteranges in Message Length rules, so chunked
multipart/byteranges were already possible.

Thank you,

Alex.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 May 2013 14:41:44 +1000

Thanks, Alex; I've made these issues #478-481, all editorial.

On 01/05/2013, at 3:09 PM, Alex Rousskov <[email-masked]> wrote:

> Hello,
> 
>    These comments are based on the "latest" snapshot dated Mon 29 Apr
> 2013 03:13:05 PM MDT at
> https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p7-auth.html
> 
> I hope these issues are "editorial in nature".
> 
> 
>> For historical reasons, senders MUST only use the quoted-string syntax.
> 
> Perhaps this can be relaxed to "MUST only generate", especially since
> another MUST prohibits proxies from modifying WWW-Authenticate and
> Authorization header fields.
> 
> 
> And here is a list of requirements that are missing an explicit actor on
> which the requirement is placed. Even though it is often possible to
> guess the actor, most of these should be easy to rephrase to place the
> requirement on the intended actor explicitly (e.g., "A proxy MUST"
> instead of "a header field MUST":
> 
>> each parameter name MUST only occur once per challenge
> 
>> This response MUST include a WWW-Authenticate header
> 
>> The 407 (Proxy Authentication Required) response message [...] MUST
>> include a Proxy-Authenticate header field
> 
>> information necessary to authenticate a request MUST be provided in
>> the request
> 
>> It MUST be included as part of a 407 (Proxy Authentication Required)
>> response.
> 
>> It MUST be included in 401 (Unauthorized) response messages
> 
> Please be careful with "send" and "generate" when fixing the above
> actorless rules so that the proxies do not accidentally become
> responsible for policing traffic where unnecessary.
> 
> 
> Thank you,
> 
> Alex.
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 May 2013 14:47:18 +1000

Now <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/483>.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 May 2013 14:54:43 +1000

Now <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/484>.

On 01/05/2013, at 4:54 AM, Alex Rousskov <[email-masked]> wrote:

> Hello,
> 
>    Summary: The specs have improved considerably since 2012. Thank you
> for not giving up on them despite HTTP/2.0 excitement!
> 
> Due to the lack of time, I have to focus on MUST-level requirements
> only. These comments are based on the "latest" snapshot dated Mon 29 Apr
> 2013 03:13:05 PM MDT at
> https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p1-messaging.html
> 
> I hope these comments can be addressed by editors alone, but I apologize
> in advance if some are found too controversial and should have been sent
> separately.
> 
> 
>> A sender MUST NOT generate protocol elements that do not match the
>> grammar defined by the ABNF rules for those protocol elements that
>> are applicable to the sender's role.
> 
> The "for those protocol elements..." part should be dropped IMO. A
> sender MUST NOT generate invalid protocol elements even if they are not
> applicable to the sender's role. Note that we are talking about
> _generation_ and not forwarding here.
> 
> 
>> If a received protocol element is processed, the recipient MUST be
>> able to parse any value that would match the ABNF rules
> 
> "processed" seems too broad because simply buffering a header may be
> called "processing". "Interpreted" may be better. Or did I miss the
> definition of "process" that clarifies this?
> 
> 
>> If a received protocol element is processed, the recipient MUST be
>> able to parse any value that would match the ABNF rules for that
>> protocol element, excluding only those rules not applicable to the
>> recipient's role.
> 
> The "excluding only those rules not applicable..." part seems to
> contradict the "processed" verb. Why would a recipient want to process
> something inapplicable? Perhaps this is related to the "process" versus
> "interpreted" issue mentioned above.
> 
> 
>> the recipient MUST be
>> able to parse any value that would match the ABNF rules for that
>> protocol element, excluding only those rules not applicable to the
>> recipient's role.
> 
> Please rephrase to avoid double negation in "excluding not applicable".
> For example: "the recipient MUST be able to parse any value matching the
> corresponding ABNF protocol element rules applicable to the recipient's
> role"
> 
> 
>> Senders MUST exclude the userinfo subcomponent (and its "@"
>> delimiter) when an "http" URI is transmitted within a message as a
>> request target or header field value.
> 
> The above MUST should not apply to proxies, right? Policing forwarded
> URLs will break applications and policing forwarded extension header
> fields is not possible at all and would violate the "MUST forward" rule.
> How about "senders MUST NOT generate"?
> 
> 
>> A server MUST be prepared to receive URIs of unbounded length
> 
> This MUST may be demoted to "ought" because "be prepared" is too vague
> (but see below for a related missing MUST).
> 
> 
>> A server MUST be prepared to receive URIs of unbounded length and
>> respond with the 414
> 
> Please insert a second MUST after "and": "and MUST respond".
> 
> 
>> Multiple header fields with the same field name can be combined into
>> one "field-name: field-value" pair
> 
> Should this be a MAY as in "The recipient MAY combine multiple ...". As
> worded now, it is not clear whether a proxy is allowed to combine
> headers when forwarding them. Note that this affects extension and other
> headers that a proxy may not understand (but may still want to combine
> if allowed to do so).
> 
> 
>> A server MUST be prepared to receive request header fields of
>> unbounded length and respond
> 
> Consider removing the above MUST but please add MUST after "and": A
> server ought to be prepared to receive ... and MUST respond ...
> See above for discussion of a similar MUST that applies to URIs of
> unbounded length.
> 
> 
>> A client MUST be prepared to receive response header fields of unbounded length.
> 
> Same here, except no new MUST is needed.
> 
> 
>> If chunked is applied to a payload body, the sender MUST NOT apply
>> chunked more than once
> 
> The precondition is bogus: If chunked is NOT [yet?] applied to a payload
> body, the sender still MUST NOT apply chunked more than once!
> 
> 
>> the sender MUST NOT apply chunked more than once
> 
> This needs to be rephrased to make it clear that proxies are not
> responsible for dechunking multiple chunked encodings to make the
> forwarded message comply with this MUST. For example, we could say: "the
> sender MUST NOT generate messages with multiple chunked encodings".
> 
> Please note that both the proposed "multiple chunked encodings" and the
> existing "more than once" wordings imply that foo,chunked,bar,chunked
> combination is also not allowed.
> 
> 
>> A server MUST send an empty trailer with the chunked transfer coding
>> unless at least one of the following is true:
> 
> This should be relaxed to "A server MUST generate ..." because a proxy,
> in general, does not know whether bullet #2 ("the trailer fields consist
> entirely of optional metadata...") is true. Even though chunking is a
> hop-by-hop mechanism, proxies ought to forward Trailers whenever
> possible, right?
> 
> 
>> a client MUST send only the absolute path and query components of the
>> target URI as the request-target
> 
>> To allow for transition to the absolute-form for all requests in some
>> future version of HTTP, HTTP/1.1 servers MUST accept the
>> absolute-form in requests
> 
> Should the first "MUST send" be relaxed to "MUST generate" so that the
> proxies do not block the apparently anticipated "transition to the
> absolute-form for all requests" by stripping URIs as they forward them?
> 
> 
>> In order to avoid request loops, a proxy that forwards requests to 
>> other proxies MUST be able to recognize and exclude all of its own
>> server names
> 
> Several intermingled issues here:
> 
> 1) The "other proxies" prerequisite is a red herring IMO. Any proxy
> should avoid request loops. If a proxy that is not configured to forward
> request to other proxies sends an "origin server" request to itself,
> such a request may still create a loop. I think the following would be
> better: "In order to avoid request loops, a proxy MUST ..."
> 
> 2) If we want the proxy to recognize and exclude, let's demand that
> instead of just demanding that the proxy is able to do that (but
> possibly does not do it): "a proxy ... MUST recognize and exclude ...".
> 
> 3) This MUST may benefit from some polishing to clarify what "exclude"
> means. I think it means "reject" in this context.
> 
> 4) The "recognize" part can be dropped because recognition is implied by
> the "exclude" requirement.
> 
> At the end, we may arrive at something like this:
> 
> "In order to avoid request loops, a proxy MUST reject requests for
> itself, including requests where the server address is formed using a
> proxy domain name, its aliases, local variations, or literal IP addresses."
> 
> 
>> A client that does not support persistent connections MUST send the
>> "close" connection option in every request message.
> 
> Including a CONNECT request message?
> 
> 
>> A client that pipelines requests MUST be prepared to retry those requests
> 
> MUST be prepared to retry but does not have to retry? Or MUST retry?
> 
> 
>> A client that pipelines requests MUST be prepared to retry those
>> requests if the connection closes before it receives all of the
>> corresponding responses.
> 
> Please clarify that the client MUST retry unanswered requests and not
> all "those requests" it pipelined.
> 
> 
>> MUST NOT pipeline on a retry connection until it knows the connection
>> is persistent.
> 
> Is it really possible to know that a connection _is_ persistent? And
> what does that really mean for a connection to be persistent "now"? I
> think all it means is that the connection seems to be "open" -- that the
> sender has not received an error or connection close notification [yet].
> 
> It is possible to know that the connection was persistent (i.e., handled
> multiple messages), but since the server may close an idle (from the
> server point of view) connection at any time, I do not think it is
> possible to know that the next message will reach the server, unless the
> client and the server are coordinating out of band somehow.
> 
> What are we trying to achieve with this requirement? Avoid multiple and
> possibly endless retries? Minimize the chances that a retry will have to
> be retried? Perhaps the MUST can be relaxed or reworded to reflect the
> true intent?
> 
> 
> 
> 
> And here is a list of MUST-level requirements that are missing an
> explicit actor on which the requirement is placed. Most of these should
> be easy to rephrase to place the requirement on the intended actor
> (e.g., "A proxy MUST" instead of "header field MUST":
> 
>> An unrecognized header field received by a proxy MUST be forwarded
>> downstream
> 
>> The host MUST NOT be empty; if an "http" URI is received with an
>> empty host, then it MUST be rejected as invalid.
> 
>> the TCP connection MUST be secured,
> 
>> These special characters MUST be in a quoted string
> 
>> the message framing is invalid and MUST be treated as an error
> 
>> a response message received by a user agent, it MUST be treated as an
>> error
> 
>> The trailer MUST NOT contain fields
> 
>> the Host field-value MUST be identical
> 
>> the Host header field MUST be sent with an empty field-value.
> 
>> The "Via" header field MUST be sent by a proxy
> 
>> the connection MUST be closed after the current request/response is
>> complete
> 
>> all messages on a connection MUST have a self-defined message length
> 
>> the first action after changing the protocol MUST be a response
>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 May 2013 15:07:53 +1000

Now <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/485> (omitting the last one, since that seems to be settled).

On 01/05/2013, at 11:18 AM, Alex Rousskov <[email-masked]> wrote:

> https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p5-range.html
> 
> I hope these comments are "editorial in nature".
> 
> 
>> A client that cannot process a multipart/byteranges response MUST NOT
>> ask for multiple ranges in a single request.
> 
> A similar concern here for "MUST NOT ask". Please reword the above to
> use "MUST NOT generate".
> 
> This is especially important because a proxy may not be able to fully
> "process" a multipart/byteranges response (whatever that means) but it
> can still forward a request for multiple ranges and correctly forward
> the 206 response back to the client because HTTPbis no longer allows
> multipart/byteranges media type to determine the message body length.
> 
> 
>> 4.1 206 Partial Content
>