Subject: Re: WGLC: p1 MUSTs

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 1 Aug 2013 03:45:38 -0700

Continuing on #484 ...

On Apr 30, 2013, at 11:54 AM, Alex Rousskov wrote:

>> Senders MUST exclude the userinfo subcomponent (and its "@"
>> delimiter) when an "http" URI is transmitted within a message as a
>> request target or header field value.
> 
> The above MUST should not apply to proxies, right? Policing forwarded
> URLs will break applications and policing forwarded extension header
> fields is not possible at all and would violate the "MUST forward" rule.
> How about "senders MUST NOT generate"?

Right, fixed.

>> A server MUST be prepared to receive URIs of unbounded length

> This MUST may be demoted to "ought" because "be prepared" is too vague
> (but see below for a related missing MUST).

Fixed.

>> A server MUST be prepared to receive URIs of unbounded length and
>> respond with the 414
> 
> Please insert a second MUST after "and": "and MUST respond".

Fixed.

>> Multiple header fields with the same field name can be combined into
>> one "field-name: field-value" pair
> 
> Should this be a MAY as in "The recipient MAY combine multiple ...". As
> worded now, it is not clear whether a proxy is allowed to combine
> headers when forwarding them. Note that this affects extension and other
> headers that a proxy may not understand (but may still want to combine
> if allowed to do so).

Fixed.

> A server MUST be prepared to receive request header fields of
>> unbounded length and respond
> 
> Consider removing the above MUST but please add MUST after "and": A
> server ought to be prepared to receive ... and MUST respond ...
> See above for discussion of a similar MUST that applies to URIs of
> unbounded length.

Fixed.

>> A client MUST be prepared to receive response header fields of unbounded length.
> 
> Same here, except no new MUST is needed.

Fixed.

>> If chunked is applied to a payload body, the sender MUST NOT apply
>> chunked more than once
> 
> The precondition is bogus: If chunked is NOT [yet?] applied to a payload
> body, the sender still MUST NOT apply chunked more than once!

Fixed.

>> the sender MUST NOT apply chunked more than once
> 
> This needs to be rephrased to make it clear that proxies are not
> responsible for dechunking multiple chunked encodings to make the
> forwarded message comply with this MUST. For example, we could say: "the
> sender MUST NOT generate messages with multiple chunked encodings".
> 
> Please note that both the proposed "multiple chunked encodings" and the
> existing "more than once" wordings imply that foo,chunked,bar,chunked
> combination is also not allowed.

Both are intentional.  A proxy is required to fix framing problems
in received messages (or reject them without forwarding).  Such
messages might be crafted to bypass security filters.

>> A server MUST send an empty trailer with the chunked transfer coding
>> unless at least one of the following is true:
> 
> This should be relaxed to "A server MUST generate ..." because a proxy,
> in general, does not know whether bullet #2 ("the trailer fields consist
> entirely of optional metadata...") is true. Even though chunking is a
> hop-by-hop mechanism, proxies ought to forward Trailers whenever
> possible, right?

Fixed.

>> a client MUST send only the absolute path and query components of the
>> target URI as the request-target
> 
>> To allow for transition to the absolute-form for all requests in some
>> future version of HTTP, HTTP/1.1 servers MUST accept the
>> absolute-form in requests
> 
> Should the first "MUST send" be relaxed to "MUST generate" so that the
> proxies do not block the apparently anticipated "transition to the
> absolute-form for all requests" by stripping URIs as they forward them?

No, since then such proxies won't interoperate with older servers.
I wish we could just delete the second requirement, since it has
been proven to be unreliable in practice and wouldn't be reliable
for any HTTP/1.x (it isn't needed for support of HTTP/2).

>> In order to avoid request loops, a proxy that forwards requests to 
>> other proxies MUST be able to recognize and exclude all of its own
>> server names
> 
> Several intermingled issues here:
> 
> 1) The "other proxies" prerequisite is a red herring IMO. Any proxy
> should avoid request loops. If a proxy that is not configured to forward
> request to other proxies sends an "origin server" request to itself,
> such a request may still create a loop.

Well, if it sends an origin server request to itself, then as
the recipient it takes on the role of an origin server or gateway,
not a proxy, and the requirement (as stated) no longer applies ...

> I think the following would be
> better: "In order to avoid request loops, a proxy MUST ..."

Since this section is about forwarding, I have changed it to

   An intermediary MUST NOT forward a message to itself unless it is
   protected from an infinite request loop. In general, an intermediary ought
   to recognize its own server names, including any aliases, local variations,
   or literal IP addresses, and respond to such requests directly.

> 2) If we want the proxy to recognize and exclude, let's demand that
> instead of just demanding that the proxy is able to do that (but
> possibly does not do it): "a proxy ... MUST recognize and exclude ...".

Fixed.

> 3) This MUST may benefit from some polishing to clarify what "exclude"
> means. I think it means "reject" in this context.

Not necessarily.

> 4) The "recognize" part can be dropped because recognition is implied by
> the "exclude" requirement.
> 
> At the end, we may arrive at something like this:
> 
> "In order to avoid request loops, a proxy MUST reject requests for
> itself, including requests where the server address is formed using a
> proxy domain name, its aliases, local variations, or literal IP addresses."

It can handle the requests itself.  I have rephrased it.

>> A client that does not support persistent connections MUST send the
>> "close" connection option in every request message.
> 
> Including a CONNECT request message?

Yes (they are orthogonal).

> A client that pipelines requests MUST be prepared to retry those requests
> 
> MUST be prepared to retry but does not have to retry? Or MUST retry?

Changed to "SHOULD retry unanswered requests".

> A client that pipelines requests MUST be prepared to retry those
>> requests if the connection closes before it receives all of the
>> corresponding responses.
> 
> Please clarify that the client MUST retry unanswered requests and not
> all "those requests" it pipelined.

Fixed.

> MUST NOT pipeline on a retry connection until it knows the connection
>> is persistent.
> 
> Is it really possible to know that a connection _is_ persistent?

Fixed, and rephrased to note that it is due to the TCP reset problem.

...

> And here is a list of MUST-level requirements that are missing an
> explicit actor on which the requirement is placed. Most of these should
> be easy to rephrase to place the requirement on the intended actor
> (e.g., "A proxy MUST" instead of "header field MUST":
> 
>> An unrecognized header field received by a proxy MUST be forwarded
>> downstream

Fixed (and then removed as it was a redundant copy).

>> The host MUST NOT be empty; if an "http" URI is received with an
>> empty host, then it MUST be rejected as invalid.

Fixed.

>> the TCP connection MUST be secured,

Fixed.

>> These special characters MUST be in a quoted string

Removed as it was redundant to ABNF.

>> the message framing is invalid and MUST be treated as an error

Fixed.

>> a response message received by a user agent, it MUST be treated as an
>> error

Fixed.

>> The trailer MUST NOT contain fields

Fixed.

>> the Host field-value MUST be identical

Fixed.

>> the Host header field MUST be sent with an empty field-value.

Fixed.

>> The "Via" header field MUST be sent by a proxy

Fixed already for another ticket.

>> the connection MUST be closed after the current request/response is
>> complete

Removed as redundant.

>> all messages on a connection MUST have a self-defined message length

Changed to "need to".

>> the first action after changing the protocol MUST be a response

Rephrased.

> Please be careful with "send" and "generate" when fixing the above
> actorless rules so that the proxies do not accidentally become
> responsible for policing traffic where unnecessary.
> 
> Thank you,
> 
> Alex.

Committed in

  http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2334

Thanks for the detailed review,

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 3 Aug 2013 18:48:22 -0700

On Apr 30, 2013, at 1:46 PM, Alex Rousskov wrote:

>> The CONNECT method requests that the recipient establish a tunnel to
>> the destination origin server [...], until the connection is closed.
> 
> The "until the connection is closed" part is misleading and inaccurate.
> 
> There are two connections in a CONNECT tunnel: (a) between a CONNECT
> sender and CONNECT recipient and (2) between CONNECT recipient the the
> next HTTP hop. The tunnel termination condition is rather complex and is
> detailed later in the same section. It may be a good idea to drop the
> "until..." part. At least I cannot suggest a way to describe it
> correctly as an ending of an already long sentence :-).

Changed to "until the tunnel is closed".

>> When a tunnel intermediary detects that either side has closed its
>> connection, any outstanding data that came from that side will first
>> be sent to the other side and then the intermediary will close both
>> connections. If there is outstanding data left undelivered, that data
>> will be discarded.
> 
> These "will"s should be rephrased as intermediary MUSTs IMO. I also
> suggest moving them higher, before the informal risk discussion.

Moved, fixed, and rephrased to "A tunnel is closed when ..."

>> A client MUST NOT send header fields in a TRACE request containing
>> sensitive data
> 
> The above rule seems too onerous to proxies. Replace "MUST NOT send"
> with "MUST NOT generate"?

Fixed.

>> 5.1.1.1 Use of the 100 (Continue) Status
>> Requirements for HTTP/1.1 clients:
>> ...
>> Requirements for HTTP/1.1 proxies:
> 
> Should we explicitly exclude proxies from the first group of
> requirements by saying "Requirements for user agents" instead of
> "Requirements for clients"?

No, the first set applies to proxies that want to use 100-continue
for their own reasons.

>> MUST contain an updated Max-Forwards field with a value decremented by one (1).
> 
> A lot of proxies violate this MUST because they cannot grok and, hence,
> cannot decrement large integer values. Interoperability problems might
> happen when a client generates Max-Forwards with a maximum value it can
> store (e.g., to count the number of hops to the origin server) but the
> proxy cannot store such a large value (e.g., 64bit vs 32bit).
> 
> Perhaps we can relax this rule by allowing proxies to decrement by "at
> least one", so that a huge value can be replaced with the maximum value
> the proxy can represent?

Changed to

  If the received Max-Forwards value is greater than zero,
  the intermediary MUST generate an updated Max-Forwards field
  in the forwarded message with a field-value that is the lesser of:
  a) the received value decremented by one (1), or
  b) the recipient's maximum supported value for Max-Forwards.

>> A client MUST be prepared to accept one or more 1xx
> 
> Drop "be prepared" to demand acceptance rather than preparedness?

Fixed in prior commit.

>> Proxies MUST forward 1xx responses, unless the connection between the
>> proxy and its client has been closed,
> 
> This "unless" clause should be dropped as implied. Otherwise, we would
> have to add it to every "MUST forward" requirement! :-)

Fixed.

>> A sender MUST generate the IMF-fixdate format when sending an
>> HTTP-date value in a header field.
> 
> Please polish to remove the implication that proxies must fix dates when
> forwarding HTTP-date values. For example: "A sender MUST use the
> IMF-fixdate format when generating a header field containing an
> HTTP-date value".
> 
> Or perhaps simply: "A sender MUST generate HTTP-date values in the
> IMF-fixdate format".

Fixed.

> And here is a list of requirements that are missing an explicit actor on
> which the requirement is placed. Most of these should be easy to
> rephrase to place the requirement on the intended actor (e.g., "A proxy
> MUST" instead of "header field MUST":
> 
>> the content codings MUST be listed in the order in which they were applied

Fixed.

>> then the resource MUST disable or disallow that action

resource owner

>> The Expect header field MUST be forwarded

Fixed.

>> the forwarded message MUST contain an updated Max-Forwards field

Fixed above.

>> The Max-Forwards header field MAY be ignored for all other request methods. 

Fixed.

>> a response with an unrecognized status code MUST NOT be cached.

Fixed.

Committed in
  http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2342

....Roy

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 04 Aug 2013 14:50:03 +1200

>    b) the recipient's maximum supported value for Max-Forwards.

Isn't Max-Forwards used for sending OPTIONS and such to a specific hop?

I know these limits are theoretically supposed to be absurdly high. But 
if the implementation decided that the limit would be 2 or something the 
above rules would break tracing. For example the client would get back a 
constant response gnenerated from hop X+2 when it was querying hop X+3 
to X+N.

I think an error response would be better if Max-Forwards is bigger than 
the implementation can support.

FWIW: there are big-math tricks that can be implemented to increment or 
decrement arbitrarily large numeric values using a counter counter as 
small as 8-bit if neccesary so while the speed issue is relevant the 
X-bit overflow shodul not be. The old Co-Advisor test for this was to 
send a 72-bit numeric value in the header and expect successful decrement.

Amos

---

From: Roy T. Fielding <[email-masked]>
Date: Fri, 23 Aug 2013 06:22:00 -0700

On Apr 30, 2013, at 5:33 PM, Alex Rousskov wrote:

>> The If-Match condition is met if and only if any of the entity-tags
>> listed in the If-Match field value match the entity-tag of the
>> selected representation using the weak comparison function (as per
>> Section 2.3.2), or if "*" is given and any current representation
>> exists for the target resource.
> 
> The "if and only if ... or if ..." construction looks funny. It is kind
> of correct because the first part applies to the non-* condition and the
> second part applies to the * condition, but still... Please consider
> dropping either "and only if" or the "if" in "or if".

Okay.

> Can a proxy with an empty cache determine whether "any current
> representation exists for the target resource" without forwarding the
> request to the origin server? In other words, does "exists" mean "exists
> in the proxy cache" or "exists on the origin server"? Perhaps you can
> add some test to clarify the scope of resource existence here from the
> caching proxy point of view.

Replaced with "has a representation" (and related changes).

> The same two concerns apply to the If-None-Match section.

Fixed.

> Here is a list of requirement-like statements that seem to be missing
> MUST/SHOULD/MAY keywords to make them formal requirements and indicate
> the level of those requirements:
> 
>> Preconditions are ignored if the server determines that an error or
>> redirect response applies before they are evaluated.

Fixed.

>> The conditional request header fields defined by this specification
>> are ignored for request methods that never involve the selection or
>> modification of a selected representation

Fixed.

>> The general rule of conditional precedence is that exact match
>> conditions are ...
> 
>> Specifically, the fields defined by this specification are evaluated
>> as follows:

Fixed.

http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2361

....Roy