Subject: Comments/Issues on P1

From: Ben Niven-Jenkins <[email-masked]>
Date: Mon, 23 Apr 2012 19:11:32 +0100

Hi,

As part of reviewing the HTTPBIS documents we had the following comments/issues on P1

1) On page 11 it states:

   All HTTP requirements
   applicable to an origin server also apply to the outbound
   communication of a gateway. [...]

   However, an HTTP-to-HTTP gateway that wishes to interoperate with
   third-party HTTP servers MUST conform to HTTP user agent requirements
   on the gateway's inbound connection

This is probably a good default assumption, but it is not always true:

* A User-Agent can, either automatically or under interactive
  user direction, decide to retry non-idempotent requests. An
  intermediate must never do this.

* An origin server is always authoritative, an intermediate is
  not and so can sometimes not make certain decisions that an
  origin could. (See If-Match below for an example.)

2) On page 13 it states:

   thereby letting the recipient know that more advanced
   features can be used in response (by servers) or in future requests
   (by clients).

and on page 56 it states:

     o  Proxies SHOULD maintain a record of the HTTP version numbers
        received from recently-referenced next-hop servers.

A server does not appear to be committed to supporting the same HTTP version from request to request, or from one URL to another on the same server. (As an example at the same address and under the same vhost, some URLs might be served by the "real" http server which fully supports HTTP/1.1, and some by CGI scripts which might only support HTTP/1.0.)

Therefore it seem unwise to rely on the upstream server supporting a particular version from request to request, unless there is a tighter coupling between them (for example, under the same organisational control) and the intermediate can be configured to specifically assume a given HTTP version for a given upstream.

The above is also unclear about what constitutes a "next-hop server": if the hostname resolves to multiple addresses, they are not all necessarily running the same version of the server, or even the same server software at all. The same is probably even more likely for different ports on the same machine.

Different vhosts might be directed to different backends with different capabilities, and as mentioned the same is true for different URLs even on the same address/port/vhost. The most conservative interpretation would be to match on address, port, vhost and URL, but that seems to require remembering an excessive amount of history.

3) On page 18 it states:

   Resources made available via the "https" scheme have no shared
   identity with the "http" scheme even if their resource identifiers
   indicate the same authority (the same host listening to the same TCP
   port).  They are distinct name spaces and are considered to be
   distinct origin servers.

While this is true for a separate shared cache, a common deployment might be gateways employed as protocol translators/accelerators. A small number of backend origins without enough oomph to serve HTTPS directly behind a larger set of more powerful HTTPS enabled gateways under the same organisational control, linked over an internal-only private network. Here the internal traffic is "http://" requests, but the external traffic is "https://" with the obvious direct mapping of pathuris from one to the other.

The same consideration applies to other protocol elements which are normally mandatory for intermediates to add, modify in specific ways, or relay untouched. For example an organisation might not want their gateways emitting Via headers at all, so that the gateway looks more like a real origin to external clients, or may want the gateway to set specific Expires/max-age values on egress which are different from those supplied by the real origin server.

Since such requirements of the specification which are currently mandatory without exception *are* going to be violated in practice, sometimes with good reason, sometimes with less good reason, it might be useful to explore the boundaries of when intermediates can get away with this and when it is definitely wrong. It would seem that a single organisation doing this with their own gateways, or a service provider of that organisation doing so under the express direction of the organisation is on pretty solid ground, whereas an intercepting proxy controlled by a completely unrelated entity should probably never do so.

4) (Related to pages 19-21) A non-conforming implementation might generate octets, or sequences of octets, that are out of spec. For example:

  * URLs containing unencoded NUL bytes
  * URLs containing unencoded CTL or high-bit bytes
  * NUL bytes anywhere elsewhere within the start-line or headers
  * CTLs or high-bit bytes elsewhere within the start-line or headers
  * non-token non-SP characters within the method
  * multiple SP characters, or alternative whitespace, separating
    start-line elements
  * non-token non-whitespace characters within a field-name
  * invalid (%ZZ) or truncated (%4z in the middle or %4 at the end)
    %-hex sequences in the URL.

No specific behaviour appears to be mandated under these conditions, so it appears to be valid for a server to attempt to satisfy the request as received, or an intermediate to relay them. An intermediate might not even notice the invalid characters if it has no other reason to parse the affected protocol element.

In practice origin servers mostly generate 4xx errors, and an intermediate might either generate a direct 4xx or relay and rely on the origin doing so. NUL bytes in URLs are especially dangerous and a known attack vector, so it would seem sensible that detecting and generating a 400 there ought to be mandatory. Probably with NULs elsewhere too. The others could be argued either way but it would be good to have some specific direction in the specification.

And although 3.1 disallows whitespace between the start-line and first header, no specific reaction appears to be mandated there either. The 3.2.2 rules seem like a suitable approach.

5) (The Note at the top of page 23)

Set-Cookie2/Cookie2 don't appear to ever have had widespread usage, and are now officially deprecaded by RFC 6265. Pointing out that they do not suffer from a problem in Set-Cookie/Cookie, which might be taken as encouraging their use, without pointing this out seems odd.

6) On page 56 it states:

   o  A proxy MUST NOT forward a 100 (Continue) response if the request
      message was received from an HTTP/1.0 (or earlier) client and did
      not include an Expect header field with the "100-continue"
      expectation.  This requirement overrides the general rule for
      forwarding of 1xx responses (see Section 7.1 of [Part2]).

Although I *think* this is an improvement over previous wordings, it does still seem to mandate that a HTTP/1.1 proxy forward a 100 Continue response to a HTTP/1.0 request which included an "Expect: 100-continue" header. Since a HTTP/1.0 downstream proxy is likely to have blindly forwarded this header, but not be capable of understanding a 100 Continue response this seems strange.

(It appears that this has been discussed several times on the httpbis list without consensus being reached.)

The downstream could do any number of bad things in response to a relayed 100 response: cache the final response headers and body as if they were both the response body. Get "stuck" waiting for the "end" of the message that it never sees. Pass the entirety of the following data on the connection to the wrong client.

It seems safest to simply state that a 100 Continue response must *never* be transmitted in response to a HTTP/1.0 request.

Secondly, the same applies to any other 1xx response, but even the above exception doesn't apply to them: a 1.1 proxy must apparently always forward a 1xx response that it knows the downstream will probably not be able to parse.

Thirdly, the client requirements state "the client SHOULD NOT wait for an indefinite period before sending the request body." I think 2616 only applied this to user-agents and not proxies, so just to clarify, is it intended that the above language now apply to intermediates as well? (I think it should, but although a user-agent can simply start transmitting the body on timeout,
an intermediate may need to synthesize a 100 Continue response to force the downstream to start transmitting. Assuming that the downstream is HTTP/1.1 capable of course...)

Thanks
Ben

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 24 Apr 2012 15:56:52 +1200

>   intermediate must never do this.

Why do you say "never"?
  there are several cases where intermediaries can and should re-try 
automatically. TCP connection handshake failures for example. All cases 
which require user interaction they can relay back to their own client 
for that interaction.

>   origin could. (See If-Match below for an example.)

An intermediary can produce authoritative answers inferred from known 
state. Not in as many cases as an origin which has access to all state, 
true. But still some cases can be responded.

Overall a proxy has all of the relevant role, with additional 
limitations on how it handles things "internally" to meet those external 
server/client role requirements.

> A server does not appear to be committed to supporting the same HTTP
> version from request to request, or from one URL to another on the
> same server. (As an example at the same address and under the same
> vhost, some URLs might be served by the "real" http server which 
> fully
> supports HTTP/1.1, and some by CGI scripts which might only support
> HTTP/1.0.)

section 2.6 paragraph 7
   "Intermediaries that process HTTP messages (i.e., all intermediaries
    other than those acting as tunnels) MUST send their own HTTP-version
    in forwarded messages."

In the example you put forward, the vhost is a gateway intermediary for 
the CGI script origin. The CGI has its own Server: header and version. 
The gateway vhost has its own Server: header and version. Te relaying 
vhost is responsible for upgrading the CGI responses to its 1.1 version.

The vhost being a gateway intermediary for the CGI is required to 
downgrade the request for the CGI capabilities, and upgrade the 
responses with its 1.1 version.

There is one exception. And that is where the client request arrives 
with an older version. The server MAY downgrade its version to that 
supported by the client.

>
> Therefore it seem unwise to rely on the upstream server supporting a
> particular version from request to request, unless there is a tighter
> coupling between them (for example, under the same organisational
> control) and the intermediate can be configured to specifically 
> assume
> a given HTTP version for a given upstream.
>
> The above is also unclear about what constitutes a "next-hop server":
> if the hostname resolves to multiple addresses, they are not all
> necessarily running the same version of the server, or even the same
> server software at all. The same is probably even more likely for
> different ports on the same machine.

They are all potential next-hop, and apparently not the same server.

>
> Different vhosts might be directed to different backends with
> different capabilities, and as mentioned the same is true for
> different URLs even on the same address/port/vhost. The most
> conservative interpretation would be to match on address, port, vhost
> and URL, but that seems to require remembering an excessive amount of
> history.

Whatever software is doing the directing is responsible for ensuring 
consistency in capabilities support and version advertisements.

> While this is true for a separate shared cache, a common deployment
> might be gateways employed as protocol translators/accelerators. A
> small number of backend origins without enough oomph to serve HTTPS
> directly behind a larger set of more powerful HTTPS enabled gateways
> under the same organisational control, linked over an internal-only
> private network. Here the internal traffic is "http://" requests, but
> the external traffic is "https://" with the obvious direct mapping of
> pathuris from one to the other.

Stop right there. This assumed URL-rewriting is directly violating the 
above clause.

The HTTP compliant behaviour in this TLS/no-TLS gateway intermediary is 
to map HTTPS (port 443) traffic to https:// URLs sent in plain HTTP 
requests over the internal non-TLS connection.

Any client facing http://example.com/ and https://example.com/ can and 
SHOULD assume they are different as are ftp://example.com/ and 
http://example.com/. Any server internal URL equivalence is exactly that 
*internal* to the server and must not be assumed by the client.

AYJ

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 24 Apr 2012 07:33:18 +0200

Hi Amos,

> with an older version. The server MAY downgrade its version to that 

I didn't think about this point on versions, but now it scares me : if
an intermediary pretends to be 1.1-compatible to a client when it talks
to an 1.0 server, it may incite the client to know this and use some of
the 1.1 extensions (eg: chunked encoding in requests) that the server
will never be able to deal with and that are not always transformable
by the intermediary. Similarly, if the intermediary pretends to the
server that the request is 1.1 while the client was 1.0, it may incite
the server to use such features (eg: chunked encoding again) that is
impossible to transform to the client's version.

I think that we should make the intermediary present the highest supported
version between his and the original message, otherwise we'll cause huge
interoperability issues.

Well in fact I think that the issues comes down again to the boundary
between tunnels and other intermediaries, because some intermediaries
will minimally change the messages (eg: add an X-Forwarded-For or Via
header) but will not affect the contents. In this regard they could be
seen as tunnels though they're slightly more active and closer to
gateways.

Anyway, I think that the rule above requires infinite buffer to be
strictly applied, which is problematic.

Regards,
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 24 Apr 2012 18:20:03 +1200

> impossible to transform to the client's version.

What features exactly do you see as non-translatable?

Chunked is not a good example there. It is trivially de-chunked into a 
unknown-lenth 1.0 entity.

The key word in yoru description is "pretends". Intermediaries that fake 
support without actually doing it will get themselves into trouble in 
some cases. The spec clauses are talking about actually compliant 
intermediaries.

> version between his and the original message, otherwise we'll cause huge

It has not caused major issues to date with the 1.0->1.1 transition. The 
bigger issue has been supposedly 1.1-compatible clients not re-trying 
when presented with 417 and 412 responses. And server ignoring explicit 
1.0 version requests and sending chunked responses regardless.

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 24 Apr 2012 09:10:52 +0200

> unknown-lenth 1.0 entity.

Except that it degrades the connection on two points :
  - loss of persistent connections
  - loss of termination indication

Also it is not possible to do that for the client. If a client wishes to
upload a payload using chunked encoding because it believes the server
supports it, the intermediary will have to upload then close the connection
to mark the end of data, which will generally not work.

Without advertising 1.1 when not necessary, each side would have done the
right thing to accomodate for the other side's version.

> some cases. The spec clauses are talking about actually compliant 

I'm not discussing "pretending" to support one version while not being
able to do it, it's precisely what causes interop issues with intercepting
intermediaries. I'm really using the word "pretends" in that it tells one
side the other is able to reliably and efficiently consume a message while
in order to make this possible, the message will have to be significantly
degraded, well below what would have been possible without advertising a
version the other side does not support.

> when presented with 417 and 412 responses.

Well this is a typical example of what can happen if an intermediary
announces itself as 1.1 when the client is only 1.0.

> And server ignoring explicit 

I've already seen this one too :-)
Another common issue is servers which don't advertise a content-length
nor chunked encoding when they get a connection: close in requests
because internally the fake an 1.0 connection !

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 Apr 2012 17:21:52 +1000

Hi Ben,

Just so it's clear: p1 is NOT in WGLC right now; you're welcome to review it, but the document is still in flux. 

If you think there are substantive issues in the document (i.e., things that are not likely to be changed in an editorial rewrite), you're welcome to raise issues now, but it may be more productive to wait until it settles a bit.

Finally, if/when you do raise issues, please include the document revision you're making comments upon, and we prefer substantive issues to be raised one-per-email.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Tue, 24 Apr 2012 16:48:45 +0100

Mark,

> Just so it's clear: p1 is NOT in WGLC right now; you're welcome to review it, but the document is still in flux. 

Understood.

> Finally, if/when you do raise issues, please include the document revision you're making comments upon, and we prefer substantive issues to be raised one-per-email.

Apologies, I forgot to include that. John is reviewing -19 of all the drafts. I didn't put the p1 comments in separate emails because it isn't in WG LC yet but I will do that from now on (and continue to put any further comments on p4-p7 in separate mails).

Ben