Subject: Re: #273: HTTP-Version should be redefined as fixed length pair of DIGIT . DIGIT

From: Mark Nottingham <[email-masked]>
Date: Wed, 22 Jun 2011 11:10:21 +1000

OK, setting the milestone for -15 for this, as there doesn't seem to be any objection.

On 27/03/2011, at 7:34 PM, Willy Tarreau wrote:

> On Sun, Mar 27, 2011 at 10:26:18AM +0200, Mark Nottingham wrote:
>> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/273>
>> 
>> Any thoughts / objections to making this change?
> 
> I think it's a good idea. Too many implementations I've seen just
> check for ver[7] == '1' or just memcmp(ver, "HTTP/1.1", 8). If
> we ever decide to use 1.10 as a valid value, we'll get a lot of
> trouble due to that.
> 
> Willy
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 22 Jun 2011 14:36:05 +0200

> OK, setting the milestone for -15 for this, as there doesn't seem to be any objection.

So how does this affect a recipient of a message with multiple digits.

Consider:

   GET / HTTP/1.10
   Host: example.com

...being received by an HTTP/1.1 server implemented according to HTTPbis.

Should it reject the message as invalid?

(I don't have a problem with that, but it seems we need to state that 
one way or the other)

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 23 Jun 2011 08:08:46 +1000

It should treat it the same as receiving a message like this:

GET / HTTP/A.B

which hopefully we cover already...

On 22/06/2011, at 10:36 PM, Julian Reschke wrote:

> (I don't have a problem with that, but it seems we need to state that one way or the other)
> 
> Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 23 Jun 2011 10:22:18 +1200

> which hopefully we cover already...

 Indeed.
 "
    When comparing HTTP versions, the numbers MUST be compared
    numerically rather than lexically.  For example, HTTP/2.4 is a lower
    version than HTTP/2.13, which in turn is lower than HTTP/12.3.
    Leading zeros MUST be ignored by recipients and MUST NOT be sent.
 "

 The implementations cited are in violation of a MUST.

 I'm more inclined to propose that this update clarify the version as 
 "HTTP/1.2" now. Which if the earlier post was correct was "merely" a 
 text omission in the 2616 development.

 AYJ

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 23 Jun 2011 09:59:29 +1000

On 23/06/2011, at 8:22 AM, Amos Jeffries wrote:

> I'm more inclined to propose that this update clarify the version as "HTTP/1.2" now. Which if the earlier post was correct was "merely" a text omission in the 2616 development.

That's explicitly out of charter for us:

"The Working Group must not introduce a new version of HTTP and should not add new functionality to HTTP."

<http://datatracker.ietf.org/wg/httpbis/charter/>

Of course, if there's consensus to change our charter, we can take that to the IESG and see what they say, but if we are going to go down that road (and I'd be quite wary of doing so), let's wait until we've got everything else done.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Thu, 23 Jun 2011 09:53:31 +0200

> which hopefully we cover already...

Really?

I would have thought that an HTTP/1.1 compliant recipient would accept 
something labeled HTTP/1.10.

If it does, and we change the spec, we will make it non-compliant.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 24 Jun 2011 08:29:54 +1000

Good point.

It appears that we can either

a) define different BNF for sending and receiving (as we have elsewhere), or

b) close the issue with no change.

Roy, this was yours -- any thoughts?

On 23/06/2011, at 5:53 PM, Julian Reschke wrote:

> I would have thought that an HTTP/1.1 compliant recipient would accept something labeled HTTP/1.10.
>

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 24 Jun 2011 06:58:43 +0200

> something labeled HTTP/1.10.

Julian, I have seen many times the version checked with :

    memcmp(version, "HTTP/1.1", 8)

One of the reasons probably is that the next char might be either a CR or
an LF, so it's not easy to quickly check for an exact version. But as a
result of the above, "HTTP/1.10" would be matched as "HTTP/1.1".

> If it does, and we change the spec, we will make it non-compliant.

I don't think this is an issue, because even if the server understands
HTTP/1.10 as major=1, minor=10, it will just not know this version, and
the draft states that such a version will not exist anyway since only
one digit can be used for the minor.

In my opinion, it would be an issue if we had already used such a version,
which is not the case. Even HTTP/0.9 was post-named with a single digit.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 24 Jun 2011 09:23:28 +0200

> result of the above, "HTTP/1.10" would be matched as "HTTP/1.1".

Yes.

> one digit can be used for the minor.

It won't know the version, but it doesn't need to, as 10 > 1.

> which is not the case. Even HTTP/0.9 was post-named with a single digit.

True.

So, summarizing: it *could* make existing implementations non-compliant, 
but it really doesn't matter as there are no real-world HTTP messages 
that are affected (like in "outside test cases").

So we would:

- simplify the ABNF (that's the whole point, right?)

- adjust the prose

- mark this as change from RFC 2616

?

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 24 Jun 2011 15:28:13 +0200

> It won't know the version, but it doesn't need to, as 10 > 1.

Agreed as long as we don't change any minor semantics between 1.1 and 1.10
(eg: the differences between 1.0 and 1.1 about persistent connections
becoming default). If we decided that we'd go up to .10, the problem would
suddenly arise that for many implementations, .10 would be matched as .1
and appear lower as .2.

> - mark this as change from RFC 2616

Yes that would seem the best solution.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 24 Jun 2011 22:16:26 +0200

On 2011-06-24 15:28, Willy Tarreau wrote:
>> It won't know the version, but it doesn't need to, as 10>  1.
>
> ...

OK, proposed patch at 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/273/273.diff>. 
The subsection would read:

2.5.  Protocol Versioning

    HTTP uses a "<major>.<minor>" numbering scheme to indicate versions
    of the protocol.  This specification defines version "1.1".  The
    protocol version as a whole indicates the sender's compliance with
    the set of requirements laid out in that version's corresponding
    specification of HTTP.

    The version of an HTTP message is indicated by an HTTP-Version field
    in the first line of the message.  HTTP-Version is case-sensitive.

      HTTP-Version   = HTTP-Prot-Name "/" DIGIT "." DIGIT
      HTTP-Prot-Name = %x48.54.54.50 ; "HTTP", case-sensitive

    The HTTP version number consists of two non-negative decimal digits
    separated by a "." (period or decimal point).  The first number
    ("major version") indicates the HTTP messaging syntax, whereas the
    second number ("minor version") indicates the highest minor version
    to which the sender is at least conditionally compliant and able to
    understand for future communication.  The minor version advertises
    the sender's communication capabilities even when the sender is only
    using a backwards-compatible subset of the protocol, thereby letting
    the recipient know that more advanced features can be used in
    response (by servers) or in future requests (by clients).

    When an HTTP/1.1 message is sent to an HTTP/1.0 recipient [RFC1945]
    or a recipient whose version is unknown, the HTTP/1.1 message is
    constructed such that it can be interpreted as a valid HTTP/1.0
    message if all of the newer features are ignored.  This specification
    places recipient-version requirements on some new features so that a
    compliant sender will only use compatible features until it has
    determined, through configuration or the receipt of a message, that
    the recipient supports HTTP/1.1.

    The interpretation of an HTTP header field does not change between
    minor versions of the same major version, though the default behavior
    of a recipient in the absence of such a field can change.  Unless
    specified otherwise, header fields defined in HTTP/1.1 are defined
    for all versions of HTTP/1.x.  In particular, the Host and Connection
    header fields ought to be implemented by all HTTP/1.x implementations
    whether or not they advertise compliance with HTTP/1.1.

    New header fields can be defined such that, when they are understood
    by a recipient, they might override or enhance the interpretation of
    previously defined header fields.  When an implementation receives an
    unrecognized header field, the recipient MUST ignore that header
    field for local processing regardless of the message's HTTP version.
    An unrecognized header field received by a proxy MUST be forwarded
    downstream unless the header field's field-name is listed in the
    message's Connection header-field (see Section 9.1).  These
    requirements allow HTTP's functionality to be enhanced without
    requiring prior update of all compliant intermediaries.

    Intermediaries that process HTTP messages (i.e., all intermediaries
    other than those acting as a tunnel) MUST send their own HTTP-Version
    in forwarded messages.  In other words, they MUST NOT blindly forward
    the first line of an HTTP message without ensuring that the protocol
    version matches what the intermediary understands, and is at least
    conditionally compliant to, for both the receiving and sending of
    messages.  Forwarding an HTTP message without rewriting the HTTP-
    Version might result in communication errors when downstream
    recipients use the message sender's version to determine what
    features are safe to use for later communication with that sender.

    An HTTP client SHOULD send a request version equal to the highest
    version for which the client is at least conditionally compliant and
    whose major version is no higher than the highest version supported
    by the server, if this is known.  An HTTP client MUST NOT send a
    version for which it is not at least conditionally compliant.

    An HTTP client MAY send a lower request version if it is known that
    the server incorrectly implements the HTTP specification, but only
    after the client has attempted at least one normal request and
    determined from the response status or header fields (e.g., Server)
    that the server improperly handles higher request versions.

    An HTTP server SHOULD send a response version equal to the highest
    version for which the server is at least conditionally compliant and
    whose major version is less than or equal to the one received in the
    request.  An HTTP server MUST NOT send a version for which it is not
    at least conditionally compliant.  A server MAY send a 505 (HTTP
    Version Not Supported) response if it cannot send a response using
    the major version used in the client's request.

    An HTTP server MAY send an HTTP/1.0 response to an HTTP/1.0 request
    if it is known or suspected that the client incorrectly implements
    the HTTP specification and is incapable of correctly processing later
    version responses, such as when a client fails to parse the version
    number correctly or when an intermediary is known to blindly forward
    the HTTP-Version even when it doesn't comply with the given minor
    version of the protocol.  Such protocol downgrades SHOULD NOT be
    performed unless triggered by specific client attributes, such as
    when one or more of the request header fields (e.g., User-Agent)
    uniquely match the values sent by a client known to be in error.

    The intention of HTTP's versioning design is that the major number
    will only be incremented if an incompatible message syntax is
    introduced, and that the minor number will only be incremented when
    changes made to the protocol have the effect of adding to the message
    semantics or implying additional capabilities of the sender.
    However, the minor version was not incremented for the changes
    introduced between [RFC2068] and [RFC2616], and this revision is
    specifically avoiding any such changes to the protocol.

and the change log (B.2) would read:

    Clarify that the string "HTTP" in the HTTP-Version ABFN production is
    case sensitive.  Restrict the version numbers to be single digits due
    to the fact that implementations are known to handle multi-digit
    version numbers incorrectly.  (Section 2.5)

I also checked for examples that use multiple digits and couldn't find any.

Feedback appreciated,

Julian

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 25 Jun 2011 00:14:10 +0200

Hi Julian,

>    The HTTP version number consists of two non-negative decimal digits

Since "integer" was changed to "digits" here, maybe we can remove the
"non-negative" precision ?

Otherwise the rest is OK to me.

> I also checked for examples that use multiple digits and couldn't find any.

I did a few quick checks on some sources I have here (some being quite outdated) :
  - thttpd : only checks if version string == HTTP/1.0, everything else is 1.1
  - mini-httpd : same
  - tux : checks for minor == 1 (first digit)
  - varnish : compares version string with "HTTP/1.0" and "HTTP/1.1", everything
       else is 0.9.
  - haproxy : sets 1.1 when length == 8 and ((major > 1) or (major == 1 and
       minor >= 1)) (one digit for each part). So 1.10 reports 1.0.
  - apache : does sscanf("%u.%u") and accepts minors up to 999.
  - teepeedee : makes use of strtoul() on both major and minor
  - squid : does sscanf("%d.%d"). Not sure what it does with negatives. This
    was a pretty old version however (2.5-stable12), that might not count.
  - lighttpd : uses strtol() on both major and minor (so might accept negatives)
       but checks for major==1 and minor==1 (or minor==0) to report 1.1 or 1.0
       respectively, the rest being rejected.
  - nginx : I was not sure

Given the diversity of methods, I think it's really nice that we can
simplify the parsing.

Regards,
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 25 Jun 2011 17:11:32 +1200

> "non-negative" precision ?

The text above is also overriding the possibility of double-digit major 
versions. Implying "the version" only contains 2 digits in total.

How about:
   consists of two sequences of decimal digits separated by one dot

Followed by clarification on the size limits of those sequences.

>    - haproxy : sets 1.1 when length == 8 and ((major>  1) or (major == 1 and
>         minor>= 1)) (one digit for each part). So 1.10 reports 1.0.

squid-2 all do that same sscanf. negative numbers get an 4xx error page.

squid-3 use isdigit() for any length numeric as per the spec. rejects 
with 505 on anything other than 1.0 or 1.1.

> Given the diversity of methods, I think it's really nice that we can

I disagree that the parsing is complex here. Just plain wrong in several 
of those cases. It is a huge amount simpler to check for valid version 
than valid method.

AYJ

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 25 Jun 2011 08:00:29 +0200

>   consists of two sequences of decimal digits separated by one dot

Mark suggested that at the speed the protocol is progression, there are
little chances that we'll ever see a two-digit major. I tend to agree a
lot with him on this, considering we remained on the same major+minor for
the last 14 years, and that emitting anything not starting with HTTP/1.
will be rejected at many places as previous check showed it. Simply send
HTTP/2.0 and we'll see various behaviours such among which :
  - a few considering it as 1.1
  - some considering it as 1.0
  - some considering it as 0.9 (since it matches neither 1.0 nor 1.1)
  - some rejecting it as invalid

(...)
> with 505 on anything other than 1.0 or 1.1.

OK thanks for the double check.

> of those cases. It is a huge amount simpler to check for valid version 

The diversity of the behaviours seems to imply that implementers disagree
with you. For instance, Apache is doing this in order to get both speed
in the common case and compliance in other cases (taken from 2.2.4) :

    /* Avoid sscanf in the common case */
    if (len == 8
        && pro[0] == 'H' && pro[1] == 'T' && pro[2] == 'T' && pro[3] == 'P'
        && pro[4] == '/' && apr_isdigit(pro[5]) && pro[6] == '.'
        && apr_isdigit(pro[7])) {
        r->proto_num = HTTP_VERSION(pro[5] - '0', pro[7] - '0');
    }
    else if (3 == sscanf(r->protocol, "%4s/%u.%u", http, &major, &minor)
             && (strcasecmp("http", http) == 0)
             && (minor < HTTP_VERSION(1, 0)) ) /* don't allow HTTP/0.1000 */
        r->proto_num = HTTP_VERSION(major, minor);
    else
        r->proto_num = HTTP_VERSION(1, 0);

All this just to decide between HTTP/1.0 or 1.1, and still it does not
support minors greater than 999. But the check is overall right (except
for the strcasecmp on the HTTP protocol, which was recently stated as
case sensitive).

With that complexity just to get a 1.0 vs 1.1 right when it's already
available at the cost of a single character compare or memcmp(), there
is no doubt we'll continue to see the simplified version for a long
time, especially in environments where this parsing is considered as
expensive for no added value.

I perfectly understand why some would prefer doing something like :

    if (memcmp(version, "HTTP/1.", 7) == 0)
       ver = 11 - (version[7] == '0');
    else
       ver = 9;

and use ver = 9, 10 or 11 for HTTP/0.9, 1.0 or 1.1. It's a lot cheaper,
always works for known versions and requires a lot less complexity to
get it right.

In fact, every time a spec will propose a wide range of possible values
where only a very small subset is used (small enough to fit in a boolean),
you can be sure that there will be substantial trade-offs between full
compliance and simplicity. This is well illustrated above by Apache which
uses a single integer for both major and minor (which is perfectly fine
in my opinion). The only drawback is that it's limited to minors between
0 and 999. We can all agree that it's enough, but it could as well have
tried to seek for even wider compliance and use 2 integers to store the
version, making the code more complex everywhere for no real benefit.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 25 Jun 2011 09:40:43 +0200

> "non-negative" precision ?

Oops. I know I had this right, but somehow it didn't end up in the 
patch. See updated patch at 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/273/273.diff>.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 25 Jun 2011 10:01:47 +0200

Hi Julian,

> <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/273/273.diff>.

Another point on wording, should we say "The first number" or "The first digit" ?

The risk of ambiguity is very low, but we're saying that the "HTTP version
number" is composed of two digits, then when we say "the first number", it
might be possible to read it as the HTTP version number composed from the
two digits. I think that talking only about digits removes any doubt :

    The HTTP version number consists of two decimal digits separated by a "." 
    (period or decimal point).  The first digit ("major version") indicates the 
    HTTP messaging syntax, whereas the second digit ("minor version") indicates 
    the highest minor version to which the sender is at least conditionally 
    compliant and able to understand for future communication.

Best regs,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 25 Jun 2011 10:39:23 +0200

>      compliant and able to understand for future communication.

Indeed; I have updated the patch accordingly.

2.5 now would read:

2.5.  Protocol Versioning

    The HTTP version number consists of two decimal digits separated by a
    "." (period or decimal point).  The first digit ("major version")
    indicates the HTTP messaging syntax, whereas the second digit ("minor
    version") indicates the highest minor version to which the sender is
    at least conditionally compliant and able to understand for future
    communication.  The minor version advertises the sender's
    communication capabilities even when the sender is only using a
    backwards-compatible subset of the protocol, thereby letting the
    recipient know that more advanced features can be used in response
    (by servers) or in future requests (by clients).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 25 Jun 2011 08:52:51 +0000

>      HTTP-Prot-Name = %x48.54.54.50 ; "HTTP", case-sensitive

Is that a syntax we've used anywhere else ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 25 Jun 2011 10:58:10 +0200

On 2011-06-25 10:52, Poul-Henning Kamp wrote:
> In message<[email-masked]>, Julian Reschke writes:
>

That's the syntax we use if things are case-sensitive (literal strings 
in RFC 5234 ABNF are not).

Best regards, Julian

---

From: Dzonatas Sol <[email-masked]>
Date: Sun, 26 Jun 2011 20:16:59 -0700

>    the set of requirements laid out in that version's corresponding

Would this help reconcile? HTTP uses the well-known "one.one" number 
scheme to indicate the version of the transfer-protocol; this 
specification defines "1.1" as the version number.

[Happy two oh one one.]

-- 
--- http://twitter.com/Dzonatas_Sol ---
Web Development, Software Engineering
Ag-Biotech, Virtual Reality, Consultant

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Jun 2011 13:41:46 +0200

> specifically avoiding any such changes to the protocol.

...applied with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1313>.

Best regards, Julian

---

From: Karl Dubost <[email-masked]>
Date: Wed, 29 Jun 2011 14:40:40 -0400

About versioning :)

>>> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/273>
[…]
>> Too many implementations I've seen just
>> check for ver[7] == '1' or just memcmp(ver, "HTTP/1.1", 8). 

Definitely an issue. It has been one of the major pain points of the Opera Browser. It has less impact I believe in the world of servers but still. 

Opera has been the first browser to get two digits by going from "9.80" to "10.0", a few years ago. When this transition happened, the browser stopped working for a number of Web sites. Libraries were identifying the User-Agent version has "1" instead of "10" by grabbing the first digit of the string.

Opera had to revert and create a funky user agent string which is still used now. Th real version number is now at the end.
User-Agent: Opera/9.80 (Macintosh; Intel Mac OS X 10.6.7; U; fr) Presto/2.9.168 Version/11.50

You could think that with the quick evolution of numbering by Google Chrome, it would have solved the issue. Well not really, people have codepath such as Opera then version number. If Opera is switching to a more reasonable string such as 

User-Agent: Opera/11.50 (Macintosh; Intel Mac OS X 10.6.7; U; fr) Presto/2.9.168
Some old scripts might break and the new scripts will break too. User agent sniffing is evil 

Back to the HTTP version number, there is indeed chances it will break. 

# Strategies: 

* Identity which are the most common libraries, programs which are likely to break
  and working with the implementers before the release of HTTPbis. 
* Not creating a new version at all, if we consider HTTPbis as just a fix of HTTP 1.1
* Creating a version number which will not break in the current deployed codes

# Questions

* Are there products which really depends on the HTTP version number for interoperability?
* If yes, in which ways
* What are these products?

#sniffkill
-- 
Karl Dubost - http://dev.opera.com/
Developer Relations & Tools, Opera Software

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 30 Jun 2011 13:20:52 +1200

On Wed, 29 Jun 2011 14:40:40 -0400, Karl Dubost wrote:
> Definitely an issue. It has been one of the major pain points of the
> Opera Browser. It has less impact I believe in the world of servers
> but still.
>
> Opera has been the first browser to get two digits by going from
> "9.80" to "10.0", a few years ago. When this transition happened, the
> browser stopped working for a number of Web sites. Libraries were
> identifying the User-Agent version has "1" instead of "10" by 
> grabbing
> the first digit of the string.
>
> Opera had to revert and create a funky user agent string which is
> still used now. Th real version number is now at the end.
> You could think that with the quick evolution of numbering by Google
> Chrome, it would have solved the issue. Well not really, people have
> codepath such as Opera then version number. If Opera is switching to 
> a
> more reasonable string such as
>

 Definitely worthwhile regardless of any consensus here.
  "Every bug unreported kills a good idea somewhere."(c)

> * What are these products?

 Any software written to comply with RFC 2616 section 3.1 as published
 "
 caching proxies MUST, gateways MAY [...] upgrade the request to the 
 highest version they support. The proxy/gateway's response to that 
 request MUST be in the same major version as the request.

       Note: Converting between versions of HTTP may involve 
 modification
       of header fields required or forbidden by the versions involved.
 "

 So any compliant middleware will be relying on HTTP version numbers for 
 both client and server to map the hop-by-hop functionality, encoding, 
 and possibly content types in transit.

 AYJ