Subject: Revising RFC2616 - what's happening

From: Mark Nottingham <[email-masked]>
Date: Wed, 18 Oct 2006 08:29:50 -0700

After the Bar Bof (summary: <http://lists.w3.org/Archives/Public/ietf- 
http-wg/2006JulSep/0010.html>), I've been talking to a lot of people  
about the prospect of revising RFC2616.

Most everyone I've talked to agrees on a few things;
    * Focus on errata and clarifications
    * Don't introduce a new version of HTTP
    * Don't break compatibility (a prerequisite for an unchanged  
version)
    * Skew towards implementation experience
    * If it's clearly uncontroversial, deprecate unused/non- 
interoperable features
    * If in doubt, don't change it (do no harm)
    * Don't try to make it perfect, just better

I've also seen a fair amount of private interest in working along  
these lines; not only from people using HTTP for machine-to-machine  
communication (e.g., Atom, AJAX) but also from traditional browser  
and server vendors.

The question now is whether that interest will become public and  
translate into activity. Jim Gettys' efforts at 2616bis didn't gain  
traction because he didn't feel that there was an appropriate pool of  
reviewers, and similarly we're concerned that while a lot of people  
might want to see this work done, we're less sure that enough people  
will actually do the work.

To help find out, a few things have been happening;
    * Yves Lafon and Julian Reschke have published an I-D that re- 
states RFC2616 using xml2rfc, so that people can verify it's a  
faithful transcription. Soon, they'll publish an -01 that  
incorporates the errata that Scott has captured in <http://purl.org/ 
NET/http-errata> (which didn't require additional discussion).
    * I'm working on an issues list that captures all of the problems  
that have popped up on the list, so that we can track proposed errata  
and clarifications. Expect to see the first revision of that soon.

We also discussed having a BoF in San Diego with the ADs, but decided  
that there hadn't yet been enough activity to make it successful.  
We're hoping that having a draft and an issues list available will  
generate discussion that will demonstrate the community's interest.

Further down the road, we'll need to figure out if, when and how a WG  
should be (re-)formed. There's also the opportunity to do interop  
work, and perhaps even a test suite (something that I've heard a few  
people express interest in), if enough people are willing to do it.

This is where we're at now; the next steps are to get the issues list  
out and start discussion, so as to give us information about the  
level of interest and commitment in moving forward. If you have  
concerns about the scope or path forward, please bring them up to the  
list; likewise, if you're willing to help, we'd love to hear about it.

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 18 Oct 2006 19:08:18 +0200

Hi Mark,

thanks for the update on the situation. I personally very much  
welcome the effort to make the HTTP/1.1 spec a better HTTP/1.1 spec.

A better spec insofar as to bring the specification and reality even  
closer together, making it easier to understand, making it easier to  
do compliant, interoperable implementations. It could also help to  
stabilize the base that other protocols like APP stand on.

This can only be a success however if the scope stays as you describe  
it. Any attempt to go beyond that will awaken tidal forces beyond  
human imagination or "open the flood gates" as Roy termed it.

I'm game to contribute what experience I have and some interop test/ 
coding sounds appealing to me  as well.

Cheers,

Stefan

Am 18.10.2006 um 17:29 schrieb Mark Nottingham:

>
> After the Bar Bof (summary: <http://lists.w3.org/Archives/Public/ 
> ietf-http-wg/2006JulSep/0010.html>), I've been talking to a lot of  
> people about the prospect of revising RFC2616.
>
> traction because he didn't feel that there was an appropriate pool  
> of reviewers, and similarly we're concerned that while a lot of  
> people might want to see this work done, we're less sure that  
> enough people will actually do the work.
>
>    * I'm working on an issues list that captures all of the  
> problems that have popped up on the list, so that we can track  
> proposed errata and clarifications. Expect to see the first  
> revision of that soon.
>
> We also discussed having a BoF in San Diego with the ADs, but  
> decided that there hadn't yet been enough activity to make it  
> successful. We're hoping that having a draft and an issues list  
> available will generate discussion that will demonstrate the  
> community's interest.
>
> Further down the road, we'll need to figure out if, when and how a  
> WG should be (re-)formed. There's also the opportunity to do  
> interop work, and perhaps even a test suite (something that I've  
> heard a few people express interest in), if enough people are  
> willing to do it.
>
> This is where we're at now; the next steps are to get the issues  
> list out and start discussion, so as to give us information about  
> the level of interest and commitment in moving forward. If you have  
> concerns about the scope or path forward, please bring them up to  
> the list; likewise, if you're willing to help, we'd love to hear  
> about it.
>
> Cheers,
>
> --

---

From: "Robert Sayre" <[email-masked]>
Date: Wed, 18 Oct 2006 21:42:28 -0400

> communication (e.g., Atom, AJAX) but also from traditional browser

I feel that a bug-fix edition of RFC 2616 would be a worthwhile
exercise. I have seen many implementors waste time on mistakes that
are covered in the errata, which are probably a subset of the issues.

I continue to be concerned by IETF management's attitude towards
security requirements.

-- 

Robert Sayre

---

From: Julian Reschke <[email-masked]>
Date: Thu, 19 Oct 2006 14:20:10 +0200

OK,

here's something cosmetic that I'd like to fix in draft 01: we have 
currently a few entries in the References section that aren't used in 
the specification text.

In particular:

1) RFC1866 (HTML 2.0) - this seems to be a leftover from RFC2068 
(definition of the LINK header, gone from RFC2616)

2) RFC2069 (Digest Auth) - again, a leftover from RFC2068, obsoleted by 
RFC2617.

3) RFC2026 (Standards Process) - only used in the front matter, I don't 
think it needs to appear in the References section.

Any objections to those being removed in draft 01?

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 19 Oct 2006 12:41:29 +0000

> are covered in the errata, which are probably a subset of the issues.

Agreed.

From experience I think it's also worthwhile to further stress the
importance of ETag uniqueness among variants of a URI. Very few
implementations get this part correct. In fact most major web servers
have issues here...

Some even strongly believe that entities with different Content-Encoding
is the same entity, arguing that since most encoding (at least the
standardized ones) can be converted to the same identity encoding so
they are in fact the same entity and should have the same strong ETag.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 19 Oct 2006 14:56:16 +0200

> they are in fact the same entity and should have the same strong ETag.

I violently agree.

Could you write up a short description of the problem, for inclusion in 
the issues list Mark N. is working on?

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 19 Oct 2006 15:21:59 +0200

>NET/http-errata> (which didn't require additional discussion).

I think it would be a mistake to keep the document in its current form
as a monolithic document; it should be split into several documents to
make the whole thing approachable. Obvious candidates for separate specs
would be caching, content negotiation, message format, and URL schemes.

>and clarifications. Expect to see the first revision of that soon.

Is there any chance to set up a usable issue tracking system?

>people express interest in), if enough people are willing to do it.

There is http://groups.yahoo.com/group/http-compliance/ btw.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 19 Oct 2006 15:40:46 +0200

> would be caching, content negotiation, message format, and URL schemes.

Not convinced. For instance, the definition of HTTP URLs currently seems 
to fit into two pages. Regarding the other areas you mention, it may be 
non-trivial to separate them properly.

> Is there any chance to set up a usable issue tracking system?

While there's nothing else in place, I'm maintaining a list (tracking 
the document changes) over at 
<http://greenbytes.de/tech/webdav/draft-lafon-rfc2616bis-issues.html>.

> There is http://groups.yahoo.com/group/http-compliance/ btw.

Well, that group doesn't seem to be active, so I'd recommend to have 
these kinds of discussions over here.

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 19 Oct 2006 16:04:05 +0200

>non-trivial to separate them properly.

I am not sure how the page count matters here. As for the other areas,
yes, that's precisely why it should be done. It may be that we do not
have the resources to actually split them, but I don't see anything
inherently bad about doing it if we do have the resources to do it,
quite the contrary.

>these kinds of discussions over here.

Perhaps, I am just saying the subscribers of that list might have use-
ful information available, or be interested to join such an effort.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 19 Oct 2006 22:56:41 +0100

> the issues list Mark N. is working on?

While on the subject of Etag and variants, it's never been clear what
a Vary header that names a header not in the original request means.

That's the obvious thing for a server to do when it consults a request
header while deciding what entity send, and the header isn't present.
Logically, the sent entity depends on the non-presence of that header
(because if it were present, the calculation on the server would
proceed differently).

For example, a server which sends different entities depending on the
value "Accept-Encoding" would usually send "Vary: Accept-Encoding" in
the response.  But what if the request doesn't have an Accept-Encoding
header?  It seems logical to me that the response should still have
"Vary: Accept-Encoding" to indicate that the response may be different
for requests that have an Accept-Encoding header, but is cacheable for
requests that don't.

But (a) is that use of Vary allowed, (b) is it consistently
interpreted that way by caches, and (c) it would be good for the
improved RFC to clarify this.

RFC 2616 doesn't say anything about variants that depend on
non-existence of request headers, inviting one to read between the
lines and make an assumption about what "same value" means for a
header that's not present.

If that use of Vary is not allowed, or is not interpreted in that way,
it implies a server would have to send "Vary:*" in those cases, which
would be unfortunate overkill as it forces round trips for subsequent
requests.

In my own servers I do set Vary to the names of all request headers
consulted in calculating the response, including those not present.
But I haven't tested the behaviour of caches, so this might be a mistake.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 01:03:56 +0200

Jamie Lokier schrieb:
> ...
> ...

For the record: I fully agree with your analysis; it seems to be the 
only way this can work in practice. We are doing the same in our server 
code, and so far haven't encountered any problems with that.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 19 Oct 2006 18:02:44 -0700

There are lots of optimisations we could make to make it cleaner,  
more clearly architected, etc., but it seems to me that going down  
that path brings a real risk of either encountering a rat-hole, or of  
making things worse.

E.g., If too much material is moved around, we risk disorienting  
implementers.

I'm not sure separate docs are necessary, but if we contemplate any  
kind of reorg, it should be *after* all errata, etc. are agreed to  
and incorporated.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 19 Oct 2006 18:02:46 -0700

Any suggestions?

For the moment, I'm looking at the XML format I've used elsewhere;
   http://www.w3.org/2002/ws/addr/issues/

The W3C's Tracker tool looks attractive, but it's pretty specific to  
W3C process and users...

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 20 Oct 2006 09:26:08 +0000

> a Vary header that names a header not in the original request means.

For me that never was a question. I immediately read Vary as the
complete list of headers the server looked for while selecting which
variant to send. But this may be just my reading of the RFC..

What isn't very clear to me was what change in Vary means on a URI.

> But (a) is that use of Vary allowed

I think so.

> (b) is it consistently interpreted that way by caches, and

Quite likely. I know at least Squid does it this way, and most others I
have seen caching Vary.

But many shared proxy caches still don't cache Vary responses at all.

> (c) it would be good for the improved RFC to clarify this.

Probably. Many get Vary wrong. But the most common error I have seen is
not sending Vary when it should have been sent and still expecting
caches to do the right thing.. But I have not seen this dependent on the
existence of the header in the request or not.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 14:28:26 +0200

> variant to send. But this may be just my reading of the RFC..

It doesn't help that "Vary" is described in two places, Section 13.6 and 
  14.44.

In 13.6 it says:

"When the cache receives a subsequent request whose Request-URI 
specifies one or more cache entries including a Vary header field, the 
cache MUST NOT use such a cache entry to construct a response to the new 
request unless all of the selecting request-headers present in the new 
request match the corresponding stored request-headers in the original 
request." (<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.13.6>)

At least this part IMHO needs to be fixed to say how to treat the case 
where a header doesn't appear in the request.

> What isn't very clear to me was what change in Vary means on a URI.

Sorry?

> But many shared proxy caches still don't cache Vary responses at all.

Well, those aren't HTTP/1.1 compliant, so there's nothing we can do 
about those.

> existence of the header in the request or not.

One of the reasons may be broken handling of Vary headers in user 
agents, mainly IE 
(<http://support.microsoft.com/default.aspx?scid=kb;en-us;824847>), but 
also Firefox (<https://bugzilla.mozilla.org/show_bug.cgi?id=269303#c30>).

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 20 Oct 2006 16:54:46 +0200

> > What isn't very clear to me was what change in Vary means on a URI.

When one cached variant has one Vary header, and then another variant is
received with a different Vary header. Lets say the first has

Vary: Accept-Language

and the second

Vary: Accept-Encoding

> Well, those aren't HTTP/1.1 compliant, so there's nothing we can do 

It's fully compliant not to cache a response. Caching is entirely
optional.

The reason why many do not cache Vary responses is

a) It wasn't very common some years ago

b) Caching of Vary quickly becomes very complex, in large due to the RFC
not being very easy to understand what Vary really means, its relations
to ETag, If-None-Match etc.

c) Many broken servers out there.

In fear of doing a non-compliant implementation or causing too much
trouble due to others malfunctions most select the safe path and doesn't
cache it at all.

> (<http://support.microsoft.com/default.aspx?scid=kb;en-us;824847>), but 

It's still HTTP compliant, just that it doesn't work when sending the
file to external applications.. If I understand the bug description
correct you should also get the same issue with their user-agent on
other "uncacheable" content.

> also Firefox (<https://bugzilla.mozilla.org/show_bug.cgi?id=269303#c30>).

The most interesting there is the reasoning about if the bug should be
considered a bug.. But it's clearly not HTTP compliant today.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 17:05:09 +0200

> received with a different Vary header. Lets say the first has

Aha.

I would say that if the value for "Vary" changes between to HTTP 
requests, the sever implementation/configuration has somehow changed, 
and a proxy should invalidate all cached entries for that URI.

> trouble due to others malfunctions most select the safe path and doesn't

Yep. Maybe we can improve proxy support by improving the spec text.

> file to external applications.. If I understand the bug description

"just"? The user follows a link and gets a completely misleading error 
message, although the server works just fine.

> other "uncacheable" content.

Yep.

Best regards, Julian

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 20 Oct 2006 08:31:54 -0700

> received with a different Vary header. Lets say the first has

The only reasonable interpretation I can think of is that
second response means that the resource has changed and
that it no longer varies by Accept-Language, and so the first
cached value should be flushed.

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 20 Oct 2006 17:56:52 +0200

> and a proxy should invalidate all cached entries for that URI.

Yes, that is what is natural to implement to the cache, but the RFC
unfortunately says nothing on the subject.

Some seem to argue that both should be cached, and the most recent match
returned by the cache. Which I personally think is both bad, nearly
impossible to implement efficiently and may results in non-deterministic
results to the end-user when there is conflicts.

Another related but less clear example which came from a server vendor
is

Vary: Accept-Language

and on another variant

Vary: Accept-Language, Accept-Encoding

seen where one language is only available in identity encoding while
another language is available in both gzip and identity encoding. The
reasoning here is that Vary lists the headers the server actually looked
for when determining which variant to return, not the headers it may
look for on all variants of the URI.

> Yep. Maybe we can improve proxy support by improving the spec text.

No doubt about that. Had to read the RFC many times before things
started to fall into place. And if every proxy, browser and server
vendor (and their coders) have to do the same no doubt there will be
broken implementations.

> message, although the server works just fine.

User-agents not behaving well to the user is not non-compliance to the
protocol specifications. And if I am not mistaken the user can select to
save the file to disk instead of opening it directly and then open it
separately outside of the browser.

How content is displayed is entirely outside of the specifications, for
good reasons.

But such issues surely drags down the deployment of affected protocol
features.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 19:46:51 +0200

> But such issues surely drags down the deployment of affected protocol

Correct. Right now this makes it impossible to me to send Vary headers 
to IE (won't display the contents to the user) or Firefox (will issue 
unneeded GET requests).

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 20 Oct 2006 11:17:15 -0700

On Oct 20, 2006, at 5:28 AM, Julian Reschke wrote:
> It doesn't help that "Vary" is described in two places, Section  
> 13.6 and  14.44.
>
> In 13.6 it says:
>
> specifies one or more cache entries including a Vary header field,  
> the cache MUST NOT use such a cache entry to construct a response  
> to the new request unless all of the selecting request-headers  
> present in the new request match the corresponding stored request- 
> headers in the original request." (<http://greenbytes.de/tech/ 
> webdav/rfc2616.html#rfc.section.13.6>)
>
> At least this part IMHO needs to be fixed to say how to treat the  
> case where a header doesn't appear in the request.

Why?  That seems completely unambiguous to me, and I have a hard time
believing that any implementer would fail to understand that a missing
header field is different from a present header field.  Would it be
more clear if we replaced "all of the" with "the set of"?  I.e.,

"When the cache receives a subsequent request whose Request-URI  
specifies one or more cache entries including a Vary header field,  
the cache MUST NOT use such a cache entry to construct a response to  
the new request unless the set of selecting request-headers present  
in the new request match the corresponding set of
stored request-headers in the original request."

> agents, mainly IE (<http://support.microsoft.com/default.aspx? 
> scid=kb;en-us;824847>), but also Firefox (<https:// 
> bugzilla.mozilla.org/show_bug.cgi?id=269303#c30>).

Yes, though for different reasons.  MSIE doesn't handle Vary because  
some
person interpreted the HTTP caching DLL to be a "shared cache" (even
though it never is in practice) because multiple user agents might use
the same DLL for requests, and it was "too complicated" to get it right.
Mainly because the library doesn't know its own header fields. *shrug*

That is why waka includes the "what this response is good for" algorithm
in each response message.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 20 Oct 2006 11:33:42 -0700

On Oct 20, 2006, at 11:27 AM, Julian Reschke wrote:
> Well, that's clearer, but not my interpretation of what it should  
> say. Don't we want a missing header to be treated the same way as  
> an empty header?

Nope.  They usually don't mean the same thing (see Accept).

> That is, if the cache has a cached response from a previous request  
> with "Accept-Language: de", and the server replied with "Vary:  
> Accept-Language", can a subsequent request without "Accept- 
> Language" header be served with the cached reply? I wouldn't have  
> thought so....

No, we have no idea what the language pref algorithm is on the server.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 20:27:17 +0200

Roy T. Fielding schrieb:
>> cache MUST NOT use such a cache entry to construct a response to the 
>> new request unless all of the selecting request-headers present in the 
>> new request match the corresponding stored request-headers in the 
>> original request." 
>> (<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.13.6>)
>>
> request unless the set of selecting request-headers present in the new 
> request match the corresponding set of

Well, that's clearer, but not my interpretation of what it should say. 
Don't we want a missing header to be treated the same way as an empty 
header?

That is, if the cache has a cached response from a previous request with 
"Accept-Language: de", and the server replied with "Vary: 
Accept-Language", can a subsequent request without "Accept-Language" 
header be served with the cached reply? I wouldn't have thought so....

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 20:46:55 +0200

> No, we have no idea what the language pref algorithm is on the server.

Right. So we don't want the cache to serve the cached response in 
absence of the request header, right?

But if we say (as you suggested):

request match the corresponding set of stored request-headers in the 
original request."

In this case, the cache would have an entry where the set of stored 
request headers is "Accept-Language", which isn't present in the new 
request. Thus, the cache would be allowed to use the cached entry, right 
(because the set of selecting response headers *present* in the request 
is empty).

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 20 Oct 2006 11:55:18 -0700

> would be caching, content negotiation, message format, and URL  

That is pretty much exactly what I said when 2616 was being edited,
and again last week when I volunteered to edit such a split.  I do not
believe that a monolithic document is reviewable, and that has been
proven twice now with 2616 and with Jim's attempt in December 2003.
I said it wasn't worth doing, and unfortunately the folks who want to
do it anyway took that as meaning I wouldn't approve of publication.

It is frustrating to have other people create a large amount of work
for me to do (reviewing other peoples' attempts to edit HTTP is easily
twice the work of editing the document myself) when I know it is going
to be a disappointment anyway, particularly given that the security
issues require an update to 2617 first.

Mark keeps noting that "the community" wants a minor update, not a
revised set of readable drafts.  I disagree.  If that is what the
community wanted, people would have reviewed Jim's draft.

I can't ignore this process because I am still responsible for one of
the primary implementations of HTTP -- Apache HTTP Server -- and we
are more directly effected by changes to the standard than any other
group due to the nature of our mission.  It is ridiculous that we have
to waste our time reviewing an unreadable document.

Cheers,

Roy T. Fielding                            <http://roy.gbiv.com/>
Chief Scientist, Day Software              <http://www.day.com/>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 20 Oct 2006 12:05:06 -0700

On Oct 20, 2006, at 11:46 AM, Julian Reschke wrote:

>>> That is, if the cache has a cached response from a previous  
>>> request with "Accept-Language: de", and the server replied with  
>>> "Vary: Accept-Language", can a subsequent request without "Accept- 
> to the new request unless the set of selecting request-headers  
> present in the new request match the corresponding set of stored  
> request-headers in the original request."
>
> request headers is "Accept-Language", which isn't present in the  
> new request. Thus, the cache would be allowed to use the cached  
> entry, right (because the set of selecting response headers  
> *present* in the request is empty).

Oh, that is not how I was interpreting "present" (the set is present,  
not
the set of present header fields).  It is actually unambiguous, but  
only if
you parse the sentence carefully.  Delete "present" if you like.

The empty set {} != {"Accept-Language: de"}, and therefore the cached
response will not be used.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 21:44:38 +0200

Roy T. Fielding schrieb:
>> new request unless the set of selecting request-headers present in the 
>> new request match the corresponding set of stored request-headers in 
>> the original request."
>>
>> request. Thus, the cache would be allowed to use the cached entry, 
>> right (because the set of selecting response headers *present* in the 
>> request is empty).
> 
> Oh, that is not how I was interpreting "present" (the set is present, not
> the set of present header fields).  It is actually unambiguous, but only if

Ah! Aha.

I think getting rid of "present" makes things clearer. How about the 
"corresponding" then? What does it refer to?

It seems to me that the spec really needs to spell out more explicitly 
what headers are being compared, what it means for a header not to be 
present, and how the comparison function for each header works.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 21:51:05 +0200

BTW: deja vu: 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2004JanMar/0021.html>

Let's try this time to write down a clarification :-)

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 20 Oct 2006 22:42:55 +0200

> absence of the request header, right?

Only if it has a matching request/response pair from a request absent of
the header.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Oct 2006 23:02:08 +0200

> Only if it has a matching request/response pair from a request absent of

I think I disagree. Maybe it's really time to come up with either 
pseudocode or precise formal definitions.

Best regards, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 20 Oct 2006 23:28:30 +0200

> pseudocode or precise formal definitions.

The "set of headers" approach mentioned resently is very easy to work
with.

Assuming a "Vary: Accept, Accept-Encoding"

GET / HTTP/1.0
Host: www.example.com

Cache key:
http://www.example.com  Vary Headers: {}

GET / HTTP/1.0
Host: www.example.com
Accept:

Cache key:
http://www.example.com  Vary Headers: {{Accept-Encoding: }}

GET / HTTP/1.0
Host: www.example.com
Accept: */*
Accept-Encoding: gzip

Cache key:
http://www.example.com  Vary Headers: {{Accept-Encoding: */*},
{Accept-Encoding: gzip}}

Absent headers only matches absent headers. Present headers only matches
present headers with the same canonical value. End.

It multiple Vary values per URI is supported by the cache then Vary as
such also need to be included in the cache key, and very careful design
to not allow out-of-sequence cached responses.

See 13.6 which among other things says

   The selecting request-headers from two requests are defined to match
   if and only if the selecting request-headers in the first request can
   be transformed to the selecting request-headers in the second request
   by adding or removing linear white space (LWS) at places where this
   is allowed by the corresponding BNF, and/or combining multiple
   message-header fields with the same field name following the rules
   about message headers in section 4.2.

which is quite unambiguous in how to compare the requests. Not sure it's
possible to define this more clearly.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 20 Oct 2006 14:36:44 -0700

On 2006/10/20, at 11:55 AM, Roy T. Fielding wrote:

> If that is what the community wanted, people would have reviewed  
> Jim's draft.

The community of HTTP  implementors today is different than it was in  
either 2003 or 1999. I think many implementers don't have the context  
of the WG at the time of publication, and are confused by the  
assumptions baked into the document. Many of the problems that people  
are tripping across could be fixed by a few extra words, not a  
wholesale re-org.

Furthermore, from what I understand there were a number of long- 
running disagreements that were buried in the interest of getting  
2616 out. While it might be easier for you if you were given carte  
blanche to re-write HTTP, I'm not convinced those disagreements  
wouldn't resurface, dragging out the process considerably, or causing  
the effort to fail.

Those are the concerns that I have. If people want a drastic re-org,  
it's achievable in a bounded amount of time, and there are enough  
people willing to do the work, great (personally, I've always wanted  
to see a more comprehensive, clearly organised document). However, so  
far only you and Bjoern have expressed interest in doing that now.  
OTOH, much of the feedback that I got when floating the idea of  
revising 2616 was concerned with overambition. An apt quote is "the  
enemy of the good is the perfect."

Roy, perhaps it would be helpful if you gave a sketch of the changes  
you have in mind, so that people have a better idea of what we're  
talking about here, and get a broader public response.

In the meantime, I think we can still talk about the small(-er) issues.

> It is ridiculous that we have to waste our time reviewing an  
> unreadable document.

I'm a little surprised by that.

If you've already implemented 2616, and a diff to the new document is  
small, it should be simple to find any changes that you need to make.  
However, if the document is radically different, it seems that you'd  
need to evaluate the new spec line-by-line to verify that you're  
conformant.

Certainly, the changes need to be weighed for interactions with the  
rest of the document, but that's true in either case. Are you saying  
that the document is so unintelligible as to justify the effort of  
coming to consensus on a whole new formulation (see above)?

Cheers,

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 20 Oct 2006 16:56:04 -0700

On Oct 20, 2006, at 2:36 PM, Mark Nottingham wrote:
> The community of HTTP  implementors today is different than it was  
> in either 2003 or 1999. I think many implementers don't have the  
> context of the WG at the time of publication, and are confused by  
> the assumptions baked into the document. Many of the problems that  
> people are tripping across could be fixed by a few extra words, not  
> a wholesale re-org.
>
> wouldn't resurface, dragging out the process considerably, or  
> causing the effort to fail.

I am sure that they would resurface, assuming that the same people  
hadn't
changed their minds given implementation experience, but they can be  
resolved
via the usual working group consensus process if the chair has the will
to enforce it.  That means nothing goes in the draft until it is proven
to be implemented, anything that wasn't implemented (aside from open
extensibility provisions) gets removed, and consensus is determined by
the implementations rather than just the number of opinions.

The biggest unresolved issue between the editors was whether to call
the payload a "representation" (as I had been doing for ages) or  
"instance"
as desired by Jeff Mogul.  We had good reasons for disagreeing, and Jim
decided not to use either, so that is all that can be said about it.
You know exactly what I would call it if I rewrote the spec.

> Those are the concerns that I have. If people want a drastic re- 
> org, it's achievable in a bounded amount of time, and there are  
> enough people willing to do the work, great (personally, I've  
> always wanted to see a more comprehensive, clearly organised  
> document). However, so far only you and Bjoern have expressed  
> interest in doing that now. OTOH, much of the feedback that I got  
> when floating the idea of revising 2616 was concerned with  
> overambition. An apt quote is "the enemy of the good is the perfect."
>
> Roy, perhaps it would be helpful if you gave a sketch of the  
> changes you have in mind, so that people have a better idea of what  
> we're talking about here, and get a broader public response.

As I said on March 6, 2006:

>> On Mar 6, 2006, at 10:39 AM, Jim Gettys wrote:
>>
>>> Still doesn't solve the problem of getting people to *read* and  
>>> *verify*
>>> the results, which is the fundamental sticking point, IMHO.  So  
>>> far, no
>>> promises from anyone I've heard.
>>
>> It isn't worth the effort.  The minimum tasks for an update include
>> re-specifying the grammar using the now-standard IETF BNF notation  
>> and
>> the RFC 3986 rules for URIs, and then split the specification into
>> readable drafts on message syntax, http/https URIs, caching, and
>> content negotiation.  I don't see that happening any time soon, since
>> it makes more sense to let MIME be updated first.

Nothing has changed between then and now.

> If you've already implemented 2616, and a diff to the new document  
> is small, it should be simple to find any changes that you need to  
> make. However, if the document is radically different, it seems  
> that you'd need to evaluate the new spec line-by-line to verify  
> that you're conformant.

The minimal changes will end up changing every line of BNF, so your
theory about this revision being identifiable from the diffs will
not be true.

Have I implemented 2616?  Why do you think there were so many editorial
mistakes, like entire paragraphs missing their first character, when  
2616
was submitted to the RFC editor?  Why do you think the RFC editor didn't
find those errors before publication?  The answer to all of those  
questions
is that NOBODY, not even the document authors, read the final submission
sufficiently to notice anything -- our eyes glazed over before we got  
that
far in the spec.  It was simply too big and the timing was wrong and the
whole "minor fixes for update to Draft Standard" was rushed through,  
while
I was trying to advance to candidacy, because any real editorial effort
was considered "too risky".

> rest of the document, but that's true in either case. Are you  
> saying that the document is so unintelligible as to justify the  
> effort of coming to consensus on a whole new formulation (see above)?

Yes!  I've been saying that since 1998, ever since section 13 was added
with 25 pages of very obscure and hard to read requirements.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 21 Oct 2006 10:58:09 +0200

Roy T. Fielding schrieb:
>>>> the results, which is the fundamental sticking point, IMHO.  So far, no
>>> re-specifying the grammar using the now-standard IETF BNF notation and

Speaking of which, *is* there any activity with the goal of updating MIME?

Anyway, I don't think doing the other updates is a waste of time. It 
reduces the work left to do once MIME is updated.

> theory about this revision being identifiable from the diffs will

The current draft has all RFC2616-style BNF fragments marked up as such 
(in the XML source).

I would imagine the activity to update the BNFs to go like this:

- use automated tools to extract all the BNFs, and produce parse trees 
for them (in some ASCII format suitable for diffing) -- if while doing 
this we find problems, fix them

- discuss how to translate to RFC4234-compliant ABNFs, in particular 
whether to keep the "#" extension (see 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.2.1.p.9>) - I 
would assume we do to keep stuff readable

- do the conversion, then run a similar set of tools against the new 
version, comparing the parse trees

Sounds like a good activity for the upcoming weeks where we can't 
publish any new drafts anyway...

> mistakes, like entire paragraphs missing their first character, when 2616

I think you're overstating the problem here. I'm not aware of any such 
problem, except the one reported in 
<http://skrb.org/ietf/http_errata.html#msg-len-chars>, which was about 
exactly one paragraph. If there are more problems like this, please 
report them.

Best regards, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 22 Oct 2006 04:35:38 +0100

> and a proxy should invalidate all cached entries for that URI.

No, no.

The natural implementation is for the server to note each time a
request header is examined to compute the response, and to emit Vary
with those headers.

That set can be different depending on the path taken by the server
logic, _without_ changing the server implementation/configuration.

Consider, for example, a server implementation that does this (this
is reasonable):

   1. Look at "Accept-Encoding" to decide if known compression methods
      are supported as a content-encoding.

   2. If compression is supported, then for some types of response,
      look at "User-Agent" to check for certain agents that do not
      behave correctly for some compressed content types.

That server would produce the following two _correct_ responses, for
different request headers:

   a. Vary: Accept-Encoding

   b. Vary: Accept-Encoding, User-Agent
      Content-Encoding: gzip

If you specify that a cache must purge all variants when receiving a
Vary header which is different from previously received Vary, then
servers will realistically have to send "Vary: Accept-Encoding,
User-Agent" even in the case that the response _doesn't_ depend on
User-Agent.

In this particular example, it's easy to imagine writing the server to
do that; i.e. to always emit the same Vary for the compression test.
Although, it would clearly be sub-optimal.

However, when the server's use of request headers is less tightly
coupled, it's _much_ harder to do that.

For example, consider a server which renders pages from templates
(PHP, Perl, etc.), as many of them do.

And imagine that a template is selected depending on the language(s)
in Accept-Language.

And that some of those templates just emit some text in an appropriate
language, but some of them contain code which looks at
Accept-Charset to make further decisions.

The server could not possibly send "Vary: Accept-Language,
Accept-Charset" for _all_ the transmited variants, because for some of
them, it executes a template which doesn't look at Accept-Charset; the
server has no information that _other_ templates would do that.

A reasonable implementation is for the server to construct Vary
depending on the headers looked at by code in the templates.  And that
results in different variants having different Vary headers, quite
reasonably, with clearly defined semantics, and quite difficult to do
otherwise.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 22 Oct 2006 14:22:42 +0200

> > and a proxy should invalidate all cached entries for that URI.

Yes yes yes ;-)

> request header is examined to compute the response, and to emit Vary

True, but this creates quite a bit of a nightmare at the cache level. So
with this requirement Vary: will still become equal to "no-store" in
most implementations, perhaps with hardcoded special cases for the most
common uses or more likely caches trying to outguess the servers and
implementing their own content negotiation schemes. This simply because
general caching of Vary entities then becomes too complicated to even
care trying to index the variants in the cache.

> User-Agent" even in the case that the response _doesn't_ depend on

Which is fine to me. Especially if the server supports ETag and
If-None-Match on larger responses.

> coupled, it's _much_ harder to do that.

True.

So question then becomes multifold:

1) Is caching of Vary responses worth the effort to get it working
proper?

2) If caching of Vary is desireable, what component of the network
should have to deal with the complexity involved?

  * The servers, where more complex servers will have a harder job
determining a proper Vary header.

  * Every cache implementation needing to have support for the most
complex cases.

What we see today is that neither component really cares. Most servers
forgets to send Vary headers when they should, instead using no-cache to
solve the problem. And most caches sees Vary too complex and reads it
the same as no-store, or in some user-agent cases reads Vary wrongly as
"no-cache" (need to revalidate on every request) and additionally
getting the validation completely wrong.

Regards
Henrik

---

From: Yves Lafon <[email-masked]>
Date: Mon, 23 Oct 2006 15:48:31 +0200 (MEST)

>   b. Vary: Accept-Encoding, User-Agent

Is it the same server with the same configuration on the same URI ? 
because if it's the case, then only b) is right. The server has the 
knowledge of the variability axis and should populate the Vary: header 
accordingly and not on a case-by-case basis.
My reading of the spec is that Vary should carry the list of all headers 
used to send compute the variant, so f(x,y), and not send f(5, y) = g(y).
Cheers,

-- 
Yves Lafon - W3C
"Baroula que barouleras, au tiéu toujou t'entourneras."

---

From: David Morris <[email-masked]>
Date: Mon, 23 Oct 2006 10:19:52 -0700 (PDT)

Encodings are supposed to be lossless ... based on my memory of the
discussion and not reading the spec recently, the spirit of ETag and
variants is that an encoding doesn't change the actual underlying entity.
A proxy does need to determine if a cached response will satisfy the
request but it may choose to perform a lossless transform to satisfy the
request or it may forward the request to the origin server. After the
requesting client unravels any encoding, the value will be identical.

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 23 Oct 2006 22:14:15 +0200

mån 2006-10-23 klockan 15:48 +0200 skrev Yves Lafon:

> because if it's the case, then only b) is right.

So I and many others say, but not all. And it isn't very clearly
specified either from what I can tell.

> The server has the 

That's just the thing. In some configurations the server does not have a
clear knowledge of all the axis the object MAY vary on. All it knows for
certain is the axis THIS variant depends on, which may differ from other
variants of the same URI.

Consider for example combinations of Accept-Language and Accept-Charset.
Many languages does not look for Accept-Charset as their content is only
available in a single charset, but some may have content available in
multiple charset.

My opinion looking at this from the cache perspective is that the server
should in such case respond with "Vary: Accept-Language,
Accept-Charset", but I do buy that server vendors may think differently
not want to examine all possible paths on all requests only to see what
the unified Vary header should look for at the URI, and instead only
wants to add the actual headers the server looked for while processing
the URI. That is Accept-Language, and then only for languages available
in multiple charsets also Accept-Charset.

I have not found anything in the RFC supporting or denying either view.
But I can tell that having caches support vary:ing Vary correctly is
quite complex if one at the same time have to care for performance.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 24 Oct 2006 00:15:35 +0200

> variants is that an encoding doesn't change the actual underlying entity.

Well, many aspects of the protocol in fact relies on ETag being unique
per response entity variant, not per original object version before
encoding.

 a) Caching of server driven negotiation logics (i.e. Vary), which uses
If-None-Match to deduce the logics applied by the server.

 b) Range merging and If-Range. Ranges is in the encoded data, not the
original.

Also the protocol as such does not guarantee that content-encoding is
lossless, no more than translations between languages is semantically
lossless. In both the the content is supposed to have the same semantic
meaning. The content-encoding is opaque to caches as they are not
required (or even supposed) to implement decoding or encoding on the
fly, only forwarding of the response as-is.

Also Content-Encoding is defined as an end-to-end entity header, which
the proxy is not supposed to muck around with. A non-transparent proxy
MAY but then has to take the full responsibilities, just as it may
perform lossy transformations on the data to reduce image resolutions
etc.

In the RFC Accept-Encoding is defined as one of many metrics for
server-driven content negotiation. It's governed by the same rules as
Accept, Accept-Language, Accept-Language, and any other headers used by
the server to decide what content to respond with. The end result of all
these decisions taken by the server is an response entity uniquely
identified by ETag, and to some extent Content-Location (the two has a
complex relation to each other, mainly related to cache invalidation on
updates).

This can be compared to the Transfer-Encoding schemes which is applied
hop-by-hop on the actual HTTP message transport with no semantic meaning
at all. Transfer-Encoding does not (MUST NOT) change ETag or any other
entity headers. Content-Encoding MUST change ETag.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 24 Oct 2006 10:12:40 +0200

Mark Nottingham schrieb:
>    * Yves Lafon and Julian Reschke have published an I-D that re-states 
> RFC2616 using xml2rfc, so that people can verify it's a faithful 
> transcription. Soon, they'll publish an -01 that incorporates the errata 
> that Scott has captured in <http://purl.org/NET/http-errata> (which 
> didn't require additional discussion).
> ...

In the meantime, the aforementioned draft 01 has been published (see 
announcement below).

For the next draft, we plan to work on re-organizing references into 
Normative/Informative (probably following Jim's work in 
<http://tools.ietf.org/html/draft-gettys-http-v11-spec-rev-00>), 
updating some of the references (leaving out the harder ones for now), 
and potentially do initial work on preparing the RFC822-to-RFC4234 
(ABNF) transition.

Best regards, Julian

--- snip ---
A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title		: Hypertext Transfer Protocol -- HTTP/1.1
	Author(s)	: Y. LAFON, et al.
	Filename	: draft-lafon-rfc2616bis-01.txt
	Pages		: 214
	Date		: 2006-10-23
	
The Hypertext Transfer Protocol (HTTP) is an application-level
    protocol for distributed, collaborative, hypermedia information
    systems.  It is a generic, stateless, protocol which can be used for
    many tasks beyond its use for hypertext, such as name servers and
    distributed object management systems, through extension of its
    request methods, error codes and headers [47].  A feature of HTTP is
    the typing and negotiation of data representation, allowing systems
    to be built independently of the data being transferred.

    HTTP has been in use by the World-Wide Web global information
    initiative since 1990.  This specification defines the protocol
    referred to as "HTTP/1.1", and is an update to RFC2616.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-lafon-rfc2616bis-01.txt

To remove yourself from the I-D Announcement list, send a message to
[email-masked] with the word unsubscribe in the body of
the message.
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the
username "anonymous" and a password of your e-mail address. After
logging in, type "cd internet-drafts" and then
"get draft-lafon-rfc2616bis-01.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-lafon-rfc2616bis-01.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/i-d-announce