Subject: [RFC] Optional header negotitation

From: Stefanos Harhalakis <[email-masked]>
Date: Fri, 17 Aug 2007 15:28:24 +0300

Hi there,

  After considering a lot your comments and other feedback I got regarding 
the "HTTP Information Request" proposal [1], I still believe that header 
negotiation is a good thing (tm) to have, so I'm resubmitting a request for 
comments to this list regarding this issue. 

  Let me explain the idea: It seems to me that web based applications can 
really use additional HTTP headers but we cannot propose them because of the 
extra overhead they will introduce (like the HTTP Timezone Header). So, one 
solution will be to find a way for supporting optional headers that will be 
send on demand. This proposal does exactly this.

  Example: A client contacts a web server and sends a GET request. Server side 
script (SC) is able to take advantage of header (H) (example: Timezone 
information) to adjust its content. SC examines the headers and finds out 
that the client did not send that header. Server side script sends an extra 
header like:

Header-Request: Timezone

and goes on sending the content. Client side browser:
a) Does not support "Header-Request" and ignores it, rendering the page
b) Supports Header-Request and the 'Timezone' optional header and resubmits a 
GET request with the optional header included (It would act as a refresh).

  I believe that this has great potentials because it can be used as the basis 
for easier future HTTP expansion, without wasting bandwidth.

  I'd appreciate any comments you may have, especially any objections.

Thank you in advance,
Harhalakis Stefanos

[1] http://www.nabble.com/-RFC--HTTP-Information-Request-tf3932478.html

---

From: "Robert Brewer" <[email-masked]>
Date: Fri, 17 Aug 2007 10:42:47 -0700

Stefanos Harhalakis wrote:
> Example: A client contacts a web server and sends a GET
> request. Server side script (SC) is able to take advantage
> of header (H) (example: Timezone information) to adjust
> its content. SC examines the headers and finds out that
> the client did not send that header. Server side script
> sends an extra header like:
> 
> Header-Request: Timezone
> 
> b) Supports Header-Request and the 'Timezone' optional
> header and resubmits a GET request with the optional
> header included (It would act as a refresh).

How is that sequence of events any better than:

 * User-agent always sends the extension-header "Timezone"
 * Server takes advantage of the "Timezone" header
   whenever it's present

? HTTP already has an "extension-header" token in the grammar,
for pete's sake. Since both parties need to know about any
extension header to take advantage of it, where's the benefit
of explicitly asking for it? Just send it every time.

> It seems to me that web based
> applications can really use additional HTTP headers but
> we cannot propose them because of the extra overhead
> they will introduce (like the HTTP Timezone Header).

Propose them all you want. If they're universally useful,
then all servers and user-agents will adopt them and accept
any overhead. If they're only useful to some servers and
user-agents, then only those parties should pay the price
of extension. The overhead of an extra request header field
for a small set of cooperating parties is far smaller than
the overhead of two extra messages for all parties whether
they use optional features or not.

Robert Brewer
[email-masked]

---

From: Stefanos Harhalakis <[email-masked]>
Date: Sat, 18 Aug 2007 14:12:58 +0300

> of explicitly asking for it? Just send it every time.

'extension-header' was a poorly chosen phrase. I didn't new it existed in 
RFC2626, but it seems that is matches :-)

Anyway, just ignore the 'extension-header' in that phrase, or remove the dash.

> of extension. The overhead of an extra request header field

The facts regarding this are:

a) I already proposed that and the feedback I got indicated that there is 
concern about the additional overhead it will introduce.

b) The 'universally useful' does not seem to apply to 'Timezone' (even though 
I strongly believe that it is) and I'm sure that other headers won't be. 
That's why I'm proposing an on-demand approach, to eliminate the 'universally 
useful' requirement for additional HTTP headers. After all it is not wise to 
say: "It seems that a part of web scripts could benefit from the 'Timezone' 
header. Lets send it to everyone, just to be sure."

  I'm also considering the possibility that this method will be used by *both* 
ends to request additional headers. For example, if we could travel back in 
time and perform some changes, clients could be able to request the 'Server' 
and 'Date' headers on demand.

> they use optional features or not.

  The way I'm thinking of (and proposing) it will result in no overhead for 
parties that don't support it. Server side support will be implemented by 
scripts and not by servers and thus those scripts will actually need this. 
Client side browsers that don't support this will not need to change anything 
and there will be no overhead. Browsers that support it will have the option 
to perform an additional request with the extra header included. Then they 
will need to cache this information for at least the current session and keep 
sending this header to the server without additional overhead. For an 
additional request to happen, both client and server sides need to support it 
and the server side needs to ask for an additional header. I find 
this 'overhead' extremely small compared to current 'Location:' header usage 
and I believe that it is the smallest possible.

  Finally, I don't understand what you mean when saying: 'two extra messages 
for all parties'.

---

From: "Robert Brewer" <[email-masked]>
Date: Sat, 18 Aug 2007 11:13:34 -0700

Stefanos Harhalakis wrote:
> 'extension-header' was a poorly chosen phrase. I didn't
> new it existed in RFC2626, but it seems that is matches :-)

Well, now you know it's there. So use it. ;)

> After all it is not wise to say:
> "It seems that a part of web scripts could benefit from the
> 'Timezone' header. Lets send it to everyone, just to be sure."

No problem. As the designer of application-specific extension
headers, that decision is up to you.

> I'm also considering the possibility that this method will
> be used by *both* ends to request additional headers.
> For example, if we could travel back in time and perform
> some changes, clients could be able to request the 'Server'

Again, no problem. My point is that that can all be handled
already by any cooperating servers and clients, and the
existing 'extension-header' is the way to do that:

   "The extension-header mechanism allows additional entity-header
    fields to be defined *without changing the protocol* ..."

Without changing the protocol:

 * a server that wants a Timezone header can, today, send an
   entity header in the response like "Header-Request: Timezone",
   and clients that understand "Header-Request: Timezone"
   can respond, or...
 * servers can define their own "Timezone-Request" header
   to do the same thing, or...
 * Timezone-aware user-agents can send the "Timezone" header
   everywhere, or...
 * only to a preselected set of servers, or...
 * response bodies could contain code (e.g. javascript) to add
   a "Timezone" header to requests for specific URLs, or ...
 * the user-agent could provide a means for the user to decide
   whether to send the "Timezone" header, or...
 * clients can send a "Header-Request: Date" entity header
   in the request, or...
 * a million other ways of organizing the flow.

But that can all be done without formalizing a new "Header-Request"
into the spec, because the spec already allows for header
extension in a generic fashion.

> Finally, I don't understand what you mean when saying:
> 'two extra messages for all parties'.

It's not critical to the point above. I was just pointing out
that your example required four messages when both parties
are Timezone-aware. There are other ways to structure the
flow so that no additional messages are needed. The beautiful
thing about the existing, generic extension-header mechanism
is that my application is free to be designed that way,
and yours is free to use more messages if you want them.
But those are workflow-level concerns (and solutions),
not protocol-level. So no change to the spec needed, and
indeed, changing the spec to add a new "Header-Request"
header would limit some of that freedom by canonicalizing
a single optional-header workflow (and a poor one,
but that can be argued elsewhere) when there's a benefit
to allowing many.

Robert Brewer
[email-masked]

---

From: Stefanos Harhalakis <[email-masked]>
Date: Sat, 18 Aug 2007 21:44:35 +0300

> Without changing the protocol:
[...]
>  * Timezone-aware user-agents can send the "Timezone" header

  That's the main reason I got negative feedback on the original proposal.  
You suggest that I should consider this as a non-issue?

>  * only to a preselected set of servers, or...

  I don't consider this an option since I believe that will not be implemented 
at all.

>    a "Timezone" header to requests for specific URLs, or ...

  There are cases where no javascript or any other *script languages are 
available.

> and yours is free to use more messages if you want them.

  Yes, but the clients need to send the extra header everywhere. This includes 
cases where this is not usefull (static content, most images, etc...). And 
the idea is that this proposal will not be specific to Timezone. Perhaps one 
day we would like to send our location/country for customized content, or... 
who knows...

  The 4 messages you're saying are in fact one extra request (2 messages) that 
will be performed only once at each session and only when both server side 
and client support and want to exchange that header. Apart from that, if one 
day the XXX (for example, Timezone) header becomes a commonplace, clients 
will be free to send it always (without waiting for a Header-Request). I'm 
not willing to suggest a restriction on that.

> but that can be argued elsewhere) when there's a benefit

  I see now that there are two meanings of the word 'optional':
* One that means that is optional for the protocol
* One that means that is optional for the content (and of course the protocol 
too)

I'm talking about the second case while you're refering to the first one. 
There is no indention at all to restrict HTTP! Just to provide a way for 
extending it with headers that aren't required everywhere. Of course other 
optional headers (as in the 1st case) will be able to be added.

p.s. Do you find this reply meaningfull or there is something you said that I 
didn't understand?

---

From: "Robert Brewer" <[email-masked]>
Date: Sat, 18 Aug 2007 16:52:50 -0700

Stefanos Harhalakis wrote:
> There is no indention at all to restrict HTTP! Just
> to provide a way for extending it with headers that
> aren't required everywhere.

It already exists: SEND THE HEADERS.

Right now, go write your script to send "Header-Request:
Timezone". I'll even write one for you in CherryPy 3:

    import cherrypy

    class App:
        
        def index(self):
            tz = cherrypy.request.headers.get("Timezone")
            if tz:
                return "Supplied timezone was: %r" % tz
            else:
                cherrypy.response.headers["Header-Request"] = "Timezone"
                return "No timezone supplied."
        index.exposed = True

    cherrypy.quickstart(App())

Run that and inspect the traffic with a tool like TamperData
for Firefox. Did the "Header-Request" header make it to the
client? (Yes.) Pretend you're a client that understands the
received "Header-Request", and send a "Timezone" header in
your next request. Did it make it to the server? (Yes.) Did
the server understand it? (Yes.) Are we violating the HTTP
spec? (No.) Did we make any changes to the HTTP spec? (No.)
Would there be any difference in the messages if we added
"Header-Request" to the spec? (No.) So why do it? Masochism?

Robert Brewer
[email-masked]

---

From: Stefanos Harhalakis <[email-masked]>
Date: Sun, 19 Aug 2007 12:05:17 +0300

[...]
> "Header-Request" to the spec? (No.) So why do it? Masochism?

  What you wrote is what I proposed and I never said that the HTTP spec need 
to be changed. I wrote to this list because Julian suggested it as a place 
for conversasion to take place for the original "Timezone" draft which I 
submitted some months ago. The 'header-request' is intended to be a separate 
RFC from 2616.

  What you described is exactly what I'm proposing... and if I get any 
comments about it, I'll summarize them and submit a draft. I'm sorry if you 
thought that I'm proposing this as something that should be included in RFC 
2616.

---

From: "Robert Brewer" <[email-masked]>
Date: Sun, 19 Aug 2007 21:21:07 -0700

Stefanos Harhalakis wrote:
> What you described is exactly what I'm proposing...
> and if I get any comments about it, I'll summarize
> them and submit a draft. I'm sorry if you thought
> that I'm proposing this as something that should
> be included in RFC 2616.

I am sorry I misinterpreted that bit. However, I don't
think that changes my central reservation: that HTTP
already allows enough ways to ask for extended headers
that there's no need to canonicalize one of those ways.
Every previous HTTP app which has needed additional
headers seems to have done just fine without a formal
"Header-Request" response header. So I'd find an extra
RFC on the matter just as spurious, serving only to
complicate web app framework and user agent code
unnecessarily.

Robert Brewer
[email-masked]

---

From: Stefanos Harhalakis <[email-masked]>
Date: Mon, 20 Aug 2007 16:58:40 +0300

> complicate web app framework and user agent code

  Can you please provide me with additional information about how apps are 
doing this? As far as I see it, javascript is not an option and I'm not aware 
of another method...

---

From: "Robert Brewer" <[email-masked]>
Date: Mon, 20 Aug 2007 22:16:32 -0700

Stefanos Harhalakis wrote:
On Monday 20 August 2007, Robert Brewer wrote:
> Can you please provide me with additional information
> about how apps are doing this? As far as I see it,
> javascript is not an option and I'm not aware of
> another method...

Different clients use different means to determine when to
send 'additional' headers:

 * wget provides the user a '--header' command-line argument.
   This puts the burden on the interactive user to decide
   which headers to send (in addition to those required by
   the HTTP spec).
 * Atom Publishing Protocol clients may send a "Name" request
   header in a POST to suggest a name for the POSTed resource.
   Similar mechanisms are used by other protocols such as WebDAV.
   This puts the burden on the authors and implementors of those
   specifications.
 * Most modern browsers allow servers to send them javascript.
   This puts the burden on the server, in the document workflow.
 * Some applications allow users to fully script the HTTP
   conversation. For example, Firefox's nsIhttpChannel, or
   Greasemonkey. This puts the burden on (the scripts of)
   the user.

Not that any of that would change if a "Header-Request" header
started to be widely used. What *would* happen is that all of
the maintainers of the above clients would come under pressure
from naive users to add more code for special-casing the receipt
of a "Header-Request" header. Server authors/maintainers, too.
I can just see Request objects the world over growing a new
"req.requested_headers" attribute so that they can...populate
those headers with the same values they were writing before,
using application-specific conventions and mechanisms. I just
don't see the benefit to generalizing the declaration, since
the above applications seem to be doing just fine without it.

Robert Brewer
[email-masked]

---

From: Stefanos Harhalakis <[email-masked]>
Date: Tue, 21 Aug 2007 13:01:19 +0300

[...]
[...]
[...]
[...]

> the above applications seem to be doing just fine without it.

  First of all, thanks for those clarifications...

  I believe that what you're saying act as a plus to the 'Header-Request' 
proposal. What you're showing here is in fact a (another ?) clue that 
something like this is actually needed but doesn't exist (yet ?), which leads 
to proprietary, non-standard "solutions". Do you actually believe that having 
a standard way to do these things will not be a benefit? Why not propose a 
standard to fit those needs?

  As for the req.requested_headers (I suppose you're referring to mod_python), 
I find it usefull... :-) (speaking for myself) 

  Finally, you should consider that newer "clients" will not need to introduce 
their own proprietary methods (like the javascript and plugin solutions you 
proposed), which of course saves us from interoperability and compatibility 
issues.

---

From: "Robert Brewer" <[email-masked]>
Date: Tue, 21 Aug 2007 08:43:00 -0700

Stefanos Harhalakis wrote:
On Tuesday 21 August 2007, Robert Brewer wrote:
> Do you actually believe that having a standard way to do
> these things will not be a benefit? Why not propose a

Yes, I actually believe that. You can't codify/systematize the
whole world, and in fact attempting to do so leads to fragile,
stale societies. Let the applications which use HTTP decide
how best to coordinate additional headers in a way that is
best suited to each one of them.

Or, you could ask yourself why none of the above applications
(and in fact, I know of no applications which) send a second
message asking for more headers, when they could easily do so
today without an additional standard.

Robert Brewer
[email-masked]

---

From: Stefanos Harhalakis <[email-masked]>
Date: Tue, 21 Aug 2007 19:43:17 +0300

> best suited to each one of them.

1) What aspect of the proposal would lead to something being 'fragile' or 
becoming 'stale'?
2) As I said previously, this will be just a method. If it is usefull then ok, 
if it is not then no-problem.
3) Can you be more specific? What you mean by saying 'coordinate'? Are you 
aware of something like this?

> today without an additional standard.

  I find this sentence non-consistent with the rest of our talking. Let me 
summarize:

Once you said: 
> I don't

(You said 'ASK')

you also said:
> "Header-Request" response header.

I asked:
> about how apps are doing this?

>  * Some applications allow users to fully script the HTTP

(where: 'wget' does not 'ask for extended headers', I already said that I 
don't see javascript as a solution and I'm not familiar with how Atom 
works...)

And now you're saying that those are *NOT* asking for 'extended headers'? That 
takes us back 3 messages, where I asked:

> about how apps are doing this?

.......

  I believe there is a missunderstanding somewhere. I'm not against 
applications using their own custom headers. But web browsers are not just 
clients for a specific application, and thus they will almost never 
send 'custom' headers, unless something like a plugin is loaded and it sends 
its own headers (everywhere?), so the only method to provide additional 
headers is by sending them all of the time (no?)

  Let me describe the proposal once more, being less abstract: Provide a way 
for server side scripts to optionally request one or more optional additional 
headers from browsers. If browsers support those headers then they may 
optionally include them in their future HTTP requests.

  I still don't understand why you're claiming that this is bad. Your main 
reasoning is that server side scripts MUST NOT have a standard that they will 
be able to optionally follow. If I understand your thoughts correctly, then 
why don't you object for HTTP Authentication which is similar? What about 
Accept-Language? Both of them could be handled without a standard by 
applications. What about 'Location'? 

  Finally I'd really like to listen to at least another one's opinion about 
this, just to find out whether I am (or not) the one that missunderstands 
something...

p.s. I'm not familiar with the rules of the list. Should we continue this 
conversation in private and post a summary of it, or we should keep posting 
at the list? (I'm in favour of posting to the list since it will allow for 
others to observe and will be available for future reference)

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 23 Aug 2007 00:57:08 +0200

> concern about the additional overhead it will introduce.

The overhead is only a minor concern. It's not a big deal. For things
like timezone the overhead of always sending this is fairly small.

The main concern is privacy and extending the protocol with mechanisms
designed to give out more information about the user without the user
knowing about it.

This said I don't see much wrong with the proposal. In fact I like if
for the context it's intended to be used as it allows for the user-agent
to implement a simple workflow to build a profile of the users security
preferences similar to what is already implemented in most useragents
for cookies. Just make the refresh optional, only needed if the response
Vary on the requested header.

> and 'Date' headers on demand.

Lets leave this for the context of user related request headers. Yes, it
may be used for other purposes as well, but can't think of many such use
cases where it do make good sense outside user related information and
privacy protection.

Most other headers is better to always send when available.. as is done
today.

Regards
Henrik

---

From: Stefanos Harhalakis <[email-masked]>
Date: Sun, 26 Aug 2007 15:36:40 +0300

> Vary on the requested header.

I've submitted this as an I-D. You can find it at:

http://www.ietf.org/internet-drafts/draft-sharhalakis-httphreq-00.txt

Thank you and Robert for your comments. I'll be more than glad to listen to 
more of them.