Subject: Deploying new expectation-extensions

From: Julian Reschke <[email-masked]>
Date: Thu, 03 Apr 2008 21:01:18 +0200

Hi,

interesting: 
<http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal>.

In particular:

"Note that section 14.20 of HTTP/1.1 indicates that "an HTTP/1.1 proxy 
MUST return a 417 (Expectation Failed) status if it receives a request 
with an expectation that it cannot meet". We expect that fully compliant 
proxies ignore Expect pragmas which they don't understand (as opposed to 
understand but cannot meet), but this remains to be verified in the wild."

So does anybody know that proxies do here?

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 10:06:54 +1100

I've tested a fairly wide variety of proxies with co-advisor; the only  
one that passed the related set of tests was very recent builds of  
Squid (2.7DEVEL0). Everything else -- including Squid 2.6STABLE4 --  
failed (it would take some digging to figure out exactly where this  
happened, unless Henrik knows; regardless, I think it's safe to say  
that a very large proportion of Squid's installed base fails as well).

Cheers,

On 04/04/2008, at 6:01 AM, Julian Reschke wrote:
>
> Hi,
>
> interesting: <http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal 
> >.
>
> In particular:
>
> "Note that section 14.20 of HTTP/1.1 indicates that "an HTTP/1.1  
> proxy MUST return a 417 (Expectation Failed) status if it receives a  
> request with an expectation that it cannot meet". We expect that  
> fully compliant proxies ignore Expect pragmas which they don't  
> understand (as opposed to understand but cannot meet), but this  
> remains to be verified in the wild."
>

--
Mark Nottingham       [email-masked]

---

From: "Charles Fry" <[email-masked]>
Date: Thu, 3 Apr 2008 19:47:48 -0400

Would you mind pointing us to the "related set of tests" which you refer to?

Also, could you specify just what you imply by passing and failing
these tests? Specifically, how is correct proxy behavior defined for
unknown Expect requests (I could see arguments either way based on my
reading of the HTTP protocol spec)?

thanks,
Charles

On Thu, Apr 3, 2008 at 7:06 PM, Mark Nottingham <[email-masked]> wrote:
>
>  I've tested a fairly wide variety of proxies with co-advisor; the only one
> that passed the related set of tests was very recent builds of Squid
> (2.7DEVEL0). Everything else -- including Squid 2.6STABLE4 -- failed (it
> would take some digging to figure out exactly where this happened, unless
> Henrik knows; regardless, I think it's safe to say that a very large
> proportion of Squid's installed base fails as well).
>
>  Cheers,
>
>
> MUST return a 417 (Expectation Failed) status if it receives a request with
> an expectation that it cannot meet". We expect that fully compliant proxies
> ignore Expect pragmas which they don't understand (as opposed to understand
> but cannot meet), but this remains to be verified in the wild."
> >

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 11:12:10 +1100

See:
   http://coad.measurement-factory.com/

A representative test is sending a request with
   Expect: 100-continueing

I don't see how you can read it both ways; e.g.,

>  This header field is defined with extensible syntax to allow for
>    future extensions. If a server receives a request containing an
>    Expect field that includes an expectation-extension that it does  
> not
>    support, it MUST respond with a 417 (Expectation Failed) status.

[...]

>    The Expect mechanism is hop-by-hop: that is, an HTTP/1.1 proxy MUST
>    return a 417 (Expectation Failed) status if it receives a request
>    with an expectation that it cannot meet. However, the Expect
>    request-header itself is end-to-end; it MUST be forwarded if the
>    request is forwarded.

Cheers,

On 04/04/2008, at 10:47 AM, Charles Fry wrote:
>> <http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal 
>> >.
>>>
>>> In particular:
>>>
>>> proxy

---

From: "Charles Fry" <[email-masked]>
Date: Thu, 3 Apr 2008 20:34:39 -0400

>  A representative test is sending a request with

And you define correct proxy behavior as sending a 417 when such a
header is encountered? In other words your previous response suggests
that most proxies _do not_ send a 417, but simply pass the header
through, except for very recent Squid builds which send a 417 on an
unknown expectation?

> >   Expect field that includes an expectation-extension that it does not

I read this as applying to a non-proxy server. In other words, the
final server for a given request. Obviously this server knows which
expectation-extensions it supports, and clearly it must respond with a
417 when it receives an unsupported expectation.

> >   request-header itself is end-to-end; it MUST be forwarded if the

This is ambiguous in that it does not specify how a proxy determines
that it cannot meet an unknown expectation. As far as I can tell, you
and Julian are reading this to suggest that when a proxy sees an
unknown expectation-extension it must return a 417. The way I read it,
if a proxy receives an unknown expectation-extension then it must go
about determining whether or not the expectation can be met by the
next-hop server. If it knows the next-hop server to be HTTP/1.0 then
it can clearly bail out immediately with a 417 (as specifically
outlined in 8.2.3 for the 100-continue case). Otherwise the only way
it can determine whether or not the expectation can be met is by
forwarding it to the next-hop server.

My reading of the utility of the hop-by-hop nature of the Expect
mechanism (which corresponds very cleanly with our desired use case)
is that Expect headers can be inserted by proxies (as specified in
10.1), allowing the proxy which inserted the header to eat any
corresponding 1xx informational response that is returned.

Now whether or not proxies in the wild behave this way or not is the
other half of the question which I am trying to determine. :-)

Charles

> one
> with

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 11:39:24 +1100

RFC 2616, section 1.3:

> server
> An application program that accepts connections in order to service  
> requests by sending back responses. Any given program may be capable  
> of being both a client and a server; our use of these terms refers  
> only to the role being performed by the program for a particular  
> connection, rather than to the program's capabilities in general.  
> Likewise, any server may act as an origin server, proxy, gateway, or  
> tunnel, switching behavior based on the nature of each request.

> origin server
> The server on which a given resource resides or is to be created.

> proxy
> An intermediary program which acts as both a server and a client for  
> the purpose of making requests on behalf of other clients. Requests  
> are serviced internally or by passing them on, with possible  
> translation, to other servers. A proxy MUST implement both the  
> client and server requirements of this specification. A "transparent  
> proxy" is a proxy that does not modify the request or response  
> beyond what is required for proxy authentication and identification.  
> A "non-transparent proxy" is a proxy that modifies the request or  
> response in order to provide some added service to the user agent,  
> such as group annotation services, media type transformation,  
> protocol reduction, or anonymity filtering. Except where either  
> transparent or non-transparent behavior is explicitly stated, the  
> HTTP proxy requirements apply to both types of proxies.

Cheers,

On 04/04/2008, at 11:34 AM, Charles Fry wrote:

>>> Would you mind pointing us to the "related set of tests" which you  
>>> refer
>> to?
>>>

--
Mark Nottingham     http://www.mnot.net/

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 4 Apr 2008 01:54:17 +0100

> corresponding 1xx informational response that is returned.

Wow, I'm amazed that anyone had that interpretation.

My reading of expectation-extensions is that it's absolutely intended
for a proxy to return 417 if it doesn't recognise the expectation
_itself_, to allow HTTP to be extended with new features in a way
which doesn't break over existing servers, including proxies.

E.g. if there was some new message boundary encoding used in a
request, you'd accompany it with an expectation so that a proxy which
doesn't know that encoding would return 417 rather than
misinterpreting the message.  The client would then retry with a more
backward compatible encoding.  Things like multiplexing extensions and
pipelining improvements would use this.  (But not if proxies aren't
returning 417s).

In general with future/backward-compatibility of protocols, file
formats etc., it's useful to have an extensible way to say "this
message has features which _may_ be supported", and "this message has
features which _must_ be supported".  For HTTP, I understood Expect is
the "must" part of that.  But not if proxies ignore it.

My reading of "Expect: 100-continue" is that the proxies _themselves_
can return "100 Continue" responses (as well as eating the one
produced by the next hop, if any.

-- Jamie

---

From: "Charles Fry" <[email-masked]>
Date: Thu, 3 Apr 2008 22:30:42 -0400

Right. I'd missed that.

So in summary, new expectations are guaranteed to break on compliant
proxies? Sad.

Charles

On Thu, Apr 3, 2008 at 8:39 PM, Mark Nottingham <[email-masked]> wrote:
> requests by sending back responses. Any given program may be capable of
> being both a client and a server; our use of these terms refers only to the
> role being performed by the program for a particular connection, rather than
> to the program's capabilities in general. Likewise, any server may act as an
> origin server, proxy, gateway, or tunnel, switching behavior based on the
> nature of each request.
> >
>
>
> > origin server
> > An intermediary program which acts as both a server and a client for the
> purpose of making requests on behalf of other clients. Requests are serviced
> internally or by passing them on, with possible translation, to other
> servers. A proxy MUST implement both the client and server requirements of
> this specification. A "transparent proxy" is a proxy that does not modify
> the request or response beyond what is required for proxy authentication and
> identification. A "non-transparent proxy" is a proxy that modifies the
> request or response in order to provide some added service to the user
> agent, such as group annotation services, media type transformation,
> protocol reduction, or anonymity filtering. Except where either transparent
> or non-transparent behavior is explicitly stated, the HTTP proxy
> requirements apply to both types of proxies.
> >
>
>
>  Cheers,
>
>
>
>
> > > > On Thu, Apr 3, 2008 at 7:06 PM, Mark Nottingham <[email-masked]>
> > > >
> > > wrote:
> > >
> > > >
> > > >
> > > > >
> > > > > would take some digging to figure out exactly where this happened,
> > > > >
> > > >
> > > unless
> > >
> > > >
> > > > > an expectation that it cannot meet". We expect that fully compliant
> > > > >
> > > >
> > > proxies
> > >
> > > >
> > > > > ignore Expect pragmas which they don't understand (as opposed to
> > > > >
> > > >
> > > understand
> > >
> > > >

---

From: "Charles Fry" <[email-masked]>
Date: Thu, 3 Apr 2008 22:35:14 -0400

Jamie, thanks for the detailed explanation. We were approaching the
spec from our current requirements without the added insight of
additional use cases. Your description went a long way in clarifying
the possible intent of Expect, and in making it seem a bit less
broken.

cheers,
Charles

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 04 Apr 2008 15:56:23 +1300

My reading of the spec was always that any server or proxy receiving an 
Expects header it didn't understand must bounce the request.  If a proxy 
relayed to the next hop there would be several problems:

a) it would no longer be a hop-by-hop header, but an end-to-end header 
(if all proxies in a chain did this, the request would make it all the 
way to the server)
b) The proxy still cannot perform any required processing because it 
doesn't know how to.  therefore such a mechanism couldn't be used to 
extend HTTP outside the bounds of the realm in which it already operates.

IMO I think this makes the Expects header pretty well useless, which is 
why I believe no-one has really bothered to implement it.  It merely 
invites all compliant agents to bounce your request, and therefore in 
the spirit of reducing round-trips to servers, and since there's 
generally another way around any problem,  people choose instead to 
avoid Expects (more reliable, fewer support calls etc).

However relying on non-compliance of proxies in this case would be 
foolhardy.  Changing the semantics of Expects I don't think is that 
great an option either (actually I'd vote to deprecate it along with 305 
Use Proxy)

However, I don't believe this use-case requires use of the Expects 
header.  What stops a server simply sending multiple 103 interim 
responses regardless of what the client sent?  Or a client could 
advertise the desire for such functionality with another header other 
than Expects. That's already explicitly allowed for in the spec... 
sending an Expects header in the request is simply asking anything along 
the chain to bounce the request if they are compliant and don't 
understand it. 

Adrien

Charles Fry wrote:
>>> Would you mind pointing us to the "related set of tests" which you refer
>>>       
>> to?
>>     

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: "Mark Baker" <[email-masked]>
Date: Thu, 3 Apr 2008 23:08:43 -0400

On Thu, Apr 3, 2008 at 10:56 PM, Adrien de Croy <[email-masked]> wrote:
> foolhardy.  Changing the semantics of Expects I don't think is that great an
> option either (actually I'd vote to deprecate it along with 305 Use Proxy)

+1

Mark.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 3 Apr 2008 20:36:27 -0700

>> option either (actually I'd vote to deprecate it along with 305  

That train has long since left the station.

  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/ 
0165.html
  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/ 
0192.html

Version numbers in protocols are a good thing.  Stupid IETF politics
and FUD regarding the so called "risk" of changing the version number
(when an incompatible change is made to the protocol) are the only
reasons they don't work, and every time they end up biting us.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 04 Apr 2008 18:54:19 +1300

Actually the concept goes back to Thomas Moore :)

The Expects header breaks the universal law that silence does not mean 
assent.  Instead it requires recipients to know that they need to 
actively deny a capability (else the sender will assume the capability 
is there).  Any recipient unaware of the Expects header requirements 
will ignore it, which is a failure condition.  So this creates all sorts 
of issues around sending Expects headers to HTTP/1.0 servers or 
intermediaries, or non-compliant (in terms of requirements to bounce) 
servers/intermediaries. 

The goal of the Expects header I believe has been captured in the spec, 
whether that goal was worthy or not is another matter.  I think mostly 
confusion arises because it's not clear in the spec that the Expects 
header is intended to be used where a required extension capability 
would NOT normally work (or degrade gracefully) through any agent that 
was not explicitly aware of it. 

This covers 2 cases that I can see:

1: where the extension was semantically incompatible with the current 
version of HTTP. 
2: where the extension was optional in the current version but for some 
reason indispensable to the client.

AFAIK the only instance referred to in the spec (100-Continue) is an 
instance of 2, but not particularly useful since it's easier for an 
agent to cope with lack of 100-continue than it is to cope with being 
disconnected with a 417.  There aren't really that many optional bits of 
HTTP/1.1 that a client could reasonably decide were indispensible, 
especially considering the prospects of daring to use an Expects header.

Version changes I think is a better way to cope with the first class of 
use-cases.  There's no difference between a chain of 
agent->intermediary->server supporting a newer version number than 
supporting an extension.  It's just a matter of labelling.

Adrien

Roy T. Fielding wrote:
>>> great an
>  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/0165.html 
>
>  http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/0192.html 
>
>

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 4 Apr 2008 07:43:27 +0100

> reason indispensable to the client.
...
> use-cases.

That's assuming agents reject a message with a newer version number...

Since they don't reject expectations, I would be surprised if they
consistently reject newer version numbers.

> There's no difference between a chain of 

Well, in principle you can try extensions privately ("Expect:
x-my-experiment") and they can be orthogonal.  Version numbering
forces linearity: if you handle feature Z, you must handle feature X
and Y too.

Also, agents tend to report they are HTTP/1.1 even when they are
grossly non-compliant.  I suspect that's due to it being a version to
aim for, and they do support some features of 1.1, and reporting the
version is advantageous.  If expectations were able to be used, naming
individual features, I suspect agents wouldn't pretend to handle all
features.

In practice it seems there aren't that many "features" to be concerned
about.

Peculiar things like chunked requests have long been unusable.... I
guess they might have fitted into the scheme.

-- Jamie

---

From: "Brian McBarron" <[email-masked]>
Date: Fri, 4 Apr 2008 09:50:55 -0400

> (if all proxies in a chain did this, the request would make it all the

Not to beat a dead horse, but Expect _is_ end-to-end, according to the last

>    request-header itself is end-to-end; it MUST be forwarded if the

However, there seem to be plenty of other reasons why Expect is not useful,
so it's probably moot.

> the chain to bounce the request if they are compliant and don't

What we need to implement
http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal is
a mechanism which:
A) Is guaranteed to reach the origin-server (not discarded or refused by a
proxy)
B) The origin-server will refuse to process if it doesn't understand
These conditions are necessary so that the continuation of a request can't
result in unintended consequences if it ends up at the wrong origin-server.
Charles and I thought that the Expect header would accomplish these two
things.  Does anyone have a suggestion for a different mechanism that could
achieve the same effect?

Thanks,
Brian

> >> <
> http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal>.
> >>
> >>>>> In particular:
> >>>>>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Apr 2008 16:01:58 +0200

Brian McBarron wrote:
> http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal 
> is a mechanism which:
> A) Is guaranteed to reach the origin-server (not discarded or refused by 
> a proxy)
> These conditions are necessary so that the continuation of a request 
> can't result in unintended consequences if it ends up at the wrong 
> origin-server.  Charles and I thought that the Expect header would 
> accomplish these two things.  Does anyone have a suggestion for a 
> different mechanism that could achieve the same effect?
> ...

It sounds you need a Must-Understand end-to-end mechanism. I don't think 
that exists in HTTP/1.1. RFC 2774 may be what you want, but that doesn't 
help in practice.

That being said, I'm not sure why you need B).

Can't the origin server just send the 103s without being asked for it? 
That would allow the client to discover support for the feature.

BR, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 05 Apr 2008 12:56:41 +1300

just off the top of my head, if you

(a) can't rely on intermediaries to pass on headers they don't 
understand (e.g. which you could use to flag a requirement) - even 
though this is a requirement
(b) can't rely on Expects to be processed incorrectly by intermediaries
(c) can't rely on intermediaries to pass on methods they don't 
understand (even though the capability to be able to do this is required 
in the spec)

then your options are limited:

* another type of content encoding (e.g gzip -> packaged), with protocol 
overhead (restart position, markers  etc) multiplexed into the content.
* meta-protocol... something layered over normal HTTP GET and POST, 
where your negotiation is carried out over the message content rather 
than the headers.

personally though I'd be looking at relying on (a) above

Adrien

Brian McBarron wrote:
> On Thu, Apr 3, 2008 at 10:56 PM, Adrien de Croy <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Not to beat a dead horse, but Expect _is_ end-to-end, according to the 
> last sentence here:
>
> However, there seem to be plenty of other reasons why Expect is not 
> useful, so it's probably moot.
>
> A) Is guaranteed to reach the origin-server (not discarded or refused 
> by a proxy)
>     >>> On Thu, Apr 3, 2008 at 7:06 PM, Mark Nottingham
>     <[email-masked] <mailto:[email-masked]>>
>     >>>
>     >> wrote:
>     >>
>     <mailto:[email-masked]>
>     >>>>
>     >>>>
>     >>>>
>     >>>>
>     >>>>
>     >>>>
>     >>  --
>     >>
>     >>
>     <mailto:[email-masked]>
>     >>
>     >>
>     >>
>     >>
>     >
>     >
>
>     --

---

From: "Charles Fry" <[email-masked]>
Date: Fri, 4 Apr 2008 20:08:53 -0400

>  Can't the origin server just send the 103s without being asked for it? That
> would allow the client to discover support for the feature.

Hmm. Now this is starting to come full-circle. As I understand it the
whole reason that Expect: 100-continue is used in conjunction with 100
Continue responses is to ensure, as the request is finding its way to
the origin server, that the response will be able to find its way
back, being properly interpreted as an intermediate response. Without
this there is the risk that a non-100-continue-aware proxy would
interpret the 100 response as a final response.

Is this not a requirement of any client-elicited 1xx response? I.e.
can we really just send 103s when they aren't asked for, with full
confidence that they won't break anything as they travel back to the
client?

Charles

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 4 Apr 2008 18:10:52 -0700

On Apr 4, 2008, at 6:50 AM, Brian McBarron wrote:
> What we need to implement http://code.google.com/p/google-gears/ 
> wiki/ResumableHttpRequestsProposal is a mechanism which:

Don't reimplement TCP on top of HTTP.  The result will be not work
as well as TCP.  Among other reasons, the connection will be reset
at the TCP level and the RST will rollback the TCP windows to the
point where your little client will never see the ack from the server.

Re-think your solution instead.

Consider this.  Any server that is capable of supporting such a
mechanism will have to maintain temporary files that are uniquely
accessible to that client for the amount of time that the server
is willing to allow resumption.  So, let's give that resource a URI.

The first request is sent to the origin server without needing
to know if it (or anything along the path) understands the extension.

    POST /my/favorites HTTP/1.1
    Host: example.com
    ....

If the server receives that message with no Via field or with a
Via field that indicates no HTTP/1.0 intermediaries, then the server
can respond immediately with

    103 Resumable
    Location: http://example.com/my/favorites;r123

Note that the URI above is created by the server, so it could be
anything, though a sensible security policy would limit it to a
suffix of the requested POST URI.  You'll have to try it to see
if this works in practice with HTTP/1.1 user agents -- you may
need an additional request header to indicate it if not.

If the connection drops, the client can then do

    HEAD /my/favorites;r123 HTTP/1.1
    Host: example.com

to determine how much of the content was successfully transferred
and stored on the origin server, and append to that content with
a simple

    POST /my/favorites;r123 HTTP/1.1
    Host: example.com

and repeat as necessary.  Auth/access control can be added as normal.
I suggest, however, that anything as fragile as a resumed request
be accompanied with some sort of integrity check, like content-md5.

The extension is therefore achieved with the addition of one status
code and should work with any HTTP/1.1 compliant server chain.
If it doesn't, you are no worse off than before (and HTTP is not
complicated and constipated by unnecessary Expect fields).

Cheers,

Roy T. Fielding                            <http://roy.gbiv.com/>
Chief Scientist, Day Software              <http://www.day.com/>

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 05 Apr 2008 14:24:08 +1300

the spec requires that any intermediary pass on 1xx messages, and that 
any number of 1xx messages may be sent

So any intermediary thinking that a 1xx response is the final one, is 
non-compliant.

Having said that, there are some that behave like that (possibly 
HTTP/1.0 proxies?)

Adrien

---

From: Julian Reschke <[email-masked]>
Date: Sat, 05 Apr 2008 10:58:54 +0200

> interpret the 100 response as a final response.

I think that is incorrect.

Expect is needed in this case, because the client will not start sending 
the request body until the 100 Continue response has been received.

So in this case, it's essential for the protocol to work at all.

BR, Julian

---

From: Bill de hOra <[email-masked]>
Date: Sat, 05 Apr 2008 13:33:15 +0100

> complicated and constipated by unnecessary Expect fields).

So this is interesting. I'm working on exactly the same thing for 
uploads from mobile networks, where connections get dropped all the 
time. Typically these are photo/video and as the cameras get to 5mp, the 
chances of actually getting a failed upload improve.

I've split the problem into two parts:

  - upload resumption
  - upload packets

Upload resumption means continuing from a known offset; I'm not using 
new response codes or expectations for that.  Upload packaging means 
splitting the image in advance, and uploading each part separately. .

== Upload resumption

For upload resumption, it goes like this:

    ...binary...

    201 Created
    Location: http://example.com/my/favorites/image5
    ETag: "c180de84f991g8"

That sequence integrates with Atom Protocol, which is important  -  a 
new response code means upgrades for the successful AtomPub case.

If the client never gets the 201, the client can retry POST again (the 
server. will have what I used to call a 'phantom' in HTTPLR, but that 
can be cleaned up).

The Etag can help with requests arriving out of order (my understanding 
is that such heisenbugs are possible over mobile networks; but I'm happy 
to be corrected on that).

If the connection drops, the client might be confused, but it can check 
Content-Length:

    HEAD /my/favorites/image5 HTTP/1.1
    Host: example.com

    200 Ok
    Location: http://example.com/my/favorites/image5
    Content-Length: nnn
    ETag: "c180de84f991g8"

Then the client can send the rest of the data offset from 'nnn'. But to 
make the semantics clear I use PATCH (or overloaded POST with 
X-HTTP-Method-Override). Definitely not PUT.

    PATCH /my/favorites/image5 HTTP/1.1
    Host: example.com
    If-None-Match: "e180ee84f0671b1"

    ...nnn+...

== Upload packets

This is where the image is split in advance and they are uploaded to the 
server, perhaps in parallel.  It *is* different and is I think a 
reinvention of TCP a few layers up, largely because unlike the 
resumption case, the packets can generally arrive out of order 
(seriously, we're talking photomosiacs).  To deal with that means being 
able to say this packet is "m of n".

It gets complicated quite quickly and the fundamental tradeoff seems to 
be this - do you tunnel the protocol through http, or do you expose each 
part as a resource, a surrogate of some overall resource that denotes 
the uploaded content?

I have some ideas on how to do this but nothing concrete. To be honest, 
part of me isn't sure whether isn't a good idea to overload http and 
resource semantics in this way so we can play at being bittorent. It 
seems to be valuable (or at least wished for) in the mobile space; 
repeating failed uploads are a real problem there. I doubt it can be 
done without extensions, but would be delighted to be wrong about that.

Anyway, I think the resumption case be dealt with straightforwardly 
without any extensions to http.

Bill

---

From: "Charles Fry" <[email-masked]>
Date: Sat, 5 Apr 2008 10:34:50 -0400

>  Anyway, I think the resumption case be dealt with straightforwardly without
> any extensions to http.

We agree completely. Our proposal does not involve extensions to HTTP,
but rather formalizing and standardizing the HTTP semantics involved
in resumption.

Obviously our current proposal uses Expects where it shouldn't, but
exactly the same behavior can be obtained using other headers (off the
top of my head something like If-Match for the ETag, Content-Range for
the byte range, and Pragma: resmue to flag resume requests, though
we'll come up with a formal proposal next week).

Further, regarding your mobile case, we are actively developing Google
Gears modules to support two different kinds of upload resumption:

  - <http://code.google.com/p/google-gears/wiki/ContentRangePostProposal>
  - <http://code.google.com/p/google-gears/wiki/ResumableHttpRequestsProposal>
(previously referenced in this thread)

Both solutions, along with Google Gears itself, are being designed in
a cross-platform, cross-browser manner, explicitly targeting mobile
devices (<http://code.google.com/apis/gears/mobile.html>).

We feel strongly that there is an important gap to fill, both from a
practical and a standardization perspective, with respect to resumable
uploads, and we would be very excited about collaborating with other
interested parties as part of this process.

Charles

---

From: "Charles Fry" <[email-masked]>
Date: Sat, 5 Apr 2008 10:37:31 -0400

> That

If you are correct, then that implies (as I understand it) that Expect
is the wrong mechanism for 100 continues, as the only requirement is
an end-to-end request and interim response, with no hop-by-hop
requirements. I am just trying to fully understand why an Expect would
be necessary for some 1xx responses and not others.

Charles

---

From: "Charles Fry" <[email-masked]>
Date: Sat, 5 Apr 2008 10:41:37 -0400

As you call out, the real problem here is HTTP/1.0 and lower proxies.
For that matter, it seems that the primary focus of section 8.2.3 is
to deal with intermediate HTTP/1.0 and lower proxies (otherwise the
origin server could simply do all the work, and Expects wouldn't even
be needed).

So why would this be necessary for 100 Continues but not other 1xx responses?

Charles

On Fri, Apr 4, 2008 at 9:24 PM, Adrien de Croy <[email-masked]> wrote:
>
>  the spec requires that any intermediary pass on 1xx messages, and that any
> number of 1xx messages may be sent
>
>  Having said that, there are some that behave like that (possibly HTTP/1.0
> proxies?)
>
>  Adrien
>
>
>
>
>  Charles Fry wrote:
>
> >

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 00:46:23 +0200

> So does anybody know that proxies do here?

Well, as written I don't see how one can end up with the impression that
proxies should forward unknown expectations..

      * A proxy MUST implement both the client and server requirements
        of this specification.
      * If a server receives a request containing an Expect field that
        includes an expectation-extension that it does not support, it
        MUST respond with a 417 (Expectation Failed) status.
      * an HTTP/1.1 proxy MUST return a 417 (Expectation Failed) status
        if it receives a request with an expectation that it cannot
        meet.

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 00:56:04 +0200

> that a very large proportion of Squid's installed base fails as well).

Squid-2.6 is an HTTP/1.0 proxy. It does not have to or even need to look
into Expect. In HTTP/1.0 Expect is just an extension header like any
other extension header.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 01:06:01 +0200

> produced by the next hop, if any.

My reading is that proxies can not inject 100 Continue, only forward
them or eat them..

An 100 Continue is a guarantee that the client can continue sending the
request entity. 

See "Use of the 100 (Continue) Status" for a quite in-depth discussion
about the purpose and use of 100 Continue.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 01:16:52 +0200

> interpret the 100 response as a final response.

Not really. It's more about avoiding the heuristics on how long a client
should wait for 100 Continue. But unfortutately it failed due to
HTTP/1.0 and RFC2068 being part of the picture...

> confidence that they won't break anything as they travel back to the

Yes, if the request was received as an HTTP/1.1 request. At least from a
specifications point of view. (not entirely sure about actual
implementations..)

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 01:21:36 +0200

> be necessary for some 1xx responses and not others.

It's correct as there is situations where it's known 100 Continue won't
come (next-hop HTTP/1.0).

For what it's worth it could have been spelled "Expect: HTTP/1.1". with
the same effect.

Regards
Henrik

---

From: Nicolas Alvarez <[email-masked]>
Date: Sat, 05 Apr 2008 20:27:08 -0300

> individual features, I suspect agents wouldn't pretend to handle all

Agreed. I'm having that discussion about an app-specific protocol too, 
suggesting devs to use a "capability list" instead of checking for 
version numbers. (even worse: the current server is checking for the 
*client version*, the protocol itself isn't versioned at all, that will 
work so nicely (not!) with alternate implementations...)

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 01:33:42 +0200

> though this is a requirement

This is effectively saying "you can't trust intermediaries".

I would be very surprised if you find a proxy which aims for semantic
transparency and which do not forward unknown headers.

> (b) can't rely on Expects to be processed incorrectly by intermediaries

I would expect any HTTP/1.1 intermediary to process Expect.

> understand (even though the capability to be able to do this is required 

I am not aware of a such requirement in the specs, but common sense says
that a semantically transparent proxy should forward extension-methods.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 6 Apr 2008 10:06:32 +1000

/me shakes head at self... good point.

The problem that this raises is that many people configure their  
proxies to not send Via headers. This breaks the algorithm that Roy  
posts later on...

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 02:18:18 +0200

> proxies to not send Via headers. This breaks the algorithm that Roy  

Yes, and the specs says...

        The Via general-header field MUST be used by gateways and
        proxies to indicate the intermediate protocols and recipients
        between the user agent and the server on requests

The only way forward on that is educating implementers about the
importance of Via, and make them expose the following feature instead or
allowing Via to be removed:

        However, if the real host is considered to be sensitive
        information, it MAY be replaced by a pseudonym.

which is the correct resolution to the issue people try to solve by
removing the Via header..

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 6 Apr 2008 10:24:41 +1000

+1

It may be worth putting a few words around Via so that people  
understand its function. The folks on this list won't be around  
forever to educate them (and although the list archives will be there,  
most won't bother with them).

Cheers,

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 06 Apr 2008 13:00:23 +1200

there was a recent discussion about Via and whether it should become a 
SHOULD level requirement (I think related to an editorial change).

In any case, I believe there are still plenty of customers that don't 
want their ISP to know they are using a proxy server.  These people 
really don't want via headers leaking out.  Yep, there are some ISPs 
still (haven't gone out of business yet) that don't like their customers 
using a proxy to obtain more use out of what they pay for.  At least the 
last time I got a customer request about this wasn't that long ago.

* Intercepting transparent proxies - should they insert a Via?  then 
they aren't transparent any more.  Even changing the request HTTP 
version is dodgy enough in that situation (let alone that the whole 
concept of intercepting transparent proxies is dodgy).
* Reverse proxies?

In any case, IMO using Via for anything except tracing is fragile 
(relies on 3rd-hand information) although I guess pretty much everything 
about a forwarded request relies on 3rd hand information at the origin 
server.  Are there any known implementations of any servers that change 
behaviour based on the path a request takes to them?

I guess this is another instance of where customer wishes and protocol 
designs are at loggerheads.  Unfortunately implementors of protocols 
have to deal with customers and commercial/competitive issues.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 06 Apr 2008 13:02:52 +1200

> transparency and which do not forward unknown headers.
me too, which is why I suggested relying on this option.

> I would expect any HTTP/1.1 intermediary to process Expect.
sure, it's in the spec, but as per the original post on this topic, few 
seem to be compliant in this matter.

> that a semantically transparent proxy should forward extension-methods.

capability to be able to, rather than the policy to actually do it.

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 03:14:06 +0200

> concept of intercepting transparent proxies is dodgy).

If you don't insert Via then you better be really transparent and not
change the protocol version, and also deal with the complications
involved in that...

> * Reverse proxies?

Just a proxy, but specs give them more freedom for as long as the
solution as a whole is conformant.

But I would say it at least MUST add Via on requests..

> behaviour based on the path a request takes to them?

Not sure. Haven't seen any but haven't looked for them either.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 6 Apr 2008 15:22:50 +0100

> My reading is that proxies can not inject 100 Continue, only forward

If there's one thing becoming clear, it's that different people have
different ideas about the behaviour of Expect and about 100 Continue,
while reading the same spec.

> An 100 Continue is a guarantee that the client can continue sending the

The client can always send the request entity, because if there's a
HTTP/1.0 proxy in the chain, the client won't receive 100 Continue, so
HTTP/1.1 clients have an unspecified short timeout and send it anyway.

The 100 Continue is a hint to continue immediately, instead of after a
short timeout.  The idea is if the server sends an error response
(quickly), the client can abort the request and close the connection
without transmitting any part of the entity.

At least, that's how _I_ understand the spec!

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 6 Apr 2008 16:05:33 +0100

> proxies should forward unknown expectations..

I don't see it either, but at least one implementor did.  For those
who might want to rely on Expect functioning, we have to know what the
implementors of most of the proxies out there have ended up doing.
Even if it's wrong.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 18:21:38 +0200

> HTTP/1.1 clients have an unspecified short timeout and send it anyway.

Thats only if 100 Continue has never been seen from the requested
server... If the client has seen a 100 Continue ever from the requested
server it should wait indefinitely.

Hmm.. when reading 2616 one more on the use of 100 Continue it's pretty
clear 2616 should have been an 1.2 version bump from 2068 to clean up
this mess... some heavy heuristics involved there to tell 2068 1.1 apart
from 2616 1.1, even with a MAY requirement overriding a SHOULD NOT...

> without transmitting any part of the entity.

That's one aspect of it. But there is more aspects of 100 Continue, and
more conditions governing it's use.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 07 Apr 2008 00:06:40 +0200

sön 2008-04-06 klockan 16:05 +0100 skrev Jamie Lokier:

> I don't see it either, but at least one implementor did.

Which one?

Squid-2.6 was mentioned here, but it's not an HTTP/1.1 proxy..

Regards
Henrik

---

From: "Charles Fry" <[email-masked]>
Date: Sun, 6 Apr 2008 21:45:49 -0400

>  For what it's worth it could have been spelled "Expect: HTTP/1.1". with

See, what we really want is exactly "Expect: HTTP/1.1" so that we can
be certain that we can spend a 1xx class response, but as far as I can
tell there is no way to actually accomplish this, other than just
spewing out our 1xx response, with utter disregard for any
non-compliant proxies in the downstream path. But at this point I
don't see any other viable option.

Charles

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 07 Apr 2008 23:45:07 +1200

I think that's possibly your best option.

For what it's worth, RFC 1945 (HTTP 1.0) specified the existence of 1xx 
response codes, and defined them as provisional, however specified that 
there aren't any valid 1xx responses to any HTTP/1.0 request (seems to 
be predicting a later iteration of the protocol).

so anyway, the concept of a provisional (and therefore to be ignored) 
1xx class of responses was defined in 1.0, so if we're lucky then people 
who implemented 1.0 proxies read that and they should be able to cope 
with them.

Actually it's quite interesting looking through HTTP 1.0 again... it was 
so much simpler!  Only 3 methods for starters, which really betrays the 
way HTTP evolved, and why we have issues negotiating transfer of message 
bodies for uploads... because HTTP was fundamentally designed as a 
retrieval protocol, not an upload protocol.

I think until we adopt proper handling of uploads (i.e. pre-authorised / 
negotiated etc) we'll have problems - esp with large uploads and auth.  
But there I go flogging that poor dead horse again...

Adrien

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 7 Apr 2008 16:38:52 +0100

> who implemented 1.0 proxies read that and they should be able to cope 

Unfortunately RFC 1945 says nothing about what 'provisional' means.  I
don't see any language that even suggests it is followed by another
response to the same request.

> But there I go flogging that poor dead horse again...

I'm looking forward to being invited to that HTTP/2.0 design list when
you (or Google/Microsoft/Adobe by the looks of things) get around to it ;-)

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 07 Apr 2008 17:48:20 +0200

> > But there I go flogging that poor dead horse again...

100 Continue + chunked encoding accomplishes this quite well, allowing
for any length of negotiation before the actual upload is sent. It's not
the specs fault these features haven't been properly adopted.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 7 Apr 2008 17:25:43 +0100

> the specs fault these features haven't been properly adopted.

Breaking with HTTP/1.0 proxies and servers is quite a good reason not
to use chunked requests for general purpose HTTP over "the internet".

I don't buy the argument that once you've seen a HTTP/1.1 response
from a domain, you can assume it's a HTTP/1.1 server and proxy chain
for all future requests to that domain.  It's very likely, but not
reliable.  Proxy routes change, reverse proxies route requests to
different servers depending on URL, etc.

As a result, there hasn't been a perceived need or any testing of
chunked requests to servers, and even today, some otherwise good
HTTP/1.1 servers don't support chunked requests.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 07 Apr 2008 21:18:45 +0200

> for all future requests to that domain.

If you are not using a proxy you can, even without knowing it's an
HTTP/1.1 server.

If you are using a proxy things do get a little fuzzier, but generally
it's a safe bet assuming the proxy follows specifications and the client
uses Expect: 100-continue.. (optional, but helps..)

> It's very likely, but not

You should use Expect: 100-continue, and be prepared to resend the
request with a content-length if you see a 411 or 417 response.

But yes, it still MAY end up in odd situations until the world is
HTTP/1.1. Fortunately things do progress towards that..

> HTTP/1.1 servers don't support chunked requests.

I assume they at least respond with a 411. If not they are not HTTP/1.1
servers..

Anyway, what I was coming to is that the mechanism needed is specified
in HTTP/1.1. Specifying it again in HTTP/2.0 isn't likely to render any
significantly better results for as long as compatibility with HTTP/1.0
is required, only add it's own set of complications..

We are now in the situation that chunked encoding for requests is
starting to be considered a useful feature if supported. With a little
pressure server vendors will fix up their support and proxy vendors
follow, and in some years (well.. several) it will quite likely be a
generally useful mechanism with failure being the exceptional situation
rather than the expected one...

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 7 Apr 2008 21:08:58 +0100

> Squid-2.6 was mentioned here, but it's not an HTTP/1.1 proxy..

Squid isn't a person.  I said implementor, not implementation.

At least one person who implements HTTP agents read RFC 2616's
description of proxy behaviour for checking Expect differently to you
and I, and explained his reading earlier in the discussion.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Apr 2008 09:40:11 +1200

I'm going to have to disagree on that one, I don't think 100 Continue 
does a particularly good job at all, for the following reasons.

1. It uses an ill-defined heuristic for what a client should do in the 
absense of 100 continue. The amount of work clients have to do to even 
calculate timeouts is unreasonable (and often impossible), and commonly 
it's not done well.  Again Thomas Moore applies: assuming silence to 
mean anything (either assent or dissent) is just plain wrong, and 
writing them into the protocol we'll bear the consequences whatever they 
are.  In this day and age there's no excuse for not designing completely 
deterministic protocols, especially for arguably the most important 
application protocol we have.
2. It doesn't fail gracefully.  If you use expects and don't get a 100 
continue, you have to wait around, terminate your connection and retry, 
resetting up auth etc etc.  The suggested smart way around this is for 
clients to "remember" which servers are HTTP/1.1, or to use chunked 
uploads.  That's just really bad design: doesn't cope with change, 
imposes unreasonable requirements on clients, and denies the ability for 
proxies to set efficient policy on upload size.
3. The common heurstics don't work through proxies well at all, due to 
timing based on connection to local proxy.  Chain more proxies and it 
just gets worse.

and finally, by no stretch of anyone's imagination is it a negotiated 
transfer.  A client can connect, and spew enormous amounts of resource 
at a server without ever having received a byte in assent from it.  
That's simply not negotiation in anyone's book.

100 Continue is a mild improvement in some scenarios (client direct 
connection to slow server), but does nothing for several common problem 
cases.

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 00:21:50 +0200

tis 2008-04-08 klockan 09:40 +1200 skrev Adrien de Croy:
> absense of 100 continue.

Which is due to HTTP/1.0 not supporting 100 Continue.

>   The suggested smart way around this is for 
> uploads.

chunked uploads and persistent connections is how it should have been
from start.

and as above the reason why it fails is HTTP/1.0 not supporting chunked.

> That's just really bad design: doesn't cope with change, 

Proxies needing up-front knowledge of the size can reject the request
with 417.

> timing based on connection to local proxy.  Chain more proxies and it 

Not by much.

But the heuristics do get quite ifuzzy if using a proxy and the request
is to a host not recently contacted. But the more of the infrastructure
becoming HTTP/1.1 the more aggressive clients can be about requiring 100
Continue.

A client which has seen a HTTP/1.1 response from it's next hop can in
fact in nearly all network configurations switch to be very aggresive
about 100 continue on requests via that hop.

> That's simply not negotiation in anyone's book.

request
4xx response
modified request to meet server requrements
100 Continue
request data
2xx response

how is that not a negotiatated transfer?

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Apr 2008 13:44:51 +1200

> Proxies needing up-front knowledge of the size can reject the request
what does the client then do in that case?

Seems to me that getting around message - length / connection 
maintaining issues by using chunking so you can send a premature final 
chunk instead of terminating the connection is asking for a whole lot of 
pain.  It's an ugly hack.

For starters, there's then no way to tell the recipient that the 
completion was an abortive one.  If there are several processes in an 
intermediary or end server that the data goes through before getting to 
the module that caused the client to abort, then you've got all manner 
of things may happen to that data which appears complete.

That's where it would have been useful to support notifying an abortive 
end ( e.g. previously discussed negative 1 chunk length ) to a 
transfer.  Abort without closing.

All in all I think using chunked uploads is bad for many reasons, apart 
from the fact that it's probably poorly supported even in existing 
HTTP/1.1 infrastructure.  Using it so you can stop sending a body is 
even worse.

> becoming HTTP/1.1 the more aggressive clients can be about requiring 100
Is it just me that finds the idea of a user agent maintaining a list of 
servers and their versions and capabilities a bit desperate?  There are 
a lot of servers out there.  If we start relying on this sort of 
behaviour, what complexity ( = bugs) does this behaviour add, and what 
about all the boundary conditions it creates.

Just seems like a bad idea to rely on that sort of heuristic.

> about 100 continue on requests via that hop.
OK, but I've never seen one.

> how is that not a negotiatated transfer?
that's just one scenario of many upload scenarios.  Take out the 100 
Continue and you don't have pre-agreement (negotiation).  And its so 
incredibly inefficient.  In that original request, many browsers even 
now send the entire entity body. Other browsers who think they might be 
challenged for auth send the initial POST with Content-Length 0, 
regardless of whether they actually will be challenged or not.  A proxy 
that tries to save the client sending the whole body can do nothing 
except terminate the connection, and then there's no guarantee the 
client won't just repeat the same request.

As I said, it's possible ("a client can") to have a connection followed 
by initiation of a large transfer without any notification of acceptance 
by the thing that will have to swallow the data or reject and disconnect.

If you want proof of the problem, try uploading a 100MB file across a 
slow WAN through a proxy that requires auth to a poorly-connected server 
that requires auth (a fairly common scenario actually).  Basically 
impossible because depending on the auth method, that 100MB may have to 
be sent up to 6 times.  Any connection-based auth cannot get around this 
in any reasonable way (yes I know how many people hate NTLM for being 
allegedly non-compliant, but tens if not hundreds of millions of people 
have to live with it).

Proper negotiation would solve this, but would need to take the form of 
establishing in advance permission to upload a body without requiring 
that body to be present in the message that establishes the permission.  
E..g a probe command, or the equivalent of HEAD for POST.

Try an analogy - e.g. a fertiliser delivery company.

the HTTP way:

* The truck turns up and starts dumping fertiliser in your driveway
* you come out and scream at it.
* it stops
* you clean up
* maybe it comes back and dumps some more fertiliser in your driveway 
soon after

Pros:
* delivery co didn't need to spend time calling in advance if you really 
did want the fertiliser
Cons:
* had to send that truck all that way. If you didn't want it, that's a 
big waste
* Even if you didn't want it, you and the delivery company both have to 
sink resources into it.

The proper/polite way (real apriori negotiation)

* the delivery company rings you and checks if it's ok to send the truck 
around
* you say no thanks or yes please.

Pros:
* don't waste money sending a truck around if the recipient didn't want it
* don't waste recipient's time and energy cleaning up
Cons:
* the cost and time relating to a phone call

Add in authentication steps and it gets even more ridiculous. 

Cheers

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 11:07:56 +0200

> chunk instead of terminating the connection is asking for a whole lot of 

I disagree.

> completion was an abortive one.

It's the recipient who asked for it to be aborted by sending an error
instead of 100 continue.

> If there are several processes in an 

> transfer.  Abort without closing.

Then propose a chunk extension for that purpose.. I.e. something like

0; aborted

> HTTP/1.1 infrastructure.  Using it so you can stop sending a body is 

I obviously don't share that opinion.

> by the thing that will have to swallow the data or reject and disconnect.

Yes.

Same is also true for request headers, or URL lengths..

> be sent up to 6 times.

Well, ignoring connection oriented auth the client SHOULD either use
chunked encoding or close the connection, aborting the request when
seeing the challenge. Clients SHOULD NOT send the whole body after
receiving an error. This is already spelled out verbatim in the specs
under "Monitoring Connections for Error Status Messages".

This got screwed up the day someone decided that connection oriented
authentication is a good thing in a message oriented protocol where each
message is supposed to be self-contained... I won't even comment on what
disaster that is to the process. The use of chunked encoding helps
avoiding some of the disaster thankfully.

> * maybe it comes back and dumps some more fertiliser in your driveway 

I see where you are coming, but you got the roles the wrong way around
for it to match HTTP. A more proper analogy would be a person trying to
place an order, or someone wanting to send a package at the post office.

Example based on the order. Note that there is a twist here in that if
you hand over the order form and there is a problem you can't get it
back and have to fill out a new copy.

- Person fills out his order form and goes to the counter where the
order is supposed to be placed.
- If the customer is polite he asks if he may place the order before
handing over the order form, or if in a hurry he hands over the order
form immediately hoping for the best.
- Gets told that he need to have some proof of his customer number in
order to place the order. 
- Goes away to fetch his customer card proving his identity in the
process.
- Back at the counter again repeating the process

Or another version where everything is in order

- Person fills out his order form and goes to the counter where the
order is supposed to be placed.
- The customer is polite and asks if he may place the order before
handing over the order form
- The processing agent is currently busy processing another order and
doesn't answer immediately.
- The person hands over the order form when the processing agent says
it's ready, or if he suspects the processing agent or the communication
with it is oldfashioned and outdated and doesn't indicate readiness.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Apr 2008 22:41:06 +1200

> It's the recipient who asked for it to be aborted by sending an error
I guess that's my point.  If you look at many proxies or servers, where 
you have installable filter modules etc, then a lot of processes can 
exist inside this one "recipient who asked for it".  They may not 
necessarily know that a module upstream of them rejected the message.

I think it's necessary if we are to try and keep connections alive, to 
be able to signal an abortive end to message body transfer without 
closing the connection.  A chunk extension for this as you propose 
sounds like a good way to do this.

> I obviously don't share that opinion.

evidently :)

Actually I think there are some places where chunked uploads are 
appropriate, but my personal opinion is that if at all possible, the 
length should be specified in Content-Length.  Unless a client is 
streaming some on-the-fly generated content up in an upload (that 
couldn't be spooled first), then it should always send a length.  Just 
like I think in SMTP all sending agents should indicate a SIZE in the 
MAIL FROM command.  It's just the most efficient way to implement 
size-based policy.

> Same is also true for request headers, or URL lengths..
sure, but not many serves would accept request headers + URL length over 
about 32kb.  I think the default for IIS is 16 kb.

I'd be surprised if any accept even 1MB.  It doesn't compare to a large 
upload.

> seeing the challenge. 
right - but then that only works with HTTP Basic auth and Digest.  NTLM 
users are out in the cold because their auth method requires a 
persistent connection.  It's a lot easier to write "ignoring connection 
oriented" than it is to ignore all those customers.

It doesn't help either that many browsers don't support receiving any 
data whilst trying to send resource, so won't stop sending even if you 
do send back a 4xx.  I gues contravening your spec reference below.  I 
understand the market-leading browser does this.

> Clients SHOULD NOT send the whole body after
> disaster that is to the process. 

I guess that's what stands HTTP apart from pretty much all other 
transfer protocols (SMTP, FTP, NNTP etc).  By optimising the protocol so 
much to minimise round-trips, we sacrificed certain protocol 
pleasantries that dealt with these sorts of situations.  When we then 
come up against the issues, we have to specify heuristics to get around 
it, and to deal with backward compatibility.  It makes me wonder what 
HTTP will look like in 30 years.

> The use of chunked encoding helps
I guess I was trying to convey the issue of the work involved and 
resources expended by all parties (therefore waste) in even fronting up 
with the request.

I like the sending the package at the post office analogy, but try it 
with 2000kg of sand, and you live on an island and see what the 
logistics are like.  Esp when the small local post office wants to see 
some id, which you have to go back home for, and then front up with 
another 2000kg of sand when you come back again with your ID, then they 
say they don't accept packages that big.

Or worse still then they refer you to the central post office on the 
mainland, which is only accessible by a small rope swing bridge which 
you have to carry this 2000kg over in small bags.  Then they send you 
home for ID as well.

And even with Digest auth and 2 entities wanting auth, you are looking 
to transfer 4 times or drop connections.  I guess that's the beauty of 
digest - you don't need to maintain the connection.

But this really doesn't help so many people who are stuck with NTLM for 
any number of reasons.

And 100 continue is just like turning up to the post office with this 
sand on the truck with the removal guys you are paying $40/hr for, and 
if someone doesn't come out of the post office within 2 seconds, you 
start unloading.  And once you start unloading, you don't stop until 
it's unloaded (what many browsers do).  100 Continue + Expects is the 
same except you toot the horn outside before waiting for 2 seconds then 
starting unloading.

A normal person would ring the post office first :)

So what comes out of all of this?

a) agents should use digest instead of NTLM
b) agents should notice rejections whilst they are sending (already 
specified but not universally observed)
c) on rejection during an upload, agents should disconnect and retry 
with credentials etc.  I can't really endorse using chunked uploads 
until there is a method to signal abortive end to a transfer, and also 
to signal size in advance if known for policy purposes.

which leaves us in the following predicament:

i) what to do about the zillions of people reliant on NTLM - seems like 
the proposal is to basically ignore them?  Or worse still try and 
educate them about using Digest :)
ii) what to do about chunked uploads to fill in the gaps (reporting size 
and signalling abortive end without closing). 

The size thing is a big problem for many sites if we move to chunked 
uploads.  Pretty much every site has a specification for max size POST 
data.  Having to receive it all in order to decide it's too big is 
incredibly wasteful. 

Many people around the world still have to pay for data per MB, and/or 
are on slow connections.

iii) praying that as a proxy we never get large chunked requests to 
process for upstream HTTP/1.0 agents.  In many cases there is no viable 
solution to this problem, since the proxy has to spool the entire 
request before submitting anything to the server - it can't send 
anything through without a Content-Length.  Actually for this reason and 
ii) above, I'd be a keen fan of using Content-Length AND chunking for 
client message bodies (even though it's utterly prohibited in the 
spec).  At least then the proxy could stream stuff through to the HTTP 
1.0 next hop as it received it (after of course first establishing that 
the next hop actually IS HTTP/1.0 by having its first request bounced).

Regards
Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 14:10:19 +0200

> oriented" than it is to ignore all those customers.

Using the chunked method works just fine for NTLM.

> do send back a 4xx.

That's a bug/misfeature. Not someting we can help by adding more
features to the protocol.

> I gues contravening your spec reference below.  I 

So someone should beat them with a big stick then...

> HTTP will look like in 30 years.

The day we can forget HTTP/1.0 things will stabilize considerably.

But in 30 years I expect that HTTP has been pretty much replaced by
something new, more targeted for interactive transfer of large amounts
of data.

> say they don't accept packages that big.

Only if you refuse to do what the specs says you should do.

Specs says you should contact the post office, say I want to send 2000kg
of sand and then wait for confirmation. The twist is that if there is
neither a confirmation or rejection within a reasonable time frame and
you don't know if the post office or your communication channel to the
post office supports confirmation then you should not wait forever and
instead assume it's acceptable if no response is seen within a
reasonable timeframe.

Specs absolutely do not say that you SHOULD drive to the post office
with all that 2000kg of sand blindly assume they will handle it for you
by default.

> you have to carry this 2000kg over in small bags.  Then they send you 

Stop assuming that you have to carry those 2000kb, it's a false
assumption. You do not have to. You can select to but it's your own
choice.

> digest - you don't need to maintain the connection.

Just follow the specs and you will be fine.

With NTLM it's harder to follow the specs, but thats not the specs
fault. NTLM should have been implemented at the message level, not
connection. Digest is one source of inspiration on how a such session
oriented authentication scheme may look like without tying it to the
transport.

> But this really doesn't help so many people who are stuck with NTLM for 

Until something better can replace NTLM/Negotiate.

> start unloading.

Now you are just too pessimistic about 100 Continue. Most web servers
out there is HTTP/1.1 and do send 100 Continue (even old ones just
implementing RFC2068). And the major commercial proxy servers it
HTTP/1.1 as well. Squid is not, but that's another story..

> And once you start unloading, you don't stop until 

I don't view it that way. I would view it that you either first send the
request in a separate car or phone it in. Then at a suitable time send
the truck with all the sand.

> A normal person would ring the post office first :)

And is what expect 100-continue does. You should view the heuristics in
the specs as what you do if the post office doesn't care to answer the
phone.

> a) agents should use digest instead of NTLM

Yes, or a replacement along the same lines (session oriented auth at the
message level, not connection oriented)

> specified but not universally observed)

YEs.

> to signal size in advance if known for policy purposes.

Well, I don't consider the intermediary issues you push on that
critical. Intermediaries should follow 'b' so you know that any data
seen after a error response is to be considered garbage. It shouldn't
even be forwarded, so you should not take any actions based on that
data.

> educate them about using Digest :)

Find a better replacement for NTLM that actually works with the HTTP
specifications, then get vendors to accept it. Not technically very
hard, and takes about the same time to roll out as any other noticeable
change.

> and signalling abortive end without closing). 

?

Signalling the abort differently than just "end of request" is purely
optional. Whoever you are talking to SHOULD have already realised the
request has failed.

> data.  Having to receive it all in order to decide it's too big is 

True. And to address this one can introduce an advisory header carrying
the expected size of the transmitted entity. It has to be different from
Content-Length for protocol reasons.

> are on stow connections.

The more important for them that their clients and any proxies they use
actually follow specs then.

> process for upstream HTTP/1.0 agents.

Proxies should respond with 411 in such case, making communication
revert to "HTTP/1.0 compatible" with all the problems that have..

> In many cases there is no viable 
> request before submitting anything to the server

No, it should not. Proxies should use the same rules on when chunked is
acceptable as any other clients, and reject to forward the message if
they think it won't work out. That's what 411 and 417 is about,
downgrading the protocol when needed. Remember that proxies act as both
servers and clients and have to fulfill both sets of requirements.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Apr 2008 00:54:44 +1200

> Using the chunked method works just fine for NTLM.
we just need to get some browsers to support it :)

> So someone should beat them with a big stick then...
I'll have to leave that to someone closer to them....

> something new, more targeted for interactive transfer of large amounts

heh - sounds like FTP...

> assumption. You do not have to. You can select to but it's your own
Coming from the proxy side, I see what the clients do, and I either see 
Firefox sending the whole thing each time, or IE trying a POST with 
Content-Length:0 first if it thinks I will bounce it with a 407.  Since 
POST with Content-Length:0 is not invalid, this creates several problems.

But if we put that aside as 2 examples of bad behaviour and move onwards...

> oriented authentication scheme may look like without tying it to the
I've been looking more into this... it does have some difficulties, 
seemingly keying on the URI, and requiring the server to maintain an 
independent cache of credential handles.

All doable, but quite different from session-oriented.  It's easy to see 
why session-oriented was chosen.  Makes association of interim / 
temporary credential handles with a user trivial in most cases.

> HTTP/1.1 as well. Squid is not, but that's another story..
I guess that's the thing.  100-continue -> timeout ->start sending is 
the optimistic option.

> the specs as what you do if the post office doesn't care to answer the
OK.

Browser authors needs to work on heuristics then.  Basing timing from 
RTT to a local proxy when a remote server may be poorly connected means 
timeouts will happen more often.

> even be forwarded, so you should not take any actions based on that

OK, fair point.. chained filters should notice outbound 4xx as well.

I still feel uncomfortable about aborting without explicitly marking it 
as such.  It may be implicit, but if not explicitly marked, that 
precludes any possible future use-case where a 4xx response could 
possibly not invalidate the data.  I can't think of any such case, but 
I'm not quite ready to stake that it will never eventuate.

Could make a SHOULD level requirement that user agents sending chunked 
requests that choose to abort them SHOULD include a "0 ; aborted" chunk 
extension.  Then at least the possibility exists that it can be noted as 
an explicitly aborted termination by anything that recognises that 
without having to assume so based on previous protocol traffic.  That 
would avoid boundary/race conditions (e.g. sent final chunk before 
receiving 4xx/5xx/3xx?)

> Content-Length for protocol reasons.
OK, I'd be happy with anything that can tell me the length.  I'm not 
aware of why it can't be Content-Length for client requests (it's more 
obvious to me when it comes to server responses), but if it could be 
converted to a Content-Length header for relaying upstream that solves 
spooling and flow-control issues.

For instance there are 3 boundary conditions for Content-Length and 
chunking.

Content-Length tag matches amount transmitted with chunking.  Everything 
is ok
Content-Length is bigger.  Send aborted or other problem discard.
Content-Length is smaller.  Client was lying.

> actually follow specs then.
so we still have a long way to go with browser behaviour then.

> revert to "HTTP/1.0 compatible" with all the problems that have..

I guess so, or at least to retry without chunking.  I guess the 411 
would come from the HTTP/1.0 server?  How to tell that the server is 
HTTP/1.0 if it doesn't recognise Transfer-Encoding, and just sees chunk 
wrappers as part of the content anyway?  Would it be purely lack of 
Content-Length?  Or would you have to rely on cached knowledge?

> servers and clients and have to fulfill both sets of requirements.
I guess this means that proxies have to start caching details of servers 
then.  Problem is that caching things like HTTP versions etc unlike 
normal caching in HTTP doesn't have the benefit of explicit cache 
support - specified expiries, dates, etc.  There's nothing governing 
that caching.

Thanks for your explanations.

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 14:55:34 +0200

> what does the client then do in that case?

I meant to say 411, but the answer is pretty much the same. Adjust the
request and try again.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Apr 2008 01:08:14 +1200

Adrien de Croy wrote:
> HTTP/1.0 if it doesn't recognise Transfer-Encoding, and just sees 
> chunk wrappers as part of the content anyway?  Would it be purely lack 
> of Content-Length?  Or would you have to rely on cached knowledge?
>

Just looked it up.. RFC1945 specifies that a server receiving a POST 
without Content-Length (or if it cannot determine the length - dunno how 
it could otherwise) SHOULD respond with a 400 (bad request).  There's no 
code specified in HTTP 1.0 for length required, so I guess it comes down 
to a heuristic.

If a proxy sends a chunked POST to a server, and that server responds 
with HTTP/1.0 and 400 (but not 401, 403 or 404) then the response the 
proxy should send to the client should be 411 Length required?

Seems like a bit of an assumption, although the HTTP/1.0 is a bit of a 
giveaway, but it's still possible the request was bounced for some other 
reason, and will bounce again even if the client can and does submit 
with Content-Length.

Cheers

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Apr 2008 01:11:12 +1200

or should the proxy always respond with the HTTP version number it 
received in a response?

I know for upstream the proxy should alway set the protocol version to 
it's own, but downstream?

Else there's no way a client can tell what the end server capabilities 
might be.

Adrien de Croy wrote:
>> chunk wrappers as part of the content anyway?  Would it be purely 
>> lack of Content-Length?  Or would you have to rely on cached knowledge?
>>
>
> without Content-Length (or if it cannot determine the length - dunno 
> how it could otherwise) SHOULD respond with a 400 (bad request).  
> There's no code specified in HTTP 1.0 for length required, so I guess 
> it comes down to a heuristic.
>
> giveaway, but it's still possible the request was bounced for some 
> other reason, and will bounce again even if the client can and does 
> submit with Content-Length.
>
> Cheers
>
> Adrien
>

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Apr 2008 01:30:04 +1200

doh, just read RFC2145, which answers that question.

although it looks like RFC2145 ought to have been superseded... couple 
of issues in it. (precludes Expects from HTTP/1.1, suggests sending 505 
response for version issues, even though this code not defined in HTTP/1.0)

Adrien de Croy wrote:
>>>>> iii) praying that as a proxy we never get large chunked requests 
>>>>> to process for upstream HTTP/1.0 agents.
>>>>>     
>>>>
>> Seems like a bit of an assumption, although the HTTP/1.0 is a bit of 
>> a giveaway, but it's still possible the request was bounced for some

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 16:06:38 +0200

> heh - sounds like FTP...

My thinking was more in the line one of the streaming protocols,
allowing seeking etc without loosing the session, and low initiation
overhead to deal with the problem of increasing bandwidth but pretty
much constant rtt..

Current HTTP beats FTP on all aspects, and FTP is likely to decline
considerably over time.

> independent cache of credential handles.

Yes, it needs a cache of the session keying material just as Digest
does.

> temporary credential handles with a user trivial in most cases.

What do you refer to here by "session-oriented"?

> I guess that's the thing.  100-continue -> timeout ->start sending is 

Yes. You start out by assuming 100 Continue is supported, and the fall
back gracefully to HTTP/1.0 behaviour if you suspect it's not..

The less we see of HTTP/1.0 in the relevant traffic, the less you need
to fall back on timeout..

> timeouts will happen more often.

Indeed.

If the client has seen 100 Continue for the server then it SHOULD be
very aggressive about wanting to see it in future.

If it hasn't seen 100 Continue from the server then it should still
start out pretty aggressive about wanting to see 100 Continue, and tune
future requests to adopt if it's suspected 100 Continue is not
supported.

> possibly not invalidate the data.

That should be implemented with an 1xx message, followed by 4xx when the
data has been received.

> extension.

I would make propose it being a MAY for 1.1, SHOULD or even MUST for
1.2.
   
> spooling and flow-control issues.

It can't be Content-Length due to interactions with current
implementations looking for Content-Length to determine the message
length. Especially if there is HTTP/1.0 hops in the chain..

When using chunked encoding Content-Length != message length.

> so we still have a long way to go with browser behaviour then.

as always..

> Content-Length?  Or would you have to rely on cached knowledge?

Most 1.0 servers probably respond either with 400 or maybe 500 or some
other error on such requests depending on the nature of how they fail.
   
> then.

Yes.

> Problem is that caching things like HTTP versions etc unlike 

And it's not very likely servers suddenly downgrade.. How long or how
many servers to keep track of is an implementation detail based on the
trust in implementers to use common sense.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 16:15:14 +0200

> proxy should send to the client should be 411 Length required?

Probably, or 400 and let the requesting client deal with it.

But the proxy should only accept forwarding the chunked encoding if it
has reason to believe the server really is HTTP/1.1. I.e. if it has seen
an HTTP/1.1 response from it..  (server in this context is the next hop,
not origin server).

So a client sending a POST to a origin server where the proxy knows it's
an HTTP/1.0 server or where it's not known SHOULD result in an 411 by
the proxy. This could happen "randomly" if the server has not been
contacted before (or expired from the version cache) and the proxy is
not willing to probe the server with OPTIONS * or similar probe request.

> reason, and will bounce again even if the client can and does submit 

Then it will bounce again, and the proxy will this time know that it's
not the lack of Content-Length causing the bounce..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 16:29:52 +0200

> it's own, but downstream?

Currently an implementation choice, but it's best if it always uses it's
own version applying the same rule as specified for requests. Actually
less impedance between the protocol versions that way than trying to
switch the protocol level depending on the server..

The day there is a need to bump the major version many things will
change and protocol gatewaying of http1<->2 will be an issue to tackle..
but we are not in that position today and unclear if it ever will be..

> Else there's no way a client can tell what the end server capabilities 

It doesn't need to. Thats best delegated to the proxy and communicated
to the client by using 411/417 which is just a single packet RTT in
overhead.

If the client really wants to it MAY inspect the Via header to tell the
capabilities of the forwarding path the request/response travelled via,
but is not required.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 16:39:34 +0200

> response for version issues, even though this code not defined in HTTP/1.0)

HTTP/1.0 isn't really relevant there..

505 is to be returned if an HTTP/1.x server gets a HTTP/2.x or higher
request.. or if an HTTP/2.x only server gets a HTTP/1.x request.

HTTP/1.X servers will never respond with 505 to HTTP/1.0 requests.

Regards
Henrik

---

From: Patrick McManus <[email-masked]>
Date: Tue, 08 Apr 2008 10:44:20 -0400

> I would make propose it being a MAY for 1.1, SHOULD or even MUST for

I think it is great for 1.2, but doesn't help at all for 1.1.. deployed
engines will just ignore the unknown extension and view the message as a
complete one.. the sender is better off terminating the connection
early, creating a malformed chunked message and thereby at least getting
its point across - which is what happens now. I've written and
interoperated with sofwtare that works that way.

And its a huge improvement over the HTTP/1.0 semantics which has no
chunking and no content-length.. why no content-length? Because
streaming is often really important, either out of concern for latency
or state (especially in an intermediary). At least in HTTP/1.1 responses
chunking can be reliably used.

FWIW, It seems pretty common nowadays for HTTP/1.1 servers to correctly
accept chunked requests. I don't mean to say it's a sure thing - but
more often than not the server gets it right. That's a change in the
last few years. I've never implemented with an auto-discover cache so I
don't have numbers (my products just used config switches for enabling
chunked requests) but I would be interested in seeing them..

streaming seems an equally important feature for client and server to
me.

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Apr 2008 02:46:55 +1200

It's quite an interesting interop issue though between 1.1 and 1.0, 
since there are more parts to a message than just the headers.

e.g the status codes.  The set for 1.0 is relatively small.  If a sender 
is sending a status code back, surely it's required to sanitise it for 
the recipient?  Or does it expect a downstream agent to pass it through 
yet only understand the major error code.

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 08 Apr 2008 17:17:57 +0200

> yet only understand the major error code.

Even in HTTP/1.0 the set of status codes is extensible and clients MUST
be prepared to deal with status codes they haven't seen before.. but
they obviously do not need to be prepared to take automatic actions to
correct errors they know nothing about..

Thats one reason why the status codes is divided into xYY groups, where
x being the major indication on how the status should be dealt with. If
the exact xYY status isn't known it's to be dealt with as x00 which is
the default status if none of the other in the relevant group applies..

Quote from rfc1945:

   applications must understand the class of any status code, as
   indicated by the first digit, and treat any unrecognized response as
   being equivalent to the x00 status code of that class, with the

the exact same text also in 2616 but more formal..

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 09 Apr 2008 10:19:28 +1200

> Current HTTP beats FTP on all aspects, and FTP is likely to decline
definitely.

> What do you refer to here by "session-oriented"?
sorry, I meant connection-oriented.

> to fall back on timeout..
but still relying on either chunking or disconnection, which breaks
connection-based auth.

OK, I'm with you.

> I would make propose it being a MAY for 1.1, SHOULD or even MUST for

I hadn't even considered for 1.1, but that would be great.  Very easy to
test for.

> length. Especially if there is HTTP/1.0 hops in the chain..
ah, because we rely on HTTP 1.0 to bounce a POST with no
Content-Length.  HTTP1.1 existing implementations should be ignoring
it.  It would only be new ones that would use both.

But I agree, another field would be better then.

>> I guess this means that proxies have to start caching details of servers 
:(

> trust in implementers to use common sense.

now where have I heard that before :)

---

From: Lisa Dusseault <[email-masked]>
Date: Wed, 9 Apr 2008 09:17:39 -0700

On PATCH for resumption of upload:

On Apr 5, 2008, at 5:33 AM, Bill de hOra wrote:
>
> Then the client can send the rest of the data offset from 'nnn'. But  
> to make the semantics clear I use PATCH (or overloaded POST with X- 
> HTTP-Method-Override). Definitely not PUT.
>

I've thought about that too; one of the use cases driving the  
submission of version 00 of draft-dusseault-http-patch was talking to  
Adobe engineers doing WebDAV, who wanted better ways for collaborating  
designers to make changes to large documents e.g. Photoshop  
documents.  Some document formats can have changes applied piecemeal,  
which can even be useful when a certain amount of document  
construction or history is retained within the document itself.  So e.g:

	LOCK ourdocument
	PATCH ourdocument
		Add a new layer and upload an image into that layer
	PATCH ourdocument
		Upload an image into an existing layer
	PATCH ourdocument
		specify an algorithm to combine two layers
	UNLOCK ourdocument

(I'm making the internal change details up, please don't quibble with  
the details :)

One of the things that worries me, in the case of PATCH used for  
partial upload of a pre-existing resource, is when there are document  
formats where it's not possible to keep the document valid and also  
break it into upload chunks.  A document should remain valid.  I guess  
I can see a couple solutions to that but I'm not sure what's best.

	1. Change the content-type to application/octet-stream during one  
PATCH, and change it back to the real valid content type at the last  
PATCH -- this requires servers to handle Content-Type as a  
modification on PATCH requests

	2.  Applications are encouraged to create tmp files through PUT and  
multiple PATCHes, then COPY that over top of the previous destination  
file.  This has the drawback of requiring WebDAV, although so does the  
LOCK example above

Lisa

---

From: "Charles Fry" <[email-masked]>
Date: Thu, 10 Apr 2008 08:44:23 -0400

Would co-advisor be the right/best way to test what existing proxies
do when they receive an unsolicited 103 response?

Charles

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Apr 2008 07:40:17 +1000

Good question; probably best to ask one of the MF guys (Alex CC'ed).

Alex - do you have user-configurable testing now?

I've also been thinking about working on an open source HTTP testing  
framework (not proxy-specific), but that's just an idea and a few bits  
of random Python at this point...

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 10 Apr 2008 16:11:38 -0600

> Alex - do you have user-configurable testing now?

Yes, but the focus of that work was not on HTTP. Regardless of whether
Co-Advisor scripting works here, I am sure we can hack a few custom test
cases in if somebody wants to test a bunch of HTTP proxies for something
outside of MUST-level RFC 2616 requirements.

On the other hand, if we are talking just about a single simple test
case, a custom Perl or Python script can do the job as well. You can
even reuse curl/wget on the client side.

Please let me know if you need some assistance from the Factory.

Thank you,

Alex.

---

From: "Charles Fry" <[email-masked]>
Date: Thu, 10 Apr 2008 20:21:27 -0400

>  outside of MUST-level RFC 2616 requirements.

We are primarily interested in testing RFC 2616 MUST-level
requirements, or discovering the results of such tests if they have
already happened. Specifically, section 10.1: "Proxies MUST forward
1xx responses, unless the connection between the proxy and its client
has been closed, or unless the proxy itself requested the generation
of the 1xx response."

So if a server sends an unrequested, unrecognized 1xx response (say
103), how do current proxies respond?

We are interested in HTTP/1.0 proxies in addition to HTTP/1.1 proxies.
Speaking of which, does anybody have any statistics on the relative
distribution of such proxies in the wild?

Charles

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Apr 2008 10:32:00 +1000

Very hard to say; people differ widely about the deployment footprint  
of proxies, regardless of version. if you can get your hands on a  
popular Web site's access logs :), you could look at the Via header's  
contents, although some number of proxies will not append it (as  
previously discussed).

I would point out that Squid is by far one of the more popular  
choices, especially in small ISPs, 2nd- and 3rd- world countries,  
etc., and it's still HTTP/1.0 (although 2.7 will have a HTTP/1.1 mode,  
and from the testing I've done, it's pretty conformant).

Cheers,

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 11 Apr 2008 15:40:21 +0200

> We are interested in HTTP/1.0 proxies in addition to HTTP/1.1 proxies.

HTTP/1.0 proxies is quite irrelevant here as you MUST NOT send 1xx
responses to an HTTP/1.0 client. Only one or two servers has been seen
not following this.

> distribution of such proxies in the wild?

Squid is fairly common and not yet HTTP/1.1. But some work is underway
on that and the upgrade cycle among the Squid users is at most in years,
not decades.. but somewhat longer cycles can be expected now with Squid
being in several enterprise distributions..

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 11 Apr 2008 15:43:46 +0200

> and from the testing I've done, it's pretty conformant).

Still no support for chunked requests. But it does respond with 411 if
seeing one..

Also no forwarding of 1xx responses yet, which is the primary reason why
it's not enabled by default.

Regards
Henrik