Subject: expect header, clarification of status

From: Stefan Eissing <[email-masked]>
Date: Tue, 28 Jan 2003 14:09:16 +0100

Dear Experts,

I'd like to get a clarification of the HTTP/1.1 Expect header (RFC  
2616, 14.20).
This feature is very useful to authoring clients, however it currently  
causes
interoperability problems far outweighting the potential benefits.

There has been discussion on the http-wg mailing list ([1]) with the  
statement
by Roy that 2616 has broken compatibility to 2068.

There is however no clarification in the errata[2].

The conclusion I draw from the mentioned discussion is that
a) clients, talking to HTTP/1.1 servers, cannot rely on correct
     Expect header support, since they may be talking to 2068
     compliant servers.
b) Requests which have a body will cause the client to hang
    on a 2068 server. Unfortunately, using Expect together with
    PUT is one of the more attractive use cases.

Assuming these conclusions are correct, do you have any opinion
which direction WebDAV should take in this matter? It is highly
desirable to offer something like Execpt: 100-continue to a client,
before it tries PUTting its 100MB file onto a server.

As far as I can see, announcing support for Expect in a OPTIONS
response seems to be one way to move forward. Are there other
ideas how to make Except usable to clients?

//Stefan

1)  
http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998MayAug/ 
0192.html
2) http://purl.org/NET/http-errata

---

From: Scott Lawrence <[email-masked]>
Date: 28 Jan 2003 09:38:51 -0500

Stefan Eissing <[email-masked]> writes:

> 2616, 14.20).  This feature is very useful to authoring clients,
> however it currently causes interoperability problems far
> outweighting the potential benefits.

What problems?
 
> statement by Roy that 2616 has broken compatibility to 2068.
>

The discussion you cited was during the development of RFC 2616; the
errata only attempts to cover issues that arose after publication.
 
>     PUT is one of the more attractive use cases.

Section 8.2.3 lays out procedures for both client and server that I
believe work for all the combinations of Expect header
support/non-support. 
 
--
Scott Lawrence        
  http://world.std.com/~lawrence/
  Actively seeking work

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 28 Jan 2003 16:34:59 +0100

Scott,

thanks for the quick reply. Comments below:

Am Dienstag, 28.01.03, um 15:38 Uhr (Europe/Berlin) schrieb Scott 

>> outweighting the potential benefits.

If a client submits a request against a HTTP/1.1 server with:

Expect: greenbytes-super-feature-enabled

it has no way, in my understanding, to know if
1) the feature is supported
2) the feature is unsupported and the server complies only to 2068

That makes the Expect header useless for extension checks until the
protocol version is incremented some (far) time in the future.

> believe work for all the combinations of Expect header
I agree that 8.2.3 covers PUT with Expect: 100-continue, however the
offered solution gives a rather unsatisfying user experience in my
opinion.  There is no good heuristic to determine how long to wait
for the 100 Continue response. It does depend not so much on
network latency, but on server load, e.g. how long the server needs
to check whatever it wants to check before accepting the PUT.

"Expect" is a very useful feature, but in order to use it reliably, one
needs a way to discover its support for a certain resource. Don't
you agree?

//Stefan

---

From: Scott Lawrence <[email-masked]>
Date: 28 Jan 2003 11:14:03 -0500

> needs a way to discover its support for a certain resource. Don't

The same might be said of several HTTP features, but there just isn't
a mechanism in the protocol to help you right now.  We tried and
failed to make a richer OPTIONS response; that might have been a good
answer. 

I'm afraid that what you are stuck with is that if you want to extend
the protocol and be able to determine reliably whether or not that
extension is supported by a given resource, then you must also specify
a robust mechanism for checking for that support.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 28 Jan 2003 09:57:03 -0700 (MST)

> outweighting the potential benefits.

True. A quick look at our test results shows that our of 6 HTTP
proxies, 5 violate at least one Expect-related test case, and some
violate most of them. I suspect that origin server support is similar.
Moreover, since Expect mechanism is hop-by-hop, you have to have
compliance all the way through to the origin server.

Furthermore, many intermediaries do not forward 1xx responses to
clients or forward the first 1xx response and then close the
connection. This breaks compatibility further.

>    Expect header support, since they may be talking to 2068

Clients, in general, do not know the version of the "server" in this
context because the mechanism is hop-by-hop and you may have a mix of
HTTP/1.1 and 1.0 intermediaries (including things like surrogates).

>     PUT is one of the more attractive use cases.

I think it is OK to assume that all servers out there are either
HTTP/1.0 or (2616-based) HTTP/1.1. This is not true in reality, but
the fact that many servers violate RFC 2616 they are based on, makes
the 2616/2068 distinction unimportant. Hopefully, we can increase the
number of RFC 2616 compliant agents in the foreseeable future. We are
successfully working with many open and proprietary projects on that.

In other words, forget about 2068, but do not assume that every
HTTP/1.1 server complies with the 2616 specs it was built on.

> response seems to be one way to move forward.

It is kind of ironic that one would put support for a MUST-level
feature into OPTIONS. An even bigger problem is that OPTIONS (and
Max-Forwards) are not well supported either.

> Are there other ideas how to make Except usable to clients?

I think you have to make a choice:

	a) Rely on (in specs) and push for (in public forums)
	  RFC 2616 compliance. Implementations will run into
	  real-life compatibility problems. In my experience,
	  the popularity of WebDAV will force the majority
	  of implementations to become compliant (in
	  relevant areas) relatively soon after WebDAV users
	  start to bombard server/proxy support teams with bug
	  reports. WebDAV already relies on things that do not work
	  perfectly in real world, such as extension request
	  methods (IIRC), and proxies/servers are improving
	  their support for that.

	b) Use just the basic RFC 2616 functionality. Never
	  use Expect/OPTIONs/chunked requests unless there
	  is a safe and clear way to back off to something
	  simpler. For example, you can send 100MB objects
	  in small chunks (one chunk per HTTP exchange) and
	  provide an extension header identifying those
	  chunks so that a server that supports WebDAV would
	  be able to assemble a complete object while
	  all intermediaries would be happy to pass a simple
	  PUT trough. Alternatively, you can invent an
	  XOPTIONS/XEXPECT pair with end-to-end semantics and
	  sent XOPTIONS request before uploading 100MB object.
	  All this introduces extra bytes on the wire.
	  This also complicates your specs and implementations,
	  but it makes immediate deployment more smooth.

Some people will call option (a) the "Right Thing To Do". Some people
will argue that option (b) is the only way it can be done.

HTH,

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 28 Jan 2003 11:14:03 -0800

FYI, there was a paper at the 7th International Workshop on 
Web Content Caching and Distribution (WCW) last year:

    Adam D. Bradley, Azer Bestavros, Assaf J. Kfoury (Boston University) 
    "Safe Composition of Web Communication Protocols for Extensible Edge
    Services"
    http://2002.iwcw.org/papers/18500001.pdf

that's actually about how to formally check the interoperability
of different versions of the Expect/Continue mechanism.  From
their abstract:

    we show how ...  a tool from the formal systems verification
    community can be used to quickly identify problematic behaviors
    of application-layer protocols with non-trivial communication
    models such as HTTP with the addition of the  100 Continue
    mechanism. As a case study, we examine several versions of the
    specification for the Continue mechanism; our experiments
    mechanically uncover multi-version interoperability problems,
    including some which motivated revisions of HTTP/1.1 and some
    which persist even in the current version of the protocol. We
    develop relations for describing arbitrarily large compositions of
    HTTP proxies using finite models, and also discuss the broader
    applicability of these techniques to open internet protocol
    development.

The lessons from this paper support Alex's belief that the only
two solutions to your problem are (1) avoid using Expect/Continue
if you want 100% reliable results, or (2) aggressively work to
get rid of non-RFC2616-compliant implementations.

My personal belief is that people who deployed RFC2068-compliant
implementations *in production use* were naive, stupid, or
obstructionist, and we should have zero sympathy for them.  The
IETF process does insist on interoperability testing of Proposed
Standard specifications, and people who did *implement* RFC2068-
compliant systems deserve our gratitude, but the IETF (see
RFC2026) neither requires nor condones "deploying implementations
of [proposed] standards into a disruption-sensitive environment."

-Jeff

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 29 Jan 2003 12:11:44 +0100

Thanks Alex for the excellent summary. I hadn't though yet about
the intermediates problem regarding Expect.

The way forward is probably a mix of your a) and b) scenarios. In case
of methods like OPTIONS and others (and WebDAV has quite a lot) most
servers nowadays have good support. Apple has support calls with
HTTP proxies regarding their WebDAV iMac service. They have real
interest ot solve it. So, I think, the method "problem" will go away
rather sooner than later.

Chunked requests are still a problem. Even when servers are ok, there
are a still servlet engines which screw up. At least we got Tomcat to
support it since 3.1.

I am currently trying to confirm that proper handling of Expect is *not*
possible with the current Java Servlet API. That would be real bad.

If this is indeed the case, we need to work on another solution in
WebDAV for the PUT/continue which works end-to-end. Currently,
a PATCH kind of method looks attractive. The discovery of
supported methods works quite well with WebDAV/OPTIONS.

//Stefan

Am Dienstag, 28.01.03, um 17:57 Uhr (Europe/Berlin) schrieb Alex