Subject: update: http progress notification

From: Adrien de Croy <[email-masked]>
Date: Mon, 24 May 2010 13:13:53 +1200

Hi all

We did some testing here using multiple 102 responses (from a perl 
script) prior to the final response.

We also made some modifications to chromium to test user experience.  We 
plan to make a few more changes (advertisement of support) then we will 
have a client available for anyone interested in testing this.  Initial 
results are very encouraging.

Tests with unmodified Firefox and IE showed no problem with multiple 102 
responses prior to the final response as well (even though this won't be 
an issue since the draft proposes the client explicitly advertises support).

Regards

Adrien

---

From: Henrik Nordström <[email-masked]>
Date: Wed, 26 May 2010 20:00:41 +0200

> an issue since the draft proposes the client explicitly advertises support).

Hmm.. is that explicit negotiation really needed?

Checking.. right the specs says "Unexpected 1xx status responses MAY be
ignored by a user agent", not MUST, so sending progress updates MAY
break HTTP/1.1 compliant clients.

The question however is if that perhaps should be a MUST?

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 26 May 2010 14:09:02 -0700

> The question however is if that perhaps should be a MUST?

No.  There is nothing wrong with the recipient doing other things,
like maintaining a sidebar with logs or displaying the status
briefly in some status area.  The fact that it isn't ignored does
not imply that the recipient keels over or explodes.

....Roy

---

From: Henrik Nordström <[email-masked]>
Date: Thu, 27 May 2010 13:04:31 +0200

> not imply that the recipient keels over or explodes.

Fair enough.

Just got a little confused about 101 which clearly must not be ignored
even if the client doesn't know about 101, and the implications of that
on introduction of new 1xx status codes.

But somehow I overlooked the first part of the same paragraph. "A client
MUST be prepared to accept one or more 1xx status responses prior to a
regular response" which covers things nicely, and the oddness of 101 is
covered by other means (Connection). We may want to clarify that to add
"or other" in the part that mentions "expect a 100"....

So no negotiation is really needed for the use of a Progress status, and
servers MAY send Progress notifications whenever they deem it's good as
long as general 1xx requirements is fulfilled. In many cases it would
help even clients without Progress support as it keeps the connection
alive avoiding timeout.

If it's found that there is clients failing due to progress notification
and getting those fixed is not an viable option then administratively
configuring the server or proxy with a blacklist is imho the best
solution to the "negotiation" problem.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 28 May 2010 00:18:52 +1200

> Hmm.. is that explicit negotiation really needed?
the hop-by-hop negotiation was the justification for

a) dropping a requirement to suppress these 1xx responses for HTTP/1.0 
clients
b) not worrying about semantics of entity headers transported back in a 
1xx response.

If we drop the requirement to advertise support from the client, we 
would need to re-think these.

An alternative could be a Progress header in the request, which then 
would normally be forwarded by proxies.  Then this progress mechanism 
could work through an HTTP/1.1 proxy that didn't know about it as long 
as it forwards 1xx responses.

Regards
Adrien

---

From: Henrik Nordström <[email-masked]>
Date: Thu, 27 May 2010 16:37:22 +0200

> a) dropping a requirement to suppress these 1xx responses for HTTP/1.0 

That's part of the HTTP specification. Only need to be repeated here in
text regarding to negotiation support the new response code.

> b) not worrying about semantics of entity headers transported back in a 

Not sure we need to worry about those, with the exception of
Content-Length.

> as it forwards 1xx responses.

Not sure if/how that makes a difference.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 28 May 2010 09:56:36 +1200

OK, so HTTP/1.0 is covered in any case, since 1xx responses are not 
allowed to HTTP/1.0 clients.

That just leaves the issue of whether a client should indicate support 
(or desire) for progress notifications or not.

I'm a little bit reluctant to suggest the progress notifications should 
not require any indication from the client.  If only because some 
clients may want to be able to suppress generation of such messages for 
whatever reason.

So I think my preferred option is a Progress token in the Connection 
header.  That way I don't have to try and dream up something useful in 
terms of parameters for a potential Progress request header.  It also 
means clients that aren't written (e.g. existing clients) to process 
these messages won't receive them for discarding and waste bandwidth 
(esp over slower high-cost circuits such as GPRS / G3 etc).

So, I'll mark up a new I-D for this, proposing 103 as the status which 
appears to be free still in the IANA registry.

Thanks all for your help.

Regards

Adrien

---

From: Henrik Nordström <[email-masked]>
Date: Fri, 28 May 2010 00:25:18 +0200

> clients may want to be able to suppress generation of such messages for 

A header saying that you do not want notifications perhaps?

But I don't see it likely to be used much, and in those few situations
where I can see it being usedul it can just as well be handled by the
closest proxy with a local policy.

> (esp over slower high-cost circuits such as GPRS / G3 etc).

It also means that every client which supports this need to add more
overhead to all their requests, plus that existing clients will not
benefit at all from servers supporting progress notifications and will
still suffer from timeouts.

By not requiring negotiation all existing clients will immediately
benefit from the new messages keeping the connection alive, and avoid
clients or network equipment timing out the transaction.

The likelyhood that the progress notifications break existing software
is fairly small, and then only when using software not compliant with
RFC2616 (MUST level requirement to not crash & burn due to unexpected
1xx responses).

Regards
Henrik

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 28 May 2010 00:42:35 +0200 (CEST)

On Fri, 28 May 2010, Henrik Nordström wrote:

> The likelyhood that the progress notifications break existing software is 
> fairly small, and then only when using software not compliant with RFC2616 
> (MUST level requirement to not crash & burn due to unexpected 1xx 
> responses).

... but OTOH, there's basically nothing today that sends 1xx responses without 
the client asking for it so I would expect a fair amount of breakage simply 
because it hasn't been tested properly before.

-- 

  / daniel.haxx.se

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 27 May 2010 15:45:20 -0700

On May 27, 2010, at 2:56 PM, Adrien de Croy wrote:
> OK, so HTTP/1.0 is covered in any case, since 1xx responses are not allowed to HTTP/1.0 clients.
> 
> That just leaves the issue of whether a client should indicate support (or desire) for progress notifications or not.
> 
> I'm a little bit reluctant to suggest the progress notifications should not require any indication from the client.  If only because some clients may want to be able to suppress generation of such messages for whatever reason.
> 
> So I think my preferred option is a Progress token in the Connection header.  That way I don't have to try and dream up something useful in terms of parameters for a potential Progress request header.  It also means clients that aren't written (e.g. existing clients) to process these messages won't receive them for discarding and waste bandwidth (esp over slower high-cost circuits such as GPRS / G3 etc).

Sending an additional 10 to 30 bytes on every single request,
in the critical path, just in case the browser might access
such a resource (a one in a billion chance, at best), is an
extremely poor trade-off.

If you want this feature to be deployable, then don't rely on
negotiation.  It isn't going to happen.  Use the user-agent.

....Roy

---

From: Henrik Nordström <[email-masked]>
Date: Fri, 28 May 2010 00:55:50 +0200

> because it hasn't been tested properly before.

Yes, but those can be handled by blacklisting based on User-Agent.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 28 May 2010 11:40:41 +1200

> extremely poor trade-off.

actually if you're behind a corporate firewall that does this, it's 
closer to a 100% chance depending on scanning rules for the firewall 
AV.  When we deploy this with WinGate and AV, that will then be millions 
of browsers (and other HTTP agents) seeing these 103s for pretty much 
all requests if we don't negotiate.  I've tested Chrome, FF and IE8, but 
I haven't tested things like windows update, BITS service etc etc etc.  
There's no way to guarantee what won't break.

I was of the impression switching behaviour based on User-Agent was 
frowned upon.  It creates at best a maintenance issue for admins 
(keeping the list of UAs current).

I agree it will be a long time before origin servers generate them.  So 
probably it makes no sense for now for a proxy to forward the Progress 
token upstream, unless it's talking to another proxy.  So actually I 
don't see many of these going over the wire onto the net. The main 
benefit is for a client talking to a local proxy.  RFC2616-compliant 
proxies that don't understand Progress will already strip it.

I don't know of any browser that doesn't already include a Connection 
header in every request already for keep-alives (even through default 
semantics for 1.1 is to keep alive so it's largely redundant), so 
overhead in practise will only be an extra 10 bytes.

Also, clients could make some choices about likelihood of benefit before 
adding Progress to the Connection header.  E.g. for retrieving images, 
JS, CSS etc it could be turned off.

Or behaviour could differ for proxy connections vs direct and be 
configurable in the browser.

Otherwise if we don't advertise support, and it's on by default, we need 
to resort to things like administrative settings, e.g. admin can turn it 
off/on per client IP or UA or Content-Type or something.  That loads up 
burden on the admin.

As for Henrik's point about benefit for non-supporting clients, where 
these messages will stop connection timeouts.  I can understand a client 
may consider something is happening, but if there is no progress 
indicated to the user, we are back at square one (where we are now), 
where the user hits retry a few times then calls his tech support or 
admin.  So I don't think the benefit will be there for browsers.  Other 
agents certainly may benefit.

Maybe the answer to that is a positive exception mechanism.  E.g. proxy 
sends progress to anyone that asks for it, plus a list of extras that 
didn't.

Adrien

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 27 May 2010 17:21:06 -0700

On May 27, 2010, at 4:40 PM, Adrien de Croy wrote:
> actually if you're behind a corporate firewall that does this, it's closer to a 100% chance depending on scanning rules for the firewall AV.  When we deploy this with WinGate and AV, that will then be millions of browsers (and other HTTP agents) seeing these 103s for pretty much all requests if we don't negotiate.  I've tested Chrome, FF and IE8, but I haven't tested things like windows update, BITS service etc etc etc.  There's no way to guarantee what won't break.

Again, that is a very tiny percentage of agents when compared to the
rest of the Internet traffic and needs to be crossed with an even
smaller percentage of resources that would benefit from a progress feature.

I see no reason for such a small instance of potential benefit to
impose a constant cost.  Services like Windows Update can be easily
bypassed (why you would want to scan such a service is beyond me,
since it often delivers virus signatures inside virus-checking
client tools).

> I was of the impression switching behaviour based on User-Agent was frowned upon.  It creates at best a maintenance issue for admins (keeping the list of UAs current).

You don't have to keep a list of current agents.  You have to keep
a list of broken agents that do not properly handle HTTP/1.1 status
codes, and you can make that configurable.  Apache does that kind of
thing all the time.  It might even be the empty list, for all we know.

> I agree it will be a long time before origin servers generate them.  So probably it makes no sense for now for a proxy to forward the Progress token upstream, unless it's talking to another proxy.  So actually I don't see many of these going over the wire onto the net. The main benefit is for a client talking to a local proxy.  RFC2616-compliant proxies that don't understand Progress will already strip it.

So you would only have UAs send it when configured to use a proxy?

> I don't know of any browser that doesn't already include a Connection header in every request already for keep-alives (even through default semantics for 1.1 is to keep alive so it's largely redundant), so overhead in practise will only be an extra 10 bytes.
> 
> Also, clients could make some choices about likelihood of benefit before adding Progress to the Connection header.  E.g. for retrieving images, JS, CSS etc it could be turned off.

Anything that relies on the client guessing the nature of a resource
is not a good plan.  Images, for example, are often generated, converted,
or cropped on the fly.

> Or behaviour could differ for proxy connections vs direct and be configurable in the browser.
> 
> Otherwise if we don't advertise support, and it's on by default, we need to resort to things like administrative settings, e.g. admin can turn it off/on per client IP or UA or Content-Type or something.  That loads up burden on the admin.

The number of scan-proxy-maintaining admins or long-resource-processing
owners is far smaller than the groups effected by the other options.
The cost should be placed on those benefitting most from the feature.

> As for Henrik's point about benefit for non-supporting clients, where these messages will stop connection timeouts.  I can understand a client may consider something is happening, but if there is no progress indicated to the user, we are back at square one (where we are now), where the user hits retry a few times then calls his tech support or admin.  So I don't think the benefit will be there for browsers.  Other agents certainly may benefit.

Or you could just send a 202 response if the software anticipates
a lengthy delay.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 28 May 2010 12:49:45 +1200

On 28/05/2010 12:21 p.m., Roy T. Fielding wrote:
>> AV. When we deploy this with WinGate and AV, that will then be 
>> millions of browsers (and other HTTP agents) seeing these 103s for 
>> pretty much all requests if we don't negotiate. I've tested Chrome, 
>> FF and IE8, but I haven't tested things like windows update, BITS 
>> service etc etc etc. There's no way to guarantee what won't break.

OK fair enough.

> since it often delivers virus signatures inside virus-checking

OK also fair.  Actually default config for many AV filtering products 
excludes many known update sites for this reason.

> thing all the time.  It might even be the empty list, for all we know.

that's what I meant.  Depending on how it works, it's either a whitelist 
or blacklist of UAs.  Hard to predict at this stage whether a white or 
black list would be more work to maintain.

> So you would only have UAs send it when configured to use a proxy?

that's an option although not my personal preference.

> is not a good plan.  Images, for example, are often generated, converted,

Agreed.  I guess the point was that the client can still choose.  If 
it's on always, then the client can't turn it off if it may need to 
(unless we add a No-Progress token).  Why a client may wish to turn it 
off I can't imagine yet, but that doesn't mean there will never be a 
need to.

> The cost should be placed on those benefitting most from the feature.

The problem with that for me, is that there's a high correlation between 
those people and my customer-base...

> Or you could just send a 202 response if the software anticipates

How does a browser recover from that to finally obtain the entity?  
RFC2616 is extremely vague on 202.  Is that where you'd post some sort 
of progress page?  Progress pages work great for downloads, but not so 
great for embedded content, and completely useless for non-browser UAs.

Does that BTW mean that the entity returned with a 202 is not the entity 
requested (c.f. recent long-running discussion on how to determine what 
entity a response carries).

Actually I like the idea of just sending the progress messages always, 
for the reason that it will potentially provide a little more incentive 
for UA authors to implement it.

The downside is that since there's no indication from the client of 
support for it, the proxy can't then tell whether it should send 
progress, or revert to previous mechanisms for keeping UAs and their 
human happy (e.g. drip-feeding part of the entity through).  We get a 
fair number of support calls from clients complaining about download 
timeouts and retries due to scanning without dripfeeding.  In fact 
getting rid of the abomination of drip-feeding is my main incentive for 
promoting this.

Adrien