Subject: http progress notification

From: Adrien de Croy <[email-masked]>
Date: Tue, 25 May 2010 16:15:53 +1200

We've made some more great progress with testing on this, which has lead 
to a fairly major change in the structure of information passed back by 
servers.

We have a Google Chrome client available for testing, and an 
experimental WinGate proxy as well using Kaspersky AV for WinGate to 
provide the updates during spooling for scanning.

Test results are very promising, and in my view provide clear benefits 
and incentives.

In a nutshell, the changes to the Internet Draft are as follows:

1. Client advertisement.

This is now done by way of adding a token to the Connection or 
Proxy-Connection request header - e.g.

Connection: Keep-Alive, Progress

2. Server / intermediary responses.

The current experimental implementation returns status codes of 102.  
The data returned is in a list of name value pairs.
Currently we have defined the following:

"state"    represents the current state, defined values are: GENERATING, 
RECEIVING, SENDING or PROCESSING (no quotes, case sensitive)
"agent"   a quoted string representing the subject of the current state
"bytes"    a number of bytes (either sent or received) which makes sense 
mostly for RECEIVING and SENDING.  Omit if not relevant for the state
"size"      the size of the entity in bytes (if known). Omit if not known.
"percent" percent complete of any stage

as an example, downloading and scanning a hypothetical 10k file (over a 
very slow link) produces something like:

HTTP/1.1 102 OK
Progress: state=RECEIVING,agent="WinGate",bytes=1024,size=10240

HTTP/1.1 102 OK
Progress: bytes=2048

HTTP/1.1 102 OK
Progress: bytes=4096

HTTP/1.1 102 OK
Progress: bytes=8192

HTTP/1.1 102 OK
Progress: bytes=10240

HTTP/1.1 102 OK
Progress: state=PROCESSING,agent="Kaspersky AV",percent=0

HTTP/1.1 102 OK
Progress: percent=10

HTTP/1.1 102 OK
Progress: percent=100

HTTP/1.1 200 OK
Content-Length: 10240
Content-Type: etc.....

We only send new values for a parameter if it changes, e.g. if the state 
changes, or agent changes, or bytes changes.  I've omitted Date headers 
in the example above.

All fields except state are optional.  What this means is that at least 
the first interim response must specify a state, but all other fields 
are optional.  It is valid simply to indicate state and changes in state.

We considered sending all progress data each time, but this redundant 
information just causes extra bandwidth consumption which I considered 
hard to justify purely to avoid storage of state in the client.  Our 
current implementation sends on any state change or data size downloaded 
change (max 2/sec).  It sends bytes for downloads, and percents for 
scanning.

The main reasons we adopted this new approach (c.f. the I-D) were:

a) it's extensible.
* We can add more data as required (e.g. entity content-type perhaps to 
enable a client to display a "save as" dialog)
* we can add more states as required, although these will require some 
form of registry.  I'd propose we give some thought to states we wish to 
be able to distinguish and get a set from the start that won't need 
extending for a long time if ever.
b) dealing with unknown lengths is simpler than the original proposal - 
just omit the size field.
c) it's closer in structure to existing headers so parsing should be easier.

I'd propose that anyone wanting to contribute to this let me know, I can 
also make available software for testing.  Then after a week or so I'll 
update the I-D.

Regards

Adrien

---

From: Henrik Nordström <[email-masked]>
Date: Tue, 25 May 2010 10:28:18 +0200

> to a fairly major change in the structure of information passed back by 

Looks good to me. Some comments

Maybe add an optional Progress request header, allowing client to
indicate for example how frequent it desires to get progress updates.
But probably overkill as the clients needing this most also is low
bandwidth clients and adding a header like this for those is
counterproductive..

I also thought about perhaps an explicit note to not respond with 102 if
the request was HTTP/1.0. But this should not be needed today. As far as
I know all HTTP/1.0 proxies deployed today support the Connection
header.

"OK" is maybe not the most appropriate status code description, and
confusingly similar to 200.. "Progress" or "Info" perhaps?

Something which may require some additional thought is how to handle
chains of proxies using Progress. For example Client <-> Transforming
proxy <-> AV Proxy <-> Server. How should an intermediary forward
progress information when it is itself also buffering and processing the
response?

And another use case to cover for progress notifications is when
intermediaries buffer the request before forwarding causing delays in
request forwarding which in effect looks the same as delays in request.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 25 May 2010 22:38:07 +1200

> bandwidth clients and adding a header like this for those is

Yes, the original I-D proposed this, but in the end I figured the most 
common use case would be where the proxy was well connected to the 
client, and the complexity it would add in the server was hardly worth it.

There are also applications for a server to use this directly to a 
client (e.g. when there's a lot of processing to do before sending a 
response, e.g. webmail).

> I know all HTTP/1.0 proxies deployed today support the Connection

we would only use this if we received a Progress token in the Connection 
header.  Do you mean some HTTP/1.0 proxies may pass the Connection 
header on verbatim so pass it on without understanding it?

I think even HTTP/1.0 required intermediaries and clients to support 
multiple 1xx responses, so this would probably even just work.

> confusingly similar to 200.. "Progress" or "Info" perhaps?

sure

> chains of proxies using Progress. For example Client<->  Transforming
> proxy<->  AV Proxy<->  Server. How should an intermediary forward

Since the interim stage only occurs prior to the actual response (e.g. 
200 ok), it's not possible to be buffering an entity at the time you are 
receiving 1xx messages from upstream.

Therefore the only agent generating 1xx updates would be the agent 
currently doing the spooling.  This means filters inside a gateway can 
individually own the task of generating these.

Any downstream agent seeing 1xx messages going past doesn't really have 
anything meaningful to add, since they are just waiting for upstream.

> request forwarding which in effect looks the same as delays in request.

yes, this is an interesting use case, and one I think will also benefit 
greatly.

Uploading a large file where you scan at a gateway typically has 
problems where the file is spooled to the gateway extremely quickly, 
then scanned and slowly then uploaded to the target server before the 
server responds and that response is relayed back to the client.

This means the user experience is a very quick upload followed by a very 
long delay before a result.  Highly confusing for users.

Progress notifications in this scenario would therefore be also very 
useful so that the client could see the progress of the scan and upload 
to the server.

Actually the upload scenario lends itself to one more state: 
CONNECTING.  This would allow the proxy to buffer the entire request and 
scan it before even connecting to the origin server.

The best thing about interim responses (compared to drip-feeding part of 
the entity - the most common solution to the problem) is that the proxy 
can respond with say a 403 if it finds a virus, and the client hasn't 
received any (possibly compromised) entity body, yet has been able to 
display progress as if it were directly connected.

One issue we are seeing.   Browsers don't know how to treat the response 
until they sees Content-Type.  This means they don't know if it's a file 
download (display save as dialog, or similar) vs just some part of a 
page or other resource to display in-line.  Chrome for instance will 
abort non-download requests when you navigate to another page, whereas 
downloads continue.

So we've discussed here possibly sending the Content-Type (and possibly 
other headers) through to the client as well, so it can know whether 
it's downloading something or not, and behave accordingly.   I can't yet 
think of any requirement for other headers but we could perhaps define a 
way to transport the headers back without making them actual headers of 
the 1xx response, e.g. prefix header names with X-Orig- or encapsulate 
them somehow into the Progress response header.  If it were just 
Content-Type, we'd just put that through as type="..."

Have to also be careful about escaping and charsets for quoted text.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Henrik Nordström <[email-masked]>
Date: Tue, 25 May 2010 16:32:33 +0200

> header on verbatim so pass it on without understanding it?

HTTP/1.0 did not specify Connection, but that was 1997. Connection have
been around for 15 years now, and most proxies that announced themselves
as HTTP/1.0 in 2001 did implement Connection.

> multiple 1xx responses, so this would probably even just work.

1xx does not exists in HTTP/1.0. Reserved for future use. 

> receiving 1xx messages from upstream.

What I was thinking about was more in line of how each step along the
path announces that it will use Progress. It's not very intuitive to the
end-user if progress indication goes up to 100% and then suddenly drops
down to 0% again when the next starts processing the response without a
prior indication that there is more steps to be taken.

> think of any requirement for other headers

Content-Disposition perhaps?

> but we could perhaps define a way to transport the headers back without
> making them actual headers of 

Should be fine to send pretty much any header in 1xx responses, but
better avoid transport headers such as an Content-Length or
Transfer-Encoding

> Have to also be careful about escaping and charsets for quoted text.

Agreed, but best if free text is avoided if possible. A token registry
for states may make sense, and application names should use the same
product bnf as used for Server.

If there is need for a free text response message then I think it's
better to add another Progress-Message header for that, defined as TEXT.
Trying to get implementations to properly quote free text tends to
always fail in a significant number of implementations.

Regards
Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 May 2010 18:12:28 +0200

On 25.05.2010 06:15, Adrien de Croy wrote:
> ...

You really really should choose a free status code; 102 has been defined 
in RFC 2518.

Best regards, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 26 May 2010 10:13:44 +1200

On 26/05/2010 4:12 a.m., Julian Reschke wrote:
> You really really should choose a free status code; 102 has been 
> defined in RFC 2518.
>

there was discussion on the code a couple of years ago.

At the time it was discussed that the existing webdav 102 status could 
be an appropriate code - in other words that the Progress response 
header could be an application for it.

Personally I've no preference for the code - if you think it's 
justifiable to register another one for Progress, I'm happy to use a 
different one.

Regards

Adrien

---

From: Elias Sinderson <[email-masked]>
Date: Tue, 25 May 2010 15:37:47 -0700

Adrien de Croy wrote:
> [...] it was discussed that the existing webdav 102 status could be an 
> appropriate code - in other words that the Progress response header 
> could be an application for it.

'102 Processing' seems like a very good fit to me (but not 102 OK / 
Progress / Info), provided that you extend the existing definition in a 
compatible way.

Thanks and Regards,
Elias

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 26 May 2010 11:09:31 +1200

> as HTTP/1.0 in 2001 did implement Connection.

OK.  This is quite a difficult issue then, since there are

a) proxies that forward the version of the request even if they aren't 
HTTP/1.1 compliant
b) proxies that don't strip headers based on Connection header tokens 
nor alter the Connection header but advertise themselves as HTTP/1.1 
compliant (e.g. only support Keep-alive)
c) proxies that don't add via headers.

So it will be possible for a server to receive an HTTP/1.1 message with 
Connection: Progress in it, even though the sender of the message 
doesn't understand progress.

I don't think we can realistically specify around this.

However excluding Progress from responses to HTTP/1.0 messages does seem 
prudent.

Also, given the amount of time it takes to roll out things like this, I 
imagine the incidents of proxies behaving like this in say another 10 
years will be low.

> 1xx does not exists in HTTP/1.0. Reserved for future use.

OK

> prior indication that there is more steps to be taken.

OK, that's the reason we added the agent parameter.  If the response 
entity is going through several stages each of which buffer the entire 
entity for processing, there will in fact be several stages of 
spooling.  Each spooling stage can distinguish itself from any other by 
its agent value.

In general though, this will only be a potential issue if several 
chained proxies are poorly connected, otherwise the transfer between 
will be very fast, and the user won't probably notice the updates.

> Content-Disposition perhaps?
I wondered about that.  It's not a common header, and I don't know of 
any browsers that use it to make a handling decision instead of 
Content-Type.

> better avoid transport headers such as an Content-Length or

OK, that could be quite good, then a downstream proxy can make policy 
decision based on content type etc as early as possible.

So it's not in violation to send entity headers when there can be no 
entity (in a 1xx response message)?

> product bnf as used for Server.

I thought about that too, but the target for the text in the agent value 
(only currently proposed text field) is a human, and so I thought it 
would be preferable not to include things like version numbers etc etc.

So the intention was just a simple label.  More like a realm for auth.  
Something the human will hopefully recognise (or learn to).

> always fail in a significant number of implementations.

OK, that's a good idea.

Thanks

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 26 May 2010 11:28:34 +1200

On 26/05/2010 10:37 a.m., Elias Sinderson wrote:
>> [...] it was discussed that the existing webdav 102 status could be 
>> an appropriate code - in other words that the Progress response 
> Progress / Info), provided that you extend the existing definition in 
> a compatible way.
>

I was under the impression that the reason phrase wasn't interpreted as 
part of the protocol.  It would only be seen by people watching packet 
captures or debug logs.

for instance from RFC2616 s 6.1.1

para 1

The Reason-Phrase is intended to give a short
    textual description of the Status-Code. The Status-Code is intended
    for use by automata and the Reason-Phrase is intended for the human
    user. The client is not required to examine or display the Reason-
    Phrase.

and para 3

The reason phrases listed here are only
    recommendations -- they MAY be replaced by local equivalents without
    affecting the protocol.

which I read to mean you can change the text to anything you like 
without affecting the protocol.

Is this not the case?

Regards

Adrien

---

From: Henrik Nordström <[email-masked]>
Date: Wed, 26 May 2010 01:32:37 +0200

ons 2010-05-26 klockan 11:28 +1200 skrev Adrien de Croy:

> I was under the impression that the reason phrase wasn't interpreted
> as part of the protocol.  It would only be seen by people watching
> packet captures or debug logs.

Correct. Machines MUST NOT base any decisions on the reason phrase.
That's purely intended for diagnostic purposes by humans.

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Tue, 25 May 2010 16:35:59 -0700 (PDT)

On Wed, 26 May 2010, Adrien de Croy wrote:

> I was under the impression that the reason phrase wasn't interpreted as part
> of the protocol.  It would only be seen by people watching packet captures or
> debug logs.
> 
> which I read to mean you can change the text to anything you like without
> affecting the protocol.

But reusing a code for more than one status value just makes life
difficult for those of us who need to examine wire level data.
It helps analysis to have meaningful phrases, in particular for status
codes which may not be familiar.

---

From: Henrik Nordström <[email-masked]>
Date: Wed, 26 May 2010 01:36:46 +0200

> a) proxies that forward the version of the request even if they aren't 

Broken proxy. Not even allowed by HTTP/1.0.

> compliant (e.g. only support Keep-alive)

Broken proxy. HTTP/1.1 is very clear on Connection.

> c) proxies that don't add via headers.

Also broken.

> doesn't understand progress.

It's possible for a server to receive anything.

It's not realistic to try to work around every broken piece of software
there is out there.

The case I am talking about is purely mismatch between HTTP/1.0 and 1.1.
Not broken software.

> However excluding Progress from responses to HTTP/1.0 messages does seem 

Personally I am not sure it needs to be mentioned for the reasons given
before. The amount of proxies in use today and not supporting Connection
SHOULD be minimal.

> imagine the incidents of proxies behaving like this in say another 10 

I argue that it's already low, and should be close to distinct in
another 10 years.

> any browsers that use it to make a handling decision instead of 

Every browser I tried..

> entity (in a 1xx response message)?

I do not think so. Especially not with this being a hop-by-hop
negotiated feature.

> would be preferable not to include things like version numbers etc etc.

I disagree, for the exact same reason those are in Server. The agent
name is mostly for diagnostic purposes, not to be human friendly.

> Something the human will hopefully recognise (or learn to).

which is not an agent description, but more like a state description.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 26 May 2010 12:11:57 +1200

> before. The amount of proxies in use today and not supporting Connection

OK fine by me.

>> any browsers that use it to make a handling decision instead of

OK, I'll have another look.  I think I didn't consider the case where 
both are present.  So Content-Disposition typically overrides handling 
which otherwise is dependent on Content-Type?

In that case we definitely need to transport that header through as 
well.  This leads us to requirements (since there are potentially many 
1xx response messages) about which 1xx message(s) should contain these 
headers.  I'd propose that only one of them should.  It may not always 
be possible for this to be the first interim response (e.g. if there is 
a 1xx sent back when the proxy is connecting upstream), but it's 
possible from the stage where the agent doing the processing receives a 
final result from upstream.

> I do not think so. Especially not with this being a hop-by-hop

OK, in which case I'm tending to side with Julian in that we should use 
another status than 102, since there is more non-standard (in terms of 
the existing 102 requirements) processing potentially required by clients.

> name is mostly for diagnostic purposes, not to be human friendly.

Our Chromium implementation actually builds the status string displayed 
to the user using the agent value.  Perhaps "agent" isn't the best name 
for this parameter (subject is the linguistic equivalent).  It's not 
like the User-Agent header.

So the user sees status messages in the status bar like

WinGate is downloading (1.2MB of 2.7MB)
WinGate is downloading (1.4MB of 2.7MB)
WinGate is downloading (1.6MB of 2.7MB)
etc
Kaspersky AV for WinGate is processing (10%)

Most people surfing through a corporate firewall at some stage become 
aware of its existence and its product name.  For instance error pages 
(cannot connect to server etc) commonly have branding on them from the 
firewall/proxy vendor.  Block pages from AV have their branding.

So there is value in using recognisable names in status updates.

the added benefit of using the "product" name here, is that product 
names are less likely to require localisation.

If we need more information for forensic purposes, the thing generating 
the 1xx response could add a complete Server header.

Regards

Adrien

---

From: Julian Reschke <[email-masked]>
Date: Wed, 26 May 2010 08:58:00 +0200

> Progress / Info), provided that you extend the existing definition in a

That's the important part: if you can make it an extension, go ahead 
(but take over the definition in the IANA registry), otherwise pick a 
new one.

Best regards, Julian

---

From: Henrik Nordström <[email-masked]>
Date: Wed, 26 May 2010 19:56:47 +0200

> Progress / Info), provided that you extend the existing definition in a 

Does it?

The concept of the two is similar, but:

The definition of 102 have another header attached which this proposal
do not use at all.

the definition of 102 Processing is based on being authorative by the
origin server with wordings such as " has accepted the complete request,
but has not yet completed it". Both clauses there may be invalid in this
proposal as the progress indication is not necessarily related at all to
the server as such, in fact most times not at all..

Regards
Henrik